vLLM

vLLM is a fast, easy-to-use library for serving large language models. It was originally developed in the Sky Computing Lab at UC Berkeley and is now one of the most active open-source AI projects, maintained by a broad community.

vLLM delivers high serving throughput through PagedAttention memory management, continuous batching, chunked prefill and prefix caching. It supports many quantization formats (FP8, INT4, GPTQ, AWQ, GGUF and more), optimised attention kernels, speculative decoding, and disaggregated prefill and decode.

vLLM is the standard engine teams use to serve open models in production at high throughput. It is Apache-licensed.

Stars

0

Forks

0

Commits

0

Repository

vllm-project/vllm

• OpenAI APIOpenAI's hosted API for its models.
• Together AIHosted inference for open models.
• LocalAINo. 200Open-source AI engine with OpenAI-compatible APIs.