vLLM

Summary

vLLM is an open-source library for LLM inference and serving, and it has no licensing cost stated in the documentation. It functions as a library that integrates with popular Hugging Face models and is intended for developers working with LLMs. The project has a history of being built and maintained by a diverse community of dozens of academic institutions and companies.

A high-throughput and memory-efficient inference and serving engine for LLMs

Command line

Values are placeholders taken from each option’s declared default. Nothing is executed here — the output shown is a recording of a run that already happened.

Release history

1. v0.28.0 Aug 26, 2026 · issue 008

   vLLM v0.28.0 adds disk KV offloading, gRPC multimodal inference, new API fields, and major model/hardware expansions across NVIDIA, AMD, and Intel.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.28.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.28.0

   • ›Adds module_path config key for out-of-tree secondary KV cache tier managers, enabling custom offloading backends without forking vLLM.
   • ›Adds count_reasoning_tokens field to the streaming parser engine for tracking reasoning token counts.
   • ›Adds content_parts field on /inference/v1/generate for richer multimodal generation requests.
   • ›Adds request priority parsed from an HTTP header, enabling priority-based request scheduling via standard HTTP.
   • ›Adds session ID plumbing into requests for stateful session tracking.
   +53 moreshow less

   • ›Adds CuMemAllocator.discard() for tag-selective GPU memory release in RL workflows.
   • ›Adds thinking_token_budget support in Model Runner V2 for controlling reasoning token budgets.
   • ›Adds disk offloading support for SimpleCPUOffloadConnector, extending tiered KV cache to persistent storage.
   • ›Adds tiering offloading metrics for observability into KV cache tier behavior.
   • ›Adds a canonical CPU layout for parallelism-agnostic KV cache offload.
   • ›Raises default max_num_batched_tokens from 8192 to 16384.
   • ›Raises the Blackwell CUDA graph capture default to 1024.
   • ›Enables prefix caching by default for Mamba models.
   • ›Adds Rust frontend standalone renderer with gRPC multimodal image inference support.
   • ›Adds explicit data-parallel rank routing and RL lifecycle control to the Rust/gRPC frontend.
   • ›Publishes protobuf schemas for the gRPC frontend to Buf.
   • ›Adds Model Runner V2 E/P/D (encode/prefill/decode) disaggregation support.
   • ›Adds encoder CUDA graphs in Model Runner V2.
   • ›Adds decoder token-wise pooling and Transformers pooling models in Model Runner V2.
   • ›Adds attention-free model support in Model Runner V2.
   • ›Adds multi-layer MTP KV cache support in Model Runner V2.
   • ›Adds weight offloading in Model Runner V2.
   • ›Adds FlashInfer XQA decode support on SM12x (Blackwell).
   • ›Adds a CuTeDSL fused query kernel on SM100.
   • ›Adds an MLA backend enabling DeepSeek-V2/V3 inference on CPU.
   • ›Adds GPTQ and AWQ quantization support on s390x.
   • ›Enables AITER and FP8 inference on GFX120x (AMD ROCm).
   • ›Adds a triton-cpu wheel for CPU-based Triton kernels.
   • ›Adds an XPU wheel to the release pipeline.
   • ›Adds online MXFP4 quantization support.
   • ›Adds batch-invariant NVFP4 MoE via CUTLASS.
   • ›Adds block-wise scaled_mm kernel for quantization.
   • ›Adds AMD Quark NVFP4 support with an emulation kernel for DeepSeek-V4.
   • ›Adds Mooncake store group semantics and tenant ID support, with official wheels in the Docker image.
   • ›Adds file:// rendezvous for single-node executors to eliminate startup port races.
   • ›Adds a CI GPU<->CPU sync check to guard against accidental synchronization regressions on the execution path.
   • ›Adds JIT warmup infrastructure with predicate filtering.
   • ›Adds sparse MLA end-to-end support for plain decode, MTP, and DSpark speculative decoding in DeepSeek V4.
   • ›Adds DFlash2 speculative decoding with local convolution and a candidate selector.
   • ›Adds DSpark confidence-scheduled verification for speculative decoding.
   • ›Adds async scheduling auto-enabled for draft models in speculative decoding.
   • ›Adds Decode Context Parallel (DCP) support for Kimi-K3.
   • ›Adds optional shared-expert sharding for Kimi-K3, saving ~17 GiB of memory per GPU.
   • ›Adds vision tower LoRA support for Gemma4, Keye, and Ultravox models.
   • ›Adds native multimodal support for Dots3 NOTE.
   • ›Adds new models: Muse Glimmer, Ling 3.0 Flash (BF16/MTP/FP8/MXFP4), and Interns2mobius.
   • ›Adds Transformers modeling backend with MLA support, hardware-agnostic model definition, logit softcapping, and generalized input embedding handling.
   • ›Adds EAGLE3 speculative decoding support on KimiLinear.
   • ›Adds NUL byte rejection in structured_outputs.regex and negative token ID rejection for structured output robustness.
   • ›Adds VLLMValidationError raised from structured output validators.
   • ›Adds dynamic tools from developer messages in the Rust frontend.
   • ›Adds output token IDs logged at DEBUG level.
   • ›Adds vLLM Recipes integration with native config-based deployment and benchmarking.
   • ›Adds a minimal Triton Proton profiling backend.
   • ›Adds partial secondary-tier KV load results support, allowing partial cache hits from offload storage.
   • ›Adds data-parallel topology exposure to offloading backends.
   • ›Adds stateful trainer send over NCCL and sparse NCCL for RL workflows.
   • ›Adds model as optional on all /derender request classes.
   └──▷ BREAKING ON UPGRADE

   • !bitsandbytes support has been migrated to an out-of-tree plugin; existing setups using bitsandbytes quantization will break without installing the plugin separately.
   • !Transformers dependency bumped to 5.15.0; models or configurations incompatible with Transformers 5.15.0 will break.
   • !The deprecated calculate_kv_scales runtime KV scale calculation has been removed; any configuration or code referencing it will fail.
   • !override_attention_dtype has been removed; any configuration or code using this option will fail.
2. v0.27.1 Aug 11, 2026 · issue 002

   vLLM v0.27.1 adds quantization support for DSpark Markov heads, enabling W4A16 and other quantized weight configurations.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.27.1 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.27.1

   • ›Adds quant_config support to DSparkMarkovHead, forwarding quantization configuration to its ParallelLMHead-based markov_w2 projection, including W4A16 configurations with weight_scale_2.
3. v0.27.0 Aug 10, 2026 · issue -009

   vLLM v0.27.0 adds Kimi K3, Qwen3.5, FlashAttention 4 FP8/headdim-256 on SM100, fault-tolerance for DP+EP, gRPC control plane, and sm_107 Rubin support.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.27.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.27.0

   └──▷ TRY IT
   Run a Triton tensor-descriptor fused MoE kernel path for MoE models to improve kernel performance.
   copy

   $ VLLM_TRITON_USE_TD=1 vllm serve <model>

   • ›Adds --linear-backend flag honored for ModelOpt W4A16 quantization.
   • ›Adds VLLM_TRITON_USE_TD environment variable to enable the Triton tensor-descriptor path for fused MoE kernels.
   • ›Adds sample_from_anchor loaded from speculators config for speculative decoding.
   • ›Adds has_pending_push_work to encoder-cache connectors for distributed disaggregated serving.
   • ›Adds vllm-bench integrated into the vllm CLI for benchmarking.
   +43 moreshow less

   • ›New model support for Kimi K3 with full stack: core model, Python and Rust frontends, AttnRes kernels, DeepGEMM, DSpark AR fusion, and optional shared-expert sharding.
   • ›New model support for Qwen3.5 text-only dense and MoE models with EVS video token pruning.
   • ›New model support for K-EXAONE-2.0-750B-A37B.
   • ›New model support for VaultGemma via the Transformers modeling backend.
   • ›New model support for jina-embeddings-v5-text-nano with EuroBERT encoder backbone.
   • ›Deepens FlashAttention 4 integration on SM100 with FP8 KV cache support and headdim-256 support.
   • ›Adds new JIT warmup infrastructure and runner-owned Triton kernel warmup before the first request, removing first-request compilation stalls.
   • ›Adds fault tolerance framework for DP+EP external load-balancer deployments.
   • ›Adds async preparation for elastic EP scaling.
   • ›Adds NIXL P/D disaggregation support for hybrid MLA+SSM models.
   • ›Adds NIXL heterogeneous P/D block sizes for hybrid models.
   • ›Adds MoRIIO heterogeneous TP<->DP prefill/decode read routing.
   • ›Adds gRPC control plane to the Rust frontend with engine-aware health reporting, abort control, server and model discovery, and KV event source discovery.
   • ›Adds sm_107 target for NVIDIA Rubin with NVLink all-reduce paths on SM107.
   • ›Enables ROCm gfx1250 architecture support.
   • ›Expands Model Runner V2 to non-generative workloads: encoder-only attention, sequence pooling for embedding/classification, encoder token classification, encoder token embedding, and BGE-M3 pooling.
   • ›Adds multi-layer MTP speculator to Model Runner V2.
   • ›Adds multimodal inference on CPU via Model Runner V2.
   • ›Adds per-request KV offload tier filtering via TierFilter/TierMatcher and self-describing KV events via TieringOffloadingSpec.
   • ›Adds pluggable eviction policies via CachePolicyFactory for KV offloading.
   • ›Adds CPUOffloadingSpec with single-copy MLA layout and migration onto SharedOffloadRegion.
   • ›Adds generic P2P secondary tier with peer lookup and serving for KV offloading.
   • ›Adds FP4 Qutlass integration for compressed-tensors quantization.
   • ›Adds CuTeDSL MoE for ReLU2 NVFP4 quantization.
   • ›Adds MXFP8 linear support in INC quantization.
   • ›Adds AutoRound W4A16 MoE and MXFP4 linear/MoE on XPU.
   • ›Adds KV quant mode for TurboQuant.
   • ›Adds ModelOpt FP8 emulation on SM80.
   • ›Adds compressed-tensors checkpoint support for DeepSeek-V4 and Kimi-K3.
   • ›Adds top_k and top_p sampling for DiffusionGemma diffusion models.
   • ›Adds weight version tagging for RL rollouts.
   • ›Adds INT8 fused MoE kernel for Arm CPUs.
   • ›Adds s390x inference optimization with oneDNN INT8 GEMM.
   • ›Adds XPU QK Norm + RoPE fusion pass.
   • ›Adds XPU FP8 o_proj with fp8_bmm and load-time scale transpose.
   • ›Adds sequence parallelism for DeepSeek-V4 serving.
   • ›Adds ReplaySSM caching for faster Mamba2 standard decode.
   • ›Adds FlashInfer Mamba SSU algorithm selection.
   • ›Adds FlexAttention epilogue hook.
   • ›Adds audio model support to the Transformers modeling backend.
   • ›Adds fused residual-add + RMSNorm compilation pass in the Transformers modeling backend.
   • ›Upgrades to PyTorch 2.13.0 with torchvision 0.28.0 and Triton 3.7.1.
   • ›Adds grammar advancement across the reasoning boundary with speculative decode for structured outputs.
   └──▷ BREAKING ON UPGRADE

   • !PyTorch upgraded to 2.13.0 (along with torchvision 0.28.0 and Triton 3.7.1), which is a breaking environment change requiring environment rebuild for all platforms including XPU and CPU.
   • !FusedMoE is renamed to FusedMoEFactory in the MoE refactor.
4. v0.27.0 Aug 10, 2026 · issue 002

   vLLM v0.27.0 adds Kimi K3, Qwen3.5, FP4FP4A 4-bit floating-point numeric format used to represent model weights and activations in machine-learning inference, trading arithmetic precision for dramatically reduced memory footprint and faster computation on hardware that supports it./FP8 quantization, gRPC control plane, fault tolerance, and PyTorch 2.13.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.27.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.27.0

   └──▷ TRY IT
   Enable the Triton tensor-descriptor path for fused MoE to improve MoE kernel throughput on supported hardware.
   copy

   $ VLLM_TRITON_USE_TD=1 vllm serve deepseek-ai/DeepSeek-V3

   • ›Adds VLLM_TRITON_USE_TD environment variable to enable the Triton tensor-descriptor path for fused MoE kernels.
   • ›Adds --linear-backend flag, now honored for ModelOpt W4A16 quantization.
   • ›Adds sample_from_anchor loaded from the speculators config for speculative decoding control.
   • ›Adds has_pending_push_work to encoder-cache connectors for disaggregated serving.
   • ›Adds vllm-bench integrated into the vllm CLI for benchmarking.
   +44 moreshow less

   • ›Adds sm_107 compilation target for NVIDIA Rubin GPU architecture, with NVLink all-reduce paths on SM107.
   • ›Supports Kimi K3 model with full stack: core model files, specialized CUDA/Triton AttnRes kernels, Python and Rust frontends, DeepGEMM, DSpark AR fusion, and compressed-tensors quantized checkpoints.
   • ›Supports Qwen3.5 text-only dense and MoE models with EVS video token pruning.
   • ›Supports K-EXAONE-2.0-750B-A37B and jina-embeddings-v5-text-nano (EuroBERT encoder backbone) models.
   • ›Supports VaultGemma via the Transformers modeling backend.
   • ›Supports top_k and top_p sampling for DiffusionGemma diffusion models.
   • ›Supports longer max audio duration for MOSS-TD multimodal model.
   • ›Adds FP4 Qutlass integration for compressed-tensors quantization.
   • ›Adds CuTeDSL MoE kernel for ReLU2 NVFP4 quantization.
   • ›Adds MXFP8 linear support in INC (Intel Neural Compressor).
   • ›Adds AutoRound W4A16 MoE and MXFP4 linear/MoE on XPU.
   • ›Adds KV quant mode for TurboQuant.
   • ›Adds ModelOpt FP8 emulation support on SM80 GPUs.
   • ›Adds compressed-tensors support for DeepSeek-V4 and Kimi-K3 checkpoints.
   • ›Adds FlashAttention 4 FP8 KV cache support and headdim-256 support on SM100 (NVIDIA Blackwell).
   • ›Adds new JIT warmup infrastructure and runner-owned Triton kernel warmup to eliminate first-request compilation stalls.
   • ›Adds sequence parallelism for DeepSeek-V4 workloads.
   • ›Adds compact MXFP4 indexer KV cache for DeepSeek-V4.
   • ›Adds CachePolicyFactory interface for pluggable and externally-defined KV cache eviction policies.
   • ›Adds generic P2P secondary KV offload tier with peer lookup and serving.
   • ›Adds per-request KV tier filtering via TierFilter/TierMatcher.
   • ›Adds self-describing KV events via TieringOffloadingSpec.
   • ›Adds ReplaySSM caching for faster Mamba2 standard decode.
   • ›Adds FlashInfer Mamba SSU algorithm selection.
   • ›Adds multi-layer MTP speculator support in Model Runner V2.
   • ›Adds encoder-only attention, sequence pooling, encoder token classification and embedding, and BGE-M3 pooling to Model Runner V2, expanding it to non-generative workloads.
   • ›Adds multimodal inference on CPU via Model Runner V2.
   • ›Adds a simplified fault tolerance framework for DP+EP external load-balancer deployments.
   • ›Adds async preparation for elastic EP (expert parallelism) scaling.
   • ›Adds NIXL P/D disaggregation support for hybrid MLA+SSM models.
   • ›Adds heterogeneous P/D block sizes for hybrid models in NIXL.
   • ›Adds MoRIIO heterogeneous TP<->DP prefill/decode read routing.
   • ›Adds gRPC control plane to the Rust frontend: engine-aware health reporting, abort control, server and model discovery, and KV event source discovery.
   • ›Adds weight version tagging for RL rollouts.
   • ›Adds grammar advancement across the reasoning boundary with speculative decode for structured outputs.
   • ›Adds ROCm gfx1250 architecture support.
   • ›Adds AITER FP8 ViT encoder attention on ROCm.
   • ›Adds INT8 fused MoE kernel for Arm CPUs.
   • ›Adds s390x inference optimization via oneDNN INT8 GEMM.
   • ›Adds XPU QK Norm + RoPE fusion pass and FP8 o_proj with fp8_bmm.
   • ›Adds audio model support to the Transformers modeling backend.
   • ›Adds FlexAttention epilogue hook for custom attention post-processing.
   • ›Adds FusedMoEFactory (renamed from FusedMoE) as part of a MoE kernel refactor.
   • ›Upgrades to PyTorch 2.13.0, torchvision 0.28.0, and Triton 3.7.1.
   └──▷ BREAKING ON UPGRADE

   • !PyTorch upgraded from a prior version to 2.13.0 (with torchvision 0.28.0 and Triton 3.7.1); existing environments must be rebuilt — this is described as a breaking environment change.
   • !FusedMoE is renamed to FusedMoEFactory; any code referencing FusedMoE directly will break.
5. v0.26.0 Jul 27, 2026 · issue -023

   vLLM v0.26.0 adds the Inkling model family, head_dtype fp32 lm_head, per-KV-cache-group attention backends, KV offloading metrics, and major API/frontend expansions.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.26.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.26.0

   └──▷ TRY IT
   Filter completions against a banned-word list using the new bad_words field on the OpenAI-compatible completions endpoint.
   copy

   $ curl http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{"model": "meta-llama/Llama-3-8B", "prompt": "Tell me about", "bad_words": ["violence", "hate"]}'

   • ›Adds head_dtype config to enable fp32 lm_head for generation models, extended to the LoRA path and given a ROCm torch.mm fast path.
   • ›Adds kv_cache_dtype field inside speculative_config to set a separate KV-cache dtype for speculative decoding.
   • ›Adds kv_cache_dtype_skip_layers support for MLA quantization.
   • ›Adds blocks_per_chunk config key for heterogeneous KV groups in KV offloading.
   • ›Adds bad_words parameter to /v1/completions for OpenAI-compatible completions.
   +46 moreshow less

   • ›Adds logprob_token_ids field exposed on Python OpenAI endpoints.
   • ›Adds include_reasoning parameter for non-Harmony models on OpenAI-compatible endpoints.
   • ›Adds num_cache_creation_tokens field populated on Messages responses.
   • ›Adds /abort_requests endpoint on the RLHF dev API router.
   • ›Adds continue_final_message handling with renderer sentinel in the Rust frontend.
   • ›Adds endpoint plugins framework for extending API routing.
   • ›Introduces the Inkling model family with full support: base modeling, piecewise CUDA graph, Hopper FA4 relative attention, MTP=1 speculative decoding, LoRA, and ModelOpt NVFP4 quantization.
   • ›Adds support for selecting a different attention backend per KV-cache group, enabling better hybrid-model support.
   • ›Makes sliding-window an explicit backend capability for per-backend declaration.
   • ›Adds partial prefix-cache hit for hybrid models and selective hybrid cache retention.
   • ›Reports prefix-cache-reused blocks in full report mode.
   • ›Adds KV offloading metrics: basic offloading gauges, CPU cache read/write split gauges, and tiering-lookup-delay sync/async histograms.
   • ›Adds object-store secondary KV tier with workload identity support and DP-replica-aware tiering.
   • ›Adds runtime draft weight update for speculative decoding.
   • ›Adds hybrid (SWA + full attention) DFlash drafters for speculative decoding.
   • ›Adds nvfp4_per_token online MoE quantization.
   • ›Adds Humming w[2-7]a[4,8] weight-only inference with compressed-tensors.
   • ›Adds int4 quantization for the emulation MoE backend.
   • ›Adds INT2 XPU weight-only quant linear.
   • ›Adds FlashInfer MoE LoRA for BF16 models.
   • ›Adds LoRA support for tower/connector in LlavaNextVideo.
   • ›Adds opt-in persistence and reuse of the memory-profiling result across boots.
   • ›Adds Rust frontend support for multimodal video and audio inputs.
   • ›Adds Seed-OSS tool parser to the Rust frontend.
   • ›Adds native vllm-bench port in the Rust frontend.
   • ›Adds Deepstream video decoding backend.
   • ›Adds overlap of preprocessing and computation for pooling models in offline inference.
   • ›Adds new models: BertForMaskedLM, RobertaForTokenClassification, XLMRobertaForTokenClassification, LongCat-Flash-Lite n-gram embedding, Cosmos3 Edge Reasoner, Cosmos3-Super, and TranslateGemma-12b-it.
   • ›Migrates Olmo/Olmo2, MistralLarge3, and HunyuanVL to the Transformers modeling backend (Transformers 5.13.0).
   • ›Adds automatic fallback to ViT data parallelism when TP is unavailable for multimodal models.
   • ›Adds NIXL pipeline-parallel prefill in push mode for PD disaggregation.
   • ›Adds Decode Context Parallel (DCP) hybrid attention support and DCP + Eagle for Tokenspeed MLA backends.
   • ›Adds native macOS arm64 CPU wheel builds.
   • ›Adds s390x NUMA topology support for CPU inference.
   • ›Adds POWER VSX math function optimization and IBM Power docker builds using prebuilt wheels.
   • ›Adds CuTeDSL compilation progress bar for UX feedback during compilation.
   • ›Adds human-readable integer formatting for more CLI arguments.
   • ›Adds expanded GPU profiler config scope and annotations.
   • ›Adds worker exit-code logging when a process dies unexpectedly.
   • ›Replaces diskcache with a non-pickle alternative to eliminate pickle deserialization risks.
   • ›Adds resource-bounds validation to derender endpoints and sanitizes server file paths from validation error responses.
   • ›Adds grammar compilation failure handling that avoids crashing the engine.
   • ›Adds timeout guard on lm-format-enforcer regex compilation.
   • ›Adds STATEFUL trainer-send abstractions for RLHF.
   • ›Adds DSpark speculative decoding for DSv4 on AMD ROCm and XPU.
   • ›Adds DFlash speculative decoding for GDN models on CPU.
   └──▷ BREAKING ON UPGRADE

   • !Models TeleChat, Persimmon, and Fuyu have been removed and are no longer supported.
6. v0.26.0 Jul 27, 2026 · issue 002

   vLLM v0.26.0 adds the Inkling model family, fp32 head_dtype, per-group attention backends, KV offloading metrics, and expanded OpenAI API fields.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.26.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.26.0

   └──▷ TRY IT
   Serve a hybrid Mamba/attention model with partial prefix-cache hits for faster multi-turn inference.
   copy

   $ vllm serve Qwen/Qwen3.5-35B-A3B-FP8 --enable-prefix-caching --mamba-cache-mode align --prefix-match-unit 16

   Use a separate KV cache dtype for the speculative draft model while keeping the main model at a different precision.
   copy

   $ vllm serve <model> --speculative-config '{"model": "<draft_model>", "kv_cache_dtype": "fp8"}' --kv-cache-dtype auto

   • ›Adds head_dtype config option to enable fp32 lm_head for generation models, improving accuracy for generation heads; extended to the LoRA path and given a ROCm torch.mm fast path.
   • ›Adds kv_cache_dtype per speculative_config so the draft model's KV cache dtype can be set independently from the main model.
   • ›Adds kv_cache_dtype_skip_layers support for MLA quantization.
   • ›Adds blocks_per_chunk config key for heterogeneous KV groups in KV offloading.
   • ›Adds prefix_match_unit option to select prefix-match granularity independently from physical KV-cache block sizes, enabling partial prefix-cache hits for hybrid (full-attention + Mamba) models.
   +42 moreshow less

   • ›Adds --mamba-cache-mode align and --prefix-match-unit CLI flags for fine-grained prefix-cache hits on hybrid models.
   • ›Exposes logprob_token_ids on Python OpenAI endpoints.
   • ›Adds bad_words parameter support to /v1/completions.
   • ›Adds include_reasoning parameter for non-Harmony models on OpenAI-compatible endpoints.
   • ›Populates num_cache_creation_tokens field on Messages (chat completions) responses.
   • ›Adds /abort_requests endpoint to the RLHF dev API router.
   • ›Adds an endpoint plugins framework for extending API routing.
   • ›Supports runtime draft weight update for speculative decoding, allowing RL training loops to update draft model weights without restart.
   • ›Adds P2P default host/port environment variables for KV offloading peer-to-peer configuration.
   • ›Adds KV offloading metrics: basic offloading gauges, split CPU cache usage into read/write gauges, and tiering-lookup-delay split into sync/async histograms.
   • ›Adds object-store secondary tier with workload identity for KV tiered storage.
   • ›Adds DP-replica-aware tiering for KV offloading.
   • ›Adds EC (encoder-cache) transfer params and a CPU-offloading EC connector.
   • ›Adds nvfp4_per_token online MoE quantization support.
   • ›Adds Humming w[2-7]a[4,8] weight-only inference with compressed-tensors.
   • ›Adds INT2 XPU weight-only quantization linear support.
   • ›Adds per-KV-cache-group attention backend selection, enabling hybrid models with mixed attention backends.
   • ›Makes sliding-window support an explicit backend capability, improving hybrid model support.
   • ›Adds DCP (Decode Context Parallel) hybrid attention support and DCP + Eagle for Tokenspeed MLA backends.
   • ›Adds NIXL pipeline-parallel prefill in push mode for PD disaggregation.
   • ›Adds Rust frontend support for multimodal video and audio, a Seed-OSS tool parser, and a native vllm-bench port.
   • ›Adds continue_final_message handling with renderer sentinel in the Rust frontend.
   • ›Adds Deepstream video decoding backend.
   • ›Adds human-readable integer formatting for more CLI arguments.
   • ›Adds CuTeDSL compilation progress bar.
   • ›Adds expanded GPU profiler config scope and annotations.
   • ›Adds worker exit code logging when a process dies unexpectedly.
   • ›Adds native macOS arm64 CPU wheel builds.
   • ›Adds DFlash speculative decoding for GDN models on CPU.
   • ›Adds new model support: Inkling family (base, piecewise CUDA graphs, Hopper FA4 relative attention, MTP=1 speculative decoding, LoRA, NVFP4), BertForMaskedLM, RobertaForTokenClassification, XLMRobertaForTokenClassification, LongCat-Flash-Lite n-gram embedding, Cosmos3 Edge Reasoner, Cosmos3-Super, TranslateGemma-12b-it.
   • ›Migrates Olmo/Olmo2, MistralLarge3, and HunyuanVL to the Transformers modeling backend (Transformers 5.13.0).
   • ›Adds FlashInfer MoE LoRA for BF16 models.
   • ›Adds LoRA support for tower/connector in LlavaNextVideo.
   • ›Adds automatic fallback to ViT data parallelism when TP is unavailable for multimodal models.
   • ›Adds STATEFUL trainer-send abstractions for RLHF.
   • ›Replaces diskcache to eliminate pickle deserialization in caching layer.
   • ›Adds resource-bounds validation to derender endpoints and bounds the completion prompt list to prevent unbounded engine fan-out.
   • ›Adds prefix-cache-reused blocks reporting in full report mode.
   • ›Adds selective hybrid cache retention (Marconi caching) for hybrid models.
   • ›Adds opt-in persistence and reuse of the memory-profiling result across boots.
   • ›Adds TRTLLM BF16 MoE modular kernel.
   • ›Adds XPU nightly/release image publishing.
   └──▷ BREAKING ON UPGRADE

   • !The VLLM_TRITON_ATTN_USE_TD environment variable is renamed to VLLM_TRITON_USE_TD.
   • !Models TeleChat, Persimmon, and Fuyu are removed and no longer supported.
7. v0.25.0 Jul 11, 2026 · issue -039

   vLLM v0.25.0 adds a Streaming Parser Engine, static HTTPS/mTLS on the Rust frontend, universal speculative decoding, and a raft of new models and API fields.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.25.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.25.0

   └──▷ TRY IT
   Turn on GPU sync checking during development to surface synchronization issues early.
   copy

   $ VLLM_GPU_SYNC_CHECK=1 vllm serve <model-id>

   • ›Adds VLLM_GPU_SYNC_CHECK environment variable for GPU synchronization checking.
   • ›Adds return_loss_mask field to Chat/Completions responses for training-data generation.
   • ›Adds model_class_overrides CLI option for development and debugging.
   • ›Adds repetition_detection sampling parameter in the Rust frontend.
   • ›Adds kv_transfer_params merging via MultiConnector for PD disaggregation.
   +51 moreshow less

   • ›Adds FLASH_ATTN_MLA_SPARSE environment variable to select the Hopper sparse-MLA attention backend.
   • ›Adds ROCM_AITER_FA backend flag to enable the AITER FlashAttention MLA prefill path on ROCm.
   • ›Adds static HTTPS and mTLS support for HTTP and gRPC in the Rust frontend.
   • ›Adds per-request timing metrics field on Chat/Completions responses.
   • ›Adds token offsets on render endpoints.
   • ›Adds profiler control routes in the Rust frontend.
   • ›Adds a DP supervisor via the Rust frontend.
   • ›Introduces a new Streaming Parser Engine — a unified tool-call and reasoning parsing framework — with a new Kimi k2.5/k2.6/k2.7 parser and ports of seed_oss and DeepSeek V4 parsers.
   • ›Adds universal speculative decoding for heterogeneous vocabularies (TLI), enabling spec decode across models with mismatched vocabularies.
   • ›Adds DSpark and DFlash speculative decoding drafters, including a Laguna XS.2.1 drafter.
   • ›Adds tok_sparse_select configuration for MiniMax-M3, replacing Triton kernels.
   • ›Adds LookupResult enum to the KV offloading subsystem.
   • ›Adds TorchCodec as a video decoding backend.
   • ›Adds TTFT and TPS metrics printing in vllm chat.
   • ›Adds usage field exposed for disaggregated serving.
   • ›Adds a dedicated Harmony renderer for gpt-oss / Harmony serving.
   • ›Adds a secondary-tier implementation for PD disaggregation.
   • ›Adds 2/3/5/6/7-bit pack-quantized weight-only inference (Humming).
   • ›Adds Triton INT4 per-token-head KV cache quantization.
   • ›Adds NVFP4 KV cache with skip-layers sliding window support.
   • ›Adds Marlin thread-tile padding extended to MoE (WNA16 + FP8/MXFP8) and int8 grouped WNA16 MoE.
   • ›Adds FlashInfer MXINT4 MoE support for gated SiLU.
   • ›Adds W8A8 FP8 linear kernel with multi-granularity quantization for Intel XPU.
   • ›Adds AITER custom all-reduce in CudaCommunicator for ROCm.
   • ›Adds INT3 quantization for quickreduce on ROCm.
   • ›Adds EPLB support for Quark OCP MXFP4 MoE on ROCm.
   • ›Adds accelerator-agnostic get_memory_info platform API and a C++ implementation for Intel XPU.
   • ›Adds RVV path for W4A8 INT4 GEMM on RISC-V, BF16 support on VLEN=256 hardware, and reduced LMUL pressure in INT4 LUT dequant.
   • ›Adds fp16 support on PowerPC.
   • ›Adds new models: LLaVA-OneVision-2, Unlimited OCR (with Triton R-SWA backend), MOSS-Transcribe-Diarize, openai/privacy-filter, and Hy3 (with token-suffix and JSON Schema array support).
   • ›Adds GLM-5 / DeepSeek-V3.2 to the model zoo and pipeline parallelism for MiniMax-M3.
   • ›Model Runner V2 is now the default execution path for all dense models, adding support for EVS, realtime embeddings, Mamba hybrid prefix caching, multimodal-prefix bidirectional attention, and dynamic speculative decoding with full CUDA graphs.
   • ›Adds sequence parallelism without requiring data parallelism, yielding 1.9–5.0% end-to-end throughput improvement.
   • ›Adds NCCL symmetric memory support extended to AllGather and ReduceScatter operations.
   • ›Adds a fault-tolerance backend to detect all-to-all peer faults and prevent corrupted output.
   • ›Adds Mooncake connector support for GDN (Qwen3.5) and MLA (DeepSeek-V4-Flash) in PD disaggregation.
   • ›Adds NIXL Mamba1 support for PD disaggregation.
   • ›Adds fused_indexer_q_rope_quant Triton kernel for GLM-5.2/DeepSeek, providing 1.9–3.3% end-to-end throughput gain.
   • ›Adds token_to_req_indices cache for DSv4, achieving 5–6x kernel speedup.
   • ›Adds Responses API namespace tools for OpenAI compatibility.
   • ›Adds HTTP 422 response for unprocessable image URLs.
   • ›Adds image decompression-bomb OOM denial-of-service prevention.
   • ›Adds pluggable sleep-mode backend abstraction with communicator-agnostic capability flags.
   • ›Adds blocking CUDA events to avoid busy-polling the driver lock.
   • ›Adds VRAM semaphore infrastructure.
   • ›Adds FlashInfer fused all-reduce tuned for world_size=16 on GB300 (Blackwell).
   • ›Adds CuTeDSL/FA4-MLA warmup infrastructure and a B12x backend for non-gated MoEs on Blackwell.
   • ›Adds Helion fused_qk_norm_rope and silu_and_mul_per_block_quant kernels.
   • ›Adds accelerated unquantized MoE for AArch64 CPUs.
   • ›Adds compressed-tensor w8a8 int8 MoE and Mamba ShortConv support on CPU.
   • ›Adds chunked prefill and prefix caching for Qwen3.5 on CPU.
   └──▷ BREAKING ON UPGRADE

   • !PagedAttention has been deleted; deployments relying on the legacy attention implementation must migrate to the V1/MRv2 backend.
   • !Models removed: Baichuan, Aquila, Grok, Tarsier/Tarsier2, AyaVision/MusicFlamingo, and Mantis are no longer supported.
   • !gptq_marlin has been removed from supported ROCm quantization schemes.
   • !Legacy api_server.py has been moved to the examples directory and is no longer the primary server entrypoint.
   • !The old FP8 online MoE quantization class is deprecated (removed in a future release).
8. v0.24.0 Jun 29, 2026 · issue -051

   vLLM v0.24.0 adds MiniMax-M3, DiffusionGemma, a Streaming Parser Engine, Rust frontend endpoints, and removes internal CUDA_VISIBLE_DEVICES management.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.24.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.24.0

   └──▷ TRY IT
   Skip Triton autotuning at startup to reduce cold-start time in latency-sensitive deployments.
   copy

   $ VLLM_TRITON_FORCE_FIRST_CONFIG=1 vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 4

   Pause the Rust frontend to drain in-flight requests during a rolling restart, then resume once the new instance is ready.
   copy

   $ curl -X POST http://localhost:8000/pause && sleep 5 && curl -X POST http://localhost:8000/resume

   • ›Adds device_ids argument to replace internal CUDA_VISIBLE_DEVICES management; vLLM no longer sets CUDA_VISIBLE_DEVICES internally.
   • ›Adds VLLM_TRITON_FORCE_FIRST_CONFIG environment variable to skip Triton autotuning and cut startup overhead.
   • ›Rust frontend gains API-key authentication, CORS, /tokenize, /detokenize, /pause, /resume, /is_paused, /abort_requests, /get_world_size, thinking_token_budget, and parallel_tool_calls=false.
   • ›Adds real /v1/embeddings support for messages with chat_template_kwargs.
   • ›Adds cache-usage reporting to the Anthropic-compatible /v1/messages endpoint.
   +29 moreshow less

   • ›Adds multimodal token counts inside usage.prompt_tokens_details in API responses.
   • ›Adds strict mode for tool calling in Chat Completions API and Responses API.
   • ›Introduces a new Streaming Parser Engine that unifies tool-call and reasoning parsing, with parsers for Qwen3, MiniMax-M2, GLM-4.7/5.1/5.2, and Nemotron V3.
   • ›Adds support for new models: MiniMax-M3, DiffusionGemma (including a CPU path), Hierarchical Reasoning Model (HrmTextForCausalLM), and OpenMOSS.
   • ›Model Runner V2 now supports quantized models by default and enables GraniteMoE by default.
   • ›Adds Dynamic Speculative Decoding and EAGLE3 support for Qwen3 targets.
   • ›Adds online FP8 per-token-per-channel (PTPC) quantization.
   • ›Extends modelopt_mixed quantization support to Ampere (SM80-86) and Turing (SM75) GPUs.
   • ›Allows fp8_e5m2 KV cache for non-FP8 checkpoints.
   • ›Adds flashinfer_cutlass as a clamped NVFP4 MoE backend.
   • ›Integrates DeepEP v2 for expert parallelism.
   • ›Adds KV push from prefill to decode via NIXL for disaggregated serving.
   • ›Adds a KV-cache watermark to reduce preemptions.
   • ›Adds a Marconi-style admission policy for hybrid KV cache.
   • ›Adds structured-output guardrails for diffusion decoders.
   • ›Adds structured outputs support for beam search.
   • ›Adds Qwen3-VL video loader and Qwen2-VL/Qwen2.5-VL processor-mapped video loader.
   • ›Adds ViT full CUDA graph support for GLM-4.1V, DeepSeek-OCR, Kimi-VL, mllama4, and Lfm2VL encoder.
   • ›Adds fastsafetensors ParallelLoader for faster weight loading.
   • ›Adds 2.5x faster ASR CPU preprocessing via multi-threading.
   • ›Adds cgroup memory-limit-aware KV cache sizing on CPU.
   • ›Adds RISC-V oneDNN W8A8 INT8 support and RVV micro-GEMM for WNA16.
   • ›Migrates GGUF quantization to a plugin.
   • ›Adds Intel XPU sequence-parallel support and W4A16 int4 group_size=32 MoE.
   • ›Adds SM90 CUTLASS FP8 odd-M support via swap_ab (180-290% kernel speedup).
   • ›Adds FlashInfer cutedsl NVFP4 GEMM backend and cute-dsl MXFP8 linear kernel.
   • ›Adds Helion kernels for FP8/RMSNorm quantization.
   • ›Completes migration of all kernels to the libtorch stable ABI.
   • ›Deprecation window begun for CUDA_VISIBLE_DEVICES on ROCm.
   └──▷ BREAKING ON UPGRADE

   • !ROCm: CUDA_VISIBLE_DEVICES support has entered a deprecation window and will be removed in a future release.
   • !P2pNcclConnector has been removed; configurations using it must migrate to an alternative KV connector.
9. v0.23.0 Jun 15, 2026 · issue -065

   vLLM v0.23.0 adds a maturing Rust frontend, multi-tier KV offloading, unified parser, and broad new model support across backends.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.23.0 https://github.com/vllm-project/vllm.git
   # already have the repo? check out this version:
   $ git checkout v0.23.0

   └──▷ TRY IT
   Enable request-ID headers in the Rust frontend so downstream systems can correlate responses to originating requests.
   copy

   $ vllm serve <model> --enable-request-id-headers

   • ›Rust frontend adds a streaming generate endpoint, dynamic LoRA endpoints, /version and /server_info endpoints, a server-router extension hook, and --enable-request-id-headers flag.
   • ›Adds per-request KV cache offloading policy via the on_new_request lifecycle hook and on_schedule_end() hook, plus an object-store secondary offloading tier.
   • ›Introduces scheduler_block_size threaded into KVCacheManager/Coordinator and max_concurrent_batches moved to VllmConfig.
   • ›Adds pluggable KVCacheSpec for configurable KV cache specifications.
   • ›Unifies reasoning and tool-call parsing behind a single Parser.parse() interface, with the Responses parser migrated to it.
   +35 moreshow less

   • ›Adds system_fingerprint field to the OpenAI/Responses API.
   • ›Adds chat_template_kwargs support in the Responses API.
   • ›Supports streaming tool/function calling with required tool choice in the OpenAI-compatible API.
   • ›Adds tool_choice="none" honored in streaming responses.
   • ›Adds Anthropic Messages API support for structured output, effort, and system-role messages inside the messages array.
   • ›Model Runner V2 (MRv2) is now the default for Llama and Mistral dense models, gaining a FlashInfer sampler, breakable CUDA graphs, and pipeline-parallel bubble elimination.
   • ›Adds new tool parsers for InternLM2, hy_v3, Phi-4-mini, and Gemma4 in the Rust frontend.
   • ›Adds new tool parsers: MiniCPM5 XML, Qwen3 XML JSON-args-first, and DeepSeek DSML incremental streaming.
   • ›Adds support for new models: Step-3.7-Flash, Cosmos3 Reasoner, Gemma 4 Unified (encoder-free), JetBrains Mellum v2, Granite Speech Plus, and Cohere Mini Code.
   • ›Adds Gemma 4 MTP and encoder-free Unified support.
   • ›Adds LMCacheMPConnector for LMCache in disaggregated serving.
   • ›Enables async EPLB by default and adds EPLB support for DeepSeek-V4 Mega-MoE.
   • ›Adds SSL support for the data-parallel supervisor.
   • ›Enables Triton MoE backend on Hopper by default.
   • ›Adds native W4A16 and fused-MoE W4A16 HIP kernels for RDNA3 (gfx1100) on AMD ROCm.
   • ›Adds AITER top-k/top-p sampler by default on AMD ROCm.
   • ›Adds Intel XPU support for CPU/tiering offloading, block_fp8_moe, block-scaled W8A8 FP8 path, and transparent sleep mode.
   • ›Adds zentorch-accelerated W8A8/W4A16 quantization on AMD Zen CPUs.
   • ›Adds RISC-V RVV WNA16 helpers and PowerPC SHM communicator.
   • ›Adds arm64 CI image.
   • ›Adds ModelOpt LM-head quantization and MXFP8 non-gated MoE support.
   • ›Adds compressed-tensors support for WNA8O8Int linears, WNInt embeddings, and asymmetric MoE WNA16 Marlin.
   • ›Adds Triton W4A16 as a CUDA fallback for non-Marlin-aligned shapes.
   • ›Adds thinking_token_budget validation in the frontend.
   • ›Adds NUMA auto-binding on DGX B300.
   • ›Adds extra_repr() for pooler classes in pooling/classification models.
   • ›Adds auto-selection of registered video loader for VLMs.
   • ›Adds Nixl zero-copy EPLB transfers and Nixl Mamba prefix-caching mode.
   • ›Adds Transformers v5 compatibility, including vendored MiniCPM-V/O processors and Voxtral fetch_audio for transformers≥5.10.
   • ›Adds DeepSeek-V4 TRTLLM-gen attention kernel, EPLB support, selective prefix-cache retention for sliding-window KV cache, and index-share feature for DSA MTP.
   • ›Adds sparse NCCL weight transfer for in-place model updates.
   • ›Adds per-GPU-worker RDMA NIC selection for data-parallel deployments.
   • ›Upgrades TPU inference to v0.21.0.
   • ›Adds CUTLASS FP8 scaled-mm padding bypass delivering ~20% performance improvement.
   • ›Adds MoE-permute buffer pre-allocation delivering 9–14% performance improvement.
   └──▷ BREAKING ON UPGRADE

   • !NixlConnector kv_both role has entered a deprecation cycle and should be migrated away from.
   • !supports_expert_map has been removed from the MoE modular-kernel oracle.
   • !The inplace fused-experts mechanism has been removed from the MoE oracle.
   • !cpu_awq is folded into awq_marlin; configurations referencing cpu_awq must be updated to use awq_marlin.
10. v0.22.1 Jun 5, 2026 · issue -075

    vLLM v0.22.1 adds Mellum v2 model support and zentorch-accelerated quantized inference on AMD Zen CPUs.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.22.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.22.1

    • ›Adds W8A8 (int8 dynamic-symmetric) and W4A16 (GPTQ) linear inference through zentorch kernels on AMD Zen CPUs, registered ahead of generic oneDNN CPU kernels, with transparent fallback on non-Zen CPUs, GPUs, and XPU.
    • ›Adds support for JetBrains' Mellum v2, an open-weights Mixture-of-Experts code-generation model.
11. v0.22.0 May 29, 2026 · issue -082

    vLLM v0.22.0 adds a Rust frontend, multi-tier KV cache offloading, batch-invariant FP8/NVFP4 paths, and broad new model/hardware support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.22.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.22.0

    └──▷ TRY IT
    Limit reasoning token budget per request to control cost when using thinking-capable models.
    copy

    $ curl http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{"model": "deepseek-ai/DeepSeek-R1", "prompt": "Explain quantum entanglement.", "thinking_token_budget": 512}'

    • ›Adds --cpu-distributed-timeout-seconds CLI flag for CPU-distributed deployments.
    • ›Adds thinking_token_budget parameter support to the Completions API.
    • ›Adds reasoning_effort mapping to enable_thinking in the Completions API.
    • ›Adds chat_template_kwargs support to the Responses API.
    • ›Adds reset_cache() method to the KV offloading framework.
    +53 moreshow less

    • ›Adds update_config support to Model Runner V2.
    • ›Adds logprob_token_ids field support in Model Runner V2.
    • ›Adds X-data-parallel-rank header forwarding for data-parallel serving.
    • ›Adds API-key authorization for /v2 endpoints.
    • ›Adds AuthenticationMiddleware path extraction simplification and utility call-ID newtype to the Rust frontend.
    • ›Adds PoolingOfflineMixin split for the pooling offline API.
    • ›Adds QuantKey with activation override, reworking quantization_config in the quantization framework.
    • ›Adds ExpertMapManager as part of the MoE refactor, with experts moved to the experts/ directory and a RoutedExperts alias for FusedMoE.
    • ›Adds truncation-side control for OpenAI-compatible endpoints.
    • ›Adds reasoning_content → reasoning normalization in the frontend.
    • ›Adds consolidated Speech-to-Text entrypoints.
    • ›Adds beam-search consolidation via BeamSearchMixin.
    • ›Adds score/rerank chat-template instructions to the frontend.
    • ›Introduces experimental Rust frontend integration with in-tree code, a DP Supervisor for data-parallel serving.
    • ›Introduces multi-tier KV cache offloading framework with a Python filesystem secondary tier, DSv4 support, and Mooncake disk offloading via MooncakeStoreConnector.
    • ›Adds DeepSeek V4 model package reorganized into dedicated vllm/models/deepseek_v4/ package with NVFP4 fused MoE, full and piecewise CUDA graph, and MTP speculative decoding.
    • ›Model Runner V2 (MRv2) is now default for Qwen3 dense models, with automatic fallback to MRv1 for unsupported features.
    • ›Adds batch-invariant inference with Cutlass FP8 support for a 28.9% end-to-end latency improvement and an NVFP4 Cutlass linear path.
    • ›Adds CutlassFP8 padding pre-processing yielding a 13.5% TTFT improvement.
    • ›Adds padded NVFP4 quantization kernel yielding 2.4–5.7% end-to-end improvement.
    • ›Adds fused RoPE+KVCache+q_concat kernel for MLA decoding.
    • ›Adds FULL CUDA graph capture for TRITON_MLA decode.
    • ›Adds breakable CUDA graph (experimental).
    • ›Adds RISC-V Vector Extension (RVV) optimized attention kernels with VLEN=256 support.
    • ›Adds fused GDN for AMX CPU and experimental Triton + MRv2 on CPU.
    • ›Adds MXFP4 linear layers with compressed-tensors integration.
    • ›Adds MXFP4 CPU W4A16 MoE and XPU mxfp8 MoE support.
    • ›Adds AutoRound W4A16 quantization support.
    • ›Adds ability to load Quark NVFP4 checkpoints.
    • ›Adds ModelOpt Qwen3.5/3.6 VLM quantized prefix mapping.
    • ›Adds GPTQ consolidation, migrating gptq_marlin → auto_gptq.
    • ›Adds support for new model architectures: MiniCPM-V 4.6, InternS2 Preview, OpenVLA, and EXAONE-4.5.
    • ›Adds custom callable proposer backend for speculative decoding.
    • ›Adds post-norm EAGLE-3 and peagle speculative decoding backends.
    • ›Adds non-MTP speculation support for NemotronH.
    • ›Adds Apertus tool parser.
    • ›Adds anyOf/oneOf/$ref resolution for Qwen3Coder tool calling.
    • ›Adds one-shot Triton kernel for MoE LoRA and simultaneous 2D & 3D MoE LoRA adapters.
    • ›Adds per-tensor FP8 CUTLASS on SM12.1 and FlashInfer b12x MoE + FP4 GEMM for SM120/121 (NVIDIA Blackwell).
    • ›Adds RMSNorm+Quant fusion for AMD gfx950 and XGMI backend for MoRI connector.
    • ›Adds Intel XPU GPTQ int4 support and FP8 block-scaled quantization.
    • ›Adds head_dim=512 for FlashInfer TRTLLM attention.
    • ›Adds manual cumem allocator enable and stream-aware free callback to the allocator.
    • ›Adds KV cache event metadata emission.
    • ›Adds penalties Triton kernel.
    • ›Adds Mamba2 SSD kernel warmup, bf16 SSM cache, and GPU-side state postprocessing fused kernel.
    • ›Adds lease-renewal TTL for KV blocks and handshake-failure policy for disaggregated serving (NIXL).
    • ›Adds Mooncake operation metrics and HMA support for DSV4.
    • ›Adds DP Supervisor and publishes request counts at engine-step start for data-parallel serving.
    • ›Adds non-root vllm-openai Docker target.
    • ›Adds setuptools-rust dependency for Rust frontend builds.
    • ›Adds cohere MoE and pipeline parallelism for Cohere vision models.
    • ›Adds runai-streamer weight loading for Qwen3.5/MTP/Qwen3-VL.
    └──▷ BREAKING ON UPGRADE

    • !Removed old locations of get_tokenizer and resolve_hf_chat_template; callers must update their import paths.
    • !Removed deprecated MLA prefill arguments; existing code using those arguments will break.
    • !Marked environment variables now covered by --moe-backend / --linear-backend as deprecated.
12. v0.21.0 May 15, 2026 · issue -096

    vLLM v0.21.0 adds Blackwell TOKENSPEED_MLA backend, KV offload HMA integration, thinking-budget spec decode, and new RLHF/Responses API endpoints.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.21.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.21.0

    └──▷ TRY IT
    Skip model name validation at startup when serving a fine-tuned or renamed model that fails the default name check.
    copy

    $ VLLM_SKIP_MODEL_NAME_VALIDATION=1 vllm serve ./my-finetuned-model --host 0.0.0.0 --port 8000

    • ›Adds VLLM_SKIP_MODEL_NAME_VALIDATION environment variable to bypass model name validation at startup.
    • ›Adds explicit /start_weight_update and /finish_weight_update API endpoints for RLHF weight-update workflows.
    • ›Adds system_fingerprint field to OpenAI-compatible chat completion responses.
    • ›Adds prompt_embeds content part support to OpenAI-compatible API.
    • ›Adds defer_loading and tool_reference support to OpenAI-compatible API.
    +40 moreshow less

    • ›Adds logprob_token_ids support in Model Runner V2.
    • ›Adds configurable safetensors checkpoint prefetch.
    • ›Adds max reasoning effort support for DeepSeek V4.
    • ›Adds new TOKENSPEED_MLA attention backend for DeepSeek-R1 and Kimi-K25 prefill and decode on NVIDIA Blackwell GPUs.
    • ›Adds MooncakeStoreConnector for distributed KV offloading.
    • ›Adds PluggableLayer interface enabling out-of-tree MoE runners.
    • ›Integrates KV offloading subsystem with the Hybrid Memory Allocator (HMA), including scheduler-side sliding window group support and multi-connector HMA.
    • ›Adds bi-directional KV cache transfers between prefill (P) and decode (D) nodes in disaggregated serving.
    • ›Adds Mooncake KVConnectorStats for KV transfer observability in disaggregated serving.
    • ›Enables independent drafter attention backend selection for speculative decoding.
    • ›Adds EAGLE speculative decoding support for Mistral, Gemma4 MTP, MiMo-V2.5 MTP, and Cohere Eagle.
    • ›Adds tool calling support via XGrammar 0.2.0 with structural tags for strict tool calling combined with reasoning.
    • ›Adds Cohere reasoning and tool parsers, plus LFM2/2.5 tool parser.
    • ›Adds Responses API streaming tool/function calling with required and named tool/function choice, plus resubmitting output items with missing fields.
    • ›Adds rendered prompt text in chat completion response.
    • ›Adds Fastokens support in the tokenizer.
    • ›Adds RayExecutorV2 enabled by default for Ray-based execution.
    • ›Adds IndexCache support for DSA models.
    • ›Enables FlashInfer top-k/top-p sampler by default.
    • ›Adds OOM prevention via max_split_size_mb during model loading.
    • ›Adds two-phase pause mechanism to prevent scheduler deadlock.
    • ›Adds NIXL connector bumped to 1.x for disaggregated serving.
    • ›Adds new model architecture support: MiMo-V2.5, Laguna XS.2, Moondream3, Qianfan-OCR, Cohere MoE, and Cohere Eagle.
    • ›Adds AMD ROCm support for DeepSeek V4, including TP4 AITER MLA and pipeline parallelism.
    • ›Adds AMD Dynamic Batch Optimization (DBO) on ROCm.
    • ›Adds CPU FP8 attention for AMX/AVX-512, FP8 W8A16 linear, FP8 W8A16 MoE, and DNNL AVX2 W8A8 Int8.
    • ›Adds Intel XPU top-k/top-p sample kernel and LoRA support.
    • ›Adds IBM Power VSX attention backend.
    • ›Adds NVFP4 KV cache support and Triton dequant/QDQ emulation kernels for Hopper and AMD.
    • ›Adds Humming MXFP4 MoE backend.
    • ›Adds TurboQuant hybrid model and uniform quantization support.
    • ›Adds configurable model weights loading tracking.
    • ›Adds Triton JIT compilation monitor.
    • ›Adds LoRA initial expert parallel (EP) support.
    • ›Adds ViT CUDA graph support for Qwen2.5-VL.
    • ›Adds Qwen3.5/Mamba hybrid model support in Model Runner V2.
    • ›Adds Gated DeltaNet Attention for Qwen 3.5/3.6 on CPU.
    • ›Adds RISC-V OMP thread auto-binding for CPU backend.
    • ›Reduces Docker image size by approximately 2.5 GB via deferred FlashInfer cubin download.
    • ›Embeds container image provenance metadata.
    └──▷ BREAKING ON UPGRADE

    • !C++20-compatible compiler is now required to build vLLM (for PyTorch compatibility); builds with older compilers will fail.
    • !transformers v4 is formally deprecated; users must migrate to transformers v5.
13. v0.20.0 Apr 27, 2026 · issue -114

    vLLM v0.20.0 adds DeepSeek V4, TurboQuant 2-bit KV cache, FA4 default MLA prefill, online quantization frontend, and CUDA 13.0 as default.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.20.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.20.0

    └──▷ TRY IT
    Run vLLM on CUDA 12.9 using the recommended torch backend to avoid the new CUDA 13.0 default wheel.
    copy

    $ uv pip install vllm --torch-backend=cu129

    Enable ViT CUDA graph for a VLM video workload to maximize GPU throughput on Qwen3-VL-style models.
    copy

    $ vllm serve Qwen/Qwen3-VL-7B-Instruct --enable-vit-cuda-graph

    Cache remote media URLs across requests to avoid re-fetching the same images or video frames on every inference call.
    copy

    $ VLLM_MEDIA_CACHE=1 vllm serve <model>

    • ›Adds --enable-vit-cuda-graph flag for VLM examples to enable ViT full CUDA graph for video inference.
    • ›Adds VLLM_MEDIA_CACHE opt-in environment variable for media URL caching.
    • ›Adds request_id field on FinishedRequestStats for per-request completion tracking.
    • ›Adds --torch-backend=cu129 recommended install option via uv for CUDA 12.9 users.
    • ›Adds donate_graph_module=True support for standalone_compile in torch.compile integration.
    +47 moreshow less

    • ›Respects TORCH_COMPILE_DISABLE at vLLM config level to disable compilation without code changes.
    • ›Adds labeled waiting-breakdown metric distinguishing capacity vs. deferred queue states.
    • ›Introduces RayExecutorV2 for unified engine process monitoring with the Ray backend.
    • ›Adds shutdown() method on OffloadingConnector for clean KV offload teardown.
    • ›Adds CONFIG_REGISTRY config-class lookup to resolve on-disk model_type mismatches.
    • ›Enables CUDAGraph memory profiling by default for clearer startup memory accounting.
    • ›Adds default_max_frames_per_batch auto-inference for ViT CUDA graph video workloads.
    • ›Introduces mamba_ssm_cache_dtype=float32 default for NemotronH with auto-hook for NemotronHNanoVLV2.
    • ›Default CUDA wheel on PyPI and vllm/vllm-openai:v0.20.0 Docker image switched to CUDA 13.0 (13.0.2).
    • ›Adds initial DeepSeek V4 model architecture support.
    • ›Adds Hunyuan v3 (Hy3) preview model with HYV3 reasoning parser.
    • ›Adds Granite 4.1 Vision as a built-in multimodal model.
    • ›Adds EXAONE-4.5, BharatGen Param2MoE, Phi-4-reasoning-vision-15B, Cheers multimodal, telechat3, FireRedLID, jina-reranker-v3, Jina Embeddings v5, and Nemotron-v3 VL Nano/Super model architectures.
    • ›Enables FlashAttention 4 as the default MLA prefill backend with head-dim 512 and paged-KV on SM90+.
    • ›Introduces TurboQuant 2-bit KV cache attention backend delivering 4× KV cache capacity, now with FA3/FA4 prefill support.
    • ›Introduces new end-to-end online quantization frontend consolidating FP8, experts_int8, and MXFP8 online quantization paths.
    • ›Adds support for transformers>=5 (HuggingFace Transformers v5).
    • ›Adds Python 3.14 to the supported version list.
    • ›Ships with PyTorch 2.11 for CUDA and XPU (XPU no longer pinned to 2.10).
    • ›Adds full CUDA graph support for Eagle prefill in Model Runner V2.
    • ›Adds auto-resolution of CUDA graph mode and sizes from the attention backend.
    • ›Adds multiple prompt-logprobs support in Model Runner V2.
    • ›Introduces vLLM IR skeleton with rms_norm op as foundation for future kernel work.
    • ›Adds Helion kernel support for torch.compile.
    • ›Adds FlexAttention non-causal support.
    • ›Adds NUMA binding for GPU workers.
    • ›Adds CPU draft-model speculative decoding.
    • ›Adds MXFP4 W4A4 CUTLASS MoE support for SM100 (Blackwell).
    • ›Adds fused_silu_mul_block_quant kernel on AMD ROCm.
    • ›Adds MLA dual RMS norm fusion pass for DeepSeek/Kimi-K2 on ROCm.
    • ›Adds ZenCPU / AMD Zen CPU backend via zentorch.
    • ›Adds RDNA 3.5/4 device IDs (gfx1150/gfx1151/gfx1201) and gfx1102/gfx1103 support on ROCm.
    • ›Adds XPU MXFP8 and MXFP4 quantization ops for Intel XPU.
    • ›Adds FP8 KV cache support on Intel XPU.
    • ›Adds per-channel FP8 linear on Intel XPU.
    • ›Adds sm_110 (Jetson Thor) to CUDA 13.0 build targets.
    • ›Adds RISC-V multiple RVV VLEN target support.
    • ›Adds Eagle3 speculative decoding for MiniMax-M2 and Gemma4.
    • ›Adds LoRA support for Qwen3ASRForConditionalGeneration, Gemma4ForConditionalGeneration, and DeepSeek V3.2.
    • ›Adds GGUF support for MiniMax-M2.1 and non-standard GGUF quant types with prefix such as UD-IQ1_S.
    • ›Adds batched KV-cache swap via cuMemcpyBatchAsync for NVIDIA GPUs.
    • ›Adds DFlash attention backend on ROCm.
    • ›Adds fused qknorm+rope kernel on SM9.0.
    • ›Adds mm-scheduler overhead reduction via get_num_embed optimization.
    • ›Adds AOT compile with batch-invariance mode and Inductor cache nested under AOT directory.
    • ›Adds IBM Z s390x torch 2.11 build targets.
    • ›Achieves 2.1% E2E latency improvement via fused RMS norm batch-invariant optimization.
    └──▷ BREAKING ON UPGRADE

    • !PyTorch upgraded to 2.11 for CUDA and XPU — existing environments pinned to an earlier torch version will break; reinstallation required.
    • !Default CUDA wheel switched to CUDA 13.0.2; users on CUDA 12.9 must explicitly install with uv and --torch-backend=cu129.
    • !XPU is no longer pinned to torch 2.10; environments depending on torch 2.10 for XPU will require updating.
    • !SharedFusedMoE removed; code or configs referencing it will fail.
    • !make_expert_params_mapping renamed; call sites using the old name will break.
    • !MoE DP chunking removed; configurations relying on it will no longer function.
    • !Piecewise-fallback disabled for Eagle draft decodes; setups that depended on this fallback path will behave differently.
14. v0.19.1 Apr 18, 2026 · issue -123

    vLLM v0.19.1 upgrades to Transformers v5.5.3 and adds Gemma4 Eagle3 and quantized MoE support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.19.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.19.1

    • ›Adds Gemma4 Eagle3 speculative decoding support.
    • ›Adds quantized MoE support for Gemma4 models.
    • ›Upgrades the Transformers dependency to v5.5.3.
15. v0.19.0 Apr 3, 2026 · issue -138

    vLLM v0.19.0 adds Gemma 4, a new /v1/chat/completions/batch endpoint, CPU KV cache offloading, zero-bubble async spec decode, and VLLM_MAX_N_SEQUENCES security control.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.19.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.19.0

    └──▷ TRY IT
    Cap the number of concurrent sequences to prevent resource exhaustion on a shared inference server.
    copy

    $ VLLM_MAX_N_SEQUENCES=512 vllm serve meta-llama/Llama-3-8B-Instruct

    • ›Adds /v1/chat/completions/batch endpoint for batched chat completions.
    • ›Adds VLLM_MAX_N_SEQUENCES environment variable to enforce a hard cap on concurrent sequences, limiting resource exhaustion risk.
    • ›Adds --lora-target-modules flag (shorthand: no shorthand listed) to restrict LoRA adapters to specific model modules.
    • ›Adds -sc as a shorthand alias for --speculative-config.
    • ›Adds --speculative-config option for per-draft-model MoE backend selection in speculative decoding.
    +39 moreshow less

    • ›Adds hard limit on thinking tokens via the API (limit thinking tokens feature).
    • ›Adds online MXFP8 quantization support for both MoE and dense models.
    • ›Adds QeRL online quantization composed with quantized reloading, enabling quantization-aware RLHF workflows.
    • ›Adds W4A16 compressed tensors quantization support for CPU backends.
    • ›Adds CompressedTensor W4A8 quantization support for Intel XPU.
    • ›Adds AWQ Marlin support for ROCm backends.
    • ›Adds full Gemma 4 architecture support including MoE, multimodal, reasoning, and tool-use; requires transformers>=5.5.0 (recommended image: vllm/vllm-openai:gemma4).
    • ›Adds zero-bubble async scheduling with speculative decoding overlap, improving throughput.
    • ›Adds general CPU KV cache offloading for V1 with pluggable CachePolicy and block-level preemption.
    • ›Adds full CUDA graph capture for Vision Transformer (ViT) encoders.
    • ›Adds piecewise CUDA graphs for pipeline parallelism in Model Runner V2.
    • ›Adds configurable acceptance rate for Model Runner V2 spec decode.
    • ›Adds DBO (Dual-Batch Overlap) microbatch optimization generalized to all model architectures.
    • ›Adds NVIDIA B300/GB300 (SM 10.3) support with allreduce fusion enabled by default and tuned all-reduce communicator.
    • ›Adds Triton autotuning disk cache enabled by default.
    • ›Adds FlexAttention custom mask modification support.
    • ›Enables NFS prefetch with automatic RAM guard.
    • ›Adds support for multiple embedding types in a single API call.
    • ›Adds numpy array embeddings for multimodal inputs.
    • ›Adds GigaChat 3.1, Kimi-K2.5, and Gemma 4 tool parsers.
    • ›Adds new model architectures: Cohere ASR, Cohere Transcribe, ColQwen3.5 4.5B, LFM2-ColBERT-350M, Granite 4.0 1B Speech, Qwen3-ForcedAligner.
    • ›Adds Eagle3 speculative decoding support for Pixtral.
    • ›Adds DeepEP as all-to-all backend for AMD ROCm.
    • ›Adds persistent MLA kernel from AITER and FP8xFP8 attention in AITER for ROCm.
    • ›Adds ROCm 7.2.1, torch 2.10, triton 3.6 build support.
    • ›Adds async scheduling interface for TPU.
    • ›Adds MLA model support and CompressedTensor W4A8 for Intel XPU with auto-detect XPU build platform.
    • ›Enables tcmalloc by default on CPU with graceful degradation when unavailable.
    • ›Adds CPU slot mapping kernel and achieves 48.9% throughput improvement for pooling models on CPU.
    • ›Adds PD kv_transfer_params support for Anthropic Messages and Responses API in disaggregated serving.
    • ›Adds Mooncake heterogeneous TP support for disaggregated serving.
    • ›Adds tensor IPC transfer for multimodal data.
    • ›Adds PluggableLayer extensibility API for custom decoder implementations (e.g., CustomQwen2Decoder).
    • ›Adds plugin-overridable metadata build for the KV connector.
    • ›Adds Mega AOT artifact compilation support for torch 2.12+.
    • ›Adds frame limit enforcement in VideoMediaIO to prevent resource exhaustion.
    • ›Adds torch profiler with stack logging option.
    • ›Adds log-once-per-node default behavior to reduce log noise in multi-node deployments.
    • ›Removes per-tensor-per-channel FP8 and Sparse24 integration and kernels.
    └──▷ BREAKING ON UPGRADE

    • !Per-tensor-per-channel FP8 support has been removed.
    • !Sparse24 integration and kernels have been removed.
    • !The reasoning_content message field has been removed from the API.
    • !--calculate-kv-scales is deprecated.
    • !The score task is deprecated.
    • !Pooling multi-task support is deprecated.
    • !Virtual engine is deprecated (V0 deprecation path).
    • !--disable-frontend-multiprocessing is deprecated.
16. v0.18.0 Mar 20, 2026 · issue -152

    vLLM v0.18.0 adds gRPC serving, GPU-less render launch, NGram GPU speculative decoding, FlexKV offloading, and streaming tool calls in the Responses API.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.18.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.18.0

    └──▷ TRY IT
    Run GPU-less multimodal preprocessing and rendering on a CPU-only node, offloading preprocessing from the GPU inference server.
    copy

    $ vllm launch render

    Load an Elastic Expert Parallelism (EP) MoE model faster by filtering non-local expert weights during startup.
    copy

    $ vllm serve deepseek-ai/DeepSeek-V3 --enable-ep-weight-filter

    • ›Adds --grpc flag to enable gRPC serving alongside the existing HTTP/REST interface for high-performance RPC-based inference.
    • ›Adds vllm launch render subcommand for GPU-less multimodal preprocessing and rendering, separating preprocessing from GPU inference.
    • ›Adds --enable-ep-weight-filter CLI option for faster Elastic Expert Parallelism model loading via NIXL-EP integration.
    • ›Adds --distributed-timeout-seconds option for configuring distributed operation timeouts.
    • ›Adds --attention-backend auto option for automatic attention backend selection.
    +34 moreshow less

    • ›Adds reasoning_effort=none API option to disable reasoning effort.
    • ›Adds PyTorch profiler schedule configuration option.
    • ›NGram speculative decoding now runs on GPU and is compatible with the async scheduler, reducing speculative decode overhead.
    • ›Adds FlexKV as a new KV cache offloading backend with support for multiple KV groups in the offloading spec.
    • ›Adds smart CPU offloading that stores only frequently-reused KV cache blocks (reuse-frequency-gated CPU stores).
    • ›Adds streaming tool/function calling support to the OpenAI Responses API.
    • ›Adds online beam search transcription support for ASR encoder/decoder models, plus offline beam search, and audio transcription for MP4/M4A/WebM formats.
    • ›Adds Cohere Embed v2 API support.
    • ›Adds Azure Blob Storage support for RunAI Model Streamer.
    • ›Adds graceful shutdown timeout for in-flight requests.
    • ›Adds Anthropic API support for redacted thinking blocks.
    • ›Adds Granite4 tool parser.
    • ›Adds NIXL-EP integration for dynamic GPU scaling of MoE experts (Elastic EP Milestone 2).
    • ›Adds HMA + NIXL KV connector and All-to-All DCP backend for KV transfer.
    • ›Adds ModelOpt MXFP8 MoE quantization support.
    • ›Adds MXFP4 MoE routing simulation override for accuracy control.
    • ›Adds FP8 LoRA dense kernel.
    • ›Adds FA4 kernel support for MLA prefill.
    • ›Adds FlashInfer Sparse MLA FP8 KV cache support and CUDA graphs on ROCm.
    • ›Adds TRTLLM FP8 MoE modular kernel.
    • ›Adds 320-dimension MLA head size support.
    • ›Adds support for RISC-V CPU backend.
    • ›Adds AMD Zen CPU backend via zentorch.
    • ›Adds aarch64 int8 matmul via OneDNN upgrade.
    • ›Adds gfx1152/gfx1153 (Krackan) GPU support on ROCm.
    • ›Adds ROCm Quark W4A8 MXFP4/FP8 quantization support for LinearLayer.
    • ›Adds Intel XPU Model Runner V2, MLA Sparse backend for DeepSeek V3.2, and LoRA via torch.compile.
    • ›Adds OOT linear method registration and custom collective ops registration for non-CUDA platforms for extensibility.
    • ›Adds new model architectures: Sarvam MoE, OLMo Hybrid, HyperCLOVAX-SEED-Think-32B VLM, HyperCLOVAX-SEED-Think-14B, Kimi-Audio-7B-Instruct, ColPali late-interaction retrieval, and ERNIE pooling models.
    • ›Adds Eagle3 speculative decoding support for Qwen3.5 and Kimi K2.5 MLA, plus Eagle for Mistral Large 3 with dense layers.
    • ›Adds Whisper LoRA support.
    • ›Adds online use_audio_in_video multimodal capability, audio extraction from MP4 for Nemotron Nano VL, and media_io_kwargs exposure at runtime.
    • ›Updates FlashInfer dependency to version 0.6.6 with performance and correctness improvements.
    • ›Removes Ray as a default dependency — must now be installed explicitly.
    └──▷ BREAKING ON UPGRADE

    • !Ray is no longer a default dependency; existing setups that rely on Ray must install it explicitly after upgrading.
    • !Cascade attention is now disabled by default; workloads that relied on cascade attention being on by default will see changed behavior.
    • !The swap_space parameter has been removed (was deprecated in V0); configs or CLI invocations using swap_space will fail.
    • !Monolithic TRTLLM MoE is disabled for renormalize routing; setups depending on monolithic TRTLLM MoE with renormalize routing will be affected.
17. v0.17.1 Mar 11, 2026 · issue -160

    vLLM v0.17.1 adds support for the Nemotron 3 Super model.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.17.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.17.1

    • ›Adds support for the Nemotron 3 Super model.
18. v0.17.0 Mar 7, 2026 · issue -164

    vLLM v0.17.0 adds FlashAttention 4, --performance-mode, Anthropic count_tokens, quantized LoRA, and PyTorch 2.10.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.17.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.17.0

    └──▷ TRY IT
    Optimize a vLLM server for interactive (low-latency) use cases without manually tuning individual parameters.
    copy

    $ vllm serve meta-llama/Llama-3.1-8B-Instruct --performance-mode interactivity

    Optimize a vLLM server for maximum throughput batch workloads.
    copy

    $ vllm serve meta-llama/Llama-3.1-8B-Instruct --performance-mode throughput

    Select a specific MoE kernel backend explicitly when serving a Mixture-of-Experts model.
    copy

    $ vllm serve deepseek-ai/DeepSeek-V3 --moe-backend trtllm

    • ›Adds --performance-mode {balanced, interactivity, throughput} flag to simplify performance tuning for common deployment scenarios.
    • ›Adds --moe-backend flag for explicit MoE kernel selection.
    • ›Adds --language-model-only flag for hybrid models.
    • ›Adds Anthropic count_tokens API support.
    • ›Adds Anthropic tool_choice=none support.
    +53 moreshow less

    • ›Adds Anthropic thinking blocks support.
    • ›Integrates FlashAttention 4 backend for next-generation attention performance.
    • ›Supports loading quantized LoRA adapters (e.g. QLoRA) directly.
    • ›Adds AOT compile support with PyTorch 2.10.
    • ›Enables AR+RMSNorm fusion by default at -O2 and SiLU+FP4 quant fusion by default at O1+ for torch.compile.
    • ›Adds MFU Prometheus counters for observability.
    • ›Adds unrecognized environment variable warnings.
    • ›Adds per-head KV cache scales in the attention selector.
    • ›Adds sleep level 0 mode with enqueue/wait pattern.
    • ›Adds native weight syncing API via IPC for RL workflows.
    • ›Adds repetitive token pattern detection flags.
    • ›Adds Triton-based top-k and top-p sampler kernels.
    • ›Adds FlashInfer Sparse MLA backend.
    • ›Adds Helion kernel framework with autotuning infrastructure.
    • ›Adds NIXL Token-based IPC API.
    • ›Adds Elastic Expert Parallelism Milestone 2 for dynamic GPU scaling on MoE models.
    • ›Adds MoERunner abstraction with modular kernel architecture.
    • ›Adds min_tokens support with speculative decoding.
    • ›Adds Nemotron-H MTP and Mamba speculative decoding.
    • ›Adds Eagle3 speculative decoding with CUDA graphs on Model Runner V2.
    • ›Adds Pipeline Parallel and Decode Context Parallel support to Model Runner V2.
    • ›Adds weight offloading V2 with prefetching to hide onloading latency, selective CPU weight offloading, and CPU offloading without pinned memory doubling.
    • ›Adds support for new model architectures: Qwen3.5 (with GDN, FP8 quantization, MTP speculative decoding, and reasoning parser), COLQwen3, ColModernVBERT, Ring 2.5, skt/A.X-K1, Ovis 2.6, nvidia/llama-nemotron-embed-vl-1b-v2, nvidia/llama-nemotron-rerank-vl-1b-v2, nvidia/nemotron-colembed.
    • ›Adds ASR model support: FunASR, FireRedASR2, Qwen3-ASR realtime streaming.
    • ›Adds OpenPangu-VL video input, audio chunking for offline LLM, and Parakeet audio encoder.
    • ›Adds Whisper automatic language detection.
    • ›Adds structured outputs support to the Responses API.
    • ›Adds structured output JSON feature validation.
    • ›Adds explicit validation error for tool calls.
    • ›Adds sparse embedding IO process plugin.
    • ›Adds IO Processor plugin simplification.
    • ›Adds LoRA support for LFM2 and Llama 4 Vision tower/connector; increases max LoRA vocab size to 258048.
    • ›Adds SM100 MXFP8 blockscaled grouped MM and quantization kernels.
    • ›Adds AITER fused RoPE+KVCache for AMD ROCm.
    • ›Adds CUDA graph support for Intel XPU.
    • ›Adds ARM BF16 cross-compilation and s390x FP16 support for CPU builds.
    • ›CPU release now supports AVX2, AVX-512, VNNI, AVX512BF16, and AMX via a multi-ISA dispatcher.
    • ›Upgrades to PyTorch 2.10.0.
    • ›Adds Async EPLB rebalance algorithm.
    • ›Adds Ray environment variable propagation to workers.
    • ›Adds mixed precision support for ModelOpt.
    • ›Adds Llama-4 attention quantization (int8, fp8).
    • ›Adds bitsandbytes quantization support on AMD ROCm.
    • ›Adds GPT-OSS Quark format support on AMD ROCm.
    • ›Adds generation_config max_tokens treated as a default rather than a ceiling.
    • ›Adds validation of non-text content in system messages.
    • ›Adds ColBERT extension to non-standard BERT backbones and multimodal scoring for late-interaction models.
    • ›Adds KV cache update ops extraction from FlashInfer forward and MLA backends.
    • ›Adds TRTLLM DSV3 Router GEMM kernel with 6% batch-1 speedup.
    • ›Adds pooling model support to Model Runner V2 with 13.9% throughput improvement.
    • ›Pipeline Parallel async send/recv yields 2.9% E2E throughput improvement.
    • ›Adds Reasoning API streaming reasoning_part events.
    • ›Adds amd-aiter package (renamed from aiter) for AMD ROCm.
    └──▷ BREAKING ON UPGRADE

    • !PyTorch upgraded to 2.10.0, which is a breaking change for environment dependencies.
    • !KV load failure policy default changed from 'recompute' to 'fail' (controlled in large-scale serving configuration).
    • !AMD ROCm aiter package renamed to amd-aiter; any environment or install referencing aiter by name must be updated.
19. v0.16.0 Feb 25, 2026 · issue -173

    vLLM v0.16.0 adds a WebSocket Realtime API, native NCCL weight-sync for RLHF, unified parallel speculative decoding, and major XPU/ARM/NVIDIA kernel expansions.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.16.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.16.0

    └──▷ TRY IT
    Suppress access log noise for health-check or metrics endpoints in high-traffic deployments.
    copy

    $ vllm serve meta-llama/Llama-3-8B --disable-access-log-for-endpoints

    Load a specific GGUF quantization variant directly from a Hugging Face repo without manually specifying a file path.
    copy

    $ vllm serve bartowski/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M

    • ›Adds --disable-access-log-for-endpoints CLI flag to suppress access logging for specific endpoints.
    • ›Adds GGUF repo_id:quant_type syntax for selecting quantization variants when loading GGUF models.
    • ›Adds nested config support in YAML files for vLLM server configuration.
    • ›Adds FlashInfer autotune control flag (#34006) to manage FlashInfer autotuning behavior at runtime.
    • ›Adds native NCCL-based weight syncing API for RLHF workflows, enabling direct weight updates between training and inference engines without third-party communication layers.
    +31 moreshow less

    • ›Adds engine pause/resume with request preservation for RLHF workflows, allowing training loops to halt inference without dropping in-flight requests.
    • ›Adds layerwise weight reloading for QeRL in RLHF workflows.
    • ›Adds WebSocket-based Realtime API for streaming audio interactions, including Voxtral realtime support.
    • ›Adds sampling parameters and token/prompt token ID return fields to the Responses API.
    • ›Adds /server_info collect_env endpoint for retrieving environment diagnostics from a running server.
    • ›Adds OTEL tracing during model loading for observability of startup phases.
    • ›Adds run_batch transcription and translation support.
    • ›Adds HF Hub LoRA resolver (#20320) for loading LoRA adapters directly from Hugging Face Hub.
    • ›Adds KV offloading connector metrics and labeled prompt token metrics for P/D disaggregation.
    • ›Adds Unified Parallel Drafting for speculative decoding, delivering throughput improvements alongside structured output support and penalty application in Model Runner V2.
    • ›Adds async scheduling with Pipeline Parallelism, delivering 30.8% end-to-end throughput improvement and 31.8% TPOT improvement.
    • ›Adds FP8 block quant for CompressedTensorsW8A16Fp8, ModelOpt MXFP8 for dense models, and NVFP4/FP8 support on Turing GPUs.
    • ›Adds TP > 4 support for FP4 Gemm quantization.
    • ›Adds Mooncake disaggregated serving connector rework with bootstrap server and cross-layer KV cache layout at NIXL Connector V2.
    • ›Adds EPLB logical expert capture with router replay for expert-parallel load balancing.
    • ›Adds MoE cold start optimization option for torch.compile to reduce compilation overhead.
    • ›Adds Intel XPU support for unquantized MoE, MXFP4 MoE, WNA16, scaled_mm, and FP8 MoE kernels via vllm-xpu-kernels (replacing IPEX).
    • ›Adds ARM CPU KleidiAI INT4 dynamic quantization with BF16 activations and NEON BFMMLA BF16 paged attention.
    • ›Adds IBM Z BF16 kernel type for s390x architecture.
    • ›Adds SM100 INT4 W4A16 kernel and SM121 (DGX Spark) CUTLASS support for NVIDIA GPUs.
    • ›Adds MNNVL protocol support for GB series NVIDIA hardware.
    • ›Adds ORJSONResponse for faster API response serialization.
    • ›Adds DeepSeek ReasoningParser with thinking enabled by default.
    • ›Adds early tokenization validation to surface malformed inputs before scheduling.
    • ›Adds LoRA expansion for Gemma3 vision components, Nemotron-H MTP models, and Qwen3 output embedding.
    • ›Adds new model architecture support for GLM-OCR with MTP, Qwen3-ASR, DeepSeek-OCR-2, Intern-S1-Pro, MiniCPM-o 4.5, openPangu7B-VL, NemotronHPuzzle heterogeneous, MusicFlamingo, FunAudioChat, ColBERT late interaction, voyage-4-nano, and GLM-5.
    • ›Adds EAGLE3 speculative decoding support for Hunyuan/HunyuanVL, AFMoE, and Mistral3.
    • ›Adds Pooling API request schema consensus for ScoreRequest and final standardization.
    • ›Adds PluggableLayer support applied to linear layers and Mamba layers.
    • ›Adds Helion kernel framework components: ConfigManager, kernel wrapper, and kernel registry.
    • ›Only includes the Authorization header in outbound requests when OPENAI_API_KEY is set, reducing credential leakage risk.
    └──▷ BREAKING ON UPGRADE

    • !Removed BitBlas quantization backend — workloads using BitBlas will fail after upgrade.
    • !Removed Marlin 24 quantization backend — workloads using Marlin 24 will fail after upgrade.
    • !Removed deprecated reasoning_content message field — code referencing this field will break.
    • !Removed deprecated pooling items — any configuration or API calls relying on the deprecated pooling endpoints/fields will break.
    • !Removed deprecated VLLM_ALL2ALL_BACKEND environment variable — scripts setting this variable will have no effect (and previously working behavior dependent on it is gone).
    • !Deprecated IPEX for Intel XPU; switched to vllm-xpu-kernels — existing XPU deployments using IPEX will need to migrate to vllm-xpu-kernels.
20. v0.15.1 Feb 4, 2026 · issue -194

    vLLM v0.15.1 adds Step-3.5-Flash model support and restores RTX Blackwell GPU compatibility.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.15.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.15.1

    • ›Adds support for the Step-3.5-Flash model.
21. v0.15.0 Jan 29, 2026 · issue -200

    vLLM v0.15.0 adds Kimi-K2.5/Molmo2/Step3vl models, Mamba prefix caching, async scheduling with PP, FIPS 140-3 hashing, and new API fields.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.15.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.15.0

    └──▷ TRY IT
    Enable Mamba prefix caching to get ~2x speedup on hybrid/Mamba models by caching Mamba states across requests.
    copy

    $ vllm serve <model> --enable-prefix-caching --mamba-cache-mode align

    Run pipeline-parallel serving with async scheduling enabled, previously a blocked combination.
    copy

    $ vllm serve <model> --pipeline-parallel-size 4 --async-scheduling

    Restrict TLS cipher suites on a vLLM API server for hardened enterprise deployments.
    copy

    $ vllm serve <model> --ssl-keyfile key.pem --ssl-certfile cert.pem --ssl-ciphers 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256'

    • ›Enables --async-scheduling with pipeline parallelism, previously unsupported.
    • ›Adds Mamba/hybrid model prefix caching via --enable-prefix-caching --mamba-cache-mode align, achieving ~2x speedup by caching Mamba states directly.
    • ›Adds --ssl-ciphers CLI argument for TLS cipher suite control.
    • ›Adds skip_special_tokens configuration to the OpenAI-compatible API.
    • ›Adds prompt_cache_key support to the Responses API.
    +26 moreshow less

    • ›Adds include_stop_str_in_output tuning parameter to the Responses API.
    • ›Adds data_1/data_2 and queries/documents flexible input formats to the score endpoint.
    • ›Adds new render endpoints for prompt preprocessing.
    • ›Adds avg_logprob and compression_ratio fields to Whisper API verbose_json segment output.
    • ›Adds FIPS 140-3 compliant hash option for enterprise/government deployments.
    • ›Auto-detects api_server_count based on dp_size to simplify multi-process server configuration.
    • ›Adds support for new model architectures: Kimi-K2.5, Molmo2, Step3vl 10B, Step1, GLM-Lite, and Eagle2.5-8B VLM.
    • ›Extends LoRA support to Nemotron-H, InternVL2, and MiniMax M2 models.
    • ›Adds EAGLE3 speculative decoding for Pixtral/LlavaForConditionalGeneration and Qwen3 VL MoE.
    • ›Adds BGE-M3 sparse embeddings and ColBERT embeddings support.
    • ›Adds VLM support to Model Runner V2.
    • ›Adds inplace LoRA loading for improved memory efficiency.
    • ›Adds torch.compile inductor AOT artifact support.
    • ›Adds torch.compile support for Whisper.
    • ›Adds MXFP4 W4A16 quantization support for compressed-tensors MoE models.
    • ›Adds quantization support for non-gated MoE with Marlin, NVFP4 CUTLASS, FP8, INT8, and compressed-tensors.
    • ›Adds Intel Quantization Toolkit integration.
    • ›Adds FP8 KV cache per-tensor and per-attention-head quantization via llmcompressor.
    • ›Adds session-based streaming input accepting async generators producing StreamingInput objects for interactive workloads like ASR.
    • ›Sets FlashInfer MLA as the default MLA backend on Blackwell, with TRTLLM as default prefill.
    • ›Adds MoRI EP high-performance all2all backend for Expert Parallel on AMD ROCm.
    • ›Adds Flash Attention Triton backend support for AMD RDNA3/RDNA4 consumer GPUs.
    • ›Adds TPU pipeline parallelism support.
    • ›Adds NUMA-aware acceleration for TP/DP inference on ARM CPU.
    • ›Adds wheel variant auto-detection during install.
    • ›Adds custom profiler URI scheme support.
    └──▷ BREAKING ON UPGRADE

    • !The vllm:time_per_output_token_seconds metric has been removed — use vllm:inter_token_latency_seconds instead.
    • !Several deprecated environment variables have been removed.
    • !DeepSpeedFp8 quantization has been removed.
    • !RTN quantization has been removed.
    • !HQQ quantization is deprecated.
22. v0.14.0 Jan 20, 2026 · issue -209

    vLLM v0.14.0 adds a gRPC server, --max-model-len auto, async scheduling by default, and broad new model/quantization support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.14.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.14.0

    └──▷ TRY IT
    Start vLLM with automatic context-length fitting to avoid OOM errors on GPU memory-constrained deployments.
    copy

    $ vllm serve meta-llama/Llama-3-8B-Instruct --max-model-len auto

    Inspect a model's attention backends, modules, and quantization config before serving.
    copy

    $ VLLM_LOG_MODEL_INSPECTION=1 vllm serve Qwen/Qwen3-8B --max-model-len 8192

    Query the new /server_info endpoint to retrieve environment and runtime details from a running vLLM instance.
    copy

    $ curl http://localhost:8000/server_info

    • ›Adds --max-model-len auto flag to automatically fit context length to available GPU memory, eliminating OOM startup failures.
    • ›Adds --no-async-scheduling flag to disable the newly default async scheduling (overlaps engine core scheduling with GPU execution for higher throughput).
    • ›Adds -ep shorthand flag for --enable-expert-parallel.
    • ›Adds --enable-log-deltas CLI flag (renamed in this release) for iteration delta logging.
    • ›Adds --default-chat-template-kwargs CLI flag to pass default kwargs to chat templates.
    +25 moreshow less

    • ›Adds --input-len flag and auto-discovery to bench serve for benchmark configuration.
    • ›Adds VLLM_LOG_MODEL_INSPECTION=1 environment variable (or printing the LLM object) to inspect a model's modules, attention backends, and quantization configuration.
    • ›Adds attention_config parameter to the LLM() constructor for programmatic attention configuration.
    • ›Adds reasoning_effort parameter to the API for controlling reasoning depth.
    • ›Adds gRPC server entrypoint as an alternative to the REST API, supporting binary protocol and HTTP/2 multiplexing.
    • ›Adds /server_info API endpoint exposing environment information.
    • ›Adds continue_final_message field to the /embeddings API endpoint.
    • ›Adds MCP streaming support in the Responses API.
    • ›Adds MFU (Model FLOP Utilization) metrics to the API.
    • ›Adds spec-decode acceptance stats reporting via CLI.
    • ›Adds offline FastAPI documentation endpoint.
    • ›Adds FunctionGemma and GLM-4.7 tool-calling parsers.
    • ›Adds new model architecture support: Grok-2 (with tiktoken tokenizer), LFM2-VL, MiMo-V2-Flash, openPangu MoE, IQuestCoder, Nemotron Parse 1.1, GLM-ASR audio, Isaac vision model v0.1/v0.2, Kanana-1.5-v-3b-instruct, and K-EXAONE-236B-A23B MoE.
    • ›Expands LoRA support to multimodal tower/connector models (LLaVA, BLIP2, PaliGemma, Pixtral, DotsOCR, GLM4-V) and additional architectures (DeepSeek-OCR, Qwen3-Next, NemotronH, PLaMo 2/3).
    • ›Adds enable_thinking: false config option for GLM-4.5/GLM-4.7 models.
    • ›Adds Marlin quantization support for Turing (sm75) GPUs.
    • ›Adds Quark int4-fp8 w4a8 MoE quantization.
    • ›Adds MXFP4 W4A16 quantization for dense models.
    • ›Adds ModelOpt FP8 variants: FP8_PER_CHANNEL_PER_TOKEN and FP8_PB_WO.
    • ›Adds XBO (Extended Dual-Batch Overlap) for large-scale serving.
    • ›Adds NIXL asymmetric TP support (P > D tensor-parallel-size).
    • ›Async scheduling now also works with speculative decoding and structured outputs.
    • ›Adds hybrid allocator + KV connector with multiple KV cache groups.
    • ›Adds SM103 hardware support.
    • ›Adds weights_only=True to torch.load calls, reducing deserialization attack surface.
    └──▷ BREAKING ON UPGRADE

    • !Async scheduling is now enabled by default; disable with --no-async-scheduling if issues arise (pipeline parallel, CPU backend, and non-MTP/Eagle spec decoding are excluded and remain unaffected).
    • !PyTorch 2.9.1 is now required and the default wheel is compiled against cu129.
    • !Deprecated quantization schemes have been removed.
    • !When using speculative decoding, unsupported sampling parameters now fail with an error rather than being silently ignored.
23. v0.13.0 Dec 19, 2025 · issue -241

    vLLM v0.13.0 adds new models, --attention-backend CLI arg, multi-vector retrieval, MCP tooling, and major Whisper/DeepSeek performance gains.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.13.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.13.0

    └──▷ TRY IT
    Switch from the old environment variable to the new CLI argument to select the Triton attention backend when launching a vLLM server.
    copy

    $ vllm serve meta-llama/Llama-3-8b-instruct --attention-backend triton

    Request binary-format embeddings when latency or bandwidth on the wire matters.
    copy

    $ curl http://localhost:8000/v1/embeddings -H 'Content-Type: application/json' -d '{"model": "BAAI/bge-base-en", "input": "Hello world", "encoding_format": "bytes_only"}'

    Reset the prefix cache on a running disaggregated prefill/decode deployment without restarting the server.
    copy

    $ curl -X POST http://localhost:8000/reset_prefix_cache

    • ›Replaces VLLM_ATTENTION_BACKEND environment variable with --attention-backend CLI argument via new AttentionConfig.
    • ›Adds encoding_format=bytes_only to the embeddings API for binary-format embedding output.
    • ›Adds /reset_prefix_cache endpoint to KV connectors for cache reset in disaggregated prefill/decode deployments.
    • ›Adds compile_ranges config for conditional/selective kernel compilation.
    • ›Adds xxHash as a high-performance hash option for prefix caching.
    +33 moreshow less

    • ›Adds multi-vector retrieval API for pooling tasks.
    • ›Adds chunked prefill support for ALL pooling tasks.
    • ›Adds MCP type infrastructure, Browser/Container MCP tools, and full MCP Python loop to the Responses API.
    • ›Adds extra body parameters support to the Responses API.
    • ›Adds min-p sampling support in Model Runner V2.
    • ›Adds NaN detection in logits in Model Runner V2.
    • ›Adds prefill KV compute metric that excludes cached tokens.
    • ›Adds layer-wise NVTX profiling and a profiling CLI config.
    • ›Adds automatic TokenClassification model conversion.
    • ›Adds support for NVIDIA Blackwell Ultra SM103 (GB300) with CUDA 13.
    • ›Adds W4A8 grouped GEMM quantization on Hopper GPUs.
    • ›Adds online FP8 quantization with streaming post-processing.
    • ›Adds FP8 weight reloading for RLHF workflows.
    • ›Adds AWQ Marlin and GPTQ Marlin quantization support for MoE + LoRA.
    • ›Adds TRITON_MLA attention support without requiring prefix-caching.
    • ›Adds PrefixLM support for FlexAttention and TritonAttention backends.
    • ›Adds CUDA graphs for 3D Triton attention.
    • ›Adds Mooncake Transfer Engine as a KV connector.
    • ›Adds KV events support and failure recovery config for KV connectors.
    • ›Adds external launcher mode for multi-node serving.
    • ›Adds NIXL compatibility checking in handshake and large-batch proxy support.
    • ›Adds AMD ROCm MXFP4 w4a4 inference and Aiter quantization kernels.
    • ›Adds Intel XPU wNa16 compressed tensor support.
    • ›Delivers ~3x Whisper throughput improvement (V1 now faster than V0), with encoder batching and CPU backend support.
    • ›Adds Eagle/Eagle3 speculative decoding via the Transformers backend.
    • ›Adds new models: BAGEL, AudioFlamingo3, JAIS 2, and latent MoE architecture.
    • ›Adds tool parsers for DeepSeek-V3.2, Gigachat 3, and Holo2 reasoning.
    • ›Adds Qwen3-VL embeddings support and Efficient Video Sampling (EVS).
    • ›Adds BitsAndBytes quantization support for Qwen3-Omni-MoE.
    • ›Adds support for multiple images/audio per embeddings request.
    • ›Adds tokenization_kwargs override for embeddings requests.
    • ›Enables DeepEP High-Throughput CUDA graph by default for ~5.3% throughput and ~4.4% TTFT improvement on DeepSeek models.
    • ›Adds CUDA 13 aarch64 wheels and x86 CPU wheel pipeline.
    └──▷ BREAKING ON UPGRADE

    • !VLLM_ATTENTION_BACKEND environment variable is replaced by the --attention-backend CLI argument; existing env-var-based attention backend configuration will no longer work.
    • !PassConfig flags have been renamed per RFC #27995; existing configs using the old flag names will break.
    • !The -O.xx flag has been removed; any scripts or configs using it will fail.
    • !Deprecated plugin and compilation fields have been removed; configs referencing them will break.
    • !Deprecated task, seed, and multimodal (MM) settings have been removed.
    • !embed_input_ids and embed_multimodal fallback fields have been removed; requests relying on them will fail.
    • !The tokenizer setter has been removed; code that set the tokenizer via this interface will break.
    • !--convert reward CLI flag is replaced by --convert embed; existing scripts using --convert reward will fail.
24. v0.12.0 Dec 3, 2025 · issue -257

    vLLM v0.12.0 adds GPU Model Runner V2, EAGLE multimodal speculative decoding, RLHF pause/resume, new optimization levels, and major quantization/API expansions.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.12.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.12.0

    └──▷ TRY IT
    Start vLLM with aggressive compilation to maximize throughput at the cost of longer startup, useful for long-running production deployments.
    copy

    $ vllm serve meta-llama/Llama-3-8B-Instruct -O3

    Serve a quantized GGUF model directly from HuggingFace using the new repo_id:quant_type syntax without manually downloading files.
    copy

    $ vllm serve bartowski/Meta-Llama-3-8B-Instruct-GGUF:Q4_K_M

    Enable fully-sharded LoRA adapters with FusedMoE for MoE models to reduce memory fragmentation across tensor-parallel ranks.
    copy

    $ vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 --fully-sharded-loras --enable-lora

    • ›Adds -O0, -O1, -O2, -O3 optimization-level flags to trade startup time for runtime performance, with more compilation flags planned.
    • ›Adds --fully-sharded-loras flag enabling FusedMoE LoRA support.
    • ›Moves flat logprob control from an environment variable into SamplingParams, making it a first-class sampling parameter.
    • ›Adds verbose_json and timestamp features to the Whisper transcription/translation API.
    • ›Adds parallel_tool_calls param compliance and parsed tool arguments support to the tool-calling API.
    +27 moreshow less

    • ›Adds inputs_embeds_size config key as a separate dimension from hidden_size for flexible embedding configuration.
    • ›Supports GGUF model loading from HuggingFace using repo_id:quant_type syntax for improved UX.
    • ›Adds parallel_tool_calls param and tool filtering support in ToolServer.
    • ›Adds Responses API multi-turn support for non-harmony requests, plus reasoning item input parsing.
    • ›Adds GPU Model Runner V2 (experimental): GPU-persistent block tables, Triton-native sampler with per-request seeds, memory-efficient prompt logprobs, and simplified DP/CUDA graph implementations.
    • ›Adds Prefill Context Parallel (PCP) to partition the sequence dimension during prefill, complementing existing Decode Context Parallel (DCP) for improved long-sequence inference.
    • ›Adds RLHF pause and resume generation support for asynchronous RL training workflows.
    • ›Adds cross-layer KV blocks support and KV cache residency metrics.
    • ›Adds audio embeddings support in chat completions.
    • ›Adds EAGLE speculative decoding with multi-step CUDA graph support, DP>1, and multimodal support including Qwen3VL.
    • ›Adds logprobs support with speculative decoding and async scheduling.
    • ›Adds iteration-level profiling for Torch and CUDA profiler.
    • ›Adds colorized log output.
    • ›Adds new model families: PLaMo-3, OpenCUA-7B, HunyuanOCR, Mistral Large 3, and Ministral 3.
    • ›Adds Gemma3 GGUF multimodal format support.
    • ›Adds Qwen3 Omni audio-in-video multimodal support.
    • ›Delivers 18.1% throughput improvement and 10.7% TTFT improvement on DeepSeek-V3.1 via batch invariant BMM optimization.
    • ›Delivers 2.2% throughput improvement and 3.6% TTFT improvement at batch size 32 via shared experts overlap with FlashInfer DeepGEMM.
    • ›Adds W4A8 Marlin kernel quantization support.
    • ›Adds NVFP4 MoE CUTLASS support for SM120 and TRTLLM MoE NVFP4 kernel.
    • ›Adds AWQ compressed-tensors support for Turing GPUs.
    • ›Adds FusedMoE LoRA Triton kernel for MXFP4 quantization.
    • ›Adds AMD ROCm support for DeepSeek v3.2 with SparseMLA, FP8 MLA decode, AITER attention backend, bitsandbytes quantization on warp-size-32 GPUs, and sliding window for AiterFlashAttentionBackend.
    • ›Adds paged attention GEMM acceleration on ARM CPUs with NEON.
    • ›Adds FlashAttention ViT support, now the default attention backend.
    • ›Adds enhanced NUMA functionality for systems with multiple NUMA nodes per socket.
    • ›Reduces Docker image size by approximately 200MB.
    └──▷ BREAKING ON UPGRADE

    • !PyTorch upgraded to 2.9.0 with CUDA 12.9 — environments must be updated to CUDA 12.9.
    • !The num_lookahead_slots parameter has been removed.
    • !The best_of parameter has been removed.
    • !LoRA extra vocab has been removed.
    • !The xformers backend is deprecated.
    • !Mistral format is now auto-detected during model loading, which may change behavior for existing Mistral model configs.
    • !seed=None is deprecated.
25. v0.11.1 Nov 18, 2025 · issue -272

    vLLM v0.11.1 adds Anthropic /v1/messages API, DeepSeek-V3.2 model, GCS streamer support, and async scheduling improvements across 1456 commits.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.11.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.11.1

    └──▷ TRY IT
    Query vLLM using an Anthropic-compatible client against the new /v1/messages endpoint.
    copy

    $ curl http://localhost:8000/v1/messages \
      -H 'Content-Type: application/json' \
      -H 'x-api-key: <your-api-key>' \
      -d '{"model": "deepseek-ai/DeepSeek-V3", "max_tokens": 256, "messages": [{"role": "user", "content": "Hello!"}]}'

    Dump torch.compile debug artifacts to disk for inspecting graph partitions and compile-cache issues.
    copy

    $ VLLM_DEBUG_DUMP_PATH=/tmp/vllm_compile_debug vllm serve meta-llama/Llama-3-8B --compilation-config '{"cudagraph_mode": "full_graph"}'

    • ›Adds VLLM_DEBUG_DUMP_PATH environment variable for dumping torch.compile debug artifacts.
    • ›Updates default CUDA build to torch==2.9.0+cu129 (PyTorch 2.9.0 + CUDA 12.9.1), enabling Inductor partitioning.
    • ›Adds --async-scheduling flag with robustness fixes covering chunked prefill, structured outputs, priority scheduling, MTP, and DeepEP/DCP; expected to become the default in the next release.
    • ›Adds support for the DeepSeek-V3.2 model.
    • ›Adds Run:ai model streamer GCS package support.
    +18 moreshow less

    • ›Adds media domain restriction filtering via add option to restrict media domains, preventing bypass via HTTP redirects.
    • ›Adds RL online quantization support with torchao.
    • ›Adds cuda2cpu support in NixlConnector for P/D disaggregated serving.
    • ›Adds KV load failure recovery support in V1 P/D disaggregated mode.
    • ›Adds MLA cache support with different latent dimensions in NIXL.
    • ›Adds explicit pooling classes for the Transformers backend.
    • ›Adds Eagle/Eagle3 multimodal support for speculative decoding, with enablement on Qwen2.5-VL.
    • ›Adds text-only mode for Qwen3-VL multimodal model.
    • ›Adds LoRA support for OPT models.
    • ›Adds ROCm support for AMD Ryzen AI MAX / AI 300 Series GPUs.
    • ›Adds Blackwell GPU family support (SM100+) including INT8 quantization fixes.
    • ›Adds batch-invariant torch.compile support across attention and MoE backends, with explicit support for DeepGEMM and FlashInfer on Hopper and Blackwell GPUs.
    • ›Updates to Transformers v4.56.2.
    • ›Adds chunk-aligned Mamba2 kernel.
    • ›Adds GC debug callback for core diagnostics.
    • ›Adds more tuned Triton configs for ExpertsInt8 and FP8 MoE kernels.
    • ›Serializes cudagraph_mode as its enum name instead of numeric value in torch.compile.
    • ›Adds chat template kwargs filtering for API security.
    └──▷ BREAKING ON UPGRADE

    • !The --endpoint-type benchmark flag, deprecated in v0.11.0, is now removed.
    • !VllmConfig is moved from config/__init__.py to config/vllm.py; imports that referenced the old path will break.
    • !vllm.worker module is removed; any code importing from vllm.worker directly will break.
26. v0.11.0 Oct 2, 2025 · issue -318

    vLLM v0.11.0 ships V1-only engine, CPU KV offloading, new model architectures, FP8/FP4 quant expansions, and major API/CLI additions.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.11.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.11.0

    └──▷ TRY IT
    Request full-vocabulary logprobs for every output token to analyze model confidence across the entire vocab.
    copy

    $ curl http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{"model": "meta-llama/Llama-3-8B-Instruct", "prompt": "Hello world", "logprobs": -1}'

    • ›Adds --enable-logging CLI flag for controlling logging output.
    • ›Adds logprobs=-1 support to return logprobs for the full vocabulary via the OpenAI-compatible API.
    • ›Adds FULL_AND_PIECEWISE as the new default CUDA graph mode (previously PIECEWISE), improving out-of-the-box performance especially for fine-grained MoEs.
    • ›Adds speculative model engine args to the Config system for configuring speculative decoding at the engine level.
    • ›Adds NVTX profiling support via config, enabling GPU timeline tracing for performance analysis.
    +57 moreshow less

    • ›Adds LLM.apply_model method for direct model access in the V1 engine.
    • ›Adds KV cache metrics reporting in GiB units via updated metrics fields.
    • ›Adds V1 TPOT (Time Per Output Token) histogram metric.
    • ›Adds KV transfer metrics for disaggregated serving.
    • ›Adds prompt logprobs for all tokens in OpenAI-compatible completions API.
    • ›Adds reasoning streaming events to the OpenAI-compatible API.
    • ›Adds MCP tools support to the Responses API.
    • ›Adds XML tool-call parser for Qwen3-Coder and Hermes-style tool-call token support.
    • ›Adds health endpoint 503 response when the engine is dead.
    • ›Adds image path format support for multimodal inputs via the API.
    • ›Adds media UUID caching: clients can skip re-uploading media data when UUIDs are provided.
    • ›Adds torchrun launcher support for data-parallel large-scale serving.
    • ›Adds CPU KV cache offloading with LRU management.
    • ›Adds prompt embeddings support in V1 engine.
    • ›Adds sharded state loading in V1 engine.
    • ›Adds FlexAttention sliding window attention in V1.
    • ›Adds shared-memory-based multimodal data caching and IPC between processes.
    • ›Adds BERT token classification / NER task support.
    • ›Adds multimodal model support for pooling tasks.
    • ›Enables DeepGEMM by default, delivering ~5.5% throughput improvement.
    • ›Enables NCCL symmetric memory by default for tensor parallelism, with 3-4% throughput improvement.
    • ›Adds NVFP4 quantization support for dense models including Gemma3 and Llama 3.1 405B.
    • ›Adds FP8 per-token-group quantization and hardware-accelerated FP8 instructions.
    • ›Adds FP8 FlashInfer MLA decode support for NVIDIA GPUs.
    • ›Adds BF16 fused MoE for Hopper/Blackwell expert parallel.
    • ›Adds FlashAttention 3 support for Vision Transformer (ViT) inference.
    • ›Adds Dual-Batch Overlap (DBO) for overlapping prefill and decode computation.
    • ›Adds EAGLE3 speculative decoding support for MiniCPM3 and GPT-OSS.
    • ›Adds SeedOSS reasoning parser.
    • ›Adds EVS video token pruning for video multimodal models.
    • ›Adds data-parallel support for vision encoders in InternVL, Qwen2-VL, and Qwen3-VL.
    • ›Adds RADIO encoder support.
    • ›Adds Transformers backend support for encoder-only models.
    • ›Adds new model architectures: DeepSeek-V3.2-Exp, Qwen3-VL, Qwen3-Next, OLMo3, LongCat-Flash, Dots OCR, Ling2.0, CWM.
    • ›Adds RISC-V 64-bit and ARM non-x86 CPU backend support.
    • ›Adds ROCm 7.0 support.
    • ›Adds ARM 4-bit fused MoE kernel.
    • ›Adds Whisper model support on Intel XPU.
    • ›Adds Hybrid SSM/Attention support in Triton.
    • ›Adds torch.compile CUDA graph Inductor partition integration.
    • ›Adds EPLB (Expert-Parallel Load Balancing) support for Hunyuan V1, Mixtral, and static placement.
    • ›Adds Mamba2 support with tensor parallelism and quantization.
    • ›Adds MRoPE + YaRN support for long-context models.
    • ›Adds --help improvements to the CLI for better discoverability.
    • ›Adds env validation for configuration, catching misconfigured environment variables at startup.
    • ›Adds guided decoding backward compatibility fixes.
    • ›Adds NIXL MLA latent dimension support for disaggregated serving.
    • ›Adds Ray placement group support for data-parallel deployments.
    • ›Adds Triton DP/EP kernels for large-scale data/expert parallelism.
    • ›Adds FlashInfer speculative decoding backend with 1.14x speedup.
    • ›Adds CUDA graph Inductor partition integration for torch.compile.
    • ›Adds optimized LoRA weight loading.
    • ›Adds blocked FP8 for MoE via compressed tensors.
    • ›Adds W4A8 faster preprocessing.
    • ›Adds FP8 torch.compile KV cache support.
    • ›Adds EPLB reduced overhead and shared expert overlap optimization for MoE.
    • ›V0 engine (AsyncLLMEngine, LLMEngine, MQLLMEngine) and all V0 attention backends have been fully removed; V1 is now the sole engine.
    └──▷ BREAKING ON UPGRADE

    • !AsyncLLMEngine, LLMEngine, and MQLLMEngine have been removed entirely — any code importing or instantiating these classes will break.
    • !All V0 attention backends have been removed; deployments relying on V0 backend selection will break.
    • !The default CUDA graph mode is now FULL_AND_PIECEWISE instead of PIECEWISE; models that only support PIECEWISE mode may need explicit configuration.
    • !C++17 is now enforced globally as a build requirement; builds using older C++ standards will fail.
    • !TPU: xm.mark_step is deprecated in favor of torch_xla.sync; code calling xm.mark_step will produce deprecation errors.
    • !max_seq_len_to_capture interface has been removed.
    • !V0 components removed: encoder-decoder support, V0 output processor, V0 sampling metadata, V0 Sequence/Sampler, V0 async output processor, MultiModalPlaceholderMap, V0 seq group methods, placeholder attention, V0 input embeddings, V0 multimodal registry, V0 attention classes, V0 hybrid model support, V0 backend suffixes, V0 compilation fallbacks, and V0 default args.
    • !--async-scheduling produces incorrect (gibberish) output in v0.11.0 under preemption and other scenarios; use v0.10.1 if this flag is required.
27. v0.10.2 Sep 13, 2025 · issue -337

    vLLM v0.10.2 adds aarch64/GB200 support, 20+ new model families, PyTorch 2.8, and major API/quantization expansions with several breaking changes.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.10.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.10.2

    └──▷ TRY IT
    Load a model using the Terratorch backend for geospatial or semantic segmentation tasks instead of standard LLM inference.
    copy

    $ vllm serve <model> --model-impl terratorch

    Speed up model weight loading when checkpoints are on a slow NFS mount.
    copy

    $ vllm serve <model> --safetensors-load-strategy <strategy>

    Install vLLM with aarch64 wheels on a GB200 or other ARM64 system.
    copy

    $ uv pip install vllm==0.10.2 --extra-index-url https://wheels.vllm.ai/0.10.2/ --torch-backend=auto

    • ›Adds --model-impl terratorch flag to enable the Terratorch backend for non-language model tasks like semantic segmentation and geospatial applications.
    • ›Adds --safetensors-load-strategy flag for accelerating model weight loading from NFS-based file systems.
    • ›Adds --log-error-stack flag to surface full error stack traces in the frontend.
    • ›Adds --mm-encoder-tp-mode flag (now formalized) to control multimodal encoder tensor-parallelism mode.
    • ›Adds VLLM_DISABLE_PAD_FOR_CUDAGRAPH environment variable to control CUDA graph padding behavior.
    +40 moreshow less

    • ›Adds FlashInfer cubin directory environment variable for offline environments and pre-cached CUDA binaries.
    • ›Adds return_token_ids parameter to the OpenAI-compatible API.
    • ›Adds EPLB configuration parameter for expert-parallel load balancing.
    • ›Adds native aarch64 support, enabling vLLM on GB200 platforms; multiplatform vllm/vllm-openai Docker image and wheels available via --extra-index-url https://wheels.vllm.ai/0.10.2/ --torch-backend=auto.
    • ›Extends V1 engine support to GPUs with compute capability below 8.0.
    • ›Adds cross-attention KV cache support for encoder-decoder models in the V1 engine.
    • ›Adds request-level logits processor integration in V1.
    • ›Adds user-configurable KV cache memory size.
    • ›Adds support for all prompt logprobs and final logprobs in sampling.
    • ›Enables full CUDA graph support by default for hybrid and Mamba models.
    • ›Adds V1 engine as the default for Mamba models.
    • ›Adds multimodal cache tracking via mm_hash and UUID-based multimodal identifiers.
    • ›Adds Apple Silicon bfloat16 support for M2+ chips.
    • ›Adds IBM Z V1 engine support (s390x).
    • ›Adds Intel XPU torch.compile, MoE data parallelism, Triton attention, and FP8 quantization support.
    • ›Adds ROCm pipeline parallelism with Ray and TorchAO quantization enablement.
    • ›Adds FP8 MLA support with CUTLASS backend for NVIDIA Blackwell/SM100 GPUs.
    • ›Adds MXFP4 fused CUTLASS MoE support, enabled by default on Blackwell.
    • ›Adds W4A8 support on Hopper GPUs.
    • ›Adds per-layer quantization routing.
    • ›Adds GGUF quantization with layer-skipping capability.
    • ›Adds NFP4+FP8 MoE support.
    • ›Adds AMD CDNA2/CDNA3 FP4 quantization support.
    • ›Adds compressed tensors transforms for linear operations, enabling SpinQuantR1R2R4 and QuIP quantization methods.
    • ›Adds LoRA support for Voxtral, Qwen-2.5-Omni, and DeepSeek V2/V3/R1-0528 models, with significantly faster LoRA startup.
    • ›Adds multi-label classification support and sigmoid normalization for pooling/classification models.
    • ›Adds Support for Decode Context Parallel (DCP) for MLA in distributed serving.
    • ›Adds new model families: Apertus, LFM2, MiDashengLM, Motif-1-Tiny, Seed-Oss, Google EmbeddingGemma-300m, GTE sequence classification, Donut OCR, KeyeVL-1.5-8B, R-4B vision, Ernie4.5 VL, MiniCPM-V 4.5, Ovis2.5, Qwen3-Next with hybrid attention, InternVL3.5 with video, Qwen2Audio embeddings, NemotronH Nano VLM, BLOOM V1, and Whisper encoder-decoder for V1.
    • ›Adds pipeline parallelism support for Hunyuan, Ovis2.5, GPT-OSS, and Kimi-VL-A3B-Thinking-2506.
    • ›Enables data parallelism for ViT across Qwen2.5VL, MiniCPM-V, Kimi-VL, and GLM-4.5V.
    • ›Adds Gemma3n audio transcription/translation endpoints to the OpenAI-compatible API.
    • ›Adds transcription response usage statistics to the API.
    • ›Adds streaming support for non-harmony responses in the Response API.
    • ›Adds non-streaming logprobs to the Response API.
    • ›Adds MCP tool background mode and streaming+background support.
    • ›Adds tool output token reporting to the Response API.
    • ›Adds collective RPC endpoint.
    • ›Adds embedding endpoint support for chat requests.
    • ›Upgrades to PyTorch 2.8.0 and FlashInfer v0.3.0.
    • ›Delivers 13.7x speedup for token conversion and symmetric memory all-reduce enabled by default.
    └──▷ BREAKING ON UPGRADE

    • !PyTorch 2.8.0 upgrade requires updated CUDA versions and environment dependency changes.
    • !FlashMLA is disabled on Blackwell GPUs due to compatibility issues.
    • !V0 Neuron backend removed.
    • !V0 pooling model support removed.
    • !V0 FlashInfer attention backend removed.
    • !Original Marlin quantization format removed.
    • !Quantized Mixtral hack implementation removed.
    • !prompt_token_ids fallback removed from LLM.generate and LLM.embed.
    • !Metrics name TPOT renamed to ITL — dashboards and alerting rules referencing TPOT will break.
28. v0.10.1 Aug 18, 2025 · issue -363

    vLLM v0.10.1 adds GPT-OSS, Command-A-Vision, mBART, Eagle multimodal, new CLI flags, reward API, and broad hardware/quantization expansions.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.10.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.10.1

    └──▷ TRY IT
    Install vLLM with FlashInfer now that it is an optional dependency, for maximum attention kernel performance.
    copy

    $ pip install vllm[flashinfer]

    • ›Replaces --task with --runner and --convert CLI options as part of V0 engine cleanup.
    • ›Renames --expand-tools-even-if-tool-choice-none to --exclude-tools-when-tool-choice-none for clearer semantics.
    • ›Deprecates --disable-log-requests in favor of --enable-log-requests.
    • ›Adds pip install vllm[flashinfer] optional-dependency install, decoupling FlashInfer from the core install.
    • ›Adds dedicated LLM.reward interface for reward models.
    +46 moreshow less

    • ›Adds PoolingParams for per-request pooling control.
    • ›Adds Unix domain socket support for local OpenAI-compatible API communication.
    • ›Adds multiple API keys support via new configuration for enhanced authentication.
    • ›Adds custom vLLM tuned configuration paths (--custom-vllm-config equivalent, #22791).
    • ›Adds environment variable control for logging statistics (#22905).
    • ›Adds multimodal cache size configuration option (#22441).
    • ›Adds V1 API support for the run-batch command.
    • ›Adds HermesToolParser for models without special tokens.
    • ›Adds request_id support for external load balancers in distributed serving.
    • ›Adds model loader plugin system for extensible model loading.
    • ›Adds rate limiting with bucket algorithm for the proxy server.
    • ›Adds tree attention backend for the V1 engine (experimental).
    • ›Adds N-gram speculative decoding with single KMP token proposal algorithm.
    • ›Adds explicit EAGLE3 interface for enhanced speculative decoding compatibility.
    • ›Adds encoder-only models without KV-cache, enabling BERT-style architectures.
    • ›Adds FlexAttention encoder-only support.
    • ›Adds multiple attention metadata builders per KV cache specification.
    • ›Adds multiple attention groups for KV sharing patterns.
    • ›Adds full CUDA graph support with FA2 and FlashInfer compatibility.
    • ›Adds CutlassMLA as the default backend for NVIDIA Blackwell (SM100).
    • ›Adds Block FP8 quantization and CUTLASS NVFP4 4-bit weights/activations support for NVIDIA RTX 5090/RTX PRO 6000 (SM120).
    • ›Adds dynamic 4-bit quantization with Kleidiai kernels for CPU inference.
    • ›Adds TensorRT-LLM FP4 quantization optimized for MoE low-latency inference.
    • ›Adds MXFP4 and bias support for the Marlin kernel.
    • ›Adds compressed-tensors mixed-precision model loading.
    • ›Adds calibration-free RTN quantization for MoE models.
    • ›Adds Flash Attention backend for Qwen-VL models on AMD ROCm.
    • ›Adds AITER HIP block quantization kernels for AMD ROCm.
    • ›Adds CPU transfer support in NixlConnector for prefill/decode disaggregation.
    • ›Adds Docker-aware precompiled wheel support for containerized deployment.
    • ›Adds multi-turn conversation benchmarking tool.
    • ›Adds optional memory profiling skip for multimodal models (#22950).
    • ›Adds enhanced hybrid distributed serving with multiple API servers in load balancing mode.
    • ›Adds chunked processing for long inputs in embedding models.
    • ›Adds custom process naming for better monitoring.
    • ›Adds new model families: GPT-OSS (with tool calling and streaming), Command-A-Vision, mBART, and SmolLM3 via Transformers backend.
    • ›Adds official Eagle multimodal support with Llama4 backend, Step3 vision-language models, Gemma3n multimodal, MiniCPM-V 4.0, Emu3 via Transformers backend, and Intern-S1.
    • ›Adds Qwen3 dual-chunk attention and EPLB support, plus native Eagle3 target support.
    • ›Adds Mamba1 and Jamba model support in V1 engine (without CUDA graphs).
    • ›Adds Ultravox support for Llama 4 and Gemma 3 backends.
    • ›Adds tensor/pipeline parallelism with Mamba2 kernel for PLaMo2.
    • ›Adds expanded tensor parallelism support in the Transformers backend.
    • ›Delivers ~6% end-to-end throughput improvement from Cutlass MLA.
    • ›Adds Triton-based multi-dimensional RoPE implementation replacing the PyTorch implementation.
    • ›Adds async tensor parallelism for scaled matrix multiplication.
    • ›Adds multithreaded async multimodal loading.
    └──▷ BREAKING ON UPGRADE

    • !The --task CLI flag is replaced by --runner and --convert; existing invocations using --task will break.
    • !The --expand-tools-even-if-tool-choice-none CLI flag is renamed to --exclude-tools-when-tool-choice-none; scripts using the old flag will break.
    • !The --disable-log-requests flag is deprecated in favor of --enable-log-requests; the old flag may no longer work.
    • !AQLM quantization support is removed; models using AQLM quantization must migrate to an alternative quantization method.
    • !V0 FlashAttention 3 (FA3) support is deprecated; FP8 KV-cache in V0 may have issues as a result.
    • !Previously deprecated API arguments and methods from the V0 engine codebase are removed (#21907).
    • !FlashInfer is moved to an optional dependency (pip install vllm[flashinfer]); environments that relied on it being installed automatically will no longer have it by default.
    • !Mamba SSM is removed from core requirements; existing setups that depend on it being included automatically will need to install it separately.
29. v0.10.0 Jul 24, 2025 · issue -364

    vLLM v0.10.0 adds async scheduling, Responses API, new model families, MXFP4/FP8 quantization, and broad hardware expansion including Blackwell and ARM.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.10.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.10.0

    └──▷ TRY IT
    Enable experimental async scheduling to overlap engine core scheduling with the GPU runner, reducing latency for concurrent requests.
    copy

    $ vllm serve Qwen/Qwen3-0.6B --async-scheduling

    Load a GGUF model directly from a HuggingFace repo URL without a local download step.
    copy

    $ vllm serve hf://bartowski/Llama-3.2-1B-Instruct-GGUF/Llama-3.2-1B-Instruct-Q4_K_M.gguf

    • ›Adds --async-scheduling flag to overlap engine core scheduling with GPU runner for experimental async scheduling.
    • ›Adds --help=page option to the CLI for enhanced paginated help documentation.
    • ›Adds get_tokenizer_info endpoint for retrieving tokenizer and chat-template information.
    • ›Adds cache_salt support for completions and responses endpoints.
    • ›Adds tokenization_kwargs for controlling embedding truncation on model-loading requests.
    +33 moreshow less

    • ›Adds logprobs mode for selecting which stage of logprobs to return (RLHF support).
    • ›Adds new RPC methods for runtime weight reloading and config updates, enabling RLHF workflows.
    • ›Introduces reproducible prefix cache hashing using SHA-256 + CBOR.
    • ›Changes default model to Qwen3-0.6B.
    • ›Adds OpenAI Responses API implementation.
    • ›Adds image object support in llm.chat.
    • ›Adds tool calling with required choice and $defs support for OpenAI compatibility.
    • ›Supports Tensorizer S3 integration with arbitrary arguments for model loading.
    • ›Supports HuggingFace repo paths and URLs for GGUF model loading.
    • ›Supports new model families: Llama 4 with EAGLE, EXAONE 4.0, Microsoft Phi-4-mini-flash-reasoning, Hunyuan V1 Dense + A13B with reasoning/tool parsing, Ling MoE, JinaVL Reranker, Nemotron-Nano-VL-8B-V1, Arcee, and Voxtral.
    • ›Adds MXFP4 quantization support for MoE models.
    • ›Adds BNB (bitsandbytes) support for Mixtral and additional MoE models.
    • ›Adds in-flight quantization for MoE models.
    • ›Adds FP8 KV cache quantization on TPU.
    • ›Adds CUTLASS block-scaled group GEMM and DeepGEMM integration for NVIDIA Blackwell/SM100.
    • ›Adds FlashInfer MoE blockscale FP8 backend and CUDNN prefill API for MLA on Blackwell.
    • ›Adds ARM CPU int8 quantization support.
    • ›Adds PPC64LE and ARM V1 engine support.
    • ›Adds Intel XPU ray distributed execution support.
    • ›Adds shared-memory pipeline parallel for CPU.
    • ›Adds FlashInfer ARM CUDA support.
    • ›Delivers 48% request duration reduction via microbatch tokenization for concurrent requests.
    • ›Adds elastic expert parallel for dynamic GPU scaling while preserving state.
    • ›Adds startup time reduction via CUDA graph capture speedup using frozen GC.
    • ›Adds multi-modal caching for the transformers backend.
    • ›Adds hybrid KV cache with local chunked attention on the V1 engine.
    • ›Adds MLA FlashInfer ragged prefill on the V1 engine.
    • ›Adds Hybrid SSM/Attention model support on V1 engine.
    • ›Adds VLM support with the transformers backend.
    • ›Adds support for models with multiple tasks, multiple poolers, and dynamic pooling parameter configuration.
    • ›Adds attention-free model support.
    • ›Updates PyTorch to 2.7.1 for CUDA builds.
    • ›Updates FlashInfer to v0.2.8rc1.
    └──▷ BREAKING ON UPGRADE

    • !V0 CPU, XPU, TPU, and HPU backends have been removed; workloads relying on those V0 backends will no longer work.
    • !Long context LoRA (V0) has been removed.
    • !Prompt Adapters have been removed.
    • !Phi3-Small and BlockSparse Attention support has been removed.
    • !V0 Spec Decode workers have been removed.
    • !Default model is changed to Qwen3-0.6B; scripts that rely on the previous default model will now load Qwen3-0.6B unless an explicit model is specified.
30. v0.9.2 Jul 7, 2025 · issue -364

    vLLM v0.9.2 adds audio translation endpoints, Expert-Parallel Load Balancer, priority scheduling in V1, and broad Blackwell/ROCm/TPU kernel upgrades.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.9.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.9.2

    └──▷ TRY IT
    Tune MoE data-parallel chunk size at runtime without code changes, useful when optimising throughput on large MoE models.
    copy

    $ MOE_DP_CHUNK_SIZE=512 vllm serve mistralai/Mixtral-8x7B-Instruct-v0.1 --tensor-parallel-size 4

    • ›Adds /v1/audio/translations endpoint and revamps /v1/audio/transcriptions for OpenAI-compatible audio support.
    • ›Adds -O/--compilation-config flag with improved parsing, batch-size-sweep benchmarking support, richer --help output, and faster startup to the CLI.
    • ›Adds Expert-Parallel Load Balancer (EPLB) for large-scale MoE serving.
    • ›Adds Priority Scheduling to the V1 engine.
    • ›Adds calibration-free RTN INT4/INT8 quantization pipeline for model compression without calibration data.
    +24 moreshow less

    • ›Adds Compressed-Tensor NVFP4 (including MoE) support with emulation mode; FP4 emulation removed on devices below SM100.
    • ›Adds MOE_DP_CHUNK_SIZE environment variable to control MoE data-parallel chunk sizing.
    • ›Adds no-privileged CPU/Docker/Kubernetes deployment mode for environments without elevated container privileges.
    • ›Adds security hardening that forbids runtime (cloud)pickle imports.
    • ›Adds image-object support in llm.chat, tool-choice expansion, and custom-arg passthroughs for multi-modal agents.
    • ›Adds token-level progress bar for LLM.beam_search and cached template-resolution speed-ups.
    • ›Adds NaN export in logits to scheduler_stats when output is corrupted.
    • ›Adds CUDA-graph live capture progress bar for debugging graph capture.
    • ›Adds full CUDA-Graph execution for all FlashAttention v3 (FA3) and FlashMLA paths including prefix-caching.
    • ›Adds full-graph capture for TritonAttention on AMD ROCm, along with quick All-Reduce and chunked pre-fill.
    • ›Adds Split-KV support to the unified Triton Attention kernel on ROCm, boosting long-context throughput.
    • ›Adds Intel GPU (V1) backend with Flash-Attention support.
    • ›Adds CUTLASS W8A8/FP8 kernels for NVIDIA Blackwell SM120 devices.
    • ›Adds block-scaled-group GEMM, INT8/FP8 vectorization, deep-GEMM kernels, activation-chunking for MoE, and group-size 64 for Machete on Blackwell SM100.
    • ›Adds support for new model families: Ernie 4.5 (+MoE), MiniMax-M1, Phi-tiny-MoE-instruct (Slim-MoE), Tencent HunYuan-MoE-V1, Keye-VL-8B-Preview, GLM-4.1 V, Gemma-3 (text-only), Tarsier 2, Qwen 3 Embedding & Reranker, dots1, and GPT-2 for Sequence Classification.
    • ›Adds embedding model support and Mamba2 support to the V1 engine.
    • ›Adds native xPyD P2P NCCL transport as base case for disaggregated PD serving without external dependencies.
    • ›Adds dynamic MoE-layer quantization for Marlin/GPTQ.
    • ›Adds FlexAttention support for any head size with FP32 fallback.
    • ›Adds TPU support for dynamic-grid KV-cache updates, head-dim less than 128, and tuned paged-attention kernels.
    • ›Adds Bits-and-Bytes 0.45+ support with improved double-quant logic and AWQ quality improvements.
    • ›Eliminates api_key and x_request_id headers middleware overhead in the API server.
    • ›Deprecates metrics with gpu_ prefix for non-GPU-specific metrics.
    • ›Hermetic builds and wheel slimming (FA2 8.0 + PTX only) reduce supply-chain surface.
    └──▷ BREAKING ON UPGRADE

    • !V0 engine code and features will be removed after this release; this is the last version where V0 stays intact — migrate to the V1 engine before upgrading beyond v0.9.2.
    • !FP4 emulation is removed on devices below SM100 (as part of Compressed-Tensor NVFP4 support).
    • !Runtime imports of (cloud)pickle are now forbidden by the security hardening; code that relied on dynamic pickle imports will break.
    • !Metrics with the gpu_ prefix are deprecated for non-GPU-specific metrics — downstream dashboards or alert rules using those metric names will need updating.
31. v0.9.1 Jun 10, 2025 · issue -365

    vLLM v0.9.1 adds CPU V1 backend, FlexAttention, run-batch CLI, rerank in batch endpoint, and broad LoRA/hardware expansions.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.9.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.9.1

    └──▷ TRY IT
    Restrict a chat completion request to a specific set of token IDs to enforce constrained generation server-side.
    copy

    $ curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "meta-llama/Llama-3-8B-Instruct", "messages": [{"role": "user", "content": "Answer yes or no."}], "allowed_token_ids": [9642, 694]}'

    • ›Adds run batch subcommand to the vLLM CLI for offline batch inference.
    • ›Adds allowed_token_ids field to ChatCompletionRequest for server-side token filtering.
    • ›Makes use_tqdm in the LLM API accept a callable for custom progress bars.
    • ›Adds rerank support to the run_batch endpoint.
    • ›Adds custom logging support for the vLLM server.
    +30 moreshow less

    • ›Adds AsyncLLMEngine.generate support for targeting a specific DP rank.
    • ›Adds FlexAttention to vLLM V1 engine.
    • ›Adds initial full support for Hybrid Memory Allocator with cross-layer KV sharing.
    • ›Adds support for inplace model weights loading for RLHF workflows.
    • ›Adds V1 engine support for the CPU backend.
    • ›Adds LoRA support to Beam Search.
    • ›Adds LoRA support for InternVL multimodal models.
    • ›Adds multi-LoRA support for Neuron hardware.
    • ›Adds Multi-Modal model support for Neuron hardware.
    • ›Adds quantization support on Neuron.
    • ›Adds multi-LoRA optimizations for the V1 TPU backend.
    • ›Adds initial SPMD model parallelism support for TPU with a single worker.
    • ›Adds Cutlass MLA backend for Blackwell (SM100) GPUs.
    • ›Enables FlashInfer by default on Blackwell GPUs.
    • ›Adds compressed-tensors NVFP4 support.
    • ›Adds CUDA kernel for applying repetition penalty (sampler performance).
    • ›Adds CUDA graph support for DP Attention + Expert Parallelism.
    • ›Adds API-server scaleout with many-to-many server-engine communications for data parallel serving.
    • ›Adds support for DP with Ray.
    • ›Enables NixlConnector FlashInfer backend.
    • ›Adds Heterogeneous Tensor Parallelism support.
    • ›Adds support for Magistral, NemotronH, and minicpm eagle models.
    • ›Adds DeepSeek-R1-0528 function call chat template.
    • ›Enables data parallel for Llama4 vision encoder.
    • ›Adds IBM POWER11 support to CPU extension detection.
    • ›Adds AITER grouped topk for DeepSeekV2 on ROCm.
    • ›Makes torch distributed process group extendable for custom platform plugins.
    • ›Adds H20-3e fused MoE kernel tuning configs for DeepSeek-R1/V3 and Qwen3-235B-A22B.
    • ›Adds benchmark_serving supports for llama.cpp backends.
    • ›Adds dataset support in vllm bench serve.
    └──▷ BREAKING ON UPGRADE

    • !Positional arguments other than model are no longer accepted when initializing LLM; all other arguments must be passed as keyword arguments.
    • !The inputs argument fallback in Engine classes has been removed.
    • !Fallbacks for the Embeddings API have been removed.
    • !The default mean pooling behavior for Qwen2EmbeddingModel has been removed; pooling must now be specified explicitly.
    • !Overriding get_dummy_text and get_dummy_mm_data is now required.
    • !Metrics that were deprecated in v0.8 have been removed.
32. v0.9.0 May 15, 2025 · issue -366

    vLLM v0.9.0 upgrades to PyTorch 2.7/CUDA 12.8, adds EAGLE3, NIXL PD support, /classify endpoint, and broad new model coverage.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.9.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.9.0

    └──▷ USE IT
    Pass custom chat template keyword arguments when running batch chat inference with LLM.chat.
    python copy

    from vllm import LLM
    llm = LLM(model='Qwen/Qwen3-8B')
    responses = llm.chat(messages=[{'role': 'user', 'content': 'Hello'}], chat_template_kwargs={'enable_thinking': True})

    • ›Adds VLLM_ATTENTION_BACKEND=FLASHINFER environment variable to enable optimized FlashInfer attention and MLP kernels on NVIDIA Blackwell GPUs.
    • ›Adds VLLM_ALLOW_INSECURE_SERIALIZATION environment variable to explicitly permit insecure serialization paths.
    • ›Adds /classify HTTP endpoint for classification tasks on the frontend.
    • ›Adds chat_template_kwargs parameter to LLM.chat for passing custom keyword arguments to chat templates.
    • ›Adds cached_tokens field to response usage payloads.
    +41 moreshow less

    • ›Adds tool_choice: required support for the Xgrammar structured-outputs backend.
    • ›Changes top_k sampling parameter: disabled with value 0 (still accepts -1 for backwards compatibility).
    • ›Sets the default random seed to 0 for the V1 Engine so repeated runs with temperature > 0 produce identical outputs.
    • ›Adds NIXL integration for Prefill-Decode (PD) disaggregated inference, including local attention optimization.
    • ›Supports multiple KV connectors simultaneously for disaggregated inference workloads.
    • ›Supports EAGLE3 speculative decoding algorithm.
    • ›Enables torch.compile and CUDA graph capture for EAGLE speculative decoding.
    • ›Adds EAGLE shared input embedding support for speculative decoding.
    • ›Enables Speculative Decoding combined with Structured Outputs.
    • ›Adds Qwen3 reasoning parser for structured outputs.
    • ›Adds Structural Tag support with Guidance backend for structured outputs.
    • ›Adds thinking compatibility for structured outputs.
    • ›Supports full CUDA graph capture in the V1 engine.
    • ›Adds Tensorizer support for fast model loading in V1 and with LoRA.
    • ›Adds MultiprocExecutor support and torchrun support for Pipeline Parallelism.
    • ›Supports sequence parallelism combined with pipeline parallelism.
    • ›Adds async tensor parallelism via compilation pass.
    • ›Adds truncation control for embedding models.
    • ›Adds KV event publishing for metrics.
    • ›Adds API for accessing in-memory Prometheus metrics.
    • ›Supports nvidia/DeepSeek-R1-FP4 quantization format.
    • ›Supports Quark MXFP4 quantization format.
    • ›Supports AutoRound quantization.
    • ›Supports torchao models with AOPerModuleConfig.
    • ›Adds CUDA Graph support for V1 GGUF quantization.
    • ›Supports cache salting to prevent side-channel attacks.
    • ›Adds default local directory LoRA resolver plugin.
    • ›Adds new models: MiMo-7B, MiniMax-VL-01, Ovis 1.6, Ovis 2, GraniteMoeHybrid 4.0, FalconH1 (requires dev transformers), LlamaGuard4.
    • ›Adds embedding models: nomic-embed-text-v2-moe, new class of GTE models.
    • ›Adds DeepSeek Function Call support.
    • ›Adds Multi-Token Prediction (MTP) in V1 for DeepSeek.
    • ›Implements dual-chunk-flash-attn backend for Qwen2.5-1M with sparse attention support.
    • ›Adds video input support for InternVL models with Qwen2.5 backbone.
    • ›Adds Multi-LoRA support on TPU.
    • ›Adds top-logprobs support on TPU.
    • ›Adds NeuronxDistributedInference support, Speculative Decoding, dynamic on-device sampling, Mistral model, and Multi-LoRA for Neuron.
    • ›Enables FP8 KV cache on AMD V1 backend.
    • ›Adds MLA support on AMD.
    • ›Adds Block-Scaled GEMM on AMD.
    • ›Upgrades default wheel from CUDA 12.4 to CUDA 12.8; CUDA 12.6 wheel distributed via GitHub artifact.
    • ›Migrates docs from Sphinx to MkDocs.
    └──▷ BREAKING ON UPGRADE

    • !PyTorch upgraded to 2.7 — existing environment dependencies must be updated; CUDA 12.4 support is removed.
    • !Default wheel now targets CUDA 12.8 (previously CUDA 12.4); environments pinned to CUDA 12.4 will break on upgrade.
    • !top_k is now disabled with 0 instead of -1; callers passing -1 still work for now but the canonical value has changed.
    • !The V1 Engine now defaults to seed 0, changing output determinism behavior for runs that previously relied on non-deterministic sampling across restarts.
33. v0.8.5 Apr 28, 2025 · issue -367

    vLLM v0.8.5 adds Qwen3/Qwen3MoE day-0 support, vllm bench CLI, structural_tag tool-calling, EAGLE-3, and a new /server_info endpoint.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.8.5 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.8.5

    └──▷ TRY IT
    Benchmark inference latency of a model directly from the CLI without writing a custom script.
    copy

    $ vllm bench latency --model meta-llama/Llama-3-8B-Instruct

    Inspect the full running vLLM configuration programmatically after server startup.
    copy

    $ curl http://localhost:8000/server_info | jq .

    Benchmark end-to-end throughput for capacity planning on a newly supported model.
    copy

    $ vllm bench throughput --model Qwen/Qwen3-8B

    • ›Adds vllm bench latency and vllm bench throughput CLI subcommands for on-demand benchmarking.
    • ›Adds structural_tag support via xgrammar for structured tool-calling in the V1 engine.
    • ›Adds /server_info API endpoint to retrieve the running vllm_config from the API server.
    • ›Adds sampling params to the v1/audio/transcriptions endpoint.
    • ›Enables dynamic LoRA loading from a remote server at runtime.
    +20 moreshow less

    • ›Adds KV Connector API V1 for disaggregated serving, plus an LMCache KV connector for V1.
    • ›Day-0 model support for Qwen3 and Qwen3MoE, including fp8 weight loading and tuned MoE configs.
    • ›Adds EAGLE-3 speculative decoding support.
    • ›Adds support for FlashInfer Attention in the V1 engine.
    • ›Adds ModernBERT model support.
    • ›Adds Granite Speech model support.
    • ›Adds PLaMo2 model support.
    • ›Adds Kimi-VL model support.
    • ›Adds Qwen2.5-Omni (thinker-only) model support.
    • ›Adds Snowflake Arctic Embed family model support.
    • ›Enables structured decoding on TPU V1.
    • ›Enables Top-P and Top-K sampling on TPU V1.
    • ›Adds Cutlass MLA support for Blackwell GPUs.
    • ›Adds AMD AITER fused MoE V1 support and integrates the AITER Paged Attention kernel and MLA.
    • ›Adds prototype sequence parallelism via compilation pass.
    • ›Adds BitBLAS (Microsoft Runtime Kernel Lib) support for low-precision computation.
    • ›Adds Triton-based rotary_emb implementation for improved performance.
    • ›Adds URL validation for multimodal content parts.
    • ›Adds property-based testing for vLLM endpoints using an OpenAPI 3.1 schema.
    • ›Adds a security guide to documentation.
    └──▷ BREAKING ON UPGRADE

    • !--enable-chunked-prefill, --multi-step-stream-outputs, and --disable-chunked-mm-input can no longer be explicitly set to False; use the --no- prefix instead (e.g., --no-enable-chunked-prefill).
34. v0.8.4 Apr 14, 2025 · issue -367

    vLLM v0.8.4 adds Qwen3, Llama4, InternVL3 support, TorchAO quantization, 3x DeepSeek MLA speedup, and structured output auto-backend.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.8.4 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.8.4

    └──▷ TRY IT
    Run benchmark_serving with custom sampling parameters to simulate realistic production traffic.
    copy

    $ python benchmarks/benchmark_serving.py --model meta-llama/Llama-3.1-8B-Instruct --dataset-name sharegpt --sampling-params '{"temperature": 0.8, "top_p": 0.95}'

    • ›Adds --max-model-len hint to the error message when KV cache memory is insufficient, showing users how to set the flag to fit their model.
    • ›Adds hf_token to EngineArgs, allowing Hugging Face authentication to be passed directly through the engine configuration.
    • ›Adds disable_chunked_mm_input argument to the V1 engine to disable partial multimodal input prefill.
    • ›Sets the structured output backend to auto by default in the V1 engine (previously required explicit selection).
    • ›Adds supports_structured_output() method to the Platform interface for V1 structured output backend negotiation.
    +19 moreshow less

    • ›Adds histogram buckets for request_latency, time_to_first_token, and time_per_output_token metrics.
    • ›Adds sampling parameters to benchmark_serving for more representative load testing.
    • ›Enables regex support with xgrammar in the V0 engine.
    • ›Supports matryoshka representation and dimensions parameter in the embedding API.
    • ›Supports TorchAO quantization for compatible models.
    • ›Adds modelopt quantization support for Mixtral models.
    • ›Enables PTPC FP8 for CompressedTensorsW8A8Fp8MoEMethod (triton fused_moe).
    • ›Supports W8A8 channel-wise weights and per-token activations in the triton fused_moe_kernel.
    • ›New merge_attn_states CUDA kernel for DeepSeek MLA delivers a 3x speedup.
    • ›Adds support for Qwen3 and Qwen3MoE models.
    • ›Adds support for SmolVLM, jinaai/jina-embeddings-v3, InternVL3, and GLM-4-0414 models.
    • ›Adds Llama4 support including chat templates for pythonic tool calling, tuned FusedMoE kernel config for Llama4 Scout (TP=8 on H100), and attention temperature tuning enabled by default for long context (>32k).
    • ›Enables multi-input by default in the V1 engine.
    • ›Adds Eagle model loading and KV cache slot support for Eagle speculative decoding heads in V1.
    • ›Enables zero-copy tensor/ndarray serialization and transmission in the V1 engine core.
    • ›Adds Intel Gaudi (HPU) multi-step scheduling implementation.
    • ›Enables @support_torch_compile for the XLA (TPU) backend.
    • ›Auto-detects bitsandbytes pre-quantized models.
    • ›Unifies engine configuration under LoadConfig and ParallelConfig via engine args.
35. v0.8.3 Apr 6, 2025 · issue -367

    vLLM v0.8.3 adds Day 0 Llama 4 support, single-node data parallel API serving, BitsAndBytes on V1, and SHA256 prefix-cache hashing.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.8.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.8.3

    • ›Adds tags parameter to the wake_up API endpoint.
    • ›Adds HTTP service metrics via the API server (#15657).
    • ›Adds BitsAndBytes quantization support in the V1 engine.
    • ›Adds Enum support for xgrammar-based structured output in V1.
    • ›Adds LoRA CPU offload support in the V1 engine.
    +21 moreshow less

    • ›Adds single-node data parallel serving with API server support in the V1 engine (AsyncLLM data parallel).
    • ›Adds Collective RPC support in the V1 engine.
    • ›Day 0 support for Llama 4 Scout and Maverick models (V1 engine only).
    • ›V1 engine now supports native sliding window attention with the hybrid memory allocator.
    • ›Adds new model support: Aya Vision, MiniMaxText01, Skywork-R1V, jina-reranker-v2.
    • ›Adds Reasoning Parser for Granite Models.
    • ›Adds Phi-4-mini function calling support.
    • ›Adds EAGLE Proposer for Speculative Decoding in V1, including speculative decoding metrics and n-gram interface update.
    • ›Adds CUTLASS grouped GEMM FP8 MoE kernel for expert parallelism.
    • ›Adds FP8 channelwise dynamic per-token GroupedGEMM support.
    • ›Adds option to use DeepGemm contiguous grouped GEMM kernel for fused MoE operations.
    • ›Supports XpYd disaggregated prefill with MooncakeStore.
    • ›Adds custom all-reduce support for ROCm (AMD GPUs).
    • ›Adds AITER integration for AMD: int8 scaled GEMM kernel and fused MoE kernel.
    • ›Adds paged attention for V1 on AMD ROCm.
    • ›Adds CPU MLA (Multi-head Latent Attention) kernel support.
    • ›TPU: adds sliding window and logit soft capping support in the paged attention kernel.
    • ›TPU: adds optimized top-p implementation that avoids scattering.
    • ›Adds middleware to log API server responses.
    • ›Adds minimum version pin for huggingface_hub to enable Xet downloads.
    • ›Adds multi-node offline DP+EP example.
36. v0.8.2 Mar 23, 2025 · issue -368

    vLLM v0.8.2 adds FP8 KV cache, guidance structured output backend, spec decode for top-p/k, and --disable-uvicorn-access-log to the V1 engine.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.8.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.8.2

    └──▷ TRY IT
    Suppress verbose per-request access logs in production deployments to reduce log noise.
    copy

    $ vllm serve meta-llama/Llama-3.1-8B-Instruct --disable-uvicorn-access-log

    • ›Adds --disable-any-whitespace option for xgrammar structured output in the V1 engine.
    • ›Adds --disable-uvicorn-access-log parameter to suppress access log output.
    • ›Adds a flag to disable cascade attention in the V1 engine.
    • ›Supports FP8 KV cache in the V1 engine (including FA3 support).
    • ›Integrates fastsafetensors loader for faster model weight loading.
    +10 moreshow less

    • ›Enables speculative decoding for top-p and top-k sampling in the V1 engine.
    • ›Supports tool calling and reasoning parser on the frontend.
    • ›Supports reset of prefix cache by specified device.
    • ›Adds pipeline parallel support to TransformersModel.
    • ›Enables CUDA graph support for Llama 3.2 Vision.
    • ›Enables Triton (ROCm) attention backend for Nvidia GPUs in the V1 engine.
    • ›Adds TPU MHA Pallas backend for the V1 engine.
    • ›Adds tensor parallel multiprocessing support for TPU in the V1 engine.
    • ›Supports the Tele-FLM model.
    • ›Adds a Kubernetes deployment guide with CPU support.
    └──▷ BREAKING ON UPGRADE

    • !OpenVINO support has been removed from core vLLM in favor of an external plugin.
37. v0.8.1 Mar 19, 2025 · issue -368

    vLLM v0.8.1 adds Zamba2 model support and LoRA for embedding models alongside V1 engine fixes.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.8.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.8.1

    • ›Adds support for Zamba2 models.
    • ›Enables LoRA support for embedding models.
    • ›Optimizes the V1 Rejection Sampler using Triton Kernels for speculative decoding.
    • ›Re-enables Gemma3 model support on the V1 engine.
38. v0.8.0 Mar 18, 2025 · issue -368

    vLLM v0.8.0 enables the V1 engine by default, adds new API endpoints, DeepSeek/MLA improvements, new models, and Blackwell GPU support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.8.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.8.0

    └──▷ TRY IT
    Force the legacy V0 engine when your model or workflow is not yet compatible with the new V1 default.
    copy

    $ VLLM_USE_V1=0 vllm serve meta-llama/Llama-3-8B-Instruct

    Check server load statistics from the new /load endpoint to inform autoscaling decisions.
    copy

    $ curl http://localhost:8000/load

    • ›V1 engine is now enabled by default; disable with VLLM_USE_V1=0 environment variable.
    • ›New /load API endpoint exposes load statistics from the API server.
    • ›New /is_sleeping API endpoint for checking server sleep state.
    • ›Enables /score endpoint for embedding models.
    • ›Adds return_tokens_as_token_ids as a request parameter to the API server.
    +33 moreshow less

    • ›Adds enable_expert_parallel argument to enable Distributed Expert Parallelism (EP) for DeepSeek models.
    • ›Adds --show-hidden-metrics-for-version CLI argument to expose hidden metrics.
    • ›Adds vllm bench CLI subcommand for benchmarking.
    • ›Adds pluggable scheduler support in the V1 engine.
    • ›Adds SupportsV0Only protocol for model definitions to opt out of V1.
    • ›Supports Structured Outputs in V1 engine.
    • ›Supports LoRA in V1 engine.
    • ›Supports ngram speculative decoding in V1 engine.
    • ›Supports reasoning output (chain-of-thought) via the API, including streaming and outlines engine integration.
    • ›Supports SSL Key Rotation in the HTTP server.
    • ›Enables streaming for the Transcription API.
    • ›Supports Image Embedding as API input.
    • ›Makes the model parameter optional in API requests.
    • ›Supports KV cache offloading and disaggregated prefill via LMCache connector, including chunked prefill.
    • ›Adds FlashMLA integration for DeepSeek MLA, including V1 support and chunked prefill.
    • ›Adds EP/TP MoE + DP Attention for DeepSeek distributed serving.
    • ›Adds MTP support for k > n_predict in DeepSeek MTP speculative decoding.
    • ›Adds streamK for block-quantized CUTLASS kernels (GEMM performance).
    • ›Adds GPTQAllSpark quantization method.
    • ›Adds Deepseek GGUF support including a MoE GGUF kernel.
    • ›Adds support for NVIDIA Blackwell nvfp4 and fp8 CUTLASS GEMM kernels.
    • ›Adds ModelOpt FP4 checkpoint support for Blackwell.
    • ›Supports new models: Gemma 3, Mistral Small 3.1, Phi-4-multimodal-instruct, Grok1, QwQ-32B (with tool calling), and Zamba2.
    • ›Adds LoRA support for TransformersModel and Gemma3ForConditionalGeneration.
    • ›Enables prefix caching by default on TPU.
    • ›Adds tensor parallel support via Ray on TPU.
    • ›Adds start_profile/stop_profile support in TPU worker.
    • ›Adds TPU multimodal model support for ragged attention.
    • ›Adds Neuron device communicator for vLLM V1.
    • ›Supports FP8 KV cache in the CPU backend.
    • ›Adds CPU inference with VXE ISA for s390x architecture.
    • ›vLLM now defaults generation_config from the model for chat template and sampling parameters such as temperature.
    • ›Updates PyTorch to 2.6.0, CUDA default to 12.4, and Ray to 2.43.
    └──▷ BREAKING ON UPGRADE

    • !The default value of seed is now None (previously a fixed integer); explicitly set seed for reproducibility.
    • !The kv_cache and attn_metadata arguments for a model's forward method have been removed; access them via forward_context on the attention backend instead.
    • !vLLM now defaults generation_config from the model for chat template and sampling parameters (e.g. temperature); existing overrides may behave differently.
    • !The metrics vllm:time_in_queue_requests, vllm:model_forward_time_milliseconds, and vllm:model_execute_time_milliseconds are deprecated and subject to removal.
39. v0.7.3 Feb 20, 2025 · issue -369

    vLLM v0.7.3 adds DeepSeek MTP, pipeline parallelism, sleep/wake endpoints, audio transcription, and broad V1 engine enhancements.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.7.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.7.3

    • ›Adds /v1/audio/transcriptions OpenAI-compatible API endpoint for audio transcription.
    • ›Adds sleep and wake-up endpoint with V1 engine support for suspending and resuming inference workers.
    • ›Adds logit_bias support to the V1 Sampler and min_p sampling support in the V1 engine.
    • ›Adds GPU prefix cache hit rate % gauge metric and iteration_tokens_total histogram plus several request timing histograms to the V1 metrics system.
    • ›Supports GPTQModel Dynamic [2, 3, 4, 8]-bit GPTQ quantization.
    +24 moreshow less

    • ›Supports Unsloth Dynamic 4-bit BnB quantization.
    • ›Supports nvfp4 quantization on NVIDIA hardware.
    • ›Adds AMD per-token-activation per-channel-weight FP8 quantization for ROCm.
    • ›Supports pipeline parallelism in the V1 engine.
    • ›Adds LoRA support to the V1 engine.
    • ›Adds logprobs and prompt logprobs support to the V1 engine.
    • ›Uses msgpack for core request serialization in the V1 engine.
    • ›Adds initial speculative decoding support with n-grams in the V1 engine.
    • ›Adds pluggable platform-specific scheduler.
    • ›Enables quantization support for the transformers backend.
    • ›Supports torch_dtype configuration in TransformersModel for the transformers backend.
    • ›Supports DeepSeek Multi-Token Prediction (MTP), delivering a 1.69x speedup in low-QPS scenarios.
    • ›Uses FlashAttention3 for MLA on Hopper GPUs.
    • ›Expands MLA to support most quantization types.
    • ›Adds initial ROCm (AMD) support to the V1 engine.
    • ›Adds TPU V1 engine support.
    • ›Enables long-context and LoRA support on Intel Gaudi.
    • ›Supports Mamba2 (Codestral Mamba) and Bamba model architectures.
    • ›Supports IBM/NASA Prithvi Geospatial model.
    • ›Supports Ultravox model v0.5.
    • ›Adds BNB support and LoRA for Qwen2.5-VL.
    • ›Reduces TTFT via concurrent partial prefills.
    • ›Adds choice-based structured output with xgrammar.
    • ›Makes vLLM compatible with veRL for RLHF colocated training.
40. v0.7.2 Feb 6, 2025 · issue -369

    vLLM v0.7.2 adds Qwen2.5-VL, a --model-impl=transformers backend, and a 43% DeepSeek throughput boost via KV cache alignment.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.7.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.7.2

    └──▷ TRY IT
    Run any Hugging Face text model that lacks a native vLLM implementation by falling back to the transformers backend.
    copy

    $ vllm serve meta-llama/Llama-3.2-1B --model-impl=transformers

    Speed up structured (grammar/JSON) decoding under high batch load by offloading logits processing to multiple threads.
    copy

    $ VLLM_LOGITS_PROCESSOR_THREADS=4 vllm serve mistralai/Mistral-7B-Instruct-v0.3 --guided-decoding-backend outlines

    • ›Adds --model-impl=transformers flag to run arbitrary Hugging Face text models through a transformers backend without a native vLLM model implementation.
    • ›Adds VLLM_LOGITS_PROCESSOR_THREADS environment variable to parallelize structured decoding and reduce latency under high batch sizes.
    • ›Adds request_success_total Prometheus counter (labelled with finish reason) for V1 engine metrics.
    • ›Adds Qwen2.5-VL vision-language model support (requires source install of Hugging Face transformers).
    • ›Enables MLA (Multi-head Latent Attention) for DeepSeek VL2 models, delivering the same KV cache compression as text-only DeepSeek V3/R1.
    +12 moreshow less

    • ›Enables DeepSeek models on ROCm/AMD GPUs.
    • ›Aligns KV cache entries to 256-byte boundaries for CUDA devices, yielding a 43% throughput improvement for DeepSeek MLA models.
    • ›Applies torch.compile to fused_moe/grouped_topk kernels, yielding a 5% throughput improvement for DeepSeek MoE models.
    • ›Enables FusedSDPA support for Intel Gaudi (HPU) hardware.
    • ›Adds BNB (bitsandbytes) quantization support for Whisper models.
    • ›Adds LoRA support for the Ultravox multimodal model.
    • ›Adds support for Sparse24Bitmask compressed model format.
    • ›Adds support for loading pure-sparsity Compressed Tensors configs.
    • ›Adds Pixtral-Large support via the Hugging Face model format using llava multimodal_projector_bias config.
    • ›Enables V1 engine support for idefics3 vision-language models.
    • ›Adds DeepSeek V3 FP8 W8A8 quantization configs for B200 GPUs.
    • ›Adds quantization and MoE configs for GH200 machines.
41. v0.7.1 Feb 1, 2025 · issue -369

    vLLM v0.7.1 delivers ~3x DeepSeek throughput via MLA kernels, FP8 block quantization, reasoning content in API, and richer V1 Prometheus metrics.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.7.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.7.1

    • ›Adds MLA decode optimization kernels for the DeepSeek model family, delivering ~3x generation throughput, ~10x token memory capacity, and horizontal context scalability via pipeline parallelism.
    • ›Integrates block-quantized CUTLASS kernels (cutlass_scaled_mm with 2D group/blockwise scaling) for DeepSeekV3 FP8 inference.
    • ›Adds FP8 Triton configs for block quantization and a fused MoE Triton kernel for GPTQ/AWQ quantization formats.
    • ›Supports reasoning content in the API for DeepSeek R1 models.
    • ›Supports overriding generation config via engine arguments.
    +9 moreshow less

    • ›Enables offline /score endpoint for embedding models.
    • ›Enables MLPSpeculator/Medusa and prompt_logprobs with ChunkedPrefill for speculative decoding.
    • ›Adds V1 engine Prometheus metrics including per-request prompt/generation token histograms, TTFT and TPOT histograms, and GPU cache usage percentage gauge.
    • ›Adds MiniCPM-o-2.6 model support (text outputs).
    • ›Adds Llama 3.2 support on AMD ROCm.
    • ›Adds NKI-based flash-attention kernel with paged KV cache for AWS Neuron.
    • ›Adds V1 support for Qwen-VL multimodal model.
    • ›Upgrades FlashInfer to 0.2.0.
    • ›Adds DeepSeek-V3 MoE tuning support on AMD.
    └──▷ BREAKING ON UPGRADE

    • !When MLA is enabled, chunked prefill and prefix caching are automatically disabled.
42. v0.7.0 Jan 27, 2025 · issue -370

    vLLM v0.7.0 ships a rewritten V1 engine, torch.compile by default, new LLM methods, Flash Attention 3, and a Rerank API

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.7.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.7.0

    └──▷ TRY IT
    Enable torch.compile optimizations on the V0 engine when V1 is not yet suitable for your deployment.
    copy

    $ vllm serve meta-llama/Llama-3.1-8B-Instruct -O3

    Use the new LLM lifecycle methods to pause a model during post-training, free GPU memory, then resume — useful in RLHF or fine-tuning loops.
    python copy

    from vllm import LLM
    llm = LLM(model='meta-llama/Llama-3.1-8B-Instruct')
    llm.sleep()   # release GPU resources between training steps
    # ... run optimizer step ...
    llm.wake_up() # restore model for next inference pass
    llm.reset_prefix_cache()  # clear stale KV cache after weight update

    • ›Enables the new V1 engine by setting the VLLM_USE_V1=1 environment variable — a fully rewritten engine focused on performance and architectural simplicity.
    • ›Adds torch.compile integration, enabled by default in V1 and toggleable via the -O3 engine parameter in the existing engine.
    • ›Adds LLM.sleep, LLM.wake_up, LLM.collective_rpc, and LLM.reset_prefix_cache methods to the LLM class for post-training framework integration.
    • ›Adds a new collective_rpc distributed abstraction for coordinating across workers.
    • ›Adds Jina- and Cohere-compatible Rerank API to the API server.
    +15 moreshow less

    • ›Adds Flash Attention 3 kernel support.
    • ›Adds native macOS Apple Silicon support.
    • ›Adds TPU support for W8A8 quantization format.
    • ›Adds x86 Multi-LoRA and MoE support.
    • ›Adds AMD MI300 FP8 format for block quantization, tuned MoE configurations, and a block size heuristic delivering an average 2.8x speedup for int8 models.
    • ›Adds support for torchrun and SPMD-style offline inference.
    • ›Adds Punica prefill kernel fusion.
    • ›Adds new generative model support: CogAgent, Deepseek-VL2, fairseq2 Llama, InternLM3, and Whisper.
    • ›Adds new pooling model support: Qwen2 PRM and InternLM2 reward models.
    • ›Adds merged multi-modal processor for VLMs, automatically supported by the V1 engine for any model implementing get_*_embeddings methods.
    • ›Adds Rank Stabilized LoRA (RSLoRA) support.
    • ›Adds LoRA support for MolmoForCausalLM.
    • ›Adds a new benchmark script for CPU offloading.
    • ›Adds out-of-tree hardware platform plugin support.
    • ›Implements Cascade Attention in the V1 engine.
43. v0.6.6.post1 Dec 27, 2024 · issue -371

    vLLM v0.6.6.post1 adds FlashInfer sampling kernels for Top-P and Top-K in V1 and restores quantized MoE support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.6.post1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.6.post1

    • ›Uses FlashInfer sampling kernel for Top-P and Top-K sampling in the V1 engine, improving sampling performance.
    • ›Reduces task switching in the V1 API server and improves abort handling for more efficient request lifecycle management.
44. v0.6.6 Dec 27, 2024 · issue -371

    vLLM v0.6.6 adds DeepSeek V3 support, online pooling API, S3 model streaming, and V1 engine milestones including VLM prefix caching.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.6 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.6

    └──▷ TRY IT
    Serve DeepSeek V3 across 8 GPUs with an 8K context window on H200 or MI300x hardware.
    copy

    $ vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 --trust-remote-code --max-model-len 8192

    • ›Adds --enable-request-id-headers flag to opt in to X-Request-ID echoing (now off by default for performance).
    • ›Adds --tensor-parallel-size flag support for serving DeepSeek V3 on 8xH200 or MI300x via vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 --trust-remote-code --max-model-len 8192.
    • ›Adds Online Pooling API for embedding/pooling workloads via the V1 frontend.
    • ›Supports streaming model weights from S3 using RunAI Model Streamer as an optional loader.
    • ›Enables loading video inputs from base64 encoding via decord in multimodal requests.
    +9 moreshow less

    • ›Adds Cutlass 2:4 sparsity kernel support combined with FP8/INT8 quantization for accelerated sparse inference.
    • ›Adds prefix caching support for vision-language models (VLMs) in the V1 engine.
    • ›Adds min-tokens, repetition, presence, and frequency penalties to the V1 sampler.
    • ›Adds Tensor Parallel Ray executor support to the V1 engine.
    • ›Supports IBM Granite 3.1 and JambaForSequenceClassification models.
    • ›Adds QVQ and QwQ to the list of supported models.
    • ›Adds PyPI index for every commit and nightly build for easier pre-release installation.
    • ›Supports automatic conversion of classification and reward models.
    • ›Adds DeepSeek V3 w8a8 FP8 block-wise quantization support.
    └──▷ BREAKING ON UPGRADE

    • !X-Request-ID header echoing is now opt-in instead of on by default; existing deployments that rely on it must add --enable-request-id-headers to re-enable it.
45. v0.6.5 Dec 17, 2024 · issue -371

    vLLM v0.6.5 adds xgrammar guided decoding, new model support (Aria, GLM-4, Cross Encoders), AWQ/GPTQ on XPU, TPU prefix caching, and a --version CLI flag.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.5 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.5

    └──▷ TRY IT
    Check the installed vLLM version directly from the CLI without starting a server.
    copy

    $ vllm --version

    • ›Adds --version flag to the vLLM CLI for quick version inspection.
    • ›Switches xgrammar as the default guided decoding backend, replacing the previous default.
    • ›Adds input_audio field support in frontend requests for audio input.
    • ›Adds automatic detection of chat content format from AST in the frontend, removing the need to manually specify chat format.
    • ›Adds AWQ and GPTQ quantization support for the XPU backend.
    +14 moreshow less

    • ›Adds GGUF quantization support on AMD ROCm.
    • ›Enables prefix caching on TPU.
    • ›Enables chunked-prefill and prefix-caching on CPU.
    • ›Adds prefix-cache aware scheduling for improved cache utilization.
    • ›Adds sliding window support to the scheduler.
    • ›Adds Machete W4A8 quantization kernel support.
    • ›Adds HQQ quantization support via Marlin kernel expansion.
    • ›Adds support for new models: Aria, Cross Encoders, GLM-4, OLMo (November 2024), Telechat2, Cohere R7B, and GritLM embeddings.
    • ›Adds LoRA support for Internlm2, glm-4v, and Pixtral-HF.
    • ›Expands torch.compile integration to cover all attention backends and encoder-based models, including dynamic FP8 fusion.
    • ›Adds ARM AARCH64 platform enablement.
    • ›Adds GH200 GPU support.
    • ›Reports multi_modal_placeholders in VLM inference output.
    • ›Adds benchmarking support for structured outputs and vision datasets, including partial prefix benchmarking and random input generation for prefix benchmarks.
    └──▷ BREAKING ON UPGRADE

    • !The default temperature for ChatCompletionRequest is changed from 0.7 to 1.0 to align with OpenAI's default.
46. v0.6.4 Nov 15, 2024 · issue -372

    vLLM v0.6.4 adds Intel Gaudi backend, V1 engine progress, torch.compile support, new LLMs/VLMs, and chat-based embeddings API.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.4 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.4

    └──▷ TRY IT
    Run a model that supports both generation and embedding in embedding mode explicitly, avoiding ambiguity when a model supports multiple tasks.
    copy

    $ vllm serve <model> --task embed

    • ›Adds user-configurable --task parameter for models that support both generation and embedding tasks.
    • ›Adds Intel Gaudi (HPU) inference backend for running vLLM on Gaudi hardware.
    • ›Adds Exllama as a backend for compressed-tensors quantization.
    • ›Adds Triton implementation scaled_mm_triton supporting fp8 and int8 SmoothQuant symmetric quantization.
    • ›Adds goodput metric support to the engine.
    +21 moreshow less

    • ›Adds stateless process group for easier integration with RLHF and disaggregated prefill workflows.
    • ›Adds tool calling parser support for Granite 3.0, Jamba, and granite-20b-functioncalling models.
    • ›Adds new LLMs and VLMs: Idefics3, H2OVL-Mississippi, Qwen2-Audio, Pixtral (HF Transformers format), FalconMamba, and Florence-2 language backbone.
    • ›Adds new encoder-only embedding models: BERT (BERTModel), RoBERTa, and XLM-RoBERTa.
    • ›Expands embedding task support to Llama, Qwen2, Qwen2-VL, VLM2Vec (Phi-3-Vision), and E5-V (LLaVA-NeXT).
    • ›Adds Math-Shepherd (Mistral reward modeling) and Qwen2 classification task support.
    • ›Adds Chat-based Embeddings API.
    • ›Adds LoRA support for Granite 3.0 MoE, Idefics3, Llama embeddings, Qwen, and Qwen2-VL.
    • ›Adds BNB quantization support for Idefics3, Mllama, Qwen2, and MiniCPMV.
    • ›Adds embedding model support for the CPU backend.
    • ›Enables combining chunked prefill with speculative decoding.
    • ›Adds ability to override HF config.json values via CLI.
    • ›Adds unified multi-modal processor for VLMs.
    • ›Adds CUDA Time Layerwise Profiler.
    • ›Adds env variable to force FlashInfer backend to enable tensor cores.
    • ›Adds LoRA request metrics tracking.
    • ›Expands torch.compile support with TorchInductor across many models.
    • ›Significant progress on V1 engine core refactor.
    • ›Benchmark throughput tool now supports image input.
    • ›Adds fused_moe performance improvements.
    • ›Supports sliding window in the Flash Attention backend.
    └──▷ BREAKING ON UPGRADE

    • !Python 3.8 is no longer supported.
    • !Block manager v1 is deprecated and block manager v2 is now the default.
47. v0.6.3.post1 Oct 17, 2024 · issue -373

    vLLM v0.6.3.post1 adds multimodal embedding support, new Mistral models, and Exllama compressed-tensors backend

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.3.post1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.3.post1

    • ›Adds Exllama as a backend for compressed-tensors quantization, expanding quantized inference options.
    • ›Adds CUDA time layer-wise profiler for per-layer GPU timing analysis.
    • ›Adds compressed-tensors INT8 W8A8 AZP support on CPU hardware.
    • ›Supports VLM2Vec, the first multimodal embedding model in vLLM, enabling vision-language embedding use cases.
    • ›Supports Ministral 3B and Ministral 8B via interleaved attention.
    +5 moreshow less

    • ›Supports multiple and interleaved images for Llama 3.2 vision models.
    • ›Supports SDPA attention for Molmo vision backbone.
    • ›Improves continuous batching for Jamba and Mamba models.
    • ›Optimizes speculative decoding ngram lookup performance.
    • ›Block manager v2 is now the default; block manager v1 is removed.
    └──▷ BREAKING ON UPGRADE

    • !Block manager v1 is removed; block manager v2 is now the default. Any configuration or workflow relying on block manager v1 will break on upgrade.
48. v0.6.3 Oct 14, 2024 · issue -373

    vLLM v0.6.3 adds Granite MoE, Mamba, GLM-4V, Molmo, NVLM-D, Molmo vision models, Llama 3.1/3.2 tool use, and VLLM_TORCH_COMPILE_LEVEL env var for full Inductor graph compilation.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.3

    └──▷ TRY IT
    Enable full Inductor graph compilation to maximize throughput when serving a model, bypassing vLLM custom ops.
    copy

    $ VLLM_TORCH_COMPILE_LEVEL=3 vllm serve meta-llama/Llama-3.1-8B-Instruct

    Skip P2P GPU connectivity checks in environments where the check hangs or incorrectly fails (e.g., certain cloud multi-GPU setups).
    copy

    $ VLLM_SKIP_P2P_CHECK=1 vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4

    • ›Adds VLLM_TORCH_COMPILE_LEVEL environment variable to control torch.compile compilation levels; setting VLLM_TORCH_COMPILE_LEVEL=3 enables Inductor full graph compilation without vLLM custom ops.
    • ›Adds VLLM_SKIP_P2P_CHECK flag to skip peer-to-peer GPU connectivity checks in distributed deployments.
    • ›Moves guided decoding params into SamplingParams, unifying the interface for structured-output control.
    • ›Makes scheduling policy settable via EngineArgs, enabling priority-based scheduling to be configured at engine startup.
    • ›Adds support for Llama 3.1 and 3.2 tool use via the tool calling interface.
    +21 moreshow less

    • ›Adds tool calling support for InternLM2.5 models.
    • ›Adds new text models: Granite MoE and Mamba.
    • ›Adds new vision-language models: GLM-4V, Molmo, and NVLM-D.
    • ›Adds reward model support for Qwen2.5-Math-RM-72B.
    • ›Adds Gemma2 embedding model support.
    • ›Adds input embeddings support for Qwen2-VL and MiniCPMV.
    • ›Adds LoRA support for MiniCPMV2.5 and MiniCPMV2.6.
    • ›Expands LoRA modules for Mixtral.
    • ›Expands bitsandbytes quantization support to Falcon, OPT, Gemma, Gemma2, and Phi models.
    • ›Adds AWQ quantization support for the CPU backend.
    • ›Adds cross-attention and encoder-decoder model support on the x86 CPU backend.
    • ›Adds async output processor for the XPU backend.
    • ›Adds on-device sampling support for Neuron.
    • ›Enables OpenVINO GPU backend support.
    • ›Enables combined multi-step scheduling, chunked prefill, and prefix caching simultaneously.
    • ›Enables CUDA Graphs for multi-step scheduling combined with chunked prefill.
    • ›Adds priority scheduling support for embedding requests and via the OpenAI-compatible API.
    • ›Extends pipeline parallelism support to remaining text and embedding models.
    • ›Adds explicit out-of-tree (OOT) interface for vLLM models including OOT embedding models.
    • ›Adds support for HuggingFace's continue_final_message parameter in the frontend.
    • ›Exposes InternVL2 max_dynamic_patch as a mm_processor_kwarg for multimodal processing control.
    └──▷ BREAKING ON UPGRADE

    • !BlockSpaceManagerV2 is now the default block manager, replacing BlockSpaceManagerV1; deployments relying on V1 behavior will use the new code path automatically on upgrade.
49. v0.6.2 Sep 25, 2024 · issue -374

    vLLM v0.6.2 adds Llama 3.2 vision support, priority scheduling, MQLLMEngine (+30% throughput), and AMD fp8 KV cache.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.2

    └──▷ TRY IT
    Serve a Llama 3.2 multimodal model with eager execution to validate the new vision model support.
    copy

    $ vllm serve meta-llama/Llama-3.2-11B-Vision-Instruct --enforce-eager --max-num-seqs 16

    • ›Serves Llama 3.2 multimodal models via vllm serve meta-llama/Llama-3.2-11B-Vision-Instruct --enforce-eager --max-num-seqs 16.
    • ›Set VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 to suppress the new beam-search deprecation error, or migrate to vllm.LLM.use_beam_search for dedicated beam search.
    • ›Introduces MQLLMEngine for the API server, boosting throughput ~30% in single-step and ~7% in multi-step scenarios.
    • ›Adds initial support for priority scheduling in the production engine.
    • ›Adds batch inference support for the llm.chat() API.
    +20 moreshow less

    • ›Exposes a revision argument in the OpenAI-compatible server frontend.
    • ›Adds LoRA lineage and base model metadata management support.
    • ›Adds prompt logprobs support in multi-step scheduling.
    • ›Adds output streaming support to multi-step + async scheduling.
    • ›Adds FlashInfer backend support for multi-step scheduling.
    • ›Adds CUDA graph support during decoding for encoder-decoder models.
    • ›Adds benchmark_serving support for sampling from Hugging Face datasets and image inputs.
    • ›Adds custom paged attention kernel for AMD ROCm.
    • ›Adds fp8 KV cache support for AMD ROCm.
    • ›Implements multi-step scheduling for TPU, using Ray as the default distributed backend.
    • ›Enables MRoPE and Qwen2-VL support on the CPU backend.
    • ›Enables 8-bit weights in Fused Marlin MoE kernel.
    • ›Adds asymmetric quantization (AQ AZP) kernels for 3-bit and 4-bit precision.
    • ›Adds tensor parallelism support with bitsandbytes quantization.
    • ›Adds bitsandbytes quantization support for Gemma2.
    • ›Adds support for Solar, MiniCPM3, and LLaVA-Onevision models.
    • ›Adds multiple-image support for Qwen-VL and pipeline parallelism for Qwen2-VL.
    • ›Adds Mistral function calling format support across all models loaded with the 'mistral' format.
    • ›Adds IQ1_M quantization implementation to the GGUF kernel.
    • ›Binds ZMQ socket to 127.0.0.1 only for local-only API server usage, reducing network exposure.
    └──▷ BREAKING ON UPGRADE

    • !Using beam search as a sampling parameter now raises an error; set VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1 to suppress, or migrate to vllm.LLM.use_beam_search.
50. v0.6.1.post2 Sep 13, 2024 · issue -374

    vLLM v0.6.1.post2 adds support for custom torch.compile backends via the plugin system.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.1.post2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.1.post2

    • ›Adds plugin support for custom torch.compile backends, allowing operators to register their own compilation strategies.
51. v0.6.1.post1 Sep 13, 2024 · issue -374

    vLLM v0.6.1.post1 adds multi-image Qwen-VL support, FlashInfer multi-step backend, delta-only engine output, and bitsandbytes for Gemma2.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.1.post1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.1.post1

    • ›Adds engine option to return only deltas or final output (rather than full accumulated text on each step).
    • ›Adds FlashInfer backend support for multi-step scheduling.
    • ›Adds bitsandbytes quantization support for Gemma2 models.
    • ›Supports multiple images for Qwen-VL models.
    • ›Removes the engine_use_ray engine option.
    └──▷ BREAKING ON UPGRADE

    • !The engine_use_ray option has been removed; usages must be dropped on upgrade.
    • !Chunked scheduling is disabled by default for multimodal (vision) models; replace --max_num_batched_tokens 16384 with --max-model-len 16384 in existing invocations.
52. v0.6.1 Sep 11, 2024 · issue -374

    vLLM v0.6.1 adds Pixtral, Qwen2-VL, and Llava-Next-Video support, LoRA hot-swap via API, and 2x AWQ throughput.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.1

    └──▷ TRY IT
    Benchmark serving latency with log-probability output to measure token-level confidence at scale.
    copy

    $ python benchmarks/benchmark_serving.py --model mistralai/Mistral-7B-v0.1 --logprobs 5 --dataset-name sharegpt --num-prompts 200

    Benchmark serving with prefix caching enabled to simulate realistic repeated-prompt workloads.
    copy

    $ python benchmarks/benchmark_serving.py --model mistralai/Mistral-7B-v0.1 --enable-prefix-caching --num-prompts 500

    • ›Adds --logprobs argument to benchmark_serving.py for log-probability output during serving benchmarks.
    • ›Enables random prefix caching in benchmark_serving.py serving profiling tool.
    • ›Supports dynamic LoRA load and unload via the API server without restart.
    • ›Adds progress reporting to the batch runner (run_batch.py).
    • ›Adds support for NVIDIA ModelOpt static scaling checkpoints.
    +11 moreshow less

    • ›Supports compressed-tensor W8A8 quantization for the Intel CPU backend.
    • ›Adds Pixtral (mistralai/Pixtral-12B-2409) multimodal model support, including loading from original Mistral format.
    • ›Adds Qwen2-VL model support.
    • ›Adds Qwen-VL / Qwen-VL-Chat multimodal model support.
    • ›Adds Llava-Next-Video model support.
    • ›Adds multi-image input support for LLaVA models.
    • ›Adds multi-image input support for InternVL2 models, including pipeline parallelism.
    • ›Memory optimization for awq_gemm and awq_dequantize kernels delivers 2x throughput improvement.
    • ›Keeps chunked prefill enabled by default with long context when prefix caching is enabled.
    • ›Adds Fused MoE Marlin support for GPTQ models.
    • ›Docker image updated to Python 3.12 for a performance bump.
    └──▷ BREAKING ON UPGRADE

    • !SqueezeLLM quantization support has been removed.
53. v0.6.0 Sep 4, 2024 · issue -374

    vLLM v0.6.0 delivers 2x throughput, multi-step scheduling, Tools API streaming, and new model/hardware support

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.6.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.6.0

    └──▷ TRY IT
    Maximize throughput on a high-end GPU by enabling multi-step scheduling — useful for batch inference workloads where latency per step matters less than overall tokens/sec.
    copy

    $ vllm serve meta-llama/Meta-Llama-3-8B-Instruct --num-scheduler-steps 8

    Serve a Mistral model with its native tokenizer mode for correct chat encoding and improved robustness, especially for instruction-tuned variants.
    copy

    $ vllm serve mistralai/Mistral-7B-Instruct-v0.3 --tokenizer-mode mistral

    Run a quantized model with bitsandbytes FP4 to serve large models on memory-constrained GPUs.
    copy

    $ vllm serve meta-llama/Meta-Llama-3-70B-Instruct --quantization bitsandbytes --load-format bitsandbytes

    • ›Enables multi-step scheduling via --num-scheduler-steps engine argument (e.g. --num-scheduler-steps 8) to unlock up to 2x throughput versus v0.5.3, with log_probs support included.
    • ›Adds json_schema support from the OpenAI protocol for structured output control.
    • ›Adds OpenAI-Compatible Tools API with streaming support for Hermes and Mistral models.
    • ›Enables chunked prefill and prefix caching simultaneously, previously mutually exclusive.
    • ›Publishes Prometheus metrics in the run_batch API.
    +18 moreshow less

    • ›Adds support for bitsandbytes 8-bit and FP4 quantized models.
    • ›Adds a new tokenizer mode for Mistral models using the native mistral-commons package.
    • ›Adds async output processing that overlaps output data-structure construction with GPU work, delivering a 12% throughput increase.
    • ›Uses FlashInfer backend for FP8 KV Cache when available.
    • ›Uses FlashInfer backend for rejection sampling in Speculative Decoding.
    • ›Adds tensor parallelism (TP) support for Vision Transformers (ViTs).
    • ›Adds multi-image input support for LLaVA-Next and Phi-3-vision models.
    • ›Adds multimodal support in offline chat and support for multiple multi-modal items in the OpenAI frontend.
    • ›Adds Ultravox support for multiple audio chunks.
    • ›Supports benchmarking the async engine in benchmark_throughput.py.
    • ›Adds new LLM models: Exaone, Granite, and Phi-3.5-MoE.
    • ›Extends CUDA graph size for H200 GPUs.
    • ›Adds Triton implementations of awq_dequantize and awq_gemm for AMD ROCm AWQ support.
    • ›Adds pipeline parallel support for Intel GPUs.
    • ›Adds context-length and token-generation bucket support for Neuron.
    • ›Adds support for single and multi-host TPUs on GKE, plus async output processing for TPU.
    • ›Adds more percentile and latency metrics to benchmark output.
    • ›Migrates mamba_ssm and causal_conv1d kernels into vLLM, removing external kernel dependencies.
54. v0.5.5 Aug 23, 2024 · issue -375

    vLLM v0.5.5 adds multi-step GPU scheduling, audio model support, GGUF loading with tensor parallelism, and new entrypoint APIs.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.5 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.5.5

    └──▷ TRY IT
    Reduce CPU scheduling overhead by pipelining 8 GPU steps ahead — useful for maximizing single-GPU throughput on large models.
    copy

    $ vllm serve meta-llama/Meta-Llama-3-8B --num-scheduler-steps 8

    Run chat-style inference directly from the LLM class without spinning up an HTTP server.
    python copy

    from vllm import LLM
    llm = LLM(model='meta-llama/Meta-Llama-3-8B-Instruct')
    outputs = llm.chat([{'role': 'user', 'content': 'Explain zero-day vulnerabilities.'}])
    print(outputs[0].outputs[0].text)

    • ›Adds --num-scheduler-steps parameter to vllm serve and AsyncLLMEngine to schedule multiple GPU steps in advance, delivering ~20% QPS improvement for single-GPU 8B and 30B models.
    • ›Adds a chat method to the LLM class for direct chat-style inference without going through the HTTP server.
    • ›Adds prompt_logprobs support in the Chat Completion API.
    • ›Supports embeddings in the run_batch API.
    • ›Supports loading GGUF models with tensor parallelism.
    +17 moreshow less

    • ›Adds support for the first audio model, UltravoxModel.
    • ›Supports Jamba 1.5 models.
    • ›Supports image embeddings as input for vision models.
    • ›Supports SigLIP encoder and alternative decoders for LLaVA models.
    • ›Supports serving encoder/decoder models with cross-attention architecture.
    • ›Adds EAGLE speculative decoding implementation with Top-1 proposer.
    • ›Uses target model max length as default for draft model in speculative decoding.
    • ›Adds W8A16 load-time quantization for TPU backend.
    • ›Adds fp8 Linear Layer support for AMD ROCm.
    • ›Expands MoE weight loading and adds Fused Marlin MoE Kernel.
    • ›Adds Machete Hopper-optimized mixed precision linear kernel.
    • ›Registers custom ops for torch.compile kernel support.
    • ›Adds span metrics for model_forward, scheduler, and sampler time.
    • ›Optimizes prefix caching performance.
    • ›Uses flashinfer sampling kernel when available, yielding ~7% decoding throughput speedup.
    • ›Reduces Python allocations for ~24% throughput speedup.
    • ›Supports multi-host inference on TPU with optimized RoPE.
55. v0.5.4 Aug 5, 2024 · issue -375

    vLLM v0.5.4 adds new models, W4A8 quantization, guided decoding for offline LLM, ZeroMQ-based server split, and TPU/Intel parallelism.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.4 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.5.4

    • ›Adds support for guided decoding for offline LLM inference.
    • ›Separates the OpenAI server's HTTP request handling from the model inference loop using zeromq, delivering a 20% speedup in time-to-first-token and 2x improvement in inter-token latency.
    • ›Adds W4A8 quantization support.
    • ›Adds support for reading bitsandbytes (bnb) pre-quantized models.
    • ›Tunes FP8 and INT8 kernels for Ada Lovelace and SM75 T4 GPUs.
    +11 moreshow less

    • ›Adds FlashInfer support in DraftModelRunner for speculative decoding.
    • ›Adds observability enhancements for speculative decoding.
    • ›Adds pipeline parallelism support for DeepSeek v2, Qwen, Qwen2, and Nemotron models.
    • ›Adds vision language model support for InternVL2 and BLIP-2.
    • ›Adds support for MiniCPM-V vision language model.
    • ›Adds H2O Danube3-4b model.
    • ›Adds Nemotron model family (Nemotron-3, Nemotron-4, Minitron).
    • ›Enables TPU collective communication and tensor parallelism (TP) for the async engine with reduced compile time.
    • ›Enables multiprocessing and tensor parallelism on the Intel CPU backend.
    • ›Delivers 15% throughput improvement in large-batch scenarios by using Python's native array data structure for padding.
    • ›Reduces log-probability computation latency from ~30ms to ~5ms in large-batch scenarios when logprobs=None.
56. v0.5.3 Jul 23, 2024 · issue -376

    vLLM v0.5.3 adds Llama 3.1, Mistral-Nemo, and Chameleon support, a new vllm CLI, --cpu-offload-gb, and AWQ-Marlin performance gains.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.5.3

    └──▷ TRY IT
    Start an OpenAI-compatible serving endpoint using the new serve subcommand.
    copy

    $ vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct

    • ›Adds --cpu-offload-gb flag to control how much GPU memory to extend into RAM via CPU offloading.
    • ›Adds new vllm CLI with three subcommands: serve, complete, and chat.
    • ›Supports chat completions input in the tokenize endpoint.
    • ›Adds AWQ support to the Marlin kernel, delivering 1.5–2x performance improvements for existing AWQ models.
    • ›Supports Meta Llama 3.1, runnable on a single 8xH100 or 8xA100 node with FP8 quantization, or across multiple nodes with BF16 via pipeline parallelism.
    +11 moreshow less

    • ›Enables chunked prefill by default for the Meta Llama 3.1 model series to support long context.
    • ›Adds a new rope extension method to support long-context workloads.
    • ›Supports dynamic loading of LoRA adapters from HuggingFace at runtime.
    • ›Adds pipeline parallel support for Mixtral.
    • ›Adds support for Mistral-Nemo.
    • ›Adds initial support for Chameleon (multimodal model).
    • ›Enables pipeline parallelism via the stdlib multiprocessing module as an alternative to Ray.
    • ›Adds InfiniBand shared-object support to wheels so multi-node serving with InfiniBand works out of the box.
    • ›Adds SPMD worker execution using Ray accelerated DAG.
    • ›Enables MoE with the Pallas GMM kernel on TPU.
    • ›Wheels now build on Ubuntu 20.04 instead of 22.04, broadening compatibility.
    └──▷ BREAKING ON UPGRADE

    • !The Attention.kv_scale field is replaced by separate k_scale and v_scale fields; any code or configuration referencing kv_scale directly will break.
57. v0.5.2 Jul 15, 2024 · issue -376

    vLLM v0.5.2 adds a new CLI, PaliGemma/Fuyu-8B support, Medusa speculative decoding, soft-tuned prompts, and cross-Python wheels.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.5.2

    • ›New experimental vllm CLI for serving and querying an OpenAI-compatible server without writing Python.
    • ›Adds support for PaliGemma and Fuyu-8B multimodal models.
    • ›Adds Medusa speculative decoding with a Top-1 proposer, enabling faster inference with draft-model-free speculation.
    • ›Adds soft-tuned prompt (prompt adapter) insertion support for customizing model behavior without fine-tuning weights.
    • ›Releases a Python-version-agnostic wheel so a single build installs across all supported Python versions.
    +5 moreshow less

    • ›Enables bonus tokens in speculative decoding for KV-cache-based models, improving output quality under speculation.
    • ›Adds ZeroMQ fallback for broadcasting large objects in distributed inference, unblocking large-model multi-node use cases.
    • ›Adds FP8 support via llm-compressor and CUTLASS kernels for FP8 layers with bias.
    • ›Exposes vllm.__commit__ attribute carrying the git commit hash for build traceability.
    • ›Adds deprecation warning when beam search is enabled, signaling planned removal in a future release.
58. v0.5.1 Jul 5, 2024 · issue -376

    vLLM v0.5.1 adds pipeline parallelism, Gemma 2/Jamba/DeepSeek-V2/Phi3-Vision support, OpenVINO backend, and speculative decoding improvements.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.5.1

    └──▷ TRY IT
    Run the vLLM API server across 4 pipeline stages to serve a large model that doesn't fit on a single GPU.
    copy

    $ python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-70B --pipeline-parallel-size 4

    • ›Adds --pipeline-parallel-size flag to the API server to enable pipeline parallelism across multiple GPUs (early stage).
    • ›Adds MLPSpeculator and Typical Acceptance Sampler to speculative decoding, plus a new Draft Model Runner.
    • ›Adds support for sharded tensorized models via the frontend and core engine.
    • ›Adds continuous streaming of OpenAI response token stats.
    • ›Adds support for FlashInfer as an optional backend for prefill, decode, and CUDA Graph for decode.
    +12 moreshow less

    • ›Adds w4a16 and Marlin 24 support to compressed-tensors quantization.
    • ›Adds a faster vectorized FP8 quantize kernel, with FP8 support extended to Ampere GPUs.
    • ›Adds OpenVINO inference backend.
    • ›Adds Intel GPU (XPU) initial inference backend.
    • ›Adds CPU inference support with AVX2 ISA.
    • ›Adds support for Gemma 2 (recommended with FlashInfer backend v0.0.8 for logits soft cap correctness).
    • ›Adds support for Jamba, vLLM's first state space model.
    • ›Adds support for DeepSeek-V2.
    • ›Adds support for Phi3-Vision, dynamic image size, and a VLM input registry; VLM prompts now use <image> directly instead of complex formatting.
    • ›Enhances TPU backend support.
    • ›Enhances distributed communication performance via shared memory.
    • ›Enhances block manager latency.
    └──▷ BREAKING ON UPGRADE

    • !All VLM-specific arguments are removed from engine APIs; they can no longer be set globally via CLI. Prompts must now pass <image> instead of the previous complex prompt formatting.
59. v0.5.0.post1 Jun 14, 2024 · issue -377

    vLLM v0.5.0.post1 adds initial TPU support, sharded tensorized models, w4a16 compressed-tensors, and AVX2 CPU inference.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.0.post1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.5.0.post1

    • ›Adds w4a16 quantization support for compressed-tensors kernel, enabling 4-bit weight inference on supported hardware.
    • ›Supports sharded tensorized models via frontend and core changes, allowing pre-sharded model loading across distributed backends.
    • ›Supports CPU inference with AVX2 ISA on Intel hardware, broadening the range of deployable host CPUs.
    • ›Adds initial TPU integration, enabling vLLM to run inference on Google TPU hardware.
    • ›Adds a vectorized FP8 quantize kernel for improved quantization throughput.
    +1 moreshow less

    • ›Adds 'input speed' metric to the tqdm progress postfix alongside existing output speed, giving real-time visibility into both throughput directions during batch inference.
60. v0.5.0 Jun 11, 2024 · issue -377

    vLLM v0.5.0 adds FP8 inference (1.5x speedup), OpenAI Vision API, Speculative Decoding, bitsandbytes/QLoRA, and multi-LoRA generation.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.5.0

    └──▷ TRY IT
    Stream a chat completion with per-token usage stats using the new stream_options field.
    copy

    $ curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "llava-hf/llava-1.5-7b-hf", "messages": [{"role": "user", "content": "Hello"}], "stream": true, "stream_options": {"include_usage": true}}'

    • ›Adds stream_options field to ChatCompletionRequest in the OpenAI protocol, enabling streaming options on chat completions.
    • ›Adds add_special_tokens field to ChatCompletionRequest (default False) for controlling special-token injection in the OpenAI API server.
    • ›Returns max_model_len in the /v1/models response, giving clients visibility into the model's context length limit.
    • ›Enables passing multiple LoRA adapters at once to generate() in a single call.
    • ›Supports named functions in the OpenAI tools API, allowing structured tool-call requests by function name.
    +10 moreshow less

    • ›Adds FP8 quantization support (via CUTLASS kernels) delivering ~1.5x inference speedup on supported hardware.
    • ›Adds OpenAI Vision API support for LLaVA and LLaVA-NeXT models via the API server.
    • ›Adds Speculative Decoding support (ready for testing, planned to be on by default in a future release).
    • ›Adds Automatic Prefix Caching support (ready for testing, planned to be on by default in a future release).
    • ›Adds bitsandbytes quantization and QLoRA support.
    • ›Adds Dynamic Per-Token Activation Quantization kernel.
    • ›Adds customizable RoPE theta, enabling control over rotary position embedding frequency.
    • ›Defaults to the multiprocessing backend for single-node distributed inference.
    • ›Updates Outlines integration from FSM to Guide for structured output generation.
    • ›Adds CUDA Graph output buffer to reduce memory footprint during inference.
    └──▷ BREAKING ON UPGRADE

    • !FP8 checkpoint format field renamed from act_scale to input_scale; existing FP8 checkpoints using the old field name will break.
61. v0.4.3 Jun 1, 2024 · issue -377

    vLLM v0.4.3 adds Falcon, Granite, Arctic, and Phi-3-Small model support, OpenAI Batch API, embedding API, TP>1 speculative decoding, and GPTQ Marlin 2:4 sparse quantization.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.4.3

    └──▷ TRY IT
    Run vLLM without Ray by using the multiprocessing-based GPU executor for multi-GPU serving.
    copy

    $ python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-7b-hf --tensor-parallel-size 2 --distributed-executor-backend mp

    Run speculative decoding across multiple tensor-parallel ranks (TP>1) to accelerate generation on large models.
    copy

    $ python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-70b-hf --tensor-parallel-size 4 --speculative-model meta-llama/Llama-2-7b-hf --num-speculative-tokens 5

    • ›Adds MultiprocessingGPUExecutor to make Ray an optional dependency for multi-GPU inference.
    • ›Supports OpenAI batch file format via the Batch API, enabling offline batch inference workloads.
    • ›Adds initial Embedding API support with e5-mistral-7b-instruct as the first supported model.
    • ›Adds GPTQ Marlin 2:4 sparse structured quantization support.
    • ›Adds initial Activation Quantization support.
    +21 moreshow less

    • ›Loads FP8 kv-cache scaling factors from checkpoints.
    • ›Adds w8a8 CUTLASS kernels for FP8 quantized inference.
    • ›Supports CUTLASS kernels inside CUDA graphs.
    • ›Supports CUDA graph acceleration for speculative decoding.
    • ›Enables tensor-parallel (TP>1) speculative decoding.
    • ›Adds support for Falcon models.
    • ›Adds support for IBM Granite Code models.
    • ›Adds blocksparse flash attention kernel with support for the Phi-3-Small model.
    • ›Adds Snowflake Arctic model implementation.
    • ›Supports Dynamic RoPE scaling.
    • ›Supports long-context LoRA.
    • ›Adds cross-attention KV caching and memory management toward encoder-decoder model support.
    • ›Adds Punica kernel support on AMD GPUs.
    • ›Adds fused MoE Triton configs for AMD GPUs.
    • ›Automatically detects SparseML models for quantization.
    • ›Automatically detects vLLM-tensorized models, removing the need for manual configuration.
    • ›Adds Pipeline Parallelism support via PyNCCL groups (PP PyNCCL Groups).
    • ›Accelerates automatic prefix caching hashing by avoiding deep copies.
    • ›Improves n-gram speculative decoding efficiency.
    • ›Adds get_name method to attention backends for programmatic backend identification.
    • ›Re-enables custom roles in the Chat Completions API.
62. v0.4.2 May 5, 2024 · issue -378

    vLLM v0.4.2 adds chunked prefill, ngram speculative decoding, FlashInfer backend, and Phi-3 support

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.4.2

    └──▷ TRY IT
    Control API server log verbosity at startup to reduce noise in production or increase detail for debugging.
    copy

    $ python -m vllm.entrypoints.api_server --model meta-llama/Llama-2-7b-hf --log-level warning

    • ›Adds --log-level option to the API server for runtime log verbosity control.
    • ›Adds chunked prefill support (ready for testing) to improve inter-token latency under high load by chunking prompt processing and prioritizing decode.
    • ›Adds ngram prompt lookup decoding for speculative decoding via the [Speculative decoding] ngram proposer.
    • ›Adds logprobs support for speculative decoding.
    • ›Adds FlashInfer as a selectable attention backend.
    +12 moreshow less

    • ›Adds support for Phi-3-mini models.
    • ›Adds full tensor parallelism for LoRA layers.
    • ›Expands Marlin kernel to support all GPTQ models, including AutoGPTQ and 8-bit GPTQ models.
    • ›Supports FP8 checkpoints (both dynamic and static) in the kernel layer.
    • ›Supports complex message content (e.g. multi-part messages) for the chat completions endpoint.
    • ›Supports dynamic num_readers configuration for Tensorizer.
    • ›Adds more Prometheus histogram metrics for monitoring.
    • ›Enables prefix caching with block manager v2.
    • ›Allows users to define a custom whitespace pattern for Outlines-based structured generation.
    • ›Centralizes and documents all environment variables for easier configuration discovery.
    • ›Upgrades to torch==2.3.0.
    • ›Upgrades to tensorizer==2.9.0.
63. v0.4.1 Apr 24, 2024 · issue -379

    vLLM v0.4.1 adds Meta Llama 3, CommandR+, Mixtral 8x22B, Intel CPU backend, tensorizer loading, LM Format Enforcer guided decoding, and FP8 dynamic scaling.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.4.1

    └──▷ TRY IT
    Run inference on an Intel CPU without a GPU by targeting the CPU backend.
    copy

    $ python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-8B --device cpu

    • ›Adds tensorizer as a model-loading backend via new load option, enabling fast deserialization of model weights.
    • ›Adds LM Format Enforcer as an option for guided decoding alongside the existing backend.
    • ›Makes detokenization and tokenizer initialization optional, allowing lighter-weight engine deployments.
    • ›Adds Intel CPU inference backend, enabling vLLM to run inference on CPU hardware without a GPU.
    • ›Adds initial support for dynamic per-tensor scaling via FP8, enabling quantized inference with FP8 precision.
    +10 moreshow less

    • ›Supports private/out-of-tree model registration, letting users register custom model architectures without upstreaming.
    • ›Adds support for new models: CommandR+, MiniCPM, Meta Llama 3, and Mixtral 8x22B.
    • ›Adds LoRA support on quantized models.
    • ›Adds prompt token truncation option to the completions API.
    • ›Supports new autogptq checkpoint_format, broadening GPTQ model compatibility.
    • ›Upgrades to PyTorch 2.2.1 and Triton 2.2.0.
    • ›AMD ROCm backend gains Triton kernel for default Flash Attention and e4m3fn FP8 KV cache support.
    • ›Enables hf_transfer by default when available, accelerating model downloads from Hugging Face Hub.
    • ›Progress toward chunked prefill scheduler (end-to-end working path included).
    • ›Progress toward speculative decoding, including lookahead scheduling and configuration object.
64. v0.4.0.post1 Apr 2, 2024 · issue -379

    vLLM v0.4.0.post1 adds Intel CPU inference backend, lookahead scheduling for speculative decoding, and new AutoGPTQ checkpoint format support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.0.post1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.4.0.post1

    • ›Adds Intel CPU inference backend, enabling vLLM inference on CPU without a GPU.
    • ›Adds lookahead scheduling for speculative decoding, improving throughput for speculative decode workloads.
    • ›Adds support for the new AutoGPTQ checkpoint_format, broadening quantized model compatibility.
    • ›Restores sm70/sm75 (Volta/Turing GPU) binary support dropped in v0.4.0.
65. v0.4.0 Mar 30, 2024 · issue -380

    vLLM v0.4.0 adds automatic prefix caching, LLaVA vision support, five new models, mTLS, and JSON-mode structured output

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.4.0

    └──▷ TRY IT
    Serve a model with automatic prefix caching enabled to speed up repeated long system prompts across API requests.
    copy

    $ python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-2-70b-chat-hf --enable-prefix-caching

    Request structured JSON output from the OpenAI-compatible server, useful for tool-calling or data-extraction pipelines.
    copy

    $ curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "meta-llama/Llama-2-70b-chat-hf", "response_format": {"type": "json_object"}, "messages": [{"role": "user", "content": "Return a JSON object with fields name and age."}]}'

    Run the server with a custom log level and mTLS to harden a production deployment.
    copy

    $ python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-v0.1 --log-level warning

    • ›Enables automatic prefix caching via --enable-prefix-caching flag, caching long system prompts across requests to reduce redundant computation.
    • ›Adds --log-level flag to control vLLM log verbosity instead of the previously fixed info level.
    • ›Adds mTLS support via new server arguments (PR #3430).
    • ›Adds min_tokens parameter for EOS suppression in the OpenAI-compatible server.
    • ›Supports json_object response format in the OpenAI-compatible server for arbitrary JSON output and Context Free Grammar-constrained generation.
    +16 moreshow less

    • ›Adds LLaVA as the first supported vision-language model.
    • ›Adds new text models: Command R, Qwen2 MoE, DBRX, XVerse, and Jais.
    • ›Adds LoRA support for Qwen2 models.
    • ›Adds GPTQ quantization support for Gemma models.
    • ›Adds asynchronous tokenization to reduce CPU-side latency.
    • ›Adds chat templates for Falcon and ChatGLM models.
    • ›Adds progress on chunked prefill scheduler and speculative decoding (worker for speculate, score, and rejection sampling).
    • ›Adds CMake-based build system for extensibility and custom kernel integration.
    • ›Improves AWS Inferentia (Neuron) support, including Mistral model inference via transformers-neuronx.
    • ›Re-enables custom all-reduce kernel after robustness improvements.
    • ›Adds engine health check connected to the OpenAI-compatible server endpoint.
    • ›Replaces cupy dependency with an alternative due to upstream bugs.
    • ›Moves model file locks from /tmp/ to ~/.cache/vllm/locks/ for better isolation.
    • ›Adds usage statistics collection.
    • ›Adds GeGLU kernel with approximate GELU for improved MoE model performance.
    • ›Adds batched RoPE kernel.
    └──▷ BREAKING ON UPGRADE

    • !Model file locks are now stored in ~/.cache/vllm/locks/ instead of /tmp/; any tooling that referenced the old path will need to be updated.
66. v0.3.3 Mar 1, 2024 · issue -380

    vLLM v0.3.3 adds StarCoder2, Orion model support, 2/3/8-bit GPTQ, Marlin Int4 kernels, and experimental structured output for the OpenAI server.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.3.3

    • ›Adds experimental structured output (JSON, Regex) via guided decoding for the OpenAI-compatible API server.
    • ›Adds logit bias support for the OpenAI API.
    • ›Adds log probabilities for Chat Completions in the OpenAI-compatible API.
    • ›Adds cache_config information to Prometheus metrics.
    • ›Supports 2/3/8-bit GPTQ quantization models.
    +10 moreshow less

    • ›Integrates Marlin kernels for Int4 GPTQ inference, accelerating quantized throughput.
    • ›Adds LoRA support for Gemma.
    • ›Supports the StarCoder2 architecture.
    • ›Supports the Orion model architecture.
    • ›Adds experimental AWS Inferentia2 support via transformers-neuronx.
    • ›Enables GQA (Grouped Query Attention) support in prefix prefill kernels.
    • ›Adds early stopping to completion APIs.
    • ›Optimizes the fused MoE (Mixture-of-Experts) kernel for improved performance.
    • ›Optimizes the GeGLU layer in Gemma for faster inference.
    • ›Migrates metrics from aioprometheus to prometheus_client (requires prometheus_client >= 0.18.0).
67. v0.3.2 Feb 21, 2024 · issue -381

    vLLM v0.3.2 adds OLMo and Gemma model support, per-request seed, multi-LoRA, and ROCm gfx908 coverage.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.3.2

    • ›Adds seed parameter support for per-request reproducible sampling control.
    • ›Adds --code-revision config argument for pinning a specific revision of a model on Hugging Face Hub.
    • ›Supports multi-LoRA adapters as extra models in the OpenAI-compatible server.
    • ›Adds metrics to RequestOutput objects for programmatic inspection of per-request stats.
    • ›Makes vLLM logging formatting optional (configurable).
    +3 moreshow less

    • ›Adds support for OLMo models.
    • ›Adds support for Gemma models.
    • ›Extends ROCm support to include the gfx908 GPU architecture.
68. v0.3.1 Feb 16, 2024 · issue -381

    vLLM v0.3.1 adds LoRA for Mixtral, InternLM2 support, request-level Prometheus metrics, and ROCm gfx1100 (Radeon 7900) inference.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.3.1

    • ›Adds request-level metrics via a refactored Prometheus integration.
    • ›Adds LoRA support for Mixtral models.
    • ›Adds InternLM2 model support.
    • ›Adds a fused top-K softmax kernel for MoE models, improving mixture-of-experts inference efficiency.
    • ›Adds support for ROCm on Radeon 7900 series (gfx1100) without requiring flash-attention.
    +2 moreshow less

    • ›Removes EOS tokens from output by default.
    • ›Enables setting local logging level via environment variable.
    └──▷ BREAKING ON UPGRADE

    • !The Yi model definition is removed; users must switch to LlamaForCausalLM instead.
    • !Custom all-reduce is disabled by default.
    • !Ray compiled DAG integration is disabled by default.
69. v0.3.0 Jan 31, 2024 · issue -382

    vLLM v0.3.0 adds multi-LoRA, prefix caching, FP8 KV cache, API token auth, batch completions, and DeepSeek MoE support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.3.0

    • ›Adds include_stop_str_in_output and length_penalty parameters to the OpenAI-compatible API.
    • ›Adds simple API token authentication to the OpenAI API server.
    • ›Supports FP8-E5M2 KV Cache to reduce memory pressure during inference.
    • ›Experimental multi-LoRA support enables serving multiple LoRA adapters concurrently from a single engine.
    • ›Experimental prefix caching support reuses KV cache across requests that share a common prompt prefix.
    +13 moreshow less

    • ›Supports batch completion requests in the OpenAI-compatible server.
    • ›Adds DeepSeek MoE model support with a fused MoE kernel for optimized throughput.
    • ›Fused MoE kernel for Mixtral improves MoE inference performance.
    • ›Adds quantized Mixtral support.
    • ›AWQ quantization delivers up to 2.66x higher throughput.
    • ›Custom all-reduce kernels replace the default NCCL all-reduce for multi-GPU communication.
    • ›Adds ROCm 6.0 and MI300 GPU support.
    • ›Adds Qwen2 model support.
    • ›Adds Stable LM 2 model support.
    • ›Adds AWS Neuron build option for running vLLM on Neuron hardware.
    • ›Supports the OpenAI API server as a target in benchmark_serving.py.
    • ›Allows setting the FastAPI root_path argument for reverse-proxy deployments.
    • ›Adds a Gradio chatbot UI for the OpenAI-compatible web server.
70. v0.2.7 Jan 4, 2024 · issue -382

    vLLM v0.2.7 adds SSL to API servers, CUDA graph support for GPTQ/SqueezeLLM, and up to 70% distributed inference throughput gains via NCCL.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.7 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.7

    • ›Adds SSL arguments to API servers, enabling TLS-secured inference endpoints.
    • ›Enables CUDA graph capture for GPTQ and SqueezeLLM quantized models, accelerating inference for those quantization formats.
    • ›Switches distributed control-plane communication from Ray to NCCL, removing serialization/deserialization overhead for up to 70% throughput improvement in distributed inference.
    • ›Adds support for DeciLM-7B and DeciLM-7B-instruct model architectures.
    • ›Adds support for GPT-NeoX models without attention biases.
71. v0.2.6 Dec 17, 2023 · issue -383

    vLLM v0.2.6 adds CUDA/HIP graph execution for faster inference and W4A16 GPTQ quantization support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.6 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.6

    • ›Adds quantization argument to enable W4A16 GPTQ support for running 4-bit quantized models, reducing memory footprint during inference.
    • ›Adds a flag to include the stop string in output text, giving callers visibility into which stop sequence terminated generation.
    • ›Accelerates model execution via CUDA/HIP graph capture, reducing per-request kernel-launch overhead for eligible models.
    • ›Adds Phi-2 to the list of supported models.
72. v0.2.5 Dec 14, 2023 · issue -383

    vLLM v0.2.5 adds Mixtral expert parallelism and grouped-query attention for MPT models.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.5 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.5

    • ›Adds expert parallelism support for Mixtral models, improving multi-GPU inference throughput.
    • ›Adds grouped-query attention (GQA) support for MPT model attention layers.
    • ›Removes the Python 3.10 minimum version requirement, broadening supported runtime environments.
73. v0.2.4 Dec 11, 2023 · issue -383

    vLLM v0.2.4 adds Mixtral 8x7B and AMD GPU support, plus ChatGLM and custom server params.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.4 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.4

    • ›Adds custom server parameters support via the OpenAI-compatible server.
    • ›Supports Mixtral 8x7B (Mixture-of-Experts) model, requiring the megablocks dependency.
    • ›Adds AMD GPU (ROCm) support in collaboration with EmbeddedLLM.
    • ›Adds support for ChatGLMForConditionalGeneration models.
    • ›Adds CUDA architecture list as a Docker build option.
    +1 moreshow less

    • ›Saves PyTorch profiler output for latency benchmarking.
74. v0.2.3 Dec 3, 2023 · issue -383

    vLLM v0.2.3 adds Prometheus metrics, echo support for the OpenAI chat API, and better Ray Serve integration.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.3

    • ›Adds production metrics exported in Prometheus format for operational observability.
    • ›Adds echo parameter support to the OpenAI-compatible chat API, mirroring the completions API behavior.
    • ›Supports chat templates in the OpenAI-compatible chat API.
    • ›Adds --profile option to the latency benchmark tool.
    • ›Supports max-model-len argument in the throughput benchmark tool.
    +3 moreshow less

    • ›Better integration with Ray Serve for production deployments.
    • ›Adds a PyTorch-native implementation of custom layers as an alternative to compiled extensions.
    • ›Initializes models directly on GPU to reduce CPU memory footprint when loading large models.
75. v0.2.2 Nov 19, 2023 · issue -384

    vLLM v0.2.2 adds Yi/ChatGLM2/Phi models, AWQ for all models, LogitsProcessor API, Min-P sampler, YaRN, and a health endpoint.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.2

    • ›Adds LogitsProcessor API to SamplingParams, enabling custom logits manipulation at inference time.
    • ›Adds AWQ quantization support for all models (previously limited to select models); quantization config is now auto-read from the HuggingFace quantization_config field.
    • ›Adds a /health HTTP endpoint to the OpenAI-compatible server for liveness checking.
    • ›Returns token usage fields in OpenAI-compatible API responses.
    • ›Adds support for the Min-P sampler in sampling parameters.
    +8 moreshow less

    • ›Adds repetition_penalty to sampling parameters.
    • ›Adds YaRN (Yet another RoPE extensioN) support for extended context length.
    • ›Adds preliminary support for SqueezeLLM quantization.
    • ›Adds new model support: Yi, ChatGLM2, and Microsoft Phi-1.5.
    • ›Upgrades base environment to PyTorch v2.1 + CUDA 12.1 (CUDA 11.8 wheels also provided).
    • ›Supports downloading models from modelscope.cn in addition to HuggingFace Hub.
    • ›Adds DeepSpeed-MII backend option to the benchmark script.
    • ›Adds official Dockerfile with CUDA 12.1.
    └──▷ BREAKING ON UPGRADE

    • !Scheduler input tensor shape changed from 1D flattened to 2D; custom integrations that depend on the internal tensor layout will break.
76. v0.2.1 Oct 16, 2023 · issue -385

    vLLM v0.2.1 adds PagedAttention V2 (up to 20% latency cut), AWQ support for Mistral 7B, and prompt log probabilities.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.1 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.1

    • ›Adds PagedAttention V2 kernel delivering up to 20% end-to-end latency reduction.
    • ›Adds AWQ (Activation-aware Weight Quantization) support for Mistral 7B models, with a workaround for Turing GPUs.
    • ›Adds prompt log probabilities — log probs are now returned for prompt tokens, not just generated tokens.
    • ›Adds support for sharding LLaMA2-70B across more than 8 GPUs.
    • ›Adds IPv4/IPv6 dual-stack support to the API server.
    +2 moreshow less

    • ›Adds tokenizer revision support.
    • ›Improves detokenization performance.
77. v0.2.0 Sep 28, 2023 · issue -386

    vLLM v0.2.0 adds AWQ quantization, Mistral-7B and LongChat support, RoPE scaling, and up to 60% faster inference.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.0 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.2.0

    • ›Makes max_model_len configurable, allowing users to override the default maximum model context length.
    • ›Adds gpu_memory_utilization and swap_space parameters to the LLM class for fine-grained memory control.
    • ›Adds stop_token_ids sampling parameter to halt generation at specific token IDs.
    • ›Adds float16 and float32 as explicit --dtype choices for model loading.
    • ›Enables TORCH_CUDA_ARCH_LIST environment variable to select target GPU architectures at build time.
    +6 moreshow less

    • ›Adds model revision support, allowing a specific model commit or revision to be pinned.
    • ›Supports AWQ (Activation-aware Weight Quantization) for LLaMA models, including safetensors format for quantized models.
    • ›Adds support for Mistral-7B-v0.1.
    • ›Adds support for LongChat and RoPE scaling (reads rope_theta and max_position_embeddings from model config).
    • ›Automatically sets max_num_batched_tokens based on available resources, removing the need for manual tuning.
    • ›Delivers up to 60% performance improvement through optimized de-tokenization (TGI-style incremental detokenization) and vectorized sampler.
78. v0.1.6 Sep 8, 2023 · issue -386

    vLLM v0.1.6 reverts a breaking API change and delivers faster startup plus background task generation.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.6 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.1.6

    • ›Reduces vLLM startup time for faster engine initialization.
    • ›Starts background tasks within AsyncLLMEngine.generate so generation can proceed asynchronously without requiring a separate task launch.
79. v0.1.5 Sep 7, 2023 · issue -386

    vLLM v0.1.5 adds CodeLlama support, safetensors loading for all models, and OpenAPI request body specs for OpenAI endpoints.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.5 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.1.5

    • ›Enables safetensors loading for all models, expanding the supported weight formats beyond the previous subset.
    • ›Adds support for CodeLlama, expanding the roster of served models.
    • ›Enables request body OpenAPI spec for OpenAI-compatible endpoints, making the API surface formally documented and machine-readable.
    • ›Stabilizes AsyncLLMEngine with a background engine loop, making async serving more robust for production use.
    • ›Aligns beam search implementation with HuggingFace hf_model.generate, improving output consistency for beam-search workloads.
    +1 moreshow less

    • ›Accelerates LLaMA model loading, reducing startup time when serving LLaMA-family models.
80. v0.1.4 Aug 25, 2023 · issue -387

    vLLM v0.1.4 adds InternLM, Qwen, and Aquila models, token-array inputs to the OpenAI API, and pre-built CUDA binaries.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.4 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.1.4

    • ›Supports tokens and arrays of tokens as inputs to the OpenAI completion API, expanding what callers can pass as prompts.
    • ›Adds support for three new model families: InternLM, Qwen (QWen-7B), and Aquila.
    • ›Publishes pre-built CUDA binaries so users no longer need to compile vLLM's CUDA kernels locally on install.
    • ›Adds max prompt length validation for the OpenAI completions API.
    • ›Implements approximate GELU kernels and optimizes the single_query_cached_kv_attention kernel for faster paged-attention throughput.
    +1 moreshow less

    • ›Adds compute capability 8.9 to default CUDA build targets, covering Ada Lovelace GPUs.
81. v0.1.3 Aug 2, 2023 · issue -387

    vLLM v0.1.3 adds LLaMA 2, Falcon, GPT-J, and Baichuan model support plus TGI-style continuous batching and MQA/GQA efficiency.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.3 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.1.3

    └──▷ TRY IT
    Serve a model whose tokenizer requires remote code execution (e.g. Baichuan) without a trust error.
    copy

    $ python -m vllm.entrypoints.api_server --model baichuan-inc/Baichuan-7B --trust-remote-code

    • ›Adds --trust-remote-code flag to the API server and benchmark tooling to handle remote tokenizers.
    • ›Adds Ray placement group support for more flexible distributed cluster scheduling.
    • ›Adds support for LLaMA 2 models.
    • ›Adds support for Falcon models.
    • ›Adds support for GPT-J models.
    +3 moreshow less

    • ›Adds support for Baichuan-7B and Baichuan-13B models.
    • ›Switches the scheduling algorithm to TGI-style continuous batching.
    • ›Adds efficient support for Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), including an optimized MQA kernel.
82. v0.1.2 Jul 5, 2023 · issue -388

    vLLM v0.1.2 adds GPTBigCode, MPT, and BLOOM model support plus a ChatCompletion endpoint in the OpenAI demo server.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.2 https://github.com/vllm-project/vllm.git
    # already have the repo? check out this version:
    $ git checkout v0.1.2

    • ›Adds ChatCompletion endpoint to the OpenAI-compatible demo server, enabling chat-style API interactions.
    • ›Adds support for GPTBigCode models.
    • ›Adds support for MPT and BLOOM models.
    • ›Supports custom tokenizers.