https://aitoolchain.io/tools/vllm New releases and features in vLLM, tracked by The AI Toolchain. en Wed, 26 Aug 2026 09:46:30 GMT https://github.com/vllm-project/vllm/releases/tag/v0.28.0 https://github.com/vllm-project/vllm/releases/tag/v0.28.0 Wed, 26 Aug 2026 09:46:30 GMT vLLM v0.28.0 adds disk KV offloading, gRPC multimodal inference, new API fields, and major model/hardware expansions across NVIDIA, AMD, and Intel. • Adds `module_path` config key for out-of-tree secondary KV cache tier managers, enabling custom offloading backends without forking vLLM. • Adds `count_reasoning_tokens` field to the streaming parser engine for tracking reasoning token counts. • Adds `content_parts` field on `/inference/v1/generate` for richer multimodal generation requests. • Adds request priority parsed from an HTTP header, enabling priority-based request scheduling via standard HTTP. • Adds `session ID` plumbing into requests for stateful session tracking. • Adds CuMemAllocator.discard() for tag-selective GPU memory release in RL workflows. • Adds `thinking_token_budget` support in Model Runner V2 for controlling reasoning token budgets. • Adds disk offloading support for `SimpleCPUOffloadConnector`, extending tiered KV cache to persistent storage. • Adds tiering offloading metrics for observability into KV cache tier behavior. • Adds a canonical CPU layout for parallelism-agnostic KV cache offload. • Raises default `max_num_batched_tokens` from 8192 to 16384. • Raises the Blackwell CUDA graph capture default to 1024. • Enables prefix caching by default for Mamba models. • Adds Rust frontend standalone renderer with gRPC multimodal image inference support. • Adds explicit data-parallel rank routing and RL lifecycle control to the Rust/gRPC frontend. • Publishes protobuf schemas for the gRPC frontend to Buf. • Adds Model Runner V2 E/P/D (encode/prefill/decode) disaggregation support. • Adds encoder CUDA graphs in Model Runner V2. • Adds decoder token-wise pooling and Transformers pooling models in Model Runner V2. • Adds attention-free model support in Model Runner V2. • Adds multi-layer MTP KV cache support in Model Runner V2. • Adds weight offloading in Model Runner V2. • Adds FlashInfer XQA decode support on SM12x (Blackwell). • Adds a CuTeDSL fused query kernel on SM100. • Adds an MLA backend enabling DeepSeek-V2/V3 inference on CPU. • Adds GPTQ and AWQ quantization support on s390x. • Enables AITER and FP8 inference on GFX120x (AMD ROCm). • Adds a triton-cpu wheel for CPU-based Triton kernels. • Adds an XPU wheel to the release pipeline. • Adds online MXFP4 quantization support. • Adds batch-invariant NVFP4 MoE via CUTLASS. • Adds block-wise `scaled_mm` kernel for quantization. • Adds AMD Quark NVFP4 support with an emulation kernel for DeepSeek-V4. • Adds Mooncake store group semantics and tenant ID support, with official wheels in the Docker image. • Adds `file://` rendezvous for single-node executors to eliminate startup port races. • Adds a CI GPU<->CPU sync check to guard against accidental synchronization regressions on the execution path. • Adds JIT warmup infrastructure with predicate filtering. • Adds sparse MLA end-to-end support for plain decode, MTP, and DSpark speculative decoding in DeepSeek V4. • Adds DFlash2 speculative decoding with local convolution and a candidate selector. • Adds DSpark confidence-scheduled verification for speculative decoding. • Adds async scheduling auto-enabled for draft models in speculative decoding. • Adds Decode Context Parallel (DCP) support for Kimi-K3. • Adds optional shared-expert sharding for Kimi-K3, saving ~17 GiB of memory per GPU. • Adds vision tower LoRA support for Gemma4, Keye, and Ultravox models. • Adds native multimodal support for Dots3 NOTE. • Adds new models: Muse Glimmer, Ling 3.0 Flash (BF16/MTP/FP8/MXFP4), and Interns2mobius. • Adds Transformers modeling backend with MLA support, hardware-agnostic model definition, logit softcapping, and generalized input embedding handling. • Adds EAGLE3 speculative decoding support on KimiLinear. • Adds NUL byte rejection in `structured_outputs.regex` and negative token ID rejection for structured output robustness. • Adds `VLLMValidationError` raised from structured output validators. • Adds `dynamic tools` from developer messages in the Rust frontend. • Adds output token IDs logged at DEBUG level. • Adds vLLM Recipes integration with native config-based deployment and benchmarking. • Adds a minimal Triton Proton profiling backend. • Adds partial secondary-tier KV load results support, allowing partial cache hits from offload storage. • Adds data-parallel topology exposure to offloading backends. • Adds stateful trainer send over NCCL and sparse NCCL for RL workflows. • Adds `model` as optional on all `/derender` request classes. Breaking changes: • bitsandbytes support has been migrated to an out-of-tree plugin; existing setups using bitsandbytes quantization will break without installing the plugin separately. • Transformers dependency bumped to 5.15.0; models or configurations incompatible with Transformers 5.15.0 will break. • The deprecated `calculate_kv_scales` runtime KV scale calculation has been removed; any configuration or code referencing it will fail. • `override_attention_dtype` has been removed; any configuration or code using this option will fail. https://github.com/vllm-project/vllm/releases/tag/v0.27.1 https://github.com/vllm-project/vllm/releases/tag/v0.27.1 Tue, 11 Aug 2026 10:47:49 GMT vLLM v0.27.1 adds quantization support for DSpark Markov heads, enabling W4A16 and other quantized weight configurations. • Adds `quant_config` support to `DSparkMarkovHead`, forwarding quantization configuration to its `ParallelLMHead`-based `markov_w2` projection, including W4A16 configurations with `weight_scale_2`. https://github.com/vllm-project/vllm/releases/tag/v0.27.0 https://github.com/vllm-project/vllm/releases/tag/v0.27.0 Mon, 10 Aug 2026 21:18:11 GMT vLLM v0.27.0 adds Kimi K3, Qwen3.5, FlashAttention 4 FP8/headdim-256 on SM100, fault-tolerance for DP+EP, gRPC control plane, and `sm_107` Rubin support. • Adds `--linear-backend` flag honored for ModelOpt W4A16 quantization. • Adds `VLLM_TRITON_USE_TD` environment variable to enable the Triton tensor-descriptor path for fused MoE kernels. • Adds `sample_from_anchor` loaded from speculators config for speculative decoding. • Adds `has_pending_push_work` to encoder-cache connectors for distributed disaggregated serving. • Adds `vllm-bench` integrated into the `vllm` CLI for benchmarking. • New model support for Kimi K3 with full stack: core model, Python and Rust frontends, AttnRes kernels, DeepGEMM, DSpark AR fusion, and optional shared-expert sharding. • New model support for Qwen3.5 text-only dense and MoE models with EVS video token pruning. • New model support for K-EXAONE-2.0-750B-A37B. • New model support for VaultGemma via the Transformers modeling backend. • New model support for jina-embeddings-v5-text-nano with EuroBERT encoder backbone. • Deepens FlashAttention 4 integration on SM100 with FP8 KV cache support and headdim-256 support. • Adds new JIT warmup infrastructure and runner-owned Triton kernel warmup before the first request, removing first-request compilation stalls. • Adds fault tolerance framework for DP+EP external load-balancer deployments. • Adds async preparation for elastic EP scaling. • Adds NIXL P/D disaggregation support for hybrid MLA+SSM models. • Adds NIXL heterogeneous P/D block sizes for hybrid models. • Adds MoRIIO heterogeneous TP<->DP prefill/decode read routing. • Adds gRPC control plane to the Rust frontend with engine-aware health reporting, abort control, server and model discovery, and KV event source discovery. • Adds `sm_107` target for NVIDIA Rubin with NVLink all-reduce paths on SM107. • Enables ROCm gfx1250 architecture support. • Expands Model Runner V2 to non-generative workloads: encoder-only attention, sequence pooling for embedding/classification, encoder token classification, encoder token embedding, and BGE-M3 pooling. • Adds multi-layer MTP speculator to Model Runner V2. • Adds multimodal inference on CPU via Model Runner V2. • Adds per-request KV offload tier filtering via `TierFilter`/`TierMatcher` and self-describing KV events via `TieringOffloadingSpec`. • Adds pluggable eviction policies via `CachePolicyFactory` for KV offloading. • Adds `CPUOffloadingSpec` with single-copy MLA layout and migration onto `SharedOffloadRegion`. • Adds generic P2P secondary tier with peer lookup and serving for KV offloading. • Adds FP4 Qutlass integration for compressed-tensors quantization. • Adds CuTeDSL MoE for ReLU2 NVFP4 quantization. • Adds MXFP8 linear support in INC quantization. • Adds AutoRound W4A16 MoE and MXFP4 linear/MoE on XPU. • Adds KV quant mode for TurboQuant. • Adds ModelOpt FP8 emulation on SM80. • Adds compressed-tensors checkpoint support for DeepSeek-V4 and Kimi-K3. • Adds `top_k` and `top_p` sampling for DiffusionGemma diffusion models. • Adds weight version tagging for RL rollouts. • Adds INT8 fused MoE kernel for Arm CPUs. • Adds s390x inference optimization with oneDNN INT8 GEMM. • Adds XPU QK Norm + RoPE fusion pass. • Adds XPU FP8 `o_proj` with `fp8_bmm` and load-time scale transpose. • Adds sequence parallelism for DeepSeek-V4 serving. • Adds ReplaySSM caching for faster Mamba2 standard decode. • Adds FlashInfer Mamba SSU algorithm selection. • Adds FlexAttention epilogue hook. • Adds audio model support to the Transformers modeling backend. • Adds fused residual-add + RMSNorm compilation pass in the Transformers modeling backend. • Upgrades to PyTorch 2.13.0 with torchvision 0.28.0 and Triton 3.7.1. • Adds grammar advancement across the reasoning boundary with speculative decode for structured outputs. Breaking changes: • PyTorch upgraded to 2.13.0 (along with torchvision 0.28.0 and Triton 3.7.1), which is a breaking environment change requiring environment rebuild for all platforms including XPU and CPU. • `FusedMoE` is renamed to `FusedMoEFactory` in the MoE refactor. https://github.com/vllm-project/vllm/releases/tag/v0.27.0 https://github.com/vllm-project/vllm/releases/tag/v0.27.0 Mon, 10 Aug 2026 21:18:11 GMT vLLM v0.27.0 adds Kimi K3, Qwen3.5, FP4/FP8 quantization, gRPC control plane, fault tolerance, and PyTorch 2.13. • Adds `VLLM_TRITON_USE_TD` environment variable to enable the Triton tensor-descriptor path for fused MoE kernels. • Adds `--linear-backend` flag, now honored for ModelOpt W4A16 quantization. • Adds `sample_from_anchor` loaded from the speculators config for speculative decoding control. • Adds `has_pending_push_work` to encoder-cache connectors for disaggregated serving. • Adds `vllm-bench` integrated into the `vllm` CLI for benchmarking. • Adds `sm_107` compilation target for NVIDIA Rubin GPU architecture, with NVLink all-reduce paths on SM107. • Supports Kimi K3 model with full stack: core model files, specialized CUDA/Triton AttnRes kernels, Python and Rust frontends, DeepGEMM, DSpark AR fusion, and compressed-tensors quantized checkpoints. • Supports Qwen3.5 text-only dense and MoE models with EVS video token pruning. • Supports K-EXAONE-2.0-750B-A37B and jina-embeddings-v5-text-nano (EuroBERT encoder backbone) models. • Supports VaultGemma via the Transformers modeling backend. • Supports `top_k` and `top_p` sampling for DiffusionGemma diffusion models. • Supports longer max audio duration for MOSS-TD multimodal model. • Adds FP4 Qutlass integration for compressed-tensors quantization. • Adds CuTeDSL MoE kernel for ReLU2 NVFP4 quantization. • Adds MXFP8 linear support in INC (Intel Neural Compressor). • Adds AutoRound W4A16 MoE and MXFP4 linear/MoE on XPU. • Adds KV quant mode for TurboQuant. • Adds ModelOpt FP8 emulation support on SM80 GPUs. • Adds compressed-tensors support for DeepSeek-V4 and Kimi-K3 checkpoints. • Adds FlashAttention 4 FP8 KV cache support and headdim-256 support on SM100 (NVIDIA Blackwell). • Adds new JIT warmup infrastructure and runner-owned Triton kernel warmup to eliminate first-request compilation stalls. • Adds sequence parallelism for DeepSeek-V4 workloads. • Adds compact MXFP4 indexer KV cache for DeepSeek-V4. • Adds `CachePolicyFactory` interface for pluggable and externally-defined KV cache eviction policies. • Adds generic P2P secondary KV offload tier with peer lookup and serving. • Adds per-request KV tier filtering via `TierFilter`/`TierMatcher`. • Adds self-describing KV events via `TieringOffloadingSpec`. • Adds `ReplaySSM` caching for faster Mamba2 standard decode. • Adds FlashInfer Mamba SSU algorithm selection. • Adds multi-layer MTP speculator support in Model Runner V2. • Adds encoder-only attention, sequence pooling, encoder token classification and embedding, and BGE-M3 pooling to Model Runner V2, expanding it to non-generative workloads. • Adds multimodal inference on CPU via Model Runner V2. • Adds a simplified fault tolerance framework for DP+EP external load-balancer deployments. • Adds async preparation for elastic EP (expert parallelism) scaling. • Adds NIXL P/D disaggregation support for hybrid MLA+SSM models. • Adds heterogeneous P/D block sizes for hybrid models in NIXL. • Adds MoRIIO heterogeneous TP<->DP prefill/decode read routing. • Adds gRPC control plane to the Rust frontend: engine-aware health reporting, abort control, server and model discovery, and KV event source discovery. • Adds weight version tagging for RL rollouts. • Adds grammar advancement across the reasoning boundary with speculative decode for structured outputs. • Adds ROCm gfx1250 architecture support. • Adds AITER FP8 ViT encoder attention on ROCm. • Adds INT8 fused MoE kernel for Arm CPUs. • Adds s390x inference optimization via oneDNN INT8 GEMM. • Adds XPU QK Norm + RoPE fusion pass and FP8 `o_proj` with `fp8_bmm`. • Adds audio model support to the Transformers modeling backend. • Adds FlexAttention epilogue hook for custom attention post-processing. • Adds `FusedMoEFactory` (renamed from `FusedMoE`) as part of a MoE kernel refactor. • Upgrades to PyTorch 2.13.0, torchvision 0.28.0, and Triton 3.7.1. Breaking changes: • PyTorch upgraded from a prior version to 2.13.0 (with torchvision 0.28.0 and Triton 3.7.1); existing environments must be rebuilt — this is described as a breaking environment change. • `FusedMoE` is renamed to `FusedMoEFactory`; any code referencing `FusedMoE` directly will break. https://github.com/vllm-project/vllm/releases/tag/v0.26.0 https://github.com/vllm-project/vllm/releases/tag/v0.26.0 Mon, 27 Jul 2026 01:06:58 GMT vLLM v0.26.0 adds the Inkling model family, `head_dtype` fp32 lm_head, per-KV-cache-group attention backends, KV offloading metrics, and major API/frontend expansions. • Adds `head_dtype` config to enable fp32 `lm_head` for generation models, extended to the LoRA path and given a ROCm `torch.mm` fast path. • Adds `kv_cache_dtype` field inside `speculative_config` to set a separate KV-cache dtype for speculative decoding. • Adds `kv_cache_dtype_skip_layers` support for MLA quantization. • Adds `blocks_per_chunk` config key for heterogeneous KV groups in KV offloading. • Adds `bad_words` parameter to `/v1/completions` for OpenAI-compatible completions. • Adds `logprob_token_ids` field exposed on Python OpenAI endpoints. • Adds `include_reasoning` parameter for non-Harmony models on OpenAI-compatible endpoints. • Adds `num_cache_creation_tokens` field populated on Messages responses. • Adds `/abort_requests` endpoint on the RLHF dev API router. • Adds `continue_final_message` handling with renderer sentinel in the Rust frontend. • Adds endpoint plugins framework for extending API routing. • Introduces the Inkling model family with full support: base modeling, piecewise CUDA graph, Hopper FA4 relative attention, MTP=1 speculative decoding, LoRA, and ModelOpt NVFP4 quantization. • Adds support for selecting a different attention backend per KV-cache group, enabling better hybrid-model support. • Makes sliding-window an explicit backend capability for per-backend declaration. • Adds partial prefix-cache hit for hybrid models and selective hybrid cache retention. • Reports prefix-cache-reused blocks in full report mode. • Adds KV offloading metrics: basic offloading gauges, CPU cache read/write split gauges, and tiering-lookup-delay sync/async histograms. • Adds object-store secondary KV tier with workload identity support and DP-replica-aware tiering. • Adds runtime draft weight update for speculative decoding. • Adds hybrid (SWA + full attention) DFlash drafters for speculative decoding. • Adds `nvfp4_per_token` online MoE quantization. • Adds Humming w[2-7]a[4,8] weight-only inference with compressed-tensors. • Adds int4 quantization for the emulation MoE backend. • Adds INT2 XPU weight-only quant linear. • Adds FlashInfer MoE LoRA for BF16 models. • Adds LoRA support for tower/connector in LlavaNextVideo. • Adds opt-in persistence and reuse of the memory-profiling result across boots. • Adds Rust frontend support for multimodal video and audio inputs. • Adds Seed-OSS tool parser to the Rust frontend. • Adds native `vllm-bench` port in the Rust frontend. • Adds Deepstream video decoding backend. • Adds overlap of preprocessing and computation for pooling models in offline inference. • Adds new models: BertForMaskedLM, RobertaForTokenClassification, XLMRobertaForTokenClassification, LongCat-Flash-Lite n-gram embedding, Cosmos3 Edge Reasoner, Cosmos3-Super, and TranslateGemma-12b-it. • Migrates Olmo/Olmo2, MistralLarge3, and HunyuanVL to the Transformers modeling backend (Transformers 5.13.0). • Adds automatic fallback to ViT data parallelism when TP is unavailable for multimodal models. • Adds NIXL pipeline-parallel prefill in push mode for PD disaggregation. • Adds Decode Context Parallel (DCP) hybrid attention support and DCP + Eagle for Tokenspeed MLA backends. • Adds native macOS arm64 CPU wheel builds. • Adds s390x NUMA topology support for CPU inference. • Adds POWER VSX math function optimization and IBM Power docker builds using prebuilt wheels. • Adds CuTeDSL compilation progress bar for UX feedback during compilation. • Adds human-readable integer formatting for more CLI arguments. • Adds expanded GPU profiler config scope and annotations. • Adds worker exit-code logging when a process dies unexpectedly. • Replaces diskcache with a non-pickle alternative to eliminate pickle deserialization risks. • Adds resource-bounds validation to derender endpoints and sanitizes server file paths from validation error responses. • Adds grammar compilation failure handling that avoids crashing the engine. • Adds timeout guard on lm-format-enforcer regex compilation. • Adds STATEFUL trainer-send abstractions for RLHF. • Adds DSpark speculative decoding for DSv4 on AMD ROCm and XPU. • Adds DFlash speculative decoding for GDN models on CPU. Breaking changes: • Models TeleChat, Persimmon, and Fuyu have been removed and are no longer supported. https://github.com/vllm-project/vllm/releases/tag/v0.26.0 https://github.com/vllm-project/vllm/releases/tag/v0.26.0 Mon, 27 Jul 2026 01:06:58 GMT vLLM v0.26.0 adds the Inkling model family, fp32 `head_dtype`, per-group attention backends, KV offloading metrics, and expanded OpenAI API fields. • Adds `head_dtype` config option to enable fp32 `lm_head` for generation models, improving accuracy for generation heads; extended to the LoRA path and given a ROCm `torch.mm` fast path. • Adds `kv_cache_dtype` per `speculative_config` so the draft model's KV cache dtype can be set independently from the main model. • Adds `kv_cache_dtype_skip_layers` support for MLA quantization. • Adds `blocks_per_chunk` config key for heterogeneous KV groups in KV offloading. • Adds `prefix_match_unit` option to select prefix-match granularity independently from physical KV-cache block sizes, enabling partial prefix-cache hits for hybrid (full-attention + Mamba) models. • Adds `--mamba-cache-mode align` and `--prefix-match-unit` CLI flags for fine-grained prefix-cache hits on hybrid models. • Exposes `logprob_token_ids` on Python OpenAI endpoints. • Adds `bad_words` parameter support to `/v1/completions`. • Adds `include_reasoning` parameter for non-Harmony models on OpenAI-compatible endpoints. • Populates `num_cache_creation_tokens` field on Messages (chat completions) responses. • Adds `/abort_requests` endpoint to the RLHF dev API router. • Adds an endpoint plugins framework for extending API routing. • Supports runtime draft weight update for speculative decoding, allowing RL training loops to update draft model weights without restart. • Adds P2P default host/port environment variables for KV offloading peer-to-peer configuration. • Adds KV offloading metrics: basic offloading gauges, split CPU cache usage into read/write gauges, and tiering-lookup-delay split into sync/async histograms. • Adds object-store secondary tier with workload identity for KV tiered storage. • Adds DP-replica-aware tiering for KV offloading. • Adds EC (encoder-cache) transfer params and a CPU-offloading EC connector. • Adds `nvfp4_per_token` online MoE quantization support. • Adds Humming w[2-7]a[4,8] weight-only inference with compressed-tensors. • Adds INT2 XPU weight-only quantization linear support. • Adds per-KV-cache-group attention backend selection, enabling hybrid models with mixed attention backends. • Makes sliding-window support an explicit backend capability, improving hybrid model support. • Adds DCP (Decode Context Parallel) hybrid attention support and DCP + Eagle for Tokenspeed MLA backends. • Adds NIXL pipeline-parallel prefill in push mode for PD disaggregation. • Adds Rust frontend support for multimodal video and audio, a Seed-OSS tool parser, and a native `vllm-bench` port. • Adds `continue_final_message` handling with renderer sentinel in the Rust frontend. • Adds Deepstream video decoding backend. • Adds human-readable integer formatting for more CLI arguments. • Adds CuTeDSL compilation progress bar. • Adds expanded GPU profiler config scope and annotations. • Adds worker exit code logging when a process dies unexpectedly. • Adds native macOS arm64 CPU wheel builds. • Adds DFlash speculative decoding for GDN models on CPU. • Adds new model support: Inkling family (base, piecewise CUDA graphs, Hopper FA4 relative attention, MTP=1 speculative decoding, LoRA, NVFP4), BertForMaskedLM, RobertaForTokenClassification, XLMRobertaForTokenClassification, LongCat-Flash-Lite n-gram embedding, Cosmos3 Edge Reasoner, Cosmos3-Super, TranslateGemma-12b-it. • Migrates Olmo/Olmo2, MistralLarge3, and HunyuanVL to the Transformers modeling backend (Transformers 5.13.0). • Adds FlashInfer MoE LoRA for BF16 models. • Adds LoRA support for tower/connector in LlavaNextVideo. • Adds automatic fallback to ViT data parallelism when TP is unavailable for multimodal models. • Adds STATEFUL trainer-send abstractions for RLHF. • Replaces diskcache to eliminate pickle deserialization in caching layer. • Adds resource-bounds validation to derender endpoints and bounds the completion prompt list to prevent unbounded engine fan-out. • Adds prefix-cache-reused blocks reporting in full report mode. • Adds selective hybrid cache retention (Marconi caching) for hybrid models. • Adds opt-in persistence and reuse of the memory-profiling result across boots. • Adds TRTLLM BF16 MoE modular kernel. • Adds XPU nightly/release image publishing. Breaking changes: • The `VLLM_TRITON_ATTN_USE_TD` environment variable is renamed to `VLLM_TRITON_USE_TD`. • Models TeleChat, Persimmon, and Fuyu are removed and no longer supported. https://github.com/vllm-project/vllm/releases/tag/v0.25.0 https://github.com/vllm-project/vllm/releases/tag/v0.25.0 Sat, 11 Jul 2026 20:06:44 GMT vLLM v0.25.0 adds a Streaming Parser Engine, static HTTPS/mTLS on the Rust frontend, universal speculative decoding, and a raft of new models and API fields. • Adds `VLLM_GPU_SYNC_CHECK` environment variable for GPU synchronization checking. • Adds `return_loss_mask` field to Chat/Completions responses for training-data generation. • Adds `model_class_overrides` CLI option for development and debugging. • Adds `repetition_detection` sampling parameter in the Rust frontend. • Adds `kv_transfer_params` merging via `MultiConnector` for PD disaggregation. • Adds `FLASH_ATTN_MLA_SPARSE` environment variable to select the Hopper sparse-MLA attention backend. • Adds `ROCM_AITER_FA` backend flag to enable the AITER FlashAttention MLA prefill path on ROCm. • Adds static HTTPS and mTLS support for HTTP and gRPC in the Rust frontend. • Adds per-request timing `metrics` field on Chat/Completions responses. • Adds token offsets on render endpoints. • Adds profiler control routes in the Rust frontend. • Adds a DP supervisor via the Rust frontend. • Introduces a new Streaming Parser Engine — a unified tool-call and reasoning parsing framework — with a new Kimi k2.5/k2.6/k2.7 parser and ports of seed_oss and DeepSeek V4 parsers. • Adds universal speculative decoding for heterogeneous vocabularies (TLI), enabling spec decode across models with mismatched vocabularies. • Adds DSpark and DFlash speculative decoding drafters, including a Laguna XS.2.1 drafter. • Adds `tok_sparse_select` configuration for MiniMax-M3, replacing Triton kernels. • Adds `LookupResult` enum to the KV offloading subsystem. • Adds TorchCodec as a video decoding backend. • Adds TTFT and TPS metrics printing in `vllm chat`. • Adds `usage` field exposed for disaggregated serving. • Adds a dedicated Harmony renderer for gpt-oss / Harmony serving. • Adds a secondary-tier implementation for PD disaggregation. • Adds 2/3/5/6/7-bit pack-quantized weight-only inference (Humming). • Adds Triton INT4 per-token-head KV cache quantization. • Adds NVFP4 KV cache with skip-layers sliding window support. • Adds Marlin thread-tile padding extended to MoE (WNA16 + FP8/MXFP8) and int8 grouped WNA16 MoE. • Adds FlashInfer MXINT4 MoE support for gated SiLU. • Adds W8A8 FP8 linear kernel with multi-granularity quantization for Intel XPU. • Adds AITER custom all-reduce in CudaCommunicator for ROCm. • Adds INT3 quantization for quickreduce on ROCm. • Adds EPLB support for Quark OCP MXFP4 MoE on ROCm. • Adds accelerator-agnostic `get_memory_info` platform API and a C++ implementation for Intel XPU. • Adds RVV path for W4A8 INT4 GEMM on RISC-V, BF16 support on VLEN=256 hardware, and reduced LMUL pressure in INT4 LUT dequant. • Adds fp16 support on PowerPC. • Adds new models: LLaVA-OneVision-2, Unlimited OCR (with Triton R-SWA backend), MOSS-Transcribe-Diarize, openai/privacy-filter, and Hy3 (with token-suffix and JSON Schema array support). • Adds GLM-5 / DeepSeek-V3.2 to the model zoo and pipeline parallelism for MiniMax-M3. • Model Runner V2 is now the default execution path for all dense models, adding support for EVS, realtime embeddings, Mamba hybrid prefix caching, multimodal-prefix bidirectional attention, and dynamic speculative decoding with full CUDA graphs. • Adds sequence parallelism without requiring data parallelism, yielding 1.9–5.0% end-to-end throughput improvement. • Adds NCCL symmetric memory support extended to AllGather and ReduceScatter operations. • Adds a fault-tolerance backend to detect all-to-all peer faults and prevent corrupted output. • Adds Mooncake connector support for GDN (Qwen3.5) and MLA (DeepSeek-V4-Flash) in PD disaggregation. • Adds NIXL Mamba1 support for PD disaggregation. • Adds `fused_indexer_q_rope_quant` Triton kernel for GLM-5.2/DeepSeek, providing 1.9–3.3% end-to-end throughput gain. • Adds `token_to_req_indices` cache for DSv4, achieving 5–6x kernel speedup. • Adds Responses API namespace tools for OpenAI compatibility. • Adds HTTP 422 response for unprocessable image URLs. • Adds image decompression-bomb OOM denial-of-service prevention. • Adds pluggable sleep-mode backend abstraction with communicator-agnostic capability flags. • Adds blocking CUDA events to avoid busy-polling the driver lock. • Adds VRAM semaphore infrastructure. • Adds FlashInfer fused all-reduce tuned for world_size=16 on GB300 (Blackwell). • Adds CuTeDSL/FA4-MLA warmup infrastructure and a B12x backend for non-gated MoEs on Blackwell. • Adds Helion `fused_qk_norm_rope` and `silu_and_mul_per_block_quant` kernels. • Adds accelerated unquantized MoE for AArch64 CPUs. • Adds compressed-tensor w8a8 int8 MoE and Mamba ShortConv support on CPU. • Adds chunked prefill and prefix caching for Qwen3.5 on CPU. Breaking changes: • PagedAttention has been deleted; deployments relying on the legacy attention implementation must migrate to the V1/MRv2 backend. • Models removed: Baichuan, Aquila, Grok, Tarsier/Tarsier2, AyaVision/MusicFlamingo, and Mantis are no longer supported. • `gptq_marlin` has been removed from supported ROCm quantization schemes. • Legacy `api_server.py` has been moved to the examples directory and is no longer the primary server entrypoint. • The old FP8 online MoE quantization class is deprecated (removed in a future release). https://github.com/vllm-project/vllm/releases/tag/v0.24.0 https://github.com/vllm-project/vllm/releases/tag/v0.24.0 Mon, 29 Jun 2026 19:41:59 GMT vLLM v0.24.0 adds MiniMax-M3, DiffusionGemma, a Streaming Parser Engine, Rust frontend endpoints, and removes internal CUDA_VISIBLE_DEVICES management. • Adds `device_ids` argument to replace internal `CUDA_VISIBLE_DEVICES` management; vLLM no longer sets `CUDA_VISIBLE_DEVICES` internally. • Adds `VLLM_TRITON_FORCE_FIRST_CONFIG` environment variable to skip Triton autotuning and cut startup overhead. • Rust frontend gains API-key authentication, CORS, `/tokenize`, `/detokenize`, `/pause`, `/resume`, `/is_paused`, `/abort_requests`, `/get_world_size`, `thinking_token_budget`, and `parallel_tool_calls=false`. • Adds real `/v1/embeddings` support for messages with `chat_template_kwargs`. • Adds cache-usage reporting to the Anthropic-compatible `/v1/messages` endpoint. • Adds multimodal token counts inside `usage.prompt_tokens_details` in API responses. • Adds strict mode for tool calling in Chat Completions API and Responses API. • Introduces a new Streaming Parser Engine that unifies tool-call and reasoning parsing, with parsers for Qwen3, MiniMax-M2, GLM-4.7/5.1/5.2, and Nemotron V3. • Adds support for new models: MiniMax-M3, DiffusionGemma (including a CPU path), Hierarchical Reasoning Model (HrmTextForCausalLM), and OpenMOSS. • Model Runner V2 now supports quantized models by default and enables GraniteMoE by default. • Adds Dynamic Speculative Decoding and EAGLE3 support for Qwen3 targets. • Adds online FP8 per-token-per-channel (PTPC) quantization. • Extends `modelopt_mixed` quantization support to Ampere (SM80-86) and Turing (SM75) GPUs. • Allows `fp8_e5m2` KV cache for non-FP8 checkpoints. • Adds `flashinfer_cutlass` as a clamped NVFP4 MoE backend. • Integrates DeepEP v2 for expert parallelism. • Adds KV push from prefill to decode via NIXL for disaggregated serving. • Adds a KV-cache watermark to reduce preemptions. • Adds a Marconi-style admission policy for hybrid KV cache. • Adds structured-output guardrails for diffusion decoders. • Adds structured outputs support for beam search. • Adds Qwen3-VL video loader and Qwen2-VL/Qwen2.5-VL processor-mapped video loader. • Adds ViT full CUDA graph support for GLM-4.1V, DeepSeek-OCR, Kimi-VL, mllama4, and Lfm2VL encoder. • Adds fastsafetensors `ParallelLoader` for faster weight loading. • Adds 2.5x faster ASR CPU preprocessing via multi-threading. • Adds cgroup memory-limit-aware KV cache sizing on CPU. • Adds RISC-V oneDNN W8A8 INT8 support and RVV micro-GEMM for WNA16. • Migrates GGUF quantization to a plugin. • Adds Intel XPU sequence-parallel support and W4A16 int4 group_size=32 MoE. • Adds SM90 CUTLASS FP8 odd-M support via swap_ab (180-290% kernel speedup). • Adds FlashInfer cutedsl NVFP4 GEMM backend and cute-dsl MXFP8 linear kernel. • Adds Helion kernels for FP8/RMSNorm quantization. • Completes migration of all kernels to the libtorch stable ABI. • Deprecation window begun for `CUDA_VISIBLE_DEVICES` on ROCm. Breaking changes: • ROCm: `CUDA_VISIBLE_DEVICES` support has entered a deprecation window and will be removed in a future release. • `P2pNcclConnector` has been removed; configurations using it must migrate to an alternative KV connector. https://github.com/vllm-project/vllm/releases/tag/v0.23.0 https://github.com/vllm-project/vllm/releases/tag/v0.23.0 Mon, 15 Jun 2026 05:27:20 GMT vLLM v0.23.0 adds a maturing Rust frontend, multi-tier KV offloading, unified parser, and broad new model support across backends. • Rust frontend adds a streaming `generate` endpoint, dynamic LoRA endpoints, `/version` and `/server_info` endpoints, a server-router extension hook, and `--enable-request-id-headers` flag. • Adds per-request KV cache offloading policy via the `on_new_request` lifecycle hook and on_schedule_end() hook, plus an object-store secondary offloading tier. • Introduces `scheduler_block_size` threaded into `KVCacheManager`/Coordinator and `max_concurrent_batches` moved to `VllmConfig`. • Adds pluggable `KVCacheSpec` for configurable KV cache specifications. • Unifies reasoning and tool-call parsing behind a single Parser.parse() interface, with the Responses parser migrated to it. • Adds `system_fingerprint` field to the OpenAI/Responses API. • Adds `chat_template_kwargs` support in the Responses API. • Supports `streaming tool/function calling` with `required` tool choice in the OpenAI-compatible API. • Adds `tool_choice="none"` honored in streaming responses. • Adds Anthropic Messages API support for structured output, effort, and system-role messages inside the messages array. • Model Runner V2 (MRv2) is now the default for Llama and Mistral dense models, gaining a FlashInfer sampler, breakable CUDA graphs, and pipeline-parallel bubble elimination. • Adds new tool parsers for InternLM2, hy_v3, Phi-4-mini, and Gemma4 in the Rust frontend. • Adds new tool parsers: MiniCPM5 XML, Qwen3 XML JSON-args-first, and DeepSeek DSML incremental streaming. • Adds support for new models: Step-3.7-Flash, Cosmos3 Reasoner, Gemma 4 Unified (encoder-free), JetBrains Mellum v2, Granite Speech Plus, and Cohere Mini Code. • Adds Gemma 4 MTP and encoder-free Unified support. • Adds `LMCacheMPConnector` for LMCache in disaggregated serving. • Enables async EPLB by default and adds EPLB support for DeepSeek-V4 Mega-MoE. • Adds SSL support for the data-parallel supervisor. • Enables Triton MoE backend on Hopper by default. • Adds native W4A16 and fused-MoE W4A16 HIP kernels for RDNA3 (gfx1100) on AMD ROCm. • Adds AITER top-k/top-p sampler by default on AMD ROCm. • Adds Intel XPU support for CPU/tiering offloading, `block_fp8_moe`, block-scaled W8A8 FP8 path, and transparent sleep mode. • Adds zentorch-accelerated W8A8/W4A16 quantization on AMD Zen CPUs. • Adds RISC-V RVV WNA16 helpers and PowerPC SHM communicator. • Adds arm64 CI image. • Adds ModelOpt LM-head quantization and MXFP8 non-gated MoE support. • Adds `compressed-tensors` support for WNA8O8Int linears, WNInt embeddings, and asymmetric MoE WNA16 Marlin. • Adds Triton W4A16 as a CUDA fallback for non-Marlin-aligned shapes. • Adds `thinking_token_budget` validation in the frontend. • Adds NUMA auto-binding on DGX B300. • Adds extra_repr() for pooler classes in pooling/classification models. • Adds auto-selection of registered video loader for VLMs. • Adds Nixl zero-copy EPLB transfers and Nixl Mamba prefix-caching mode. • Adds Transformers v5 compatibility, including vendored MiniCPM-V/O processors and Voxtral `fetch_audio` for transformers≥5.10. • Adds DeepSeek-V4 TRTLLM-gen attention kernel, EPLB support, selective prefix-cache retention for sliding-window KV cache, and index-share feature for DSA MTP. • Adds sparse NCCL weight transfer for in-place model updates. • Adds per-GPU-worker RDMA NIC selection for data-parallel deployments. • Upgrades TPU inference to v0.21.0. • Adds CUTLASS FP8 scaled-mm padding bypass delivering ~20% performance improvement. • Adds MoE-permute buffer pre-allocation delivering 9–14% performance improvement. Breaking changes: • `NixlConnector` `kv_both` role has entered a deprecation cycle and should be migrated away from. • `supports_expert_map` has been removed from the MoE modular-kernel oracle. • The inplace fused-experts mechanism has been removed from the MoE oracle. • `cpu_awq` is folded into `awq_marlin`; configurations referencing `cpu_awq` must be updated to use `awq_marlin`. https://github.com/vllm-project/vllm/releases/tag/v0.22.1 https://github.com/vllm-project/vllm/releases/tag/v0.22.1 Fri, 05 Jun 2026 10:10:00 GMT vLLM v0.22.1 adds Mellum v2 model support and zentorch-accelerated quantized inference on AMD Zen CPUs. • Adds W8A8 (int8 dynamic-symmetric) and W4A16 (GPTQ) linear inference through zentorch kernels on AMD Zen CPUs, registered ahead of generic oneDNN CPU kernels, with transparent fallback on non-Zen CPUs, GPUs, and XPU. • Adds support for JetBrains' Mellum v2, an open-weights Mixture-of-Experts code-generation model. https://github.com/vllm-project/vllm/releases/tag/v0.22.0 https://github.com/vllm-project/vllm/releases/tag/v0.22.0 Fri, 29 May 2026 10:28:13 GMT vLLM v0.22.0 adds a Rust frontend, multi-tier KV cache offloading, batch-invariant FP8/NVFP4 paths, and broad new model/hardware support. • Adds `--cpu-distributed-timeout-seconds` CLI flag for CPU-distributed deployments. • Adds `thinking_token_budget` parameter support to the Completions API. • Adds `reasoning_effort` mapping to `enable_thinking` in the Completions API. • Adds `chat_template_kwargs` support to the Responses API. • Adds reset_cache() method to the KV offloading framework. • Adds `update_config` support to Model Runner V2. • Adds `logprob_token_ids` field support in Model Runner V2. • Adds `X-data-parallel-rank` header forwarding for data-parallel serving. • Adds API-key authorization for `/v2` endpoints. • Adds `AuthenticationMiddleware` path extraction simplification and `utility call-ID` newtype to the Rust frontend. • Adds `PoolingOfflineMixin` split for the pooling offline API. • Adds `QuantKey` with activation override, reworking `quantization_config` in the quantization framework. • Adds `ExpertMapManager` as part of the MoE refactor, with experts moved to the `experts/` directory and a `RoutedExperts` alias for FusedMoE. • Adds truncation-side control for OpenAI-compatible endpoints. • Adds `reasoning_content` → `reasoning` normalization in the frontend. • Adds consolidated Speech-to-Text entrypoints. • Adds beam-search consolidation via `BeamSearchMixin`. • Adds score/rerank chat-template instructions to the frontend. • Introduces experimental Rust frontend integration with in-tree code, a DP Supervisor for data-parallel serving. • Introduces multi-tier KV cache offloading framework with a Python filesystem secondary tier, DSv4 support, and Mooncake disk offloading via `MooncakeStoreConnector`. • Adds DeepSeek V4 model package reorganized into dedicated `vllm/models/deepseek_v4/` package with NVFP4 fused MoE, full and piecewise CUDA graph, and MTP speculative decoding. • Model Runner V2 (MRv2) is now default for Qwen3 dense models, with automatic fallback to MRv1 for unsupported features. • Adds batch-invariant inference with Cutlass FP8 support for a 28.9% end-to-end latency improvement and an NVFP4 Cutlass linear path. • Adds CutlassFP8 padding pre-processing yielding a 13.5% TTFT improvement. • Adds padded NVFP4 quantization kernel yielding 2.4–5.7% end-to-end improvement. • Adds fused RoPE+KVCache+q_concat kernel for MLA decoding. • Adds FULL CUDA graph capture for TRITON_MLA decode. • Adds breakable CUDA graph (experimental). • Adds RISC-V Vector Extension (RVV) optimized attention kernels with VLEN=256 support. • Adds fused GDN for AMX CPU and experimental Triton + MRv2 on CPU. • Adds MXFP4 linear layers with compressed-tensors integration. • Adds MXFP4 CPU W4A16 MoE and XPU mxfp8 MoE support. • Adds AutoRound W4A16 quantization support. • Adds ability to load Quark NVFP4 checkpoints. • Adds ModelOpt Qwen3.5/3.6 VLM quantized prefix mapping. • Adds GPTQ consolidation, migrating `gptq_marlin` → `auto_gptq`. • Adds support for new model architectures: MiniCPM-V 4.6, InternS2 Preview, OpenVLA, and EXAONE-4.5. • Adds custom callable proposer backend for speculative decoding. • Adds post-norm EAGLE-3 and peagle speculative decoding backends. • Adds non-MTP speculation support for NemotronH. • Adds Apertus tool parser. • Adds `anyOf`/`oneOf`/`$ref` resolution for Qwen3Coder tool calling. • Adds one-shot Triton kernel for MoE LoRA and simultaneous 2D & 3D MoE LoRA adapters. • Adds per-tensor FP8 CUTLASS on SM12.1 and FlashInfer b12x MoE + FP4 GEMM for SM120/121 (NVIDIA Blackwell). • Adds RMSNorm+Quant fusion for AMD gfx950 and XGMI backend for MoRI connector. • Adds Intel XPU GPTQ int4 support and FP8 block-scaled quantization. • Adds `head_dim=512` for FlashInfer TRTLLM attention. • Adds manual cumem allocator enable and stream-aware free callback to the allocator. • Adds KV cache event metadata emission. • Adds penalties Triton kernel. • Adds Mamba2 SSD kernel warmup, bf16 SSM cache, and GPU-side state postprocessing fused kernel. • Adds lease-renewal TTL for KV blocks and handshake-failure policy for disaggregated serving (NIXL). • Adds Mooncake operation metrics and HMA support for DSV4. • Adds DP Supervisor and publishes request counts at engine-step start for data-parallel serving. • Adds non-root `vllm-openai` Docker target. • Adds `setuptools-rust` dependency for Rust frontend builds. • Adds `cohere` MoE and pipeline parallelism for Cohere vision models. • Adds `runai-streamer` weight loading for Qwen3.5/MTP/Qwen3-VL. Breaking changes: • Removed old locations of `get_tokenizer` and `resolve_hf_chat_template`; callers must update their import paths. • Removed deprecated MLA prefill arguments; existing code using those arguments will break. • Marked environment variables now covered by `--moe-backend` / `--linear-backend` as deprecated. https://github.com/vllm-project/vllm/releases/tag/v0.21.0 https://github.com/vllm-project/vllm/releases/tag/v0.21.0 Fri, 15 May 2026 08:44:26 GMT vLLM v0.21.0 adds Blackwell TOKENSPEED_MLA backend, KV offload HMA integration, thinking-budget spec decode, and new RLHF/Responses API endpoints. • Adds `VLLM_SKIP_MODEL_NAME_VALIDATION` environment variable to bypass model name validation at startup. • Adds explicit `/start_weight_update` and `/finish_weight_update` API endpoints for RLHF weight-update workflows. • Adds `system_fingerprint` field to OpenAI-compatible chat completion responses. • Adds `prompt_embeds` content part support to OpenAI-compatible API. • Adds `defer_loading` and `tool_reference` support to OpenAI-compatible API. • Adds `logprob_token_ids` support in Model Runner V2. • Adds configurable safetensors checkpoint prefetch. • Adds `max` reasoning effort support for DeepSeek V4. • Adds new `TOKENSPEED_MLA` attention backend for DeepSeek-R1 and Kimi-K25 prefill and decode on NVIDIA Blackwell GPUs. • Adds `MooncakeStoreConnector` for distributed KV offloading. • Adds `PluggableLayer` interface enabling out-of-tree MoE runners. • Integrates KV offloading subsystem with the Hybrid Memory Allocator (HMA), including scheduler-side sliding window group support and multi-connector HMA. • Adds bi-directional KV cache transfers between prefill (P) and decode (D) nodes in disaggregated serving. • Adds Mooncake `KVConnectorStats` for KV transfer observability in disaggregated serving. • Enables independent drafter attention backend selection for speculative decoding. • Adds EAGLE speculative decoding support for Mistral, Gemma4 MTP, MiMo-V2.5 MTP, and Cohere Eagle. • Adds tool calling support via XGrammar 0.2.0 with structural tags for strict tool calling combined with reasoning. • Adds Cohere reasoning and tool parsers, plus LFM2/2.5 tool parser. • Adds Responses API streaming tool/function calling with `required` and named tool/function choice, plus resubmitting output items with missing fields. • Adds rendered prompt text in chat completion response. • Adds Fastokens support in the tokenizer. • Adds `RayExecutorV2` enabled by default for Ray-based execution. • Adds IndexCache support for DSA models. • Enables FlashInfer top-k/top-p sampler by default. • Adds OOM prevention via `max_split_size_mb` during model loading. • Adds two-phase pause mechanism to prevent scheduler deadlock. • Adds NIXL connector bumped to 1.x for disaggregated serving. • Adds new model architecture support: MiMo-V2.5, Laguna XS.2, Moondream3, Qianfan-OCR, Cohere MoE, and Cohere Eagle. • Adds AMD ROCm support for DeepSeek V4, including TP4 AITER MLA and pipeline parallelism. • Adds AMD Dynamic Batch Optimization (DBO) on ROCm. • Adds CPU FP8 attention for AMX/AVX-512, FP8 W8A16 linear, FP8 W8A16 MoE, and DNNL AVX2 W8A8 Int8. • Adds Intel XPU top-k/top-p sample kernel and LoRA support. • Adds IBM Power VSX attention backend. • Adds NVFP4 KV cache support and Triton dequant/QDQ emulation kernels for Hopper and AMD. • Adds Humming MXFP4 MoE backend. • Adds TurboQuant hybrid model and uniform quantization support. • Adds configurable model weights loading tracking. • Adds Triton JIT compilation monitor. • Adds LoRA initial expert parallel (EP) support. • Adds ViT CUDA graph support for Qwen2.5-VL. • Adds Qwen3.5/Mamba hybrid model support in Model Runner V2. • Adds Gated DeltaNet Attention for Qwen 3.5/3.6 on CPU. • Adds RISC-V OMP thread auto-binding for CPU backend. • Reduces Docker image size by approximately 2.5 GB via deferred FlashInfer cubin download. • Embeds container image provenance metadata. Breaking changes: • C++20-compatible compiler is now required to build vLLM (for PyTorch compatibility); builds with older compilers will fail. • `transformers` v4 is formally deprecated; users must migrate to `transformers` v5. https://github.com/vllm-project/vllm/releases/tag/v0.20.0 https://github.com/vllm-project/vllm/releases/tag/v0.20.0 Mon, 27 Apr 2026 21:20:28 GMT vLLM v0.20.0 adds DeepSeek V4, TurboQuant 2-bit KV cache, FA4 default MLA prefill, online quantization frontend, and CUDA 13.0 as default. • Adds `--enable-vit-cuda-graph` flag for VLM examples to enable ViT full CUDA graph for video inference. • Adds `VLLM_MEDIA_CACHE` opt-in environment variable for media URL caching. • Adds `request_id` field on `FinishedRequestStats` for per-request completion tracking. • Adds `--torch-backend=cu129` recommended install option via `uv` for CUDA 12.9 users. • Adds `donate_graph_module=True` support for `standalone_compile` in torch.compile integration. • Respects `TORCH_COMPILE_DISABLE` at vLLM config level to disable compilation without code changes. • Adds labeled waiting-breakdown metric distinguishing capacity vs. deferred queue states. • Introduces `RayExecutorV2` for unified engine process monitoring with the Ray backend. • Adds shutdown() method on `OffloadingConnector` for clean KV offload teardown. • Adds `CONFIG_REGISTRY` config-class lookup to resolve on-disk `model_type` mismatches. • Enables CUDAGraph memory profiling by default for clearer startup memory accounting. • Adds `default_max_frames_per_batch` auto-inference for ViT CUDA graph video workloads. • Introduces `mamba_ssm_cache_dtype=float32` default for NemotronH with auto-hook for NemotronHNanoVLV2. • Default CUDA wheel on PyPI and `vllm/vllm-openai:v0.20.0` Docker image switched to CUDA 13.0 (13.0.2). • Adds initial DeepSeek V4 model architecture support. • Adds Hunyuan v3 (Hy3) preview model with HYV3 reasoning parser. • Adds Granite 4.1 Vision as a built-in multimodal model. • Adds EXAONE-4.5, BharatGen Param2MoE, Phi-4-reasoning-vision-15B, Cheers multimodal, telechat3, FireRedLID, jina-reranker-v3, Jina Embeddings v5, and Nemotron-v3 VL Nano/Super model architectures. • Enables FlashAttention 4 as the default MLA prefill backend with head-dim 512 and paged-KV on SM90+. • Introduces TurboQuant 2-bit KV cache attention backend delivering 4× KV cache capacity, now with FA3/FA4 prefill support. • Introduces new end-to-end online quantization frontend consolidating FP8, `experts_int8`, and MXFP8 online quantization paths. • Adds support for `transformers>=5` (HuggingFace Transformers v5). • Adds Python 3.14 to the supported version list. • Ships with PyTorch 2.11 for CUDA and XPU (XPU no longer pinned to 2.10). • Adds full CUDA graph support for Eagle prefill in Model Runner V2. • Adds auto-resolution of CUDA graph mode and sizes from the attention backend. • Adds multiple prompt-logprobs support in Model Runner V2. • Introduces vLLM IR skeleton with `rms_norm` op as foundation for future kernel work. • Adds Helion kernel support for torch.compile. • Adds FlexAttention non-causal support. • Adds NUMA binding for GPU workers. • Adds CPU draft-model speculative decoding. • Adds MXFP4 W4A4 CUTLASS MoE support for SM100 (Blackwell). • Adds `fused_silu_mul_block_quant` kernel on AMD ROCm. • Adds MLA dual RMS norm fusion pass for DeepSeek/Kimi-K2 on ROCm. • Adds ZenCPU / AMD Zen CPU backend via zentorch. • Adds RDNA 3.5/4 device IDs (`gfx1150`/`gfx1151`/`gfx1201`) and `gfx1102`/`gfx1103` support on ROCm. • Adds XPU MXFP8 and MXFP4 quantization ops for Intel XPU. • Adds FP8 KV cache support on Intel XPU. • Adds per-channel FP8 linear on Intel XPU. • Adds `sm_110` (Jetson Thor) to CUDA 13.0 build targets. • Adds RISC-V multiple RVV VLEN target support. • Adds Eagle3 speculative decoding for MiniMax-M2 and Gemma4. • Adds LoRA support for `Qwen3ASRForConditionalGeneration`, `Gemma4ForConditionalGeneration`, and DeepSeek V3.2. • Adds GGUF support for MiniMax-M2.1 and non-standard GGUF quant types with prefix such as `UD-IQ1_S`. • Adds batched KV-cache swap via `cuMemcpyBatchAsync` for NVIDIA GPUs. • Adds DFlash attention backend on ROCm. • Adds fused `qknorm+rope` kernel on SM9.0. • Adds `mm-scheduler` overhead reduction via `get_num_embed` optimization. • Adds AOT compile with batch-invariance mode and Inductor cache nested under AOT directory. • Adds IBM Z s390x torch 2.11 build targets. • Achieves 2.1% E2E latency improvement via fused RMS norm batch-invariant optimization. Breaking changes: • PyTorch upgraded to 2.11 for CUDA and XPU — existing environments pinned to an earlier torch version will break; reinstallation required. • Default CUDA wheel switched to CUDA 13.0.2; users on CUDA 12.9 must explicitly install with `uv` and `--torch-backend=cu129`. • XPU is no longer pinned to torch 2.10; environments depending on torch 2.10 for XPU will require updating. • `SharedFusedMoE` removed; code or configs referencing it will fail. • `make_expert_params_mapping` renamed; call sites using the old name will break. • MoE DP chunking removed; configurations relying on it will no longer function. • Piecewise-fallback disabled for Eagle draft decodes; setups that depended on this fallback path will behave differently. https://github.com/vllm-project/vllm/releases/tag/v0.19.1 https://github.com/vllm-project/vllm/releases/tag/v0.19.1 Sat, 18 Apr 2026 05:44:42 GMT vLLM v0.19.1 upgrades to Transformers v5.5.3 and adds Gemma4 Eagle3 and quantized MoE support. • Adds Gemma4 Eagle3 speculative decoding support. • Adds quantized MoE support for Gemma4 models. • Upgrades the Transformers dependency to v5.5.3. https://github.com/vllm-project/vllm/releases/tag/v0.19.0 https://github.com/vllm-project/vllm/releases/tag/v0.19.0 Fri, 03 Apr 2026 02:19:12 GMT vLLM v0.19.0 adds Gemma 4, a new `/v1/chat/completions/batch` endpoint, CPU KV cache offloading, zero-bubble async spec decode, and `VLLM_MAX_N_SEQUENCES` security control. • Adds `/v1/chat/completions/batch` endpoint for batched chat completions. • Adds `VLLM_MAX_N_SEQUENCES` environment variable to enforce a hard cap on concurrent sequences, limiting resource exhaustion risk. • Adds `--lora-target-modules` flag (shorthand: no shorthand listed) to restrict LoRA adapters to specific model modules. • Adds `-sc` as a shorthand alias for `--speculative-config`. • Adds `--speculative-config` option for per-draft-model MoE backend selection in speculative decoding. • Adds hard limit on thinking tokens via the API (limit thinking tokens feature). • Adds online `MXFP8` quantization support for both MoE and dense models. • Adds `QeRL` online quantization composed with quantized reloading, enabling quantization-aware RLHF workflows. • Adds `W4A16` compressed tensors quantization support for CPU backends. • Adds `CompressedTensor W4A8` quantization support for Intel XPU. • Adds `AWQ Marlin` support for ROCm backends. • Adds full Gemma 4 architecture support including MoE, multimodal, reasoning, and tool-use; requires `transformers>=5.5.0` (recommended image: `vllm/vllm-openai:gemma4`). • Adds zero-bubble async scheduling with speculative decoding overlap, improving throughput. • Adds general CPU KV cache offloading for V1 with pluggable `CachePolicy` and block-level preemption. • Adds full CUDA graph capture for Vision Transformer (ViT) encoders. • Adds piecewise CUDA graphs for pipeline parallelism in Model Runner V2. • Adds configurable acceptance rate for Model Runner V2 spec decode. • Adds DBO (Dual-Batch Overlap) microbatch optimization generalized to all model architectures. • Adds NVIDIA B300/GB300 (SM 10.3) support with allreduce fusion enabled by default and tuned all-reduce communicator. • Adds Triton autotuning disk cache enabled by default. • Adds `FlexAttention` custom mask modification support. • Enables NFS prefetch with automatic RAM guard. • Adds support for multiple embedding types in a single API call. • Adds numpy array embeddings for multimodal inputs. • Adds GigaChat 3.1, Kimi-K2.5, and Gemma 4 tool parsers. • Adds new model architectures: Cohere ASR, Cohere Transcribe, ColQwen3.5 4.5B, LFM2-ColBERT-350M, Granite 4.0 1B Speech, Qwen3-ForcedAligner. • Adds Eagle3 speculative decoding support for Pixtral. • Adds DeepEP as all-to-all backend for AMD ROCm. • Adds persistent MLA kernel from AITER and FP8xFP8 attention in AITER for ROCm. • Adds ROCm 7.2.1, torch 2.10, triton 3.6 build support. • Adds async scheduling interface for TPU. • Adds MLA model support and `CompressedTensor W4A8` for Intel XPU with auto-detect XPU build platform. • Enables `tcmalloc` by default on CPU with graceful degradation when unavailable. • Adds CPU slot mapping kernel and achieves 48.9% throughput improvement for pooling models on CPU. • Adds `PD kv_transfer_params` support for Anthropic Messages and Responses API in disaggregated serving. • Adds Mooncake heterogeneous TP support for disaggregated serving. • Adds tensor IPC transfer for multimodal data. • Adds `PluggableLayer` extensibility API for custom decoder implementations (e.g., `CustomQwen2Decoder`). • Adds plugin-overridable metadata build for the KV connector. • Adds `Mega AOT artifact` compilation support for torch 2.12+. • Adds frame limit enforcement in `VideoMediaIO` to prevent resource exhaustion. • Adds `torch profiler with stack` logging option. • Adds log-once-per-node default behavior to reduce log noise in multi-node deployments. • Removes per-tensor-per-channel FP8 and Sparse24 integration and kernels. Breaking changes: • Per-tensor-per-channel FP8 support has been removed. • Sparse24 integration and kernels have been removed. • The `reasoning_content` message field has been removed from the API. • `--calculate-kv-scales` is deprecated. • The `score` task is deprecated. • Pooling multi-task support is deprecated. • Virtual engine is deprecated (V0 deprecation path). • `--disable-frontend-multiprocessing` is deprecated. https://github.com/vllm-project/vllm/releases/tag/v0.18.0 https://github.com/vllm-project/vllm/releases/tag/v0.18.0 Fri, 20 Mar 2026 21:31:36 GMT vLLM v0.18.0 adds gRPC serving, GPU-less render launch, NGram GPU speculative decoding, FlexKV offloading, and streaming tool calls in the Responses API. • Adds `--grpc` flag to enable gRPC serving alongside the existing HTTP/REST interface for high-performance RPC-based inference. • Adds `vllm launch render` subcommand for GPU-less multimodal preprocessing and rendering, separating preprocessing from GPU inference. • Adds `--enable-ep-weight-filter` CLI option for faster Elastic Expert Parallelism model loading via NIXL-EP integration. • Adds `--distributed-timeout-seconds` option for configuring distributed operation timeouts. • Adds `--attention-backend auto` option for automatic attention backend selection. • Adds `reasoning_effort=none` API option to disable reasoning effort. • Adds PyTorch profiler schedule configuration option. • NGram speculative decoding now runs on GPU and is compatible with the async scheduler, reducing speculative decode overhead. • Adds FlexKV as a new KV cache offloading backend with support for multiple KV groups in the offloading spec. • Adds smart CPU offloading that stores only frequently-reused KV cache blocks (reuse-frequency-gated CPU stores). • Adds streaming tool/function calling support to the OpenAI Responses API. • Adds online beam search transcription support for ASR encoder/decoder models, plus offline beam search, and audio transcription for MP4/M4A/WebM formats. • Adds Cohere Embed v2 API support. • Adds Azure Blob Storage support for RunAI Model Streamer. • Adds graceful shutdown timeout for in-flight requests. • Adds Anthropic API support for redacted thinking blocks. • Adds Granite4 tool parser. • Adds NIXL-EP integration for dynamic GPU scaling of MoE experts (Elastic EP Milestone 2). • Adds HMA + NIXL KV connector and All-to-All DCP backend for KV transfer. • Adds ModelOpt MXFP8 MoE quantization support. • Adds MXFP4 MoE routing simulation override for accuracy control. • Adds FP8 LoRA dense kernel. • Adds FA4 kernel support for MLA prefill. • Adds FlashInfer Sparse MLA FP8 KV cache support and CUDA graphs on ROCm. • Adds TRTLLM FP8 MoE modular kernel. • Adds 320-dimension MLA head size support. • Adds support for RISC-V CPU backend. • Adds AMD Zen CPU backend via zentorch. • Adds aarch64 int8 matmul via OneDNN upgrade. • Adds gfx1152/gfx1153 (Krackan) GPU support on ROCm. • Adds ROCm Quark W4A8 MXFP4/FP8 quantization support for LinearLayer. • Adds Intel XPU Model Runner V2, MLA Sparse backend for DeepSeek V3.2, and LoRA via torch.compile. • Adds OOT linear method registration and custom collective ops registration for non-CUDA platforms for extensibility. • Adds new model architectures: Sarvam MoE, OLMo Hybrid, HyperCLOVAX-SEED-Think-32B VLM, HyperCLOVAX-SEED-Think-14B, Kimi-Audio-7B-Instruct, ColPali late-interaction retrieval, and ERNIE pooling models. • Adds Eagle3 speculative decoding support for Qwen3.5 and Kimi K2.5 MLA, plus Eagle for Mistral Large 3 with dense layers. • Adds Whisper LoRA support. • Adds online `use_audio_in_video` multimodal capability, audio extraction from MP4 for Nemotron Nano VL, and `media_io_kwargs` exposure at runtime. • Updates FlashInfer dependency to version 0.6.6 with performance and correctness improvements. • Removes Ray as a default dependency — must now be installed explicitly. Breaking changes: • Ray is no longer a default dependency; existing setups that rely on Ray must install it explicitly after upgrading. • Cascade attention is now disabled by default; workloads that relied on cascade attention being on by default will see changed behavior. • The `swap_space` parameter has been removed (was deprecated in V0); configs or CLI invocations using `swap_space` will fail. • Monolithic TRTLLM MoE is disabled for renormalize routing; setups depending on monolithic TRTLLM MoE with renormalize routing will be affected. https://github.com/vllm-project/vllm/releases/tag/v0.17.1 https://github.com/vllm-project/vllm/releases/tag/v0.17.1 Wed, 11 Mar 2026 10:24:34 GMT vLLM v0.17.1 adds support for the Nemotron 3 Super model. • Adds support for the Nemotron 3 Super model. https://github.com/vllm-project/vllm/releases/tag/v0.17.0 https://github.com/vllm-project/vllm/releases/tag/v0.17.0 Sat, 07 Mar 2026 00:46:41 GMT vLLM v0.17.0 adds FlashAttention 4, `--performance-mode`, Anthropic `count_tokens`, quantized LoRA, and PyTorch 2.10. • Adds `--performance-mode {balanced, interactivity, throughput}` flag to simplify performance tuning for common deployment scenarios. • Adds `--moe-backend` flag for explicit MoE kernel selection. • Adds `--language-model-only` flag for hybrid models. • Adds Anthropic `count_tokens` API support. • Adds Anthropic `tool_choice=none` support. • Adds Anthropic thinking blocks support. • Integrates FlashAttention 4 backend for next-generation attention performance. • Supports loading quantized LoRA adapters (e.g. QLoRA) directly. • Adds AOT compile support with PyTorch 2.10. • Enables AR+RMSNorm fusion by default at `-O2` and SiLU+FP4 quant fusion by default at `O1+` for `torch.compile`. • Adds MFU Prometheus counters for observability. • Adds unrecognized environment variable warnings. • Adds per-head KV cache scales in the attention selector. • Adds `sleep level 0` mode with enqueue/wait pattern. • Adds native weight syncing API via IPC for RL workflows. • Adds repetitive token pattern detection flags. • Adds Triton-based top-k and top-p sampler kernels. • Adds FlashInfer Sparse MLA backend. • Adds Helion kernel framework with autotuning infrastructure. • Adds NIXL Token-based IPC API. • Adds Elastic Expert Parallelism Milestone 2 for dynamic GPU scaling on MoE models. • Adds MoERunner abstraction with modular kernel architecture. • Adds `min_tokens` support with speculative decoding. • Adds Nemotron-H MTP and Mamba speculative decoding. • Adds Eagle3 speculative decoding with CUDA graphs on Model Runner V2. • Adds Pipeline Parallel and Decode Context Parallel support to Model Runner V2. • Adds weight offloading V2 with prefetching to hide onloading latency, selective CPU weight offloading, and CPU offloading without pinned memory doubling. • Adds support for new model architectures: Qwen3.5 (with GDN, FP8 quantization, MTP speculative decoding, and reasoning parser), COLQwen3, ColModernVBERT, Ring 2.5, skt/A.X-K1, Ovis 2.6, nvidia/llama-nemotron-embed-vl-1b-v2, nvidia/llama-nemotron-rerank-vl-1b-v2, nvidia/nemotron-colembed. • Adds ASR model support: FunASR, FireRedASR2, Qwen3-ASR realtime streaming. • Adds OpenPangu-VL video input, audio chunking for offline LLM, and Parakeet audio encoder. • Adds Whisper automatic language detection. • Adds structured outputs support to the Responses API. • Adds structured output JSON feature validation. • Adds explicit validation error for tool calls. • Adds sparse embedding IO process plugin. • Adds IO Processor plugin simplification. • Adds LoRA support for LFM2 and Llama 4 Vision tower/connector; increases max LoRA vocab size to 258048. • Adds SM100 MXFP8 blockscaled grouped MM and quantization kernels. • Adds AITER fused RoPE+KVCache for AMD ROCm. • Adds CUDA graph support for Intel XPU. • Adds ARM BF16 cross-compilation and s390x FP16 support for CPU builds. • CPU release now supports AVX2, AVX-512, VNNI, AVX512BF16, and AMX via a multi-ISA dispatcher. • Upgrades to PyTorch 2.10.0. • Adds Async EPLB rebalance algorithm. • Adds Ray environment variable propagation to workers. • Adds mixed precision support for ModelOpt. • Adds Llama-4 attention quantization (int8, fp8). • Adds bitsandbytes quantization support on AMD ROCm. • Adds GPT-OSS Quark format support on AMD ROCm. • Adds `generation_config` max_tokens treated as a default rather than a ceiling. • Adds validation of non-text content in system messages. • Adds ColBERT extension to non-standard BERT backbones and multimodal scoring for late-interaction models. • Adds KV cache update ops extraction from FlashInfer forward and MLA backends. • Adds TRTLLM DSV3 Router GEMM kernel with 6% batch-1 speedup. • Adds pooling model support to Model Runner V2 with 13.9% throughput improvement. • Pipeline Parallel async send/recv yields 2.9% E2E throughput improvement. • Adds Reasoning API streaming `reasoning_part` events. • Adds `amd-aiter` package (renamed from `aiter`) for AMD ROCm. Breaking changes: • PyTorch upgraded to 2.10.0, which is a breaking change for environment dependencies. • KV load failure policy default changed from 'recompute' to 'fail' (controlled in large-scale serving configuration). • AMD ROCm `aiter` package renamed to `amd-aiter`; any environment or install referencing `aiter` by name must be updated. https://github.com/vllm-project/vllm/releases/tag/v0.16.0 https://github.com/vllm-project/vllm/releases/tag/v0.16.0 Wed, 25 Feb 2026 19:58:49 GMT vLLM v0.16.0 adds a WebSocket Realtime API, native NCCL weight-sync for RLHF, unified parallel speculative decoding, and major XPU/ARM/NVIDIA kernel expansions. • Adds `--disable-access-log-for-endpoints` CLI flag to suppress access logging for specific endpoints. • Adds `GGUF repo_id:quant_type` syntax for selecting quantization variants when loading GGUF models. • Adds nested config support in YAML files for vLLM server configuration. • Adds `FlashInfer autotune control flag` (#34006) to manage FlashInfer autotuning behavior at runtime. • Adds native NCCL-based weight syncing API for RLHF workflows, enabling direct weight updates between training and inference engines without third-party communication layers. • Adds engine pause/resume with request preservation for RLHF workflows, allowing training loops to halt inference without dropping in-flight requests. • Adds layerwise weight reloading for QeRL in RLHF workflows. • Adds WebSocket-based Realtime API for streaming audio interactions, including Voxtral realtime support. • Adds sampling parameters and token/prompt token ID return fields to the Responses API. • Adds `/server_info collect_env` endpoint for retrieving environment diagnostics from a running server. • Adds OTEL tracing during model loading for observability of startup phases. • Adds `run_batch` transcription and translation support. • Adds HF Hub LoRA resolver (#20320) for loading LoRA adapters directly from Hugging Face Hub. • Adds KV offloading connector metrics and labeled prompt token metrics for P/D disaggregation. • Adds Unified Parallel Drafting for speculative decoding, delivering throughput improvements alongside structured output support and penalty application in Model Runner V2. • Adds async scheduling with Pipeline Parallelism, delivering 30.8% end-to-end throughput improvement and 31.8% TPOT improvement. • Adds FP8 block quant for `CompressedTensorsW8A16Fp8`, ModelOpt MXFP8 for dense models, and NVFP4/FP8 support on Turing GPUs. • Adds TP > 4 support for FP4 Gemm quantization. • Adds Mooncake disaggregated serving connector rework with bootstrap server and cross-layer KV cache layout at NIXL Connector V2. • Adds EPLB logical expert capture with router replay for expert-parallel load balancing. • Adds MoE cold start optimization option for `torch.compile` to reduce compilation overhead. • Adds Intel XPU support for unquantized MoE, MXFP4 MoE, WNA16, `scaled_mm`, and FP8 MoE kernels via vllm-xpu-kernels (replacing IPEX). • Adds ARM CPU KleidiAI INT4 dynamic quantization with BF16 activations and NEON BFMMLA BF16 paged attention. • Adds IBM Z BF16 kernel type for s390x architecture. • Adds SM100 INT4 W4A16 kernel and SM121 (DGX Spark) CUTLASS support for NVIDIA GPUs. • Adds MNNVL protocol support for GB series NVIDIA hardware. • Adds ORJSONResponse for faster API response serialization. • Adds DeepSeek ReasoningParser with thinking enabled by default. • Adds early tokenization validation to surface malformed inputs before scheduling. • Adds LoRA expansion for Gemma3 vision components, Nemotron-H MTP models, and Qwen3 output embedding. • Adds new model architecture support for GLM-OCR with MTP, Qwen3-ASR, DeepSeek-OCR-2, Intern-S1-Pro, MiniCPM-o 4.5, openPangu7B-VL, NemotronHPuzzle heterogeneous, MusicFlamingo, FunAudioChat, ColBERT late interaction, voyage-4-nano, and GLM-5. • Adds EAGLE3 speculative decoding support for Hunyuan/HunyuanVL, AFMoE, and Mistral3. • Adds Pooling API request schema consensus for `ScoreRequest` and final standardization. • Adds PluggableLayer support applied to linear layers and Mamba layers. • Adds Helion kernel framework components: `ConfigManager`, kernel wrapper, and kernel registry. • Only includes the `Authorization` header in outbound requests when `OPENAI_API_KEY` is set, reducing credential leakage risk. Breaking changes: • Removed BitBlas quantization backend — workloads using BitBlas will fail after upgrade. • Removed Marlin 24 quantization backend — workloads using Marlin 24 will fail after upgrade. • Removed deprecated `reasoning_content` message field — code referencing this field will break. • Removed deprecated pooling items — any configuration or API calls relying on the deprecated pooling endpoints/fields will break. • Removed deprecated `VLLM_ALL2ALL_BACKEND` environment variable — scripts setting this variable will have no effect (and previously working behavior dependent on it is gone). • Deprecated IPEX for Intel XPU; switched to vllm-xpu-kernels — existing XPU deployments using IPEX will need to migrate to vllm-xpu-kernels. https://github.com/vllm-project/vllm/releases/tag/v0.15.1 https://github.com/vllm-project/vllm/releases/tag/v0.15.1 Wed, 04 Feb 2026 20:48:08 GMT vLLM v0.15.1 adds Step-3.5-Flash model support and restores RTX Blackwell GPU compatibility. • Adds support for the Step-3.5-Flash model. https://github.com/vllm-project/vllm/releases/tag/v0.15.0 https://github.com/vllm-project/vllm/releases/tag/v0.15.0 Thu, 29 Jan 2026 10:21:01 GMT vLLM v0.15.0 adds Kimi-K2.5/Molmo2/Step3vl models, Mamba prefix caching, async scheduling with PP, FIPS 140-3 hashing, and new API fields. • Enables `--async-scheduling` with pipeline parallelism, previously unsupported. • Adds Mamba/hybrid model prefix caching via `--enable-prefix-caching --mamba-cache-mode align`, achieving ~2x speedup by caching Mamba states directly. • Adds `--ssl-ciphers` CLI argument for TLS cipher suite control. • Adds `skip_special_tokens` configuration to the OpenAI-compatible API. • Adds `prompt_cache_key` support to the Responses API. • Adds `include_stop_str_in_output` tuning parameter to the Responses API. • Adds `data_1`/`data_2` and `queries`/`documents` flexible input formats to the score endpoint. • Adds new render endpoints for prompt preprocessing. • Adds `avg_logprob` and `compression_ratio` fields to Whisper API verbose_json segment output. • Adds FIPS 140-3 compliant hash option for enterprise/government deployments. • Auto-detects `api_server_count` based on `dp_size` to simplify multi-process server configuration. • Adds support for new model architectures: Kimi-K2.5, Molmo2, Step3vl 10B, Step1, GLM-Lite, and Eagle2.5-8B VLM. • Extends LoRA support to Nemotron-H, InternVL2, and MiniMax M2 models. • Adds EAGLE3 speculative decoding for Pixtral/LlavaForConditionalGeneration and Qwen3 VL MoE. • Adds BGE-M3 sparse embeddings and ColBERT embeddings support. • Adds VLM support to Model Runner V2. • Adds inplace LoRA loading for improved memory efficiency. • Adds `torch.compile` inductor AOT artifact support. • Adds `torch.compile` support for Whisper. • Adds MXFP4 W4A16 quantization support for compressed-tensors MoE models. • Adds quantization support for non-gated MoE with Marlin, NVFP4 CUTLASS, FP8, INT8, and compressed-tensors. • Adds Intel Quantization Toolkit integration. • Adds FP8 KV cache per-tensor and per-attention-head quantization via llmcompressor. • Adds session-based streaming input accepting async generators producing `StreamingInput` objects for interactive workloads like ASR. • Sets FlashInfer MLA as the default MLA backend on Blackwell, with TRTLLM as default prefill. • Adds MoRI EP high-performance all2all backend for Expert Parallel on AMD ROCm. • Adds Flash Attention Triton backend support for AMD RDNA3/RDNA4 consumer GPUs. • Adds TPU pipeline parallelism support. • Adds NUMA-aware acceleration for TP/DP inference on ARM CPU. • Adds wheel variant auto-detection during install. • Adds custom profiler URI scheme support. Breaking changes: • The `vllm:time_per_output_token_seconds` metric has been removed — use `vllm:inter_token_latency_seconds` instead. • Several deprecated environment variables have been removed. • DeepSpeedFp8 quantization has been removed. • RTN quantization has been removed. • HQQ quantization is deprecated. https://github.com/vllm-project/vllm/releases/tag/v0.14.0 https://github.com/vllm-project/vllm/releases/tag/v0.14.0 Tue, 20 Jan 2026 09:20:31 GMT vLLM v0.14.0 adds a gRPC server, `--max-model-len auto`, async scheduling by default, and broad new model/quantization support. • Adds `--max-model-len auto` flag to automatically fit context length to available GPU memory, eliminating OOM startup failures. • Adds `--no-async-scheduling` flag to disable the newly default async scheduling (overlaps engine core scheduling with GPU execution for higher throughput). • Adds `-ep` shorthand flag for `--enable-expert-parallel`. • Adds `--enable-log-deltas` CLI flag (renamed in this release) for iteration delta logging. • Adds `--default-chat-template-kwargs` CLI flag to pass default kwargs to chat templates. • Adds `--input-len` flag and auto-discovery to `bench serve` for benchmark configuration. • Adds `VLLM_LOG_MODEL_INSPECTION=1` environment variable (or printing the `LLM` object) to inspect a model's modules, attention backends, and quantization configuration. • Adds `attention_config` parameter to the LLM() constructor for programmatic attention configuration. • Adds `reasoning_effort` parameter to the API for controlling reasoning depth. • Adds gRPC server entrypoint as an alternative to the REST API, supporting binary protocol and HTTP/2 multiplexing. • Adds `/server_info` API endpoint exposing environment information. • Adds `continue_final_message` field to the `/embeddings` API endpoint. • Adds MCP streaming support in the Responses API. • Adds MFU (Model FLOP Utilization) metrics to the API. • Adds spec-decode acceptance stats reporting via CLI. • Adds offline FastAPI documentation endpoint. • Adds FunctionGemma and GLM-4.7 tool-calling parsers. • Adds new model architecture support: Grok-2 (with tiktoken tokenizer), LFM2-VL, MiMo-V2-Flash, openPangu MoE, IQuestCoder, Nemotron Parse 1.1, GLM-ASR audio, Isaac vision model v0.1/v0.2, Kanana-1.5-v-3b-instruct, and K-EXAONE-236B-A23B MoE. • Expands LoRA support to multimodal tower/connector models (LLaVA, BLIP2, PaliGemma, Pixtral, DotsOCR, GLM4-V) and additional architectures (DeepSeek-OCR, Qwen3-Next, NemotronH, PLaMo 2/3). • Adds `enable_thinking: false` config option for GLM-4.5/GLM-4.7 models. • Adds Marlin quantization support for Turing (sm75) GPUs. • Adds Quark int4-fp8 w4a8 MoE quantization. • Adds MXFP4 W4A16 quantization for dense models. • Adds ModelOpt FP8 variants: `FP8_PER_CHANNEL_PER_TOKEN` and `FP8_PB_WO`. • Adds XBO (Extended Dual-Batch Overlap) for large-scale serving. • Adds NIXL asymmetric TP support (P > D tensor-parallel-size). • Async scheduling now also works with speculative decoding and structured outputs. • Adds hybrid allocator + KV connector with multiple KV cache groups. • Adds SM103 hardware support. • Adds `weights_only=True` to `torch.load` calls, reducing deserialization attack surface. Breaking changes: • Async scheduling is now enabled by default; disable with `--no-async-scheduling` if issues arise (pipeline parallel, CPU backend, and non-MTP/Eagle spec decoding are excluded and remain unaffected). • PyTorch 2.9.1 is now required and the default wheel is compiled against cu129. • Deprecated quantization schemes have been removed. • When using speculative decoding, unsupported sampling parameters now fail with an error rather than being silently ignored. https://github.com/vllm-project/vllm/releases/tag/v0.13.0 https://github.com/vllm-project/vllm/releases/tag/v0.13.0 Fri, 19 Dec 2025 03:02:22 GMT vLLM v0.13.0 adds new models, `--attention-backend` CLI arg, multi-vector retrieval, MCP tooling, and major Whisper/DeepSeek performance gains. • Replaces `VLLM_ATTENTION_BACKEND` environment variable with `--attention-backend` CLI argument via new `AttentionConfig`. • Adds `encoding_format=bytes_only` to the embeddings API for binary-format embedding output. • Adds `/reset_prefix_cache` endpoint to KV connectors for cache reset in disaggregated prefill/decode deployments. • Adds `compile_ranges` config for conditional/selective kernel compilation. • Adds xxHash as a high-performance hash option for prefix caching. • Adds multi-vector retrieval API for pooling tasks. • Adds chunked prefill support for ALL pooling tasks. • Adds MCP type infrastructure, Browser/Container MCP tools, and full MCP Python loop to the Responses API. • Adds `extra body parameters` support to the Responses API. • Adds min-p sampling support in Model Runner V2. • Adds NaN detection in logits in Model Runner V2. • Adds prefill KV compute metric that excludes cached tokens. • Adds layer-wise NVTX profiling and a profiling CLI config. • Adds automatic TokenClassification model conversion. • Adds support for NVIDIA Blackwell Ultra SM103 (GB300) with CUDA 13. • Adds W4A8 grouped GEMM quantization on Hopper GPUs. • Adds online FP8 quantization with streaming post-processing. • Adds FP8 weight reloading for RLHF workflows. • Adds AWQ Marlin and GPTQ Marlin quantization support for MoE + LoRA. • Adds `TRITON_MLA` attention support without requiring prefix-caching. • Adds PrefixLM support for FlexAttention and TritonAttention backends. • Adds CUDA graphs for 3D Triton attention. • Adds Mooncake Transfer Engine as a KV connector. • Adds KV events support and failure recovery config for KV connectors. • Adds external launcher mode for multi-node serving. • Adds NIXL compatibility checking in handshake and large-batch proxy support. • Adds AMD ROCm MXFP4 w4a4 inference and Aiter quantization kernels. • Adds Intel XPU wNa16 compressed tensor support. • Delivers ~3x Whisper throughput improvement (V1 now faster than V0), with encoder batching and CPU backend support. • Adds Eagle/Eagle3 speculative decoding via the Transformers backend. • Adds new models: BAGEL, AudioFlamingo3, JAIS 2, and latent MoE architecture. • Adds tool parsers for DeepSeek-V3.2, Gigachat 3, and Holo2 reasoning. • Adds Qwen3-VL embeddings support and Efficient Video Sampling (EVS). • Adds BitsAndBytes quantization support for Qwen3-Omni-MoE. • Adds support for multiple images/audio per embeddings request. • Adds `tokenization_kwargs` override for embeddings requests. • Enables DeepEP High-Throughput CUDA graph by default for ~5.3% throughput and ~4.4% TTFT improvement on DeepSeek models. • Adds CUDA 13 aarch64 wheels and x86 CPU wheel pipeline. Breaking changes: • `VLLM_ATTENTION_BACKEND` environment variable is replaced by the `--attention-backend` CLI argument; existing env-var-based attention backend configuration will no longer work. • PassConfig flags have been renamed per RFC #27995; existing configs using the old flag names will break. • The `-O.xx` flag has been removed; any scripts or configs using it will fail. • Deprecated plugin and compilation fields have been removed; configs referencing them will break. • Deprecated `task`, `seed`, and multimodal (MM) settings have been removed. • `embed_input_ids` and `embed_multimodal` fallback fields have been removed; requests relying on them will fail. • The tokenizer setter has been removed; code that set the tokenizer via this interface will break. • `--convert reward` CLI flag is replaced by `--convert embed`; existing scripts using `--convert reward` will fail. https://github.com/vllm-project/vllm/releases/tag/v0.12.0 https://github.com/vllm-project/vllm/releases/tag/v0.12.0 Wed, 03 Dec 2025 09:36:17 GMT vLLM v0.12.0 adds GPU Model Runner V2, EAGLE multimodal speculative decoding, RLHF pause/resume, new optimization levels, and major quantization/API expansions. • Adds `-O0`, `-O1`, `-O2`, `-O3` optimization-level flags to trade startup time for runtime performance, with more compilation flags planned. • Adds `--fully-sharded-loras` flag enabling FusedMoE LoRA support. • Moves flat logprob control from an environment variable into `SamplingParams`, making it a first-class sampling parameter. • Adds `verbose_json` and `timestamp` features to the Whisper transcription/translation API. • Adds `parallel_tool_calls` param compliance and parsed tool arguments support to the tool-calling API. • Adds `inputs_embeds_size` config key as a separate dimension from `hidden_size` for flexible embedding configuration. • Supports GGUF model loading from HuggingFace using `repo_id:quant_type` syntax for improved UX. • Adds `parallel_tool_calls` param and tool filtering support in ToolServer. • Adds Responses API multi-turn support for non-harmony requests, plus reasoning item input parsing. • Adds GPU Model Runner V2 (experimental): GPU-persistent block tables, Triton-native sampler with per-request seeds, memory-efficient prompt logprobs, and simplified DP/CUDA graph implementations. • Adds Prefill Context Parallel (PCP) to partition the sequence dimension during prefill, complementing existing Decode Context Parallel (DCP) for improved long-sequence inference. • Adds RLHF pause and resume generation support for asynchronous RL training workflows. • Adds cross-layer KV blocks support and KV cache residency metrics. • Adds audio embeddings support in chat completions. • Adds EAGLE speculative decoding with multi-step CUDA graph support, DP>1, and multimodal support including Qwen3VL. • Adds logprobs support with speculative decoding and async scheduling. • Adds iteration-level profiling for Torch and CUDA profiler. • Adds colorized log output. • Adds new model families: PLaMo-3, OpenCUA-7B, HunyuanOCR, Mistral Large 3, and Ministral 3. • Adds Gemma3 GGUF multimodal format support. • Adds Qwen3 Omni audio-in-video multimodal support. • Delivers 18.1% throughput improvement and 10.7% TTFT improvement on DeepSeek-V3.1 via batch invariant BMM optimization. • Delivers 2.2% throughput improvement and 3.6% TTFT improvement at batch size 32 via shared experts overlap with FlashInfer DeepGEMM. • Adds W4A8 Marlin kernel quantization support. • Adds NVFP4 MoE CUTLASS support for SM120 and TRTLLM MoE NVFP4 kernel. • Adds AWQ compressed-tensors support for Turing GPUs. • Adds FusedMoE LoRA Triton kernel for MXFP4 quantization. • Adds AMD ROCm support for DeepSeek v3.2 with SparseMLA, FP8 MLA decode, AITER attention backend, bitsandbytes quantization on warp-size-32 GPUs, and sliding window for AiterFlashAttentionBackend. • Adds paged attention GEMM acceleration on ARM CPUs with NEON. • Adds FlashAttention ViT support, now the default attention backend. • Adds enhanced NUMA functionality for systems with multiple NUMA nodes per socket. • Reduces Docker image size by approximately 200MB. Breaking changes: • PyTorch upgraded to 2.9.0 with CUDA 12.9 — environments must be updated to CUDA 12.9. • The `num_lookahead_slots` parameter has been removed. • The `best_of` parameter has been removed. • LoRA extra vocab has been removed. • The `xformers` backend is deprecated. • Mistral format is now auto-detected during model loading, which may change behavior for existing Mistral model configs. • `seed=None` is deprecated. https://github.com/vllm-project/vllm/releases/tag/v0.11.1 https://github.com/vllm-project/vllm/releases/tag/v0.11.1 Tue, 18 Nov 2025 23:03:42 GMT vLLM v0.11.1 adds Anthropic `/v1/messages` API, DeepSeek-V3.2 model, GCS streamer support, and async scheduling improvements across 1456 commits. • Adds `VLLM_DEBUG_DUMP_PATH` environment variable for dumping torch.compile debug artifacts. • Updates default CUDA build to `torch==2.9.0+cu129` (PyTorch 2.9.0 + CUDA 12.9.1), enabling Inductor partitioning. • Adds `--async-scheduling` flag with robustness fixes covering chunked prefill, structured outputs, priority scheduling, MTP, and DeepEP/DCP; expected to become the default in the next release. • Adds support for the DeepSeek-V3.2 model. • Adds Run:ai model streamer GCS package support. • Adds media domain restriction filtering via `add option to restrict media domains`, preventing bypass via HTTP redirects. • Adds RL online quantization support with torchao. • Adds cuda2cpu support in `NixlConnector` for P/D disaggregated serving. • Adds KV load failure recovery support in V1 P/D disaggregated mode. • Adds MLA cache support with different latent dimensions in NIXL. • Adds explicit pooling classes for the Transformers backend. • Adds Eagle/Eagle3 multimodal support for speculative decoding, with enablement on Qwen2.5-VL. • Adds text-only mode for Qwen3-VL multimodal model. • Adds LoRA support for OPT models. • Adds ROCm support for AMD Ryzen AI MAX / AI 300 Series GPUs. • Adds Blackwell GPU family support (SM100+) including INT8 quantization fixes. • Adds batch-invariant `torch.compile` support across attention and MoE backends, with explicit support for DeepGEMM and FlashInfer on Hopper and Blackwell GPUs. • Updates to Transformers `v4.56.2`. • Adds chunk-aligned Mamba2 kernel. • Adds GC debug callback for core diagnostics. • Adds more tuned Triton configs for ExpertsInt8 and FP8 MoE kernels. • Serializes `cudagraph_mode` as its enum name instead of numeric value in torch.compile. • Adds chat template kwargs filtering for API security. Breaking changes: • The `--endpoint-type` benchmark flag, deprecated in v0.11.0, is now removed. • `VllmConfig` is moved from `config/__init__.py` to `config/vllm.py`; imports that referenced the old path will break. • `vllm.worker` module is removed; any code importing from `vllm.worker` directly will break. https://github.com/vllm-project/vllm/releases/tag/v0.11.0 https://github.com/vllm-project/vllm/releases/tag/v0.11.0 Thu, 02 Oct 2025 19:17:04 GMT vLLM v0.11.0 ships V1-only engine, CPU KV offloading, new model architectures, FP8/FP4 quant expansions, and major API/CLI additions. • Adds `--enable-logging` CLI flag for controlling logging output. • Adds `logprobs=-1` support to return logprobs for the full vocabulary via the OpenAI-compatible API. • Adds `FULL_AND_PIECEWISE` as the new default CUDA graph mode (previously `PIECEWISE`), improving out-of-the-box performance especially for fine-grained MoEs. • Adds speculative model engine args to the `Config` system for configuring speculative decoding at the engine level. • Adds NVTX profiling support via config, enabling GPU timeline tracing for performance analysis. • Adds `LLM.apply_model` method for direct model access in the V1 engine. • Adds KV cache metrics reporting in GiB units via updated metrics fields. • Adds V1 TPOT (Time Per Output Token) histogram metric. • Adds KV transfer metrics for disaggregated serving. • Adds prompt logprobs for all tokens in OpenAI-compatible completions API. • Adds reasoning streaming events to the OpenAI-compatible API. • Adds MCP tools support to the Responses API. • Adds XML tool-call parser for Qwen3-Coder and Hermes-style tool-call token support. • Adds health endpoint 503 response when the engine is dead. • Adds image path format support for multimodal inputs via the API. • Adds media UUID caching: clients can skip re-uploading media data when UUIDs are provided. • Adds `torchrun` launcher support for data-parallel large-scale serving. • Adds CPU KV cache offloading with LRU management. • Adds prompt embeddings support in V1 engine. • Adds sharded state loading in V1 engine. • Adds FlexAttention sliding window attention in V1. • Adds shared-memory-based multimodal data caching and IPC between processes. • Adds BERT token classification / NER task support. • Adds multimodal model support for pooling tasks. • Enables DeepGEMM by default, delivering ~5.5% throughput improvement. • Enables NCCL symmetric memory by default for tensor parallelism, with 3-4% throughput improvement. • Adds NVFP4 quantization support for dense models including Gemma3 and Llama 3.1 405B. • Adds FP8 per-token-group quantization and hardware-accelerated FP8 instructions. • Adds FP8 FlashInfer MLA decode support for NVIDIA GPUs. • Adds BF16 fused MoE for Hopper/Blackwell expert parallel. • Adds FlashAttention 3 support for Vision Transformer (ViT) inference. • Adds Dual-Batch Overlap (DBO) for overlapping prefill and decode computation. • Adds EAGLE3 speculative decoding support for MiniCPM3 and GPT-OSS. • Adds SeedOSS reasoning parser. • Adds EVS video token pruning for video multimodal models. • Adds data-parallel support for vision encoders in InternVL, Qwen2-VL, and Qwen3-VL. • Adds RADIO encoder support. • Adds Transformers backend support for encoder-only models. • Adds new model architectures: DeepSeek-V3.2-Exp, Qwen3-VL, Qwen3-Next, OLMo3, LongCat-Flash, Dots OCR, Ling2.0, CWM. • Adds RISC-V 64-bit and ARM non-x86 CPU backend support. • Adds ROCm 7.0 support. • Adds ARM 4-bit fused MoE kernel. • Adds Whisper model support on Intel XPU. • Adds Hybrid SSM/Attention support in Triton. • Adds torch.compile CUDA graph Inductor partition integration. • Adds EPLB (Expert-Parallel Load Balancing) support for Hunyuan V1, Mixtral, and static placement. • Adds Mamba2 support with tensor parallelism and quantization. • Adds MRoPE + YaRN support for long-context models. • Adds `--help` improvements to the CLI for better discoverability. • Adds env validation for configuration, catching misconfigured environment variables at startup. • Adds guided decoding backward compatibility fixes. • Adds NIXL MLA latent dimension support for disaggregated serving. • Adds Ray placement group support for data-parallel deployments. • Adds Triton DP/EP kernels for large-scale data/expert parallelism. • Adds FlashInfer speculative decoding backend with 1.14x speedup. • Adds CUDA graph Inductor partition integration for `torch.compile`. • Adds optimized LoRA weight loading. • Adds blocked FP8 for MoE via compressed tensors. • Adds W4A8 faster preprocessing. • Adds FP8 `torch.compile` KV cache support. • Adds EPLB reduced overhead and shared expert overlap optimization for MoE. • V0 engine (`AsyncLLMEngine`, `LLMEngine`, `MQLLMEngine`) and all V0 attention backends have been fully removed; V1 is now the sole engine. Breaking changes: • `AsyncLLMEngine`, `LLMEngine`, and `MQLLMEngine` have been removed entirely — any code importing or instantiating these classes will break. • All V0 attention backends have been removed; deployments relying on V0 backend selection will break. • The default CUDA graph mode is now `FULL_AND_PIECEWISE` instead of `PIECEWISE`; models that only support `PIECEWISE` mode may need explicit configuration. • C++17 is now enforced globally as a build requirement; builds using older C++ standards will fail. • TPU: `xm.mark_step` is deprecated in favor of `torch_xla.sync`; code calling `xm.mark_step` will produce deprecation errors. • `max_seq_len_to_capture` interface has been removed. • V0 components removed: encoder-decoder support, V0 output processor, V0 sampling metadata, V0 Sequence/Sampler, V0 async output processor, `MultiModalPlaceholderMap`, V0 seq group methods, placeholder attention, V0 input embeddings, V0 multimodal registry, V0 attention classes, V0 hybrid model support, V0 backend suffixes, V0 compilation fallbacks, and V0 default args. • `--async-scheduling` produces incorrect (gibberish) output in v0.11.0 under preemption and other scenarios; use v0.10.1 if this flag is required. https://github.com/vllm-project/vllm/releases/tag/v0.10.2 https://github.com/vllm-project/vllm/releases/tag/v0.10.2 Sat, 13 Sep 2025 06:37:01 GMT vLLM v0.10.2 adds aarch64/GB200 support, 20+ new model families, PyTorch 2.8, and major API/quantization expansions with several breaking changes. • Adds `--model-impl terratorch` flag to enable the Terratorch backend for non-language model tasks like semantic segmentation and geospatial applications. • Adds `--safetensors-load-strategy` flag for accelerating model weight loading from NFS-based file systems. • Adds `--log-error-stack` flag to surface full error stack traces in the frontend. • Adds `--mm-encoder-tp-mode` flag (now formalized) to control multimodal encoder tensor-parallelism mode. • Adds `VLLM_DISABLE_PAD_FOR_CUDAGRAPH` environment variable to control CUDA graph padding behavior. • Adds `FlashInfer cubin directory` environment variable for offline environments and pre-cached CUDA binaries. • Adds `return_token_ids` parameter to the OpenAI-compatible API. • Adds `EPLB` configuration parameter for expert-parallel load balancing. • Adds native aarch64 support, enabling vLLM on GB200 platforms; multiplatform `vllm/vllm-openai` Docker image and wheels available via `--extra-index-url https://wheels.vllm.ai/0.10.2/ --torch-backend=auto`. • Extends V1 engine support to GPUs with compute capability below 8.0. • Adds cross-attention KV cache support for encoder-decoder models in the V1 engine. • Adds request-level logits processor integration in V1. • Adds user-configurable KV cache memory size. • Adds support for all prompt logprobs and final logprobs in sampling. • Enables full CUDA graph support by default for hybrid and Mamba models. • Adds V1 engine as the default for Mamba models. • Adds multimodal cache tracking via `mm_hash` and UUID-based multimodal identifiers. • Adds Apple Silicon bfloat16 support for M2+ chips. • Adds IBM Z V1 engine support (s390x). • Adds Intel XPU `torch.compile`, MoE data parallelism, Triton attention, and FP8 quantization support. • Adds ROCm pipeline parallelism with Ray and TorchAO quantization enablement. • Adds FP8 MLA support with CUTLASS backend for NVIDIA Blackwell/SM100 GPUs. • Adds MXFP4 fused CUTLASS MoE support, enabled by default on Blackwell. • Adds W4A8 support on Hopper GPUs. • Adds per-layer quantization routing. • Adds GGUF quantization with layer-skipping capability. • Adds NFP4+FP8 MoE support. • Adds AMD CDNA2/CDNA3 FP4 quantization support. • Adds compressed tensors transforms for linear operations, enabling SpinQuantR1R2R4 and QuIP quantization methods. • Adds LoRA support for Voxtral, Qwen-2.5-Omni, and DeepSeek V2/V3/R1-0528 models, with significantly faster LoRA startup. • Adds multi-label classification support and sigmoid normalization for pooling/classification models. • Adds Support for Decode Context Parallel (DCP) for MLA in distributed serving. • Adds new model families: Apertus, LFM2, MiDashengLM, Motif-1-Tiny, Seed-Oss, Google EmbeddingGemma-300m, GTE sequence classification, Donut OCR, KeyeVL-1.5-8B, R-4B vision, Ernie4.5 VL, MiniCPM-V 4.5, Ovis2.5, Qwen3-Next with hybrid attention, InternVL3.5 with video, Qwen2Audio embeddings, NemotronH Nano VLM, BLOOM V1, and Whisper encoder-decoder for V1. • Adds pipeline parallelism support for Hunyuan, Ovis2.5, GPT-OSS, and Kimi-VL-A3B-Thinking-2506. • Enables data parallelism for ViT across Qwen2.5VL, MiniCPM-V, Kimi-VL, and GLM-4.5V. • Adds Gemma3n audio transcription/translation endpoints to the OpenAI-compatible API. • Adds transcription response usage statistics to the API. • Adds streaming support for non-harmony responses in the Response API. • Adds non-streaming logprobs to the Response API. • Adds MCP tool background mode and streaming+background support. • Adds tool output token reporting to the Response API. • Adds collective RPC endpoint. • Adds embedding endpoint support for chat requests. • Upgrades to PyTorch 2.8.0 and FlashInfer v0.3.0. • Delivers 13.7x speedup for token conversion and symmetric memory all-reduce enabled by default. Breaking changes: • PyTorch 2.8.0 upgrade requires updated CUDA versions and environment dependency changes. • FlashMLA is disabled on Blackwell GPUs due to compatibility issues. • V0 Neuron backend removed. • V0 pooling model support removed. • V0 FlashInfer attention backend removed. • Original Marlin quantization format removed. • Quantized Mixtral hack implementation removed. • `prompt_token_ids` fallback removed from `LLM.generate` and `LLM.embed`. • Metrics name `TPOT` renamed to `ITL` — dashboards and alerting rules referencing `TPOT` will break. https://github.com/vllm-project/vllm/releases/tag/v0.10.1 https://github.com/vllm-project/vllm/releases/tag/v0.10.1 Mon, 18 Aug 2025 04:39:12 GMT vLLM v0.10.1 adds GPT-OSS, Command-A-Vision, mBART, Eagle multimodal, new CLI flags, reward API, and broad hardware/quantization expansions. • Replaces `--task` with `--runner` and `--convert` CLI options as part of V0 engine cleanup. • Renames `--expand-tools-even-if-tool-choice-none` to `--exclude-tools-when-tool-choice-none` for clearer semantics. • Deprecates `--disable-log-requests` in favor of `--enable-log-requests`. • Adds `pip install vllm[flashinfer]` optional-dependency install, decoupling FlashInfer from the core install. • Adds dedicated `LLM.reward` interface for reward models. • Adds `PoolingParams` for per-request pooling control. • Adds Unix domain socket support for local OpenAI-compatible API communication. • Adds multiple API keys support via new configuration for enhanced authentication. • Adds custom vLLM tuned configuration paths (`--custom-vllm-config` equivalent, `#22791`). • Adds environment variable control for logging statistics (`#22905`). • Adds `multimodal cache size` configuration option (`#22441`). • Adds V1 API support for the `run-batch` command. • Adds `HermesToolParser` for models without special tokens. • Adds `request_id` support for external load balancers in distributed serving. • Adds model loader plugin system for extensible model loading. • Adds rate limiting with bucket algorithm for the proxy server. • Adds tree attention backend for the V1 engine (experimental). • Adds N-gram speculative decoding with single KMP token proposal algorithm. • Adds explicit EAGLE3 interface for enhanced speculative decoding compatibility. • Adds encoder-only models without KV-cache, enabling BERT-style architectures. • Adds FlexAttention encoder-only support. • Adds multiple attention metadata builders per KV cache specification. • Adds multiple attention groups for KV sharing patterns. • Adds full CUDA graph support with FA2 and FlashInfer compatibility. • Adds CutlassMLA as the default backend for NVIDIA Blackwell (SM100). • Adds Block FP8 quantization and CUTLASS NVFP4 4-bit weights/activations support for NVIDIA RTX 5090/RTX PRO 6000 (SM120). • Adds dynamic 4-bit quantization with Kleidiai kernels for CPU inference. • Adds TensorRT-LLM FP4 quantization optimized for MoE low-latency inference. • Adds MXFP4 and bias support for the Marlin kernel. • Adds compressed-tensors mixed-precision model loading. • Adds calibration-free RTN quantization for MoE models. • Adds Flash Attention backend for Qwen-VL models on AMD ROCm. • Adds AITER HIP block quantization kernels for AMD ROCm. • Adds CPU transfer support in NixlConnector for prefill/decode disaggregation. • Adds Docker-aware precompiled wheel support for containerized deployment. • Adds multi-turn conversation benchmarking tool. • Adds optional memory profiling skip for multimodal models (`#22950`). • Adds enhanced hybrid distributed serving with multiple API servers in load balancing mode. • Adds chunked processing for long inputs in embedding models. • Adds custom process naming for better monitoring. • Adds new model families: GPT-OSS (with tool calling and streaming), Command-A-Vision, mBART, and SmolLM3 via Transformers backend. • Adds official Eagle multimodal support with Llama4 backend, Step3 vision-language models, Gemma3n multimodal, MiniCPM-V 4.0, Emu3 via Transformers backend, and Intern-S1. • Adds Qwen3 dual-chunk attention and EPLB support, plus native Eagle3 target support. • Adds Mamba1 and Jamba model support in V1 engine (without CUDA graphs). • Adds Ultravox support for Llama 4 and Gemma 3 backends. • Adds tensor/pipeline parallelism with Mamba2 kernel for PLaMo2. • Adds expanded tensor parallelism support in the Transformers backend. • Delivers ~6% end-to-end throughput improvement from Cutlass MLA. • Adds Triton-based multi-dimensional RoPE implementation replacing the PyTorch implementation. • Adds async tensor parallelism for scaled matrix multiplication. • Adds multithreaded async multimodal loading. Breaking changes: • The `--task` CLI flag is replaced by `--runner` and `--convert`; existing invocations using `--task` will break. • The `--expand-tools-even-if-tool-choice-none` CLI flag is renamed to `--exclude-tools-when-tool-choice-none`; scripts using the old flag will break. • The `--disable-log-requests` flag is deprecated in favor of `--enable-log-requests`; the old flag may no longer work. • AQLM quantization support is removed; models using AQLM quantization must migrate to an alternative quantization method. • V0 FlashAttention 3 (FA3) support is deprecated; FP8 KV-cache in V0 may have issues as a result. • Previously deprecated API arguments and methods from the V0 engine codebase are removed (`#21907`). • FlashInfer is moved to an optional dependency (`pip install vllm[flashinfer]`); environments that relied on it being installed automatically will no longer have it by default. • Mamba SSM is removed from core requirements; existing setups that depend on it being included automatically will need to install it separately. https://github.com/vllm-project/vllm/releases/tag/v0.10.0 https://github.com/vllm-project/vllm/releases/tag/v0.10.0 Thu, 24 Jul 2025 22:43:46 GMT vLLM v0.10.0 adds async scheduling, Responses API, new model families, MXFP4/FP8 quantization, and broad hardware expansion including Blackwell and ARM. • Adds `--async-scheduling` flag to overlap engine core scheduling with GPU runner for experimental async scheduling. • Adds `--help=page` option to the CLI for enhanced paginated help documentation. • Adds `get_tokenizer_info` endpoint for retrieving tokenizer and chat-template information. • Adds `cache_salt` support for completions and responses endpoints. • Adds `tokenization_kwargs` for controlling embedding truncation on model-loading requests. • Adds `logprobs` mode for selecting which stage of logprobs to return (RLHF support). • Adds new RPC methods for runtime weight reloading and config updates, enabling RLHF workflows. • Introduces reproducible prefix cache hashing using SHA-256 + CBOR. • Changes default model to `Qwen3-0.6B`. • Adds OpenAI Responses API implementation. • Adds image object support in `llm.chat`. • Adds tool calling with `required` choice and `$defs` support for OpenAI compatibility. • Supports Tensorizer S3 integration with arbitrary arguments for model loading. • Supports HuggingFace repo paths and URLs for GGUF model loading. • Supports new model families: Llama 4 with EAGLE, EXAONE 4.0, Microsoft Phi-4-mini-flash-reasoning, Hunyuan V1 Dense + A13B with reasoning/tool parsing, Ling MoE, JinaVL Reranker, Nemotron-Nano-VL-8B-V1, Arcee, and Voxtral. • Adds MXFP4 quantization support for MoE models. • Adds BNB (bitsandbytes) support for Mixtral and additional MoE models. • Adds in-flight quantization for MoE models. • Adds FP8 KV cache quantization on TPU. • Adds CUTLASS block-scaled group GEMM and DeepGEMM integration for NVIDIA Blackwell/SM100. • Adds FlashInfer MoE blockscale FP8 backend and CUDNN prefill API for MLA on Blackwell. • Adds ARM CPU int8 quantization support. • Adds PPC64LE and ARM V1 engine support. • Adds Intel XPU ray distributed execution support. • Adds shared-memory pipeline parallel for CPU. • Adds FlashInfer ARM CUDA support. • Delivers 48% request duration reduction via microbatch tokenization for concurrent requests. • Adds elastic expert parallel for dynamic GPU scaling while preserving state. • Adds startup time reduction via CUDA graph capture speedup using frozen GC. • Adds multi-modal caching for the transformers backend. • Adds hybrid KV cache with local chunked attention on the V1 engine. • Adds MLA FlashInfer ragged prefill on the V1 engine. • Adds Hybrid SSM/Attention model support on V1 engine. • Adds VLM support with the transformers backend. • Adds support for models with multiple tasks, multiple poolers, and dynamic pooling parameter configuration. • Adds attention-free model support. • Updates PyTorch to 2.7.1 for CUDA builds. • Updates FlashInfer to v0.2.8rc1. Breaking changes: • V0 CPU, XPU, TPU, and HPU backends have been removed; workloads relying on those V0 backends will no longer work. • Long context LoRA (V0) has been removed. • Prompt Adapters have been removed. • Phi3-Small and BlockSparse Attention support has been removed. • V0 Spec Decode workers have been removed. • Default model is changed to `Qwen3-0.6B`; scripts that rely on the previous default model will now load `Qwen3-0.6B` unless an explicit model is specified. https://github.com/vllm-project/vllm/releases/tag/v0.9.2 https://github.com/vllm-project/vllm/releases/tag/v0.9.2 Mon, 07 Jul 2025 17:05:24 GMT vLLM v0.9.2 adds audio translation endpoints, Expert-Parallel Load Balancer, priority scheduling in V1, and broad Blackwell/ROCm/TPU kernel upgrades. • Adds `/v1/audio/translations` endpoint and revamps `/v1/audio/transcriptions` for OpenAI-compatible audio support. • Adds `-O`/`--compilation-config` flag with improved parsing, batch-size-sweep benchmarking support, richer `--help` output, and faster startup to the CLI. • Adds Expert-Parallel Load Balancer (EPLB) for large-scale MoE serving. • Adds Priority Scheduling to the V1 engine. • Adds calibration-free RTN INT4/INT8 quantization pipeline for model compression without calibration data. • Adds Compressed-Tensor NVFP4 (including MoE) support with emulation mode; FP4 emulation removed on devices below SM100. • Adds `MOE_DP_CHUNK_SIZE` environment variable to control MoE data-parallel chunk sizing. • Adds no-privileged CPU/Docker/Kubernetes deployment mode for environments without elevated container privileges. • Adds security hardening that forbids runtime `(cloud)pickle` imports. • Adds image-object support in `llm.chat`, tool-choice expansion, and custom-arg passthroughs for multi-modal agents. • Adds token-level progress bar for `LLM.beam_search` and cached template-resolution speed-ups. • Adds NaN export in logits to `scheduler_stats` when output is corrupted. • Adds CUDA-graph live capture progress bar for debugging graph capture. • Adds full CUDA-Graph execution for all FlashAttention v3 (FA3) and FlashMLA paths including prefix-caching. • Adds full-graph capture for TritonAttention on AMD ROCm, along with quick All-Reduce and chunked pre-fill. • Adds Split-KV support to the unified Triton Attention kernel on ROCm, boosting long-context throughput. • Adds Intel GPU (V1) backend with Flash-Attention support. • Adds CUTLASS W8A8/FP8 kernels for NVIDIA Blackwell SM120 devices. • Adds block-scaled-group GEMM, INT8/FP8 vectorization, deep-GEMM kernels, activation-chunking for MoE, and group-size 64 for Machete on Blackwell SM100. • Adds support for new model families: Ernie 4.5 (+MoE), MiniMax-M1, Phi-tiny-MoE-instruct (Slim-MoE), Tencent HunYuan-MoE-V1, Keye-VL-8B-Preview, GLM-4.1 V, Gemma-3 (text-only), Tarsier 2, Qwen 3 Embedding & Reranker, dots1, and GPT-2 for Sequence Classification. • Adds embedding model support and Mamba2 support to the V1 engine. • Adds native xPyD P2P NCCL transport as base case for disaggregated PD serving without external dependencies. • Adds dynamic MoE-layer quantization for Marlin/GPTQ. • Adds FlexAttention support for any head size with FP32 fallback. • Adds TPU support for dynamic-grid KV-cache updates, head-dim less than 128, and tuned paged-attention kernels. • Adds Bits-and-Bytes 0.45+ support with improved double-quant logic and AWQ quality improvements. • Eliminates `api_key` and `x_request_id` headers middleware overhead in the API server. • Deprecates metrics with `gpu_` prefix for non-GPU-specific metrics. • Hermetic builds and wheel slimming (FA2 8.0 + PTX only) reduce supply-chain surface. Breaking changes: • V0 engine code and features will be removed after this release; this is the last version where V0 stays intact — migrate to the V1 engine before upgrading beyond v0.9.2. • FP4 emulation is removed on devices below SM100 (as part of Compressed-Tensor NVFP4 support). • Runtime imports of `(cloud)pickle` are now forbidden by the security hardening; code that relied on dynamic pickle imports will break. • Metrics with the `gpu_` prefix are deprecated for non-GPU-specific metrics — downstream dashboards or alert rules using those metric names will need updating. https://github.com/vllm-project/vllm/releases/tag/v0.9.1 https://github.com/vllm-project/vllm/releases/tag/v0.9.1 Tue, 10 Jun 2025 18:30:42 GMT vLLM v0.9.1 adds CPU V1 backend, FlexAttention, run-batch CLI, rerank in batch endpoint, and broad LoRA/hardware expansions. • Adds `run batch` subcommand to the vLLM CLI for offline batch inference. • Adds `allowed_token_ids` field to `ChatCompletionRequest` for server-side token filtering. • Makes `use_tqdm` in the `LLM` API accept a callable for custom progress bars. • Adds rerank support to the `run_batch` endpoint. • Adds custom logging support for the vLLM server. • Adds `AsyncLLMEngine.generate` support for targeting a specific DP rank. • Adds FlexAttention to vLLM V1 engine. • Adds initial full support for Hybrid Memory Allocator with cross-layer KV sharing. • Adds support for inplace model weights loading for RLHF workflows. • Adds V1 engine support for the CPU backend. • Adds LoRA support to Beam Search. • Adds LoRA support for InternVL multimodal models. • Adds multi-LoRA support for Neuron hardware. • Adds Multi-Modal model support for Neuron hardware. • Adds quantization support on Neuron. • Adds multi-LoRA optimizations for the V1 TPU backend. • Adds initial SPMD model parallelism support for TPU with a single worker. • Adds Cutlass MLA backend for Blackwell (SM100) GPUs. • Enables FlashInfer by default on Blackwell GPUs. • Adds compressed-tensors NVFP4 support. • Adds CUDA kernel for applying repetition penalty (sampler performance). • Adds CUDA graph support for DP Attention + Expert Parallelism. • Adds API-server scaleout with many-to-many server-engine communications for data parallel serving. • Adds support for DP with Ray. • Enables NixlConnector FlashInfer backend. • Adds Heterogeneous Tensor Parallelism support. • Adds support for Magistral, NemotronH, and minicpm eagle models. • Adds DeepSeek-R1-0528 function call chat template. • Enables data parallel for Llama4 vision encoder. • Adds IBM POWER11 support to CPU extension detection. • Adds AITER grouped topk for DeepSeekV2 on ROCm. • Makes torch distributed process group extendable for custom platform plugins. • Adds H20-3e fused MoE kernel tuning configs for DeepSeek-R1/V3 and Qwen3-235B-A22B. • Adds `benchmark_serving` supports for llama.cpp backends. • Adds dataset support in `vllm bench serve`. Breaking changes: • Positional arguments other than `model` are no longer accepted when initializing `LLM`; all other arguments must be passed as keyword arguments. • The `inputs` argument fallback in Engine classes has been removed. • Fallbacks for the Embeddings API have been removed. • The default mean pooling behavior for `Qwen2EmbeddingModel` has been removed; pooling must now be specified explicitly. • Overriding `get_dummy_text` and `get_dummy_mm_data` is now required. • Metrics that were deprecated in v0.8 have been removed. https://github.com/vllm-project/vllm/releases/tag/v0.9.0 https://github.com/vllm-project/vllm/releases/tag/v0.9.0 Thu, 15 May 2025 03:38:25 GMT vLLM v0.9.0 upgrades to PyTorch 2.7/CUDA 12.8, adds EAGLE3, NIXL PD support, `/classify` endpoint, and broad new model coverage. • Adds `VLLM_ATTENTION_BACKEND=FLASHINFER` environment variable to enable optimized FlashInfer attention and MLP kernels on NVIDIA Blackwell GPUs. • Adds `VLLM_ALLOW_INSECURE_SERIALIZATION` environment variable to explicitly permit insecure serialization paths. • Adds `/classify` HTTP endpoint for classification tasks on the frontend. • Adds `chat_template_kwargs` parameter to `LLM.chat` for passing custom keyword arguments to chat templates. • Adds `cached_tokens` field to response usage payloads. • Adds `tool_choice: required` support for the Xgrammar structured-outputs backend. • Changes `top_k` sampling parameter: disabled with value `0` (still accepts `-1` for backwards compatibility). • Sets the default random seed to `0` for the V1 Engine so repeated runs with `temperature > 0` produce identical outputs. • Adds NIXL integration for Prefill-Decode (PD) disaggregated inference, including local attention optimization. • Supports multiple KV connectors simultaneously for disaggregated inference workloads. • Supports EAGLE3 speculative decoding algorithm. • Enables `torch.compile` and CUDA graph capture for EAGLE speculative decoding. • Adds EAGLE shared input embedding support for speculative decoding. • Enables Speculative Decoding combined with Structured Outputs. • Adds Qwen3 reasoning parser for structured outputs. • Adds Structural Tag support with Guidance backend for structured outputs. • Adds thinking compatibility for structured outputs. • Supports full CUDA graph capture in the V1 engine. • Adds Tensorizer support for fast model loading in V1 and with LoRA. • Adds MultiprocExecutor support and `torchrun` support for Pipeline Parallelism. • Supports sequence parallelism combined with pipeline parallelism. • Adds async tensor parallelism via compilation pass. • Adds truncation control for embedding models. • Adds KV event publishing for metrics. • Adds API for accessing in-memory Prometheus metrics. • Supports `nvidia/DeepSeek-R1-FP4` quantization format. • Supports Quark MXFP4 quantization format. • Supports AutoRound quantization. • Supports torchao models with `AOPerModuleConfig`. • Adds CUDA Graph support for V1 GGUF quantization. • Supports cache salting to prevent side-channel attacks. • Adds default local directory LoRA resolver plugin. • Adds new models: MiMo-7B, MiniMax-VL-01, Ovis 1.6, Ovis 2, GraniteMoeHybrid 4.0, FalconH1 (requires dev `transformers`), LlamaGuard4. • Adds embedding models: nomic-embed-text-v2-moe, new class of GTE models. • Adds DeepSeek Function Call support. • Adds Multi-Token Prediction (MTP) in V1 for DeepSeek. • Implements dual-chunk-flash-attn backend for Qwen2.5-1M with sparse attention support. • Adds video input support for InternVL models with Qwen2.5 backbone. • Adds Multi-LoRA support on TPU. • Adds `top-logprobs` support on TPU. • Adds NeuronxDistributedInference support, Speculative Decoding, dynamic on-device sampling, Mistral model, and Multi-LoRA for Neuron. • Enables FP8 KV cache on AMD V1 backend. • Adds MLA support on AMD. • Adds Block-Scaled GEMM on AMD. • Upgrades default wheel from CUDA 12.4 to CUDA 12.8; CUDA 12.6 wheel distributed via GitHub artifact. • Migrates docs from Sphinx to MkDocs. Breaking changes: • PyTorch upgraded to 2.7 — existing environment dependencies must be updated; CUDA 12.4 support is removed. • Default wheel now targets CUDA 12.8 (previously CUDA 12.4); environments pinned to CUDA 12.4 will break on upgrade. • `top_k` is now disabled with `0` instead of `-1`; callers passing `-1` still work for now but the canonical value has changed. • The V1 Engine now defaults to seed `0`, changing output determinism behavior for runs that previously relied on non-deterministic sampling across restarts. https://github.com/vllm-project/vllm/releases/tag/v0.8.5 https://github.com/vllm-project/vllm/releases/tag/v0.8.5 Mon, 28 Apr 2025 21:13:09 GMT vLLM v0.8.5 adds Qwen3/Qwen3MoE day-0 support, `vllm bench` CLI, structural_tag tool-calling, EAGLE-3, and a new `/server_info` endpoint. • Adds `vllm bench latency` and `vllm bench throughput` CLI subcommands for on-demand benchmarking. • Adds `structural_tag` support via xgrammar for structured tool-calling in the V1 engine. • Adds `/server_info` API endpoint to retrieve the running `vllm_config` from the API server. • Adds sampling params to the `v1/audio/transcriptions` endpoint. • Enables dynamic LoRA loading from a remote server at runtime. • Adds KV Connector API V1 for disaggregated serving, plus an LMCache KV connector for V1. • Day-0 model support for Qwen3 and Qwen3MoE, including fp8 weight loading and tuned MoE configs. • Adds EAGLE-3 speculative decoding support. • Adds support for FlashInfer Attention in the V1 engine. • Adds ModernBERT model support. • Adds Granite Speech model support. • Adds PLaMo2 model support. • Adds Kimi-VL model support. • Adds Qwen2.5-Omni (thinker-only) model support. • Adds Snowflake Arctic Embed family model support. • Enables structured decoding on TPU V1. • Enables Top-P and Top-K sampling on TPU V1. • Adds Cutlass MLA support for Blackwell GPUs. • Adds AMD AITER fused MoE V1 support and integrates the AITER Paged Attention kernel and MLA. • Adds prototype sequence parallelism via compilation pass. • Adds BitBLAS (Microsoft Runtime Kernel Lib) support for low-precision computation. • Adds Triton-based `rotary_emb` implementation for improved performance. • Adds URL validation for multimodal content parts. • Adds property-based testing for vLLM endpoints using an OpenAPI 3.1 schema. • Adds a security guide to documentation. Breaking changes: • `--enable-chunked-prefill`, `--multi-step-stream-outputs`, and `--disable-chunked-mm-input` can no longer be explicitly set to `False`; use the `--no-` prefix instead (e.g., `--no-enable-chunked-prefill`). https://github.com/vllm-project/vllm/releases/tag/v0.8.4 https://github.com/vllm-project/vllm/releases/tag/v0.8.4 Mon, 14 Apr 2025 06:14:20 GMT vLLM v0.8.4 adds Qwen3, Llama4, InternVL3 support, TorchAO quantization, 3x DeepSeek MLA speedup, and structured output auto-backend. • Adds `--max-model-len` hint to the error message when KV cache memory is insufficient, showing users how to set the flag to fit their model. • Adds `hf_token` to `EngineArgs`, allowing Hugging Face authentication to be passed directly through the engine configuration. • Adds `disable_chunked_mm_input` argument to the V1 engine to disable partial multimodal input prefill. • Sets the structured output backend to `auto` by default in the V1 engine (previously required explicit selection). • Adds supports_structured_output() method to the `Platform` interface for V1 structured output backend negotiation. • Adds histogram buckets for `request_latency`, `time_to_first_token`, and `time_per_output_token` metrics. • Adds sampling parameters to `benchmark_serving` for more representative load testing. • Enables regex support with xgrammar in the V0 engine. • Supports matryoshka representation and `dimensions` parameter in the embedding API. • Supports TorchAO quantization for compatible models. • Adds modelopt quantization support for Mixtral models. • Enables PTPC FP8 for `CompressedTensorsW8A8Fp8MoEMethod` (triton `fused_moe`). • Supports W8A8 channel-wise weights and per-token activations in the triton `fused_moe_kernel`. • New merge_attn_states CUDA kernel for DeepSeek MLA delivers a 3x speedup. • Adds support for Qwen3 and Qwen3MoE models. • Adds support for SmolVLM, jinaai/jina-embeddings-v3, InternVL3, and GLM-4-0414 models. • Adds Llama4 support including chat templates for pythonic tool calling, tuned FusedMoE kernel config for Llama4 Scout (TP=8 on H100), and attention temperature tuning enabled by default for long context (>32k). • Enables multi-input by default in the V1 engine. • Adds Eagle model loading and KV cache slot support for Eagle speculative decoding heads in V1. • Enables zero-copy tensor/ndarray serialization and transmission in the V1 engine core. • Adds Intel Gaudi (HPU) multi-step scheduling implementation. • Enables `@support_torch_compile` for the XLA (TPU) backend. • Auto-detects bitsandbytes pre-quantized models. • Unifies engine configuration under `LoadConfig` and `ParallelConfig` via engine args. https://github.com/vllm-project/vllm/releases/tag/v0.8.3 https://github.com/vllm-project/vllm/releases/tag/v0.8.3 Sun, 06 Apr 2025 04:11:42 GMT vLLM v0.8.3 adds Day 0 Llama 4 support, single-node data parallel API serving, BitsAndBytes on V1, and SHA256 prefix-cache hashing. • Adds `tags` parameter to the `wake_up` API endpoint. • Adds HTTP service metrics via the API server (#15657). • Adds BitsAndBytes quantization support in the V1 engine. • Adds Enum support for xgrammar-based structured output in V1. • Adds LoRA CPU offload support in the V1 engine. • Adds single-node data parallel serving with API server support in the V1 engine (`AsyncLLM` data parallel). • Adds Collective RPC support in the V1 engine. • Day 0 support for Llama 4 Scout and Maverick models (V1 engine only). • V1 engine now supports native sliding window attention with the hybrid memory allocator. • Adds new model support: Aya Vision, MiniMaxText01, Skywork-R1V, jina-reranker-v2. • Adds Reasoning Parser for Granite Models. • Adds Phi-4-mini function calling support. • Adds EAGLE Proposer for Speculative Decoding in V1, including speculative decoding metrics and n-gram interface update. • Adds CUTLASS grouped GEMM FP8 MoE kernel for expert parallelism. • Adds FP8 channelwise dynamic per-token GroupedGEMM support. • Adds option to use DeepGemm contiguous grouped GEMM kernel for fused MoE operations. • Supports XpYd disaggregated prefill with MooncakeStore. • Adds custom all-reduce support for ROCm (AMD GPUs). • Adds AITER integration for AMD: int8 scaled GEMM kernel and fused MoE kernel. • Adds paged attention for V1 on AMD ROCm. • Adds CPU MLA (Multi-head Latent Attention) kernel support. • TPU: adds sliding window and logit soft capping support in the paged attention kernel. • TPU: adds optimized top-p implementation that avoids scattering. • Adds middleware to log API server responses. • Adds minimum version pin for `huggingface_hub` to enable Xet downloads. • Adds multi-node offline DP+EP example. https://github.com/vllm-project/vllm/releases/tag/v0.8.2 https://github.com/vllm-project/vllm/releases/tag/v0.8.2 Sun, 23 Mar 2025 21:05:21 GMT vLLM v0.8.2 adds FP8 KV cache, guidance structured output backend, spec decode for top-p/k, and `--disable-uvicorn-access-log` to the V1 engine. • Adds `--disable-any-whitespace` option for xgrammar structured output in the V1 engine. • Adds `--disable-uvicorn-access-log` parameter to suppress access log output. • Adds a flag to disable cascade attention in the V1 engine. • Supports FP8 KV cache in the V1 engine (including FA3 support). • Integrates `fastsafetensors` loader for faster model weight loading. • Enables speculative decoding for top-p and top-k sampling in the V1 engine. • Supports tool calling and reasoning parser on the frontend. • Supports reset of prefix cache by specified device. • Adds pipeline parallel support to `TransformersModel`. • Enables CUDA graph support for Llama 3.2 Vision. • Enables Triton (ROCm) attention backend for Nvidia GPUs in the V1 engine. • Adds TPU MHA Pallas backend for the V1 engine. • Adds tensor parallel multiprocessing support for TPU in the V1 engine. • Supports the Tele-FLM model. • Adds a Kubernetes deployment guide with CPU support. Breaking changes: • OpenVINO support has been removed from core vLLM in favor of an external plugin. https://github.com/vllm-project/vllm/releases/tag/v0.8.1 https://github.com/vllm-project/vllm/releases/tag/v0.8.1 Wed, 19 Mar 2025 17:40:43 GMT vLLM v0.8.1 adds Zamba2 model support and LoRA for embedding models alongside V1 engine fixes. • Adds support for Zamba2 models. • Enables LoRA support for embedding models. • Optimizes the V1 Rejection Sampler using Triton Kernels for speculative decoding. • Re-enables Gemma3 model support on the V1 engine. https://github.com/vllm-project/vllm/releases/tag/v0.8.0 https://github.com/vllm-project/vllm/releases/tag/v0.8.0 Tue, 18 Mar 2025 17:52:20 GMT vLLM v0.8.0 enables the V1 engine by default, adds new API endpoints, DeepSeek/MLA improvements, new models, and Blackwell GPU support. • V1 engine is now enabled by default; disable with `VLLM_USE_V1=0` environment variable. • New `/load` API endpoint exposes load statistics from the API server. • New `/is_sleeping` API endpoint for checking server sleep state. • Enables `/score` endpoint for embedding models. • Adds `return_tokens_as_token_ids` as a request parameter to the API server. • Adds `enable_expert_parallel` argument to enable Distributed Expert Parallelism (EP) for DeepSeek models. • Adds `--show-hidden-metrics-for-version` CLI argument to expose hidden metrics. • Adds `vllm bench` CLI subcommand for benchmarking. • Adds pluggable scheduler support in the V1 engine. • Adds `SupportsV0Only` protocol for model definitions to opt out of V1. • Supports Structured Outputs in V1 engine. • Supports LoRA in V1 engine. • Supports ngram speculative decoding in V1 engine. • Supports reasoning output (chain-of-thought) via the API, including streaming and outlines engine integration. • Supports SSL Key Rotation in the HTTP server. • Enables streaming for the Transcription API. • Supports Image Embedding as API input. • Makes the model parameter optional in API requests. • Supports KV cache offloading and disaggregated prefill via LMCache connector, including chunked prefill. • Adds FlashMLA integration for DeepSeek MLA, including V1 support and chunked prefill. • Adds EP/TP MoE + DP Attention for DeepSeek distributed serving. • Adds MTP support for k > n_predict in DeepSeek MTP speculative decoding. • Adds streamK for block-quantized CUTLASS kernels (GEMM performance). • Adds GPTQAllSpark quantization method. • Adds Deepseek GGUF support including a MoE GGUF kernel. • Adds support for NVIDIA Blackwell `nvfp4` and `fp8` CUTLASS GEMM kernels. • Adds ModelOpt FP4 checkpoint support for Blackwell. • Supports new models: Gemma 3, Mistral Small 3.1, Phi-4-multimodal-instruct, Grok1, QwQ-32B (with tool calling), and Zamba2. • Adds LoRA support for TransformersModel and Gemma3ForConditionalGeneration. • Enables prefix caching by default on TPU. • Adds tensor parallel support via Ray on TPU. • Adds `start_profile`/`stop_profile` support in TPU worker. • Adds TPU multimodal model support for ragged attention. • Adds Neuron device communicator for vLLM V1. • Supports FP8 KV cache in the CPU backend. • Adds CPU inference with VXE ISA for s390x architecture. • vLLM now defaults `generation_config` from the model for chat template and sampling parameters such as temperature. • Updates PyTorch to 2.6.0, CUDA default to 12.4, and Ray to 2.43. Breaking changes: • The default value of `seed` is now `None` (previously a fixed integer); explicitly set `seed` for reproducibility. • The `kv_cache` and `attn_metadata` arguments for a model's `forward` method have been removed; access them via `forward_context` on the attention backend instead. • vLLM now defaults `generation_config` from the model for chat template and sampling parameters (e.g. temperature); existing overrides may behave differently. • The metrics `vllm:time_in_queue_requests`, `vllm:model_forward_time_milliseconds`, and `vllm:model_execute_time_milliseconds` are deprecated and subject to removal. https://github.com/vllm-project/vllm/releases/tag/v0.7.3 https://github.com/vllm-project/vllm/releases/tag/v0.7.3 Thu, 20 Feb 2025 17:08:41 GMT vLLM v0.7.3 adds DeepSeek MTP, pipeline parallelism, sleep/wake endpoints, audio transcription, and broad V1 engine enhancements. • Adds `/v1/audio/transcriptions` OpenAI-compatible API endpoint for audio transcription. • Adds sleep and wake-up endpoint with V1 engine support for suspending and resuming inference workers. • Adds `logit_bias` support to the V1 Sampler and `min_p` sampling support in the V1 engine. • Adds GPU prefix cache hit rate % gauge metric and `iteration_tokens_total` histogram plus several request timing histograms to the V1 metrics system. • Supports GPTQModel Dynamic [2, 3, 4, 8]-bit GPTQ quantization. • Supports Unsloth Dynamic 4-bit BnB quantization. • Supports `nvfp4` quantization on NVIDIA hardware. • Adds AMD per-token-activation per-channel-weight FP8 quantization for ROCm. • Supports pipeline parallelism in the V1 engine. • Adds LoRA support to the V1 engine. • Adds logprobs and prompt logprobs support to the V1 engine. • Uses msgpack for core request serialization in the V1 engine. • Adds initial speculative decoding support with n-grams in the V1 engine. • Adds pluggable platform-specific scheduler. • Enables quantization support for the `transformers` backend. • Supports `torch_dtype` configuration in `TransformersModel` for the `transformers` backend. • Supports DeepSeek Multi-Token Prediction (MTP), delivering a 1.69x speedup in low-QPS scenarios. • Uses FlashAttention3 for MLA on Hopper GPUs. • Expands MLA to support most quantization types. • Adds initial ROCm (AMD) support to the V1 engine. • Adds TPU V1 engine support. • Enables long-context and LoRA support on Intel Gaudi. • Supports Mamba2 (Codestral Mamba) and Bamba model architectures. • Supports IBM/NASA Prithvi Geospatial model. • Supports Ultravox model v0.5. • Adds BNB support and LoRA for Qwen2.5-VL. • Reduces TTFT via concurrent partial prefills. • Adds choice-based structured output with xgrammar. • Makes vLLM compatible with veRL for RLHF colocated training. https://github.com/vllm-project/vllm/releases/tag/v0.7.2 https://github.com/vllm-project/vllm/releases/tag/v0.7.2 Thu, 06 Feb 2025 07:30:12 GMT vLLM v0.7.2 adds Qwen2.5-VL, a `--model-impl=transformers` backend, and a 43% DeepSeek throughput boost via KV cache alignment. • Adds `--model-impl=transformers` flag to run arbitrary Hugging Face text models through a `transformers` backend without a native vLLM model implementation. • Adds `VLLM_LOGITS_PROCESSOR_THREADS` environment variable to parallelize structured decoding and reduce latency under high batch sizes. • Adds `request_success_total` Prometheus counter (labelled with finish reason) for V1 engine metrics. • Adds Qwen2.5-VL vision-language model support (requires source install of Hugging Face `transformers`). • Enables MLA (Multi-head Latent Attention) for DeepSeek VL2 models, delivering the same KV cache compression as text-only DeepSeek V3/R1. • Enables DeepSeek models on ROCm/AMD GPUs. • Aligns KV cache entries to 256-byte boundaries for CUDA devices, yielding a 43% throughput improvement for DeepSeek MLA models. • Applies `torch.compile` to `fused_moe`/`grouped_topk` kernels, yielding a 5% throughput improvement for DeepSeek MoE models. • Enables FusedSDPA support for Intel Gaudi (HPU) hardware. • Adds BNB (bitsandbytes) quantization support for Whisper models. • Adds LoRA support for the Ultravox multimodal model. • Adds support for Sparse24Bitmask compressed model format. • Adds support for loading pure-sparsity Compressed Tensors configs. • Adds Pixtral-Large support via the Hugging Face model format using `llava` multimodal_projector_bias config. • Enables V1 engine support for idefics3 vision-language models. • Adds DeepSeek V3 FP8 W8A8 quantization configs for B200 GPUs. • Adds quantization and MoE configs for GH200 machines. https://github.com/vllm-project/vllm/releases/tag/v0.7.1 https://github.com/vllm-project/vllm/releases/tag/v0.7.1 Sat, 01 Feb 2025 18:02:54 GMT vLLM v0.7.1 delivers ~3x DeepSeek throughput via MLA kernels, FP8 block quantization, reasoning content in API, and richer V1 Prometheus metrics. • Adds MLA decode optimization kernels for the DeepSeek model family, delivering ~3x generation throughput, ~10x token memory capacity, and horizontal context scalability via pipeline parallelism. • Integrates block-quantized CUTLASS kernels (`cutlass_scaled_mm` with 2D group/blockwise scaling) for DeepSeekV3 FP8 inference. • Adds FP8 Triton configs for block quantization and a fused MoE Triton kernel for GPTQ/AWQ quantization formats. • Supports reasoning content in the API for DeepSeek R1 models. • Supports overriding generation config via engine arguments. • Enables offline `/score` endpoint for embedding models. • Enables `MLPSpeculator`/Medusa and `prompt_logprobs` with `ChunkedPrefill` for speculative decoding. • Adds V1 engine Prometheus metrics including per-request prompt/generation token histograms, TTFT and TPOT histograms, and GPU cache usage percentage gauge. • Adds MiniCPM-o-2.6 model support (text outputs). • Adds Llama 3.2 support on AMD ROCm. • Adds NKI-based flash-attention kernel with paged KV cache for AWS Neuron. • Adds V1 support for Qwen-VL multimodal model. • Upgrades FlashInfer to 0.2.0. • Adds DeepSeek-V3 MoE tuning support on AMD. Breaking changes: • When MLA is enabled, chunked prefill and prefix caching are automatically disabled. https://github.com/vllm-project/vllm/releases/tag/v0.7.0 https://github.com/vllm-project/vllm/releases/tag/v0.7.0 Mon, 27 Jan 2025 05:50:13 GMT vLLM v0.7.0 ships a rewritten V1 engine, torch.compile by default, new LLM methods, Flash Attention 3, and a Rerank API • Enables the new V1 engine by setting the `VLLM_USE_V1=1` environment variable — a fully rewritten engine focused on performance and architectural simplicity. • Adds `torch.compile` integration, enabled by default in V1 and toggleable via the `-O3` engine parameter in the existing engine. • Adds `LLM.sleep`, `LLM.wake_up`, `LLM.collective_rpc`, and `LLM.reset_prefix_cache` methods to the `LLM` class for post-training framework integration. • Adds a new `collective_rpc` distributed abstraction for coordinating across workers. • Adds Jina- and Cohere-compatible Rerank API to the API server. • Adds Flash Attention 3 kernel support. • Adds native macOS Apple Silicon support. • Adds TPU support for `W8A8` quantization format. • Adds x86 Multi-LoRA and MoE support. • Adds AMD MI300 FP8 format for block quantization, tuned MoE configurations, and a block size heuristic delivering an average 2.8x speedup for int8 models. • Adds support for `torchrun` and SPMD-style offline inference. • Adds Punica prefill kernel fusion. • Adds new generative model support: CogAgent, Deepseek-VL2, fairseq2 Llama, InternLM3, and Whisper. • Adds new pooling model support: Qwen2 PRM and InternLM2 reward models. • Adds merged multi-modal processor for VLMs, automatically supported by the V1 engine for any model implementing `get_*_embeddings` methods. • Adds Rank Stabilized LoRA (RSLoRA) support. • Adds LoRA support for MolmoForCausalLM. • Adds a new benchmark script for CPU offloading. • Adds out-of-tree hardware platform plugin support. • Implements Cascade Attention in the V1 engine. https://github.com/vllm-project/vllm/releases/tag/v0.6.6.post1 https://github.com/vllm-project/vllm/releases/tag/v0.6.6.post1 Fri, 27 Dec 2024 06:24:26 GMT vLLM v0.6.6.post1 adds FlashInfer sampling kernels for Top-P and Top-K in V1 and restores quantized MoE support. • Uses FlashInfer sampling kernel for Top-P and Top-K sampling in the V1 engine, improving sampling performance. • Reduces task switching in the V1 API server and improves abort handling for more efficient request lifecycle management. https://github.com/vllm-project/vllm/releases/tag/v0.6.6 https://github.com/vllm-project/vllm/releases/tag/v0.6.6 Fri, 27 Dec 2024 00:12:05 GMT vLLM v0.6.6 adds DeepSeek V3 support, online pooling API, S3 model streaming, and V1 engine milestones including VLM prefix caching. • Adds `--enable-request-id-headers` flag to opt in to `X-Request-ID` echoing (now off by default for performance). • Adds `--tensor-parallel-size` flag support for serving DeepSeek V3 on 8xH200 or MI300x via `vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 --trust-remote-code --max-model-len 8192`. • Adds Online Pooling API for embedding/pooling workloads via the V1 frontend. • Supports streaming model weights from S3 using RunAI Model Streamer as an optional loader. • Enables loading video inputs from base64 encoding via `decord` in multimodal requests. • Adds Cutlass 2:4 sparsity kernel support combined with FP8/INT8 quantization for accelerated sparse inference. • Adds prefix caching support for vision-language models (VLMs) in the V1 engine. • Adds min-tokens, repetition, presence, and frequency penalties to the V1 sampler. • Adds Tensor Parallel Ray executor support to the V1 engine. • Supports IBM Granite 3.1 and `JambaForSequenceClassification` models. • Adds `QVQ` and `QwQ` to the list of supported models. • Adds PyPI index for every commit and nightly build for easier pre-release installation. • Supports automatic conversion of classification and reward models. • Adds DeepSeek V3 w8a8 FP8 block-wise quantization support. Breaking changes: • `X-Request-ID` header echoing is now opt-in instead of on by default; existing deployments that rely on it must add `--enable-request-id-headers` to re-enable it. https://github.com/vllm-project/vllm/releases/tag/v0.6.5 https://github.com/vllm-project/vllm/releases/tag/v0.6.5 Tue, 17 Dec 2024 23:10:37 GMT vLLM v0.6.5 adds xgrammar guided decoding, new model support (Aria, GLM-4, Cross Encoders), AWQ/GPTQ on XPU, TPU prefix caching, and a `--version` CLI flag. • Adds `--version` flag to the vLLM CLI for quick version inspection. • Switches `xgrammar` as the default guided decoding backend, replacing the previous default. • Adds `input_audio` field support in frontend requests for audio input. • Adds automatic detection of chat content format from AST in the frontend, removing the need to manually specify chat format. • Adds AWQ and GPTQ quantization support for the XPU backend. • Adds GGUF quantization support on AMD ROCm. • Enables prefix caching on TPU. • Enables chunked-prefill and prefix-caching on CPU. • Adds prefix-cache aware scheduling for improved cache utilization. • Adds sliding window support to the scheduler. • Adds Machete W4A8 quantization kernel support. • Adds HQQ quantization support via Marlin kernel expansion. • Adds support for new models: Aria, Cross Encoders, GLM-4, OLMo (November 2024), Telechat2, Cohere R7B, and GritLM embeddings. • Adds LoRA support for Internlm2, glm-4v, and Pixtral-HF. • Expands `torch.compile` integration to cover all attention backends and encoder-based models, including dynamic FP8 fusion. • Adds ARM AARCH64 platform enablement. • Adds GH200 GPU support. • Reports `multi_modal_placeholders` in VLM inference output. • Adds benchmarking support for structured outputs and vision datasets, including partial prefix benchmarking and random input generation for prefix benchmarks. Breaking changes: • The default temperature for `ChatCompletionRequest` is changed from `0.7` to `1.0` to align with OpenAI's default. https://github.com/vllm-project/vllm/releases/tag/v0.6.4 https://github.com/vllm-project/vllm/releases/tag/v0.6.4 Fri, 15 Nov 2024 07:32:45 GMT vLLM v0.6.4 adds Intel Gaudi backend, V1 engine progress, torch.compile support, new LLMs/VLMs, and chat-based embeddings API. • Adds user-configurable `--task` parameter for models that support both generation and embedding tasks. • Adds Intel Gaudi (HPU) inference backend for running vLLM on Gaudi hardware. • Adds Exllama as a backend for compressed-tensors quantization. • Adds Triton implementation `scaled_mm_triton` supporting fp8 and int8 SmoothQuant symmetric quantization. • Adds goodput metric support to the engine. • Adds stateless process group for easier integration with RLHF and disaggregated prefill workflows. • Adds tool calling parser support for Granite 3.0, Jamba, and granite-20b-functioncalling models. • Adds new LLMs and VLMs: Idefics3, H2OVL-Mississippi, Qwen2-Audio, Pixtral (HF Transformers format), FalconMamba, and Florence-2 language backbone. • Adds new encoder-only embedding models: BERT (`BERTModel`), RoBERTa, and XLM-RoBERTa. • Expands embedding task support to Llama, Qwen2, Qwen2-VL, VLM2Vec (Phi-3-Vision), and E5-V (LLaVA-NeXT). • Adds Math-Shepherd (Mistral reward modeling) and Qwen2 classification task support. • Adds Chat-based Embeddings API. • Adds LoRA support for Granite 3.0 MoE, Idefics3, Llama embeddings, Qwen, and Qwen2-VL. • Adds BNB quantization support for Idefics3, Mllama, Qwen2, and MiniCPMV. • Adds embedding model support for the CPU backend. • Enables combining chunked prefill with speculative decoding. • Adds ability to override HF `config.json` values via CLI. • Adds unified multi-modal processor for VLMs. • Adds CUDA Time Layerwise Profiler. • Adds env variable to force FlashInfer backend to enable tensor cores. • Adds LoRA request metrics tracking. • Expands `torch.compile` support with TorchInductor across many models. • Significant progress on V1 engine core refactor. • Benchmark throughput tool now supports image input. • Adds `fused_moe` performance improvements. • Supports sliding window in the Flash Attention backend. Breaking changes: • Python 3.8 is no longer supported. • Block manager v1 is deprecated and block manager v2 is now the default. https://github.com/vllm-project/vllm/releases/tag/v0.6.3.post1 https://github.com/vllm-project/vllm/releases/tag/v0.6.3.post1 Thu, 17 Oct 2024 17:26:12 GMT vLLM v0.6.3.post1 adds multimodal embedding support, new Mistral models, and Exllama compressed-tensors backend • Adds Exllama as a backend for compressed-tensors quantization, expanding quantized inference options. • Adds CUDA time layer-wise profiler for per-layer GPU timing analysis. • Adds compressed-tensors INT8 W8A8 AZP support on CPU hardware. • Supports VLM2Vec, the first multimodal embedding model in vLLM, enabling vision-language embedding use cases. • Supports Ministral 3B and Ministral 8B via interleaved attention. • Supports multiple and interleaved images for Llama 3.2 vision models. • Supports SDPA attention for Molmo vision backbone. • Improves continuous batching for Jamba and Mamba models. • Optimizes speculative decoding ngram lookup performance. • Block manager v2 is now the default; block manager v1 is removed. Breaking changes: • Block manager v1 is removed; block manager v2 is now the default. Any configuration or workflow relying on block manager v1 will break on upgrade. https://github.com/vllm-project/vllm/releases/tag/v0.6.3 https://github.com/vllm-project/vllm/releases/tag/v0.6.3 Mon, 14 Oct 2024 20:20:28 GMT vLLM v0.6.3 adds Granite MoE, Mamba, GLM-4V, Molmo, NVLM-D, Molmo vision models, Llama 3.1/3.2 tool use, and `VLLM_TORCH_COMPILE_LEVEL` env var for full Inductor graph compilation. • Adds `VLLM_TORCH_COMPILE_LEVEL` environment variable to control `torch.compile` compilation levels; setting `VLLM_TORCH_COMPILE_LEVEL=3` enables Inductor full graph compilation without vLLM custom ops. • Adds `VLLM_SKIP_P2P_CHECK` flag to skip peer-to-peer GPU connectivity checks in distributed deployments. • Moves guided decoding params into `SamplingParams`, unifying the interface for structured-output control. • Makes scheduling policy settable via `EngineArgs`, enabling priority-based scheduling to be configured at engine startup. • Adds support for Llama 3.1 and 3.2 tool use via the tool calling interface. • Adds tool calling support for InternLM2.5 models. • Adds new text models: Granite MoE and Mamba. • Adds new vision-language models: GLM-4V, Molmo, and NVLM-D. • Adds reward model support for Qwen2.5-Math-RM-72B. • Adds Gemma2 embedding model support. • Adds input embeddings support for Qwen2-VL and MiniCPMV. • Adds LoRA support for MiniCPMV2.5 and MiniCPMV2.6. • Expands LoRA modules for Mixtral. • Expands bitsandbytes quantization support to Falcon, OPT, Gemma, Gemma2, and Phi models. • Adds AWQ quantization support for the CPU backend. • Adds cross-attention and encoder-decoder model support on the x86 CPU backend. • Adds async output processor for the XPU backend. • Adds on-device sampling support for Neuron. • Enables OpenVINO GPU backend support. • Enables combined multi-step scheduling, chunked prefill, and prefix caching simultaneously. • Enables CUDA Graphs for multi-step scheduling combined with chunked prefill. • Adds priority scheduling support for embedding requests and via the OpenAI-compatible API. • Extends pipeline parallelism support to remaining text and embedding models. • Adds explicit out-of-tree (OOT) interface for vLLM models including OOT embedding models. • Adds support for HuggingFace's `continue_final_message` parameter in the frontend. • Exposes `InternVL2` `max_dynamic_patch` as a `mm_processor_kwarg` for multimodal processing control. Breaking changes: • `BlockSpaceManagerV2` is now the default block manager, replacing `BlockSpaceManagerV1`; deployments relying on V1 behavior will use the new code path automatically on upgrade. https://github.com/vllm-project/vllm/releases/tag/v0.6.2 https://github.com/vllm-project/vllm/releases/tag/v0.6.2 Wed, 25 Sep 2024 21:50:41 GMT vLLM v0.6.2 adds Llama 3.2 vision support, priority scheduling, MQLLMEngine (+30% throughput), and AMD fp8 KV cache. • Serves Llama 3.2 multimodal models via `vllm serve meta-llama/Llama-3.2-11B-Vision-Instruct --enforce-eager --max-num-seqs 16`. • Set `VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1` to suppress the new beam-search deprecation error, or migrate to `vllm.LLM.use_beam_search` for dedicated beam search. • Introduces `MQLLMEngine` for the API server, boosting throughput ~30% in single-step and ~7% in multi-step scenarios. • Adds initial support for priority scheduling in the production engine. • Adds batch inference support for the llm.chat() API. • Exposes a `revision` argument in the OpenAI-compatible server frontend. • Adds LoRA lineage and base model metadata management support. • Adds prompt logprobs support in multi-step scheduling. • Adds output streaming support to multi-step + async scheduling. • Adds FlashInfer backend support for multi-step scheduling. • Adds CUDA graph support during decoding for encoder-decoder models. • Adds `benchmark_serving` support for sampling from Hugging Face datasets and image inputs. • Adds custom paged attention kernel for AMD ROCm. • Adds fp8 KV cache support for AMD ROCm. • Implements multi-step scheduling for TPU, using Ray as the default distributed backend. • Enables MRoPE and Qwen2-VL support on the CPU backend. • Enables 8-bit weights in Fused Marlin MoE kernel. • Adds asymmetric quantization (AQ AZP) kernels for 3-bit and 4-bit precision. • Adds tensor parallelism support with bitsandbytes quantization. • Adds bitsandbytes quantization support for Gemma2. • Adds support for Solar, MiniCPM3, and LLaVA-Onevision models. • Adds multiple-image support for Qwen-VL and pipeline parallelism for Qwen2-VL. • Adds Mistral function calling format support across all models loaded with the 'mistral' format. • Adds `IQ1_M` quantization implementation to the GGUF kernel. • Binds ZMQ socket to `127.0.0.1` only for local-only API server usage, reducing network exposure. Breaking changes: • Using beam search as a sampling parameter now raises an error; set `VLLM_ALLOW_DEPRECATED_BEAM_SEARCH=1` to suppress, or migrate to `vllm.LLM.use_beam_search`. https://github.com/vllm-project/vllm/releases/tag/v0.6.1.post2 https://github.com/vllm-project/vllm/releases/tag/v0.6.1.post2 Fri, 13 Sep 2024 18:35:43 GMT vLLM v0.6.1.post2 adds support for custom torch.compile backends via the plugin system. • Adds plugin support for custom `torch.compile` backends, allowing operators to register their own compilation strategies. https://github.com/vllm-project/vllm/releases/tag/v0.6.1.post1 https://github.com/vllm-project/vllm/releases/tag/v0.6.1.post1 Fri, 13 Sep 2024 04:40:28 GMT vLLM v0.6.1.post1 adds multi-image Qwen-VL support, FlashInfer multi-step backend, delta-only engine output, and bitsandbytes for Gemma2. • Adds engine option to return only deltas or final output (rather than full accumulated text on each step). • Adds FlashInfer backend support for multi-step scheduling. • Adds bitsandbytes quantization support for Gemma2 models. • Supports multiple images for Qwen-VL models. • Removes the `engine_use_ray` engine option. Breaking changes: • The `engine_use_ray` option has been removed; usages must be dropped on upgrade. • Chunked scheduling is disabled by default for multimodal (vision) models; replace `--max_num_batched_tokens 16384` with `--max-model-len 16384` in existing invocations. https://github.com/vllm-project/vllm/releases/tag/v0.6.1 https://github.com/vllm-project/vllm/releases/tag/v0.6.1 Wed, 11 Sep 2024 21:44:44 GMT vLLM v0.6.1 adds Pixtral, Qwen2-VL, and Llava-Next-Video support, LoRA hot-swap via API, and 2x AWQ throughput. • Adds `--logprobs` argument to `benchmark_serving.py` for log-probability output during serving benchmarks. • Enables random prefix caching in `benchmark_serving.py` serving profiling tool. • Supports dynamic LoRA load and unload via the API server without restart. • Adds progress reporting to the batch runner (`run_batch.py`). • Adds support for NVIDIA ModelOpt static scaling checkpoints. • Supports compressed-tensor W8A8 quantization for the Intel CPU backend. • Adds Pixtral (`mistralai/Pixtral-12B-2409`) multimodal model support, including loading from original Mistral format. • Adds Qwen2-VL model support. • Adds Qwen-VL / Qwen-VL-Chat multimodal model support. • Adds Llava-Next-Video model support. • Adds multi-image input support for LLaVA models. • Adds multi-image input support for InternVL2 models, including pipeline parallelism. • Memory optimization for `awq_gemm` and `awq_dequantize` kernels delivers 2x throughput improvement. • Keeps chunked prefill enabled by default with long context when prefix caching is enabled. • Adds Fused MoE Marlin support for GPTQ models. • Docker image updated to Python 3.12 for a performance bump. Breaking changes: • SqueezeLLM quantization support has been removed. https://github.com/vllm-project/vllm/releases/tag/v0.6.0 https://github.com/vllm-project/vllm/releases/tag/v0.6.0 Wed, 04 Sep 2024 23:35:42 GMT vLLM v0.6.0 delivers 2x throughput, multi-step scheduling, Tools API streaming, and new model/hardware support • Enables multi-step scheduling via `--num-scheduler-steps` engine argument (e.g. `--num-scheduler-steps 8`) to unlock up to 2x throughput versus v0.5.3, with log_probs support included. • Adds `json_schema` support from the OpenAI protocol for structured output control. • Adds OpenAI-Compatible Tools API with streaming support for Hermes and Mistral models. • Enables chunked prefill and prefix caching simultaneously, previously mutually exclusive. • Publishes Prometheus metrics in the `run_batch` API. • Adds support for bitsandbytes 8-bit and FP4 quantized models. • Adds a new tokenizer mode for Mistral models using the native `mistral-commons` package. • Adds async output processing that overlaps output data-structure construction with GPU work, delivering a 12% throughput increase. • Uses FlashInfer backend for FP8 KV Cache when available. • Uses FlashInfer backend for rejection sampling in Speculative Decoding. • Adds tensor parallelism (TP) support for Vision Transformers (ViTs). • Adds multi-image input support for LLaVA-Next and Phi-3-vision models. • Adds multimodal support in offline chat and support for multiple multi-modal items in the OpenAI frontend. • Adds Ultravox support for multiple audio chunks. • Supports benchmarking the async engine in `benchmark_throughput.py`. • Adds new LLM models: Exaone, Granite, and Phi-3.5-MoE. • Extends CUDA graph size for H200 GPUs. • Adds Triton implementations of `awq_dequantize` and `awq_gemm` for AMD ROCm AWQ support. • Adds pipeline parallel support for Intel GPUs. • Adds context-length and token-generation bucket support for Neuron. • Adds support for single and multi-host TPUs on GKE, plus async output processing for TPU. • Adds more percentile and latency metrics to benchmark output. • Migrates `mamba_ssm` and `causal_conv1d` kernels into vLLM, removing external kernel dependencies. https://github.com/vllm-project/vllm/releases/tag/v0.5.5 https://github.com/vllm-project/vllm/releases/tag/v0.5.5 Fri, 23 Aug 2024 18:37:46 GMT vLLM v0.5.5 adds multi-step GPU scheduling, audio model support, GGUF loading with tensor parallelism, and new entrypoint APIs. • Adds `--num-scheduler-steps` parameter to `vllm serve` and `AsyncLLMEngine` to schedule multiple GPU steps in advance, delivering ~20% QPS improvement for single-GPU 8B and 30B models. • Adds a `chat` method to the `LLM` class for direct chat-style inference without going through the HTTP server. • Adds `prompt_logprobs` support in the Chat Completion API. • Supports embeddings in the `run_batch` API. • Supports loading GGUF models with tensor parallelism. • Adds support for the first audio model, `UltravoxModel`. • Supports Jamba 1.5 models. • Supports image embeddings as input for vision models. • Supports SigLIP encoder and alternative decoders for LLaVA models. • Supports serving encoder/decoder models with cross-attention architecture. • Adds EAGLE speculative decoding implementation with Top-1 proposer. • Uses target model max length as default for draft model in speculative decoding. • Adds W8A16 load-time quantization for TPU backend. • Adds fp8 Linear Layer support for AMD ROCm. • Expands MoE weight loading and adds Fused Marlin MoE Kernel. • Adds Machete Hopper-optimized mixed precision linear kernel. • Registers custom ops for `torch.compile` kernel support. • Adds span metrics for model_forward, scheduler, and sampler time. • Optimizes prefix caching performance. • Uses flashinfer sampling kernel when available, yielding ~7% decoding throughput speedup. • Reduces Python allocations for ~24% throughput speedup. • Supports multi-host inference on TPU with optimized RoPE. https://github.com/vllm-project/vllm/releases/tag/v0.5.4 https://github.com/vllm-project/vllm/releases/tag/v0.5.4 Mon, 05 Aug 2024 22:38:28 GMT vLLM v0.5.4 adds new models, W4A8 quantization, guided decoding for offline LLM, ZeroMQ-based server split, and TPU/Intel parallelism. • Adds support for guided decoding for offline `LLM` inference. • Separates the OpenAI server's HTTP request handling from the model inference loop using `zeromq`, delivering a 20% speedup in time-to-first-token and 2x improvement in inter-token latency. • Adds W4A8 quantization support. • Adds support for reading bitsandbytes (`bnb`) pre-quantized models. • Tunes FP8 and INT8 kernels for Ada Lovelace and SM75 T4 GPUs. • Adds FlashInfer support in `DraftModelRunner` for speculative decoding. • Adds observability enhancements for speculative decoding. • Adds pipeline parallelism support for DeepSeek v2, Qwen, Qwen2, and Nemotron models. • Adds vision language model support for InternVL2 and BLIP-2. • Adds support for MiniCPM-V vision language model. • Adds H2O Danube3-4b model. • Adds Nemotron model family (Nemotron-3, Nemotron-4, Minitron). • Enables TPU collective communication and tensor parallelism (TP) for the async engine with reduced compile time. • Enables multiprocessing and tensor parallelism on the Intel CPU backend. • Delivers 15% throughput improvement in large-batch scenarios by using Python's native array data structure for padding. • Reduces log-probability computation latency from ~30ms to ~5ms in large-batch scenarios when `logprobs=None`. https://github.com/vllm-project/vllm/releases/tag/v0.5.3 https://github.com/vllm-project/vllm/releases/tag/v0.5.3 Tue, 23 Jul 2024 07:01:03 GMT vLLM v0.5.3 adds Llama 3.1, Mistral-Nemo, and Chameleon support, a new `vllm` CLI, `--cpu-offload-gb`, and AWQ-Marlin performance gains. • Adds `--cpu-offload-gb` flag to control how much GPU memory to extend into RAM via CPU offloading. • Adds new `vllm` CLI with three subcommands: `serve`, `complete`, and `chat`. • Supports chat completions input in the tokenize endpoint. • Adds AWQ support to the Marlin kernel, delivering 1.5–2x performance improvements for existing AWQ models. • Supports Meta Llama 3.1, runnable on a single 8xH100 or 8xA100 node with FP8 quantization, or across multiple nodes with BF16 via pipeline parallelism. • Enables chunked prefill by default for the Meta Llama 3.1 model series to support long context. • Adds a new rope extension method to support long-context workloads. • Supports dynamic loading of LoRA adapters from HuggingFace at runtime. • Adds pipeline parallel support for Mixtral. • Adds support for Mistral-Nemo. • Adds initial support for Chameleon (multimodal model). • Enables pipeline parallelism via the stdlib multiprocessing module as an alternative to Ray. • Adds InfiniBand shared-object support to wheels so multi-node serving with InfiniBand works out of the box. • Adds SPMD worker execution using Ray accelerated DAG. • Enables MoE with the Pallas GMM kernel on TPU. • Wheels now build on Ubuntu 20.04 instead of 22.04, broadening compatibility. Breaking changes: • The `Attention.kv_scale` field is replaced by separate `k_scale` and `v_scale` fields; any code or configuration referencing `kv_scale` directly will break. https://github.com/vllm-project/vllm/releases/tag/v0.5.2 https://github.com/vllm-project/vllm/releases/tag/v0.5.2 Mon, 15 Jul 2024 18:01:34 GMT vLLM v0.5.2 adds a new CLI, PaliGemma/Fuyu-8B support, Medusa speculative decoding, soft-tuned prompts, and cross-Python wheels. • New experimental `vllm` CLI for serving and querying an OpenAI-compatible server without writing Python. • Adds support for PaliGemma and Fuyu-8B multimodal models. • Adds Medusa speculative decoding with a Top-1 proposer, enabling faster inference with draft-model-free speculation. • Adds soft-tuned prompt (prompt adapter) insertion support for customizing model behavior without fine-tuning weights. • Releases a Python-version-agnostic wheel so a single build installs across all supported Python versions. • Enables bonus tokens in speculative decoding for KV-cache-based models, improving output quality under speculation. • Adds ZeroMQ fallback for broadcasting large objects in distributed inference, unblocking large-model multi-node use cases. • Adds FP8 support via `llm-compressor` and CUTLASS kernels for FP8 layers with bias. • Exposes `vllm.__commit__` attribute carrying the git commit hash for build traceability. • Adds deprecation warning when beam search is enabled, signaling planned removal in a future release. https://github.com/vllm-project/vllm/releases/tag/v0.5.1 https://github.com/vllm-project/vllm/releases/tag/v0.5.1 Fri, 05 Jul 2024 19:47:01 GMT vLLM v0.5.1 adds pipeline parallelism, Gemma 2/Jamba/DeepSeek-V2/Phi3-Vision support, OpenVINO backend, and speculative decoding improvements. • Adds `--pipeline-parallel-size` flag to the API server to enable pipeline parallelism across multiple GPUs (early stage). • Adds `MLPSpeculator` and `Typical Acceptance Sampler` to speculative decoding, plus a new Draft Model Runner. • Adds support for sharded tensorized models via the frontend and core engine. • Adds continuous streaming of OpenAI response token stats. • Adds support for FlashInfer as an optional backend for prefill, decode, and CUDA Graph for decode. • Adds `w4a16` and Marlin 24 support to `compressed-tensors` quantization. • Adds a faster vectorized FP8 quantize kernel, with FP8 support extended to Ampere GPUs. • Adds OpenVINO inference backend. • Adds Intel GPU (XPU) initial inference backend. • Adds CPU inference support with AVX2 ISA. • Adds support for Gemma 2 (recommended with FlashInfer backend v0.0.8 for logits soft cap correctness). • Adds support for Jamba, vLLM's first state space model. • Adds support for DeepSeek-V2. • Adds support for Phi3-Vision, dynamic image size, and a VLM input registry; VLM prompts now use `<image>` directly instead of complex formatting. • Enhances TPU backend support. • Enhances distributed communication performance via shared memory. • Enhances block manager latency. Breaking changes: • All VLM-specific arguments are removed from engine APIs; they can no longer be set globally via CLI. Prompts must now pass `<image>` instead of the previous complex prompt formatting. https://github.com/vllm-project/vllm/releases/tag/v0.5.0.post1 https://github.com/vllm-project/vllm/releases/tag/v0.5.0.post1 Fri, 14 Jun 2024 02:43:10 GMT vLLM v0.5.0.post1 adds initial TPU support, sharded tensorized models, w4a16 compressed-tensors, and AVX2 CPU inference. • Adds `w4a16` quantization support for `compressed-tensors` kernel, enabling 4-bit weight inference on supported hardware. • Supports sharded tensorized models via frontend and core changes, allowing pre-sharded model loading across distributed backends. • Supports CPU inference with AVX2 ISA on Intel hardware, broadening the range of deployable host CPUs. • Adds initial TPU integration, enabling vLLM to run inference on Google TPU hardware. • Adds a vectorized FP8 quantize kernel for improved quantization throughput. • Adds 'input speed' metric to the tqdm progress postfix alongside existing output speed, giving real-time visibility into both throughput directions during batch inference. https://github.com/vllm-project/vllm/releases/tag/v0.5.0 https://github.com/vllm-project/vllm/releases/tag/v0.5.0 Tue, 11 Jun 2024 18:16:12 GMT vLLM v0.5.0 adds FP8 inference (1.5x speedup), OpenAI Vision API, Speculative Decoding, bitsandbytes/QLoRA, and multi-LoRA generation. • Adds `stream_options` field to `ChatCompletionRequest` in the OpenAI protocol, enabling streaming options on chat completions. • Adds `add_special_tokens` field to `ChatCompletionRequest` (default `False`) for controlling special-token injection in the OpenAI API server. • Returns `max_model_len` in the `/v1/models` response, giving clients visibility into the model's context length limit. • Enables passing multiple LoRA adapters at once to generate() in a single call. • Supports named functions in the OpenAI `tools` API, allowing structured tool-call requests by function name. • Adds FP8 quantization support (via CUTLASS kernels) delivering ~1.5x inference speedup on supported hardware. • Adds OpenAI Vision API support for LLaVA and LLaVA-NeXT models via the API server. • Adds Speculative Decoding support (ready for testing, planned to be on by default in a future release). • Adds Automatic Prefix Caching support (ready for testing, planned to be on by default in a future release). • Adds bitsandbytes quantization and QLoRA support. • Adds Dynamic Per-Token Activation Quantization kernel. • Adds customizable RoPE theta, enabling control over rotary position embedding frequency. • Defaults to the multiprocessing backend for single-node distributed inference. • Updates Outlines integration from `FSM` to `Guide` for structured output generation. • Adds CUDA Graph output buffer to reduce memory footprint during inference. Breaking changes: • FP8 checkpoint format field renamed from `act_scale` to `input_scale`; existing FP8 checkpoints using the old field name will break. https://github.com/vllm-project/vllm/releases/tag/v0.4.3 https://github.com/vllm-project/vllm/releases/tag/v0.4.3 Sat, 01 Jun 2024 00:25:13 GMT vLLM v0.4.3 adds Falcon, Granite, Arctic, and Phi-3-Small model support, OpenAI Batch API, embedding API, TP>1 speculative decoding, and GPTQ Marlin 2:4 sparse quantization. • Adds `MultiprocessingGPUExecutor` to make Ray an optional dependency for multi-GPU inference. • Supports OpenAI batch file format via the Batch API, enabling offline batch inference workloads. • Adds initial Embedding API support with e5-mistral-7b-instruct as the first supported model. • Adds GPTQ Marlin 2:4 sparse structured quantization support. • Adds initial Activation Quantization support. • Loads FP8 kv-cache scaling factors from checkpoints. • Adds `w8a8` CUTLASS kernels for FP8 quantized inference. • Supports CUTLASS kernels inside CUDA graphs. • Supports CUDA graph acceleration for speculative decoding. • Enables tensor-parallel (TP>1) speculative decoding. • Adds support for Falcon models. • Adds support for IBM Granite Code models. • Adds blocksparse flash attention kernel with support for the Phi-3-Small model. • Adds Snowflake Arctic model implementation. • Supports Dynamic RoPE scaling. • Supports long-context LoRA. • Adds cross-attention KV caching and memory management toward encoder-decoder model support. • Adds Punica kernel support on AMD GPUs. • Adds fused MoE Triton configs for AMD GPUs. • Automatically detects SparseML models for quantization. • Automatically detects vLLM-tensorized models, removing the need for manual configuration. • Adds Pipeline Parallelism support via PyNCCL groups (`PP PyNCCL Groups`). • Accelerates automatic prefix caching hashing by avoiding deep copies. • Improves n-gram speculative decoding efficiency. • Adds `get_name` method to attention backends for programmatic backend identification. • Re-enables custom roles in the Chat Completions API. https://github.com/vllm-project/vllm/releases/tag/v0.4.2 https://github.com/vllm-project/vllm/releases/tag/v0.4.2 Sun, 05 May 2024 04:31:08 GMT vLLM v0.4.2 adds chunked prefill, ngram speculative decoding, FlashInfer backend, and Phi-3 support • Adds `--log-level` option to the API server for runtime log verbosity control. • Adds chunked prefill support (ready for testing) to improve inter-token latency under high load by chunking prompt processing and prioritizing decode. • Adds ngram prompt lookup decoding for speculative decoding via the `[Speculative decoding]` ngram proposer. • Adds logprobs support for speculative decoding. • Adds FlashInfer as a selectable attention backend. • Adds support for Phi-3-mini models. • Adds full tensor parallelism for LoRA layers. • Expands Marlin kernel to support all GPTQ models, including AutoGPTQ and 8-bit GPTQ models. • Supports FP8 checkpoints (both dynamic and static) in the kernel layer. • Supports complex message content (e.g. multi-part messages) for the chat completions endpoint. • Supports dynamic `num_readers` configuration for Tensorizer. • Adds more Prometheus histogram metrics for monitoring. • Enables prefix caching with block manager v2. • Allows users to define a custom whitespace pattern for Outlines-based structured generation. • Centralizes and documents all environment variables for easier configuration discovery. • Upgrades to `torch==2.3.0`. • Upgrades to `tensorizer==2.9.0`. https://github.com/vllm-project/vllm/releases/tag/v0.4.1 https://github.com/vllm-project/vllm/releases/tag/v0.4.1 Wed, 24 Apr 2024 02:28:08 GMT vLLM v0.4.1 adds Meta Llama 3, CommandR+, Mixtral 8x22B, Intel CPU backend, tensorizer loading, LM Format Enforcer guided decoding, and FP8 dynamic scaling. • Adds `tensorizer` as a model-loading backend via new load option, enabling fast deserialization of model weights. • Adds LM Format Enforcer as an option for guided decoding alongside the existing backend. • Makes detokenization and tokenizer initialization optional, allowing lighter-weight engine deployments. • Adds Intel CPU inference backend, enabling vLLM to run inference on CPU hardware without a GPU. • Adds initial support for dynamic per-tensor scaling via FP8, enabling quantized inference with FP8 precision. • Supports private/out-of-tree model registration, letting users register custom model architectures without upstreaming. • Adds support for new models: CommandR+, MiniCPM, Meta Llama 3, and Mixtral 8x22B. • Adds LoRA support on quantized models. • Adds prompt token truncation option to the completions API. • Supports new autogptq `checkpoint_format`, broadening GPTQ model compatibility. • Upgrades to PyTorch 2.2.1 and Triton 2.2.0. • AMD ROCm backend gains Triton kernel for default Flash Attention and e4m3fn FP8 KV cache support. • Enables `hf_transfer` by default when available, accelerating model downloads from Hugging Face Hub. • Progress toward chunked prefill scheduler (end-to-end working path included). • Progress toward speculative decoding, including lookahead scheduling and configuration object. https://github.com/vllm-project/vllm/releases/tag/v0.4.0.post1 https://github.com/vllm-project/vllm/releases/tag/v0.4.0.post1 Tue, 02 Apr 2024 20:01:54 GMT vLLM v0.4.0.post1 adds Intel CPU inference backend, lookahead scheduling for speculative decoding, and new AutoGPTQ checkpoint format support. • Adds Intel CPU inference backend, enabling vLLM inference on CPU without a GPU. • Adds lookahead scheduling for speculative decoding, improving throughput for speculative decode workloads. • Adds support for the new AutoGPTQ `checkpoint_format`, broadening quantized model compatibility. • Restores sm70/sm75 (Volta/Turing GPU) binary support dropped in v0.4.0. https://github.com/vllm-project/vllm/releases/tag/v0.4.0 https://github.com/vllm-project/vllm/releases/tag/v0.4.0 Sat, 30 Mar 2024 01:54:27 GMT vLLM v0.4.0 adds automatic prefix caching, LLaVA vision support, five new models, mTLS, and JSON-mode structured output • Enables automatic prefix caching via `--enable-prefix-caching` flag, caching long system prompts across requests to reduce redundant computation. • Adds `--log-level` flag to control vLLM log verbosity instead of the previously fixed `info` level. • Adds mTLS support via new server arguments (PR #3430). • Adds `min_tokens` parameter for EOS suppression in the OpenAI-compatible server. • Supports `json_object` response format in the OpenAI-compatible server for arbitrary JSON output and Context Free Grammar-constrained generation. • Adds LLaVA as the first supported vision-language model. • Adds new text models: Command R, Qwen2 MoE, DBRX, XVerse, and Jais. • Adds LoRA support for Qwen2 models. • Adds GPTQ quantization support for Gemma models. • Adds asynchronous tokenization to reduce CPU-side latency. • Adds chat templates for Falcon and ChatGLM models. • Adds progress on chunked prefill scheduler and speculative decoding (worker for speculate, score, and rejection sampling). • Adds CMake-based build system for extensibility and custom kernel integration. • Improves AWS Inferentia (Neuron) support, including Mistral model inference via transformers-neuronx. • Re-enables custom all-reduce kernel after robustness improvements. • Adds engine health check connected to the OpenAI-compatible server endpoint. • Replaces cupy dependency with an alternative due to upstream bugs. • Moves model file locks from `/tmp/` to `~/.cache/vllm/locks/` for better isolation. • Adds usage statistics collection. • Adds GeGLU kernel with approximate GELU for improved MoE model performance. • Adds batched RoPE kernel. Breaking changes: • Model file locks are now stored in `~/.cache/vllm/locks/` instead of `/tmp/`; any tooling that referenced the old path will need to be updated. https://github.com/vllm-project/vllm/releases/tag/v0.3.3 https://github.com/vllm-project/vllm/releases/tag/v0.3.3 Fri, 01 Mar 2024 20:58:44 GMT vLLM v0.3.3 adds StarCoder2, Orion model support, 2/3/8-bit GPTQ, Marlin Int4 kernels, and experimental structured output for the OpenAI server. • Adds experimental structured output (JSON, Regex) via guided decoding for the OpenAI-compatible API server. • Adds logit bias support for the OpenAI API. • Adds log probabilities for Chat Completions in the OpenAI-compatible API. • Adds `cache_config` information to Prometheus metrics. • Supports 2/3/8-bit GPTQ quantization models. • Integrates Marlin kernels for Int4 GPTQ inference, accelerating quantized throughput. • Adds LoRA support for Gemma. • Supports the StarCoder2 architecture. • Supports the Orion model architecture. • Adds experimental AWS Inferentia2 support via `transformers-neuronx`. • Enables GQA (Grouped Query Attention) support in prefix prefill kernels. • Adds early stopping to completion APIs. • Optimizes the fused MoE (Mixture-of-Experts) kernel for improved performance. • Optimizes the GeGLU layer in Gemma for faster inference. • Migrates metrics from `aioprometheus` to `prometheus_client` (requires `prometheus_client >= 0.18.0`). https://github.com/vllm-project/vllm/releases/tag/v0.3.2 https://github.com/vllm-project/vllm/releases/tag/v0.3.2 Wed, 21 Feb 2024 19:50:54 GMT vLLM v0.3.2 adds OLMo and Gemma model support, per-request seed, multi-LoRA, and ROCm gfx908 coverage. • Adds `seed` parameter support for per-request reproducible sampling control. • Adds `--code-revision` config argument for pinning a specific revision of a model on Hugging Face Hub. • Supports multi-LoRA adapters as extra models in the OpenAI-compatible server. • Adds metrics to `RequestOutput` objects for programmatic inspection of per-request stats. • Makes vLLM logging formatting optional (configurable). • Adds support for OLMo models. • Adds support for Gemma models. • Extends ROCm support to include the `gfx908` GPU architecture. https://github.com/vllm-project/vllm/releases/tag/v0.3.1 https://github.com/vllm-project/vllm/releases/tag/v0.3.1 Fri, 16 Feb 2024 23:06:09 GMT vLLM v0.3.1 adds LoRA for Mixtral, InternLM2 support, request-level Prometheus metrics, and ROCm gfx1100 (Radeon 7900) inference. • Adds request-level metrics via a refactored Prometheus integration. • Adds LoRA support for Mixtral models. • Adds InternLM2 model support. • Adds a fused top-K softmax kernel for MoE models, improving mixture-of-experts inference efficiency. • Adds support for ROCm on Radeon 7900 series (gfx1100) without requiring flash-attention. • Removes EOS tokens from output by default. • Enables setting local logging level via environment variable. Breaking changes: • The Yi model definition is removed; users must switch to `LlamaForCausalLM` instead. • Custom all-reduce is disabled by default. • Ray compiled DAG integration is disabled by default. https://github.com/vllm-project/vllm/releases/tag/v0.3.0 https://github.com/vllm-project/vllm/releases/tag/v0.3.0 Wed, 31 Jan 2024 08:07:57 GMT vLLM v0.3.0 adds multi-LoRA, prefix caching, FP8 KV cache, API token auth, batch completions, and DeepSeek MoE support. • Adds `include_stop_str_in_output` and `length_penalty` parameters to the OpenAI-compatible API. • Adds simple API token authentication to the OpenAI API server. • Supports FP8-E5M2 KV Cache to reduce memory pressure during inference. • Experimental multi-LoRA support enables serving multiple LoRA adapters concurrently from a single engine. • Experimental prefix caching support reuses KV cache across requests that share a common prompt prefix. • Supports batch completion requests in the OpenAI-compatible server. • Adds DeepSeek MoE model support with a fused MoE kernel for optimized throughput. • Fused MoE kernel for Mixtral improves MoE inference performance. • Adds quantized Mixtral support. • AWQ quantization delivers up to 2.66x higher throughput. • Custom all-reduce kernels replace the default NCCL all-reduce for multi-GPU communication. • Adds ROCm 6.0 and MI300 GPU support. • Adds Qwen2 model support. • Adds Stable LM 2 model support. • Adds AWS Neuron build option for running vLLM on Neuron hardware. • Supports the OpenAI API server as a target in `benchmark_serving.py`. • Allows setting the FastAPI `root_path` argument for reverse-proxy deployments. • Adds a Gradio chatbot UI for the OpenAI-compatible web server. https://github.com/vllm-project/vllm/releases/tag/v0.2.7 https://github.com/vllm-project/vllm/releases/tag/v0.2.7 Thu, 04 Jan 2024 01:36:53 GMT vLLM v0.2.7 adds SSL to API servers, CUDA graph support for GPTQ/SqueezeLLM, and up to 70% distributed inference throughput gains via NCCL. • Adds SSL arguments to API servers, enabling TLS-secured inference endpoints. • Enables CUDA graph capture for GPTQ and SqueezeLLM quantized models, accelerating inference for those quantization formats. • Switches distributed control-plane communication from Ray to NCCL, removing serialization/deserialization overhead for up to 70% throughput improvement in distributed inference. • Adds support for DeciLM-7B and DeciLM-7B-instruct model architectures. • Adds support for GPT-NeoX models without attention biases. https://github.com/vllm-project/vllm/releases/tag/v0.2.6 https://github.com/vllm-project/vllm/releases/tag/v0.2.6 Sun, 17 Dec 2023 18:35:42 GMT vLLM v0.2.6 adds CUDA/HIP graph execution for faster inference and W4A16 GPTQ quantization support. • Adds `quantization` argument to enable W4A16 GPTQ support for running 4-bit quantized models, reducing memory footprint during inference. • Adds a flag to include the stop string in output text, giving callers visibility into which stop sequence terminated generation. • Accelerates model execution via CUDA/HIP graph capture, reducing per-request kernel-launch overhead for eligible models. • Adds Phi-2 to the list of supported models. https://github.com/vllm-project/vllm/releases/tag/v0.2.5 https://github.com/vllm-project/vllm/releases/tag/v0.2.5 Thu, 14 Dec 2023 07:58:13 GMT vLLM v0.2.5 adds Mixtral expert parallelism and grouped-query attention for MPT models. • Adds expert parallelism support for Mixtral models, improving multi-GPU inference throughput. • Adds grouped-query attention (GQA) support for MPT model attention layers. • Removes the Python 3.10 minimum version requirement, broadening supported runtime environments. https://github.com/vllm-project/vllm/releases/tag/v0.2.4 https://github.com/vllm-project/vllm/releases/tag/v0.2.4 Mon, 11 Dec 2023 19:50:34 GMT vLLM v0.2.4 adds Mixtral 8x7B and AMD GPU support, plus ChatGLM and custom server params. • Adds custom server parameters support via the OpenAI-compatible server. • Supports Mixtral 8x7B (Mixture-of-Experts) model, requiring the `megablocks` dependency. • Adds AMD GPU (ROCm) support in collaboration with EmbeddedLLM. • Adds support for `ChatGLMForConditionalGeneration` models. • Adds CUDA architecture list as a Docker build option. • Saves PyTorch profiler output for latency benchmarking. https://github.com/vllm-project/vllm/releases/tag/v0.2.3 https://github.com/vllm-project/vllm/releases/tag/v0.2.3 Sun, 03 Dec 2023 20:30:03 GMT vLLM v0.2.3 adds Prometheus metrics, echo support for the OpenAI chat API, and better Ray Serve integration. • Adds production metrics exported in Prometheus format for operational observability. • Adds `echo` parameter support to the OpenAI-compatible chat API, mirroring the completions API behavior. • Supports chat templates in the OpenAI-compatible chat API. • Adds `--profile` option to the latency benchmark tool. • Supports `max-model-len` argument in the throughput benchmark tool. • Better integration with Ray Serve for production deployments. • Adds a PyTorch-native implementation of custom layers as an alternative to compiled extensions. • Initializes models directly on GPU to reduce CPU memory footprint when loading large models. https://github.com/vllm-project/vllm/releases/tag/v0.2.2 https://github.com/vllm-project/vllm/releases/tag/v0.2.2 Sun, 19 Nov 2023 05:58:18 GMT vLLM v0.2.2 adds Yi/ChatGLM2/Phi models, AWQ for all models, LogitsProcessor API, Min-P sampler, YaRN, and a health endpoint. • Adds `LogitsProcessor` API to `SamplingParams`, enabling custom logits manipulation at inference time. • Adds AWQ quantization support for all models (previously limited to select models); quantization config is now auto-read from the HuggingFace `quantization_config` field. • Adds a `/health` HTTP endpoint to the OpenAI-compatible server for liveness checking. • Returns token usage fields in OpenAI-compatible API responses. • Adds support for the Min-P sampler in sampling parameters. • Adds `repetition_penalty` to sampling parameters. • Adds YaRN (Yet another RoPE extensioN) support for extended context length. • Adds preliminary support for SqueezeLLM quantization. • Adds new model support: Yi, ChatGLM2, and Microsoft Phi-1.5. • Upgrades base environment to PyTorch v2.1 + CUDA 12.1 (CUDA 11.8 wheels also provided). • Supports downloading models from modelscope.cn in addition to HuggingFace Hub. • Adds DeepSpeed-MII backend option to the benchmark script. • Adds official Dockerfile with CUDA 12.1. Breaking changes: • Scheduler input tensor shape changed from 1D flattened to 2D; custom integrations that depend on the internal tensor layout will break. https://github.com/vllm-project/vllm/releases/tag/v0.2.1 https://github.com/vllm-project/vllm/releases/tag/v0.2.1 Mon, 16 Oct 2023 20:01:21 GMT vLLM v0.2.1 adds PagedAttention V2 (up to 20% latency cut), AWQ support for Mistral 7B, and prompt log probabilities. • Adds PagedAttention V2 kernel delivering up to 20% end-to-end latency reduction. • Adds AWQ (Activation-aware Weight Quantization) support for Mistral 7B models, with a workaround for Turing GPUs. • Adds prompt log probabilities — log probs are now returned for prompt tokens, not just generated tokens. • Adds support for sharding LLaMA2-70B across more than 8 GPUs. • Adds IPv4/IPv6 dual-stack support to the API server. • Adds tokenizer revision support. • Improves detokenization performance. https://github.com/vllm-project/vllm/releases/tag/v0.2.0 https://github.com/vllm-project/vllm/releases/tag/v0.2.0 Thu, 28 Sep 2023 22:31:44 GMT vLLM v0.2.0 adds AWQ quantization, Mistral-7B and LongChat support, RoPE scaling, and up to 60% faster inference. • Makes `max_model_len` configurable, allowing users to override the default maximum model context length. • Adds `gpu_memory_utilization` and `swap_space` parameters to the `LLM` class for fine-grained memory control. • Adds `stop_token_ids` sampling parameter to halt generation at specific token IDs. • Adds `float16` and `float32` as explicit `--dtype` choices for model loading. • Enables `TORCH_CUDA_ARCH_LIST` environment variable to select target GPU architectures at build time. • Adds model revision support, allowing a specific model commit or revision to be pinned. • Supports AWQ (Activation-aware Weight Quantization) for LLaMA models, including safetensors format for quantized models. • Adds support for Mistral-7B-v0.1. • Adds support for LongChat and RoPE scaling (reads `rope_theta` and `max_position_embeddings` from model config). • Automatically sets `max_num_batched_tokens` based on available resources, removing the need for manual tuning. • Delivers up to 60% performance improvement through optimized de-tokenization (TGI-style incremental detokenization) and vectorized sampler. https://github.com/vllm-project/vllm/releases/tag/v0.1.6 https://github.com/vllm-project/vllm/releases/tag/v0.1.6 Fri, 08 Sep 2023 07:08:25 GMT vLLM v0.1.6 reverts a breaking API change and delivers faster startup plus background task generation. • Reduces vLLM startup time for faster engine initialization. • Starts background tasks within `AsyncLLMEngine.generate` so generation can proceed asynchronously without requiring a separate task launch. https://github.com/vllm-project/vllm/releases/tag/v0.1.5 https://github.com/vllm-project/vllm/releases/tag/v0.1.5 Thu, 07 Sep 2023 23:16:58 GMT vLLM v0.1.5 adds CodeLlama support, safetensors loading for all models, and OpenAPI request body specs for OpenAI endpoints. • Enables safetensors loading for all models, expanding the supported weight formats beyond the previous subset. • Adds support for CodeLlama, expanding the roster of served models. • Enables request body OpenAPI spec for OpenAI-compatible endpoints, making the API surface formally documented and machine-readable. • Stabilizes `AsyncLLMEngine` with a background engine loop, making async serving more robust for production use. • Aligns beam search implementation with HuggingFace `hf_model.generate`, improving output consistency for beam-search workloads. • Accelerates LLaMA model loading, reducing startup time when serving LLaMA-family models. https://github.com/vllm-project/vllm/releases/tag/v0.1.4 https://github.com/vllm-project/vllm/releases/tag/v0.1.4 Fri, 25 Aug 2023 03:31:13 GMT vLLM v0.1.4 adds InternLM, Qwen, and Aquila models, token-array inputs to the OpenAI API, and pre-built CUDA binaries. • Supports tokens and arrays of tokens as inputs to the OpenAI completion API, expanding what callers can pass as prompts. • Adds support for three new model families: InternLM, Qwen (QWen-7B), and Aquila. • Publishes pre-built CUDA binaries so users no longer need to compile vLLM's CUDA kernels locally on install. • Adds max prompt length validation for the OpenAI completions API. • Implements approximate GELU kernels and optimizes the `single_query_cached_kv_attention` kernel for faster paged-attention throughput. • Adds compute capability 8.9 to default CUDA build targets, covering Ada Lovelace GPUs. https://github.com/vllm-project/vllm/releases/tag/v0.1.3 https://github.com/vllm-project/vllm/releases/tag/v0.1.3 Wed, 02 Aug 2023 23:56:22 GMT vLLM v0.1.3 adds LLaMA 2, Falcon, GPT-J, and Baichuan model support plus TGI-style continuous batching and MQA/GQA efficiency. • Adds `--trust-remote-code` flag to the API server and benchmark tooling to handle remote tokenizers. • Adds Ray placement group support for more flexible distributed cluster scheduling. • Adds support for LLaMA 2 models. • Adds support for Falcon models. • Adds support for GPT-J models. • Adds support for Baichuan-7B and Baichuan-13B models. • Switches the scheduling algorithm to TGI-style continuous batching. • Adds efficient support for Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), including an optimized MQA kernel. https://github.com/vllm-project/vllm/releases/tag/v0.1.2 https://github.com/vllm-project/vllm/releases/tag/v0.1.2 Wed, 05 Jul 2023 04:51:21 GMT vLLM v0.1.2 adds GPTBigCode, MPT, and BLOOM model support plus a ChatCompletion endpoint in the OpenAI demo server. • Adds `ChatCompletion` endpoint to the OpenAI-compatible demo server, enabling chat-style API interactions. • Adds support for GPTBigCode models. • Adds support for MPT and BLOOM models. • Supports custom tokenizers.