SGLang

Summary

SGLang is an open-source local-llm-runtime that serves models and handles LLM inference, available under the MIT license. It runs as a Python library and is intended for practitioners building and deploying LLM applications. Its documentation positions it alongside general inference engines. The project has several recent updates, demonstrating active development.

SGLang is a high-performance serving framework for large language models and multimodal models.

Command line

Values are placeholders taken from each option’s declared default. Nothing is executed here — the output shown is a recording of a run that already happened.

Release history

1. docs update Aug 29, 2026 · issue 010

   SGLang adds --speculative-dsa-topk-backend to independently control the DSA top-k backend for speculative draft workers.

   └──▷ TRY IT
   Override the draft worker top-k backend to flashinfer while keeping the target model on sgl-kernel, for DSA-based MTP speculative decoding tuning.
   copy

   $ python -m sglang.launch_server --speculative-dsa-topk-backend flashinfer --dsa-topk-backend sgl-kernel

   • ›Adds --speculative-dsa-topk-backend flag to select the DSA indexer top-k backend (sgl-kernel, torch, or flashinfer) for speculative draft workers independently of --dsa-topk-backend (which controls the target model); note that torch requires SGLANG_DSA_FUSE_TOPK=false and the default is sgl-kernel.
2. docs update Aug 29, 2026 · issue 010

   SGLang adds NVIDIA Dynamo KV event publishing and runtime-load PUB socket flags for load-aware routing

   └──▷ TRY IT
   Expose a runtime-load PUB socket on an explicit TCP address so a load-aware router can subscribe to engine load updates.
   copy

   $ python -m sglang.launch_server --load-publish-endpoint tcp://*:6000

   • ›Adds --load-publish-endpoint flag to opt in to a runtime-load PUB socket that load-aware routers can subscribe to; rank r binds port+r and advertises the base address under the kv_events block.
   • ›Adds a JSON config flag for NVIDIA Dynamo KV event publishing; KV event publishing is enabled when this flag is set.
   • ›Adds --speculative-dsa-topk-backend flag to choose the DSA indexer top-k backend for speculative draft workers.
3. v0.5.18 Aug 22, 2026 · issue 004

   SGLang v0.5.18 adds overlapped checkpoint staging, FlashInfer MNNVL allreduce, unified cache dir, 7 new models, and extensible serve backend plugins.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.5.18 https://github.com/sgl-project/sglang.git
   # already have the repo? check out this version:
   $ git checkout v0.5.18

   └──▷ TRY IT
   Dramatically cut server cold-start time on large models by overlapping weight loading with CUDA graph capture.
   copy

   $ python -m sglang.launch_server --model-path Qwen/Qwen3-32B --startup-weight-load-mode overlap

   Boost small-batch decode throughput on non-DeepSeek models by routing allreduce through the FlashInfer MNNVL workspace.
   copy

   $ python -m sglang.launch_server --model-path <model> --enable-flashinfer-pure-allreduce

   Redirect all SGLang kernel caches to a shared persistent volume to avoid recompilation across container restarts.
   copy

   $ export SGLANG_CACHE_DIR=/mnt/shared/sglang_cache
   python -m sglang.launch_server --model-path <model>

   • ›Adds --startup-weight-load-mode overlap flag to overlap checkpoint page staging with CUDA graph capture at startup, cutting Qwen3-32B on H100 startup time from 84.8s to 35.6s (2.38x faster).
   • ›Adds --enable-flashinfer-pure-allreduce flag to route non-fused allreduce sites through the FlashInfer MNNVL workspace instead of NCCL; auto-enabled for DeepSeek-V3/V3.2/V4, delivering up to +6.9% decode throughput at small batches on Blackwell.
   • ›Adds --dit-layerwise-residency-policy flag for strided DiT residency control in diffusion model serving.
   • ›Consolidates Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches under a single SGLANG_CACHE_DIR environment variable.
   • ›Adds extensible serve backend plugins via feat(cli): add extensible serve backend plugins.
   +14 moreshow less

   • ›Enables delayed sampling for general overlap decode via SGLANG_ENABLE_DELAY_SAMPLE environment variable, allowing sampling overhead to overlap with model execution.
   • ›Adds native multimodal processing for Qwen VL in the Rust server via integrated sglang-mm.
   • ›Adds support for new models: Muse Glimmer (autoregressive multimodal), Intern-S2-Mobius (autoregressive), SANA-Video (diffusion), LingBot-Video-MoE (diffusion), LTX-2.5 (diffusion), Cosmos3 Edge & Distilled (diffusion), and LongCat-Image (diffusion).
   • ›Supports logprobs output with DSpark and DFlash speculative decoding.
   • ›Supports MegaMoE for DSpark under DP attention.
   • ›Adds FlashInfer CuTe DSL NVFP4 MoE quantization support.
   • ›Adds FlashInfer mHC fusion for DeepSeek-V4, now enabled by default.
   • ›Adds SM12x FA4 architecture-owned kernels.
   • ›Adds a Triton MoE TMA up kernel.
   • ›Adds explicit EPLB balancedness reporting modes.
   • ›Adds a content-addressed JIT build cache generated from a custom ninja build system.
   • ›Adds tuned Triton tile configs for channelwise FP8 GEMM.
   • ›Replaces allgather + scatter in TP LMHead with a single all-to-all for pure-DP dp-attention, reducing LMHead time from 320us to 169us on DeepSeek-V4-Pro B200.
   • ›Updates core dependencies: torch 2.13.0 with triton 3.7.1, flashinfer 0.6.17, CuTeDSL 4.6.2, DeepEP from sgl-deep-ep wheels, and sgl-kernel 0.4.6.post1.
   └──▷ BREAKING ON UPGRADE

   • !All kernel caches (Triton, FlashInfer, Inductor, DeepGEMM, CUDA driver) now live under SGLANG_CACHE_DIR; the first launch after upgrading will recompile all kernels once.
   • !The --torchao-config flag and its torchao integration have been removed.
4. v0.5.17 Aug 8, 2026 · issue -011

   SGLang v0.5.17 adds Kimi K3 and MiniMax-H3 day-0 support, a Rust frontend, new DCP backends, DWDP prefill, and session-aware radix caching.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.5.17 https://github.com/sgl-project/sglang.git
   # already have the repo? check out this version:
   $ git checkout v0.5.17

   └──▷ TRY IT
   Enable session-aware radix cache eviction for an agentic workload so the server does not evict prefixes an active session still references.
   copy

   $ python -m sglang.launch_server --model-path <model> --enable-session-radix-cache

   Switch DeepSeek-MLA DCP decode to the FlashInfer MNNVL kernel on GB200 and skip the per-layer Q all-gather to reduce cross-rank communication overhead.
   copy

   $ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --dcp-comm-backend fi_a2a --dcp-replicate-q-proj

   Accelerate large MoE prefill by using DWDP to prefetch expert weights over NVLink P2P instead of dispatching tokens all-to-all across expert-parallel ranks.
   copy

   $ python -m sglang.launch_server --model-path <moe-model> --dwdp-size 4

   • ›Adds --dcp-comm-backend {ag_rs, a2a, fi_a2a} to select pluggable communication backends for the DeepSeek-MLA decode context-parallel path; a2a exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer with fp8 KV as uint8 byte transport, and fi_a2a delegates cross-rank exchange to the FlashInfer MNNVL kernel on GB200.
   • ›Adds --dcp-replicate-q-proj flag to project full-head Q locally and skip the per-layer Q head-dim all-gather in DCP decode.
   • ›Adds --dwdp-size to enable DWDP (Download-While-Dispatch-Prefill) parallelism for MoE prefill, which prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch; achieves 1.92x over DEP4 at MNT 32K / ISL 32K on 4x B200 with gpt-oss-120b.
   • ›Adds --enable-session-radix-cache flag and /close_session endpoint for session-reference-aware Unified Radix Cache, allowing requests to carry a stable session_id so eviction respects prefixes still referenced by active sessions — targeted at agentic and RL-rollout workloads.
   • ›Adds SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 environment variable to enable the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation.
   +12 moreshow less

   • ›Adds SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D environment variable to copy oversized or non-contiguous CPU weight views into contiguous storage before H2D transfer, cutting DeepSeek-V4-Pro TP8 load time from ~35 minutes to 6m20s (5.6x) and GPT-OSS-20B BF16 from 545s to 70s (7.8x); off by default.
   • ›Adds day-0 support for Kimi K3, a 2.8T-parameter multimodal LatentMoE with 1M-token context, served with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on quantized weights, and OpenAI-compatible serving; verified on NVIDIA GB300 and AMD MI35x.
   • ›Adds day-0 support for MiniMax-H3 video-and-audio generation on SGLang-Diffusion across task profiles t2va (text-to-video-and-audio), fl2va (first/last-frame conditioning), and ref2va (image/video/audio reference conditioning); verified on B200 (TP2 + Ulysses4), H100, AMD MI300X/MI355X, and 2x RTX 5090 with layerwise offload.
   • ›Adds an initial Rust frontend that replaces the Python server path from network ingress through tokenization and request hand-off to the GPU scheduler with a multi-threaded Rust implementation, including tokenizer manager, ingress validation, egress, OpenAI-compatible API server, and PD disaggregation support; shipped as prebuilt release artifacts.
   • ›Adds a weight-cache daemon that holds GPU weights so a restarting engine can recover from cache rather than reloading from storage and recapturing CUDA graphs, reducing restart time for large models (e.g., Qwen3-235B FP8 on 4 GPUs from ~6.5 minutes).
   • ›Adds support for EmbeddingGemma and LFM2.5 embedding models, nvidia/MiniMax-M3-NVFP4, and cookbook recipes for Poolside Laguna-S-2.1 and Inkling-Small.
   • ›Adds trtllm_mha support for Gemma 4 MTP draft attention backend in speculative decoding.
   • ›Adds grammar-constrained decoding support (including tool_choice=auto) for DSpark speculative decoding.
   • ›Adds grammar-constrained decoding support in DFLASH speculative verify.
   • ›Adds grammar overlap scheduling for standalone speculative decoding.
   • ›Removes unnecessary expert padding on AMD, dropping MI355X FP4 MoE model weights from 159.07 GB to 112.36 GB and cutting colocated RL training-phase footprint from ~143 GiB to ~87 GiB per GPU.
   • ›Updates dependencies: flashinfer 0.6.15.post1, sgl-deep-gemm 0.1.5.post1, helion 1.4, mooncake 0.3.12.post1, dynamo-tokenizers 1.7.0; PyTorch remains at 2.11.0 and CUDA base image at 13.0.1.
5. v0.5.17 Aug 8, 2026 · issue 002

   SGLang v0.5.17 adds Kimi K3 and MiniMax-H3 day-0 support, a Rust frontend, DWDP MoE prefill, session-aware radix cache, and major load-time speedups.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.5.17 https://github.com/sgl-project/sglang.git
   # already have the repo? check out this version:
   $ git checkout v0.5.17

   └──▷ TRY IT
   Enable session-reference-aware prefix caching for an agentic workload so active-session prefixes survive eviction until explicitly released.
   copy

   $ python -m sglang.launch_server --model-path <model> --enable-session-radix-cache

   Speed up large-MoE model loading (e.g. DeepSeek-V4-Pro) by copying non-contiguous CPU weight views before the H2D transfer.
   copy

   $ SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 python -m sglang.launch_server --model-path <model> --tp 8

   Use the FlashInfer MNNVL comm backend for DCP decode on GB200, and skip the per-layer Q all-gather to reduce cross-rank traffic.
   copy

   $ python -m sglang.launch_server --model-path <model> --dcp-comm-backend fi_a2a --dcp-replicate-q-proj

   • ›Adds --dcp-comm-backend {ag_rs, a2a, fi_a2a} to select the DCP decode context-parallel communication backend: a2a exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer with FP8 KV as uint8 byte transport; fi_a2a delegates to the FlashInfer MNNVL kernel on GB200.
   • ›Adds --dcp-replicate-q-proj to project full-head Q locally and skip the per-layer Q head-dim all-gather in the DCP decode path.
   • ›Adds --dwdp-size to enable DWDP (weight-and-data parallelism) prefill, which prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch; benchmarked at 1.92x over DEP4 on 4x B200 at MNT 32K / ISL 32K.
   • ›Adds --enable-session-radix-cache so requests carrying a stable session_id use session-reference-aware eviction in the Unified Radix Cache; active-session prefixes are not evicted until the session is released via /close_session.
   • ›Adds SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 environment variable to enable the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation.
   +13 moreshow less

   • ›Adds SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D environment variable to copy oversized or non-contiguous CPU weight views into contiguous storage before H2D transfer, cutting DeepSeek-V4-Pro TP8 load time from ~35 minutes to ~6m20s (5.6x) and GPT-OSS-20B BF16 from 545s to 70s (7.8x).
   • ›Ships an initial native Rust serving layer covering tokenizer manager, ingress validation, egress, OpenAI-compatible API server, and PD disaggregation support, as prebuilt release artifacts.
   • ›Adds day-0 support for Kimi K3 (2.8T-parameter multimodal LatentMoE, 1M-token context, native MXFP4) with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on quantized weights, and reasoning/tool-call/OpenAI-compatible serving, verified on NVIDIA GB300 and AMD MI35x.
   • ›Adds day-0 support for MiniMax-H3 video-generation model producing synchronized video and stereo audio in one request, across task profiles t2va (text-to-video-and-audio), fl2va (first/last-frame conditioning), and ref2va (image/video/audio reference conditioning), verified on B200, H100, AMD MI300X/MI355X, and 2x RTX 5090 with layerwise offload.
   • ›Adds support for EmbeddingGemma and LFM2.5 embedding models, plus nvidia/MiniMax-M3-NVFP4.
   • ›Adds a weight-cache daemon that holds weights per GPU so a restarting engine can recover from cache instead of reloading from storage, reducing Qwen3-235B FP8 restart time from ~6.5 minutes.
   • ›Adds grammar-constrained decoding support to DSpark speculative decoding, including tool_choice=auto.
   • ›Adds trtllm_mha as a supported draft attention backend for Gemma 4 MTP speculative decoding.
   • ›Adds the SGLANG_SIMULATE_ACC_LEN environment variable for use with DFlash speculative decoding.
   • ›Reduces DeepSeek-V4 AMD memory footprint by removing unnecessary expert padding, dropping MI355X FP4 MoE model weights from 159.07 GB to 112.36 GB, and reclaims the HIP compress-state pool for RL colocated training, cutting training-phase footprint from ~143 GiB to ~87 GiB per GPU.
   • ›Updates dependencies to flashinfer 0.6.15.post1, sgl-deep-gemm 0.1.5.post1, helion 1.4, mooncake 0.3.12.post1, and dynamo-tokenizers 1.7.0; PyTorch remains at 2.11.0 and the CUDA base image at 13.0.1.
   • ›Adds support for Qwen3.5-MoE modelopt_fp4 checkpoints that quantize attention and load baked FP8 KV scales.
   • ›Adds grammar overlap scheduling for standalone speculative decoding.
6. v0.5.16 Jul 25, 2026 · issue -025

   SGLang v0.5.16 adds DSpark speculative decoding, Inkling 975B model support, DSA cache layer split, and ReplaySSM Ring Spec-Verify across 574 PRs.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.5.16 https://github.com/sgl-project/sglang.git
   # already have the repo? check out this version:
   $ git checkout v0.5.16

   └──▷ TRY IT
   Run DSpark speculative decoding on a single-user DeepSeek workload to maximize token throughput with confidence-adaptive verify windows.
   copy

   $ SGLANG_RAGGED_VERIFY_MODE=compact python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --speculative-algorithm DSPARK --speculative-dspark-block-size 8 --tp 8

   Enable DSA cache layer split to cut per-rank KV memory ~74% when running GLM-5.2 with prefill context parallelism across 4 ranks.
   copy

   $ python -m sglang.launch_server --model-path THUDM/GLM-5.2-FP8 --enable-prefill-cp --cp-strategy interleave --enable-dsa-cache-layer-split --tp 8

   Reduce speculative decoding GPU scratch memory 6.4x by enabling GDN ReplaySSM Ring Spec-Verify with a custom ring cache length.
   copy

   $ python -m sglang.launch_server --model-path Qwen/Qwen3.5-35B-A3B --enable-gdn-replayssm-spec --linear-replayssm-cache-len 16 --speculative-eagle-topk 1 --tp 1

   • ›Adds --speculative-algorithm DSPARK flag for DSpark confidence-driven speculative decoding, with --speculative-dspark-block-size to tune block size; requires SGLANG_RAGGED_VERIFY_MODE=compact environment variable — reaches 383.7 tok/s at accept length ~5 on DeepSeek-V4-Pro TP8.
   • ›Adds --enable-dsa-cache-layer-split flag (requires --enable-prefill-cp --cp-strategy interleave) for GLM-5.2 DSA cache layer sharding across CP ranks, cutting per-rank KV memory ~74% (0.77 to 0.20 GB/rank) at 8192 tokens.
   • ›Adds --enable-gdn-replayssm-spec flag for ReplaySSM Ring Spec-Verify (GDN), reducing speculative scratch from 11.5 GB to 1.8 GB per GPU (6.4x smaller) on Qwen3.5-35B-A3B at TP1; tune ring size with --linear-replayssm-cache-len. Requires --speculative-eagle-topk in {None, 1}.
   • ›Adds native support for Inkling, a 975B-parameter multimodal MoE with 1M-token context, NVFP4 MoE, optional vision/audio towers, and native MTP — verified on Blackwell TP4/TP8, H200, and AMD MI350X/MI355X.
   • ›Makes UnifiedRadixTree the default cache tree for SWA, Mamba, and DSA models, with Replay SSM and Mamba int8 checkpoints synced onto it and cache-hit state reset scoped to only the used state.
   +6 moreshow less

   • ›Adds support for new models: LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5 (VLA), and LongLive 2.0 (diffusion).
   • ›Adds linear attention decode kernel (recurrent_kda) for Blackwell SM100, running at 29.6 µs vs 36.8 µs for Triton at B=64, with 1.35x throughput at B=256.
   • ›Adds DeepSeek-V4 Wint4Abf16 and Wint4Afp8 quantization support.
   • ›Adds SM120 DeepSeek V4 flashinfer_mxfp4 MoE runner backend with TP2 support.
   • ›Integrates Q8KV8 FP8 Sparse MLA Prefill into the DSA backend for DeepSeek-V3.2.
   • ›Fuses state I/O into the chunk-h kernel for GDN/KDA CuteDSL prefill.
   └──▷ BREAKING ON UPGRADE

   • !The experimental QServe (QoQ) W4A8 and FBGEMM FP8 quantization paths are removed; any server launch using these will fail.
   • !CUTLASS FP8 blockwise kernels are removed for SM90/SM100; SM120 is moved to JIT.
   • !--fp4-gemm-backend cutlass is removed along with in-tree NVFP4 JIT kernels; NVFP4 GEMM now requires FlashInfer. Use auto, which picks flashinfer_cutedsl on SM100 and flashinfer_cutlass on SM120.
   • !UnifiedRadixTree is now the default for SWA, Mamba, and DSA models, changing cache behavior on those architectures.
   • !Chunked input-logprob processing is now on by default to cap peak memory.
   • !FA3 sparse mask kernels are off by default.
   • !Kernels are relocated to the sglang.kernels namespace; code reaching past public wrappers to internal import paths must update.
   • !num_tokens_per_bs is renamed to num_tokens_per_req across spec-decoding runners.
   • !--enable-deepep-waterfill is renamed to --enable-waterfill with no deprecated alias; existing launch commands will fail with unrecognized arguments.
   • !--optimistic-prefill-retries is renamed to --optimistic-prefill-attempts with no deprecated alias.
   • !The SGLang-Diffusion post-training rollout endpoint now returns application/msgpack instead of JSON, with tensors as raw msgpack bytes; tensor_to_base64 / base64_to_tensor are replaced by tensor_to_bytes / bytes_to_tensor — RL rollout consumers must be upgraded in lockstep with the server.
7. v0.5.16 Jul 25, 2026 · issue 002

   SGLang v0.5.16 adds DSpark speculative decoding, Inkling 975B model support, GLM-5.2 DSA cache layer split, and ReplaySSM ring spec-verify.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.5.16 https://github.com/sgl-project/sglang.git
   # already have the repo? check out this version:
   $ git checkout v0.5.16

   └──▷ TRY IT
   Enable DSpark speculative decoding for high-throughput single-batch inference on a DeepSeek-class model, using confidence-driven verify windows instead of a fixed draft length.
   copy

   $ SGLANG_RAGGED_VERIFY_MODE=compact python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --speculative-algorithm DSPARK --speculative-dspark-block-size 4 --tp 8

   Reduce speculative decoding GPU memory from ~11.5 GB to ~1.8 GB per GPU when using ReplaySSM-based drafting on a Qwen MoE model.
   copy

   $ python -m sglang.launch_server --model-path Qwen/Qwen3.5-35B-A3B --enable-gdn-replayssm-spec --speculative-eagle-topk 1 --linear-replayssm-cache-len 64 --tp 1

   Cut per-rank KV memory by ~74% when serving GLM-5.2 at large context with context parallelism by splitting DSA cache layers across CP ranks.
   copy

   $ python -m sglang.launch_server --model-path THUDM/GLM-5.2-FP8 --enable-prefill-cp --cp-strategy interleave --enable-dsa-cache-layer-split

   • ›Adds --speculative-algorithm DSPARK and --speculative-dspark-block-size flags for DSpark confidence-driven speculative decoding, which drafts semi-autoregressively in blocks and sizes each verify window from the draft's own confidence; also requires SGLANG_RAGGED_VERIFY_MODE=compact environment variable.
   • ›Adds --enable-dsa-cache-layer-split flag for GLM-5.2 DSA cache layer sharding across CP ranks, cutting per-rank KV memory by ~74% (0.77 to 0.20 GB/rank) at 8192 tokens on GLM-5.2-FP8 with cp_size=4; requires --enable-prefill-cp --cp-strategy interleave.
   • ›Adds --enable-gdn-replayssm-spec flag for ReplaySSM Ring Spec-Verify (GDN), reducing speculative scratch memory from 11.5 GB to 1.8 GB per GPU (6.4x smaller); tunable via --linear-replayssm-cache-len; requires --speculative-eagle-topk in {None, 1}.
   • ›Adds support for Inkling, a 975B-parameter multimodal MoE with 1M-token context, mixing sliding-window, full and Mamba2 linear attention, NVFP4 MoE, optional vision/audio towers, and native MTP; verified on Blackwell TP4/TP8, H200, and AMD MI350X/MI355X.
   • ›UnifiedRadixTree is now the default KV cache tree for SWA, Mamba, and DSA models, with SSM and Mamba int8 checkpoint replay synced onto it and cache hits now resetting only the state they used.
   +9 moreshow less

   • ›Adds new model support for LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5 (VLA), and LongLive 2.0 (diffusion).
   • ›Adds linear attention (KDA) decode kernel for Blackwell (SM100) via FlashInfer, reaching 29.6 µs vs 36.8 µs for Triton at B=64, and 1.35x at B=256.
   • ›Adds SGLANG_MORI_COMBINE_DTYPE=fp4 environment variable for FP4 combine dtype in the MORI EP backend.
   • ›Supports n>1 outputs for GLM-Image generation.
   • ›Supports RL rollout for the WAN diffusion pipeline via a per-request scheduler switch.
   • ›Adds Q8KV8 FP8 Sparse MLA Prefill integration into the DSA backend for DeepSeek-V3.2.
   • ›Adds DeepSeek-V4 Wint4Abf16 and Win4Afp8 quantization support.
   • ›Adds SM120 DeepSeek V4 flashinfer_mxfp4 MoE runner backend with TP2 support.
   • ›Adds GDN/KDA CuteDSL prefill kernel fusing state I/O into the chunk-h kernel on Blackwell.
   └──▷ BREAKING ON UPGRADE

   • !The experimental QServe (QoQ) W4A8 and FBGEMM FP8 quantization paths are removed; models relying on these quantization methods will no longer load.
   • !CUTLASS FP8 blockwise is deleted for SM90/SM100; SM120 is moved to JIT.
   • !--fp4-gemm-backend cutlass is removed along with in-tree NVFP4 JIT kernels; NVFP4 GEMM now requires FlashInfer. Use auto, which picks flashinfer_cutedsl on SM100 and flashinfer_cutlass on SM120.
   • !UnifiedRadixTree is now the default for SWA, Mamba, and DSA models, changing cache behavior on those architectures.
   • !Chunked input-logprob processing is now on by default to cap peak memory.
   • !FA3 sparse mask kernels are off by default.
   • !The sglang.kernels namespace relocates internal kernel paths; code reaching past public wrappers to internal paths must update import paths.
   • !num_tokens_per_bs is renamed to num_tokens_per_req across spec-decoding runners; code or configs referencing the old name will break.
   • !--enable-deepep-waterfill is renamed to --enable-waterfill with no deprecated alias; existing launch commands will fail with unrecognized arguments.
   • !--optimistic-prefill-retries is renamed to --optimistic-prefill-attempts with no deprecated alias; existing launch commands will fail.
   • !The SGLang-Diffusion post-training rollout endpoint now returns application/msgpack instead of JSON, with tensors as raw msgpack bytes (tensor_to_bytes / bytes_to_tensor) instead of base64 (tensor_to_base64 / base64_to_tensor); RL rollout consumers must be upgraded in lockstep with the server.
8. v0.5.15 Jul 10, 2026 · issue -040

   SGLang v0.5.15 adds native Exa web search, decode context parallelism, new FlashKDA kernels, and expands model support to Hunyuan 3, HRM-Text, and more.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.5.15 https://github.com/sgl-project/sglang.git
   # already have the repo? check out this version:
   $ git checkout v0.5.15

   • ›Adds flashinfer_trtllm_routed MoE runner with FlashInfer all-to-all for routed Mixture-of-Experts inference.
   • ›Adds opt-in non-paged indexer for DeepSeek-V4 long-context prefill, now enabled by default for large prefill chunks, with >5% end-to-end throughput gain.
   • ›Enables Breakable CUDA Graph as the default capture path, reducing per-step kernel-launch overhead; full CUDA Graph for the prefill phase lands as experimental.
   • ›Adds web_search built-in tool backed by Exa for native web search support.
   • ›Adds FlashKDA prefill backend for safe-gate KDA linear attention, plus ReplaySSM buffered output-only decode for linear attention.
   +10 moreshow less

   • ›Adds Spec V2 as the default speculative decoding path with CUDA-graphable DSA draft-extend, dropped D2H/H2D syncs, and fused metadata ops for +11% end-to-end TPS.
   • ›Adds IndexShare MTP, reusing indexer top-k across draft steps for up to 1.9x lower draft-step cost at long context.
   • ›Adds TopK V2 fusing top-k selection with the page-table transform, supporting runtime k up to 2048.
   • ›Enables FlashMLA sparse prefill by default for DeepSeek-V4, providing >10% throughput gain on long context.
   • ›Enables decode context parallelism for MLA models including DeepSeek V3 and Kimi K2 series.
   • ›Adds shape-specialized JIT router GEMM and CuteDSL BF16 GEMM support for Blackwell (SM120).
   • ›Extends FlashInfer autotune coverage to draft-model graphs.
   • ›Adds new model support: Hunyuan 3 (Hy3), Hierarchical Reasoning Model (HRM-Text), NVIDIA LocateAnything-3B, Baidu Unlimited-OCR, JoyEcho multi-shot A/V, and Qwen3.6 NVFP4.
   • ›Adds speculative decoding support on XPU.
   • ›Bumps transformers to 5.12.1 and upgrades tvm-ffi, sgl-deep-gemm, and tilelang.
9. v0.5.14 Jun 26, 2026 · issue -054

   SGLang v0.5.14 adds 7 new models, LPLB/Waterfill MoE load balancing, NVFP4 MoE, MSCCL++ allreduce fusion, and AMD breakable CUDA graph support.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v0.5.14 https://github.com/sgl-project/sglang.git
   # already have the repo? check out this version:
   $ git checkout v0.5.14

   └──▷ TRY IT
   Enable the LPLB linear-programming load balancer to balance token routing across redundant MoE expert replicas for DeepSeek-V3/V4 workloads.
   copy

   $ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4 --ep-dispatch-algorithm lp

   Enable NVFP4 MoE quantization on a Blackwell GPU for higher DeepSeek-V4 MoE decode throughput.
   copy

   $ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4 --moe-runner-backend flashinfer_trtllm_routed

   • ›Adds --ep-dispatch-algorithm=lp to enable the new LPLB linear-programming load balancer for MoE expert parallelism, balancing token routing across redundant expert replicas by solving a per-layer LP; default behavior unchanged.
   • ›Adds --moe-runner-backend flashinfer_trtllm_routed to enable NVFP4 MoE quantization for DeepSeek-V4 on Blackwell GPUs for higher MoE throughput.
   • ›Adds --flashinfer-allreduce-fusion-backend flag (values: auto, trtllm, mnnvl) to control the FlashInfer fused allreduce + residual + RMSNorm MNNVL backend.
   • ›Adds SGLANG_ENABLE_WAR_BARRIER environment variable to force-enable the overlap scheduler WAR barrier on non-CUDA platforms such as AMD ROCm.
   • ›Adds support for online NVFP4 quantization via new online nvfp4 quantization path.
   +41 moreshow less

   • ›Adds support for asymmetric compressed-tensors MoE.
   • ›Adds Mooncake group semantics for disaggregated serving parallelism.
   • ›Adds bucketed multi-directory layout for NIXL file storage.
   • ›Adds get_parallel() structured accessor for parallel-topology state.
   • ›Adds graceful scheduler shutdown with hisparse host buffer freed on exit.
   • ›Adds support for GLM-4.7 function calling via structural tags.
   • ›Adds request/TTFT/worker metrics and a Grafana dashboard to the experimental sgl-router.
   • ›Adds W8A8 MXFP8 quantization support for Qwen3 Dense on Ascend NPU.
   • ›Adds Ascend NPU support for DeepSeek-V4.
   • ›Adds Gemma4 Sliding Window Attention support on the Ascend backend.
   • ›Adds LoRA support for DSA indexer LoRA targets for GLM-5.1 and DeepSeek-V3.2-family models.
   • ›Adds rejection sampling for MTP speculative decoding.
   • ›Adds sync-free fast_prefill_plan for EAGLE draft-extend CUDA graph.
   • ›Adds FlashInfer CUDA graph support for EAGLE draft-extend.
   • ›Adds ngram speculative decoding v2 (ngram spec v2).
   • ›Adds sliding window attention draft layer support for Dflash.
   • ›Adds piecewise CUDA graph support for Dflash.
   • ›Adds CuteDSL prefill kernel on SM100 (Blackwell) for Kimi-Linear (KDA), delivering 1.08–1.52x faster prefill than the Triton path via a reusable scratch workspace.
   • ›Adds int8 checkpoint pool for the linear-attention prefix cache (KDA/GDN models), substantially increasing prefix-cache capacity.
   • ›Deduplicates speculative conv-window intermediate cache via a sliding-window layout, halving its footprint with no numerical change.
   • ›Enables breakable CUDA graph execution on AMD ROCm/HIP.
   • ›Migrates MSCCL++ to the upstream mscclpp Python package (Executor + DSL compiler) with auto-tuned collectives for TP=8 single-node and TP=16 two-node configurations.
   • ›Adds context parallel strategy abstractions for new parallelism configurations.
   • ›Adds prefill context parallel support for DeepSeek-V4 unified KV attention on AMD.
   • ›Adds EPD disaggregated encode tracing.
   • ›Adds MPServer and embedded server support for granian to enable multiple tokenizer workers.
   • ›Adds Nemotron DP attention and MTP support for the hybrid Nemotron-H architecture.
   • ›Adds GB10 FP8 fused MoE Triton configuration.
   • ›Adds SwiGLU activation fused into gate gather_qmv for SwitchGLU MoE blocks on Apple Silicon (MLX).
   • ›Adds MTP support for DeepSeek-V4 Online Compress.
   • ›Adds MLA decode q-head padding to 64 under attention-TP, dispatching the cheaper FlashMLA head64 kernel instead of head128.
   • ›Prewarms the MHC prenorm kernel at startup to eliminate first-run JIT slowdown.
   • ›Supports BF16 mixed-dtype compression states on C4/C128 paths for DeepSeek-V4.
   • ›Adds new model support for GLM-5.2, LiquidAI LFM2.5, Kimi-K2.7-Code, Poolside Laguna-M.1, DiffusionGemma, Zyphra ZAYA1, and MiMo-V2-ASR.
   • ›Adds DeepSeek-V4 MTP acceptance length checks.
   • ›Fuses sigmoid + mul attention output gate into a single Triton kernel on AMD.
   • ›Enables fused GDN QKV split Triton kernel on HIP (AMD).
   • ›Fuses sigmoid + mul into a single Triton kernel for shared expert gating on AMD.
   • ›Adds LocalAttention for mistral3 encoder in diffusion models.
   • ›Defers MoE finalize and fuses it with main-stream add for DeepSeek-V3.
   • ›Fuses UE8M0 scale rounding directly into the FP8 group quantization kernel, removing a separate rounding pass.
10. v0.5.13 Jun 13, 2026 · issue -067

    SGLang v0.5.13 ships Spec V2 as default, nine new models, DeepSeek V4 context parallelism, and HiCache for hybrid models.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.13 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.13

    • ›Adds flash_mla_sparse_fwd sparse FlashMLA kernel for DeepSeek V4, exposed via sgl_kernel.flashmla in sgl-kernel 0.4.3.
    • ›Spec V2 is now the default speculative-decoding path; tree drafting with topk > 1 and page_size > 1 is production-ready across triton, FA3, MLA, and aiter backends including Mamba/hybrid-linear models.
    • ›Adds adaptive speculative decoding with batch-size-aware num_steps and observability metrics.
    • ›Extends Spec V2 to the FlashMLA backend.
    • ›HybridModel (SWA/Mamba) now launches HiCache via UnifiedTree by default, bringing hierarchical KV-cache offload to sliding-window and Mamba hybrids out of the box.
    +21 moreshow less

    • ›Adds decode-side HiCache integration for incremental KV-cache transfer in PD disaggregation.
    • ›Adds HiSparse support for DeepSeek V4 with PD disaggregation.
    • ›Adds Pipeline Parallelism (PP) + PD support for DeepSeek V4.
    • ›Adds context-parallel serving for DeepSeek V4, including Context Parallel + MTP and Context Parallel + fused MoE kernel (non-DeepEP).
    • ›Adds DeepEP waterfill load balancing for DeepSeek V4.
    • ›Adds FP4 indexer support and SM120 support for DeepSeek V4.
    • ›Adds Piecewise CUDA Graph (PCG) support for DSA models and Kimi-K2.5.
    • ›Adds Breakable CUDA Graph (BCG) support for DeepSeek V4.
    • ›Adds draft-extend CUDA Graph for the trtllm mha attention backend.
    • ›Adds CuTeDSL MLA attention kernels from FlashInfer.
    • ›Adds FlashInfer Gated DeltaNet (GDN) kernels and a CuTeDSL GDN prefill kernel for Qwen 3.5 on Blackwell GPUs.
    • ›Adds prefill context parallelism for MLA models (Kimi K2.5, DeepSeek V3) and support for batch size > 1 during prefill CP.
    • ›Adds optimistic prefill for improved TTFT in PD disaggregation.
    • ›Adds EPD disaggregation support for MiMo-V2.
    • ›Heterogeneous CPU + GPU EPD disaggregation: offloads VLM vision encoding onto Intel Xeon CPUs, delivering up to ~1.3x P99 TTFT and request-throughput gains under load.
    • ›Adds MoRI communication library integration for AMD Instinct MI355X, enabling cost-competitive DeepSeek-R1 disaggregated inference.
    • ›Adds realtime video generation for SGLang-Diffusion with msgpack frame streaming, a standalone browser WebUI, continuous camera controls, super-resolution controls, and progressive-resolution growing across FLUX, FLUX.2, Qwen-Image, Wan, and Z-Image.
    • ›Adds support for new autoregressive models: Nemotron 3 Ultra (Day-0 with kernel optimizations), Step-3.7-Flash, and Command A+ (Cohere 2 family).
    • ›Adds support for new diffusion models: Cosmos3-Nano/Super (T2V/I2V/T2I), Ernie-Image, LingBot-World (realtime/causal-DMD), SANA-WM (streaming + realtime), FLUX.2-Klein 4B/9B, and Ideogram 4 (FP8/NVFP4 with tensor parallelism).
    • ›Reduces per-step scheduler overhead via unified async value passing through FutureMap and moving prefill input transfer onto the forward stream.
    • ›Spec V1 is deprecated; EAGLE/MTP now run on the unified V2 worker.
    └──▷ BREAKING ON UPGRADE

    • !Spec V1 is deprecated; EAGLE/MTP now run on the unified Spec V2 worker — configurations relying on the Spec V1 path will need to migrate to Spec V2.
11. v0.5.12 May 16, 2026 · issue -095

    SGLang v0.5.12 adds DeepSeek V4 inference, TokenSpeed MLA backend, HiCache/UnifiedRadixTree, and a wave of new model support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.12 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.12

    └──▷ TRY IT
    Skip KV pool allocation to run a prefill-only benchmark without wasting GPU memory on a KV cache.
    copy

    $ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --prefill-only-disable-kv-cache

    Force every radix-cache lookup to miss, useful for isolating cold-start latency during profiling.
    copy

    $ SGLANG_RADIX_FORCE_MISS=1 python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3

    Tokenize a chat-formatted message via the new /v1/tokenize endpoint to inspect token counts before committing to a full generate call.
    copy

    $ curl http://localhost:30000/v1/tokenize -H 'Content-Type: application/json' -d '{"model": "deepseek-ai/DeepSeek-V3", "messages": [{"role": "user", "content": "Hello!"}]}'

    • ›Adds --prefill-only-disable-kv-cache flag to skip KV pool allocation entirely for prefill-only workloads.
    • ›Adds SGLANG_TRACE_LEVEL environment variable to control startup trace verbosity.
    • ›Adds SGLANG_MAX_KV_CHUNK_CAPACITY environment variable to cap KV chunk capacity.
    • ›Adds SGLANG_RADIX_FORCE_MISS environment variable to force radix-cache misses.
    • ›Adds --enable-strict-thinking server flag enabling two-phase reasoning grammar for structured thinking output.
    +41 moreshow less

    • ›Adds --random-input-len argument to send_one.py for synthetic load generation.
    • ›Renames SGLANG_USE_JIT_ALL_REDUCE to SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2; the old name no longer applies.
    • ›Enables SGLANG_OPT_FP8_WO_A_GEMM by default, activating FP8 weight-only GEMM optimizations without manual configuration.
    • ›Adds /v1/tokenize endpoint with chat-completion-style request support.
    • ›Adds sglang:get_loads_duration_seconds Prometheus metric for load-duration observability.
    • ›Adds fwd_occupancy metric to SchedulerStats plus a matching Prometheus collector.
    • ›Adds per-iteration forward-pass metrics published over ZMQ PUB socket.
    • ›Adds SWA and Mamba radix-cache KV event metrics.
    • ›Adds full inference path for DeepSeek V4, including Tensor/Expert/Context/Data Parallel Attention, Prefill-Decode Disaggregation, HiSparse KV offload, reasoning and tool-call parsers, DeepGemm and FlashMLA kernels, W4A4/W4A8 MoE kernels, and TP16 on H100/H20.
    • ›Adds unified Docker image tag lmsysorg/sglang:v0.5.12 covering all Nvidia GPU variants.
    • ›Adds TokenSpeed MLA attention backend for Blackwell (SM100) with FP8 KV cache support for low-latency MLA serving.
    • ›Adds HiCache framework support for UnifiedRadixTree (including SWA) with SSD offload via Mooncake store.
    • ›Adds custom speculative-algorithm registry, enabling user-defined spec-decoding strategies.
    • ›Adds EagleDraftExtendInput split for Spec V1 draft-extend paths.
    • ›Adds trtllm decode kernel for draft extend in speculative decoding.
    • ›Adds Adaptive Spec V2, EAGLE-3 SWA support, Kimi K2.5 EAGLE-3 MLA, and Gemma 3/4 + EAGLE-3 speculative decoding.
    • ›Adds MLA attention LoRA support for q_b_proj and kv_b_proj projections.
    • ›Adds CSGMV backend with virtual experts for MoE LoRA.
    • ›Adds LoRADrainer to reduce P99 TTFT under heavy LoRA workloads.
    • ›Adds Torch-Native LoRA embedding and graph optimization.
    • ›Adds NIXL staging buffer for heterogeneous-TP KV transfer in PD disaggregation.
    • ›Adds NIXL async transfer support.
    • ›Adds Mooncake incremental transfer and SSD offload for PD disaggregation.
    • ›Adds PrefillDelayer NCCL all-gather for cross-DP info sync.
    • ›Adds adaptive queue-based prefill-delayer trigger.
    • ›Adds retry-with-backoff for prefill bootstrap registration.
    • ›Adds Azure Blob Storage connector supporting az:// URIs and *.blob.core.windows.net endpoints.
    • ›Adds Crusoe managed-inference backend.
    • ›Adds structural tags for strict tool calling and reasoning across additional models.
    • ›Adds auto-detection of reasoning and tool-call parser from chat template.
    • ›Adds OpenAI reasoning.enabled mapping to thinking and enable_thinking fields.
    • ›Adds multi-detokenizer support for concurrent detokenization.
    • ›Enforces rejection of repetition_penalty=0 in SamplingParams.verify().
    • ›Adds new model support: Intern-S2-Preview, MiniCPM-V 4.6, Laguna-XS.2, Ring-2.6-1T (1T-parameter reasoning model), Gemma 4 MTP, Trinity-mini (Ascend NPU), HunyuanVideo ModelOpt FP8, and Qwen Image ModelOpt FP8.
    • ›Adds CFG parallelism framework with multi-branch CFG for LTX-2 in SGLang-Diffusion.
    • ›Adds initial dynamic batching for SGLang-Diffusion.
    • ›Adds dit_precision config key for diffusion models, replacing hardcoded bf16.
    • ›Adds component attention-backend override CLI for SGLang-Diffusion.
    • ›Adds AMD online MXFP4 and FP8 diffusion quantization, aiter RMSNorm, and dual-stream MoE for ROCm.
    • ›Migrates DeepEP source to deepseek-ai/DeepEP@hybrid-ep for clean CUDA 13 builds; pins FlashInfer at 0.6.11.post1.
    • ›Adds TMA bulk-store set_mla_kv_buffer kernel delivering up to 12x throughput over baseline.
    └──▷ BREAKING ON UPGRADE

    • !SGLANG_USE_JIT_ALL_REDUCE is renamed to SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2; configurations or scripts referencing the old name will no longer take effect.
    • !SGLANG_OPT_FP8_WO_A_GEMM is now enabled by default; deployments that previously relied on it being off must explicitly disable it.
    • !VAE CPU offload is disabled by default in SGLang-Diffusion; workflows that depended on automatic CPU offload must re-enable it explicitly.
12. v0.5.11 May 5, 2026 · issue -106

    SGLang v0.5.11 adds DFLASH spec-decode, decode-side radix cache for PD disaggregation, LoRA on DeepSeek-V3/Kimi-K2, and 14+ new model architectures.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.11 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.11

    └──▷ TRY IT
    Monitor speculative-decoding efficiency per request using the new split metrics instead of the old aggregate accept_length.
    copy

    $ # After upgrade, query your metrics endpoint — accept_length is gone:
    curl http://localhost:30000/metrics | grep -E 'num_accepted_drafts|num_accepted_tokens'

    Scrape SGLang's Prometheus metrics from a gRPC-mode server (previously only available in HTTP mode).
    copy

    $ curl http://localhost:30000/metrics

    • ›Splits accept_length metric into num_accepted_drafts and num_accepted_tokens for finer speculative-decoding observability.
    • ›Adds --page-size support with speculative decoding on AMD ROCm (previously caused a memory access fault).
    • ›Exposes raw KV cache pool token counts as Prometheus gauges.
    • ›Adds a Prometheus metrics endpoint for gRPC mode.
    • ›Adds HTTP sidecar endpoints and a FlushCache gRPC RPC for gRPC mode.
    +33 moreshow less

    • ›Surfaces pending token count in prefill log and get_load.
    • ›Supports moe_dp_size = 1 paired with arbitrary attention_cp_size so MoE and attention parallelism can be tuned independently.
    • ›New DFLASH speculative-decoding kernel, enabled across additional model backends and AMD ROCm.
    • ›Speculative Decoding V2 (Spec V2) with overlap scheduling is now the default, reducing per-step CPU cost for EAGLE/MTP/DFLASH paths.
    • ›Adds adaptive speculative_num_steps for EAGLE topk=1.
    • ›Enables piecewise CUDA graph with speculative decoding.
    • ›Adds OpenTelemetry tracing for speculative decoding, pipeline parallelism, and DiffGenerator.
    • ›Adds decode-side radix cache (Decode Radix Cache) for prefill/decode (PD) disaggregation, recovering cache hit rates and TTFT savings for long shared prefixes.
    • ›Adds incremental transfer for the Mooncake transfer engine in PD disaggregation.
    • ›Adds NIXL heterogeneous TP KV transfer for non-MLA models.
    • ›Adds NIXL Mamba state slice transfer for heterogeneous TP.
    • ›Enables LoRA on DeepSeek-V3 MLA (including quantization) and Kimi-K2.
    • ›Adds LoRADrainer to reduce high P99 TTFT under LoRA workloads.
    • ›Adds decoupled LoRA MoE backend with Marlin support.
    • ›Adds dual MoE CUDA graph capture for lora/nolora batches.
    • ›Adds CSGMV kernel offline auto-tuning for LoRA.
    • ›New FlashInferCuteDslMoE layer for the standard FP4 MoE path.
    • ›Integrates community-contributed FA3 kernels alongside FA4 as a high-performance alternative.
    • ›Adds NVFP4 KV cache with a quantization strategy abstraction and kernel.
    • ›Adds GPTQ / AWQ 4-bit quantization on CPU.
    • ›Adds GGUF quantization for Ascend NPU (dense and MoE models).
    • ›Adds TP communications compression for Qwen3 on Ascend NPU.
    • ›Adds pipeline parallelism support on Intel XPU.
    • ›New SequenceClassification model architecture, powering the Score API.
    • ›Adds day-0 support for Gemma 4, GLM-5.1, Qwen3.6, MiMo-V2.5/V2.5-Pro, Ling-2.6-Flash, Mistral Medium 3.5, Kimi-K2.6, Hunyuan v3 (preview), Voxtral, Parakeet, Moss-VL, and Qwen3-ASR (chunk-based streaming).
    • ›Adds diffusion model support for LTX-2.3 (including two-stage and TI2V), ERNIE-Image, FLUX.2-small-decoder, JoyAI-Image-Edit, FLUX.1-dev ModelOpt NVFP4, Qwen Image ModelOpt FP8, and Stable Diffusion 3 medium.
    • ›Adds ModelOpt diffusion FP8 support for Flux1/Flux2 and Wan2.2.
    • ›Adds Standalone Rollout API, Denoising Environment Backpass, and SP-Aligned Log-Prob for T2I post-training in SGLang-Diffusion.
    • ›Adds disaggregated diffusion support.
    • ›Adds dynamic batching v0 for SGLang-Diffusion.
    • ›Adds CPU platform support for SGLang-Diffusion.
    • ›Enables in-memory loading for URL/base64 image inputs in SGLang-Diffusion (now default).
    • ›Default CUDA version moves to 13.0 across SGLang, sgl-kernel, and Docker images; PyTorch upgraded from 2.9 to 2.11.
13. v0.5.10 Apr 6, 2026 · issue -135

    SGLang v0.5.10 adds elastic failure tolerance, LoRA-on-MoE, native Apple Silicon, and a raft of new model and kernel capabilities.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.10 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.10

    └──▷ TRY IT
    Control sparse MLA attention kernel activation threshold during DeepSeek V3.2 prefill to tune accuracy vs. speed trade-off.
    copy

    $ SGLANG_NSA_DENSE_ATTN_KV_LEN_THRESHOLD=4096 python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 4

    Include token usage stats in every streaming response for billing or monitoring without per-request opt-in.
    copy

    $ python -m sglang.launch_server --model meta-llama/Llama-3-8B-Instruct --stream-response-default-include-usage

    • ›Adds SGLANG_NSA_DENSE_ATTN_KV_LEN_THRESHOLD environment variable to control the KV length threshold at which the sparse MLA attention kernel is applied during prefill for DeepSeek V3.2.
    • ›Adds --stream-response-default-include-usage server flag to control whether usage stats are included in streaming responses by default.
    • ›Adds --strict-ports option for predictable, deterministic port assignment.
    • ›Adds NetworkAddress abstraction for IPv6-safe address handling, enabling IPv6 dual-stack deployments.
    • ›Enables piecewise CUDA graph capture by default, reducing memory overhead and improving throughput for models with complex control flow.
    +34 moreshow less

    • ›Integrates Elastic NIXL-EP (Elastic EP) for partial failure tolerance in DeepSeek MoE deployments — when a GPU fails, expert weights are redistributed from DRAM and serving continues without a full restart.
    • ›Adds GPU staging buffer with dynamic ring allocator for PD disaggregation, gathering scattered GQA head slices into contiguous memory for bulk RDMA transfer (~1000x fewer RDMA requests, ~5x TPS/GPU at large concurrency on Qwen3.5 with Prefill TP4+Decode DEP4).
    • ›Integrates HiSparse sparse attention backend for efficient long-context inference with sparsity-aware attention, also enabling HiSparse direct cache transfer from Prefill to Decode DRAM.
    • ›Integrates FlashInfer MXFP8 kernels for GEMM and MoE operations, enabling mixed-precision FP8 inference with microscaling for RL and general workloads.
    • ›Adds LoRA fine-tuning support for Mixture-of-Experts layers, including JIT alignment kernels, fused Triton kernels, TP support, CUDA graph support, and auto-detection of LoRA target modules.
    • ›Upgrades from transformers 4.57.1 to 5.3.0, adding support for latest HuggingFace model architectures including GLM-5 on the main branch.
    • ›Upgrades to the official Flash Attention 4 package, bringing Blackwell GPU support and the latest attention optimizations.
    • ›Enables speculative decoding for the FA4 attention backend.
    • ›Multi-modal attention now uses FA4 by default on SM100 (Blackwell) hardware for improved VLM performance.
    • ›Enables context parallelism during prefill for multi-head attention models (e.g. Qwen3 MoE), distributing long sequences across GPUs.
    • ›Adds Score API via EngineScoreMixin implementation.
    • ›Adds subprocess liveness monitor to detect scheduler crashes.
    • ›Adds direct model loading from object storage with RunAI Model Streamer.
    • ›Adds MFU (Model FLOPs Utilization) metrics exposed via Prometheus.
    • ›Enables IndexCache for DeepSeek V3.2, improving throughput by more than 10% on high-concurrency workloads.
    • ›Adds TRT-LLM prefill/decode DSA kernels as the default for Blackwell (SM100/SM103) hardware.
    • ›Enables NCCL/RCCL pre-warming to reduce P99 TTFT cold-start latency.
    • ›Enables multi-thread weight loading by default.
    • ›Adds CuTeDSL KDA decode kernel support and fused GDN projection Triton kernel for improved Qwen3.5 performance.
    • ›Adds enhanced transformers modeling backend with full TP, PP, MoE, VLM support, and torch.compile compatibility.
    • ›Renames the kernel package from sgl-kernel to sglang-kernel (v0.4.1), consolidating kernels and removing deprecated ops.
    • ›Replaces decord with torchcodec for video decoding in VLM pipelines.
    • ›Replaces soundfile+torchaudio with torchcodec AudioDecoder in load_audio for VLM audio ingestion.
    • ›Adds chunk-aware ViT encoding with per-image cache and lazy device transfer for VLM workloads.
    • ›Adds macOS platform support for diffusion models.
    • ›Integrates Cache-DiT optimizations into the diffusers backend for improved diffusion performance.
    • ›Adds NVFP4 support for Flux.2 diffusion model.
    • ›Adds LTX-2 two-stage pipeline support for diffusion.
    • ›Adds EPLB rebalance support for Kimi K2.5 in Elastic EP.
    • ›Adds reasoning tokens usage reporting.
    • ›Adds new model support: Nemotron-3-Super (bf16/fp8/nvfp4), Mistral Small 4 (Pixtral), LFM2-VL, Voxtral (speech-to-text), Helios, Hunyuan3D-2, LTX-2, MOVA, FireRed-Image-Edit.
    • ›Adds AMD: FP8 prefill integration with radix cache path for DeepSeek models, MHA FP8-KV support, AMD MXFP4 support for Qwen3.5-397B-A17B, and FP8 KV cache + FP8 attention kernel for NSA on MI300/MI355 with TileLang.
    • ›Adds CPU backend kernel apply_rotary_pos_emb_cpu for Qwen3-VL and Qwen3-Omni, and MXFP4 GEMM kernels for Intel AMX to support GPT-OSS series.
    • ›Adds NPU/Ascend support for Kimi-K2.5-w4a8 and diffusion models (with enable_torch_compile), plus GLM-4.7-Flash on NPU.
    └──▷ BREAKING ON UPGRADE

    • !The kernel package is renamed from sgl-kernel to sglang-kernel (v0.4.1); any install scripts, requirements.txt, or import paths referencing sgl-kernel or sgl_kernel must be updated.
    • !Transformers upgraded from 4.57.1 to 5.3.0; model configs or custom modeling code that relied on transformers 4.x APIs may require updates.
    • !decord is replaced by torchcodec for video decoding; environments with decord installed for VLM video pipelines must install torchcodec instead.
    • !soundfile+torchaudio replaced by torchcodec AudioDecoder in load_audio; environments relying on those libraries for audio ingestion in VLM pipelines must install torchcodec.
14. v0.5.9 Feb 24, 2026 · issue -174

    SGLang v0.5.9 adds Anthropic-compatible API, NSA kernel integration, LoRA overlap loading, FA4 multimodal support, and 10+ new models.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.9 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.9

    └──▷ TRY IT
    Run DeepSeek V3.2 on a Blackwell GPU with TRT-LLM Native Sparse Attention enabled for both prefill and decode to get 3x–5x throughput improvement.
    copy

    $ python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3-2 --nsa-prefill-backend trtllm --nsa-decode-backend trtllm

    • ›Adds --nsa-prefill-backend and --nsa-decode-backend flags integrating TRT-LLM Native Sparse Attention kernels for DeepSeek V3.2, delivering 3x–5x throughput gains on Blackwell platforms.
    • ›Adds native Anthropic API compatibility endpoint, enabling direct use of tools and clients built for the Anthropic API format.
    • ›Adds SGLANG_MOONCAKE_CUSTOM_MEM_POOL environment variable for configuring Mooncake intra-node NVLink KV transfer memory pool.
    • ›Adds routed-experts field to /completions and /chat/completions endpoints for MoE models.
    • ›Introduces LoRA weight loading overlap with computation, reducing TTFT by ~78% and TPOT by ~34.88% on large adaptors.
    +16 moreshow less

    • ›Adds Flashinfer all-to-all MoE dispatcher for efficient expert parallelism communication in MoE models.
    • ›Introduces FA4 (FP4 attention) backend and variable-length attention function for multimodal encoders, enabling lower-precision inference for vision-language models.
    • ›Adds KV transfer support via MORI-IO for prefill-decode disaggregation.
    • ›Adds INT4 kernels for the CPU backend.
    • ›Adds JointThreshold algorithm for joint M2T and T2T decoding in diffusion LLMs.
    • ›Adds basic dLLM scheduling strategy and implementation.
    • ›Removes CUDA graph batch size limitation for diffusion LLMs.
    • ›Updates AITER to v0.1.10.post3, adding FP8 Prefill, FP8 Decode, and FP8 KV Cache support on AMD hardware.
    • ›Adds commit-to-version lookup tool at docs.sglang.io/references/release_lookup.html for tracing PRs to official releases.
    • ›Adds new model support for Kimi-K2.5, GLM-5, Qwen 3.5, MiniMax 2.5, Ernie4.5-VL, Step3-VL, Step-3.5-Flash, LLaDA 2.1, Ring 2.5 1T / Ling 2.5 1T, MOVA (Diffusion), GLM-OCR, and DeepSeek-OCR-2.
    • ›Adds diffusion pipeline support for parallel VAE decoding, token-level sequence sharding, Nunchaku and FP8 support, fused kernels, LTX-2, and multiple new models in the ComfyUI plugin.
    • ›Adds MxINT4 Flashinfer TRT-LLM MoE GEMM support.
    • ›Adds Kimi K2.5 Day 0 ROCm support and standardizes on ROCm 7.
    • ›Adds Qwen 3.5 support on Ascend NPU.
    • ›Adds MiniCPM3-4B model support on NPU/Ascend.
    • ›Migrates GPTQ-Marlin and AWQ Marlin repack kernels to JIT compilation.
    └──▷ BREAKING ON UPGRADE

    • !ROCm 6.3 is deprecated; ROCm 7 is now the standard for AMD hardware.
15. v0.5.8 Jan 23, 2026 · issue -206

    SGLang v0.5.8 adds Flash Attention 4, chunked pipeline parallelism, Multi-LoRA diffusion inference, and support for GLM 4.7 Flash, LFM2, Qwen3-VL, and DeepSeek V3.2 NVFP4.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.8 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.8

    • ›Supports Flash Attention 4 decoding kernels for improved attention performance.
    • ›Adds chunked pipeline parallelism enabling close-to-linear scaling for million-token context windows.
    • ›Adds Multi-LoRA inference support in SGLang-Diffusion, along with SLA attention backends, a warmup switch in the CLI, and a ComfyUI Plugin.
    • ›Adds configurable generator device and seed support for the diffusion API.
    • ›Supports running SGLang-Diffusion with the diffusers backend.
    +16 moreshow less

    • ›Adds Context Parallelism Optimization for DeepSeek V3.2 with support for fused MoE, multi-batch, and FP8 KV cache.
    • ›Supports pipeline parallelism (PP) x PD decode disaggregation with the nixl backend.
    • ›Supports decode pipeline parallelism for PD disaggregation.
    • ›Adds MXFP8 Grouped GEMM support for Blackwell (B200) GPUs via sgl-kernel.
    • ›Adds day-0 model support for GLM 4.7 Flash.
    • ›Adds model support for LFM2.
    • ›Adds model support for Qwen3-VL-Embedding and Qwen3-VL-Reranker.
    • ›Adds model support for DeepSeek V3.2 NVFP4.
    • ›Adds model support for black-forest-labs/FLUX.2-klein-9B diffusion model.
    • ›Adds model support for Mistral Large 3.
    • ›Adds single batch overlap for MoE models, improving GPU utilization.
    • ›Adds positional embedding ID cache for the Qwen-VL model family, reducing recomputation overhead.
    • ›Adds CPU support for the chunk_gated_delta_rule kernel for Qwen3-Next.
    • ›Adds CPU support for mamba causal conv1d for Qwen3-Next.
    • ›Delivers up to 1.5x throughput improvement across all major diffusion models.
    • ›Achieves 65% faster TTFT for GLM4-MoE in production via EPD disaggregation and elastic encoder scaling for vision-language models.
16. gateway-v0.3.1 Jan 9, 2026 · issue -220

    SGLang gateway v0.3.1 delivers 10-12x faster cache-aware routing, 99% memory reduction, JWT/OIDC auth, and a new Classification API.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.3.1 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.3.1

    • ›Adds JWT/OIDC authentication for control plane APIs (tokenizer management, worker registration, admin endpoints) with native support for Google, Azure, Oracle, and GitHub identity providers.
    • ›Adds PrefixHash load balancing policy — a new KV cache-aware routing strategy using prefix hashing to improve cache hit rates in multi-tenant environments.
    • ›Adds native Classification API support with dedicated pipeline stages and protocol types for serving classification models alongside inference workloads.
    • ›Adds Nemotron Nano V3 reasoning parser support.
    • ›Adds in-flight request age metrics for observability and SLA monitoring.
    +3 moreshow less

    • ›Radix tree / cache-aware routing engine overhauled: INSERT throughput up from 38 MB/s to 440 MB/s, MATCH throughput up from 83 MB/s to 253 MB/s, latency down from 52.9 µs to 4.6 µs per operation, and memory per node down from ~180 KB to ~1.4 KB (~99% reduction).
    • ›Organizes CLI arguments into logical groups for improved discoverability.
    • ›HTTP Router fan-out replaced from serial execution to concurrent streams for reduced latency under load.
17. v0.5.7 Jan 1, 2026 · issue -228

    SGLang v0.5.7 adds day-0 support for Mimo-V2-Flash, LLaDA 2.0, EAGLE 3 speculative decoding, Model Gateway v0.3.0, and --dit-layerwise-offload for diffusion models.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.7 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.7

    └──▷ TRY IT
    Reduce peak VRAM by up to 30 GB when serving a diffusion model on a memory-constrained GPU.
    copy

    $ python -m sglang.launch_server --model-path <diffusion-model-path> --dit-layerwise-offload true

    Override protected sampling fields and control frame count when generating video with the diffusion CLI.
    copy

    $ python -m sglang.diffusion.cli --model-path <diffusion-model-path> --adjust-frames 16 --override-protected-fields

    • ›Adds --dit-layerwise-offload true CLI flag for SGLang-Diffusion to reduce peak VRAM usage by up to 30 GB and improve performance by up to 58% across all diffusion models.
    • ›Adds --adjust-frames and --override-protected-fields CLI arguments to the diffusion pipeline.
    • ›Day-0 support for Mimo-V2-Flash, Nemotron-Nano-v3, and LLaDA 2.0 models.
    • ›Day-0 support for Qwen-Image-Edit-2509, Qwen-Image-Edit-2511, Qwen-Image-2512, and Qwen-Image-Layered in SGLang-Diffusion.
    • ›Releases Model Gateway v0.3.0 with multimodal initialization and image processors for LLaVA, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Phi3-Vision, Phi4-Vision, LLaMA-4-Vision, and Mistral 3.
    +15 moreshow less

    • ›EAGLE 3 speculative decoding draft models now available for popular models.
    • ›Adds scalable pipeline parallelism with dynamic chunking for ultra-long contexts.
    • ›Adds encoder disaggregation support for multi-modal models.
    • ›Enables RadixCache for Mamba2 models.
    • ›Supports pipeline parallelism (PP) combined with PD decode disaggregation via the nixl backend.
    • ›Adds MXFP8 Grouped GEMM support on Blackwell (B200) GPUs via sgl-kernel.
    • ›Adds Cache-dit integration for SGLang-Diffusion to improve performance by up to 165%.
    • ›Extends SGLang-Diffusion to support AMD, RTX 4090, and RTX 5090 GPUs, additional attention backends (sage-attn, sage-attn3), tensor parallelism (TP), and Google Vertex AI in the HTTP API.
    • ›Adds single-batch overlap for MoE models to improve throughput.
    • ›Adds Mistral Large 3 and Mistral Large 3 EAGLE speculative decoding support.
    • ›Adds Ministral 3 model support.
    • ›Supports data-parallelism for GLM-V vision models.
    • ›Adds configurable generator device and seed support for the diffusion HTTP API.
    • ›Adds a lightweight runtime Docker stage via Dockerfile refactor.
    • ›Adds model architecture and type fields to the server info endpoint for gateway harmony.
18. gateway-v0.3.0 Dec 24, 2025 · issue -236

    SGLang Gateway v0.3.0 adds unified IGW mode, tokenize/detokenize endpoints, parser endpoints, gRPC embeddings, TLS, and a Go implementation.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.3.0 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.3.0

    • ›Adds /parse/reasoning and /parse/function_call HTTP endpoints for parsing reasoning outputs and function call responses, including a GLM-4 function call parser.
    • ›Adds tokenize/detokenize HTTP endpoints with a dynamic tokenizer control plane (add, list, get, remove tokenizers on-the-fly) backed by a TokenizerRegistry.
    • ›Adds native embeddings endpoint for the gRPC router, expanding the gateway beyond text generation to embedding workloads.
    • ›Adds server-side TLS support with graceful shutdown for both TLS and non-TLS servers.
    • ›Adds X-SMG-Error-Code response headers and unified error codes across all router types.
    +8 moreshow less

    • ›Adds streaming metrics (TTFT, TPOT, tokens, duration) for the gRPC router, plus smg_worker_*, smg_router_*, MCP, and discovery metric layers in a new 6-layer Prometheus metrics architecture.
    • ›Adds custom Prometheus duration buckets.
    • ›Introduces Unified Inference Gateway (IGW) mode: a single gateway instance now supports gRPC (PD and regular), HTTP (PD and regular), and OpenAI routers simultaneously, auto-enabled when Kubernetes service discovery is turned on.
    • ›Adds retry and circuit breaker support for OpenAI and gRPC routers; returns HTTP 503 when all workers are circuit-broken.
    • ›Adds a complete Go implementation of the SGLang Model Gateway with an OpenAI-compatible API server, contributed by the iFlytek MaaS team.
    • ›Adds multi-architecture Docker builds supporting Linux, macOS, Windows, and ARM.
    • ›Introduces a DAG-based parallel workflow engine with pre-computed dependency graphs and concurrent event processing for worker lifecycle orchestration.
    • ›Replaces PolicyRegistry RwLock with DashMap for lock-free policy lookups and adds lock-free router snapshots to reduce CPU overhead.
    └──▷ BREAKING ON UPGRADE

    • !Prometheus metric names and structure have changed (complete 6-layer architecture redesign) — update all Prometheus dashboards and alerting rules.
    • !Workers are now identified by UUIDs instead of endpoints — update any tooling or scripts that interact with the worker API.
    • !SmgMetrics is renamed to Metrics and smg_labels is renamed to metrics_labels — update any code referencing these names.
    • !Legacy RouterMetrics has been removed — migrate to the new unified metrics layer.
19. gateway-v0.2.4 Dec 10, 2025 · issue -250

    SGLang Model Gateway v0.2.4 adds WASM middleware, full OpenTelemetry tracing, multimodal VL model support, and broad performance optimizations.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.2.4 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.2.4

    • ›Adds WASM middleware support, enabling safe, isolated WebAssembly plugins for custom routing logic, request/response transformation, and proprietary system integration without touching core code.
    • ›Adds OpenTelemetry distributed tracing to the HTTP router with native trace context propagation across the inference stack.
    • ›Adds OpenTelemetry integration to the gRPC router, extending distributed tracing coverage to both HTTP and gRPC transports.
    • ›Makes the tokenizer builder aware of environment variables such as HF_ENDPOINT for flexible model loading configuration.
    • ›Adds a version command to the SGLang Model Gateway CLI, exposed via the Python binding.
    +13 moreshow less

    • ›Adds support for routing Vision-Language (VL) models in the gateway router.
    • ›Adds image processors for LLaVA, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Phi3 Vision, Phi4 Vision, Llama4 Vision, and Mistral 3 multimodal models.
    • ›Adds a workflow for routing to external model providers.
    • ›Adds ModelCard and ProviderType support to worker metadata for richer model configuration.
    • ›Adds audio and moderation entries to the model card.
    • ›Adds HTTP response status code metrics to the router.
    • ›Implements TokenGuardBody for managing and controlling token return in responses.
    • ›Optimizes the radix tree used in cache-aware load balancing for lower routing overhead.
    • ›Optimizes the tokenizer for dramatically reduced CPU and memory footprint.
    • ›Supports numactl CPU and memory binding before the gateway process starts.
    • ›Allows the token refill rate to be set to zero.
    • ›Promotes conversation handling to first-class routing in the OpenAI router.
    • ›Renames the Rust package and binary from sgl-router to sgl-model-gateway.
    └──▷ BREAKING ON UPGRADE

    • !The Rust package and binary have been renamed from sgl-router to sgl-model-gateway; any scripts, container images, or CI pipelines referencing sgl-router by name will break.
20. v0.5.6 Dec 3, 2025 · issue -257

    SGLang v0.5.6 adds DeepSeek V3.2, new diffusion models, JIT kernels, multimodal cache, bucket routing policy, and scheduler metrics.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.6 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.6

    • ›Adds SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL environment variable for controlling EPLB heatmap collection intervals in PD deployments.
    • ›Adds intel_xpu attention backend for Llama 4 on Intel XPU hardware.
    • ›Adds server-level multimodal cache support for VLMs.
    • ›Adds bucket routing policy for the SGLang router.
    • ›Adds scheduler and hiradix cache metrics via the metrics system.
    +20 moreshow less

    • ›Supports structured model output for OpenAI and gRPC router endpoints.
    • ›Introduces JIT kernels support.
    • ›Supports piecewise CUDA graph for MLA (Multi-head Latent Attention).
    • ›Adds support for DeepSeek V3.2 and DeepSeek V3.2 Speciale models, including MHA pathway for short-sequence prefill on B200 (SM100) via NSA.
    • ›Adds support for blockwise diffusion language models.
    • ›Adds support for new diffusion models: Flux2 and Z-image.
    • ›Supports Kimi-K2-Thinking model enhancements.
    • ›Adds Jet-Nemotron model support.
    • ›Adds pipeline parallelism (PP) support for dots_vlm.
    • ›Adds aux_hidden_states capture support for Minimax M2.
    • ›Upgrades to Torch 2.9.
    • ›Updates AMD wave-lang to 3.8.2.
    • ›Supports hidden_dim % 4 == 0 in per_token_quant_fp8 kernel, broadening FP8 quantization coverage.
    • ›Adds fallback to Triton mm_persistent kernel when DeepGEMM fails.
    • ›Refactors KTransformers heterogeneous compute with a unified GPU-quantization backend.
    • ›Adds row_starts parameter to the topk kernel for DeepSeek V3.2.
    • ›Reduces radix tree syscalls and optimizes collection filtering.
    • ›Supports WAN-2.2-TI2V and sequence parallelism for diffusion models.
    • ›Adds validation for served model name to reserve : for LoRA adapter syntax.
    • ›Refactors custom memory pool and adds bare-metal PD (prefill-decode disaggregation) support.
21. gateway-v0.2.3 Nov 17, 2025 · issue -273

    SGLang Gateway v0.2.3 adds bucket-mode routing (20-30% TTFT gain), PostgreSQL chat history, and expanded tool/structured-output support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.2.3 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.2.3

    • ›Adds PostgreSQL as a supported data connector for chat history management, joining OracleDB and in-memory storage options.
    • ›Introduces bucket-based routing policy for PD mode, delivering 20-30% improvements in TTFT and overall throughput.
    • ›Adds structured model output support for the OpenAI and gRPC router.
    • ›Adds streaming parsing with Tool Choice in the chat completions API for the gRPC router.
    • ›Adds tool_choice support for the Responses API in the gRPC router.
    +7 moreshow less

    • ›Emits OutputItemDone events and stores output item arrays in the gRPC router for improved observability.
    • ›Adds support for the MinMax M2 model, including an XML tool parser and reasoning parser.
    • ›Adds vLLM backend support for the gRPC router.
    • ›Adds support for complex assistant and tool messages in /chat/completions.
    • ›Adds timing metrics aggregation for requests in sgl-router.
    • ›Adds engine metrics aggregation support in sgl-router.
    • ›Removes the worker URL requirement for router configuration.
22. gateway-v0.2.2 Nov 17, 2025 · issue -273

    SGLang Gateway v0.2.2 adds Responses API for OSS models, gRPC health checks, mTLS, and universal platform support in a single wheel.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.2.2 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.2.2

    └──▷ TRY IT
    Install the new gateway release with universal platform and Python version support.
    copy

    $ pip install sglang-router==0.2.2

    • ›Adds v1/responses API endpoint to the gateway, bringing OpenAI Responses API support to open-source models including Llama, DeepSeek, and Qwen with built-in chat history management and multi-turn conversations.
    • ›Adds native gRPC standard health check endpoint support for production Kubernetes deployments with proper health monitoring and orchestration.
    • ›Adds mTLS support for router-to-worker communication, securing gateway-to-SGLang traffic across edge, remote cloud, multi-cloud, and hybrid environments.
    • ›Adds function calling support for the OpenAI router Responses API.
    • ›Adds web_search_preview tool support for the OpenAI router.
    +6 moreshow less

    • ›Adds multiple worker URL support for the OpenAI router for better load distribution.
    • ›Introduces the Harmony Pipeline: a unified OpenAI-native architecture supporting GPT OSS models for both Responses API and Chat Completion with MCP integration.
    • ›Adds MCP connection pooling, tool inventory, and per-server proxy configuration.
    • ›Adds workflow retry and timeout configuration via routerConfig.
    • ›Extends platform support to Linux, Windows, macOS, x86, and ARM in a single wheel file covering Python 3.8 through 3.14, while reducing wheel size by more than 40%.
    • ›Adds fine-grained request latency tracing (Part 2) for detailed performance tracking.
23. gateway-v0.2.1 Nov 17, 2025 · issue -273

    SGLang Gateway v0.2.1 adds a two-level tokenizer cache, a new /v1/classifications endpoint, and a worker management workflow engine.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.2.1 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.2.1

    └──▷ TRY IT
    Run a classification request against the new OpenAI-compatible endpoint to score or label text without a separate model service.
    copy

    $ curl -X POST http://localhost:8080/v1/classifications \
      -H 'Content-Type: application/json' \
      -d '{"model": "my-model", "input": "This transaction looks fraudulent."}'

    • ›Adds GET /v1/classifications endpoint, bringing OpenAI-style classification API support to the router.
    • ›Adds configurable two-level tokenizer caching: L0 exact-match cache for repeated prompts and L1 prefix-aware cache at special-token boundaries.
    • ›Introduces a Worker Management Workflow Engine with improved async registration, worker self-discovery, and health orchestration.
    • ›Adds parallel queue puts in the gRPC request manager and removes the mutex on the gRPC client, enabling concurrent gRPC request handling.
    └──▷ BREAKING ON UPGRADE

    • !Removes continue_final_message from ChatTemplateParams in the gRPC router.
24. gateway-v0.2.0 Nov 17, 2025 · issue -273

    SGLang Model Gateway v0.2.0 adds multi-model IGW mode, Rust-powered gRPC data plane, pluggable history storage, and native MCP integration.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.2.0 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.2.0

    └──▷ TRY IT
    Use the conversation CRUD API to retrieve stored conversation history from the gateway.
    copy

    $ curl -X GET http://localhost:8080/v1/chat/conversations

    • ›Adds serve_grpc to launch_server for a Rust-native gRPC data plane with tokenizer, reasoning parser, and tool parser running at native speed with real-time token streaming.
    • ›Adds CLI flags for queue size, timeout, and rate limit (--queue-size, --timeout, --rate-limit) to configure the token-bucket rate limiter (disabled by default).
    • ›Adds reasoning and tool parser arguments to the router CLI for controlling parsing behavior in gRPC and HTTP modes.
    • ›Adds --chat-template CLI argument to allow specifying a custom chat template path (directory or file).
    • ›Adds gRPC PD (prefill-disaggregation) mode for both /v1/chat/completions and /generate endpoints via the gRPC router.
    +12 moreshow less

    • ›Adds logprobs support to the gRPC router for /v1/chat/completions.
    • ›Adds full ToolChoice support and streaming tool-call parsing in the gRPC router.
    • ›Introduces Multi-Model Inference Gateway (IGW) mode, letting a single gateway manage multiple models — each with its own routing policy, priorities, and metadata — with dynamic model registration via /workers.
    • ›Adds pluggable conversation history storage via memory, none, or oracle backends; oracle provides full persistence via Oracle ATP with connection pooling and credentials support.
    • ›Adds GET /v1/chat/conversations and conversation item CRUD APIs (create, retrieve, delete) for history management.
    • ›Adds native MCP integration across all transports (STDIO, HTTP, SSE, Streamable) with multi-turn tool-calling loop support and streaming MCP tool-call responses for agentic workflows.
    • ›Adds OpenAI-compatible gateway mode to proxy requests to OpenAI while keeping conversation history and responses flowing through the local gateway, including support for Grok models.
    • ›Adds worker self-discovery for metadata, enabling automatic registration of worker capabilities and labels.
    • ›Adds IPv6 support across all gateway components.
    • ›Adds Prometheus metrics for latency, load, queue depth, PD pipelines, tokenizer speed, and MCP activity, along with structured tracing and request-ID propagation.
    • ›Adds per-worker circuit breakers, retries with exponential backoff and jitter, and FIFO queuing for reliability.
    • ›Adds active request cancellation on client disconnect using RAII-based cleanup.
25. gateway-v0.1.9 Nov 17, 2025 · issue -273

    SGLang Gateway v0.1.9 adds configurable retry logic for the PD router and completes the OpenAI API spec for the router.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.1.9 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.1.9

    • ›Adds configurable retry logic to the PD router to reduce backend pressure under load.
    • ›Completes OpenAI API spec coverage for the router, expanding compatible endpoint support.
    • ›Introduces a data-parallel (DP) worker abstraction in the router for improved worker management.
    • ›Implements an HTTP dependency injection pattern for the router system, enabling more modular handler composition.
26. gateway-v0.1.7 Nov 17, 2025 · issue -273

    SGLang gateway-v0.1.7 adds dp-aware routing and per-node policies for PD disaggregation in sgl-router

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.1.7 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.1.7

    • ›Adds a dp-aware routing strategy to sgl-router, enabling data-parallel-aware request distribution across worker nodes.
    • ›Adds separate routing policies for prefill (p) nodes and decode (d) nodes in sgl-router, allowing independent policy configuration per node type.
    • ›Migrates sgl-router HTTP layer from actix to axum, improving request handling internals.
    • ›Improves router logging and request ID header propagation in sgl-router.
27. gateway-v0.1.6 Nov 17, 2025 · issue -273

    SGLang Gateway v0.1.6 adds stream_options forwarding, request-completion metrics updates, and empty-worker router support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch gateway-v0.1.6 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout gateway-v0.1.6

    • ›Forwards stream_options in routed requests so streaming configuration is correctly propagated to backend workers.
    • ›Updates router metrics when a request completes, enabling accurate in-flight and throughput tracking.
    • ›Allows the router to start and operate with an empty worker pool, removing the requirement for at least one worker to be registered at startup.
28. v0.5.5 Nov 6, 2025 · issue -284

    SGLang v0.5.5 adds Kimi-K2, MiniMax-M2, NVILA model support, mTLS for router workers, MCP connection pooling, and ~10% B200 MoE perf gains.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.5 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.5

    └──▷ TRY IT
    Run MiniMax-M2 with the new model support on a multi-GPU node.
    copy

    $ python -m sglang.launch_server --model-path MiniMaxAI/MiniMax-M2 --tp 8

    • ›Adds mTLS support for router-to-worker communication, securing inter-component traffic in multi-node deployments.
    • ›Adds MCP Manager to the router with connection pooling, tool inventory, and proxy support.
    • ›Adds flashinfer_trtllm MoE runner backend, delivering ~10% throughput improvement on B200 FP8 DeepSeek workloads.
    • ›Adds deterministic inference support for DeepSeek architecture models on a single GPU.
    • ›Adds DeepGEMM support for deterministic inference.
    +10 moreshow less

    • ›Adds DeepSeek-R1 w4a8 low-latency DeepEP support.
    • ›Adds CUDA graph support for trtllm_mla attention backend spec decoding, improving speculative decoding throughput.
    • ›Adds support for MiniMax-M2 model.
    • ›Adds support for NVILA and NVILA Lite vision-language models.
    • ›Adds support for Kimi-K2-Thinking model.
    • ›Adds Llama 4 + TRTLLM MHA support.
    • ›Adds ResponsesContext to the gRPC router responses API with improved error propagation.
    • ›Adds MHA chunked-prefix optimization that merges prefix and extend KV cache to run MHA in a single pass.
    • ›Removes the deprecated --enable-beta-spec CLI argument.
    • ›Decouples GGUF quantization implementation from the vLLM dependency, and adds gguf as a first-class project dependency for CPU/XPU targets.
    └──▷ BREAKING ON UPGRADE

    • !The --enable-beta-spec argument has been removed; invocations passing this flag will fail on upgrade.
29. v0.5.4 Oct 26, 2025 · issue -295

    SGLang v0.5.4 adds overlap scheduler for speculative decoding, piecewise CUDA graph prefill, new model support, and ModelOpt quantization.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.4 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.4

    └──▷ TRY IT
    Tokenize a prompt using the new OpenAI-compatible endpoint, useful for counting tokens before sending a generation request.
    copy

    $ curl http://localhost:30000/tokenize -H 'Content-Type: application/json' -d '{"model": "<model>", "text": "Hello, world!"}'

    • ›Adds --sampling-defaults argument to the OAI serving chat interface to set server-wide sampling parameter defaults for ChatCompletionRequest.
    • ›Adds /tokenize and /detokenize OpenAI-compatible endpoints.
    • ›Adds reasoning and tool parser argument support in the router.
    • ›Adds IPv6 support across all router components.
    • ›Adds OpenAI router conversation API CRUD (create, retrieve, delete) for history management using conversation items.
    +19 moreshow less

    • ›Adds get server info and get model info methods in the gRPC server.
    • ›Adds pipeline parallelism (PP) support for NIXL in prefill/decode disaggregation.
    • ›Adds prefix cache support for Qwen3 Next and GDN/Mamba models.
    • ›Adds native ModelOpt quantization support.
    • ›[beta] Adds overlap scheduler for speculative decoding.
    • ›[beta] Adds piecewise CUDA graph support for prefill.
    • ›Adds support for passing prefix keys for L3 store in HiCache.
    • ›Adds multi-item scoring with custom attention mask via the Generative Score API.
    • ›Adds MHA prefill support with FlashAttention 4.
    • ›Adds fullset optimizations for DeepSeek-V3.2 including MTP, PD disaggregation, and function calling.
    • ›Adds DGX Spark hardware support.
    • ›Adds KTransformer integration.
    • ›Adds LoRA support in the bench_serving OAI interface.
    • ›Adds float64 support to the moe_sum_reduce CUDA kernel.
    • ›Adds LongBench v2 evaluation utilities.
    • ›Adds various Blackwell kernel optimizations.
    • ›Adds manifest to versioned Docker releases.
    • ›Upgrades FlashInfer to 0.4.0.
    • ›Adds dedicated TOML configuration files for CPU and XPU targets.
    └──▷ BREAKING ON UPGRADE

    • !The max_micro_batch_size argument is renamed to pp_max_micro_batch_size.
30. v0.5.3 Oct 6, 2025 · issue -315

    SGLang v0.5.3 adds DeepSeek-V3.2 sparse attention, FlashAttention 4 prefill kernels, gRPC scheduler, and new model support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.3 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.3

    • ›Adds a standalone gRPC server for the SGLang Python Scheduler, enabling direct gRPC-based scheduling workloads.
    • ›Adds profile args support in the Engine API, exposing profiling controls programmatically.
    • ›Adds polling interval configuration for prefill-decode disaggregation decode stage.
    • ›Adds customer labels specified via request header to metrics, enabling per-request metric segmentation.
    • ›Adds rerank routing logic in the regular router.
    +19 moreshow less

    • ›Adds embedding routing logic in the router.
    • ›Allows one router to support different model families and serving modes simultaneously.
    • ›Adds Basic OAI Response API to the router, along with get and cancel methods for the response API.
    • ›Integrates FlashAttention 4 prefill kernels for accelerated attention computation.
    • ›Supports loading model weights from a remote instance.
    • ›Supports offloading in fp8.
    • ›Supports memory_pool_host page-first direct layout.
    • ›Adds Dynamic Batch Tokenizer for improved prefill throughput.
    • ›Adds Generative Score API prefill-only scoring optimizations.
    • ›Adds fine-grained request latency tracing (SGLang Tracing, Part 1).
    • ›Adds latency histogram metrics for each stage of generate requests in PD disaggregation.
    • ›Adds get_load watching mode and shortest-queue strategy to DP refactor.
    • ›Adds split tile size option for Triton attention.
    • ›Adds RTX 5880 MoE Triton kernel config.
    • ›Supports global scale (in addition to per-expert scale) for CuteDSL MoE.
    • ›Supports Qwen3-Next with MTP, DP, optimized kernels, and multiple hardware platforms including Ascend NPU and Blackwell.
    • ›Supports new models: Qwen3-VL series, dots.vlm1, Ling-V2, Apertus, SOLAR.
    • ›Adds ROCm Docker image support for sgl-router.
    • ›Supports pip install sglang for simplified installation.
31. v0.5.2 Sep 12, 2025 · issue -338

    SGLang v0.5.2 adds HiCache hierarchical KV caching, MCP tool handler, token-bucket rate limiter, and broad new model/parser support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.2 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.2

    └──▷ TRY IT
    Serve Kimi-K2 with its dedicated tool-call parser so function-calling works correctly end-to-end.
    copy

    $ python -m sglang.launch_server --model-path moonshotai/Kimi-K2-Instruct --tool-call-parser kimi-k2

    • ›Adds response_format support for the completion API, enabling structured output control on non-chat endpoints.
    • ›Adds HiCacheStorageConfig generic configuration object for managing HiCache storage backends, with default policy changed to write-through.
    • ›Adds token-bucket rate limiter to the sgl-router.
    • ›Adds MCP Tool Handler to the sgl-router ([router] Add MCP Tool Handler).
    • ›Adds gRPC router bootstrapping support to sgl-router.
    +28 moreshow less

    • ›Adds tool parsers to sgl-router for Mistral, Qwen, Llama, Llama 3.2 (including multi-JSON streaming), Pythonic, DeepSeek, Step3, Kimi-K2, GPT-OSS, and GLM4 models.
    • ›Adds reasoning_effort param to TiktokenTokenizer.apply_chat_template.
    • ›Supports nvidia/Llama-3_1-Nemotron-Ultra-253B-v1 model.
    • ›Supports nvidia/gpt-oss-120b-Eagle3 speculative decoding model.
    • ›Supports DeepSeek-V3.1 tool call parsing.
    • ›Supports speculative decoding in the hybrid attention backend.
    • ›Supports NPUGraph for DeepSeek on Ascend NPU.
    • ›Supports FIA fusion kernel on the Ascend backend.
    • ›Supports CUDA 13.0 and TRT-LLM kernels, including compiling sgl-kernel on CUDA 13.0.
    • ›Enables SM100 FP8 kernels on SM103 (Blackwell+) hardware.
    • ›Enables hierarchical KV caching (HiCache) on AMD GPUs.
    • ›Supports draft_model for RemoteModelLoader in speculative decoding setups.
    • ›Adds bench_mix.py benchmarking script for mixed workloads.
    • ›Adds ROCm 7.0 support.
    • ›Adds A100 fused MoE kernel configs for DeepSeek models.
    • ›Adds tuned fused MoE config for GLM-4.5-Air-FP8 at tp=4 on B200.
    • ›Installs py-spy by default in container images for easier live profiling.
    • ›Improves prefill performance on the CPU backend.
    • ›Reduces GPU memory overhead via weakref-based tensor lifecycle management.
    • ›Supports get_model_info interface for mini_lb in the PD (prefill-decode disaggregation) router.
    • ›Improves disaggregation metrics output in PD mode to reflect real-time stats.
    • ›Supports Mooncake store DP attention and zero-copy MLA metadata.
    • ›Adds batch-send optimization from the Tokenizer Manager for improved throughput.
    • ›Adds target module validation for adapter initialization.
    • ›Accommodates JSON schema in the schema field (not only json_schema) of response_format.
    • ›Supports InternVL3.5 series models via updated processor loading.
    • ›Updates CUTLASS to 4.2 and enables K-Major Scale Factor for SM90 FP8 Blockwise Group GEMM.
    • ›Upgrades FlashInfer to 0.2.14.post1.
    └──▷ BREAKING ON UPGRADE

    • !HiCache default caching policy is changed to write-through; deployments relying on the previous default policy behavior will behave differently after upgrade.
32. v0.5.1 Aug 23, 2025 · issue -358

    SGLang v0.5.1 adds NIXL-backed HiCache, C++ radix tree, NCCL symmetric memory TP allreduces, NVFP4 decode kernels, and ModelScope model download.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.5.1 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.5.1

    └──▷ TRY IT
    Use minimum-token load balancing for data-parallel attention to distribute work more evenly across DP ranks.
    copy

    $ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --schedule-policy lof --dp-size 4 --enable-dp-attention

    • ›Adds lof as a valid choice for the --schedule-policy server argument (was previously missing from the option list).
    • ›Adds support for limiting the maximum number of LoRA adapters loaded in CPU memory.
    • ›Adds NCCL symmetric memory backend for tensor-parallel allreduces, reducing communication overhead on supported hardware.
    • ›Adds SGLang HiCache NIXL Connector, enabling KV-cache offload/transfer via the NIXL transport layer.
    • ›Adds support for downloading models from ModelScope as an alternative to Hugging Face Hub.
    +17 moreshow less

    • ›Adds a C++ implementation of the Radix Tree for prefix caching, replacing the Python implementation.
    • ›Adds minimum-token load balancing strategy for data-parallel attention.
    • ›Adds low-latency NVFP4 decode kernels from Flashinfer for NVIDIA GPUs.
    • ›Adds cutlass_moe_fp8 kernel support for FusedMoE on SM90 (H100/H200) GPUs.
    • ›Enables Flashinfer MoE blockscale FP8 backend for tensor-parallel MoE on NVIDIA hardware.
    • ›Adds new MoE Triton kernel tuned for NVIDIA RTX 6000 Ada.
    • ›Enables renormalize=False in Triton MoE kernels.
    • ›Adds TRTLLM-Gen MLA decode kernel integration for multi-latent attention decoding.
    • ›Adds DP Attention support for the Step3-VL model.
    • ›Extends dsv3_router_gemm kernel to support NUM_EXPERTS == 384 (Kimi K2 architecture).
    • ›Updates dsv3_router_gemm kernel batch size limit to 16.
    • ›Disables Python GC during CUDA graph capture, speeding up capture by up to 3x.
    • ›Saves CUDA graph memory by using next_token_logits_buffer and FA3 graph optimizations.
    • ›Integrates Triton kernels into sgl-kernel package.
    • ›Upgrades xgrammar dependency to 0.1.22.
    • ›Adds HTTP dependency injection pattern to the router system.
    • ›Introduces DP worker abstraction in the router.
    └──▷ BREAKING ON UPGRADE

    • !lora_path is renamed to lora_id in batch requests — existing clients passing lora_path in batches will break.
33. v0.4.10 Jul 31, 2025 · issue -364

    SGLang v0.4.10 adds dynamic LoRA hot-swap, KV metrics export, hybrid KV cache for LLaMA 4, MTP+two-batch-overlap compatibility, and new RL weight-update controls.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.10 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.10

    └──▷ TRY IT
    Skip warmup during RL training runs to reduce startup time when iterating on policy updates.
    copy

    $ python -m sglang.launch_server --model deepseek-ai/DeepSeek-R1 --skip-warmup --port 30000

    Benchmark throughput against an SGLang server using the OpenAI chat completions API endpoint.
    copy

    $ python -m sglang.bench_serving --backend openai-chat --host 127.0.0.1 --port 30000 --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 500

    • ›Adds --model as an alias for --model-path in server args, letting users use the shorter flag name interchangeably.
    • ›Adds --skip-warmup flag for RL workloads to skip the warmup phase on startup.
    • ›Supports dynamic LoRA loading and unloading via the engine and server API at runtime without restarting the server.
    • ›Supports update_weights_from_distributed with different process groups and multiple simultaneous weight updates for RL training workflows.
    • ›Adds KV metrics emission from the SGLang scheduler, enabling external monitoring of cache utilization.
    +19 moreshow less

    • ›Adds OpenAI chat completions API support in the bench_serving benchmarking script.
    • ›Adds Kimi reasoning parser and fixes stream reasoning parser for streaming inference.
    • ›Supports hybrid KV cache for LLaMA 4 models.
    • ›Enables compatibility between Multiple Token Prediction (MTP) and two-batch-overlap scheduling.
    • ›Adds Expert Parallelism Load Balancing (EPLB) support for MTP.
    • ›Supports different Tensor Parallelism sizes for prefill/decode (PD) disaggregation with non-MLA models.
    • ›Adds Tencent HunYuanMoEV1 model support.
    • ›Supports EAGLE3 speculative decoding for LLaMA 4.
    • ›Adds multi-thread model weight loading for faster startup.
    • ›Adds CPU-core and memory-node binding via a new C++ kernel for CPU deployments.
    • ›Adds INT8 and FP8 optimizations for DeepSeek models on CPU, calling fused_experts_cpu, weight_packed_linear, and bmm_cpu kernels.
    • ›Adds wna16marlin kernel for MoE weight-only quantization.
    • ›Enables mixed modality processing through refactored multimodal processors.
    • ›Adds centralized configuration module for sgl-router.
    • ›Enables aiter fused MoE, aiter_biased_grouped_topk, and FP8 blockscale quantization kernels on AMD hardware.
    • ›Adds dsv3_fused_a_gemm and DSv3 router GEMM kernels, applied for DeepSeek-R1 FP4.
    • ›Adds hidden-states return support at async generation time.
    • ›Upgrades FlashInfer to v0.2.7.post1 and sgl-kernel to v0.2.1.
    • ›Adds fbgemm benchmark bandwidth reporting and fbgemm_cutlass_gmm support in the benchmark tooling.
34. v0.4.8 Jun 24, 2025 · issue -365

    SGLang v0.4.8 adds DeepSeek R1 FP4 on Blackwell GPUs, reranking support, logit bias, hidden states API, and VILA model support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.8 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.8

    └──▷ USE IT
    Update an existing import after the OpenAI API module was relocated to sglang/srt/entrypoints/openai.
    python copy

    # Before (v0.4.7 and earlier)
    from sglang.srt.openai_api.protocol import Tool
    
    # After (v0.4.8+)
    from sglang.srt.entrypoints.openai.protocol import Tool

    • ›Moves the OpenAI-compatible API module from sglang/srt/openai_api to sglang/srt/entrypoints/openai; update imports such as from sglang.srt.entrypoints.openai.protocol import Tool.
    • ›Adds support for DeepSeek R1 with FP4 quantization and MTP on NVIDIA Blackwell GPUs, integrating FlashInfer NVFP4 MoE with TP, EP, and DP; achieves up to 90 tokens/sec per user on B200 at isl/osl/bs = 1k/1k/16.
    • ›Adds support for reranking via the OpenAI-compatible server (feat/support rerank).
    • ›Adds logit bias support to sampling parameters.
    • ›Adds hidden states exposure through the OpenAI-compatible API.
    +7 moreshow less

    • ›Adds support for VILA vision-language models.
    • ›Adds support for Phi-4-mm as a supported VLM.
    • ›Adds gfx950 GPU support to sgl-kernel.
    • ›Adds support for the new DeepGEMM input format in per-token group quantization, including silu_and_mul_masked_post_quant_fwd.
    • ›Supports 2-stream shared expert execution for DeepSeek MoE models on Blackwell.
    • ›Adds support for separate reasoning in the frontend language (frontend reasoning API).
    • ›Refactors the OpenAI-compatible server with consistent metrics, unified error handling, and improved request tracking for production and enterprise environments.
    └──▷ BREAKING ON UPGRADE

    • !The sglang/srt/openai_api directory has been removed and replaced with sglang/srt/entrypoints/openai; any import such as from sglang.srt.openai_api.protocol import Tool must be updated to from sglang.srt.entrypoints.openai.protocol import Tool.
35. v0.4.7 Jun 11, 2025 · issue -365

    SGLang v0.4.7 adds pipeline parallelism, PD disaggregation with NIXL backend, Qwen3/Kimi-VL/InternVL3/MiMo model support, CUDA graph for LoRA, and full Blackwell GPU support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.7 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.7

    └──▷ TRY IT
    Enable thinking mode for a Qwen3 or DeepSeek R1 model at request time, useful for reasoning or chain-of-thought workflows.
    copy

    $ curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Solve step by step: 12 * 47"}], "chat_template_kwargs": {"enable_thinking": true}}'

    Cap output length on a Chat Completions request using the OpenAI-compatible max_completion_tokens field, useful for controlling cost and latency in production.
    copy

    $ curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "meta-llama/Llama-4-Scout", "messages": [{"role": "user", "content": "Summarize this document."}], "max_completion_tokens": 512}'

    • ›Adds chat_template_kwargs.enable_thinking to enable thinking mode via the chat template, allowing reasoning/tool-call workflows to be toggled per request.
    • ›Supports max_completion_tokens parameter for OpenAI Chat Completions API, aligning with the OpenAI spec.
    • ›Adds pipeline parallelism support, enabling multi-node inference pipelines across GPU clusters.
    • ›Adds NIXL backend for PD (prefill-decode) disaggregation, now supporting Prefill TP and Decode TP+DP configurations.
    • ›Supports expert parallelism (EP) MoE for Qwen3, extending large-scale EP to the Qwen3 model family.
    +21 moreshow less

    • ›Adds Cutlass MLA attention backend as a new attention backend option for MLA-based models.
    • ›Supports CUDA graph capture for LoRA adapters, enabling faster LoRA inference.
    • ›Adds service discovery for the SGLang router.
    • ›Adds structured logging, file-based log output, and log tracing for the SGLang runtime.
    • ›Supports automatic chat template selection (auto chat template) based on the loaded model.
    • ›Automatically sets draft model path for MTP (multi-token prediction), removing the need to specify it manually.
    • ›Adds support for Kimi-VL multimodal model inference.
    • ›Adds support for InternVL3 multimodal model inference.
    • ›Adds support for XiaomiMiMo/MiMo model inference.
    • ›Enables full Blackwell GPU support (sm_120) for DeepSeek V3/R1, Llama 4, and Qwen3.
    • ›Scales FlashAttention 3 (FA3) kernel to sm8x architectures, broadening hardware support.
    • ›Enables overlap scheduler for multimodal models.
    • ›Adds support for Qwen-1M context via block sparse attention backend kernel updates.
    • ›Adds pythonic tool call support and index field in tool call streaming responses.
    • ›Bumps FlashInfer to 0.2.5.
    • ›Adds VLM benchmark profiling support.
    • ›Adds concurrency evaluation logic in the MMMU benchmark.
    • ›Adds MMMU benchmark support for InternVL.
    • ›Enhances platform compatibility for ARM (including Thor and Spark boards).
    • ›Adds PD fake transfer support for warmup, reducing cold-start latency in disaggregated deployments.
    • ›SGLang DeepSeek V3/R1 reaches 190 tokens-per-second on a single H200, a 50%+ improvement over comparable frameworks.
36. v0.4.6 Apr 27, 2025 · issue -367

    SGLang v0.4.6 adds FlashAttention3 as default backend, PD disaggregation with Mooncake/NIXL, FP4 inference, and preliminary Blackwell GPU support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.6 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.6

    └──▷ TRY IT
    Serve a Llama 4 multimodal model with multimodal support explicitly enabled via the new CLI flag.
    copy

    $ python -m sglang.launch_server --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct --enable-llama4-multimodal

    • ›Adds --enable-llama4-multimodal flag to enable Llama 4 multimodal inference support.
    • ›Enables FlashAttention3 as the default attention backend for mainstream models (DeepSeek, Qwen, Llama), including FA3 MLA by default on Hopper GPUs.
    • ›Adds PD (prefill-decode) disaggregation support with Mooncake and NIXL KV transfer backends.
    • ›Adds FP4 weight loading and inference support (NV FP4 precision).
    • ›Enables DeepGEMM by default for DeepSeek models, plus additional kernel fusions for improved DeepSeek performance.
    +11 moreshow less

    • ›Adds preliminary support for NVIDIA Blackwell GPUs, including a Blackwell Dockerfile and Cutlass MLA kernel.
    • ›Adds sparse attention kernel to sgl-kernel.
    • ›Adds flash_attn_varlen_func to sgl-kernel.
    • ›Automatically inspects whether a model is ModelOpt quantized and sets the quantization method accordingly, including ModelOpt KV cache support.
    • ›Adds in-queue metrics to the metrics reporting surface.
    • ›Adds H20 dtype fp8_w8a8 fused MoE kernel tuning configs for DeepSeek V3/R1.
    • ›Disables grammar restrictions within reasoning sections, allowing structured output to work alongside chain-of-thought reasoning.
    • ›Enables bench_one_batch to support enable_dp_attention.
    • ›Adopts fast image processor by default for VLM (vision-language model) inputs.
    • ›Supports server-based rollout in VerlEngine for RL training workloads.
    • ›Adds Llama 4 FP8 inference support.
37. v0.4.5 Apr 7, 2025 · issue -367

    SGLang v0.4.5 adds Llama 4, FlashAttention 3, EAGLE3 speculative decoding, DeepEP MoE integration, and disaggregated prefill/decode.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.5 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.5

    • ›Adds support for Llama 4 models (Llama-4-Scout-17B-16E-Instruct and Llama-4-Maverick-17B-128E-Instruct), achieving zero-shot MMLU Pro scores of 75.2 and 80.7 respectively.
    • ›Adds FlashAttention 3 backend for significant acceleration on long-context inference tasks.
    • ›Adds EAGLE3 speculative decoding support, delivering substantial gains in decoding throughput.
    • ›Integrates DeepEP for enhanced performance in MoE (Mixture-of-Experts) inference.
    • ›Introduces a prototype for disaggregated prefill and decoding.
    +13 moreshow less

    • ›Adds support for Gemma 3 IT (gemma-3-it) model.
    • ›Adds support for Deepseek-VL2 multimodal model.
    • ›Adds support for QwenMoe model.
    • ›Adds FlashMLA backend support.
    • ›Adds online quantization support for W8A8 (weight 8-bit, activation 8-bit).
    • ›Adds support for serving DeepSeek-R1-Channel-INT8 on 32 L40S GPUs.
    • ›Adds tensor parallelism and weight slicing support for LoRA adapters.
    • ›Adds code completion serving support.
    • ›Sets xgrammar as the default grammar backend.
    • ›Adds hierarchical caching (HiCache) support for MLA (Multi-head Latent Attention).
    • ›Adds tool call with text support.
    • ›Supports variable remote backend for model loader.
    • ›Auto-detects device if not specified in server arguments.
38. v0.4.4 Mar 13, 2025 · issue -368

    SGLang v0.4.4 adds FlashInfer MLA with radix cache, DeepGEMM via SGL_ENABLE_JIT_DEEPGEMM, INT8 DeepSeek R1, hierarchical caching, and SageMaker support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.4 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.4

    └──▷ TRY IT
    Enable FlashInfer MLA with radix cache and chunked prefill for faster DeepSeek V3/R1 prefill on NVIDIA hardware.
    copy

    $ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1 --enable-flashinfer-mla

    Activate DeepGEMM JIT compilation for NVIDIA Hopper (H100/H200) to maximize DeepSeek inference throughput.
    copy

    $ export SGL_ENABLE_JIT_DEEPGEMM=1
    python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1 --enable-flashinfer-mla

    • ›Enables FlashInfer MLA support — fully compatible with radix cache, chunked prefill, and MTP — via --enable-flashinfer-mla.
    • ›Integrates DeepGEMM for NVIDIA Hopper architectures, enabled by setting export SGL_ENABLE_JIT_DEEPGEMM=1.
    • ›Adds Multi-Token Prediction (MTP) speculative decoding for DeepSeek-V3/R1 with both Triton and FlashInfer backends, compatible with radix cache and chunked prefill.
    • ›Adds hierarchical caching support for SGLang.
    • ›Adds SageMaker support.
    +11 moreshow less

    • ›Adds support for OpenAI API o1 model.
    • ›Adds support for Qwen 2.5 VL (vision-language) model.
    • ›Adds support for Qwen reward model (Qwen RM).
    • ›Adds INT8 quantization support for DeepSeek R1 models (meituan/DeepSeek-R1-Channel-INT8 and meituan/DeepSeek-R1-Block-INT8).
    • ›Adds Blackwell architecture Block Scale FP8 GEMM support.
    • ›Adds support for page size greater than 1.
    • ›Adds nvidia modelopt FP8 KV cache support.
    • ›Adds support for RDMA in Docker deployments.
    • ›Adds multi-node inference deployment via the LWS method in Kubernetes clusters.
    • ›Adds enhanced distributed parallelism support (e.g., two-node configurations with DP 2, TP 16).
    • ›Delivers optimized W8A8 FP8 implementation with 15%+ performance improvement on sm89, with gains across sm80, sm89, and sm90 architectures.
39. v0.4.3 Feb 14, 2025 · issue -369

    SGLang v0.4.3 adds Function Calling, regex/EBNF-constrained decoding, custom sampling processors, LoRA in Triton, and 4x long-context speedup via FlashInfer MLA.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.3 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.3

    • ›Introduces Function Calling capabilities for LLM inference.
    • ›Adds regex pattern and EBNF support in the XGrammar backend for constrained decoding.
    • ›Adds update_weights_from_tensor API for updating model weights at runtime.
    • ›Adds Engine.generate() support for returning token IDs alongside generated text.
    • ›Enables Flash Attention 3 (FA3) by default for prefill (requires CUDA 12.4, now the default).
    +9 moreshow less

    • ›Integrates FlashInfer MLA Attention, delivering a 4x performance improvement for long-context DeepSeek V3/R1 inference.
    • ›Adds torch.compile support for FP8, achieving ~50 tokens/s for online inference with DeepSeek V3/R1.
    • ›Implements CUTLASS block-wise FP8 kernels for enhanced inference efficiency.
    • ›Implements custom sampling processor support for flexible inference control.
    • ›Integrates LoRA support in the Triton backend.
    • ›Extends EAGLE 2 speculative decoding support to the FlashInfer and Triton backends.
    • ›Supports loading pre-sharded MoE weights.
    • ›Enables DeepSeek V3 inference on AMD GPUs.
    • ›Upgrades to FlashInfer v0.2.
40. v0.4.1 Dec 25, 2024 · issue -371

    SGLang v0.4.1 adds DeepSeek V3 support, a dynamic-scaling fault-tolerant router with /add_worker and remove-worker APIs, MoE Expert Parallel, gemlite quantization, and the new sgl-kernel package.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.1 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.1

    └──▷ TRY IT
    Dynamically add a new worker to a running SGLang router without restarting the cluster.
    copy

    $ curl -X POST http://<router-host>:<port>/add_worker -H 'Content-Type: application/json' -d '{"worker_url": "http://<new-worker-host>:<worker-port>"}'

    • ›Adds /add_worker API to the SGLang native router, enabling dynamic worker registration at runtime.
    • ›Adds remove-worker API to the SGLang native router, enabling dynamic worker deregistration and cleanup.
    • ›Adds lora_path field to chat completion requests and lora_paths to v1_chat_generate_request, enabling per-request LoRA adapter selection.
    • ›Introduces sgl-kernel as a standalone PyPI package for custom CUDA/HIP kernels, including a TensorRT-LLM-derived custom all-reduce implementation.
    • ›Adds support for DeepSeek V3 (FP8) on both NVIDIA and AMD GPUs, including MLA optimization and DP attention.
    +13 moreshow less

    • ›Adds MoE Expert Parallel support.
    • ›Adds gemlite weight-only quantization integration.
    • ›Adds chunked prefill support for LLaVA-OneVision multimodal models.
    • ›Adds support for IBM Granite 3.x models.
    • ›Adds in-batch prefix caching via delay scheduling.
    • ›Adds retry-based fault tolerance to the cache-aware SGLang router.
    • ›Adds health-check on workers before they are added to the router.
    • ›Adds expanded support for Intel Gaudi accelerators.
    • ›Adds ROCm support for sglang.check_env.
    • ›Adds FP8 MoE optimizations for AMD GPUs.
    • ›Optimizes the Triton decoding kernel for long-context inference.
    • ›Makes the maximum request payload size configurable.
    • ›Makes Tensor Parallelism composable with both torch.compile and torchao.
41. v0.4.0 Dec 4, 2024 · issue -371

    SGLang v0.4 adds cache-aware load balancing, xgrammar structured outputs, GGUF support, LoRA in Completion API, and Intel Gaudi accelerator support.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.4.0 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.4.0

    └──▷ TRY IT
    Enable cache-aware load balancing across multiple SGLang workers to maximize KV-cache reuse and throughput.
    copy

    $ python -m sglang_router.launch_router --policy cache_aware --worker-urls http://worker1:30000 http://worker2:30000

    Set CPU core affinity for SGLang workers on a multi-socket host to reduce NUMA overhead.
    copy

    $ SGLANG_SET_CPU_AFFINITY=1 python -m sglang.launch_server --model-path meta-llama/Llama-3-8B-Instruct --tp 2

    • ›Adds SGLANG_SET_CPU_AFFINITY environment variable to control CPU core affinity for multi-socket systems (off by default).
    • ›Renames data-parallelism rank environment variable from DP_RANK to SGLANG_DP_RANK.
    • ›Adds get_server_info unified API endpoint, merging three previously separate native APIs into one (also exposes SGLang version and max_total_num_tokens).
    • ›Introduces cache-aware load-balancing router (sglang-router v0.0.10) delivering up to 1.9x throughput increase and 3.8x higher cache hit rate.
    • ›Adds more API routes to the router (completion, health, and others previously missing).
    +15 moreshow less

    • ›Supports LoRA in the Completion API.
    • ›Supports GGUF model format loading.
    • ›Supports input_embeds as input to the engine API.
    • ›Adds initial support for Intel Gaudi accelerators.
    • ›Integrates xgrammar for structured outputs, delivering up to 10x faster constrained decoding.
    • ›Adds the OLMo2 model.
    • ›Adds CPU offloading support.
    • ›Adds zero-overhead batch scheduler yielding a 1.1x throughput increase.
    • ›Adds data parallelism attention for DeepSeek models with up to 1.9x decoding throughput improvement.
    • ›Adds prefix match routing for specific tenants in the router.
    • ›Adds profiling support in the offline benchmark tool.
    • ›Adds fused_moe_triton benchmark and tuning tools for AMD Instinct GPUs.
    • ›Adds ability to update weights from disk and retrieve weights by parameter name (Llama).
    • ›Adds a development Dockerfile.
    • ›Exposes max_total_num_tokens from the Runtime and Engine API.
    └──▷ BREAKING ON UPGRADE

    • !The environment variable DP_RANK is renamed to SGLANG_DP_RANK; any deployment setting DP_RANK must update to SGLANG_DP_RANK.
42. v0.3.6 Nov 22, 2024 · issue -372

    SGLang v0.3.6 adds cache-aware load balancing, xgrammar backend, Prometheus metrics, graceful termination, and a Rust-based router.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.6 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.3.6

    └──▷ TRY IT
    Scrape Prometheus metrics from a running SGLang server for real-time throughput and latency monitoring.
    copy

    $ curl http://localhost:30000/metrics

    • ›Adds a cache-aware load balancer delivering 4x higher cache hit rate across multi-server deployments.
    • ›Adds xgrammar backend for grammar-guided (constrained) decoding, joinable with the existing outlines backend.
    • ›Adds Prometheus metrics support for observability and monitoring.
    • ›Adds graceful termination: server drains in-flight requests before exit on SIGTERM.
    • ›Adds a watchdog thread to detect and surface server hangs.
    +11 moreshow less

    • ›Adds a Rust-based router with radix-tree cache-aware load balancing.
    • ›Supports data parallelism for attention and MLA, yielding 1.5x higher decoding throughput.
    • ›Enables the overlap scheduler by default, delivering 1.1x higher throughput with reduced CPU overhead.
    • ›Adds torch-native tensor parallelism support.
    • ›Supports PyTorch 2.5.1.
    • ›Adds support for new models: Phi3-small, Gemma-2 reward model, and GPT-2.
    • ›Adds notebook-style (.ipynb) documentation at https://sgl-project.github.io/.
    • ›Adds an offline benchmark script.
    • ›Adds get memory pool size API endpoint.
    • ›Allows token IDs to be passed directly in engine.generate.
    • ›Makes decode log interval configurable.
43. v0.3.4.post1 Oct 22, 2024 · issue -373

    SGLang v0.3.4.post1 adds an offline Engine API, overlap scheduler, async/streaming engine support, and new model/hardware coverage.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.4.post1 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.3.4.post1

    └──▷ TRY IT
    Tune continuous decode batch size from the CLI to reduce CPU overhead on high-throughput serving deployments.
    copy

    $ python -m sglang.launch_server --model-path meta-llama/Llama-3.2-8B --num-continuous-decode-steps 4

    • ›Exposes --num-continuous-decode-steps as a CLI argument to control continuous decode step count.
    • ›Adds an offline Engine API for inference without an HTTP server, with reduced overhead and support for async and streaming generation.
    • ›Adds get_tokenizer function to the Engine class for direct tokenizer access in offline workflows.
    • ›Adds HTTP API endpoints to enable and disable the PyTorch profiler at runtime.
    • ›Adds an overlap scheduler to hide CPU overhead during inference.
    +8 moreshow less

    • ›Supports min_tokens parameter in sgl.gen for minimum output length control.
    • ›Adds support for reward models.
    • ›Adds support for Intel XPU GPUs via PyTorch.
    • ›Adds support for new models: Llama 3.2, QWen-VL2, OLMo, and GLM 4.
    • ›Accelerates multi-LoRA serving throughput.
    • ›Adds float8 dynamic quantization support via torchao_utils.
    • ›Adds support for double sparsity attention.
    • ›Adds MoE tuning optimizations for AMD MI300x GPUs.
44. v0.3.2 Oct 2, 2024 · issue -373

    SGLang v0.3.2 adds multi-LoRA serving, AMD/ROCm support, torchao quantization, and a new --attention-backend flag.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.2 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.3.2

    └──▷ TRY IT
    Switch to the Triton attention backend (with torch.compile and CUDA graph) instead of the default FlashInfer backend.
    copy

    $ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B --attention-backend triton

    • ›Introduces --attention-backend CLI flag to select the attention backend, replacing the deprecated --disable-flashinfer.
    • ›Adds constrained_json_whitespace_pattern to ServerArgs for fine-grained control over JSON whitespace in constrained decoding.
    • ›Passes model_override_args to launch_server via the CLI, enabling runtime model config overrides without code changes.
    • ›Supports OpenAI API json_schema response format for structured output.
    • ›Supports response prefill via the OpenAI-compatible API.
    +7 moreshow less

    • ›Initial support for multi-LoRA serving, including LoRA path renaming.
    • ›Integrates torchao quantization (int4, int8, fp8) for Llama, Mixtral, and Qwen MoE models.
    • ›Enables torch.compile and CUDA graph support for the Triton attention backend, including DeepSeek MLA.
    • ›Enables Multi-head Latent Attention (MLA) by default for DeepSeek-v2.
    • ›Adds AMD GPU support via PyTorch for ROCm backend.
    • ›Adds new model support: MiniCPM3, OLMoE, BaiChuan2, XVERSE (Dense and MoE), and SmolLM.
    • ›Adds bench_server_latency.py for server latency benchmarking.
    └──▷ BREAKING ON UPGRADE

    • !The --disable-flashinfer flag is deprecated and replaced by --attention-backend; existing scripts using --disable-flashinfer will need to be updated.
45. v0.3.0 Sep 19, 2024 · issue -374

    SGLang v0.3.0 adds MLA throughput gains, torch.compile, fp8 KV cache, min-p sampling, JSON constraints, and live weight updates.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.3.0 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.3.0

    └──▷ TRY IT
    Verify the inference server is live and capable of generating tokens (not just process-up).
    copy

    $ curl http://localhost:30000/v1/health

    • ›Adds --disable-custom-all-reduce flag to opt out of custom all-reduce kernels.
    • ›Adds fp8 e5m2 KV cache support via both FlashInfer and Triton backends for reduced memory footprint during inference.
    • ›Adds min-p sampling parameter for nucleus-style sampling with a minimum probability floor.
    • ›Enables JSON-constrained generation (structured output) usable directly from the gen call via json_schema.
    • ›Adds /v1/health endpoint that validates the server by generating a single token.
    +13 moreshow less

    • ›Supports live weight updates without restarting the server via the new update-weights API.
    • ›Supports batch cancellation requests.
    • ›Supports optional start length for logprobs via new logprobs sampling parameter.
    • ›Enables streaming for multi-prompt and parallel sampling requests.
    • ›Chunked prefill is now on by default, with support for separate or mixed prefill-and-decode modes.
    • ›Adds LLaVA-OneVision model support, including SigLIP encoder, Qwen2 decoder, interleaved text and multi-image/video input, and OpenAI-compatible API server.
    • ›Adds EXAONE 3.0 model support.
    • ›Adds Phi-3 Mini and Medium model support.
    • ›Adds Alibaba-NLP/gte-Qwen2-7B-instruct embedding model support.
    • ›Delivers up to 7x higher throughput for DeepSeek Multi-head Latent Attention (MLA) via optimized Triton decoding kernels and offline weight transposition.
    • ›Delivers up to 1.5x lower latency with torch.compile on small batch sizes.
    • ›Extends Gemma-2 context length 2x via interleaved window attention support.
    • ›Adds SM75 (Turing) GPU support via FlashInfer v0.1.6.
    └──▷ BREAKING ON UPGRADE

    • !model_overide_args is renamed to model_override_args; any config or code referencing the old spelling will break.
46. v0.2.13 Sep 19, 2024 · issue -374

    SGLang v0.2.13 adds window attention for Gemma-2, embedding model support, all sampling penalties, chunked-prefill by default, and 2x MLA throughput for DeepSeek v2.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.13 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.2.13

    └──▷ TRY IT
    Generate embeddings for a list of strings using the new OpenAI-compatible embeddings endpoint backed by e5-mistral.
    copy

    $ curl http://localhost:30000/v1/embeddings \
      -H 'Content-Type: application/json' \
      -d '{"model": "intfloat/e5-mistral-7b-instruct", "input": ["query: what is SGLang?", "query: how does MLA work?"]}'

    Apply repetition and frequency penalties during generation to reduce token repetition in long outputs.
    copy

    $ curl http://localhost:30000/v1/completions \
      -H 'Content-Type: application/json' \
      -d '{"model": "meta-llama/Meta-Llama-3-8B", "prompt": "Explain quantum entanglement:", "repetition_penalty": 1.1, "frequency_penalty": 0.5, "presence_penalty": 0.3, "min_new_tokens": 50}'

    • ›Adds skip_tokenizer_init argument to bypass tokenizer initialization for advanced serving setups.
    • ›Adds sampling penalties — frequency_penalty, presence_penalty, repetition_penalty, and min_new_tokens — completing OpenAI-compatible penalty support.
    • ›Enables chunked-prefill by default, improving throughput without requiring manual configuration.
    • ›Supports the e5-mistral embedding model with a fully OpenAI-compatible embeddings API, including accepting input as a list.
    • ›Adds window attention support for Gemma-2 models.
    +4 moreshow less

    • ›Accelerates Multi-head Latent Attention (MLA) for DeepSeek v2 via a Triton kernel, delivering a ~2x end-to-end throughput improvement.
    • ›Supports loading models from www.modelscope.cn in addition to HuggingFace.
    • ›Adds alternative choices selection methods for constrained generation.
    • ›Uses FlashInfer kernels for rmsnorm and silu operations, reducing per-token overhead.
47. v0.2.9 Aug 2, 2024 · issue -375

    SGLang v0.2.9 adds chunked prefill, DeepSeek v2 support, AWQ Marlin quantization, and logprobs in the OpenAI chat API.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.9 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.2.9

    └──▷ TRY IT
    Cap the total number of tokens the server will hold in flight to avoid OOM on large batch workloads.
    copy

    $ python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --max-total-tokens 131072

    Expose a custom model name when serving a local checkpoint so OpenAI-compatible clients see a stable identifier.
    copy

    $ python -m sglang.launch_server --model-path /weights/my-llama --served-model-name my-llama-v1

    • ›Adds --max-total-tokens CLI argument to control the total token budget for the server.
    • ›Adds --disable_ignore_eos support in bench_serving.py for benchmarking scenarios that require EOS suppression.
    • ›Adds served_model_name to customize the model ID when running in local mode via the OpenAI-compatible API server.
    • ›Renames response fields: prefill_token_logprobs is now input_token_logprobs; decode_token_logprobs is now output_token_logprobs.
    • ›Adds chunked prefill support, including operation when radix cache is disabled.
    +6 moreshow less

    • ›Adds support for logprobs in the OpenAI chat completion API.
    • ›Adds support for offline batch (file) processing via the OpenAI API interface.
    • ›Adds AWQ Marlin quantization support (awq_marlin).
    • ›Adds DeepSeek v2 model support.
    • ›Adds chat template for InternLM2-Chat.
    • ›Vectorizes logprob computation for a performance improvement in log-probability workloads.
    └──▷ BREAKING ON UPGRADE

    • !prefill_token_logprobs is renamed to input_token_logprobs and decode_token_logprobs is renamed to output_token_logprobs — any client parsing these field names will break on upgrade.
48. v0.2.0 Jul 25, 2024 · issue -376

    SGLang v0.2.0 adds Llama3 405B, Deepseek MoE, Mistral-Nemo, GPTBigCode, and InternLM2 support with OpenAI parallel sampling and VertexAI safety settings.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.2.0 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.2.0

    • ›Adds support for VertexAI safety settings in API calls.
    • ›Adds OpenAI API parallel sampling support.
    • ›Adds new model support for Llama3 405B (including FP8), Deepseek MoE, InternLM2, GPTBigCode, and Mistral-Nemo.
    • ›Adds incremental detokenization when streaming, reducing time-to-first-token (TTFT) and inter-token latency (ITL) at stream_interval 1.
    • ›Adds benchmark serving tooling (bench_serving.py) supporting random datasets, non-streaming mode, and TensorRT-LLM comparison benchmarks.
    +1 moreshow less

    • ›Delivers superior or competitive throughput versus TensorRT-LLM and vLLM across Llama-8B to Llama-405B on A100 and H100 GPUs in both FP8 and FP16.
49. v0.1.20 Jul 14, 2024 · issue -376

    SGLang v0.1.20 enables CUDA graphs by default, adds Gemma2/MiniCPM/Qwen2 MoE support, and ships Docker images.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.20 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.20

    └──▷ TRY IT
    Run the server with P2P connectivity checking enabled to validate GPU-to-GPU links at startup.
    copy

    $ python -m sglang.launch_server --model-path <model> --enable-p2p-check

    • ›Adds --enable-p2p-check option to the CLI for peer-to-peer connectivity validation.
    • ›Exposes logprob options to the sgl.gen API via new LogitsMetadata support.
    • ›Enables CUDA graph execution by default, delivering 1.5x–2x speedup for small-batch decoding.
    • ›Adds model support for Gemma2, MiniCPM, and Qwen2 MoE.
    • ›Adds Docker support with a new Dockerfile.
50. v0.1.18 Jul 4, 2024 · issue -376

    SGLang v0.1.18 adds multi-node tensor parallelism, 2x large-batch prefill throughput, ChatGLM support, and new CLI arguments.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.18 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.18

    └──▷ TRY IT
    Suppress verbose per-request HTTP logs in production while keeping model-level logging intact.
    copy

    $ python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --log_level_http warning

    Run an offline latency benchmark against a model without standing up a full server.
    copy

    $ python -m sglang.bench_latency --model-path meta-llama/Llama-2-7b-chat-hf

    • ›Adds --log_level_http argument to control HTTP request logging independently of the main log level.
    • ›Adds --dtype argument to expose model dtype selection at server launch.
    • ›Adds sglang.bench_latency module for offline latency benchmarking.
    • ›Enables flashinfer kernels by default, delivering a 2x throughput improvement for large-batch prefill workloads.
    • ›Supports multi-node tensor parallelism, enabling model serving across multiple machines.
    +1 moreshow less

    • ›Adds ChatGLM model support.
51. v0.1.17 Jun 8, 2024 · issue -377

    SGLang v0.1.17 adds data parallelism, speculative execution for OpenAI API, new model support, and a LiteLLM backend.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.17 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.17

    └──▷ TRY IT
    Enable speculative execution on an OpenAI-compatible chat request to speed up inference with a draft model.
    copy

    $ curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "meta-llama/Llama-2-7b-chat-hf", "messages": [{"role": "user", "content": "Hello"}], "num_api_spec_tokens": 5}'

    • ›Adds num_api_spec_tokens parameter enabling speculative execution for the OpenAI chat API.
    • ›Adds PUT method support to the generate API endpoint.
    • ›Adds finish_reason field to OpenAI API responses.
    • ›Supports static data parallelism for serving.
    • ›Adds LiteLLM backend integration.
    +3 moreshow less

    • ›Ports FP8 Mixtral quantization support via vLLM v0.4.3 upgrade, enabling new quantization features.
    • ›Adds LLaVA-Qwen and LLaVA-Mistral model support.
    • ›Adds logit cap to improve generation control.
    └──▷ BREAKING ON UPGRADE

    • !The parameter api_num_spec_tokens is renamed to num_api_spec_tokens.
52. v0.1.16 May 14, 2024 · issue -378

    SGLang v0.1.16 adds DBRX, Command-R, Gemma, and LLaVA-Video support alongside Marlin quantization kernels and cache optimizations.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.16 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.16

    └──▷ TRY IT
    Send pre-tokenized input directly to the inference server when you already have token IDs, bypassing text encoding.
    copy

    $ curl -X POST http://localhost:30000/generate -H 'Content-Type: application/json' -d '{"input_ids": [1, 2308, 338, 278, 7483, 310, 3444, 29973], "max_new_tokens": 64}'

    • ›Adds spaces_between_special_tokens argument to SamplingParams for finer control over token output formatting.
    • ›Allows input_ids to be passed directly in the body of the /generate endpoint, enabling token-level input without text encoding.
    • ›Includes finish_reason in the meta info response returned by the inference API.
    • ›Adds support for DBRX, Command-R, and Gemma model architectures.
    • ›Adds support for LLaVA-Video multimodal inference.
    +5 moreshow less

    • ›Enables Marlin quantization kernels for faster quantized model inference.
    • ›Adds Llama 3 instruct chat template.
    • ›Adds Cohere Command-R chat template.
    • ›Optimizes radix tree prefix matching for improved cache hit performance.
    • ›Reduces memory usage of the logits processor.
53. v0.1.13 Mar 11, 2024 · issue -380

    SGLang v0.1.13 adds Gemma model support, API key auth, SSL, and Together/AzureOpenAI backend examples.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.13 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.13

    • ›Adds API key authentication support to the SGLang server via Add Support for API Key Authentication.
    • ›Adds SSL certificate functionality to the server for encrypted transport.
    • ›Adds set_var method to interpreter.py for programmatic variable assignment in the runtime interpreter.
    • ›Adds completion_tokens_wo_jump_forward field to decoding logprob output, exposing token counts that exclude jump-forward tokens.
    • ›Adds ability to modify the context length at server launch via server_args.
    +3 moreshow less

    • ›Replaces skip_embed with input_embeds in the API surface for passing raw embeddings.
    • ›Adds Gemma model support.
    • ›Adds Together and AzureOpenAI backend usage examples.
    └──▷ BREAKING ON UPGRADE

    • !skip_embed is replaced by input_embeds — any code passing skip_embed will break.
54. v0.1.12 Feb 11, 2024 · issue -381

    SGLang v0.1.12 adds fast JSON decoding, decode token logprobs, GPTQ support, and a --disable-disk-cache flag

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.12 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.12

    └──▷ TRY IT
    Prevent the server from writing cache to disk — useful in ephemeral or read-only container environments.
    copy

    $ python -m sglang.launch_server --model-path <model> --disable-disk-cache

    • ›Adds --disable-disk-cache flag to the SRT server to explicitly turn off on-disk KV cache persistence.
    • ›Adds auth token support to RuntimeEndpoint for authenticated inference endpoints.
    • ›Supports decode token logprobs, enabling callers to retrieve per-token log-probabilities from decoding.
    • ›Adds GPTQ quantized model support, allowing inference on GPTQ-quantized weights.
    • ›Supports regex extra field in the OpenAI-compatible API for constrained decoding via regular expressions.
    +3 moreshow less

    • ›Fast JSON decoding (compressed FSM / jump-forward) now supported for LLaVA models in addition to base LLMs.
    • ›Adds server warmup on SRT server startup to reduce cold-start latency on first requests.
    • ›Adds OpenAI error handler with automatic retry and logging for more resilient API calls.
55. v0.1.11 Feb 3, 2024 · issue -381

    SGLang v0.1.11 adds LLaVA 1.6, Yi-VL, QWen2 support, faster JSON decoding, flush cache API, and a health endpoint.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.11 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.11

    • ›Adds max_prefill_num_token server argument to control prefill batch size limits.
    • ›Adds a flush cache API endpoint to the SGLang runtime server.
    • ›Adds a health endpoint to the SGLang runtime server.
    • ›Supports specifying all ports in advance via server arguments.
    • ›Supports LLaVA v1.6 (including HD variant) and fine-tuned LLaVA model loading.
    +9 moreshow less

    • ›Supports Yi-VL multimodal model.
    • ›Supports QWen2 model.
    • ›Adds image content support in OpenAI-compatible chat routes.
    • ›Returns logprob for choices in API responses.
    • ›Adds cache metrics reporting.
    • ›Supports speculative execution for the OpenAI API backend.
    • ›Faster JSON decoding via compressed finite-state machine approach.
    • ›Dynamic model class loading for extensibility.
    • ›Improves streaming control and reduces first-token latency in streaming mode.
56. v0.1.6 Jan 21, 2024 · issue -382

    SGLang v0.1.6 adds an OpenAI-compatible API server with streaming Completions and ChatCompletions endpoints.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.6 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.6

    └──▷ TRY IT
    Stream completions from a running SGLang server using the new OpenAI-compatible endpoint.
    copy

    $ curl http://localhost:30000/v1/completions -H 'Content-Type: application/json' -d '{"model": "default", "prompt": "The capital of France is", "stream": true}'

    Send a chat-style request to the new ChatCompletion endpoint for OpenAI-compatible clients.
    copy

    $ curl http://localhost:30000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "default", "messages": [{"role": "user", "content": "Hello!"}]}'

    • ›Adds v1/chat/completions endpoint for OpenAI-compatible ChatCompletion API support.
    • ›Adds stream=True support to the v1/completions endpoint for streaming token output.
    • ›Adds disk-based FSM cache to persist finite-state-machine state across runs.
    • ›Increases interpreter parallelism for higher throughput during multi-call workloads.
    • ›Adds a LLaVA multimodal example and a Vicuna chat template.
57. v0.1.5 Jan 18, 2024 · issue -382

    SGLang v0.1.5 adds a VertexAI backend, metadata in async streaming, and expanded scheduling controls.

    └──▷ GET THIS VERSION
    copy

    $ git clone --branch v0.1.5 https://github.com/sgl-project/sglang.git
    # already have the repo? check out this version:
    $ git checkout v0.1.5

    • ›Adds a VertexAI (formerly Gemini) backend, enabling SGLang programs to target Google's VertexAI models.
    • ›Adds an option to return metadata in async streaming responses.
    • ›Exposes additional arguments to control the scheduling policy.
    • ›Adds memory fraction tuning to give operators finer control over GPU memory allocation.
    └──▷ BREAKING ON UPGRADE

    • !The image_url field is renamed to image_file; any existing code or configs that reference image_url will break.