https://aitoolchain.io/tools/sglang New releases and features in SGLang, tracked by The AI Toolchain. en Sat, 22 Aug 2026 00:09:15 GMT https://github.com/sgl-project/sglang/releases/tag/v0.5.18 https://github.com/sgl-project/sglang/releases/tag/v0.5.18 Sat, 22 Aug 2026 00:09:15 GMT SGLang v0.5.18 adds overlapped checkpoint staging, FlashInfer MNNVL allreduce, unified cache dir, 7 new models, and extensible serve backend plugins. • Adds `--startup-weight-load-mode overlap` flag to overlap checkpoint page staging with CUDA graph capture at startup, cutting Qwen3-32B on H100 startup time from 84.8s to 35.6s (2.38x faster). • Adds `--enable-flashinfer-pure-allreduce` flag to route non-fused allreduce sites through the FlashInfer MNNVL workspace instead of NCCL; auto-enabled for DeepSeek-V3/V3.2/V4, delivering up to +6.9% decode throughput at small batches on Blackwell. • Adds `--dit-layerwise-residency-policy` flag for strided DiT residency control in diffusion model serving. • Consolidates Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches under a single `SGLANG_CACHE_DIR` environment variable. • Adds extensible serve backend plugins via `feat(cli): add extensible serve backend plugins`. • Enables delayed sampling for general overlap decode via `SGLANG_ENABLE_DELAY_SAMPLE` environment variable, allowing sampling overhead to overlap with model execution. • Adds native multimodal processing for Qwen VL in the Rust server via integrated sglang-mm. • Adds support for new models: Muse Glimmer (autoregressive multimodal), Intern-S2-Mobius (autoregressive), SANA-Video (diffusion), LingBot-Video-MoE (diffusion), LTX-2.5 (diffusion), Cosmos3 Edge & Distilled (diffusion), and LongCat-Image (diffusion). • Supports logprobs output with DSpark and DFlash speculative decoding. • Supports MegaMoE for DSpark under DP attention. • Adds FlashInfer CuTe DSL NVFP4 MoE quantization support. • Adds FlashInfer mHC fusion for DeepSeek-V4, now enabled by default. • Adds SM12x FA4 architecture-owned kernels. • Adds a Triton MoE TMA up kernel. • Adds explicit EPLB balancedness reporting modes. • Adds a content-addressed JIT build cache generated from a custom ninja build system. • Adds tuned Triton tile configs for channelwise FP8 GEMM. • Replaces allgather + scatter in TP LMHead with a single all-to-all for pure-DP dp-attention, reducing LMHead time from 320us to 169us on DeepSeek-V4-Pro B200. • Updates core dependencies: torch 2.13.0 with triton 3.7.1, flashinfer 0.6.17, CuTeDSL 4.6.2, DeepEP from `sgl-deep-ep` wheels, and sgl-kernel 0.4.6.post1. Breaking changes: • All kernel caches (Triton, FlashInfer, Inductor, DeepGEMM, CUDA driver) now live under `SGLANG_CACHE_DIR`; the first launch after upgrading will recompile all kernels once. • The `--torchao-config` flag and its torchao integration have been removed. https://github.com/sgl-project/sglang/releases/tag/v0.5.17 https://github.com/sgl-project/sglang/releases/tag/v0.5.17 Sat, 08 Aug 2026 00:19:16 GMT SGLang v0.5.17 adds Kimi K3 and MiniMax-H3 day-0 support, a Rust frontend, new DCP backends, DWDP prefill, and session-aware radix caching. • Adds `--dcp-comm-backend {ag_rs, a2a, fi_a2a}` to select pluggable communication backends for the DeepSeek-MLA decode context-parallel path; `a2a` exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer with fp8 KV as uint8 byte transport, and `fi_a2a` delegates cross-rank exchange to the FlashInfer MNNVL kernel on GB200. • Adds `--dcp-replicate-q-proj` flag to project full-head Q locally and skip the per-layer Q head-dim all-gather in DCP decode. • Adds `--dwdp-size` to enable DWDP (Download-While-Dispatch-Prefill) parallelism for MoE prefill, which prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch; achieves 1.92x over DEP4 at MNT 32K / ISL 32K on 4x B200 with gpt-oss-120b. • Adds `--enable-session-radix-cache` flag and `/close_session` endpoint for session-reference-aware Unified Radix Cache, allowing requests to carry a stable `session_id` so eviction respects prefixes still referenced by active sessions — targeted at agentic and RL-rollout workloads. • Adds `SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1` environment variable to enable the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation. • Adds `SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D` environment variable to copy oversized or non-contiguous CPU weight views into contiguous storage before H2D transfer, cutting DeepSeek-V4-Pro TP8 load time from ~35 minutes to 6m20s (5.6x) and GPT-OSS-20B BF16 from 545s to 70s (7.8x); off by default. • Adds day-0 support for Kimi K3, a 2.8T-parameter multimodal LatentMoE with 1M-token context, served with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on quantized weights, and OpenAI-compatible serving; verified on NVIDIA GB300 and AMD MI35x. • Adds day-0 support for MiniMax-H3 video-and-audio generation on SGLang-Diffusion across task profiles `t2va` (text-to-video-and-audio), `fl2va` (first/last-frame conditioning), and `ref2va` (image/video/audio reference conditioning); verified on B200 (TP2 + Ulysses4), H100, AMD MI300X/MI355X, and 2x RTX 5090 with layerwise offload. • Adds an initial Rust frontend that replaces the Python server path from network ingress through tokenization and request hand-off to the GPU scheduler with a multi-threaded Rust implementation, including tokenizer manager, ingress validation, egress, OpenAI-compatible API server, and PD disaggregation support; shipped as prebuilt release artifacts. • Adds a weight-cache daemon that holds GPU weights so a restarting engine can recover from cache rather than reloading from storage and recapturing CUDA graphs, reducing restart time for large models (e.g., Qwen3-235B FP8 on 4 GPUs from ~6.5 minutes). • Adds support for EmbeddingGemma and LFM2.5 embedding models, nvidia/MiniMax-M3-NVFP4, and cookbook recipes for Poolside Laguna-S-2.1 and Inkling-Small. • Adds `trtllm_mha` support for Gemma 4 MTP draft attention backend in speculative decoding. • Adds grammar-constrained decoding support (including `tool_choice=auto`) for DSpark speculative decoding. • Adds grammar-constrained decoding support in DFLASH speculative verify. • Adds grammar overlap scheduling for standalone speculative decoding. • Removes unnecessary expert padding on AMD, dropping MI355X FP4 MoE model weights from 159.07 GB to 112.36 GB and cutting colocated RL training-phase footprint from ~143 GiB to ~87 GiB per GPU. • Updates dependencies: flashinfer 0.6.15.post1, sgl-deep-gemm 0.1.5.post1, helion 1.4, mooncake 0.3.12.post1, dynamo-tokenizers 1.7.0; PyTorch remains at 2.11.0 and CUDA base image at 13.0.1. https://github.com/sgl-project/sglang/releases/tag/v0.5.17 https://github.com/sgl-project/sglang/releases/tag/v0.5.17 Sat, 08 Aug 2026 00:19:16 GMT SGLang v0.5.17 adds Kimi K3 and MiniMax-H3 day-0 support, a Rust frontend, DWDP MoE prefill, session-aware radix cache, and major load-time speedups. • Adds `--dcp-comm-backend {ag_rs, a2a, fi_a2a}` to select the DCP decode context-parallel communication backend: `a2a` exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer with FP8 KV as uint8 byte transport; `fi_a2a` delegates to the FlashInfer MNNVL kernel on GB200. • Adds `--dcp-replicate-q-proj` to project full-head Q locally and skip the per-layer Q head-dim all-gather in the DCP decode path. • Adds `--dwdp-size` to enable DWDP (weight-and-data parallelism) prefill, which prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch; benchmarked at 1.92x over DEP4 on 4x B200 at MNT 32K / ISL 32K. • Adds `--enable-session-radix-cache` so requests carrying a stable `session_id` use session-reference-aware eviction in the Unified Radix Cache; active-session prefixes are not evicted until the session is released via `/close_session`. • Adds `SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1` environment variable to enable the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation. • Adds `SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D` environment variable to copy oversized or non-contiguous CPU weight views into contiguous storage before H2D transfer, cutting DeepSeek-V4-Pro TP8 load time from ~35 minutes to ~6m20s (5.6x) and GPT-OSS-20B BF16 from 545s to 70s (7.8x). • Ships an initial native Rust serving layer covering tokenizer manager, ingress validation, egress, OpenAI-compatible API server, and PD disaggregation support, as prebuilt release artifacts. • Adds day-0 support for Kimi K3 (2.8T-parameter multimodal LatentMoE, 1M-token context, native MXFP4) with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on quantized weights, and reasoning/tool-call/OpenAI-compatible serving, verified on NVIDIA GB300 and AMD MI35x. • Adds day-0 support for MiniMax-H3 video-generation model producing synchronized video and stereo audio in one request, across task profiles `t2va` (text-to-video-and-audio), `fl2va` (first/last-frame conditioning), and `ref2va` (image/video/audio reference conditioning), verified on B200, H100, AMD MI300X/MI355X, and 2x RTX 5090 with layerwise offload. • Adds support for EmbeddingGemma and LFM2.5 embedding models, plus nvidia/MiniMax-M3-NVFP4. • Adds a weight-cache daemon that holds weights per GPU so a restarting engine can recover from cache instead of reloading from storage, reducing Qwen3-235B FP8 restart time from ~6.5 minutes. • Adds grammar-constrained decoding support to DSpark speculative decoding, including `tool_choice=auto`. • Adds `trtllm_mha` as a supported draft attention backend for Gemma 4 MTP speculative decoding. • Adds the `SGLANG_SIMULATE_ACC_LEN` environment variable for use with DFlash speculative decoding. • Reduces DeepSeek-V4 AMD memory footprint by removing unnecessary expert padding, dropping MI355X FP4 MoE model weights from 159.07 GB to 112.36 GB, and reclaims the HIP compress-state pool for RL colocated training, cutting training-phase footprint from ~143 GiB to ~87 GiB per GPU. • Updates dependencies to flashinfer 0.6.15.post1, sgl-deep-gemm 0.1.5.post1, helion 1.4, mooncake 0.3.12.post1, and dynamo-tokenizers 1.7.0; PyTorch remains at 2.11.0 and the CUDA base image at 13.0.1. • Adds support for Qwen3.5-MoE modelopt_fp4 checkpoints that quantize attention and load baked FP8 KV scales. • Adds grammar overlap scheduling for standalone speculative decoding. https://github.com/sgl-project/sglang/releases/tag/v0.5.16 https://github.com/sgl-project/sglang/releases/tag/v0.5.16 Sat, 25 Jul 2026 00:13:18 GMT SGLang v0.5.16 adds DSpark speculative decoding, Inkling 975B model support, DSA cache layer split, and ReplaySSM Ring Spec-Verify across 574 PRs. • Adds `--speculative-algorithm DSPARK` flag for DSpark confidence-driven speculative decoding, with `--speculative-dspark-block-size` to tune block size; requires `SGLANG_RAGGED_VERIFY_MODE=compact` environment variable — reaches 383.7 tok/s at accept length ~5 on DeepSeek-V4-Pro TP8. • Adds `--enable-dsa-cache-layer-split` flag (requires `--enable-prefill-cp --cp-strategy interleave`) for GLM-5.2 DSA cache layer sharding across CP ranks, cutting per-rank KV memory ~74% (0.77 to 0.20 GB/rank) at 8192 tokens. • Adds `--enable-gdn-replayssm-spec` flag for ReplaySSM Ring Spec-Verify (GDN), reducing speculative scratch from 11.5 GB to 1.8 GB per GPU (6.4x smaller) on Qwen3.5-35B-A3B at TP1; tune ring size with `--linear-replayssm-cache-len`. Requires `--speculative-eagle-topk` in `{None, 1}`. • Adds native support for Inkling, a 975B-parameter multimodal MoE with 1M-token context, NVFP4 MoE, optional vision/audio towers, and native MTP — verified on Blackwell TP4/TP8, H200, and AMD MI350X/MI355X. • Makes `UnifiedRadixTree` the default cache tree for SWA, Mamba, and DSA models, with Replay SSM and Mamba int8 checkpoints synced onto it and cache-hit state reset scoped to only the used state. • Adds support for new models: LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5 (VLA), and LongLive 2.0 (diffusion). • Adds linear attention decode kernel (`recurrent_kda`) for Blackwell SM100, running at 29.6 µs vs 36.8 µs for Triton at B=64, with 1.35x throughput at B=256. • Adds DeepSeek-V4 Wint4Abf16 and Wint4Afp8 quantization support. • Adds SM120 DeepSeek V4 `flashinfer_mxfp4` MoE runner backend with TP2 support. • Integrates Q8KV8 FP8 Sparse MLA Prefill into the DSA backend for DeepSeek-V3.2. • Fuses state I/O into the chunk-h kernel for GDN/KDA CuteDSL prefill. Breaking changes: • The experimental QServe (QoQ) W4A8 and FBGEMM FP8 quantization paths are removed; any server launch using these will fail. • CUTLASS FP8 blockwise kernels are removed for SM90/SM100; SM120 is moved to JIT. • `--fp4-gemm-backend cutlass` is removed along with in-tree NVFP4 JIT kernels; NVFP4 GEMM now requires FlashInfer. Use `auto`, which picks `flashinfer_cutedsl` on SM100 and `flashinfer_cutlass` on SM120. • `UnifiedRadixTree` is now the default for SWA, Mamba, and DSA models, changing cache behavior on those architectures. • Chunked input-logprob processing is now on by default to cap peak memory. • FA3 sparse mask kernels are off by default. • Kernels are relocated to the `sglang.kernels` namespace; code reaching past public wrappers to internal import paths must update. • `num_tokens_per_bs` is renamed to `num_tokens_per_req` across spec-decoding runners. • `--enable-deepep-waterfill` is renamed to `--enable-waterfill` with no deprecated alias; existing launch commands will fail with `unrecognized arguments`. • `--optimistic-prefill-retries` is renamed to `--optimistic-prefill-attempts` with no deprecated alias. • The SGLang-Diffusion post-training rollout endpoint now returns `application/msgpack` instead of JSON, with tensors as raw msgpack bytes; `tensor_to_base64` / `base64_to_tensor` are replaced by `tensor_to_bytes` / `bytes_to_tensor` — RL rollout consumers must be upgraded in lockstep with the server. https://github.com/sgl-project/sglang/releases/tag/v0.5.16 https://github.com/sgl-project/sglang/releases/tag/v0.5.16 Sat, 25 Jul 2026 00:13:18 GMT SGLang v0.5.16 adds DSpark speculative decoding, Inkling 975B model support, GLM-5.2 DSA cache layer split, and ReplaySSM ring spec-verify. • Adds `--speculative-algorithm DSPARK` and `--speculative-dspark-block-size` flags for DSpark confidence-driven speculative decoding, which drafts semi-autoregressively in blocks and sizes each verify window from the draft's own confidence; also requires `SGLANG_RAGGED_VERIFY_MODE=compact` environment variable. • Adds `--enable-dsa-cache-layer-split` flag for GLM-5.2 DSA cache layer sharding across CP ranks, cutting per-rank KV memory by ~74% (0.77 to 0.20 GB/rank) at 8192 tokens on GLM-5.2-FP8 with cp_size=4; requires `--enable-prefill-cp --cp-strategy interleave`. • Adds `--enable-gdn-replayssm-spec` flag for ReplaySSM Ring Spec-Verify (GDN), reducing speculative scratch memory from 11.5 GB to 1.8 GB per GPU (6.4x smaller); tunable via `--linear-replayssm-cache-len`; requires `--speculative-eagle-topk` in {None, 1}. • Adds support for Inkling, a 975B-parameter multimodal MoE with 1M-token context, mixing sliding-window, full and Mamba2 linear attention, NVFP4 MoE, optional vision/audio towers, and native MTP; verified on Blackwell TP4/TP8, H200, and AMD MI350X/MI355X. • `UnifiedRadixTree` is now the default KV cache tree for SWA, Mamba, and DSA models, with SSM and Mamba int8 checkpoint replay synced onto it and cache hits now resetting only the state they used. • Adds new model support for LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5 (VLA), and LongLive 2.0 (diffusion). • Adds linear attention (KDA) decode kernel for Blackwell (SM100) via FlashInfer, reaching 29.6 µs vs 36.8 µs for Triton at B=64, and 1.35x at B=256. • Adds `SGLANG_MORI_COMBINE_DTYPE=fp4` environment variable for FP4 combine dtype in the MORI EP backend. • Supports n>1 outputs for GLM-Image generation. • Supports RL rollout for the WAN diffusion pipeline via a per-request scheduler switch. • Adds Q8KV8 FP8 Sparse MLA Prefill integration into the DSA backend for DeepSeek-V3.2. • Adds DeepSeek-V4 Wint4Abf16 and Win4Afp8 quantization support. • Adds SM120 DeepSeek V4 `flashinfer_mxfp4` MoE runner backend with TP2 support. • Adds GDN/KDA CuteDSL prefill kernel fusing state I/O into the chunk-h kernel on Blackwell. Breaking changes: • The experimental QServe (QoQ) W4A8 and FBGEMM FP8 quantization paths are removed; models relying on these quantization methods will no longer load. • CUTLASS FP8 blockwise is deleted for SM90/SM100; SM120 is moved to JIT. • `--fp4-gemm-backend cutlass` is removed along with in-tree NVFP4 JIT kernels; NVFP4 GEMM now requires FlashInfer. Use `auto`, which picks `flashinfer_cutedsl` on SM100 and `flashinfer_cutlass` on SM120. • `UnifiedRadixTree` is now the default for SWA, Mamba, and DSA models, changing cache behavior on those architectures. • Chunked input-logprob processing is now on by default to cap peak memory. • FA3 sparse mask kernels are off by default. • The `sglang.kernels` namespace relocates internal kernel paths; code reaching past public wrappers to internal paths must update import paths. • `num_tokens_per_bs` is renamed to `num_tokens_per_req` across spec-decoding runners; code or configs referencing the old name will break. • `--enable-deepep-waterfill` is renamed to `--enable-waterfill` with no deprecated alias; existing launch commands will fail with `unrecognized arguments`. • `--optimistic-prefill-retries` is renamed to `--optimistic-prefill-attempts` with no deprecated alias; existing launch commands will fail. • The SGLang-Diffusion post-training rollout endpoint now returns `application/msgpack` instead of JSON, with tensors as raw msgpack bytes (`tensor_to_bytes` / `bytes_to_tensor`) instead of base64 (`tensor_to_base64` / `base64_to_tensor`); RL rollout consumers must be upgraded in lockstep with the server. https://github.com/sgl-project/sglang/releases/tag/v0.5.15 https://github.com/sgl-project/sglang/releases/tag/v0.5.15 Fri, 10 Jul 2026 22:58:33 GMT SGLang v0.5.15 adds native Exa web search, decode context parallelism, new FlashKDA kernels, and expands model support to Hunyuan 3, HRM-Text, and more. • Adds `flashinfer_trtllm_routed` MoE runner with FlashInfer all-to-all for routed Mixture-of-Experts inference. • Adds opt-in non-paged indexer for DeepSeek-V4 long-context prefill, now enabled by default for large prefill chunks, with >5% end-to-end throughput gain. • Enables Breakable CUDA Graph as the default capture path, reducing per-step kernel-launch overhead; full CUDA Graph for the prefill phase lands as experimental. • Adds `web_search` built-in tool backed by Exa for native web search support. • Adds FlashKDA prefill backend for safe-gate KDA linear attention, plus ReplaySSM buffered output-only decode for linear attention. • Adds Spec V2 as the default speculative decoding path with CUDA-graphable DSA draft-extend, dropped D2H/H2D syncs, and fused metadata ops for +11% end-to-end TPS. • Adds IndexShare MTP, reusing indexer top-k across draft steps for up to 1.9x lower draft-step cost at long context. • Adds TopK V2 fusing top-k selection with the page-table transform, supporting runtime k up to 2048. • Enables FlashMLA sparse prefill by default for DeepSeek-V4, providing >10% throughput gain on long context. • Enables decode context parallelism for MLA models including DeepSeek V3 and Kimi K2 series. • Adds shape-specialized JIT router GEMM and CuteDSL BF16 GEMM support for Blackwell (SM120). • Extends FlashInfer autotune coverage to draft-model graphs. • Adds new model support: Hunyuan 3 (Hy3), Hierarchical Reasoning Model (HRM-Text), NVIDIA LocateAnything-3B, Baidu Unlimited-OCR, JoyEcho multi-shot A/V, and Qwen3.6 NVFP4. • Adds speculative decoding support on XPU. • Bumps `transformers` to 5.12.1 and upgrades `tvm-ffi`, `sgl-deep-gemm`, and `tilelang`. https://github.com/sgl-project/sglang/releases/tag/v0.5.14 https://github.com/sgl-project/sglang/releases/tag/v0.5.14 Fri, 26 Jun 2026 22:57:30 GMT SGLang v0.5.14 adds 7 new models, LPLB/Waterfill MoE load balancing, NVFP4 MoE, MSCCL++ allreduce fusion, and AMD breakable CUDA graph support. • Adds `--ep-dispatch-algorithm=lp` to enable the new LPLB linear-programming load balancer for MoE expert parallelism, balancing token routing across redundant expert replicas by solving a per-layer LP; default behavior unchanged. • Adds `--moe-runner-backend flashinfer_trtllm_routed` to enable NVFP4 MoE quantization for DeepSeek-V4 on Blackwell GPUs for higher MoE throughput. • Adds `--flashinfer-allreduce-fusion-backend` flag (values: `auto`, `trtllm`, `mnnvl`) to control the FlashInfer fused allreduce + residual + RMSNorm MNNVL backend. • Adds `SGLANG_ENABLE_WAR_BARRIER` environment variable to force-enable the overlap scheduler WAR barrier on non-CUDA platforms such as AMD ROCm. • Adds support for online NVFP4 quantization via new online nvfp4 quantization path. • Adds support for asymmetric compressed-tensors MoE. • Adds Mooncake group semantics for disaggregated serving parallelism. • Adds bucketed multi-directory layout for NIXL file storage. • Adds get_parallel() structured accessor for parallel-topology state. • Adds graceful scheduler shutdown with hisparse host buffer freed on exit. • Adds support for GLM-4.7 function calling via structural tags. • Adds request/TTFT/worker metrics and a Grafana dashboard to the experimental `sgl-router`. • Adds W8A8 MXFP8 quantization support for Qwen3 Dense on Ascend NPU. • Adds Ascend NPU support for DeepSeek-V4. • Adds Gemma4 Sliding Window Attention support on the Ascend backend. • Adds LoRA support for DSA indexer LoRA targets for GLM-5.1 and DeepSeek-V3.2-family models. • Adds rejection sampling for MTP speculative decoding. • Adds sync-free `fast_prefill_plan` for EAGLE draft-extend CUDA graph. • Adds FlashInfer CUDA graph support for EAGLE draft-extend. • Adds ngram speculative decoding v2 (`ngram spec v2`). • Adds sliding window attention draft layer support for Dflash. • Adds piecewise CUDA graph support for Dflash. • Adds CuteDSL prefill kernel on SM100 (Blackwell) for Kimi-Linear (KDA), delivering 1.08–1.52x faster prefill than the Triton path via a reusable scratch workspace. • Adds int8 checkpoint pool for the linear-attention prefix cache (KDA/GDN models), substantially increasing prefix-cache capacity. • Deduplicates speculative conv-window intermediate cache via a sliding-window layout, halving its footprint with no numerical change. • Enables breakable CUDA graph execution on AMD ROCm/HIP. • Migrates MSCCL++ to the upstream `mscclpp` Python package (Executor + DSL compiler) with auto-tuned collectives for TP=8 single-node and TP=16 two-node configurations. • Adds context parallel strategy abstractions for new parallelism configurations. • Adds prefill context parallel support for DeepSeek-V4 unified KV attention on AMD. • Adds EPD disaggregated encode tracing. • Adds MPServer and embedded server support for granian to enable multiple tokenizer workers. • Adds Nemotron DP attention and MTP support for the hybrid Nemotron-H architecture. • Adds GB10 FP8 fused MoE Triton configuration. • Adds SwiGLU activation fused into gate `gather_qmv` for SwitchGLU MoE blocks on Apple Silicon (MLX). • Adds MTP support for DeepSeek-V4 Online Compress. • Adds MLA decode q-head padding to 64 under attention-TP, dispatching the cheaper FlashMLA head64 kernel instead of head128. • Prewarms the MHC prenorm kernel at startup to eliminate first-run JIT slowdown. • Supports BF16 mixed-dtype compression states on C4/C128 paths for DeepSeek-V4. • Adds new model support for GLM-5.2, LiquidAI LFM2.5, Kimi-K2.7-Code, Poolside Laguna-M.1, DiffusionGemma, Zyphra ZAYA1, and MiMo-V2-ASR. • Adds DeepSeek-V4 MTP acceptance length checks. • Fuses sigmoid + mul attention output gate into a single Triton kernel on AMD. • Enables fused GDN QKV split Triton kernel on HIP (AMD). • Fuses sigmoid + mul into a single Triton kernel for shared expert gating on AMD. • Adds LocalAttention for mistral3 encoder in diffusion models. • Defers MoE finalize and fuses it with main-stream add for DeepSeek-V3. • Fuses UE8M0 scale rounding directly into the FP8 group quantization kernel, removing a separate rounding pass. https://github.com/sgl-project/sglang/releases/tag/v0.5.13 https://github.com/sgl-project/sglang/releases/tag/v0.5.13 Sat, 13 Jun 2026 00:17:49 GMT SGLang v0.5.13 ships Spec V2 as default, nine new models, DeepSeek V4 context parallelism, and HiCache for hybrid models. • Adds `flash_mla_sparse_fwd` sparse FlashMLA kernel for DeepSeek V4, exposed via `sgl_kernel.flashmla` in sgl-kernel 0.4.3. • Spec V2 is now the default speculative-decoding path; tree drafting with `topk > 1` and `page_size > 1` is production-ready across triton, FA3, MLA, and aiter backends including Mamba/hybrid-linear models. • Adds adaptive speculative decoding with batch-size-aware `num_steps` and observability metrics. • Extends Spec V2 to the FlashMLA backend. • HybridModel (SWA/Mamba) now launches HiCache via UnifiedTree by default, bringing hierarchical KV-cache offload to sliding-window and Mamba hybrids out of the box. • Adds decode-side HiCache integration for incremental KV-cache transfer in PD disaggregation. • Adds HiSparse support for DeepSeek V4 with PD disaggregation. • Adds Pipeline Parallelism (PP) + PD support for DeepSeek V4. • Adds context-parallel serving for DeepSeek V4, including Context Parallel + MTP and Context Parallel + fused MoE kernel (non-DeepEP). • Adds DeepEP waterfill load balancing for DeepSeek V4. • Adds FP4 indexer support and SM120 support for DeepSeek V4. • Adds Piecewise CUDA Graph (PCG) support for DSA models and Kimi-K2.5. • Adds Breakable CUDA Graph (BCG) support for DeepSeek V4. • Adds draft-extend CUDA Graph for the trtllm `mha` attention backend. • Adds CuTeDSL MLA attention kernels from FlashInfer. • Adds FlashInfer Gated DeltaNet (GDN) kernels and a CuTeDSL GDN prefill kernel for Qwen 3.5 on Blackwell GPUs. • Adds prefill context parallelism for MLA models (Kimi K2.5, DeepSeek V3) and support for batch size > 1 during prefill CP. • Adds optimistic prefill for improved TTFT in PD disaggregation. • Adds EPD disaggregation support for MiMo-V2. • Heterogeneous CPU + GPU EPD disaggregation: offloads VLM vision encoding onto Intel Xeon CPUs, delivering up to ~1.3x P99 TTFT and request-throughput gains under load. • Adds MoRI communication library integration for AMD Instinct MI355X, enabling cost-competitive DeepSeek-R1 disaggregated inference. • Adds realtime video generation for SGLang-Diffusion with msgpack frame streaming, a standalone browser WebUI, continuous camera controls, super-resolution controls, and progressive-resolution growing across FLUX, FLUX.2, Qwen-Image, Wan, and Z-Image. • Adds support for new autoregressive models: Nemotron 3 Ultra (Day-0 with kernel optimizations), Step-3.7-Flash, and Command A+ (Cohere 2 family). • Adds support for new diffusion models: Cosmos3-Nano/Super (T2V/I2V/T2I), Ernie-Image, LingBot-World (realtime/causal-DMD), SANA-WM (streaming + realtime), FLUX.2-Klein 4B/9B, and Ideogram 4 (FP8/NVFP4 with tensor parallelism). • Reduces per-step scheduler overhead via unified async value passing through FutureMap and moving prefill input transfer onto the forward stream. • Spec V1 is deprecated; EAGLE/MTP now run on the unified V2 worker. Breaking changes: • Spec V1 is deprecated; EAGLE/MTP now run on the unified Spec V2 worker — configurations relying on the Spec V1 path will need to migrate to Spec V2. https://github.com/sgl-project/sglang/releases/tag/v0.5.12 https://github.com/sgl-project/sglang/releases/tag/v0.5.12 Sat, 16 May 2026 18:23:23 GMT SGLang v0.5.12 adds DeepSeek V4 inference, TokenSpeed MLA backend, HiCache/UnifiedRadixTree, and a wave of new model support. • Adds `--prefill-only-disable-kv-cache` flag to skip KV pool allocation entirely for prefill-only workloads. • Adds `SGLANG_TRACE_LEVEL` environment variable to control startup trace verbosity. • Adds `SGLANG_MAX_KV_CHUNK_CAPACITY` environment variable to cap KV chunk capacity. • Adds `SGLANG_RADIX_FORCE_MISS` environment variable to force radix-cache misses. • Adds `--enable-strict-thinking` server flag enabling two-phase reasoning grammar for structured thinking output. • Adds `--random-input-len` argument to `send_one.py` for synthetic load generation. • Renames `SGLANG_USE_JIT_ALL_REDUCE` to `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2`; the old name no longer applies. • Enables `SGLANG_OPT_FP8_WO_A_GEMM` by default, activating FP8 weight-only GEMM optimizations without manual configuration. • Adds `/v1/tokenize` endpoint with chat-completion-style request support. • Adds `sglang:get_loads_duration_seconds` Prometheus metric for load-duration observability. • Adds `fwd_occupancy` metric to `SchedulerStats` plus a matching Prometheus collector. • Adds per-iteration forward-pass metrics published over ZMQ PUB socket. • Adds SWA and Mamba radix-cache KV event metrics. • Adds full inference path for DeepSeek V4, including Tensor/Expert/Context/Data Parallel Attention, Prefill-Decode Disaggregation, HiSparse KV offload, reasoning and tool-call parsers, DeepGemm and FlashMLA kernels, W4A4/W4A8 MoE kernels, and TP16 on H100/H20. • Adds unified Docker image tag `lmsysorg/sglang:v0.5.12` covering all Nvidia GPU variants. • Adds TokenSpeed MLA attention backend for Blackwell (SM100) with FP8 KV cache support for low-latency MLA serving. • Adds HiCache framework support for UnifiedRadixTree (including SWA) with SSD offload via Mooncake store. • Adds custom speculative-algorithm registry, enabling user-defined spec-decoding strategies. • Adds `EagleDraftExtendInput` split for Spec V1 draft-extend paths. • Adds `trtllm` decode kernel for draft extend in speculative decoding. • Adds Adaptive Spec V2, EAGLE-3 SWA support, Kimi K2.5 EAGLE-3 MLA, and Gemma 3/4 + EAGLE-3 speculative decoding. • Adds MLA attention LoRA support for `q_b_proj` and `kv_b_proj` projections. • Adds CSGMV backend with virtual experts for MoE LoRA. • Adds `LoRADrainer` to reduce P99 TTFT under heavy LoRA workloads. • Adds Torch-Native LoRA embedding and graph optimization. • Adds NIXL staging buffer for heterogeneous-TP KV transfer in PD disaggregation. • Adds NIXL async transfer support. • Adds Mooncake incremental transfer and SSD offload for PD disaggregation. • Adds `PrefillDelayer` NCCL all-gather for cross-DP info sync. • Adds adaptive queue-based prefill-delayer trigger. • Adds retry-with-backoff for prefill bootstrap registration. • Adds Azure Blob Storage connector supporting `az://` URIs and `*.blob.core.windows.net` endpoints. • Adds Crusoe managed-inference backend. • Adds structural tags for strict tool calling and reasoning across additional models. • Adds auto-detection of reasoning and tool-call parser from chat template. • Adds OpenAI `reasoning.enabled` mapping to `thinking` and `enable_thinking` fields. • Adds multi-detokenizer support for concurrent detokenization. • Enforces rejection of `repetition_penalty=0` in SamplingParams.verify(). • Adds new model support: Intern-S2-Preview, MiniCPM-V 4.6, Laguna-XS.2, Ring-2.6-1T (1T-parameter reasoning model), Gemma 4 MTP, Trinity-mini (Ascend NPU), HunyuanVideo ModelOpt FP8, and Qwen Image ModelOpt FP8. • Adds CFG parallelism framework with multi-branch CFG for LTX-2 in SGLang-Diffusion. • Adds initial dynamic batching for SGLang-Diffusion. • Adds `dit_precision` config key for diffusion models, replacing hardcoded bf16. • Adds component attention-backend override CLI for SGLang-Diffusion. • Adds AMD online MXFP4 and FP8 diffusion quantization, aiter RMSNorm, and dual-stream MoE for ROCm. • Migrates DeepEP source to `deepseek-ai/DeepEP@hybrid-ep` for clean CUDA 13 builds; pins FlashInfer at 0.6.11.post1. • Adds TMA bulk-store `set_mla_kv_buffer` kernel delivering up to 12x throughput over baseline. Breaking changes: • `SGLANG_USE_JIT_ALL_REDUCE` is renamed to `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2`; configurations or scripts referencing the old name will no longer take effect. • `SGLANG_OPT_FP8_WO_A_GEMM` is now enabled by default; deployments that previously relied on it being off must explicitly disable it. • VAE CPU offload is disabled by default in SGLang-Diffusion; workflows that depended on automatic CPU offload must re-enable it explicitly. https://github.com/sgl-project/sglang/releases/tag/v0.5.11 https://github.com/sgl-project/sglang/releases/tag/v0.5.11 Tue, 05 May 2026 21:28:56 GMT SGLang v0.5.11 adds DFLASH spec-decode, decode-side radix cache for PD disaggregation, LoRA on DeepSeek-V3/Kimi-K2, and 14+ new model architectures. • Splits `accept_length` metric into `num_accepted_drafts` and `num_accepted_tokens` for finer speculative-decoding observability. • Adds `--page-size` support with speculative decoding on AMD ROCm (previously caused a memory access fault). • Exposes raw KV cache pool token counts as Prometheus gauges. • Adds a Prometheus metrics endpoint for gRPC mode. • Adds HTTP sidecar endpoints and a `FlushCache` gRPC RPC for gRPC mode. • Surfaces pending token count in prefill log and `get_load`. • Supports `moe_dp_size = 1` paired with arbitrary `attention_cp_size` so MoE and attention parallelism can be tuned independently. • New `DFLASH` speculative-decoding kernel, enabled across additional model backends and AMD ROCm. • Speculative Decoding V2 (`Spec V2`) with overlap scheduling is now the default, reducing per-step CPU cost for EAGLE/MTP/DFLASH paths. • Adds adaptive `speculative_num_steps` for EAGLE topk=1. • Enables piecewise CUDA graph with speculative decoding. • Adds OpenTelemetry tracing for speculative decoding, pipeline parallelism, and DiffGenerator. • Adds decode-side radix cache (`Decode Radix Cache`) for prefill/decode (PD) disaggregation, recovering cache hit rates and TTFT savings for long shared prefixes. • Adds incremental transfer for the Mooncake transfer engine in PD disaggregation. • Adds NIXL heterogeneous TP KV transfer for non-MLA models. • Adds NIXL Mamba state slice transfer for heterogeneous TP. • Enables LoRA on DeepSeek-V3 MLA (including quantization) and Kimi-K2. • Adds `LoRADrainer` to reduce high P99 TTFT under LoRA workloads. • Adds decoupled LoRA MoE backend with Marlin support. • Adds dual MoE CUDA graph capture for lora/nolora batches. • Adds CSGMV kernel offline auto-tuning for LoRA. • New `FlashInferCuteDslMoE` layer for the standard FP4 MoE path. • Integrates community-contributed FA3 kernels alongside FA4 as a high-performance alternative. • Adds NVFP4 KV cache with a quantization strategy abstraction and kernel. • Adds GPTQ / AWQ 4-bit quantization on CPU. • Adds GGUF quantization for Ascend NPU (dense and MoE models). • Adds TP communications compression for Qwen3 on Ascend NPU. • Adds pipeline parallelism support on Intel XPU. • New `SequenceClassification` model architecture, powering the Score API. • Adds day-0 support for Gemma 4, GLM-5.1, Qwen3.6, MiMo-V2.5/V2.5-Pro, Ling-2.6-Flash, Mistral Medium 3.5, Kimi-K2.6, Hunyuan v3 (preview), Voxtral, Parakeet, Moss-VL, and Qwen3-ASR (chunk-based streaming). • Adds diffusion model support for LTX-2.3 (including two-stage and TI2V), ERNIE-Image, FLUX.2-small-decoder, JoyAI-Image-Edit, FLUX.1-dev ModelOpt NVFP4, Qwen Image ModelOpt FP8, and Stable Diffusion 3 medium. • Adds ModelOpt diffusion FP8 support for Flux1/Flux2 and Wan2.2. • Adds Standalone Rollout API, Denoising Environment Backpass, and SP-Aligned Log-Prob for T2I post-training in SGLang-Diffusion. • Adds disaggregated diffusion support. • Adds dynamic batching v0 for SGLang-Diffusion. • Adds CPU platform support for SGLang-Diffusion. • Enables in-memory loading for URL/base64 image inputs in SGLang-Diffusion (now default). • Default CUDA version moves to 13.0 across SGLang, sgl-kernel, and Docker images; PyTorch upgraded from 2.9 to 2.11. https://github.com/sgl-project/sglang/releases/tag/v0.5.10 https://github.com/sgl-project/sglang/releases/tag/v0.5.10 Mon, 06 Apr 2026 04:42:12 GMT SGLang v0.5.10 adds elastic failure tolerance, LoRA-on-MoE, native Apple Silicon, and a raft of new model and kernel capabilities. • Adds `SGLANG_NSA_DENSE_ATTN_KV_LEN_THRESHOLD` environment variable to control the KV length threshold at which the sparse MLA attention kernel is applied during prefill for DeepSeek V3.2. • Adds `--stream-response-default-include-usage` server flag to control whether usage stats are included in streaming responses by default. • Adds `--strict-ports` option for predictable, deterministic port assignment. • Adds `NetworkAddress` abstraction for IPv6-safe address handling, enabling IPv6 dual-stack deployments. • Enables piecewise CUDA graph capture by default, reducing memory overhead and improving throughput for models with complex control flow. • Integrates Elastic NIXL-EP (Elastic EP) for partial failure tolerance in DeepSeek MoE deployments — when a GPU fails, expert weights are redistributed from DRAM and serving continues without a full restart. • Adds GPU staging buffer with dynamic ring allocator for PD disaggregation, gathering scattered GQA head slices into contiguous memory for bulk RDMA transfer (~1000x fewer RDMA requests, ~5x TPS/GPU at large concurrency on Qwen3.5 with Prefill TP4+Decode DEP4). • Integrates HiSparse sparse attention backend for efficient long-context inference with sparsity-aware attention, also enabling HiSparse direct cache transfer from Prefill to Decode DRAM. • Integrates FlashInfer MXFP8 kernels for GEMM and MoE operations, enabling mixed-precision FP8 inference with microscaling for RL and general workloads. • Adds LoRA fine-tuning support for Mixture-of-Experts layers, including JIT alignment kernels, fused Triton kernels, TP support, CUDA graph support, and auto-detection of LoRA target modules. • Upgrades from transformers 4.57.1 to 5.3.0, adding support for latest HuggingFace model architectures including GLM-5 on the main branch. • Upgrades to the official Flash Attention 4 package, bringing Blackwell GPU support and the latest attention optimizations. • Enables speculative decoding for the FA4 attention backend. • Multi-modal attention now uses FA4 by default on SM100 (Blackwell) hardware for improved VLM performance. • Enables context parallelism during prefill for multi-head attention models (e.g. Qwen3 MoE), distributing long sequences across GPUs. • Adds Score API via `EngineScoreMixin` implementation. • Adds subprocess liveness monitor to detect scheduler crashes. • Adds direct model loading from object storage with RunAI Model Streamer. • Adds MFU (Model FLOPs Utilization) metrics exposed via Prometheus. • Enables `IndexCache` for DeepSeek V3.2, improving throughput by more than 10% on high-concurrency workloads. • Adds TRT-LLM prefill/decode DSA kernels as the default for Blackwell (SM100/SM103) hardware. • Enables NCCL/RCCL pre-warming to reduce P99 TTFT cold-start latency. • Enables multi-thread weight loading by default. • Adds CuTeDSL KDA decode kernel support and fused GDN projection Triton kernel for improved Qwen3.5 performance. • Adds enhanced transformers modeling backend with full TP, PP, MoE, VLM support, and `torch.compile` compatibility. • Renames the kernel package from `sgl-kernel` to `sglang-kernel` (v0.4.1), consolidating kernels and removing deprecated ops. • Replaces `decord` with `torchcodec` for video decoding in VLM pipelines. • Replaces `soundfile`+`torchaudio` with `torchcodec` `AudioDecoder` in `load_audio` for VLM audio ingestion. • Adds chunk-aware ViT encoding with per-image cache and lazy device transfer for VLM workloads. • Adds macOS platform support for diffusion models. • Integrates Cache-DiT optimizations into the diffusers backend for improved diffusion performance. • Adds NVFP4 support for Flux.2 diffusion model. • Adds LTX-2 two-stage pipeline support for diffusion. • Adds EPLB rebalance support for Kimi K2.5 in Elastic EP. • Adds reasoning tokens usage reporting. • Adds new model support: Nemotron-3-Super (bf16/fp8/nvfp4), Mistral Small 4 (Pixtral), LFM2-VL, Voxtral (speech-to-text), Helios, Hunyuan3D-2, LTX-2, MOVA, FireRed-Image-Edit. • Adds AMD: FP8 prefill integration with radix cache path for DeepSeek models, MHA FP8-KV support, AMD MXFP4 support for Qwen3.5-397B-A17B, and FP8 KV cache + FP8 attention kernel for NSA on MI300/MI355 with TileLang. • Adds CPU backend kernel `apply_rotary_pos_emb_cpu` for Qwen3-VL and Qwen3-Omni, and MXFP4 GEMM kernels for Intel AMX to support GPT-OSS series. • Adds NPU/Ascend support for Kimi-K2.5-w4a8 and diffusion models (with `enable_torch_compile`), plus GLM-4.7-Flash on NPU. Breaking changes: • The kernel package is renamed from `sgl-kernel` to `sglang-kernel` (v0.4.1); any install scripts, `requirements.txt`, or import paths referencing `sgl-kernel` or `sgl_kernel` must be updated. • Transformers upgraded from 4.57.1 to 5.3.0; model configs or custom modeling code that relied on transformers 4.x APIs may require updates. • `decord` is replaced by `torchcodec` for video decoding; environments with `decord` installed for VLM video pipelines must install `torchcodec` instead. • `soundfile`+`torchaudio` replaced by `torchcodec` `AudioDecoder` in `load_audio`; environments relying on those libraries for audio ingestion in VLM pipelines must install `torchcodec`. https://github.com/sgl-project/sglang/releases/tag/v0.5.9 https://github.com/sgl-project/sglang/releases/tag/v0.5.9 Tue, 24 Feb 2026 01:14:21 GMT SGLang v0.5.9 adds Anthropic-compatible API, NSA kernel integration, LoRA overlap loading, FA4 multimodal support, and 10+ new models. • Adds `--nsa-prefill-backend` and `--nsa-decode-backend` flags integrating TRT-LLM Native Sparse Attention kernels for DeepSeek V3.2, delivering 3x–5x throughput gains on Blackwell platforms. • Adds native Anthropic API compatibility endpoint, enabling direct use of tools and clients built for the Anthropic API format. • Adds `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable for configuring Mooncake intra-node NVLink KV transfer memory pool. • Adds routed-experts field to `/completions` and `/chat/completions` endpoints for MoE models. • Introduces LoRA weight loading overlap with computation, reducing TTFT by ~78% and TPOT by ~34.88% on large adaptors. • Adds Flashinfer all-to-all MoE dispatcher for efficient expert parallelism communication in MoE models. • Introduces FA4 (FP4 attention) backend and variable-length attention function for multimodal encoders, enabling lower-precision inference for vision-language models. • Adds KV transfer support via MORI-IO for prefill-decode disaggregation. • Adds INT4 kernels for the CPU backend. • Adds JointThreshold algorithm for joint M2T and T2T decoding in diffusion LLMs. • Adds basic dLLM scheduling strategy and implementation. • Removes CUDA graph batch size limitation for diffusion LLMs. • Updates AITER to v0.1.10.post3, adding FP8 Prefill, FP8 Decode, and FP8 KV Cache support on AMD hardware. • Adds commit-to-version lookup tool at docs.sglang.io/references/release_lookup.html for tracing PRs to official releases. • Adds new model support for Kimi-K2.5, GLM-5, Qwen 3.5, MiniMax 2.5, Ernie4.5-VL, Step3-VL, Step-3.5-Flash, LLaDA 2.1, Ring 2.5 1T / Ling 2.5 1T, MOVA (Diffusion), GLM-OCR, and DeepSeek-OCR-2. • Adds diffusion pipeline support for parallel VAE decoding, token-level sequence sharding, Nunchaku and FP8 support, fused kernels, LTX-2, and multiple new models in the ComfyUI plugin. • Adds MxINT4 Flashinfer TRT-LLM MoE GEMM support. • Adds Kimi K2.5 Day 0 ROCm support and standardizes on ROCm 7. • Adds Qwen 3.5 support on Ascend NPU. • Adds MiniCPM3-4B model support on NPU/Ascend. • Migrates GPTQ-Marlin and AWQ Marlin repack kernels to JIT compilation. Breaking changes: • ROCm 6.3 is deprecated; ROCm 7 is now the standard for AMD hardware. https://github.com/sgl-project/sglang/releases/tag/v0.5.8 https://github.com/sgl-project/sglang/releases/tag/v0.5.8 Fri, 23 Jan 2026 22:09:28 GMT SGLang v0.5.8 adds Flash Attention 4, chunked pipeline parallelism, Multi-LoRA diffusion inference, and support for GLM 4.7 Flash, LFM2, Qwen3-VL, and DeepSeek V3.2 NVFP4. • Supports Flash Attention 4 decoding kernels for improved attention performance. • Adds chunked pipeline parallelism enabling close-to-linear scaling for million-token context windows. • Adds Multi-LoRA inference support in SGLang-Diffusion, along with SLA attention backends, a warmup switch in the CLI, and a ComfyUI Plugin. • Adds configurable generator device and seed support for the diffusion API. • Supports running SGLang-Diffusion with the diffusers backend. • Adds Context Parallelism Optimization for DeepSeek V3.2 with support for fused MoE, multi-batch, and FP8 KV cache. • Supports pipeline parallelism (PP) x PD decode disaggregation with the nixl backend. • Supports decode pipeline parallelism for PD disaggregation. • Adds MXFP8 Grouped GEMM support for Blackwell (B200) GPUs via sgl-kernel. • Adds day-0 model support for GLM 4.7 Flash. • Adds model support for LFM2. • Adds model support for Qwen3-VL-Embedding and Qwen3-VL-Reranker. • Adds model support for DeepSeek V3.2 NVFP4. • Adds model support for black-forest-labs/FLUX.2-klein-9B diffusion model. • Adds model support for Mistral Large 3. • Adds single batch overlap for MoE models, improving GPU utilization. • Adds positional embedding ID cache for the Qwen-VL model family, reducing recomputation overhead. • Adds CPU support for the `chunk_gated_delta_rule` kernel for Qwen3-Next. • Adds CPU support for mamba causal conv1d for Qwen3-Next. • Delivers up to 1.5x throughput improvement across all major diffusion models. • Achieves 65% faster TTFT for GLM4-MoE in production via EPD disaggregation and elastic encoder scaling for vision-language models. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.3.1 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.3.1 Fri, 09 Jan 2026 06:18:26 GMT SGLang gateway v0.3.1 delivers 10-12x faster cache-aware routing, 99% memory reduction, JWT/OIDC auth, and a new Classification API. • Adds JWT/OIDC authentication for control plane APIs (tokenizer management, worker registration, admin endpoints) with native support for Google, Azure, Oracle, and GitHub identity providers. • Adds `PrefixHash` load balancing policy — a new KV cache-aware routing strategy using prefix hashing to improve cache hit rates in multi-tenant environments. • Adds native Classification API support with dedicated pipeline stages and protocol types for serving classification models alongside inference workloads. • Adds Nemotron Nano V3 reasoning parser support. • Adds in-flight request age metrics for observability and SLA monitoring. • Radix tree / cache-aware routing engine overhauled: INSERT throughput up from 38 MB/s to 440 MB/s, MATCH throughput up from 83 MB/s to 253 MB/s, latency down from 52.9 µs to 4.6 µs per operation, and memory per node down from ~180 KB to ~1.4 KB (~99% reduction). • Organizes CLI arguments into logical groups for improved discoverability. • HTTP Router fan-out replaced from serial execution to concurrent streams for reduced latency under load. https://github.com/sgl-project/sglang/releases/tag/v0.5.7 https://github.com/sgl-project/sglang/releases/tag/v0.5.7 Thu, 01 Jan 2026 10:01:57 GMT SGLang v0.5.7 adds day-0 support for Mimo-V2-Flash, LLaDA 2.0, EAGLE 3 speculative decoding, Model Gateway v0.3.0, and `--dit-layerwise-offload` for diffusion models. • Adds `--dit-layerwise-offload true` CLI flag for SGLang-Diffusion to reduce peak VRAM usage by up to 30 GB and improve performance by up to 58% across all diffusion models. • Adds `--adjust-frames` and `--override-protected-fields` CLI arguments to the diffusion pipeline. • Day-0 support for Mimo-V2-Flash, Nemotron-Nano-v3, and LLaDA 2.0 models. • Day-0 support for Qwen-Image-Edit-2509, Qwen-Image-Edit-2511, Qwen-Image-2512, and Qwen-Image-Layered in SGLang-Diffusion. • Releases Model Gateway v0.3.0 with multimodal initialization and image processors for LLaVA, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Phi3-Vision, Phi4-Vision, LLaMA-4-Vision, and Mistral 3. • EAGLE 3 speculative decoding draft models now available for popular models. • Adds scalable pipeline parallelism with dynamic chunking for ultra-long contexts. • Adds encoder disaggregation support for multi-modal models. • Enables RadixCache for Mamba2 models. • Supports pipeline parallelism (PP) combined with PD decode disaggregation via the nixl backend. • Adds MXFP8 Grouped GEMM support on Blackwell (B200) GPUs via sgl-kernel. • Adds Cache-dit integration for SGLang-Diffusion to improve performance by up to 165%. • Extends SGLang-Diffusion to support AMD, RTX 4090, and RTX 5090 GPUs, additional attention backends (`sage-attn`, `sage-attn3`), tensor parallelism (TP), and Google Vertex AI in the HTTP API. • Adds single-batch overlap for MoE models to improve throughput. • Adds Mistral Large 3 and Mistral Large 3 EAGLE speculative decoding support. • Adds Ministral 3 model support. • Supports data-parallelism for GLM-V vision models. • Adds configurable generator device and seed support for the diffusion HTTP API. • Adds a lightweight runtime Docker stage via Dockerfile refactor. • Adds model architecture and type fields to the server info endpoint for gateway harmony. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.3.0 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.3.0 Wed, 24 Dec 2025 22:00:56 GMT SGLang Gateway v0.3.0 adds unified IGW mode, tokenize/detokenize endpoints, parser endpoints, gRPC embeddings, TLS, and a Go implementation. • Adds `/parse/reasoning` and `/parse/function_call` HTTP endpoints for parsing reasoning outputs and function call responses, including a GLM-4 function call parser. • Adds tokenize/detokenize HTTP endpoints with a dynamic tokenizer control plane (add, list, get, remove tokenizers on-the-fly) backed by a `TokenizerRegistry`. • Adds native embeddings endpoint for the gRPC router, expanding the gateway beyond text generation to embedding workloads. • Adds server-side TLS support with graceful shutdown for both TLS and non-TLS servers. • Adds `X-SMG-Error-Code` response headers and unified error codes across all router types. • Adds streaming metrics (TTFT, TPOT, tokens, duration) for the gRPC router, plus `smg_worker_*`, `smg_router_*`, MCP, and discovery metric layers in a new 6-layer Prometheus metrics architecture. • Adds custom Prometheus duration buckets. • Introduces Unified Inference Gateway (IGW) mode: a single gateway instance now supports gRPC (PD and regular), HTTP (PD and regular), and OpenAI routers simultaneously, auto-enabled when Kubernetes service discovery is turned on. • Adds retry and circuit breaker support for OpenAI and gRPC routers; returns HTTP 503 when all workers are circuit-broken. • Adds a complete Go implementation of the SGLang Model Gateway with an OpenAI-compatible API server, contributed by the iFlytek MaaS team. • Adds multi-architecture Docker builds supporting Linux, macOS, Windows, and ARM. • Introduces a DAG-based parallel workflow engine with pre-computed dependency graphs and concurrent event processing for worker lifecycle orchestration. • Replaces `PolicyRegistry` `RwLock` with `DashMap` for lock-free policy lookups and adds lock-free router snapshots to reduce CPU overhead. Breaking changes: • Prometheus metric names and structure have changed (complete 6-layer architecture redesign) — update all Prometheus dashboards and alerting rules. • Workers are now identified by UUIDs instead of endpoints — update any tooling or scripts that interact with the worker API. • `SmgMetrics` is renamed to `Metrics` and `smg_labels` is renamed to `metrics_labels` — update any code referencing these names. • Legacy `RouterMetrics` has been removed — migrate to the new unified metrics layer. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.4 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.4 Wed, 10 Dec 2025 01:09:08 GMT SGLang Model Gateway v0.2.4 adds WASM middleware, full OpenTelemetry tracing, multimodal VL model support, and broad performance optimizations. • Adds WASM middleware support, enabling safe, isolated WebAssembly plugins for custom routing logic, request/response transformation, and proprietary system integration without touching core code. • Adds OpenTelemetry distributed tracing to the HTTP router with native trace context propagation across the inference stack. • Adds OpenTelemetry integration to the gRPC router, extending distributed tracing coverage to both HTTP and gRPC transports. • Makes the tokenizer builder aware of environment variables such as `HF_ENDPOINT` for flexible model loading configuration. • Adds a `version` command to the SGLang Model Gateway CLI, exposed via the Python binding. • Adds support for routing Vision-Language (VL) models in the gateway router. • Adds image processors for LLaVA, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Phi3 Vision, Phi4 Vision, Llama4 Vision, and Mistral 3 multimodal models. • Adds a workflow for routing to external model providers. • Adds `ModelCard` and `ProviderType` support to worker metadata for richer model configuration. • Adds audio and moderation entries to the model card. • Adds HTTP response status code metrics to the router. • Implements `TokenGuardBody` for managing and controlling token return in responses. • Optimizes the radix tree used in cache-aware load balancing for lower routing overhead. • Optimizes the tokenizer for dramatically reduced CPU and memory footprint. • Supports `numactl` CPU and memory binding before the gateway process starts. • Allows the token refill rate to be set to zero. • Promotes conversation handling to first-class routing in the OpenAI router. • Renames the Rust package and binary from `sgl-router` to `sgl-model-gateway`. Breaking changes: • The Rust package and binary have been renamed from `sgl-router` to `sgl-model-gateway`; any scripts, container images, or CI pipelines referencing `sgl-router` by name will break. https://github.com/sgl-project/sglang/releases/tag/v0.5.6 https://github.com/sgl-project/sglang/releases/tag/v0.5.6 Wed, 03 Dec 2025 05:11:51 GMT SGLang v0.5.6 adds DeepSeek V3.2, new diffusion models, JIT kernels, multimodal cache, bucket routing policy, and scheduler metrics. • Adds `SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL` environment variable for controlling EPLB heatmap collection intervals in PD deployments. • Adds `intel_xpu` attention backend for Llama 4 on Intel XPU hardware. • Adds server-level multimodal cache support for VLMs. • Adds bucket routing policy for the SGLang router. • Adds scheduler and hiradix cache metrics via the metrics system. • Supports structured model output for OpenAI and gRPC router endpoints. • Introduces JIT kernels support. • Supports piecewise CUDA graph for MLA (Multi-head Latent Attention). • Adds support for DeepSeek V3.2 and DeepSeek V3.2 Speciale models, including MHA pathway for short-sequence prefill on B200 (SM100) via NSA. • Adds support for blockwise diffusion language models. • Adds support for new diffusion models: Flux2 and Z-image. • Supports Kimi-K2-Thinking model enhancements. • Adds Jet-Nemotron model support. • Adds pipeline parallelism (PP) support for dots_vlm. • Adds `aux_hidden_states` capture support for Minimax M2. • Upgrades to Torch 2.9. • Updates AMD wave-lang to 3.8.2. • Supports `hidden_dim % 4 == 0` in `per_token_quant_fp8` kernel, broadening FP8 quantization coverage. • Adds fallback to Triton `mm_persistent` kernel when DeepGEMM fails. • Refactors KTransformers heterogeneous compute with a unified GPU-quantization backend. • Adds `row_starts` parameter to the topk kernel for DeepSeek V3.2. • Reduces radix tree syscalls and optimizes collection filtering. • Supports WAN-2.2-TI2V and sequence parallelism for diffusion models. • Adds validation for served model name to reserve `:` for LoRA adapter syntax. • Refactors custom memory pool and adds bare-metal PD (prefill-decode disaggregation) support. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.3 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.3 Mon, 17 Nov 2025 11:23:30 GMT SGLang Gateway v0.2.3 adds bucket-mode routing (20-30% TTFT gain), PostgreSQL chat history, and expanded tool/structured-output support. • Adds PostgreSQL as a supported data connector for chat history management, joining OracleDB and in-memory storage options. • Introduces bucket-based routing policy for PD mode, delivering 20-30% improvements in TTFT and overall throughput. • Adds structured model output support for the OpenAI and gRPC router. • Adds streaming parsing with Tool Choice in the chat completions API for the gRPC router. • Adds `tool_choice` support for the Responses API in the gRPC router. • Emits `OutputItemDone` events and stores output item arrays in the gRPC router for improved observability. • Adds support for the MinMax M2 model, including an XML tool parser and reasoning parser. • Adds vLLM backend support for the gRPC router. • Adds support for complex assistant and tool messages in `/chat/completions`. • Adds timing metrics aggregation for requests in sgl-router. • Adds engine metrics aggregation support in sgl-router. • Removes the worker URL requirement for router configuration. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.2 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.2 Mon, 17 Nov 2025 11:19:03 GMT SGLang Gateway v0.2.2 adds Responses API for OSS models, gRPC health checks, mTLS, and universal platform support in a single wheel. • Adds `v1/responses` API endpoint to the gateway, bringing OpenAI Responses API support to open-source models including Llama, DeepSeek, and Qwen with built-in chat history management and multi-turn conversations. • Adds native gRPC standard health check endpoint support for production Kubernetes deployments with proper health monitoring and orchestration. • Adds mTLS support for router-to-worker communication, securing gateway-to-SGLang traffic across edge, remote cloud, multi-cloud, and hybrid environments. • Adds function calling support for the OpenAI router Responses API. • Adds `web_search_preview` tool support for the OpenAI router. • Adds multiple worker URL support for the OpenAI router for better load distribution. • Introduces the Harmony Pipeline: a unified OpenAI-native architecture supporting GPT OSS models for both Responses API and Chat Completion with MCP integration. • Adds MCP connection pooling, tool inventory, and per-server proxy configuration. • Adds workflow retry and timeout configuration via `routerConfig`. • Extends platform support to Linux, Windows, macOS, x86, and ARM in a single wheel file covering Python 3.8 through 3.14, while reducing wheel size by more than 40%. • Adds fine-grained request latency tracing (Part 2) for detailed performance tracking. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.1 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.1 Mon, 17 Nov 2025 11:13:12 GMT SGLang Gateway v0.2.1 adds a two-level tokenizer cache, a new /v1/classifications endpoint, and a worker management workflow engine. • Adds `GET /v1/classifications` endpoint, bringing OpenAI-style classification API support to the router. • Adds configurable two-level tokenizer caching: L0 exact-match cache for repeated prompts and L1 prefix-aware cache at special-token boundaries. • Introduces a Worker Management Workflow Engine with improved async registration, worker self-discovery, and health orchestration. • Adds parallel queue puts in the gRPC request manager and removes the mutex on the gRPC client, enabling concurrent gRPC request handling. Breaking changes: • Removes `continue_final_message` from `ChatTemplateParams` in the gRPC router. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.0 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.2.0 Mon, 17 Nov 2025 11:03:01 GMT SGLang Model Gateway v0.2.0 adds multi-model IGW mode, Rust-powered gRPC data plane, pluggable history storage, and native MCP integration. • Adds `serve_grpc` to `launch_server` for a Rust-native gRPC data plane with tokenizer, reasoning parser, and tool parser running at native speed with real-time token streaming. • Adds CLI flags for queue size, timeout, and rate limit (`--queue-size`, `--timeout`, `--rate-limit`) to configure the token-bucket rate limiter (disabled by default). • Adds reasoning and tool parser arguments to the router CLI for controlling parsing behavior in gRPC and HTTP modes. • Adds `--chat-template` CLI argument to allow specifying a custom chat template path (directory or file). • Adds gRPC PD (prefill-disaggregation) mode for both `/v1/chat/completions` and `/generate` endpoints via the gRPC router. • Adds logprobs support to the gRPC router for `/v1/chat/completions`. • Adds full `ToolChoice` support and streaming tool-call parsing in the gRPC router. • Introduces Multi-Model Inference Gateway (IGW) mode, letting a single gateway manage multiple models — each with its own routing policy, priorities, and metadata — with dynamic model registration via `/workers`. • Adds pluggable conversation history storage via `memory`, `none`, or `oracle` backends; `oracle` provides full persistence via Oracle ATP with connection pooling and credentials support. • Adds `GET /v1/chat/conversations` and conversation item CRUD APIs (`create`, `retrieve`, `delete`) for history management. • Adds native MCP integration across all transports (STDIO, HTTP, SSE, Streamable) with multi-turn tool-calling loop support and streaming MCP tool-call responses for agentic workflows. • Adds OpenAI-compatible gateway mode to proxy requests to OpenAI while keeping conversation history and responses flowing through the local gateway, including support for Grok models. • Adds worker self-discovery for metadata, enabling automatic registration of worker capabilities and labels. • Adds IPv6 support across all gateway components. • Adds Prometheus metrics for latency, load, queue depth, PD pipelines, tokenizer speed, and MCP activity, along with structured tracing and request-ID propagation. • Adds per-worker circuit breakers, retries with exponential backoff and jitter, and FIFO queuing for reliability. • Adds active request cancellation on client disconnect using RAII-based cleanup. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.1.9 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.1.9 Mon, 17 Nov 2025 10:58:59 GMT SGLang Gateway v0.1.9 adds configurable retry logic for the PD router and completes the OpenAI API spec for the router. • Adds configurable retry logic to the PD router to reduce backend pressure under load. • Completes OpenAI API spec coverage for the router, expanding compatible endpoint support. • Introduces a data-parallel (DP) worker abstraction in the router for improved worker management. • Implements an HTTP dependency injection pattern for the router system, enabling more modular handler composition. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.1.7 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.1.7 Mon, 17 Nov 2025 10:51:42 GMT SGLang gateway-v0.1.7 adds dp-aware routing and per-node policies for PD disaggregation in sgl-router • Adds a dp-aware routing strategy to `sgl-router`, enabling data-parallel-aware request distribution across worker nodes. • Adds separate routing policies for prefill (`p`) nodes and decode (`d`) nodes in `sgl-router`, allowing independent policy configuration per node type. • Migrates `sgl-router` HTTP layer from actix to axum, improving request handling internals. • Improves router logging and request ID header propagation in `sgl-router`. https://github.com/sgl-project/sglang/releases/tag/gateway-v0.1.6 https://github.com/sgl-project/sglang/releases/tag/gateway-v0.1.6 Mon, 17 Nov 2025 10:44:25 GMT SGLang Gateway v0.1.6 adds stream_options forwarding, request-completion metrics updates, and empty-worker router support. • Forwards `stream_options` in routed requests so streaming configuration is correctly propagated to backend workers. • Updates router metrics when a request completes, enabling accurate in-flight and throughput tracking. • Allows the router to start and operate with an empty worker pool, removing the requirement for at least one worker to be registered at startup. https://github.com/sgl-project/sglang/releases/tag/v0.5.5 https://github.com/sgl-project/sglang/releases/tag/v0.5.5 Thu, 06 Nov 2025 17:54:32 GMT SGLang v0.5.5 adds Kimi-K2, MiniMax-M2, NVILA model support, mTLS for router workers, MCP connection pooling, and ~10% B200 MoE perf gains. • Adds mTLS support for router-to-worker communication, securing inter-component traffic in multi-node deployments. • Adds MCP Manager to the router with connection pooling, tool inventory, and proxy support. • Adds `flashinfer_trtllm` MoE runner backend, delivering ~10% throughput improvement on B200 FP8 DeepSeek workloads. • Adds deterministic inference support for DeepSeek architecture models on a single GPU. • Adds DeepGEMM support for deterministic inference. • Adds DeepSeek-R1 w4a8 low-latency DeepEP support. • Adds CUDA graph support for `trtllm_mla` attention backend spec decoding, improving speculative decoding throughput. • Adds support for MiniMax-M2 model. • Adds support for NVILA and NVILA Lite vision-language models. • Adds support for Kimi-K2-Thinking model. • Adds Llama 4 + TRTLLM MHA support. • Adds `ResponsesContext` to the gRPC router responses API with improved error propagation. • Adds MHA chunked-prefix optimization that merges prefix and extend KV cache to run MHA in a single pass. • Removes the deprecated `--enable-beta-spec` CLI argument. • Decouples GGUF quantization implementation from the vLLM dependency, and adds `gguf` as a first-class project dependency for CPU/XPU targets. Breaking changes: • The `--enable-beta-spec` argument has been removed; invocations passing this flag will fail on upgrade. https://github.com/sgl-project/sglang/releases/tag/v0.5.4 https://github.com/sgl-project/sglang/releases/tag/v0.5.4 Sun, 26 Oct 2025 02:37:45 GMT SGLang v0.5.4 adds overlap scheduler for speculative decoding, piecewise CUDA graph prefill, new model support, and ModelOpt quantization. • Adds `--sampling-defaults` argument to the OAI serving chat interface to set server-wide sampling parameter defaults for `ChatCompletionRequest`. • Adds `/tokenize` and `/detokenize` OpenAI-compatible endpoints. • Adds reasoning and tool parser argument support in the router. • Adds IPv6 support across all router components. • Adds OpenAI router conversation API CRUD (create, retrieve, delete) for history management using conversation items. • Adds `get server info` and `get model info` methods in the gRPC server. • Adds pipeline parallelism (PP) support for NIXL in prefill/decode disaggregation. • Adds prefix cache support for Qwen3 Next and GDN/Mamba models. • Adds native ModelOpt quantization support. • [beta] Adds overlap scheduler for speculative decoding. • [beta] Adds piecewise CUDA graph support for prefill. • Adds support for passing prefix keys for L3 store in HiCache. • Adds multi-item scoring with custom attention mask via the Generative Score API. • Adds MHA prefill support with FlashAttention 4. • Adds fullset optimizations for DeepSeek-V3.2 including MTP, PD disaggregation, and function calling. • Adds DGX Spark hardware support. • Adds KTransformer integration. • Adds LoRA support in the `bench_serving` OAI interface. • Adds `float64` support to the `moe_sum_reduce` CUDA kernel. • Adds LongBench v2 evaluation utilities. • Adds various Blackwell kernel optimizations. • Adds manifest to versioned Docker releases. • Upgrades FlashInfer to 0.4.0. • Adds dedicated TOML configuration files for CPU and XPU targets. Breaking changes: • The `max_micro_batch_size` argument is renamed to `pp_max_micro_batch_size`. https://github.com/sgl-project/sglang/releases/tag/v0.5.3 https://github.com/sgl-project/sglang/releases/tag/v0.5.3 Mon, 06 Oct 2025 18:45:52 GMT SGLang v0.5.3 adds DeepSeek-V3.2 sparse attention, FlashAttention 4 prefill kernels, gRPC scheduler, and new model support. • Adds a standalone gRPC server for the SGLang Python Scheduler, enabling direct gRPC-based scheduling workloads. • Adds `profile args` support in the Engine API, exposing profiling controls programmatically. • Adds polling interval configuration for prefill-decode disaggregation decode stage. • Adds customer labels specified via request header to metrics, enabling per-request metric segmentation. • Adds rerank routing logic in the regular router. • Adds embedding routing logic in the router. • Allows one router to support different model families and serving modes simultaneously. • Adds Basic OAI Response API to the router, along with `get` and `cancel` methods for the response API. • Integrates FlashAttention 4 prefill kernels for accelerated attention computation. • Supports loading model weights from a remote instance. • Supports offloading in fp8. • Supports `memory_pool_host` page-first direct layout. • Adds Dynamic Batch Tokenizer for improved prefill throughput. • Adds Generative Score API prefill-only scoring optimizations. • Adds fine-grained request latency tracing (SGLang Tracing, Part 1). • Adds latency histogram metrics for each stage of generate requests in PD disaggregation. • Adds `get_load` watching mode and shortest-queue strategy to DP refactor. • Adds split tile size option for Triton attention. • Adds RTX 5880 MoE Triton kernel config. • Supports global scale (in addition to per-expert scale) for CuteDSL MoE. • Supports Qwen3-Next with MTP, DP, optimized kernels, and multiple hardware platforms including Ascend NPU and Blackwell. • Supports new models: Qwen3-VL series, dots.vlm1, Ling-V2, Apertus, SOLAR. • Adds ROCm Docker image support for `sgl-router`. • Supports `pip install sglang` for simplified installation. https://github.com/sgl-project/sglang/releases/tag/v0.5.2 https://github.com/sgl-project/sglang/releases/tag/v0.5.2 Fri, 12 Sep 2025 03:50:52 GMT SGLang v0.5.2 adds HiCache hierarchical KV caching, MCP tool handler, token-bucket rate limiter, and broad new model/parser support. • Adds `response_format` support for the `completion` API, enabling structured output control on non-chat endpoints. • Adds `HiCacheStorageConfig` generic configuration object for managing HiCache storage backends, with default policy changed to write-through. • Adds token-bucket rate limiter to the sgl-router. • Adds MCP Tool Handler to the sgl-router (`[router] Add MCP Tool Handler`). • Adds gRPC router bootstrapping support to sgl-router. • Adds tool parsers to sgl-router for Mistral, Qwen, Llama, Llama 3.2 (including multi-JSON streaming), Pythonic, DeepSeek, Step3, Kimi-K2, GPT-OSS, and GLM4 models. • Adds `reasoning_effort` param to `TiktokenTokenizer.apply_chat_template`. • Supports `nvidia/Llama-3_1-Nemotron-Ultra-253B-v1` model. • Supports `nvidia/gpt-oss-120b-Eagle3` speculative decoding model. • Supports DeepSeek-V3.1 tool call parsing. • Supports speculative decoding in the hybrid attention backend. • Supports NPUGraph for DeepSeek on Ascend NPU. • Supports FIA fusion kernel on the Ascend backend. • Supports CUDA 13.0 and TRT-LLM kernels, including compiling sgl-kernel on CUDA 13.0. • Enables SM100 FP8 kernels on SM103 (Blackwell+) hardware. • Enables hierarchical KV caching (HiCache) on AMD GPUs. • Supports `draft_model` for `RemoteModelLoader` in speculative decoding setups. • Adds `bench_mix.py` benchmarking script for mixed workloads. • Adds ROCm 7.0 support. • Adds A100 fused MoE kernel configs for DeepSeek models. • Adds tuned fused MoE config for GLM-4.5-Air-FP8 at `tp=4` on B200. • Installs `py-spy` by default in container images for easier live profiling. • Improves prefill performance on the CPU backend. • Reduces GPU memory overhead via weakref-based tensor lifecycle management. • Supports `get_model_info` interface for `mini_lb` in the PD (prefill-decode disaggregation) router. • Improves disaggregation metrics output in PD mode to reflect real-time stats. • Supports Mooncake store DP attention and zero-copy MLA metadata. • Adds batch-send optimization from the Tokenizer Manager for improved throughput. • Adds target module validation for adapter initialization. • Accommodates JSON schema in the `schema` field (not only `json_schema`) of `response_format`. • Supports InternVL3.5 series models via updated processor loading. • Updates CUTLASS to 4.2 and enables K-Major Scale Factor for SM90 FP8 Blockwise Group GEMM. • Upgrades FlashInfer to 0.2.14.post1. Breaking changes: • HiCache default caching policy is changed to write-through; deployments relying on the previous default policy behavior will behave differently after upgrade. https://github.com/sgl-project/sglang/releases/tag/v0.5.1 https://github.com/sgl-project/sglang/releases/tag/v0.5.1 Sat, 23 Aug 2025 19:57:46 GMT SGLang v0.5.1 adds NIXL-backed HiCache, C++ radix tree, NCCL symmetric memory TP allreduces, NVFP4 decode kernels, and ModelScope model download. • Adds `lof` as a valid choice for the `--schedule-policy` server argument (was previously missing from the option list). • Adds support for limiting the maximum number of LoRA adapters loaded in CPU memory. • Adds NCCL symmetric memory backend for tensor-parallel allreduces, reducing communication overhead on supported hardware. • Adds SGLang HiCache NIXL Connector, enabling KV-cache offload/transfer via the NIXL transport layer. • Adds support for downloading models from ModelScope as an alternative to Hugging Face Hub. • Adds a C++ implementation of the Radix Tree for prefix caching, replacing the Python implementation. • Adds minimum-token load balancing strategy for data-parallel attention. • Adds low-latency NVFP4 decode kernels from Flashinfer for NVIDIA GPUs. • Adds `cutlass_moe_fp8` kernel support for FusedMoE on SM90 (H100/H200) GPUs. • Enables Flashinfer MoE blockscale FP8 backend for tensor-parallel MoE on NVIDIA hardware. • Adds new MoE Triton kernel tuned for NVIDIA RTX 6000 Ada. • Enables `renormalize=False` in Triton MoE kernels. • Adds TRTLLM-Gen MLA decode kernel integration for multi-latent attention decoding. • Adds DP Attention support for the Step3-VL model. • Extends `dsv3_router_gemm` kernel to support `NUM_EXPERTS == 384` (Kimi K2 architecture). • Updates `dsv3_router_gemm` kernel batch size limit to 16. • Disables Python GC during CUDA graph capture, speeding up capture by up to 3x. • Saves CUDA graph memory by using `next_token_logits_buffer` and FA3 graph optimizations. • Integrates Triton kernels into `sgl-kernel` package. • Upgrades xgrammar dependency to 0.1.22. • Adds HTTP dependency injection pattern to the router system. • Introduces DP worker abstraction in the router. Breaking changes: • `lora_path` is renamed to `lora_id` in batch requests — existing clients passing `lora_path` in batches will break. https://github.com/sgl-project/sglang/releases/tag/v0.4.10 https://github.com/sgl-project/sglang/releases/tag/v0.4.10 Thu, 31 Jul 2025 18:48:25 GMT SGLang v0.4.10 adds dynamic LoRA hot-swap, KV metrics export, hybrid KV cache for LLaMA 4, MTP+two-batch-overlap compatibility, and new RL weight-update controls. • Adds `--model` as an alias for `--model-path` in server args, letting users use the shorter flag name interchangeably. • Adds `--skip-warmup` flag for RL workloads to skip the warmup phase on startup. • Supports dynamic LoRA loading and unloading via the engine and server API at runtime without restarting the server. • Supports `update_weights_from_distributed` with different process groups and multiple simultaneous weight updates for RL training workflows. • Adds KV metrics emission from the SGLang scheduler, enabling external monitoring of cache utilization. • Adds OpenAI chat completions API support in the `bench_serving` benchmarking script. • Adds Kimi reasoning parser and fixes stream reasoning parser for streaming inference. • Supports hybrid KV cache for LLaMA 4 models. • Enables compatibility between Multiple Token Prediction (MTP) and two-batch-overlap scheduling. • Adds Expert Parallelism Load Balancing (EPLB) support for MTP. • Supports different Tensor Parallelism sizes for prefill/decode (PD) disaggregation with non-MLA models. • Adds Tencent HunYuanMoEV1 model support. • Supports EAGLE3 speculative decoding for LLaMA 4. • Adds multi-thread model weight loading for faster startup. • Adds CPU-core and memory-node binding via a new C++ kernel for CPU deployments. • Adds INT8 and FP8 optimizations for DeepSeek models on CPU, calling `fused_experts_cpu`, `weight_packed_linear`, and `bmm_cpu` kernels. • Adds `wna16marlin` kernel for MoE weight-only quantization. • Enables mixed modality processing through refactored multimodal processors. • Adds centralized configuration module for `sgl-router`. • Enables aiter fused MoE, `aiter_biased_grouped_topk`, and FP8 blockscale quantization kernels on AMD hardware. • Adds `dsv3_fused_a_gemm` and DSv3 router GEMM kernels, applied for DeepSeek-R1 FP4. • Adds hidden-states return support at async generation time. • Upgrades FlashInfer to v0.2.7.post1 and sgl-kernel to v0.2.1. • Adds fbgemm benchmark bandwidth reporting and `fbgemm_cutlass_gmm` support in the benchmark tooling. https://github.com/sgl-project/sglang/releases/tag/v0.4.8 https://github.com/sgl-project/sglang/releases/tag/v0.4.8 Tue, 24 Jun 2025 18:43:35 GMT SGLang v0.4.8 adds DeepSeek R1 FP4 on Blackwell GPUs, reranking support, logit bias, hidden states API, and VILA model support. • Moves the OpenAI-compatible API module from `sglang/srt/openai_api` to `sglang/srt/entrypoints/openai`; update imports such as `from sglang.srt.entrypoints.openai.protocol import Tool`. • Adds support for DeepSeek R1 with FP4 quantization and MTP on NVIDIA Blackwell GPUs, integrating FlashInfer NVFP4 MoE with TP, EP, and DP; achieves up to 90 tokens/sec per user on B200 at isl/osl/bs = 1k/1k/16. • Adds support for reranking via the OpenAI-compatible server (`feat/support rerank`). • Adds logit bias support to sampling parameters. • Adds hidden states exposure through the OpenAI-compatible API. • Adds support for VILA vision-language models. • Adds support for Phi-4-mm as a supported VLM. • Adds `gfx950` GPU support to `sgl-kernel`. • Adds support for the new DeepGEMM input format in per-token group quantization, including `silu_and_mul_masked_post_quant_fwd`. • Supports 2-stream shared expert execution for DeepSeek MoE models on Blackwell. • Adds support for separate reasoning in the frontend language (frontend reasoning API). • Refactors the OpenAI-compatible server with consistent metrics, unified error handling, and improved request tracking for production and enterprise environments. Breaking changes: • The `sglang/srt/openai_api` directory has been removed and replaced with `sglang/srt/entrypoints/openai`; any import such as `from sglang.srt.openai_api.protocol import Tool` must be updated to `from sglang.srt.entrypoints.openai.protocol import Tool`. https://github.com/sgl-project/sglang/releases/tag/v0.4.7 https://github.com/sgl-project/sglang/releases/tag/v0.4.7 Wed, 11 Jun 2025 19:14:40 GMT SGLang v0.4.7 adds pipeline parallelism, PD disaggregation with NIXL backend, Qwen3/Kimi-VL/InternVL3/MiMo model support, CUDA graph for LoRA, and full Blackwell GPU support. • Adds `chat_template_kwargs.enable_thinking` to enable thinking mode via the chat template, allowing reasoning/tool-call workflows to be toggled per request. • Supports `max_completion_tokens` parameter for OpenAI Chat Completions API, aligning with the OpenAI spec. • Adds pipeline parallelism support, enabling multi-node inference pipelines across GPU clusters. • Adds NIXL backend for PD (prefill-decode) disaggregation, now supporting Prefill TP and Decode TP+DP configurations. • Supports expert parallelism (EP) MoE for Qwen3, extending large-scale EP to the Qwen3 model family. • Adds Cutlass MLA attention backend as a new attention backend option for MLA-based models. • Supports CUDA graph capture for LoRA adapters, enabling faster LoRA inference. • Adds service discovery for the SGLang router. • Adds structured logging, file-based log output, and log tracing for the SGLang runtime. • Supports automatic chat template selection (`auto chat template`) based on the loaded model. • Automatically sets draft model path for MTP (multi-token prediction), removing the need to specify it manually. • Adds support for Kimi-VL multimodal model inference. • Adds support for InternVL3 multimodal model inference. • Adds support for XiaomiMiMo/MiMo model inference. • Enables full Blackwell GPU support (sm_120) for DeepSeek V3/R1, Llama 4, and Qwen3. • Scales FlashAttention 3 (FA3) kernel to sm8x architectures, broadening hardware support. • Enables overlap scheduler for multimodal models. • Adds support for Qwen-1M context via block sparse attention backend kernel updates. • Adds pythonic tool call support and index field in tool call streaming responses. • Bumps FlashInfer to 0.2.5. • Adds VLM benchmark profiling support. • Adds concurrency evaluation logic in the MMMU benchmark. • Adds MMMU benchmark support for InternVL. • Enhances platform compatibility for ARM (including Thor and Spark boards). • Adds PD fake transfer support for warmup, reducing cold-start latency in disaggregated deployments. • SGLang DeepSeek V3/R1 reaches 190 tokens-per-second on a single H200, a 50%+ improvement over comparable frameworks. https://github.com/sgl-project/sglang/releases/tag/v0.4.6 https://github.com/sgl-project/sglang/releases/tag/v0.4.6 Sun, 27 Apr 2025 21:47:10 GMT SGLang v0.4.6 adds FlashAttention3 as default backend, PD disaggregation with Mooncake/NIXL, FP4 inference, and preliminary Blackwell GPU support. • Adds `--enable-llama4-multimodal` flag to enable Llama 4 multimodal inference support. • Enables FlashAttention3 as the default attention backend for mainstream models (DeepSeek, Qwen, Llama), including FA3 MLA by default on Hopper GPUs. • Adds PD (prefill-decode) disaggregation support with Mooncake and NIXL KV transfer backends. • Adds FP4 weight loading and inference support (NV FP4 precision). • Enables DeepGEMM by default for DeepSeek models, plus additional kernel fusions for improved DeepSeek performance. • Adds preliminary support for NVIDIA Blackwell GPUs, including a Blackwell Dockerfile and Cutlass MLA kernel. • Adds sparse attention kernel to `sgl-kernel`. • Adds `flash_attn_varlen_func` to `sgl-kernel`. • Automatically inspects whether a model is ModelOpt quantized and sets the quantization method accordingly, including ModelOpt KV cache support. • Adds in-queue metrics to the metrics reporting surface. • Adds H20 `dtype fp8_w8a8` fused MoE kernel tuning configs for DeepSeek V3/R1. • Disables grammar restrictions within reasoning sections, allowing structured output to work alongside chain-of-thought reasoning. • Enables `bench_one_batch` to support `enable_dp_attention`. • Adopts fast image processor by default for VLM (vision-language model) inputs. • Supports server-based rollout in VerlEngine for RL training workloads. • Adds Llama 4 FP8 inference support. https://github.com/sgl-project/sglang/releases/tag/v0.4.5 https://github.com/sgl-project/sglang/releases/tag/v0.4.5 Mon, 07 Apr 2025 08:33:25 GMT SGLang v0.4.5 adds Llama 4, FlashAttention 3, EAGLE3 speculative decoding, DeepEP MoE integration, and disaggregated prefill/decode. • Adds support for Llama 4 models (`Llama-4-Scout-17B-16E-Instruct` and `Llama-4-Maverick-17B-128E-Instruct`), achieving zero-shot MMLU Pro scores of 75.2 and 80.7 respectively. • Adds FlashAttention 3 backend for significant acceleration on long-context inference tasks. • Adds EAGLE3 speculative decoding support, delivering substantial gains in decoding throughput. • Integrates DeepEP for enhanced performance in MoE (Mixture-of-Experts) inference. • Introduces a prototype for disaggregated prefill and decoding. • Adds support for Gemma 3 IT (`gemma-3-it`) model. • Adds support for Deepseek-VL2 multimodal model. • Adds support for QwenMoe model. • Adds FlashMLA backend support. • Adds online quantization support for W8A8 (weight 8-bit, activation 8-bit). • Adds support for serving DeepSeek-R1-Channel-INT8 on 32 L40S GPUs. • Adds tensor parallelism and weight slicing support for LoRA adapters. • Adds code completion serving support. • Sets `xgrammar` as the default grammar backend. • Adds hierarchical caching (HiCache) support for MLA (Multi-head Latent Attention). • Adds tool call with text support. • Supports variable remote backend for model loader. • Auto-detects device if not specified in server arguments. https://github.com/sgl-project/sglang/releases/tag/v0.4.4 https://github.com/sgl-project/sglang/releases/tag/v0.4.4 Thu, 13 Mar 2025 18:21:23 GMT SGLang v0.4.4 adds FlashInfer MLA with radix cache, DeepGEMM via `SGL_ENABLE_JIT_DEEPGEMM`, INT8 DeepSeek R1, hierarchical caching, and SageMaker support. • Enables FlashInfer MLA support — fully compatible with radix cache, chunked prefill, and MTP — via `--enable-flashinfer-mla`. • Integrates DeepGEMM for NVIDIA Hopper architectures, enabled by setting `export SGL_ENABLE_JIT_DEEPGEMM=1`. • Adds Multi-Token Prediction (MTP) speculative decoding for DeepSeek-V3/R1 with both Triton and FlashInfer backends, compatible with radix cache and chunked prefill. • Adds hierarchical caching support for SGLang. • Adds SageMaker support. • Adds support for OpenAI API o1 model. • Adds support for Qwen 2.5 VL (vision-language) model. • Adds support for Qwen reward model (Qwen RM). • Adds INT8 quantization support for DeepSeek R1 models (`meituan/DeepSeek-R1-Channel-INT8` and `meituan/DeepSeek-R1-Block-INT8`). • Adds Blackwell architecture Block Scale FP8 GEMM support. • Adds support for page size greater than 1. • Adds nvidia modelopt FP8 KV cache support. • Adds support for RDMA in Docker deployments. • Adds multi-node inference deployment via the LWS method in Kubernetes clusters. • Adds enhanced distributed parallelism support (e.g., two-node configurations with DP 2, TP 16). • Delivers optimized W8A8 FP8 implementation with 15%+ performance improvement on sm89, with gains across sm80, sm89, and sm90 architectures. https://github.com/sgl-project/sglang/releases/tag/v0.4.3 https://github.com/sgl-project/sglang/releases/tag/v0.4.3 Fri, 14 Feb 2025 02:50:47 GMT SGLang v0.4.3 adds Function Calling, regex/EBNF-constrained decoding, custom sampling processors, LoRA in Triton, and 4x long-context speedup via FlashInfer MLA. • Introduces Function Calling capabilities for LLM inference. • Adds regex pattern and EBNF support in the XGrammar backend for constrained decoding. • Adds `update_weights_from_tensor` API for updating model weights at runtime. • Adds Engine.generate() support for returning token IDs alongside generated text. • Enables Flash Attention 3 (FA3) by default for prefill (requires CUDA 12.4, now the default). • Integrates FlashInfer MLA Attention, delivering a 4x performance improvement for long-context DeepSeek V3/R1 inference. • Adds `torch.compile` support for FP8, achieving ~50 tokens/s for online inference with DeepSeek V3/R1. • Implements CUTLASS block-wise FP8 kernels for enhanced inference efficiency. • Implements custom sampling processor support for flexible inference control. • Integrates LoRA support in the Triton backend. • Extends EAGLE 2 speculative decoding support to the FlashInfer and Triton backends. • Supports loading pre-sharded MoE weights. • Enables DeepSeek V3 inference on AMD GPUs. • Upgrades to FlashInfer v0.2. https://github.com/sgl-project/sglang/releases/tag/v0.4.1 https://github.com/sgl-project/sglang/releases/tag/v0.4.1 Wed, 25 Dec 2024 23:27:01 GMT SGLang v0.4.1 adds DeepSeek V3 support, a dynamic-scaling fault-tolerant router with `/add_worker` and remove-worker APIs, MoE Expert Parallel, gemlite quantization, and the new `sgl-kernel` package. • Adds `/add_worker` API to the SGLang native router, enabling dynamic worker registration at runtime. • Adds remove-worker API to the SGLang native router, enabling dynamic worker deregistration and cleanup. • Adds `lora_path` field to chat completion requests and `lora_paths` to `v1_chat_generate_request`, enabling per-request LoRA adapter selection. • Introduces `sgl-kernel` as a standalone PyPI package for custom CUDA/HIP kernels, including a TensorRT-LLM-derived custom all-reduce implementation. • Adds support for DeepSeek V3 (FP8) on both NVIDIA and AMD GPUs, including MLA optimization and DP attention. • Adds MoE Expert Parallel support. • Adds gemlite weight-only quantization integration. • Adds chunked prefill support for LLaVA-OneVision multimodal models. • Adds support for IBM Granite 3.x models. • Adds in-batch prefix caching via delay scheduling. • Adds retry-based fault tolerance to the cache-aware SGLang router. • Adds health-check on workers before they are added to the router. • Adds expanded support for Intel Gaudi accelerators. • Adds ROCm support for `sglang.check_env`. • Adds FP8 MoE optimizations for AMD GPUs. • Optimizes the Triton decoding kernel for long-context inference. • Makes the maximum request payload size configurable. • Makes Tensor Parallelism composable with both `torch.compile` and torchao. https://github.com/sgl-project/sglang/releases/tag/v0.4.0 https://github.com/sgl-project/sglang/releases/tag/v0.4.0 Wed, 04 Dec 2024 02:14:48 GMT SGLang v0.4 adds cache-aware load balancing, xgrammar structured outputs, GGUF support, LoRA in Completion API, and Intel Gaudi accelerator support. • Adds `SGLANG_SET_CPU_AFFINITY` environment variable to control CPU core affinity for multi-socket systems (off by default). • Renames data-parallelism rank environment variable from `DP_RANK` to `SGLANG_DP_RANK`. • Adds `get_server_info` unified API endpoint, merging three previously separate native APIs into one (also exposes SGLang version and `max_total_num_tokens`). • Introduces cache-aware load-balancing router (sglang-router v0.0.10) delivering up to 1.9x throughput increase and 3.8x higher cache hit rate. • Adds more API routes to the router (completion, health, and others previously missing). • Supports LoRA in the Completion API. • Supports GGUF model format loading. • Supports `input_embeds` as input to the engine API. • Adds initial support for Intel Gaudi accelerators. • Integrates xgrammar for structured outputs, delivering up to 10x faster constrained decoding. • Adds the OLMo2 model. • Adds CPU offloading support. • Adds zero-overhead batch scheduler yielding a 1.1x throughput increase. • Adds data parallelism attention for DeepSeek models with up to 1.9x decoding throughput improvement. • Adds prefix match routing for specific tenants in the router. • Adds profiling support in the offline benchmark tool. • Adds `fused_moe_triton` benchmark and tuning tools for AMD Instinct GPUs. • Adds ability to update weights from disk and retrieve weights by parameter name (Llama). • Adds a development Dockerfile. • Exposes `max_total_num_tokens` from the Runtime and Engine API. Breaking changes: • The environment variable `DP_RANK` is renamed to `SGLANG_DP_RANK`; any deployment setting `DP_RANK` must update to `SGLANG_DP_RANK`. https://github.com/sgl-project/sglang/releases/tag/v0.3.6 https://github.com/sgl-project/sglang/releases/tag/v0.3.6 Fri, 22 Nov 2024 11:36:16 GMT SGLang v0.3.6 adds cache-aware load balancing, xgrammar backend, Prometheus metrics, graceful termination, and a Rust-based router. • Adds a cache-aware load balancer delivering 4x higher cache hit rate across multi-server deployments. • Adds `xgrammar` backend for grammar-guided (constrained) decoding, joinable with the existing `outlines` backend. • Adds Prometheus metrics support for observability and monitoring. • Adds graceful termination: server drains in-flight requests before exit on `SIGTERM`. • Adds a watchdog thread to detect and surface server hangs. • Adds a Rust-based router with radix-tree cache-aware load balancing. • Supports data parallelism for attention and MLA, yielding 1.5x higher decoding throughput. • Enables the overlap scheduler by default, delivering 1.1x higher throughput with reduced CPU overhead. • Adds torch-native tensor parallelism support. • Supports PyTorch 2.5.1. • Adds support for new models: Phi3-small, Gemma-2 reward model, and GPT-2. • Adds notebook-style (`.ipynb`) documentation at https://sgl-project.github.io/. • Adds an offline benchmark script. • Adds `get memory pool size` API endpoint. • Allows token IDs to be passed directly in `engine.generate`. • Makes decode log interval configurable. https://github.com/sgl-project/sglang/releases/tag/v0.3.4.post1 https://github.com/sgl-project/sglang/releases/tag/v0.3.4.post1 Tue, 22 Oct 2024 04:30:03 GMT SGLang v0.3.4.post1 adds an offline Engine API, overlap scheduler, async/streaming engine support, and new model/hardware coverage. • Exposes `--num-continuous-decode-steps` as a CLI argument to control continuous decode step count. • Adds an offline `Engine` API for inference without an HTTP server, with reduced overhead and support for async and streaming generation. • Adds `get_tokenizer` function to the `Engine` class for direct tokenizer access in offline workflows. • Adds HTTP API endpoints to enable and disable the PyTorch profiler at runtime. • Adds an overlap scheduler to hide CPU overhead during inference. • Supports `min_tokens` parameter in `sgl.gen` for minimum output length control. • Adds support for reward models. • Adds support for Intel XPU GPUs via PyTorch. • Adds support for new models: Llama 3.2, QWen-VL2, OLMo, and GLM 4. • Accelerates multi-LoRA serving throughput. • Adds `float8` dynamic quantization support via `torchao_utils`. • Adds support for double sparsity attention. • Adds MoE tuning optimizations for AMD MI300x GPUs. https://github.com/sgl-project/sglang/releases/tag/v0.3.2 https://github.com/sgl-project/sglang/releases/tag/v0.3.2 Wed, 02 Oct 2024 17:19:13 GMT SGLang v0.3.2 adds multi-LoRA serving, AMD/ROCm support, torchao quantization, and a new `--attention-backend` flag. • Introduces `--attention-backend` CLI flag to select the attention backend, replacing the deprecated `--disable-flashinfer`. • Adds `constrained_json_whitespace_pattern` to `ServerArgs` for fine-grained control over JSON whitespace in constrained decoding. • Passes `model_override_args` to `launch_server` via the CLI, enabling runtime model config overrides without code changes. • Supports OpenAI API `json_schema` response format for structured output. • Supports response prefill via the OpenAI-compatible API. • Initial support for multi-LoRA serving, including LoRA path renaming. • Integrates torchao quantization (int4, int8, fp8) for Llama, Mixtral, and Qwen MoE models. • Enables `torch.compile` and CUDA graph support for the Triton attention backend, including DeepSeek MLA. • Enables Multi-head Latent Attention (MLA) by default for DeepSeek-v2. • Adds AMD GPU support via PyTorch for ROCm backend. • Adds new model support: MiniCPM3, OLMoE, BaiChuan2, XVERSE (Dense and MoE), and SmolLM. • Adds `bench_server_latency.py` for server latency benchmarking. Breaking changes: • The `--disable-flashinfer` flag is deprecated and replaced by `--attention-backend`; existing scripts using `--disable-flashinfer` will need to be updated. https://github.com/sgl-project/sglang/releases/tag/v0.3.0 https://github.com/sgl-project/sglang/releases/tag/v0.3.0 Thu, 19 Sep 2024 10:09:00 GMT SGLang v0.3.0 adds MLA throughput gains, torch.compile, fp8 KV cache, min-p sampling, JSON constraints, and live weight updates. • Adds `--disable-custom-all-reduce` flag to opt out of custom all-reduce kernels. • Adds fp8 e5m2 KV cache support via both FlashInfer and Triton backends for reduced memory footprint during inference. • Adds `min-p` sampling parameter for nucleus-style sampling with a minimum probability floor. • Enables JSON-constrained generation (structured output) usable directly from the `gen` call via `json_schema`. • Adds `/v1/health` endpoint that validates the server by generating a single token. • Supports live weight updates without restarting the server via the new update-weights API. • Supports batch cancellation requests. • Supports optional start length for logprobs via new `logprobs` sampling parameter. • Enables streaming for multi-prompt and parallel sampling requests. • Chunked prefill is now on by default, with support for separate or mixed prefill-and-decode modes. • Adds LLaVA-OneVision model support, including SigLIP encoder, Qwen2 decoder, interleaved text and multi-image/video input, and OpenAI-compatible API server. • Adds EXAONE 3.0 model support. • Adds Phi-3 Mini and Medium model support. • Adds Alibaba-NLP/gte-Qwen2-7B-instruct embedding model support. • Delivers up to 7x higher throughput for DeepSeek Multi-head Latent Attention (MLA) via optimized Triton decoding kernels and offline weight transposition. • Delivers up to 1.5x lower latency with `torch.compile` on small batch sizes. • Extends Gemma-2 context length 2x via interleaved window attention support. • Adds SM75 (Turing) GPU support via FlashInfer v0.1.6. Breaking changes: • `model_overide_args` is renamed to `model_override_args`; any config or code referencing the old spelling will break. https://github.com/sgl-project/sglang/releases/tag/v0.2.13 https://github.com/sgl-project/sglang/releases/tag/v0.2.13 Thu, 19 Sep 2024 10:08:50 GMT SGLang v0.2.13 adds window attention for Gemma-2, embedding model support, all sampling penalties, chunked-prefill by default, and 2x MLA throughput for DeepSeek v2. • Adds `skip_tokenizer_init` argument to bypass tokenizer initialization for advanced serving setups. • Adds sampling penalties — `frequency_penalty`, `presence_penalty`, `repetition_penalty`, and `min_new_tokens` — completing OpenAI-compatible penalty support. • Enables chunked-prefill by default, improving throughput without requiring manual configuration. • Supports the e5-mistral embedding model with a fully OpenAI-compatible embeddings API, including accepting input as a list. • Adds window attention support for Gemma-2 models. • Accelerates Multi-head Latent Attention (MLA) for DeepSeek v2 via a Triton kernel, delivering a ~2x end-to-end throughput improvement. • Supports loading models from `www.modelscope.cn` in addition to HuggingFace. • Adds alternative choices selection methods for constrained generation. • Uses FlashInfer kernels for rmsnorm and silu operations, reducing per-token overhead. https://github.com/sgl-project/sglang/releases/tag/v0.2.9 https://github.com/sgl-project/sglang/releases/tag/v0.2.9 Fri, 02 Aug 2024 08:55:00 GMT SGLang v0.2.9 adds chunked prefill, DeepSeek v2 support, AWQ Marlin quantization, and logprobs in the OpenAI chat API. • Adds `--max-total-tokens` CLI argument to control the total token budget for the server. • Adds `--disable_ignore_eos` support in `bench_serving.py` for benchmarking scenarios that require EOS suppression. • Adds `served_model_name` to customize the model ID when running in local mode via the OpenAI-compatible API server. • Renames response fields: `prefill_token_logprobs` is now `input_token_logprobs`; `decode_token_logprobs` is now `output_token_logprobs`. • Adds chunked prefill support, including operation when radix cache is disabled. • Adds support for logprobs in the OpenAI chat completion API. • Adds support for offline batch (file) processing via the OpenAI API interface. • Adds AWQ Marlin quantization support (`awq_marlin`). • Adds DeepSeek v2 model support. • Adds chat template for InternLM2-Chat. • Vectorizes logprob computation for a performance improvement in log-probability workloads. Breaking changes: • `prefill_token_logprobs` is renamed to `input_token_logprobs` and `decode_token_logprobs` is renamed to `output_token_logprobs` — any client parsing these field names will break on upgrade. https://github.com/sgl-project/sglang/releases/tag/v0.2.0 https://github.com/sgl-project/sglang/releases/tag/v0.2.0 Thu, 25 Jul 2024 15:58:24 GMT SGLang v0.2.0 adds Llama3 405B, Deepseek MoE, Mistral-Nemo, GPTBigCode, and InternLM2 support with OpenAI parallel sampling and VertexAI safety settings. • Adds support for VertexAI safety settings in API calls. • Adds OpenAI API parallel sampling support. • Adds new model support for Llama3 405B (including FP8), Deepseek MoE, InternLM2, GPTBigCode, and Mistral-Nemo. • Adds incremental detokenization when streaming, reducing time-to-first-token (TTFT) and inter-token latency (ITL) at `stream_interval` 1. • Adds benchmark serving tooling (`bench_serving.py`) supporting random datasets, non-streaming mode, and TensorRT-LLM comparison benchmarks. • Delivers superior or competitive throughput versus TensorRT-LLM and vLLM across Llama-8B to Llama-405B on A100 and H100 GPUs in both FP8 and FP16. https://github.com/sgl-project/sglang/releases/tag/v0.1.20 https://github.com/sgl-project/sglang/releases/tag/v0.1.20 Sun, 14 Jul 2024 00:33:05 GMT SGLang v0.1.20 enables CUDA graphs by default, adds Gemma2/MiniCPM/Qwen2 MoE support, and ships Docker images. • Adds `--enable-p2p-check` option to the CLI for peer-to-peer connectivity validation. • Exposes logprob options to the `sgl.gen` API via new `LogitsMetadata` support. • Enables CUDA graph execution by default, delivering 1.5x–2x speedup for small-batch decoding. • Adds model support for Gemma2, MiniCPM, and Qwen2 MoE. • Adds Docker support with a new Dockerfile. https://github.com/sgl-project/sglang/releases/tag/v0.1.18 https://github.com/sgl-project/sglang/releases/tag/v0.1.18 Thu, 04 Jul 2024 06:35:42 GMT SGLang v0.1.18 adds multi-node tensor parallelism, 2x large-batch prefill throughput, ChatGLM support, and new CLI arguments. • Adds `--log_level_http` argument to control HTTP request logging independently of the main log level. • Adds `--dtype` argument to expose model dtype selection at server launch. • Adds `sglang.bench_latency` module for offline latency benchmarking. • Enables flashinfer kernels by default, delivering a 2x throughput improvement for large-batch prefill workloads. • Supports multi-node tensor parallelism, enabling model serving across multiple machines. • Adds ChatGLM model support. https://github.com/sgl-project/sglang/releases/tag/v0.1.17 https://github.com/sgl-project/sglang/releases/tag/v0.1.17 Sat, 08 Jun 2024 02:58:55 GMT SGLang v0.1.17 adds data parallelism, speculative execution for OpenAI API, new model support, and a LiteLLM backend. • Adds `num_api_spec_tokens` parameter enabling speculative execution for the OpenAI chat API. • Adds `PUT` method support to the generate API endpoint. • Adds `finish_reason` field to OpenAI API responses. • Supports static data parallelism for serving. • Adds LiteLLM backend integration. • Ports FP8 Mixtral quantization support via vLLM v0.4.3 upgrade, enabling new quantization features. • Adds LLaVA-Qwen and LLaVA-Mistral model support. • Adds logit cap to improve generation control. Breaking changes: • The parameter `api_num_spec_tokens` is renamed to `num_api_spec_tokens`. https://github.com/sgl-project/sglang/releases/tag/v0.1.16 https://github.com/sgl-project/sglang/releases/tag/v0.1.16 Tue, 14 May 2024 00:36:05 GMT SGLang v0.1.16 adds DBRX, Command-R, Gemma, and LLaVA-Video support alongside Marlin quantization kernels and cache optimizations. • Adds `spaces_between_special_tokens` argument to `SamplingParams` for finer control over token output formatting. • Allows `input_ids` to be passed directly in the body of the `/generate` endpoint, enabling token-level input without text encoding. • Includes `finish_reason` in the meta info response returned by the inference API. • Adds support for DBRX, Command-R, and Gemma model architectures. • Adds support for LLaVA-Video multimodal inference. • Enables Marlin quantization kernels for faster quantized model inference. • Adds Llama 3 instruct chat template. • Adds Cohere Command-R chat template. • Optimizes radix tree prefix matching for improved cache hit performance. • Reduces memory usage of the logits processor. https://github.com/sgl-project/sglang/releases/tag/v0.1.13 https://github.com/sgl-project/sglang/releases/tag/v0.1.13 Mon, 11 Mar 2024 12:52:58 GMT SGLang v0.1.13 adds Gemma model support, API key auth, SSL, and Together/AzureOpenAI backend examples. • Adds API key authentication support to the SGLang server via `Add Support for API Key Authentication`. • Adds SSL certificate functionality to the server for encrypted transport. • Adds `set_var` method to `interpreter.py` for programmatic variable assignment in the runtime interpreter. • Adds `completion_tokens_wo_jump_forward` field to decoding logprob output, exposing token counts that exclude jump-forward tokens. • Adds ability to modify the context length at server launch via `server_args`. • Replaces `skip_embed` with `input_embeds` in the API surface for passing raw embeddings. • Adds Gemma model support. • Adds Together and AzureOpenAI backend usage examples. Breaking changes: • `skip_embed` is replaced by `input_embeds` — any code passing `skip_embed` will break. https://github.com/sgl-project/sglang/releases/tag/v0.1.12 https://github.com/sgl-project/sglang/releases/tag/v0.1.12 Sun, 11 Feb 2024 14:49:18 GMT SGLang v0.1.12 adds fast JSON decoding, decode token logprobs, GPTQ support, and a `--disable-disk-cache` flag • Adds `--disable-disk-cache` flag to the SRT server to explicitly turn off on-disk KV cache persistence. • Adds auth token support to `RuntimeEndpoint` for authenticated inference endpoints. • Supports decode token logprobs, enabling callers to retrieve per-token log-probabilities from decoding. • Adds GPTQ quantized model support, allowing inference on GPTQ-quantized weights. • Supports `regex` extra field in the OpenAI-compatible API for constrained decoding via regular expressions. • Fast JSON decoding (compressed FSM / jump-forward) now supported for LLaVA models in addition to base LLMs. • Adds server warmup on SRT server startup to reduce cold-start latency on first requests. • Adds OpenAI error handler with automatic retry and logging for more resilient API calls. https://github.com/sgl-project/sglang/releases/tag/v0.1.11 https://github.com/sgl-project/sglang/releases/tag/v0.1.11 Sat, 03 Feb 2024 10:57:53 GMT SGLang v0.1.11 adds LLaVA 1.6, Yi-VL, QWen2 support, faster JSON decoding, flush cache API, and a health endpoint. • Adds `max_prefill_num_token` server argument to control prefill batch size limits. • Adds a flush cache API endpoint to the SGLang runtime server. • Adds a health endpoint to the SGLang runtime server. • Supports specifying all ports in advance via server arguments. • Supports LLaVA v1.6 (including HD variant) and fine-tuned LLaVA model loading. • Supports Yi-VL multimodal model. • Supports QWen2 model. • Adds image content support in OpenAI-compatible chat routes. • Returns logprob for choices in API responses. • Adds cache metrics reporting. • Supports speculative execution for the OpenAI API backend. • Faster JSON decoding via compressed finite-state machine approach. • Dynamic model class loading for extensibility. • Improves streaming control and reduces first-token latency in streaming mode. https://github.com/sgl-project/sglang/releases/tag/v0.1.6 https://github.com/sgl-project/sglang/releases/tag/v0.1.6 Sun, 21 Jan 2024 10:09:46 GMT SGLang v0.1.6 adds an OpenAI-compatible API server with streaming Completions and ChatCompletions endpoints. • Adds `v1/chat/completions` endpoint for OpenAI-compatible ChatCompletion API support. • Adds `stream=True` support to the `v1/completions` endpoint for streaming token output. • Adds disk-based FSM cache to persist finite-state-machine state across runs. • Increases interpreter parallelism for higher throughput during multi-call workloads. • Adds a LLaVA multimodal example and a Vicuna chat template. https://github.com/sgl-project/sglang/releases/tag/v0.1.5 https://github.com/sgl-project/sglang/releases/tag/v0.1.5 Thu, 18 Jan 2024 02:40:44 GMT SGLang v0.1.5 adds a VertexAI backend, metadata in async streaming, and expanded scheduling controls. • Adds a VertexAI (formerly Gemini) backend, enabling SGLang programs to target Google's VertexAI models. • Adds an option to return metadata in async streaming responses. • Exposes additional arguments to control the scheduling policy. • Adds memory fraction tuning to give operators finer control over GPU memory allocation. Breaking changes: • The `image_url` field is renamed to `image_file`; any existing code or configs that reference `image_url` will break.