SGLang v0.5.19

v0.5.19
Added 16
  • Support for beam search with configurable beam_width parameter to return the n best sequences
  • DeepEP v2 ElasticBuffer engine available as --moe-a2a-backend deepep_v2 for DeepSeek-V3/V4 and Qwen3-MoE in FP8
  • LayerNorm sequence parallelism with --enable-layernorm-sp flag to reduce prefill latency for dense Qwen3 models
  • W4A8 MoE quantization on Hopper with --flashinfer-mxfp4-moe-precision fp8 option
  • Decode context parallelism support on trtllm_mla default Blackwell MLA backend
  • Opt-in fused accept path for KDA models with SGLANG_OPT_KDA_FUSED_ACCEPT_STATE=1 environment variable
Changed 2
  • Unified radix tree is now the default cache for all models instead of only hybrid models
  • DSA prefill top-k kernel moved to v2 version for improved performance on B200
Fixed 2
  • PD decode workers can now reuse cached prefixes for SWA hybrid models like gpt-oss
  • Pipeline parallelism with HiCache L3 now stays consistent across ranks

From SGLang

Highlights

786 PRs from 214 contributors.

New models in this release (see the cookbook for all supported models):

ModelTypePRsCookbook
Qwen3.8 (2.4T-A95B)Autoregressive#35758 ⭐link
Qwen3.8-27BAutoregressive#34859link
dots3.noteAutoregressive#33829 ⭐link
Ling-3.0-flashAutoregressive#33561link
Ling-3.0-tinyAutoregressive#33561link
Spark2.5Autoregressive#35963 ⭐
MiniCPM-SALAAutoregressive#30360
Granite 4.2Autoregressive#36286link
LongCat-Image-Edit & Edit-TurboDiffusion#35829

Cookbook updates:

  • GLM-5.3 deployment guide: link
  • PaddleOCR-VL deployment guide: link
  • Kimi-K3 on Ascend A3: #35508
  • Kimi-K2.7-Code-MXFP4 on MI355X: #36246
  • Qwen3.5 MXFP4 on MI355X with an FP8 KV cache or a HiCache host-memory tier: #35445, #36245
  • MiniMax-H3 on a 24 GB GPU or DGX Spark, with a consumer-GPU tuning guide: #36169, #35816
  • Ling-3.0-flash on DGX Spark: #36364
  • Qwen3.8-27B on RTX 5090, RTX PRO 6000, and DGX Spark, re-measured: #35825

Beam search. SGLang can now do beam search. Pass beam_width in your request and you get back the n best sequences instead of a single sample. It works out of the box next to regular requests, though it does not yet mix with speculative decoding, disaggregation, DP attention, or HiCache (#31626).

DeepEP v2. DeepEP's new ElasticBuffer engine is available as --moe-a2a-backend deepep_v2 for DeepSeek-V3/V4 and Qwen3-MoE in FP8. Its buffers have a fixed size, so decode can run under CUDA graphs even across nodes. Performance is on par with the classic backend (#35634, #34923).

LayerNorm sequence parallelism. With --enable-layernorm-sp, each tensor-parallel rank normalizes only its own share of the prefill tokens instead of all of them. That takes 3.5% off Qwen3-8B prefill on H100 and 5.6% on B200, and the saving grows with the TP degree. Dense Qwen3 models only for now (#30915).

W4A8 MoE on Hopper. If you serve MXFP4 experts on Hopper, you can now quantize the activations to FP8 as well with --flashinfer-mxfp4-moe-precision fp8. DeepSeek-V4-Flash gains about 12% output throughput with no change in GSM8K accuracy. Needs FlashInfer 0.6.18 (#34967).

DCP on the default Blackwell MLA backend. Decode context parallelism now runs on trtllm_mla, not just CuTe DSL and Tokenspeed. It pays off at long context: at 128K input, plain TP stops scaling around 680 tokens per second on eight B200s, while DCP keeps going as concurrency grows (#33926).

Faster speculative kernels. DSA prefill top-k moves to the v2 kernel, 1.3 to 1.8 times faster on B200 (#35175). KDA models get an opt-in fused accept path, SGLANG_OPT_KDA_FUSED_ACCEPT_STATE=1, that cuts MTP verify-and-commit time by 45% to 63% on Kimi-Linear shapes with bit-identical output (#33722).

Unified radix tree by default. The unified tree is now the cache for every model, not just hybrid ones (#35081, see Breaking Changes). It also picked up three things this cycle: PD decode workers can reuse cached prefixes for SWA hybrid models like gpt-oss (#27770), you can attach or detach L3 storage on a running server (#35269), and pipeline parallelism with HiCache L3 stays consistent across ranks (#27010).

Lean attention on AMD. Long or uneven decode batches used to leave many compute units idle on MI300X and MI355X. The new persistent Lean kernel spreads the work across all of them, for up to 1.52x more throughput and up to 3.62x lower inter-token latency on MI355X. It turns on by itself where it helps, and SGLANG_DISABLE_LEAN_ATTENTION=1 turns it off (#33576).

DSA models on ROCm. Disaggregated GLM-5.2 serving now uses the fused top-k seed remap, which brings decode TPOT from 23 ms down to 8 ms on eight MI355Xs (#36714). DeepSeek-V4 gets the v2 top-k kernel, up to three times faster (#36684), and a shared-experts gate fix gives GLM-5.2 up to 16% better TPOT (#36124).

Dependencies. FlashInfer moves to 0.6.18 (#36954), sgl-deep-ep to 0.1.2 (#35450), sgl-deep-gemm to 0.1.7 (#37279), and mooncake to 0.3.13 (#36493). There is a new CUDA 13.4 preview image for Rubin (#36233) and new ROCm 10 images for gfx942, gfx950, and gfx1250 (#36434, #36871).

Full release notes by category below; breaking changes and known issues are at the end.

Rust Server
  • [Feature] Add process-local in-memory KV indexer and Router integration: #33370
  • [Rust Server] Add e2e latency metadata and fix Sarashina import: #35125
  • [Rust] Derive server address and accept signed env values: #37221
  • [Rust] Keep sampling and scheduler wire schemas in sync: #37222
  • Add configurable HTTP/2 connection window: #36920
  • fix(gateway): bump wfaas to 1.0.2 so ContinueNextStep unblocks dependents: #37249
Speculative Decoding
  • [KDA] Fused-accept state advance for FlashInfer KDA MTP verify: #33722 ⭐
  • [Spec] DFlash2: local convolution + candidate selector (3.43x over no-spec at batch 1 and about 24% over DFlash at concurrency 64): #35371
  • [Spec] Support quantized target lm_head in the DFlash2 selector: #35496
  • [Spec] Add LFM2 and LFM2-MoE DSpark speculative decoding support (1.05 to 2.42x faster decoding on LFM2.5 targets, 1xH100): #31041
  • [Model] Support Nemotron 3.5 Lightning speculative decoding (GSM8K 94.6% to 95.8% across MTP, DFlash, and DSpark on GB300): #36186
  • [Spec][LoRA] Support multi-adapter LoRA with EAGLE/NEXTN/DFLASH/DSPARK speculative decoding: #34337
  • [Spec][DSA] Add --speculative-dsa-topk-backend: #36313
  • [2/N][Mixed] Mixed chunk prefill with spec enabled: #36933
  • Support custom draft worker classes in DSpark: #35397
  • Make draft attention backends extensible: #35932
  • [Memory] Borrow CUDA graph pool storage for EAGLE sampling: #35375
  • [Spec] Fix Dspark and Dflash state divergence across TP rank: #33614
  • [Fix][Spec] fix startup crash and reduce CUDA graph memory usage for speculative adaptive: #35275
  • Fix DSV4 DSpark sample-from-anchor initialization: #36419
  • [Fix] Drop the duplicated DSpark draft sample_block call: #36934
  • fix(gemma4): quantize MTP bridge projections (Gemma-4 FP8 MTP acceptance from 0% to 60%): #32440
Piecewise & Breakable CUDA Graph
  • [Feature] Support PP in full prefill CUDA graphs: #35451
  • [Feature] Coordinate FullCG prefill across DP-attention ranks: #35640
  • [PP] Support prefill CUDA graph proxy tensors (PP prefill up to 2.48x at 2K-token forwards on Qwen3.5-397B, GB300): #36248
  • [Performance] Reduce idle DP work in breakable prefill CUDA graphs (median TTFT -18.9%, QPS +12.9%): #33871
  • [Cohere Command-A-Plus] Optimize decode and BCG capture on SM10X (decode TPOT 2.28x faster at batch 1, up to 9,152 tok/s on 4xGB300): #36624
  • [Fix] Fix full prefill CUDA graph padding and EAGLE capture: #35588
  • Converge the two SWA predicates, and stop conditioning the capture sink on the pool: #37550
Attention Backends
  • [DCP] Support decode context parallelism on the trtllm_mla decode path: #33926 ⭐
  • [DSA] Route the ragged prefill top-k to the v2 kernel: #35175 ⭐
  • [DSA] Trim top-k v2 output modes and tighten its PDL waits: #35041
  • Add fmha_v2 attention backend for SM90/120 (about 15% faster than FA3 at the kernel level): #23112
  • [FlashInfer 0.6.18] feat(dsv4): support --dsa-topk-backend flashinfer with fused top-k: #33237
  • [DeepSeek-V4] Add Q8KV8 sparse MLA prefill runtime backend (prefill throughput +4.4% to +7.5% from 3.5K to 160K input on H20): #32327
  • Split TRTLLM MHA decode batches by KV sequence length (mixed-length decode step -10.2% on GB300): #34888
  • [Performance] Tune FlashInfer EXTEND for DP prefill (DP prefill throughput +4.9% to +6.8%): #36219
  • [Triton] Bound the sliding-window extend-attention KV loop: -86.6% on SWA layers, -9.4% prefill GPU, bit-identical: #34462
  • [Kernel] Skip reserved writes in MLA KV cache: #36003
  • [Fix] Skip padded state slots in the chunked GDN kernel: #33431
  • [Fix] Fix Nemotron-H Mamba illegal memory access under DP attention with CUDA graph: #34561
MoE & Expert Parallelism
  • [Feature] Add DeepEPv2 (ElasticBuffer) MoE A2A backend: #35634 ⭐
  • Apply latest DeepEP branch: #34923 ⭐
  • [MoE] Add FlashInfer SM90 MXFP4 W4A8 CUTLASS MoE (W4A8 kernel 1.63 to 2.08x faster than W4A16, +11.7% output throughput on DeepSeek-V4-Flash): #34967 ⭐
  • [DeepSeek V4] Add W4A4 MegaMoE server flag: #35918
  • [DeepSeek V4] Default FP4 checkpoints to FlashInfer MXFP4 MoE: #35919
  • [DeepSeek-V4] Enable shared-experts fusion on the flashinfer_mxfp4 (trtllm-gen) MoE path (TTFT -13% to -21%, warmed GSM8K e2e +5-7%): #35505
  • Support deepseek v4 and kimi k3 on ssd (DeepSeek-V4-Flash prefill 2.24x and decode 1.49x faster than llama.cpp on one 5090): #35314
  • perf: overlap Qwen shared expert with DeepEP routed experts (+2.1% output throughput on 24xGB300 disaggregated Qwen3.5-397B): #34938
  • feat(humming): FP8 DeepEP dispatch for humming MoE backend (TPOT -35% and TTFT -23% vs BF16 dispatch on H20): #31429
  • feat(humming): support native W4AFP8 checkpoint schemas (TPOT -22% to -29%, TTFT -10% to -11% on H20): #32033
  • [MoE] Make simulated expert routing support DP>1, and fuse into one triton kernel: #29718
  • [Blackwell] Reserve SMs for DeepGEMM MegaMoE grid barriers: #36657
  • [DeepGEMM] Deduplicate JIT precompile across local ranks: #36231
  • [Perf] Tune the W4AFP8 DeepEP low-latency requant launch geometry (requant kernel up to 10.6x faster at 256 experts): #35760
  • [Kimi K3] Select FlashInfer MXFP4 for SM107 auto MoE: #35554
  • [Fix] Route the Mooncake MoE A2A backend through Kimi K3's EP-A2A / SP-MoE fast path: #36862
  • Laguna: config-driven MoE router scoring: #35362
  • Sync FlashInfer autotune tactic choice across TP ranks: #35343
  • Fix OOB read in mxfp4 MoE weight scales on Hopper: #36456
  • Fix stale GLM MoE routing after runtime weight updates: #35883
Quantization
  • [FlashInfer 0.6.18] add FlashInfer CuTe DSL NVFP4 W4A16 mode (Qwen3-30B-A3B W4A16 GSM8K 0.965 to 0.980, 1,662 to 1,858 tok/s): #35120
  • [Quant] Load compressed-tensors quantized lm_head instead of value-casting it: #35228
  • [Quant] Load compressed-tensors kv_cache_scheme scales: #35455
  • Support mxfp8 KV cache in PD transfer: #35718
  • Support CPU offload for mxfp8 KV cache: #35888
  • [Kernel] Fix SM90 FP8 decode regression with benchmarked M/K/N routing (restores about 215 tok/s decode from 190 after the SM90 FP8 routing regression): #37018
  • [NVIDIA] Fix SM107 MXFP8 activation prep: #35405
  • Fix GPU kernel ordering and MXFP8 quantization dispatch: #37331
  • fix(kimi-k3): preserve dense ModelSlim MLA weights: #36603
  • Support Intern-S2-Mobius FP8: #34908
Parallelism & Disaggregation
  • [Feature] Megatron LayerNorm sequence parallelism (--enable-layernorm-sp): #30915 ⭐
  • [PD] Decode-side radix cache for SWA hybrid models (unified radix tree): #27770 ⭐
  • [Refactor] New EPD: #30398
  • [PD] Pack DCP1→DCP-N PD KV transfers into dest-contiguous RDMA blocks (PD transfer time -28% to -39% on Kimi-Linear-48B, 8xB300): #35762
  • [PD] Deferred decode-side KV release for aborts mid-transfer: #35049
  • [PD] Deferred decode-side KV release for the NIXL backend: #35360
  • [PD] Overlap prefill DP-rank bootstrap queries: #35071
  • [PD] Fix MORI-IO ABORT bootstrap message handling: #29133
  • [Fix] Assert the page-aligned SWA evict floor on both PD decode prealloc paths: #35396
  • fix(disagg): refresh stale prefill bootstrap metadata: #36029
  • [Fix] DP attention: correct the decode->extend prefix off-by-one (DP attention GSM8K back from 70.4 to 94.2 on Qwen3-30B-A3B): #37505
  • [Fix] Select custom all-reduce v2 by topology capability: #35061
  • [Kernel] Split the custom all-reduce communicator into push/pull planes: #35735
  • TP/PP Consensus checker: #34406
  • [CP V1 Deprecation 1/5] Migrate tests to strategy-based prefill CP: #36222
Scheduler & Runtime
  • [Feature] Beam search support: #31626 ⭐
  • [1/N][Mix] Mixed Chunk Prefill Base: #36288
  • [Scheduler] Add configurable decode interval after prefill: #35017
  • Publish per-scheduler load on a dedicated socket for load-aware routers: #34608
  • [Fix] Fix min-new-token EOS handling: #31378
  • fix: preserve output logprobs without input logprobs: #34627
  • [Sampling] Restore finite top-k requirement for sampling masks: #35205
  • [Sampling] Fix int32 offset overflow in top-k renorm Triton kernels: #35571
  • Handle unlimited tokenizer context lengths: #37054
  • [Constrained] Support MistralCommon tokenizers in the XGrammar backend: #35215
  • [OpenAI] Drop empty assistant turns for mistral_common tokenizers: #35915
  • [Fix] Pass Anthropic thinking history as reasoning_content for custom chat encoders: #35480
  • [Fix] Resolve tool argument types through top-level anyOf/oneOf/allOf: #36626
  • feat(openai): Accept the input_audio content part in chat completions: #33606
  • Make the scheduler track the published weight version: #35925
  • Report per-token weight-version spans in generation meta info: #35926
  • [FEAT] Weight Daemon abstraction: #33279
  • [Weight Cache] Support static DP/EP layouts: #33684
  • [Fix] Account resident weight memory in KV sizing: #34053
  • Fix KV cache pool sized far too small when weight-loading memory is still referenced: #36583
  • Profiling Enhancements [2/3]: detailed execution step annotations: #24911
HiCache & Radix Cache
  • Using unified radix tree by default for all case: #35081
  • [Unified Tree] feat: support runtime attach/detach for historage: #35269 ⭐
  • [HiCache] Fix PP inconsistency with HiCache L3 (#22607): #27010 ⭐
  • [Radix Cache] Add Rust TreeCore backend with shared parity tests: #32710
  • [Unified Cache][1/N]: Support cache contract for external linker: #37091
  • [Unified Cache][4/N]: Add Mooncake backend for external linker: #37205
  • [HiCache] Buffer-only mode for HiCache host memory layer: #34798
  • [HiCache] buffer mode: decide staged-fetch fate against the live tree: #36834
  • Fix buffer-mode HiCache load-back ownership races; add optional prefetch anchor lock: #35769
  • [HiCache] Batch PP write and load completion sync: #33473
  • [HiCache] Stop populating host-pool mmaps twice (-13% allocation time): #36705
  • [HiCache] Retry L3 storage prefetch after a missed attempt: #36227
  • [HiSparse] Support hisparse multi-step swap io kernel (batch-64 swap latency -48.2% on H20): #32162
  • feat(unified-memory): dense KV views for uniform-row MHA/SWA models: #34602
  • feat(unified-memory): read unified pool from attention backends fa3/flashinfer/trtllm_mha/flashmla: #34613
  • feat(unified-memory): three sub-pools for mamba + hybrid-SWA models: #35177
  • feat(unified-memory): byte-budget sizing, feasibility floor, and a conservation verifier: #35158
  • [Unified Memory] Stop eviction when shared allocation capacity is sufficient (cache survival 35.7% to 67.9%, replay E2E -37.9%): #33091
Multimodal
  • [Perf] PaddleOCR-VL: overlap page preprocessing, pack the ViT, enable prefill CUDA graph (1080p page TTFT 235 ms to 120 ms, 1.84x req/s on H200): #35318
  • [VLM] Route every multimodal processor through the worker pool's call site: #35342
  • [VLM] Split Pixtral multi-image features before the CUDA IPC wrap: #35463
  • Scatter mm embeddings with row index_copy_ instead of masked_scatter_ to cut transient GPU memory: #37070
  • [Fix] Re-encode multimodal embeddings after cache mismatch: #36595
  • [Fix][VLM] keep Qwen3-VL MoE inference deepstack order: #34690
  • [Kimi-K3] Fix "wrong grids" crash in DP-sharded vision preprocessing: #35305
  • Fix transcription & audio-understanding for ASR/audio/speech models: #32611
Model Support & Optimizations
  • Add SGLang Granite SWA support via existing Granite models: #35794
  • [Fix] Transformers-fallback (GPT-NeoX) + KV pool config (DeepSeek-VL2): #35244
  • Fix BailingMoeV3 reading enable_dp_lm_head off live topology instead of config: #36584
  • [k3] declare packed_modules_mapping on KimiK3ForConditionalGeneration: #36211
  • [Fix][Qwen3.8] Support Qwen3.8-MXFP4 DCP by registing Qwen3_5 text-only archs in mamba radix cache whitelists: #35297
  • Fix DeepSeek V4 multistream QKV buffer lifetime: #36547
Kernel Library
  • [Kernel] Add H200 MoE configs for Qwen3.5 and Qwen3.6 (+3.8% throughput on Qwen3.5 FP8 and +7.1% on Qwen3.6 FP8, H200): #35374
  • [Kernel] Add GB300 Triton MoE configs for GLM-4.5 FP8 (decode throughput +21.9% on GB300): #37159
  • [Kernel] Raise shape limits in shared FLA and MoE kernels (ported from #36507): #37317
  • [Kernel] Declare the PyTorch ABI dependency in sglang-kernel wheels: #36465
Configuration System
  • [Core] Refactor server argument choices: #36586
  • config: constructing a config no longer resolves it: #35907
  • config: ServerArgs holds the raw input: #36255
  • config: resolution declares, and nothing writes a field: #36618
  • [Config] Round 5.1: the published-side readers ask the bags, and a platform fact gets one address: #37086
  • [Config] Round 5.2: the per-model declarations get their own modules: #37087
SGLang-Diffusion
  • [Diffusion] Optimizing MiniMax-H3 for consumer-level GPUs: INT8 Linear + pluggable DiT attention backends (MiniMax-H3 on one 24 GB GPU, denoise 1.34x faster with exact attention and up to 2.48x with approximate attention): #34581
  • [Diffusion] feat: load GGUF transformer checkpoints (MiniMax-H3): #35370
  • [Diffusion] Support MiniMax-H3 pruned safetensors checkpoints: #35418
  • [Diffusion] Load Comfy NVFP4 MiniMax H3 checkpoints: #36044
  • [Diffusion] Load MiniMax H3 GGUF text encoders: #36055
  • [Diffusion] Load Diffusers MiniMax H3 components natively: #36067
  • [Diffusion][MiniMax H3] support subblock sparse attention on SM90: #34680
  • [Diffusion][MiniMax H3] Restrict MiniMax-H3 SubBlock sparsity to video queries (1.31 to 1.46x faster with more consistent audio): #35850
  • [Diffusion] ComfyUI: add a MiniMax-H3 node and a generic extra-fields passthrough: #35352
  • fix(diffusion): MiniMax-H3 dp_size>1 deadlock and cross-request audio determinism: #36398
  • [Cosmos3] Add cosmos3 transfer capability: #34747
  • [Diffusion] RL rollout support for the Cosmos3 pipeline: #34197
  • [Diffusion] Optimize Pi0.5 inference and bounded graph serving (Pi0.5 end to end -9% eager, -68% with bounded graphs on H200): #34599
  • [Diffusion] Per-request lossy accelerations: Cache-DiT, CFG gating, attention backend override: #35339
  • [Diffusion] Allow Cache-DiT with DiT layerwise offload (Cache-DiT with layerwise offload 1.84x faster at 30.1 dB PSNR on a 24 GB 4090 D): #35858
  • [Diffusion] feat: support out-of-tree models and pipelines: #35713
  • [Diffusion] Decouple encoder parallelism from the DiT parallel layout: #34713
  • [Diffusion] Reuse SRT Qwen vision and text modules: #35006
  • [Diffusion] Reuse SRT quantization contracts and MXFP8 kernels: #36063
  • [Diffusion] Load serialized Comfy ConvRot INT8 DiTs: #35994
  • [Diffusion] Load serialized Comfy W4A8 checkpoints: #36036
  • [Diffusion] Add component-scoped quantization overrides: #36035
  • [Diffusion] Add composable component weight path CLI: #36078
  • [Diffusion] feat: let every layerwise component be configurable: #35688
  • [Diffusion] feat: plan pinned host memory against the cgroup cap not the machine (text encoding 2.01x faster on RTX 4090): #35641
  • [Diffusion] Enable LongCat breakable CUDA graphs (LongCat-Image denoise -11.9%, e2e -3.4% on B300): #35724
  • [Diffusion] Enable SANA-Video breakable CUDA graphs (SANA-Video denoise -13.5%): #35729
  • [Diffusion] Accelerate SANA-Video linear attention in quality=high (SANA-Video quality=high e2e -7.2% on B300): #35728
  • [Diffusion] Accelerate LingBot Video RMSNorm in quality=high (LingBot-Video quality=high denoise -25.7%, e2e -22%): #35969
  • [Diffusion] Fuse SANA-Video interleaved RoPE (SANA-Video e2e -8.2%): #35695
  • [Diffusion] Fuse LTX-2.5 decoder 3D RoPE (LTX-2.5 decoder -17%): #35698
  • [Diffusion] Fuse LongCat-Image QKNorm and interleaved RoPE (LongCat-Image e2e -17%): #35995
  • [Diffusion] Fuse Cosmos3 Nano T2I attention on Hopper (Cosmos3 Nano denoise -12%, e2e -8%): #36571
  • [Diffusion][Kernel] Fuse Wan FFN GELU epilogue (Wan2.1 denoise -1.9%): #36592
  • [Diffusion][Kernel] Fuse Wan2.2 NVFP4 bias + GELU on Blackwell (Wan2.2 NVFP4 denoise -3.0%): #37075
  • [Diffusion] Optimize Qwen-Image TP collectives and attention (Qwen-Image-2512 TP2 e2e -8.6%): #36680
  • [Diffusion] Cache Qwen-Image modulation across serial CFG branches: #37090
  • [Diffusion] Fuse Qwen-Image FP8 QKV projection and Blackwell epilogue: #37123
  • [Diffusion] Fuse Qwen-Image residual norm and NVFP4 quantization (denoise -1.2% on GB300): #37129
  • [Diffusion] Fuse FLUX.2 gated residual normalization on Blackwell: #37112
  • [Diffusion] Fuse FLUX.2 token concatenation and NVFP4 quantization: #37141
  • [Diffusion] Fuse FLUX.2 ModelOpt FP8 producers and QKV packing: #37162
  • [Diffusion] Default Hunyuan VAE to tiled decode (Hunyuan e2e +18% and FastHunyuan no longer OOMs in VAE decode on 4xH200): #36012
Local & Desktop AI
  • [Multimodal] Fix NVFP4 diffusion models on sm_120 (RTX PRO 6000 / RTX 50xx): #35739
  • [SM120] flash_mla: allocate the page-split buffer outside inference mode: #35116
  • [MLX] Upgrade to Torch 2.13/MLX 0.32+ and redesign the Torch-MLX tensor bridge: #32984
  • [MLX] Fix startup crash when reporting preloaded weights: #37035
AMD / ROCm
  • [AMD] Add Work-Centric (Lean) Attention: a persistent-CTA decode kernel for long-context serving (up to 1.52x throughput and 3.62x lower median ITL on MI355X): #33576
  • [AMD] support gfx1250 on ROCM 10: #36871
  • [AMD] Add ROCm 10 (gfx942 / gfx950) release images: #36434
  • [AMD] Enable Mori-EP on kimi-k3 (GSM8K 0.951): #35630
  • [AMD][Fix] Route MoRI through the Qwen MoE all-to-all path (Qwen3.5 MoRI GSM8K 0.006 to 0.967): #32039
  • [AMD] Enable moe_a2a_backend=mori for DeepSeek-V4 prefill context parallelism: #35611
  • [AMD] Add Radix-4 MoE top-k router kernel for Kimi-K3 routing: #34490
  • [AMD] Enable aiter mla asm path through padding attn heads for Kimi K3 (GSM8K 0.947 to 0.958): #36356
  • [AMD] Improve K3 dspark draft attn kernel perf: #35499
  • [AMD] Enable 12-head MLA aiter fp8 Gluon decode (batched bh16bn128).: #34647
  • [AMD] DeepSeek-V4: add aiter fused mHC post+pre with cross-layer boundary dispatch: #32577
  • [AMD] DeepSeek-V4: route decode wo_a bf16 batched matmul to aiter batched_gemm_bf16: #33313
  • [AMD] DeepSeek-V4 MI355X: eliminate bpreshuffle fp8-scale copies at producer sites (MoE down, MLA o_proj bmm): #33166
  • [AMD][DSV4] perf: bound the MoRI receive buffer during decode: #36130
  • [AMD] Enable deepseek-v4 topk_transform v2 kernel (top-k kernel 2.66 to 2.97x faster, up to 1.07x end to end): #36684
  • [AMD] Enable draft_extend CUDA graph for HIP DSA backend: #35672
  • [AMD][Spec][PD] Enable the PD DSA fused-TopK seed remap on ROCm (GLM-5.2 PD decode TPOT 23.16 ms to 7.94 ms on 8xMI355X): #36714
  • [AMD] Qwen3.5 ASM FMHA chunked-prefill context attention (context-chunk kernel 14.24 ms to 4.04 ms, E2E latency -24.6% to -28.5% at 70K input): #36758
  • [AMD] Optimize Qwen3.5 MTP unified attention on gfx950 (+9.6% throughput at concurrency 8 with 70K input): #36330
  • [AMD] MiniMax-M3 : Fuse QKV+index proj for block-fp8: #32099
  • [AMD] feat(moe): fold padded-topk_ids fill into fused shared-experts append+remap: #31370
  • [AMD] [sgl-kernel] Bypass caches for peer traffic in ROCm custom all-reduce: #32832
  • [ROCm] Extend the gfx950 extend-attention tile to head_dim <= 128: -43% kernel, -14% TTFT, bit-identical: #34461
  • [ROCm] Fix QuickReduce fp16 saturation corrupting bf16 all-reduces (106M non-finite -> 0, +0.3%): #34484
  • [ROCm] Define the DSA head-gate graph helpers on HIP: #37118
  • [ROCm][Fix] Cap the DSA MQA-logits budget at AITER's buffer_store limit: #36960
  • [ROCm][Fix] Use token-level KV indices in the aiter ASM context-prefill gather: #36852
  • [AMD][Spec] Fix aiter GQA packing + split-KV routing in NEXTN spec attention (verify & draft_extend) (draft-extend at 88K from ~773 ms to ~5 ms, NEXTN now 1.35 to 1.76x faster than no-spec): #30105
  • [AMD] Fix DSv4 draft extend taking the target compression path during prefill: #37713
  • [AMD] Fix v4 topk issue: #37439
  • [AMD] fix aiter cannot get heuristic kernel regression: #37438
  • [AMD] Gate the aiter memory-reserve exemption behind an env var: #37242
  • [AMD] Fix Qwen3.5 MTP dropping fused shared-expert weights: #35719
  • [AMD] Quark shared-experts gate: recognise a trailing MTP layer (GLM-5.2 median TPOT -16.0% at concurrency 4, -8.4% at 64): #36124
  • [AMD] Fix the QuickReduce bf16 cast failing to build for CDNA: #37132
  • fix(lora): build the MoE LoRA align JIT kernel on ROCm: #36379
  • [AMD] Fix nightly ROCm 7.0 image build: patch missing <optional> include in AITER topk kernel: #36216
  • [AMD] Keep the PTX-inline-asm diffusion norm fusions off on ROCm (fix FLUX warmup crash): #34481
  • [AMD] Let the diffusion AITer backend take grouped-query K/V (fix Cosmos3-Nano startup): #34485
  • [AMD] [Docker] Upgrade Python 3.12 + torch 2.11 + triton 3.7 in ROCm 7.2.4: #30984
  • [AMD] Update ROCm AITER pin to c16d44b: #35810
  • [AMD][MORI] Bump MoRI to 7c51d18 for ionic RoCE dmabuf fix (#509): #37286
NPU / Ascend
  • [NPU] DeepSeek-V4 adapt sgl-kernel-npu ops (compressor/sparse-attn/sparse-attn-metadata): #35676
  • [NPU] Add mxfp4-w4a4 MOE Quantization Support for NPU: #30319
  • [NPU] add causal conv1d for ascend kda backend: #35021
  • [NPU] [Diffusion] Support MiniMax H3 on Ascend NPU's (Laser attention 1.26 to 1.45x faster than Ascend flash attention): #33569
  • [NPU] [Diffusion] support distributed inference pipeline for GLM-Image (14-denoiser disaggregation 3.7x faster than monolithic batching): #31320
  • [NPU] [Diffusion] Fix critical Ascend NPU Diffusion regression/bugs & restore 2-NPU CI testcase: #34855
  • Support streaming session on NPU: #32597
  • [NPU] fix: reach torch>=2.8 CUDA memory-pool APIs lazily via torch._C: #29100
  • [NPU][Fix] Fix OOB gather in decode KV allocation when free pool is tight: #35727
  • [NPU][Fix] Fix discontinuous input for FIA operator in GLM4.7‑Flash: #36170
  • Fix NPUMHATokenToKVPool missing k_data_ptrs/v_data_ptrs: #35820
CPU / Intel / XPU
  • [XPU] Support INT4 dense linear (AWQ/GPTQ) for XPU: #30236
  • [XPU] Use SYCL kernels for DeepSeek V4 MHC on XPU: #32166
  • [XPU] Use SYCL kernels for topk_transform on XPU: #33318
  • [XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (Qwen3.5 GDN prefill 165 ms to 59 ms): #33354
  • [Intel XPU] support prefill only models for xpu: #35072
  • fix(xpu): enable compressed-tensors FP8 W8A8 on XPU (RedHatAI FP8-dynamic models): #33057
  • XPU: remove SGLANG_USE_SGL_XPU flag: #34492
  • [Intel XPU] Fix cross-encoder rerank hang on B580 runners: #36360
  • [CPU] Enable ERNIE models on CPU: #35222
  • [CPU] Fix NUMA/core binding for DP ranks: #32856
  • Fix DP attention on CPU: #12961
  • [CPU] Fix wrongly causal-masked bidirectional attention: #35434
  • [CPU] Fix truncated KV prefix in intel_amx spec verify: #36430
  • [CPU] Fix weight missing issue in fused_input_proj_cpu for GPTQ INT4 for Qwen 3.5: #35805
Security
  • fix: bound CUDA memory for fast image preprocessing: #36295
  • fix(frontend): bound stop strings and regex patterns: #37029
  • [CI] Gate /rerun-test on commenter trust and remove /rerun-stage: #35750
Dependencies
  • Update deepep for SBO feature: #35450
  • [Deps] Bump FlashInfer to 0.6.18: #36954 ⭐
  • Use Flashinfer 0.6.18 release for CUDA 13.4 package: #37073
  • Bump sgl-deep-gemm to v0.1.6: #36998
  • Bump sgl-deep-gemm to 0.1.7: #37279
  • chore: bump mooncake version to 0.3.13: #36493
  • chore: bump tilelang to 0.1.12: #30874
  • fix(deps): pin compressed-tensors to 0.18.0: #36794
  • [NVIDIA] Add CUDA 13.4 container for initial Rubin support: #36233
Breaking Changes & Upgrade Notes
  • The unified radix tree is now the default cache for every configuration, including Full-attention-only models; SGLANG_ENABLE_UNIFIED_RADIX_TREE is deprecated and can be unset: #35081
  • Spark3 is renamed Spark2.5 across config (Spark2_5Config, model_type spark2_5), model classes, and the function-call parser; the tool-call parser name changes from spark to spark25: #36416
  • DeepSeek-V4 FP4 expert checkpoints default to the FlashInfer MXFP4 MoE runner on SM90, SM100, and SM120 when --moe-runner-backend is left at auto; an explicit backend is preserved: #35919
  • W4A4 MegaMoE moves to --enable-w4a4-megamoe; SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS and SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND now only warn: #35918
  • FlashInfer 0.6.18 is required; --dsa-topk-backend flashinfer and the CuTe DSL NVFP4 W4A16 mode have no fallback for earlier FlashInfer APIs: #36954, #33237, #35120
  • Constructing a ServerArgs no longer resolves it. Programs that build their own record must call resolve_once() before reading resolved values; the launcher does this for the engine: #35907
  • Kimi hybrid models honor --enable-symm-mem under CUDA graphs again, since torch 2.13.0 fixes the allocator bug that forced it off: #34842
  • Lean (work-centric) decode attention is auto-enabled on the ROCm Triton backend where its gate predicts a win, including under CUDA graphs; force it off with --enable-lean-attention=False or SGLANG_DISABLE_LEAN_ATTENTION=1: #33576
  • ROCm 7.2.4 images move to Python 3.12, torch 2.11, and triton 3.7; new ROCm 10 images ship for gfx942, gfx950, and gfx1250: #30984, #36434, #36871
  • The SGLANG_USE_SGL_XPU flag is removed; XPU now uses the sgl-kernel MoE kernels by default, and --moe-runner-backend triton restores the Triton path: #34492
  • Diffusion Cache-DiT, CFG gating, and the DiT attention backend become per-request sampling params; the SGLANG_CACHE_DIT_* env vars remain as server defaults and are slated for deprecation: #35339
  • Hunyuan video VAE defaults to tiled decode: #36012
  • Requests are limited to 32 stop strings and 32 stop regex patterns of at most 256 bytes each, returning HTTP 400 beyond that: #37029
  • /rerun-stage is removed and /rerun-test is gated on commenter trust (zero cooldown or repo write): #35750
  • kill_process_tree waits for the reap by default in CLI, benchmark, and test teardowns: #36589
New Contributors
View original

Upgraded? How did it go?

Discussion