SGLang v0.5.18

v0.5.18
Added 15
  • Support for Muse Glimmer autoregressive multimodal model
  • Support for Intern-S2-Mobius autoregressive model
  • Support for SANA-Video diffusion model
  • Support for LingBot-Video-MoE diffusion model
  • Support for LTX-2.5 diffusion model
  • Support for Cosmos3 Edge & Distilled diffusion models
Changed 5
  • TP LMHead allgather and scatter now uses single all-to-all for pure-DP dp-attention
  • Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches consolidated under SGLANG_CACHE_DIR
  • Updated torch to 2.13.0 with triton 3.7.1
  • Updated flashinfer to 0.6.17
  • Updated sgl-kernel to 0.4.6.post1

From SGLang

Highlights

710 PRs from 212 contributors.

New models in this release (see the cookbook for all supported models):

ModelTypePRsCookbook
Muse GlimmerAutoregressive (Multimodal)#34262link
Intern-S2-MobiusAutoregressive#33691link
SANA-VideoDiffusion#32921link
LingBot-Video-MoEDiffusion#32341link
LTX-2.5Diffusion#34471link
Cosmos3 Edge & DistilledDiffusion#31590link
LongCat-ImageDiffusion#23274

Plus cookbook recipes for the Qwen3.8 family, Ling-3.0, Nemotron 3.5 Lightning, Dots3-Note, and DeepSeek-V4-Pro-0813 (#34809).

Overlapped checkpoint staging at startup: Checkpoint pages now stage from storage while CUDA graphs capture. Qwen3-32B on H100 starts 8.6-11.7% faster than serial with prefetch, and 2.38x faster (35.6s vs 84.8s) than the plain default. Opt in with --startup-weight-load-mode overlap (#32017).

TP LMHead with All-to-All: The TP LMHead's allgather + scatter becomes a single all-to-all for pure-DP dp-attention. On DeepSeek-V4-Pro B200 decode, LMHead time drops 320us to 169us and TPOT improves 36.97ms to 35.67ms (#32313).

FlashInfer MNNVL for pure allreduce: Non-fused allreduce sites now reuse the FlashInfer MNNVL workspace instead of falling back to NCCL. DeepSeek-V4-Flash TP4 decode on Blackwell gains up to +6.9% at small batches. Auto-enabled for DeepSeek-V3/V3.2/V4; elsewhere --enable-flashinfer-pure-allreduce (#30700).

One compiled-kernel cache directory: Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches all move under SGLANG_CACHE_DIR. The first launch after upgrading recompiles once; see Breaking Changes (#32434).

Dependencies: torch 2.13.0 with triton 3.7.1 (#28836), flashinfer 0.6.17 (#33997), CuTeDSL 4.6.2, fixing an FA4 startup regression on Blackwell (#34372), DeepEP now installed from released sgl-deep-ep wheels (#33932), and sgl-kernel 0.4.6.post1 (#33842).

Full release notes by category below; breaking changes and known issues are at the end.

Rust Server
  • [mm] rust-server: native multimodal processing for Qwen VL (integrate sglang-mm, e2e): #32365
  • refactor error responses into shared utils::response helpers: #33894
  • move the PD bootstrap registry under api_server::disaggregation: #33895
  • Build Rust extensions on demand in source checkouts: #34994
Speculative Decoding
  • [spec decoding] support inkling dspark: #31847
  • [Spec] Support logprobs with DSpark speculative decoding: #34696
  • [Spec] Support output logprobs with DSpark: #34478
  • [Spec] Support logprobs with DFlash: #33459
  • [Spec] Wire DFLASH aux-hidden capture into the Qwen3.5 text-only wrapper: #34771
  • [Spec] Support mamba-radix-cache-strategy extra_buffer_lazy with DFLASH: #34763
  • [Spec] Support MegaMoE for DSpark under dp attention: #34844
  • [unified memory] Support DSPARK speculative decoding + fix two NaN root causes (page hand-out zeroing, CuTe int32 slot-stride wrap): #33974
  • [GDN] Honor configured linear-attn verify backend in the kernel dispatcher: #34592
  • [Spec] Relay ngram accept tokens through the FutureMap: #35198
  • [Spec] Reduce host-side overhead in ngram draft prep: #35207
  • [Spec] Point multi-layer eagle's last shared-read runner at the draft runner: #35057
  • Fix DFlash sliding attention causality defaults: #34524
  • [Spec] Budget the DFLASH draft KV pool from its own attention geometry: #34234
  • fix(dflash): account for DCP in draft KV pool sizing: #33912
  • [DSV4] Fix silent KV corruption when speculative draft tokens > 4: #34189
  • [Fix] Speculative decoding crashes with DP-Attention: #33892
  • [bugfix] Stop/EOS inside a spec accept run beats the max_new_tokens finish: #33758
  • [DSpark] Fix EP1 decode performance regression: #34759
  • [Fix: RL] Snapshot async state-capture outputs before overlap: #34319
Piecewise & Breakable CUDA Graph
  • [BCG][5/N] MLA Fully Support: #33661
  • [BCG][6/N] Allow prefill breakable CUDA graph for the Kimi archs: #34245
  • fix: always capture default prefill CUDA graph: #33352
  • Fix padded positions in breakable CUDA Graph attention: #33253
  • fix: avoid piecewise prefill graph for trtllm_mla: #32785
  • Reenable breakable CUDA graph for NemotronH: #34538
  • Fix prefill CP graph overflow with larger bucket search: #33906
  • Fix stale track rows corrupting conv checkpoints under the prefill graph: #34184
  • Fix sconv track refresh on graph capture: #35042
  • Increase post-capture decode memory reserve: #34996
Attention Backends
  • feat(attention): add architecture-owned SM12x FA4 kernels: #32991
  • fix: support FA4 backend for GLM4.7-flash: #33436
  • feat: Add flashinfer mHC fusion for DSV4: #33616
  • [DSV4] Turn on mhc post pre fusion by default: #35214
  • [SM12x] Default the fused MHC post+pre path on: #34019
  • [trtllm_mha] perf: Stop allocating per-layer scratch inside the decode CUDA graph: #33063
  • Select DeepGEMM standard layouts by memory budget: #33474
  • [NVIDIA] Enable CuTe DSL BF16 GEMM on SM107: #33617
  • add flashinfer cute-dsl backend for mxfp8 gemm: #34042
  • [BugFix] Fix race in c128 prefill plan kernel on ragged extend: #32467
  • [JIT Kernel][DSv4] Optimize epilogue of c128: #26671
  • fix(dsa): use FlashInfer fused top-k for packed PAGED rows: #33006
  • [DSA] Fix top-k v2 dropping non-primary ranks' output on CUDA 13.1+ (root cause for #33835): #34167
  • [DSA] Skip indexer KV cache for skip-topk layers: #30531
  • [Perf] Skip trivial DSV4 nonpaged indexer logits: #33857
  • [Fix] Restore layer-level DSV4 RoPE policy: #34788
  • Fix _pa_swa_prefill_lens off-by-one in FlashAttentionBackend: #33379
  • Support unified SWA page mapping in attention metadata: #35000
MoE & Expert Parallelism
  • [MoE] Fuse swiglu moe up gemm epilogue: #32944
  • Enable MoE deferred finalize by default and drop its expert_weights dtype workaround: #33618
  • [kernel] add triton moe TMA up support: #33559
  • [MoE] Single-launch moe_align for tiny batches with many experts: #32395
  • [MiniMax-M3] Overlap shared and routed experts: #34542
  • [MoE] Route every trtllm-gen MoE call site through one PDL guard: #34789
  • fix(qwen3): support DeepEP-class backends and early EPLB state: #34810
  • Add explicit EPLB balancedness reporting modes: #34998
  • Fix the router GEMM inaccuracy when using _front_w in Kimi-K3: #33764
  • [Kimi-K3] Use explicit SiTU activation for MegaMoE: #34883
  • Fix Qwen3-MoE producing garbage with the mori a2a backend: #34006
  • [Fix][Qwen]: fused shared-expert detection PP-safe protection: #34447
  • [FlashInfer v0.6.16] Support FlashInfer CuTe DSL NVFP4 MoE quantization: #28354
  • [Fix] Correct W4AFP8 DeepEP scaling and mode-specific dtypes: #33669
  • [Fix] Support 128-aligned hidden sizes in the W4AFP8 DeepEP low-latency requant kernel: #35593
  • perf: add H200 Triton MoE configs for E256 N512: #34150
Quantization
  • Support ModelOpt MXFP8 checkpoints: #32538
  • [ModelOpt FP4] Support online MoE weight quantization: #33115
  • [Feature] Support NVFP4 token embedding in ModelOpt mixed-precision checkpoints: #34222
  • [MXFP8] Use FlashInfer CUTLASS for dense GEMM on SM120, delete Triton path: #33208
  • [Quantization] Route per-tensor FP8 checkpoints to FlashInfer on SM90: #33148
  • Optimize FP32 LM head for bf16/fp16: #32370
  • Pin online NVFP4 4over6 quantization settings: #33621
  • [Fix] Support Kimi-K3 ModelOpt mixed NVFP4/FP8 checkpoint: #35077
  • [Quantization] Fix GPTQ scheme attachment broken by LinearBase.scheme default: #34962
  • [quantization] Add tuned Triton tile configs for channelwise FP8 GEMM…: #34331
  • [Fix] Correct dense FP8 Marlin bias ordering: #35020
  • Remove the torchao integration (--torchao-config): #34304
Parallelism & Disaggregation
  • [Feature] Optimize TP LMHead with All-to-All: #32313 ⭐
  • [NVIDIA] Add flashinfer MNNVL backend for allreduce only: #30700 ⭐
  • [CP]: Support CP V2 Strategy for dsv4: #33532
  • fix: preserve GQA head mapping in Triton DCP prefill: #34161
  • Fix DCP KV head mapping for GQA models: #32858
  • [DCP] Reuse partial output in natural-log LSE merge: #34213
  • [DCP] Drop two per-layer launches from the MLA target-verify path: #34240
  • [DCP] Fuse the a2a pack/unpack copies in the MLA LSE reduce: #34614
  • [DCP] Share one pack kernel between both a2a backends: #34651
  • [DCP] Drop the prefill index-selection syncs by taking each rank's rows by stride: #35084
  • [Fix] DCP: advertise the logical KV-event block size: #35298
  • Fix DeepSeek-V4/DeepSeek-V4-Pro DP-attention gather semantics: #31700
  • [perf] Collapse the DP attention scheduler sync to a single D2H copy: #34338
  • Multiple flexibility fixes for DP attention: #33537
  • Fix world-size-one aliasing in MLP batch sync: #34997
  • [PD] Add the missing Prefill bootstrap timeout for NIXL: #34692
  • [PD] Support pipeline-parallel prefill with Mooncake staging buffer: #33807
  • [PD] Support --enable-unified-memory with PD disaggregation (kimi-linear MLA hybrid-Mamba): #33362
  • [PD] Skip speculative verify scratch on prefill servers (saves num_draft_tokens x mamba pool per rank): #34191
  • [PD] Avoid unused PREBUILT prompt tensor transfer: #35070
  • feat(grpc): expose KV event discovery metadata: #35714
Scheduler & Runtime
  • [Model Loading] Overlap checkpoint staging with CUDA graph capture during startup: #32017 ⭐
  • [core] Consolidate compiled-kernel caches under SGLANG_CACHE_DIR: #32434 ⭐
  • [Engine] Freeze GC after server warmup: #34999
  • feat: add cache salt support to KV cache events: #30827
  • perf(kv-events): coalesce cache events: #31479
  • [Scheduler] Honor explicit min-free-slots thresholds: #33403
  • [Scheduler] Gate SWA eviction on accumulated tokens: #33404
  • [Scheduler] Fix to restrict the SWA chunk-cap escape hatch to true head-of-line livelock: #32700
  • [srt] Batch scheduler cache frees: #33475
  • O(1) slot allocation in ReqToTokenPool.alloc(): #32208
  • fix(scheduler): track max prefill batch size over recent real admissions: #34284
  • fix tpot by adjusting the sliding max-prefill-size window size: #34856
  • fix: route scheduler aborts to multi-tokenizer workers: #33940
  • [tracing] sglang tracing v2: support exporting tracing data asynchronously: #30023
  • Profiling Enhancements [1/3]: cuda graph profile traces: #24370
  • [metrics] Fix prefill FLOPs estimate to count prefix and per-request causal pairs: #34316
  • Fix default dtype restoration after model loader errors: #34440
  • fix: honor explicit model loader classes: #34880
  • feat(cli): add extensible serve backend plugins: #34753
  • Clean up environ.py: remove dead env vars, unify deprecation handling, move examples to a unit test: #35060
  • fix: make automatic NUMA binding configurable: #30394
  • Raise PD zmq per-context socket cap via SGLANG_DISAGGREGATION_ZMQ_MAX_SOCKETS: #34450
Deterministic Inference
  • Fix deterministic inference for Inkling: #33417
  • Fix deterministic inference all-reduce for tp>1: #34159
  • Deterministic gumbel sampling: clamp u=1 so masked tokens can't be sampled: #33423
  • feat: support deterministic FA4 for GLM-4.7-Flash: #33945
  • Fix NaN logits from deterministic Triton extend on the unified memory pool: #33517
  • Add bit-exact class for MTP: #35143
  • Add bit-exact unified radix cache KL test for hybrid SWA + mamba: #34607
  • Add bit-exact hicache logprob-consistency test: #34356
  • Add bit-exact guard for extra_buffer_lazy: #35030
  • Skip inkling sheared bias under batch invariance: #35161
HiCache & Radix Cache
  • [HiCache] Support packed and sidecar draft caches for MTP/EAGLE/DSpark: #30393
  • [Hicache][2/2]Support Mamba branching in Unified Radix Cache with HiCache: #33639
  • [HiCache] Support DCP with DSpark: #35221
  • Enable unified cache out-of-window slot freeing by default: #34653
  • Skip oow slot freeing under eagle: #34823
  • [HiCache] write_back: reclaim duplicated host copy first under host pressure: #33777
  • [HiCache] Optimize LogicalHostPool free-list release: #33998
  • refactor(hicache): flatten L2 transfer execution: #34793
  • [PD] Preserve decode KV across retraction in HiCache: #34801
  • fix(hicache/umbp): support DeepSeek-V4 hybrid HostPoolGroup (multi-po…: #30762
  • [Unified Radix Cache] Complete the tree-core interface boundary: #33580
  • [Fix] Snapshot req.prefix_indices when the prefix cache is disabled: #34644
  • Fix swa eviction frontier for bigram keys: #34870
  • Retain SWA down to the last state checkpoint: #34729
  • Fix mamba checkpoint depth under dcp: #34808
  • HiSparse: shared-index (IndexShare) plan-then-IO swap-in prefetch: #34329
  • Observability enhancement for HiCache: #32388
Multimodal
  • [VLM] split multimodal scheduling from mm_utils: #32415
  • [VLM] add content-addressed preprocessing cache infrastructure: #34398
  • Add CUDA VMM multimodal feature transport: #33899
  • feat(vlm): auto-select cuda vmm on multi-node mnnvl: #33936
  • Fix CUDA 13.0 VMM handle type compatibility: #34431
  • Gate multimodal feature transport by model capability: #33653
  • Reserve multimodal runtime allocations and keep padded inputs aligned: #34141
  • [VLM] Avoid synchronizing multimodal placeholder counts: #34995
  • fix(vlm): stream-order cuda-ipc feature pool lifecycle and streamline multimodal transport module: #33949
  • [EPD] feat: pipeline owner-only multimodal preprocessing: #34206
  • [FEAT] Decouple multimodal global cache from Mooncake: #30392
  • [Kimi K3] optimize: preprocess cpu-transport images on the vision owner: #33921
  • fix(vlm): preserve Kimi-K3 GPU JPEG accuracy: #34163
  • vlm: cache kimi-k3 per-image processor artifacts: #34404
  • [Fix] Carry the backend on Kimi-K3 deferred preprocessing configs: #34766
  • Fix Whisper transcription for audio over 30 seconds: #33604
  • fix: preserve priority for batched embedding requests: #32977
Model Support & Optimizations
  • fix(muse-glimmer): parse required/named tool calls natively: #34781
  • Responses support: #32689
  • [Frontend] Apply request header overrides to chat completions: #35001
  • Stop losing Kimi-K3 tool calls to reasoning, constraint conflicts, and truncation: #34881
  • [Fix] Work around xgrammar 0.2.1 negative integer minimum in Kimi-K3 structural tags: #34778
  • [Fix] Make DeepSeek-V4 reasoning and tool-call streaming parsing chunk-invariant: #34458
  • [Fix] Require JSON booleans for response_format json_schema.strict: #34777
  • Support thinking budget for Inkling: #33146
  • [inkling] Let Anthropic thinking=disabled map to reasoning effort "none": #33913
  • [inkling] Render tool-result media instead of coercing content to str: #33898
  • [DSV4] Add official DSV4 reasoning effort support: #33140
  • [feat] Add language_model_only parameter support for Qwen35: #22867
  • Support model-defined prefill input embedding width: #35002
  • Fix rope config compatibility and VL/transformers-fallback weight loading: #31575
  • [LoRA] Laguna: per-layer LoRA hidden-dim resolution for packed attention: #30298
  • [Laguna] fix YaRN mscale double-application in rope config: #33558
  • enable TRT-LLM for MiniMax M3 by preserving SwiGLU params: #33962
  • fix(minimax): use routed TRT-LLM for NVFP4 MoE auto on SM100: #32229
  • fix(glm5.2): restrict MoE weights to local PP layers: #33793
  • [Fix] Fix Qwen3.5 MTP startup with HiCache: #34560
  • Fix DeepSeek-OCR batching crash on variable local-crop counts: #33214
Kernel Library
  • [jit_kernel] Move JIT kernels into namespace sglang: #33400
  • [kernel] Content-addressed JIT build cache, generated from our own ninja: #34274
  • [JIT Kernel] Migrate per-token FP8 quantization from AOT to JIT: #34257
  • [JIT Kernel] Migrate moe_topk_softmax from AOT to JIT: #34509
  • [JIT Kernel] Migrate causal_conv1d_fwd and causal_conv1d_update from AOT to JIT: #35031
  • kernels: scalar scale A support for fp8_gemm: #33469
  • [Kernel] Unify BaseFusedOp and MultiPlatformOp dispatch: #33205
  • [Kernel] Enable Helion backend for Kimi Delta-Attention: #32593
  • [Kernel] cutedsl_bf16_gemm: trailing cluster barrier for 2-CTA TGV kernel exit (#32907): #32954
Configuration System
  • config: retire ServerArgs.derive; per-runner values are constructor arguments: #33887
  • config: business code no longer reads the published ServerArgs: #34081
  • config: the runner and scheduler read resolved config from the bags: #34095
  • config: the KV-cache configurator reads the bags: #34096
  • config: pin that resolution is reproducible from the raw input: #34094
  • config: one control-plane log for the process: #35028
  • config: make ServerArgs read-only with a single audited mutation entry: continues in this release across the runner, scheduler, spec, and DCP surfaces (#33925, #34133, #35022, #35025)
SGLang-Diffusion
  • [diffusion] feat: data-parallel serving (--dp-size): #33725
  • [diffusion] feat: cross-node sequence parallelism (Ulysses x Ring): #33327
  • [diffusion] feat: support K/V-gather style sequence parallel (CP-like) attention: #32667
  • [diffusion] feat: support sol-attn sparse attention backend for h3: #33702
  • [MiniMax-H3] SubBlock: training-free block-sparse attention for the DiT: #34148
  • [diffusion] feat: add SageAttention packed varlen path for minimax-h3: #33703
  • fix: make Cache-DiT actually cache on MiniMax-H3: #33827
  • [diffusion] feat: load quantized H3 text encoder checkpoints: #34986
  • [diffusion] feat: support native and peft minimax h3 loras: #34359
  • [diffusion] Per-section LoRA adapters on fused linear layers: #34933
  • [diffusion] feat: support quant-videogen prq kv-cache quantization (memory-saving) for causal-dit: #32581
  • [diffusion] Enable breakable CUDA graph for LTX-2 (H200 two-stage e2e 10.75 s -> 6.90 s, 1.56x): #33885
  • [diffusion] Enable breakable CUDA graph for SANA (H200 1024px e2e -26%, bit-exact): #33989
  • [diffusion] Enable breakable CUDA graphs for LTX-2.3: #34929
  • [diffusion] Z-Image single-GPU BCG: fix the replay crash and make output bit-exact vs eager (#34183): #34210
  • [diffusion] BCG: auto-capture the default warmup resolution instead of hard-requiring --warmup-resolutions (H200 SANA denoise 0.73->0.457 s with a single flag): #34174
  • [diffusion] weight-only FP8: dequantize linear weights once at first use (Ideogram-4 denoise -18.8% H200 / -7.8% H100, bit-exact): #34305
  • [diffusion] Bit-exact data-movement elimination for the Wan causal VAE decoder (H200 LongLive2 704x1280x61f: decode 2.80->2.32 s lossless / 2.12->1.67 s quality=high, e2e -10.7%): #34125
  • [diffusion] FLUX.1 fused adaLN modulate (bit-exact) + RoPE cache hoist, LN-affine folding behind quality=high (H200 e2e -3.5% lossless / -6.9% high): #34004
  • [diffusion] Ideogram-4: fuse Qwen3-style RoPE and SwiGLU silu-mul (denoise -5.1% H100 / -4.7% H200, bit-exact): #34314
  • [diffusion] ERNIE-Image: fuse rotate-half RoPE + GELU-mul and hoist rope cos/sin (denoise -16.2% H100 / -12.7% H200, bit-exact): #34306
  • [diffusion] Wan2.2-TI2V: fuse per-token adaLN table add into contiguous slices + hoist rope cache (denoise -13.1% H100 / -12.6% H200, bit-exact; eager beats compile): #34584
  • [diffusion] Z-Image bit-exact fused qk-norm (H200 Turbo 1024px e2e -6.4%): #33886
  • [diffusion] Add --dit-layerwise-residency-policy for strided DiT residency: #34534
  • [diffusion] feat: support dynamically cpu offload components: #34391
  • [diffusion] feat: gate /health and /health_generate on warmup completion and add liveness endpoint: #33787
  • [diffusion] feat: support --served-model-name in sglang serve: #34228
  • [Diffusion] Add online FP8 support for Krea-2: #34136
  • [diffusion] feat: expose cosmos3 policies through the Action API: #34243
Local & Desktop AI
  • [MLX] Support gpt-oss: sliding-window attention, attention sinks, sm_scale: #30050
  • [MLX] Window-bounded SWA KV storage and in-graph sampling: #34166
  • feat(dgx-spark): add inkling-small MoE support for sm_121: #33108
  • [misc] Pass FP8 scales in FlashInfer SWA prefill, autotune fp8 on SM120, and tighten is_image_understandable_model: #34217
AMD / ROCm
  • [AMD] Support prefill context parallel two batch overlap for DeepSeek V4: #33480
  • [AMD] Support DeepSeek V4 DSpark on AMD HIP platform: #30964
  • [AMD] Optimize KIMI-K3 with Triton MLA decode kernel by tuning the stage-1 geometry for gfx950: #34580
  • [AMD] Add concat_and_cast_mha_k_pad_kernel to support 12-head and enable K3 aiter prefill kernel: #34837
  • [AMD] Add K3 verified mla kernel for DSpark on triton backend: #33981
  • [AMD] Restore K3 MLA verify kernel path blocked by can_handle() guard: #34261
  • [AMD] perf: compact Triton extend-attention for ragged prefill (AMD/HIP-only): #29677
  • [AMD] Accelerate AITER unified-attention decode with scaled FP8 Q: #31856
  • [AMD] Run V4 MTP target-verify through the decode kernel: #34597
  • [AMD] Enable draft-extend CUDA graph and reduce bubble for MTP: #29202
  • [AMD][Spec] Accelerate Qwen3.5 verification with grouped-head shared KV: #34517
  • [AMD][Perf] Fuse GatedDeltaNet QKVZBA split/reshape/cat into a single Triton kernel for Qwen3.5-architecture MoE on HIP: #34421
  • [AMD] Fuse shared_expert_gate GEMV into the MoE append kernel (HIP/aiter): #28666
  • [AMD] [GLM5] fp8 MLA absorbed bmm for GLM-5.2 on gfx950: #30519
  • [AMD] [GLM5] Skip DSA decode indexer when kv_len <= index_topk (dense k-only fast path): #31324
  • [AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs: #29328
  • [AMD] Fix AITER custom reduce-scatter CUDA-graph capture crash under torch_memory_saver: #34203
  • [AMD] Preserve the AITER expert mask across torch_memory_saver pause/resume: #34220
  • [AMD] Fix Triton 3.7 gfx950 extend-attention spills: #34741
  • [AMD] Guard ROCm 7.0 build from using hipMemcpyBatchAsync: #35128
NPU / Ascend
  • [Kimi-K3][NPU] Support Kimi-K3 on NPU: #33465
  • [NPU] Support DeepSeek-V4 DSpark and refactor DSV4 cache management: #33676
  • [NPU] Add mxfp4-w4a8 MOE Quantization Support for NPU: #30318
  • [minimax m3][npu]Adaptation of Minimax M3(w8a8) for NPU platforms [1/2]: #32941
  • [NPU] Adding a fast layernorm for diffusion models and fix BSA: #29027
  • [NPU] Add causal conv1d: #28267
  • [npu] [bugfix] Fix PD‑disaggregation error: #33523
  • [npu] [bugfix] Fix HiCache MHA backup for NPU: #34341
  • [NPU] Modified kernel tag version to 8.10: #34254
CPU / Intel / XPU
  • [CPU] Add support for Gemma4 on Xeon: #22498
  • [CPU][QUANT] add amx cpu support for auto-round: #29593
  • Optimize MiniMax-M2.7 on CPU: #31956
  • [XPU] upgrade sglang xpu backend to PyTorch 2.13: #31751
  • [XPU] xpu kernel release workflow: #33679
  • [XPU] Enable fused GDN QKV split Triton kernel on XPU: #30144
  • [XPU] Pad MoE expert weight row stride to avoid L3 aliasing: #33905
  • [Intel GPU] DeepSeek V4 8/N: use sgl-kernel implementation of fused_k_norm_rope_flashmla on XPU: #28040
  • [Intel GPU] DeepSeek V4 15/N: Add silu_and_mul_clamp support to triton fused_moe for XPU: #33808
  • [XPU] Fix NemotronH (hybrid mamba2) launch on --device xpu: #32227
Security
  • feat: add safeguards for remote media URLs: #34892
  • fix(constrained): reject NUL bytes in grammar specs to stop an xgrammar segfault: #34679
Dependencies
  • [Deps] Upgrade CUDA PyTorch stack to 2.13: #28836 ⭐
  • Bump FlashInfer to 0.6.17 and remove Kimi K3 workarounds: #33997 ⭐
  • Bump CuTeDSL to 4.6.2: #34372 ⭐
  • Install DeepEP from release wheels: #33932 ⭐
  • Build and release sgl-deep-ep wheels: #33498
  • Update sgl-deep-ep release workflow for DeepEP v2: #34914
  • Update AOT kernels for Torch 2.13: #33841
  • chore: bump sgl-kernel version to 0.4.6.post1: #33842
  • [Fix] Pin cuda-tile to 1.6.0rc5 to unblock Python 3.10 x86_64 installs: #34321
  • [AMD] Bump mori to latest in sglang: #33462
Breaking Changes & Upgrade Notes
  • The CUDA PyTorch stack moves to torch 2.13.0, with triton 3.7.1, torchvision 0.28.0, and torchcodec 0.15.0 (torchaudio stays at 2.11.0); AOT kernels are rebuilt for Torch 2.13 and the XPU backend follows: #28836, #33841, #31751
  • Every compiled-kernel cache moves under SGLANG_CACHE_DIR, so the first launch after upgrading recompiles once. Copy or symlink the old directories if you pre-warm or volume-mount them (deep_gemm and flashinfer are the expensive ones): ~/.triton to {SGLANG_CACHE_DIR}/triton, ~/.cache/flashinfer to {SGLANG_CACHE_DIR}/.cache/flashinfer, ~/.cache/deep_gemm to {SGLANG_CACHE_DIR}/deep_gemm, /tmp/torchinductor_$USER to {SGLANG_CACHE_DIR}/inductor, ~/.nv/ComputeCache to {SGLANG_CACHE_DIR}/nv. Setting any of TRITON_CACHE_DIR, FLASHINFER_WORKSPACE_BASE, SGLANG_DG_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR, or CUDA_CACHE_PATH keeps that one where it is: #32434
  • The torchao integration is removed (--torchao-config had raised ImportError for every accepted value since the torchao pin moved to 0.17.0): #34304
  • DeepEP now installs from released sgl-deep-ep wheels instead of building from source: #33932
  • MoE deferred finalize is on by default on the NVFP4 + flashinfer_trtllm path with an unfused, non-TP1 shared expert (the DeepSeek-V3 family); set SGLANG_ENABLE_MOE_DEFERRED_FINALIZE=False to fall back: #33618
  • Unified-cache out-of-window SWA slot freeing is on by default for the unified radix cache; set SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=False to restore the old behavior: #34653
  • Remote media downloads are bounded to 64 MiB by default, with redirect validation and opt-in exact-hostname allowlisting for image, video, and audio URLs: #34892
  • The DSV4 fused MHC post+pre path is on by default, including on SM12x: #35214, #34019
  • torch.compile becomes opt-in for diffusion speed mode: #34173
  • 22 unmaintained benchmarks are removed: #34520
Known Issues
  • The Kimi K3 MLA gate-projection fusion into the QKV-A GEMM was landed and reverted this cycle (#33623, #34642). It is not in this release.
  • The AMD GLM-5.2 fused shared-expert append into aiter grouped-topk was landed and reverted (#31323, #35105). It is not in this release.
  • The parallel request lifecycle tracking that shipped with gRPC generation semantics in v0.5.17 (#32588) is reverted: #34160
  • Several failing NPU test cases are disabled pending fixes: #34377

Full Changelog: v0.5.17...v0.5.18

New Contributors
View original

Upgraded? How did it go?

Discussion