llama.cpp b12077

b12077Pre-release
Added 1
  • Support for the GLM-5.3 next model's sm tensor split in the meta backend
Fixed 2
  • Per-stream views of permuted tensors can now be split across devices instead of aborting with "view of permuted tensor not implemented" when the view stays within one slab of the split axis
  • MLA attention heads are split across devices in pairs so the HIP tile flash attention kernel for D=512 receives an even GQA ratio, fixing an abort with 3 devices

From llama.cpp

meta: support GLM-5.3 next sm tensor (#30499)

  • meta: support GLM-5.3 next sm tensor

  • glm5-next : cont q_absorbed for -sm tensor with non-unified KV

  • meta : keep GGML_OP_ARANGE in ggml.h order

  • glm5-next : split MLA heads in pairs for -sm tensor

The MLA cache has a single KV head, so the heads on a device are its GQA ratio. The HIP tile FA kernel for D=512 needs an even GQA ratio; with 3 devices the 16 heads of the test model were split 5/5/6 and it aborted.

  • meta : keep the split for views inside one slab of the split axis

With a non-unified KV cache, build_attn_mha splits q into streams with a view. For a permuted q, e.g. the absorbed query of glm5-next, that view aborted with "view of permuted tensor not implemented". The split is now kept if every other dim of the source and the view stays inside one slab of the split axis. The per-device strides are only rescaled above it.

From tetherto/qvac-fabric-llm.cpp#353.

Co-authored-by: Pratik Narola pratik.narola@tether.io

  • glm5-next : drop the q_absorbed cont

The meta backend can now split the per-stream view of the permuted q, so the cont is not needed for -sm tensor with a non-unified KV cache. With the cont, the q of flash attention was not contiguous, and when FA ran on the CPU (WebGPU) the meta get_tensor aborted.


Co-authored-by: Pratik Narola pratik.narola@tether.io

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

View original

Upgraded? How did it go?

Discussion