llama.cpp b12099

b12099Pre-release
Added 8
  • Hexagon backend adds support for the LIGHTNING_INDEXER operation, including initial support for the op.
  • Hexagon LIGHTNING_INDEXER adds an HMX path for prefill.
  • Hexagon LIGHTNING_INDEXER adds a q8_0 decode gather transpose and two-block decode compute.
  • Hexagon LIGHTNING_INDEXER adds a pipelined HMX path for both decode and prefill.
  • Hexagon LIGHTNING_INDEXER adds support for K in f32 format.
  • Hexagon LIGHTNING_INDEXER adds support for q4_0 quantized K.
  • Hexagon LIGHTNING_INDEXER adds dimension collapse support.
  • Hexagon backend adds a GGML_HEXAGON_LI_SELECT environment option to force the HMX or HVX paths.
Changed 4
  • Hexagon LIGHTNING_INDEXER removes scalar VTCM reads in the HMX q4_0 K preparation and removes vector spills.
  • Hexagon LIGHTNING_INDEXER host and kernel kparams alignment is updated and software devices are removed from the kernel.
  • Hexagon LIGHTNING_INDEXER DMA pipeline uses canonical double buffering in all paths.
  • Hexagon LIGHTNING_INDEXER kparams are made consistent with other ops.

From llama.cpp

hexagon: add LIGHTNING_INDEXER support (#31289)

  • hexagon: initial support for LIGHTNING_INDEXER op

  • hex-lght-idxr: hmx for prefill

  • hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

  • hex-lght-idxr: pipelined hmx path for decode and prefill

  • hex-lid: add support for K (f32)

  • hex-lid: add support for K (f32)

  • hex-lid: add support for q4_0

  • hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

  • hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

  • hex-lit: update dma pipeline to use canonical dbl buffering in all paths

  • hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

  • hex-lit: add support for dims collapse

  • hex-lit: make kparams constistent with other ops


Co-authored-by: Max Krasnyansky maxk@qti.qualcomm.com

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

View original

Upgraded? How did it go?

Discussion