llama.cpp b12079

b12079Pre-release
Added 2
  • Add a backend interface for collectives across ranks running in separate processes, implemented in the CUDA backend using NCCL
  • Use the NCCL-based collective backend in the RPC server when available, adding nccl support for multi-node tensor parallelism
Changed 3
  • Set maxCTAs in ncclConfig_t to 8 for DGX Spark in ggml_backend_cuda_comm_init_rank
  • Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces
  • Reuse ggml_backend_comm_allreduce_tensor for per-rank communicators and drop ggml_backend_comm_allreduce_rank_t
Removed 1
  • Remove the helper function backend_comm_proc_address

From llama.cpp

Add nccl support for multi-node tensor parallelism (#28967)

  • Add a backend interface for collectives across ranks running in separate processes, implement it in the CUDA backend using NCCL, and use it in the RPC server when available.

  • Drop ggml_backend_comm_allreduce_rank_t and reuse ggml_backend_comm_allreduce_tensor for per-rank communicators

  • reuse ggml_backend_cuda_comm_allreduce_nccl for per-rank communicators. Remove the helper function backend_comm_proc_address

  • Change ggml_backend_comm_get_unique_id_t and ggml_backend_comm_init_rank_t to more generic multi-process backend communicator interfaces

  • set maxCTAs in ncclConfig_t to 8 for dgx spark in ggml_backend_cuda_comm_init_rank

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

View original

Upgraded? How did it go?

Discussion