LocalAI

🎉 LocalAI 4.9.0 Release! 🚀

LocalAI 4.9.0 is out!

Thirteen days and 146 pull requests, spent on the parts of LocalAI you touch every day rather than on new engines. Authentication is now deny-by-default, chat gained end-to-end context compression, models and backends each have one canonical page instead of three, and vllm-cpp grew a video modality serving MiniMax-H3 with a real audio track.

Highlights:

• 🔐 Authentication is deny-by-default - every HTTP route requires credentials unless it appears in an explicit public registry. This closes a class of bypass in which unprefixed aliases such as /moderations, /models, /backends and /mcp/chat/completions fell outside the old protected-prefix list. Reported by Naor Yaacov.
• 🗜️ Chat context compression - opt-in per model, older complete turns are compressed through a LocalAI model before inference, preserving system prompts, the newest messages and whole tool-call units. Ratio and duration come back as response metadata and metrics.
• 🖥️ One page per resource - /app/models now owns Explore and Installed, /app/backends owns Catalog and Installed, and the nested Host view is gone. Old /app/manage bookmarks still work.
• 🎬 MiniMax-H3 video generation - vllm-cpp opens a second engine handle for the H3 checkpoint set and renders video and audio jointly, so the MP4 arrives with a real AAC track. Ask for speech in the prompt and the model lip-syncs it.
• 🗣️ Qwen3-TTS on llama.cpp - text-to-speech on the full accelerator matrix already shipped for text generation (CUDA, ROCm, SYCL, Vulkan, Metal, L4T), using upstream's own GGUF conversion.
• 🧭 KNN as a first-class router - similarity-weighted voting over a curated, persisted corpus of labelled prompts. No classifier model, and a prompt unlike anything labelled is treated as undecidable rather than guessed.
• 📊 Global admission control and live backend traces - process-wide HTTP admission bounds, in-flight backend operations are represented while they run, and the UI links straight to their logs.
• 🕵️ Reversible PII pseudonyms - masked values become request-scoped deterministic pseudonyms (EMAIL_001) and are restored if the backend echoes them, across JSON and SSE tokens split over writes.
• 📦 Parallel Hugging Face downloads - snapshot materialization runs up to N whole-file transfers at once, so a repository split into many shards stops spending its wall clock in per-file latency.
• 🖧 Cold model loads are durable jobs - the per-model advisory lock no longer spans a multi-GB transfer, which had made a 35.7 GB load look permanently broken from the operator's seat while staging progressed normally underneath.
• 🎮 vllm-cpp covers the cards you own - CUDA builds went from one or two architectures to eight on amd64 and five on arm64, picking up A100, L4, 4090, H100/H200, B200, Jetson Orin and Jetson Thor.

Plus a single shared WebRTC UDP port for Realtime, Metal actually enabled in the macOS Stable Diffusion and Parakeet builds, backend crash diagnostics at the default log level, and Portuguese (Brazil) and Indonesian UI translations.

---

📊 This release in numbers

Pull requests merged	146
Commits	148
Files changed	420 (+27,788 / -4,670)
Development window	13 days (2026-08-07 to 2026-08-20)
Human contributors	10, of whom 3 first-time
Gallery entries	1,622 to 1,707 (+85)

Where the work landed:

Area	Change
core/	+17,193 / -4,187 across 279 files
gallery/	+3,808 / -62
backend/	+3,298 / -177 across 60 files
pkg/	+1,180 / -113 across 31 files
swagger/	+1,095 / -5
docs/	+952 / -98 across 23 files

---

📌 TL;DR

Area	Summary
🔐 Auth by default	Every method and path requires credentials unless listed in an explicit public registry. API instructions, Swagger GETs, the LocalAI well-known document, and the health, login, OAuth, SPA, asset, branding and node registration-token bootstrap flows stay public. Migration: with database auth or legacy API keys configured, /version and generated audio, image, video and 3D URLs now require credentials. Embedded deployments can add narrow prefixes through ApplicationConfig.PathWithoutAuth, and the legacy GET exemption flags remain as explicit compatibility overrides.
🗜️ Context compression	Opt-in per-model compression config. Older complete turns are compressed through a configured LocalAI model before inference, after PII filtering and Assistant/MCP prompt injection (including later MCP iterations). Leading system/developer prompts, the newest messages and complete tool-call/result units are preserved; tool schemas and completion headroom are accounted for with a conservative offline token bound. Metadata rides non-streaming responses and streaming usage trailers, with event, ratio and duration metrics exported. Disabled by default; cloud-proxy passthrough is rejected (translate mode works).
🖥️ Unified lifecycle UI	/app/models owns Explore and Installed with URL-backed search, state and selection; /app/backends owns Catalog and Installed while keeping variants, development builds and target-node scope. Explore offers capability-aware Open and Manage installation; destructive model controls stay in Installed. Operate Overview shows host capacity from its shared summary poller, and the nested Host destination is removed. /app/manage redirects while preserving legacy query state. No API change.
📥 Import form rebuild	The import page moves to page--medium with a work column and the format reference beside it rather than behind a closed chevron. The source field is the hero and carries its own Import button, which removes the aria-hidden submit that existed only because the real action sat outside the <form>. Simple and Advanced modes are gone (about 80% the same surface); the real distinction, a source or YAML, is now two tabs. Also fixes two class bugs: a primary button with no className at all falling through to browser chrome, and class="btn btn-primary fas fa-save fa-upload" setting Font Awesome as the button's own font while two icons fought over one ::before.
🎬 MiniMax-H3 video	vllm-cpp over vllm.cpp ABI v12. A second engine handle loads the H3 checkpoint set (the DiT is parameters.model, the text encoder and two VAEs are named in options:), and GenerateVideo renders video and audio jointly into an MP4 with a real AAC track. The DiT partition is declared, not detected: community quantizations strip the release metadata and the FL2VA and Ref2VA DiTs are byte-structurally identical, so checkPartitionConditioning refuses a reference-conditioned FL2VA request before the engine runs (it would otherwise render for hours and return a coloured lattice). ffmpeg comes from the host: libvllm composes the mux argv and spawns nothing. New gallery entry minimax-h3-fl2va-q4.
🗣️ Qwen3-TTS on llama-cpp	TTS through the llama-cpp backend on CUDA, ROCm, SYCL, Vulkan, Metal and L4T, using upstream's GGUF conversion. Implemented as a slot-based SERVER_TASK_TYPE_TTS task, which is the concurrency-safe integration given that server_context owns the llama_context and runs the slot scheduler on its own thread. Carries the still-draft upstream server hunks as patches/0002-add-server-task-type-tts.patch (delete on merge of ggml-org/llama.cpp#26603). Gallery: qwen3-tts-llamacpp and qwen3-tts-llamacpp-q4. The existing qwen3-tts-cpp backend is untouched and remains a separate path.
🧭 KNN routing	classifier: knn routes by similarity-weighted voting over labelled example prompts, so no classifier model is needed and label knowledge lives in a corpus you seed and curate. Entries below knn.similarity_threshold cannot vote; when none clears it the router takes the fallback, and nearest_similarity is recorded on decisions and fallbacks alike. One JSONL file per router under <data path>/router-corpus is the source of truth, with the in-memory index rebuilt at classifier build time and entries re-embedded when the embedding model changed. Corpus input is API-only by design: POST /api/router/{name}/corpus, GET .../corpus/stats (label counts only, texts are never returned), DELETE .../corpus, admin-gated and exposed as MCP tools.
📊 Admission and traces	Process-wide HTTP admission control, bounding what was previously only per backend. Backend operations are represented while in flight, and running backend traces surface in the UI with immediate log links.
🕵️ PII pseudonyms	Opt-in pii.reverse_in_response. Masked request values become unique deterministic pseudonyms within the request (EMAIL_001, EMAIL_002) and are restored if the backend returns them, including SSE tokens split across response writes. Substitution maps are request-local and never persisted. Irreversible [REDACTED:...] remains the default.
📦 Parallel HF downloads	DownloadFilesWithConcurrency runs up to N whole-file transfers through an errgroup with SetLimit. Single files are never split, so .partial resume and per-file SHA verification are untouched, and the two non-artifact callers keep sequential ordering and fail-fast behaviour through a limit-of-1 wrapper. completedBytes became an atomic.Int64 (the race detector reported three races otherwise) and the caller's status callback stays serialized.
📞 Realtime WebRTC port	--web-rtc-udp-port / LOCALAI_WEBRTC_UDP_PORT reuses one Pion ICE UDP mux across Realtime calls, with bind failures surfaced through signaling and container/firewall setup documented. The follow-up fix keeps LOCALAI_WEBRTC_ICE_INTERFACES effective when a fixed port is set, which had been silently ignored: a wildcard mux made pion enumerate every interface itself, handing browsers unroutable 172.x candidates that dropped once ICE consent checks failed.
🖧 Durable cold loads	The per-model advisory lock is a dedup decision measured in milliseconds, not a transfer's lifetime. Cold loads now run as durable jobs instead of holding it across backend install, multi-GB staging and checkpoint load, and WithLockCtx now defends against statement_timeout as well as lock_timeout (both abort the same blocking pg_advisory_lock, only the latter was overridden).
🎮 vllm-cpp CUDA coverage	amd64 goes from 120a;121a to 80;86;89;90a;100a;103a;120a;121a, arm64 from 121a to 87;90a;100a;110;121a, split by where the silicon exists. An unlisted card did not run slower, it died at the first request with no kernel image is available for execution on the device, long after install reported success. The CUDA 13 guard now covers both branches, and Triton-AOT stays on.
🌍 Two new languages	Portuguese (Brazil), a complete 14-namespace translation at full key parity with en/, and Indonesian for the admin, media and navigation surfaces.
🧠 Models	85 new gallery entries: Qwen3.8 (9B, 27B, Ridge and small variants), Gemma 4 agentic and Scotoma 2, DeepSeek V4 Pro 0813, Ling 3.0 Flash, Nemotron 3.5 Lightning 30B, Tess 4 27B, Ornith 1.0 and 1.5 9B, LFM2.5 230M and VL 1.6B, HunyuanOCR and OvisOCR2, Higgs Audio v3 TTS, MiniMax-H3 Ref2VA, plus vllm.cpp text-generation entries and a first Carbon genomics family.

---

🚀 New Features & Major Enhancements

🔐 Authentication now denies by default

The previous classifier gated selected API-style paths by prefix. Anything whose path was not on that list was public, which meant unprefixed aliases (/mcp/chat/completions, /moderations, /models, /backends, /import-model) could bypass global authentication, and any newly registered route inherited the same weakness by default.

The middleware is now method-aware and denies by default: a route is public only if its method and path appear in an explicit public registry. What stays public is the set required to bootstrap and to be discoverable: API instructions, Swagger GET routes, the LocalAI well-known document, and the health, login, OAuth, SPA, asset, branding and node registration-token flows. Whole-router coverage is asserted in tests, so a new route cannot become public by omission.

Migration impact. When database authentication or legacy API keys are configured, /version and generated audio, image, video and 3D URLs now require credentials. Embedded deployments can still add narrow prefixes via ApplicationConfig.PathWithoutAuth, and the legacy GET exemption flags remain available as explicit compatibility overrides.

Thanks to Naor Yaacov for reporting this class of authentication bypass.

🔗 PRs: #11602

🗜️ End-to-end context compression

A long conversation eventually stops fitting. Compression is opt-in per model, and when enabled it compresses older complete turns through a configured LocalAI model before inference rather than truncating them away.

What it will not touch: leading system and developer safety prompts, the newest messages, and complete tool-call/result units, which are kept whole so a compressed history never leaves a call without its result. It runs after PII filtering and after Assistant/MCP prompt injection, including on later MCP iterations, so what gets compressed is the prompt that would actually have been sent. Tool schemas and completion headroom are accounted for with a conservative offline token bound.

Compression metadata is exposed in non-streaming responses and in streaming usage trailers, and compression events, ratios and durations are exported as metrics. Cloud-proxy passthrough configurations reject compression because LocalAI cannot safely rewrite an opaque provider payload; translate mode is supported. A late failure in an already-started stream is returned as an in-band SSE error followed by [DONE].

🔗 PRs: #11556

🖥️ One canonical page per resource

Models had a gallery and a separate Host management surface. Backends had a nested Host view for installed binaries. Between them it was not obvious where a resource lived, and the common lifecycle actions sat one level deeper than they needed to.

Each resource now has one page. /app/models owns Explore and Installed, /app/backends owns Catalog and Installed, both with URL-backed search, state and selection, and backends keep their variants, development builds and target-node scope. Explore presents capability-aware Open and Manage installation actions while destructive model controls stay in Installed. Operate Overview reads host capacity from its shared summary poller, and the nested Host destination is removed. Narrow list/detail views restore focus to the originating row when you come back from a detail view.

Nothing in the API changed, and existing /app/manage bookmarks keep working through a replace redirect that preserves legacy model and backend query state.

🔗 PRs: #11548

📥 The import form, rebuilt

The import page had taken the new palette but kept its old layout: a 760px column with the primary action detached from the form it submits. Two of its problems were outright bugs.

ImportModel.jsx:808 carried no className at all, so the page's single most important control fell through to the user-agent button, with system chrome, system font, the wrong radius and no design-system focus ring. Next to it, class="btn btn-primary fas fa-save fa-upload" set Font Awesome as the button's own font family, which its label text inherited, while fa-save and fa-upload fought over one ::before.

The layout moves to page--medium with a work column and the format reference beside it, since that reference answers the only question a first-time admin has and used to sit behind a chevron that was closed by default. Below 1024px it becomes a disclosure instead of disappearing. The source field is the hero, monospace because it holds something you paste, and it carries its own Import button, which removes the aria-hidden submit that existed only to compensate for the real action sitting outside the <form>. Simple and Advanced modes are gone: they were about 80% the same surface, and the overlap cost a mode switch, a localStorage key and a three-button Keep/Discard/Cancel dialog whose only job was protecting state the switch would have hidden. What genuinely differs is the kind of input, which is now two tabs: a source, or YAML. The size and VRAM estimate reports under the field that produced it instead of as a banner above the page header.

A follow-up swept the same class of bug across the rest of the UI: eight header controls on seven pages had two or three elements' classes collapsed into one string.

🔗 PRs: #11461, #11462, #11488

🎬 MiniMax-H3: video and audio, jointly

vllm.cpp's stable C ABI grew a video slice at v12, and vllm-cpp now serves two things. Text generation is unchanged. When a model config declares the H3 checkpoint set, Load opens a video engine instead and GenerateVideo renders a clip through LocalAI's existing /video endpoint, with video and audio generated jointly, so the MP4 comes back with a real AAC track rather than silent. Ask for speech in the prompt and the model lip-syncs it.

Three things shape the integration:

The video engine is a second handle, not a mode of the first. H3 is not a model directory. The DiT, the text encoder and two VAEs are separate artifacts, and vllm.cpp's two loaders refuse each other's checkpoints. parameters.model is the DiT; the rest of the set is named in options:.

The DiT partition is declared, not detected. The FL2VA DiT serves t2va and fl2va; ref2va is a different checkpoint. Community GGUF and NVFP4 quantizations strip the release metadata and the two DiTs are byte-structurally identical, so the engine refuses to generate until it is told which it has. Handing reference conditioning to an FL2VA DiT renders for hours and returns a coloured lattice over the frame, so checkPartitionConditioning rejects that combination before the engine is ever called.

ffmpeg comes from the host. libvllm writes the frames and the WAV and composes the mux argv, then spawns nothing, which is a deliberate upstream process boundary. The backend substitutes argv[0] and execs it.

Gallery entries minimax-h3-fl2va-q4 (Q4_K_M FL2VA set) and minimax-h3-ref2va-q4 ship with it.

🔗 PRs: #11424, #11439

🗣️ Qwen3-TTS through the llama.cpp backend

Qwen3-TTS now runs on the llama-cpp backend, which means text-to-speech on the same accelerator matrix already shipped for text generation (CUDA, ROCm, SYCL, Vulkan, Metal, L4T) using upstream's own GGUF conversion.

grpc-server.cpp is an adapter over llama.cpp's shared server_context, which owns the llama_context and runs the slot scheduler on its own thread, so a gRPC handler driving the gen-audio loop itself would race that scheduler. Making TTS a slot-based SERVER_TASK_TYPE_TTS task is the concurrency-safe integration, following the 0001-add-server-task-type-score.patch precedent already in the tree. llama.cpp merged Qwen3-TTS in ggml-org/llama.cpp#26254; the server plumbing in #26603 is still a draft, so it is carried as patches/0002-add-server-task-type-tts.patch and should be deleted once that merges. disable-tts-task.sh keeps turboquant and bonsai compiling, since they copy grpc-server.cpp into forks without our patches.

Both paths were verified end to end on CPU returning valid 24 kHz mono 16-bit WAV containing real speech, measured rather than eyeballed. Gallery entries: qwen3-tts-llamacpp and qwen3-tts-llamacpp-q4.

The existing qwen3-tts-cpp backend over qwentts.cpp is untouched. This is a second, independent path, not a replacement.

🔗 PRs: #11392

🧭 KNN as a first-class router

classifier: knn promotes KNN search from a cache for the classifier to a primary request router. Unlike score or colbert it needs no classifier model: label knowledge lives in a corpus of labelled example prompts that you seed and curate through the admin API, so routing decisions are deterministic, auditable, and grounded in graded experience rather than a model's opinion.

There is an explicit epistemic gate. Corpus entries below knn.similarity_threshold cannot vote, and when none clears it the classifier activates no labels and the router takes the fallback: a prompt unlike all labelled experience is treated as undecidable, not guessed. Decisions record nearest_similarity, on fallback rows too, so you can see how far the nearest labelled experience actually was, and the Routing tab explains out-of-corpus fallbacks and shows per-label corpus counts.

Persistence is one JSONL file per router under <data path>/router-corpus holding text, labels, vector and embedder fingerprint. That file is the source of truth; the local-store index is rebuilt from it at classifier build time and stays a pure in-memory index, and entries recorded under a different embedding model re-embed on load. This also corrects the docs' claim that local-store collections persist: the embedding cache never survived restarts and still does not, while the corpus does.

Corpus input is API-only by design, since entries may contain example user content: POST /api/router/{name}/corpus seeds (labels validated against declared policies, embedded server-side, indexed immediately), GET .../corpus/stats inspects and returns label counts only (entry texts are never returned by any surface), and DELETE .../corpus wipes. All admin-gated like the sibling router endpoints and exposed as MCP tools.

🔗 PRs: #10652

📊 Global admission control and running backend traces

Admission control existed per backend, which left nothing bounding the process as a whole, and the traces list could grow without limit. HTTP admission is now bounded process-wide.

Alongside it, backend operations are represented while they are still in flight rather than only once they finish, and running backend traces surface in the UI with immediate links to their logs, so an operation that is taking too long is something you can look at instead of something you wait out.

🔗 PRs: #11560

🕵️ PII pseudonyms that survive the round trip

The PII middleware masked values irreversibly, which is right for logs and wrong for a conversation: a model that is handed [REDACTED:EMAIL] twice cannot tell whether it saw one address or two, and anything it says about them comes back unusable.

Opt-in pii.reverse_in_response turns masked request values into unique deterministic pseudonyms within the request (EMAIL_001, EMAIL_002) and restores them if the backend returns them. Restoration handles normal JSON and SSE tokens split across response writes. Substitution maps stay request-local and are never persisted. Irreversible [REDACTED:...] remains the default.

🔗 PRs: #11272

📦 Bounded parallel Hugging Face downloads

Snapshot materialization fetched every file through the sequential executor, so a repository split into many shards spent most of its wall clock in per-file request latency rather than moving bytes.

DownloadFilesWithConcurrency now runs up to N whole-file transfers at once through an errgroup with SetLimit. Only whole files run in parallel: a single file is never split, so the .partial resume machinery and the per-file SHA check are untouched. The two non-artifact callers (core/gallery/models.go and core/config/model_config_loader.go) keep exactly their previous behaviour through a wrapper passing a limit of 1, so tasks still run in slice order and the first failure still returns before any later task starts.

Two consequences of the parallel path are worth knowing: completedBytes is now an atomic.Int64, which is not a precaution (with a plain int64 the race detector reports three races), and the caller's status callback is serialized to preserve the guarantee the sequential path gave it implicitly. AfterDownload is deliberately not serialized, because it does the verify-and-promote work that the parallelism exists to overlap.

🔗 PRs: #11162

📞 Realtime WebRTC on one UDP port

--web-rtc-udp-port / LOCALAI_WEBRTC_UDP_PORT reuses a single Pion ICE UDP mux across Realtime WebRTC calls, so a container or firewall needs one rule rather than a range. UDP bind failures are surfaced through signaling instead of failing opaquely, and the container and firewall setup is documented.

A follow-up closed the gap it opened. LOCALAI_WEBRTC_ICE_INTERFACES was silently ignored whenever a fixed UDP port was set, which is exactly the combination an operator reaches for: pinning a port to write a firewall rule and restricting interfaces to keep unreachable docker0 and veth addresses out of the candidate list usually go together. A wildcard mux made pion derive candidate addresses by enumerating interfaces itself with a nil filter, so on a host with docker bridges the browser received 172.18.0.1, 172.17.0.1, 10.10.10.1 and friends, connected on a good pair, then dropped when ICE consent checks failed on the others.

🔗 PRs: #11436, #11466

🖧 Cold model loads run as durable jobs

On a two-replica frontend, loading a 35.7 GB GGUF onto a newly added Jetson Thor worker made the model permanently unloadable from the operator's seat, while staging was in fact progressing normally underneath. Replica A held the per-model advisory lock through roughly twenty minutes of transfer; replica B blocked on pg_advisory_lock for the same model and was killed at 60s by the role's statement_timeout, and every UI retry reproduced it.

Two defects sat behind that one symptom. The lock's lifetime was the transfer's lifetime, with Route wrapping backend install, multi-GB staging and checkpoint load in advisorylock.WithLockCtx, which turns a millisecond dedup decision into a cluster-wide outage for that model. And WithLockCtx overrode lock_timeout but not statement_timeout, though both abort the same blocking call.

Cold loads now run as durable jobs, so the lock is held only for the decision it exists to make.

🔗 PRs: #11514

🎮 vllm-cpp builds for the cards people own

The vllm-cpp CUDA images were built for Blackwell only: 120a;121a on amd64 and 121a alone on arm64, out of the ten architectures vllm.cpp's own release archive builds.

What makes it worth calling out is the failure mode. An unlisted card is not slower, it dies at the first request with no kernel image is available for execution on the device, long after local-ai backends install vllm-cpp reported success. This was found on a Jetson Thor node that had the backend installed and could serve nothing.

	before	after
amd64	120a;121a	80;86;89;90a;100a;103a;120a;121a
arm64	121a	87;90a;100a;110;121a

The split follows where the silicon exists: Jetson (87 Orin, 110 Thor) is arm64-only, desktop 120a is amd64-only, and 90a/100a are on both because of the SBSA parts. That adds A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin and Jetson Thor. The CUDA 13 guard now covers both branches rather than amd64 alone, and Triton-AOT stays on.

🔗 PRs: #11512

🌍 Portuguese (Brazil) and Indonesian

A complete pt-BR translation of the WebUI: 14 namespaces at full key parity with en/ including modelEditor.json, with every i18next interpolation variable and _one/_other plural key preserved. Indonesian covers the admin, media and navigation strings. Both keep brand, model and technical identifiers untranslated, matching existing locale conventions.

🔗 PRs: #11427, #11493

🧰 Smaller features worth knowing about

• Metal is actually on in two macOS backends. stablediffusion-ggml gated its Metal flags on an OS=Darwin variable the runner never defines, so the Darwin workflow's BUILD_TYPE=metal produced a build without GGML_METAL_EMBED_LIBRARY=ON, shipping a runtime source path that failed to expose kernel_mul_mv_ext_bf16_f32_r1_5. parakeet-cpp never forwarded BUILD_TYPE=metal to PARAKEET_GGML_METAL at all: on an M1 Air the same five-minute sample went from 82.57s to 50.18s with byte-identical output. A dry-run build-contract test now guards the Stable Diffusion flags.
• Backend crashes say why. An unexpected runtime exit logged its stderr only at debug level, so at default log level operators saw an exit code and nothing else. The final non-empty stderr line now rides the unexpected-exit warning, and a failed gRPC readiness preserves the process exit code plus the last stderr diagnostic bounded to 4 KiB.
• MCP servers stay visible when they fail. Model-level MCP servers disappeared from Chat whenever connection setup or tool discovery failed, hiding container DNS, routing and VPN reachability problems. They now stay listed as disabled error rows with a per-server error, discovery retries while safely closing partially created sessions, and the model editor documents the expected mcp.remote and mcp.stdio formats.
• Checksum mismatches retry. A post-download SHA mismatch is now treated as a transient transfer failure: LocalAI removes the mismatched partial and lets the bounded planner retry a stale or corrupted CDN response, while still refusing unverified bytes. Direct URI.DownloadFile callers still see the mismatch immediately.
• Audio transform rejects the wrong contract. The audio-transform WebSocket accepted realtime_audio models and opened the frame-based AudioTransformStream RPC, which failed after the handshake with NotImplementedError for any-to-any models like liquid-audio. The use case is now validated before the backend loads, and any-to-any callers are pointed at the OpenAI Realtime API.
• Invalid preload JSON names itself. PRELOAD_MODELS / --preload-models now identifies which input was invalid and rejects non-array top-level values (including booleans and null) with the expected shape in the error.

🔗 PRs: #11531, #11492, #11532, #11447, #11495, #11536, #11565, #11434

---

🐛 Bug Fixes (recap)

• fix(auth): protect HTTP routes by default - #11602
• fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock - #11514
• fix(vllm-cpp): build every CUDA architecture the platform can host - #11512
• fix(realtime): keep the ICE interface allow-list working with a fixed UDP port - #11466
• fix(stablediffusion): embed Metal library - #11531
• fix(parakeet-cpp): enable Metal in macOS builds - #11492
• fix(model): report backend crash diagnostics - #11532
• fix(model): surface backend startup exits - #11447
• fix(downloader): retry checksum mismatches - #11536
• fix(audio): reject incompatible transform streams - #11565
• fix(gallery): parse harmony output of gpt-oss-* models correctly - #11518
• fix(gallery): identify invalid preload JSON - #11434
• fix(gallery): repair DeepSeek V4 fallback - #11480
• fix(gallery): correct Higgs Audio v3 checksum - #11459
• fix(fish-speech): preserve ROCm PyTorch - #11568
• fix(vllm): align Intel basekit runtime to oneAPI 2025.3.2 - #11437
• fix(kokoros): add the missing upscale_image stub to the Backend trait impl - #11414
• fix: show MCP connection errors in the UI - #11495
• fix(ui): unmerge the class strings that left buttons in browser chrome - #11462
• fix(ui): keep agent import action visible - #11488
• fix: wrap long TTS request text instead of widening the page - #11576

---

🧠 Models

85 new gallery entries this cycle, taking the index from 1,622 to 1,707.

Text generation: Qwen3.8 in 9B, 27B, Ridge and small variants, Gemma 4 agentic and Gemma 4 Scotoma 2, DeepSeek V4 Pro 0813, Ling 3.0 Flash, Nemotron 3.5 Lightning 30B, Tess 4 27B, Ornith 1.0 and 1.5 9B, Muse Glimmer 30B, Grug 12B, BigBang v1, Genesis Hermes V7, TwIL-LM3, BTL-4 Compact, XYZ Aquila mini, North Mini Code, AREX Turbo, Fara1.5 4B, MiniCPM5 1B Q8, Shieldstral 1.0 3B and UI-Mate 9B.

Vision and OCR: HunyuanOCR, OvisOCR2, LFM2.5 VL 1.6B, and LFM2.5 230M alongside it.

Audio and video: Higgs Audio v3 TTS, the Qwen3-TTS llama.cpp entries, and the MiniMax-H3 FL2VA and Ref2VA video sets.

Also a first Carbon genomics family, and text-generation entries for the vllm-cpp backend.

🔗 PRs: #11622, #11603, #11599, #11598, #11594, #11584, #11573, #11571, #11561, #11559, #11557, #11552, #11551, #11549, #11547, #11540, #11533, #11526, #11519, #11511, #11490, #11479, #11478, #11477, #11458, #11456, #11455, #11449, #11446, #11443, #11441, #11439, #11438, #11435

---

👒 Dependencies

Submodule and pin bumps this cycle:

Project	Bumps
CrispStrobe/CrispASR	10
ikawrakow/ik_llama.cpp	9
vllm-metal (darwin)	7
mudler/vllm.cpp	6
0xShug0/audio.cpp	5
ggml-org/llama.cpp	4
ggml-org/whisper.cpp	3
leejet/stable-diffusion.cpp	2
antirez/ds4, mudler/parakeet.cpp, mudler/depth-anything.cpp, NVIDIA/NeMo-Speech.cpp, vllm-project/vllm cu130 wheel	1 each

Plus golang.org/x/net to v0.55.0, vllm 0.26.0 and transformers >=5.15.0 in the Python backends, sentence-transformers 5.7.0, packaging 26.3, dompurify 3.4.13, the Kokoros source pin, inference defaults refreshed from unsloth, and seven gallery checksum refreshes.

---

📖 Documentation

New pages for context compression and vllm-cpp, a substantially expanded middleware page covering the PII pseudonym and compression surfaces, and a rewritten authentication page documenting the public and protected route surfaces after the deny-by-default change.

Video generation gained the MiniMax-H3 setup, text-to-audio the Qwen3-TTS llama.cpp path, distributed mode the durable cold-load behaviour, MCP the configuration formats and container networking implications, and Realtime the shared UDP port and firewall guidance. Embeddings, model gallery, backends and API discovery all picked up corrections, and the broken stars counter came out of the site.

🔗 PRs: #11448, #11581, #11582, #11415

---

🙌 New Contributors

• @tom-mi made their first contribution in #11518
• @kassane made their first contribution in #11427
• @fieryWaters made their first contribution in #11492

Thanks also to @richiejp, @jimmykarily, @Dennisadira, @dedyf5, @ALameLlama and @walcz-de, and to Naor Yaacov for the authentication bypass report.

What's Changed

Breaking Changes 🛠

• fix(auth): protect HTTP routes by default by @localai-bot in https://github.com/mudler/LocalAI/pull/11602

Bug fixes :bug:

• fix(kokoros): add missing upscale_image stub to Backend trait impl by @mudler with @Copilot in https://github.com/mudler/LocalAI/pull/11414
• fix(gallery): identify invalid preload JSON by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11434
• fix(vllm): align Intel basekit runtime by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11437
• fix(model): surface backend startup exits by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11447
• fix(ui): unmerge the class strings that left buttons in browser chrome by @localai-bot in https://github.com/mudler/LocalAI/pull/11462
• fix(realtime): keep the ICE interface allow-list working with a fixed UDP port by @jimmykarily in https://github.com/mudler/LocalAI/pull/11466
• fix(parakeet-cpp): enable Metal in macOS builds by @fieryWaters in https://github.com/mudler/LocalAI/pull/11492
• fix: Show MCP connection errors in the UI by @richiejp in https://github.com/mudler/LocalAI/pull/11495
• fix(vllm-cpp): build every CUDA architecture the platform can host by @localai-bot in https://github.com/mudler/LocalAI/pull/11512
• fix (gallery): Parse harmony output of gpt-oss-* models correctly (#8037) by @tom-mi in https://github.com/mudler/LocalAI/pull/11518
• fix(ui): keep agent import action visible by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11488
• fix(stablediffusion): embed Metal library by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11531
• fix(model): report backend crash diagnostics by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11532
• fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock by @localai-bot in https://github.com/mudler/LocalAI/pull/11514
• fix(downloader): retry checksum mismatches by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11536
• fix(fish-speech): preserve ROCm PyTorch by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11568
• fix(audio): reject incompatible transform streams by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11565
• fix: tts text wrap by @ALameLlama in https://github.com/mudler/LocalAI/pull/11576

Exciting New Features 🎉

• feat(modelartifacts): support bounded parallel Hugging Face file downloads by @Dennisadira in https://github.com/mudler/LocalAI/pull/11162
• feat(i18n): add pt-BR translation by @kassane in https://github.com/mudler/LocalAI/pull/11427
• feat(vllm-cpp): serve MiniMax-H3 video+audio generation by @localai-bot in https://github.com/mudler/LocalAI/pull/11424
• feat(pii): restore request-scoped pseudonyms by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11272
• feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend by @localai-bot in https://github.com/mudler/LocalAI/pull/11392
• feat(realtime): add shared WebRTC UDP port by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11436
• feat(ui): rebuild the import form on the restyled design language by @localai-bot in https://github.com/mudler/LocalAI/pull/11461
• i18n(id): translate admin, media, and nav UI strings to Indonesian by @dedyf5 in https://github.com/mudler/LocalAI/pull/11493
• feat(ui): unify model and backend lifecycle by @localai-bot in https://github.com/mudler/LocalAI/pull/11548
• feat: bound global admission and expose running backend traces by @richiejp in https://github.com/mudler/LocalAI/pull/11560
• feat(router): make KNN a first-class classifier with a persisted, curated corpus by @richiejp in https://github.com/mudler/LocalAI/pull/10652
• feat(chat): add end-to-end context compression by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11556

🧠 Models

• feat(gallery): add Grug 12B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11438
• feat(gallery): add Shieldstral 1.0 3B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11441
• feat(gallery): add Genesis Hermes V7 variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11435
• feat(gallery): add minimax-h3-ref2va-q4 - the Ref2VA partition, verified coherent by @localai-bot in https://github.com/mudler/LocalAI/pull/11439
• feat(gallery): add BigBang v1 variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11443
• feat(gallery): add Gemma 4 Scotoma 2 variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11458
• feat(gallery): add Muse Glimmer 30B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11446
• feat(gallery): add Higgs Audio v3 TTS by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11456
• feat(gallery): add BTL-4 Compact by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11449
• feat(gallery): add XYZ Aquila mini variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11455
• fix(gallery): correct Higgs Audio v3 checksum by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11459
• chore: bump inference defaults from unsloth by @github-actions[bot] in https://github.com/mudler/LocalAI/pull/11270
• fix(gallery): align Higgs Audio checksum test by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11476
• feat(gallery): add Nemotron 3.5 Lightning 30B by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11477
• feat(gallery): add TwIL-LM3 variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11478
• fix(gallery): repair DeepSeek V4 fallback by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11480
• feat(gallery): add Fara1.5 4B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11479
• feat(gallery): add LFM2.5 VL 1.6B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11490
• feat(gallery): add Qwen3.8 27B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11519
• feat(gallery): add DeepSeek V4 Pro 0813 by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11533
• feat(gallery): add LFM2.5 230M variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11526
• feat(gallery): add vllm.cpp text-generation models by @localai-bot in https://github.com/mudler/LocalAI/pull/11511
• feat(gallery): add HunyuanOCR variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11540
• feat(gallery): add Tess 4 27B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11547
• feat(gallery): add AREX Turbo variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11551
• feat(gallery): add OvisOCR2 variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11549
• feat(gallery): add Qwen3.8 9B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11557
• feat(gallery): add Ornith 1.0 9B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11552
• feat(gallery): add Qwen3.8 Ridge variant by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11573
• feat(gallery): add Qwen3.8 small variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11571
• feat(gallery): add MiniCPM5 1B Q8 variant by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11561
• chore(model gallery): :robot: add 1 new models via gallery agent by @localai-bot in https://github.com/mudler/LocalAI/pull/11596
• feat(gallery): add North Mini Code variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11559
• feat(gallery): add UI-Mate 9B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11584
• feat(gallery): add Ling 3.0 Flash variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11598
• feat(gallery): add Hy-MT2 1.8B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11599
• feat(gallery): add Carbon genomics variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11594
• chore(model gallery): :robot: add 1 new models via gallery agent by @localai-bot in https://github.com/mudler/LocalAI/pull/11605
• feat(gallery): add Ornith 1.5 9B variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11603
• feat(gallery): add Gemma 4 agentic variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11622

📖 Documentation and examples

• docs: :arrow_up: update docs version mudler/LocalAI by @localai-bot in https://github.com/mudler/LocalAI/pull/11415
• docs: remove broken stars counter by @richiejp in https://github.com/mudler/LocalAI/pull/11448
• docs(adopters): add walcz.de by @walcz-de in https://github.com/mudler/LocalAI/pull/11581
• docs(ecosystem): add Walcz to contributors by @walcz-de in https://github.com/mudler/LocalAI/pull/11582

👒 Dependencies

• chore: :arrow_up: Update ggml-org/whisper.cpp to 592feef04a1802b18cbeffd0fd0eb5d02570c2ec by @localai-bot in https://github.com/mudler/LocalAI/pull/11416
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11418
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to 40dffce6857b4fe051f096379dc464764c718458 by @localai-bot in https://github.com/mudler/LocalAI/pull/11403
• chore: :arrow_up: Update CrispStrobe/CrispASR to 17a6cc99422bfafadf7161e96dd7294c89da9c36 by @localai-bot in https://github.com/mudler/LocalAI/pull/11404
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to f2328aa0c19954d0ab31a3de60fbf50e47c2429f by @localai-bot in https://github.com/mudler/LocalAI/pull/11421
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11423
• chore: :arrow_up: Update antirez/ds4 to 84cc882352757baf628a1776badf7cc54d584e28 by @localai-bot in https://github.com/mudler/LocalAI/pull/11432
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to a7c81affa48c6800d63111bdb33469a01d062daa by @localai-bot in https://github.com/mudler/LocalAI/pull/11431
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11433
• chore: :arrow_up: Update 0xShug0/audio.cpp to 748c5e28f6a7228b8f38ad7142ca97d29584544b by @localai-bot in https://github.com/mudler/LocalAI/pull/11401
• chore: :arrow_up: Update CrispStrobe/CrispASR to 8898b0fee199b1765321a5d0d8314aee51a34583 by @localai-bot in https://github.com/mudler/LocalAI/pull/11422
• chore(deps): bump dompurify from 3.4.12 to 3.4.13 in /core/http/react-ui in the npm_and_yarn group across 1 directory by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11425
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to 5763a901de3f525b65ed5574fd57623905685755 by @localai-bot in https://github.com/mudler/LocalAI/pull/11452
• chore: :arrow_up: Update CrispStrobe/CrispASR to 8b91f4c5490b88ae6193673c35644d91d8695821 by @localai-bot in https://github.com/mudler/LocalAI/pull/11450
• chore: :arrow_up: Update 0xShug0/audio.cpp to 9d6e7b39236e0151ad28a70fab0d538b84ce8718 by @localai-bot in https://github.com/mudler/LocalAI/pull/11451
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11453
• chore: :arrow_up: Update ggml-org/llama.cpp to 030ebb558a5820b444a8f836ed5cdd46c9b4bd7a by @localai-bot in https://github.com/mudler/LocalAI/pull/11454
• chore: :arrow_up: Update mudler/vllm.cpp to 24306364ab8beaed9197604a042a56aaccfde493 by @localai-bot in https://github.com/mudler/LocalAI/pull/11385
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to 26ceed9d4091a1696cf50e2ed87e5767d5811d81 by @localai-bot in https://github.com/mudler/LocalAI/pull/11475
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11471
• chore: :arrow_up: Update mudler/depth-anything.cpp to 54abd5c0abfd1f394e01cb3c38f2e3af4daedf85 by @localai-bot in https://github.com/mudler/LocalAI/pull/11481
• chore: :arrow_up: Update mudler/vllm.cpp to 9fd9e8f34408d5dd21d7f9385e96fc755708950b by @localai-bot in https://github.com/mudler/LocalAI/pull/11472
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to c46ffaa5665cfb2d6cf372c9a054dbab896e14fe by @localai-bot in https://github.com/mudler/LocalAI/pull/11482
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260812005333 by @localai-bot in https://github.com/mudler/LocalAI/pull/11467
• chore: :arrow_up: Update vllm-project/vllm cu130 wheel to 0.27.1 by @localai-bot in https://github.com/mudler/LocalAI/pull/11468
• chore: :arrow_up: Update CrispStrobe/CrispASR to ce521ee178867ceaa5fdc11803616578c8936c19 by @localai-bot in https://github.com/mudler/LocalAI/pull/11470
• chore: :arrow_up: Update leejet/stable-diffusion.cpp to de298c225bed97c3f9026b73cd7b71e7879bd41b by @localai-bot in https://github.com/mudler/LocalAI/pull/11469
• chore(deps): bump backend/rust/kokoros/sources/Kokoros from 7089168 to 29e99ad by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11496
• chore(deps): bump packaging from 26.2 to 26.3 in /backend/python/coqui by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11497
• chore: :arrow_up: Update ggml-org/llama.cpp to 84e908c625fb60992b4cdef8180fb12fa9b4c4bf by @localai-bot in https://github.com/mudler/LocalAI/pull/11473
• chore(deps): update transformers requirement from >=5.14.1 to >=5.15.0 in /backend/python/transformers by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11500
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260813121949 by @localai-bot in https://github.com/mudler/LocalAI/pull/11507
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to 981e5ea0d7579b4803c86afbb09a7cd7d7bf3bb8 by @localai-bot in https://github.com/mudler/LocalAI/pull/11509
• chore: :arrow_up: Update CrispStrobe/CrispASR to cb082743c456ac77aec0947de36e6420a933da04 by @localai-bot in https://github.com/mudler/LocalAI/pull/11510
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to 43afea46c25a12aae6db1e3105643267164898b4 by @localai-bot in https://github.com/mudler/LocalAI/pull/11527
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11525
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260814013332 by @localai-bot in https://github.com/mudler/LocalAI/pull/11522
• chore: :arrow_up: Update CrispStrobe/CrispASR to cc498701f1a68d88dd489803ebad10053a924322 by @localai-bot in https://github.com/mudler/LocalAI/pull/11523
• chore: :arrow_up: Update ggml-org/whisper.cpp to 1fe009caeda75f69bc864d6370b10674e45a92bd by @localai-bot in https://github.com/mudler/LocalAI/pull/11524
• chore(deps): bump sentence-transformers from 5.6.1 to 5.7.0 in /backend/python/transformers by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11499
• chore(deps): bump vllm from 0.25.1 to 0.26.0 in /backend/python/vllm by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11402
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to 8337e4cd3861406fc04e0854b1409cd1b027fbc9 by @localai-bot in https://github.com/mudler/LocalAI/pull/11542
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260815085651 by @localai-bot in https://github.com/mudler/LocalAI/pull/11541
• chore(deps): bump golang.org/x/net to v0.55.0 by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11544
• chore: :arrow_up: Update mudler/vllm.cpp to 4880c5715f36445a30bd39d3349a06dc96085a11 by @localai-bot in https://github.com/mudler/LocalAI/pull/11515
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260816085229 by @localai-bot in https://github.com/mudler/LocalAI/pull/11553
• chore: :arrow_up: Update CrispStrobe/CrispASR to 3721d402f7bcc911dd4143a58e3da1cc67f09cc2 by @localai-bot in https://github.com/mudler/LocalAI/pull/11554
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260817081527 by @localai-bot in https://github.com/mudler/LocalAI/pull/11570
• chore: :arrow_up: Update CrispStrobe/CrispASR to 007fda47128aaba7e2ce2279b0947a1c13800d68 by @localai-bot in https://github.com/mudler/LocalAI/pull/11569
• chore: :arrow_up: Update ggml-org/llama.cpp to 0021a77de0a8966059dc94548fb3b96654e0bb12 by @localai-bot in https://github.com/mudler/LocalAI/pull/11508
• chore: :arrow_up: Update mudler/vllm.cpp to 76f2a6d84e41cc9a593146a9a958f63645542731 by @localai-bot in https://github.com/mudler/LocalAI/pull/11555
• chore: :arrow_up: Update 0xShug0/audio.cpp to 3407559d7c83b543bd60ca8fa2b630374e71083f by @localai-bot in https://github.com/mudler/LocalAI/pull/11474
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260818075955 by @localai-bot in https://github.com/mudler/LocalAI/pull/11586
• chore: :arrow_up: Update 0xShug0/audio.cpp to 92816fc5c48d33a58171b49678ccf884d70142fb by @localai-bot in https://github.com/mudler/LocalAI/pull/11587
• chore: :arrow_up: Update ggml-org/whisper.cpp to 4834a2327d008ace3ec5a9ed00f51454bcabbc1c by @localai-bot in https://github.com/mudler/LocalAI/pull/11588
• chore: :arrow_up: Update mudler/vllm.cpp to 6e99e4540b8d0811542a9d7690dc9aa0bda5812f by @localai-bot in https://github.com/mudler/LocalAI/pull/11589
• chore: :arrow_up: Update ggml-org/llama.cpp to 60addddf3c567c43ec3caf70fc953fba3572d96f by @localai-bot in https://github.com/mudler/LocalAI/pull/11590
• chore: :arrow_up: Update CrispStrobe/CrispASR to a117d433855ecc36b0cb20bd1abddd852902ed13 by @localai-bot in https://github.com/mudler/LocalAI/pull/11591
• chore: :arrow_up: Update mudler/parakeet.cpp to e75de9b6b9b688fd293aa22f7e27aa724ea286f8 by @localai-bot in https://github.com/mudler/LocalAI/pull/11592
• chore: :arrow_up: Update mudler/vllm.cpp to 438305e1577768ec0f75729456a4c8b9f425e2ee by @localai-bot in https://github.com/mudler/LocalAI/pull/11612
• chore: :arrow_up: Update 0xShug0/audio.cpp to a61da671b6a81c79071500954eea3c91c1a383dd by @localai-bot in https://github.com/mudler/LocalAI/pull/11613
• chore: :arrow_up: Update leejet/stable-diffusion.cpp to 97d2990807fe6d558e395f8764198d7c7e7b411c by @localai-bot in https://github.com/mudler/LocalAI/pull/11616
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11615
• chore: :arrow_up: Update CrispStrobe/CrispASR to a153b09b37c90cd55cd9336fccbdf3ba7a289596 by @localai-bot in https://github.com/mudler/LocalAI/pull/11611
• chore: :arrow_up: Update NVIDIA/NeMo-Speech.cpp to 4f9676226f667d14608487df744f375db87127f8 by @localai-bot in https://github.com/mudler/LocalAI/pull/11417

Other Changes

• fix(ci): parse current vllm-metal version pins by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11457
• fix(ci): refresh site counters through a PR by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11572
• feat(swagger): update swagger by @localai-bot in https://github.com/mudler/LocalAI/pull/11585
• fix(ci): test stale chunks in split bundle by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11595

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.8.2...v4.9.0

What's Changed

👒 Dependencies

• chore(deps): bump actions/stale from 10.4.0 to 11.0.0 by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11395
• chore(deps): bump actions/checkout from 4 to 7 by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/11396

Other Changes

• feat(gallery): fall back to mirrors and a cached index when the primary source fails by @localai-bot in https://github.com/mudler/LocalAI/pull/11389
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11405
• docs: :arrow_up: update docs version mudler/LocalAI by @localai-bot in https://github.com/mudler/LocalAI/pull/11397
• feat(swagger): update swagger by @localai-bot in https://github.com/mudler/LocalAI/pull/11398
• feat(gallery): default to index.localai.io with GitHub as a mirror by @localai-bot in https://github.com/mudler/LocalAI/pull/11409
• feat(nemo-speech-cpp): add the NVIDIA NeMo-Speech.cpp backend by @localai-bot in https://github.com/mudler/LocalAI/pull/11406
• fix(ci): remove unsupported cosign bundle flag by @localai-bot in https://github.com/mudler/LocalAI/pull/11413

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.8.1...v4.8.2

What's Changed

Other Changes

• docs(blog): cover the terminal agent in the 4.8 post by @localai-bot in https://github.com/mudler/LocalAI/pull/11372
• fix(vram): contain malformed GGUF metadata by @richiejp in https://github.com/mudler/LocalAI/pull/11374
• docs: :arrow_up: update docs version mudler/LocalAI by @localai-bot in https://github.com/mudler/LocalAI/pull/11377
• chore: :arrow_up: Update 0xShug0/audio.cpp to 7efbb58def443722ea540d931dd3debee3e4d5e8 by @localai-bot in https://github.com/mudler/LocalAI/pull/11378
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to cf1aa57e1a0fabfd015831718fc99d1aec01ada5 by @localai-bot in https://github.com/mudler/LocalAI/pull/11380
• chore: :arrow_up: Update leejet/stable-diffusion.cpp to c6beeef35526c6dc94b74a7fb69f9d2e6a2a7a12 by @localai-bot in https://github.com/mudler/LocalAI/pull/11384
• chore: :arrow_up: Update CrispStrobe/CrispASR to 21901d3f7c23554f072964828363e49ddbc2dc68 by @localai-bot in https://github.com/mudler/LocalAI/pull/11383
• fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again by @localai-bot in https://github.com/mudler/LocalAI/pull/11386
• fix(react-ui): stop traces page crash when switching trace tabs by @nandanadileep in https://github.com/mudler/LocalAI/pull/11387
• chore: :arrow_up: Update antirez/ds4 to b0309611041655f4e45671cfd9c9886aff161406 by @localai-bot in https://github.com/mudler/LocalAI/pull/11381
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/11382
• gallery: add Qwen3.5 9B HauhauCS variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11339
• gallery: add Qwen3.5 9B Defiant Fable variants by @localai-org-maint-bot in https://github.com/mudler/LocalAI/pull/11335
• feat(vllm-cpp): wire the full engine config surface through engine_args by @localai-bot in https://github.com/mudler/LocalAI/pull/11159
• fix(react-ui): restore 3D Studio results and history by @richiejp in https://github.com/mudler/LocalAI/pull/11393
• fix(cli): ignore a half-populated socket activation environment by @localai-bot in https://github.com/mudler/LocalAI/pull/11394

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.8.0...v4.8.1

🎉 LocalAI 4.8.0 Release! 🚀

LocalAI 4.8.0 is out!

Twenty-two days, 386 pull requests, and three new modalities. This release introduces vllm.cpp, a C++20 engine maintained by the LocalAI team, which began as a vLLM port and now carries its own featureset, shipping as the vllm-cpp backend in alpha development builds. Around it: 3D generation as a new modality, a multi-family audio.cpp engine, gallery entries that install the build your hardware can actually run, and a deep reliability pass on distributed mode driven by production incidents.

Highlights:

• 🚀 vllm.cpp (alpha) - a C++20 engine maintained by the LocalAI team, which began as a vLLM port and keeps vLLM as its reference implementation: V1 serving architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) with no Python, PyTorch or ggml at inference. Measured at 1.045x vLLM on Qwen3.6-27B NVFP4 at concurrency 1, with token-for-token identical output. Loads safetensors and GGUF, enforces structured output in-engine, and runs on CPU, CUDA, Metal and Vulkan. The Apple Silicon build ships the MLX GEMM provider, measured at 1.5x to 2.2x on an M4. Shipping as alpha development builds: try it, do not depend on it.
• 🧊 3D generation - a new modality end to end: Generate3D RPC, FLAG_3D capability, POST /v1/3d/generations, the trellis2cpp image-to-3D backend, and a UI page with a native GLB viewer and print remeshing.
• 🔊 audio.cpp - one backend process serving six audio endpoints across many model families, picked from the GGUF's own metadata: speech, transcription, VAD, diarization, source separation and sound generation.
• 🎛️ One model, many builds - a gallery entry can declare variants:, and LocalAI installs the largest build that your host can actually run. No more hunting through the gallery for the right quantization.
• ⚡ A much lighter web UI - gzip on the wire, immutable caching for hashed assets, and paginated trace endpoints: the React bundle is 3.48x smaller and the trace poll dropped from 21 MB to 7 KB.
• 📊 An Activity page - the stacked operations bar collapses to one line, and a new admin Activity page keeps the record of what installed, failed or was cancelled, instead of dropping it the moment it finished.
• 📦 Hugging Face artifact materialization - immutable snapshot resolution, authenticated downloads with real progress, and staged artifacts that remote workers can bind to.
• 🎚️ VRAM budgets - cap how much of a card LocalAI may use, per node, as a percentage (80%) or an absolute amount (12GB).
• 🗣️ Two new TTS engines - magpie-tts-cpp (NVIDIA Magpie Multilingual, 5 voices, 9+ languages) and moss-tts-cpp (48 kHz stereo with reference-audio voice cloning).
• 🌳 Sub-2-bit models - a new bonsai backend serves the 1-bit and ternary Bonsai quantizations of Qwen3 and Qwen3.6-27B.
• 🖧 Distributed mode hardening - the reaper no longer deletes rows for backends that are alive and busy, phantom replicas are cleaned up, and in_flight counters stop leaking.

Plus a Valkey vector store, systemd socket activation, persistent trace history, two security fixes, a documentation overhaul aimed squarely at onboarding, and a new localai.io.

---

📊 This release in numbers

Pull requests merged	386
Commits	392
Files changed	1,204 (+152,918 / -42,012)
Development window	22 days (2026-07-14 to 2026-08-05)
Contributors	25, of whom 11 first-time
New backends	7 (vllm-cpp, audio-cpp, trellis2cpp, valkey-store, bonsai, magpie-tts-cpp, moss-tts-cpp)
Gallery entries	1,221 to 1,515 (+294)

Where the work landed:

Area	Change
core/	+54,974 / -8,291 across 546 files
gallery/	+39,193 / -29,119 (variant ladders rewrite most of the index)
backend/	+25,668 / -1,305 across 256 files
pkg/	+9,678 / -385
.github/	+7,526 / -61
docs/	+4,285 / -2,461 (near-flat by design: the dedup pass removed as much as it added)
website/	+4,479 (new project site)
scripts/	+3,373 / -175

---

📌 TL;DR

Area	Summary
🚀 vllm.cpp (alpha)	An Apache-2.0 C++20 engine maintained by the LocalAI team and developed in its own repository, which began as a vLLM port, shipping as alpha development builds of the vllm-cpp backend over its stable C ABI v5. It uses vLLM as its reference implementation and benchmark, and implements vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) with no Python, PyTorch or ggml at inference. Safetensors + GGUF, in-engine structured output (JSON schema / regex / choice / GBNF). Chat and tool calling ride the llama.cpp autoparser path: full minja templates, tool_choice: auto as a lazy structural-tag constraint, 30 tool dialects and 7 reasoning parsers, streamed ChatDelta/ToolCallDelta. CPU amd64/arm64, CUDA 12/13 (Blackwell), L4T, Vulkan and Darwin Metal, the last with the MLX GEMM provider vendored in (1.5x to 2.2x on an M4).
🧊 3D generation	A new modality, wired end to end: Generate3D RPC, FLAG_3D capability, POST /v1/3d/generations, the trellis2cpp image-to-3D backend over TRELLIS.2, and a UI page with a native GLB viewer, IndexedDB history and previewable print remeshing.
🔊 audio.cpp	New native C++ backend over audio.cpp, a multi-family ggml audio engine: one process serves /v1/audio/speech (supertonic, chatterbox, irodori-voicedesign), /v1/audio/transcriptions (citrinet, nemotron, forced-aligner), /v1/audio/vad, /v1/audio/diarize (sortformer), /audio/transform (htdemucs 4-stem separation, voice conversion, speech-to-speech) and /v1/sound-generation. Family comes from the GGUF's own audiocpp.model_spec.family key, so no per-model backend options. 13 gallery entries. CPU, CUDA 12/13, Vulkan, Metal.
📊 Activity page	The stacked operations bar becomes a permanent one-line strip (✕ now hides rather than cancels), with a new admin /app/activity page: in-progress detail with per-node breakdown, a "needs attention" lane with Cancel and Retry, and a bounded 50-entry record of what finished.
🗄️ Valkey vector store	New valkey-store backend adding Valkey Search as a vector store option.
🌐 New localai.io	The site splits into a project site at the root and docs under /docs/, with 214 generated redirect stubs so every published URL keeps working. Adds an engines page driven by YAML, a blog, an ecosystem band and ADOPTERS.md.
🎛️ Gallery variants	An entry may declare variants: referencing other entries. Install-time selection drops builds the host cannot run (IsBackendCompatible) or cannot fit (VRAM, or cgroup-aware RAM on CPU hosts), then picks the largest that fits. Override with variant on POST /models/apply, local-ai models install --variant, the install_model MCP tool, or the UI split-button. GET /api/models?has_variants=true narrows the list. Older clients ignore the key and install the entry as before.
📦 HF artifacts	Immutable snapshot resolution, authenticated downloads with progress, gallery install and preload materialization, runtime binding to staged artifacts, and UI progress reporting. Python backends reuse the Go download path.
⚡ HTTP performance	gzip middleware (--disable-http-compression, --http-compression-min-length), with streaming paths explicitly skipped. /assets/* served immutable, index.html no-cache. /api/traces and /api/backend-traces accept limit/offset/full and summarize by default, with GET /api/traces/{id} for the full record. React bundle 2,815,513 B to 807,918 B; backend-trace poll 21,131,097 B to 7,201 B.
🎚️ VRAM budget	LOCALAI_VRAM_BUDGET=80% or =12GB (also --vram-budget), on local-ai and local-ai worker. Standalone it is a hard per-process cap inherited by context-fit, GGUF warnings and the watchdog; distributed it is a placement ceiling the scheduler respects. Admin override via PUT/DELETE /api/nodes/:id/vram-budget and the set_node_vram_budget MCP tool. Unset means all detected VRAM.
🗣️ magpie-tts-cpp	New Go/purego backend over magpie-tts.cpp, a ggml port of NVIDIA Magpie TTS Multilingual 357M with NanoCodec embedded. 5 voices, 9+ languages, 22.05 kHz mono, one self-contained GGUF.
🗣️ moss-tts-cpp	New Go/purego backend over moss-tts.cpp for MOSS-TTS-Local v1.5. 48 kHz stereo, optional reference-audio voice cloning, no Python at inference.
🌳 bonsai	New backend on the PrismML llama.cpp fork, which is the only decoder for the Q1_0 and Q2_0 quant formats. Eight gallery entries across Bonsai 8B/27B and Ternary-Bonsai 8B/27B, from ~1.15 GB.
🖧 Distributed reliability	A busy backend is no longer reaped: the worker is asked directly over a new models.running subject, and the port-probe fallback distinguishes DeadlineExceeded (busy) from Unavailable (gone), requiring three consecutive misses. Frontend model stubs are dropped when no healthy replica remains, in_flight leaks are closed, and model-load deadlines scale with checkpoint size.
🛡️ Security	Inline GRPO reward code in POST /api/fine-tuning/jobs is refused unless the operator sets LOCALAI_TRL_ALLOW_INLINE_REWARD=true; the previous builtin allowlist was escapable to arbitrary code execution on an endpoint that is unauthenticated by default. Also picks up hono 4.12.25 for CVE-2026-54290.
🧠 Models	MiniMax-M3, Gemma 4 llama.cpp MTP variants, Qwen3.5-4B DFlash, MOSS-TTS-Local v1.5, the APEX families as variant ladders, and the Bonsai families. Duplicate entries removed and linted against recurring.
📖 Docs	Onboarding overhaul: one model carried through install to first API call, a new "Build your first agent" walkthrough, a runtime-errors reference keyed on literal error strings, an agent actions catalog, and a new Operations section.

---

🚀 New Features & Major Enhancements

🚀 Introducing vllm.cpp (alpha)

vllm.cpp is Apache-2.0, maintained by the LocalAI team, and began as a C++20 port of vLLM. We want it community-first rather than a LocalAI-only engine, so it lives in its own repository with its own docs, benchmark record and issue tracker, and it is usable without LocalAI anywhere in the picture. It implements vLLM's V1 serving architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) on a portable tensor runtime with no Python, no PyTorch and no ggml at inference time, and uses vLLM itself as its reference implementation: correctness is checked by comparing output against it, and the benchmark scoreboard is kept against it.

It has since grown a featureset vLLM does not have, which is what the port was for. It loads GGUF as well as Hugging Face safetensors, runs on CPU, Apple Metal and Vulkan alongside NVIDIA CUDA, ships speculative decoding and KV offload, and enforces structured output in-engine (JSON schema, regex, choice, GBNF). Its benchmark page now measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because those are the engines it actually competes with on that hardware.

The project is expected to be renamed, with the new name still to be decided. It is drifting far enough from vLLM that calling it a port undersells it and calling it vllm.cpp will eventually mislead.

Numbers, from the project's own scoreboard

These come from vllm.cpp's BENCHMARKS.md, which reports ties as ties and losses as losses. Throughput is vllm.cpp over the reference, so above 1.0 is ahead.

Reference	Workload	Result
vLLM	Qwen3.6-27B NVFP4, GB10	1.045x at concurrency 1, 1.007x to 1.017x at c2 to c32, output token-for-token identical
vLLM	Qwen3.6-35B-A3B NVFP4, GB10	1.010x at c16 and 1.013x at c32; behind at c1 to c8 (0.817x at c1)
vLLM	DeepSeek-V2-Lite MLA, GB10	0.86x to 0.95x throughput, TTFT ahead at c4 and c8
llama.cpp	Qwen3.5-2B GGUF, CPU aarch64	prefill 1.18x, decode a tie, memory parity, byte-identical output
MLX-LM	Qwen3-0.6B, Apple M4	97.6% of warm total, prefill ahead
DwarfStar (ds4)	DeepSeek-V4-Flash IQ2_XXS, one DGX Spark	18.69 vs 16.33 tok/s decode, 1.144x, same output
vLLM	Laguna-XS-2.1 NVFP4, GB10	44.46 vs 43.10 tok/s, 1.03x, same output

The upstream page is careful about its own noise band: on the 27B grid it calls c2 through c32 ties rather than wins, because the run-to-run spread is 0.5% and those margins land between 0.7% and 1.7%. The c1 result is the one it stands behind.

The DeepSeek-V4-Flash row is the one that shows how far the project has moved from being a vLLM port. It runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark, decoding at 18.69 tok/s against DwarfStar's 16.33. At 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits inside the Spark's 119 GiB unified pool, and reading GGUF is what makes that possible.

That figure moved twice in a week, and the second move came from one lever. The dense Q8_0 projection tower was being read from the GGUF mmap over unified memory, which the GB10 reads about 20% slower per-GEMV than device memory. Staging that ~6 GiB tower device-resident once at load, same bytes and same kernels, took decode from 16.23 to 18.69, generating the same tokens and using no more peak memory. The same change took Laguna-XS-2.1 from 87% of vLLM to 1.03x ahead of it.

Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 generates the same tokens as vLLM's MTP and runs about 4% faster at concurrency 1.

It ships here as the vllm-cpp backend, which dlopens the engine's stable C ABI (v5) through purego. Concurrent requests batch continuously inside the engine's shared scheduler rather than serializing, so the backend runs on base.Base rather than SingleThread.

Tool calling is at llama.cpp parity, by construction, because chat reuses the same autoparser path. With use_tokenizer_template the engine renders the model's own chat template (GGUF tokenizer.chat_template or tokenizer_config.json, full minja) and handles the rest itself:

• tool_choice: auto lowers to a lazy structural-tag decode constraint; required and named choices force the family's native syntax where expressible.
• Streaming per-dialect parsers cover 30 tool dialects and 7 reasoning parsers, with <think> reasoning split before tool parsing.
• ChatDelta, ToolCallDelta and reasoning stream exactly as the llama-cpp backend does.

tool_parser: and reasoning_parser: are model options, auto-detected when unset.

Getting started is a normal backend install:

name: qwen3-vllm
backend: vllm-cpp
context_size: 8192
parameters:
  model: Qwen3-4B      # a safetensors directory or a .gguf file
options:
- max_num_seqs:16      # also: block_size:<n>, num_blocks:<n>

The build matrix covers CPU amd64/arm64, CUDA 12/13 (including Blackwell 120a;121a), L4T arm64 for GB10, Vulkan and Darwin Metal, with a gallery meta plus 12 image entries. The llama-cpp GGUF and vllm safetensors importers gained preference swaps, so the backend can be chosen at import time.

Apple Silicon gets the MLX GEMM provider. The darwin build vendors vllm.cpp's optional MLX backend, which upstream keeps off by default on the position that it has to earn its ~124 MB. Measured on an M4 (Qwen3-1.7B-bf16, p=512 g=128, arms toggled on one binary so there is no build-difference confound):

Batch	MLX tok/s	native tok/s	speedup	MLX TTFT	native TTFT
1	5.79	3.08	1.88x	3.32 s	7.68 s
4	15.75	10.24	1.54x	9.63 s	18.77 s
16	38.65	17.69	2.19x	18.33 s	54.48 s

Read those as indicative rather than binding: two reps with a spread reaching 9.4%, so the multipliers carry about +/-10%. The gap is far larger than the noise, and time-to-first-token roughly halves across the range.

These are alpha development builds, not a released backend. vllm.cpp is early. It ships in 4.8 so people who want to try it can, not because it is ready for anything you depend on, and llama-cpp stays the default for real use. Expect rough edges.

The CPU path is end-to-end verified against Qwen3.5-2B-UD-Q8_K_XL.gguf with the full Ginkgo suite: blocking and streaming byte-parity, greedy determinism, stop words, GBNF-constrained generation, concurrent streams, real template rendering, reasoning split, a required tool call returning schema-valid arguments, and an auto run where the engine engages the tool itself and streams parsed deltas. The GPU images build and ship, but their runtime behavior has not been through that gate. No throughput comparison against upstream vLLM is claimed. Please report what breaks.

🔗 PRs: #11100, #11137

🧊 3D generation, end to end

LocalAI gains a new modality. Image-to-3D is wired through the whole stack rather than bolted onto an existing endpoint: a Generate3D RPC in backend.proto, a FLAG_3D capability so the loader knows which backends can serve it, and POST /v1/3d/generations.

The first engine behind it is trellis2cpp, a native image-to-3D backend over TRELLIS.2. The React UI gets a 3D generation page with a native GLB viewer, IndexedDB-backed history so your generations survive a reload, and previewable print remeshing for output you intend to actually print.

🔗 PRs: #10979

🔊 audio.cpp: one backend, six audio endpoints

audio-cpp wraps audio.cpp, a multi-family ggml audio engine. Rather than one backend per model family, a single backend process serves several unrelated families through one runtime vocabulary, and picks the family from the GGUF's own audiocpp.model_spec.family metadata key, so a model needs no backend-specific options to load.

Endpoint	Families
/v1/audio/speech (batch + streaming)	supertonic, chatterbox (voice cloning), irodori-voicedesign (voice design via instructions)
/v1/audio/transcriptions (batch, streaming, live)	citrinet, nemotron, forced-aligner
/v1/audio/vad	silero-vad, marblenet-vad
/v1/audio/diarize	sortformer
/audio/transform	htdemucs (4-stem separation), chatterbox (voice conversion), seedvc-singing, vevo2 (speech to speech)
/v1/sound-generation	stable-audio-sfx

Thirteen gallery entries ship with it, one representative model per task kind the engine can actually serve. Where it cannot honestly back an RPC it returns UNIMPLEMENTED with a reason rather than an empty success, and a failed load is a gRPC error rather than success: false, so the loader's greedy backend probe never silently selects it for a model it cannot serve.

Two changes reach beyond the backend. backend.proto gains AudioTransformStem and AudioTransformResult.stems, so source separation can return the whole stem set instead of a single mixdown. And /audio/transform no longer hardcodes a 16 kHz mono fold: that fold made 4-stem separation unreachable by construction, so it became a per-backend capability, with existing backends keeping it explicitly and the default for an unregistered backend being to leave the upload alone.

Platforms: CPU (amd64 and arm64), CUDA 12, CUDA 13 and Vulkan on Linux, plus Metal on darwin-arm64. No ROCm, which upstream does not support.

🔗 PRs: #11141

📊 A one-line strip, and an Activity page

The operations bar rendered one row per in-flight operation above every page. Queue four model installs and a backend and it took most of the viewport, on every route, until the last one finished. Two things were conflated: a global "something is happening" signal, which needs one line, and the detail of what is happening, which needs a page.

The strip now collapses to a single line permanently, showing one operation (a failure first, otherwise the least-advanced running one) with a +N more pill. Its ✕ hides the strip and never cancels, a deliberate change: the same glyph previously cancelled a 17 GB download in one row and dismissed a message in the next. Cancelling moved to the page, behind a labelled button.

The new admin-only Activity page at /app/activity carries the detail the strip has to drop (phase, bytes, derived time remaining, and a per-node breakdown for cluster installs), a "needs attention" lane for unacknowledged failures with Cancel and Retry, and a record of what finished. That record is a bounded 50-entry ring, which closes a real gap: /api/operations dropped an operation the moment it succeeded, so a user who stepped away had no way to learn whether an install finished, failed, or never started.

Several latent UI bugs were fixed along the way: retrying a failed removal re-downloaded the model, queued operations rendered as "Installing" with a spinner, a long error message pushed every page ~270px past the viewport, and the ETA blanked for every operation whenever one was verifying.

🔗 PRs: #11163

🎛️ One gallery entry, several builds

A gallery entry can now declare variants:, a list of references to other gallery entries that are alternative builds of the same weights:

- name: nanbeige4.1-3b-q4      # still a normal, complete, installable entry
  url: github:mudler/LocalAI/gallery/nanbeige4.1.yaml@master
  overrides: {parameters: {model: nanbeige4.1-3b-q4_k_m.gguf}}
  files: [...]
  variants:
    - model: nanbeige4.1-3b-q8

Selection at install time, in order:

1. Drop variants whose backend cannot run here. MLX disappears on Linux, CUDA on a Mac. Derived from the backend name, so authors never write hardware conditions.
2. Drop what does not fit: VRAM on GPU hosts, cgroup-aware system RAM on CPU hosts, so a container sees its own limit.
3. Take the largest that remains, on the basis that a bigger footprint is a better build of the same weights.

The entry's own build competes in that ranking and is never filtered out, so selection always ends with something installable.

Sizes come from the existing pkg/vram estimator (remote GGUF header, HTTP HEAD, the declared size:, then the HF repo listing). Nothing is downloaded to decide, and a probe failure never fails an install.

Auto-selection is the default and every surface can override it: variant on POST /models/apply and POST /api/models/install/:id, local-ai models install <name> --variant <variant>, the variant parameter on the install_model MCP tool, and a split-button menu in the models table. An explicit selection is honored even when it does not fit, with a warning, since that is a deliberate operator override.

Existing installations are unaffected: every released LocalAI reads gallery/index.yaml live and ignores keys it does not understand, so an older client drops variants: and installs the entry exactly as before. A spec re-parses the real index through a legacy-shaped struct to keep that true.

Known gaps worth stating: in distributed mode InstallModel resolves against the frontend rather than the worker that will serve the model, so a cluster with a small frontend and large workers selects conservatively. Probing within a single entry is still serial and uncapped.

🔗 PRs: #10943, #10983, #10992, #11027, #11139

📦 Hugging Face model artifacts

Model artifacts from Hugging Face are now materialized as a managed snapshot flow: immutable snapshot resolution, authenticated downloads with progress reporting, materialization on gallery install and preload, runtime binding to the staged artifacts, and progress surfaced in the UI. Python backends reuse the Go download path rather than fetching on their own.

A substantial run of follow-ups landed alongside it: per-file resume of interrupted materialization rather than starting over, each writer staging into its own partial tree, companion artifacts persisted so remote workers receive the base_model option, single-file HF snapshots loaded from the file rather than the directory, inferred materialization gated by backend, CIFS EACCES treated as lock contention rather than failure, and multi-file install progress kept proportional during verification.

🔗 PRs: #10825, #10908, #10909, #10910, #10949, #10986, #10995, #11071, #11075, #11117

⚡ A much lighter web UI and trace API

Three HTTP-layer problems, all measured on a live deployment, fixed together because they all shape what goes on the wire.

The server sent no Content-Encoding at all, regardless of Accept-Encoding. There is now gzip middleware, on by default and controllable with --disable-http-compression / LOCALAI_DISABLE_HTTP_COMPRESSION and --http-compression-min-length / LOCALAI_HTTP_COMPRESSION_MIN_LENGTH (default 1024). Streaming responses are skipped explicitly, since buffering them behind a gzip writer defeats incremental flushing and reads as a hung stream: SSE Accept headers, WebSocket upgrades, and the completion, realtime, speech, transcription, agent-job and log-tail path prefixes. Already-compressed formats are skipped too, because gzip made those marginally larger.

Vite content-hashes the bundle filenames, so an /assets/ URL can never change content, yet they shipped with no Cache-Control, ETag or Last-Modified. They now carry public, max-age=31536000, immutable, index.html is explicitly no-cache so deploys are picked up, and unhashed locale JSONs get a 5 minute policy.

/api/traces was returning a 21 MB unpaginated blob that the UI polled every 5 seconds. Both list endpoints now accept limit (default 50, max 1000, 0 for all), offset and full, and summarize by default: bodies and headers are dropped, the byte counters kept so the UI can still report what went missing. Every trace carries a process-lifetime id, and GET /api/traces/{id} serves the full record on expand or export. Paging metadata rides in X-Total-Count, X-Trace-Offset and X-Trace-Limit, so the list body stays a plain JSON array for existing consumers.

	Before	After	Change
React JS + CSS over the wire	2,815,513 B	807,918 B	3.48x smaller
All embedded assets (incl. fonts)	3,953,917 B	1,559,787 B	2.53x smaller
Repeat navigation asset transfer	full re-download	0 bytes	eliminated
/api/backend-traces poll payload	21,131,097 B	7,201 B	~2900x smaller

🔗 PRs: #11056

🎚️ Per-node VRAM allocation budgets

Operators can now cap how much VRAM LocalAI uses for model allocation on a node, as a percentage (80%) or an absolute amount (12GB). Everywhere LocalAI reads VRAM to make an allocation decision it now uses min(detected, budget), a hard ceiling that never raises usable VRAM above physical. Percentages above 100% are rejected; absolute values above physical are clamped.

LOCALAI_VRAM_BUDGET=80%
LOCALAI_VRAM_BUDGET=12GB

It applies to both local-ai and local-ai worker (also --vram-budget), and is editable live from the standalone Settings page and per node in the distributed node UI.

The two paths are deliberately asymmetric:

• Standalone, a hard per-process cap. xsysinfo holds it as a process-global default, so hardware defaults, context auto-fit, GGUF warnings and the watchdog all inherit it.
• Distributed, a placement ceiling. The worker reports raw VRAM plus its budget string; the registry resolves it and caps stored available_vram on registration and heartbeat, so the SQL scheduler needs no query change. The worker still sees its full card for its own context-fit.

Admin overrides via PUT/DELETE /api/nodes/:id/vram-budget survive worker restarts, and are exposed as the set_node_vram_budget MCP tool.

Default unset means all detected VRAM, so existing deployments are unchanged.

🔗 PRs: #10833

🗣️ Two new text-to-speech engines

magpie-tts-cpp wraps magpie-tts.cpp, a C++17/ggml port of NVIDIA's Magpie TTS Multilingual 357M with its NanoCodec vocoder embedded: 5 voices (Aria, Jason, John, Leo, Sofia), 9+ languages, 22.05 kHz mono, from one self-contained GGUF with no Python or PyTorch at inference. GGUFs are published at mudler/magpie-tts.cpp-gguf. A live gRPC check returns a valid non-silent WAV that round-trips exactly through ASR, and the upstream engine is parity-gated against NeMo per component (teacher-forced replay max abs diff 3.6e-5).

moss-tts-cpp wraps moss-tts.cpp, the ggml port of the OpenMOSS MOSS-TTS family, serving MOSS-TTS-Local v1.5 (a GPT-J local transformer decoded through MOSS-Audio-Tokenizer-v2). It produces 48 kHz stereo with optional reference-audio voice cloning. GGUFs are at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF. Images cover CPU, CUDA 12/13, Intel SYCL f16/f32, Vulkan, ROCm, NVIDIA L4T and Darwin Metal.

🔗 PRs: #11115, #10860, #10877

🌳 Sub-2-bit models with the bonsai backend

The Bonsai models are 1-bit (Q1_0) and ternary / 1.58-bit (Q2_0) quantizations of Qwen3 8B dense and Qwen3.6-27B hybrid attention. Their quant formats are only decodable by the PrismML fork of llama.cpp, since stock llama.cpp has no Q1_0/Q2_0 kernels, so they need a dedicated fork backend in the same shape as ik-llama-cpp and turboquant.

The backend reuses backend/cpp/llama-cpp/grpc-server.cpp against the fork's libllama through a thin wrapper Makefile that only swaps LLAMA_REPO and LLAMA_VERSION, so these models are served over the same OpenAI-compatible API as stock llama-cpp. The reused server compiles against the fork with zero skew patches.

Eight gallery entries ship with it:

Family	Variants	Notes
bonsai-8b-1bit	Q1_0	Qwen3-8B, ~1.15 GB
ternary-bonsai-8b	Q2_0, -q2-g64, -pq2	Qwen3-8B, ~2.18 GB
bonsai-27b-1bit	Q1_0	Qwen3.6-27B hybrid attention, vision, ~3.9 GB
ternary-bonsai-27b	Q2_0, -pq2, -q2-g64	Qwen3.6-27B hybrid attention, vision, ~7.2 GB

If Q1_0 and Q2_0 land in mainline llama.cpp, this backend can retire in favor of a routine LLAMA_VERSION bump on stock llama-cpp.

🔗 PRs: #10834, #10866

🖧 Distributed mode reliability

Investigating a model that showed as loaded on the home page but appeared on no node in the cluster turned up four separate bugs, all fixed here.

The reaper was deleting rows for backends that were alive and working. probeLoadedModels reaped a node_models row after one failed 1 second health check, and a busy backend cannot answer one: a single-threaded Python backend blocks for minutes inside a request.

• A new models.running subject asks the worker directly, since it holds the process handle and is not blocked by the backend. The reconciler diffs its process keys against the registry before any port probe.
• A worker that does not answer is skipped, not assumed empty, so a NATS blip cannot delete a node's rows.
• The port probe now separates DeadlineExceeded (busy) from Unavailable (gone), and only the latter counts, after three consecutive misses.

Every routed model also left an in-process stub in the frontend's ModelLoader, and removal paths deleted only the database row, so the stub outlived the replica and the model was reported as loaded forever. The replica-removed hook became a list, and a new local-stub invalidator drops the stub once no healthy replica remains cluster-wide.

Alongside those: in_flight counters could leak high and pin a replica's VRAM against eviction; model-load deadlines now scale with checkpoint size and with progress rather than wall-clock; staging verification counts as progress rather than a stall; backend discovery no longer hides worker-installed or GPU-only backends behind the controller's filesystem and capability; the scheduler will not place a model on a node that cannot store it; and open responses are visible and cancellable across replicas.

Worker-side, a backend process whose directory a reinstall replaced is never reused, the gRPC port allocator is bounded and stops leaking dead backends' ports, deleted backends are reaped, and the worker has a real health endpoint with a mode-aware HEALTHCHECK.

🔗 PRs: #11142, #11121, #11030, #11029, #11026, #11019, #11000, #10999, #10990, #10970, #10968, #10967, #10966, #10956, #10948, #10947, #10838, #11054, #10551

🧰 Smaller features worth knowing about

• Valkey Search vector store (#11196): a new valkey-store backend adds Valkey as a vector store option alongside the existing ones.
• systemd socket activation (#11169): local-ai consumes a TCP listener inherited through the systemd socket-activation protocol on Linux, so it can start on demand. Ordinary --address / LOCALAI_ADDRESS binding is unchanged when no activation listener is present, ambiguous multiple listeners are rejected, and the public-bind auth safety check runs against the actual inherited address. Documented alongside the Podman descriptor-passing requirement.
• Persistent trace history (#11203): API and backend trace histories now persist under the configured data path and survive a restart, as bounded per-record JSON files under traces/api and traces/backend. No database dependency, existing tracing_max_items bounds preserved, restored IDs advanced to avoid collisions, corrupt records skipped rather than blocking startup.
• Edit saved chat messages (#11189): inline Edit / Save / Cancel on saved user prompts and assistant responses, persisted through local chat history with no inference request, preserving structured content blocks and attachment metadata.
• Self-contained Intel SYCL backend (#10991): the Intel llama.cpp backend now runs on any host rather than requiring a matching oneAPI runtime.
• Configurable VAE tiling (#11216) for stablediffusion-ggml, and voice control on low-power devices (#10804) via the classifier/VAD path.
• Anthropic prompt-cache breakpoints (#11158): optional cache breakpoints in cloud-proxy translate mode.
• /v1/detokenize (#9620), and deterministic, type-filtered backend auto-detection (#10286) so backend selection stops depending on probe order.
• MLX TTS routing (#11267): MLX TTS models now import to mlx-audio rather than a generic text backend.

🛡️ Inline fine-tuning reward code is now opt-in

POST /api/fine-tuning/jobs accepts reward_functions[].code, an inline Python body that was executed against a hand-rolled builtin allowlist. That allowlist was not a security boundary: standard CPython introspection reaches the real os module and yields arbitrary code execution on the host. Execution happened synchronously during a smoke test at job start, and the fine-tuning endpoint is unauthenticated by default.

Rather than trying to harden the allowlist, inline reward code is now refused unless the operator explicitly opts in with LOCALAI_TRL_ALLOW_INLINE_REWARD=true on the backend. Builtin reward functions are unaffected and keep working with no configuration. The documentation no longer describes the allowlist as a sandbox and states plainly that inline code is arbitrary execution.

Two further hardening fixes landed in the same cycle:

• Tar hardlinks that escape the extraction root are rejected (#11266). ExtractArchive pre-scanned members and rejected symlinks, but tar hardlink entries carry a regular file mode and passed that check, and Header.Linkname was never validated, so an archive could create a link to a path outside the destination directory. Linkname now gets the same path check as member names; hardlinks resolving inside the root still extract, so ordinary archives are unaffected.
• Cyclic $ref in a JSON-schema grammar is rejected (#11041) rather than recursing into a stack-overflow crash.

This release also picks up hono 4.12.25 for CVE-2026-54290.

🔗 PRs: #11068, #11266, #11041, #11023

🔍 Traces gain request identity

The API Traces panel recorded who issued each request but never showed it, and never captured the caller's network identity. The table now has a sortable User column, and the expanded row carries User, Client IP and User Agent, from echo's RealIP() (honouring X-Forwarded-For / X-Real-IP behind a trusted proxy). Fields render only when present, so older buffered traces and unauthenticated local requests degrade cleanly. The time column now shows the date too.

🔗 PRs: #10907, #10905

---

🐛 Bug Fixes (recap)

• fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway - #11142
• fix(distributed): scale the remote model-load deadline with checkpoint size - #11030
• fix(distributed): make the cold-load hold scale with progress, not wall-clock - #11019
• fix(distributed): count staging verification as progress, not as a stall - #11026
• fix(distributed): reject wrong-model requests at the backend and on the remaining modalities - #10970, #10990
• fix(distributed): backend discovery hid worker-installed and GPU-only backends - #10967, #10947
• fix(distributed): configurable remote model-load timeout, and reap the load when it times out - #10948
• fix(distributed): make per-node backend upgrade actually upgrade - #10838
• fix(nodes): never schedule a model onto a node that cannot store it - #11054
• fix(openresponses): make responses visible and cancellable across replicas - #11000
• fix(worker): never reuse a backend process whose directory a reinstall replaced - #11029
• fix(worker): bound the gRPC port allocator and stop leaking dead backends' ports - #10968
• fix(worker): reap deleted backends and stop models that live on a worker - #10956
• fix(worker): give the worker a real health endpoint and a mode-aware HEALTHCHECK - #10999
• fix(downloader): hash the partial file before issuing the resume request - #11099
• fix(downloader): bound the wait for response headers so a wedged origin cannot hang an install - #11053
• fix(downloader): distinguish read from write failures and retry transient ones - #10985
• fix(modelartifacts): resume interrupted materialization per-file, not from scratch - #11071
• fix(modelartifacts): stage each writer's artifact in its own partial tree - #10995
• fix(modelartifacts): treat CIFS EACCES as lock contention, not failure - #10986
• fix(model-artifacts): persist companion artifacts so remote workers get the base_model option - #11075
• fix(model-artifacts): load single-file HF snapshots from the file, not the directory - #10909
• fix(model-artifacts): gate inferred artifact materialization by backend - #10910
• fix(model-artifacts): materialize longcat-video on the controller, and support companion repos - #10949
• fix(gallery): coalesce Hugging Face artifact progress - #11117
• fix(gallery): keep multi-file HF install progress proportional during verify - #10908
• fix(galleryop): make admitted operations queryable and survive a failed op - #11044
• fix(gpu-libs): bundle cuDNN only where it is used, and complete it when it is - #10946
• fix(gpu): detect GPUs via sysfs when no pci.ids database is present - #10966
• fix(watchdog): force-kill stuck-busy backends instead of deadlocking the loader - #10578
• fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close - #10859
• fix(config): only inject llama.cpp serving options on the llama.cpp path - #10822
• fix(runtime-settings): apply persisted threads/context_size/f16 at startup - #10853
• fix(model): make backend shutdown model-scoped - #10865
• fix(model): only announce a load at INFO when a load actually happens - #11017
• fix(completions): reject empty PromptStrings in streaming to avoid an index-out-of-range panic - #11028
• fix(tts): forward the OpenAI speed field to the backend - #11120
• fix(realtime): accept the legacy modalities alias for output_modalities - #11104
• fix(vision): probe the media marker for pinned llama.cpp backend variants - #10955
• fix(audio-transform): serialize WebSocket writes to avoid a concurrent-write panic - #10857
• fix(qwen-asr): map ISO language codes to the names Qwen3-ASR expects - #10959
• fix(ollama): cap num_ctx so it cannot wrap negative when cast to int32 - #11032
• fix(ollama): set ContextSize via the embedded LLMConfig so the package builds - #11049
• fix(webui): use relative asset base so fonts and lazy chunks honor X-Forwarded-Prefix - #10904
• fix(agent-ui): reset streamed text at generation boundaries in agent chat - #10664
• fix(mcp): bound MCP session connect so an unreachable server cannot hang the widget - #10884
• fix(http): make /readyz reflect startup readiness - #10989
• fix(upgrade-check): don't filter upgrade candidates by controller capability - #11024
• fix(cloud-proxy): publish backend gallery entries - #10858
• fix(backend): don't crash the whole process on an invalid cutstrings/extract_regex - #10855
• fix(backends): derive the protoc generator from the protobuf runtime - #11057
• fix(backend/python): don't await sync servicer behaviors in AsyncModelIdentityInterceptor - #10980
• fix(sglang): implement the Status RPC to unblock backend-monitor polling - #10867
• fix(vllm): generate protobuf 6 compatible stubs - #10944
• fix(vibevoice): install diffusers from PyPI instead of git main - #10972
• fix(kokoro): pin a compatible Intel XPU runtime - #10823
• fix(ace-step): drop nonexistent Get* proto accessors in SoundGeneration - #11072
• fix(trl): disable inline GRPO reward code by default (RCE) - #11068
• fix(turboquant): supersede stale dependency bump - #11064
• fix(turboquant,bonsai): do not apply vendored llama.cpp patches to fork trees - #10866
• fix(llama-cpp): retain CPU variants in GPU builds - #11255, and the same for turboquant - #11276
• fix(llama-cpp): preserve GPU layers during option passthrough - #11193
• fix(utils): reject tar hardlinks that escape the extraction root - #11266
• fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent a stack-overflow crash - #11041
• fix(grammars): restore backslash escaping in the llama31 grammar fixture - #11242
• fix(model): deterministic, type-filtered backend auto-detection - #10286
• fix(oci): install backends on filesystems without symlinks - #11166
• fix(oci): identify signature verification requests - #11244
• fix(realtime): echo response.metadata on response.created and response.done - #11198
• fix(worker): report RAM alongside GPU memory - #11167
• fix(vllm): apply Options[] engine flags before engine init - #11147
• fix(mlx-vlm): install torch dependencies on Metal - #11164
• fix(kokoro): add a CPU backend fallback - #11161
• fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads - #11074
• fix(gallery): correct Nanbeige 4.2 artifacts - #11269
• Video: WAN 2.1 GGML entries never set known_usecases, so they resolved to image rather than video and /video rejected them - #11214

🖧 P2P area

• fix(p2p): serialize access to p2pCtx/p2pCancel - #10861

---

👒 Dependencies

121 dependency updates landed this cycle, mostly nightly engine bumps:

Engine	Bumps
CrispStrobe/CrispASR	17
ikawrakow/ik_llama.cpp	12
leejet/stable-diffusion.cpp	10
ServeurpersoCom/qwentts.cpp	8
ggml-org/llama.cpp	7
ServeurpersoCom/omnivoice.cpp	6
PrismML-Eng/llama.cpp	5
mudler/parakeet.cpp	4
ggml-org/whisper.cpp	4
antirez/ds4	3
0xShug0/audio.cpp	2
magpie-tts.cpp, locate-anything.cpp, depth-anything.cpp, trellis2cpp, rf-detr.cpp, ced.cpp, llama-cpp-turboquant	1 each

Plus 20 dependabot updates across Python, JavaScript and GitHub Actions, and a go-processmanager bump for the concurrent-Run fix.

Note: one ggml-org/llama.cpp bump (d2a8182) was reverted within the cycle and is excluded from these notes.

---

📖 Documentation

The documentation received an onboarding-focused overhaul (#10895), driven by a mapped per-page audit rather than page dates, with every factual claim verified against the code, backends, gallery and CLI:

• Accuracy: the CPU image tag corrected to localai/localai:latest (there is no latest-cpu), the registry unified, the dead llama-stable backend removed from examples, the mitm-proxy flag documentation corrected, a non-existent /sound endpoint removed, the Voice Activity Detection example made runnable, and the CLI reference refreshed with agent, mcp-server, agent-worker and p2p-worker.
• Deduplication: duplicate and stale pages folded into canonical homes, with all inbound links repointed and old URLs preserved via aliases.
• Onboarding: one concrete model (qwen3-4b) now carries through install, Web UI chat and API curl, plus a new Build your first agent walkthrough that states plainly that LocalAGI is embedded.
• Errors: a new Runtime errors and troubleshooting reference keyed on the literal error strings users see, plus a new Agent actions catalog taken from the shipped action registry.
• Structure: installation merged under Getting started for one linear install-to-first-run spine, a new Operations section for operator-facing pages, and journey-ordered navigation.
• Process: a docs checkbox in the PR template and a docs-with-code rule in the agent instructions, so user-facing code changes update docs in the same change.

Also: grpc.attempts timing and tuning guidance (#10868), a fix to the Opus backend installation instructions for realtime (#11018), reverse-proxy and long-inference timeout guidance (#11195, #11065), persistent container storage clarified (#11190), and ROCm 7.x / RDNA 3.5 (Strix Halo, gfx1151) added to the GPU acceleration guide (#9229).

🌐 A new localai.io

The site splits in two: the project site at the root, and the documentation moved under /docs/. The Hugo docs site had always been localai.io, which left nowhere to explain what LocalAI is or to show what the team builds.

Every previously published URL keeps working. GitHub Pages has no server-side rewrites, so a generator walks the built docs output and leaves a meta refresh, a canonical link and a noindex at each old root path: 214 redirect stubs, covering bare .html files as well as directory indexes, and never overwriting a path the root site owns.

The new site adds an /engines/ page driven entirely by a YAML data file (so adding an engine is one edit, not hand-written HTML in two places), a /blog/, a real POSIX install.sh and a Kubernetes manifest wired to the actual /readyz and /healthz endpoints. An ecosystem band lists the companies whose engineers have contributed, the projects that integrate LocalAI, and where LocalAI has been written about, each backed by a different and stated standard of evidence, with ADOPTERS.md as the self-service mechanism for anyone who wants to be listed.

🔗 PRs: #11243, #11248

🧹 CI cost and correctness

A sustained pass on the build pipeline, most of it invisible to users but responsible for how quickly changes land: the full backend matrix now only rebuilds on breaking backend.proto edits (#11192), image and Go PR workflows skip content they cannot see (#11218, #11223, #11224), the native engine builds in a layer the registry cache can actually restore (#11221), and three workflows that stacked runs on every PR push were deduplicated (#11058). Go backends now rebuild on linked pkg/ changes and matrix-entry edits (#10988, #10975).

---

🙌 New Contributors

Eleven people landed their first LocalAI contribution this cycle:

• @ankit-aglawe in #10930
• @anupamme in #11023
• @futurehua in #10879
• @ghshhf in #10323
• @jimmykarily in #10991
• @nandanadileep in #10578
• @owezzy in #11195
• @ResearchForumOnline in #11138
• @wuisabel-gif in #11068
• @Zelys-DFKH in #11266
• @zjuzhongwen in #10960

Thank you all, and thanks to everyone who filed issues, tested builds and reported regressions this cycle.

---

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.7.1...v4.8.0

What's Changed

Other Changes

• fix(config): only inject llama.cpp serving options on the llama.cpp path by @localai-bot in https://github.com/mudler/LocalAI/pull/10822

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.7.0...v4.7.1

🎉 LocalAI 4.7.0 Release! 🚀

LocalAI 4.7.0 is out!

This release widens what LocalAI can generate and how you drive it: a UI-managed voice cloning library, local video and audio-driven avatar generation, and interleaved reasoning that travels with tool calls. It also lands new audio engines (one-pass diarized transcription, F5-TTS, true streaming TTS) and a batch of reliability fixes across auth, transcription, and the model gallery.

Highlights:

• 🎙️ Managed voice cloning - record or upload a consented reference in the UI, save it as a reusable profile, and reference it from any cloning-capable TTS backend with a stable localai://voice-profiles/<id> URI. No more hand-edited YAML or copying audio into model folders.
• 🎬 Local video & avatars - a new longcat-video backend brings text-to-video, image-to-video, and audio-driven talking-avatar generation, wired into the Studio UI with audio and reference-image controls.
• 🧠 Interleaved thinking with tool calls - an assistant turn can now carry reasoning and tool_calls together and keep the reasoning across the tool-result loop, with a reasoning_content inbound alias and Anthropic thinking block support.
• 🗣️ New audio engines - one-pass diarized transcription (moss-transcribe-cpp), F5-TTS voice cloning in CrispASR, and true streaming TTS in vibevoice-cpp (time-to-first-audio 2.38s vs 39.96s on CPU).
• 🎚️ Auto full context - context_size: -1 runs a model at its full trained context window, read from GGUF metadata per-model, with a VRAM-fit warning.
• 🖥️ Sharper control - pick which GPUs llama.cpp offloads to (devices:), and a new model-load cooldown stops a deterministically-failing model from respawning its backend on every poll and leaking VRAM.

Plus DFlash speculative-decoding gallery models, an OIDC fix for EC/PS/EdDSA-signed tokens, transcription language/translate settings that reach the backend, and the usual set of dependency updates.

---

📌 TL;DR

Area	Summary
🎙️ Voice Library	Admin-managed voice cloning profiles: record/upload consented reference audio in the UI, preview, and reference via localai://voice-profiles/<id>. New GET/POST/DELETE /api/voice-profiles, MCP tools (list/create/delete_voice_profile), and a typed tts.voice_cloning config. Cloning declared as a capability across 12+ TTS backends (self-discovered, no hardcoded backend names).
🎬 LongCat video & avatars	New longcat-video Python backend: text-to-video, image-to-video, and LongCat-Video-Avatar 1.5 audio-driven avatars. Gallery entries longcat-video and longcat-video-avatar-1.5; new known_input_modalities/known_output_modalities config fields; /video endpoint extended with staged audio. CUDA 12/13 x86_64 + CUDA 13 ARM64 images.
🧠 Interleaved thinking	Assistant turns carry reasoning + tool_calls together and preserve reasoning across the tool loop. reasoning_content accepted as an inbound alias; Anthropic Messages local path round-trips thinking blocks (gated on thinking: {type: "enabled"}).
🗣️ moss-transcribe-cpp	New Go backend over the C++/ggml MOSS-Transcribe-Diarize port: joint multi-speaker transcription + diarization + timestamps in one pass. Gallery model moss-transcribe-cpp-0.9b. Offline; 1.6-2.2x faster than reference on CPU, bit-exact on CUDA.
⚡ vibevoice streaming TTS	Real incremental streaming via vv_capi_tts_stream: TTFA 2.38s vs 39.96s batch (~17x) on CPU for VibeVoice-Realtime-0.5B.
🎨 F5-TTS	F5-TTS linked into CrispASR + f5-tts-crispasr gallery model (24kHz, voice cloning via voice:/voice_text: options).
🎚️ context_size: -1	Negative context_size (or LOCALAI_CONTEXT_SIZE=-1) resolves to the model's n_ctx_train from GGUF metadata, with a GPU-only VRAM-fit warning and a safe clamp so no backend ever sees a negative window.
🖥️ llama.cpp device selection	options: [devices:CUDA1,CUDA2] restricts offload to named GPUs (from --list-devices).
🛡️ Model-load cooldown	A failed load enters a cooldown (default 10s, geometric growth capped at 5m) so polling clients get 503 + Retry-After instead of respawning a crashing backend and leaking GPU memory.
🧠 Speculative decoding models	Four Qwen DFlash gallery entries (4B / 9B / 27B / 35B-A3B), each bundling target + drafter, spec_type:draft-dflash.

---

🚀 New Features & Major Enhancements

🎙️ Managed voice cloning profiles (Voice Library)

Voice cloning becomes a first-class, UI-driven workflow. Record or upload consented reference audio in the React UI, normalize it to WAV, preview it, and save it as a named profile. Reference the profile from the TTS UI or /v1/audio/speech with a stable localai://voice-profiles/<id> URI, no YAML editing and no copying audio into model directories.

• New REST surface: GET /api/voice-profiles, GET /api/voice-profiles/:id/audio, POST /api/voice-profiles (admin), DELETE /api/voice-profiles/:id (admin), surfaced in Swagger, /api/instructions, and the auth capability registry. MCP admin tools list_voice_profiles, create_voice_profile, delete_voice_profile.
• New typed config tts.voice_cloning (bool) opts custom model names in or out of Voice Library compatibility; false also rejects saved profile references with HTTP 400. Request precedence is voice -> tts.voice -> tts.audio_path; existing options still work with no breaking change.
• Compatibility is server-discovered from backend capabilities (the frontend hardcodes no backend names) and offers gallery models to install when none are present. Cloning is declared in the capability registry for vllm-omni, vibevoice-cpp, coqui, pocket-tts, qwen-tts, qwen3-tts-cpp, faster-qwen3-tts, fish-speech, neutts, chatterbox, voxcpm, omnivoice-cpp, and model-dependent crispasr.
• Cross-backend contract: reference WAV + exact transcript via params.ref_text. E2E verified on qwen3-tts-cpp (4.47s reference produced a non-silent 3.44s 24kHz WAV).

🔗 PRs: #10799

🎬 LongCat video and avatar generation

A new longcat-video Python backend brings local video generation to LocalAI: text-to-video, image-to-video, and LongCat-Video-Avatar 1.5 for audio-driven talking avatars. It is wired into the React Studio UI with audio and reference-image controls.

• Gallery entries longcat-video (text+image input, video output) and longcat-video-avatar-1.5 (text+image+audio input, video output). Backend images: CUDA 12 and CUDA 13 x86_64, plus CUDA 13 ARM64 (DGX Spark / NVIDIA ARM64 guidance included).
• Introduces declarative capability metadata: new known_input_modalities / known_output_modalities config fields so generic code discovers what a checkpoint accepts instead of branching on backend or checkpoint names. The HF importer emits matching self-describing recipe metadata.
• The /video endpoint gains staged audio and video-generation parameters, with distributed audio staging (input bounded to 128 MiB). Avatar mode supports multi-segment generation, BF16 or optional INT8 quantization, and distillation settings, documented in a new features/longcat-video.md page.

🔗 PRs: #10792

🧠 Interleaved thinking with tool calls

An assistant turn can now carry reasoning and tool_calls together, and the reasoning survives the tool-result loop across turns. This is uniform, tested, and documented behavior rather than a per-backend accident.

• OpenAI chat messages accept reasoning_content as an inbound alias for the canonical reasoning field (vLLM/DeepSeek/cogito-style clients emit it); emission is unchanged and the canonical field wins when both are present.
• The Anthropic Messages local path round-trips thinking blocks: inbound thinking blocks parse into the message reasoning, and a thinking block is emitted before tool_use on both non-streaming and streaming responses, gated on the request param thinking: {type: "enabled"}. The cloud-proxy passthrough path is untouched.
• Verified live: lfm2.5-8b-a1b and gemma-4-e2b return reasoning plus structured tool_calls in a single turn. A new features/interleaved-thinking.md doc is cross-linked from the model-configuration, text-generation, and functions guides.

🔗 PRs: #10744

🗣️ New audio engines: diarized transcription, F5-TTS, and streaming TTS

Three additions widen the audio surface:

• moss-transcribe-cpp (#10756): a new Go backend that dlopens the C++/ggml MOSS-Transcribe-Diarize port to do joint multi-speaker transcription, diarization, and timestamps in a single offline pass. Gallery model moss-transcribe-cpp-0.9b (default q5_k GGUF). The ggml port is byte-exact to the reference PyTorch and 1.6-2.2x faster on CPU, bit-exact on CUDA (verified on Blackwell / Jetson Thor). Builds across the full Linux matrix plus Darwin/Metal.
• F5-TTS in CrispASR (#10753): links the F5-TTS static runtime into the CrispASR build (SWivid, MIT; a 22-layer DiT flow-matching model with a built-in Vocos vocoder) and ships an f5-tts-crispasr gallery model. Produces 24kHz mono audio and auto-detects as f5-tts (no backend: selector). Voice cloning via options: [voice:/path/ref.wav, voice_text:Transcript...]; note F5-TTS runs a 32-step ODE solver, so CPU synthesis is compute-heavy.
• vibevoice-cpp true streaming (#10764): replaces whole-clip-then-chunk synthesis with real incremental streaming through the new vv_capi_tts_stream callback ABI. Time-to-first-audio drops from 39.96s (batch) to 2.38s (streaming), about 17x, on a CPU-only box for VibeVoice-Realtime-0.5B. Scope is the realtime-0.5B model; the non-streaming path is unchanged.

🔗 PRs: #10756, #10753, #10764

🎚️ Auto full context with context_size: -1

A context_size: -1 sentinel (any negative value) now makes a model run at its full trained context, resolved per-model from the GGUF n_ctx_train metadata at load. This lets you opt a model into its true maximum window even when a gallery YAML already pins a value, without hardcoding a number. The global equivalents LOCALAI_CONTEXT_SIZE=-1 / --context-size -1 make every model resolve to its own trained max, while an explicit per-model value still wins.

Three defense layers keep it safe: GGUF resolution degrades to the default with a warning if metadata lacks a usable max; a GPU-only warnIfContextExceedsVRAM helper logs (never blocks) when the window likely will not fit; and the backend options layer clamps any residual negative to the default so no backend receives a negative n_ctx. The previously-only path (unset context_size) is unchanged.

🔗 PRs: #10752

🖥️ GPU device selection and model-load cooldown

• llama.cpp device selection (#10724): a new device / devices option in the llama.cpp options: array maps to upstream --device, so you can restrict offload to specific GPUs (for example excluding a display or debug GPU). Example: options: [devices:CUDA1,CUDA2,CUDA3]. Device names come from llama-server --list-devices.
• Model-load failure cooldown (#10728): after a load fails, new independent load triggers are refused during a cooldown window, returning a typed error mapped to 503 + Retry-After instead of respawning the backend on every poll. This stops a deterministically-failing model from leaking GPU/CUDA state (a reporter saw leaked contexts climb to ~58 GB) and, under LOCALAI_SINGLE_ACTIVE_BACKEND, from stealing the active slot from healthy models. Configurable via --model-load-failure-cooldown / LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN (default 10s, 0 disables); the cooldown doubles per consecutive failure, caps at 5m, and resets on a successful load. Coalesced followers of a genuine concurrent burst still get their one retry.

🔗 PRs: #10724, #10728

---

🧠 Models

• Qwen DFlash speculative-decoding models (#10791): four gallery entries for the llama-cpp backend, each bundling a full target model plus its small block-diffusion drafter (drafters are not standalone chat models): qwen3-4b-dflash, qwen3.5-9b-dflash, qwen3.6-27b-dflash, qwen3.6-35b-a3b-dflash. Per-entry config is flash_attention: on, draft_model:, use_jinja:true, spec_type:draft-dflash, spec_n_max:15. Requires the pinned llama.cpp with upstream DFlash support; every bundled drafter is verified general.architecture = dflash (fork-only dflash-draft GGUFs are intentionally excluded because they fail to load). GPU recommended.
• Inference defaults (#10741): adds recommended sampling defaults for deepseek-v4 (auto-generated from unsloth), so LocalAI applies correct generation parameters for the family automatically.
• Plus new gallery models added via the gallery agent (#10743, #10755).

---

🐛 Bug Fixes (recap)

• fix(auth): accept EC/PS/EdDSA-signed OIDC ID tokens, not just RS256 (OIDC login was 500ing at callback with EC-signed tokens, e.g. Authentik) - #10736
• fix(transcription): honor model-config parameters.language/parameters.translate and the OpenAI language form field, which were silently ignored for multipart uploads - #10731
• fix(backends): opus and local-store now refuse foreign model loads, so an LLM with no explicit backend cannot silently bind to the audio codec or vector store during backend probing - #10769
• fix(gallery): backend (re)install is now a clean atomic replace (stage, validate, swap, rollback) instead of an overlay, so stale files from a prior version no longer shadow the new one - #10726
• fix(vram): report the largest single GGUF quant instead of summing every quant in an HF repo, so a 9B model no longer shows as 71 GB / "May not fit" - #10707
• fix(logs): capture backend stdout/stderr into the log store by default in single mode, so the Backend Logs page is populated out of the box - #10742
• fix(vllm): pin the L4T arm64 backend to vllm==0.24.0 for GB10 / DGX Spark stability (0.23 crashes deterministically on cold loads and pins GPU memory) - #10725
• fix(ds4): bundle the full transitive runtime dependency closure so the from-scratch DS4 image no longer exits 127 on a missing gRPC library - #10783
• fix(diffusers,vllm-omni,tinygrad): save generated images as PNG explicitly, fixing an unknown file extension: .tmp crash right after successful inference - #10729
• fix(react-ui): preserve uploaded file content when regenerating a non-last answer (attachments were silently dropped after forking and regenerating a file turn) - #10819
• fix(ui): prevent a large data table from breaking the flexbox layout and forcing a page-wide horizontal scrollbar - #10754

---

👒 Dependencies

Submodule and backend bumps this cycle:

• ggml-org/llama.cpp x7
• CrispStrobe/CrispASR x7
• vllm-metal (darwin) x6
• ServeurpersoCom/qwentts.cpp x4
• ServeurpersoCom/omnivoice.cpp x4
• leejet/stable-diffusion.cpp x3
• ikawrakow/ik_llama.cpp x3
• ggml-org/whisper.cpp x2
• mudler/moss-transcribe.cpp x1
• mudler/locate-anything.cpp x1
• vllm-project/vllm 0.24.0 -> 0.25.0 (Python backend) and cu130 wheel to 0.25.0

Plus grpcio 1.81.1 -> 1.82.1, charset-normalizer >=3.4.9, and GitHub Actions bumps (actions/cache 4 -> 6, actions/stale 10.3.0 -> 10.4.0).

---

📖 Documentation

• Refreshed the LocalAI homepage to frame the project as a modular multimodal AI runtime: breadth lanes (reason, listen and speak, create, see, act), the small-core plus on-demand-backends architecture, the native engines built by the LocalAI team, and the scale path from laptop to team server to cluster - #10780
• Docs-site version bump - #10709

---

🙌 New Contributors

• @rvmz made their first contribution in #10724
• @hogeheer499-commits made their first contribution in #10783
• @ajuijas made their first contribution in #10819

---

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.6.2...v4.7.0

What's Changed

👒 Dependencies

• chore(deps): bump actions/cache from 4 to 6 by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/10704

Other Changes

• refactor: use slices.Contains to simplify code by @weifanglab in https://github.com/mudler/LocalAI/pull/10702
• fix(ci): shard single-arch backend matrix under GitHub's 256-job limit by @localai-bot in https://github.com/mudler/LocalAI/pull/10703
• chore(model gallery): add MiniCPM series models by @ZMXJJ in https://github.com/mudler/LocalAI/pull/10699

New Contributors

• @weifanglab made their first contribution in https://github.com/mudler/LocalAI/pull/10702
• @ZMXJJ made their first contribution in https://github.com/mudler/LocalAI/pull/10699

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.6.1...v4.6.2

What's Changed

Other Changes

• fix(auth): log the real cause of OIDC/OAuth user-info failures by @localai-bot in https://github.com/mudler/LocalAI/pull/10679
• feat(api): add GET /v1/models/capabilities endpoint by @localai-bot in https://github.com/mudler/LocalAI/pull/10687
• chore: :arrow_up: Update CrispStrobe/CrispASR to 1109cb3fcae2e242c2b3d42ec0e3fd6e813f2ce7 by @localai-bot in https://github.com/mudler/LocalAI/pull/10685
• chore: :arrow_up: Update ggml-org/llama.cpp to 665892536dfb1b7532161e3182304bd35c33e768 by @localai-bot in https://github.com/mudler/LocalAI/pull/10681
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/10686
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260704102955 by @localai-bot in https://github.com/mudler/LocalAI/pull/10668
• fix(ui): center the home empty-state wizard by @localai-bot in https://github.com/mudler/LocalAI/pull/10691
• docs: :arrow_up: update docs version mudler/LocalAI by @localai-bot in https://github.com/mudler/LocalAI/pull/10680
• chore: :arrow_up: Update CrispStrobe/CrispASR to 09df654e304947f7521e1f52992ceacccf03c300 by @localai-bot in https://github.com/mudler/LocalAI/pull/10693
• chore: :arrow_up: Update ServeurpersoCom/omnivoice.cpp to daedb763fd442e0916eb130a479fdd74947291c0 by @localai-bot in https://github.com/mudler/LocalAI/pull/10682
• chore: :arrow_up: Update ServeurpersoCom/qwentts.cpp to 73fe0c67bbf0898ba2999535e0680a02a7f8537d by @localai-bot in https://github.com/mudler/LocalAI/pull/10683
• feat(agents): native Prometheus metrics for agent chat runs by @walcz-de in https://github.com/mudler/LocalAI/pull/10689
• chore: :arrow_up: Update ggml-org/llama.cpp to 2da668617612d2df773f966e3b0ee22dc2beef7b by @localai-bot in https://github.com/mudler/LocalAI/pull/10694
• fix(reasoning): don't persist request-scoped reasoning_effort as an operator disable (#10622) by @Anai-Guo in https://github.com/mudler/LocalAI/pull/10623
• fix(startup): scope generated-content and upload dirs to the current user by @localai-bot in https://github.com/mudler/LocalAI/pull/10698
• fix(config): cap auto-derived context to fit VRAM by @localai-bot in https://github.com/mudler/LocalAI/pull/10696
• fix(llama-cpp): cap single-pass embedding batch to fit VRAM by @localai-bot in https://github.com/mudler/LocalAI/pull/10695

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.6.0...v4.6.1

🎉 LocalAI 4.6.0 Release! 🚀

LocalAI 4.6.0 is out!

This is a reliability-focused release: AMD ROCm backends now run on-GPU at full speed, distributed model loads no longer wedge when a worker dies, and realtime sessions warm up predictably. It also brings conversation forking to the built-in chat UI, a Prometheus counter for PII/audit events, and an SSRF fix for the model gallery.

Highlights:

• 🔴 AMD ROCm runs correctly - ggml audio backends offload to the GPU, hipBLASLt kernel-tuning data is bundled (no more slow generic kernels), rocm-vllm installs the right wheel, and the ASIC ID table is found.
• 🎙️ Predictable realtime - sessions eagerly warm the whole pipeline (VAD, ASR, LLM, TTS) up front, so the first turn no longer pays per-model cold-start stalls, plus a new POST /backend/load API and "Load into memory" UI button.
• 🌿 Forking chat - retry any assistant answer, branch a new chat from any point, duplicate, or copy the whole conversation, directly in the built-in UI.
• 🛡️ Distributed hardening - a dead worker can no longer pin the model-load advisory lock (the ~15-minute wedge is gone), and orphaned backend workers self-terminate instead of holding VRAM.
• 📊 PII/audit metrics - PII detections/masks/blocks are exported as a Prometheus counter, so you can alert when the filter stops firing.
• 🔒 Gallery SSRF fix - POST /models/apply config-URL fetches are validated against private/loopback/metadata addresses.

Plus idempotent backend installs, tool-calling and reasoning fixes across the vLLM and Python/MLX backends, cloud-proxy compatibility with the newest reasoning models, and the usual set of dependency updates.

---

📌 TL;DR

Area	Summary
🔴 AMD ROCm reliability	ggml audio backends now compile with -DGGML_HIP=ON and link HIP (real GPU offload); hipBLASLt TensileLibrary data bundled + HIPBLASLT_TENSILE_LIBPATH exported; rocm-vllm installs from the AMD wheel index on Python 3.12; amdgpu.ids symlinked so the ASIC table is found.
🎙️ Realtime warm-up + load API	Sessions block-warm the full pipeline at start (errors surface up front); new POST /backend/load / POST /v1/backend/load, a "Load into memory" UI action, and a load_model MCP tool. Opt out per pipeline with disable_warmup: true.
🌿 Forking chat	Regenerate any assistant answer (not just the last), branch a new chat from any turn, duplicate a chat, or copy it as Markdown - all client-side in the React UI.
🛡️ Process & distributed lifecycle	A dead worker no longer pins the per-model PostgreSQL advisory lock (bounded load ceiling + context-scoped lock_timeout); backend workers self-terminate on parent death (LOCALAI_BACKEND_PARENT_WATCH); the watchdog stops logging optional Free() as an error.
⚙️ Idempotent backend installs	POST /backends/apply and the LOCALAI_EXTERNAL_BACKENDS boot loop no longer re-pull an already-installed backend unless force: true.
📊 PII/audit Prometheus counter	localai_pii_events_total{kind,origin,action,direction} on /metrics, complementing the /api/pii/events ring buffer.
🔒 Gallery SSRF hardening	Gallery config URL fetches run through ValidateExternalURL, blocking private, loopback, link-local, and cloud-metadata addresses.
🧩 Tool-calling & reasoning fixes	Non-streaming vLLM tool calls restored; MLX/Python backends decode tool-call arguments for chat templates and split closing-only </think> reasoning blocks.

---

🚀 New Features & Major Enhancements

🔴 AMD ROCm backends run correctly on-GPU

Four coupled fixes make ROCm/hipBLAS backends actually run on AMD hardware, and at full speed, instead of silently falling back to CPU or slow generic kernels:

• GPU offload for ggml audio backends (#10667): rocm-qwen3-tts-cpp, rocm-omnivoice-cpp, acestep-cpp, and vibevoice-cpp were building CPU-only because their Makefiles passed the no-op -DGGML_HIPBLAS=ON (upstream ggml only understands -DGGML_HIP=ON) and the CMake link loop omitted hip. They now use the same hipblas recipe as llama-cpp and link the HIP backend.
• hipBLASLt kernel-tuning data (#10660, #10672): the packager bundled rocBLAS data but not the parallel hipBLASLt TensileLibrary_lazy_gfx*.dat files, so every arch silently used slow kernels and logged Cannot read "TensileLibrary_lazy_gfx*.dat". The data is now bundled and HIPBLASLT_TENSILE_LIBPATH is exported by the llama-cpp and turboquant run.sh.
• rocm-vllm installs the right wheel (#10642, #10651): the backend was pulling the CUDA-only PyPI vllm (fatal ModuleNotFoundError: No module named 'vllm' on AMD). It now pins CPython 3.12 and installs vLLM from the ROCm wheel index (https://wheels.vllm.ai/rocm/).
• ASIC ID table found (#10624, #10627): the compute-only hipblas image lacks /opt/amdgpu/share/libdrm/amdgpu.ids, so every model load warned. Ubuntu's libdrm-common copy is now symlinked into place.

🔗 PRs: #10667, #10672, #10651, #10627

🎙️ Realtime: eager pipeline warm-up + a load-into-memory API

Realtime voice sessions now eagerly and blockingly warm the entire pipeline (VAD, transcription, LLM, TTS, sound detection, voice recognition) at session start instead of lazy-loading each sub-model on first use. The first turn no longer pays per-model cold-start stalls, and model-load errors surface up front at session start (as model_load_error) rather than mid-stream. Pipeline sub-models load concurrently, so a session warms in the time of its slowest stage, not the sum, and a failed stage names every broken model in a joined error.

This also adds a LocalAI-native POST /backend/load (and /v1/backend/load), the inverse of /backend/shutdown, exposed as a "Load into memory" UI action and a load_model MCP admin tool, so admins can pre-warm any model (including full pipelines) on demand. The --load-to-memory startup flag now routes through the same engine. Opt out per pipeline with disable_warmup: true.

🔗 PRs: #10662

🌿 Forking chat in the built-in UI

The React chat UI gains conversation-management tools: regenerate any assistant answer (not just the last), branch a new chat from any answer, duplicate a chat into an independent copy, or copy the whole conversation to the clipboard as Markdown. Retrying a mid-conversation answer correctly truncates the conversation before re-asking, both in the DOM and in the request payload (this also fixes a latent stale-closure bug where a mid-conversation retry sent the downstream turns back to the model). All client-side, no backend changes.

🔗 PRs: #10654

🛡️ Sturdier process and distributed lifecycle

• Dead-worker advisory-lock wedge (#10600): a distributed worker going mid-load could pin a per-model PostgreSQL advisory lock and fail every subsequent request to that model with 55P03 for ~15 minutes. The detached load context is now bounded by a model-load ceiling, the install wait honors cancellation via singleflight.DoChan, and lock_timeout is scoped to the caller's context budget instead of a deployment-global GUC.
• Parent-death safety net (#10639): if LocalAI is SIGKILLed before teardown, spawned backend workers used to get reparented to init and linger, holding VRAM and their port. Each backend now polls its parent PID and self-terminates on reparenting. Configurable via LOCALAI_BACKEND_PARENT_WATCH (default on, auto-off on Windows) and LOCALAI_BACKEND_PARENT_WATCH_INTERVAL (default 2s). C++ coverage is llama-cpp for now; Python covers all backends.
• Quieter watchdog (#10602, #10607): the optional Free() RPC returns gRPC Unimplemented for many backends and the federation proxy, so the watchdog no longer logs a misleading Error freeing GPU resources on eviction. A new grpcerrors.IsUnimplemented helper distinguishes it from genuine failures.
• Idempotent backend installs (#10643): POST /backends/apply and the LOCALAI_EXTERNAL_BACKENDS boot loop no longer re-download and re-extract an already-installed backend on every apply/boot. Pass "force": true (the UI's install button still does, doubling as "Reinstall").

🔗 PRs: #10600, #10639, #10607, #10643

📊 PII/audit events as a Prometheus counter

The PII middleware / MITM audit pipeline now emits a single monotonic counter, localai_pii_events_total{kind, origin, action, direction}, on /metrics, instrumented at the EventStore.Record choke point. Labels are cardinality-bounded (no pattern or user IDs). This complements the capacity-bound /api/pii/events ring buffer and, crucially, makes silent filter failure alertable: rate() on the counter detects that the PII filter stopped firing after a deploy.

🔗 PRs: #10641

🔒 Gallery SSRF hardening

POST /models/apply with an empty id fetches the supplied url directly; in a default Docker setup (no API key) any reachable client could probe internal services or cloud-metadata (169.254.169.254) and exfiltrate a slice via the job error. Gallery config fetches now run through the existing ValidateExternalURL guard (the same one protecting the CORS proxy and media downloads), blocking private, loopback, link-local, unspecified, and metadata addresses. Only plain http(s):// is validated; huggingface://, github:, oci://, ollama://, and file:// are untouched.

🔗 PRs: #10673

---

🐛 Bug Fixes (recap)

• fix(vllm): restore non-streaming tool-call extraction that regressed after #10351 (a capability flag was mistaken for run state) - #10638
• fix(python-backends): decode tool-call arguments for chat templates (unbreaks MLX/Qwen3.5 agent loops) and split reasoning when a model emits only a closing </think> - #10658
• fix(cloud-proxy): drop temperature/top_p and send max_completion_tokens so routing to the newest reasoning models (Claude Opus 4.x, GPT-5.x) stops 400ing - #10640
• fix(config): revert defaulting swa_full:true for sliding-window-attention models (restores the memory-light reduced KV cache; still available as an explicit per-model opt-in) - #10674
• fix(kokoros): implement the AudioTranscriptionLive trait stub so the backend compiles against the updated proto - #10612
• fix(launcher): keep the desktop launcher's data/config under ~/.localai instead of the GUI's working directory - #10610, #10613

---

👒 Dependencies

Submodule and backend bumps this cycle:

• ggml-org/llama.cpp x4
• ikawrakow/ik_llama.cpp x4
• CrispStrobe/CrispASR x4
• leejet/stable-diffusion.cpp x3
• vllm-metal (darwin) x3
• ggml-org/whisper.cpp x2
• mudler/parakeet.cpp x1
• localai-org/privacy-filter.cpp x1
• vllm-project/vllm cu130 wheel to 0.24.0

Plus new gallery models added via the gallery agent (#10663, #10644).

---

📖 Documentation

• Docs version bump for the release - #10614

---

🙌 New Contributors

• @alaningtrump made their first contribution in #10657

---

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.5.6...v4.6.0

What's Changed

👒 Dependencies

• chore(deps): bump actions/cache from 4 to 6 by @dependabot[bot] in https://github.com/mudler/LocalAI/pull/10593

Other Changes

• docs: :arrow_up: update docs version mudler/LocalAI by @localai-bot in https://github.com/mudler/LocalAI/pull/10560
• fix(distributed): missing agent NATS permission by @ALameLlama in https://github.com/mudler/LocalAI/pull/10549
• feat(distributed): SyncedMap component + migrate finetune/quant/agent-tasks to cross-replica state by @localai-bot in https://github.com/mudler/LocalAI/pull/10542
• chore(fish-speech): drop the darwin/metal build target by @localai-bot in https://github.com/mudler/LocalAI/pull/10561
• fix(config): fall back to DefaultContextSize for unparseable GGUFs; pin NVFP4 gallery context_size by @localai-bot in https://github.com/mudler/LocalAI/pull/10563
• ci(vibevoice): skip the ASR transcription e2e on release tag builds by @localai-bot in https://github.com/mudler/LocalAI/pull/10567
• fix(gallery): match mmproj/model quant as a whole token so F16 no longer selects BF16 (#10559) by @localai-bot in https://github.com/mudler/LocalAI/pull/10564
• fix(distributed): return empty backend list for agent nodes instead of failing backend.list (#10545) by @localai-bot in https://github.com/mudler/LocalAI/pull/10565
• feat(distributed): add LOCALAI_DISTRIBUTED_SHARED_MODELS to skip staging on shared volumes (#10556) by @localai-bot in https://github.com/mudler/LocalAI/pull/10566
• chore: :arrow_up: Update leejet/stable-diffusion.cpp to 9956436c925a367daeab097598b1ea1f32d3503f by @localai-bot in https://github.com/mudler/LocalAI/pull/10533
• fix(openresponses): bound resume-stream buffer and enforce response ownership by @localai-bot in https://github.com/mudler/LocalAI/pull/10569
• chore: :arrow_up: Update ggml-org/whisper.cpp to 0ae02cdb2c7317b50991367c165736ce42ed96ac by @localai-bot in https://github.com/mudler/LocalAI/pull/10532
• chore: :arrow_up: Update CrispStrobe/CrispASR to 6514c9da00b03a2f0f1b49a43fae4f3a01a41844 by @localai-bot in https://github.com/mudler/LocalAI/pull/10535
• chore: :arrow_up: Update ggml-org/llama.cpp to 0ed235ea2c17a19fc8238668653946721ed136fd by @localai-bot in https://github.com/mudler/LocalAI/pull/10536
• fix(ik-llama): port multimodal path to mtmd API and bump to f96eaddb (#10534) by @localai-bot in https://github.com/mudler/LocalAI/pull/10568
• feat(backends): add voice-detect + face-detect ggml backends (replace Python insightface/speaker-recognition) by @localai-bot in https://github.com/mudler/LocalAI/pull/10441
• fix(kokoro): add explicit click dep so spacy CLI works on intel build by @localai-bot in https://github.com/mudler/LocalAI/pull/10572
• fix(launcher): robust binary download/upgrade (resume, rate-limit, UX) by @localai-bot in https://github.com/mudler/LocalAI/pull/10575
• fix(distributed): missing agent NATS permissions by @ALameLlama in https://github.com/mudler/LocalAI/pull/10571
• fix(fish-speech): allow invalid_reference_casting so tokenizers builds on darwin by @localai-bot in https://github.com/mudler/LocalAI/pull/10573
• fix(oci): retry layer downloads on transient network errors by @localai-bot in https://github.com/mudler/LocalAI/pull/10579
• chore(model-gallery): :arrow_up: update checksum by @localai-bot in https://github.com/mudler/LocalAI/pull/10585
• chore: :arrow_up: Update leejet/stable-diffusion.cpp to c1790754d31bec0731ed5fddc9d5b9ff22ee19cd by @localai-bot in https://github.com/mudler/LocalAI/pull/10584
• chore: :arrow_up: Update CrispStrobe/CrispASR to 6b50f76e59700665358a1aabf5295597fa318e06 by @localai-bot in https://github.com/mudler/LocalAI/pull/10583
• chore: :arrow_up: Update ggml-org/llama.cpp to dbdaece23de9ac63f2e7ca9e6bfcdc4fc156a3fa by @localai-bot in https://github.com/mudler/LocalAI/pull/10582
• chore: :arrow_up: Update mudler/voice-detect.cpp to 3d510772357538c5182808ac7de2278b84824e24 by @localai-bot in https://github.com/mudler/LocalAI/pull/10581
• chore: :arrow_up: Update mudler/face-detect.cpp to 06914b077d52f90d5421299138e7be6bdd06b5e8 by @localai-bot in https://github.com/mudler/LocalAI/pull/10580
• chore: :arrow_up: Update vllm-metal (darwin) to v0.3.0.dev20260628073537 by @localai-bot in https://github.com/mudler/LocalAI/pull/10562
• chore(recon): re-pin voice/face-detect to squashed release commits (+ graph-cache fix) by @localai-bot in https://github.com/mudler/LocalAI/pull/10591
• fix(sglang): parse tool_call function arguments before applying the chat template by @pos-ei-don in https://github.com/mudler/LocalAI/pull/10558
• feat(realtime): Semantic VAD EOU token by @richiejp in https://github.com/mudler/LocalAI/pull/10444
• fix(openai): stop max_tokens streaming retry loop on reasoning models (#9716) by @Dennisadira in https://github.com/mudler/LocalAI/pull/10448
• fix(import): derive model name from selected GGUF for repo-root URIs by @Dennisadira in https://github.com/mudler/LocalAI/pull/10589
• fix(functions): avoid quadratic-time debug logging in CleanupLLMResult / ParseFunctionCall by @pos-ei-don in https://github.com/mudler/LocalAI/pull/10592
• chore: :arrow_up: Update leejet/stable-diffusion.cpp to 3b6c9ca97cfcda8e68e719e6670d06379fcbe943 by @localai-bot in https://github.com/mudler/LocalAI/pull/10594
• chore: :arrow_up: Update ggml-org/llama.cpp to 6f4f53f2b7da54fcdbbecaaa734337c337ad6176 by @localai-bot in https://github.com/mudler/LocalAI/pull/10595
• chore: :arrow_up: Update localai-org/privacy-filter.cpp to 595f59630c69d361b5196f2aba2c71c873d0c13c by @localai-bot in https://github.com/mudler/LocalAI/pull/10596
• chore: :arrow_up: Update CrispStrobe/CrispASR to 3b93758f9725d400eca82976f895e4cec3f31260 by @localai-bot in https://github.com/mudler/LocalAI/pull/10597
• chore: :arrow_up: Update ikawrakow/ik_llama.cpp to f74a6fb87b315b2c3154166e075360e15021a61d by @localai-bot in https://github.com/mudler/LocalAI/pull/10598
• fix(import): strip file:// scheme from model path for local imports by @Dennisadira in https://github.com/mudler/LocalAI/pull/10599
• fix(tests): align openresponses test model name with GGUF-derived naming (#10589) by @localai-bot in https://github.com/mudler/LocalAI/pull/10609
• fix(macos): staple the notarization ticket to the .app, not just the dmg by @localai-bot in https://github.com/mudler/LocalAI/pull/10606
• fix(watchdog): persist UI-saved Check Interval across restarts (#10601) by @localai-bot in https://github.com/mudler/LocalAI/pull/10605
• feat(config): default swa_full:true for sliding-window-attention models by @localai-bot in https://github.com/mudler/LocalAI/pull/10611

New Contributors

• @ALameLlama made their first contribution in https://github.com/mudler/LocalAI/pull/10549

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.5.5...v4.5.6

What's Changed

Other Changes

• fix(backends): repair release CI build/test breaks (kokoros, fish-speech, llama-cpp-quantization, sglang) by @localai-bot in https://github.com/mudler/LocalAI/pull/10547
• chore(model gallery): :robot: add 1 new models via gallery agent by @localai-bot in https://github.com/mudler/LocalAI/pull/10544
• fix(backends): whisper darwin run.sh loads whichever fallback lib exists (.so/.dylib) by @localai-bot in https://github.com/mudler/LocalAI/pull/10553

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.5.4...v4.5.5

What's Changed

Other Changes

• fix(backends): derive darwin RUN_BINARY from the exec line only by @localai-bot in https://github.com/mudler/LocalAI/pull/10541

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.5.3...v4.5.4

What's Changed

Other Changes

• feat(macos): sign and notarize the DMG, app, and server binary by @localai-bot in https://github.com/mudler/LocalAI/pull/10510
• fix(backends): set rpath on the piper darwin binary so it can load its bundled libs by @localai-bot in https://github.com/mudler/LocalAI/pull/10525
• fix(backends): darwin packaging for silero-vad (last Linux-only Go backend) by @localai-bot in https://github.com/mudler/LocalAI/pull/10528
• chore(model gallery): :robot: add 1 new models via gallery agent by @localai-bot in https://github.com/mudler/LocalAI/pull/10526
• fix(nodes): show a node's existing labels on the detail view by @localai-bot in https://github.com/mudler/LocalAI/pull/10529
• docs: :arrow_up: update docs version mudler/LocalAI by @localai-bot in https://github.com/mudler/LocalAI/pull/10531
• chore: :arrow_up: Update mudler/parakeet.cpp to f469a57270a1cc4554acb15febf60e56619673b9 by @localai-bot in https://github.com/mudler/LocalAI/pull/10530
• fix(gpu-libs): bundle transitive deps of GPU runtime libs (#10537) by @localai-bot in https://github.com/mudler/LocalAI/pull/10539
• fix(distributed): broadcast admin model-config changes across replicas by @localai-bot in https://github.com/mudler/LocalAI/pull/10540
• fix(llama-cpp): stop reinterpreting plain-string message content as JSON (#10524) by @localai-bot in https://github.com/mudler/LocalAI/pull/10538

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.5.2...v4.5.3

What's Changed

Other Changes

• fix(backends): make the opus backend build and package on macOS/Darwin by @localai-bot in https://github.com/mudler/LocalAI/pull/10523

Full Changelog: https://github.com/mudler/LocalAI/compare/v4.5.1...v4.5.2