# transformers changelog
> π€ Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
- Vendor: huggingface
- Category: AI
- Official site: https://huggingface.co/transformers
- Tracked by: What's New (https://whatsnew.fyi/product/transformers)
- Harvested from: GitHub (huggingface/transformers)
- Entries below: 10 (newest first)
What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog.
## Releases
### v5.14.1 β Patch release: v5.14.1
- Date: 2026-07-16
- Version: v5.14.1
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.14.1
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.14.1
- **fixed** β Fix sdpa prefill with position_bias
- **fixed** β Fix assisted decoding for models with EncoderDecoderCache
- **changed** β Bump FP8 kernels version
- **fixed** β Fix deepgemm on multiple devices
#### Patch release v5.14.1
This patch solves a few issues which appeared when integrating Inkling model, most notably an issue affecting models using EncoderDecoderCache during assisted generation. It also fixes an issue that could appear during prefill with StaticCache and sdpa without padding for Inkling which uses a position_bias.
It contains the following commits:
- Fix sdpa prefill with position_bias (#47359) by @Cyrilvallez
- Fix assisted decoding for models with EncoderDecoder cache & OlmoHybrid (#47361) by @Cyrilvallez
- [FP8] Bump kernels version (#47344) by @vasqu
- Fix deepgemm on multiple devices (#47323) by @IlyasMoutawwakil
### v5.14.0 β Release v5.14.0
- Date: 2026-07-15
- Version: v5.14.0
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.14.0
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.14.0
- **added** β Add Inkling model, a 975B total/41B active multimodal model that accepts text, image and audio inputs
- **added** β Add TIPSv2 model
- **added** β Add TIPSv2 DPT model
- **added** β Add Multi-Token Prediction (MTP) decoding support in generation
- **added** β Add static ensemble verification for speculative decoding to improve draft token acceptance rates
- **changed** β GPTNeoX now remaps embed_out to lm_head
- **changed** β GPTBigCode has _supports_attention_backend = True enabled for vLLM compatibility
- **changed** β Simplify cache dispatch based on layer_types with explicit layer-type mappings for sliding and static layers
- **changed** β SDPA prefill now leverages FlashAttention kernel with StaticCache for significant performance gains
- **fixed** β Fix misleading synced_gpus warning in continuous batching mode
- **fixed** β Fix crash in greedy assisted generation with different tokenizers
- **fixed** β Fix Flash Attention performance regression affecting models like Qwen3-VL
- **fixed** β Fix MoE decode optimization bug where grouped-to-batched matrix multiplication switch was not applied to experts in submodels
- **fixed** β Fix CI read-only cache failures by patching cached_files in conftest
- **fixed** β Fix MPS graph cache growth during variable-length batch training on Apple Silicon via torch_empty_cache_steps
- **fixed** β Fix half-precision torch.compile crash in DETR-family sine position embeddings
- **fixed** β Fix hardcoded paths in siglip checkpoint and vocab loading
- **fixed** β Make DeepGEMM Triton fallback more robust when CUDA_HOME is unset or misconfigured
- **fixed** β Remove deprecated package_name argument from LocalLayerRepository
- **fixed** β DeepGEMM shouldn't pad on SM90
- **fixed** β Resolve continuous batching XPU availability checks at runtime
#### Release v5.14.0
##### New Model additions
###### Inkling (fresh from Thinking Machines): 975B total, 41B active
* Add Inkling model #47347 by @molbap @Cyrilvallez @eustlb and @zucchini-nlp
Inkling is a general-purpose multimodal model that accepts text, image and audio inputs and
generates text outputs. It is intended for use in English and other languages, and across
multiple coding languages. The model is designed to be used by developers building AI-
powered applications, including agentic and tool-use systems, coding assistants, chatbots, and
retrieval-augmented generation systems, and is suitable for general-purpose conversational
use, instruction-following, and other natural language and multimodal tasks. It is released with
open weights to support research, fine-tuning and integration into third-party products by
downstream developers.
###### TIPSv2
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/tipsv2)
* Add TIPSv2 (#46347) by @Ternura143 in [#46347](https://github.com/huggingface/transformers/pull/46347)
###### TIPSv2 DPT
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/tipsv2_dpt)
* Add TIPSv2 (#46347) by @Ternura143 in [#46347](https://github.com/huggingface/transformers/pull/46347)
##### :rotating_light: Breaking changes
GPTNeoX now remaps `embed_out` to `lm_head` and GPTBigCode has `_supports_attention_backend = True` enabled for vLLM compatibility; users relying on the previous weight naming or attention backend behavior for these models should update their code accordingly.
* :rotating_light: Fix GPTBigCode and GPTNeoX for the Transformers modelling backend for vLLM (#47198) by @hmellor
##### Kernels
Several kernel-related fixes and improvements were made, including pinning the `kernels` dependency to a compatible version in the benchmark workflow, removing a deprecated `package_name` argument from `LocalLayerRepository`, and making the DeepGEMM Triton fallback more robust when `CUDA_HOME` is unset or misconfigured. Additionally, SDPA prefill was updated to leverage the FlashAttention kernel with `StaticCache`, yielding significant performance gains (up to 260% faster for large input sizes).
* Pin kernels to compatible version in benchmark workflow (#47339) by @tarekziade in [#47339]
* [Fix] Remove deprecated argument from `kernels` call (#47100) by @remi-or in [#47100]
* [Fix] Make DeepGEMM triton fallback more robust (#47126) by @remi-or in [#47126]
* [sdpa] Allow prefill to use FA kernel with StaticCache (#47094) by @Cyrilvallez in [#47094]
##### Generation
Generation improvements include adding Multi-Token Prediction (MTP) decoding support, static ensemble verification for speculative decoding to improve draft token acceptance rates, and a fix for crashes in greedy assisted generation with different tokenizers. A misleading double-negative warning message for `synced_gpus` in continuous batching mode was also corrected.
* [generation] Fix misleading synced_gpus warning in continuous batching (#47158) by @Partha-Shankar in [#47158]
* [generate] Add proper MTP support (#46229) by @Cyrilvallez in [#46229]
* Fix crash in greedy assisted generation with different tokenizers (#46936) by @Sunt-ing in [#46936]
* [Generation] Add static ensemble verification for lossy speculative decoding (#45979) by @kasakh in [#45979]
##### Performance
Fixed a Flash Attention performance regression affecting models like Qwen3-VL and resolved a MoE decode optimizatio
_[Truncated at 4000 characters β full notes: https://github.com/huggingface/transformers/releases/tag/v5.14.0]_
### v5.13.1 β Patch release v5.13.1
- Date: 2026-07-11
- Version: v5.13.1
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.13.1
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.13.1
- **fixed** β Be more defensive with remap_legacy_layer_types for custom models
- **fixed** β Fix custom code which doesn't know about the new linear layer type names
- **fixed** β Fix case where _LazyAutoMapping.register is passed a str key
#### Patch release v5.13.1
This patch is focused on enabling `transformers` for the latest release of vllm!
- Be more defensive with remap_legacy_layer_types for custom models (#47245) from @hmellor
- Fix custom code which doesn't know about the new linear layer type names (#47174) from @hmellor
- Fix case where _LazyAutoMapping.register is passed a str key (#47148) from @hmellor
### v5.13.0 β Release v5.13.0
- Date: 2026-07-03
- Version: v5.13.0
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.13.0
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.13.0
- **added** β Add Kimi K2.5 model architecture supporting versions 2.5, 2.6, and 2.7
- **added** β Add MiMo-V2-Flash Mixture-of-Experts language model with 256K context window support
- **added** β Add Nemotron 3.5 ASR multilingual speech recognition model with streaming support
- **added** β Add NemotronAsrStreaming English speech recognition model with cache-aware FastConformer-RNNT architecture
- **added** β Add Qwen3 ASR automatic speech recognition model with forced aligner for transcript timestamping
- **added** β Add ZAYA1 760M active parameter MoE language model
#### Release v5.13.0
##### New Model additions
###### KimiK 2.5, 2.6, and 2.7
This release includes the architecture for Kimi 2.5 which is used by 2.5-2.7:
Kimi K2.5 is an open-source, native multimodal agentic model that advances practical capabilities in long-horizon coding, coding-driven design, proactive autonomous execution, and swarm-based task orchestration. The model was proposed in [Kimi K2.5: Visual Agentic Intelligence](https://www.kimi.com/en/blog/kimi-k2-5) and further improved in [Kimi K2.6: Advancing Open-Source Coding](Kimi K2.5: Visual Agentic Intelligence).
Kimi K2.5 achieves significant improvements on complex, end-to-end coding tasks, generalizing robustly across programming languages (Rust, Go, Python) and domains spanning front-end, DevOps, and performance optimization. The model is capable of transforming simple prompts and visual inputs into production-ready interfaces and lightweight full-stack workflows, generating structured layouts, interactive elements, and rich animations with deliberate aesthetic precision.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/kimi_k25)
* Add new model: Kimi2-6 (#45630) by @zucchini-nlp in [#45630](https://github.com/huggingface/transformers/pull/45630)
###### MiMo-V2-Flash
**MiMo-V2-Flash** is a Mixture-of-Experts (MoE) language model developed by the Xiaomi MiMo team. Designed to establish a new balance between long-context modeling capabilities and inference efficiency, the model is built for strong performance in complex reasoning and agentic tasks. Trained on 27T tokens with native 32k sequence lengths, MiMo-V2-Flash seamlessly supports an extended **256K context window** while significantly reducing KV-cache storage compared to standard global attention models.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/mimo_v2_flash)
* Add Xiaomi MiMo-V2 (#45144) by @casinca in [#45144](https://github.com/huggingface/transformers/pull/45144)
###### Nemotron 3.5 ASR
Nemotron 3.5 ASR is a 600M-parameter multilingual speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizesβ80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cache-aware FastConformer-RNNT architecture is central to this capability: unlike traditional buffered streaming, which repeatedly reprocesses overlapping audio windows, the model processes only each new incoming chunk while reusing cached encoder context from prior chunks. This eliminates redundant computation, significantly improves efficiency, and minimizes end-to-end delay without sacrificing accuracy, making it well suited to real-time transcription workloads.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/nemotron3_5_asr)
* Add Nemotron 3.5 ASR Streaming (#46565) by @eustlb in [#46565](https://github.com/huggingface/transformers/pull/46565)
###### NemotronAsrStreaming
Nemotron ASR Streaming is a 600M-parameter English speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizesβ80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cac
_[Truncated at 4000 characters β full notes: https://github.com/huggingface/transformers/releases/tag/v5.13.0]_
### v5.12.1 β Patch release v5.12.1
- Date: 2026-06-15
- Version: v5.12.1
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.12.1
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.12.1
- **fixed** β Fix peft lower bound
- **fixed** β Fix auto tokenizer to properly resolve the mistral tokenizer when mistral-common is installed
#### Patch release v5.12.1
Updated the lower bound for PEFT and a fix for auto tokenizer to properly resolve the mistral tokenizer (when `mistral-common` is installed). This is similar to v.5.10.3 minus the fixes that were already included in the main release - vLLM will first target 5.10.3 :hugs:
* Fix `peft` lower bound #46605 by @hmellor (#46605)
* mistral common backend fix #46667 by @itazap (#46667)
**Full Changelog**: https://github.com/huggingface/transformers/compare/v5.12.0...v5.12.1
### v5.10.3 β Patch release v5.10.4
- Date: 2026-06-15
- Version: v5.10.3
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.10.3
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.10.3
- **fixed** β regression introduced by #45534
- **fixed** β image/video/audio_token_ids in ProcessorMixin
- **fixed** β InternVL models
- **fixed** β offsets in processing
- **fixed** β peft lower bound
- **fixed** β mistral common backend
#### Patch release v5.10.4
Update: Note that on pypi `5.10.3` doesn't exist and this this saved under `5.10.4` (so essentially a minor version skipped). Sorry about that, that's on me. Just wanted to clarify to make this less confusing!
A few fixes needed for vLLM to sync with transformers :hugs:
* [fix] regression introduced by #45534 #46456 by @eustlb (#46456)
* Fix {image/video/audio}_token_ids in ProcessorMixin #46500 by @hmellor (#46500)
* Fix InternVL models #46524 by @hmellor (#46524)
* Fix the offsets in processing #46525 by @zucchini-nlp (#46525)
* Fix `peft` lower bound #46605 by @hmellor (#46605)
* mistral common backend fix #46667 by @itazap (#46667)
**Full Changelog**: https://github.com/huggingface/transformers/compare/v5.10.2...v5.10.3
### v5.12.0 β Release v5.12.0
- Date: 2026-06-12
- Version: v5.12.0
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.12.0
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.12.0
- **added** β Add MiniMax-M3-VL model, a vision-language model that pairs a CLIP-style vision tower with 3D rotary position embeddings and the MiniMax-M3 text backbone
- **added** β Add PP-OCRv6 model with official weights, a lightweight OCR system with MetaFormer-style building blocks in three model tiers
- **added** β Add Parakeet-RNNT model with Fast Conformer Encoder and RNN-T decoder
- **fixed** β Fix stop string matching for byte-fragment tokens
- **fixed** β Fix torchaudio version not tied to torch version in docker file
- **fixed** β Fix hf_hub_download not placing file in current dir for url_to_local_path
- **fixed** β Honor a concrete dtype in AutoModel for composite checkpoints
- **changed** β Make DiffusionGemma trainable
- **changed** β Require trust_remote_code to run a local-directory custom_generate
- **changed** β Remove unnecessary expand_as in get_placeholder_mask across VLMs
#### Release v5.12.0
##### New Model additions
###### MiniMax-M3-VL
MiniMax-M3-VL is the vision-language member of the MiniMax-M3 family that pairs a CLIP-style vision tower with 3D rotary position embeddings with the MiniMax-M3 text backbone. It uses a mixed dense/sparse Mixture-of-Experts decoder with SwiGLU-OAI gated experts and a lightning indexer for block-sparse attention. The model processes images through a Conv3d patch embedding system and includes specialized components for efficient multimodal understanding and generation.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/minimax_m3_vl)
* Add minimax m3vl (#46600) by @ArthurZucker in [#46600](https://github.com/huggingface/transformers/pull/46600)
###### PP-OCRv6: update documentation and slow tests (#46576)
The official weights for PP-OCRv6 are out: PP-OCRv6 is a lightweight OCR system that combines architectural innovation with data-centric optimization. It redesigns the backbone, detection neck, and recognition neck around a unified MetaFormer-style building block with structural reparameterization. Three model tiers (medium, small, tiny) share the same block primitives, covering deployment scenarios from server to edge.
* PP-OCRv6: update documentation and slow tests (#46576) by @ zhang-prog
###### Add Parakeet-RNNT (#46331)
ParakeetForRNNT: a Fast Conformer Encoder + an RNN-T (RNN Transducer) decoder
- RNN-T Decoder: Standard neural transducer:
- LSTM prediction network maintains language context across token predictions.
- Joint network combines encoder and decoder outputs.
- Greedy transducer decoding for inference: a blank emission advances the encoder frame by one, a non-blank emission stays on the same frame.
* Add Parakeet-RNNT (#46331) by @eustlb
##### Bugfixes and improvements
* [CI] don't export OTELs within the tests (#46602) by @tarekziade in [#46602]
* [CI] capture checkers output in OTEL (#46601) by @tarekziade in [#46601]
* Lfm2: thread `seq_idx` through ShortConv for packed/varlen inputs (#46588) by @ChangyiYang in [#46588]
* put output_hidden_states into filter_output_hidden_states (#46422) by @molbap in [#46422]
* a11 for checkers (#46599) by @tarekziade in [#46599]
* Fix stop string matching for byte-fragment tokens (#46530) by @Incheonkirin in [#46530]
* [DiffusionGemma] better docs and links (#46569) by @gante in [#46569]
* Require `trust_remote_code` to run a local-directory `custom_generate` (#46483) by @LinZiyuu in [#46483]
* Fix torchaudio version not tied to torch version in docker file (#46594) by @ydshieh in [#46594]
* [CI] Enable PR CI for all fork PRs via security gate (#46591) by @ydshieh in [#46591]
* [CB] [Minor] Add parameter to tune default compile level (#46533) by @remi-or in [#46533]
* Make DiffusionGemma trainable (#46568) by @kashif in [#46568]
* docs: π add Turkish translation for README file (#46312) by @onuralpszr in [#46312]
* fix-trainer-tests (#46541) by @SunMarc in [#46541]
* Remove unnecessary expand_as in get_placeholder_mask across VLMs (#44907) by @syncdoth in [#44907]
* [CI] Catch all shell/process execution issues in security gate via Bandit JSON report (#46560) by @ydshieh in [#46560]
* Honor a concrete dtype in AutoModel for composite checkpoints (#46514) by @qflen in [#46514]
* [CI] Implement real security check in PR CI security gate (#46557) by @ydshieh in [#46557]
* [CI] Add 60s delay in security gate for flow observation (#46555) by @ydshieh in [#46555]
* [TBC] [CI] Auto-approve PR CI for fork PRs via security gate (#46553) by @ydshieh in [#46553]
* [CI] fix and make less flaky (#46543) by @zucchini-nlp in [#46543
_[Truncated at 4000 characters β full notes: https://github.com/huggingface/transformers/releases/tag/v5.12.0]_
### v5.11.0 β Release v5.11.0
- Date: 2026-06-10
- Version: v5.11.0
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.11.0
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.11.0
- **added** β Add DiffusionGemma model with block-autoregressive text generation using multi-canvas sampling for improved inference speed
- **added** β Add DeepSeek-V3.2 model with DeepSeek Sparse Attention (DSA) for improved training and inference efficiency in long-context scenarios
- **changed** β Extend KernelConfig API to support n-to-1 module fusion and parameter transformation
- **added** β Add fine-grained fp8/fp4 Triton kernel support
- **fixed** β Fix dtype mismatch in NemotronH/Zamba2 Mamba2 CUDA-kernel path for out_proj
- **changed** β Update FalconMamba fast-path warning to recommend pip install kernels instead of mamba-ssm
- **fixed** β Fix model parallel beam search bugs in Qwen2-VL, Qwen2.5-VL, and Qwen3-VL MoE model families
- **added** β Add documentation for tensor parallelism support with continuous batching
- **fixed** β Fix the offsets in processing
- **fixed** β Fix trailing comma bug in DataCollatorForLanguageModeling example
- **fixed** β Fix missing Gemma4Processor._compute_audio_num_tokens
- **fixed** β Fix InternVL models
- **added** β Add max_requests_per_batch parameter to continuous batching
- **fixed** β Fix regression introduced by previous changes
- **changed** β Use torchvision's native LANCZOS interpolation instead of PIL fallback
- **added** β Add Cosmos3 model
- **fixed** β Fix fbgemm_fp8 to keep the current device aligned with the input tensor
- **fixed** β Fix video classification pipeline video processor
- **changed** β Support non-weight param names in TorchAo safetensors loading
- **fixed** β Fix Cohere2MoeModel device map attribute error in tests
#### Release v5.11.0
##### New Model additions
###### DiffusionGemma
DiffusionGemma is engineered to reduce the sequential bottlenecks of standard causal language models by employing an encoder-decoder architecture specifically optimized for inference speed. During inference, DiffusionGemma leverages multi-canvas sampling, where rather than generating one token at a time, the model iteratively denoises a full block of tokens using a diffusion sampler. This block-autoregressive approach facilitates text generation at higher speeds compared to traditional sequential generation methods.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/diffusion_gemma)
* GPU go brr (#46540) by @gante in [#46540](https://github.com/huggingface/transformers/pull/46540)
###### DeepSeek-V3.2
DeepSeek-V3.2-Exp is an experimental model from DeepSeek-AI that introduces DeepSeek Sparse Attention (DSA), a trainable, fine-grained sparse attention mechanism designed to improve training and inference efficiency in long-context scenarios. Built on top of DeepSeek-V3.1-Terminus with a 685B-parameter Mixture-of-Experts backbone, it reduces the quadratic cost of attention over long sequences by attending only to a selected subset of past tokens while maintaining virtually identical benchmark performance. The work was extended in DeepSeek-V3.2 which pairs DSA with scalable reinforcement learning and achieves gold-medal level results on competition math and competitive programming benchmarks.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/deepseek_v32) | [Paper](https://huggingface.co/papers/2512.02556)
* Add deepseek 3.2 exp (#41251) by @ArthurZucker in [#41251](https://github.com/huggingface/transformers/pull/41251)
##### Kernels
The `KernelConfig` API was extended to support n-to-1 module fusion and parameter transformation, simplifying how custom kernels are integrated with Transformers modules. Additional fixes include resolving a dtype mismatch in the Mamba2 CUDA kernel path for NemotronH/Zamba2, adding fine-grained fp8/fp4 Triton kernel support, and correcting the FalconMamba fast-path warning to recommend `pip install kernels` instead of `mamba-ssm`.
* Extended & simplified n-to-1 kernel fusion via KernelConfig (#46339) by @michaelbenayoun in [#46339]
* Triton finegrained fp8/fp4 (#46407) by @IlyasMoutawwakil in [#46407]
* Fix dtype mismatch in NemotronH/Zamba2 Mamba2 CUDA-kernel path (`out_proj`) (#46487) by @yuekaizhang in [#46487]
* fix(falcon_mamba): recommend `pip install kernels` in fast-path warning (#46343) by @Anai-Guo in [#46343]
##### Parallelization
Fixed model parallel beam search bugs in the Qwen2-VL, Qwen2.5-VL, and Qwen3-VL MoE model families, and added documentation for tensor parallelism support with continuous batching.
* [docs] tp for continuous batching (#46019) by @stevhliu in [#46019]
* revisit history parallel beam search tests to avoid unnecessary fix (#46495) by @kaixuanliu in [#46495]
* fix qwen series VL model's model parallel bug (#46316) by @kaixuanliu in [#46316]
##### Bugfixes and improvements
* Fix the offsets in processing (#46525) by @zucchini-nlp in [#46525]
* Fix buggy action sha pin (#46534) by @ydshieh in [#46534]
* Fix trailing comma bug in DataCollatorForLanguageModeling example (#46527) by @JemmaUZH in [#46527]
* Fix missing Gemma4Processor._compute_audio_num_tokens (#46416) by @csantosbh in [#46416]
* Fix InternVL models (#46524) by @hmellor in [#46524]
* fix(afmoe): reduce tokens in test_compile_static_cache to avoid flaky bfloat16 drift (#46521) by @ydshieh in [#46521]
* [CB] Add a "max_requests_per_b
_[Truncated at 4000 characters β full notes: https://github.com/huggingface/transformers/releases/tag/v5.11.0]_
### v5.10.2 β Patch release v5.10.2
- Date: 2026-06-04
- Version: v5.10.2
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.10.2
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.10.2
- **fixed** β Fix conversion for clip models
#### Patch release v5.10.2
There was a big bug in the model conversion of models related to clip, this affected models like sam3 and others. Please make sure to update :pray:
* Fix conversion for clip models by @zucchini-nlp (#46406)
**Full Changelog**: https://github.com/huggingface/transformers/compare/v5.10.1...v5.10.2
### v5.10.1 β Release v5.10.1
- Date: 2026-06-03
- Version: v5.10.1
- Original notes: https://github.com/huggingface/transformers/releases/tag/v5.10.1
- Permalink: https://whatsnew.fyi/product/transformers/releases/v5.10.1
- **added** β Add Gemma4 unified multimodal model with encoder-free architecture that projects raw inputs directly into language model embedding space
- **added** β Add Sapiens2 family of high-resolution vision transformers for human-centric computer vision tasks including pose estimation and body-part segmentation
- **added** β Add DeepSeek-OCR-2 vision-language model for OCR and document understanding with hybrid attention mechanism
- **added** β Add Mellum code-focused Mixture-of-Experts language model with 12B total parameters and 2.5B active parameters per token
- **changed** β Gemma4 vision pooler now casts inputs to float32 before scaling to prevent float16 overflow with large checkpoints, which may cause minor numerical differences in outputs for users running Gemma-4 vision models in float16
- **changed** β Audio Language Models now have a dedicated base model class without a language modeling head, aligning with Vision Language Models design
- **fixed** β Fix dsv4 dequant with tensor parallelism and expert parallelism
- **fixed** β Rework continuous batching manager for clearer control flow and improved tensor parallelism race condition handling
- **fixed** β Fix model parallel beam search across multiple models
- **fixed** β Fix model parallel issues for AltCLIP and ChineseClip models
- **fixed** β Fix model parallel device mismatch issue in create_bidirectional_mask
- **fixed** β Fix loss over-counting under tensor parallelism and expert parallelism configurations in Trainer.compute_loss
- **added** β Add FSDP initialization via from_pretrained
#### Release v5.10.1
v5.10.0 was yanked as we publish on a corrupted branch. Sorry everyone, this happens when we rush a release!!!
##### New Model additions
###### Gemma4 unified+ Gemma4 MTP
Gemma 4 12B Unified is an **encoder-free** multimodal model with pretrained and instruction-tuned variants. Unlike [standard Gemma 4](./gemma4), which uses dedicated encoder towers, Gemma 4 12B Unified projects raw inputs directly into the language model's embedding space through lightweight linear pipelines. This results in a simpler architecture while maintaining strong multimodal performance.
Key differences from standard Gemma 4:
- **No Vision Tower**: Raw pixel patches are projected directly into LM space via a `Dense + LayerNorm` pipeline with factorized 2D positional embeddings, replacing the vision encoder.
- **No Audio Tower**: Raw 16 kHz waveform samples are chunked into fixed-length frames and projected through a simple `RMSNorm β Linear` pipeline, replacing the mel spectrogram + Conformer encoder.
- **Shared Multimodal Pipeline**: Both vision and audio use the same `Gemma4UnifiedMultimodalEmbedder` (RMSNorm β Linear) for the final projection to text hidden space.
You can find the original Gemma 4 12B Unified checkpoints under the [Gemma 4](https://huggingface.co/collections/google/gemma-4) release.
* who needs encoders? (#46385) by @douglas-reid @sgerrard @vasqu @molbap
###### Sapiens2
Sapiens2 is a family of high-resolution vision transformers pretrained on ~1 billion curated human images, designed for human-centric computer vision tasks including pose estimation, body-part segmentation, surface normal estimation, and pointmap estimation. The models scale from 0.4B to 5B parameters and train at native 1K resolution, with hierarchical 4K variants for extended spatial reasoning. Sapiens2 achieves substantial improvements over its predecessor with +4 mAP in pose estimation, +24.3 mIoU in body-part segmentation, and 45.6% error reduction in normal estimation.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/sapiens2) | [Paper](https://huggingface.co/papers/2604.21681)
* Add Sapiens2 Model (#45919) by @guarin in [#45919](https://github.com/huggingface/transformers/pull/45919)
###### DeepSeek-OCR-2
DeepSeek-OCR-2 is an OCR-specialized vision-language model built on a distinctive architecture that combines a SAM ViT-B vision encoder with a Qwen2 hybrid attention encoder, connected through an MLP projector to a DeepSeek-V2 Mixture-of-Experts (MoE) language model. The model features a hybrid attention mechanism that applies bidirectional attention over image tokens and causal attention over query tokens, enabling efficient and accurate document understanding. It supports both plain OCR tasks and grounding capabilities with coordinate-aware output for document conversion to markdown format.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/deepseek_ocr2)
* Add Deepseek-OCR-2 model (#45075) by @thisisiron in [#45075](https://github.com/huggingface/transformers/pull/45075)
###### Mellum
Mellum is a code-focused Mixture-of-Experts language model developed by JetBrains. It is derived from the Qwen3-MoE architecture with per-layer-type RoPE and interleaved sliding window attention. The model has 12B total parameters with 2.5B active parameters per token, using 64 routed experts with 8 activated per token across 28 layers.
**Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/mellum)
* feat: Add support for JetBrains' `Mellum` v2 code generation model (#46112) by @shadeMe in [#46112](https://github.com/huggingface/transformers/pull/46112)
##### Breaking changes
The Gemma4 vision pooler now casts inputs to float32 before scaling to prevent float16
_[Truncated at 4000 characters β full notes: https://github.com/huggingface/transformers/releases/tag/v5.10.1]_