# transformers v5.16.0 — Release: v5.16.0 - Product: transformers (https://whatsnew.fyi/product/transformers) - Vendor: huggingface - Date: 2026-08-26 - Version: v5.16.0 - Original notes: https://github.com/huggingface/transformers/releases/tag/v5.16.0 - Permalink: https://whatsnew.fyi/product/transformers/releases/v5.16.0 What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog. Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'. --- - **added** — Add Qwen4Exp model with GatedResidual architecture, Qwen Sparse Attention, and Per-Layer Embedding components - **added** — Add Granite Speech 5.0 Turbo CTC, a lightweight conformer encoder for automatic speech recognition with frame stacking, block attention, and self-conditioned CTC - **added** — Add Step-3.7-Flash, a 198B-parameter sparse Mixture-of-Experts vision-language model with gated attention and multi-token prediction support - **added** — Add CohereCompass base architecture for small specialized vision-language models - **added** — Add ESMC and ESMFold2 state-of-the-art protein language and folding models from BioHub with support for sequence and token classification tasks #### Release v5.16.0 ##### New Model additions ###### Qwen4-Exp image Qwen4-Exp builds on Qwen3.5's hybrid text and multimodal architecture with three key components: GatedResidual (GR), Qwen Sparse Attention (QSA), and Per-Layer Embedding (PLE). GR is a Qwen-developed residual architecture that combines Hyper-Connection with GatedNorm. It mixes multiple residual streams with fine-grained elementwise gating before each attention and Mixture-of-Experts (MoE) block, then controls how much of the block output is injected back into each stream. QSA uses multiple query heads to score compressed key blocks, selects the most relevant contiguous token blocks, and keeps the incomplete trailing block uncompressed. This block-level selection reduces indexing overhead and improves memory locality for long sequences. Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first hybrid architecture to integrate linear and sparse attention, substantially improving inference efficiency for long-context workloads. PLE enriches selected decoder layers with layer-specific lexical features derived from hashed token n-grams and a dilated depthwise convolution. **Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/qwen4_exp) * Add Qwen4Exp model (#48337) by @Cyrilvallez in [#48337](https://github.com/huggingface/transformers/pull/48337) ###### GraniteSpeech5 image Granite Speech 5.0 Turbo CTC is a lightweight (~470M parameters) conformer encoder for automatic speech recognition, trained with Connectionist Temporal Classification (CTC) on BPE targets. It is a fast, encoder-only member of the [Granite Speech](https://huggingface.co/papers/2505.08699) family: transcription requires a single forward pass followed by greedy CTC decoding, with no autoregressive decoder. Architecturally, it extends the Granite Speech conformer CTC encoder with: 1. **Frame stacking + block-wise time subsampling**: the feature extractor stacks pairs of log-mel(+delta) frames (2x), and the first two conformer blocks each subsample time by 2 through a stride-2 depthwise convolution (with a mean-pooled residual), for a total 8x time reduction at 10 ms mel hop. 2. **Block attention with Shaw's relative positional embeddings**: attention is computed over fixed-size blocks (the sequence is right-padded to a whole number of blocks, with padded frames masked out), using separate bias-free query/key/value projections. 3. **Self-conditioned CTC**: the CTC posteriors of the middle layer are projected and fed back into the hidden states, and the CTC head is shared between this mid-layer self-conditioning and the final prediction. **Links:** [Documentation](https://huggingface.co/docs/transformers/main/en/model_doc/granite_speech5) * Add Granite Speech 5.0 - (#48288) by @eustlb in [#48288](https://github.com/huggingface/transformers/pull/48288) ###### Step3p7 Step-3.7-Flash was proposed in [Step 3.7 Flash](https://static.stepfun.com/blog/step-3.7-flash/) by StepFun. It is a 198B-parameter sparse Mixture-of-Experts vision-language model, pairing a 196B-parameter MoE language backbone with a 1.8B-parameter vision encoder for native image understanding. StepFun hasn't published a technical report for Step-3.7-Flash, so the details below are drawn from the released checkpoint's configuration rather than a paper. - **Sparse MoE decoder**: all but the first 3 decoder layers route through a MoE block of 288 routed experts (top-8 per token) plus a single shared expert. The router scores experts with a sigmoid and a learned per-expert bias instead of an auxiliary load-balancing loss, the same strategy as [DeepSeek-V3](./deepseek_v3). - **Gated attention* _[Truncated at 4000 characters — full notes: https://github.com/huggingface/transformers/releases/tag/v5.16.0]_