# Accelerate changelog

> Runs the same PyTorch training script on any distributed or mixed-precision setup.

- Vendor: Hugging Face
- Category: AI
- Official site: https://huggingface.co/docs/accelerate
- Tracked by: What's New (https://whatsnew.fyi/product/accelerate)
- Harvested from: GitHub (huggingface/accelerate)
- Entries below: 10 (newest first)

What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog.

Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'.

## Releases

### v1.14.0 — v1.14.0: AMD ROCm support, FSDP2 hardening

- Date: 2026-06-11
- Version: v1.14.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.14.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.14.0

- **added** — Support dynamic batch size in BatchSamplerShard with even_batches
- **added** — Add padded allgather and broadcast for Neuron devices to reduce recompilation
- **added** — Add max reduction type to utils
- **changed** — Make accelerate work end-to-end on AMD ROCm devices
- **changed** — Torchao offload support for quantized models
- **changed** — Allow kernels flash-attn in DeepSpeed SP
- **changed** — Rename pytorch-triton-xpu to triton-xpu
- **fixed** — Fix FSDP2 fully_shard embedding and norm layers
- **fixed** — Fix FSDP2 load full state dict dtype mismatch
- **fixed** — Fix region compilation fsdpv2
- **fixed** — Cast model to uniform dtype before fully_shard to fix mixed-dtype AssertionError in FSDP2
- **fixed** — Auto-exclude non-floating frozen Params4bit from fully_shard to prevent QLoRA crash
- **fixed** — Fix FSDP2 auto-wrap policy ignoring _no_split_modules fallback
- **fixed** — Use key-based matching in fsdp2_load_full_state_dict
- **fixed** — Add missing model_has_params4bit guard to fsdp2_load_full_state_dict call
- **fixed** — Drop REMOVED and NOT_YET_IMPLEMENTED FSDP1 keys in to-fsdp2 instead of leaking them
- **fixed** — Prevent double-wrapping models in prepare_model()
- **fixed** — Fix int8 offload hook detachment statistics restoration
- **fixed** — Fix keep_in_fp32_modules not working for tied weights in load_and_quantize_model
- **fixed** — Fix dtype_byte_size for FP8 fnuz and e8m0fnu dtypes

##### FSDP2 Improvements

This release brings a large batch of FSDP2 fixes and quality-of-life improvements: correct dtype handling on load, sharding of embeddings/norms, QLoRA crash prevention, and a more robust auto-wrap policy.

  - Fsdp2 fully_shard embedding and norm by @SunMarc in #4015
  - Fix fsdp2 load full state dict dtype mismatch by @SunMarc in #4021
  - Fix region compilation fsdpv2 by @SunMarc in #4022
  - [FSDP2] Cast model to uniform dtype before fully_shard to fix mixed-dtype AssertionError by @roycho96 in #3985
  - [FSDP2] Auto-exclude non-floating frozen Params4bit from fully_shard to prevent QLoRA crash by @roycho96 in #3987
  - fix(FSDP2): auto-wrap policy ignoring _no_split_modules fallback by @JohnGiorgi in #3999
  - fix: use key-based matching in fsdp2_load_full_state_dict by @roycho96 in #3982
  - fix: add missing model_has_params4bit guard to fsdp2_load_full_state_dict call by @roycho96 in #3981
  - Fix to-fsdp2: drop REMOVED / NOT_YET_IMPLEMENTED FSDP1 keys instead of leaking them by @lollinng in #4065
  - Prevent double-wrapping models in prepare_model() by @joshuaswanson in #3977

##### AMD ROCm support

Accelerate now works end-to-end on AMD ROCm devices. Thanks @Abdennacer-Badaoui!

- Make accelerate work end-to-end on AMD ROCm by @Abdennacer-Badaoui in #4025

##### Neuron

Further Neuron improvements to reduce recompilation and cover missing device cases.

- Add padded allgather and broadcast for Neuron devices to reduce recompilation by @czkkkkkk in #4000
- fix: add missing neuron device case by @michaelbenayoun in #4042

##### Quantization & Offloading

We improved offloading support for quantized models, including Torchao, int8, and tied-weight handling.

- Torchao offload by @SunMarc in #3973
- Fix int8 offload hook detachment statistics restoration by @jiqing-feng in #4044
- Fix keep_in_fp32_modules not working for tied weights in load_and_quantize_model by @jiqing-feng in #4043
- Fix dtype_byte_size for FP8 fnuz / e8m0fnu dtypes by @lollinng in #4063

##### Data Loading

  - Feat: Support dynamic batch size in BatchSamplerShard with even_batches by @yuxinyuan in #3969
  - Fix iterable dataset sharding condition when n_shards == num_processes by @SunMarc in #3958
  - Fix implicit padding in split_between_processes when apply_padding=False and num_samples < num_processes by @3manifold in #4052

##### Minor fixes

  - [DeepSpeed] allow kernels flash-attn in SP by @kashif in #3959
  - Fix: Conditionally import torch.distributed.algorithms.join in accelerator.py by @0xDELUXA in #3962
  - Fix is_hf_initialized attribute by @SunMarc in #3976
  - feat(utils): add max reduction type by @imstevenpmwork in #4027
  - fix(state): make MLU backend part of the _prepare_backend elif chain by @Anai-Guo in #4057
  - fix notebook launcher cuda init by @SunMarc in #4059
  - pytorch-triton-xpu rename to triton-xpu by @sywangyi in #4007
  - Relax numerical tolerance for XPU in test_big_modeling by @YangKai0616 in #4001
  - Fix gloo backend error in test_load_checkpoint_and_dispatch_with_broadcast on XPU by @kaixuanliu in #4056
  - Raise ValueError instead of a bare string in ParallelismConfig.get_device_mesh by @lollinng in #4064
  - tests: Gracefully handle missing set_device for mps by @booxter in #4028
  - test: add regression test for no_split_module_classes accepting set type by @UFO0506 in #4048
  - Fix all tests by @SunMarc in #4072
  - docs: add aggregate profiler memory example by @aryanputta in #4054
  - DOC: document missing parameters in load_accelerator_state, find_executable_batch_size, and send_to_device by @kratos0718 in #4051
  - docs: Fix docstring of fsdp2_prepare_auto_wrap_policy by @slocoro in #4037
  - Fix DistributedType documentation by @3manifold in #3980
  - Fix grammar, spelling, and consistency issues across docs and examples by @cihandemir in #3961
  - docs: fix typos in docstrings, comments, and user docs by @mokashang

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/accelerate/releases/tag/v1.14.0]_

### v1.13.0 — v1.13.0: Neuron support, IPEX removal, and distributed training fixes

- Date: 2026-03-04
- Version: v1.13.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.13.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.13.0

- **added** — Support for AWS Neuron (Trainium/Inferentia) devices
- **changed** — Use spawn instead of fork for XPU device
- **removed** — Remove IPEX dependency
- **changed** — Enhance new codes to XPU and make them device agnostic
- **fixed** — Fix KMP_AFFINITY incorrectly set for non-CPU training
- **fixed** — Upcast FSDP2 parameters only if requires_grad
- **fixed** — Fix FSDP2 tied embedding errors with targeted ValueError guidance
- **fixed** — Fix FSDP2 cannot load optimizer state using dcp
- **fixed** — Fix crash in optimizer.step when fsdp2 is enabled and model is bfloat16
- **fixed** — Fix FSDP2 crash with ignored_params on torch < 2.7.0
- **fixed** — Fix loss computation example for DeepSpeed Sequence Parallelism
- **fixed** — Error out if both CP and SP enabled in DeepSpeed
- **changed** — DeepSpeed has its own process group
- **fixed** — Skip device mesh creation when deepspeed and sp_size > 1
- **added** — Enable evaluation during deepspeed Sequence Parallel
- **fixed** — Fix FP8 torchao default config with padding and FSDP2 all-gather support
- **fixed** — Fix execution with Transformer Engine for FP8
- **deprecated** — Add MS-AMP deprecation warnings
- **changed** — Accelerate now imports faster by deferring heavy dependencies
- **changed** — Torch.compile hooks are disabled lazily

##### AWS Neuron support
We now have support for  AWS Neuron (Trainium/Inferentia) devices. Thanks @michaelbenayoun for adding this. 
- Neuron integration by @michaelbenayoun in https://github.com/huggingface/accelerate/pull/3935

###### XPU Improvements
We've removed IPEX dependency and improved device-agnostic code for XPU.
- using spawn instead of fork for XPU device by @kaixuanliu in https://github.com/huggingface/accelerate/pull/3884
- Remove ipex by @yao-matrix in https://github.com/huggingface/accelerate/pull/3883
- enhance new codes to XPU, and make them be device agnostic by @yao-matrix in https://github.com/huggingface/accelerate/pull/3890
- Fix KMP_AFFINITY incorrectly set for non-CPU training by @hexfaker in
https://github.com/huggingface/accelerate/pull/3912

##### FSDP2 Improvements
We've added a bunch of important fixes for FSDP2 users: upcasting only grad-requiring params, better tied embedding errors, DCP optimizer loading, bf16 optimizer step crash fix, and torch < 2.7.0 compatibility.

- Upcast FSDP2 parameters only if requires_grad by @ojh31 in https://github.com/huggingface/accelerate/pull/3848
- Fix FSDP2 tied embedding errors with targeted ValueError guidance by @amanzoni1 in https://github.com/huggingface/accelerate/pull/3878
- bug: fsdp cannot load optimizer state using dcp by @flymin in https://github.com/huggingface/accelerate/pull/3904
- fix crash in optimizer.step when fsdp2 is enabled and model is bfloat16 by @sywangyi in https://github.com/huggingface/accelerate/pull/3905
- Fix FSDP2 crash with ignored_params on torch < 2.7.0 by @Mr-Neutr0n in https://github.com/huggingface/accelerate/pull/3924

##### DeepSpeed Sequence Parallelism
We've added several fixes to the DeepSpeed + Sequence Parallelism integration introduced in v1.12.0, including evaluation support during SP training and proper process group handling.
- [SP] fix loss computation example by @kashif in https://github.com/huggingface/accelerate/pull/3858
- [SP and CP] error out if both CP and SP enabled by @kashif in https://github.com/huggingface/accelerate/pull/3862
- DeepSpeed has its own process group by @kashif in https://github.com/huggingface/accelerate/pull/3916
- [Deepspeed] skip device mesh creation when deepspeed and sp_size >1 by @kashif in https://github.com/huggingface/accelerate/pull/3914
- Enable evaluation during deepspeed Sequence Parallel by @jp1924 in https://github.com/huggingface/accelerate/pull/3917

###### FP8
We've enhanced FP8 training. Thanks @shimizust for fixing torchao support. 
  - Fix FP8 torchao default config with padding and FSDP2 all-gather support by @shimizust in https://github.com/huggingface/accelerate/pull/3831
  - Fix execution with Transformer Engine by @ksivaman in https://github.com/huggingface/accelerate/pull/3852
  - add MS-AMP deprecation warnings by @neha222222 in https://github.com/huggingface/accelerate/pull/3857

###### Performance
Accelerate now imports faster by deferring heavy dependencies, and torch.compile hooks are disabled lazily.
- Faster import by @SunMarc in https://github.com/huggingface/accelerate/pull/3953
- lazy compile disable by @SunMarc in https://github.com/huggingface/accelerate/pull/3947
- Disable hook compile by @SunMarc in https://github.com/huggingface/accelerate/pull/3888

 ### Minor fixes 
- Allow non-Tensor values in a batch with dispatch_batches=True by @tomaarsen in https://github.com/huggingface/accelerate/pull/3850
- fix module and optimizer parameter mismatch before prepare_tp_ by @naomili0924 in https://github.com/huggingface/accelerate/pull/3845
- Fix KeyError in extract_model_from_parallel for partial torch.compile by @amanzoni1 in https://github.com/huggingface/accelerate/pull/3881
- Fix hf_device_map device index comparison in prepare_model by @rezaqorbani in https://github.com/huggingface/accelerate/pull/3895
- Fix StatefulDataLoader KeyError with num_workers > 0 by @veeceey in https://github.com/huggingfa

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/accelerate/releases/tag/v1.13.0]_

### v1.12.0 — v1.12.0: Deepspeed Ulysses/ALST

- Date: 2025-11-21
- Version: v1.12.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.12.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.12.0

- **added** — Add Deepspeed Ulysses/ALST integration for efficient training on long sequences through sequence parallelism and attention head parallelism
- **fixed** — Update torch.optim.Optimizer parameter states after tensor parallelism
- **removed** — Remove warning for cpu_ram_efficient_loading
- **changed** — Add device type helper
- **fixed** — Fix typo in broadcast_object_list docstring
- **fixed** — Update typo in bnb quantisation 4bit flag docstring

##### Deepspeed Ulysses/ALST integration

Deepspeed Ulysses/ALST is an efficient way of training on long sequences by employing sequence parallelism and attention head parallelism. You can learn more about this technology in this paper https://arxiv.org/abs/2506.13996 or this deepspeed tutorial https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism/.

<img width="2368" height="1250" alt="0d8bd9e0" src="https://github.com/user-attachments/assets/b94e90c9-4368-4711-ad57-58de3c714ebc" />


To enable Deepspeed Ulysses, you first need to create `ParallelismConfig` and setting `sp` related args: 

```python
parallelism_config = ParallelismConfig(
    sp_backend="deepspeed",
    sp_size=2,
    sp_handler=DeepSpeedSequenceParallelConfig(...),
)
```
Then, you need to make sure to compute the correct loss as described on our [docs](https://huggingface.co/docs/accelerate/main/en/concept_guides/sequence_parallelism)

```python 
        ...
        losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)
        good_tokens = (shift_labels != -100).view(-1).sum()
        good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)
        total_loss = sum(
            losses_per_rank[rank] * good_tokens_per_rank[rank]
            for rank in range(sp_world_size)
            if good_tokens_per_rank[rank] > 0
        )
        total_good_tokens = sum(good_tokens_per_rank)
        loss = total_loss / max(total_good_tokens, 1)
```

Thanks @S1ro1  for starting this work and for @stas00 for finishing this work. Also thanks @kashif for adding docs and reviewing/testing this PR !

This feature will also be available in HF Trainer thanks for this PR from @stas00: https://github.com/huggingface/transformers/pull/41832


##### Minor changes

* Remove warning for `cpu_ram_efficient_loading` by @SunMarc in https://github.com/huggingface/accelerate/pull/3816
* update typo in bnb quantisation 4bit flag docstring by @hbraith in https://github.com/huggingface/accelerate/pull/3828
* ArXiv -> HF Papers by @qgallouedec in https://github.com/huggingface/accelerate/pull/3834
* Fix typo in broadcast_object_list docstring by @wsntxxn in https://github.com/huggingface/accelerate/pull/3823
* [Bug] Update torch.optim.Optimizer parameter states after tensor parallelism by @naomili0924 in https://github.com/huggingface/accelerate/pull/3835
* use self hosted runner by @SunMarc in https://github.com/huggingface/accelerate/pull/3841
* device type helper by @kashif in https://github.com/huggingface/accelerate/pull/3843

##### New Contributors
* @hbraith made their first contribution in https://github.com/huggingface/accelerate/pull/3828
* @wsntxxn made their first contribution in https://github.com/huggingface/accelerate/pull/3823
* @naomili0924 made their first contribution in https://github.com/huggingface/accelerate/pull/3835

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.11.0...v1.12.0

### v1.11.0 — v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10

- Date: 2025-10-20
- Version: v1.11.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.11.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.11.0

- **added** — Support for MXFP8 in TransformerEngine integration with use_mxfp8_block_scaling option in fp8_config
- **added** — BF16 and FP16 mixed precision support for MPS devices on Mac
- **added** — Support for ignored_params in FSDPv2
- **added** — Support for no_sync() gradient synchronization control in FSDPv2
- **added** — Allow mixed precision policy as dtype string from accelerate CLI flag or fsdp_config
- **added** — Add to_json method for nd-parallelism
- **changed** — Make torch_native_parallelism examples device agnostic
- **changed** — Update nd-parallel examples and cleanup
- **changed** — Refactor Accelerator.autocast() to use with statement instead of __enter__() and __exit__() methods
- **changed** — Drop support for Python 3.9 and bump minimum version to Python 3.10
- **fixed** — CPU RAM efficient loading for nd or HSDP parallelisms
- **fixed** — XPU INT64 all_gather issue
- **fixed** — Specify device_ids in torch.distributed.barrier for PartialState
- **fixed** — Specify device for process_tensor in example usage
- **fixed** — Reduce complexity of get_balanced_memory by adding a set
- **fixed** — Skip CUDA cache flush when origin device is cpu and offloaded to meta
- **fixed** — Convert LayerNorm without bias to FP8
- **fixed** — FSDP2 test case failure on XPU
- **fixed** — SWANLAB_MODE and swanlab tracking
- **removed** — Remove deprecated FindTiedParametersResult

##### TE MXFP8 support

We've added support for MXFP8 in our TransformerEngine integration. To use that, you need to set  `use_mxfp8_block_scaling` in `fp8_config`. See nvidia docs [here]. (https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#MXFP8-and-block-scaling) 

* Add support for TE MXFP8 recipe in accelerate by @pstjohn in https://github.com/huggingface/accelerate/pull/3688

##### FP16/BF16 Training for MPS devices

BF16 and FP16 support for MPS devices is finally here. You can now pass `mixed_precision = "fp16" or "bf16"` when training on a mac (`fp16` requires torch 2.8 and `bf16` requires torch 2.6)
* Add bf16/fp16 support for amp with mps device by @SunMarc in https://github.com/huggingface/accelerate/pull/3373

##### FSDP updates

The following PRs  add respectively support to `ignored_params` and `no_sync()` for FSDPv2:
* feat: add ignored_params support for fsdp2 by @kmehant in https://github.com/huggingface/accelerate/pull/3731
* fix: model.set_requires_gradient_sync(False) should be called to turn off gradient synchronization in FSDP2 by @EquationWalker in https://github.com/huggingface/accelerate/pull/3762

Mixed precision can now be passed as a dtype string from accelerate cli flag or `fsdp_config` in accelerate config file:
* feat: allow mixed precision policy as dtype by @kmehant in https://github.com/huggingface/accelerate/pull/3751

##### Nd-parallel updates

Some minor updates concerning nd-parallelism. 

* Context Parallelism docs typos fixed by @sergiopaniego in https://github.com/huggingface/accelerate/pull/3761
* Feat: add to_json by @S1ro1 in https://github.com/huggingface/accelerate/pull/3743
* make torch_native_parallelism examples device agnostic by @yao-matrix in https://github.com/huggingface/accelerate/pull/3759
* [ND Parallel] Update examples, cleanup by @S1ro1 in https://github.com/huggingface/accelerate/pull/3737

##### Bump to Python 3.10 

We've dropped support for python 3.9 as it reached EOL in October. 
* Bump to python3.10 + update linter by @SunMarc in https://github.com/huggingface/accelerate/pull/3809

###### Lots of minor fixes: 
* fix: CPU RAM efficient loading for nd or HSDP parallelisms by @kmehant in https://github.com/huggingface/accelerate/pull/3740
* xpu INT64 all_gather issue fixed in 2.9 by @yao-matrix in https://github.com/huggingface/accelerate/pull/3756
* Specify device_ids in torch.distributed.barrier for PartialState by @qgallouedec in https://github.com/huggingface/accelerate/pull/3744
* fix: specify device for process_tensor in example usage by @qgallouedec in https://github.com/huggingface/accelerate/pull/3755
* Lower complexity of get_balanced_memory by adding a set by @SamuelBarryCS in https://github.com/huggingface/accelerate/pull/3776
* Fix (skip) cuda cache flush when origin device is `cpu` and offloaded to `meta` by @Qubitium in https://github.com/huggingface/accelerate/pull/3796
* Fix convert LayerNorm without bias to fp8 by @mjun0812 in https://github.com/huggingface/accelerate/pull/3725
* Add optional typing by @cyyever in https://github.com/huggingface/accelerate/pull/3769
* refactor: Use  `with`  in Accelerator.autocast()instead of ` __enter__()` and` __exit__()` for more elegant style. by @EquationWalker in https://github.com/huggingface/accelerate/pull/3767
* switch XPU ccl backend to torch-builtin xccl in test_zero3_integration by @yao-matrix in https://github.com/huggingface/accelerate/pull/3773
* fix FSDP2 test case failure on XPU by @yao-matrix in https://github.com/huggingface/accelerate/pull/3771
* Fix tests by @SunMarc in https://github.com/huggingface/accelerate/pull/3722
* Protect import for device_mesh  by @SunMarc in https://github.com/huggingface/accelerate/pull/3742
* Fix `SWANLAB_MODE` by @SunMarc in https://github.com/huggingface/accelerate/pull/3808
* Fix tracking swanlab by @SunMarc in https://github.com/huggingface/accelerate/pull/3810
* refacto

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/accelerate/releases/tag/v1.11.0]_

### v1.10.1 — v1.10.1: Patchfix

- Date: 2025-08-25
- Version: v1.10.1
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.10.1
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.10.1

- **added** — add to_json
- **fixed** — Protect import for device_mesh

- Feat: add to_json by @S1ro1 in https://github.com/huggingface/accelerate/pull/3743
- Protect import for device_mesh by @SunMarc in https://github.com/huggingface/accelerate/pull/3742. 

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.10.0...v1.10.1

### v1.10.0 — v1.10.0: N-D Parallelism

- Date: 2025-08-07
- Version: v1.10.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.10.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.10.0

- **added** — Introduce ParallelismConfig to allow applying any combination of parallelism strategies (TP, CP, DP) directly in training scripts
- **added** — Add Parallelism getter property to Accelerator class
- **added** — Allow FSDP ignored modules to be specified as regex patterns
- **changed** — Update context parallel to v2.0
- **changed** — Set parallelism_config in Accelerator constructor to prevent reset of State by Trainer
- **changed** — Optimize CpuOffload pre_forward to skip moving tensors if already on device
- **changed** — Make environment variable values case-insensitive in Accelerate
- **changed** — Remove use_ipex
- **fixed** — Set default submesh_tp_size to prevent unset local variable error
- **fixed** — Fix prepare method to work when only tensor parallelism is specified
- **fixed** — Fix tensor parallelism size to be read from environment variables
- **fixed** — Fix FSDP ignored modules attribute to enable training PEFT models with MoE layers containing q_proj and v_proj parameters

#### N-D Parallelism

Training large models across multiple GPUs can be complex, especially when combining [different parallelism strategies](https://huggingface.co/spaces/nanotron/ultrascale-playbook) (e.g TP, CP, DP). To simplify this process, we've collaborated with [Axolotl](https://github.com/axolotl-ai-cloud/axolotl/) to introduce an easy-to-use integration that allows you to apply any combination of parallelism strategies directly in your training script. Just pass a `ParallelismConfig` specifying the size of each parallelism type—it's that simple.
Learn more about how it works in our latest [blogpost](https://github.com/huggingface/blog/pull/3006).

```python
parallelism_config = ParallelismConfig(
    dp_shard_size=2,
    dp_replicate_size=2,
    cp_size=2,
    tp_size=2,
)
accelerator = Accelerator(
    parallelism_config=parallelism_config,
   ...
)
model = AutoModelForCausalLM.from_pretrained("your-model-name", device_mesh=accelerator.torch_device_mesh)
model = accelerator.prepare(model)
```

* Parallelism config + TP + HSDP +  BYODM (Bring Your Own Device Mesh) by @SalmanMohammadi in https://github.com/huggingface/accelerate/pull/3682
* Feat: context parallel v2.0 by @S1ro1 in https://github.com/huggingface/accelerate/pull/3700
* set default submesh_tp_size to prevent unset local variable error by @winglian in https://github.com/huggingface/accelerate/pull/3687
* Add Parallelism getter property to Accelerator class by @WoosungMyung in https://github.com/huggingface/accelerate/pull/3703
* Fix: prepare works even if nothing except tp specified (rare) by @S1ro1 in https://github.com/huggingface/accelerate/pull/3707
* Set parallelism_config in constructor due to Trainer reset of State by @winglian in https://github.com/huggingface/accelerate/pull/3713
* Fix: tp size wouldn't read from env by @S1ro1 in https://github.com/huggingface/accelerate/pull/3716
* Remove `ParallelismConfig` from `PartialState` by @SunMarc in https://github.com/huggingface/accelerate/pull/3720


#### FSDP improvements

We've fixed ignored modules attribute. With this, it is now possible to train PEFT model that moe layers that contrains `q_proj` and `v_proj` parameters. This is especially important for fine-tuning `gpt-oss` model. 

* ENH: Allow FSDP ignored modules to be regex by @BenjaminBossan in https://github.com/huggingface/accelerate/pull/3698
* TST Add test for FSDP ignored_modules as str by @BenjaminBossan in https://github.com/huggingface/accelerate/pull/3719

#### Minor improvements
* feature: CpuOffload pre_forward don't attempt to move if already on device by @JoeGaffney in https://github.com/huggingface/accelerate/pull/3695
* Fix: Ensure environment variable values are case-insensitive in Accelerate by @jp1924 in https://github.com/huggingface/accelerate/pull/3712
* remove use_ipex by @SunMarc in https://github.com/huggingface/accelerate/pull/3721

#### New Contributors
* @SalmanMohammadi made their first contribution in https://github.com/huggingface/accelerate/pull/3682
* @WoosungMyung made their first contribution in https://github.com/huggingface/accelerate/pull/3703
* @jp1924 made their first contribution in https://github.com/huggingface/accelerate/pull/3712
* @JoeGaffney made their first contribution in https://github.com/huggingface/accelerate/pull/3695

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.9.0...v1.10.0

### v1.9.0 — v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

- Date: 2025-07-16
- Version: v1.9.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.9.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.9.0

- **added** — Add support for trackio experiment tracking library with local-first design and Hugging Face Spaces integration
- **changed** — Improve model loading speed by 4-5x when using set_module_tensor_to_device with new clear_device option and non_blocking transfers
- **changed** — Change find_executable_batch_size() default back-off strategy from halving batch size to multiplying by 0.9 to reduce GPU capacity waste
- **added** — Add support for e5e2 and default to hybrid mode when launcher is used
- **fixed** — Fix FP8 to work without direct Accelerator() configuration
- **fixed** — Fix proper error handling when using DDP with Dtensor model
- **fixed** — Fix fsdp2 example typo
- **fixed** — Add check in no_sync() to avoid errors when using deepspeed zero2/3
- **fixed** — Fix wandb config not being saved in offline mode
- **fixed** — Prevent yielding from data loader when base_dataloader is empty
- **changed** — Add warnings for invalid keys
- **changed** — Update Gaudi runner image to latest SynapseAI and enable previously disabled tests
- **removed** — Unpin datasets dependency in examples requirements

#### Trackio tracker support
We've added support for a trackio, lightweight, 💯 free experiment tracking Python library built on top of 🤗 Datasets and Spaces.

![Screen Recording 2025-06-11 at 5 39 32 PM](https://github.com/user-attachments/assets/5cf12286-54e7-4119-8a20-88c2cbd37ab6)

Main features are: 
- *Local-first* design: dashboard runs locally by default. You can also host it on Spaces by specifying a `space_id`.
- Persists logs locally (or in a private Hugging Face Dataset)
- Visualize experiments with a Gradio dashboard locally (or on Hugging Face Spaces)
- Everything here, including hosting on Hugging Faces, is **free**!

To use it with accelerate, you need to set `log_with` and initialize the trackers 
```python 
accelerator = Accelerator(log_with="trackio")
config={"learning_rate": 0.001, "batch_size": 32}
#### init_kwargs in order to host the dashboard on spaces
init_kwargs = {"trackio": {"space_id": "hf_username/space_name"}
accelerator.init_trackers("example_project", config=config, init_kwargs=init_kwargs})
```
Thanks @pcuenca for the integration ! 
* trackio by @pcuenca in https://github.com/huggingface/accelerate/pull/3669

##### Model loading speedup when relying `set_module_tensor_to_device `
Setting tensor while clearing cache is very slow, so we added `clear_device` option to disable it. 
Another small optimization is using `non_blocking` everywhere and syncing just before returning control to the user. This makes the loading slightly faster.
* Speedup model loading by 4-5x in Diffusers ⚡ by @a-r-r-o-w in https://github.com/huggingface/accelerate/pull/3674

##### FDSP, Deepspeed, FP8 minor improvements

* Add support for e5e2 and default to hybrid when launcher is used by @IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3640
* Fix FP8 tests, enable FP8 to be used without direct `Accelerator()` configuring by @pstjohn in https://github.com/huggingface/accelerate/pull/3677
* Bunch of FSDP improvements by @S1ro1 in https://github.com/huggingface/accelerate/pull/3671
* Fix: properly error when DDP + Dtensor model by @S1ro1 in https://github.com/huggingface/accelerate/pull/3629
* Fix fsdp2 example typo by @shimizust in https://github.com/huggingface/accelerate/pull/3657
* Added a check in no_sync() to avoid errors when using deepspeed zero2/3 by @xliu0105 in https://github.com/huggingface/accelerate/pull/3656

##### 🚨🚨🚨 Breaking changes 🚨🚨🚨
`find_executable_batch_size()` will no longer halves the batch after every OOM. Instead, we will multiply the batch size by 0.9. This should help user not waste gpu capacity. 

* “Stop Halving My Batch!” · Default back-off 0.5 → 0.9 by @SunMarc in https://github.com/huggingface/accelerate/pull/3684

##### What's Changed

* [typo] shards instead of shard  by @SunMarc in https://github.com/huggingface/accelerate/pull/3645
* Docs: Fix typos in gradient accumulation guide by @kilavvy in https://github.com/huggingface/accelerate/pull/3649
* xpu enablement on left cases  by @yao-matrix in https://github.com/huggingface/accelerate/pull/3654
* unpin datasets in examples requirements by @SunMarc in https://github.com/huggingface/accelerate/pull/3681
* fix: wandb config not saved in offline mode by @ved1beta in https://github.com/huggingface/accelerate/pull/3648
* accelerate/data_loader.py: do not yield if the base_dataloader is empty by @0xnightwind in https://github.com/huggingface/accelerate/pull/3659
* warn for invalid keys by @ved1beta in https://github.com/huggingface/accelerate/pull/3613
* Update Gaudi runner image to latest SynapseAI and enable previously disabled tests by @IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3653

##### New Contributors
* @kilavvy made their first contribution in https://github.com/huggingface/accelerate/pull/3649
* @shimizust made their first contribution in https://github.com/huggingface/accelerate/pull/3657
* @xliu0105 made their first contribu

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/accelerate/releases/tag/v1.9.0]_

### v1.8.1 — v1.8.1: Patchfix

- Date: 2025-06-20
- Version: v1.8.1
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.8.1
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.8.1

- **added** — Add support for e5e2 and default to hybrid when launcher is used
- **changed** — Shards

- Add support for e5e2 and default to hybrid when launcher is used by @IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3640
- shards by @SunMarc in https://github.com/huggingface/accelerate/pull/3645

**Full Changelog**: https://github.com/huggingface/accelerate/compare/v1.8.0...v1.8.1

### v1.8.0 — v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

- Date: 2025-06-19
- Version: v1.8.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.8.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.8.0

- **changed** — Simplified FSDPv2 model preparation to reduce composition complexity and improve user experience
- **added** — Added FP8 support for distributed training
- **changed** — Updated CCL_WORKER_COUNT variable and added KMP parameters for Intel CPU users to improve distributed training performance
- **added** — Added support for regional compilation with the DeepSpeed engine
- **deprecated** — ipex.optimize is deprecated in favor of PyTorch optimizations
- **changed** — Expanded and stabilized support for Intel XPUs including FSDP2, big model inference, and tensor parallelism
- **added** — Added SwanLab as an experiment tracking backend
- **changed** — Deferred all tracker initializations to prevent premature setup of distributed environments
- **fixed** — Fixed bf16 training with tensor parallelism
- **changed** — Improved FP8 handling with and without DeepSpeed
- **changed** — Updated Gaudi runners
- **removed** — Removed torch_ccl dependency
- **added** — Added support for standalone mode when default port is occupied on single node
- **added** — Added kwargs support to optimizer, scheduler, and dataloader in accelerator().load_state() function
- **fixed** — Fixed list object has no attribute keys error
- **fixed** — Fixed missing te.LayerNorm in intel_transformer_engine
- **added** — Added fp8_e5m2 support in dtype_byte_size
- **added** — Added DeepSpeed automatic gradient accumulation
- **added** — Added device-agnostic GradScaler for XPU support
- **added** — Added use of datasets.IterableDataset shard if possible

#### FSDPv2 refactor + FP8 support

We've simplified how to prepare FSDPv2 models, as there were too many ways to compose FSDP2 with other features (e.g., FP8, torch.compile, activation checkpointing, etc.). Although the setup is now more restrictive, it leads to fewer errors and a more performant user experience. We’ve also added support for FP8. You can read about the results [here](https://github.com/huggingface/accelerate/tree/main/examples/fsdp2). Thanks to @S1ro1 for this contribution!

* [FSDP2] Refactor + FP8 by @S1ro1 in https://github.com/huggingface/accelerate/pull/3585

#### Faster Distributed Training on Intel CPUs

We updated the `CCL_WORKER_COUNT` variable and added `KMP` parameters for Intel CPU users. This significantly improves distributed training performance (e.g., Tensor Parallelism), with up to a 40% speed-up on Intel 4th Gen Xeon when training transformer TP models.

* Set ccl and KMP param in simple launch by @jiqing-feng in https://github.com/huggingface/accelerate/pull/3575

#### Regional Compilation for DeepSpeed

We added support for regional compilation with the DeepSpeed engine. DeepSpeed’s .compile() modifies models in-place using torch.nn.Module.compile(...), rather than the out-of-place torch.compile(...), so we had to account for that. Thanks @IlyasMoutawwakil for this feature!

* Fix deepspeed regional compilation by @IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3609

#### ipex.optimize deprecation
`ipex.optimize` is being deprecated. Most optimizations have been upstreamed to PyTorch, and future improvements will land there directly. For users without PyTorch 2.8, we’ll continue to rely on IPEX for now.

* remove ipex.optimize in accelerate by @yao-matrix in https://github.com/huggingface/accelerate/pull/3608

#### Better XPU Support
We've greatly expanded and stabilized support for Intel XPUs:

* enable fsdp2 benchmark on XPU by @yao-matrix in https://github.com/huggingface/accelerate/pull/3590
* enable big_model_inference on xpu by @yao-matrix in https://github.com/huggingface/accelerate/pull/3595
* enable test_load_checkpoint_and_dispatch_with_broadcast cases on XPU by @yao-matrix in 
* enable test_cli & test_example cases on XPU by @yao-matrix in https://github.com/huggingface/accelerate/pull/3578
* enable torchao and pippy test cases on XPU by @yao-matrix in https://github.com/huggingface/accelerate/pull/3599 
* enable regional_compilation benchmark on xpu by @yao-matrix in https://github.com/huggingface/accelerate/pull/3592
* fix xpu 8bit value loading by @jiqing-feng in https://github.com/huggingface/accelerate/pull/3623
* add device-agnostic GradScaler by @yao-matrix in https://github.com/huggingface/accelerate/pull/3588
* add xpu support in TorchTensorParallelPlugin by @yao-matrix in https://github.com/huggingface/accelerate/pull/3627

#### Trackers

We've added support for [SwanLab](https://github.com/SwanHubX/SwanLab) as an experiment tracking backend. Huge thanks to @ShaohonChen for this contribution ! We also deferred all tracker initializations to prevent premature setup of distributed environments.

* Integrate SwanLab for offline/online experiment tracking for Accelerate by @ShaohonChen in https://github.com/huggingface/accelerate/pull/3605
* Fix: Defer Tracker Initialization to Prevent Premature Distributed Setup by @yuanjua in https://github.com/huggingface/accelerate/pull/3581


##### What's Changed
* Fix bf16 training with TP  by @SunMarc in https://github.com/huggingface/accelerate/pull/3610
* better handle FP8 with and without deepspeed by @IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3611
* Update Gaudi Runners by @IlyasMoutawwakil in https://github.com/huggingface/accelerate/pull/3593
* goodbye torch_ccl by @yao-matrix in https://github.com/huggingface/accelerate/pull/3580
* Add support for standalone mode when default port is occupied on single node by @laitifranz in https://

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/accelerate/releases/tag/v1.8.0]_

### v1.7.0 — v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

- Date: 2025-05-15
- Version: v1.7.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.7.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.7.0

- **added** — Regional compilation targets repeated blocks during model compilation to cache and reuse optimized code, reducing cold start compilation time
- **added** — Layerwise casting hook enables per-layer upcasting and downcasting during inference to run models with separate storage and compute dtypes
- **added** — Support for FULL_STATE_DICT in FSDP2, enabling .save_pretrained() to work with FSDP2 wrapped models
- **added** — QLoRA training support for FSDP2
- **added** — Support for custom function for reducing the batch size
- **added** — HPU support for Intel Gaudi hardware with documentation
- **changed** — Updated logic for torch.compile dynamic argument to explicitly preserve None rather than defaulting to False when USE_DYNAMIC environment variable is unset
- **changed** — Use device agnostic torch.OutOfMemoryError from PyTorch 2.5.0
- **fixed** — Backend issue related to parameter offloading to CPU in FSDP2
- **fixed** — Memory spike when cpu_ram_efficient_loading=True is enabled
- **fixed** — Check tied parameters in config for multimodal models
- **fixed** — FP8 DeepSpeed config
- **fixed** — Unsafe serialization option in merge-weights command
- **fixed** — Tensor parallelism training compatibility with new transformers
- **fixed** — Warning error in accelerate
- **removed** — TorchAO sequential offloading no longer creates new parameters due to weak backward compatibility guarantees

#### Regional compilation

Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start compilation time typically seen during the first inference. Thanks @IlyasMoutawwakil  for the feature ! You can view the full benchmark [here](https://github.com/huggingface/accelerate/tree/main/benchmarks/torch.compile), and check out our updated [compilation guide](https://huggingface.co/docs/accelerate/en/usage_guides/compilation) for more details!

![compilation_time-1](https://github.com/user-attachments/assets/38795d12-6ee7-4a10-84c6-d29a0877e36c)

To enable this feature, set `use_regional_compilation=True` in the `TorchDynamoPlugin` configuration.

```python
#### Configure the compilation backend
dynamo_plugin = TorchDynamoPlugin(
    use_regional_compilation=True,
    ... # other parameters
)
#### Initialize accelerator with the plugin
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
#### This will apply compile_regions to your model
model = accelerator.prepare(model)
```

#### Layerwise casting hook

We've introduced a new hook that enables per-layer upcasting and downcasting (e.g., for Linear layers) during inference. This allows users to run models with separate storage and compute dtypes, resulting in memory savings. The concept was first implemented in [diffusers](https://huggingface.co/docs/diffusers/main/en/optimization/memory#layerwise-casting), where downcasting models to FP8 proved effective without major quality degradation. Contributed by @sayakpaul in https://github.com/huggingface/accelerate/pull/3427

```python
model = ....
storage_dtype = torch.float8_e4m3fn
compute_dtype = torch.bfloat16
attach_layerwise_casting_hooks(
            model,
            storage_dtype=storage_dtype,
            compute_dtype=compute_dtype,
        )
```


#### Better FSDP2 support 

This release includes numerous new features and bug fixes. Notably, we’ve added support for `FULL_STATE_DICT`, a widely used option in FSDP, now enabling `.save_pretrained()` in transformers to work with FSDP2 wrapped models. QLoRA training is now supported as well but more testing is needed. We have also resolved a backend issue related to parameter offloading to CPU. Additionally, a significant memory spike that occurred when `cpu_ram_efficient_loading=True` was enabled has been fixed. Several other minor improvements and fixes are also included—see the **What’s Changed** section for full details.

- `FULL_STATE_DICT` have been enabled by @S1ro1 in https://github.com/huggingface/accelerate/pull/3527
-  QLoRA support by @winglian in https://github.com/huggingface/accelerate/pull/3546
-  set backend correctly for CUDA+FSDP2+cpu-offload in https://github.com/huggingface/accelerate/pull/3574
-  memory spike fixed when using `cpu_ram_efficient_loading=True` by @S1ro1 in https://github.com/huggingface/accelerate/pull/3482

#### Better HPU support:

We have added a [documentation](https://huggingface.co/docs/accelerate/en/usage_guides/gaudi) for Intel Gaudi hardware ! 
The support is already available since v1.5.0 through this [PR](https://github.com/huggingface/accelerate/pull/3378). 

- Add the HPU into accelerate config by @yuanwu2017 in https://github.com/huggingface/accelerate/pull/3495
- Add Gaudi doc by @regisss in https://github.com/huggingface/accelerate/pull/3537

#### Torch.compile breaking change for `dynamic` argument

We've updated the logic for setting `self.dynamic` to explicitly preserve None rather than defaulting to `False` when the `USE_DYNAMIC` environment variable is unset. This change aligns the behavior with the PyTorch documentation for [torch.compile](https://docs.pytorch.org/stable/generated/torch.compile.html). Thanks to @yafshar for contributing this improvement in [#3567](https://github.com/hugg

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/accelerate/releases/tag/v1.7.0]_