Accelerate

AI

Runs the same PyTorch training script on any distributed or mixed-precision setup.

Latest v1.14.0 · by Hugging FaceWebsitehuggingface/accelerate

Release activity

Release activity — 5 releases across 5 days in the last year. Each cell is one day; darker means more releases that day. Older weeks are hidden at this screen width.
MayJunJulAug
SundayNo releases on Apr 26, 2026No releases on May 3, 2026No releases on May 10, 2026No releases on May 17, 2026No releases on May 24, 2026No releases on May 31, 2026No releases on Jun 7, 2026No releases on Jun 14, 2026No releases on Jun 21, 2026No releases on Jun 28, 2026No releases on Jul 5, 2026No releases on Jul 12, 2026No releases on Jul 19, 2026No releases on Jul 26, 2026No releases on Aug 2, 2026No releases on Aug 9, 2026
MondayNo releases on Apr 27, 2026No releases on May 4, 2026No releases on May 11, 2026No releases on May 18, 2026No releases on May 25, 2026No releases on Jun 1, 2026No releases on Jun 8, 2026No releases on Jun 15, 2026No releases on Jun 22, 2026No releases on Jun 29, 2026No releases on Jul 6, 2026No releases on Jul 13, 2026No releases on Jul 20, 2026No releases on Jul 27, 2026No releases on Aug 3, 2026No releases on Aug 10, 2026
TuesdayNo releases on Apr 28, 2026No releases on May 5, 2026No releases on May 12, 2026No releases on May 19, 2026No releases on May 26, 2026No releases on Jun 2, 2026No releases on Jun 9, 2026No releases on Jun 16, 2026No releases on Jun 23, 2026No releases on Jun 30, 2026No releases on Jul 7, 2026No releases on Jul 14, 2026No releases on Jul 21, 2026No releases on Jul 28, 2026No releases on Aug 4, 2026No releases on Aug 11, 2026
WednesdayNo releases on Apr 29, 2026No releases on May 6, 2026No releases on May 13, 2026No releases on May 20, 2026No releases on May 27, 2026No releases on Jun 3, 2026No releases on Jun 10, 2026No releases on Jun 17, 2026No releases on Jun 24, 2026No releases on Jul 1, 2026No releases on Jul 8, 2026No releases on Jul 15, 2026No releases on Jul 22, 2026No releases on Jul 29, 2026No releases on Aug 5, 2026
ThursdayNo releases on Apr 30, 2026No releases on May 7, 2026No releases on May 14, 2026No releases on May 21, 2026No releases on May 28, 2026No releases on Jun 4, 20261 release on Jun 11, 2026No releases on Jun 18, 2026No releases on Jun 25, 2026No releases on Jul 2, 2026No releases on Jul 9, 2026No releases on Jul 16, 2026No releases on Jul 23, 2026No releases on Jul 30, 2026No releases on Aug 6, 2026
FridayNo releases on May 1, 2026No releases on May 8, 2026No releases on May 15, 2026No releases on May 22, 2026No releases on May 29, 2026No releases on Jun 5, 2026No releases on Jun 12, 2026No releases on Jun 19, 2026No releases on Jun 26, 2026No releases on Jul 3, 2026No releases on Jul 10, 2026No releases on Jul 17, 2026No releases on Jul 24, 2026No releases on Jul 31, 2026No releases on Aug 7, 2026
SaturdayNo releases on May 2, 2026No releases on May 9, 2026No releases on May 16, 2026No releases on May 23, 2026No releases on May 30, 2026No releases on Jun 6, 2026No releases on Jun 13, 2026No releases on Jun 20, 2026No releases on Jun 27, 2026No releases on Jul 4, 2026No releases on Jul 11, 2026No releases on Jul 18, 2026No releases on Jul 25, 2026No releases on Aug 1, 2026No releases on Aug 8, 2026

5 releases in the last year

Changelog

v1.14.0

v1.14.0: AMD ROCm support, FSDP2 hardening

Added 3
  • Support dynamic batch size in BatchSamplerShard with even_batches
  • Add padded allgather and broadcast for Neuron devices to reduce recompilation
  • Add max reduction type to utils
Changed 4
  • Make accelerate work end-to-end on AMD ROCm devices
  • Torchao offload support for quantized models
  • Allow kernels flash-attn in DeepSpeed SP
  • Rename pytorch-triton-xpu to triton-xpu
Fixed 13
  • Fix FSDP2 fully_shard embedding and norm layers
  • Fix FSDP2 load full state dict dtype mismatch
  • Fix region compilation fsdpv2
  • Cast model to uniform dtype before fully_shard to fix mixed-dtype AssertionError in FSDP2
  • Auto-exclude non-floating frozen Params4bit from fully_shard to prevent QLoRA crash
  • Fix FSDP2 auto-wrap policy ignoring _no_split_modules fallback
FSDP2 Improvements

This release brings a large batch of FSDP2 fixes and quality-of-life improvements: correct dtype handling on load, sharding of embeddings/norms, QLoRA crash prevention, and a more robust auto-wrap policy.

  • Fsdp2 fully_shard embedding and norm by @SunMarc in #4015
  • Fix fsdp2 load full state dict dtype mismatch by @SunMarc in #4021
  • Fix region compilation fsdpv2 by @SunMarc in #4022
  • [FSDP2] Cast model to uniform dtype before fully_shard to fix mixed-dtype AssertionError by @roycho96 in #3985
  • [FSDP2] Auto-exclude non-floating frozen Params4bit from fully_shard to prevent QLoRA crash by @roycho96 in #3987
  • fix(FSDP2): auto-wrap policy ignoring _no_split_modules fallback by @JohnGiorgi in #3999
  • fix: use key-based matching in fsdp2_load_full_state_dict by @roycho96 in #3982
  • fix: add missing model_has_params4bit guard to fsdp2_load_full_state_dict call by @roycho96 in #3981
  • Fix to-fsdp2: drop REMOVED / NOT_YET_IMPLEMENTED FSDP1 keys instead of leaking them by @lollinng in #4065
  • Prevent double-wrapping models in prepare_model() by @joshuaswanson in #3977
AMD ROCm support

Accelerate now works end-to-end on AMD ROCm devices. Thanks @Abdennacer-Badaoui!

  • Make accelerate work end-to-end on AMD ROCm by @Abdennacer-Badaoui in #4025
Neuron

Further Neuron improvements to reduce recompilation and cover missing device cases.

  • Add padded allgather and broadcast for Neuron devices to reduce recompilation by @czkkkkkk in #4000
  • fix: add missing neuron device case by @michaelbenayoun in #4042
Quantization & Offloading

We improved offloading support for quantized models, including Torchao, int8, and tied-weight handling.

  • Torchao offload by @SunMarc in #3973
  • Fix int8 offload hook detachment statistics restoration by @jiqing-feng in #4044
  • Fix keep_in_fp32_modules not working for tied weights in load_and_quantize_model by @jiqing-feng in #4043
  • Fix dtype_byte_size for FP8 fnuz / e8m0fnu dtypes by @lollinng in #4063
Data Loading
  • Feat: Support dynamic batch size in BatchSamplerShard with even_batches by @yuxinyuan in #3969
  • Fix iterable dataset sharding condition when n_shards == num_processes by @SunMarc in #3958
  • Fix implicit padding in split_between_processes when apply_padding=False and num_samples < num_processes by @3manifold in #4052
Minor fixes
  • [DeepSpeed] allow kernels flash-attn in SP by @kashif in #3959
  • Fix: Conditionally import torch.distributed.algorithms.join in accelerator.py by @0xDELUXA in #3962
  • Fix is_hf_initialized attribute by @SunMarc in #3976
  • feat(utils): add max reduction type by @imstevenpmwork in #4027
  • fix(state): make MLU backend part of the _prepare_backend elif chain by @Anai-Guo in #4057
  • fix notebook launcher cuda init by @SunMarc in #4059
  • pytorch-triton-xpu rename to triton-xpu by @sywangyi in #4007
  • Relax numerical tolerance for XPU in test_big_modeling by @YangKai0616 in #4001
  • Fix gloo backend error in test_load_checkpoint_and_dispatch_with_broadcast on XPU by @kaixuanliu in #4056
  • Raise ValueError instead of a bare string in ParallelismConfig.get_device_mesh by @lollinng in #4064
  • tests: Gracefully handle missing set_device for mps by @booxter in #4028
  • test: add regression test for no_split_module_classes accepting set type by @UFO0506 in #4048
  • Fix all tests by @SunMarc in #4072
  • docs: add aggregate profiler memory example by @aryanputta in #4054
  • DOC: document missing parameters in load_accelerator_state, find_executable_batch_size, and send_to_device by @kratos0718 in #4051
  • docs: Fix docstring of fsdp2_prepare_auto_wrap_policy by @slocoro in #4037
  • Fix DistributedType documentation by @3manifold in #3980
  • Fix grammar, spelling, and consistency issues across docs and examples by @cihandemir in #3961
  • docs: fix typos in docstrings, comments, and user docs by @mokashang in #4040
  • chore: update doc-builder workflow SHA by @rtrompier in #4009
  • chore: bump doc-builder SHA for main doc build workflow by @rtrompier in #4018
  • [CI] Bump style-bot SHA + switch to GitHub App by @paulinebm in #4031
  • Fix TrackioTracker.log() ignoring step parameter by @joshuaswanson in #3975
  • fix: pass step parameter in TrackioTracker.log() by @liuyun7345 in #3970
  • fix(tracking): default step=None on tracker.log and accept extra kwargs in MLflowTracker by @1fanwang in #4039
  • Fix MLflowTracker.store_init_configuration mutating the caller's config dict by @ATOM00blue in #4046
  • fix(tracker): guard init_trackers and log against None kwargs by @xodn348 in #4026
  • 🔒 Pin GitHub Actions to commit SHAs by @paulinebm in #3992
  • chore: update build-docker-images-release.yml by @hf-security-analysis[bot] in #4069
  • chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in #4049
  • Bump the actions group with 8 updates by @dependabot[bot] in #4068

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.13.0...v1.14.0

View originalPermalink
How v1.14.0 went
v1.13.0

v1.13.0: Neuron support, IPEX removal, and distributed training fixes

Added 2
  • Support for AWS Neuron (Trainium/Inferentia) devices
  • Enable evaluation during deepspeed Sequence Parallel
Changed 5
  • Use spawn instead of fork for XPU device
  • Enhance new codes to XPU and make them device agnostic
  • DeepSpeed has its own process group
  • Accelerate now imports faster by deferring heavy dependencies
  • Torch.compile hooks are disabled lazily
Fixed 11
  • Fix KMP_AFFINITY incorrectly set for non-CPU training
  • Upcast FSDP2 parameters only if requires_grad
  • Fix FSDP2 tied embedding errors with targeted ValueError guidance
  • Fix FSDP2 cannot load optimizer state using dcp
  • Fix crash in optimizer.step when fsdp2 is enabled and model is bfloat16
  • Fix FSDP2 crash with ignored_params on torch < 2.7.0
Removed 1
  • Remove IPEX dependency
Deprecated 1
  • Add MS-AMP deprecation warnings
AWS Neuron support

We now have support for AWS Neuron (Trainium/Inferentia) devices. Thanks @michaelbenayoun for adding this.

XPU Improvements

We've removed IPEX dependency and improved device-agnostic code for XPU.

FSDP2 Improvements

We've added a bunch of important fixes for FSDP2 users: upcasting only grad-requiring params, better tied embedding errors, DCP optimizer loading, bf16 optimizer step crash fix, and torch < 2.7.0 compatibility.

DeepSpeed Sequence Parallelism

We've added several fixes to the DeepSpeed + Sequence Parallelism integration introduced in v1.12.0, including evaluation support during SP training and proper process group handling.

FP8

We've enhanced FP8 training. Thanks @shimizust for fixing torchao support.

Performance

Accelerate now imports faster by deferring heavy dependencies, and torch.compile hooks are disabled lazily.

Minor fixes
View originalPermalink
How v1.13.0 went
v1.12.0

v1.12.0: Deepspeed Ulysses/ALST

Added 1
  • Add Deepspeed Ulysses/ALST integration for efficient training on long sequences through sequence parallelism and attention head parallelism
Changed 1
  • Add device type helper
Fixed 3
  • Update torch.optim.Optimizer parameter states after tensor parallelism
  • Fix typo in broadcast_object_list docstring
  • Update typo in bnb quantisation 4bit flag docstring
Removed 1
  • Remove warning for cpu_ram_efficient_loading
Deepspeed Ulysses/ALST integration

Deepspeed Ulysses/ALST is an efficient way of training on long sequences by employing sequence parallelism and attention head parallelism. You can learn more about this technology in this paper https://arxiv.org/abs/2506.13996 or this deepspeed tutorial https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism/.

To enable Deepspeed Ulysses, you first need to create ParallelismConfig and setting sp related args:

parallelism_config = ParallelismConfig(
    sp_backend="deepspeed",
    sp_size=2,
    sp_handler=DeepSpeedSequenceParallelConfig(...),
)

Then, you need to make sure to compute the correct loss as described on our docs

        ...
        losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)
        good_tokens = (shift_labels != -100).view(-1).sum()
        good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)
        total_loss = sum(
            losses_per_rank[rank] * good_tokens_per_rank[rank]
            for rank in range(sp_world_size)
            if good_tokens_per_rank[rank] > 0
        )
        total_good_tokens = sum(good_tokens_per_rank)
        loss = total_loss / max(total_good_tokens, 1)

Thanks @S1ro1 for starting this work and for @stas00 for finishing this work. Also thanks @kashif for adding docs and reviewing/testing this PR !

This feature will also be available in HF Trainer thanks for this PR from @stas00: https://github.com/huggingface/transformers/pull/41832

Minor changes
New Contributors

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.11.0...v1.12.0

View originalPermalink
How v1.12.0 went
v1.11.0

v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10

Added 6
  • Support for MXFP8 in TransformerEngine integration with use_mxfp8_block_scaling option in fp8_config
  • BF16 and FP16 mixed precision support for MPS devices on Mac
  • Support for ignored_params in FSDPv2
  • Support for no_sync() gradient synchronization control in FSDPv2
  • Allow mixed precision policy as dtype string from accelerate CLI flag or fsdp_config
  • Add to_json method for nd-parallelism
Changed 4
  • Make torch_native_parallelism examples device agnostic
  • Update nd-parallel examples and cleanup
  • Refactor Accelerator.autocast() to use with statement instead of __enter__() and __exit__() methods
  • Drop support for Python 3.9 and bump minimum version to Python 3.10
Fixed 9
  • CPU RAM efficient loading for nd or HSDP parallelisms
  • XPU INT64 all_gather issue
  • Specify device_ids in torch.distributed.barrier for PartialState
  • Specify device for process_tensor in example usage
  • Reduce complexity of get_balanced_memory by adding a set
  • Skip CUDA cache flush when origin device is cpu and offloaded to meta
Removed 1
  • Remove deprecated FindTiedParametersResult
TE MXFP8 support

We've added support for MXFP8 in our TransformerEngine integration. To use that, you need to set use_mxfp8_block_scaling in fp8_config. See nvidia docs [here]. (https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#MXFP8-and-block-scaling)

FP16/BF16 Training for MPS devices

BF16 and FP16 support for MPS devices is finally here. You can now pass mixed_precision = "fp16" or "bf16" when training on a mac (fp16 requires torch 2.8 and bf16 requires torch 2.6)

FSDP updates

The following PRs add respectively support to ignored_params and no_sync() for FSDPv2:

Mixed precision can now be passed as a dtype string from accelerate cli flag or fsdp_config in accelerate config file:

Nd-parallel updates

Some minor updates concerning nd-parallelism.

Bump to Python 3.10

We've dropped support for python 3.9 as it reached EOL in October.

Lots of minor fixes:
New Contributors

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.10.1...v1.11.0

View originalPermalink
How v1.11.0 went
v1.10.0

v1.10.0: N-D Parallelism

Added 3
  • Introduce ParallelismConfig to allow applying any combination of parallelism strategies (TP, CP, DP) directly in training scripts
  • Add Parallelism getter property to Accelerator class
  • Allow FSDP ignored modules to be specified as regex patterns
Changed 5
  • Update context parallel to v2.0
  • Set parallelism_config in Accelerator constructor to prevent reset of State by Trainer
  • Optimize CpuOffload pre_forward to skip moving tensors if already on device
  • Make environment variable values case-insensitive in Accelerate
  • Remove use_ipex
Fixed 4
  • Set default submesh_tp_size to prevent unset local variable error
  • Fix prepare method to work when only tensor parallelism is specified
  • Fix tensor parallelism size to be read from environment variables
  • Fix FSDP ignored modules attribute to enable training PEFT models with MoE layers containing q_proj and v_proj parameters

N-D Parallelism

Training large models across multiple GPUs can be complex, especially when combining different parallelism strategies (e.g TP, CP, DP). To simplify this process, we've collaborated with Axolotl to introduce an easy-to-use integration that allows you to apply any combination of parallelism strategies directly in your training script. Just pass a ParallelismConfig specifying the size of each parallelism type—it's that simple. Learn more about how it works in our latest blogpost.

parallelism_config = ParallelismConfig(
    dp_shard_size=2,
    dp_replicate_size=2,
    cp_size=2,
    tp_size=2,
)
accelerator = Accelerator(
    parallelism_config=parallelism_config,
   ...
)
model = AutoModelForCausalLM.from_pretrained("your-model-name", device_mesh=accelerator.torch_device_mesh)
model = accelerator.prepare(model)

FSDP improvements

We've fixed ignored modules attribute. With this, it is now possible to train PEFT model that moe layers that contrains q_proj and v_proj parameters. This is especially important for fine-tuning gpt-oss model.

Minor improvements

New Contributors

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.9.0...v1.10.0

View originalPermalink
How v1.10.0 went
v1.9.0

v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

Added 2
  • Add support for trackio experiment tracking library with local-first design and Hugging Face Spaces integration
  • Add support for e5e2 and default to hybrid mode when launcher is used
Changed 4
  • Improve model loading speed by 4-5x when using set_module_tensor_to_device with new clear_device option and non_blocking transfers
  • Change find_executable_batch_size() default back-off strategy from halving batch size to multiplying by 0.9 to reduce GPU capacity waste
  • Add warnings for invalid keys
  • Update Gaudi runner image to latest SynapseAI and enable previously disabled tests
Fixed 6
  • Fix FP8 to work without direct Accelerator() configuration
  • Fix proper error handling when using DDP with Dtensor model
  • Fix fsdp2 example typo
  • Add check in no_sync() to avoid errors when using deepspeed zero2/3
  • Fix wandb config not being saved in offline mode
  • Prevent yielding from data loader when base_dataloader is empty
Removed 1
  • Unpin datasets dependency in examples requirements

Trackio tracker support

We've added support for a trackio, lightweight, 💯 free experiment tracking Python library built on top of 🤗 Datasets and Spaces.

Screen Recording 2025-06-11 at 5 39 32 PM

Main features are:

  • Local-first design: dashboard runs locally by default. You can also host it on Spaces by specifying a space_id.
  • Persists logs locally (or in a private Hugging Face Dataset)
  • Visualize experiments with a Gradio dashboard locally (or on Hugging Face Spaces)
  • Everything here, including hosting on Hugging Faces, is free!

To use it with accelerate, you need to set log_with and initialize the trackers

accelerator = Accelerator(log_with="trackio")
config={"learning_rate": 0.001, "batch_size": 32}
# init_kwargs in order to host the dashboard on spaces
init_kwargs = {"trackio": {"space_id": "hf_username/space_name"}
accelerator.init_trackers("example_project", config=config, init_kwargs=init_kwargs})

Thanks @pcuenca for the integration !

Model loading speedup when relying set_module_tensor_to_device

Setting tensor while clearing cache is very slow, so we added clear_device option to disable it. Another small optimization is using non_blocking everywhere and syncing just before returning control to the user. This makes the loading slightly faster.

FDSP, Deepspeed, FP8 minor improvements
🚨🚨🚨 Breaking changes 🚨🚨🚨

find_executable_batch_size() will no longer halves the batch after every OOM. Instead, we will multiply the batch size by 0.9. This should help user not waste gpu capacity.

What's Changed
New Contributors

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.8.1...v1.9.0

View originalPermalink
How v1.9.0 went
v1.8.1

v1.8.1: Patchfix

Added 1
  • Add support for e5e2 and default to hybrid when launcher is used
Changed 1
  • Shards

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.8.0...v1.8.1

View originalPermalink
How v1.8.1 went
v1.8.0

v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

Added 9
  • Added FP8 support for distributed training
  • Added support for regional compilation with the DeepSpeed engine
  • Added SwanLab as an experiment tracking backend
  • Added support for standalone mode when default port is occupied on single node
  • Added kwargs support to optimizer, scheduler, and dataloader in accelerator().load_state() function
  • Added fp8_e5m2 support in dtype_byte_size
Changed 6
  • Simplified FSDPv2 model preparation to reduce composition complexity and improve user experience
  • Updated CCL_WORKER_COUNT variable and added KMP parameters for Intel CPU users to improve distributed training performance
  • Expanded and stabilized support for Intel XPUs including FSDP2, big model inference, and tensor parallelism
  • Deferred all tracker initializations to prevent premature setup of distributed environments
  • Improved FP8 handling with and without DeepSpeed
  • Updated Gaudi runners
Fixed 3
  • Fixed bf16 training with tensor parallelism
  • Fixed list object has no attribute keys error
  • Fixed missing te.LayerNorm in intel_transformer_engine
Removed 1
  • Removed torch_ccl dependency
Deprecated 1
  • ipex.optimize is deprecated in favor of PyTorch optimizations

FSDPv2 refactor + FP8 support

We've simplified how to prepare FSDPv2 models, as there were too many ways to compose FSDP2 with other features (e.g., FP8, torch.compile, activation checkpointing, etc.). Although the setup is now more restrictive, it leads to fewer errors and a more performant user experience. We’ve also added support for FP8. You can read about the results here. Thanks to @S1ro1 for this contribution!

Faster Distributed Training on Intel CPUs

We updated the CCL_WORKER_COUNT variable and added KMP parameters for Intel CPU users. This significantly improves distributed training performance (e.g., Tensor Parallelism), with up to a 40% speed-up on Intel 4th Gen Xeon when training transformer TP models.

Regional Compilation for DeepSpeed

We added support for regional compilation with the DeepSpeed engine. DeepSpeed’s .compile() modifies models in-place using torch.nn.Module.compile(...), rather than the out-of-place torch.compile(...), so we had to account for that. Thanks @IlyasMoutawwakil for this feature!

ipex.optimize deprecation

ipex.optimize is being deprecated. Most optimizations have been upstreamed to PyTorch, and future improvements will land there directly. For users without PyTorch 2.8, we’ll continue to rely on IPEX for now.

Better XPU Support

We've greatly expanded and stabilized support for Intel XPUs:

Trackers

We've added support for SwanLab as an experiment tracking backend. Huge thanks to @ShaohonChen for this contribution ! We also deferred all tracker initializations to prevent premature setup of distributed environments.

What's Changed
New Contributors

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.7.0...v1.8.0

View originalPermalink
How v1.8.0 went
v1.7.0

v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

Added 6
  • Regional compilation targets repeated blocks during model compilation to cache and reuse optimized code, reducing cold start compilation time
  • Layerwise casting hook enables per-layer upcasting and downcasting during inference to run models with separate storage and compute dtypes
  • Support for FULL_STATE_DICT in FSDP2, enabling .save_pretrained() to work with FSDP2 wrapped models
  • QLoRA training support for FSDP2
  • Support for custom function for reducing the batch size
  • HPU support for Intel Gaudi hardware with documentation
Changed 2
  • Updated logic for torch.compile dynamic argument to explicitly preserve None rather than defaulting to False when USE_DYNAMIC environment variable is unset
  • Use device agnostic torch.OutOfMemoryError from PyTorch 2.5.0
Fixed 7
  • Backend issue related to parameter offloading to CPU in FSDP2
  • Memory spike when cpu_ram_efficient_loading=True is enabled
  • Check tied parameters in config for multimodal models
  • FP8 DeepSpeed config
  • Unsafe serialization option in merge-weights command
  • Tensor parallelism training compatibility with new transformers
  • Warning error in accelerate
Removed 1
  • TorchAO sequential offloading no longer creates new parameters due to weak backward compatibility guarantees

Regional compilation

Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start compilation time typically seen during the first inference. Thanks @IlyasMoutawwakil for the feature ! You can view the full benchmark here, and check out our updated compilation guide for more details!

compilation_time-1

To enable this feature, set use_regional_compilation=True in the TorchDynamoPlugin configuration.

# Configure the compilation backend
dynamo_plugin = TorchDynamoPlugin(
    use_regional_compilation=True,
    ... # other parameters
)
# Initialize accelerator with the plugin
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
# This will apply compile_regions to your model
model = accelerator.prepare(model)

Layerwise casting hook

We've introduced a new hook that enables per-layer upcasting and downcasting (e.g., for Linear layers) during inference. This allows users to run models with separate storage and compute dtypes, resulting in memory savings. The concept was first implemented in diffusers, where downcasting models to FP8 proved effective without major quality degradation. Contributed by @sayakpaul in https://github.com/huggingface/accelerate/pull/3427

model = ....
storage_dtype = torch.float8_e4m3fn
compute_dtype = torch.bfloat16
attach_layerwise_casting_hooks(
            model,
            storage_dtype=storage_dtype,
            compute_dtype=compute_dtype,
        )

Better FSDP2 support

This release includes numerous new features and bug fixes. Notably, we’ve added support for FULL_STATE_DICT, a widely used option in FSDP, now enabling .save_pretrained() in transformers to work with FSDP2 wrapped models. QLoRA training is now supported as well but more testing is needed. We have also resolved a backend issue related to parameter offloading to CPU. Additionally, a significant memory spike that occurred when cpu_ram_efficient_loading=True was enabled has been fixed. Several other minor improvements and fixes are also included—see the What’s Changed section for full details.

Better HPU support:

We have added a documentation for Intel Gaudi hardware ! The support is already available since v1.5.0 through this PR.

Torch.compile breaking change for dynamic argument

We've updated the logic for setting self.dynamic to explicitly preserve None rather than defaulting to False when the USE_DYNAMIC environment variable is unset. This change aligns the behavior with the PyTorch documentation for torch.compile. Thanks to @yafshar for contributing this improvement in #3567.

What's Changed
New Contributors

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.6.0...v1.7.0

View originalPermalink
How v1.7.0 went

More in AI

View all
codex

codex release notes.

codex
Ollama

Get up and running with large language models locally.

Ollama
openclaw

Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

openclaw
superpowers

An agentic skills framework & software development methodology that works.

obra
ECC

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

affaan-m
hermes-agent

The agent that grows with you

NousResearch

Discussion