# Accelerate v1.7.0 — v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

- Product: Accelerate (https://whatsnew.fyi/product/accelerate)
- Vendor: Hugging Face
- Date: 2025-05-15
- Version: v1.7.0
- Original notes: https://github.com/huggingface/accelerate/releases/tag/v1.7.0
- Permalink: https://whatsnew.fyi/product/accelerate/releases/v1.7.0

What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog.

Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'.

---

- **added** — Regional compilation targets repeated blocks during model compilation to cache and reuse optimized code, reducing cold start compilation time
- **added** — Layerwise casting hook enables per-layer upcasting and downcasting during inference to run models with separate storage and compute dtypes
- **added** — Support for FULL_STATE_DICT in FSDP2, enabling .save_pretrained() to work with FSDP2 wrapped models
- **added** — QLoRA training support for FSDP2
- **added** — Support for custom function for reducing the batch size
- **added** — HPU support for Intel Gaudi hardware with documentation
- **changed** — Updated logic for torch.compile dynamic argument to explicitly preserve None rather than defaulting to False when USE_DYNAMIC environment variable is unset
- **changed** — Use device agnostic torch.OutOfMemoryError from PyTorch 2.5.0
- **fixed** — Backend issue related to parameter offloading to CPU in FSDP2
- **fixed** — Memory spike when cpu_ram_efficient_loading=True is enabled
- **fixed** — Check tied parameters in config for multimodal models
- **fixed** — FP8 DeepSpeed config
- **fixed** — Unsafe serialization option in merge-weights command
- **fixed** — Tensor parallelism training compatibility with new transformers
- **fixed** — Warning error in accelerate
- **removed** — TorchAO sequential offloading no longer creates new parameters due to weak backward compatibility guarantees

#### Regional compilation

Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start compilation time typically seen during the first inference. Thanks @IlyasMoutawwakil  for the feature ! You can view the full benchmark [here](https://github.com/huggingface/accelerate/tree/main/benchmarks/torch.compile), and check out our updated [compilation guide](https://huggingface.co/docs/accelerate/en/usage_guides/compilation) for more details!

![compilation_time-1](https://github.com/user-attachments/assets/38795d12-6ee7-4a10-84c6-d29a0877e36c)

To enable this feature, set `use_regional_compilation=True` in the `TorchDynamoPlugin` configuration.

```python
#### Configure the compilation backend
dynamo_plugin = TorchDynamoPlugin(
    use_regional_compilation=True,
    ... # other parameters
)
#### Initialize accelerator with the plugin
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
#### This will apply compile_regions to your model
model = accelerator.prepare(model)
```

#### Layerwise casting hook

We've introduced a new hook that enables per-layer upcasting and downcasting (e.g., for Linear layers) during inference. This allows users to run models with separate storage and compute dtypes, resulting in memory savings. The concept was first implemented in [diffusers](https://huggingface.co/docs/diffusers/main/en/optimization/memory#layerwise-casting), where downcasting models to FP8 proved effective without major quality degradation. Contributed by @sayakpaul in https://github.com/huggingface/accelerate/pull/3427

```python
model = ....
storage_dtype = torch.float8_e4m3fn
compute_dtype = torch.bfloat16
attach_layerwise_casting_hooks(
            model,
            storage_dtype=storage_dtype,
            compute_dtype=compute_dtype,
        )
```


#### Better FSDP2 support 

This release includes numerous new features and bug fixes. Notably, we’ve added support for `FULL_STATE_DICT`, a widely used option in FSDP, now enabling `.save_pretrained()` in transformers to work with FSDP2 wrapped models. QLoRA training is now supported as well but more testing is needed. We have also resolved a backend issue related to parameter offloading to CPU. Additionally, a significant memory spike that occurred when `cpu_ram_efficient_loading=True` was enabled has been fixed. Several other minor improvements and fixes are also included—see the **What’s Changed** section for full details.

- `FULL_STATE_DICT` have been enabled by @S1ro1 in https://github.com/huggingface/accelerate/pull/3527
-  QLoRA support by @winglian in https://github.com/huggingface/accelerate/pull/3546
-  set backend correctly for CUDA+FSDP2+cpu-offload in https://github.com/huggingface/accelerate/pull/3574
-  memory spike fixed when using `cpu_ram_efficient_loading=True` by @S1ro1 in https://github.com/huggingface/accelerate/pull/3482

#### Better HPU support:

We have added a [documentation](https://huggingface.co/docs/accelerate/en/usage_guides/gaudi) for Intel Gaudi hardware ! 
The support is already available since v1.5.0 through this [PR](https://github.com/huggingface/accelerate/pull/3378). 

- Add the HPU into accelerate config by @yuanwu2017 in https://github.com/huggingface/accelerate/pull/3495
- Add Gaudi doc by @regisss in https://github.com/huggingface/accelerate/pull/3537

#### Torch.compile breaking change for `dynamic` argument

We've updated the logic for setting `self.dynamic` to explicitly preserve None rather than defaulting to `False` when the `USE_DYNAMIC` environment variable is unset. This change aligns the behavior with the PyTorch documentation for [torch.compile](https://docs.pytorch.org/stable/generated/torch.compile.html). Thanks to @yafshar for contributing this improvement in [#3567](https://github.com/hugg

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/accelerate/releases/tag/v1.7.0]_