Accelerate v1.7.0

v1.7.0

v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

Added 6
  • Regional compilation targets repeated blocks during model compilation to cache and reuse optimized code, reducing cold start compilation time
  • Layerwise casting hook enables per-layer upcasting and downcasting during inference to run models with separate storage and compute dtypes
  • Support for FULL_STATE_DICT in FSDP2, enabling .save_pretrained() to work with FSDP2 wrapped models
  • QLoRA training support for FSDP2
  • Support for custom function for reducing the batch size
  • HPU support for Intel Gaudi hardware with documentation
Changed 2
  • Updated logic for torch.compile dynamic argument to explicitly preserve None rather than defaulting to False when USE_DYNAMIC environment variable is unset
  • Use device agnostic torch.OutOfMemoryError from PyTorch 2.5.0
Fixed 7
  • Backend issue related to parameter offloading to CPU in FSDP2
  • Memory spike when cpu_ram_efficient_loading=True is enabled
  • Check tied parameters in config for multimodal models
  • FP8 DeepSpeed config
  • Unsafe serialization option in merge-weights command
  • Tensor parallelism training compatibility with new transformers
  • Warning error in accelerate
Removed 1
  • TorchAO sequential offloading no longer creates new parameters due to weak backward compatibility guarantees

Regional compilation

Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start compilation time typically seen during the first inference. Thanks @IlyasMoutawwakil for the feature ! You can view the full benchmark here, and check out our updated compilation guide for more details!

compilation_time-1

To enable this feature, set use_regional_compilation=True in the TorchDynamoPlugin configuration.

# Configure the compilation backend
dynamo_plugin = TorchDynamoPlugin(
    use_regional_compilation=True,
    ... # other parameters
)
# Initialize accelerator with the plugin
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
# This will apply compile_regions to your model
model = accelerator.prepare(model)

Layerwise casting hook

We've introduced a new hook that enables per-layer upcasting and downcasting (e.g., for Linear layers) during inference. This allows users to run models with separate storage and compute dtypes, resulting in memory savings. The concept was first implemented in diffusers, where downcasting models to FP8 proved effective without major quality degradation. Contributed by @sayakpaul in https://github.com/huggingface/accelerate/pull/3427

model = ....
storage_dtype = torch.float8_e4m3fn
compute_dtype = torch.bfloat16
attach_layerwise_casting_hooks(
            model,
            storage_dtype=storage_dtype,
            compute_dtype=compute_dtype,
        )

Better FSDP2 support

This release includes numerous new features and bug fixes. Notably, we’ve added support for FULL_STATE_DICT, a widely used option in FSDP, now enabling .save_pretrained() in transformers to work with FSDP2 wrapped models. QLoRA training is now supported as well but more testing is needed. We have also resolved a backend issue related to parameter offloading to CPU. Additionally, a significant memory spike that occurred when cpu_ram_efficient_loading=True was enabled has been fixed. Several other minor improvements and fixes are also included—see the What’s Changed section for full details.

Better HPU support:

We have added a documentation for Intel Gaudi hardware ! The support is already available since v1.5.0 through this PR.

Torch.compile breaking change for dynamic argument

We've updated the logic for setting self.dynamic to explicitly preserve None rather than defaulting to False when the USE_DYNAMIC environment variable is unset. This change aligns the behavior with the PyTorch documentation for torch.compile. Thanks to @yafshar for contributing this improvement in #3567.

What's Changed
New Contributors

Full Changelog: https://github.com/huggingface/accelerate/compare/v1.6.0...v1.7.0

View original

Upgraded? How did it go?

Discussion