# PEFT v0.16.0 — 0.16.0: LoRA-FA, RandLoRA, C³A, and much more

- Product: PEFT (https://whatsnew.fyi/product/peft)
- Vendor: Hugging Face
- Date: 2025-07-03
- Version: v0.16.0
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.16.0
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.16.0

What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog.

Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'.

---

- **added** — Add LoRA-FA optimizer based on AdamW to increase memory efficiency of LoRA training
- **added** — Add RandLoRA method that uses non-learnable random low rank matrices combined through learnable matrices, with support for quantized models with bitsandbytes
- **added** — Add Circular Convolution Adaptation (C3A) as a new PEFT method to overcome low rank adaptation limits
- **added** — Add support for LoRA with Conv2d layers where groups != 1 (requires rank r divisible by groups)
- **added** — Add Intel Neural Compressor (INC) quantization support to LoRA
- **added** — Add DoRA support for Conv1d layers
- **added** — Add orthogonal weight initialization for LoRA via init_lora_weights='orthogonal'
- **added** — Add Quantization-Aware LoRA training with GPTQ support
- **changed** — Refactor Orthogonal Finetuning (OFT) for improved speed and memory efficiency, but this is incompatible with old OFT checkpoints
- **fixed** — Fix LoRA hotswapping with compiled models to no longer cause CUDA graph re-records
- **changed** — Set required_grads_ of modules_to_save to True when used directly with inject_adapter for PEFT integrations
- **changed** — Update compatibility with Transformers vision language model refactor; apply prompt learning methods to vlm directly instead of vlm.language_model
- **changed** — Update prompt learning methods for Transformers attention mask refactoring, which may result in numerical differences for prefix tuning and affect models using 4d attention masks like Gemma

#### Highlights

![peft-v0 16 0](https://github.com/user-attachments/assets/fcced016-7237-410f-b569-53e0f932208d)

##### New Methods

###### LoRA-FA

In #2468, @AaronZLT added the [LoRA-FA optimizer](https://huggingface.co/docs/peft/main/en/developer_guides/lora#lora-fa-optimizer) to PEFT. This optimizer is based on `AdamW` and it increases memory efficiency of LoRA training. This means that you can train LoRA with less memory, or, with the same memory budget, use higher LoRA ranks, potentially getting better results.

###### RandLoRA

Thanks to @PaulAlbert31, a new PEFT method called [`RandLoRA`](https://huggingface.co/docs/peft/main/en/package_reference/randlora) was added to PEFT (#2464). Similarly to VeRA, it uses non-learnable random low rank matrices that are combined through learnable matrices. This way, RandLoRA can approximate full rank updates of the weights. Training models quantized with bitsandbytes is supported.

###### C³A

@Phoveran added [Circular Convolution Adaptation](https://huggingface.co/docs/peft/main/en/package_reference/c3a), C3A, in #2577. This new PEFT method can overcome the limit of low rank adaptations as seen e.g. in LoRA while still promising to be fast and memory efficient.

##### Enhancements

Thanks to @gslama12 and @SP1029, LoRA now supports `Conv2d` layers with `groups != 1`. This requires the rank `r` being divisible by `groups`. See #2403 and #2567 for context.

@dsocek added support for Intel Neural Compressor (INC) quantization to LoRA in #2499.

DoRA now supports `Conv1d` layers thanks to @EskildAndersen (#2531).

Passing `init_lora_weights="orthogonal"` now enables orthogonal weight initialization for LoRA (#2498).

@gapsong brought us Quantization-Aware LoRA training in #2571. This can make QLoRA training more efficient, please check the [included example](https://github.com/huggingface/peft/tree/main/examples/qalora_finetuning). Right now, only GPTQ is supported.

There has been a big refactor of Orthogonal Finetuning, [OFT](https://huggingface.co/docs/peft/package_reference/oft), thanks to @zqiu24 (#2575). This makes the PEFT method run more quickly and require less memory. It is, however, incompatible with old OFT checkpoints. If you have old OFT checkpoints, either pin the PEFT version to `<0.16.0` or retrain it with the new PEFT version.

Thanks to @keepdying, LoRA hotswapping with compiled models no longer leads to CUDA graph re-records (#2611).

#### Changes

##### Compatibility

- #2481: The value of `required_grads_` of `modules_to_save` is now set to `True` when used directly with `inject_adapter`. This is relevant for PEFT integrations, e.g. Transformers or Diffusers.
- Due to a [big refactor of vision language models]( https://github.com/huggingface/transformers/pull/37033) (VLMs) in Transformers, the model architecture has been slightly adjusted. One consequence of this is that if you use a PEFT prompt learning method that is applied to `vlm.language_model`, it will no longer work, please apply it to `vlm` directly (see #2554 for context). Morever, the refactor results in different checkpoints. We managed to ensure _backwards compatability_ in PEFT, i.e. old checkpoints can be loaded successfully. There is, however, no _forward compatibility_, i.e. loading checkpoints trained after the refactor is not possible with package versions from before the refactor. In this case, you need to upgrade PEFT and transformers. More context in #2574.
- #2579: There have been bigger refactors in Transformers concerning attention masks. This required some changes on the PEFT side which can affect prompt learning methods. For prefix tuning specifically, this can result in numerical differences but overall performance should be the same. For other prompt learning methods, numerical values should be the same, except if the base model uses 4d attention masks, like Gemma. If you load old prompt learning checkpoints, please double-check that they still

_[Truncated at 4000 characters — full notes: https://github.com/huggingface/peft/releases/tag/v0.16.0]_