# PEFT changelog
> Parameter-efficient fine-tuning — LoRA and friends — for large pretrained models.
- Vendor: Hugging Face
- Category: AI
- Official site: https://huggingface.co/docs/peft
- Tracked by: What's New (https://whatsnew.fyi/product/peft)
- Harvested from: GitHub (huggingface/peft)
- Entries below: 10 (newest first)
What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog.
Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'.
## Releases
### v0.20.0
- Date: 2026-07-28
- Version: v0.20.0
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.20.0
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.20.0
- **added** — Add HiRA (Hadamard High-Rank Adaptation) method which multiplies low-rank product with frozen base weight using Hadamard product instead of additive combination
- **added** — Add GLoRA (Generalized LoRA) method extending LoRA with configurable weight, activation, and bias adaptation
- **added** — Add BEFT (Bias-Efficient Fine-Tuning) method that trains only bias terms, targeting the value projection by default for roughly 0.01% trainable parameters
- **added** — Add MonteCLoRA method treating low-rank parameters as distributions and using Monte Carlo estimation for more robust training
- **added** — Add VeLoRA method compressing intermediate activations with rank-1 sub-token projections to reduce activation memory
- **added** — Add Uni-LoRA method using single global projection across the model for cross-layer parameter sharing
- **added** — Add FRoD (Full-Rank Efficient Fine-Tuning with Rotational Degrees) method reconstructing weights with shared rotational subspaces and sparse trainable coefficients
- **added** — Add MiCA method initializing from SVD of base weight and training only on smallest singular value directions
- **added** — Add DEFT (Decompositional Efficient Fine-Tuning) method for text-to-image models splitting weight updates into projection and low-rank components
- **added** — Add automatic LoRA target selection enhancement
#### Highlights
This release adds no less than nine new PEFT methods and puts a lot of work into the surrounding infrastructure, for example adding a new image generation benchmark for the method comparison suite and greatly improving the documentation structure.
##### New Methods
###### HiRA
@hqsiswiliam added ["HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models"](https://openreview.net/forum?id=TwJrTz9cRS) to PEFT (#2668). Instead of adding the low-rank product `BA` to the base weight, HiRA multiplies it elementwise (Hadamard product) with the frozen base weight. Because the base weight itself is full rank, the resulting update is no longer constrained to be low rank, while the trainable parameter count stays the same as LoRA's.
###### GLoRA
@not-lain contributed GLoRA: ["One-for-All: Generalized LoRA for Parameter-Efficient Fine-Tuning"](https://arxiv.org/abs/2306.07967) in #3098. It is a flexible PEFT method that extends LoRA with configurable weight, activation, and bias adaptation, delivering richer fine-tuning with no extra inference cost. Use it when you need per-layer flexibility or stronger adaptation than vanilla LoRA. Skip it for non-Linear layers (e.g. Conv/Embedding) or when standard LoRA is already sufficient and simplicity matters.
###### BEFT
@whubaichuan added ["BEFT: Bias-Efficient Fine-Tuning of Language Models"](https://arxiv.org/abs/2509.15974v2) in #3195. BEFT builds on the observation that fine-tuning bias terms alone can be competitive in low-data regimes, but goes further: rather than training *all* biases, it targets the value projection by default, as the authors found this to be most efficient. This brings the trainable parameter count down to roughly 0.01% of the total parameters.
###### MonteCLoRA
@victor7246 integrated MonteCLoRA, ["Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation"](https://openreview.net/forum?id=2HFmicB8kh) in #2943. LoRA is known to be sensitive to hyperparameters like learning rate and batch size. This new LoRA variant addresses this by treating the low-rank parameters as a distribution rather than a point estimate, using Monte Carlo estimation to obtain a low-variance posterior estimate. This should make training noticeably more robust, at the cost of only _O(rank)_ additional parameters.
###### VeLoRA
@roymiles added VeLoRA: ["Memory Efficient Training using Rank-1 Sub-Token Projections"](https://arxiv.org/abs/2405.17991) in #3159. Unlike most PEFT methods, this LoRA variant targets activation memory rather than parameter count: intermediate activations are split into sub-tokens and compressed with a fixed rank-1 projection before being cached for the backward pass, then reconstructed during backpropagation. Conceptually, it is similar to gradient checkpointing, with a lower memory saving but running faster.
###### Uni-LoRA
@KaiyangLi1992 contributed Uni-LoRA: ["One Vector is All You Need"](https://arxiv.org/abs/2506.00799) in #3257. The paper shows that parameter-efficient LoRA variants like VeRA and VB-LoRA can all be described as projecting the full LoRA parameter space down to a much smaller subspace, differing only in the choice of projection. Uni-LoRA uses a single global projection across the whole model instead of layer-wise ones, which allows cross-layer parameter sharing and thus very low parameter counts.
###### FRoD
@Bane-Elvin added FRoD, ["Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence"](https://arxiv.org/abs/2512.23485), in #3270. Instead of adding low-rank deltas like LoRA, it reconstructs selected weights with shared rotational subspaces and sparse trainable coefficients. It is especially useful when fast convergence and a higher full-rank capacity ceiling are im
_[Truncated at 4000 characters — full notes: https://github.com/huggingface/peft/releases/tag/v0.20.0]_
### v0.19.1
- Date: 2026-04-16
- Version: v0.19.1
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.19.1
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.19.1
A small patch release containing these fixes:
- #3161
- #3165
**Full Changelog**: https://github.com/huggingface/peft/compare/v0.19.0...v0.19.1
### v0.19.0
- Date: 2026-04-14
- Version: v0.19.0
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.19.0
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.19.0
- **added** — GraLoRA method that subdivides base weights into smaller blocks and applies LoRA to them for increased expressiveness and improved performance at higher ranks
- **added** — BD-LoRA method that implements LoRA weights in block-diagonal form to reduce communication overhead when using tensor parallelism
- **added** — Cartridges method to train a prefix that compresses long context to short size and save on tokens
- **added** — PVeRA method as an extension of VeRA that adds probabilistic element by sampling from shared parameters
- **added** — PSOFT method that improves efficiency of orthogonal fine-tuning by constraining adaptation to low-rank principal subspace
- **added** — Lily method with sophisticated parameter sharing scheme where A parameters are shared blockwise and B parameters are chosen in data-dependent way through a router
- **added** — PEANuT method that adds small neural networks called weight-aware neural tweakers to the base model
- **added** — TinyLoRA method for training extremely small number of parameters much lower than LoRA rank 1
- **added** — AdaMSS method that segments base weights into smaller subspaces with dynamic parameter budget assignment to less important subspaces during training
- **added** — Functions to convert checkpoints of many non-LoRA PEFT methods into LoRA checkpoints for compatibility with other packages
#### Highlights
This PEFT release contains no less than nine new PEFT methods, described below. It also contains numerous enhancements that should make PEFT more useful to many users.
##### New Methods
###### GraLoRA
@yeonjoon-jung01 added ["GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning"](https://arxiv.org/abs/2505.20355) to PEFT (#2851). This method subdivides the base weight into smaller blocks and applies LoRA to those. This more granular adaptation promises to increase expressiveness and improve performance, especially at higher ranks (64+), closing the gap to full fine-tuning.
###### BD-LoRA
@Conzel contributed BD-LoRA: ["Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving"](https://openreview.net/forum?id=1cjLvtFOmL) (#2895). With BD-LoRA, the LoRA weights are implemented in a block-diagonal way. This allows to reduce communication overhead when using tensor parallelism (TP) and thus faster serving.
There is an experiment branch for BD-LoRA support in vLLM: vllm-project/vllm#28136.
###### Cartridges
Thanks to @kashif, PEFT now also supports [Cartridges](https://arxiv.org/abs/2506.06266) (#2953). The main purpose of this method is to train a prefix to [compress a long context to a short size](https://hazyresearch.stanford.edu/blog/2025-06-08-cartridges) and thus save on tokens. On a low level, this is similar to [prefix tuning](https://huggingface.co/docs/peft/package_reference/prefix_tuning). The PR also added an [example recipe](https://github.com/huggingface/peft/tree/main/examples/cartridge_self_study) to quickly get started.
###### PVeRA
["PVeRA: Probabilistic Vector-Based Random Matrix Adaptation"](https://arxiv.org/abs/2512.07703) was added to PEFT by @leofillioux in #2952. It is an extension of [VeRA](https://huggingface.co/docs/peft/package_reference/vera), a PEFT method that uses weight sharing between layers to be especially parameter efficient. PVeRA builds on top of that by adding a probabilistic element, sampling from the shared parameters and promising better performance overall.
###### PSOFT
@fei407 added PSOFT, ["Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation"](https://openreview.net/forum?id=FSHrinMArK), to PEFT in #3037. Orthogonal fine-tuning techniques like [OFT](https://huggingface.co/docs/peft/package_reference/oft) and [BOFT](https://huggingface.co/docs/peft/package_reference/boft) are good at preserving the structure and thus capabilities of the underlying base model. PSOFT improves efficiency of this technique by constraining the adaptation to low-rank principal subspace.
###### Lily
@yibozhong added Lily: ["Low-Rank Interconnected Adaptation across Layers"](https://arxiv.org/abs/2407.09946) to PEFT in #2563. Lily is on the surface similar to LoRA but has a sophisticated parameter sharing scheme. The A parameters are shared blockwise (e.g. 4 consecutive q_proj layers share the same A). There is a pool of B parameters that is shared globally, the actual B's are chosen in a data-dependent way through a router. This allows Lily to use higher ranks than LoRA while maintaining a low trainable parameter count.
###### PEANuT
In #3084, ["PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers"](https://arxiv.org/abs/2410.01870) was added to PEFT, again by @yibozhong. PEANuT adds a small, neural net (so called weight-aware neural tweakers) to the base model. Compared to LoRA, this increases expressivity for the same trainable parameter count or allows to greatly lower the parameter count without sacrificing expressivity. This comes at the expensive of a higher memory requirement for the same parameter count and decreased speed.
###### TinyLoRA
We have another serial contributor in @kashif, who also contribu
_[Truncated at 4000 characters — full notes: https://github.com/huggingface/peft/releases/tag/v0.19.0]_
### v0.18.1 — 0.18.1
- Date: 2026-01-09
- Version: v0.18.1
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.18.1
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.18.1
- **fixed** — Small fixes required for some special cases to work with the upcoming transformers v5 release
- **fixed** — Enable PEFT to run with AMD ROCm
- **fixed** — Fix a regression that inadvertently required transformers >= 4.52
Small patch release containing the following changes:
- #2934: Small fixes required for some special cases to work with the upcoming transformers v5 release
- #2963: Fix to enable PEFT to run with AMD ROCm thanks to @vladmandic
- #2976: Fix a regression that inadvertently required transformers >= 4.52
### v0.18.0 — 0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more
- Date: 2025-11-13
- Version: v0.18.0
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.18.0
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.18.0
- **added** — Added RoAd (2D Rotary Adaptation) method that learns 2D rotation matrices applied via element-wise multiplication and supports mixed adapter batches
- **added** — Added Activated LoRA (ALoRA) technique for causal language models allowing selective enabling of LoRA adapters based on token invocation sequences
- **added** — Added Arrow dynamic routing algorithm for switching between multiple loaded LoRAs
- **added** — Added GenKnowSub technique built upon Arrow that modifies the library of LoRAs by subtracting general knowledge adapters
- **added** — Added Wavelet Fine-Tuning (WaveFT) method that trains sparse updates in the wavelet domain of residual matrices
- **added** — Added DeLoRA (Decoupled Low-rank Adaptation) method that decouples angle and magnitude of learned adapter weights with norm constraints via delora_lambda parameter
- **added** — Added OSF (Orthogonal Fine-Tuning) method that freezes high-rank subspace and projects gradient updates to low-rank subspace for continual learning
- **added** — Added text generation benchmark framework to determine and compare metrics for different PEFT methods including runtime and memory usage
- **added** — Added stable functional interface for integrations with other libraries like Transformers and Diffusers with commitment to backwards compatibility
- **added** — Added ensure_weight_tying argument to LoraConfig to enforce weight tying of modules targeted with modules_to_save
#### Highlights
FIXME update list of all changes, so some more commits were added
##### New Methods
###### RoAd
@ppetrushkov added [RoAd: 2D Rotary Adaptation](https://arxiv.org/pdf/2409.00119) to PEFT in #2678. RoAd learns 2D rotation matrices that are applied using only element-wise multiplication, thus promising very fast inference with adapters in unmerged state.
Remarkably, besides LoRA, RoAd is the only PEFT method that supports _mixed adapter batches_. This means that when you have loaded a model with multiple RoAd adapters, you can use all of them for different samples in the same batch, which is much more efficient than switching adapters between batches:
```python
model = PeftModel.from_pretrained(base_model, , adapter_name="adapter-A")
model.add_adapter("adapter-B", )
inputs = ... # input with 3 samples
#### apply adapter A to sample 0, adapter B to sample 1, and use the base model for sample 2:
adapter_names = ["adapter-A", "adapter-B", "__base__"]
output_mixed = model(**inputs, adapter_names=adapter_names)
gen_mixed = model.generate(**inputs, adapter_names=adapter_names)
```
###### ALoRA
Activated LoRA is a technique added by @kgreenewald in #2609 for causal language models, allowing to selectively enable LoRA adapters depending on a specific token invocation sequence in the input. This has the major benefit of being able to re-use most of the KV cache during inference when the adapter is only used to generate part of the response, after which the base model takes over again.
###### Arrow & GenKnowSub
@TheTahaaa contributed not only support for [Arrow](https://huggingface.co/papers/2405.11157), a dynamic routing algorithm between multiple loaded LoRAs in #2644, but also [GenKnowSub](https://huggingface.co/papers/2505.10939), a technique built upon Arrow where the 'library' of LoRAs available to Arrow is first modified by subtracting general knowledge adapters (e.g., trained on subsets of Wikipedia) to enhance task-specific performance.
###### WaveFT
Thanks to @Bilican, [Wavelet Fine-Tuning](https://arxiv.org/abs/2505.12532) (WaveFT) was added to PEFT in #2560. This method trains sparse updates in the wavelet domain of residual matrices, which is especially parameter efficient. It is very interesting for image generation, as it promises to generate diverse outputs while preserving subject fidelity.
###### DeLoRA
[Decoupled Low-rank Adaptation](https://arxiv.org/abs/2503.18225) (DeLoRA) was added by @mwbini in #2780. This new PEFT method is similar to DoRA in so far as it decouples the angle and magnitude of the learned adapter weights. However, DeLoRA implements this in a way that promises to better prevent divergence. Moreover, it constrains the deviation of the learned weight by imposing an upper limit of the norm, which can be adjusted via the `delora_lambda` parameter.
###### OSF
[Orthogonal Fine-Tuning](https://huggingface.co/papers/2504.07097) (OSF) was added by @NikhilNayak-debug in #2685. By freezing the high-rank subspace of the targeted weight matrices and projecting gradient updates to a low-rank subspace, OSF achieves good performance on continual learning tasks. While it is a bit memory intensive for standard fine-tuning processes, it is definitely worth checking out on tasks where performance degradation of previously learned tasks is a concern.
##### Enhancements
###### Text generation benchmark
In #2525, @ved1beta added the [text generation benchmark](https://github.com/huggingface/peft/tree/main/method_comparison/text_generation_benchmark) to PEFT. This is a framework to determine and compare metrics with regard to text generation of different PEFT methods, e.g. runtime and memory usage. Right now, this benchmark is still lacking experimental se
_[Truncated at 4000 characters — full notes: https://github.com/huggingface/peft/releases/tag/v0.18.0]_
### v0.17.1 — 0.17.1
- Date: 2025-08-21
- Version: v0.17.1
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.17.1
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.17.1
- **fixed** — PEFT no longer removes existing parametrizations from the parameter when using the target_parameters feature
- **changed** — PEFT now raises an error when attempting to add multiple adapters to prevent incorrect behavior
This patch release contains a few fixes (via #2710) for the newly introduced [`target_parameters`](https://huggingface.co/docs/peft/main/en/developer_guides/lora#targeting-nnparameter-directly) feature, which allows LoRA to target `nn.Parameter`s directly (useful for mixture of expert layers). Most notably:
- PEFT no longer removes possibly [existing parametrizations](https://docs.pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html) from the parameter.
- Adding multiple adapters (via `model.add_adapter` or `model.load_adapter`) did not work correctly. Since a solution is not trivial, PEFT now raises an error to prevent this situation.
### v0.17.0 — 0.17.0: SHiRA, MiSS, LoRA for MoE, and more
- Date: 2025-08-01
- Version: v0.17.0
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.17.0
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.17.0
- **added** — Add Sparse High Rank Adapters (SHiRA) method for improved performance over LoRAs, especially when using multiple adapters
- **added** — Add MiSS (Matrix Shard Sharing) method as an evolution of Bone with improved performance and memory efficiency
- **added** — Enable LoRA to target nn.Parameter directly using target_parameters config attribute, useful for models with Mixture of Expert layers
- **added** — Support injecting adapters based on a state_dict without needing to specify target_modules in the config
- **fixed** — Fix bug in prompt learning methods where modules_to_save was ignored, preventing classification layers from being trained or stored
- **fixed** — Fix create mask function signature change in transformers 4.53.1
- **fixed** — Fix faulty OFT parameter device test
- **fixed** — Allow peft_type to be a string
- **fixed** — Fix prefix tuning after transformers PR 38635
- **deprecated** — Deprecate Bone method in favor of MiSS, to be removed in PEFT v0.19.0
#### Highlights
##### New Methods
###### SHiRA
@kkb-code contributed [Sparse High Rank Adapters](https://huggingface.co/docs/peft/main/en/package_reference/shira) (SHiRA, [paper](https://huggingface.co/papers/2406.13175)) which promise to offer a potential gain in performance over LoRAs - especially the concept loss when using multiple adapters is improved. Since the adapters only train on 1-2% of the weights and are inherently sparse, switching between adapters may be cheaper than with LoRAs. (#2584)
###### MiSS
@JL-er added a new PEFT method, MiSS ([Matrix Shard Sharing](https://arxiv.org/abs/2409.15371)) in #2604. This method is an evolution of [Bone](https://huggingface.co/docs/peft/package_reference/bone), which, according to our [PEFT method comparison benchmark](https://huggingface.co/spaces/peft-internal-testing/PEFT-method-comparison), gives excellent results when it comes to performance and memory efficiency. If you haven't tried it, you should do so now.
At the same time, Bone will be deprecated in favor of MiSS and will be removed in PEFT v0.19.0. If you already have a Bone checkpoint, you can use [`scripts/convert-bone-to-miss.py`](https://github.com/huggingface/peft/tree/main/scripts/convert-bone-to-miss.py) to convert it into a MiSS checkpoint and proceed with training using MiSS.
##### Enhancements
###### LoRA for `nn.Parameter`
LoRA is now able to target `nn.Parameter` directly (#2638, #2665)! Ever had this complicated `nn.Module` with promising parameters inside but it was too custom to be supported by your favorite fine-tuning library? No worries, now you can target `nn.Parameters` directly using the [`target_parameters`](https://huggingface.co/docs/peft/main/en/developer_guides/lora#targeting-nnparameter-directly) config attribute which works similarly to `target_modules`.
This option can be especially useful for models with **Mixture of Expert** (MoE) layers, as those often use `nn.Parameter`s directly and cannot be targeted with `target_modules`. For example, for the [Llama4 family of models](https://huggingface.co/collections/meta-llama/llama-4-67f0c30d9fe03840bc9d0164), use the following config to target the MoE weights:
```python
config = LoraConfig(
...,
target_modules=[], # <= prevent targeting any modules
target_parameters=["feed_forward.experts.down_proj", "feed_forward.experts.gate_up_proj"],
)
```
Note that this feature is still experimental as it comes with a few caveats and therefore might change in the future. Also, MoE weights with many experts can be quite huge, so expect a higher memory usage than compared to targeting normal `nn.Linear` layers.
###### Injecting adapters based on a `state_dict`
Sometimes, it is possible that there is a PEFT adapter checkpoint but the corresponding PEFT config is not known for whatever reason. To inject the PEFT layers for this checkpoint, you would usually have to reverse-engineer the corresponding PEFT config, most notably the `target_modules` argument, based on the `state_dict` from the checkpoint. This can be cumbersome and error prone. To avoid this, it is also possible to call `inject_adapter_in_model` and pass the loaded `state_dict` as an argument:
```python
from safetensors.torch import load_file
from peft import LoraConfig, inject_adapter_in_model
model = ...
state_dict = load_file()
lora_config = LoraConfig() # <= no need to specify further
model = inject_adapter_in_model(lora_config, model, state_dict=state_dict)
```
Find more on [`state_dict` based injection in the docs](https://huggingface.co/docs/peft/main/en/developer_guides/low_level_api#injection-based-on-a-statedict).
##### Changes
###### Compatibility
A bug in prompt learning methods caused `modules_to_save` to be ignored. Especially clas
_[Truncated at 4000 characters — full notes: https://github.com/huggingface/peft/releases/tag/v0.17.0]_
### v0.16.0 — 0.16.0: LoRA-FA, RandLoRA, C³A, and much more
- Date: 2025-07-03
- Version: v0.16.0
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.16.0
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.16.0
- **added** — Add LoRA-FA optimizer based on AdamW to increase memory efficiency of LoRA training
- **added** — Add RandLoRA method that uses non-learnable random low rank matrices combined through learnable matrices, with support for quantized models with bitsandbytes
- **added** — Add Circular Convolution Adaptation (C3A) as a new PEFT method to overcome low rank adaptation limits
- **added** — Add support for LoRA with Conv2d layers where groups != 1 (requires rank r divisible by groups)
- **added** — Add Intel Neural Compressor (INC) quantization support to LoRA
- **added** — Add DoRA support for Conv1d layers
- **added** — Add orthogonal weight initialization for LoRA via init_lora_weights='orthogonal'
- **added** — Add Quantization-Aware LoRA training with GPTQ support
- **changed** — Refactor Orthogonal Finetuning (OFT) for improved speed and memory efficiency, but this is incompatible with old OFT checkpoints
- **fixed** — Fix LoRA hotswapping with compiled models to no longer cause CUDA graph re-records
- **changed** — Set required_grads_ of modules_to_save to True when used directly with inject_adapter for PEFT integrations
- **changed** — Update compatibility with Transformers vision language model refactor; apply prompt learning methods to vlm directly instead of vlm.language_model
- **changed** — Update prompt learning methods for Transformers attention mask refactoring, which may result in numerical differences for prefix tuning and affect models using 4d attention masks like Gemma
#### Highlights
##### New Methods
###### LoRA-FA
In #2468, @AaronZLT added the [LoRA-FA optimizer](https://huggingface.co/docs/peft/main/en/developer_guides/lora#lora-fa-optimizer) to PEFT. This optimizer is based on `AdamW` and it increases memory efficiency of LoRA training. This means that you can train LoRA with less memory, or, with the same memory budget, use higher LoRA ranks, potentially getting better results.
###### RandLoRA
Thanks to @PaulAlbert31, a new PEFT method called [`RandLoRA`](https://huggingface.co/docs/peft/main/en/package_reference/randlora) was added to PEFT (#2464). Similarly to VeRA, it uses non-learnable random low rank matrices that are combined through learnable matrices. This way, RandLoRA can approximate full rank updates of the weights. Training models quantized with bitsandbytes is supported.
###### C³A
@Phoveran added [Circular Convolution Adaptation](https://huggingface.co/docs/peft/main/en/package_reference/c3a), C3A, in #2577. This new PEFT method can overcome the limit of low rank adaptations as seen e.g. in LoRA while still promising to be fast and memory efficient.
##### Enhancements
Thanks to @gslama12 and @SP1029, LoRA now supports `Conv2d` layers with `groups != 1`. This requires the rank `r` being divisible by `groups`. See #2403 and #2567 for context.
@dsocek added support for Intel Neural Compressor (INC) quantization to LoRA in #2499.
DoRA now supports `Conv1d` layers thanks to @EskildAndersen (#2531).
Passing `init_lora_weights="orthogonal"` now enables orthogonal weight initialization for LoRA (#2498).
@gapsong brought us Quantization-Aware LoRA training in #2571. This can make QLoRA training more efficient, please check the [included example](https://github.com/huggingface/peft/tree/main/examples/qalora_finetuning). Right now, only GPTQ is supported.
There has been a big refactor of Orthogonal Finetuning, [OFT](https://huggingface.co/docs/peft/package_reference/oft), thanks to @zqiu24 (#2575). This makes the PEFT method run more quickly and require less memory. It is, however, incompatible with old OFT checkpoints. If you have old OFT checkpoints, either pin the PEFT version to `<0.16.0` or retrain it with the new PEFT version.
Thanks to @keepdying, LoRA hotswapping with compiled models no longer leads to CUDA graph re-records (#2611).
#### Changes
##### Compatibility
- #2481: The value of `required_grads_` of `modules_to_save` is now set to `True` when used directly with `inject_adapter`. This is relevant for PEFT integrations, e.g. Transformers or Diffusers.
- Due to a [big refactor of vision language models]( https://github.com/huggingface/transformers/pull/37033) (VLMs) in Transformers, the model architecture has been slightly adjusted. One consequence of this is that if you use a PEFT prompt learning method that is applied to `vlm.language_model`, it will no longer work, please apply it to `vlm` directly (see #2554 for context). Morever, the refactor results in different checkpoints. We managed to ensure _backwards compatability_ in PEFT, i.e. old checkpoints can be loaded successfully. There is, however, no _forward compatibility_, i.e. loading checkpoints trained after the refactor is not possible with package versions from before the refactor. In this case, you need to upgrade PEFT and transformers. More context in #2574.
- #2579: There have been bigger refactors in Transformers concerning attention masks. This required some changes on the PEFT side which can affect prompt learning methods. For prefix tuning specifically, this can result in numerical differences but overall performance should be the same. For other prompt learning methods, numerical values should be the same, except if the base model uses 4d attention masks, like Gemma. If you load old prompt learning checkpoints, please double-check that they still
_[Truncated at 4000 characters — full notes: https://github.com/huggingface/peft/releases/tag/v0.16.0]_
### v0.15.2
- Date: 2025-04-15
- Version: v0.15.2
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.15.2
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.15.2
- **fixed** — fix a bug that prevented prompt learning methods like P-tuning from working
This patch fixes a bug that resulted in prompt learning methods like P-tuning not to work (#2477).
### v0.15.1
- Date: 2025-03-27
- Version: v0.15.1
- Original notes: https://github.com/huggingface/peft/releases/tag/v0.15.1
- Permalink: https://whatsnew.fyi/product/peft/releases/v0.15.1
- **fixed** — Fix handling of modules_to_save when used in conjunction with DeepSpeed ZeRO stage 3 to prevent those modules from being placeholder values in saved checkpoints
This patch includes a fix for #2450. In this bug `modules_to_save` was not handled correctly when used in conjunction with DeepSpeed ZeRO stage 3 which resulted in those modules being placeholder values in the saved checkpoints.
**Full Changelog**: https://github.com/huggingface/peft/compare/v0.15.0...v0.15.1