# PEFT v0.18.0 — 0.18.0: RoAd, ALoRA, Arrow, WaveFT, DeLoRA, OSF, and more - Product: PEFT (https://whatsnew.fyi/product/peft) - Vendor: Hugging Face - Date: 2025-11-13 - Version: v0.18.0 - Original notes: https://github.com/huggingface/peft/releases/tag/v0.18.0 - Permalink: https://whatsnew.fyi/product/peft/releases/v0.18.0 What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog. Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'. --- - **added** — Added RoAd (2D Rotary Adaptation) method that learns 2D rotation matrices applied via element-wise multiplication and supports mixed adapter batches - **added** — Added Activated LoRA (ALoRA) technique for causal language models allowing selective enabling of LoRA adapters based on token invocation sequences - **added** — Added Arrow dynamic routing algorithm for switching between multiple loaded LoRAs - **added** — Added GenKnowSub technique built upon Arrow that modifies the library of LoRAs by subtracting general knowledge adapters - **added** — Added Wavelet Fine-Tuning (WaveFT) method that trains sparse updates in the wavelet domain of residual matrices - **added** — Added DeLoRA (Decoupled Low-rank Adaptation) method that decouples angle and magnitude of learned adapter weights with norm constraints via delora_lambda parameter - **added** — Added OSF (Orthogonal Fine-Tuning) method that freezes high-rank subspace and projects gradient updates to low-rank subspace for continual learning - **added** — Added text generation benchmark framework to determine and compare metrics for different PEFT methods including runtime and memory usage - **added** — Added stable functional interface for integrations with other libraries like Transformers and Diffusers with commitment to backwards compatibility - **added** — Added ensure_weight_tying argument to LoraConfig to enforce weight tying of modules targeted with modules_to_save #### Highlights peft-v0 18 0 FIXME update list of all changes, so some more commits were added ##### New Methods ###### RoAd @ppetrushkov added [RoAd: 2D Rotary Adaptation](https://arxiv.org/pdf/2409.00119) to PEFT in #2678. RoAd learns 2D rotation matrices that are applied using only element-wise multiplication, thus promising very fast inference with adapters in unmerged state. Remarkably, besides LoRA, RoAd is the only PEFT method that supports _mixed adapter batches_. This means that when you have loaded a model with multiple RoAd adapters, you can use all of them for different samples in the same batch, which is much more efficient than switching adapters between batches: ```python model = PeftModel.from_pretrained(base_model, , adapter_name="adapter-A") model.add_adapter("adapter-B", ) inputs = ... # input with 3 samples #### apply adapter A to sample 0, adapter B to sample 1, and use the base model for sample 2: adapter_names = ["adapter-A", "adapter-B", "__base__"] output_mixed = model(**inputs, adapter_names=adapter_names) gen_mixed = model.generate(**inputs, adapter_names=adapter_names) ``` ###### ALoRA Activated LoRA is a technique added by @kgreenewald in #2609 for causal language models, allowing to selectively enable LoRA adapters depending on a specific token invocation sequence in the input. This has the major benefit of being able to re-use most of the KV cache during inference when the adapter is only used to generate part of the response, after which the base model takes over again. ###### Arrow & GenKnowSub @TheTahaaa contributed not only support for [Arrow](https://huggingface.co/papers/2405.11157), a dynamic routing algorithm between multiple loaded LoRAs in #2644, but also [GenKnowSub](https://huggingface.co/papers/2505.10939), a technique built upon Arrow where the 'library' of LoRAs available to Arrow is first modified by subtracting general knowledge adapters (e.g., trained on subsets of Wikipedia) to enhance task-specific performance. ###### WaveFT Thanks to @Bilican, [Wavelet Fine-Tuning](https://arxiv.org/abs/2505.12532) (WaveFT) was added to PEFT in #2560. This method trains sparse updates in the wavelet domain of residual matrices, which is especially parameter efficient. It is very interesting for image generation, as it promises to generate diverse outputs while preserving subject fidelity. ###### DeLoRA [Decoupled Low-rank Adaptation](https://arxiv.org/abs/2503.18225) (DeLoRA) was added by @mwbini in #2780. This new PEFT method is similar to DoRA in so far as it decouples the angle and magnitude of the learned adapter weights. However, DeLoRA implements this in a way that promises to better prevent divergence. Moreover, it constrains the deviation of the learned weight by imposing an upper limit of the norm, which can be adjusted via the `delora_lambda` parameter. ###### OSF [Orthogonal Fine-Tuning](https://huggingface.co/papers/2504.07097) (OSF) was added by @NikhilNayak-debug in #2685. By freezing the high-rank subspace of the targeted weight matrices and projecting gradient updates to a low-rank subspace, OSF achieves good performance on continual learning tasks. While it is a bit memory intensive for standard fine-tuning processes, it is definitely worth checking out on tasks where performance degradation of previously learned tasks is a concern. ##### Enhancements ###### Text generation benchmark In #2525, @ved1beta added the [text generation benchmark](https://github.com/huggingface/peft/tree/main/method_comparison/text_generation_benchmark) to PEFT. This is a framework to determine and compare metrics with regard to text generation of different PEFT methods, e.g. runtime and memory usage. Right now, this benchmark is still lacking experimental se _[Truncated at 4000 characters — full notes: https://github.com/huggingface/peft/releases/tag/v0.18.0]_