Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation while preventing catastrophic forgetting, but its high runtime and memory demands limit practical deployment. We identify the core computational bottleneck in OFT as its weight-centric implementation, which relies on costly matrix-matrix multiplications with cubic complexity. To overcome this, we propose OFTv2, an input-centric reformulation that instead uses matrix-vector multiplications (i.e., matrix-free computation), reducing the computational cost to quadratic. We further introduce the Cayley{--}Neumann parameterization, an efficient orthogonal parameterization that approximates the matrix inversion in the Cayley transform via a truncated Neumann series. These modifications allow OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage without compromising performance. In addition, we extend OFTv2 to support finetuning quantized foundation models and show that it outperforms the popular QLoRA in training stability, efficiency, and memory usage.
| Author(s): | Qiu*, Z. and Liu*, W. and Weller, A. and Schölkopf, B. |
| Book Title: | Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP) |
| Pages: | 31946--31963 |
| Year: | 2025 |
| Month: | November |
| Editors: | Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet |
| Publisher: | Association for Computational Linguistics |
| BibTeX Type: | Conference Paper (conference) |
| DOI: | 10.18653/v1/2025.emnlp-main.1627 |
| Event Name: | EMNLP 2025 |
| Event Place: | Suzhou, China |
| State: | Published |
| URL: | https://aclanthology.org/2025.emnlp-main.1627/ |
| Note: | *equal contribution |
BibTeX
@conference{QiuLiuWelSch25,
title = {Orthogonal Finetuning Made Scalable},
booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
abstract = {Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation while preventing catastrophic forgetting, but its high runtime and memory demands limit practical deployment. We identify the core computational bottleneck in OFT as its weight-centric implementation, which relies on costly matrix-matrix multiplications with cubic complexity. To overcome this, we propose OFTv2, an input-centric reformulation that instead uses matrix-vector multiplications (i.e., matrix-free computation), reducing the computational cost to quadratic. We further introduce the Cayley{--}Neumann parameterization, an efficient orthogonal parameterization that approximates the matrix inversion in the Cayley transform via a truncated Neumann series. These modifications allow OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage without compromising performance. In addition, we extend OFTv2 to support finetuning quantized foundation models and show that it outperforms the popular QLoRA in training stability, efficiency, and memory usage.},
pages = {31946--31963},
editors = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
publisher = {Association for Computational Linguistics},
month = nov,
year = {2025},
note = {*equal contribution},
author = {Qiu*, Z. and Liu*, W. and Weller, A. and Sch{\"o}lkopf, B.},
doi = {10.18653/v1/2025.emnlp-main.1627},
url = {https://aclanthology.org/2025.emnlp-main.1627/},
month_numeric = {11}
}