Paper link:

https://arxiv.org/pdf/2404.15758

Table of contents:

1. Introduction
2. Related Work
3. Methodology
4. Results
   1. LoRA learns less
   2. LoRA forgets less
   3. LoRA regularization properties
   4. LoRA rank dynamics
5. Discussion
   1. Practical LoRA tips
   2. Future research directions
   3. Questions

1. Introduction

The common assumption is that LoRA maintains accuracy as a fully finetuned model (FFT) while reducing trainable parameters and compute usage on a new target domain. Seems too good to be true?

Few papers have shown a direct model-to-model comparison with billions of parameters between LoRA and FFT.

The current available papers in this field:

• some of them showed mixed results
• other papers were using older models and benchmark (i.e. RoBERTa model and GLUE benchmark) which are not too relevant with contemporary LLMs.
• others also reported that LoRA struggles in more sensitive domain tasks like coding.

The authors hence performed a rigorous analysis on the performance on LoRA finetuning and FFT Llama-2 7B models.