LoRA adapter
A low-rank adaptation (LoRA) adapter contains compact updates to a pretrained model's weight matrices.
LoRA represents each update using two smaller matrices. Training learns those matrices while keeping the original weight matrix fixed 1.
At inference time, the update can be applied alongside the original weights or merged into them. Different adapters let the same base model serve different tasks without storing a separate copy of all its weights for each task 1.
Related
Sources
- Tier A. E. J. Hu et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations (ICLR 2022). Source recordSupports: frozen base weight matrices; trainable low-rank updates; merging or separately applying updates; switching tasks · §4.1–4.2