QLoRA

The idea, in one analogy

LoRA already shrinks the trainable part of fine-tuning down to a small adapter. QLoRA asks the next question: what about the frozen part? The base model's weights still have to sit in GPU memory even though they never change during training, so QLoRA compresses them down to 4 bits each, the way you might vacuum-pack clothes you're not using right now but still need on hand. The compressed weights take a quarter the space of a normal 16-bit copy, and only get "unpacked" briefly, on the fly, for each computation.

Why this compounds with LoRA

LoRA's memory savings come from not needing gradients or optimizer state for the frozen weights, only for the small adapter. But the frozen weights themselves still have to be stored somewhere, and at standard 16-bit precision, a 70B-parameter model's weights alone take 140GB, out of reach for a single consumer GPU regardless of how cheap the adapter is. QLoRA quantizes those frozen weights to 4-bit, cutting that storage to roughly 35GB, while keeping the small LoRA adapter (A and B in LoRA's notation) at full precision so training quality doesn't suffer. The two techniques solve different halves of the same memory problem: LoRA shrinks what you train, QLoRA shrinks what you have to store to train it.

Note

What makes QLoRA's quantization specifically work for training: NF4 (4-bit NormalFloat) is a 4-bit datatype QLoRA introduces, designed around the fact that pretrained weights are typically distributed close to a normal (bell-curve) distribution. Instead of spacing the 16 representable values evenly (standard int4 or fp4), NF4 spaces them so that each of the 16 values covers an equal fraction of probability mass under a normal distribution, meaning it needs fewer bits to represent the values that occur often.

Double quantization addresses the fact that quantization itself introduces small per-block scaling constants (a separate small number for every block of weights, used to map the compressed 4-bit values back to their real range) — double quantization quantizes those scaling constants themselves to save additional memory, since a large model has many thousands of them.

Paged optimizers use NVIDIA's unified memory to page optimizer state between GPU and CPU memory automatically during rare memory spikes (e.g. a longer-than-usual sequence in a batch), instead of the training run crashing on an out-of-memory error.

None of these change the LoRA math itself (h = Wx + BAx from LoRA). W is simply stored and dequantized-on-the-fly in this compressed format, while A and B train normally at full precision.

What this buys you, concretely

Dettmers et al.'s original benchmark: fine-tuning a 65B-parameter model, which would need over 780GB of GPU memory for full 16-bit fine-tuning, dropped to fitting on a single 48GB GPU with QLoRA, while matching full 16-bit fine-tuning performance on their benchmarks. That's the headline result: 4-bit QLoRA fine-tuning reaching parity with 16-bit full fine-tuning, not just "close enough."

Where to look further

Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs": the original paper, including the NF4 datatype and double quantization.
bitsandbytes: the library implementing NF4 quantization and paged optimizers, used by Hugging Face's PEFT/Transformers integration.
LoRA: the adapter mechanism QLoRA builds on top of.

QLoRA

The idea, in one analogy#

Why this compounds with LoRA#

What this buys you, concretely#

Where to look further#

The idea, in one analogy

Why this compounds with LoRA

What this buys you, concretely

Where to look further