QLoRA: Fine-Tuning Under Memory Constraints

LoRA makes the trainable update small, but the frozen base still takes space. On a constrained device, those unchanged weights can decide whether the training job fits at all. QLoRA addresses that part of the budget by quantizing the base: storing approximations of its weights with less numerical detail. The adapter still learns in higher precision against that approximated base.

Four-bit storage offers sixteen possible codes per stored value — so it cannot preserve every float32 weight exactly. The method trades numerical precision in the frozen model for a smaller representation. The question the chapter answers is what that trade costs — in routing quality, in training time, and in the memory counters we can actually read.

To isolate the change, the rank-eight query/value adapter, the 150 requests, the two epochs and the evaluation interface all stay as they were in Chapter 8. Only the base representation changes.

Run against: bitsandbytes 0.50.2, PEFT 0.17.1, Transformers 4.56.2, PyTorch 2.8.0; pinned Qwen3-0.6B, NF4 with double quantization, float32 compute and adapters, CPU quantization followed by Apple M1 Ultra/MPS training, eager execution, dataset router-v2. The printed progress lines and result fields are from the recorded QLoRA run’s log and results.json.

Not run. CUDA kernels, bfloat16 or float16 compute, paged optimizers, quantized adapter merging, a separate unadapted four-bit test arm, and a true MPS accelerator-memory peak are outside this experiment.

Separate representation from arithmetic

A floating-point number stores a sign, an exponent and a run of significant digits, which together describe its value. A four-bit quantization code instead selects a value from a small set, called a codebook. A scale adjusts the range those values cover. Many nearby original weights select the same code, so reconstruction from the code and scale gives an approximation. The stored code identifies a choice; it is not the original weight.

The storage format does not mean every calculation uses four-bit arithmetic. A linear operation can reconstruct blocks into a higher-precision compute type, apply them to the activations, and combine the result with the adapter branch. Which operations are fused and which use temporary dequantized tensors is the implementation’s business. Our configuration uses float32 compute — a compatibility choice that keeps the comparison on the same footing as the float32 full and LoRA arms. A CUDA recipe with bfloat16 compute has different memory and performance behavior and should not inherit these timings just because both are called QLoRA.

The base stays frozen. The optimizer changes the adapter parameters, not the four-bit codes, and gradients still flow through the computation to the adapter. That is how the method avoids trying to optimize packed discrete codes with an ordinary floating-point update. A useful conceptual expression is:

output = dequantize(W_4bit) × input
         + (alpha / rank) × B × A × input

The first contribution comes from the frozen base after its stored codes are reconstructed into approximate weights. The second is the familiar trainable adapter branch. Their outputs are added, just as in ordinary LoRA. The expression does not require the backend to reconstruct the whole model at once: implementations can work blockwise or use specialized low-level routines. Memory therefore has to be measured at the implementation rather than read off the formula.

See how a scale can hide small values

Quantization error is easier to follow with a deliberately tiny codebook. Suppose a two-bit scheme uses four normalized levels, -1, -1/3, 1/3 and 1. A block is a group of weights that share a scale; here, that scale is the largest absolute weight in the group. Divide each weight by the scale to normalize it, then choose the nearest level. Multiplying that level by the scale reconstructs the approximate weight. This is a teaching example, not NF4.

With scale one, a weight of 0.1 reconstructs as about 0.3333, an absolute error of about 0.2333. Now put a weight of 8 in the same block. The scale becomes eight, and the small weight reconstructs as about 2.6667. The outlier has stretched the range the tiny codebook must cover — the error on an unrelated value grew more than tenfold.

Figure 09.1. Both lines use the same two-bit codebook and the same weight; only the block scale differs, because one weight of 8 has joined the block. Amber marks the original weight and teal its reconstruction. The gap between them is the quantization error the chapter calculates.

That is why block size, scale storage and the distribution of values matter, and why bit width alone does not describe an approximation’s quality. Smaller blocks keep an outlier from setting the scale for as many unrelated weights, at the cost of more scaling metadata. Larger blocks amortize the metadata and share a range more broadly. Real NF4 uses sixteen nonuniform levels and a more careful representation than this toy, but the mechanism is the same.

Understand what NF4 chooses

NormalFloat 4, or NF4, chooses its quantization levels around a normal-distribution assumption for weights — a bell-shaped distribution. Its levels are not uniformly spaced, so they allocate representational capacity differently from uniformly spaced integer levels. As in the toy example, blocks carry scaling information so the codes can represent weights of different magnitudes. Two four-bit formats can therefore produce different approximations, need different metadata and use different low-level routines. Recording only the bit width leaves out how the approximation was made.

Double quantization compresses the quantization constants themselves. Scales are values that take storage too, and representing them more compactly cuts overhead beyond the packed codes. It reduces metadata cost without restoring precision lost in the first quantization step. The QLoRA paper develops NF4, double quantization and paged optimizers together. Our run uses the first two with ordinary PyTorch AdamW for the small adapter, so calling it a QLoRA-style four-bit adapter experiment names its central mechanism without claiming every part of the paper’s large-model system is present.

For an idealized array of N values, four-bit codes occupy N / 2 bytes. Real storage adds scales, packing details, modules left unquantized and the adapter’s own parameters. Half a byte per base parameter is a floor, not the memory a training job needs.

Quantize the checkpoint before the adapter learns

The runner loads the pinned checkpoint with a bitsandbytes configuration, then moves the quantized model to MPS. The essential configuration is:

quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float32,
)

The loading path keeps the model on the CPU while it is quantized and then calls model.to('mps'). That is a practical detail of the recorded backend, not a requirement of QLoRA. The initial direct-to-MPS loading probes did not complete within the diagnostic window: one kept attempting compiler paths before falling back, and disabling compilation removed those attempts without resolving the stall. Quantizing on CPU and moving the result completed the forward, backward and evaluation probe. Those observations establish the path used here. They do not establish an upstream defect, and the cause could lie in tensor size, package versions or this environment.

Optional torch.compile is disabled before the training stack is imported, for every comparison arm. That keeps the four-bit fallback from repeatedly attempting compilation under the pinned PyTorch, and it means the results describe eager execution rather than the best attainable optimized backend. The bitsandbytes installation documentation lists Apple Silicon support for this release. The working probe was still necessary, because platform support does not say whether a particular model, precision and backward path works in a given environment.

Prepare the base without accidentally training it

prepare_model_for_kbit_training() sets up the frozen quantized base before the adapter is attached. The companion disables gradient checkpointing — a memory-saving option that recomputes activations during backward instead of storing them — in that call, because this short-sequence experiment does not need it. PEFT then creates the same query/value adapter as the ordinary LoRA arm:

model = prepare_model_for_kbit_training(
    model,
    use_gradient_checkpointing=False,
)
model = get_peft_model(model, config)

Here config is the rank-eight, alpha-sixteen configuration from Chapter 8. The order matters: preparing the base first and adding the trainable parameters second makes the intended parameter set explicit. The trainable count is 1,146,880 again — the frozen representation changed, the adapter shapes did not.

The loaded model also reports a different total. Summing numel() over its tensors, adapter included, gives 376,995,840, where the float32 LoRA model gave 597,196,800 for the same base plus the same adapter. Nothing has been removed. Packed storage puts more than one four-bit code inside each integer element, so the storage-element count and the logical weight count are different quantities, and the architecture is the same.

The watched base query weight stays unchanged through optimization while the adapter factors change, which checks the intended separation. The frozen base can still differ from the float32 model, because it was approximated during loading, before the first optimizer update. That starting-point difference is visible in the first printed loss below.

Train against the four-bit base

The command differs from Chapter 8’s by the mode name, which selects the quantized-base preparation just described. It still trains an adapter and needs a new output directory. You can also read the recorded run below to compare its losses and results with the ordinary adapter:

.venv/bin/python train_router.py \
  --mode qlora --size 150 --epochs 2 \
  --output runs/local-qlora \
  --evaluate test --reload

The recorded run printed:

qlora 150 {'step': 1, 'epoch': 0, 'loss': 0.728108286857605} elapsed 0.94
qlora 150 {'step': 10, 'epoch': 0, 'loss': 0.6342746615409851} elapsed 6.77
qlora 150 {'step': 20, 'epoch': 0, 'loss': 0.020670821890234947} elapsed 13.25
qlora 150 {'step': 30, 'epoch': 0, 'loss': 0.044854044914245605} elapsed 19.74
qlora 150 {'step': 40, 'epoch': 1, 'loss': 0.38943010568618774} elapsed 26.09
qlora 150 {'step': 50, 'epoch': 1, 'loss': 0.39867234230041504} elapsed 32.61
qlora 150 {'step': 60, 'epoch': 1, 'loss': 0.0015424429439008236} elapsed 39.1
qlora 150 {'step': 70, 'epoch': 1, 'loss': 0.0015169470570981503} elapsed 45.58

The first line shows the starting task loss with the quantized setup. Chapter 8’s adapter started at 0.7232056856155396 on this batch, identical to the full fine-tune because a zero-initialized adapter leaves the base untouched. This run starts at 0.7281 on the same batch with the same adapter initialization and the same seed. The loss is about 0.005 higher before training has applied an update. That difference describes the desired tokens’ average negative log-probability on this batch; it does not measure the error between original and reconstructed weights. The adapter then learns against the approximated base. Its trajectory is noisier than LoRA’s, with step 50 still at 0.40, and its last recorded value in loss.jsonl is 0.00074.

Each step also took longer — about 0.65 seconds against LoRA’s 0.52. Reconstructing weights adds work in this backend and may contribute to that difference, but these whole-step timings do not isolate its cost. Profiling the operations would be needed to attribute the slowdown. The run finishes by evaluating, reloading the adapter onto a freshly quantized base, and printing:

"optimizer_steps": 76
"training_seconds": 49.245049957884476
"accuracy": 1.0
"macro_f1": 1.0
"reload_predictions_equal": true

Compare the actual memory counters

Measurement	LoRA	QLoRA
Accuracy	100%	100%
Macro-F1	1.000	1.000
Training seconds	40.28	49.25
Parameter elements reported by the loaded model, adapter included	597,196,800	376,995,840
MPS allocated MiB after training	2284.82	838.40
MPS driver MiB after training	9855.78	9103.78
Process peak RSS MiB	5212.05	3837.44

Both adapters reach fifty correct labels with no invalid output, and QLoRA’s reloaded artifact reproduces them. The test finds no quality difference; its ten template families keep that from being a claim of general equivalence. What four-bit storage bought is in the allocation row. The memory retained on the accelerator after training fell from about 2,285 MiB to 838 MiB, in exchange for 49.25 seconds of training against 40.28. Smaller frozen storage came with a slower update loop in this environment — a real tradeoff, not a law.

The three memory rows measure three different things — the table is only useful if they are read separately. As in Chapter 8, the allocation snapshot is taken after optimizer deletion and saving. It can miss peaks during loading, quantization, backward, saving or evaluation. The driver counter covers a broader scope and can retain cached resources. Process peak RSS is the operating system’s view of the whole process lifecycle. None of them is the true accelerator peak, which this harness could not read. The lower allocation says what was retained — not that the highest demand fell by the same amount.

Two further reasons the saving is smaller than an eightfold reduction of an all-float32 model. Quantization targets the supported linear layers and leaves embeddings, normalization values and the float32 adapter in higher precision. And the installed MPS backend includes a fallback that reconstructs a weight for a linear operation when a specialized path does not apply. Compact storage does not mean every operation ran on compact operands.

Ordinary LoRA already fits comfortably on this machine, so here QLoRA is a tradeoff to judge rather than a requirement to run the job. A larger checkpoint or a smaller device changes that. Before extending the result to either, memory that scales with weights has to be told apart from memory that scales with activations.

Treat quality as a measured cost

Quantization perturbs the frozen model, and the adapter learns its correction with limited capacity and limited examples, so the result need not match ordinary LoRA on every decision boundary. The evaluation therefore keeps the same strict parser and no-demonstration prompt, and it keeps the raw responses, so an invalid output can be told from a valid but wrong destination. When two candidates tie or nearly tie, read the per-class recalls and the paired disagreements on the same requests: those say which routes changed, which a headline score cannot. Repeated seeds would show whether the ranking is stable, and changed test families whether it depends on a wording pattern; this experiment used one seed and one set of families.

Keep the quantization recipe with the adapter

The QLoRA artifact is still an adapter, not a self-contained four-bit base, so reconstructing the candidate needs the pinned checkpoint and the same quantization settings. The runner’s reload path repeats the CPU quantization, moves the result to MPS, attaches the saved adapter and compares labels with the in-memory model. That verifies the reconstruction this chapter relies on. Loading the same adapter onto a float32 base produces a related but numerically different candidate. Adapters trained against different quantization formats are not interchangeable even when the shapes let them load, so format, scale configuration, compute dtype, base revision and runtime all belong in the deployment manifest.

Merging needs separate care. Adding the update into dequantized weights and saving a floating-point model changes storage and execution, and requantizing the merged result adds another approximation. A quantized merge therefore needs its own quality and artifact checks — the ordinary LoRA merge result does not cover it. The routing policy and decoder belong in the contract too. A model trained to return one destination under a non-thinking prefix can behave differently if a serving wrapper injects another system instruction or enables a different chat mode, and quantization does not insulate it from those changes.

Decide whether the saved memory is useful

Ask what constraint the extra complexity solves. If ordinary LoRA fits, trains faster and gives acceptable quality, four-bit storage does little for this small router. If the quantized base is the only version that fits, the tradeoff is compelling. The saving has to matter to the job you are trying to run.

The bottleneck should pick the next experiment, so a useful planning sheet lists the memory categories separately, with what drives each and what reduces it:

Memory category	What drives it	What reduces it
Frozen weights	Model size and storage precision	Quantizing the base, as QLoRA does
Trainable weights, their gradients and optimizer state	The number of trainable parameters	Training an adapter, as LoRA does
Activations kept for backward	Batch size and sequence length	A smaller batch, shorter inputs, or gradient checkpointing
Temporary workspace	The backend’s implementation and fallbacks	Backend-specific changes, guided by profiling

Cutting frozen-weight storage may let a larger checkpoint load while a long batch still exhausts memory during backward, and reducing the batch may let training run without changing the serving footprint at all. The point where the process fails names the row to investigate; combining every mechanism at once makes that diagnosis harder.

Two practical consequences follow. CPU quantization needs host memory while the checkpoint loads and converts, so a compact final representation does not imply a small startup requirement. This machine had enough; a smaller host might need a prequantized checkpoint or a different loading strategy. And the saved adapter directory does not contain the quantized base, so its size is not the download budget for deployment. The recipient obtains the pinned checkpoint, reconstructs the quantized representation and loads the adapter, and a package that caches the quantized base should record that choice. Memory savings are most useful tied to an actual constraint — a larger batch, a larger model, less per-process allocation, a shared device. Without one, a smaller counter becomes a goal of its own while quality and throughput slip.

Exercises

1. Separate storage and compute

A configuration stores the base in NF4 and sets compute dtype to float32. Are all matrix calculations four-bit operations?

Answer

No. Four-bit codes describe storage. The backend reconstructs values or uses specialized low-level routines to perform the linear operation with the declared compute behavior. The adapter remains higher precision, and temporary representations depend on the implementation.

2. Interpret a parameter count

Summing tensor numel() after quantization gives a smaller number than the logical base parameter count. Has the architecture lost layers?

Answer

Not necessarily. Packed tensors can store more than one four-bit code per integer element. Storage element count and logical weight count are different quantities. Architecture inspection and quantization metadata determine what the representation contains.

3. Read a memory claim

After-training MPS allocation falls by one gigabyte. Can the report say peak accelerator memory fell by one gigabyte?

Answer

No. A snapshot does not measure the maximum over the run. Temporary allocations may have peaked earlier. The report should name the actual snapshot and keep the unmeasured true peak separate.

4. Reconstruct the candidate

A QLoRA adapter is loaded onto the correct base revision in float32. Why is that not an exact reproduction of this experiment?

Answer

The adapter learned against an NF4 base with double quantization. A float32 base changes the numerical transformation beneath it. Reproduction also needs the recorded quantization configuration, compute dtype, tokenizer, policy, and decoder.

Final thoughts

Four-bit storage can make an experiment possible when the original base will not fit. Once it fits, the job is the same as before — judge the quality and the runtime. Keeping the memory counters, the errors and the reconstruction path explicit is what lets a quantized candidate be compared with ordinary LoRA honestly. The first printed loss is a reminder that the approximation is already in the model before training begins.

Next: How Much Training Data Do You Need?