Every QLoRA VRAM Table Leaves Out Sequence Length

Every QLoRA guide gives you a table like this: 8B needs 10 GB, 14B needs 16 GB, 70B fits on a 48 GB card. You rent or buy accordingly, start the run, and it dies with an out-of-memory error on the first step.

The table quoted one number from a bill with four lines and left out the line that grows fastest.

I ran the numbers on five models using the same architecture-aware formulas as our AI VRAM calculator, with parameter counts read from each model's Hugging Face config. For Llama 3.1 8B at rank 16 and a 2,048-token sequence, QLoRA needs 8.59 GiB. Push the sequence to 8,192 tokens and the same job needs 18.49 GiB. The model stayed the same. Sequence length did the damage.

Table of contents

• Four components of QLoRA memory
• LoRA rank barely moves memory
• Activations grow with sequence length
• Maximum sequence length per card
• Llama 3.3 70B on a 48 GB card
• Full fine-tuning compared with QLoRA
• Gradient checkpointing, packing and NF4
• FAQ
• Sources

Four components of QLoRA memory

A QLoRA run is billed for four things, and each one scales on a different axis.

Base weights are fixed. Llama 3.1 8B in NF4 is 4.19 GiB whatever else you change, because the base model is frozen. Adapter state is small, as the next section shows. Activations are the intermediate tensors the backward pass needs, and they scale with batch size times sequence length times layers times hidden size, not with parameter count. Runtime overhead is the CUDA context, allocator fragmentation and kernel scratch. I budget 10% for it, and going lower is a gamble.

For the 8B at 2,048 tokens the split is 48.7% base weights, 7.27% adapters, 34.9% activations, and the rest overhead. Published tables usually quote the first two lines at a short sequence, and readers then apply the figure to a 4k or 8k training set.

LoRA rank barely moves memory

Rank is the first knob most people reach for, and it has about the smallest effect.

At rank 16 targeting all seven projection modules, Llama 3.1 8B gets 41.94 million adapter parameters, which is 0.522% of the base model. They carry the full mixed-precision training cost of 16 bytes per parameter (a bf16 weight, a bf16 gradient, an FP32 master copy and two FP32 AdamW moments). On 42 million parameters that comes to 640 MiB.

Going from rank 8 to rank 32 quadruples trainable capacity (4x) and costs about 1 GiB. Doubling the sequence length from 2,048 to 4,096 costs 3.3 GiB at any rank. If you run out of memory at rank 16, rank 8 will probably not save you.

Activations grow with sequence length

Here is the same 8B model at rank 16, batch size 1 and gradient checkpointing on, with only the sequence length changing.

Past 4,096 tokens, activations are more than half of your VRAM. At 32k they are 82.6% of it, and the 4-bit weights everyone optimizes are a footnote.

The rate differs per model. With gradient checkpointing on, bf16 and batch 1:

Round those to 1.5 GiB per 1,024 tokens for an 8B, 2.3 for a 14B and 7.5 for a 70B. Multiply by your sequence length in thousands of tokens and by your batch size, add the 4-bit weights, add 10%, and you have an estimate that holds up in a real run. Batch size behaves exactly like sequence length here: batch 4 at 2,048 tokens costs the same 12 GiB as batch 1 at 8,192.

Maximum sequence length per card

Turn the question around. Instead of asking how much VRAM a model needs, ask what sequence length your card can train at. These figures are for Llama 3.1 8B, QLoRA, rank 16, batch 1, gradient checkpointing on and 10% runtime overhead.

An 8 GB card cannot train an 8B model at 2,048 tokens. It tops out at 1,664. Most "8B on 8 GB" claims assume 512 tokens, which is a real configuration but a poor fit for instruction data with long answers. A 12 GB card handles 4,096 tokens, enough for most instruction tuning. A 16 GB card reaches 6,528, and 24 GB nearly doubles that.

A few long outliers in your dataset set the memory ceiling, not the average. Check the token-length distribution of your training set before you pick a card. If you need to inspect the JSONL you are about to train on, our JSON formatter will pretty-print it in the browser without uploading it anywhere.

Llama 3.3 70B on a 48 GB card

This claim is repeated everywhere, and it is true only in the narrowest sense.

For Llama 3.3 70B with QLoRA at rank 16, the NF4 base weights are 36.80 GiB. Add 3.09 GiB of adapter state and 10% overhead and you reach 48.00 GiB before a single activation, which happens to be where a 512-token sequence lands.

So a 48 GB card fits 70B QLoRA at 512 tokens with no headroom, and at 1,024 tokens the job no longer fits. An 80 GB H100 tops out at 4,480 tokens. The claim is a specification with the sequence length removed, and with that variable gone you can make almost anything fit almost anything.

Full fine-tuning compared with QLoRA

Parameter count finally dominates in full fine-tuning, because every parameter carries optimizer baggage.

Mixed-precision AdamW holds a bf16 weight, a bf16 gradient, an FP32 master copy and two FP32 optimizer moments, 16 bytes per parameter. On 8.03 billion parameters that is 119.66 GiB of parameter state before any activations.

Full fine-tuning needs roughly 15 to 19 times more memory (about 16x for the 8B models), and the gap grows with model size. It also comes from different places. In full fine-tuning of the 8B, parameter state is 88.7% of the bill and activations are a rounding error. In QLoRA at the same sequence length, parameter state is under 9% and activations are a third. They are different problems under one name, so advice that works for one often fails for the other.

Switching AdamW to 8-bit states drops the cost from 16 bytes per parameter to 10, which takes the 8B parameter state from 119.66 GiB to 74.79 GiB. That still needs several 80 GB cards for a model you can QLoRA on a laptop GPU.

Gradient checkpointing, packing and NF4

Check gradient checkpointing first. It recomputes activations during the backward pass instead of storing them, which roughly halves the activation term for a small time penalty. On the 8B at 8,192 tokens it takes activations from 24.00 GiB to 12.00 GiB and the total from 31.69 GiB to 18.49 GiB. A 24 GB card that can't run the job will run it with one flag flipped.

Sequence packing is next. If your examples average 600 tokens and you set a 4,096-token maximum, you pay for 4,096 on every step and use about a seventh of it. Set the maximum near your real distribution, or pack short examples together so the tokens you pay for carry data.

Two things help less than people expect. One is lowering rank, as the table above shows. The other is swapping NF4 for int4. NF4's extra scaling metadata costs 0.56 bytes per parameter against int4's 0.5, which on 8B is 4.19 GiB versus 3.74 GiB. That saves 450 MiB and gives up the quantization quality NF4 was designed for.

If you are sizing a rental or choosing between a 16 GB and a 24 GB card, enter your real sequence length and batch size into the AI VRAM calculator and read the component breakdown rather than the total.

Training memory and serving memory are separate budgets. Once the adapter is trained and exported to GGUF, running it is a different job with a different tool. The Unsloth vs Ollama comparison covers which tool handles which half.

FAQ

How much VRAM does QLoRA fine-tuning of an 8B model need?

Between 6.12 GiB and 18.49 GiB for Llama 3.1 8B at rank 16 and batch 1, depending on sequence length: 6.12 GiB at 512 tokens, 8.59 GiB at 2,048, 11.89 GiB at 4,096 and 18.49 GiB at 8,192. A figure quoted without a sequence length is incomplete.

Why did my fine-tune run out of memory when the guide said it would fit?

Almost always sequence length or batch size. Guides usually quote short sequences, and activation memory scales linearly with both. For Llama 3.1 8B each 1,024 tokens costs 1.5 GiB with gradient checkpointing on, so a jump from 512 to 4,096 tokens adds 5.25 GiB to a job whose weights never changed.

Is gradient checkpointing worth the slowdown?

Usually, because it halves the term that dominates your bill. On the 8B at 8,192 tokens it takes the total from 31.69 GiB to 18.49 GiB, the difference between needing an 80 GB card and running on a 24 GB one.

Why is NF4 listed at 0.56 bytes per parameter instead of 0.5?

NF4 stores quantization scaling factors alongside the 4-bit weights, so the effective cost is slightly above four bits per parameter. For an 8B model that is 4.19 GiB rather than the 3.74 GiB a pure 4-bit encoding would give.

Sources

• Every VRAM figure in this post was computed locally from the architecture-aware formulas in ToolMintX's AI VRAM calculator, using parameter counts, layer counts, hidden sizes, KV head counts and intermediate sizes taken from each model's Hugging Face config: Llama 3.1 8B Instruct (8,030,261,248 params, 32 layers, hidden 4096), Qwen3 8B (8,190,735,360, 36 layers), Qwen3 14B (14,768,307,200, 40 layers), Qwen3 32B (32,762,123,264, 64 layers) and Llama 3.3 70B Instruct (70,553,706,496, 80 layers).
• Adapter parameter counts follow the standard LoRA placement of rank-r matrices on all seven projections (q, k, v, o, gate, up, down), with the KV projections sized by KV heads rather than attention heads to account for grouped-query attention.
• Mixed-precision AdamW memory accounting of 16 bytes per trainable parameter (bf16 weight, bf16 gradient, FP32 master copy, two FP32 moments) and the 8-bit optimizer variant at 10 bytes per parameter.
• Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" (2023), for the NF4 data type and the double-quantization scaling factors that put NF4's effective cost slightly above four bits per parameter.
• Unsloth documentation, Unsloth requirements and the LoRA hyperparameters guide, for gradient checkpointing behaviour and the recommended rank and learning-rate ranges for QLoRA fine-tuning.