Session 15

How do you teach a big model
your company’s rules?

Today: fine-tuning, PEFT, LoRA, quantization, and QLoRA. One story. Lots of pictures. Hit Play on each demo.

The problem

ChatGPT is smart.
It still does not know your return policy.

Generic chatbot
Your company bot

Customer: “I bought shoes 20 days ago. Can I return them?”

Your real policy is 14 days. A generic model has never seen that document. Next: RAG can inject that document. Fine-tuning teaches how to talk.
Ready
First fork in the road

You can fix “missing facts” with RAG too.
Fine-tuning is for a different job.

Same return-policy question. Two different tools.

RAG

At answer time, search your PDF/wiki, paste the matching text into the prompt, ask the frozen model.

Changes: what the model sees right now (fresh facts).
Does not change: the model’s weights.

Fine-tuning

Train on your past chats so the model picks up your tone, format, and habits.

Changes: how the model behaves.
Weak at: facts that change every week (you would retrain).

Return policy text that changes often → start with RAG. Brand voice and short WhatsApp-style replies that stay the same → fine-tune. Real products often use both: RAG for today’s facts, fine-tune for manners.
When to use which

Decision table you can teach from memory.

If you need…Go withWhy
Up-to-date facts (menu, policy, prices)RAGUpdate the document. No retrain.
Citations / “show me the source”RAGYou can point at the retrieved chunk.
Brand tone, empathy, reply formatFine-tuneBehavior lives in the weights.
Domain slang / how your team talksFine-tuneHard to enforce with retrieval alone.
Fresh facts + your voiceBothFine-tune the style. RAG the facts.
This class is about the fine-tuning path (PEFT → LoRA → QLoRA). You already know RAG from earlier modules. Do not treat them as enemies.
Fine-tuning

Fine-tuning means: keep training the model on your examples.

You show it hundreds of real support chats. It picks up your tone, your short answers, and stable habits. For changing policy text, still prefer RAG.

1
Collect your chats
2
Train a bit more
3
Bot answers like your team
The catch

Updating every weight in a 7B model
needs about 112 GB of GPU memory.

112 GB
Rough cost of full fine-tuning
2
bytes weight
2
bytes gradient
~12
bytes optimizer
7 billion × ~16 bytes ≈ 112 GB. A free Colab GPU has about 15 GB. Full fine-tuning is the right idea with the wrong price.
PEFT

Don’t retrain every number.
Freeze the model. Train a tiny add-on.

PEFT = Parameter-Efficient Fine-Tuning. Lock the big brain. Only train small adapters.

Those orange chips are small trainable matrices (adapters), not magic “tone buttons.” LoRA is the PEFT method that builds those adapters as B and A.
Ready
Hugging Face PEFT docs
PEFT · worked example

Count the numbers. That is all PEFT is.

Imagine one tiny layer with a 4×4 weight matrix. Full fine-tuning touches every cell. PEFT freezes those 16 cells and trains only a small adapter.

ApproachWhat you trainCount
Full fine-tuneAll of W (4×4)16
PEFT (tiny adapter)Adapter only (example size 4)4
SavingsFrozen W never gets gradients75% fewer trainable params
Same idea at real scale: a 7B model has 7,000,000,000 weights. PEFT might train ~20–50 million (under 1%). Formula: trainable_PEFT ≪ N_full. LoRA is one way to build that small trainable set.
What training changes

Training changes the weights.
That change is called ΔW.

Before training the weights are W₀. After training they are W. Subtract them: ΔW = W − W₀. Full fine-tuning tries to learn that entire change sheet.

1.2
0.1
0.0
0.9
W · after
−
1.0
0.0
0.0
1.0
W₀ · before
=
?
?
?
?
ΔW · the change
Ready
LoRA

LoRA idea: don’t store the whole ΔW.
Store two thin pieces that multiply into it.

LoRA = Low-Rank Adaptation. Freeze W₀. Learn B and A only. The update is B × A.

Rule: each cell of ΔW is one number from B times one number from A.

ΔW[i, j] = B[i] × A[j]

1
0
0
1
W₀ frozen
+
2
0.5
B · 2×1
×
0.5
1
A · 1×2
=
(0,0)
(0,1)
(1,0)
(1,1)
ΔW
→
?
?
?
?
W₀ + ΔW
Ready
Toy is break-even (4 numbers in B+A vs 4 in ΔW). Real win: 512×512 layer with r=8 stores 8,192 instead of 262,144. W₀ stays locked; only B and A train.
LoRA · full worked example

Follow one input through the math.

Frozen W₀, LoRA update BA, input x. You can check every multiply by hand.

Step 1 · frozen base
W₀ = [[1, 0], [0, 1]]
Step 2 · LoRA pieces
B = [[2], [0.5]]   A = [[0.5, 1]]
Step 3 · build ΔW = B×A
ΔW = [[1, 2], [0.25, 0.5]]
Step 4 · effective weight
W = W₀ + ΔW = [[2, 2], [0.25, 1.5]]

Input x = [2, 3]. Hit Play.

?
W₀ · x (no LoRA)
?
W · x (with LoRA)
Without LoRA: W₀x = [2, 3]
With LoRA: Wx = [2·2 + 2·3, 0.25·2 + 1.5·3] = [10, 5]
Ready
General formula (paper): h = W₀x + (α/r)·BAx. We only train B and A. W₀ stays locked. If α = r, the scale is 1 (as in this toy).
Why this saves money

Same trick on a real layer size.

Now d is big and r stays small. For a 512×512 weight matrix with rank r = 8:

What we storeShapeNumbers
Full ΔW512 × 512262,144
LoRA B512 × 84,096
LoRA A8 × 5124,096
LoRA totalB + A8,192 (about 3%)
Formula to remember: trainable ≈ 2 × d × r, not d × d. Bigger layers, bigger savings. Paper: Hu et al. 2021.
LoRA paper (arXiv:2106.09685)
Still stuck

LoRA made training cheap.
The frozen base is still huge.

A 7B model in FP16 is about 14 GB just for weights. A free Colab T4 has about 15 GB total. You are one sneeze away from out-of-memory.

14
GB base
vs
15
GB free GPU
Next move: shrink each number in the base model. That is quantization.
Quantization · the point

We are not teaching the model new facts here.
We are storing the same weights with fewer bits.

Goal: fit a big frozen base on a small GPU so QLoRA training is possible.

14
GB · FP16
2 bytes each
→
3.5
GB · 4-bit
0.5 bytes each
Rule: memory ≈ N × (bytes per weight).
7B × 2 bytes ≈ 14 GB.
7B × 0.5 bytes ≈ 3.5 GB.
Trade: a little rounding error. Gain: the model fits. That is the whole reason quantization appears before QLoRA.
Quantization · how

A precise float gets rounded into a small set of allowed labels.

Think “lossy zip for numbers.” Same idea as JPEG for images: smaller file, almost the same picture.

Step 1 · few buckets (INT2 toy)

−2very neg
−1neg
0near 0
1pos
Ready

Step 2 · finer labels (INT8 math)

scale ≈ 0.00784
q = round(−0.91 / 0.00784) = −116
restore = −116 × 0.00784 ≈ −0.9098
−0.91
float
÷
0.00784
scale
→
?
INT8
Ready
QLoRA uses NF4 (4-bit levels chosen for normal-looking weights). Same story: fewer bits, small error, much less VRAM.
QLoRA

QLoRA = compress the base, then train LoRA on top.

Freeze the model in 4-bit NF4. Attach LoRA adapters. Train only the adapters. Gradients never rewrite the base.

14
GB FP16
→
3.5
GB NF4
1 · Load base model14 GB
2 · Quantize to NF4 and freeze3.5 GB
3 · Attach LoRA adapterstiny
4 · Train on free Colab T4fits
LoRAQLoRA
BaseFP16 frozenNF4 frozen
Adapterstraintrain
7B in training (ballpark)~16 GB+ (base + activations)~6 GB
Ready
QLoRA paper (arXiv:2305.14314)
QLoRA · worked example

Same 7B model. Do the memory arithmetic.

Memory ≈ (number of parameters) × (bytes per parameter). That one line explains why QLoRA fits on Colab.

SetupBytes / param7B size
Base in FP16 (LoRA case)27e9 × 2 = 14 GB
Base in NF4 (QLoRA case)0.57e9 × 0.5 = 3.5 GB
LoRA adapters (example 24M)2 (FP16)24e6 × 2 ≈ 0.05 GB
QLoRA total (weights only)NF4 base + FP16 adapters≈ 3.55 GB
What trains? Only the adapters (~24M). The 7B NF4 base is frozen.
What is “Q”? Quantized base. What is “LoRA”? The B and A adapters on top.
Forward pass idea: dequantize a weight block to 16-bit just long enough to multiply, then keep storing it as 4-bit.
How the words fit

Three names. One family.

1
PEFTThe family name: freeze the big model, train a small add-on.
2
LoRAThe popular PEFT method: the add-on is two thin matrices B and A.
3
QLoRALoRA plus a 4-bit compressed base, so it fits on a small GPU.
If someone says “we fine-tuned with QLoRA,” they mean: quantized base + LoRA adapters. Not a totally different algorithm from LoRA.
Recap

The story you should be able to retell

1
Generic models don’t know your company rules
2
RAG for fresh facts · fine-tune for behavior (often both)
3
Full fine-tuning is too expensive (~112 GB)
4
PEFT (via LoRA) trains under 1% using B × A
5
Quantization = fewer bits so the base fits (14 GB → 3.5 GB)
6
QLoRA = 4-bit base + LoRA → fits Colab
LoRA paper QLoRA paper HF PEFT

Next lab: Unsloth on Colab with Llama 3.2 3B and your own support-style examples.