The Journey So Far

Zero to GenAI Engineer: Full Recap

Sessions 00 → 05 · Everything in one story

Mohamed Noordeen Alaudeen

Before We Start

"You did not just learn tools. You learned how AI thinks."

One month ago, most of you were asking: "How does Google even work?"

Today, after 8 sessions, you can:

• Build a keyword search engine from scratch
• Explain self-attention to a colleague
• Trace the full history of GPT
• Calculate token costs before calling an API
• Run an LLM on your own laptop, for free
• Optimise prompts automatically with DSPy
• Call any LLM provider with a single LangChain interface

That is not a small thing. Let's make sure it sticks.

The Map: What We've Covered Together

The Story in One Sentence

Computers only understand numbers. We spent 70 years inventing better and better ways to turn language into numbers, and somewhere along the way, we accidentally built intelligence.

That is what S00 → S05 is about.

Every session left one gap open. The next session filled it.

The MISSING Chain: How Each Session Connects

S00  Keyword search (TF-IDF)
      ⚠️  MISSING: "car crash" ≠ "automobile accident", same meaning, different words
S01  Word embeddings (Word2Vec, GloVe, FastText)
      ⚠️  MISSING: "bank" (river) and "bank" (money) get the SAME vector
S02  Transformers + Self-Attention
      ⚠️  MISSING: Training from scratch = weeks of GPU time, thousands of dollars
S03  Pre-trained GPT → ChatGPT via RLHF
      ⚠️  MISSING: What actually happens between "prompt" and "response"?
S04  BPE tokens + Temperature + Top-K + Top-P
      ⚠️  MISSING: API costs money, needs internet, can go down
S05  Local LLMs (Ollama) + Multi-provider (OpenRouter)
      ⚠️  MISSING: We're still writing prompts by hand, guessing what works
S06  DSPy + Chain-of-Thought + Prompt Optimisers
      ⚠️  MISSING: Every provider has a different SDK, auth, and response format
S07  LangChain, one unified interface for every provider

Chapter 1

🔍 How Search Engines Work

Session 00: The Foundation of Everything

"Before AI could understand you, it first had to find you."

The Problem: Computers Cannot Read

A computer sees this sentence:

"The cat sat on the mat"

As this:

[84, 104, 101, 32, 99, 97, 116, 32, 115, 97, 116, ...]

Just bytes. No meaning. No structure. Just numbers.

First challenge in AI: How do we rank documents by relevance to a query, without any intelligence?

How Google (Used to) Work: The Inverted Index

Every word in every document gets catalogued:

"python"  →  [doc_12, doc_47, doc_203, doc_891]
"snake"   →  [doc_9,  doc_47, doc_891]
"Django"  →  [doc_47, doc_203]

Search "python Django" → intersection → doc_47, doc_203

Fast. Scalable. But naive, every word counts equally.

"the" and "python" get the same weight. That's a problem.

TF-IDF: The First Smart Ranking

Fix the naive count with two signals combined:

TF (Term Frequency), how often does this word appear in this document?

IDF (Inverse Document Frequency), how rare is this word across all documents?

TF-IDF = TF × log(Total docs / Docs containing word)

"the"       → appears everywhere → IDF ≈ 0 → score ≈ 0
"tensorflow" → rare → IDF is high  → score is high

Key insight: A word that appears often in one document but rarely everywhere else is the signal word for that document.

Where TF-IDF Breaks

Search query: "automobile accident"

TF-IDF finds: documents containing "automobile" and "accident" ✅

TF-IDF misses: "The car crash happened on Highway 5 this morning" ❌

Same meaning. Completely different words. TF-IDF cannot bridge that gap.

⚠️ This is the gap that Session 01 fills.

🎯 Interview Questions: Session 00

Q1: What is an inverted index and why does it make search fast?

A data structure that maps each unique word to the list of documents containing it. Instead of scanning every document per query, you look up the word directly and get the document list in sub-millisecond time. A query like "python training" looks up both words, gets their document lists, then intersects them. This is how Elasticsearch, Lucene, and every major search engine indexes content.

Q2: What does TF-IDF measure and how does it rank better than simple word count?

TF (Term Frequency) measures how often a word appears in one document. IDF = log(total docs / docs containing word). Words in every document like "the" get IDF close to 0 and contribute nothing. Rare words get high IDF scores. TF x IDF is high for relevant, rare words and near zero for common filler words. This gives automatic stopword filtering built into the math, no explicit stopword list needed.

Q3: What is the core limitation of TF-IDF and how was it solved?

TF-IDF is purely lexical: it matches exact character strings. "car" and "automobile" have zero computed relationship. No understanding of synonyms, context, or meaning. This semantic gap was solved by word embeddings (Word2Vec, GloVe), which map words to dense vectors where semantically similar words land geometrically close. Modern systems use hybrid search: BM25 for exact keyword recall plus vector search for semantic understanding.

Q4: What is BM25 and why do most search engines use it over raw TF-IDF?

BM25 extends TF-IDF with two fixes. Term saturation: raw TF-IDF grows linearly with word frequency, so a word appearing 100 times scores 10x higher than appearing 10 times. BM25 adds a saturation parameter k1 (typically 1.5) so additional occurrences have diminishing returns. Document length normalization: parameter b penalizes long documents, because finding a rare word in a short document is more significant than in a 50,000-word one. BM25 is the default in Elasticsearch and a strong baseline even against neural retrievers.

Q5: What is IDF when a word appears in every document? Why does this matter?

IDF = log(N/N) = log(1) = 0. The TF-IDF score becomes 0, so this word contributes nothing to ranking regardless of how often it appears. This is the automatic stopword filtering property: "the", "is", "a" appear in every document and score zero without needing an explicit stopword list. Conversely, a word appearing in 3 of 1000 documents gets IDF = log(333) ≈ 5.8, making it a very strong relevance signal.

Q6: Design a production search architecture. Where does the inverted index fit?

Ingestion pipeline: crawl documents, tokenize text, build the inverted index offline. At query time: tokenize the query, look up each token in the inverted index (O(1)), retrieve candidate lists, intersect them, score with BM25, return top-K results. For semantic search: add a vector index alongside and run approximate nearest-neighbor search. Use a reranker to improve precision on top results. This two-stage retrieve-then-rerank pattern is the same architecture used in production RAG systems.

Q7: What are the main limitations of bag-of-words models?

Bag-of-words completely discards word order, so "dog bites man" and "man bites dog" produce identical vectors. It creates sparse, high-dimensional representations with vocabulary sizes of 100,000 or more. It treats all words as independent: "good" and "excellent" are as unrelated as "good" and "airplane." The word "bank" gets the same representation regardless of context. These limitations led directly to word embeddings (semantic gap) and then to Transformers (context-dependence gap).

Chapter 2

🔢 Text to Numbers

Session 01: Teaching Computers to Understand Meaning

"What if we could capture what a word means, not just what it says?"

Five Methods, Six Decades of Progress

1954   Bag of Words        count words, ignore order
1972   TF-IDF              weight rare words higher
2013   Word2Vec            meaning from surrounding context
2014   GloVe               meaning from global co-occurrence
2016   FastText            meaning from character pieces (handles typos)

Each method fixed what the previous one got wrong.

Bag of Words: The Simplest Possible Approach

Sentence: "I love Python. Python loves me."
Vocab:    {I, love, Python, loves, me}
Vector:   [1,  2,     2,     1,    1 ]

✅ Simple, fast, works for basic classification tasks

❌ "Dog bites man" and "Man bites dog" → identical vector Word order is completely destroyed.

❌ "good" and "excellent" → zero relationship in this model

Word2Vec: The 2013 Breakthrough

Core insight: You shall know a word by the company it keeps., J.R. Firth, 1957

Training task: Given surrounding words, predict the missing one.

"The _____ ran across the road"   → model sees: cat, dog, child all fit
"She deposited money at the ___"  → model sees: bank fits, not river

After training on billions of sentences, words with similar contexts end up with similar vectors.

Result: geometry encodes meaning.

The Magic: Meaning Becomes Arithmetic

king  - man + woman  ≈  queen       ✅
Paris - France + Italy  ≈  Rome      ✅
doctor - man + woman  ≈  nurse       ✅ (reflects training data bias)

You can add and subtract concepts.

Cosine similarity score:

• ("car", "automobile") → ~0.92 (nearly identical)
• ("car", "cloud") → ~0.12 (unrelated)

This is why RAG works. This is why semantic search works. Everything from M05 onwards relies on this.

GloVe and FastText: What Came Next

GloVe (2014), instead of a sliding window, use global co-occurrence statistics across the entire corpus. More stable, faster to train.

FastText (2016), instead of whole words, train on character n-grams:

"playing" → ["pla", "lay", "ayi", "yin", "ing", "play", "layi", ...]

Advantage: handles typos and words not seen in training:

"playin" (typo) → still gets a reasonable vector from subword pieces ✅

Why Cosine Similarity: Not Euclidean Distance?

Short review: "Good film" vs "A remarkably brilliant, absolutely wonderful, outstanding film that I thoroughly enjoyed"

Euclidean distance: large, different vector lengths → far apart ❌

Cosine similarity: high, same direction → same meaning ✅

Cosine measures the ANGLE between vectors, not the distance.
Angle = meaning direction. Length = how many words you used.

Length doesn't determine meaning. Direction does.

🎯 Interview Questions: Session 01

Q1: What is a word embedding and how is it different from one-hot encoding?

A word embedding maps each word to a dense vector (typically 100 to 300 dimensions) where semantic relationships are preserved as geometry. One-hot encoding maps each word to a sparse binary vector with a 1 at the vocabulary index and 0 everywhere else. Problems with one-hot: no semantic relationship between words, and a 100k vocabulary needs 100k-dimensional vectors. Word embeddings solve both: "king" and "queen" have high cosine similarity, and 300 dimensions represent the entire vocabulary meaningfully.

Q2: How does Word2Vec learn embeddings without human labels?

Word2Vec uses self-supervised learning on a prediction task derived from text structure. Skip-gram: given a center word, predict surrounding context words. CBOW: given context words, predict the center word. Words that appear in similar contexts end up with similar weight vectors. After training on billions of sentences, the network weights converge to vectors that encode usage patterns, which turn out to encode semantic meaning.

Q3: Why cosine similarity and not Euclidean distance for word vectors?

Word vectors have magnitudes that depend on word frequency during training. Common words end up with larger magnitudes. Euclidean distance is magnitude-sensitive: two vectors pointing in the same direction but with different lengths would score as different. Cosine similarity measures only the angle between vectors, ignoring magnitude. Same semantic meaning equals same direction equals high cosine similarity regardless of frequency. Range is -1 (opposite) to +1 (identical direction).

Q4: What is the difference between Skip-gram and CBOW, and which handles rare words better?

Skip-gram: given center word, predict surrounding context words. Each center word generates multiple training examples, so the model focuses deeply on each individual word. CBOW: given context words, predict the center word. CBOW is 3 to 5 times faster but averages context signals. Skip-gram performs better for rare words because each step focuses on one word's context relationships. CBOW works better for frequent words and when training speed matters. Skip-gram with Negative Sampling (SGNS) is the industry standard for high-quality embeddings.

Q5: What is negative sampling and why was it needed?

The naive Word2Vec update computes a gradient over the entire vocabulary (100k words) for each training step: O(vocabulary size) operations per step. Negative sampling replaces the full softmax with a binary classifier: is this (center word, context word) pair real or noise? For each real pair, sample k noise words (5 to 20), and train to distinguish real from noise. This reduces computation from O(vocabulary) to O(k) per update, a 5000 to 20000x speedup. The model still learns meaningful embeddings because distinguishing real context from random noise forces it to capture genuine co-occurrence patterns.

Q6: What is the critical limitation of Word2Vec and GloVe that Transformers solve?

Both Word2Vec and GloVe produce static embeddings: every word gets one fixed vector regardless of context. "Bank" gets the same vector whether it appears near "money" or "river." This polysemy problem is exactly what Transformers solve. Self-attention builds context-aware representations where the same word gets a different vector depending on its surrounding words. That is why BERT largely replaced Word2Vec for downstream NLP tasks.

Q7: What is FastText and when would you use it over Word2Vec?

FastText trains on character n-grams rather than whole words. "Playing" becomes ["pla", "lay", "ayi", "yin", "ing"]. This lets FastText generate a vector for words it never saw during training by composing character pieces. Use FastText when your data has typos or misspellings, you work with morphologically rich languages (German, Turkish), you have domain-specific technical vocabulary, or you work with code tokens. The cost is larger model size and slower inference. For most production tasks today, sentence-transformers (BERT-based) outperform FastText, but it remains a fast baseline with no GPU required.

Chapter 3

⚡ The Attention Revolution

Session 02: The Architecture That Powers Everything You Use Today

"What if every word could look at every other word simultaneously, and decide what matters?"

The Context Problem Word2Vec Left

"I went to the bank to deposit money."

"I sat on the bank of the river."

Word2Vec gives "bank" the exact same vector in both sentences.

It was trained as an average across all contexts. It has no ability to adapt.

We need a model that builds a different representation for "bank" based on whether it's surrounded by "money" or "river".

Why RNNs Failed at This

Before 2017, we used Recurrent Neural Networks:

word₁ → word₂ → word₃ → ···  → word₁₀₀ → output

Problems:

• Sequential, can't parallelise → slow on modern GPUs
• Vanishing gradient, by word 100, word 1 is largely forgotten
• Fixed bottleneck, entire 100-word sentence compressed into one vector

The solution wasn't to make RNNs better. It was to replace them entirely.

Self-Attention: The Core Mechanism

For every word, ask: "Which other words in this sentence are most relevant to understanding me?"

"The animal didn't cross the street because it was too tired."

Question: What does "it" refer to?
Self-attention says: "animal" (high weight), "street" (low weight)

This assignment happens automatically, learned from data.

Every word in the sentence votes on how relevant every other word is, simultaneously.

Query, Key, Value: The Search Engine Inside Attention

Think of it as running a search engine inside the sentence:

Attention score = dot product of Q with each K (how relevant?)

Output = weighted sum of all V, using attention scores as weights

Multi-Head Attention: 8 Perspectives at Once

Run self-attention 8 times in parallel, each with different learned projections:

Head 1 → might learn: who does what (subject → verb)
Head 2 → might learn: what describes what (noun → adjective)
Head 3 → might learn: what refers to what (pronoun → antecedent)
Head 4–8 → other patterns, discovered automatically during training

Concatenate all 8 outputs → project down → richer representation

"One person reading the sentence 8 times with 8 different questions."

Positional Encoding: Giving the Model a Sense of Order

Self-attention looks at all words simultaneously, it has no built-in sense of word order.

Without positional encoding: "Dog bites man" = "Man bites dog" (same words, same attention)

Solution: Add a unique positional signal to each word's embedding:

Position 1 → add [sin(1/10000⁰), cos(1/10000⁰), sin(1/10000²/⁵¹²), ...]
Position 2 → add [sin(2/10000⁰), cos(2/10000⁰), sin(2/10000²/⁵¹²), ...]

Sine and cosine at different frequencies → each position gets a unique "fingerprint".

The Full Architecture: What You Built

You trained this on English → Italian translation.

🎯 Interview Questions: Session 02

Q1: Explain self-attention. Walk through Q, K, V.

Each token generates three learned linear projections: Query (what this token is searching for), Key (what this token offers as a label to others), and Value (the actual content this token contributes). Attention score = dot product of Q with each K, scaled by the square root of d_k to prevent softmax saturation, then softmax to get weights. Output = weighted sum of all V vectors using the attention weights. This runs in parallel for all tokens simultaneously, resolving contextual meaning in one pass.

Q2: What is the difference between BERT and GPT architecturally?

BERT uses encoder-only Transformer layers with bidirectional self-attention: every token can attend to every other token in both directions. Pre-trained with Masked Language Modeling. Strong for classification, extraction, and sentence similarity. GPT uses decoder-only Transformer layers with causal self-attention: each token attends only to previous tokens. Pre-trained on next-token prediction. In the era of conversational AI, generation wins: GPT, Claude, Gemini, LLaMA, and Mistral are all decoder-only.

Q3: Why does multi-head attention work better than single-head?

A single attention head must collapse all relationship types into one pattern. Multiple heads run in parallel, each with different learned weight matrices, so each head can simultaneously specialize. One head might track subject-verb dependencies, another pronoun-antecedent resolution, another semantic similarity. The outputs are concatenated and projected back down. Richer representations at the same computational cost as a single larger head.

Q4: Why divide by the square root of d_k in the attention formula?

As the embedding dimension d_k grows, dot products between Q and K grow in magnitude proportionally. Very large dot products push softmax into regions where one token gets probability close to 1 and all others close to 0. In this saturated regime, gradients drop to near zero and the model gets stuck. Dividing by the square root of d_k normalizes the scale, keeping softmax numerically stable with useful gradients.

Q5: Why does a Transformer need positional encoding?

Self-attention treats the input as a set with no inherent order. "The cat sat" and "sat cat The" would produce identical attention patterns without positional encoding. The original paper adds sinusoidal waves at different frequencies to each position. Modern models like Llama and Mistral use RoPE (Rotary Position Embedding), which rotates the Q and K vectors by position-dependent angles, encoding relative position directly into the attention computation and generalizing better to long sequences.

Q6: What does the Feed-Forward sublayer do in each Transformer block?

Each Transformer block has two sublayers. Multi-Head Self-Attention handles relationships between tokens. The Feed-Forward Network (FFN) is applied independently to each token. FFN(x) = max(0, xW1 + b1) x W2 + b2. The FFN is typically 4x wider than the embedding dimension. Research has traced factual knowledge (like "Paris is the capital of France") to specific FFN layer weights. Attention routes context across the sequence; FFN applies per-token transformations and retrieves memorized knowledge.

Chapter 4

🚀 How GPT Changed the World

Session 03: From Text Prediction to Aligned Intelligence

"A model trained on one task, predict the next word, accidentally learned how to reason, translate, code, and debate."

The Timeline That Rewrote the Rules

2017  Attention Is All You Need   → The Transformer (Vaswani et al.)
      │
2018  GPT-1    117M params        → Pre-train + fine-tune works
      BERT-base 110M params        → Bidirectional understanding
      │
2019  GPT-2    1.5B params        → Zero-shot. Too dangerous to release.
      │
2020  GPT-3    175B params        → Few-shot learning at unprecedented scale
      │
2022  InstructGPT  → RLHF  → ChatGPT (OpenAI)
      HH-RLHF  → Constitutional AI  → Claude (Anthropic)
      │
2023  GPT-4 · LLaMA · Mistral · Gemini
      │
2024+ What you use every day

GPT-1: The First Proof (2018)

The old way: Train a separate model for every task (translation, classification, Q&A) from scratch. Needs thousands of labelled examples for each task.

GPT-1's insight: Pre-train on massive unlabelled text first. Language itself teaches the model grammar, facts, and reasoning. Then fine-tune on a small labelled dataset for your specific task.

Pre-training:  predict next word on 7000 books
Fine-tuning:   show 500 labelled examples for sentiment analysis

Result: beats task-specific models trained on 50,000 examples

"You don't need labels to learn language. Language is its own teacher."

GPT-2: Scale Breaks Expectations (2019)

1.5 Billion parameters. 40GB of internet text.

Something unexpected happened: zero-shot task completion.

No fine-tuning. Just a prompt:

Prompt: "Translate to French: The sky is blue"
GPT-2:  "Le ciel est bleu"    ✅

Prompt: "Summarise this article: ..."
GPT-2:  Reasonable summary    ✅

OpenAI held back the full model for 9 months, worried about misuse.

Scale is a feature. Emergent capabilities appear at certain parameter counts, not programmed in, just... there.

GPT-3: The World Stopped (2020)

175 Billion parameters. 570GB of filtered text data (from 45TB of raw Common Crawl).

Few-shot learning, teach it in the prompt:

Prompt:
"Translate to French:
  Hello → Bonjour
  Thank you → Merci
  Good morning → ???"

GPT-3: "Bonjour"   ✅

No gradient updates. No retraining. The model learned from the prompt at inference time.

This is called in-context learning, and it changes everything about how we build AI systems.

The Alignment Problem

GPT-3 was powerful. But it was not helpful.

It would:

• Complete your sentence instead of answering your question
• Hallucinate facts confidently
• Occasionally produce harmful or offensive content

The alignment problem: A model trained to predict text is NOT the same as a model trained to be helpful, harmless, and honest.

Training task (next-word prediction) ≠ Deployment goal (helpful assistant)

This gap had to be closed. Enter RLHF.

RLHF: How ChatGPT Was Born

Reinforcement Learning from Human Feedback:

Step 1  Supervised Fine-Tuning
        Collect examples of ideal responses from human writers
        Fine-tune GPT-3 on those responses

Step 2  Train a Reward Model
        Show human raters pairs of responses → "which is better?"
        Train a model to predict human preference scores

Step 3  Reinforcement Learning (PPO)
        Update the LLM using the reward model as the "judge"
        The LLM learns to produce responses that score high

Step 4  Repeat, the reward model improves, the LLM improves

Result: a model that tries to help instead of just predicting text.

Hallucination: Why It Still Happens

GPT is trained to predict the most probable next token.

When training data contains confidently-written wrong statements, the model learns to sound confident, even when wrong.

It has no internal "I don't know" signal.

Question: "Who invented the telephone?"
GPT (unhelpful): "The telephone was invented by
                  Elisha Gray in 1876."   ← confident, wrong

RLHF reduces hallucination but doesn't eliminate it.

This is why RAG (Retrieval-Augmented Generation) exists, you give the model the right answer in the prompt so it doesn't have to guess.

BERT vs GPT: Why GPT Dominated

BERT understood text better. GPT generated text better. In the era of conversational AI, generation wins.

Today: GPT-4, Claude, Gemini, LLaMA, Mistral, all decoder-only.

🎯 Interview Questions: Session 03

Q1: What is in-context learning and which model demonstrated it at scale?

In-context learning is the ability to perform new tasks from examples given in the prompt, with zero parameter updates. GPT-3 (2020, 175B parameters) demonstrated this at unprecedented scale. You give 2 to 3 input/output examples and the model generalizes to new inputs without any gradient updates. This was genuinely surprising: prior models needed explicit fine-tuning for every new task. In practice, it means you can turn a general-purpose LLM into a specialized classifier, extractor, or formatter purely through prompting, with no labeled training data or GPU training required.

Q2: Walk through RLHF step by step. What is the KL penalty for?

RLHF has three stages. Stage 1: Supervised Fine-Tuning (SFT), start from a base model and fine-tune on human-written demonstrations of good responses. Stage 2: Reward Model Training, generate multiple responses to the same prompt, have humans rank them, train a separate model on these preference comparisons. Stage 3: RL with PPO, use Proximal Policy Optimization to update the SFT model to maximize reward model scores. The KL penalty keeps the model from drifting too far from the SFT checkpoint and exploiting the reward model with degenerate outputs.

Q3: What is hallucination and what are the main mitigations?

Hallucination is when a model confidently states something factually false: a fabricated citation, a wrong date, a person who does not exist. It happens because LLMs are trained to predict plausible next tokens, not to verify factual accuracy. There is no built-in "I don't know" signal. Mitigations: RAG grounds answers in retrieved documents (the model must cite specific text), system prompts instruct the model to say it does not know rather than invent, low temperature reduces creative deviations for factual tasks, and validation LLM calls check if the answer is supported by retrieved context.

Q4: What is DPO and how does it differ from RLHF?

DPO (Direct Preference Optimization) achieves the same alignment goal as RLHF but eliminates both the reward model and the RL training loop. RLHF requires three separate training stages: SFT, reward model, then PPO. This is expensive and unstable. DPO's insight: the mathematically optimal RLHF policy has a closed-form solution. You can convert the RL problem directly into a classification task on preference pairs. Given (prompt, preferred response, rejected response), DPO trains the LM to increase the likelihood ratio of preferred over rejected. Simpler, more stable, computationally cheaper. Most modern open-source aligned models use DPO or its variants rather than RLHF.

Q5: What is Constitutional AI?

Constitutional AI (CAI, Anthropic 2022) addresses the scalability bottleneck of RLHF: human labelers must evaluate every output. CAI replaces human safety evaluators with the model itself, guided by a written "constitution" of principles. The model critiques its own response against these principles, generates a revised response, and these AI-generated preference pairs are used for training. Benefits: scales cheaply (principles applied to millions of examples automatically), and alignment criteria are explicit and auditable rather than implicit in human judgments. Claude's alignment training uses Constitutional AI as a core component.

Q6: What are emergent abilities in LLMs?

Emergent abilities appear suddenly at certain model scales, barely present in smaller models and then fully present in larger ones, without being explicitly trained for. A model at 7B parameters might score 5% on a task, and a 70B model suddenly scores 85%: not a gradual improvement but a phase transition. Examples: chain-of-thought reasoning, multi-step arithmetic, in-context learning. For engineers: you cannot predict capabilities by extrapolating from small models. A task that does not work at 7B may become trivial at 70B.

Q7: What were the key differences between GPT-1, GPT-2, and GPT-3?

GPT-1 (2018): 117M parameters, trained on roughly 1B tokens. Proved that unsupervised pre-training followed by fine-tuning works. Required fine-tuning for each downstream task. GPT-2 (2019): 1.5B parameters, trained on WebText (40GB of web text). Same architecture, dramatically bigger. Demonstrated zero-shot task completion: just a prompt, no fine-tuning. OpenAI initially withheld it over misuse concerns. GPT-3 (2020): 175B parameters, trained on 570GB of filtered internet text. Demonstrated in-context few-shot learning at scale: show 3 examples in the prompt and the model generalizes with zero gradient updates. One model could write code, translate, summarize, and answer questions from examples in the prompt alone. The pattern: scale (parameters plus data) was the primary lever, architecture barely changed.

Chapter 5

⚙️ Inside the Pipeline

Session 04: BPE, Temperature, Top-K, Top-P

"You've been calling the API like a black box. Time to open it."

You Are Not Paying Per Word

When you call an LLM API, the billing unit is not a word. It is a token.

A token is a subword unit, not a character, not a full word.

"ChatGPT"       → ["Chat", "G", "PT"]          = 3 tokens
"tokenization"  → ["token", "ization"]          = 2 tokens
"unbelievable"  → ["un", "believ", "able"]      = 3 tokens
"Hello!"        → ["Hello", "!"]                = 2 tokens
"2024"          → ["20", "24"]                  = 2 tokens

Rule of thumb: 1 token ≈ 0.75 English words

A 10,000-word document ≈ 13,000 tokens. Miscalculate and you burn budget.

BPE: How Tokens Are Built

Byte Pair Encoding, used in GPT-2, GPT-3, GPT-4, LLaMA, Mistral, and nearly every modern LLM.

Algorithm:

Start:   vocabulary = all individual characters
         [h, e, l, l, o, " ", w, o, r, l, d]

Round 1: Most frequent pair: "l" + "o" → merge into "lo"
Round 2: Most frequent pair: "h" + "e" → merge into "he"
Round 3: Most frequent pair: "w" + "o" → merge into "wo"
...
Repeat until vocabulary reaches target size (e.g. 50,000 tokens)

Common words → single tokens. Rare words → split into meaningful pieces. Unknown words → handled gracefully, never <UNK>.

Temperature: The Creativity Dial

After the Transformer runs, it produces raw scores called logits for every word in the vocabulary.

Temperature T divides all logits before converting to probabilities:

Prompt: "The sky is ___"

T = 0.2  →  blue: 97%, grey: 2%, red: 1%   (almost always "blue")
T = 0.7  →  blue: 65%, grey: 20%, red: 10%  (mostly "blue", some variety)
T = 1.5  →  blue: 32%, grey: 25%, purple: 20%, ...  (creative/random)

Top-K Sampling

After applying temperature, filter to only the K most likely tokens:

Full vocabulary: 50,000 tokens
Top-K = 5:       keep only top 5, zero the rest, renormalise, then sample

"The capital of France is ___"
K=5 keeps: Paris(78%), Lyon(8%), Nice(6%), Marseille(5%), Bordeaux(3%)

• K = 1 → greedy decoding (always pick the top token, no randomness)
• K = 50 → common default, good balance
• K = vocabulary size → no filtering, pure sampling

Top-P: Nucleus Sampling (Better Than Top-K)

Sort tokens by probability (highest first). Keep adding to the "nucleus" until cumulative probability reaches P:

P = 0.9, keep the smallest set of tokens summing to ≥ 90% probability

Confident model (one clear answer):
  "Paris": 95% → nucleus = just ["Paris"] (tiny)

Uncertain model (many valid words):
  "happy", "sad", "curious", "excited", "nervous", ... → nucleus expands

Why Top-P is better than Top-K: Top-K always keeps exactly K tokens regardless of confidence. Top-P adapts, small when the model is sure, large when uncertain.

Production systems usually use temperature + top-P together.

The Correct Order: A Common Interview Trap

# The ONLY correct order:

# Step 1: Apply temperature
logits = logits / temperature          # rescale raw scores

# Step 2: Apply Top-K filter
# Zero out everything outside the top K

# Step 3: Apply Top-P filter
# Zero out everything outside the nucleus

# Step 4: Sample
next_token = sample_from(remaining_probabilities)

Getting the order wrong is a real bug, and a common interview question.

Temperature must come first because it reshapes the distribution before filtering.

🎯 Interview Questions: Session 04

Q1: Walk through the BPE algorithm step by step.

Start with a character-level vocabulary. Count all adjacent character pairs across the training corpus. Merge the most frequent pair into a new token. Repeat until the vocabulary reaches the target size (typically 50,000 to 100,000 tokens). Why not word-level: any word not in training becomes UNK, and every inflected form needs its own entry. Why not character-level: sequences become extremely long and attention computation (O(n squared)) becomes 16 to 25 times more expensive. Byte-level BPE (GPT-2 onwards) operates on raw bytes, guaranteeing no unknown tokens ever.

Q2: What is the correct order: temperature, softmax, Top-K, Top-P, sample?

Temperature always comes first because it reshapes the distribution before filtering. Correct order: (1) run Transformer forward pass to get raw logits, (2) divide logits by temperature, (3) apply softmax to convert to probabilities, (4) apply Top-K filter by zeroing out everything below rank K, (5) apply Top-P filter by keeping only the smallest set reaching cumulative probability P, (6) renormalize and sample. Getting this order wrong is a common bug: applying temperature after softmax changes what it does mathematically.

Q3: GPT-4o charges $5 per 1M input tokens. A RAG system sends 3,000 tokens per query at 10,000 queries per day. What is the monthly input cost?

3,000 tokens x 10,000 queries = 30 million input tokens per day. At $5 per 1M tokens = $150 per day for input alone. Monthly: $150 x 30 = $4,500 in input costs. Add output (500 tokens x 10,000 queries = 5M output tokens per day, at $15 per 1M = $75 per day, $2,250 per month). Total: roughly $6,750 per month. Optimization: route simple queries to GPT-4o-mini (30 to 40 times cheaper), reduce context size by keeping top 3 retrieved chunks instead of 10, use prefix caching for repeated system prompts.

Q4: What is the difference between Top-K and Top-P sampling?

Top-K always keeps exactly K candidates. Problem: if the model is very confident (one token at 95%), K=40 still forces sampling from 40 tokens including near-zero probability noise. If the model is uncertain (20 tokens at 5% each), K=40 might be too restrictive. Top-P keeps the smallest set whose cumulative probability reaches P. When confident, Top-P might keep just 2 to 3 tokens. When uncertain, it adapts to keep more. Best practice: use both together (Top-K=40, Top-P=0.95).

Q5: What is the "lost in the middle" problem in RAG?

Research shows LLMs systematically under-attend to information in the middle of long context windows. They perform best on information at the beginning (primacy) and end (recency) of the context, with accuracy dropping for content in the middle. In RAG: if you retrieve 10 documents and the most relevant one lands in the middle, it may effectively be ignored. Mitigations: put the most important retrieved context first or last, limit to top 3 to 5 chunks rather than 10 to 15, use a reranker to ensure the most relevant chunk gets a priority position.

Q6: A model generates "Eiffel Tower is in Berlin" confidently. Is this hallucination or retrieval failure?

This is a factual hallucination: the model generates false information with confidence. Retrieval failure means the correct document was not found. Hallucination means even with the correct document retrieved, the model generates something not supported by it. How to diagnose: check if the correct fact was in the retrieved context. If yes, it is hallucination (the model ignored the grounding evidence). If no, it is retrieval failure (fix chunking, reranking, or the embedding model). They require different fixes: retrieval failure needs better search; hallucination needs stricter grounding prompts and citation enforcement.

Chapter 6

💻 Running AI Yourself

Session 05: Local LLMs & API Providers

"What if you could run GPT-quality AI on your own laptop, for free, without sending a single byte to any company?"

Why Not Just Use OpenAI Every Time?

The answer: Run locally for dev/privacy OR use a multi-provider strategy for production.

Ollama: LLMs on Your Laptop

# Install (Mac / Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull phi3          # 2.7B params, runs on 8GB RAM, free
ollama pull llama3.2      # 3B  params, runs on 8GB RAM, free
ollama pull mistral       # 7B  params, needs 16GB RAM

# Ask it a question
ollama run phi3
>>> "Explain transformers like I'm 10 years old"

Ollama exposes an OpenAI-compatible REST API at http://localhost:11434

Your Python code doesn't change. Just swap the base_url.

The Universal Chat Format

Every provider, OpenAI, Gemini, Anthropic, Ollama, Databricks, accepts this same structure:

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant."
    },
    {
        "role": "user",
        "content": "Explain the Transformer architecture in one paragraph."
    },
    {
        "role": "assistant",
        "content": "The Transformer was introduced in..."   # previous turn
    },
    {
        "role": "user",
        "content": "Now explain RLHF."
    }
]

This is the skeleton of every LLM application you will ever build.

OpenRouter: One Key, 100+ Models

# Identical code. Change one string. Different provider.

model = "openai/gpt-4o"
model = "anthropic/claude-3-5-sonnet"
model = "google/gemini-1.5-pro"
model = "meta-llama/llama-3-70b-instruct"
model = "mistralai/mistral-7b-instruct"

No 5 different SDKs. No 5 API keys. One account, one key, transparent pricing.

When to use OpenRouter: Prototyping, model comparison, avoiding vendor lock-in.

Student tip: Start with Gemini (free tier). Use OpenRouter when you want to compare models.

Map-Reduce: Handling Documents Longer Than the Context Window

Every LLM has a context limit (8K, 32K, 128K tokens). What if your document is longer?

200-page PDF (300,000 tokens), won't fit
          │
    SPLIT into 20 chunks of ~15,000 tokens each
          │
    MAP: Summarise each chunk independently   ← 20 parallel LLM calls
          │
    REDUCE: Merge all 20 chunk summaries       ← 1 final LLM call
          │
    Final summary of the entire 200-page document

Same pattern used in document intelligence pipelines (M11), research assistants (M14), and any production RAG system handling large documents.

Local vs Cloud: The Decision Table

🎯 Interview Questions: Session 05

Q1: What is GGUF format and what does Q4_K_M mean?

GGUF is the standard format for quantized LLM weights created by the llama.cpp project. All model weights, hyperparameters, tokenizer, and metadata are packed into one portable file with no separate config files needed. Q4_K_M breaks down as: Q4 = 4-bit quantization (each weight stored in 4 bits instead of 32), K = K-quant method (groups of roughly 32 weights share a scale factor to minimize quantization error), M = medium size variant (balances accuracy vs speed). Q4_K_M achieves roughly 3% perplexity increase vs FP16, imperceptible in most applications, while reducing a 7B model from 14GB to about 4GB.

Q2: What are the three message roles in the universal chat format?

The universal format is a list of {role, content} dicts supported by every major provider. System: sets overall behavior, persona, and constraints for the entire conversation. User: represents a human turn. Assistant: represents a model turn. Including prior assistant turns is how you inject conversation history for multi-turn dialogue. Every provider (OpenAI, Anthropic, Google, Ollama) accepts this same format, which is why your conversation history logic works across providers without changes.

Q3: What are the limitations of OpenRouter for production use?

OpenRouter is a proxy, not a first-party provider. Limitations: (1) additional latency from the proxy hop (typically 50 to 200ms), (2) you depend on OpenRouter's uptime on top of the underlying provider's uptime, (3) pricing includes a small markup over direct API pricing, (4) some advanced features like fine-tuned model endpoints may not be available through the proxy, (5) enterprise SLAs come from OpenRouter, not the underlying providers. Use it for prototyping, development, and model comparison. For high-throughput production, connect directly to the provider.

Q4: How does INT4 quantization reduce memory while preserving quality?

FP32 weights use 32 bits per parameter. INT4 uses 4 bits, an 8 times reduction. A 7B model goes from roughly 28GB (FP32) to roughly 3.5GB (INT4). The accuracy tradeoff: quantization rounds each weight to the nearest representable value in the 16-step INT4 range. For most weights, this rounding error is small because weight distributions are bell-shaped and concentrated near zero. K-quant methods use per-group scale factors to minimize this error further. The result is models that fit in consumer RAM while performing close to full-precision.

Q5: How much RAM do you need to run a 7B model locally?

A 7B model at Q4_K_M requires roughly 4GB for weights plus 1 to 2GB for KV cache and overhead. Practical minimum: 6 to 8GB available RAM. Guide: 8GB RAM (MacBook Air M1/M2) runs 7B comfortably at 15 to 25 tokens per second using Apple Metal. 16GB handles 7B well or 13B at Q4_K_M. 32GB allows 13B comfortably or a try at 30B. 64GB handles 70B at Q4_K_M (roughly 40GB). Key rule: the model must fit entirely in RAM or VRAM for interactive speeds. Paging to disk is too slow for real use.

Q6: Compare Ollama, LM Studio, and OpenRouter: when would you choose each?

Ollama: CLI-first tool, runs a REST server at localhost:11434 with an OpenAI-compatible API. Best for developers, scripting, Jupyter notebooks, and production local inference. LM Studio: desktop GUI with drag-and-drop model loading, built-in chat interface, and a local OpenAI-compatible server. Best for non-developers and students exploring models visually. OpenRouter: cloud API gateway, no local install. Best for accessing frontier models through one API key, model comparison studies, or when your hardware cannot run the required model size. Decision: need privacy or offline use → Ollama or LM Studio. Need frontier model quality → OpenRouter or direct provider. Building production code → Ollama (stable REST API, Docker-friendly).

Q7: How does map-reduce work for documents that exceed the context window?

A single context window cannot fit a full book or large codebase. Map-reduce splits the document into chunks (map step), processes each chunk independently with the LLM, then combines results (reduce step). For summarization: map generates a summary of each chunk, reduce merges summaries into a final coherent output. For Q&A: map extracts relevant passages, reduce synthesizes the final answer. An alternative is the refine strategy: process chunk 1, pass its output plus chunk 2 to the LLM for refinement, repeat until done. Refine maintains cross-chunk coherence but is sequential and slower. Map-reduce is faster because chunks are processed in parallel. In LangChain: load_summarize_chain(chain_type="map_reduce").

Chapter 7

🧠 Prompt Engineering + DSPy

Session 06: Stop Guessing. Start Optimising.

"What if instead of writing the perfect prompt yourself, you let code find it for you?"

How to Write a Good Prompt

A well-written prompt has five parts. You don't need all five every time, but knowing them helps you debug when the model gives you the wrong answer.

Zero-Shot: The Gap Between Vague and Specific

Bad prompt:

Summarize this.

The model doesn't know: how long? What style? What to focus on?

Good prompt:

You are a business analyst.
Summarize the following customer feedback in exactly 3 bullet points.
Each bullet: main complaint + why it matters to the business.

Feedback: "The checkout took 10 minutes and the payment kept
failing. I had to try 4 times. Really frustrating."

Role + task + output format. The model has no room for misunderstanding.

Role Prompting: Give the AI a Job

The same question produces completely different answers depending on the role you set.

Without role:

Prompt: "Explain recursion."
Result: A dry textbook definition.

With role:

Prompt: "You are a patient tutor teaching a 16-year-old
who knows Python basics but has never heard of recursion.
Start with a real-world analogy. Then show the simplest
possible code example. Keep it under 150 words."

Result: An analogy about a camera inside a mirror, then
5 lines of Python, in exactly the right tone and length.

Why does this work? The model has seen millions of examples of teaching-style writing. Saying "patient tutor" activates those patterns.

Few-Shot: Show, Don't Tell

You give 2-3 examples of input and output before asking your real question. The model learns the exact format and labels you want.

Without examples (zero-shot), the model picks its own format:

Review: "Arrived late but product is great."
→ "Mixed sentiment" or "Neutral" or "3/5 stars" (you get something different every time)

With examples (the model locks into your format):

Review: "Fast delivery, great quality."       → Positive
Review: "Product broke after one day."        → Negative
Review: "Good price but slow shipping."       → Mixed
Review: "Arrived late but product is great."  → 

The model now knows: exactly three labels, one word, no explanation. You don't have to say any of that, because the examples show it.

Chain-of-Thought: Make It Show Its Working

Without CoT (the model guesses on reasoning tasks):

Q: Train A leaves at 9am at 80km/h, Train B at 10am at 100km/h,
   stations are 210km apart. When do they meet?

A: "10:45am"   ← often wrong

With CoT (just add "think step by step"):

Q: Same question. Think step by step.

A: At 10am, Train A has traveled 80km. Distance remaining: 130km.
   Combined closing speed = 80 + 100 = 180 km/h.
   Time to close 130km = 130/180 = 0.72h = 43 minutes.
   They meet at 10:43am.   ✓

Generating intermediate steps gives the model more context for each next token. It is especially powerful for math, logic, and classification with subtle distinctions. In DSPy, dspy.ChainOfThought adds this automatically.

Output Format: Tell It Exactly What You Want

If the response needs to go into your code, always lock down the format.

Unstructured (your code cannot reliably parse this):

Analyze this review and tell me the sentiment, the main topics,
and whether the customer would recommend it.

→ "The customer seems fairly happy overall, mentioning
   delivery speed and product quality. They would likely..."

Structured (your code does json.loads(response) and it works):

Reply with ONLY this JSON. No markdown. No explanation.
{
  "sentiment": "positive" | "negative" | "mixed",
  "topics": ["list", "of", "topics"],
  "would_recommend": true | false
}

One rule: if the LLM output goes into code downstream, always control the format.

The Problem with Hand-Written Prompts

You've been writing prompts like this:

"You are a helpful assistant. Classify the intent of this customer message.
 Message: {text}
 Intent options: {labels}
 Answer:"

Then tweaking it manually. Testing. Tweaking again.

This does not scale. You have no way to know if your prompt is optimal. You're guessing.

DSPy's answer: Define what you want in code. Let optimisers find the best prompt automatically.

DSPy: Programmatic Prompt Engineering

DSPy (Stanford NLP) treats prompts as learnable parameters, not hand-written strings.

Key concepts:

DSPy Signatures: Define the Task, Not the Prompt

class Classification(dspy.Signature):
    """Classify the customer message into one of the intent labels."""

    customer_message = dspy.InputField(
        desc="Customer message during customer service interaction"
    )
    intent_labels = dspy.InputField(
        desc="Labels that represent customer intent"
    )
    answer = dspy.OutputField(
        desc="A label best matching the customer's intent"
    )

# Chain-of-Thought: think step by step before answering
cot_predictor = dspy.ChainOfThought(Classification)

The docstring becomes the instruction. The fields become the prompt structure.

You define the task. DSPy writes the prompt.

The Three Optimisers: From Simple to Powerful

What BootstrapFewShot Actually Does

Your labelled training data
         │
   Run the LLM on each training example
         │
   Check: does the LLM's output pass the metric?
         │
   YES → keep this (input, output) pair as a demo
   NO  → discard
         │
   Inject only the PASSING demos into the final prompt

The key insight: Instead of hand-picking examples, DSPy runs your metric against the model's own generations and keeps the good ones automatically.

MIPROv2: Optimise the Instruction Itself

The most powerful DSPy optimiser: it searches over the instruction wording, not just the examples.

// From MiproV2Prompt.json, the optimised instruction found automatically
{
  "predict": {
    "signature": {
      "instructions": "Classify the customer's airline service query into
                       the correct intent category. Analyze the message
                       carefully to identify the primary action or
                       information need expressed..."
    }
  }
}

Nobody wrote that instruction. The optimiser found it by testing thousands of phrasings against the validation set.

Chain-of-Thought: Why "Think Step by Step" Works

Standard prompting:

Q: "I need to change my flight booking"
A: "flight_change"

Chain-of-Thought:

Q: "I need to change my flight booking"
Reasoning: "The customer is asking to modify an existing reservation.
            The key words are 'change' and 'flight booking'.
            This is a modification request, not a cancellation."
A: "flight_change"

CoT forces the model to show its work. Errors in reasoning are visible and correctable. Accuracy improves significantly on complex classification and reasoning tasks.

🎯 Interview Questions: Session 06

Q1: What are the five parts of a well-written prompt and why does each matter?

Role (who the AI should be): activates relevant writing patterns from training. "You are a senior data analyst" produces very different output than the same task with no role. Task (what exactly to do): be specific, for example "summarize in 3 bullet points" not just "summarize." Context (background it needs): "the audience has no technical background" changes every word choice. Examples (2 to 3 input/output pairs): the most powerful lever for locking in format and label names, the model copies your exact output structure. Output format: critical when code must parse the output, use "Reply ONLY with valid JSON, no markdown, no explanation." Most bad prompts fail because only the task is included.

Q2: What is Chain-of-Thought prompting and why does forcing reasoning help?

CoT prompting forces the model to generate intermediate reasoning steps before producing the final answer. Adding "think step by step" produces explicit reasoning tokens that give the model more context for each subsequent decision. Classic example: "Is 9.11 > 9.9?" Without CoT: "Yes" (wrong). With CoT: "9.11 means 0.11 after the decimal. 9.9 means 0.90. 0.11 < 0.90, so 9.11 < 9.9. No." (correct). CoT helps on multi-step math, logic puzzles, nuanced classification, and code generation. Does not help on simple factual lookups.

Q3: What is DSPy and how does it differ from writing prompts manually?

DSPy (Stanford NLP) treats prompts as learnable parameters rather than hand-written strings. You define a Signature (input/output schema with field descriptions), a metric (function that scores output quality), and a training dataset. An optimizer automatically searches for the best prompt, instruction wording, and few-shot examples. The model never sees your raw prompt: DSPy assembles the optimal one from your signature and the optimized instructions. Manual prompting requires expert iteration; DSPy automates it and is reproducible across model versions.

Q4: What is the difference between LabeledFewShot and BootstrapFewShot?

LabeledFewShot randomly picks k examples from your training set and injects them as few-shot demonstrations. No actual optimization: just in-context learning with randomly selected examples. Quality depends on which k examples happen to be picked. BootstrapFewShot runs the model on training examples, evaluates each prediction against your metric, and keeps only examples where the model was correct. The injected demos are guaranteed to pass your validation criterion, not random samples. This produces significantly higher quality signal than random selection. BootstrapFewShot requires no human-labeled outputs, only human-labeled inputs with a metric to evaluate outputs.

Q5: What can MIPROv2 optimize that BootstrapFewShot cannot?

BootstrapFewShot selects which demonstrations to include but cannot modify the instruction text itself. MIPROv2 searches over both the instruction wording AND the few-shot examples simultaneously. It uses Bayesian optimization to generate and evaluate candidate instruction variants, focusing the search on promising phrasings. MIPROv2 may find "let's work through this step by step to ensure accuracy" outperforms your hand-written "answer the following question" by several points in F1. Use BootstrapFewShot first; upgrade to MIPROv2 when you need further gains and have labeled validation data.

Q6: Give a concrete example where few-shot is strictly necessary over zero-shot.

Format consistency for downstream code parsing. Zero-shot "Classify sentiment as positive, negative, or mixed" produces "Positive", "positive", "POSITIVE", "The sentiment is positive", "I believe this is positive": all different strings, none reliably parseable by code. Few-shot with 3 examples ("Love it!" → "positive", "Awful!" → "negative") forces the model to lock into exactly one of three strings you control. Your if answer == "positive" check works 100% of the time. Rule: whenever downstream code must parse the output or you need consistent label strings for a classifier, few-shot is not optional. It is the only reliable approach.

Chapter 8

🔗 LangChain Fundamentals

Session 07: One Interface. Every Provider.

"You've called OpenAI. You've called Gemini. You've called Ollama. Why are you writing three different codebases?"

The Problem Before LangChain

Every provider has different:

# OpenAI
response = client.chat.completions.create(model=..., messages=[...])
answer = response.choices[0].message.content

# Anthropic / Claude
response = client.messages.create(model=..., messages=[...])
answer = response.content[0].text

# Gemini
response = client.models.generate_content(model=..., contents=...)
answer = response.text

Different authentication, different response shapes, different streaming APIs.

Every provider switch = rewrite a significant chunk of code.

LangChain: One Interface for Everything

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_ollama import ChatOllama

# ALL return an AIMessage with .content: identical interface
models = {
    "openai":  ChatOpenAI(model="gpt-4o-mini"),
    "claude":  ChatAnthropic(model="claude-haiku-4-5-20251001"),
    "gemini":  ChatGoogleGenerativeAI(model="gemini-2.5-flash-lite"),
    "llama":   ChatOllama(model="llama3.2:3b"),
}

for name, model in models.items():
    response = model.invoke(["Explain neural networks simply"])
    print(f"{name}: {response.content}")

Change one variable. Same code. Every provider.

ChatPromptTemplate: Structured Prompts

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful assistant."),
    ("human", "{question}"),
    MessagesPlaceholder(variable_name="history")
])

MessagesPlaceholder, injects the full conversation history at that position automatically.

Build the prompt once. Reuse it across any model, any session.

Conversation History: InMemoryChatMessageHistory

from langchain_core.chat_history import InMemoryChatMessageHistory

# Each user session gets its own history object
history = InMemoryChatMessageHistory()

# Messages are added automatically as the conversation progresses
history.add_user_message("What is self-attention?")
history.add_ai_message("Self-attention is a mechanism that...")
history.add_user_message("How is it different from RNNs?")

# history.messages → passed into MessagesPlaceholder in the prompt

The model sees the full conversation every turn, this is how chatbots remember.

Streaming: Token by Token

# Without streaming: user waits for full response
response = model.invoke(["Write me a short essay on AI"])

# With streaming: user sees tokens appear in real time
for chunk in model.stream(["Write me a short essay on AI"]):
    print(chunk.content, end="", flush=True)

Why streaming matters in production:

• Perceived latency drops dramatically, users see output immediately
• First token in ~300ms vs waiting 5–10s for full response
• Every major chat app (ChatGPT, Claude.ai) uses streaming

The LCEL Chain: Pipe Operator

from langchain_core.output_parsers import StrOutputParser

# Build a chain: prompt → model → parse to string
chain = prompt | model | StrOutputParser()

# Invoke it
answer = chain.invoke({
    "question": "What is LangChain?",
    "history": history.messages
})

| pipes the output of one component into the next.

This is LCEL (LangChain Expression Language), composable, readable, provider-agnostic.

🎯 Interview Questions: Session 07

Q1: What is LCEL and what does the pipe operator do technically?

LCEL (LangChain Expression Language) is a declarative way to compose LangChain components into pipelines using the | operator. Every component implements the Runnable interface. The | operator chains Runnables: output of the left component becomes input to the right. A chain like prompt | llm | StrOutputParser() means: (1) prompt formats the input dict into a messages list, (2) llm sends the messages to the provider and returns an AIMessage, (3) StrOutputParser extracts the .content string. The chain itself is also a Runnable, so you can pipe it into more components. Switching providers means replacing only the llm component without touching prompt or parser.

Q2: What is ChatPromptTemplate and what is MessagesPlaceholder used for?

ChatPromptTemplate defines the message structure sent to the LLM as a template with typed variable slots. It takes a list of message tuples: [("system", "You are a {persona}"), ("human", "{question}")]. At runtime, .format_messages() fills in the variables and produces the final messages list. MessagesPlaceholder is a special slot for injecting a dynamic list of messages, specifically used for conversation history. Instead of manually building message dicts for every prior turn, you define MessagesPlaceholder("history") once in the template and pass the history object at runtime. This cleanly separates template definition from runtime data.

Q3: How does streaming work in LangChain and why is it critical for production?

chain.stream(input) returns a Python generator that yields text chunks as the LLM produces them, typically word by word. chain.invoke(input) waits for the complete response. Why streaming matters: users see the first word in under 1 second instead of waiting 5 to 20 seconds for a blank screen. Perceived speed is 5 to 10 times better even if total generation time is the same. Users can stop generation early if the response is going in the wrong direction. In Streamlit: st.write_stream(chain.stream(...)) handles the UI update automatically. The same .stream() call works across all providers: LangChain normalizes provider-specific streaming APIs.

Q4: What is the Runnable interface and what are its 5 core methods?

The Runnable interface is the standard protocol every LangChain component implements, enabling composition with |. Five core methods: (1) .invoke(input): synchronous single call, use for simple scripts. (2) .stream(input): synchronous streaming, yields output chunks, use for real-time UIs. (3) .batch(inputs): processes a list of inputs efficiently, often in parallel, use for bulk processing. (4) .ainvoke(input): async version of invoke, use in FastAPI. (5) .astream(input): async streaming generator, use in async web servers. Every component that implements these methods works with |: prompts, LLMs, parsers, and custom RunnableLambda wrappers.

Q5: How would you switch a production chain from GPT-4 to Claude with minimal changes?

Because all LLM wrappers implement the same Runnable interface, you change exactly one line: replace ChatOpenAI with ChatAnthropic and update the model name. The chain prompt | llm | parser is unchanged. For production-grade switchability, define a factory function that reads from an environment variable: def get_llm(provider=None). Then switching in production means changing one environment variable, no code deployment needed. Caveat: models differ in context length, function calling format, and system message handling. Always test your specific use case after switching.

Q6: What is RunnableWithMessageHistory and how does it simplify multi-user chat?

Manual approach: maintain a message list, append each user and AI message, pass the full list to the chain on every turn. Works but requires boilerplate for every app, and multi-user isolation needs manual session keying. RunnableWithMessageHistory wraps any LCEL chain to automate all of this. You provide a get_session_history_fn that maps session_id to a history store. On each call, it retrieves history for the session, injects it into the MessagesPlaceholder, runs the chain, and appends the new turn automatically. Multi-user isolation: pass config={"configurable": {"session_id": user_id}}. Each user ID gets completely isolated history. Session persistence across restarts requires a Redis or database backend.

The Full Picture: One Diagram

Raw Text
   │  "How does Google find a result?"
   ▼
S00  Inverted Index + TF-IDF        Keyword matching and ranking
   │  ⚠️ "car" ≠ "automobile"
   ▼
S01  Word Embeddings                 Semantic meaning in vector space
   │  ⚠️ "bank" one vector, context ignored
   ▼
S02  Transformers + Self-Attention   Context-aware representations
   │  ⚠️ Training from scratch = weeks + $$$$
   ▼
S03  Pre-trained GPT → RLHF          Aligned, helpful intelligence
   │  ⚠️ How does token-by-token generation actually work?
   ▼
S04  BPE + Temperature + Top-K/P    Inside the generation pipeline
   │  ⚠️ Cloud APIs cost money and require internet
   ▼
S05  Ollama + OpenRouter             You own the infrastructure
   │  ⚠️ We're still writing prompts by hand, guessing what works
   ▼
S06  DSPy + CoT + BootstrapFewShot  Prompts become learnable parameters
   │  ⚠️ Every provider needs different code to call
   ▼
S07  LangChain                       One interface. Every provider.

Every line is a problem. Every arrow is a solution.

The Interview Cheat Sheet: One Slide

What's Coming Next

Already Covered: All 8 Sessions Complete

What's Coming with Mohamed

You Are Not Beginners Anymore

Eight sessions ago you asked: "How does Google find a result?"

Now you can explain:

• How a search engine scores and ranks documents
• How a neural network learns that "car" and "automobile" mean the same thing
• How Transformers read context that RNNs could not
• How GPT went from predicting text to passing the bar exam
• How tokens are built and how sampling controls generation
• How to run GPT-quality AI on your own machine for free
• How to let code write and optimise prompts instead of guessing
• How to call any LLM with the same code using LangChain

This foundation is permanent. Every LLM system in the world, no matter how advanced, sits on exactly these concepts. You understand the ground floor.

Next session, we build higher.