🧠

Sessions 00–07 Full Recap 8 Sessions · 1 Story

Zero to GenAI Engineer

One continuous story, from how search engines rank documents to how we automate prompt engineering. Every session solved a gap left by the previous one.

8

Sessions

30+

Concepts

48

Interview Q&As

1

Career Goal

↓scroll to explore

📊 Session Dependency Map

8 sessions

S00 · Foundation

🔍 Keyword Search, TF-IDF, Inverted Index

⚠ "car" and "automobile" are completely unrelated to TF-IDF

S01 · Semantics

🧭 Word Embeddings, Word2Vec, GloVe, Cosine Similarity

⚠ Words have meaning but no long-range context

S02 · Architecture

🧠 Transformers, Attention, Multi-Head, Encoder/Decoder

⚠ Great architecture, but only trained to predict text

S03 · Intelligence

🚀 GPT Evolution, RLHF, ChatGPT, Alignment, Hallucination

⚠ Powerful, but how do we use it efficiently and cheaply?

S04 · Mechanics

🔢 BPE Tokenization, Temperature, Top-K, Top-P

⚠ Cloud APIs cost money and send your data to third parties

S05 · Infrastructure

💻 Local LLMs, Ollama, LM Studio, OpenRouter

⚠ We can run models but we're still guessing at good prompts

S06 · Optimization

✍️ Prompt Engineering + DSPy + CoT + MIPROv2

⚠ Different providers have different SDKs, too much boilerplate

S07 · Unification

🔗 LangChain, One interface for every model provider

✅ After 8 sessions, you can

Build a keyword search engine from scratch
Explain self-attention to a colleague
Trace the full history of GPT from 2017 to today
Calculate token costs before calling an API
Run an LLM on your own laptop, for free
Optimise prompts automatically with DSPy
Call any LLM provider with a single LangChain interface

💡

That is not a small thing. Let's make sure it sticks.

🧠 The story in one line

S00 → S01 → S02 → S03 → S04 → S05 → S06 → S07

Session	Central Question
S00	How does Google decide which result to show?
S01	How does a computer understand the meaning of a word?
S02	How does the same word mean different things in context?
S03	How did next-word prediction become ChatGPT?
S04	How does an LLM decide which word to generate next?
S05	How do we run and own the infrastructure ourselves?
S06	How do we stop guessing at good prompts?
S07	How do we call any LLM with the same code?

🔍

Session 00 · Foundation

The Foundation of Everything

Before embeddings. Before Transformers. Before ChatGPT. This is how computers understood text for 50 years, and why it wasn't enough.

📋 What you'll recall in this section

Concept	What It Is	Why It Matters
Inverted Index	Word → list of documents map	Makes search sub-millisecond
TF-IDF	Word importance score per document	Ranks documents by relevance
Semantic Gap	"car" ≠ "automobile" in this model	The gap that led to embeddings

⚠️

The MISSING: TF-IDF is purely lexical, it matches exact words only. No understanding of synonyms, context, or meaning. This gap is what S01 solves.

📂 Inverted Index

Instead of scanning every document per query, pre-build a lookup: word → document list. O(1) retrieval regardless of corpus size.

Word	Documents containing it
"python"	Doc 1, Doc 5, Doc 12, Doc 20
"neural"	Doc 3, Doc 7, Doc 12
"model"	Doc 1, Doc 3, Doc 7, Doc 9
"training"	Doc 5, Doc 9, Doc 20

ℹ️

Query "python training" → intersect lists → Doc 5, Doc 20. Sub-millisecond.

📊 TF-IDF Scoring

TF: Term Frequency

How often does the word appear in this document?

IDF: Inverse Doc Frequency

How rare is the word across all documents?

TF-IDF(word, doc) = TF(word, doc) × IDF(word) -- "the" → TF=high, IDF≈0 → score near 0 (too common) -- "GAN" → TF=medium, IDF=high → high score (rare, relevant)

⚠️

Hard limit: "automobile" and "car" are completely unrelated in TF-IDF. Exact word matching only, no semantic understanding.

📊 IDF in Practice: why common words score near zero

IDF = log(total docs / docs containing word)

TF-IDF = TF × IDF. "the" appears in 200 of 200 docs → IDF = log(200/200) = 0 → score = 0, no matter how often it appears. "RLHF" appears in 3 of 200 docs → IDF = log(200/3) ≈ 4.9 → strong relevance signal. This is why stopwords automatically get filtered out without an explicit stopword list.

💬 Expand each to reveal the answer

7 questions

A data structure that maps each unique word to the list of documents containing it. Instead of scanning every document for every query, you look up the word directly and get the document list instantly, sub-millisecond lookup regardless of corpus size. Query "python training" → look up "python" (Docs 1,5,12,20) and "training" (Docs 5,9,20) → intersect → [Doc 5, Doc 20]. This is the core of how Elasticsearch, Lucene, and every search engine indexes content. Inverted indexes are the foundation of every major search engine; Google added PageRank on top to score results by link authority.

TF (Term Frequency) measures how often a word appears in a specific document. IDF (Inverse Document Frequency) = log(total_docs / docs_containing_word). Words in every document (like "the") get IDF = log(1) = 0 and are automatically filtered. Rare words get high IDF scores. TF × IDF is high for relevant, rare words and near zero for common filler words. Example: "neural" appears 5 times in Doc A and once in Doc B. Doc A is more relevant for the query "neural." But "the" appearing 500 times is irrelevant because IDF crushes the score to near zero.

TF-IDF is purely lexical: it matches exact character strings. "car" and "automobile" have zero computed relationship. It has no understanding of synonyms, context, or meaning. This "semantic gap" was solved by word embeddings (Word2Vec, GloVe) which map words to dense vectors in a high-dimensional space where semantically similar words land geometrically close to each other. Modern production systems use hybrid search: BM25 for exact keyword recall plus vector search for semantic understanding, because each catches what the other misses.

BM25 (Best Match 25) extends TF-IDF with two critical fixes: (1) Term saturation: in raw TF-IDF, a word appearing 100× scores 10× higher than appearing 10×, which is misleading. BM25 adds a saturation parameter k1 (typically 1.5). Beyond a certain frequency, additional occurrences have diminishing returns. (2) Document length normalization: parameter b (typically 0.75) penalizes long documents, because finding "python" once in a 100-word document is more significant than finding it once in a 50,000-word document. BM25 is the default algorithm in Elasticsearch and Lucene and remains a strong baseline even against neural retrievers. In modern RAG systems, BM25 + vector search together outperform either alone. This is called hybrid search.

IDF = log(N / df), where N = total documents and df = documents containing the word. If a word appears in every document: IDF = log(N / N) = log(1) = 0. The word contributes nothing to ranking regardless of how many times it appears. This is exactly correct: "the", "is", "a" carry zero discriminating information about what a document is about. This property means TF-IDF has automatic stopword filtering built into the math, with no explicit stopword list needed. Conversely, a word appearing in 3 of 1000 documents has IDF = log(1000/3) ≈ 5.8, making it a very strong relevance signal. This is why "RLHF" or "quantization" would dominate a TF-IDF ranking in a general corpus.

A complete search system architecture: (1) Ingestion pipeline: crawl or receive documents → tokenize text → build/update the inverted index offline (write-heavy, can be batched). (2) Query time: receive user query → tokenize → look up each token in the inverted index (O(1)) → retrieve candidate document lists → intersect/union them → score candidates with BM25 or TF-IDF → return top-K ranked results. (3) Improvements: add vector index alongside inverted index for semantic search (ANN search using FAISS or Pinecone), rerank top-N results using a cross-encoder for precision, cache frequent queries. Real systems like Elasticsearch combine the inverted index with a distributed storage layer for horizontal scaling. This two-stage "retrieve then rerank" pattern is the same architecture used in modern RAG pipelines.

Bag-of-words completely discards word order, so "dog bites man" and "man bites dog" produce identical vectors. It creates sparse, high-dimensional representations with vocabulary sizes of 100,000 or more, expensive to store and compute. It treats all words as independent: "good" and "excellent" are as unrelated as "good" and "airplane." The word "bank" gets the same representation whether the context is financial or geographic. These limitations mean bag-of-words handles recall (finding documents containing the right words) but fails at understanding meaning. Word embeddings solve the semantic gap by mapping words to dense vectors where similar meanings cluster together. Transformers then solved the context-dependence gap by making representations sensitive to surrounding words.

🧭

Session 01 · Semantics

Words as Numbers

What if every word had a location in a 300-dimensional space, and words with similar meanings lived close together? Meaning becomes arithmetic.

📐 The Famous Equation

king − man + woman = queen

No rules were hand-coded. The model learned this purely from text co-occurrence. Meaning is captured as geometry, relationships become vector arithmetic.

📋 Three Algorithms

Model	Year	Key Idea
Word2Vec	2013	Learns from a sliding context window. "Know a word by its company."
GloVe	2014	Learns from global co-occurrence matrix across entire corpus (not local context window). Strong on word analogy tasks like king−man+woman=queen.
FastText	2016	Trains on character n-grams. Handles typos and unknown words.

🗺️ Word Vector Space: similar meanings cluster together

Conceptual visualization

📏 Cosine Similarity vs Euclidean Distance

❌ Euclidean Distance

Measures the straight-line distance between two points. Problem: a word appearing 1000× vs 10× creates different vector magnitudes, but the meaning is the same. Euclidean distance calls them different.

✅ Cosine Similarity

Measures only the angle between two vectors. Ignores magnitude entirely. "cat" appearing 10× or 1000× still points in the same direction, same meaning. Range: -1 (opposite) to +1 (identical).

💬 Expand each to reveal the answer

7 questions

A word embedding maps each word to a dense vector (e.g. 300 dimensions) where semantic relationships are preserved as geometry. One-hot encoding maps each word to a sparse binary vector, with a 1 at the word's vocabulary index and 0 everywhere else. Problems with one-hot: (1) No semantic relationship: "king" and "queen" are as far apart as "king" and "banana." (2) Dimensionality explosion: a 100k-word vocabulary needs 100k-dimensional vectors. Word embeddings solve both: "king" and "queen" have high cosine similarity, and 300 dimensions can represent the entire vocabulary meaningfully. Pre-trained embeddings (word2vec, GloVe) transfer learned meaning to downstream tasks without retraining from scratch.

Word2Vec uses self-supervised learning on a prediction task derived from the text structure itself, no human annotation needed. Skip-gram: given the center word "brown" in "the quick brown fox jumps," predict the surrounding context words. CBOW: given context words, predict the center word. The model trains a shallow neural network on billions of such examples. Words that appear in similar contexts (e.g., "dog" and "cat" often appear near "pet," "food," "vet") end up with similar weight vectors. The key insight from Wittgenstein's philosophy: "the meaning of a word is its use in the language." The network weights converge to vectors that encode usage patterns, which turn out to encode meaning.

Word vectors have magnitudes that depend on word frequency during training. A very common word like "dog" will have larger weights than a rare word like "dachshund," even if they're semantically related. Euclidean distance is magnitude-sensitive: two vectors pointing in the same direction (identical meaning) but with different lengths would be measured as different. Cosine similarity measures only the angle between vectors, completely ignoring magnitude. Two words with the same semantic meaning (same directional orientation) have cosine similarity = 1.0 regardless of their frequencies. Formula: cos(A,B) = (A·B) / (|A||B|). Range: -1 (opposite meaning) to +1 (identical direction/meaning). In practice: "dog" and "puppy" → cosine ≈ 0.85, "dog" and "quantum" → cosine ≈ 0.1.

Skip-gram: given the center word (e.g., "brown"), predict the surrounding context words (e.g., "quick", "fox"). Each center word generates multiple training examples. CBOW (Continuous Bag of Words): given the context words, predict the center word, the inverse task. Performance difference: Skip-gram performs better for rare words, because each training step focuses deeply on one word's context relationships. CBOW is 3-5× faster to train and performs better for frequent words, because it averages many context signals. In practice: Skip-gram with Negative Sampling (SGNS) is the industry standard for high-quality embeddings and produces the famous analogy results (king − man + woman ≈ queen). CBOW is used when training speed matters more than quality on rare vocabulary.

The naive Skip-gram training objective requires a softmax over the entire vocabulary (50k-100k words) for every training step: P(context_word | center_word) = exp(v_o · v_c) / Σ exp(v_w · v_c). That denominator sum over all vocabulary words makes each update extremely expensive (O(vocab_size) operations). Negative sampling replaces the full softmax with a binary classification task: for each real (center, context) pair, randomly sample k "noise" words (typically k=5-20) that should NOT be predicted. Train a binary classifier: is this pair real or noise? This reduces computation from O(vocabulary) to O(k) per update, a 5000-20000x speedup. The model still learns meaningful embeddings because distinguishing real context from random noise forces it to capture genuine co-occurrence patterns.

GloVe (Global Vectors, Stanford 2014) pre-computes a global co-occurrence matrix: for every word pair (i, j), count how many times they appear within a context window across the entire corpus. Then train vectors so that v_i · v_j ≈ log(X_ij), the log co-occurrence count. Word2Vec learns from local context windows one sentence at a time. It sees global patterns only indirectly through many local examples. GloVe explicitly encodes global corpus statistics. Result: GloVe tends to perform particularly well on word analogy benchmarks (king − man + woman = queen) because it explicitly models the ratio of co-occurrence probabilities. Pre-trained GloVe vectors (glove.6B.300d, trained on Wikipedia + Gigaword, 840B tokens) are freely available and widely used as a strong baseline. In practice, both Word2Vec and GloVe have been largely superseded by contextual embeddings (BERT) for most tasks.

FastText (Facebook, 2016) trains on character n-grams rather than whole words. "Playing" becomes ["pla", "lay", "ayi", "yin", "ing"]. The word vector is the sum of its subword piece vectors. Key advantage: FastText generates a vector for words it never saw during training by composing character pieces. Word2Vec assigns an unknown vector to "playin" (a typo); FastText recognizes the subword overlap and produces a reasonable embedding. Use FastText when: your data has typos or misspellings, you work with morphologically rich languages (German, Turkish, Finnish), you have domain-specific jargon that won't appear in a general pre-trained vocabulary, or you work with code tokens. Trade-offs: larger model size (stores subword vectors), slower inference. For most production tasks today, sentence-transformers (BERT-based) outperform both FastText and Word2Vec, but FastText remains a fast, strong baseline with no GPU required.

🧠

Session 02 · Architecture

The Transformer Architecture

"Attention Is All You Need" (2017) introduced self-attention, which lets every word look at every other word simultaneously. This changed everything.

🔑 Self-Attention: Q, K, V

Component	Analogy	Role
Query (Q)	Your search question to the library	What this token is looking for
Key (K)	The label on each library book	What this token offers to others
Value (V)	The book's actual content	What this token contributes when attended to

Attention = softmax(Q·Kᵀ / √d_k) × V -- d_k = head dimension (e.g. 512-dim model ÷ 8 heads = d_k 64) -- Score = how well Q matches each K -- Output = weighted average of V values

🏗️ BERT vs GPT

🔵 BERT (Encoder only)

Reads the whole sentence at once (bidirectional). Good at understanding, classification, named entity recognition. Cannot auto-regressively generate text: the bidirectional mask processes all tokens simultaneously, making sequential left-to-right generation architecturally undefined.

🟢 GPT (Decoder only)

Reads left-to-right only (masked). Good at generation, text, code, answers. Cannot see future tokens while predicting.

ℹ️

Positional Encoding: Self-attention has no built-in sense of order. PE injects position info using sine/cosine functions so the model knows which token came first.

🏗️ Transformer Architecture (Original Paper)

Vaswani et al. 2017 · Wikimedia CC BY-SA

Transformer encoder-decoder architecture diagram from Attention Is All You Need

Left = Encoder (BERT). Right = Decoder (GPT). The Nx means these blocks are stacked N times.

⚙️ Single Attention Head: Q/K/V in Detail

Wikimedia CC BY-SA 4.0

Process of a single attention head showing Q K V transformation and weighted output

Each token projects to Q, K, V vectors. Attention = softmax(QKᵀ/√d) × V.

💬 Expand each to reveal the answer

6 questions

Every token generates three learned linear projections: Q (Query) = what this token is searching for. K (Key) = what this token offers as a label to others. V (Value) = the actual content this token contributes when attended to. Math: (1) Compute raw scores: score_ij = Q_i · K_j (dot product, O(n²) pairs). (2) Scale: divide by √d_k (head dimension) to prevent softmax saturation. (3) Softmax to get attention weights (sum to 1). (4) Weighted sum of V vectors: output_i = Σ softmax_ij × V_j. In the sentence "The bank by the river," the word "bank" attends strongly to "river" (clarifying it means riverbank, not financial institution). This is attention in action, resolving ambiguity through context.

BERT uses the Transformer Encoder only. Every token attends to all other tokens (bidirectional). Trained with Masked Language Modeling: 15% of tokens are masked, the model predicts them. Excellent contextual understanding: "bank" in "river bank" vs. "bank account" gets different embeddings. Cannot auto-regressively generate text: the bidirectional mask processes all tokens simultaneously, making sequential left-to-right generation architecturally undefined. Use BERT for: classification, NER, extractive Q&A, sentence similarity. GPT uses the Decoder with causal (masked) self-attention, where each token attends only to previous tokens. Trained with next-token prediction. Autoregressive generation: one token at a time, each output fed back as input. Use GPT for: text generation, code, chat, summarization.

Multi-head attention runs h independent attention computations in parallel (GPT-3: 96 heads, BERT-base: 12 heads, GPT-2: 12 heads). Each head gets a projected subspace of Q, K, V (dimension d_k = d_model / h). All heads run independently and their outputs are concatenated and linearly projected back to d_model. Why it helps: a single attention head must collapse all relationship types into one pattern. Multiple heads can simultaneously specialize. One might track syntactic dependencies ("subject → verb"), another semantic similarity, another coreferences ("it → the model"). The key insight: multi-head attention has the same compute cost as single-head (each head is proportionally smaller), yet provides qualitatively richer representations by attending to multiple relationship types simultaneously.

The dot product Q·K is the sum of d_k multiplications of random variables. As d_k grows, the expected variance of the dot products grows proportionally to d_k. For large d_k (e.g., 64), raw dot products become very large in magnitude. When large values enter softmax, it saturates: one element gets probability ≈1, all others get ≈0. In this saturated regime, gradients drop to near zero. The model gets stuck. Dividing by √d_k normalizes the dot products to approximately unit variance regardless of d_k, keeping softmax numerically stable with flowing gradients. This is the "scaled" in "Scaled Dot-Product Attention" from the original paper (Vaswani et al. 2017). For a 512-dim model with 8 heads: d_k = 64, √64 = 8, so divide all scores by 8.

Self-attention treats the input as a set, so "The cat sat" and "sat cat The" would produce identical attention patterns because attention is permutation-invariant. RNNs process tokens sequentially (one by one), so position is implicit in the order of processing. Transformers process all tokens in parallel, so position must be injected explicitly. The original paper adds sine/cosine waves: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)). Modern variants: BERT uses learned positional embeddings (trainable, one per position). Llama/Mistral/GPT-NeoX use RoPE (Rotary Position Embedding), which rotates Q and K vectors by position-dependent angles during attention, rather than adding PE to embeddings. RoPE generalizes better to sequence lengths not seen during training.

Each Transformer block has two sublayers: (1) Multi-Head Self-Attention, which handles relationships between tokens. (2) Feed-Forward Network (FFN), applied independently to each token. FFN(x) = max(0, xW₁ + b₁)W₂ + b₂. The inner hidden dimension is typically 4× the model dimension (512 → 2048 → 512). While attention routes information across the sequence, the FFN is where most factual knowledge is believed to be stored. Research ("Locating Factual Associations in GPT", 2022) traced factual claims like "Paris is the capital of France" to specific FFN layer weights. The two sublayers are complementary: attention gathers context from the whole sequence, FFN applies position-wise transformations and retrieves memorized knowledge. Both are wrapped with LayerNorm and residual connections.

🚀

Session 03 · Intelligence

From Text Predictor to ChatGPT

The same Transformer architecture. Trained on more data. Then fine-tuned with human feedback. That's the complete story of how we got ChatGPT.

📅 GPT Evolution Timeline

🌱

2018

GPT-1

117M params. Proved pre-train + fine-tune works.

📝

2019

GPT-2

1.5B params. "Too dangerous to release."

⚡

2020

GPT-3

175B params. In-context learning at scale.

🎯

2022

InstructGPT

RLHF added. Follows instructions, not just predicts.

💬

Nov 2022

ChatGPT

100M users in 2 months. The product is born.

🔄 RLHF Explained

Generate multiple responses to the same prompt
Human rankers order them from best to worst
Train a reward model on those human rankings
Use RL (PPO) to update GPT to maximise the reward, with a KL-divergence penalty against the original model to prevent reward hacking and output collapse
Result: a model that's helpful, harmless, and honest

⚠️ The Alignment Problem

⚠️

Hallucination: Base GPT optimises for "predict next token", not "say true things." It generates plausible-sounding text regardless of factual accuracy.

🛡️

Mitigations: RLHF, Constitutional AI, DPO, RAG (provide ground truth in the prompt), fine-tuning on factual data.

🔄 RLHF Process Diagram: how human feedback reshapes the model

Wikimedia Commons · CC BY-SA 4.0

RLHF diagram showing the three-stage process: supervised fine-tuning, reward model training, and RL optimization

Stage 1: Supervised fine-tuning on human-written responses.

Stage 2: Train a reward model, humans rank multiple responses from best to worst. The reward model learns to predict human preference.

Stage 3: RL (PPO) fine-tunes the language model to maximise the reward model's score. This is what makes GPT say "I don't know" instead of making something up.

💬 Expand each to reveal the answer

7 questions

In-context learning is the ability to perform new tasks from examples given directly in the prompt, with zero parameter updates or fine-tuning. GPT-3 (2020, 175B parameters) demonstrated this at scale for the first time. You give 2-3 input/output examples (few-shot) and the model generalizes to new inputs without any gradient updates. This was genuinely surprising. Prior models needed explicit fine-tuning for every new task. The emergent behavior is believed to come from the model effectively doing implicit gradient descent in its forward pass over the provided examples. Practical implications: you can turn a general-purpose LLM into a specialized classifier, extractor, or formatter purely through prompting, with no labeled training data or GPU training required. This is the foundation of prompt engineering as a discipline.

RLHF (Reinforcement Learning from Human Feedback) aligns model outputs with human preference. Three stages. Stage 1: Supervised Fine-Tuning (SFT): start from a base GPT, fine-tune on high-quality human-written demonstrations of good responses. Produces a model that follows instructions but imperfectly. Stage 2: Reward Model Training: generate multiple responses to the same prompt, have human raters rank them from best to worst. Train a separate model (the reward model) on these preference comparisons. The reward model learns to predict human preference scores. Stage 3: RL with PPO: use Proximal Policy Optimization to update the SFT model to maximize the reward model's score, with a KL-divergence penalty against the original SFT model to prevent reward hacking and output collapse (the model learns to produce absurd text that tricks the reward model). This transforms GPT-3 → InstructGPT → ChatGPT. The KL penalty is critical: without it, the model exploits reward model weaknesses.

Hallucination is when a model confidently states something factually false: a fabricated citation, a wrong date, a person who doesn't exist. It happens because LLMs are trained to predict the most likely next token, not to verify factual accuracy. They have no mechanism to distinguish "I know this" from "I'm generating a plausible-sounding completion." Three types: (1) Factual hallucination: wrong facts stated with confidence. (2) Temporal hallucination: events or information outside the training cutoff presented as current. (3) Intrinsic hallucination: claims contradicted by the source document in the prompt. Main mitigations: RAG (ground answers in retrieved documents, force the model to cite sources), Constitutional AI (training-time constraints that reduce confident false claims), fact-checking pipelines, and calibrated uncertainty expressions in system prompts ("If you don't know, say you don't know").

DPO (2023, Stanford) achieves the same alignment goal as RLHF but eliminates both the reward model and the RL training loop. RLHF requires 3 separate training stages: SFT → reward model → PPO. This is expensive, unstable, and requires careful hyperparameter tuning of the RL phase. DPO's key insight: the mathematically optimal RLHF policy has a closed-form solution. You can reparameterize the reward in terms of the language model itself, converting the RL problem directly into a supervised classification problem on preference pairs. Given (prompt, preferred_response, rejected_response), DPO trains the LM to increase the likelihood ratio of preferred/rejected responses with a built-in KL constraint. DPO is: simpler (no separate reward model to train), more stable (no RL instability), computationally cheaper, and achieves comparable alignment quality to RLHF. Most modern open-source aligned models (Llama 2-3 Chat, Mistral Instruct, Phi-3) use DPO or its variants (IPO, SimPO) rather than RLHF.

Constitutional AI (CAI, Anthropic 2022) addresses the scalability bottleneck of RLHF: human labelers must evaluate every output. CAI replaces human safety evaluators with the model itself, guided by a written "constitution" of principles. Process: (1) Generate a potentially harmful response from an unaligned model. (2) Use the same LLM (or a stronger one) to critique its own response against constitutional principles: "Is this response harmful? How could it be revised to avoid harm?" (3) Generate a revised response based on the self-critique. (4) Use these (prompt, revision) pairs as SFT training data, and preference pairs for reward model training. This enables scalable supervision: principles like "avoid assisting with illegal activities," "don't provide instructions for creating weapons," and "be honest about uncertainty" can be applied to millions of examples without human evaluation of each one. Claude's alignment training uses Constitutional AI as a core component.

Emergent abilities are capabilities that appear suddenly at a certain model scale, absent in smaller models and present in larger ones, without being explicitly trained for. Examples: Chain-of-thought reasoning (appears around 100B parameters), three-digit arithmetic, multi-step word problems. The defining feature is non-linearity: a model at 7B parameters might score 5% on a task, and a 70B model suddenly scores 85%, not a gradual improvement. Why it matters for engineers: (1) You cannot predict capabilities by extrapolating from smaller model benchmarks. (2) A company with only 7B models cannot know whether the 70B model can solve their problem. (3) Scaling laws predict loss, but not capability emergence. The "emergent abilities" paper (Wei et al. 2022) is controversial. Some argue emergence is an artifact of measuring with discontinuous metrics. For practical ML engineering: when a task doesn't work with a 7B model, try a larger model before concluding LLMs can't solve it. Many "unsolvable" tasks at 7B become trivial at 70B.

GPT-1 (2018): 117M parameters, 12 Transformer decoder layers, trained on roughly 1B tokens from 7,000 books. Proved that unsupervised pre-training followed by fine-tuning works. Required explicit fine-tuning for each downstream task. GPT-2 (2019): 1.5B parameters (13x larger), trained on WebText (40GB of curated web text, roughly 10x more data). Same decoder-only architecture, just much bigger. Demonstrated zero-shot task completion with no fine-tuning: just a prompt. OpenAI initially withheld the full model over misuse concerns, then released it 9 months later. GPT-3 (2020): 175B parameters (117x GPT-1's size), trained on 570GB of filtered internet text. 96 Transformer layers, 96 attention heads. In-context few-shot learning became a core capability: show 3 examples in the prompt and the model generalizes with zero gradient updates. One model could write code, translate, summarize, and answer questions from prompt examples alone. The pattern across all three: scale (parameters plus data) was the primary lever. Architecture barely changed; capabilities emerged from scale.

🔢

Session 04 · Mechanics

Tokens, Temperature, and Sampling

How text becomes token IDs, how IDs become probabilities, and how probabilities become the next word. This controls cost, randomness, and output quality.

🔤 BPE Tokenization

BPE starts with characters and repeatedly merges the most frequent adjacent pairs into a vocabulary of ~50k-100k subwords.

-- "unbelievable" → 3 tokens (GPT-4 cl100k_base) "un" + "believ" + "able" → [3456, 12890, 7823] -- Cost formula (approximate) ~4 chars = 1 token 1000 tokens ≈ 750 words cost = (input_tokens + output_tokens) × price

Token visualisation of "Hello, how are you?"

Hello

,

how

are

you

?

🌡️ Temperature and Sampling

Parameter	What It Does	Typical Value
Temperature	Scales logits before softmax. Low = focused. High = creative.	0 → greedy, 0.7 → balanced
Top-K	Keep only the K most likely tokens. Cuts off the long tail.	40 or 50 typical
Top-P (Nucleus)	Keep tokens until cumulative probability ≥ P. Dynamic cut-off.	0.9 or 0.95

ℹ️

Correct order: Apply Temperature → Softmax → Top-K → Top-P → Sample. Reversing Top-P and Temperature gives wrong results.

🌡️ How Temperature Changes the Probability Distribution

Visual diagram

💬 Expand each to reveal the answer

6 questions

BPE (Byte-Pair Encoding) starts with individual characters and iteratively merges the most frequent adjacent pair until reaching a target vocabulary size (~50k-100k subwords). Why not word-level: (1) unknown word problem: any word not seen in training becomes [UNK]; (2) morphological explosion: every inflected form needs a separate entry. Why not character-level: produces sequence lengths 4-5× longer, making attention computation (O(n²)) 16-25× more expensive. BPE's advantage: handles new words through composition ("unbelievable" → ["un", "believ", "able"]), works across all languages without language-specific rules, and balances sequence length vs. vocabulary size. GPT-4 uses ~100k BPE tokens. Byte-level BPE (GPT-2 onwards) operates on raw bytes, guaranteeing no unknown tokens. Every possible byte sequence can be represented.

Temperature T scales raw logits before softmax: logit_i / T. High T (>1.0) flattens the distribution, more tokens get comparable probabilities, output is more creative/diverse. Low T (<1.0) sharpens it: the top token dominates, output is more focused/deterministic. T=0 is greedy decoding: always pick argmax(logits), deterministic, no sampling. Correct pipeline order: (1) Divide logits by T to scale. (2) Softmax → convert to probabilities summing to 1. (3) Top-K → zero out all but the K most probable tokens (e.g., K=40). (4) Top-P → further filter: keep the smallest set of tokens whose cumulative probability ≥ P (e.g., P=0.95). (5) Renormalize remaining probabilities. (6) Sample from the filtered, renormalized distribution. Reversing Top-P and Top-K, or applying temperature after softmax, gives incorrect results. Most production APIs accept temperature + top_p + top_k and apply them in this order internally.

3,000 tokens/query × 10,000 queries/day = 30,000,000 input tokens/day. At $5 per 1M tokens = $150/day for input tokens alone. × 30 days = $4,500/month in input costs. Add output tokens (say 500 tokens/query × 10k/day = 5M output tokens/day, GPT-4o output = $15/1M → $75/day → $2,250/month). Total: ~$6,750/month. Optimization strategies: (1) Reduce context size: compress retrieved chunks, keep only top-3 results. (2) Cache repeated system prompt with prefix caching (OpenAI offers ~50% discount on cached prompt tokens). (3) Route simple queries to GPT-4o-mini ($0.15/$0.60 per 1M in/out vs $5/$15 for GPT-4o), 30-40x cheaper. (4) Use batch API for non-real-time queries (50% discount). Lesson: in production, token budgeting directly determines whether a product is economically viable.

Top-K always keeps exactly K candidates. Problem: if the model is very confident (one token at 95% probability), K=40 still forces sampling from 40 tokens, introducing unnecessary randomness. If the model is very uncertain (20 tokens all at ~5%), K=40 might be too restrictive. Top-P (nucleus sampling) keeps the smallest set of tokens whose cumulative probability ≥ P. When the model is confident, Top-P might keep only 2-3 tokens. When uncertain, it adapts to keep more. This dynamic adaptation is why Top-P is generally preferred for open-ended generation. When to prefer Top-K: code generation where you want consistent syntax, or when you've empirically found a K that works well for your use case. Best practice: use both together (Top-K=40, Top-P=0.95). Top-K prevents catastrophic sampling from very low probability tail tokens, Top-P provides the adaptive benefit. Temperature controls how sharp/flat the distribution is before either K or P filtering.

Research (Liu et al. 2023, "Lost in the Middle") showed that LLMs systematically under-attend to information in the middle of long context windows. They perform best on information at the beginning (primacy effect) and end (recency effect) of the context, with accuracy dropping significantly for content in positions 15-60% through a long prompt. Concretely: if you retrieve 10 documents and concatenate them, the most relevant document placed in the middle will be effectively ignored, while a less relevant document at position 1 or 10 gets disproportionate weight. RAG implications: (1) Put the most important/highest-scored retrieved context at the beginning or end of your prompt, not the middle. (2) Limit context to top 3-5 chunks rather than 10-15. (3) Use a reranker to ensure the most relevant chunk is first. (4) For very long contexts, consider sending multiple shorter prompts instead of one long one. Models trained specifically for long-context understanding (Gemini 1.5 Pro, Claude 3.5 Sonnet) mitigate but do not fully eliminate this problem.

This is a factual hallucination: the model generates false information with confidence. Distinguish from retrieval failure: retrieval failure means the system couldn't find the right document. Hallucination means even with the correct document retrieved, the model generates something not supported (or contradicted) by the document. How to diagnose: (1) Check if the correct fact was in the retrieved context. If yes → hallucination (the model ignored grounding evidence). If no → retrieval failure (fix chunking, reranking, or embedding model). Root cause of hallucination: the next-token prediction objective optimizes for plausibility, not accuracy. "Eiffel Tower is in P..." → "Paris" and "Berlin" both have high probability in training data because both follow "Eiffel Tower is in" in some proportion of training text. Mitigations: (1) Force the model to cite specific spans from retrieved text ("quote the relevant passage before answering"). (2) Use a validation LLM call to check if the answer is supported by the retrieved context. (3) Lower temperature for factual tasks.

💻

Session 05 · Infrastructure

Local LLMs and API Providers

Run Llama on your own laptop, free, private, offline. And use one API key to access every cloud provider through OpenRouter.

⚖️ Local vs Cloud

Factor	☁️ Cloud API	💻 Local (Ollama)
Cost	Per token	Free
Privacy	Data leaves machine	100% local
Internet	Required	Works offline
Quality	Best available	Depends on hardware
Latency	~1s (fast GPU)	Slower on CPU
Setup	API key only	Install + pull model

🛠️ Tools You Used

🦙

Ollama

Run Llama 3, Mistral, Gemma locally. CLI + REST API. ollama run llama3.2

🔀

OpenRouter

One API key for 100+ models. OpenAI-compatible endpoint. Compare providers without multiple keys.

📦

LM Studio

GUI for local models. Drag-and-drop GGUF loading. Built-in OpenAI-compatible server.

# Universal chat format, same across all providers messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is attention?"}, {"role": "assistant", "content": "Attention is..."}, {"role": "user", "content": "Give me an example."}, ]

🏗️ Architecture: Local Inference vs Cloud API

How Ollama works

💬 Expand each to reveal the answer

7 questions

GGUF is a self-describing binary file format for quantized LLM weights, created by the llama.cpp project (it replaced the older GGML format; GGUF stands for GGML Universal Format). It matters because: (1) all model weights, hyperparameters, tokenizer, and metadata are packed into one portable file, no separate config files needed. (2) Supports quantization: a 7B model shrinks from ~14GB (FP16, the standard distribution format) to ~4.1GB at Q4_K_M, approximately 3% perplexity increase vs FP16. (3) Optimized for CPU/GPU inference via llama.cpp, with Apple Metal support on M-series chips. Ollama uses GGUF internally. Q4_K_M decoded: Q4 = 4-bit quantization; K = K-quant method (group-wise, more accurate than naive INT4); M = Medium quality/speed tradeoff within the K-quant family.

The standard chat format uses messages with role/content pairs. Consistent across all providers (OpenAI, Anthropic, Gemini, Ollama): system (sets the AI's persona, constraints, and context), user (the human's message in each turn), assistant (the AI's prior response, for multi-turn history). Provider-specific: the "tool" role for function call results exists in OpenAI's format, but Anthropic uses user messages with tool_result content blocks. The three-role structure (system/user/assistant) is the universal standard. Code for any provider: messages = [{"role": "system", "content": "You are..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}].

OpenRouter is an API gateway providing access to 100+ models (GPT-4o, Claude Sonnet, Gemini, Llama, Mistral) through a single OpenAI-compatible endpoint with one API key. You only change the base_url and model name in your existing code. Limitations vs. direct API calls: (1) Extra latency: ~50-200ms routing overhead. (2) Outage masking: if a provider is down, errors are less diagnostic. (3) Model versioning: model names may map to different underlying versions. (4) Compound rate limits: you hit both OpenRouter's AND the provider's limits simultaneously. (5) Inconsistent tool calling: function calling standardization across providers is imperfect. Use OpenRouter for: development, model comparison experiments, prototyping. For production with SLA requirements: use direct provider API for better error handling and contractual guarantees.

Quantization reduces numerical precision of model weights to save memory and speed up inference. Memory per parameter: FP32 = 4 bytes, FP16 = 2 bytes, INT8 = 1 byte, INT4 = 0.5 bytes. For a 7B parameter model: FP32 = 28GB, FP16 = 14GB, INT8 = 7GB, Q4_K_M ≈ 4.1GB. Q4_K_M decoded: "Q4" = 4-bit quantization. "K" = K-quant method: weights quantized in groups (~32 weights/group), with a per-group scale factor calibrated to minimize quantization error, more accurate than naive INT4. "M" = Medium size variant (S = smaller/faster but slightly less accurate, L = larger/more accurate but slower). Q4_K_M achieves approximately 3% perplexity increase vs FP16, imperceptible in most applications. This 3.5x memory reduction is what lets you run a 7B model on a laptop with 8GB RAM.

A 7B model at Q4_K_M requires ~4.1GB for model weights. Add 1-2GB for KV cache and overhead. Practical minimum: 6-8GB available RAM. Hardware guide: 8GB RAM (MacBook Air M1/M2): runs 7B at Q4_K_M at ~15-25 tok/s using Apple Metal. M-series chips use unified memory, where RAM is also GPU memory. 16GB RAM: 7B comfortably or 13B at Q4_K_M. 32GB: 13B well or attempt 30B. 64GB+: 70B at Q4_K_M (~40GB). Generation speed factors: (1) CPU vs. Metal/GPU: CPU is 5-10x slower. (2) Context length: longer KV cache = more memory bandwidth. (3) Quantization level: Q2 is faster than Q8 but lower quality. Apple M-series chips with unified memory are the best consumer-grade local inference hardware. Memory bandwidth rivals discrete GPUs for inference workloads.

Ollama: CLI-first tool, runs a REST server at localhost:11434 with OpenAI-compatible /api/chat. Best for developers, scripting, Jupyter notebooks, production local inference servers. Command: "ollama run llama3.2" downloads GGUF automatically. LM Studio: GUI desktop app for local models. Drag-and-drop GGUF loading, built-in chat interface, OpenAI-compatible server at localhost:1234. Best for non-developers, students exploring models visually, quick local chat without CLI. OpenRouter: cloud API gateway, no local install. Best for accessing frontier models (GPT-4o, Claude 3.5 Sonnet) through one key, model comparison studies, or teams needing one billing source. Decision framework: need privacy/offline/free → Ollama (dev) or LM Studio (GUI). Need frontier model quality → OpenRouter or direct provider. Need it in production code → Ollama (stable REST API, Docker-friendly, no GUI dependency).

A single context window (even 128K tokens) cannot fit a full book, large codebase, or thousands of emails. Map-reduce splits the document into chunks (map step), each processed independently by the LLM, then combines results (reduce step). For summarization: map each chunk to a summary, reduce merges all chunk summaries into one final coherent output. For Q&A over long documents: map extracts relevant passages from each chunk, reduce synthesizes the final answer. An alternative is the refine strategy: process chunk 1 to get an initial output, then pass that output plus chunk 2 to the LLM for refinement, repeat until done. Refine produces more coherent output because each step builds on the previous, but it is sequential and cannot be parallelized. Map-reduce is faster because all chunks are processed in parallel. Trade-off: map-reduce can lose cross-chunk context; refine maintains it but is much slower. In LangChain: load_summarize_chain(chain_type="map_reduce").

✍️

Session 06 · Optimization

Prompt Engineering and DSPy

First: how to write a good prompt manually. Then: how to let code find the optimal prompt automatically using DSPy, MIPROv2, and GEPA.

📋 Two Parts to This Session

#	Part	What You Learn	Why It Matters
Part 1	Manual Prompt Engineering	Role, Task, Context, Examples, Output Format. Zero-shot, Few-shot, CoT.	Foundation skill. Every AI engineer needs this.
Part 2	Automated Optimization	DSPy Signatures, BootstrapFewShot, MIPROv2, GEPA	Removes guesswork. Finds prompts better than human-written.

🧩 Five Parts of a Prompt

Most prompts fail because only "Task" is included

Part	What It Does	Example
Role	Who the AI should be, activates the right writing patterns from training data	"You are a senior data analyst writing for executives"
Task	What exactly you want done, be specific, not vague	"Summarize the following in exactly 3 bullet points"
Context	Background the AI needs to answer correctly	"The audience has no technical background"
Examples	2-3 input/output pairs showing the format and style you want	Few-shot examples before the actual question
Format	How to structure the output, critical if your code needs to parse it	"Reply ONLY with valid JSON. No markdown. No explanation."

❌ Bad Prompt

Summarize this.

The model doesn't know the length, audience, format, or what to focus on. You'll get something different every time.

✅ Good Prompt

You are a business analyst writing for non-technical managers.
Summarize the following customer feedback in exactly 3 bullet points.
Each bullet: main complaint + why it matters to the business.

Role + Task + Format. Consistent output every single time.

Zero-Shot

No examples. Just the instruction. Works for simple tasks.

Classify: "I love this!"
→ Positive

⚠️

Fails when format consistency matters, you get different label names every run.

Few-Shot

Show 2-3 examples. The model infers format and labels from them.

"Love it!" → Positive "Awful!" → Negative "It's ok" → Mixed "I love this!" →

✅

Model locks into your exact labels and format, no description needed.

Chain-of-Thought

Add "think step by step." Forces visible reasoning before the answer.

-- Without CoT: Q: 9.11 > 9.9 ? A: Yes ❌ (wrong) -- With CoT: Q: 9.11 > 9.9? Think step by step. A: 9.11 has 0.11 after decimal. 9.9 has 0.90. So 9.9 > 9.11. No. ✅

ℹ️

Intermediate tokens give more context for each next prediction. Works especially well for reasoning and classification.

📐 Output Format: Always lock it down if your code needs to parse it

❌ Unstructured (cannot reliably parse)

"The customer seems fairly happy overall, particularly mentioning delivery speed...", Your code has to parse natural language. That breaks constantly.

✅ Structured (json.loads() just works)

Reply ONLY with this JSON. No markdown.
{"sentiment": "positive"|"negative"|"mixed",
 "would_recommend": true|false}

🧱 Prompt Anatomy: from vague to specific

5-part structure

📝 Signatures: Define the Task, Not the Prompt

DSPy treats prompts as learnable parameters. You define what you want in code. The optimizer finds the best prompt automatically.

class Classification(dspy.Signature): """Classify customer message intent.""" customer_message = dspy.InputField() intent_labels = dspy.InputField() answer = dspy.OutputField() # ChainOfThought adds reasoning field automatically cot = dspy.ChainOfThought(Classification)

ℹ️

The docstring becomes the system instruction. The field names become prompt slots. You never write raw prompt strings.

🏆 The 4 Optimizers: From Simple to Powerful

LabeledFewShot

Baseline

Randomly picks k training examples and injects them. No actual optimization.

BootstrapFewShot

Better

Runs the model on training data, keeps only examples where the model was already correct. Higher quality signal.

BootstrapFewShot + RandomSearch

Strong

Runs Bootstrap multiple times with different random seeds. Picks the best-performing program.

MIPROv2

Most Powerful

Searches over the instruction wording itself, not just examples. Found the instruction in MiproV2Prompt.json automatically, nobody wrote it.

💬 Expand each to reveal the answer

6 questions

Role (who the AI should be): activates relevant writing patterns from training. "You are a senior data analyst writing for executives" produces very different output than the same task with no role. Task (what exactly to do): be specific: for example, "summarize in 3 bullet points" not "summarize." Context (background it needs): "the audience has no technical background" changes every word choice. Examples (2-3 input/output pairs): the most powerful tool for locking in format and label names: the model copies your exact output structure. Output format (how to structure the response): critical if code must parse the output: "Reply ONLY with valid JSON, no markdown, no explanation." Most bad prompts fail because only the Task is included, and the model guesses at tone, length, format, and labels differently every run. Including all 5 produces consistent, parseable output you can deploy to production.

CoT prompting adds "think step by step" or explicitly asks the model to reason before answering. Classic example: "Is 9.11 > 9.9?" → without CoT: "Yes" (wrong). With CoT: "9.11 means 0.11 after decimal. 9.9 means 0.90. 0.11 < 0.90, so 9.11 < 9.9. No." (correct). Why it works mechanically: intermediate reasoning tokens provide additional context for each subsequent token prediction. The model is effectively doing more computation by generating the reasoning chain. Works in two modes: zero-shot ("think step by step" added to any prompt) and few-shot (show 2-3 examples with explicit reasoning → answer chains). CoT dramatically improves accuracy on multi-step reasoning, math word problems, complex classification, and code generation. DSPy's ChainOfThought module automates this by adding a "reasoning" field before the answer field in the signature.

DSPy (Stanford NLP, 2023) treats prompts as learnable parameters rather than hand-written strings. Manual workflow: write a prompt → test → tweak wording → repeat. This is slow, doesn't scale, and the optimal prompt is highly model-specific. DSPy workflow: (1) Define a Signature: a Python class specifying input/output fields and a docstring describing the task. (2) Choose a Module (Predict, ChainOfThought, ReAct). (3) Provide training examples + a scoring metric. (4) Run an Optimizer (LabeledFewShot, BootstrapFewShot, MIPROv2). The optimizer automatically searches over example selection and instruction wording to maximize your metric. The model never sees your raw prompt. DSPy assembles the optimal prompt from signature + examples + instructions at runtime. This enables prompt optimization across model versions: re-run the optimizer when you upgrade from GPT-4 to GPT-4o and the prompt adapts automatically.

LabeledFewShot: the simplest optimizer. Randomly picks k examples from your training set (which must have ground-truth labels) and injects them as few-shot demonstrations. No actual optimization: you're just doing in-context learning with randomly selected examples. Quality depends on which k examples happen to be picked. BootstrapFewShot: runs the current program on every training example, evaluates each prediction against your metric, and keeps only examples where the model was already correct. The demonstrations injected into the prompt are guaranteed to pass your validation criterion. They're not random, they're validated examples of the model succeeding. This produces significantly higher quality signal than random selection. The "bootstrap" name: you're using the model's own correct predictions to create high-quality training data for itself. BootstrapFewShot requires no human-labeled outputs, only human-labeled inputs with a metric to evaluate outputs.

BootstrapFewShot selects which demonstrations to include, but it cannot modify the instruction text of the prompt itself. MIPROv2 (Multi-prompt Instruction PRoposal Optimizer v2) searches over both the instruction wording AND the few-shot examples simultaneously. MIPROv2 uses Bayesian optimization (TPE-style) to explore the instruction space: it generates candidate instruction variants, evaluates each on validation data, and focuses the search on promising phrasings. Example: MIPROv2 might find "Let's work through this step by step to ensure accuracy" outperforms your hand-written "Answer the following question" by 8% F1, something no human thought to write. MIPROv2 is most valuable when: (1) you have labeled validation data, (2) your metric is well-defined and computationally cheap, (3) manual prompting has already been tried and isn't sufficient. Downside: MIPROv2 requires many LLM calls for the optimization sweep (can be expensive). Use BootstrapFewShot first; upgrade to MIPROv2 when you need further gains.

Format consistency for downstream code parsing. Zero-shot "Classify the sentiment as positive, negative, or mixed" produces: "Positive", "positive", "POSITIVE", "The sentiment is positive", "positive sentiment", "I believe this is positive", all different strings, none reliably parseable by code. Few-shot with 3 examples ("Love it!" → "positive", "Awful!" → "negative", "It's ok" → "mixed") forces the model to lock into exactly one of three strings you control. Your json.loads() or if answer == "positive" code works 100% of the time. A second example: extracting a date from a document. Zero-shot produces: "2024-01-15", "January 15, 2024", "15/01/2024", "the 15th of January." One few-shot example showing input → "2024-01-15" forces ISO format consistently. Rule: whenever your downstream code must parse the output, or you need consistent label strings for a classifier, few-shot is not optional, it is the only reliable approach.

🔗

Session 07 · Unification

LangChain: One Interface, Every Provider

You've called OpenAI. You've called Gemini. You've called Ollama. Why are you maintaining three different codebases for the same thing?

📋 What LangChain Unifies

Component	Before LangChain	After LangChain
Provider calls	3 different SDKs, 3 different response parsers	Same `.invoke()` for all providers
Prompt management	f-strings and manual string building	`ChatPromptTemplate` with typed variables
Conversation history	Manually pass message list each turn	`MessagesPlaceholder` + `InMemoryChatMessageHistory`
Streaming	Different streaming API per provider	`chain.stream()` works identically everywhere
Output parsing	Access different fields per provider	`StrOutputParser()` extracts content universally

⚙️ LCEL Pipeline

ℹ️

The | operator chains Runnable components. Output of left becomes input of right. Same interface for every component.

from langchain_openai import ChatOpenAI from langchain_anthropic import ChatAnthropic from langchain_ollama import ChatOllama # Build the chain, provider-agnostic chain = prompt_template | llm | StrOutputParser() # Invoke, same for ALL providers answer = chain.invoke({"question": "Explain RAG"}) # Streaming, tokens appear as generated for chunk in chain.stream({"question": "Explain RAG"}): print(chunk, end="", flush=True)

💬 Conversation Memory

from langchain_core.prompts import ( ChatPromptTemplate, MessagesPlaceholder ) from langchain_core.chat_history import ( InMemoryChatMessageHistory ) # Template with a slot for conversation history prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful assistant."), MessagesPlaceholder("history"), # ← history slot ("human", "{question}"), ]) history = InMemoryChatMessageHistory() history.add_user_message("What is RAG?") history.add_ai_message("RAG stands for...")

✅

MessagesPlaceholder automatically injects the full history list into the template. You define the structure once, the history is passed at runtime.

⛓️ LCEL Pipe Chain: how components connect

Visual diagram

💬 Expand each to reveal the answer

6 questions

LCEL (LangChain Expression Language) is a declarative way to compose LangChain components into pipelines using the | operator. Every component implements the Runnable interface. The | operator overloads Python's bitwise OR to chain Runnables: output of left becomes input of right. So chain = prompt | llm | StrOutputParser() means: (1) prompt formats the input dict into a messages list, (2) llm sends the messages list to the provider and returns an AIMessage, (3) StrOutputParser extracts the .content string from the AIMessage. The chain object itself is also a Runnable: you can pipe it into another component. You can swap any component without touching the others: change the provider by replacing llm, change output format by replacing the parser. The same chain.invoke({"question": "..."}) call works for OpenAI, Anthropic, Google, or Ollama.

ChatPromptTemplate defines the message structure sent to the LLM as a template with typed variable slots ({variable_name}). It takes a list of message tuples: [("system", "You are a {persona}"), ("human", "{question}")]. At runtime, .format_messages(persona="analyst", question="What is RAG?") produces the final messages list. MessagesPlaceholder is a special slot for injecting a dynamic list of messages, specifically used for conversation history. Instead of manually building {"role": "user", "content": "..."} dicts for every prior turn, you define MessagesPlaceholder("history") once in the template and pass the history object at runtime. The placeholder expands into the full list of all prior messages automatically. This cleanly separates template definition from runtime data, and works with any history store (in-memory, Redis, database).

chain.stream(input) returns a Python generator that yields text chunks as the LLM produces them, typically word-by-word or a few tokens at a time. chain.invoke(input) waits for the complete response (5-20 seconds for long outputs). Why streaming matters: (1) Perceived latency: users see the first word in under 1 second instead of a blank screen for 10 seconds, and perceived speed is 5-10× better even if total time is the same. (2) User trust: a streaming response feels like conversation; a 10-second wait followed by sudden output feels broken. (3) Early termination: users can stop generation if the response is going in the wrong direction. You save tokens. LangChain streaming API: for chunk in chain.stream({"question": "..."}): print(chunk, end="", flush=True). In Streamlit: st.write_stream(chain.stream({"question": "..."})) handles the UI update automatically. The same .stream() call works across all providers. LangChain normalizes provider-specific streaming APIs.

The Runnable interface is the standard protocol every LangChain component implements, making them composable with |. Five core methods: (1) .invoke(input): synchronous single call, returns one output. Use for simple scripts. (2) .stream(input): synchronous streaming, yields output chunks. Use for real-time UIs. (3) .batch(inputs): synchronous, processes a list of inputs efficiently, often in parallel. Use for bulk processing. (4) .ainvoke(input): async version of invoke, returns awaitable. Use in async web frameworks (FastAPI). (5) .astream(input): async streaming generator. Use in async web servers with streaming responses. Every component that implements these 5 methods works with |: ChatPromptTemplate, any LLM wrapper, StrOutputParser, RunnableLambda (wraps any Python function), RunnablePassthrough (passes input unchanged). You can create custom Runnables: class MyTransformer(Runnable): def invoke(self, input, config=None): ...

Because all LLM wrappers implement the same Runnable interface, you change exactly one line: from langchain_openai import ChatOpenAI → from langchain_anthropic import ChatAnthropic; llm = ChatOpenAI(model="gpt-4") → llm = ChatAnthropic(model="claude-sonnet-4-5"). The chain prompt | llm | parser is unchanged. For production-grade switchability: define a factory function def get_llm(provider=None): provider = provider or os.getenv("LLM_PROVIDER", "openai"). if provider == "openai": return ChatOpenAI(model=os.getenv("OPENAI_MODEL", "gpt-4o")). if provider == "anthropic": return ChatAnthropic(model=os.getenv("ANTHROPIC_MODEL", "claude-sonnet-4-5")). if provider == "google": return ChatGoogleGenerativeAI(model=os.getenv("GOOGLE_MODEL", "gemini-1.5-pro")). Then: llm = get_llm(). Switching in production = change one environment variable, no code deployment. Caveat: models differ in context length, function calling format, and system message handling. Always test your specific use case after switching.

Manual approach: maintain a list, append each user message and AI response, pass the full list to the chain on every turn. Works, but: (1) you write boilerplate for every app, (2) persistence across restarts needs separate code, (3) multi-user isolation needs manual session keying. RunnableWithMessageHistory wraps any LCEL chain to automate all of this: chain_with_history = RunnableWithMessageHistory(chain, get_session_history_fn, input_messages_key="question", history_messages_key="history"). get_session_history_fn maps session_id → a history store object (InMemoryChatMessageHistory, RedisChatMessageHistory, or custom). On each call, it: (1) retrieves history for the session_id, (2) injects it into the MessagesPlaceholder, (3) runs the chain, (4) appends the new turn to the store automatically. Multi-user isolation: chain_with_history.invoke({"question": "..."}, config={"configurable": {"session_id": user_id}}). Each user_id gets completely isolated history. Session persistence survives restarts with Redis backend. This eliminates all the boilerplate of building multi-user stateful chat.

🎯 What to say in your next GenAI interview

22 concepts

Concept	What to say in 10 seconds
TF-IDF	Rare words that appear often in one doc are important. Formula: TF x log(N / df).
BM25	TF-IDF with saturation (k1) and document length normalization (b). Default in Elasticsearch.
Inverted Index	Word-to-document lookup for sub-millisecond search. O(1) retrieval instead of full scan.
Word Embedding	Dense vector where similar words cluster together. Geometry encodes meaning.
Cosine Similarity	Measures angle between vectors, ignores magnitude. Range: -1 to 1.
Skip-gram	Word2Vec mode: predict context words from a center word. Better for rare words.
CBOW	Word2Vec mode: predict center word from context words. Faster to train.
Negative Sampling	Replaces full softmax with binary classifier on noise words. 5000x faster training.
GloVe	Embeddings from global co-occurrence statistics instead of local context windows.
Self-Attention	Each token weighs every other token for relevance using Q, K, V projections.
Multi-Head Attention	Run attention in parallel with different learned projections for different relationship types.
Positional Encoding	Adds order information. Transformers process in parallel so position must be injected.
BERT vs GPT	BERT: encoder-only, bidirectional, understands context. GPT: decoder-only, generates left-to-right.
RLHF	Three stages: SFT, reward model from human rankings, PPO with KL penalty to align LLM.
Hallucination	Model predicts plausible text, not verified facts. No built-in "I don't know" signal.
BPE	Iteratively merge frequent character pairs until target vocabulary size is reached.
Temperature	Divide logits by T before softmax. Low = deterministic. High = creative.
Top-K / Top-P	Top-K: keep K most probable. Top-P: keep smallest set summing to P. Use both together.
GGUF / Q4_K_M	Quantized model file. Q4 = 4-bit weights, K = group-wise scaling, M = medium accuracy.
Prompt Engineering	Role + Task + Context + Examples + Output format. All 5 parts for consistent output.
DSPy / MIPROv2	Prompts as learnable parameters. Optimizer finds best instruction text automatically.
LCEL / Runnable	LangChain pipe operator. Every component implements .invoke, .stream, .batch, .ainvoke, .astream.

Now you can explain all of this

How a search engine scores and ranks documents (TF-IDF, BM25, inverted index)
How a neural network learns that "car" and "automobile" mean the same thing (Word2Vec)
How Transformers read context that RNNs could not (self-attention, Q/K/V)
How GPT went from predicting text to passing the bar exam (scale, emergence, RLHF)

How tokens are built and how sampling controls generation (BPE, temperature, Top-P)
How to run GPT-quality AI on your own laptop for free (Ollama, GGUF, quantization)
How to let code write and optimise prompts (DSPy, BootstrapFewShot, MIPROv2)
How to call any LLM with the same code (LangChain, LCEL, Runnable interface)

💪

This foundation is permanent. Every LLM system in the world, no matter how advanced, sits on exactly these concepts. You understand the ground floor. Next session, we build higher.

🗃️ Upcoming: RAG

Next module

Connect LLMs to your own data, documents, databases, APIs
Stop hallucinations on your private knowledge base
Chunk → Embed → Store → Retrieve → Generate with citations
ChromaDB, sentence-transformers, production patterns

🤖 Upcoming: Agents

Coming soon

LLMs that can use tools, search the web, write and run code
ReAct pattern, Thought, Action, Observation loop
LangGraph for stateful multi-step workflows
CrewAI for multi-agent teams with specialist roles

📊 Your Progress

Session	Topic	Status
S00	Keyword Search, TF-IDF	Complete
S01	Word Embeddings, Word2Vec	Complete
S02	Transformer Architecture	Complete
S03	GPT Evolution, RLHF, Alignment	Complete
S04	BPE, Temperature, Sampling	Complete
S05	Local LLMs, OpenRouter	Complete
S06	Prompt Engineering, DSPy, MIPROv2	Complete
S07	LangChain, LCEL, Streaming	Complete

🎯

The gap between you and a GenAI engineer role is the next few modules. Every session you've completed has closed a real gap. Keep going.