Notebook 06: Why BM25

Before we start, the obvious question

We already have dense embeddings. Why learn a second retriever?

Last class we saw embeddings place meaning into a few hundred dimensions, so "how do plants make food using light" lands right next to "photosynthesis converts sunlight into glucose" even though the two sentences share almost no words. That part is real and it works.

What embeddings are genuinely good at: paraphrase, synonyms, intent, meaning that survives when the wording changes completely.

So here is the fair question to ask before we spend a whole notebook on something older than embeddings: if meaning already works, what is left to fix?

No guarantee on exact tokens

An error code, a SKU, a citation gets folded into a vector like every other word. Embeddings compare "roughly similar," they were never built to promise "this exact string is present."

The wins are often razor thin

In our own notebook 4 numbers, dense embeddings found the right document on a tricky query, but its top score beat the second place score by as little as 0.06. Correct today, one similar document away from wrong.

Every query costs a model call

Dense search needs an embedding model to run at query time. That is latency and, on a hosted API, money. Per query, every time.

You cannot ask it why

A cosine similarity of 0.548 is just a number. It does not tell you which words mattered or how rare they were. Nothing to debug when it ranks something oddly.

Dense embeddings are not wrong, they are fuzzy by design. That fuzziness is exactly what makes them bad at exact, rare, cheap, explainable matching. BM25 is the tool built for precisely that gap, which is why production search almost always runs both.

So, a 30 second recap

TF-IDF in one line

Score a word by how often it shows up in this document, and how rare it is across every document. Common words like "the" score low everywhere. Rare words like "ERR_2048" score high wherever they appear.

scorewhat we compute

TFcount in this doc

IDFrarity across all docs

This part is fine. Nobody is replacing IDF today. The part we are about to break is TF, the raw count.

The problem

Raw counts have no ceiling

Two documents, both about dogs. Same query: "dog".

1×

Doc 1 says "dog" once

You read it. It's about a dog.

100×

Doc 2 says "dog" a hundred times

You already knew that by the fifth mention.

TF-IDF's answer: Doc 2 is exactly 100 times more relevant than Doc 1. Double the count, double the score, forever, with no limit.

Is that actually true?

What BM25 changes

Two curves, one argument

Same chart, both formulas

TF-IDF keeps climbing forever. BM25 rises fast at first, then flattens and hits a wall.

Zoomed in: just the BM25 shape

The first few mentions matter the most. Mention six, ten and fifty barely move the needle.

TF-IDF trusts the raw count and rewards repetition forever. BM25 says the first few mentions prove the topic, and every mention after that matters less and less. That is called saturation, and it is the whole idea.

The same argument, in numbers

How much more relevant, really?

Same word, same document, just counted more times. Score shown relative to saying the word once.

word appears this many times	1	5	10	20	50	100
TF-IDF says	1×	5×	10×	20×	50×	100×
BM25 says	1×	1.9×	2.2×	2.3×	2.4×	2.5×

By the time you've repeated a word 20 times, BM25 has basically made up its mind. TF-IDF is still climbing, and will keep climbing however long you keep typing the word.

Under the hood

What actually changed in the formula

unchanged

IDF, rarity

Same idea as TF-IDF. A word that appears in 1 of 16 documents is worth far more than a word in all 16.

new

TF, but capped

Controlled by a knob called k1. It sets how quickly repetition stops paying off. Typical value: 1.2 to 2.0.

new

Length normalization

Controlled by a knob called b. A long document doesn't win just for having more words to repeat things in. Typical value: 0.75.

as word count → infinity, BM25's term score → (k1 + 1), never further

Back to our own corpus

This is why BM25 won decisively today

doc 0: "Error ERR_CONN_TIMEOUT occurs when the database connection exceeds the 30 second limit..."

query: "ERR_CONN_TIMEOUT"

BM25 top1, score 0.826 Dense top1, score 0.548, but fuzzy

ERR_CONN_TIMEOUT appears in exactly one of our sixteen documents. That gives it a huge IDF. It only needs to appear once for BM25 to be certain, so the saturation cap never even gets tested. Rare, exact tokens are BM25's home turf, and its score tells you exactly why it's confident. Dense also finds it here, but as a softer, harder to explain signal.

Being honest about the limit

Saturation doesn't fix everything

doc 4: "Myocardial infarction results from coronary artery occlusion, causing ischemic damage to cardiac muscle."

query: "what is a heart attack?"

BM25 score, 0.000

Not one word is shared between the query and the document. BM25 only ever looks at literal tokens, so a perfect saturation formula still scores this a flat zero. Fixing raw counts was never going to fix vocabulary gaps, that is exactly where dense embeddings take over, and neither method alone covers both jobs.

A third option

Meet SPLADE: sparse, but learned

BM25 counts words that are literally there. Dense turns a whole sentence into one blurry vector. SPLADE sits in between: it still builds a sparse vector, one slot per vocabulary word, exactly like BM25's index. The difference is who decides the weight in each slot.

input

one document

→

a trained model

weighs every vocabulary word

→

output

30,522 numbers,
99.66% exactly zero

Doc 0 only ever says "Error ERR_CONN_TIMEOUT ... database connection ... limit ... pool timeout." But the model is also allowed to switch on words that fit the meaning, even if that exact word was never typed.

error database connection limit pool errors timing

word actually in the document expansion: never written, added by the model

Put it to the same test

SPLADE matches dense, and it still has BM25 inside it

Same six queries we've been using all class, one retriever added to the table. Look closely: SPLADE does not beat dense here, it ties it. Both hit all six.

query	BM25	Dense	SPLADE
exact code	HIT	HIT	HIT
synonym gap	miss	HIT	HIT
paraphrase	miss	HIT	HIT
exact SKU	HIT	HIT	HIT
lay → technical	miss	HIT	HIT
version number	miss	HIT	HIT

On this small, clean corpus, dense already hits everything, so SPLADE has nothing left to prove on hit rate alone. Its real edge is underneath the hit or miss: it's a sparse, inverted-index-friendly vector like BM25, so it plugs into keyword-style search infrastructure, and it stays inspectable, you can list the exact weighted words behind a score, which a dense vector never gives you. The cost is real too: a full neural network runs per document and per query, closer to dense's cost than to BM25's instant lookup.

Why this isn't just academic

Every major search engine made this switch

1970s–90s

TF-IDF, the original ranking idea

2015–16

Lucene, then Elasticsearch, switch their default to BM25

Today

BM25 is still the standard sparse retriever, paired with dense search

This is not a minor tuning detail. It is the reason keyword search on any real product, from your codebase search to your email search, behaves the way it does.

Notebook 06 · today's plan

What we'll actually run, in order

1Index the same 16 documents with BM25, dense embeddings, and SPLADE

2Watch BM25 win instantly on exact codes and SKUs

3Watch BM25 score exactly 0.000 on vocabulary gap questions

4Turn the k1 and b knobs ourselves and watch scores shift

5Look inside SPLADE's own vocabulary vector and see the expansion terms ourselves

6Fuse BM25 and dense results with Reciprocal Rank Fusion, a preview of notebook 07

Let's open notebook 06 →

Why BM25, and not just TF-IDF?