Notebook 07 · Retrieval

Two retrievers, one wrong answer alone

Notebook 06 showed BM25 and dense fail on different queries. Today we follow one real query all the way through fusion, and prove on a labeled corpus that it actually wins.

BM25

blind to meaning

Dense

fuzzy on exact terms

→

Hybrid

covers both blind spots

The setup

A bigger, harder test bed than last class

Sixteen toy documents was enough to see the two failure modes. To prove fusion actually helps, we need a corpus large enough for the wins and losses to average out honestly.

passages indexed

gold-labeled queries

retrievers to fuse

5 exact 8 semantic 3 mixed

BM25

same sparse retriever as notebook 06

bge-small-en-v1.5

a small, fast bi-encoder, chosen because that's what production actually runs at first-stage retrieval, not a giant API model

One query, carried through the whole notebook

Two retrievers, two different top picks

BM25, top 5

#	doc	text	score
1	85	"Python 3.10 introduced structural pattern matching..."	3.920
2	91	"Python 3.11 introduced fine-grained error locations..." ← gold	3.761
3	52	"Increase the application's log level to ERROR..."	2.678

Dense (bge-small), top 5

#	doc	text	score
1	91	"Python 3.11 introduced fine-grained error locations..." ← gold	0.782
2	81	"Python 2.7 reached end of life in 2020..."	0.733
3	85	"Python 3.10 introduced structural pattern matching..."	0.716

BM25 puts the wrong document, doc 85, in first place, most likely the same version-fragmentation issue from notebook 06, "3.11" and "3.10" share tokenized pieces. Dense gets it right. Neither one is safe to trust alone, so let's try to combine them.

The obvious first move

Just add the two scores?

For our two contenders, doc 85 and doc 91 (the gold document), add BM25's score and dense's score directly.

doc 85, wrong answer

"Python 3.10 introduced structural pattern matching..."

3.920 (BM25) + 0.716 (dense)

= 4.636

wins the sum

doc 91, gold document

"Python 3.11 introduced fine-grained error locations..."

3.761 (BM25) + 0.782 (dense)

= 4.543

loses by 0.093

Adding raw scores puts the wrong document on top. BM25's number is so much bigger to begin with that its small 0.16-point lead over the gold document survives the addition, while dense's correct, larger vote barely moves the total. Dense's opinion is real, it just can't out-shout a scale it was never on.

Two judges score a talent show. One scores out of 100, the other out of 5. Add their scores directly and the second judge might as well stay home, her strongest possible vote can't overturn even a small gap in the first judge's number. Before two opinions can be combined, they need to be on the same scale.

Fix attempt 1

Put both scores on the same 0-to-1 scale, then blend

Min-max normalize each retriever's full score list first, using that query's own minimum and maximum across all 96 passages: BM25 ranged 0.00 to 3.920, dense ranged 0.288 to 0.782.

norm(s) = (s − min) / (max − min)

doc	BM25 raw	BM25 norm	dense raw	dense norm
85, wrong	3.920	1.000	0.716	0.866
91, gold	3.761	0.959	0.782	1.000

Now blend with α=0.5: final = 0.5·dense_norm + 0.5·bm25_norm

doc 85, wrong answer

0.5×0.866 + 0.5×1.000

= 0.933

doc 91, gold document

0.5×1.000 + 0.5×0.959

= 0.980

now wins

Normalizing first genuinely fixes it. The gold document is back in first place, because now both retrievers' opinions live on the same 0-to-1 footing before they're added.

Before we call this solved

The catch with normalizing every time

Weighted fusion worked here. But look at what it needed to work:

Recomputed from scratch, every query

Min and max above came from this query's 96 scores. A different query produces a different min-max, so the normalization has to be redone at query time, every time.

A degenerate range breaks the formula

If a query is a pure vocabulary gap, the kind BM25 scores 0.000 on across the board, then min equals max, and (s-min)/(max-min) divides by almost zero. The code has to add a tiny epsilon just to survive that case.

α needs labeled data to tune

0.5 wasn't a guess, we'll see it swept against 16 labeled queries next. Without that labeled set, picking α is a shot in the dark.

None of this makes weighted fusion wrong. It's just extra machinery running on every single query. Is there a fix that skips all three problems at once?

Fix attempt 2

Forget the scores entirely, use only the rank

Reciprocal Rank Fusion never looks at 3.920 or 0.782. It only asks: where did this document land in each list?

RRF(d) = Σ 1 / (k + rank)

k=60. The top result in a list contributes 1/60, the second contributes 1/61, and so on. A document missing from a list contributes 0 from it.

doc 85, wrongBM25 #1 → 1/60 + dense #3 → 1/62= 0.0328

doc 91, goldBM25 #2 → 1/61 + dense #1 → 1/60= 0.0331

Gold wins again, this time without ever normalizing anything, without an epsilon safeguard, and without needing a single labeled example to pick a setting.

Olympic judging, the version that ignores raw points. One judge is a harsh scorer, another generous, it doesn't matter, you only ask "where did each judge place this athlete?" An athlete ranked 2nd by every judge beats one ranked 1st by a single judge and 8th by everyone else.

Now zoom out to all 16 queries

Finding the right α, with real data

That α=0.5 wasn't arbitrary. Sweep it across every labeled query and measure MRR, mean reciprocal rank, how high the correct answer lands on average.

α=0.0

0.875

0.25

0.906

0.4

0.906

0.5

0.969

0.6

0.969

0.75

0.969

α=1.0

0.849

Pure sparse: 0.875. Pure dense: 0.849, actually the worse extreme here. The blend from 0.5 to 0.75 wins outright at 0.969. Neither retriever alone is the answer, the middle ground is, exactly what we just watched happen on one query, now confirmed across sixteen.

The actual proof

Across all 16 queries, hybrid wins on every metric

Method	MRR	R@1	R@3	nDCG@10
BM25 alone	0.875	0.75	1.00	0.908
Dense alone	0.849	0.75	0.94	0.887
Hybrid, RRF	0.969	0.94	1.00	0.977
Hybrid, weighted (α=0.5)	0.969	0.94	1.00	0.977

Both fusion methods land on identical numbers in this run. R@1 is the number to feel: the gold document is the very top hit 75% of the time for a single retriever, and 94% of the time once you fuse, exactly the jump we watched happen with doc 91.

Same numbers, as a picture

Both single-stage retrievers sit lower

BM25 (sparse)

0.875

Dense (bge-small)

0.849

Hybrid, RRF

0.969

Hybrid, weighted

0.969

Fusion works because BM25 and dense miss different queries. Combining them recovers the union of their strengths, not the average of their weaknesses.

RRF vs weighted

When to pick which

	RRF	Weighted convex
Uses	ranks only	normalized scores
Tuning	just k, default 60 is robust	must tune α on a dev set
Score-scale safe	immune by construction	only after normalization
Best when	no labeled data, want a safe default	you have a dev set, want max quality

Start with RRF. It's the hybrid default in Elasticsearch, Weaviate, Qdrant, and Pinecone, for a reason.

Move to weighted once you have labeled data and want to bias toward lexical or semantic traffic on purpose.

Always normalize before weighting. Mismatched score axes are the number one hybrid bug, as we just watched firsthand.

Notebook 07 · summary

Where this leaves us

Production retrieval is hybrid, not either/or. Dense and sparse cover each other's blind spots.

RRF fuses ranks, robust, no tuning. Weighted convex fuses normalized scores, tunable but higher maintenance.

On our labeled corpus, both fusion methods beat either retriever alone, on every metric.

Never add raw BM25 and cosine directly. We watched the wrong document win by 0.093 when we tried.

Fusion fixes recall and rough ordering. It doesn't sharpen what's exactly at the top.

Next, notebook 08: reranking →