RAG Strategy Matrix — Evaluation Presentation

RAG Studio — Evaluation Presentation

RAG Strategy Matrix
Evaluation Report

A live, end-to-end comparison of 14 retrieval-augmented-generation pipeline configurations, scored by both RAGAS and DeepEval against a 7-question golden set over a purpose-written policy handbook. Every score on every slide came from a real model call — nothing here is simulated.

14 variants 7 golden questions 6 RAGAS + DeepEval metrics gpt-4o-mini generation gpt-4o judge (DeepEval)

0.881

best composite score

14/14

variants completed

23

indexed chunks

2

systemic bugs found

01 / 14

Context

What was tested, and against what document

The evaluation needed a document where the correct answer to every question is knowable in advance — so reference-dependent metrics like RAGAS context recall are meaningful. A purpose-written policy handbook made that possible.

Nimbus Cloud Storage — Customer Handbook

~2,000 words · 8 sections

Company overview & product tiers (Free / Pro / Business $99/mo with SSO / Enterprise)
Security & encryption — AES-256 at rest, HSM-managed keys, Private Vault
API rate limits — per-tier request ceilings, 429 behavior
Data retention & deletion timelines — trash window, cancellation grace period
Refund policy — 14-day guarantee, first-payment-only exclusion
Support SLAs — response times and uptime service credits by tier
Compliance certifications — SOC 2, ISO 27001, HIPAA status

Fixed across every variant

Embeddingsminilm (local)

Vector storein-memory

Generation modelgpt-4o-mini @ T=0.0

Chunkingrecursive, 500 / 60

Retrieved / reranked totop_k 20 → 5

Caching / guardrailsoff

02 / 14

Design of the matrix

4 axes, 14 configurations

Rather than a full combinatorial sweep of every field (which would run into the thousands of configurations), the matrix targeted the axes known to actually move answer quality: how candidates are retrieved, whether they're reranked, and whether the query or the retrieved context is transformed before generation.

Retrieval mode

dense · bm25 · hybrid_rrf · hybrid_weighted

Reranker

none · local cross-encoder

Query transform

none · HyDE · multi-query

Post-retrieval

none · contextual compression

Coverage

8 variants cover every retrieval × reranker pair. 6 more variants layer HyDE, multi-query, or contextual compression onto the strongest baseline (hybrid_rrf), each with and without the reranker — isolating each transform's individual effect.

03 / 14

How quality was measured

Metric definitions

Two independent judge frameworks were used so no single scoring methodology's blind spots go unchecked.

RAGAS

reference-dependent

Faithfulness

Does every claim in the answer trace back to the retrieved context? Penalizes fabrication.

Answer relevancy

Does the answer actually address the question asked, without padding or drift?

Context precision

Of the chunks retrieved, how many were actually relevant / ranked highly?

Context recall

Did retrieval surface everything needed to construct the reference answer?

DeepEval

gpt-4o judge

Faithfulness

Same intent as RAGAS's version, scored independently by a different judge model.

Answer relevancy

Same intent, independent judge — used as a cross-check against RAGAS's score.

Pass rate

Fraction of the 7 golden questions where every DeepEval metric individually cleared its threshold.

Composite score (used to rank variants throughout this presentation) is simply (RAGAS metric average + DeepEval metric average) / 2 — a single number for at-a-glance comparison, with the full metric breakdown always available in the table.

04 / 14

Executive summary

Four things worth knowing before the details

0.881

Best composite — hybrid_rrf + HyDE or multi-query

Tied for first, reranker made no measurable difference here

0.837–0.881

Composite range across all 14 variants

A real but modest 4.4-point spread

14 / 14

Variants failed the same golden question

A chunking blind spot, not a strategy problem

~30%

Of grounded answers show zero citations

A citation-format parsing bug, present in every variant

05 / 14

Full results

All 14 variants, every metric

Sorted by composite score, descending. Scroll horizontally on smaller screens.

Variant	Config	RAGAS faith.	RAGAS relev.	RAGAS ctx-prec.	RAGAS ctx-recall	DeepEval faith.	DeepEval relev.	DE pass%	Retrieval ms	Generate ms	Composite

06 / 14

Ranking

Composite score, ranked

07 / 14

Finding 01 · Biggest lever

Query transforms beat every other lever tested

HyDE and multi-query each added +0.018 composite over the plain hybrid_rrf baseline — identically whether or not a reranker was also enabled. That's a bigger single effect than switching retrieval mode or adding a reranker at all.

The cost side of the ledger

real latency & token cost

Multi-query retrieval time54ms → 125ms (~2.3×)

Extra LLM call per questionHyDE: ~1,000ms · multi-query: ~1,650ms

Whymulti-query retrieves once per paraphrase, then fuses

08 / 14

Finding 02 · Inconsistent effect

Reranking helped some modes, hurt one

Paired composite-score deltas, reranker off → on, for each retrieval mode:

Retrieval mode	No reranker	Cross-encoder	Δ
bm25	0.855	0.864	+0.009
dense	0.853	0.856	+0.004
hybrid_rrf	0.864	0.864	+0.000
hybrid_weighted	0.850	0.837	−0.013

For a 23-chunk corpus this small, first-stage retrieval is already strong enough that reranking has little room to improve on it. For hybrid_weighted specifically it actively hurt — most likely by reordering a couple of fusion-scored candidates in a way that didn't match reference expectations.

09 / 14

Findings 03 & 04

Compression didn't earn its keep; BM25 was the speed champion

Contextual compression

wash to slightly negative

−0.013 composite with no reranker, +0.002 with the cross-encoder. At this corpus size the reranker already narrows to 5 chunks — there's little left to trim, so compression mostly adds an extra embedding-similarity pass for no benefit. It would likely earn its keep on a much larger, noisier corpus.

BM25 retrieval speed

25–30× faster

Avg 2.2ms vs. 53–125ms for embedding-based modes, since it's pure local lexical scoring with no embedding call in the loop — yet it scored within 0.01 composite of the best embedding-based baseline. On a keyword-rich corpus, BM25 is a legitimately strong default for latency-sensitive deployments, not just a fallback.

10 / 14

Finding 05 · Universal failure

One question broke every single variant

"Am I eligible for a refund if I've been on the Growth plan for six months and just got charged for a renewal?" — every one of the 14 variants, including both HyDE and multi-query (designed to help with exactly this kind of paraphrased question), answered "answer not found in provided context."

Reference: "No. The 14-day money-back guarantee only applies to a customer's first payment on a tier, not to renewal charges." All 14 variants: "answer not found in provided context"

This is a chunking problem, not a retrieval-strategy problem: the 500-character recursive splitter most likely separated the sentence naming the 14-day window from the sentence excluding renewal charges into two different chunks — so no single retrieved chunk contained the complete, qualifying policy fact. No amount of reranking or query rewriting fixes a fact that's been chunked apart from its own qualifier. The fix is chunking strategy (larger chunk size, or clause-aware splitting for policy documents), not retrieval strategy.

11 / 14

Finding 06 · Present in every variant

A citation-format bug hides good answers

~30% of answers show zero extracted citations despite being fully grounded — because the model sometimes cites like a footnote ([19]) instead of the instructed [chunk_id] format, and the extractor only recognizes exact chunk ids.

Q: "What is the guaranteed support response time for a critical ticket on the Enterprise tier...?" A: "...is 1 hour [19]. If Nimbus misses its Enterprise uptime SLA, the..." citations extracted: 0 (the answer is correct and grounded — the tag just doesn't parse)

This is a prompt/parsing gap in the generation stage, independent of retrieval strategy — it shows up identically regardless of which of the 14 configs is used, so no amount of strategy tuning fixes it. The fix belongs in the citation prompt or the extraction regex.

12 / 14

Reading the numbers correctly

Why DeepEval's pass rate sits at 50% almost everywhere

13 of 14 variants show exactly a 50% DeepEval pass rate — not because half of every variant's answers are bad, but because pass/fail requires every one of the 7 golden questions to individually clear the relevancy threshold, and the refund-renewal failure scores 0.00 on relevancy in every variant.

DeepEval's own reason string: "The score is 0.00 because the response failed to address the question about refund eligibility for the Growth plan, instead providing unrelated information."

The underlying average relevancy score is still 0.79–0.93 across variants — healthy — but one universally-failing question caps every variant's binary pass/fail at 50%. Worth loosening to a majority- or threshold-based pass rule if this is meant to be a go/no-go production gate.

13 / 14

Recommendation

What to actually ship

Configuration

recommended

hybrid_rrf retrieval + HyDE query transform, reranker optional. Tied for best composite (0.881) with and without the reranker — meaning the reranker adds no measurable value once HyDE is in place, so skip it and save the latency.

If retrieval latency matters more than the last ~2% of quality: bm25, no reranker, no query transform is a strong lower-cost fallback (0.855 composite, a fraction of the latency, zero extra LLM calls).

Fix before shipping either config

blocking

Loosen the citation regex/prompt so numeric footnote-style citations still resolve to a real chunk id, or re-emphasize the exact [chunk_id] format more forcefully.
Re-chunk clause-heavy policy sections (refunds, SLAs, deletion timelines) with a larger chunk size or a semantic/clause-aware splitter.