Every RAG pipeline so far reads text beautifully and throws every image straight in the trash.
Real documents hide their most important numbers inside charts. This deck walks through three real,
working architectures — built and run against a real 63-page thesis, not a toy example.
Our loader from Notebook 02 (PyMuPDF4LLMLoader) extracts text beautifully.
It throws every embedded image straight in the trash — and business documents put their most
important findings inside those images.
An embedding model can't search raw image bytes unless you pick one of three real architectures. As of 2026 the market has converged on exactly these three — and this notebook builds and runs all three against the same real document, not just describes them.
| Architecture | How it works | Best for | Tradeoff |
|---|---|---|---|
| ① Caption-and-index | A vision LLM writes a detailed text caption of each image; embed the caption; keep the raw image linked by ID | Business/report documents — reads exact chart numbers correctly | Extra LLM call per image; needs generation grounded on real pixels |
| ② Unified embeddings | One model (CLIP here; Cohere Embed-v4 / Voyage multimodal-3.5 in production) embeds text and images in the same vector space — no captioning at all | Genuinely diverse visual corpora — zero LLM calls | Can't read the fine legend text that tells two similar charts apart |
| ③ Page-as-image / late interaction | Skip parsing — embed at patch level (dozens of regions per image), score with MaxSim (ColBERT-style) | Visually dense documents, layout-sensitive matching | Needs a purpose-trained backbone (real ColPali); heavier infra |
A vector index needs searchable text. A user answer needs the real content. For an image chunk those are two different things — and treating them as the same thing is the root cause of most broken multimodal RAG demos.
docstore (id → raw content, text
or image) kept separate from the vector store (id → embedding of the searchable representation).
Retrieval searches the vector store; once you have the winning ID, you look up the real content in the docstore.No new retrieval machinery beyond Notebooks 05–08 — dense (Chroma) + sparse (BM25) + RRF fusion + cross-encoder rerank. The index has no idea it's holding a mix of modalities.
fitz pulls real embedded raster images. GPT-4o Vision (temperature=0)
captions each one with exact numbers.The obvious first design: one unified ranking — dense + BM25 fusion over all 252 chunks, then a single cross-encoder reranker picks the winners, text and image captions competing head to head. Checking the actual scores against a known-correct answer exposed a real problem.
Guaranteeing charts reach context doesn't mean every chart retrieved was actually relevant.
GPT-4o cites its sources inline as [1], [2]... — parse those citation numbers and
display only the chart(s) genuinely used in the final answer.
CLIP is trained by contrastive learning: millions of (image, caption) pairs, pulled together in one shared vector space, everything else pushed apart. The result — a single model that embeds raw pixels and raw text into the same space, so cosine similarity alone can match them. Zero LLM calls.
Instead of one pooled vector per image, keep every patch — roughly 49 small regions, each with its own vector. Score with MaxSim (the ColBERT "late interaction" rule): for every query token, find its single best-matching patch anywhere in the image, then sum those best matches.
No architecture wins everything — that's the actual lesson from running all three against one real document. The right choice depends on how visually diverse your images are.
Every answer below was checked against the actual chart pixels by hand before being accepted — not just "the code ran without an error."
What you search on and what you show the user don't have to be the same object. That's the multi-vector pattern — and it's one instance of a bigger idea: search doesn't have to happen in text-space at all (CLIP), and it doesn't have to happen at one-vector-per-item granularity either (patch-level late interaction).
| Architecture | What we actually saw it do well | What we actually saw it struggle with |
|---|---|---|
| ① Caption-and-index | Correctly read exact survey numbers off 8 different real charts | Extra LLM call per image; needs generation grounded on pixels |
| ② CLIP embeddings | Told word cloud / pie / bar chart types apart, zero LLM calls | Couldn't tell which pie chart — no OCR ability |
| ③ Late interaction | Real, working MaxSim mechanism, end to end | Worse than pooled CLIP without a purpose-trained backbone |