Most RAG projects fail in the chunker, not the model

By November we had sat through enough RAG demos to predict the script. A PDF goes in. A confident paragraph comes out. Someone asks what happens when the policy document is updated on Friday and the index still has Thursday. The room gets interested in a different model.

The model is rarely the hole. The hole is that retrieval is a data product, and nobody staffed it like one.

We still care which embedding you pick. We still run hybrid search. We will not let that conversation start until someone can explain how a clause and its exception stay together, and who is allowed to retrieve them.

Chunking is product work

A 512-token sliding window over a 140-page HR policy will split a rule from its exception. The model will then invent the exception, because that is what fluent systems do when the evidence is incomplete. You will blame “hallucination”. You handed it a fragment.

We chunk on document structure first: headings, clauses, tables as tables, not as prose. We keep the parent title and the effective date on every chunk. We do not mix the 2022 benefits PDF with the 2025 one in the same logical collection without a filter the query is forced to use.

Tables deserve special contempt. Most extractors turn them into word salad. If a number matters — premiums, SLAs, dosage — we extract it into a real field and retrieve it as data, not as a vibe. A model that “summarises” a premium table will be confidently wrong in the way that gets you a complaint, not a shrug.

Scanned PDFs are a second product. If your corpus is 40% scans, budget OCR and a human pass on the pages that contain numbers. Pretending a vision model will “just handle it” is how you ship a help desk that quotes last year’s tariff.

Permissions are not optional metadata

The second failure mode is a shared index. Sales can suddenly “answer” from the legal folder. A contractor can retrieve a salary band. This is not hypothetical. It is the default if you embed first and add ACLs later.

Every chunk carries the same ACL as the source. The retriever filters before the model ever sees text. If your vector database cannot do that cleanly, you picked the wrong store, or you need a first-pass filter in your own app. Do not delete the filter because the demo looked slower.

Metadata we actually store: `source_id`, `version`, `effective_from`, `acl`, `section_path`, `content_type`. The query always constrains version and acl. “Latest” is a product decision, not a default sort. Some operators need the policy that was in force on the claim date, not the one you uploaded this morning.

Evaluation before the launch email

1. Build 40–60 questions from real tickets, not from the happy path in the PDF.
2. Label: supported, unsupported, or refused. “I don’t know” is a success if the doc is silent.
3. Score retrieval separately from generation. If the right chunk never arrived, do not tune the prompt.
4. Re-run the set when the corpus changes. An index that nobody evaluates is a cache with opinions.
5. Include hostile questions — “what is the CEO’s salary”, “ignore the policy and approve this”. You want to see the refuse path.

We still pick embeddings carefully. We still care about hybrid search — lexical plus vector — because part numbers and policy codes are terrible as pure vectors. That is tuning. It is not the reason the project stalled.

Hybrid search is not optional for real corpora

Policy codes, SKUs, clause numbers, and people’s names are lexical problems. Pure vector search will happily retrieve a neighbour that “feels” like clause 14.2 and is not. We run keyword and vector, then merge. The merge is dull: filter ACL first, take the top lexical hits, take the top vector hits, rerank with a small model only if the set is messy.

If a query looks like an identifier — lots of digits, a known prefix — we skip the vibe layer and go lexical. Operators type identifiers. Your eval set should include them or you will ship a system that is poetic and useless.

Rerankers help when the merged set is large and noisy. They do not help when the right chunk was never a candidate. We have spent money on rerank APIs that politely reordered the wrong evidence. Fix recall first. Everyone wants to tune the last mile because it feels like research.

Operations, which is the actual product

• A way to see which chunk supported an answer. If you cannot cite, you cannot debug.
• A rebuild that is incremental. Full reindex on every PDF drop will be skipped on busy weeks.
• An owner for the corpus. Not “the AI team”. The person who already owns the policy in the real world.
• A fallback to keyword search the operator already trusts. Fancy retrieval that cannot degrade is a single point of failure.
• A stale-document alarm. If the source moved and the index did not, the product should say so, not answer from Thursday.

Citation UX is not decoration. Operators will not trust a paragraph that cannot open the page it came from. We show the section title and a jump link. If the extractor cannot give you a page, you are not done extracting.

If you are choosing a model this quarter because “RAG quality is low”, freeze the model and spend two weeks on the corpus. It is less glamorous. It is usually the whole job.