RAG: when an LLM isn't smart enough on its own
What RAG is, why you need it, and the real problems when you build it: chunking, embedding drift, model swaps, latency. Lessons, not a tutorial.
An LLM knows a lot. But it doesn’t know your codebase. It doesn’t know your internal docs. It doesn’t know last month’s Slack thread about that production bug.
Fine-tuning costs money and time, and every time the data changes you have to train again. Stuffing everything into the context window works, but $0.03/1K tokens at 100K tokens per request isn’t sustainable.
RAG is another way.
What RAG is
Retrieval-Augmented Generation: instead of stuffing all the context into the prompt, you fetch only the relevant parts, then ask the LLM.
The basic flow:
- Index time: chunk documents → embed them as vectors → store them in a vector DB
- Query time: embed the question → similarity search → take the top-k chunks → put them in the prompt → call the LLM
The LLM no longer needs to “remember” everything. It only needs to read the right documents and answer from them.
Sounds simple. Once you’ve built it, you find the devil in the details.
Real-world difficulties
1. Chunking strategy
How to chunk is the first question, and the most important one.
Fixed-size chunking: split every 512 tokens with a 50-token overlap. Simple, easy to implement. But it cuts through sentences and paragraphs, and semantic context gets lost.
Semantic chunking: split by paragraph or section, or use sentence embeddings to detect topic boundaries. Better results, but more complex. And structured data like code needs yet another strategy.
Hierarchical chunking: document → sections → paragraphs, store all 3 levels, and have queries return small chunks along with their parent context. Good, but 3x the storage and higher latency.
There’s no one-size-fits-all. I’ve used fixed-size for markdown docs, semantic for prose, function-level for code. Each type gets its own pipeline.
2. Embedding quality
The embedding model decides retrieval quality. A bad model → search returns the wrong chunks → the LLM hallucinates even with context.
The problem: you don’t see this error right away. The LLM still answers, confidently, but from unrelated chunks. Silent failure: the most dangerous kind in AI systems.
You need an eval pipeline: a golden dataset of (question, expected_chunk_ids), measuring recall@k and MRR. Without numbers, you don’t know where you stand.
3. Reranking
The top-k from vector search isn’t always the best top-k. Vector similarity ≠ semantic relevance in every case.
Cross-encoder reranker: after fetching the top-20 chunks, use a smaller model to score each (question, chunk) pair and reorder them. Noticeably better results. The cost: ~100-300ms of extra latency.
For a production system, that’s a trade-off to weigh.
4. Context assembly
You have 5 relevant chunks. In what order do they go into the prompt? Most important chunk first, or last?
LLMs have the lost-in-the-middle problem: information in the middle of the context window gets “forgotten” more than information at the start and end. Put the most important chunk at the start or end of the prompt, not in the middle.
And don’t forget: the context window has a limit. If the chunks exceed the budget, you have to trim. Trim carelessly and you lose information.
The core problem: models change
This one is rarely talked about, but it hurts the most in operation.
Embedding drift
You index 100K documents with text-embedding-ada-002. Six months later OpenAI releases text-embedding-3-large: better and cheaper.
You can’t just switch the model for queries and keep the index. Embeddings from 2 different models aren’t compatible: their vector spaces are completely different. Similarity search will return garbage.
The fix: re-index everything. With 100K documents, that means:
- A non-trivial embedding API bill
- Downtime, or the complexity of running 2 indexes side by side
- Risk: what if the re-index fails halfway?
How to handle it:
Track embedding_model for each document in its metadata. When you swap models, run an incremental migration job: process batch by batch, mark migrated: true, and keep the old index until the migration is 100% done. Then flip the pointer.
documents/
- id: doc_123
content: "..."
embedding_model: "text-embedding-ada-002"
migrated: false
chunks: [...]
Dual-write during the transition if you need zero downtime. Twice the storage, but safe.
LLM version changes
gpt-4-turbo → gpt-4o: behavior changes. A prompt format that was optimal for the old model may not be optimal for the new one. Some prompt tricks that worked with Claude 2 are no longer needed with Claude 3.
Every time you swap the LLM, run a regression test over the whole eval set. Without testing, you don’t know how quality changed.
Version pinning: pin the model version in config, never "latest". Upgrade on purpose, not automatically. One painful upgrade beats production degrading for no clear reason.
llm:
model: "claude-3-5-sonnet-20241022" # pinned, not "claude-3-5-sonnet-latest"
embedding:
model: "text-embedding-3-small"
version: "2024-01-01"
Context length changes
New models usually have larger context windows. You can add more chunks or make them bigger, and suddenly response quality changes, because the model behaves differently with long context.
Changing the context budget needs re-testing, just like changing the model.
A few more gotchas
Stale index: documents get updated but nobody re-embeds them. You need change detection: hash the content, compare on every ingest, re-embed only when the content changed.
Query expansion: users’ short questions often don’t match long chunks well. Use an LLM to rewrite the query into a better form for retrieval before embedding it. One more LLM call, but retrieval quality goes up clearly.
Metadata filtering: don’t search the whole index if you know which category a document belongs to. Filter first, search second. Faster and more accurate.
Wrapping up
RAG isn’t plug-and-play. It’s a system with many moving parts: indexing pipeline, embedding model, vector DB, reranker, prompt engineering, eval.
Each part has its own failure mode. Silent failure is the most dangerous, because the LLM always returns something that sounds right.
Invest in an eval pipeline from the start. Track the embedding model version from day one. Pin everything. Test before you upgrade.
And chunk strategy: try a lot, measure carefully, don’t trust your intuition.
Related posts
AI for office work: 7 things you can do today, no coding needed
How to use ChatGPT, Gemini, Claude or Copilot at work: emails, document summaries, Excel formulas, meeting minutes. Copy-ready prompts included.
2026-09-29Jev: when AI only makes decisions, and doesn't write
TypeSafe AI launched Jev, a model that returns only typed decisions with probabilities and generates no text. How it works, and the open-source alternatives.