Anthropic ran a specific experiment in September 2024. They measured how often a retrieval system failed to surface the right passage in its top 20 results, then tried to improve it. Baseline failure rate: 5.7%. Adding generated context to each chunk before embedding brought it to 3.7%. Combining that with keyword search brought it to 2.9%. Adding a reranking step brought it to 1.9%, a 67% reduction overall.
Notice what that sequence tells you. The single biggest gains in retrieval-augmented generation come from unglamorous engineering on the retrieval side, not from swapping models or fine-tuning anything. Also notice that even the best configuration still failed about 2% of the time.
This article covers how RAG actually works, the four options a business realistically has when a model does not know something it should, and how to pick between them using cost arithmetic instead of vibes.
What RAG is and where it came from
The term comes from a 2020 paper by Patrick Lewis and colleagues at what was then Facebook AI Research, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, presented at NeurIPS that year. Their framing is still the clearest way to think about it: a model has parametric memory, the knowledge baked into its weights during training, and you can bolt on non-parametric memory, an external index the model queries at answer time.
Their original setup used a dense vector index of Wikipedia and a neural retriever feeding a sequence-to-sequence generator. The reported outcome was state-of-the-art results on three open-domain question answering benchmarks, plus generated language they described as more specific and more factual than the parametric-only baseline.
Six years later the architecture is the same. The pipeline is: split your documents into chunks, convert each chunk into a vector, store the vectors, convert the user’s question into a vector, find the closest chunks, paste them into the prompt, and ask the model to answer using only that material.
The pipeline, stage by stage
Chunking
You cannot embed a 300-page manual as one vector and expect useful retrieval. Documents get split into chunks, typically a few hundred tokens each. Anthropic’s write-up describes working with chunks usually no more than a few hundred tokens.
Chunking is where most RAG systems are quietly broken. Split at a fixed character count and you cut tables in half and orphan headings from the text beneath them. Split too large and each chunk contains mostly irrelevant material that dilutes the embedding.
The context problem is worse. A chunk reading “revenue increased 3% quarter over quarter” is useless in isolation because nothing in it says which company or which quarter. Anthropic’s fix, contextual retrieval, uses a cheap model to write a 50 to 100 token situating description for each chunk and prepends it before embedding. Using prompt caching, they put the one-time cost at $1.02 per million document tokens.
Embeddings and vector storage
Each chunk becomes a vector. OpenAI’s embeddings documentation describes two current models: text-embedding-3-small at 1,536 dimensions and text-embedding-3-large at 3,072, both accepting up to 8,192 input tokens. A dimensions parameter lets you truncate vectors to save storage; the docs note the large model shortened to 256 dimensions still beats the older ada-002 model at its full size.
Those vectors need to live somewhere searchable. You do not necessarily need a dedicated vector database. pgvector adds vector similarity search to PostgreSQL, keeping vectors alongside relational data with normal ACID guarantees and joins. It supports two approximate-nearest-neighbour index types with a clear tradeoff: HNSW gives better query performance but builds slowly and uses more memory, while IVFFlat builds faster and lighter with weaker query performance.
For most companies with under a few million chunks, adding an extension to a database you already run beats adopting new infrastructure.
Hybrid search
Vector search finds semantic matches. It is notably bad at exact tokens: part numbers, error codes, surnames, statute references. Ask for “error TS2345” and a pure embedding search may return chunks about TypeScript errors generally while missing the one document containing that exact string.
Keyword search, usually BM25, does the opposite. It nails exact matches and fails on paraphrase. Hybrid search runs both and merges the ranked lists. Anthropic’s numbers show why this is standard practice: contextual embeddings alone cut failure rate 35%, but combined with contextual BM25 the reduction reached 49%.
Reranking
Retrieval is tuned for recall. You pull 50 or 100 candidate chunks to make sure the right one is somewhere in there, then you need to decide which 5 to actually put in the prompt.
A reranker is a smaller model that scores each candidate against the query directly, rather than comparing pre-computed vectors. It is slower per item but far more accurate, and running it over 100 candidates is cheap. In Anthropic’s experiment, reranking took the failure rate from 2.9% to 1.9%.
Reranking also addresses a subtler problem. The Lost in the Middle research found models retrieve information best at the start or end of the input and worse from the middle. Fewer, better-ordered chunks beat more chunks.
Evaluating RAG without fooling yourself
The most common mistake is judging a RAG system by reading a dozen answers and concluding it seems fine. RAG has two failure surfaces, and they need separate measurement: retrieval can fetch the wrong material, or generation can misuse the right material.
The open-source Ragas framework separates these explicitly. Its metric set includes context precision (was the retrieved material relevant), context recall (did retrieval find everything needed), faithfulness (is the answer actually grounded in what was retrieved), response relevancy (does the answer address the question), and noise sensitivity (how badly does irrelevant context derail the answer).
Build a set of 100 or so real questions with known correct answers before you build anything else. Without it, every change to chunk size or retrieval depth is a guess.
The four options, honestly compared
When a model does not know something, you have four moves. They are not competitors so much as tools for different problems.
| Approach | Fixes | Does not fix | Setup effort | Ongoing cost driver |
|---|---|---|---|---|
| Prompt engineering | Format, tone, task framing | Missing knowledge | Hours | Slightly longer prompts |
| RAG | Missing or changing facts | Behaviour, style, output format | Weeks | Retrieval infra plus context tokens |
| Long context | Small, stable document sets | Large corpora, cost at volume | Days | Input tokens, heavily |
| Fine-tuning | Consistent behaviour and format | Facts that change | Weeks plus data collection | Training runs and retraining |
OpenAI’s own model optimization guidance is refreshingly unsalesy: prompt engineering may be sufficient for many use cases. It lists fine-tuning’s genuine wins as handling a wider variety of inputs than fit in one context window, cutting token cost and latency through shorter prompts, and being able to run a smaller cheaper model on a specific task. It describes four methods: supervised fine-tuning, vision fine-tuning, direct preference optimization for tone and style, and reinforcement fine-tuning for complex reasoning on reasoning models.
Fine-tuning is also cheaper than it used to be. The LoRA method (Hu et al., 2021) freezes the pretrained weights and trains small low-rank matrices instead, which the authors reported cutting trainable parameters by up to 10,000 times versus full fine-tuning of a 175-billion-parameter model, with roughly a threefold reduction in GPU memory and no added inference latency.
None of that changes the core rule: fine-tuning teaches behaviour, not facts. If your policy document changes monthly, fine-tuning bakes in a snapshot that goes stale and cannot be corrected without another training run.
The long-context temptation
With million-token context windows widely available, dumping the whole corpus into the prompt is tempting. Run the arithmetic before you commit.
Using Anthropic’s published API pricing, Claude Sonnet 5 costs $2 per million input tokens. A RAG query carrying 4,000 tokens of retrieved context costs about $0.008 in input. Stuffing 200,000 tokens of documents into every request costs about $0.40. At 10,000 queries a month that is roughly $80 versus $4,000.
Prompt caching narrows the gap considerably when the same large prefix repeats. Cache reads are billed at 0.1x the base input rate, with cache writes at 1.25x for a five-minute time-to-live or 2x for an hour. A fully cached 200,000-token prefix drops to roughly $0.04 per call. Better, still five times the RAG cost, and only if your traffic pattern actually keeps the cache warm.
Long context also does not solve accuracy. The Lost in the Middle result applies directly: more material in the window means more chances the relevant passage sits in the position models handle worst.
A decision framework that works
Run these questions in order and stop at the first yes.
- Does the model already know this, and you just need better instructions? Fix the prompt. Add examples. This is hours of work and it resolves a surprising share of “the AI is wrong” complaints.
- Is your knowledge base small, stable, and used by few queries? Under roughly 100,000 tokens and low volume, put it in the prompt with caching. Skip the retrieval infrastructure entirely.
- Does the answer depend on facts that change, or on a corpus too big for one window? Build RAG. This covers most customer support, internal documentation, policy lookup, and product catalogue use cases.
- Do you need consistent format or domain behaviour that prompting cannot pin down, across thousands of calls a day? Fine-tune. Structured extraction, classification into a fixed taxonomy, and house style are the strong cases.
- Both? Fine-tune for behaviour and use RAG for facts. These compose cleanly and the combination is common in production.
One more filter. If you cannot articulate what “correct” looks like well enough to build a 100-question evaluation set, you are not ready for either RAG or fine-tuning. Both require that set to be worth anything.
What this means for you
If you are the person deciding this for a team, the practical takeaways:
- Budget most of your effort for retrieval quality, not model selection. Chunking, hybrid search and reranking moved failure rate by 67% in Anthropic’s test. Model choice moves it far less.
- Start with your existing database. pgvector on Postgres is enough for millions of chunks and avoids a new operational dependency.
- Always run hybrid search. Pure vector search fails on exactly the identifiers your users type most.
- Measure retrieval and generation separately. Otherwise you will spend weeks tuning prompts for a retrieval problem.
- Do not fine-tune to add knowledge. It is the wrong tool, it goes stale, and it is much harder to audit than a citation.
- Price your long-context plan at real volume before adopting it. The per-call difference looks trivial and the monthly difference does not.
Frequently asked questions
Does RAG stop hallucinations?
It reduces them and, more importantly, makes them checkable, because the answer cites retrieved passages a human can verify. It does not eliminate them. A model can still misread correct context, and if retrieval fails the model may answer from parametric memory anyway.
How big should chunks be?
Commonly a few hundred tokens, but the honest answer is that it depends on your documents and should be tested against your evaluation set. Split on structural boundaries such as sections and headings rather than raw character counts.
Do I need a dedicated vector database?
Usually not at first. Postgres with pgvector handles substantial workloads. Consider specialised systems when scale, filtering complexity, or query latency actually become the constraint.
Is fine-tuning worth it for a small business?
Rarely as a first move. It requires curated training data, an evaluation harness, and retraining discipline. Parameter-efficient methods like LoRA lowered the compute cost sharply, but the data work is still the expensive part.
Will bigger context windows make RAG obsolete?
Not on current evidence. Cost scales with tokens in the window, and retrieval accuracy degrades as it fills. Long context and RAG increasingly work together: retrieve broadly, then let a large window hold more of what you retrieved.
Where to put your engineering hours
Most disappointing RAG deployments are not model failures. They are chunking that severed context, vector-only search that missed exact terms, and no evaluation set to reveal either problem.
The measured wins are all in that unglamorous layer. Contextualise your chunks. Run keyword and vector search together. Rerank before you assemble the prompt. Test retrieval separately from generation. Those four moves cost less than a fine-tuning project and, on the published numbers, do considerably more.
Reach for fine-tuning when the problem is genuinely how the model behaves rather than what it knows. Reach for long context when your corpus is small and stable. Everything else is a retrieval problem, and retrieval problems reward engineering, not spending.
Sources
- arXiv — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Anthropic — Introducing Contextual Retrieval
- arXiv — LoRA: Low-Rank Adaptation of Large Language Models
- arXiv — Lost in the Middle: How Language Models Use Long Contexts
- OpenAI — Embeddings Guide
- OpenAI — Model Optimization and Fine-Tuning
- GitHub — pgvector: Open-Source Vector Similarity Search for Postgres
- Ragas — Available Evaluation Metrics
- Anthropic — Prompt Caching
- Anthropic — Model Pricing
Image credit: Photo: Mennonite Church USA Archives — No restrictions (via Wikimedia Commons)
