RAG vs fine-tuning: how to choose, with a decision rule

Short answer: use RAG when the answer depends on facts that change, or when you need citations. Fine-tune when the problem is behaviour — a fixed format, a specific tone, or a small model that must match a large one on one narrow task. Around 80 percent of applications start with RAG, and most mature systems end up with both.

Posed as a duel, this question cannot be answered, because the two techniques do not compete. They fix different failures.

The useful framing is this: RAG operates on the knowledge layer — what the model can see. Fine-tuning operates on the behaviour layer — how the model responds. They are independent. You can use either, or both, and the choice follows from which layer is failing.

Diagnose the failure first

Before comparing architectures, name what is going wrong. Almost every real complaint falls into one of two buckets.

"It gets facts wrong, or it does not know our stuff." That is a knowledge problem. The model was trained on a snapshot of the internet that does not contain your pricing, your policies, your codebase, or last week's incident. No amount of behavioural training fixes not knowing something. This is RAG.

"It knows the answer but says it badly." Wrong tone, inconsistent JSON, ignores the format, drifts over a long conversation, reasons in a way an expert in your field would not recognise. This is a behaviour problem, and retrieval will not fix it. This is fine-tuning.

If both are true — and they often are — you need both, and the order matters: retrieval first, because it is faster to build and it tells you how much of the problem was really about knowledge.

Six questions that decide it

Run your use case through these in order. The first "yes" is usually your answer.

1. Does the answer depend on data that changes — prices, policies, tickets, documentation, code? → RAG. Fine-tuning bakes facts into weights, and those weights go stale the moment the data changes. With retrieval you replace a document; with fine-tuning you retrain.

2. Do you need to cite sources, show provenance, or pass an audit? → RAG. A fine-tuned model cannot point at the paragraph that justified its answer. Retrieval can, and in regulated settings that is not a nice-to-have.

3. Must you be able to delete someone's data on request? → RAG. Deleting a document from a vector store takes seconds. Removing information from model weights means retraining from a clean dataset.

4. Is the output a strict schema — JSON, a regulatory form, a structured extraction — that prompting cannot hold reliably? → Fine-tune. Format discipline drifts over long contexts; weights do not.

5. Do you need a small open model to match a frontier model on one narrow, high-volume task, for cost or latency? → Fine-tune. This is the strongest commercial case in 2026: distil a large model into a tuned small one and cut inference cost by an order of magnitude on a task the base model already half-handles.

6. None of the above? → RAG first, then revisit fine-tuning only for the specific behaviours retrieval could not fix.

What each one actually costs

	RAG	Fine-tuning
Time to first working version	Days	Weeks
Main up-front work	Chunking, embedding, retrieval quality	Building and cleaning a training set
Updating knowledge	Replace a document	Retrain
Inference cost	Higher — you pay for retrieved context every call	Lower — no retrieved context
Latency	Adds roughly 0.5 to 2 seconds for retrieval	None added
Explains its answer	Yes, with sources	No
Fails by	Retrieving the wrong passage	Confidently reproducing a stale pattern

Two lines in that table are the ones teams underestimate. The hardest part of RAG is not the vector database, it is retrieval quality: chunking strategy, embedding choice, hybrid search, reranking, and knowing when to return nothing. And the hardest part of fine-tuning is not the training run, it is the dataset — a few thousand examples that genuinely represent the behaviour you want, which somebody has to produce and review.

The hybrid pattern, which is where most systems land

A customer support assistant is the canonical example. It must answer from current product documentation, and it must answer in the company's voice, at a consistent level of detail, in a fixed structure.

Retrieval supplies the facts at query time. A light fine-tune fixes the voice and the structure. Neither alone is sufficient, and the split is clean: fine-tune for how, retrieve for what.

A practical sequence for building it:

1. Ship RAG with a good base model and a careful prompt. Measure.
2. Look at what is still wrong. Classify every failure as knowledge or behaviour.
3. If most failures are knowledge, improve retrieval — chunking, reranking, metadata filters — not the model.
4. Only when behavioural failures dominate, and you have a few thousand curated examples, fine-tune.
5. Keep the evaluation set from step 1. Without it you cannot tell which leg is doing the work.

That last point is the one that separates teams who improve from teams who keep swapping architectures. Build the evaluation harness before the second architecture, or you will be arguing from anecdotes.

Where prompt engineering fits

Before either technique, exhaust the prompt. Few-shot examples, a clear system message, structured output constraints and a reasoning step will fix a surprising share of problems at zero infrastructure cost. The rule of thumb: if a careful prompt with three good examples cannot get you to about 70 percent, the problem is architectural and you should move on. If it gets you to 85 percent, you are tuning the wrong thing by reaching for training.

Learning this properly

The concepts above are cheap to read and expensive to get right. If you want the implementation depth — chunking strategies, embeddings, hybrid retrieval, reranking, evaluation harnesses, and where each one breaks — our LLM Development course is built around exactly that pipeline. For the training side, Machine Learning covers the fundamentals that make a fine-tune more than a script you copied, and MLflow and MLOps covers the part everyone skips: tracking which version of which model, trained on which data, is currently answering your customers.

Frequently asked questions

What is the difference between RAG and fine-tuning?

RAG retrieves relevant documents at query time and puts them in the prompt, so the model answers from current external knowledge. Fine-tuning continues training the model on examples so it internalises a behaviour — a tone, a format, a reasoning style. RAG changes what the model knows, fine-tuning changes how it responds.

Which is cheaper, RAG or fine-tuning?

RAG is cheaper to build and far cheaper to update, since changing knowledge means replacing a document rather than retraining. Fine-tuning is cheaper per request at high volume, because you are not paying for retrieved context on every call. For a low-volume application RAG almost always wins on total cost.

Does fine-tuning stop hallucinations?

No, and expecting it to is the most common mistake. Fine-tuning on facts teaches the model the shape of your answers, not a reliable memory of them, and it will produce confident, well-formatted, wrong answers. Grounding responses in retrieved documents with citations is what reduces hallucination.

When should I fine-tune instead of using RAG?

Three cases: you need a strict output schema that prompting cannot hold, you need a specific tone or reasoning style that is itself the product, or you want a small open model to match a much larger one on one narrow task to cut cost and latency. The LLM Development course covers both pipelines.

Can you use RAG and fine-tuning together?

Yes, and most mature production systems do. The usual split is a light fine-tune for voice and output format, plus retrieval for the facts. They operate on different layers, so combining them is normal rather than exotic.

How much data do you need to fine-tune a model?

For style and format, a few hundred to a few thousand high-quality examples is often enough. For specialised reasoning, expect tens of thousands. Quality and consistency matter far more than volume: a thousand carefully reviewed examples beat fifty thousand scraped ones.

Where to go next

Build the retrieval version first and measure it. If you want the full pipeline with evaluation and deployment rather than a demo notebook, start with LLM Development, then AI Deployment for what happens after the model works on your laptop.