Choosing between fine-tuning, RAG and prompt engineering

Almost every AI project we scope starts with the same question: should we fine-tune a model on our data? The answer is usually "not yet". Fine-tuning, retrieval-augmented generation (RAG) and prompt engineering solve different problems, and picking the wrong one costs months. Here is how we decide.

What each technique actually changes

• Prompt engineering changes the instructions. It is the cheapest lever: you describe the task, the tone, the format and a few examples. It cannot teach the model facts it does not have.
• RAG changes what the model can see. At question time you retrieve relevant passages from your own content and place them in the prompt. It is how a general model answers questions about last week's policy.
• Fine-tuning changes how the model behaves. You train on examples of inputs and ideal outputs so the model learns a style, a format or a narrow skill. It is a poor way to store facts.

The most common mistake is using fine-tuning to inject knowledge. The model absorbs patterns, not a reliable database, so it will answer fluently and sometimes wrongly, and you cannot see where an answer came from.

Start with the cost of a wrong answer

Before choosing any technique, ask what happens when the assistant is wrong. If a wrong answer is mildly annoying, such as a bad product description draft, a good prompt may be enough. If it can mislead a customer, trigger a refund or breach a regulation, you need answers you can trace to a source, and that points to RAG with citations and a refusal path.

Then look at how fast the knowledge changes

If your source material changes weekly, such as prices, policies, stock or documentation, fine-tuning is a trap. Every change would mean retraining and re-validating. With RAG you update the index and the new answer is live in minutes. If the knowledge is stable and small enough to fit comfortably in the prompt, you may not need retrieval at all: put it in the instructions and move on.

A decision path we use

1. Write a good prompt with five to ten examples and run it against 50 real cases. If quality is acceptable, stop here.
2. If failures are mostly "it did not know this", add retrieval. Measure whether the right passages are found before you touch the model.
3. If failures are mostly format, tone or consistency, for example a strict JSON schema or a house style, collect 500 to 2,000 good examples and consider fine-tuning a smaller model.
4. If you need both fresh knowledge and a specific behaviour, combine them: fine-tune for behaviour, retrieve for facts.

When fine-tuning does pay off

We recommend it in three situations. One is high-volume, narrow tasks, such as classifying support emails into 40 categories, where a small tuned model is cheaper and faster than a large general one. Another is strict output formats that prompting cannot hold reliably. The third is domain language, such as clinical shorthand or legal drafting conventions, where general models consistently miss the register.

In each case you need clean labelled examples, an evaluation set the training data never touched, and a plan for retraining when the task shifts. If you cannot describe where the examples will come from, you are not ready.

Costs people forget

Prompt changes cost an afternoon. RAG costs an ingestion pipeline, an index and ongoing evaluation, typically a few weeks to build well. Fine-tuning costs data preparation first, which is the expensive part, plus training runs, hosting for a custom model and re-validation whenever the base model is upgraded.

The order matters. Move up the ladder only when you can show, with numbers from your own cases, that the current step is the bottleneck. Most teams find that a clear prompt plus retrieval gets them 90% of the way, and the money saved goes into evaluation, which is where reliability actually comes from.