Fine-tuning vs RAG
Use retrieval (RAG) when the model needs facts from your documents, especially facts that change or must be cited. Use fine-tuning when the facts are fine but the model's behavior is wrong: format, style, a narrow task, or a smaller model doing one job. Try prompting first, because it tells you which of the two problems you have.
How do fine-tuning, RAG and prompting compare?
The same seven questions, answered for each approach. Billing rows describe how providers charge; actual prices are on the prices page.
| Question | Retrieval (RAG) | Fine-tuning | Prompting |
|---|---|---|---|
| What it changes | What the model can see at answer time: passages pulled from your documents are added to the prompt. | The model itself: its weights (or a small adapter) are trained on your examples. | Only the instructions and examples sent with each request. |
| Best at | Facts that live in documents, change often, or must be cited back to a source. | Behavior: a fixed output format, a house style, a narrow classification or extraction task, a smaller model doing one job well. | Getting a first version working and learning what the task really needs. |
| Weak at | Teaching a new skill or format; answers are only as good as the passages retrieved. | Keeping facts current. New facts mean new training data and another run. | Long, fragile instructions; cost and latency grow with prompt length. |
| Data you need | The documents themselves, cleaned and split into passages, plus a set of test questions. | Hundreds to thousands of input and ideal-output pairs, plus a held-out evaluation set. | A handful of good examples and a test set. |
| How it is billed | Embedding the documents, storing vectors, and per-token inference on a base model (longer prompts cost more). | Training billed by training tokens (dataset tokens times epochs) or by training hour, then serving: per token on some platforms, per GPU-second or hour on dedicated hardware. | Per-token inference only. |
| Time to a first result | Days to a few weeks, mostly spent on document cleanup and retrieval quality. | Weeks, mostly spent on building the training and evaluation sets. | Hours to days. |
| Updating it later | Add or replace documents; re-embed what changed. | Train again on the new data, then re-run the evaluation set. | Edit the prompt. |
How do you tell a knowledge problem from a behavior problem?
Collect fifty wrong answers from your current prototype and sort them. The pile tells you where to spend.
Signs it is a knowledge problem (retrieval)
- The answer exists in a document the model never saw: a policy, a product sheet, a contract clause.
- The answer was right last quarter and is wrong now.
- Users need to see where an answer came from.
Signs it is a behavior problem (fine-tuning)
- The model has the information but returns it in the wrong structure, length or tone.
- It ignores instructions that are already in the prompt, because there are too many of them.
- The task is narrow and repeated thousands of times a day, so a smaller, cheaper model would do if it were trained for it.
- You need it to prefer one kind of answer over another, and you can show it ranked pairs (preference tuning, such as DPO).
If most of the pile is neither, the problem is often the evaluation itself: unclear correct answers, or test inputs that do not look like production traffic.
What does each approach cost to run?
The cost shapes differ more than the totals. Retrieval adds a little to every request (embedding lookups and longer prompts) and almost nothing up front beyond the engineering. Fine-tuning moves cost to the front: data preparation, the training job and evaluation, and then a hosting decision.
Training itself is often the smallest line. Platforms bill it by the token (dataset tokens multiplied by the number of epochs) or by the training hour, and published rates for small open models run to cents or dollars per million training tokens. Building a clean set of examples and the evaluation that proves the tuned model is better usually costs more than the run. The project scoper lists the questions that decide the total.
Hosting is the line teams forget. A tuned open model either runs on a provider's shared serving (where one exists for tuned models) or on dedicated GPUs billed by the second, minute or hour whether or not traffic arrives. Our post on hosting costs walks through the options.
Sources behind this section
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", arXiv 2005.11401 (2020)
- Rafailov et al., "Direct Preference Optimization", arXiv 2305.18290 (2023)
- Billing units: Google Vertex AI, Together AI and Fireworks AI pricing pages, read 7 Nov 2026 (links on the prices page)
Fine-tuning vs RAG: common questions
Is fine-tuning better than RAG?
Neither is better in general; they fix different problems. Retrieval gives the model facts it does not have. Fine-tuning changes how it behaves. If wrong answers come from missing or outdated facts, start with retrieval. If the facts are fine but the format, tone or judgment is off, fine-tuning is the stronger tool.
Can fine-tuning teach a model our company knowledge?
Partly, but it is an unreliable way to store facts. A tuned model can still state things that are not in its training data, it cannot point to a source, and every change to the facts means another training run. Most teams keep facts in retrieval and use fine-tuning for behavior.
Should we use RAG and fine-tuning together?
Often, yes. A common pattern is a retrieval layer for the facts and a model tuned to read retrieved passages, follow the house format and say when the passages do not contain the answer. Build and measure the retrieval part first, because it is cheaper to change.
When is prompting enough?
When a strong base model already does the task with clear instructions and a few examples, and the cost per request is acceptable. Prompting is also the right way to start any project: it shows what the model gets wrong before you spend money on data or training.
Does fine-tuning reduce cost?
It can. A small model tuned for one narrow task can replace a much larger model sent long prompts, so each request is cheaper and faster. Whether it pays back depends on volume: training, evaluation and hosting costs are fixed, so low-volume tasks rarely recover them. Our price tables show how each provider bills.
How do we know which approach worked?
Write the evaluation set before choosing. Two or three hundred real inputs with agreed correct outputs, scored the same way for every option, will settle most fine-tuning versus RAG arguments faster than opinion.
Get a quote
Only three things are needed: the company's ZIP code, where to send our answer, and a short sketch of the project in taps. The answer names providers whose published offers line up with it.
Keep data out of this form: no samples, records, weights or keys. Describing the task in words gives us enough to work with.
This site is an introduction service, and a provider it introduces you to may pay it a fee. Your details stay with us until you approve a named provider.
Received. Thanks for the outline.
- Next we set it against the published offers of fine-tuning providers: their models, methods and deployment options.
- An email follows naming the ones that fit, with a reason for each. Your details go nowhere until you approve one.
- From there, that provider deals with your team directly and prices the job. This site plays no part in the contract.