RAG vs Fine-Tuning: How to Choose for Company Data
Every company that wants a language model to know its own data faces the same fork: retrieve the data at question time, or train it into the model. The vendors on each side present it as a technology choice. It is really an operations choice, and most businesses should take the same side of the fork most of the time.
How RAG Works
Retrieval-augmented generation keeps your data outside the model. When a question arrives, the system searches your documents, hands the best passages to the model as context, and the model answers from them. The knowledge lives in an index that updates when your documents do.
The consequences follow from that design. Answers can cite sources. Permissions can be enforced at retrieval time, so people only get answers from documents they may see. New knowledge arrives by updating documents, not by retraining. And when the documents contain no answer, a well-built system says so.
How Fine-Tuning Works
Fine-tuning continues a model's training on your examples, changing its weights. The knowledge, or more precisely the behaviour, moves inside the model.
Fine-tuning is excellent at teaching form: your tone of voice, your output format, a specialised task performed the same way thousands of times. It is poor at teaching facts you expect back reliably: the model may absorb them, but it cannot cite them, cannot respect per-user permissions on them, and forgets nothing when a fact becomes outdated short of training again.
The Comparison That Decides It
- Freshness. RAG reflects a document change in minutes. A fine-tune reflects it at the next training run.
- Traceability. RAG answers carry sources, which auditors and users can check. Fine-tuned knowledge is unattributable, which in regulated settings can be disqualifying on its own.
- Permissions. RAG can inherit document-level access control. Fine-tuning bakes everyone's data into one model for all its users.
- Cost shape. RAG costs engineering up front and retrieval at query time. Fine-tuning costs training runs every time the data or the base model changes, plus evaluation to prove the new version regressed nothing.
- Failure modes. RAG fails visibly, usually by retrieving the wrong passage, which you can see and fix. A fine-tune fails invisibly, by confidently blending stale and current knowledge.
Costs in Practice
For a mid-market knowledge assistant, the market ranges we use in our AI development cost guide apply: a production RAG system typically lands in the low-to-mid six figures of euros including integration and evaluation, then runs at query-time model costs. A fine-tuning programme looks cheaper per training run, current API pricing makes small fine-tunes cheap, but it buys a recurring obligation: every data refresh and every base-model upgrade means retraining and re-evaluating, forever. Teams consistently underestimate that annuity.
A Worked Example: The Mayster Parts Assistant
When we built an AI parts assistant for Mayster, the tempting path was fine-tuning on the parts catalogue. We went with retrieval instead, for reasons that generalise: the catalogue changes continuously, answers had to point at the exact catalogue entry, and a wrong part number presented confidently is worse than no answer. Fine-tuning on catalogue data would have baked thousands of part numbers into weights where they could neither be cited nor corrected; we made the same argument in general form in fine-tuning is the wrong default for catalogue data. The full build is in the Mayster case study.
Decision Checklist
Choose RAG when any of these is true: the data changes weekly or faster; answers must cite sources; different users may see different data; you are in a regulated industry; you want to swap base models without losing your investment.
Consider fine-tuning, usually on top of RAG rather than instead of it, when: you need a specific tone or output format at scale; you have thousands of high-quality input-output examples of a repeated task; latency or cost pressure argues for a small specialised model; and the knowledge itself still comes from retrieval.
The pattern that wins in production is boring: retrieval for knowledge, fine-tuning, if at all, for behaviour. If you want this decided properly for your data, our RAG implementation services start with exactly this assessment, and our guide on how to implement RAG covers the build that follows.