A team wants their AI assistant to answer questions about company policy. Someone says the model needs to be fine tuned on the policy documents so it "knows" them. Weeks of data preparation follow. The result answers in the right tone and still gets the leave policy wrong, because policies changed after training and the model cannot tell.
For most business use cases, the better first move is retrieval: find the relevant text at the moment of the question and give it to the model. Fine tuning has a place, but it is usually the second step, not the first.
Two different jobs
The confusion comes from treating both techniques as ways to teach a model facts. They solve different problems.
Retrieval supplies knowledge. The system searches your documents, picks the passages most relevant to the question and includes them in the prompt. The model answers using what it was just shown. This is often called retrieval augmented generation.
Fine tuning adjusts behaviour. It trains the model further on examples so that it follows a style, a format or a task pattern more reliably. It changes how the model responds, and it is a poor way to store changing facts.
Put simply: retrieval is for what the model should know, fine tuning is for how it should act.
Why retrieval comes first
Knowledge stays current. Update a document and the next answer reflects it. With fine tuning, every change means another training run.
You can show sources. Because the answer is built from specific passages, you can cite them. A person can check the original. That is hard to do with knowledge baked into weights.
Access control is possible. You can filter what is retrieved by who is asking, so a user only receives answers drawn from documents they may see. A fine tuned model cannot forget a document for one user and remember it for another.
It is cheaper to start. Building a search index over documents is faster and less expensive than curating training data and running experiments. You learn quickly whether the idea works.
Errors are easier to diagnose. When an answer is wrong, you can ask whether the right passage was found. That is a concrete, testable question.
What good retrieval needs
Retrieval is not a quick demo that works on its own. Quality depends on a few unglamorous details.
Chunking. Documents are split into pieces. Too small and each piece lacks context. Too large and the relevant sentence is drowned. Splitting along natural boundaries such as headings and sections usually works better than fixed sizes.
Search method. Semantic search uses embeddings to match meaning, and it handles paraphrase well. Keyword search is better for exact names, codes and identifiers. Combining both often beats either alone.
Metadata. Store the document title, date, owner and permissions beside each chunk. This lets you filter, prefer recent versions and cite properly.
Ranking. Retrieving the top few passages and then reordering them with a more careful scoring step can lift quality noticeably.
Clean source material. Contradictory, outdated or badly scanned documents produce confidently wrong answers. Fixing the documents often does more than any tuning.
Evaluate before you decide
Do not argue about the approach in the abstract. Build a small test set of real questions with known good answers, perhaps fifty to a hundred. For each, record whether the correct passage was retrieved and whether the final answer was right.
This separates two failure types. If the right passage was not found, improve search. If it was found and the model still answered badly, the problem lies in the prompt or the model. Only the second kind sometimes points towards fine tuning.
When fine tuning earns a place
Once retrieval is working, fine tuning can help when:
- You need a consistent output format or house style that prompting cannot hold reliably.
- A narrow, repetitive task, such as classifying messages into your categories, needs higher accuracy or lower cost per call.
- You want a smaller, cheaper model to match a larger one on a specific task.
- Domain language is unusual enough that the base model keeps misreading it.
Even then, the two combine well. A tuned model that handles format and tone, fed by retrieval for facts, is a common and sensible design.
Data and privacy
Whichever route you take, know where your data goes. Documents sent to a provider for embedding or tuning may include personal or confidential information, so check retention terms and your duties under the Nigeria Data Protection Act 2023 or GDPR. Retrieval has a small advantage here: documents can stay in your own store, and only the relevant passages travel with each request.
A practical order of work
Start with a clear question set. Build simple retrieval with sensible chunking and citations. Measure. Improve search and source quality until results plateau. Only then ask whether the remaining errors are about behaviour, and if they are, consider tuning with a clear target and a way to measure the gain.
Most teams who skip to fine tuning are reaching for the more impressive tool. Retrieval is less glamorous, and it solves the problem more often.