The demo that worked
Every AI feature has a good demo. Someone types a sensible question, the model answers well, and the room nods. Then the feature ships, real users arrive with typos, half formed requests, other languages and odd intentions, and the quality you saw turns out to describe about one input in a hundred.
The gap between the demo and production is not a model problem. It is a measurement problem. Most teams do not know how good their feature is, because they have never defined "good" in a way they can test. They find out from support tickets.
Evaluation is the work of finding out first. It is unglamorous, and it is the difference between a feature you can improve and one you can only hope about.
Decide what good means
Before writing any test, write down what the feature is for. Not "answer questions", but something a reviewer could check: "give the correct refund policy for the customer's plan, cite the relevant clause, and say so when it does not know."
From that sentence you can pull out separate qualities, and they rarely move together:
- Correctness: is the content true and complete for the task?
- Groundedness: does it stay within the source material, or invent?
- Format: does the output follow the structure your code depends on?
- Tone and safety: is it appropriate for your users and your brand?
- Refusal behaviour: does it decline what it should, and only that?
A response can be fluent and wrong, correct and badly formatted, or safe and useless. If you collapse all of this into one impression of "seems fine", you will miss the failures that matter most.
Build a test set from real work
The test set is the centre of the whole exercise. A good one is a collection of inputs, each with a note on what a good answer looks like.
Start small. Fifty carefully chosen cases beat five thousand careless ones. Take them from places that reflect reality: support logs, existing search queries, tickets, the questions your own staff ask each other. Where you have no history yet, write them yourself, and ask people who are not on the project to write some too, because the builders always ask the questions the system handles well.
Make sure the set covers more than the happy path:
- Ordinary requests, in the range of phrasing real people use.
- Ambiguous requests where the right move is a clarifying question.
- Requests the feature should refuse or redirect.
- Inputs that are empty, very long, misspelt or in a mix of languages.
- Cases where the source material is missing, outdated or contradictory.
- Attempts to push the feature outside its job, including instructions hidden in the input.
Keep the set versioned alongside your code. When a real failure appears in production, add it. Over time the test set becomes a record of everything you have learned the hard way.
Score it in more than one way
Some checks can be automatic. If the output must be valid JSON with certain fields, test that with code. If the answer must include a policy reference that exists in your documents, check it exists. These checks are cheap, exact and can run on every change.
Others need judgement. Here you have two options, and most teams should use both.
Human review is the reference. Have people who understand the domain grade a sample against a short, written rubric. Keep the scale simple, such as pass, partial, fail, and require a reason for anything that is not a pass. Check that two reviewers mostly agree. If they do not, your rubric is vague, not your reviewers.
Model graded review lets you scale. A second model scores answers against the rubric. It is fast and useful for catching regressions, but treat it as an instrument that needs calibration. Compare its grades with human grades on the same cases. Where they diverge, fix the rubric or stop trusting the grader for that quality. Models can favour longer, more confident answers whatever their accuracy, so do not let the grader be the only judge.
Test the system, not just the model
Most AI features are not a model. They are a pipeline: retrieval, prompts, tools, post processing, fallbacks. A failure may live in any stage, and judging only the final text hides which one.
If your feature retrieves documents, evaluate retrieval on its own. Did the right passage come back? If it did not, no prompt will save the answer. If your feature calls tools, check the sequence and the arguments, not only the final reply. If you post process, test that step with fixed inputs.
This separation turns a vague complaint ("it got worse") into a specific one ("retrieval for pricing questions dropped"). That is the point where you can actually fix something.
Run it every time something changes
An evaluation that runs once is a report. One that runs on every change is a safety net.
Changing a prompt, swapping a model version, adjusting retrieval, updating the documents behind it: each can shift behaviour in ways no one expected. Run the full set before the change goes out, compare against the previous run, and read the differences, not only the average. A score that holds steady can hide one category that collapsed while another improved.
Set thresholds for what blocks a release, especially for safety and refusal behaviour. A small drop in tone is a discussion. A drop in correct refusals is a stop.
Remember also that models are not deterministic. Run uncertain cases more than once and look at the spread. A feature that is right two times in three is not right.
Keep learning after launch
No test set anticipates everyone. Once users arrive, watch what happens, with proper regard for their privacy and your obligations on personal data.
- Give people a simple way to mark an answer as unhelpful.
- Sample real conversations regularly and grade them with the same rubric.
- Track the rate of fallbacks, retries and abandoned sessions.
- Feed each confirmed failure back into the test set.
Be careful with the cheapest signal, a thumbs up or down. Most people never click it, and those who do are not typical. Use it as a pointer to where to look, not as a score.
Where this leaves you
You will not reach certainty. You can reach a clear, written statement of what the feature does well, where it fails, and how you would know if that changed. That is enough to ship responsibly, to decide when a new model is worth adopting, and to answer the director who asks, reasonably, how you know it works.
The teams that do this are not the ones with the best models. They are the ones who found out about the problems before their users did.