"A human reviews everything" is one of the most comforting sentences in AI products. It appears in policies, in sales decks, in risk assessments. Often it means that a person somewhere clicks "approve" on a screen, forty times an hour, without time to think.
That is a disclaimer. It moves blame to a person while giving them no real ability to catch a mistake. A genuine human in the loop is something you design.
Why review decays
People are poor at checking work that is usually right. If the system is correct ninety nine times in a hundred, the reviewer learns to expect correctness. Attention drops. The hundredth case, the one that mattered, goes through with the same quick click as the rest.
This is not laziness. It is how attention behaves under repetition. Aviation and industrial safety have studied the same effect for decades. If you design a review step that depends on constant vigilance, you have designed a step that will fail.
Volume makes it worse. If a reviewer is given the whole queue, the queue wins. They will clear it by approving.
What a real loop needs
Show the reviewer why. An answer alone invites a yes or no. An answer with its source, the fields it relied on and what it was unsure about invites a check. If the reviewer has to rebuild the context from scratch, they will not.
Route by risk, not by habit. Not every item deserves a person. Send the low risk, high confidence cases straight through, with sampling to check them, and concentrate people on the cases that are unusual, large or uncertain. Fewer items, closer attention.
Make disagreement cheap. If rejecting an output takes five clicks and a free text explanation, people will accept. Give them a fast way to correct, flag or send back, and a way to say why in one tap.
Authority has to be real. A reviewer who is judged on throughput is not free to stop the line. The person must have both the time and the standing to say no, and no penalty for doing so often.
Close the loop. Corrections should go somewhere useful: into a list of failures the team reviews, into test cases, into changes to the prompts or rules. If feedback vanishes, people stop giving it.
A hypothetical
Imagine an assistant that drafts replies to customer complaints at a bank. The weak design shows the agent a drafted reply with a "send" button. The stronger design highlights the claims in the draft that depend on account data, shows the data next to them, flags any promise of a refund as needing approval, and passes simple acknowledgements through with a daily sample check by a team lead.
Same model, same people. The second design gets attention where it counts.
Test the loop itself
You can measure whether review works. Seed the queue with known errors, a few at a time, and see how many are caught. Track how long each review takes, and be suspicious when it drops towards two seconds. Look at the rate at which reviewers change the output. A rate near zero is not good news. It usually means nobody is really looking.
Where this runs out
There are decisions where review is not enough. If the cost of one error is severe and cannot be undone, a rushed approval is no safeguard and the better answer may be limiting what the system is allowed to do at all. There are also tasks so cheap and so reversible that heavy review costs more than the errors it prevents. Match the weight of the control to the weight of the risk.
Alert fatigue is the same problem
The same pattern appears wherever a system raises flags that are mostly harmless. Hospital staff learn to tune out alarms that rarely matter. Security teams learn to skim alerts. An AI product that asks a human to confirm every output has created a new alert stream with the same flaw.
The remedy is the same as well: fewer, better flags. A system that interrupts rarely, and is right when it does, earns attention. A system that interrupts constantly trains people to stop listening.
The next step
Look at one human review step in your product today. Find out how long reviewers actually spend per item, and what fraction they change. Those two numbers will tell you whether you have a loop or a disclaimer.