A team spends two weeks wiring a large language model into a support tool. It classifies tickets into twelve categories. It works, costs a little more each month than anyone expected, and takes a second or two per ticket. Then someone trains a small model on the company's own labelled tickets. It runs on a modest server, answers in milliseconds, and does the job as well or better.
This keeps happening. Bigger is not always better, and for narrow, repeated tasks, it is often worse.
What small means
"Small model" is a loose term. We use it for models that are small enough to run cheaply on your own hardware, or even on a phone, and often specialised: a classifier, an embedding model, a compact language model tuned for one job. They sit well below the largest general models in size, and they are not trying to do everything.
Where they win
Narrow, high volume tasks. Routing tickets, detecting language, tagging documents, spotting duplicates, extracting a fixed set of fields. When the task is the same ten thousand times a day, a specialised model is cheaper and faster, and its behaviour is more predictable.
Latency. Some products need an answer inside a human blink: autocomplete, live suggestions, fraud checks during a payment. A round trip to a large hosted model often will not fit.
Cost at scale. Per request prices look small until volume arrives. A small model on a fixed server turns a variable cost into a flat one.
Privacy and control. Running a model inside your own environment means the data does not leave it. For regulated records, that can simplify a long conversation with a compliance team.
Poor connectivity. On device models work when the network does not. In markets where data is expensive or patchy, that is a feature users notice.
Where they lose
Small models are less capable at open ended reasoning, long documents, unusual requests and anything that needs wide general knowledge. Ask a compact classifier something outside its training and it fails without warning.
They also need work. Someone has to gather labelled examples, train or tune, evaluate honestly and maintain the result as the data changes. A large hosted model needs only a prompt. For a first version, or a low volume task, that convenience is worth paying for.
How we choose
We usually start with a large model, because it lets us learn what the task really is. We write down examples, measure how well it does, and see where the failures cluster.
Then we ask three questions. Is the task narrow enough to name? Is the volume high enough that cost or speed matters? Do we have, or can we cheaply get, labelled examples? If the answer is yes to all three, we try a small model against the large one on the same test set. Often we end up with both: a small model handles the routine ninety percent, and the large one handles the odd cases the small one is unsure about.
That pattern, a cheap first pass with an escalation path, is the most reliable win we have found.
Be careful with the comparison
Test on your own data, not on a public benchmark. Check performance on the rare, expensive cases as well as the average. And include the full cost: the engineer who maintains the small model is not free.
Try this next
Pick the single most repeated task your product sends to a large model. Collect a few hundred real examples with correct answers, run both options, and compare accuracy, speed and monthly cost side by side. The result may surprise you in either direction, and either answer is useful.