A decision that looks like a technical one
A team wants to add a language model to a product. Someone asks whether to call a hosted API or run an open weights model on their own servers. It sounds like an infrastructure question. It is really a question about cost, control, risk and who you want to depend on.
There is no universal answer, and anyone who offers one is selling something. What follows is a way to think it through, based on the trade offs rather than on any specific model, since specific models change every few months and this advice should outlast them.
Two terms, used carefully
Hosted APIs mean you send requests to a provider's servers and receive responses. The provider runs the model, the hardware and the scaling. You pay per use, typically by the amount of text processed.
Open weights means the trained model's parameters are published so you can download and run them yourself. This is not always the same as open source. Licences vary, and some restrict commercial use or impose conditions, so read the terms for the specific model you intend to use. Running the model is then your job, on your own hardware or rented machines.
There is also a middle path: managed services that host open weights models for you. That gives you some of the portability of open weights with less operational burden, at the price of some of the control.
What hosted APIs do well
Speed to a working product is the main advantage. You can have something running in an afternoon. There is no hardware to buy, no serving software to tune, and no capacity planning. Providers also tend to offer their most capable models through their APIs first, and those models are often hard to match with anything you can run yourself.
Costs scale with usage, which suits experiments and uneven demand. If you have ten users, you pay for ten users. You also inherit the provider's investment in reliability, safety measures and ongoing improvement.
The weaknesses are the mirror image. You depend on a third party for availability, pricing and terms. Models are updated or retired on the provider's schedule, which can change your product's behaviour. Your data leaves your environment, which raises questions that depend on the provider's terms and on your obligations.
What open weights do well
Control is the central advantage. You decide where the model runs, which version you use and when, if ever, it changes. Behaviour stays stable because nothing updates beneath you. You can run it inside your own network, which can simplify some data protection and residency questions, since personal data need not be sent to an external provider.
You can also adapt the model. Fine tuning on your own material, constraining outputs and tuning for a narrow task are all open to you, within the model's licence. For a well defined, high volume task, a smaller model that you run yourself can be both cheaper and fast enough.
The cost is operational. You need people who can deploy, monitor and scale model serving. You pay for hardware whether or not it is busy. Capable GPUs can be expensive or hard to obtain, and power and cooling matter, especially where electricity is unreliable. Quality may lag behind the strongest hosted models for demanding reasoning tasks, though the gap varies and shifts over time.
Where the money really goes
Comparing the two on cost is trickier than it looks.
- Hosted APIs have a variable cost with almost no fixed cost. The bill grows with traffic and with the length of prompts and outputs.
- Self hosting has a largely fixed cost: hardware or rented capacity, plus the engineering time to keep it running. The cost per request falls as utilisation rises.
So the crossover depends on volume and on how steady it is. At low or spiky volume, hosted is usually cheaper once you count staff time honestly. At high, steady volume, self hosting can win, provided you keep the hardware busy. Many teams underestimate the people cost, and overestimate their utilisation.
Questions that settle most cases
Rather than debate in the abstract, answer these for your specific use.
- What data goes in? If it is sensitive or regulated, can you accept a provider's terms, or must it stay within your environment? Check the data protection rules that apply to you.
- How hard is the task? Classification, extraction and routing often work well on smaller models. Open ended reasoning or complex writing may need the strongest available.
- How stable must behaviour be? If your product breaks when outputs shift slightly, pinning a version you control has real value.
- What is your volume? Estimate it honestly, including peaks.
- Who will run it? Do you have, or want to hire, the skills to operate model serving?
- What if the provider changes? Prices, limits and terms can move. How exposed are you?
Risks people forget
With hosted APIs, the forgotten risk is concentration. If one provider sits under several of your features, a pricing change, an outage or a policy shift hits all of them together. Rate limits can also bite at exactly the moment your product succeeds. Read the terms on how your inputs may be stored or used, and whether you can opt out of any retention.
With open weights, the forgotten risk is staleness and security. A model you run does not improve unless you upgrade it, and the serving software around it needs patching like any other service. Safety behaviour, such as refusing harmful requests or resisting manipulation, is also partly your responsibility rather than the provider's. Budget for testing it.
In both cases, a model's outputs can be wrong in fluent, convincing ways. Neither choice removes the need for human review where mistakes are costly.
You can mix them
The choice is not permanent, and it is rarely all or nothing. A sensible pattern is to start with a hosted API to prove the use case and learn what the task really requires. Keep the model behind a thin interface in your own code, so swapping it is a configuration change rather than a rewrite. Build an evaluation set from real examples, so you can test alternatives on your own task rather than on a public leaderboard.
Later, move the high volume, well understood parts to a model you run, and keep the hosted API for the hard or rare cases. Route requests to the cheapest option that does the job well enough.
Evaluate on your own work
Public benchmarks tell you how a model performs on someone else's questions. Collect a few dozen to a few hundred real examples from your own workflow, with the answers a careful person would give. Run each candidate against them and read the failures yourself. A cheaper model that gets your task right most of the time may beat a famous one, and you will only know by testing.
A closing rule of thumb
Start hosted, measure honestly, and move to open weights only when volume, data constraints or the need for control give you a clear reason. Whichever you choose, keep your own evaluation and your own abstraction layer. They are the assets that stay yours, and they are what let you change your mind without paying for it twice.