Module 01 · Lesson 6 of 13Free

Hosted and open-weight models: the tradeoffs

Level

Intermediate

Topic

AI engineering fundamentals

Reading time

16 min

A shop wants its tickets to stay in Germany

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team.

Feldkamp, a large garden-supply shop in Germany, wants to sign up. Its head of support, Anja, sends one question first: “Can our ticket text be processed only in Germany?” Feldkamp’s contracts with its own customers promise that. It is a data residency rule, and it covers processing, not only storage.

Tomás, the Deskora engineer who builds the AI features, checks the summary feature. It sends each long ticket thread to Claude Haiku 5.5 through the Claude API. As of October 2026, that API lets a request choose where the model runs from two values only: global or us ^{1Data residencyAnthropic, Claude API documentation}. Germany isn’t one of them.

So Tomás compares staying on a hosted API with running a model Deskora downloads. This lesson answers one question: what does each option cost Deskora, and which one should it choose for Feldkamp?

Feldkamp’s numbers are estimates for this lesson’s scenario: 6,000 new tickets a day, 2,000 of them long enough to get a summary, one in four of those very long.

Three ways to run the same summary

The summary call itself doesn’t change. What changes is who holds the model, whose computers run it, and how Deskora pays.

Most readers guess the downloaded model wins. Here is where that belief comes from, and where it breaks.

What the three options are

A model is a large file of numbers learned in training, called its weights. Running the model on an input is called inference.

• Hosted model API. The provider keeps the weights and runs the model. You send text and pay per token, as in Calling a model API. Claude is offered only this way: you can’t download its weights.
• Open-weight model. The publisher lets anyone download the weights ^{4Dual-Use Foundation Models with Widely Available Model Weights ReportNational Telecommunications and Information Administration, U.S. Department of Commerce, 2024}. With the file and enough GPUs, you can run the model yourself. That is self-hosting. “Open-weight” isn’t the same as open source: the training data and code are usually not published, and the licence sets what you may do.
• Inference provider. A company that runs open-weight models on its own GPUs and sells a per-token API for them. It is the middle option: you choose the model, and often the region, but someone else runs the GPUs.

There is a fourth route for the hosted model itself. Cloud platforms that resell Claude, such as Amazon Bedrock, set the region through the endpoint you call, with their own regional prices ^{1Data residencyAnthropic, Claude API documentation}. If one offers Claude Haiku 5.5 with processing pinned to Germany, Deskora could keep its current model.

Per token against per GPU-hour

Hosted and inference-provider APIs charge for work done. Self-hosting charges for time rented. That one difference drives the whole cost comparison.

The unit of cost is one completed summary. It takes one call; this estimate leaves out retries and output checks. Token counts are the ones from lesson 03: a typical summary is about 4,150 input tokens and 380 output tokens, with 300 of those assumed thinking tokens. A long one is about 15,150 input and 900 output, with 800 assumed thinking tokens. On Claude Haiku 5.5, thinking is billed as output, and usage.output_tokens_details.thinking_tokens reports the real amount.

The GPU price and throughput are assumptions for this lesson, not quotes. Change them in the calculator below; the shape of the result holds.

Utilization is the share of paid time the GPUs spend working. It decides the self-hosted cost per summary: the same $120 spread over 2,000 summaries or 150,000.

Where the lines cross

A per-token bill starts at zero and rises with every summary. A GPU bill is flat until you need more GPUs. The break-even volume is where they meet:

So break-even isn’t a property of self-hosting. It depends on four numbers: the GPU price, the throughput, how busy you keep the GPUs, and which hosted price you compare against. Against a cheap model it is far away. Against an expensive one it is close, but only if the open model does the job as well, and that is a separate question.

Staff time isn’t in these numbers either. Someone has to run those GPUs.

Data, quality, operations and versions

Cost is one of five differences. For Feldkamp, the first one decides whether an option is allowed at all.

Data control. Running open weights lets you use a model without sharing data with a third party ^{4Dual-Use Foundation Models with Widely Available Model Weights ReportNational Telecommunications and Information Administration, U.S. Department of Commerce, 2024}. On rented GPUs the cloud host still owns the hardware, but no model provider sees the text. Deskora picks the region and how long anything is kept.

Quality. An open model isn’t better or worse by label. Tomás runs the candidate on a few hundred real tickets and scores the summaries the same way he scores Haiku’s. Evals teaches how. Weighing quality against price is the subject of Choosing a model.

Operations. Self-hosting turns a function call into a service Deskora runs. Someone sizes the GPUs for the busy hours, keeps the serving software patched, watches for a GPU failing at 3 a.m. and plans capacity for 60-message tickets. If both GPUs fail, Feldkamp’s agents read every long thread in full until someone fixes it. Self-hosting open models shows how it is done.

Version pinning. A hosted model is retired one day, and requests to a retired model fail ^{2Model deprecationsAnthropic, Claude API documentation}. Anthropic promises at least 60 days’ notice, and lists Claude Haiku 5.5 as active until at least October 7, 2027, as of October 2026 ^{2Model deprecationsAnthropic, Claude API documentation}. A weights file Deskora stores never changes unless Deskora changes it. That matters when every summary must keep behaving the same for years. Model upgrades covers switching models safely.

Shrinking a model to fit a cheaper GPU is Quantization; training it further on Deskora’s tickets is Fine-tuning basics. Neither changes this decision for Feldkamp.

Choosing for Feldkamp

Try it yourself

You can see where a Claude request ran. This call asks for US-only processing, which costs 1.1 times the normal price on Claude 4.6 and later models ^{1Data residencyAnthropic, Claude API documentation}. With 200 output tokens at most, it costs under $0.0002. If you don’t have an API key, skip this.

curl https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -d '{"model": "claude-haiku-5-5", "max_tokens": 200, "inference_geo": "us", "messages": [{"role": "user", "content": "Summarize in one line: my garden hose arrived with a split in it."}]}'

An example of the usage you might see, shortened; your numbers will differ:

"usage": {"input_tokens": 24, "output_tokens": 61, "output_tokens_details": {"thinking_tokens": 38}, "inference_geo": "us"}

What to look for:

• inference_geo in usage says where the model ran. Remove the field from the request and it may say global.
• Try "inference_geo": "de". The API rejects it, which is Feldkamp’s problem in one line.

Where this shows up

• Design: an enterprise document Q&A product: large customers ask the same question about where their documents go.
• Design: an AI customer support platform: high, steady volume is where the GPU break-even becomes real.

Whichever route Tomás picks, a 60-message ticket must still fit in what one call can read. A smaller open model may hold less than Haiku 5.5, which is the problem in Context windows.

Common mistakes

• Comparing at full load. A GPU price divided by its peak throughput gives the best case. Divide the daily bill by the summaries you really serve.
• Counting one GPU and no people. One GPU is one failure away from no summaries, and on-call time is a cost too.
• Reading “open-weight” as “use it any way”. Licences differ. Read the model’s licence before building on it.
• Answering the cost question first. Check what the data rule allows, then compare the costs of the options that remain.

Check your understanding

Summary