Module 10 · Lesson 4 of 11Free

Model routing: small models first, large when needed

Level

Intermediate

Topic

Cost, latency and scaling

Reading time

18 min

Malik’s draft answered one order of two

Deskora is a help desk app. A shop’s support agents use it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team. Deskora’s drafting assistant writes a reply for every ticket, and an agent edits and sends it.

Priya is a support agent at a kitchenware shop that uses Deskora. This morning she opens a ticket from Malik: 64 messages, two orders, and a customer who is angry now. The draft apologises for order 48213 and never mentions order 48377, the one that arrived broken. Priya rewrites it from scratch.

Tomás, the Deskora engineer who builds the AI features, runs the drafts on Claude Haiku 5.5, the cheapest Claude model. On Deskora’s eval set of 300 labelled tickets, Haiku’s drafts pass 258. Claude Sonnet 5.5, a larger model, passes 282, at about 20 times the price per draft. Most failures sit in three kinds of ticket: long threads, angry customers and tickets about several orders.

Tomás considers sending only those to Sonnet. This lesson answers one question: which drafts should go to the larger model, how does code decide, and what does that decision cost?

Pass counts, the share of each type of ticket, token counts and timings are estimates for this lesson’s scenario. The 300-ticket eval set, 60,000 drafts a day and the draft call’s token counts come from earlier lessons.

Most drafts don’t need the larger model

The eval set is stratified: it holds far more hard tickets than real traffic does, so every kind gets enough cases to measure (Building an eval set from real examples). Each ticket is assigned to one kind only, checked in the order long, angry, several orders.

Sonnet fixes 30 of Haiku’s 42 failures, but it also fails 6 tickets that Haiku passed. And 12 failures defeat both models: in most, search returned the wrong article, which no model can fix.

Now the price. The unit of cost is one draft call, the only call that changes model. The query rewrite and the reranker from Design: a support assistant over company docs stay as they are. Prices are as of October 2026 ^{3PricingAnthropic, Claude API documentation}.

All-Sonnet buys 24 more passes in 300 for about $1,950 a day. Most of that money goes to typical-length tickets, and on typical tickets Sonnet passes only 2 more of 200. A router should send the hard tickets to Sonnet and keep the rest on Haiku. The hard part is knowing which ones are hard.

Three ways to choose the model

Each way answers “Haiku or Sonnet?” at a different moment, with different information. So each has its own cost, latency and way of failing.

Rules. Code reads facts it already has: the thread is longer than 40 messages, the thread holds two or more order numbers, the language is one where the eval shows Haiku is weaker, or the text matches a list of angry words. A match goes to Sonnet. Rules cost nothing, and anyone can read why a ticket went where. They fail on what they can’t see. Many angry customers use no angry words, so the word list catches only 12 of the 30 angry tickets. And “long” sends every long ticket, though Haiku already handled 34 of the 50.

A routing classifier. A small model call reads the ticket and predicts one thing: will Haiku’s draft fail? A score above a threshold sends the ticket to Sonnet. RouteLLM trains such routers on human preference data, choosing between a strong and a weak model per request, and reports costs cut by over 2 times in some cases without lower quality ^{2RouteLLM: Learning to Route LLMs with Preference DataIsaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, Ion Stoica, arXiv:2406.18665, 2024}. For Deskora, a Haiku call reading the latest messages costs about $0.000175, about $13 a day. Its weakness is timing: it guesses before the draft exists. It also needs labelled examples where both models ran.

A cascade. Haiku writes the draft. A check tests it. If the check fails, Sonnet writes a second draft and the first is thrown away. FrugalGPT studied cascades like this over a list of models and reports matching the best single model with up to 98% lower cost on its benchmarks ^{1FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving PerformanceLingjiao Chen, Matei Zaharia, James Zou, arXiv:2305.05176, 2023}. A cascade judges the real output, not a prediction. It pays for that twice: a check on every draft, and two drafts plus extra waiting on every escalated ticket.

The cascade’s cost depends on the check, so the check needs its own price before any comparison.

The check is a second bill

A check that runs on 75,000 drafts a day must be cheap. Tomás has three kinds to choose from.

• Code checks, free. The draft stopped early (stop_reason is max_tokens), an order number from the thread is missing from the draft, or the marks from the drafting assistant’s existing checks fire: an invalid citation, a possible promise of a refund.
• A small model check: one Claude Haiku 5.5 call at low effort that grades the draft on four criteria and quotes its evidence.
• The Sonnet judge from LLM-as-judge, the one that grades the eval set.

So code checks run first, and a draft that fails one escalates without a model check. Everything else gets the Haiku check. The Sonnet judge stays on 300 eval tickets, not 75,000 drafts.

The real drafting call sends sources as document blocks with citations; plain text keeps this sketch short.

import json
from pathlib import Path

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
SMALL, LARGE = "claude-haiku-5-5", "claude-sonnet-5-5"
CRITERIA = ["answers_latest_message", "covers_every_order", "tone_fits_customer", "stays_within_sources"]
GRADE = {"type": "object", "additionalProperties": False, "required": ["verdict", "evidence"],
         "properties": {"verdict": {"type": "string", "enum": ["pass", "fail"]}, "evidence": {"type": "string"}}}
CHECK = {"type": "json_schema", "schema": {"type": "object", "additionalProperties": False,
         "required": CRITERIA, "properties": {c: GRADE for c in CRITERIA}}}


def call(model: str, prompt_file: str, user: str, output_config: dict) -> anthropic.types.Message:
    return client.messages.create(model=model, max_tokens=4000, system=Path(prompt_file).read_text(),
                                  output_config=output_config, timeout=30,
                                  messages=[{"role": "user", "content": user}])


def text_of(message: anthropic.types.Message) -> str:
    return "".join(block.text for block in message.content if block.type == "text")


def failed_checks(draft: anthropic.types.Message, case: str, orders: list[str]) -> list[str]:
    if draft.stop_reason == "max_tokens":
        return ["cut_off"]  # code checks first: free, and they skip the model check
    if any(number not in text_of(draft) for number in orders):
        return ["missing_order"]
    grades = call(SMALL, "prompts/check-v3.txt", f"{case}\n<draft>\n{text_of(draft)}\n</draft>",
                  {"effort": "low", "format": CHECK})
    if grades.stop_reason != "end_turn":
        return ["no_verdict"]  # a check that can't answer counts as a fail
    verdicts = json.loads(text_of(grades))
    return [c for c in CRITERIA if verdicts[c]["verdict"] == "fail"]


def draft_reply(ticket: str, sources: str, orders: list[str]) -> dict:
    case = f"<sources>\n{sources}\n</sources>\n<ticket>\n{ticket}\n</ticket>"
    ask = case + "\nDraft a reply to the customer's latest messages."
    first = call(SMALL, "prompts/draft-v9-haiku.txt", ask, {"effort": "medium"})
    failed = failed_checks(first, case, orders)
    if not failed:
        return {"model": SMALL, "text": text_of(first), "escalated_for": []}
    second = call(LARGE, "prompts/draft-v9-sonnet.txt", ask, {"effort": "medium"})  # own prompt, effort set
    return {"model": LARGE, "text": text_of(second), "escalated_for": failed}

escalated_for goes into the draft log, so Tomás can count why drafts escalate. The Sonnet draft gets no second model check; Priya still reads every draft before sending.

Five options on the same 300 tickets

Tomás runs every option on the eval set, with the same judge and code checks. Daily costs use production volume, where hard tickets are rarer.

The cascade passes the most of the routed options: its check caught 34 of Haiku’s 42 failures, and Sonnet fixed 23 of those. The check also escalated 16 good drafts, and Sonnet failed 1 of them. Rules spend almost twice as much for fewer passes, because they send drafts Haiku had already done well.

Decision: a cascade, escalating on any failed check

Including the rewrite and the reranker, drafting goes from about $267 to about $741 a day. Running a small open model yourself is another way to cut this bill; Self-hosting open models asks when they pay.

Watching the router after launch

A router drifts quietly as tickets, prompts and models change. Tomás launches the cascade in shadow mode for a week: Sonnet drafts are made for escalated tickets but not shown, so he can measure the real escalation rate before paying for it in front of Priya.

Any change to a prompt, a model or the check reruns the eval set on the whole cascade.

Try it yourself

Send one hard message to both models and compare what each bills. You need a Claude API key in ANTHROPIC_API_KEY; without one, skip this.

for model in claude-haiku-5-5 claude-sonnet-5-5; do
  curl -s https://api.anthropic.com/v1/messages \
    -H "content-type: application/json" \
    -H "x-api-key: $ANTHROPIC_API_KEY" \
    -H "anthropic-version: 2026-10-21" \
    -d '{"model": "'"$model"'", "max_tokens": 1500, "system": "Draft a short reply for a support agent. Never promise a refund.", "messages": [{"role": "user", "content": "Customer: This is the third time I write. Order 48213 is late and order 48377 came broken. Fix both today."}]}'
  echo
done

An example of the end of each response, from one run; yours will differ:

{"usage": {"input_tokens": 61, "output_tokens": 412, "output_tokens_details": {"thinking_tokens": 270}}}
{"usage": {"input_tokens": 61, "output_tokens": 735, "output_tokens_details": {"thinking_tokens": 560}}}

Check whether each reply covers both orders, and compare thinking_tokens: with no effort set, Sonnet runs at its default, high, and Haiku at medium. Both calls together cost under 2 cents, as of October 2026.

Where this shows up

• Design: scaling a chat assistant to a million users: routing where a person waits for every token, so a cascade’s second draft is felt.
• Design: an AI customer support platform: one routing decision per feature, each measured on its own eval set.

Escalated drafts still run as soon as a ticket goes quiet. Some of Deskora’s model work could wait hours, and the next lesson asks what that is worth.

Common mistakes

Going deeper

Check your understanding

Summary