Module 01 · Lesson 4 of 13Free

Tokens: how models read text and charge for it

Level

Beginner

Topic

AI engineering fundamentals

Reading time

8 min

A 38-word message is not 38 tokens

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s conversation with the support team. Tomás, a Deskora engineer, is building a short summary for long tickets.

Here is the first message of one ticket, from a customer named Meera:

Hi, I ordered a pressure cooker on 12 March (order 58214) and it has not arrived yet. The tracking page says it is in transit, but nothing has changed for six days. Can you check where it is?

It has 38 words. But the model API bills tokens, not words. Tomás’s estimate of about $38 a day for summaries, in Calling a model API, rests on a guess: about 4,000 tokens per thread.

This lesson answers one question: what exactly is Tomás paying for, and how can he count it before he sends a ticket? The token counts here are illustrative; the code near the end gives real ones.

A token is a common piece of text

Before the model sees Meera’s message, a tokenizer cuts it into tokens: pieces of text from a fixed list called the vocabulary. Each token becomes a number, and the model reads and writes those numbers, as Next-token prediction shows.

Why pieces, and not words or letters? A vocabulary of whole words would need every name, typo and order number, and would still miss new ones. Single characters make every message several times longer, and the model’s work grows with length. Tokens sit between: frequent words get one token, and everything else is built from several.

Anthropic doesn’t publish the tokenizer for current Claude models, so the real boundaries may differ. Only the API gives you a real count.

The same meaning, very different counts

The same shop also gets tickets in Hindi. Here is Meera’s message, translated:

नमस्ते, मैंने 12 मार्च को एक प्रेशर कुकर ऑर्डर किया था (ऑर्डर 58214), लेकिन वह अभी तक नहीं पहुँचा है। ट्रैकिंग पेज पर लिखा है कि वह रास्ते में है, पर छह दिनों से कुछ नहीं बदला। क्या आप पता कर सकते हैं कि वह कहाँ है?

A common rule says “a token is about three-quarters of a word”. It breaks for other languages, code and numbers: the count depends on the tokenizer, the language and the type of content ^{2PricingAnthropic, Claude API documentation}.

This gap is not special to Claude. One study found the same text in different languages up to 15 times apart in tokens ^{3Language Model Tokenizers Introduce Unfairness Between LanguagesAleksandar Petrov, Emanuele La Malfa, Philip H.S. Torr, Adel Bibi, NeurIPS 2023, 2023}. The model matters too: Claude models from 4.7 on use a newer tokenizer that produces about 30% more tokens for the same text ^{2PricingAnthropic, Claude API documentation}.

Price, context and max_tokens all count tokens

Here is lesson 03’s cost formula applied to a Hindi ticket. Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens, as of October 2026 ^{2PricingAnthropic, Claude API documentation}.

If Deskora charges every shop the same, a shop with mostly Hindi tickets costs about twice as much to serve; Charging for AI features deals with that. The Hindi thread also fills nearly three times as much of the context window, covered in Context windows. One ratio can’t serve every ticket, so Tomás needs a real count.

Count tokens before you send

The endpoint, POST /v1/messages/count_tokens, takes the same model, system and messages as a normal call and returns one field, input_tokens ^{1Token countingAnthropic, Claude API documentation}. It counts input only: output, thinking included, is unknown until the model writes it.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

MODEL = "claude-haiku-5-5"
INPUT_PRICE = 0.10  # US dollars per million input tokens, as of October 2026
LONG_PROMPT = 100_000  # above this, Haiku 5.5 charges 5 times per token
SYSTEM = (
    "You summarize a support ticket for the agent who picks it up. "
    "Write exactly three lines: the problem, what has been tried, "
    "and what the customer is waiting for."
)


def count_summary_input(thread: str) -> int:
    result = client.messages.count_tokens(
        model=MODEL,
        system=SYSTEM,
        messages=[{"role": "user", "content": thread}],
    )
    return result.input_tokens


if __name__ == "__main__":
    with open("ticket.txt", encoding="utf-8") as f:
        thread = f.read()
    tokens = count_summary_input(thread)
    cost = tokens * INPUT_PRICE / 1_000_000
    print(f"{len(thread)} characters, {tokens} input tokens, ${cost:.6f} of input")
    if tokens > LONG_PROMPT:
        print("Over 100,000 tokens: this call would pay the long-prompt price.")

Running it costs nothing, because counting is free; you need an API key. Put Meera’s message in ticket.txt in English, then in Hindi, and compare.

Tomás can now count a ticket’s tokens. The model they go to sets the price of each one, which is the subject of Choosing a model.

Check your understanding

Summary