Module 01 · Lesson 7 of 13Free

Sampling and temperature: controlling randomness

Level

Beginner

Topic

AI engineering fundamentals

Reading time

12 min

Priya gets a different summary the second time

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team, message by message.

Priya is a support agent at a kitchenware shop that uses Deskora. She opens a 20-message ticket from Oskar, whose pressure cooker arrived without its lid. At the top sits a three-line summary. Tomás, the Deskora engineer who builds the AI features, is testing it, and his test page has a Regenerate button. Priya clicks it. The thread hasn’t changed, but the second summary uses different words.

Tomás asks a colleague. “Set temperature to 0,” the colleague says. “Then you get the same answer every time.” Tomás adds it to his request to Claude Haiku 5.5.

This lesson answers one question: if Tomás can’t turn the randomness down, how does he give Priya summaries she can rely on?

The thread size and costs below are estimates for this lesson’s scenario.

The model draws each word from a spread

A model writes one token at a time; a token is a small piece of text, often part of a word. At each step, the model gives every possible next token a probability. Then one token is drawn from that spread, by chance, in proportion to those probabilities. A likely token wins most of the time, not every time. A summary of 80 tokens is 80 draws, so two runs almost always differ somewhere. Under the hood: how decoding picks each token shows the mechanics.

Here is one step from an example ticket in an earlier lesson:

Most splits change the wording, not the facts. Now and then a pick changes a fact, and Priya can’t tell from one summary which kind she got. So different wording is expected. What remains is which settings shape the draw.

Three settings used to shape the draw

For years, model APIs let you shape the draw with three settings. Other providers’ APIs and the libraries that run open-weight models still have them.

Temperature 0 was the classic advice for steady output. It made the top token win at every step. That gives the same words on every run, but not the right words: if the top token is wrong, it is wrong every time.

On Claude, none of the three is available any more.

On the Claude API, the request fails

On API version 2026-10-21, a request that sends temperature, top_p or top_k returns a 400 error, even at the parameter’s default value ^{1Migrating to API version 2026-10-21Anthropic, Claude API documentation, 2026}. The API version is the date in the anthropic-version header of every request. Without the three settings, every request samples with the provider’s defaults, including a temperature of 1.0 ^{1Migrating to API version 2026-10-21Anthropic, Claude API documentation, 2026}.

Tomás’s Python code fails even earlier. The anthropic Python SDK 2.3.0 sends version 2026-10-21 and no longer has the three parameters ^{1Migrating to API version 2026-10-21Anthropic, Claude API documentation, 2026}. Passing one raises a Python TypeError before any request leaves his server.

Staying on the older version, 2023-06-01, doesn’t help him. Claude 4.7 and later models, Haiku 5.5 included, accept only the default values there too ^{1Migrating to API version 2026-10-21Anthropic, Claude API documentation, 2026}. The migration guide also says no other setting replaces a lowered temperature ^{1Migrating to API version 2026-10-21Anthropic, Claude API documentation, 2026}.

Try it yourself

If you have a Claude API key, send the request Tomás sent and watch it fail. Readers without a key can skip this.

curl https://api.anthropic.com/v1/messages \
  -s -w "\nstatus %{http_code}\n" \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -d '{"model": "claude-haiku-5-5", "max_tokens": 200, "temperature": 0, "messages": [{"role": "user", "content": "Summarize in one line: my parcel is late."}]}'

An example of what comes back, with the status line that -w adds:

{
  "type": "error",
  "error": {"type": "invalid_request_error", "message": "..."},
  "request_id": "req_..."
}
status 400

Look for invalid_request_error and status 400. Now delete "temperature": 0, and send it again. The same call returns a message and status 200. The API key stays in your shell’s environment, never in the command. With max_tokens at 200, the working call costs at most about 200 × $0.50 per million = $0.0001 on Claude Haiku 5.5, as of October 2026, plus a few input tokens.

So Tomás can’t make the draw less random. He can make the right summary far more likely, and stop drawing again.

What Tomás controls instead

These are the levers he has:

1. A clear prompt with an example. The more exactly the prompt says what goes on each line, the more probability lands on the right tokens at each step. One worked example shows the format; Few-shot prompting goes further.
2. Structured outputs. They constrain the reply to JSON that matches a schema you send ^{5Structured outputsAnthropic, Claude API documentation}, so every summary has the same three fields. Structured outputs teaches them.
3. Effort. It sets how much the model thinks before it answers ^{6EffortAnthropic, Claude API documentation}. It controls how much work, not how much randomness. Write it out, so a change of default can’t change the feature. Reasoning effort covers the tradeoff.
4. Store the result. Make the summary once when the ticket changes, save it, and show the saved copy.
5. Check over several runs. Run the same tickets several times and compare the facts, which Evals turns into a habit.

Here is the request without any sampling settings. It adds an example, a schema and an explicit effort to lesson 03’s call:

import json

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

SYSTEM = """You summarize support tickets for the agent who picks them up.
Fill three fields: the problem, what was tried, what the customer is waiting for.
Use only facts from the thread. One short sentence per field.

Example:
problem: Kettle arrived cracked.
tried: Agent sent a replacement on 3 May.
waiting_for: Tracking number for the replacement."""

SCHEMA: dict[str, object] = {
    "type": "object",
    "properties": {
        "problem": {"type": "string"},
        "tried": {"type": "string"},
        "waiting_for": {"type": "string"},
    },
    "required": ["problem", "tried", "waiting_for"],
    "additionalProperties": False,
}


def summarize(thread: str) -> dict[str, str]:
    response = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        system=SYSTEM,
        messages=[{"role": "user", "content": thread}],
        output_config={
            "effort": "medium",  # Haiku 5.5's default, written out on purpose
            "format": {"type": "json_schema", "schema": SCHEMA},
        },
    )
    text = next(block.text for block in response.content if block.type == "text")
    return json.loads(text)

Deskora saves the result with the ticket and shows the saved copy. A Regenerate click would cost another call, about $0.0006 for a typical ticket on Claude Haiku 5.5, as of October 2026. That is lesson 03’s estimate: 4,150 input tokens × $0.10 per million + (300 assumed thinking + 80 summary tokens) × $0.50 per million ≈ $0.0006.

Design: a ticket summary feature builds the background job and the stored row.

Variety helps drafts, not labels

Different runs are a problem for some tasks and the point of others.

For alternatives, don’t rely on chance. Two calls with the same prompt may give two drafts that say the same thing in other words. Ask in one call for two drafts with a stated difference: one short and direct, one that explains the shop’s policy. A schema with two fields keeps them apart. The difference then comes from the prompt, and Priya always gets a real choice.

For labels, runs that disagree are a signal. If three runs on one ticket say delivery, refund, delivery, the ticket or the prompt is unclear. Tighten the category definitions with examples before picking a winner.

Open-weight models still have temperature

This section is about open-weight models, not Claude.

An open-weight model is one whose weights you download and run yourself. Sampling then runs in your own code, so the three settings are yours. In Hugging Face Transformers, they are arguments to generate() ^{3GenerationHugging Face, Transformers documentation}. It picks the top token every time unless sampling is turned on ^{2Generation strategiesHugging Face, Transformers documentation}. Many models ship their own settings: Qwen2.5-0.5B-Instruct turns sampling on, with temperature 0.7, top_p 0.8 and top_k 20 ^{4Qwen2.5-0.5B-Instruct: generation_config.jsonQwen team, Hugging Face Hub}. Values you pass to generate() override those ^{3GenerationHugging Face, Transformers documentation}.

This script runs a small open-weight model on your own computer, twice at a low temperature and twice at a high one:

# pip install transformers==5.21.0 torch==2.15.0 (an open-weight model, not Claude)
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID: str = "Qwen/Qwen2.5-0.5B-Instruct"  # downloads about 1 GB the first time

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)

messages: list[dict[str, str]] = [
    {"role": "user", "content": "Write a one-line apology for a late parcel."}
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
)
prompt_length: int = inputs["input_ids"].shape[1]

for temperature in (0.2, 1.2):
    for run in (1, 2):
        output = model.generate(
            **inputs,
            max_new_tokens=40,
            do_sample=True,
            temperature=temperature,
            top_p=0.9,
            top_k=50,
        )
        reply = tokenizer.decode(output[0][prompt_length:], skip_special_tokens=True)
        print(f"temperature {temperature}, run {run}: {reply}")

Look at the pairs. At 0.2, the two runs are usually close or identical. At 1.2, they usually differ more. It costs nothing per run beyond your own computer’s time. The same rule holds here: a low temperature makes runs alike, not right, so store results here too.

Where this shows up

• Design: an invoice data extractor: the same fields on every run come from a schema and validation, not from a setting.
• Design: a writing assistant inside an editor: offering several suggestions needs runs that differ, asked for on purpose.

Tomás’s request now carries a system prompt, an example and a 4,000-token thread. Context windows shows how much one call can hold, and what filling it costs.

Common mistakes

• Copying temperature=0 from an older tutorial into Claude code. On API version 2026-10-21 it fails, as a 400 or as a TypeError in the SDK. Remove it.
• Treating new wording as a bug. Compare the facts and the fields, not the words.
• Calling the model on every view. Each view is a new draw and a new bill. Store the result.
• Expecting two calls to give two useful alternatives. Ask for the difference in the prompt.

Check your understanding

Summary