Module 01 · Lesson 9 of 13Free

Streaming responses: showing output as it arrives

Level

Beginner

Topic

AI engineering fundamentals

Reading time

13 min

Priya watches a spinner for six seconds

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team, message by message.

Priya is a support agent at a kitchenware shop that uses Deskora. A customer, Amara, writes that her new pan set arrived without one of its lids. Priya presses a new button, Draft a reply. A spinner turns. About six seconds later, a whole draft appears at once.

Tomás, the Deskora engineer who builds the AI features, made that button. His ticket summary runs in the background, so nobody waits for it. The draft is different: Priya asks for it and watches the screen. Six seconds of nothing looks broken, so some agents press the button again, which pays for a second call.

This lesson answers one question: how can Priya see the reply while the model is still writing it, and what changes in Tomás’s code when she does?

The draft-reply feature, its token counts and its speeds are estimates for this lesson’s scenario.

Streaming moves the first word, not the last

Without streaming, the call works as in Calling a model API: the API waits until the model has finished, then sends the whole response as one JSON body. With streaming, the API sends the response in small pieces while the model produces them. The model writes in tokens, small pieces of text often shorter than a word.

Here is where Priya’s 5.5 seconds go. The draft runs on Claude Haiku 5.5, which thinks before it answers by default; thinking tokens are billed as output but not shown ^{2ThinkingAnthropic, Claude API documentation}.

The wait for the first output token is called time to first token. Priya feels the time to first visible text; hidden thinking sits between the two. On an easy input the model may skip thinking, and text starts almost at once ^{2ThinkingAnthropic, Claude API documentation}.

Streaming hides the time spent writing, not the time spent thinking. A long ticket shows it:

The long row assumes 1 second before the first token, since more input takes longer to read. To shorten the pause itself, ask the model to think less; Reasoning effort covers that control.

Streaming doesn’t change the price. On Claude Haiku 5.5, as of October 2026, a typical draft costs 4,150 input tokens × $0.10 per million + 500 output tokens (thinking included) × $0.50 per million = about $0.0007 ^{3PricingAnthropic, Claude API documentation}. A second press doubles that.

So what does a streamed response look like on the wire?

The response arrives as a sequence of events

Tomás adds "stream": true to the request body. The API answers with status 200 and a content-type of text/event-stream, then keeps the HTTP response open and writes each piece as soon as it is ready. This format is called server-sent events ^{5HTML Living Standard: Server-sent eventsWHATWG, HTML Living Standard}. Each event is an event: line naming its type, a data: line holding JSON, and a blank line ^{1Streaming messagesAnthropic, Claude API documentation}.

One example stream for Amara’s draft, shortened; the text differs on every run:

event: message_start
data: {"type": "message_start", "message": {"id": "msg_01...", "type": "message", "role": "assistant", "content": [], "model": "claude-haiku-5-5", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 4163, "output_tokens": 1}}}

event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "thinking", "thinking": "", "signature": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "signature_delta", "signature": "EqQB..."}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 0}

event: content_block_start
data: {"type": "content_block_start", "index": 1, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "text_delta", "text": "Hi Amara, I'm sorry"}}

... more text_delta events ...

event: content_block_stop
data: {"type": "content_block_stop", "index": 1}

event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}, "usage": {"output_tokens": 512}}

event: message_stop
data: {"type": "message_stop"}

The content blocks of a normal response arrive one at a time. Each opens, receives deltas (small pieces of the block) and closes; index is its position in the final content list ^{1Streaming messagesAnthropic, Claude API documentation}.

With the thinking text hidden, the thinking block carries one empty thinking_delta and a signature_delta, the encrypted thinking ^{2ThinkingAnthropic, Claude API documentation}. That is the pause on Priya’s screen. A stream may also include ping events, and new event types may be added later, so code skips any event it doesn’t know ^{1Streaming messagesAnthropic, Claude API documentation}.

These events reach Tomás’s server. Priya’s browser still sees nothing.

Deskora’s server relays the text to Priya

The browser can’t call the model API, the provider’s HTTP service that runs the model, because the call carries Deskora’s API key. So there are two streams: Claude to Deskora’s server, and Deskora’s server to Priya’s browser.

The browser needs only the text and how the draft ended, so the server sends its own events instead of Claude’s: status, text, then one of done, cut_off or failed. The provider’s format stays on the server, which also logs the cost. The code uses the SDK’s stream helper and Python’s built-in HTTP server ^{4anthropic-sdk-pythonAnthropic, GitHub}:

import json
from collections.abc import Iterator
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
SYSTEM = ("Draft a short, polite reply to the customer's last message. "
          "A support agent will edit it before sending.")


def sse(event: str, data: str) -> bytes:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n".encode("utf-8")


def draft_reply(thread: str) -> Iterator[bytes]:
    try:
        with client.messages.stream(
            model="claude-haiku-5-5",
            max_tokens=4000,  # room for thinking plus a few paragraphs
            system=SYSTEM,
            messages=[{"role": "user", "content": thread}],
        ) as stream:
            for event in stream:
                if (event.type == "content_block_start"
                        and event.content_block.type == "thinking"):
                    yield sse("status", "writing")
                elif (event.type == "content_block_delta"
                        and event.delta.type == "text_delta"):
                    yield sse("text", event.delta.text)
            final = stream.get_final_message()
    except Exception as err:  # an error event, or the connection dropped
        yield sse("failed", type(err).__name__)
        return
    print(f"in={final.usage.input_tokens} out={final.usage.output_tokens}")
    ending = "done" if final.stop_reason == "end_turn" else "cut_off"
    yield sse(ending, final.stop_reason or "")


class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        # A real handler checks the session and loads the thread by ticket id.
        length = int(self.headers["content-length"])
        thread = self.rfile.read(length).decode("utf-8")
        self.send_response(200)
        self.send_header("content-type", "text/event-stream")
        self.send_header("cache-control", "no-cache")
        self.end_headers()
        for frame in draft_reply(thread):
            self.wfile.write(frame)
            self.wfile.flush()  # send each event now, not when the buffer fills


if __name__ == "__main__":
    ThreadingHTTPServer(("127.0.0.1", 8000), Handler).serve_forever()

client.messages.stream(...) sends the request with streaming on, and looping over stream gives each event as it arrives. Afterwards, stream.get_final_message() returns the whole message built from the events, the same object a normal call returns ^{1Streaming messagesAnthropic, Claude API documentation}. If you need only the text, stream.text_stream yields just the text pieces.

In the browser, Deskora reads the response body piece by piece with fetch and adds each text event to the reply box. A WebSocket would also work, but traffic here goes one way; WebSockets, SSE and long polling compared weighs the options.

Priya now sees words at 3.5 s. But some things still arrive last.

The ending comes last, and it can go wrong

In a stream, stop_reason is null in message_start and arrives only in message_delta, after every word is on screen ^{1Streaming messagesAnthropic, Claude API documentation}. If it says max_tokens, the draft Priya has been reading was cut off. Deskora keeps the text and marks it: “This draft stopped early.”

usage also arrives at the end, and its counts are cumulative ^{1Streaming messagesAnthropic, Claude API documentation}. Take the last value; don’t add them up.

The harder case is an error. Status 200 and the headers went out first, so a later failure can’t change them. Instead, the API sends an error event, such as an overloaded_error during heavy load, which a normal call would return as HTTP 529 ^{1Streaming messagesAnthropic, Claude API documentation}.

When to retry, and how often, is the subject of Model API errors, rate limits and retries.

Try it yourself

Watch a raw stream with curl; -N makes it print events as they arrive. One run costs at most about $0.0005 on Claude Haiku 5.5: at most 1,000 output tokens × $0.50 per million, plus a few dozen input tokens. Without an API key, skip this.

curl -N https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -d '{"model": "claude-haiku-5-5", "max_tokens": 1000, "stream": true, "messages": [{"role": "user", "content": "Write a two-sentence reply to a customer whose pan set arrived without a lid."}]}'

Example output, shortened; your events and text will differ:

event: message_start
data: {"type":"message_start","message":{...,"stop_reason":null,...}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"thinking",...}}
...
event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"text_delta","text":"Hi there,"}}
...
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn",...},"usage":{"output_tokens":131}}

What to look for: a pause before the first text_delta, which is hidden thinking (none if the model skipped it), and stop_reason set only in message_delta.

Where this shows up

• Design: a writing assistant inside an editor streams text into a document as the writer watches.
• Design: scaling a chat assistant to a million users keeps many streams open at once.
• Design: an AI customer support platform mixes streamed drafts with background jobs.

Half a draft and a Try again button raise the next question: when the model API fails, which calls should Deskora retry, and how soon?

Common mistakes

• Reading stop_reason from message_start. It is null there; read it from message_delta.
• Adding up usage across message_delta events. The counts are cumulative.
• Treating the end of the stream as success. A dropped connection also ends it. Only done means complete.
• A buffering proxy in the path. A reverse proxy that buffers responses holds every event until the end, and the spinner is back.

Check your understanding

Summary