Module 01 · Lesson 4 of 14Free

Latency and throughput: measuring speed and capacity

Level

Beginner

Topic

System design fundamentals

Reading time

14 min

During Kartly’s festival sale, traffic jumps to 1,000 requests a second. The dashboard shows a product-page average of 138 ms. Yet Asha’s checkout spinner stays visible for 2 seconds.

Kartly still has one app server with 8 CPU cores and one PostgreSQL database. How can its speed measurements look healthy when a customer is stuck?

One request and many completions

Latency is the elapsed time for one request, measured between a stated start and finish. Throughput is the number of requests completed per unit of time. A service can have high throughput and still have slow individual requests.^{1Monitoring Distributed SystemsRob Ewaschuk, Site Reliability Engineering, O’Reilly Media, 2016}

For this lesson, server latency runs from request arrival to response completion. It includes waiting inside the server. The customer’s experience also includes network travel and browser work, outside this boundary.

As Server concurrency showed, overlapping requests let a server work during other requests’ waits.

Suppose an illustrative sale window contains these completed product-page requests:

The average looks close to the fast group because that group is much larger. We need a measurement that exposes the slow group before choosing what to improve.

Averages hide the waiting customers

These are invented samples, not Kartly production measurements:

A percentile locates a position in sorted measurements. For this example, use the nearest-rank rule: multiply the sample size by the percentile fraction, round up, then read that position.

At least 99% of these observations finished within the p99 value. It is not the maximum: the last ten requests took longer.

The slow end of a distribution is called tail latency. Averages alone can hide it; recording latency distributions makes it visible.^{1Monitoring Distributed SystemsRob Ewaschuk, Site Reliability Engineering, O’Reilly Media, 2016}

Keep product-page and checkout measurements separate. A large stream of quick product reads must not hide a smaller stream of slow purchases.

Percentiles reveal who is waiting. They do not yet reveal where the time went.

The dependency path sets the budget

A latency budget divides an allowed elapsed time among the work needed for a response.

Use Kartly’s earlier product-page example:

• Initial CPU work takes 2 ms.
• The database call takes 40 ms.
• The pricing call takes 50 ms.
• Building the response takes 3 ms.

These remain illustrative durations. For one request, sequential operations add. Independent operations started together overlap; waiting for all of them takes roughly the longest duration, plus coordination overhead.

The longest required chain is the path that determines completion. Making the 40 ms call faster would not improve this parallel example until it changes that path.

For an illustrative checkout budget, reserve 300 ms inside the server:

These are allowances, not measured percentiles or promises. Required sequential phases total 260 ms, leaving the margin shown above.

Do not give every downstream call the full request budget. That would let several sequential calls each consume time the customer does not have.

For a request that must finish within its budget, record elapsed time before starting each phase. Compare the remaining time with the work still required. A payment timeout must not be presented as proof that no payment happened; safe retry behavior belongs in Idempotency.

Even independent calls introduce another problem: waiting for all of them exposes you to every call’s slow cases.

More branches expose more slow cases

Fan-out means one incoming request starts several downstream operations. If every result is required, one slow operation can delay the entire response.

Large systems can turn occasional component delays into frequent user-visible delays. That is the problem described by The Tail at Scale.^{3The Tail at ScaleJeffrey Dean and Luiz André Barroso, Communications of the ACM, volume 56, pages 74–80, 2013}

Consider a hypothetical Kartly product-page redesign. It waits for ten independent lookups. Assume each lookup has a 1% chance of exceeding 200 ms.

A shared database or network can make these delays correlated. Then the independence calculation no longer applies.

For Kartly, an optional recommendation should not hold up the product’s price and purchase button. Keep required price and stock checks on the response path. Treat optional content separately when the product permits it.

Do not add downstream p99 values and call the result the page’s p99. Different calls can have their slowest observations on different requests. Measure complete requests to obtain the end-to-end percentile.

Reducing the number of required waits helps, but every remaining operation still competes for finite resources.

Nearly full leaves little recovery room

Utilization is the fraction of a resource’s available time spent busy. High utilization leaves fewer idle intervals to absorb uneven arrivals or unusually long work.

Latency can deteriorate before a resource reaches full utilization. Watch the constrained resource, not only the machine’s overall CPU percentage.^{1Monitoring Distributed SystemsRob Ewaschuk, Site Reliability Engineering, O’Reilly Media, 2016}

Here is a deliberately simplified model, not a capacity estimate for Kartly. Imagine one busy resource that can complete 100 identical operations per second. An extra burst adds 20 operations.

Assume arrivals and completions behave as smooth rates during recovery.

These are backlog recovery times, not individual request latencies. They show why small increases near full utilization have large consequences.

Requests arriving during recovery wait in a queue behind unfinished work. Their elapsed time includes that wait. If arrivals keep exceeding completions, the backlog cannot clear under these assumptions.

This is why “keep everything 100% busy” is a poor goal for an interactive checkout. Real arrivals are uneven. Without recovery room, one delay affects later customers.

Throughput may initially rise as Kartly accepts more concurrent work. Near a bottleneck, completions stop rising proportionally, but waiting continues to increase. Request cost also matters: a checkout and a product read need not consume equal resources.^{2Handling OverloadAlejandro Forero Cuervo, Site Reliability Engineering, O’Reilly Media, 2016}

The reverse tradeoff appears with batching. Processing several operations together can share setup work and improve throughput. Waiting to collect that batch adds latency before any operation starts. Choose that delay for background work only when its completion requirements permit it.

The useful operating point therefore depends on acceptable customer waiting, not the largest completion count alone.

Measure outcomes before declaring improvement

For Kartly’s next sale rehearsal, use the same request mix across runs. Increase arrivals gradually and compare:

Keep successful and failed response latencies distinct. Fast errors can otherwise make a broken service appear faster.^{1Monitoring Distributed SystemsRob Ewaschuk, Site Reliability Engineering, O’Reilly Media, 2016}

If checkout slows, compare its complete duration with time spent waiting and time inside each dependency. Optimize the part that explains the extra elapsed time, rather than assuming the database is responsible.

After a mitigation, require both recovered latency and successful completions. A shorter queue caused by rejecting every customer is not a successful checkout recovery.

Where this shows up

• Design: a food delivery API: freshness targets and repeated status requests create competing demands.
• Design: a fast product catalog page: required dependencies determine when a page can respond.

Kartly now needs to decide which checkout actions must work, and how fast they must finish: the questions in Functional and non-functional requirements.

Common mistakes

• Calling arrivals throughput. Receiving work does not mean completing it.
• Treating p99 as the maximum. Some observations can take longer.
• Averaging percentiles across windows. Combine the underlying observations or distribution counts, then calculate the percentile.
• Parallelizing dependent work. Starting early cannot produce an input that does not exist yet.
• Reading latency without outcomes. A fast failure is still a failed purchase.

Check your understanding

Summary