Module 01 · Lesson 3 of 14Free

Server concurrency: handling many requests at once

Level

Beginner

Topic

System design fundamentals

Reading time

15 min

At the start of a festival sale, Kartly receives 1,000 product-page requests in one second. Its app server has 8 CPU cores. If every request needs its own core, most customers must wait before anything happens.

But a request does not need a core for its entire lifetime.

How can those cores keep many customers’ requests moving, and what eventually limits them?

Waiting is not CPU work

Follow one product-page request from the request lifecycle:

These are Kartly’s teaching-scenario timings, not measured production results. The database and pricing calls happen sequentially.

The app uses its CPU for only 5 ms. During the remaining time, other systems are working or data is travelling.

Concurrency means several requests remain in progress. Parallelism means work executes at the same moment on different cores.

For our calculations, assume each core executes one CPU task at a time. Waiting requests do not count against that limit.

The server needs somewhere to keep unfinished requests. The important choice is what else each waiting request holds.

A worker can remain occupied

A thread executes application instructions. A thread pool keeps a fixed number of reusable threads.

In a blocking request handler, one thread handles a request from start to finish. With blocking I/O, the thread stops until the requested data arrives. It remains assigned to that request, but another runnable thread can use the CPU. Platform threads are comparatively scarce resources.^{2Virtual ThreadsOracle, Java Platform, Standard Edition 21: Core Libraries, 2023}

Imagine Kartly has a pool of 200 threads. A free thread takes the next request. When every thread is occupied, incoming work waits in a queue.

The request’s path is straightforward:

1. Take a request from the queue.
2. Check access.
3. Ask the database and wait.
4. Ask pricing and wait.
5. Build the response, then return the thread to the pool.

The pool avoids creating a new thread for every arrival. Its limit is equally clear: a thread waiting on pricing cannot take another request.

More threads can overlap more waiting. They cannot make the database answer sooner or create more CPU time.

Parking keeps the worker available

An event loop runs ready pieces of work on one thread. With non-blocking I/O, a request can pause without holding that thread.

For network operations, the runtime asks the operating system to monitor connections. When data becomes ready, the runtime runs the corresponding callback. It does not need one thread checking each connection repeatedly.^{1Don't Block the Event Loop (or the Worker Pool)Node.js, Node.js Learn, 2026}

Here is Kartly’s database wait under that model:

1. Request A starts its query.
2. The runtime keeps A’s saved execution state and connection information.
3. A pauses, leaving the loop available for request B.
4. Database data becomes available.
5. The loop resumes A when it can run its continuation.

A continuation is the work that remains after the pause.

The database still takes 40 ms in this scenario. Parking changes who can use the waiting time, not how long the query takes.

An event-loop callback runs synchronously until it returns or yields control. A long calculation on that loop delays other callbacks, including callbacks for completed network operations.^{1Don't Block the Event Loop (or the Worker Pool)Node.js, Node.js Learn, 2026}

That creates a choice: keep simple blocking handlers, or make waiting cheaper.

Choose the model for the workload

A separate process has its own memory. Multiple processes are another way to place work on several cores, with stronger memory separation and extra memory cost. They can contain thread pools or event loops; process separation and request scheduling are different decisions.^{2Virtual ThreadsOracle, Java Platform, Standard Edition 21: Core Libraries, 2023}

We can now estimate whether workers or cores limit Kartly first.

Count occupied workers and cores separately

Assume sustained arrivals rather than a single burst. Also assume timings stay unchanged under load.

The useful relationship is Little’s Law: average work in progress equals arrival rate multiplied by average time in the system. Use matching boundaries: handler time gives requests inside handlers, not requests still waiting outside them.

For this scenario, the pool is not the first predicted constraint. Keep it and measure actual behaviour before changing architecture.

These averages do not predict how long the last request in a sudden burst waits. The next lesson, Latency and throughput, separates response time from completion rate.

Async workers still need a limit

This small experiment isolates waiting. It models the database and pricing delays with timers, omitting CPU work and network failures.

The app owns two asynchronous worker functions. Each claims one request before pausing. Both run on one event loop; they are not two operating-system threads.

import asyncio

async def main() -> None:
    pending = iter(range(6))

    async def worker() -> None:
        for request_id in pending:
            print("start", request_id)
            await asyncio.sleep(0.040)
            await asyncio.sleep(0.050)
            print("done", request_id)

    await asyncio.gather(worker(), worker())

if __name__ == "__main__":
    asyncio.run(main())

async function main(): Promise<void> {
  const pending: Iterator<number> = [0, 1, 2, 3, 4, 5].values();
  const sleep = (ms: number): Promise<void> =>
    new Promise<void>((resolve) => setTimeout(resolve, ms));

  async function worker(): Promise<void> {
    for (;;) {
      const item = pending.next();
      if (item.done) return;
      const requestId: number = item.value;
      console.log("start", requestId);
      await sleep(40);
      await sleep(50);
      console.log("done", requestId);
    }
  }

  await Promise.all([worker(), worker()]);
}

void main().catch(console.error);

Predict the output before running it. Two requests start before either finishes. Completion ordering is not a guarantee.

There is no race between claiming an ID and entering the first wait: each worker runs that part without yielding. Sharing this iterator across actual threads would require separate analysis.

Change the worker count to one. Waiting no longer overlaps between requests. Changing it to six overlaps all six waits, but would also increase simultaneous dependency calls.

In real handlers, use asynchronous network clients instead of timers. await does not make a synchronous calculation or blocking library call non-blocking.^{1Don't Block the Event Loop (or the Worker Pool)Node.js, Node.js Learn, 2026}

The example bounds work for a finite batch. A live server also needs a limit on incoming work.

A slow dependency fills every model

Suppose total handler time rises to one second, mostly waiting. Assume the dependencies could otherwise sustain the incoming rate.

PostgreSQL limits concurrent connections. Raising that limit also increases some resource allocations; it does not create unlimited query capacity.^{319.3. Connections and AuthenticationPostgreSQL Global Development Group, PostgreSQL 17 Documentation, 2024}

Kartly should bound admitted work rather than move an unlimited waiting line into memory.

For this product-page path, there is no business-data write. The app reads product data, requests pricing, then returns the assembled response.

For this lesson’s design, an expired request stops waiting and releases its app-side slot. Remote work may continue unless cancellation reaches the dependency.

Track queue length, occupied handlers, dependency waiting time and CPU use. These distinguish “no worker available” from “no CPU available”.

When the dependency recovers, bounded pending work can drain. Capacity estimates alone cannot promise what survives a failure.

Where this shows up

• Design: a URL shortener on a single server: distinguish database waiting from CPU work.
• Design: a food delivery API: keep many tracking connections open without assigning each a platform thread.
• Design: a notification service: bound simultaneous calls to delivery providers.

After sizing these resources, Kartly must decide what customers can expect when one fails; Core system properties develops those guarantees.

Common mistakes

• Treating worker occupancy as CPU usage. A fully occupied blocking pool can spend most of its time waiting.
• Treating parked requests as free. They still retain state and may hold scarce dependency resources.
• Adding threads without identifying the limit. More overlap helps waiting-heavy work only while downstream capacity remains.
• Putting long calculations on the loop. Other ready requests cannot resume until that work yields or finishes.^{1Don't Block the Event Loop (or the Worker Pool)Node.js, Node.js Learn, 2026}

Check your understanding

Summary