Module 07 · Lesson 7 of 12Free

Cache stampede: many requests hitting the database at once

Level

Advanced

Topic

Caching and CDNs

Reading time

19 min

At the festival-sale peak, 2,000 requests a second ask Kartly for the same product description. Its cached entry expires. Before the replacement arrives, hundreds of requests ask the database to rebuild the same answer.

How can Kartly refill one missing entry without turning every waiting customer into another database query?

These are illustrative numbers for one product within Kartly’s many-service stage. This lesson caches description data, not checkout prices or stock.

One missing entry multiplies work

Start with the cache-aside read path. The catalog service checks the shared cache. On a miss, it reads the database and stores the result.

That sequence has no owner. Two requests can both observe “missing” before either stores an answer. Each request follows the rule correctly, yet together they repeat work.

A cache stampede is a burst of overlapping attempts to rebuild missing or expired cached data. It can affect one popular entry or many entries expiring together.

Deleting an entry after a seller edit creates the same gap. So does eviction under memory pressure. Expiry is one trigger, not the underlying cause.

The gap lasts until a replacement becomes visible. Its duration determines how many requests can enter the rebuild path.

Count arrivals during the gap

Assume rebuilding this description takes about 200 ms without contention. It includes several reads and assembly work, not one simple indexed lookup.

The last calculation assumes every attempt reaches the database and keeps the same execution time. A connection limit may make requests wait earlier. Contention may make execution slower.

Either way, customers pay for repeated work through a frozen product page. Unrelated catalog requests can also lose access to database connections.

A longer rebuild creates a larger missing-entry window. That admits more duplicate work, which can lengthen the rebuild again. A small expiry event can therefore continue beyond the first burst.

Share the work before adding locks

The simplest correction is to keep a local map of work already running. Its key is the cache key. Its value is a shared asynchronous result.

The first request creates that result and becomes the leader. Later requests become followers: they wait for the same result instead of starting another query.

This is request coalescing, also called singleflight. Go’s singleflight package provides this duplicate-call suppression for overlapping calls using the same key.^{1singleflightThe Go Authors, Go Packages}

The catalog worker owns this map

A worker follows this sequence:

1. Read the shared cache.
2. On a miss, look up the key in its local work map.
3. Join an existing result, or install a new shared result.
4. Inside the leader’s operation, check the cache again.
5. If still missing, read the database and store the replacement.
6. Complete the shared result, then remove it from the map.

The second cache check closes a useful race. Another worker may have filled the entry since the first check.

The key must include every input affecting the answer. A description in one language must not share work with another language. Include the same tenant, locale and representation fields used by the cache key.

Implement the sharing mechanism

These implementations share overlapping calls within one event loop. The supplied load operation owns the second cache check, database read and cache write.

import asyncio
from collections.abc import Awaitable, Callable
from typing import Generic, TypeVar

T = TypeVar("T")

class SingleFlight(Generic[T]):
    def __init__(self) -> None:
        self._tasks: dict[str, asyncio.Task[T]] = {}

    async def run(
        self, key: str, load: Callable[[], Awaitable[T]]
    ) -> T:
        task = self._tasks.get(key)
        if task is None:
            async def invoke() -> T:
                return await load()

            task = asyncio.create_task(invoke())
            self._tasks[key] = task

            def finish(done: asyncio.Task[T]) -> None:
                if self._tasks.get(key) is done:
                    del self._tasks[key]
                if not done.cancelled():
                    done.exception()

            task.add_done_callback(finish)

        return await asyncio.shield(task)

class SingleFlight<T> {
  private readonly tasks = new Map<string, Promise<T>>();

  run(key: string, load: () => Promise<T>): Promise<T> {
    const existing = this.tasks.get(key);
    if (existing !== undefined) return existing;

    const pending: Promise<T> = Promise.resolve()
      .then(load)
      .finally(() => {
        if (this.tasks.get(key) === pending) {
          this.tasks.delete(key);
        }
      });

    this.tasks.set(key, pending);
    return pending;
  }
}

Removal happens after success or failure. Otherwise a failed operation could remain in the map and fail every future request.

Python’s shield keeps one cancelled waiter from cancelling the shared task. The TypeScript example passes no caller cancellation signal into the shared operation. Give that operation its own deadline rather than inheriting the first customer’s deadline.

Neither implementation stores completed values. That is the cache’s responsibility. Neither limits waiting callers or distinct keys; those limits belong around this small mechanism.

There is also a boundary: a local map coordinates only callers that can access that map.

Ownership must match the deployment

Suppose Kartly has 20 catalog worker processes. Each owns a separate work map, even when several processes run on one machine.

For one overlapping burst, local coalescing permits up to one rebuild per process. That means up to 20 rebuilds here, not one across the fleet.

This may be acceptable. It fits the scenario’s spare database capacity. Expanding to 80 processes changes that calculation.

The shared-cache topology from Distributed caching does not change this boundary. Sharing values and sharing refresh ownership are separate choices.

A lease coordinates local leaders

If needed, only each process’s local leader competes for the shared lease. Followers remain attached to their local result.

For a single Redis instance, an acquisition can use SET refresh:42 unique-token NX PX 1000. It creates the record only when absent and expires it after one second.^{2Distributed Locks with RedisRedis, Redis documentation}

The token identifies this attempt. Release must atomically compare the token and delete only a matching record. Unconditional deletion could remove a newer owner’s lease.^{2Distributed Locks with RedisRedis, Redis documentation}

Kartly’s proposed lease flow is:

1. Acquire the lease, then check the data cache again.
2. If still missing, start a database read with a shorter operation deadline.
3. Publish the replacement only through an atomic operation that checks current ownership.
4. Release matching ownership after publication.
5. If acquisition fails, recheck for a completed value after a short randomized delay.

For this design, the ownership check and publication happen on the same cache instance. A separate “check token, then write” sequence leaves a race between those operations.

A lease is not a promise of exactly one rebuild. Its owner can pause past expiry. Redis failover can also lose ownership state before replication completes.^{2Distributed Locks with RedisRedis, Redis documentation}

Accept occasional duplicate reads here, but reject publication by an expired owner. Concurrent seller updates require separate protection, covered in Cache race conditions.

Put the pieces on the request path

With leases enabled, losing local leaders recheck the cache, not the database. Bound their polling and stop at the caller’s deadline.

Without such a bound, coordination protects database execution but leaves customers waiting indefinitely. The next decision is what they should see during that wait.

Keep old data for readers

Hard expiry removes the only answer just before Kartly starts producing its replacement.

For description text, Kartly can instead keep a short period where an old answer remains usable. Stale-while-revalidate means serving that answer while refreshing it in the background. RFC 5861 describes this behavior for HTTP caches.^{3RFC 5861: HTTP Cache-Control Extensions for Stale ContentMark Nottingham, RFC Editor, 2010}

Kartly can apply the same idea inside its catalog service. The service must implement it; an HTTP header does not configure an application cache.

Store two deadlines, not one

For this scenario, keep descriptions fresh for one minute. Permit stale serving for another half-minute.

The entry stores the payload, a freshness deadline and a stale-serving deadline. Physical expiry must retain the payload until at least the stale-serving deadline.

At 65 seconds after publication, a customer receives the old description immediately. One background operation refreshes it. Another customer arriving during that operation receives the old description too, without starting another operation.

After the stale limit, the old payload is no longer an acceptable response under this policy. Wait briefly for a refresh or return an explicit unavailable response.

A failed refresh must not move either deadline forward. Otherwise repeated failures can make an old description appear usable forever.

Do not attach this background work to a response that has already completed. A worker-level supervisor must retain the task, observe errors and enforce its deadline.

This removes much of the waiting during normal expiry. A cold start still has no old value to return.

Failure needs a separate budget

Coalescing limits duplicate work for one key. It does not limit work across different missing keys.

If thousands of descriptions become missing after a cache restart, each can legitimately elect a different leader. The database still needs an independent rebuild budget.

Choose that budget from measured spare database capacity. Allocate it across workers, and account for process growth. A per-process limit silently multiplies when the fleet expands.

A cooldown records a short “do not retry before” deadline. It prevents each newly arriving request from becoming the next failed leader. Randomizing that delay reduces synchronized retries.

Do not cache a temporary database failure as “product not found”. Those answers mean different things to a customer.

A caller timeout must not become permission to query the database independently. That would rebuild the original stampede precisely when the dependency is already slow.

Broader cache-outage handling belongs in Cache failures and database fallback.

Measure rebuilds, not just hits

Record refresh starts, joined callers, duration, publication failures and stale responses. Track how often the stale limit is reached.

Compare refresh starts with logical misses during expiry. Many joined callers and few starts suggest coalescing works. Growing joins with no successful publication suggest a stuck or failing refresh.

Avoid a metric label for every product ID. Use bounded dimensions and sampled traces to investigate individual keys.

To exercise the design, remove one test entry during steady traffic. Check that rebuild starts match the intended coordination scope. Then fail the refresh and confirm that cooldown prevents repeated immediate attempts.

Prewarming can reduce cold starts for known sale products. It does not replace coordination: the warmup itself needs a budget, and entries can disappear later.

Where this shows up

• Design: a fast product catalog page combines description freshness with a responsive page.
• Design: search autocomplete needs to control repeated rebuilds for popular cached suggestions.

Even after duplicate rebuilds disappear, every cache hit for this product may still reach one overloaded cache node—the next problem in Hot keys.

Common mistakes

• Using one global refresh lock. A slow description then blocks unrelated products. Coordinate per key and separately bound total database work.
• Letting every stale reader schedule a job. Moving duplicate work into the background does not remove it. Coalesce before scheduling.
• Setting the lease from average rebuild time. Slow operations can outlive ownership. Use deadlines and reject publication after ownership expires.
• Extending stale lifetime after failure. That hides an outage by weakening the freshness promise. Keep the original stale limit.
• Claiming one rebuild across the fleet. Local maps have a process boundary. Shared leases also have expiry and failure limits.

Check your understanding

Summary