Module 01 · Lesson 9 of 14Free

Vertical and horizontal scaling: when to use each

Level

Beginner

Topic

System design fundamentals

Reading time

14 min

At the festival opening, Kartly receives 1,000 requests a second. Its application server has 8 CPU cores. Someone proposes replacing it with a larger machine; someone else proposes adding another server.

Which change would help, and what evidence would justify it?

Bigger and more solve different problems

Vertical scaling changes one machine’s resources: more CPU cores, more memory, or a different machine size. Horizontal scaling changes the number of application instances running the workload. An instance is one running copy of the application.^{1AutoscalingMicrosoft, Azure Architecture Center}

Picture a checkout area. A larger counter gives one worker more space. Opening another counter lets another worker serve a different customer. Neither change fixes a payment terminal that every counter must share.

Kartly currently has one application server and one PostgreSQL database. The multi-server designs below are possible next steps, not changes already made to the running example.

As Server concurrency explains, requests can wait without keeping a CPU core busy. A high number of requests in progress therefore does not prove that Kartly needs more cores.

The first decision is not “bigger or more”. It is “what resource is preventing useful work?”

Find the limit before adding capacity

A bottleneck is the resource currently limiting useful throughput. Google’s overload guidance warns that requests per second alone can hide large differences in request cost.^{2Handling OverloadAlejandro Forero Cuervo, Site Reliability Engineering, 2016}

Use the measurements from Latency and throughput, then connect the delay to a resource.

For this lesson’s next scenario, assume a new page renderer doubles CPU work to 10 ms per request. Database and pricing waits remain unchanged.

Assume load measurements now show CPU saturation under the sale workload. The team has ruled out an easy code improvement. It wants a planning limit of 70% CPU usage.

That percentage is a scenario choice, not a universal safe threshold. It leaves room for uneven work and unexpected demand.

These estimates assume comparable cores and unchanged request cost. They are a reason to test a candidate configuration, not a performance guarantee.

A larger server preserves simplicity

Vertical scaling keeps the request path familiar: the application receives a request, reads PostgreSQL, calls pricing and returns the page.

For this scenario, assume the replacement server has comparable per-core performance. Also assume Kartly’s application can use its additional cores.

More cores do not automatically shorten one request. Under our assumption, each request still needs 10 ms of CPU work. The extra cores let independent requests run at the same time.

The upgrade solves the estimated CPU shortage. It does not remove the application server as a failure point.

More servers divide independent work

Now add a product requirement: browsing must continue if one application server fails.

Horizontal scaling needs a load balancer, a component that selects a server for incoming traffic. Kartly’s proposed balancer sends each new request to a healthy application instance. Instances must be ready before receiving traffic. Departing instances should stop receiving new work and finish existing requests where possible.^{1AutoscalingMicrosoft, Azure Architecture Center}

For this proposed version, assume all durable customer data lives in PostgreSQL. No request needs data held only in another application server’s memory.

The read path becomes:

1. The customer requests a product page.
2. The balancer selects a healthy application server.
3. That server reads PostgreSQL and asks pricing for the price.
4. It builds the response and sends it back through the balancer.

A cart write follows the same route, but the selected application server saves the change in PostgreSQL. Adding application instances does not copy or divide the database.

The balancer can stop routing to a failed server, but it cannot recover a response already lost during that failure. Several servers therefore require both spare capacity and failure handling.

Spare capacity makes failover useful

Two servers sound redundant. But redundancy is not enough if the survivor cannot handle the traffic.

Under our CPU model, two 8-core servers provide a combined planning capacity of 1,120 requests a second. Losing either leaves only 560.

The customer still sees slow pages, even though a server remains alive.

Health checks take time to detect failure. During that interval, some customers can still receive errors. This design targets enough surviving capacity; it does not promise uninterrupted requests.

Shared limits move the bottleneck

A tempting belief is “twice as many servers means twice the capacity”. That only holds if other limits do not intervene. Microsoft’s guidance explicitly calls for checking databases and downstream APIs before adding compute.^{1AutoscalingMicrosoft, Azure Architecture Center}

Consider a separate scenario assumption: PostgreSQL can complete 900 representative queries a second within Kartly’s response-time target. Each page request requires one query.

Even if the application tier supports more traffic, the database limits these pages to at most 900 a second under those conditions.

Adding application servers may move waiting into a database queue. Customers still wait for the same scarce resource. Overload handling must therefore remain part of the design, even with traffic distribution.^{2Handling OverloadAlejandro Forero Cuervo, Site Reliability Engineering, 2016}

Connections create another shared limit. Suppose every application instance allows 40 open database connections. Three instances could request 120 connections.

PostgreSQL’s typical default connection limit is 100. Some slots are reserved. Raising the limit also increases certain resource allocations; it does not establish that queries will execute faster.^{319.3. Connections and AuthenticationPostgreSQL Global Development Group, PostgreSQL 18 Documentation}

Set a fleet-wide connection budget, then divide it across instances. Investigate expensive queries before introducing more database infrastructure. Scaling a relational database covers the later choices.

The useful question becomes “what limits completed requests now?”, not “how many machines can we add?”

Add capacity before customers need it

For Kartly’s fixed-time sale, prepare capacity before opening. Scheduled scaling can account for startup time; reactive scaling must first detect demand and then start usable instances.^{1AutoscalingMicrosoft, Azure Architecture Center}

Use this release checklist:

• Confirm the expected request mix, not only total traffic.
• Measure response times and errors at the target load.
• Remove one application instance and repeat the measurement.
• Check database connections and dependency waiting.
• Confirm new instances are ready before receiving requests.

Treat the estimates as hypotheses. Keep the change only if completed requests improve without moving unacceptable delays elsewhere.

Where this shows up

• Design: a URL shortener for millions of users: distinguish redirect-server capacity from storage capacity.
• Design: a fast product catalog page: reduce repeated database work instead of only adding compute.

Before Kartly adds its second application server, it must answer another question: what happens when that server cannot see a customer’s session?

Common mistakes

• Scaling from CPU usage alone. Check whether customers are waiting on another resource before buying compute.
• Calling two servers failure-ready. Calculate whether the survivors can handle the required traffic.
• Assuming scaling out fixes every layer. Application instances still share database and pricing limits.
• Expecting an immediate traffic response. Allow time for detection, startup and readiness.

Check your understanding

Summary