Hide outline
Feedback
Guide
Was this lesson helpful?
Dive Deeper
Generate a follow-up sub-lesson on any aspect of this topic
Guide complete
Scaling Up vs. Scaling Out Decisions
Generate a follow-up sub-lesson on any aspect of this topic
Scaling Up vs. Scaling Out Decisions
TLDR;
- One bigger server can buy time, but it concentrates downtime and caps you at a single box’s limits.
- Adding more app servers only works cleanly after you move state out of the process.
- Database scaling usually forces your first real architecture fork, replicas for reads or shards for writes.
A checkout API is a small system with a hard promise. It takes POST /checkout requests, validates a cart, charges a payment method, and creates an order that customer support can find later. The users are web and mobile clients, and the business depends on it during peak minutes when a slow or failed checkout is lost revenue. On one machine, the API, in memory session state, and a single database all share the same CPU, RAM, disk, and network link, so the box hits a limit long before the code feels complex. For a concrete target, assume the service needs to sustain 1,000 requests per second at acceptable latency, and any reboot is a full outage because there is nowhere else to send traffic. Let’s anchor the baseline in a single-box picture.
In that baseline, the key constraint is that every resource is shared and finite. CPU saturation shows up as longer queueing time per request, RAM pressure turns into garbage collection churn or swapping, and the single network interface caps inbound and outbound throughput for clients and the database. The hardest operational limit is downtime because patching the OS, replacing a disk, or restarting the process takes the whole checkout path offline. That is the single-machine limit that drives the next decision.
Scaling up means running the same architecture on a bigger machine, often by moving to a larger instance type with more vCPUs, memory, and sometimes faster local storage. The immediate win is simplicity because there is still one runtime to deploy, one local cache, one database connection pool, and no need to coordinate multiple application copies. For the checkout API, that can be the fastest way to get from 1,000 requests per second to something higher when the bottleneck is straightforward, like CPU for JSON parsing or encryption. The cost is that vertical capacity has hard ceilings and every improvement increases the blast radius of a failure because the whole workload now depends on an even more expensive single box. The tradeoff is easiest to feel by sliding between instance size and failure impact.
When the slider moves toward a bigger instance, throughput headroom usually increases but the shape of failure stays the same. A kernel panic, a bad deploy, or a noisy neighbor event still drops 100 percent of traffic, and the bigger the box, the more revenue rides on that one failure domain. Maintenance windows also stay binary because even a quick reboot is downtime unless you add redundancy, which is already the first step toward scaling out.
Design rule
If a restart is unacceptable, scaling up alone cannot be the final architecture.
Scaling out means running multiple copies of the application and spreading requests across them, but it only behaves predictably when each request can land on any instance. Statelessness means the app process does not hold user specific or request critical state that must be present on the next request, beyond what the client sends or what a shared store can retrieve. Checkout flows often violate this in subtle ways, like storing a cart snapshot in memory, keeping a session cookie that maps to a local in memory session object, or writing temporary files to local disk during an upload. Once you add more instances, those choices turn into intermittent bugs because the next request might hit a different machine that does not have the state. The pieces that usually need a redesign show up clearly when you enumerate them.
The building blocks fall into two buckets. State you can move becomes a shared dependency, like sessions in Redis, uploads in object storage, and scheduled jobs in a queue where a single worker claims work. State you cannot safely share becomes a reason to pin traffic, like long lived WebSockets that effectively create a per client affinity, or in process caches that are correct only on one node. The practical goal is not purity. It is that losing one app instance does not lose a checkout in progress or corrupt an order.
A load balancer sits in front of multiple app instances and decides which instance receives each request. A load balancer is a routing component that forwards traffic using a policy such as round robin or least connections, and it can stop sending traffic to instances that fail health checks. For checkout, this is how you get redundancy and rolling deploys. If one instance dies, the load balancer routes around it and the fleet continues to serve requests, assuming the app is stateless and the shared stores are healthy. The routing behavior and the failure behavior belong together, especially when timeouts and retries enter the picture. Seeing the fleet wiring makes the failure modes easier to reason about.
The important moving parts are clients, the load balancer, and at least three app instances so one can disappear without dropping capacity to zero. Health checks usually probe a lightweight endpoint like GET /healthz, and an instance that fails checks is removed from rotation. The tricky edge is retry amplification. If clients retry quickly on 5xx and the load balancer also retries upstream, a slow instance can trigger extra load that makes the whole fleet slower, which triggers more retries. Timeouts need to be consistent across client, load balancer, and app so the system fails fast instead of piling work onto already overloaded instances.
Failure smell
When p95 latency rises and retries rise with it, the load balancer may be multiplying load.
App servers scale out almost linearly for CPU bound work, but a single database usually does not, because writes serialize on shared structures like the write ahead log, indexes, and locks. For the checkout API, each successful checkout must durably write an order, inventory changes, and a payment record, and those writes define the real capacity limit. The first database split is usually between adding read replicas and doing sharding. A read replica is a copy that applies changes from the primary so you can serve read queries without loading the primary, while sharding partitions the data so different primaries handle different subsets of writes. The best choice depends on whether your pain is read load, write load, or latency. The fork is easier to compare side by side.
The comparison usually comes out like this. A single primary is simplest and gives strong consistency for reads after writes, but it runs out of write capacity first. A primary with read replicas improves read throughput and can reduce read latency when replicas are near the app, but replicas lag, so reading from them can show stale order status. Sharding can increase write scalability by splitting the write workload across multiple primaries, but it increases operational complexity because queries, transactions, and rebalancing now cross partitions. The key is that replicas mostly protect the primary from reads, while shards are a commitment to distributing writes.
Sharding works when each request maps cleanly to one shard most of the time. A shard key is the field you partition on, such as user_id or order_id, and it determines both load distribution and what queries become expensive. If you shard by user_id, each user’s orders land together, which helps user centric reads, but a celebrity flash sale can create a hot shard if many checkouts come from a small key range. If you shard by order_id, writes may distribute better but user history becomes a cross shard query. Rebalancing is the cost you pay later because shard sizes drift, and moving data between shards consumes bandwidth and complicates correctness during the move. Putting small numbers on the scenario helps expose whether you are actually facing a write scalability problem yet.
Given inputs like user count, orders per day, and shard count, the most useful outputs are per shard writes per second and whether a single hot key can dominate capacity. Even if average writes per shard look fine, the risk is amplification when many requests target the same shard key, like a single merchant account or a single promotional SKU. The mental model to keep is that sharding fixes average throughput only when the distribution stays flat enough under peak conditions.
A practical path for the checkout scenario is to scale up once, then scale out the stateless app tier, and postpone database sharding until a measured write bottleneck forces it. That often means moving sessions to Redis, moving uploads to object storage, running background work from a queue, and putting three or more app instances behind a load balancer. For the database, start with one primary and add read replicas only when read traffic is the cause, keeping checkout writes and read after write flows on the primary. The deciding constraint is the write path. If the primary cannot sustain peak writes with acceptable p95 latency, you either shard writes or introduce a coordination layer for stronger guarantees across partitions, like consensus for a single order ID generator or inventory reservation. The interesting breaking points tend to be non obvious until they bite.
In practice, redesign triggers show up as specific signals rather than a vague sense of load. Sticky sessions appear because the app was not truly stateless, and they reduce the load balancer’s ability to smooth spikes. Replica lag appears when reads are served from replicas and order status looks wrong immediately after checkout. Noisy neighbor shards appear when a few shard keys dominate traffic and one shard becomes the new single point of failure. Scaling up is still the right move when the bottleneck is local and predictable, but scaling out is the right move when downtime and burst tolerance are the constraint, and sharding is the right move when sustained write throughput is the constraint that will not move with bigger hardware.