Hide outline
Feedback
Guide
Was this lesson helpful?
Dive Deeper
Generate a follow-up sub-lesson on any aspect of this topic
Guide complete
Caching Decisions That Actually Speed Systems Up
Generate a follow-up sub-lesson on any aspect of this topic
Caching Decisions That Actually Speed Systems Up
TLDR;
- A 1 percent miss rate at 50k RPS can still melt a database when misses synchronize.
- Picking the wrong cache layer can improve p50 while making p95 worse under partial outages.
- Versioned keys for price and inventory can protect correctness without paying global purge costs.
A product detail page that does a single database read per request looks fine right up until it does not. At 50k requests per second, a small bump in traffic or a momentary slowdown in the database turns a clean p95 latency target into a queue, then a timeout storm, then retries that multiply load. The important detail is that nothing is logically wrong with the code path. The system collapses because the database is a shared bottleneck and every request insists on touching it in the critical path. Let’s anchor the baseline read path before we change it.
In that diagram, clients hit a load balancer, which fans out to stateless app servers that all call the same primary database for the product record. The numbers matter because queuing is nonlinear. Once the database hits saturation, each extra millisecond of service time converts into more in flight queries, which increases lock contention and buffer pressure, which further increases service time. Caching only helps if it removes work from the database during the exact moments when the database would otherwise fall behind.
Caching helps in practice because it cuts the number of requests that pay database latency at all, not because it makes the fastest requests faster. Tail latency is where users feel the system. A page render that depends on a single read inherits the slowest component in the path. When the database has a slow spike from compaction, a noisy neighbor, or a replica failover, those slow reads become the p95 and p99. A cache hit avoids that spike entirely and keeps the app servers from piling onto a struggling database. We can treat each request as either a hit that costs cache latency or a miss that costs cache latency plus a database trip.
At 50k RPS, even a small miss rate is a large absolute number of database reads. If the miss rate is , then database QPS from this endpoint is . So is still 1,000 QPS, and those misses often cluster after deploys, restarts, or eviction waves. That clustering is the hidden cost. A cache strategy that looks fine under steady state can still fail if many app servers miss on the same key at once and synchronize their retries against the database.
Design rule
A cache is successful when it reduces peak database work, not when it improves average latency.
Where to place caches is a question of which network hop and which failure mode the cache should shield. A browser cache is closest to the user and can eliminate requests entirely, but it cannot reliably coordinate freshness across users. A CDN cache can absorb global read traffic for static or mostly static content, but it struggles with per user personalization and rapidly changing fields like inventory. An app side cache such as Redis sits on the request path next to the app servers and is good at hot objects and low latency reads, but it becomes another distributed system that can fail. A database cache such as the buffer pool is always there, but it only helps after the query reaches the database, which does not protect connection limits or query CPU.
The building blocks map cleanly onto a product page if we split the page into fields with different change rates. Product description and images can live comfortably in a CDN with long Cache-Control because staleness is rarely harmful. Price and inventory usually need tighter control because they change under business operations and correctness matters. The key is that each layer should have an explicit job. If the CDN is expected to protect the database from read bursts, then those bursts must be satisfiable from CDN keys, not from personalized HTML that forces an origin fetch.
The critical difference between cache-aside and read-through is who is responsible for loading data on a miss. In cache-aside, the app checks the cache first and on a miss it queries the database and then writes the value into the cache. In read-through, the app reads from the cache and the cache itself fetches from the database on a miss, acting like a proxy with storage. Cache-aside gives application code full control over keying and serialization, but it also makes every app instance a potential miss amplifier during outages. Read-through centralizes miss behavior, which can make stampede protection easier, but it couples cache availability to database access patterns.
In the side by side flows, watch what happens during a cache outage or a partial cache partition. With cache-aside, app servers can bypass the cache and hammer the database unless the app has explicit circuit breaking. With read-through, a cache outage often turns into a hard read failure unless there is a fallback path. Operationally, cache-aside tends to fail open and overload the database, while read-through tends to fail closed and shed traffic earlier. The right choice depends on whether protecting the database or maximizing availability is the priority for this endpoint.
Write strategies decide whether the cache is a mirror of the database or a separate source of truth. In write-through, the app writes to the cache and the database in one logical operation, and reads can trust the cache to be fresh if the write succeeded. In write-back, the app writes to the cache and the cache flushes to the database asynchronously, which reduces write latency but risks data loss if the cache is lost. In write-around, the app writes only to the database and lets the cache fill on reads, which avoids polluting the cache with write heavy keys but guarantees that a just written key will be a miss until it is read again.
The decision matrix should make one thing uncomfortable. Write-back is rarely acceptable for price or inventory unless the cache has durability assumptions close to the database. That typically pushes teams away from pure write-back in Redis unless they are willing to treat Redis as a primary store, which changes backup, replication, and failover requirements. For product pages, write-around is common for fields where correctness matters and the read path can tolerate a brief miss penalty. Write-through is common when the system already has a canonical write service that can enforce ordering and idempotency.
Practical caveat
If cache and database writes are not atomic, write-through still permits brief inconsistency under retries.
Eviction and freshness policies define what the cache optimizes for when memory is constrained and keys have uneven popularity. Least Recently Used (LRU) tries to keep what was accessed most recently, which is good for recency driven workloads but can be fooled by scans. Least Frequently Used (LFU) tries to keep what is accessed most often, which is good for stable hot keys but reacts slowly to shifts. Time To Live (TTL) forces expiry after a fixed time, which bounds staleness but can cause synchronized expiry waves. Each policy behaves differently when one product becomes extremely hot and the rest are cold.
In the table, focus on what gets evicted under skew. With LRU, a burst of unique product views can evict a genuinely hot product, creating self inflicted misses. With LFU, a hot product stays, but a formerly hot product can linger and waste memory. With TTL only, memory use is less predictable, and a fleet wide synchronized TTL boundary can create a miss spike even though nothing meaningful changed. If the product detail traffic is heavy tailed, which is typical in ecommerce, LFU or LRU plus TTL often beats TTL alone because it keeps the head of the distribution resident while still bounding staleness.
Cache invalidation becomes manageable when each field on the product page has an explicit freshness contract. TTL only is simplest, but it can allow stale price or stale inventory that violates user expectations. Explicit purge is precise, but it requires reliable fanout to every cache layer and is vulnerable to missed messages. Versioned keys avoid global deletes by making the key change when the underlying data version changes, so new reads naturally bypass stale entries. Versioning works best when there is a single write path that can bump a version number in the database and publish it with the response.
A workable contract for this page usually splits along change rate and harm of staleness. Description can take a long TTL and rely on background propagation. Price often needs a short TTL plus versioning so promotions take effect quickly without needing to purge every edge cache. Inventory is usually best treated as separate from the page cache entirely, or cached with very short TTL and a stale tolerant UI, because strict correctness often implies a read from a strongly consistent store at checkout time anyway. The widget should show how keys like product:{id}:desc:v{n} and product:{id}:price:v{n} let old entries die naturally while new reads shift to the new version.
At 50k RPS, the main cache failures are coordinated. A cache stampede happens when many requests miss on the same key and all run the expensive fallback at once. A thundering herd is the broader version where many keys expire together and the system floods the backend with recomputation. Staleness becomes visible when caches are layered, because a purge or version bump might reach Redis quickly but take longer at the CDN. Mitigations exist, but each adds its own operational surface area.
After the reveal, the pattern should be clear. Request coalescing makes the app or cache ensure only one in flight load per key, so a hot product miss becomes one database read instead of thousands. Jittered TTLs randomize expirations so the backend sees a spread of misses instead of a cliff. Stale while revalidate serves a slightly stale value quickly while one background refresh updates the cache, which protects p95 at the cost of bounded staleness. The thread through all three is that the system must avoid synchronized misses, because synchronized misses are what turn a cache into a load amplifier.
A defensible caching design for the product detail page starts with one decision. Which component must be protected at peak load, the database, the cache, or the user experience. If the p95 target is strict and the database is the single point of collapse, a common posture is CDN caching for description and media, Redis cache-aside for product aggregates, write-around for correctness sensitive fields, and versioned keys for price with a short TTL. Inventory stays out of the main page cache or gets a very short TTL with explicit UI messaging that checkout is authoritative. Put those choices against the scenario constraints before committing.
What forces a redesign is usually a tighter correctness requirement, not more traffic. If price must be globally consistent within seconds, versioned keys plus TTL might still be too weak because edge caches and client caches create multiple stale reservoirs. If inventory must never be oversold based on a cached value, then inventory cannot be treated as cacheable truth and the system shifts toward quorum reads, reservations, or a strongly consistent store for that field. Caching speeds systems up when it is scoped to data whose staleness is explicitly acceptable and whose miss behavior is explicitly controlled.