Why the Database Goes Down: The Anatomy of a Cache Stampede

Why the Database Goes Down: The Anatomy of a Cache Stampede

Adding Redis to the backend to offload the database is a classic solution. Everything runs fast — until the exact moment a popular key's TTL expires, and the system runs into a non‑obvious trap: the very tool meant to speed things up becomes the trigger for a crash (Cache Stampede). Let's break down this chain reaction.

The anatomy of the problem:

- TTL expiration. A key with a massive request flow — like a product catalog — suddenly disappears from the cache.
- Thundering herd on the database. Thousands of parallel backend threads simultaneously get an empty response and rush to the primary database for fresh data.
- Connection collapse. The database chokes on the unexpected queue, hits the connection pool limit, and users start seeing timeouts en masse.

How to protect the system at the code and architecture level:

- Distributed lock via SET NX. Instead of heavy Redlock, a simple atomic key in Redis is enough. It guarantees that only one request across the entire cluster will go to the database for fresh data.
- SingleFlight inside the pod. A pattern that merges identical parallel requests within a single application instance on the fly, while the first thread fetches the data.
- Stale‑While‑Revalidate. A pragmatic approach where the application instantly returns the stale version from the cache to the user, while triggering a heavy recalculation in the background — sparing the system from waiting.
- Jitter for TTL. Adding a random spread to the base key lifetime, which shifts their expiration times and effectively dampens mass load spikes.

The bottom line is simple: the cache only shields you from load when the application is protected from its own misses. If you let things run their course, a single expired line in memory will bring down a live database faster than any DDoS attack.