Thundering Herd: From Blackouts to Ad Breaks
When the power comes back after a blackout, every fridge and air conditioner starts in the same second, and the grid can trip again. Software has the same failure: a cache key expires, a server restarts, the clock hits midnight, and every client acts at once. Nine places where the herd shows up, the four ways out, and how Pluto TV got ahead of its ad breaks.

After a long blackout, the power doesn't simply come back. When a utility re-energizes a feeder, every fridge, freezer and air conditioner on it has been off long enough to want to start, and they all start together. A paper from Pacific Northwest National Laboratory describes the resulting load as "significantly higher than pre-outage levels, even exceeding the normal distribution feeder peak demand", and the standard mitigation is to "restore the feeder in sections" (Schneider et al., IEEE Trans. Power Systems, 2016). Grid engineers call it cold load pickup. A GE protection guide puts the first-seconds inrush at roughly four times the normal peak-load current, and notes that reclosing after a long outage "throws all of these loads on at once without the usual diversity" (GE Vernova, GET-6450).
Without the usual diversity is the whole problem in four words. On a normal evening, fridges cycle at random moments and their load averages out. After a blackout, the outage synchronized them.
Software calls this a thundering herd: one trigger, many clients that react to it in the same instant, and one shared resource that was sized for their average, not for all of them at once. It is the mirror image of the previous article on debounce and throttle. There, one actor fires many times and the system wants one action. Here, many actors fire once each, at the same moment, and the system needs them not to.
Scroll sideways to see the full diagram →
Every fix changes when clients arrive or how many of them actually reach the shared resource. None of them changes what the clients want. They fall into four families:
- Spread: make the arrivals random or staggered, so they stop lining up.
- Let one do the work: when everyone wants the same thing, one fetches it and the rest share the result.
- Pace: make clients back off, or let them in at the rate the system can serve.
- Get ahead: do the expensive work before the moment everyone arrives.
Nine places where the herd shows up, grouped by the fix that tames it.
Spread it out
1. The power grid
The herd. Thermostatic appliances that normally cycle at random all start together when power returns, because the outage put them in the same state.
The fix. Restore the feeder in sections, so each section's inrush settles before the next one adds its own. The total demand is the same; only the timing changes. That is the picture in the visual at the top of this article.
2. Cron jobs at midnight
The herd. People pick round times. Google's SRE book describes it directly: so many teams schedule jobs at midnight that it causes "the thundering herd" and "substantial spikes in datacenter usage" (Google SRE book, Distributed Periodic Scheduling with Cron).
The fix. Google added a ? value to its crontab syntax: the job says it doesn't care about the exact minute, and the system picks one by hashing the job's configuration, which spreads jobs evenly. On a single Linux machine, systemd timers offer RandomizedDelaySec=, which adds a random delay so timers configured the same way don't all fire together (systemd.timer). Its default is 0, so you have to ask for it.
Where it doesn't help. Jobs tied to a real external time (a market open, a daily report due at 9:00) still line up. For those, the next families apply.
3. Push notifications at the top of the hour
The herd. Marketing teams schedule sends on the hour too. Firebase says it plainly: "FCM receives more than double traffic during the first 30 seconds to 2 minutes of each hour" (Firebase, scaling FCM).
The fix. Firebase recommends avoiding the two minutes around :00, :15, :30 and :45, and ramping a large send from zero to full rate over at least 60 seconds. The same advice protects your own backend: a notification that reaches ten million phones at once can bring a good share of them into your app in the same minute.
Let one do the work
4. The expired cache key
The herd. A popular key expires, and every request that needed it misses the cache in the same instant and goes to the database for the same value. This is the cache stampede.
The fix. Facebook's memcache paper describes leases: on a miss, the cache gives a token to one client, at most "once every 10 seconds per key", and tells the others to wait briefly and retry the cache. Only the token holder refills the key. Over a week of measurements on keys prone to stampedes, the peak database query rate dropped from 17,000 per second to 1,300 (Nishtala et al., Scaling Memcache at Facebook, NSDI 2013).
Scroll sideways to see the full diagram →
Or refresh before it expires. A second technique belongs to the "get ahead" family: recompute a hot key slightly before its expiry, at a random moment that gets more likely as the expiry approaches. Vattani et al. prove that an exponentially distributed early refresh is optimal and call it XFetch (PVLDB, 2015). Each server decides on its own, and the randomness keeps them from all deciding at once.
5. The CDN cache miss
The herd. A popular object expires at the edge, and a burst of viewers requests it before the first fetch from the origin comes back.
The fix. Request collapsing: the CDN combines "multiple requests for the same object into a single request to origin" and answers all of them with the one response, preventing "an immediate flood of requests to an origin server" (Fastly, request collapsing). Inside an application, Go's singleflight package does the same: it ensures "only one execution is in-flight for a given key at a time" and hands the result to every caller that asked (pkg.go.dev).
6. Sleeping server threads, where the name comes from
The herd. The term comes from operating systems. Many processes wait for a new connection on the same socket; one arrives, the kernel wakes all of them, one wins, and the rest go back to sleep having burned CPU for nothing.
The fix. Wake just one. Linux 4.5 added EPOLLEXCLUSIVE, which the man page describes as "useful for avoiding thundering herd problems in certain scenarios" (epoll_ctl(2)). The original patch came from an engineer at Akamai, a company that knows something about many clients arriving at once (LWN, 2015).
Pace it, or get ahead of it
7. Apps reconnecting after an outage
The herd. The server goes down, every client notices within seconds, and when it comes back they all reconnect in the same second, which can take it down again. In October 2017, a Slack deploy disconnected all of its customers and left them unable to reconnect; the fix was rate limits "to allow for gradual reconnection" (Slack status, 2017-10-31).
The fix. Exponential backoff with jitter. Backoff alone doesn't help much: clients that failed together back off together and come back together. Marc Brooker's simulation at AWS shows that "the no-jitter exponential backoff approach is the clear loser", and that adding randomness to the wait cuts the total work by more than half with 100 contending clients (AWS Architecture Blog, 2015). Google's SRE book makes it a rule: "Always use randomized exponential backoff when scheduling retries" (Addressing Cascading Failures). The same chapter shows why retries need a budget too: three layers that each retry three times turn one user request into 64 attempts.
The idea is older than the web. The 1976 Ethernet paper already resolved collisions by having each station wait a random interval whose mean doubles after every collision, and called it binary exponential backoff (Metcalfe & Boggs, CACM 1976).
Kubernetes applies the same pacing to itself: a container that keeps crashing is restarted with an "exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds" (Kubernetes, Pod lifecycle).
8. The ticket on-sale
The herd. Tickets go on sale at 10:00, and everyone arrives at 9:59. Unlike a cron job, the time can't move: the moment is the product.
The fix. A virtual waiting room paces the arrivals instead of spreading them. Queue-it describes one design: visitors who arrive early wait in a pre-queue with no position; when the sale opens, "all visitors in the pre-queue are randomized, just like a raffle", later arrivals join in order, and the sale site admits people at the rate it can serve (Queue-it). The randomization also removes the reason to hammer refresh at 9:59:59.
Scroll sideways to see the full diagram →
9. The live ad break
The herd. On a live or linear stream, every viewer of a channel reaches the ad break at the same second, and every player needs ads for it at once. AWS Elemental MediaTailor documents prefetching for exactly this: it helps "reduce peak load on ad decision servers (ADS) and decrease manifest delivery latency at the start of each ad break", and mitigates lower fill rates caused by "ad request and transcoding timeouts" (AWS MediaTailor, prefetching ads).
The fix. Get ahead of it: fetch and prepare the ads before the break, so the break reads from storage instead of calling the ad servers all at once. This is the one place in the list where I have built the fix in production, so it gets its own case below.
Choosing the way out
| If the moment… | And the clients… | Reach for |
|---|---|---|
| can move | don't care about the exact time | Spread: jitter, hashing, sections (cron, sends, grid) |
| can't move | all want the same thing | Let one do it: leases, request collapsing, singleflight |
| can't move | each want something different | Pace: backoff with jitter, a waiting room, rate limits |
| is predictable | need something expensive | Get ahead: prefetch, pre-scale, refresh early |
How the fixes fail
- Backoff without jitter. Clients that failed together wait the same time and fail together again. Randomness is the fix; the exponent alone is not.
- Retries at every layer. Each layer's retries multiply the ones below. Retry at one layer, with a budget.
- One lease holder that dies. If the client holding the lease crashes, everyone waits on a refill that never comes. Leases need a timeout, which is why Facebook's are handed out at most once every 10 seconds per key.
- Prefetching the wrong thing. Getting ahead only works if you can predict the moment and the demand. A wrong forecast wastes work at best and serves stale results at worst, so the prediction needs a feedback loop.
- Fixing the herd and forgetting the limit. All four families buy time and smoothness; none of them creates capacity. If the steady demand after the spike is above the limit, you still need to scale.
Real-world case: getting ahead of ad breaks at Pluto TV
Scroll sideways to see the full diagram →
Situation. Pluto TV is a free, ad-supported streaming service with linear channels: viewers on the same channel reach each ad break at the same moment. Its revenue services decide, fetch and prepare the ads for every break, calling third-party ad servers and an external transcoding platform. The supply of ads was limited, and different platforms competed for the same ads, so the moment of highest demand was also the moment when getting ads was hardest.
Task. Make sure the breaks expected to see the most demand were filled, without depending on a burst of calls to the ad servers at the exact second each break started.
Action.
- Forecast the demand. A statistical forecast picked the points expected to see the most demand, using the event being streamed, the day and outside forecasts as inputs.
- Prefetch ahead of the break. For those points, ads were fetched and processed ahead of time and kept in our own storage, so the break was served from storage instead of from a herd of live requests.
- Feed the forecast back. The statistics were fed with what actually happened, so the forecast adjusted over time instead of staying a fixed guess.
Result. For the breaks the forecast flagged, the expensive part (asking the ad servers and processing the ads) happened ahead of time, on a schedule we controlled, instead of in the same second as every viewer of the channel. It is the same idea AWS MediaTailor documents for prefetching, built in-house and driven by our own forecast. The lesson I keep from it: when you can predict the herd, don't make it faster at the door; do the work before the door opens.
Read the full case: forecast-driven ad prefetch for Pluto TV's ad breaks →
The takeaway
A thundering herd is not a load problem; it is a timing problem. The same houses, the same requests, the same jobs are fine when they arrive spread over a minute and fatal when they arrive in the same second. Before you add capacity, ask what synchronized your clients (an outage, an expiry, a round number on a clock, a break in a live stream) and pick the way out: spread them, let one of them do the work, pace them, or get there first.
References
- Schneider et al., Evaluating the Magnitude and Duration of Cold Load Pick-up on Residential Distribution Feeders (IEEE Transactions on Power Systems, 2016).
- GE Vernova, Distribution System Feeder Overcurrent Protection (GET-6450).
- Google SRE book, Distributed Periodic Scheduling with Cron and Addressing Cascading Failures.
- systemd, systemd.timer.
- Firebase, Scale your FCM app.
- Nishtala et al., Scaling Memcache at Facebook (NSDI 2013).
- Vattani, Chierichetti, Lowenstein, Optimal Probabilistic Cache Stampede Prevention (PVLDB, 2015).
- Fastly, Request collapsing; Go, singleflight.
- Linux, epoll_ctl(2); LWN, epoll: add EPOLLEXCLUSIVE flag (2015).
- Slack, incident of 2017-10-31.
- Marc Brooker, Exponential Backoff And Jitter (AWS Architecture Blog, 2015).
- Metcalfe & Boggs, Ethernet: Distributed Packet Switching for Local Computer Networks (CACM, 1976).
- Kubernetes, Pod lifecycle: container restarts.
- Queue-it, First-in-first-out randomization.
- AWS Elemental MediaTailor, Prefetching ads.