Mario Ruiz Díaz
← All articles

Thundering Herd: From Blackouts to Ad Breaks

When the power comes back after a blackout, every fridge and air conditioner starts in the same second, and the grid can trip again. Software has the same failure: a cache key expires, a server restarts, the clock hits midnight, and every client acts at once. Nine places where the herd shows up, the four ways out, and how Pluto TV got ahead of its ad breaks.

Mario Ruiz Díaz11 min read

Thundering Herd: From Blackouts to Ad Breaks

After a long blackout, the power doesn't simply come back. When a utility re-energizes a feeder, every fridge, freezer and air conditioner on it has been off long enough to want to start, and they all start together. A paper from Pacific Northwest National Laboratory describes the resulting load as "significantly higher than pre-outage levels, even exceeding the normal distribution feeder peak demand", and the standard mitigation is to "restore the feeder in sections" (Schneider et al., IEEE Trans. Power Systems, 2016). Grid engineers call it cold load pickup. A GE protection guide puts the first-seconds inrush at roughly four times the normal peak-load current, and notes that reclosing after a long outage "throws all of these loads on at once without the usual diversity" (GE Vernova, GET-6450).

Without the usual diversity is the whole problem in four words. On a normal evening, fridges cycle at random moments and their load averages out. After a blackout, the outage synchronized them.

Software calls this a thundering herd: one trigger, many clients that react to it in the same instant, and one shared resource that was sized for their average, not for all of them at once. It is the mirror image of the previous article on debounce and throttle. There, one actor fires many times and the system wants one action. Here, many actors fire once each, at the same moment, and the system needs them not to.

Every thundering herd has the same shape: one trigger, many clients that react to it in the same instant, and one shared resource with a limit. The four ways out all change the timing or the number of arrivals, never the clients' intent.FOUR WAYS OUTOne triggerpower back, key expires, 00:00Many clientsall react in the same secondOne shared resourcewith a limitSpreadjitter, sectionsLet one do itlease, collapsePacebackoff, waiting roomGet aheadprefetch, refresh earlyall at oncechange the timing

Scroll sideways to see the full diagram →

Every thundering herd has the same shape: one trigger, many clients that react to it in the same instant, and one shared resource with a limit. The four ways out all change the timing or the number of arrivals, never the clients' intent.

Every fix changes when clients arrive or how many of them actually reach the shared resource. None of them changes what the clients want. They fall into four families:

  • Spread: make the arrivals random or staggered, so they stop lining up.
  • Let one do the work: when everyone wants the same thing, one fetches it and the rest share the result.
  • Pace: make clients back off, or let them in at the rate the system can serve.
  • Get ahead: do the expensive work before the moment everyone arrives.

Nine places where the herd shows up, grouped by the fix that tames it.

Spread it out

1. The power grid

The herd. Thermostatic appliances that normally cycle at random all start together when power returns, because the outage put them in the same state.

The fix. Restore the feeder in sections, so each section's inrush settles before the next one adds its own. The total demand is the same; only the timing changes. That is the picture in the visual at the top of this article.

2. Cron jobs at midnight

The herd. People pick round times. Google's SRE book describes it directly: so many teams schedule jobs at midnight that it causes "the thundering herd" and "substantial spikes in datacenter usage" (Google SRE book, Distributed Periodic Scheduling with Cron).

The fix. Google added a ? value to its crontab syntax: the job says it doesn't care about the exact minute, and the system picks one by hashing the job's configuration, which spreads jobs evenly. On a single Linux machine, systemd timers offer RandomizedDelaySec=, which adds a random delay so timers configured the same way don't all fire together (systemd.timer). Its default is 0, so you have to ask for it.

Where it doesn't help. Jobs tied to a real external time (a market open, a daily report due at 9:00) still line up. For those, the next families apply.

3. Push notifications at the top of the hour

The herd. Marketing teams schedule sends on the hour too. Firebase says it plainly: "FCM receives more than double traffic during the first 30 seconds to 2 minutes of each hour" (Firebase, scaling FCM).

The fix. Firebase recommends avoiding the two minutes around :00, :15, :30 and :45, and ramping a large send from zero to full rate over at least 60 seconds. The same advice protects your own backend: a notification that reaches ten million phones at once can bring a good share of them into your app in the same minute.

Let one do the work

4. The expired cache key

The herd. A popular key expires, and every request that needed it misses the cache in the same instant and goes to the database for the same value. This is the cache stampede.

The fix. Facebook's memcache paper describes leases: on a miss, the cache gives a token to one client, at most "once every 10 seconds per key", and tells the others to wait briefly and retry the cache. Only the token holder refills the key. Over a week of measurements on keys prone to stampedes, the peak database query rate dropped from 17,000 per second to 1,300 (Nishtala et al., Scaling Memcache at Facebook, NSDI 2013).

A hot cache key expires and every request misses at once. With leases, the cache hands a token to one client, which refills the key from the database; the others wait briefly and read the refilled value. Facebook reports this cut the peak database query rate on such keys from 17,000 to 1,300 per second.1,000 requestssame key, same secondCachekey just expiredDatabase1 query, not 1,000Lease to one clientthe rest wait brieflymissone tokenrefillrefilled value

Scroll sideways to see the full diagram →

A hot cache key expires and every request misses at once. With leases, the cache hands a token to one client, which refills the key from the database; the others wait briefly and read the refilled value. Facebook reports this cut the peak database query rate on such keys from 17,000 to 1,300 per second.

Or refresh before it expires. A second technique belongs to the "get ahead" family: recompute a hot key slightly before its expiry, at a random moment that gets more likely as the expiry approaches. Vattani et al. prove that an exponentially distributed early refresh is optimal and call it XFetch (PVLDB, 2015). Each server decides on its own, and the randomness keeps them from all deciding at once.

5. The CDN cache miss

The herd. A popular object expires at the edge, and a burst of viewers requests it before the first fetch from the origin comes back.

The fix. Request collapsing: the CDN combines "multiple requests for the same object into a single request to origin" and answers all of them with the one response, preventing "an immediate flood of requests to an origin server" (Fastly, request collapsing). Inside an application, Go's singleflight package does the same: it ensures "only one execution is in-flight for a given key at a time" and hands the result to every caller that asked (pkg.go.dev).

6. Sleeping server threads, where the name comes from

The herd. The term comes from operating systems. Many processes wait for a new connection on the same socket; one arrives, the kernel wakes all of them, one wins, and the rest go back to sleep having burned CPU for nothing.

The fix. Wake just one. Linux 4.5 added EPOLLEXCLUSIVE, which the man page describes as "useful for avoiding thundering herd problems in certain scenarios" (epoll_ctl(2)). The original patch came from an engineer at Akamai, a company that knows something about many clients arriving at once (LWN, 2015).

Pace it, or get ahead of it

7. Apps reconnecting after an outage

The herd. The server goes down, every client notices within seconds, and when it comes back they all reconnect in the same second, which can take it down again. In October 2017, a Slack deploy disconnected all of its customers and left them unable to reconnect; the fix was rate limits "to allow for gradual reconnection" (Slack status, 2017-10-31).

The fix. Exponential backoff with jitter. Backoff alone doesn't help much: clients that failed together back off together and come back together. Marc Brooker's simulation at AWS shows that "the no-jitter exponential backoff approach is the clear loser", and that adding randomness to the wait cuts the total work by more than half with 100 contending clients (AWS Architecture Blog, 2015). Google's SRE book makes it a rule: "Always use randomized exponential backoff when scheduling retries" (Addressing Cascading Failures). The same chapter shows why retries need a budget too: three layers that each retry three times turn one user request into 64 attempts.

The idea is older than the web. The 1976 Ethernet paper already resolved collisions by having each station wait a random interval whose mean doubles after every collision, and called it binary exponential backoff (Metcalfe & Boggs, CACM 1976).

Kubernetes applies the same pacing to itself: a container that keeps crashing is restarted with an "exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds" (Kubernetes, Pod lifecycle).

8. The ticket on-sale

The herd. Tickets go on sale at 10:00, and everyone arrives at 9:59. Unlike a cron job, the time can't move: the moment is the product.

The fix. A virtual waiting room paces the arrivals instead of spreading them. Queue-it describes one design: visitors who arrive early wait in a pre-queue with no position; when the sale opens, "all visitors in the pre-queue are randomized, just like a raffle", later arrivals join in order, and the sale site admits people at the rate it can serve (Queue-it). The randomization also removes the reason to hammer refresh at 9:59:59.

A virtual waiting room in front of a ticket on-sale. Early arrivals wait in a pre-queue; when the sale opens they are randomized, later arrivals queue in order, and the sale site receives visitors at the rate it can serve.Fansarrive before 10:00Pre-queueno position yetRandomized at 10:00then first in, first outSale siteadmits at a set rateLate arrivalsjoin the backN per minutequeue in order

Scroll sideways to see the full diagram →

A virtual waiting room in front of a ticket on-sale. Early arrivals wait in a pre-queue; when the sale opens they are randomized, later arrivals queue in order, and the sale site receives visitors at the rate it can serve.

9. The live ad break

The herd. On a live or linear stream, every viewer of a channel reaches the ad break at the same second, and every player needs ads for it at once. AWS Elemental MediaTailor documents prefetching for exactly this: it helps "reduce peak load on ad decision servers (ADS) and decrease manifest delivery latency at the start of each ad break", and mitigates lower fill rates caused by "ad request and transcoding timeouts" (AWS MediaTailor, prefetching ads).

The fix. Get ahead of it: fetch and prepare the ads before the break, so the break reads from storage instead of calling the ad servers all at once. This is the one place in the list where I have built the fix in production, so it gets its own case below.

Choosing the way out

If the moment…And the clients…Reach for
can movedon't care about the exact timeSpread: jitter, hashing, sections (cron, sends, grid)
can't moveall want the same thingLet one do it: leases, request collapsing, singleflight
can't moveeach want something differentPace: backoff with jitter, a waiting room, rate limits
is predictableneed something expensiveGet ahead: prefetch, pre-scale, refresh early

How the fixes fail

  • Backoff without jitter. Clients that failed together wait the same time and fail together again. Randomness is the fix; the exponent alone is not.
  • Retries at every layer. Each layer's retries multiply the ones below. Retry at one layer, with a budget.
  • One lease holder that dies. If the client holding the lease crashes, everyone waits on a refill that never comes. Leases need a timeout, which is why Facebook's are handed out at most once every 10 seconds per key.
  • Prefetching the wrong thing. Getting ahead only works if you can predict the moment and the demand. A wrong forecast wastes work at best and serves stale results at worst, so the prediction needs a feedback loop.
  • Fixing the herd and forgetting the limit. All four families buy time and smoothness; none of them creates capacity. If the steady demand after the spike is above the limit, you still need to scale.

Real-world case: getting ahead of ad breaks at Pluto TV

The prefetch loop at Pluto TV. A forecast fed by the event calendar, the day and outside forecasts picks the ad breaks expected to see the most demand; the ads for them are fetched and processed ahead of time and kept in storage, so the break is served from storage instead of a burst of calls when it starts. What actually happened feeds the forecast back.AHEAD OF THE BREAKWHEN THE BREAK STARTSDemand signalsevent, day, outside forecastsForecastbreaks with most demandPrefetchask ad servers earlyPrepared adsprocessed and storedViewers at the breaksame second, same channelAd breakserved from storageActual demandfed back to the forecastready adswhat happenedadjusts

Scroll sideways to see the full diagram →

The prefetch loop at Pluto TV. A forecast fed by the event calendar, the day and outside forecasts picks the ad breaks expected to see the most demand; the ads for them are fetched and processed ahead of time and kept in storage, so the break is served from storage instead of a burst of calls when it starts. What actually happened feeds the forecast back.

Situation. Pluto TV is a free, ad-supported streaming service with linear channels: viewers on the same channel reach each ad break at the same moment. Its revenue services decide, fetch and prepare the ads for every break, calling third-party ad servers and an external transcoding platform. The supply of ads was limited, and different platforms competed for the same ads, so the moment of highest demand was also the moment when getting ads was hardest.

Task. Make sure the breaks expected to see the most demand were filled, without depending on a burst of calls to the ad servers at the exact second each break started.

Action.

  • Forecast the demand. A statistical forecast picked the points expected to see the most demand, using the event being streamed, the day and outside forecasts as inputs.
  • Prefetch ahead of the break. For those points, ads were fetched and processed ahead of time and kept in our own storage, so the break was served from storage instead of from a herd of live requests.
  • Feed the forecast back. The statistics were fed with what actually happened, so the forecast adjusted over time instead of staying a fixed guess.

Result. For the breaks the forecast flagged, the expensive part (asking the ad servers and processing the ads) happened ahead of time, on a schedule we controlled, instead of in the same second as every viewer of the channel. It is the same idea AWS MediaTailor documents for prefetching, built in-house and driven by our own forecast. The lesson I keep from it: when you can predict the herd, don't make it faster at the door; do the work before the door opens.

Read the full case: forecast-driven ad prefetch for Pluto TV's ad breaks →

The takeaway

A thundering herd is not a load problem; it is a timing problem. The same houses, the same requests, the same jobs are fine when they arrive spread over a minute and fatal when they arrive in the same second. Before you add capacity, ask what synchronized your clients (an outage, an expiry, a round number on a clock, a break in a live stream) and pick the way out: spread them, let one of them do the work, pace them, or get there first.

References