9 Distributed Systems Patterns and the Failures They Prevent
Every distributed systems pattern exists because something breaks: a node dies, a call times out, a retry charges twice, an event never arrives. Nine patterns, the failure each one prevents, the failure each one introduces, and when not to use it.
Lists of distributed systems patterns usually define them: replication copies data, sharding splits it, a circuit breaker stops calls. The definitions are right and almost useless, because they skip the only question that matters when you are designing: what breaks if I don't do this, and what breaks because I do?
Every pattern below exists because of a specific failure. Each one also brings a failure of its own, which is why none of them is free and none of them is a default. This article goes through nine of them, grouped by the problem they attack: spreading data and work, surviving failures and keeping state consistent. For each one: the failure it prevents, how it works, how it fails, and when not to use it.
The list leaves out consensus internals such as leader election and quorums on purpose: most teams consume them through etcd, ZooKeeper or their database rather than build them. Instead it includes three patterns application teams implement themselves and that are easy to get wrong: idempotency keys, the transactional outbox and reconciliation.
Spreading data and work
1. Replication
Prevents: losing data, or the whole service, when one machine dies.
How it works. Every write goes to a primary, which copies it to replicas. Reads can go to any replica. If the primary dies, a replica is promoted and the system keeps serving.
How it fails. The copy is either synchronous (the write waits for replicas, so it is slower and stalls when a replica is slow) or asynchronous (fast, but the replicas lag). With asynchronous replication, a failover can lose the last writes the old primary acknowledged and never shipped, and a client that reads from a lagging replica can see its own write disappear. Promotion itself needs a reliable way to pick exactly one new primary, which is why failover managers lean on a consensus store (Patroni for PostgreSQL uses etcd, Consul or ZooKeeper), built on algorithms such as Raft (Raft paper).
Don't use it for scaling writes. Replicas multiply read capacity and durability; every write still lands on one primary. For write scale you need sharding.
2. Sharding
Prevents: one database hitting the limit of one machine, in storage, writes or both.
How it works. Split the data horizontally by a shard key: users A–H on one machine, I–Q on another, R–Z on a third. A routing layer sends each request to the shard that owns its key (Microsoft Azure Architecture Center).
How it fails.
- Hot keys. A bad key concentrates traffic: shard by country and one country melts its shard while the rest idle.
- Cross-shard work. Joins, aggregations and transactions that span shards become application problems.
- Resharding. Adding a shard means moving data. With a naive
hash(key) % Nplacement, changing N moves almost every key; consistent hashing limits the move to about 1/N of the keys (Karger et al., 1997; Amazon Dynamo).
Don't use it before you need it. One well-indexed database with replicas goes very far, and a premature shard key is one of the hardest decisions to undo.
3. Pub/Sub
Prevents: a sender waiting on, or failing because of, every receiver.
How it works. The publisher writes an event to a topic and moves on. Each subscriber reads it at its own pace. A subscriber that is down catches up when it comes back; the publisher never knew it was gone (Microsoft Azure Architecture Center).
How it fails.
- Duplicates. Most brokers deliver at least once, so every consumer must be idempotent (pattern 6).
- Lost events at the source. If the service writes to its database and then publishes, a crash in between loses the event (pattern 7).
- Ordering. It is usually guaranteed only per partition or key, not globally.
- Poison messages and silent lag. A message that always fails loops a consumer until it is moved to a dead-letter queue, and the decoupling hides a consumer falling behind, so consumer lag has to be a monitored metric.
In production. At Pluto TV, the frequency capping system I designed is built on this: an ad-beacon API publishes one NEW_IMPRESSION event per watched ad to Kafka, and independent consumers keep the impression history, update quotas and clean up expired windows, each scaling and failing on its own (the full design).
Don't use it when the caller needs the answer to continue. A request that needs a response is request/reply; forcing it through a topic adds latency and correlation logic for nothing.
Surviving failures
4. Retry with backoff
Prevents: a brief glitch (a dropped connection, a throttled call, a failover in progress) becoming a user-facing error.
How it works. Retry the failed call, waiting longer each time: 1 s, 2 s, 4 s, with jitter (a random spread) so thousands of clients that failed together do not retry together. Stop after a few attempts (AWS Builders' Library).
How it fails.
- Retry storms. Retries multiply load exactly when a dependency is struggling, and they compound across layers: three layers retrying three times each turn one failure into 27 calls.
- Duplicate side effects. Retrying a call that already succeeded, but whose response was lost, repeats it. That is pattern 6.
- Messages that never arrive. A retry only covers failures the system can see; an event that was never delivered is never retried. That is pattern 9.
Retry only transient errors, only idempotent operations, and inside a retry budget.
Don't use it for errors that will not go away on their own: a validation error, a missing permission, a 404.
5. Circuit breaker
Scroll sideways to see the full diagram →
Prevents: cascading failure, where callers pile up threads and connections waiting on a dependency that is already down, until they go down too.
How it works. A breaker wraps calls to a dependency and counts failures. Closed: calls pass. When failures cross a threshold, it trips open: calls fail immediately, ideally to a fallback, and the dependency gets room to recover. After a cool-down it goes half-open and lets one trial call through; success closes it, failure opens it again (Martin Fowler; popularized by Michael Nygard's Release It!).
How it fails. Thresholds are hard to tune: too sensitive and it opens on noise, too lax and it opens after the damage. It needs a meaningful fallback, or "fail fast" just means "fail". And a breaker per instance sees only its own traffic, so instances can disagree about the same dependency.
Don't use it for calls to local, in-process resources, or where a failure must surface immediately anyway. Pair it with timeouts: without a timeout, a slow dependency never counts as a failure.
6. Idempotency key
Scroll sideways to see the full diagram →
Prevents: a retried request doing its work twice: a double charge, a duplicate email, an event counted twice.
How it works. The client generates one unique key per logical operation and sends it with every attempt. The server stores the key together with the result of the first execution; a retry with the same key gets the stored result instead of running again (Stripe; IETF Idempotency-Key header draft). For events, the same idea is a deduplication key: consumers record the ids they have already processed.
How it fails.
- Concurrent attempts. Two attempts can arrive at the same time, so the key must be claimed atomically (a unique constraint, a conditional write), or both run.
- Retention. Keys kept too briefly let a late retry run again; kept forever, the store only grows.
- Key reuse. A key sent with a different payload must be rejected, not answered with the old result.
Don't use it for operations that are naturally idempotent, such as setting a value or deleting by id: the operation already gives the guarantee.
Keeping state consistent
7. Transactional outbox
Scroll sideways to see the full diagram →
Prevents: the dual write. The database commits but the event is never published, or the event goes out and the commit fails, so other services act on a state that does not exist.
How it works. In the same local transaction that changes the business data, insert the event into an outbox table: both commit or neither does. A separate relay reads the outbox and publishes to the broker, either by polling the table or by tailing the database's change log with change data capture (microservices.io; Debezium outbox event router).
How it fails. The relay delivers at least once: if it crashes after publishing and before marking the row as sent, the event goes out again, so consumers need idempotency (pattern 6). Ordering holds only if the relay preserves it per entity. And the outbox needs cleanup, or it becomes the largest table in the database.
Don't use it when nothing downstream depends on the event being delivered, such as best-effort analytics.
8. Saga
Scroll sideways to see the full diagram →
Prevents: a business operation left half-done across services, such as an order created and stock reserved but never charged.
How it works. Split the operation into local transactions, one per service, each with a compensating action that semantically undoes it. If a step fails, the saga runs the compensations of the steps that already completed, newest first (Garcia-Molina & Salem, 1987; microservices.io). Steps are coordinated by an orchestrator or by services reacting to each other's events (choreography), which is where the outbox comes in.
How it fails. There is no isolation: other requests see the intermediate states (stock reserved for an order that will be cancelled). Compensations must be idempotent and retryable, because they can fail too. Some actions cannot be undone, such as an email that was already sent, and need to be ordered last or designed as reversible.
Don't use it when the data fits in one database: a local ACID transaction is simpler and stronger.
9. Reconciliation
Prevents: state that silently drifts from the source of truth because an event was lost, a webhook never arrived or a bug skipped a write. These are the failures no retry can see.
How it works. Periodically compare your copy with the source of truth and repair the difference: fetch what the source has, find what you are missing or have wrong, and enqueue the fix. Dynamo applies the same idea between replicas, comparing Merkle trees in the background (Amazon Dynamo), and Kubernetes controllers are built as loops that keep moving the actual state toward the desired one (Kubernetes).
How it fails. It is only as good as the source's API: rate limits and pagination bound how often and how far back you can compare. The repair must be idempotent, because the event it recovers may still arrive late. And a gap lives until the next run, so the interval is a product decision.
In production. At Afterhour it recovers the calls whose webhook never arrived (the real case below).
Don't use it as a substitute for fixing a delivery path you control. It is the safety net for the paths you don't.
Side by side
| Pattern | Prevents | Costs you | Common tools |
|---|---|---|---|
| Replication | Data loss, downtime on node failure | Replica lag, failover complexity | PostgreSQL streaming replication, MySQL, MongoDB replica sets |
| Sharding | One machine's limits | Hot keys, cross-shard queries, resharding | Vitess, Citus, MongoDB, DynamoDB partitions |
| Pub/Sub | Senders coupled to receivers | Duplicates, ordering, hidden lag | Kafka, SNS/SQS, Pub/Sub, NATS |
| Retry with backoff | Transient errors reaching users | Retry storms, duplicate side effects | SDK retry policies, resilience4j, Polly |
| Circuit breaker | Cascading failure | Tuning, fallbacks, per-instance views | resilience4j, Envoy outlier detection, Polly |
| Idempotency key | Double side effects on retry | A key store, atomic claims, retention | Unique constraints, conditional writes, Redis |
| Transactional outbox | Saved data with a lost event | A relay, duplicates, table cleanup | Debezium, polling relays |
| Saga | Half-done cross-service operations | No isolation, compensation logic | Temporal, Step Functions, Camunda |
| Reconciliation | Silent drift from the source of truth | Delay until the next run, API limits | Scheduled jobs, Kubernetes-style controllers |
Patterns compose
These nine are a chain more than a list: each one covers a failure the previous ones introduce.
- Retry needs an idempotency key. Retries absorb blips; keys make sure a retry of something that already succeeded does not do it twice.
- Pub/Sub needs an outbox. Topics decouple the services; the outbox makes sure the event is published if and only if the data was saved.
- Sagas need both. Each step is a local transaction that publishes its event through an outbox, and every step and compensation is idempotent because they will be retried.
- Everything needs reconciliation. Retries, outboxes and sagas cover the failures the system can see. A periodic compare against the source of truth catches the rest.
- Circuit breakers keep retries honest. Without one, every layer keeps retrying a dependency that is already down.
Real-world case: the call pipeline at Afterhour
Scroll sideways to see the full diagram →
Situation. Afterhour is a multi-tenant AI voice receptionist for law firms, which I built for Big Voodoo Interactive. For a firm, a missed call is a missed case, so every call that ends has to be processed: structured extraction, recording archival, notifications and CRM sync. The voice provider reports finished calls through a webhook.
Task. Make post-call processing reliable when any step can fail and when the upstream webhook itself is not guaranteed to arrive.
Action.
- Persist first, process later. The webhook handler (API Gateway + Lambda) only stores the raw payload and enqueues the call id on SQS. Processing can fail, retry or be replayed without ever losing the original call.
- Typed jobs with retry schedules and dead-letter queues. Each job type declares its own retry schedule; a job that exhausts its retries lands in a dead-letter queue instead of looping or disappearing.
- Reconciliation for what retries cannot see. Retries only cover failures that reach the system. Roughly 7% of calls never arrived via webhook at all, so an EventBridge job compares against the provider every 5 minutes and enqueues the missing ones.
- Observability before features. Custom metrics, dashboards and alerts made the silent failures visible, which is how the missing-calls problem was found and measured in the first place.
Result. Calls lost by webhook delivery, about 7%, are now recovered automatically, and the platform has processed more than 25,000 calls in production.
The lesson I keep from it: retry with backoff protects you from the failures you can see; reconciliation finds the ones you can't. Retries, a dead-letter queue and a reconcile loop are three different answers to three different failures.
Read the full case: Afterhour, a multi-tenant AI voice receptionist that never loses a call →
Where to start
- Name the failure first. Write down what breaks and how often, before picking a pattern.
- Prefer the boring option. One database with replicas, synchronous calls with timeouts, a local transaction. Add a pattern when the failure it prevents is real.
- Make operations idempotent early. Retries, pub/sub, outboxes and sagas all depend on it.
- Never write to a database and a broker separately. Use an outbox, or make the broker the source of truth.
- Add a reconcile loop for every external source you depend on: webhooks, third-party APIs, other teams' events.
- Measure the new failure mode. Replica lag, consumer lag, breaker state, outbox backlog, reconciled gaps: each pattern adds a metric you now have to watch.
The takeaway
A distributed systems pattern is a trade: it removes one failure and adds another. Replication trades data loss for lag, retries trade transient errors for duplicates, idempotency keys trade duplicates for a key store, outboxes trade lost events for a relay, sagas trade half-done operations for compensation logic. Learn the failure each pattern prevents and the one it introduces, and you will know when to use it, and when not to.
References
- Karger et al., Consistent hashing and random trees (1997).
- DeCandia et al., Dynamo: Amazon's highly available key-value store (2007).
- Ongaro & Ousterhout, In search of an understandable consensus algorithm (Raft).
- AWS Builders' Library, Timeouts, retries and backoff with jitter.
- Martin Fowler, CircuitBreaker.
- Stripe, Designing robust and predictable APIs with idempotency, and IETF, The Idempotency-Key HTTP header field (draft).
- Chris Richardson, Transactional outbox and Saga; Debezium, Outbox event router.
- Garcia-Molina & Salem, Sagas (1987).
- Kubernetes, Controllers.
- Microsoft Azure Architecture Center, Publisher-Subscriber, Sharding, Retry and Circuit Breaker.