System design glossary

A result stating that a replicated system experiencing a network partition must choose between staying available on both sides and preserving linearizable consistency. It constrains behavior only while a partition is active and describes nothing about the trade-offs a system makes when the network is healthy.

Use CAP to reason about what happens the moment a link drops, and expect real designs to sit on a spectrum through tunable quorums. The framing misleads people into treating AP or CP as a permanent identity, when the same database offers either depending on the consistency level chosen per query. Brewer's own follow-up argues for planning partition detection, degraded mode, and recovery instead of picking letters.

Source: Brewer: CAP Twelve Years Later

A partitioning scheme that maps keys and nodes onto the same hash ring and assigns each key to the next node clockwise, so adding or removing one node relocates roughly 1/n of the keys. Virtual nodes give each machine many ring positions to smooth the distribution.

Use it wherever the node set changes and a full reshuffle would be ruinous, as in Dynamo, Cassandra, and most cache fleets. Plain consistent hashing still produces load skew above 30 percent without enough virtual nodes, and it imposes no ceiling on any single node's share. Bounded-load variants and rendezvous hashing address the skew that operators usually discover from one node running hot in production.

Source: Karger et al.: Consistent hashing and random trees

A replicated data type whose merge operation is commutative, associative, and idempotent, so replicas receiving the same updates in any order converge to identical state without coordination. Convergence follows from the type's algebra, and the merged value can still discard what a user intended.

A good fit for collaborative editing, presence, counters, and offline-first mobile sync, which is how Automerge, Yjs, and Figma's multiplayer layer work. Costs are metadata growth from tombstones and per-actor state, plus semantics you cannot express, since a set that merges by union cannot enforce an inventory floor. Teams adopt CRDTs to avoid conflict handling and then reimplement conflict handling above them to get merges users accept.

Source: crdt.tech: CRDT resources and papers

A liveness property guaranteeing that if writes stop, every replica eventually converges on the same value. It places no bound on how long convergence takes and allows reads to return stale or out-of-order values for the whole interval before it completes.

Adequate for view counters, feeds, and DNS, where a few seconds of staleness costs nothing measurable. The word eventually carries no timing promise: convergence depends on anti-entropy and repair, and a partitioned replica can serve stale data for as long as the partition lasts. When users must observe their own writes, layer session guarantees such as read-your-writes on top rather than hoping the window stays small.

Source: Vogels: Eventually Consistent

A protocol by which a group of nodes agrees that one member holds a role for a bounded term, usually backed by a lease or a majority vote. It guarantees agreement among nodes following the protocol, and it cannot prevent a partitioned or paused former leader from believing it still holds the role.

Every primary-based system needs one, and the property people over-read is exclusivity. A leader that pauses for a long GC or loses its network can wake after its lease expired and issue writes. Pair election with fencing tokens or lease-checked writes so the storage layer refuses stale leaders, and account for the detection gap: the cluster learns a leader is gone only after the lease times out.

Source: Google: The Chubby lock service

A correctness condition for operations on a single object requiring that each operation appear to take effect atomically at some instant between its invocation and its response, consistent with real time. Once a write returns, every later read anywhere in the system observes that write or a newer one.

This is the guarantee that lets clients treat a distributed store as one register, and it costs a coordination round trip on every operation. Serializability constrains the ordering of multi-object transactions with no real-time requirement, so a system can be serializable while serving a stale snapshot. Compare-and-swap, distributed locks, and leader leases all assume linearizable storage underneath, and they misbehave in exactly the way people find hardest to debug when they get eventual consistency instead.

Source: Herlihy & Wing: Linearizability

An extension of CAP stating that during a partition (P) a system trades availability (A) against consistency (C), and else (E), when the network is healthy, it trades latency (L) against consistency (C). It describes the steady-state behavior CAP leaves unspecified.

PACELC is the more useful framing for a design review, because systems spend nearly all their time partition-free while paying latency for every synchronous cross-region acknowledgment. Spanner is PC/EC and buys strict consistency with commit-wait; Dynamo-style stores are PA/EL. Ask which quadrant a datastore occupies before promising a p99 that a cross-continent round trip cannot support.

Source: Abadi: Consistency tradeoffs in modern distributed database system design

A family of consensus protocols in which proposers, acceptors, and learners agree on a single value through two phases of majority voting, tolerating crash failures of a minority of participants. Multi-Paxos amortizes the first phase by keeping one stable leader across many instances.

Paxos underlies Chubby, Spanner, and Megastore, and it remains the reference point for correctness arguments about consensus. The published protocol leaves membership change, log compaction, and leader stability as exercises, which is why teams implementing it from the paper ship subtle bugs that surface only under partition. Pick an existing consensus library and spend the saved effort on failure injection testing.

Source: Lamport: Paxos Made Simple

A rule requiring that a read or write be acknowledged by a minimum number of replicas, sized so that read and write sets overlap. With N replicas, R + W > N guarantees a read touches at least one replica holding the most recent acknowledged write.

Quorums let you tune each operation between durability and latency: W=N waits for the slowest replica, W=1 is fast and leaves a window of data loss. The intersection property alone falls short of linearizability, since concurrent clients can still observe different orders without read repair or a consensus protocol on top. Sloppy quorums with hinted handoff, as in Dynamo, abandon the intersection guarantee during failures.

Source: Amazon Dynamo (SOSP 2007)

A consensus algorithm that elects a leader by majority vote, replicates an append-only log from that leader to followers, and commits an entry once a majority has stored it. Term numbers and log-matching rules ensure any newly elected leader already holds every committed entry.

Raft backs etcd, Consul, TiKV, and CockroachDB, and it wins over Paxos mostly by having a specification engineers can implement correctly. Throughput is bounded by the single leader and one majority round trip, so a five-node group spread across regions pays inter-region latency on every write. Operators run even-numbered clusters or place three of five members in one failure domain, which forfeits the majority guarantee they deployed Raft to get.

Source: Ongaro & Ousterhout: In Search of an Understandable Consensus Algorithm

A session guarantee that a client always observes its own earlier writes, even when the underlying store is otherwise eventually consistent. It promises nothing about when other clients see those writes, and it holds only inside the boundary the session defines.

This is the guarantee that keeps a user from posting a comment and finding it missing after the redirect. Implementations pin the session to the primary for a window, route by a stored log position, or use a sticky replica. The boundary breaks when a user switches devices or a load balancer moves them to another region, so define the session around user identity and carry the log position in a token the client sends back.

Source: MongoDB: causal consistency and read/write concerns

An atomic commit protocol in which a coordinator asks every participant to prepare and commits only when all vote yes. A participant that voted yes must hold its locks and await the decision, which blocks it for as long as the coordinator stays down.

2PC delivers real atomicity across resources and earns its place inside one datacenter with a recoverable coordinator, as in XA transactions or Kafka's transactional writes. The blocking window is the price: locks stay held across a coordinator failure, turning one process outage into a distributed deadlock. Across service boundaries most teams choose sagas with compensations, because holding locks over a network multiplies the blast radius of any single slow participant.

Source: MySQL: XA transactions

A causality-tracking structure holding one counter per node, where comparing two vectors reveals whether one event happened before another or whether the two are concurrent. It detects conflicting concurrent updates and supplies no rule for resolving them.

Dynamo-style stores attach vector clocks to objects and hand a client the conflicting siblings to merge, which pushes resolution into application code that usually picks the higher wall-clock timestamp and loses a write. Clock size grows with the number of writers, so implementations truncate entries and then report false concurrency. Version vectors and dotted version vectors handle client-driven writes with less growth.

Source: Lamport: Time, Clocks, and the Ordering of Events

A balanced on-disk index structure that stores sorted keys in fixed-size pages with a fan-out in the hundreds, so a lookup touches only three or four pages even on large tables. Updates modify pages in place, which keeps reads cheap and makes writes random.

The default for OLTP engines such as PostgreSQL, InnoDB, and SQL Server because point lookups and range scans both stay predictable. The cost is random write IO and page splits that fragment the index as the table churns. Engineers add an index for every query pattern and then watch write throughput halve, since every index multiplies the work on the write path and enlarges the working set held in the buffer pool.

Source: PostgreSQL: B-Tree indexes

A probabilistic set-membership structure that answers "definitely absent" or "possibly present" using a bit array and k independent hash functions. It never returns a false negative, and it trades a tunable false-positive rate for a few bits of space per element.

Standard in LSM engines such as Cassandra and RocksDB, where one filter per SSTable removes most disk reads for keys that do not exist. Roughly 10 bits per key buys a 1 percent false-positive rate, and a standard Bloom filter supports no deletion. Engineers size filters from today's row count and then watch the false-positive rate climb as the dataset outgrows the assumption, which quietly returns disk reads to the read path.

Source: Apache Cassandra: Bloom filters

A technique that turns committed database changes into an ordered event stream by reading the transaction log, such as the MySQL binlog or the PostgreSQL WAL. Consumers receive inserts, updates, and deletes in commit order without polling application tables.

CDC gives search indexes, caches, and warehouses a replication-quality feed without dual writes, and Debezium with Kafka Connect is the usual assembly. The costs are schema evolution handling, snapshotting existing rows at bootstrap, and the care a replication slot demands, since an unconsumed slot retains WAL until the primary's disk fills. Treat the slot lag metric as a page-worthy alert on the source database.

Source: Debezium: architecture

A write-optimized storage structure that buffers updates in an in-memory table, flushes them to disk as immutable sorted files, and merges those files in background compactions. A read may consult several levels, so read cost grows with the number of levels on disk.

Reach for an LSM engine when writes dominate and you can afford background compaction stealing IO, which is why RocksDB, Cassandra, and ScyllaDB use one. The cost arrives as write amplification of 10x or more under leveled compaction, plus latency spikes when compaction falls behind ingest. Teams size disks for logical data volume and then run out of space during a compaction backlog, since compaction needs headroom to rewrite files it has yet to delete.

Source: RocksDB: leveled compaction

A concurrency control scheme in which each write creates a new row version and each transaction reads the versions visible in its snapshot, so readers never block writers. Obsolete versions remain on disk until a cleanup process proves no live snapshot can still see them.

MVCC is why one long analytical query in PostgreSQL can pin dead tuples and bloat a table for hours, since vacuum cannot reclaim a version any open snapshot might need. Budget for the cleanup path: autovacuum tuning, monitoring the xmin horizon, and killing runaway transactions. Snapshot isolation also permits write skew, so an invariant spanning two rows needs explicit locking or serializable isolation.

Source: PostgreSQL: MVCC introduction

The delay between a write committing on a primary and that write becoming visible on a replica, measured as seconds of staleness or as bytes of unshipped log. Asynchronous replication leaves the lag unbounded during load spikes, long-running replica queries, or network stalls.

Any read served by a replica can return data older than a write the same user just made, which breaks create-then-redirect flows. Fixes include pinning a session to the primary for a few seconds or waiting on a log position, as MySQL does with GTIDs and PostgreSQL with LSN comparisons. Alerting on average lag hides the tail, so track the maximum across replicas and the age of the oldest unreplayed transaction.

Source: PostgreSQL: hot standby and replication delay

Horizontal partitioning of a dataset across independent database instances, where a shard key decides which instance owns each row. Every shard runs its own storage and query engine, so queries and transactions that span shards need coordination in the application or a proxy layer.

Shard when one primary can no longer absorb the write volume or the working set exceeds memory, and choose the key from the dominant access pattern. Resharding a live system is the expensive part, which is why Vitess and Citus exist. The frequent mistake is a monotonically increasing key such as an auto-increment ID or timestamp, which routes every new write to a single shard and leaves the rest of the fleet idle.

Source: Vitess: shard concepts

A durability mechanism in which every change is appended to a sequential log and flushed to stable storage before the corresponding data pages are modified. Crash recovery replays the log to restore committed transactions and discard partial ones.

The WAL makes fsync latency your commit latency, so a slow or contended disk shows up directly as transaction throughput. Group commit and batched flushes amortize that cost, which is why PostgreSQL exposes commit_delay and synchronous_commit. The common shortcut is relaxing synchronous commit for speed, which buys throughput and creates a window of acknowledged transactions that vanish after an unclean shutdown.

Source: PostgreSQL: Write-Ahead Logging

A failure mode where a popular cached entry expires and many concurrent requests miss at once, each recomputing the same expensive value and saturating the backing store. Load at the moment of expiry scales with request concurrency, independent of the steady-state hit rate.

Any high-traffic key with a costly recompute is a candidate, and the remedy is to let one request recompute while the rest wait or receive stale data. Single-flight coalescing, a short-lived lock key, and probabilistic early expiration all work, and the last one needs no coordination. Teams add caching to shield a slow database and then take that database down with the first synchronized expiry under peak traffic.

Source: Vattani et al.: Optimal probabilistic cache stampede prevention

A caching pattern where the application reads the cache, and on a miss loads from the database, stores the value in the cache, and returns it. The cache holds only data that has actually been requested, and stale entries survive until eviction or explicit invalidation.

The default pattern because it degrades sanely: a cache outage degrades into database load that the origin can absorb. Two failure modes dominate. A miss on a hot key sends every concurrent request to the database until one loader wins, which single-flight coalescing fixes, and invalidation after a write races with an in-flight read that repopulates the old value, which versioned keys or short TTLs contain.

Source: Azure Architecture Center: Cache-Aside pattern

A cache replacement policy that discards the entry unused for the longest time, treating recency of access as a predictor of future use. Production implementations usually sample a handful of candidates rather than maintaining exact ordering, since true LRU needs bookkeeping on every read.

LRU suits workloads with temporal locality and gets flattened by a large scan, which walks the key space and evicts the working set behind it. LFU and segmented policies resist that, which is why Redis ships allkeys-lfu and Caffeine uses W-TinyLFU. Watch hit rate together with eviction rate: a steady hit rate with climbing evictions means the cache is sized below the working set, and the hit rate will drop as soon as traffic shifts.

Source: Redis: key eviction policies

A contention pattern where one event wakes many blocked waiters simultaneously and they all compete for a resource only a few can use, wasting scheduling and lock work. It shows up in accept queues, mass client reconnects after a restart, and cron jobs aligned to the same minute.

Recovery paths are where this hurts: a service restarts, every client reconnects within the same second, and the cold instance falls over again before it warms up. Exponential backoff with full jitter, staggered schedules, and EPOLLEXCLUSIVE on accept queues are the standard mitigations. AWS measured full jitter as the variant that genuinely reduces contention, since backoff without randomness keeps clients synchronized while spacing their attempts.

Source: AWS Architecture Blog: exponential backoff and jitter

An expiry duration attached to a cached entry, after which the entry is treated as absent and the next request repopulates it. TTL places an upper bound on staleness without requiring the writer to send any invalidation message.

TTL is the cheapest correctness mechanism in caching and the thing that saves you when invalidation logic has a bug. Uniform TTLs assigned at deploy time expire together and produce a synchronized miss wave, so add 10 to 20 percent jitter. For expensive values, serve the stale entry while refreshing in the background, which keeps expiry from turning into a latency spike visible to users.

Source: Redis: EXPIRE

A caching pattern where every write updates the cache and the backing store synchronously before the write is acknowledged, keeping the two in step. Write latency includes both hops, and the cache fills only with data that has been written through it.

Useful when reads follow writes closely and a stale read is unacceptable, such as session state or account settings. The costs are a slower write path and a cache full of entries nobody reads, so it pairs with a TTL to bound waste. Failure handling is the part teams skip: when the cache write succeeds and the store write fails, the acknowledgment must be withheld and the cache entry removed, or the cache starts serving values the database never accepted.

Source: AWS ElastiCache: caching strategies

A delivery guarantee under which a message is redelivered until the consumer acknowledges it, so no message is lost and duplicates are expected. Handling those duplicates belongs to the consumer, normally through idempotent operations or a deduplication store.

This is the default in SQS, in Kafka with manual offset commits, and in most brokers, because the alternative drops messages when a consumer dies between processing and acknowledgment. Duplicates arrive in real volume during redeploys, rebalances, and network timeouts. Design every consumer around a natural idempotency key from the start, since retrofitting deduplication into a payment or email path after launch means reconciling data you already sent.

Source: AWS SQS: standard queues

A set of consumer instances sharing one subscription, where each partition is assigned to exactly one member so every message is processed once per group while parallelism scales up to the partition count. Members beyond the partition count sit idle.

Partition count is the parallelism ceiling in Kafka, and raising it later changes key-to-partition mapping and breaks per-key ordering, so over-provision modestly at design time. Rebalances are the operational cost: every join or leave pauses consumption for the whole group. Processing that exceeds max.poll.interval.ms triggers a rebalance loop that presents as a broker problem and is actually a slow handler.

Source: Confluent: Kafka consumer documentation

An architectural pattern that separates the write model from the read model so each can use storage and a schema suited to its access pattern. The read side is updated from the write side, usually asynchronously, which leaves reads eventually consistent with writes.

Justified when read and write loads differ by an order of magnitude or their shapes conflict, such as a normalized write model feeding denormalized search documents. The cost lands on the UI, which must handle a read model that has yet to reflect the write a user just made. CQRS and event sourcing are separable, and adopting both simultaneously is the usual reason a migration stalls.

Source: Fowler: CQRS

A separate destination that receives messages a consumer failed to process after a configured number of attempts, removing them from the main flow so processing continues. Messages retain the original payload and failure metadata for inspection and replay.

Without one, a poison message blocks a partition or redelivers forever; with one, the queue fills up and nobody looks. Alert on arrival rate as well as depth, since a slow trickle of failures matters more than a stale backlog. Build the replay tooling at the same time as the queue, because messages sitting in a DLQ with no path back are indistinguishable from lost messages to everything downstream.

Source: AWS SQS: dead-letter queues

A persistence approach that stores state as an append-only sequence of domain events and derives current state by replaying them, using snapshots to bound replay cost. The event log becomes the system of record and supports reconstructing the state as of any past moment.

Worth it for audit-heavy domains, temporal queries, and rebuilding read models under new logic. The costs are permanent: event schema versioning with no migration escape hatch, replay time that grows with the log, and a team that has to model in events. Most systems that reach for it need an audit table and a change feed, and the pattern is difficult to back out of once the log is authoritative.

Source: Fowler: Event Sourcing

A property meaning each message affects system state one time, achieved by combining at-least-once transport with deduplication or transactional state updates at the consumer. A network cannot promise that a message crosses it exactly once, so the guarantee is built at the endpoints.

Kafka's transactional producer with read_committed consumers provides this for Kafka-to-Kafka pipelines by making offset commits and output writes atomic, and that boundary ends the moment a consumer writes somewhere else. For external side effects such as charging a card, the guarantee comes from an idempotency key the receiver checks. When a vendor advertises exactly-once, ask where its transaction boundary starts and stops.

Source: Confluent: exactly-once semantics in Apache Kafka

A client-supplied unique identifier attached to a request so the server can recognize a retry and return the original result instead of performing the operation a second time. The server stores the key with its outcome for a retention window, making retries safe over an unreliable network.

Any non-idempotent operation exposed over HTTP needs one, which is why Stripe requires it on charge creation. The details decide whether it works: persist the key in the same transaction as the effect, replay the original response body, and reject a reused key that arrives with a different payload. A key generated server-side or regenerated per attempt provides no protection, because each retry then looks like a fresh request.

Source: Stripe API: idempotent requests

A messaging model where publishers send messages to a topic with no knowledge of subscribers, and the broker delivers a copy to every interested subscriber. Producers and consumers scale and fail independently, and the broker owns fan-out, retention, and delivery guarantees.

Choose it when several independent consumers need the same event and you want to add the next consumer without touching the producer. The cost is observability: a message with no subscriber vanishes silently, and one debugging session spans producer, broker, and consumer logs. Decide retention and replay before launch, since the question of who consumed what becomes unanswerable without durable offsets or acknowledgment tracking.

Source: Google Cloud Pub/Sub: overview

A way to maintain consistency across services without distributed locks by splitting a transaction into a sequence of local transactions, each paired with a compensating action that undoes it. Intermediate states are visible to other readers, so the sequence delivers atomicity of outcome and no isolation.

Use it when a workflow crosses service boundaries and holding a 2PC lock is impractical, such as order, payment, and fulfillment. The real engineering is in compensation: refunding a charge is business logic with its own failure modes, and some steps such as a sent email cannot be undone. Orchestration keeps the state machine in one service and stays legible past three steps, while choreography spreads the flow across every participant's event handlers.

Source: Richardson: Saga pattern

A pattern where a service writes its state change and an event row to the same database inside one transaction, and a separate relay process reads the outbox table and publishes the event. Database atomicity removes the dual-write failure where one of the two writes succeeds alone.

Reach for it whenever a service must update its own data and tell other services, since writing to the database and the broker separately drops events whenever the second write fails. The relay usually rides on CDC through Debezium or a polling loop, and publication stays at-least-once, so consumers still need idempotency. Prune published rows on a schedule, or the outbox becomes the largest table in the database.

Source: Richardson: Transactional outbox pattern

A distributed network of caching proxies near users that serves content from an edge location and contacts the origin only on a miss. It cuts origin load and round-trip time, and its effectiveness depends almost entirely on cache key design and freshness headers.

The lever most teams underuse is the cache key: query strings, cookies, and Vary headers fragment the cache and sink the hit ratio with no error to alert on. Measure hit ratio per path before adding origin capacity. Stale-while-revalidate and origin shielding matter more than the provider's PoP count once you are serving anything dynamic, and a single Set-Cookie on a static asset can disable caching for an entire route.

Source: Cloudflare Learning Center: What is a CDN

A delay pattern where the first item in an ordered queue stalls and everything behind it waits, even when the later items could have been handled independently. It appears in HTTP/1.1 pipelines, TCP streams carrying multiplexed HTTP/2 requests, and single-partition message consumers.

HTTP/2 removed the application-layer version and left the TCP-layer one, which is why HTTP/3 moved to QUIC over UDP with per-stream loss recovery. The same shape appears in Kafka, where one poison message in a partition stalls every later message for that consumer group. Partition-level parallelism and a dead letter queue apply the same remedy in messaging, routing the stuck item off the path everything else shares.

Source: RFC 9114: HTTP/3

Layer 4 load balancing forwards TCP or UDP connections using addresses and ports without inspecting payloads, while layer 7 terminates the connection and routes on HTTP attributes such as path, header, or cookie. Layer 7 enables per-request routing and retries at the cost of termination and parsing.

Choose L4 for raw throughput, protocol independence, and long-lived connections, and L7 when you need path-based routing, header rewriting, or per-request spreading. The trap with L4 is multiplexing: HTTP/2 and gRPC carry many requests on one connection, so connection-level balancing pins a busy client to one backend and leaves request load badly skewed. That is why gRPC fleets rely on an L7 proxy or client-side balancing with subsetting.

Source: AWS: Elastic Load Balancing overview

A control that caps how many requests a client may issue in a time window, rejecting or delaying the excess to protect capacity and enforce fairness. Limits attach to an identity such as API key, user, or IP address, and that choice determines what the limit actually protects.

Derive limits from measured capacity and return 429 with Retry-After so well-behaved clients back off instead of hammering. Distributed enforcement is the hard part: per-instance counters let a client reach N times the intended rate, and a shared Redis counter adds a network hop to every request. Stripe runs separate limiters for request rate, concurrency, and expensive endpoints, which catches abuse that a single global limit lets through.

Source: Stripe: Scaling your API with rate limiters

An infrastructure layer that moves service-to-service concerns such as mTLS, retries, timeouts, and traffic splitting into proxies deployed beside each workload, configured from a central control plane. Application code issues plain calls while the data plane enforces policy.

Worth adopting once you have enough services in enough languages that per-library implementations of retries and TLS have drifted apart. The costs are concrete: an extra proxy hop on every call, typically single-digit milliseconds added to p99, plus a control plane that becomes a critical dependency during incidents. A small fleet gets most of the value from one shared client library and an ingress proxy, at a fraction of the operational surface.

Source: Istio: architecture

A rate-limiting algorithm that holds a bucket of tokens refilled at a fixed rate, where each request consumes one token and a request arriving at an empty bucket is rejected or queued. Bucket capacity sets the allowed burst and the refill rate sets the sustained rate.

Token bucket is the default because it tolerates a burst after an idle period, matching how real clients behave, and it gives you two independent knobs. AWS API Gateway and most API proxies implement exactly this. A leaky bucket instead smooths output to a constant rate and gives up burst tolerance, which suits shaping traffic toward a downstream with fixed capacity such as a payment processor.

Source: AWS API Gateway: request throttling

A technique that keeps a set of reusable open connections to a backend so callers avoid per-request handshake and setup cost. Pool size caps concurrency against that backend, which makes the pool a queueing point that has to be sized and monitored like any other.

PostgreSQL allocates a process per connection, so a few hundred direct connections from application instances will exhaust memory, and PgBouncer in transaction mode is the standard answer. Size the pool near the backend's useful concurrency, usually far lower than intuition suggests, since a pool wider than the database's parallelism just relocates the queue. Monitor time spent waiting for a connection separately from query time, because saturation appears there first.

Source: PgBouncer: features and pooling modes

A measurement error where a load generator stalls while waiting on a slow response and therefore never issues the requests that would have arrived during the stall, deleting the worst latencies from its sample. Reported percentiles then understate real user-visible latency, sometimes by an order of magnitude.

Any closed-loop benchmark that sends the next request only after the previous one returns carries this bug, which covers most hand-rolled load test scripts. Correct it by recording latency against intended send time, which wrk2, JMeter throughput shaping, and HdrHistogram's interval correction all support. A benchmark reporting a p99 within a few milliseconds of its median is showing the signature of the problem.

Source: HdrHistogram: corrected value recording

A partition that receives a disproportionate share of traffic because the partition key concentrates access, capping throughput at what a single partition can serve no matter how much total capacity is provisioned. Timestamp keys, one dominant tenant, and low-cardinality keys are the usual causes.

DynamoDB, Kafka, and Cassandra all present this the same way: aggregate metrics look healthy while one partition throttles and a subset of users sees timeouts. Fixes include salting the key with a bounded random suffix, hashing a prefix, or splitting the dominant tenant across sub-partitions. Salting costs you ordered range scans, so decide before launch, since changing a partition key later means rewriting the dataset.

Source: AWS: DynamoDB partition key design

A data access anti-pattern in which fetching a collection of N records triggers one extra query per record to load a related field, producing N+1 round trips where a join or batched fetch would need one or two. Latency then scales linearly with result size.

It hides well in ORMs and GraphQL resolvers, where lazy loading looks like an ordinary property access, and it stays invisible in development against twenty rows. Fixes are eager loading, explicit joins, or a batching layer such as DataLoader. Catch it by asserting query counts per request in tests, because code review rarely surfaces a round trip that the framework generates on your behalf.

Source: Rails Guides: eager loading and the N+1 problem

The latency value below which 99 percent of requests complete during a measurement window, describing the slow tail that an average conceals. On a page that assembles 50 backend calls, roughly 40 percent of page loads contain at least one call from that slowest one percent.

Percentiles are the only latency numbers worth alerting on, and they do not average across hosts or time windows, so combining ten servers' p99s produces a number that describes nothing. Compute from histograms instead, using Prometheus buckets or HdrHistogram. Closed-loop load generators that wait for each response before sending the next also understate the tail, an artifact known as coordinated omission.

Source: Prometheus: histograms and summaries

The effect where a request that fans out to many backends inherits the slowest response, so overall latency tracks the tail of its components instead of their median. Fanning out to 100 servers that are each slow 1 percent of the time makes a slow response nearly certain.

This is why a service with an excellent median can post a terrible p99 once it fans out, documented by Dean and Barroso in The Tail at Scale. Mitigations include hedged requests issued after a p95 delay, tied requests cancelled on first response, and returning partial results at a deadline. Adding replicas without addressing the tail simply multiplies the number of chances to be slow.

Source: Dean & Barroso: The Tail at Scale

A flow-control mechanism in which a slow consumer signals upstream to reduce the production rate, bounding queue depth and memory use. A pipeline without it converts overload into unbounded buffering, which becomes latency growth and then memory exhaustion.

Any pipeline with a rate mismatch needs an explicit answer: block the producer, drop messages, or shed load at the edge. TCP windows, gRPC flow control, and Reactive Streams all implement the blocking variant. Unbounded queues are the common bug, because they hide the mismatch until the process dies and takes every in-flight message with it, then refill at the same rate once the service restarts.

Source: Reactive Streams specification

An isolation pattern that partitions a shared resource such as a thread pool or connection pool, so saturation caused by one dependency cannot consume the capacity other work needs. Failure stays confined to the partition assigned to the failing dependency.

Reach for it when one slow downstream can drain a shared pool and stall unrelated endpoints, the classic path from a single dependency's latency to a full outage. The cost is lower utilization, because reserved capacity idles while its partition is quiet. Sizing partitions from average load leaves no headroom for the burst that causes the incident, so size from peak and accept the waste.

Source: Azure Architecture Center: Bulkhead pattern

A deployment technique that routes a small fraction of production traffic to a new version, compares its error and latency metrics against the version already running, and expands or rolls back based on that comparison. Blast radius stays bounded by the traffic fraction.

Worth the pipeline investment for any service where a bad deploy costs more than an hour of engineering time. Automated metric comparison against a control group is what separates a canary from a slow rollout with someone watching a dashboard. Two failure modes recur: a canary too small to produce a statistically meaningful signal, and canary traffic that misrepresents production, such as a canary receiving only requests from one healthy region.

Source: Fowler: CanaryRelease

A resilience component that tracks failures against a dependency and, past a threshold, short-circuits further calls by failing immediately for a cooldown period, then admits trial requests before closing again. It converts slow cascading failure into fast bounded failure.

Wrap any remote dependency where timeouts would otherwise consume threads or connections, and trip on error rate over a rolling window instead of consecutive failures. The dangerous configurations sit at both extremes: a threshold so high the breaker never opens, or one so twitchy that a single slow request cuts off a healthy dependency. Every breaker needs a defined fallback, since opening one only relocates the decision to the caller.

Source: Fowler: CircuitBreaker

The amount of unreliability an SLO permits over its window, computed as one minus the target, so a 99.9 percent objective allows about 43 minutes of failure every 30 days. It turns reliability into a quantity that release decisions can draw down.

The budget only works when exhausting it triggers something, typically freezing risky launches until the service is back inside its objective. Teams publish budgets, blow through them, and keep shipping, which reduces the SLO to decoration. Burn-rate alerts across two windows, a fast burn measured over an hour and a slow burn over a day, catch both sharp outages and the slow degradation that a monthly review would miss.

Source: Google SRE Workbook: Implementing SLOs

A runtime switch that separates deploying code from enabling behavior, letting a change ship dark and activate for a chosen segment. Flags allow a broken path to be disabled in seconds without a deploy, and each one adds a code path that needs testing in both states.

Use them for progressive rollout, kill switches, and experiments, and treat flag evaluation as a hot-path dependency with local caching and a safe default when the flag service is unreachable. The cost compounds combinatorially: twenty live flags describe a million configurations, nearly all untested. Set a removal date at creation, since Knight Capital's 2012 trading loss came from a repurposed flag that reactivated dormant code on partially deployed servers.

Source: Hodgson: Feature Toggles

A monotonically increasing number issued alongside a lock or lease, which the client must present to the resource it writes to; the resource rejects any token lower than the highest it has already seen. This stops a delayed or paused lock holder from corrupting state after its lease expired.

Any distributed lock that guards writes needs one, because a client can pause for a full GC cycle, wake after its lease expired, and issue a write while still believing it holds the lock. The enforcement burden lands on the storage layer, which must record and compare tokens, and that is why a lock service by itself cannot deliver mutual exclusion. Kleppmann's critique of Redlock is the canonical treatment of this gap.

Source: Kleppmann: How to do distributed locking

An SLI is a measured indicator of service behavior such as request latency or error rate, an SLO is the internal target for that indicator over a window, and an SLA is the contractual promise with financial consequences, normally set looser than the SLO.

Pick indicators users actually feel and measure them as close to the user as possible, since a server-side latency histogram misses the load balancer queue. A 100 percent objective removes your ability to ship anything, and an objective nobody enforces removes the reason to have one. Google's guidance holds: derive the target from what users need, then spend the remaining error budget deciding between features and stability.

Source: Google SRE Book: Service Level Objectives

A failure state in which a network partition leaves two subsets of a cluster each believing it holds the authoritative role, so both accept writes and the datasets diverge. Healing the partition then requires discarding or merging writes one side already acknowledged to clients.

Majority quorum prevents it by construction, which is why three-node and five-node clusters are standard and two-node clusters with a naive heartbeat are the classic trap. Where quorum is impractical, use an external witness, STONITH-style node fencing, or fencing tokens checked at the storage layer. RabbitMQ, Elasticsearch, and Redis Sentinel deployments all have documented incidents that trace back to a minority side accepting writes.

Source: RabbitMQ: clustering and network partitions

An access control flaw in which an application exposes a reference to an internal object, such as a database ID in a URL, and returns that object without verifying the caller is authorized for that specific record. Changing the identifier in a request reaches another tenant's data.

This is the most common serious bug in multi-tenant APIs, and it survives review because the endpoint looks correct and authentication passes cleanly. The fix is per-object authorization in every handler, enforced at the data access layer so a query cannot be written without a tenant predicate. Unguessable identifiers reduce discovery and stop nothing once an ID leaks, so treat UUIDs as an obstacle and keep the authorization check.

Source: OWASP: Insecure Direct Object Reference Prevention Cheat Sheet

A compact, URL-safe token format carrying signed JSON claims that a receiver can verify without a database lookup. The signature proves issuer and integrity, the payload stays readable unless the token is also encrypted, and a valid signature says nothing about whether the token was revoked.

Stateless verification is the reason to adopt JWTs and revocation is the reason they hurt, so keep access token lifetimes to minutes and put revocation in the refresh path. Pin the accepted algorithm server-side, since trusting the token's own alg header enabled both the alg=none bypass and RS256-to-HS256 key confusion. Assume the client reads every claim, and re-check authorization server-side on each request.

Source: RFC 7519: JSON Web Token

A TLS handshake in which both parties present and validate certificates, so the server authenticates the client and the client authenticates the server. Identity binds to the connection and its private key, which removes the bearer-token weakness where anyone holding a copied credential can use it.

Standard for service-to-service authentication on zero-trust networks, and what SPIFFE and service meshes automate away. The hard part is lifecycle: issuance, short rotation intervals, revocation, and clock skew, which is why hand-rolled mTLS across a few hundred services usually breaks at renewal time. Certificate expiry has caused multi-hour outages at major providers, so alert on remaining validity in days.

Source: RFC 8446: TLS 1.3

An authorization framework that lets a user grant a third-party application scoped access to their resources without sharing credentials, by having an authorization server issue access tokens. It covers delegated authorization and specifies nothing about how the user was authenticated or how identity reaches the application.

Use the authorization code flow with PKCE for every client type, including confidential ones, since the implicit flow is deprecated and leaks tokens through URLs and logs. Scopes are the security boundary and most implementations draw them too coarse, so a token meant for reading a profile can also write to it. RFC 9700 collects current practice, including exact redirect URI matching and sender-constrained tokens.

Source: RFC 6749: The OAuth 2.0 Authorization Framework

An identity layer on top of OAuth 2.0 that adds an ID token, a signed JWT carrying authenticated user claims, along with a standard UserInfo endpoint and a discovery document. It defines how an authentication result is conveyed, which OAuth 2.0 by itself leaves undefined.

Use OIDC when your application needs to know who the user is, treating the ID token as proof of authentication and reserving the access token for calling APIs. Validation means checking issuer, audience, expiry, nonce, and signature against the published JWKS, with key rotation handled. Passing an access token to your own backend as evidence of identity is the common mistake, since it may carry a different audience entirely.

Source: OpenID Connect Core 1.0

A practice where each use of a refresh token issues a replacement and invalidates the previous one, making theft detectable: any reuse of an invalidated token signals compromise and triggers revocation of the entire token family. It bounds the value of a leaked long-lived credential.

Required for public clients such as single-page apps and mobile apps, where a refresh token cannot be kept secret. Implementation details decide whether it helps: track the token family, treat reuse as a security event that revokes the family, and handle the race where a client crashes after the server issued a replacement. Without rotation, a leaked refresh token grants access for its full lifetime, often months.

Source: RFC 9700: OAuth 2.0 Security Best Current Practice