CAP Theorem

“Pick two: consistency, availability, partition tolerance.” That’s how most people learn the CAP theorem, and it’s basically wrong. Or at least, it’s so oversimplified that it leads you to the wrong conclusions.

The CAP theorem, proved by Seth Gilbert and Nancy Lynch in 2002 (building on Eric Brewer’s 2000 conjecture), says something more specific: in the presence of a network partition, a distributed system must choose between consistency and availability. You can’t have both at the same time, during a partition.

That “during a partition” qualifier changes everything. When the network is healthy, you can have all three. CAP only forces a choice when things go wrong. And most of the time, things aren’t going wrong.

What the terms actually mean

This is where people get tripped up, because CAP uses familiar words with very specific definitions.

Consistency in CAP means linearizability - the strongest form of consistency. Every read returns the most recent write, across all nodes, as if there were a single copy of the data. This is much stricter than what most people mean when they say “consistent.” It’s not eventual consistency, not read-your-writes, not causal consistency. It’s the whole thing.

Availability means every request to a non-failing node gets a response. Not “the system is up most of the time” - every single request, to any node that hasn’t crashed, must return a result. No timeouts, no errors.

Partition tolerance means the system keeps functioning when network messages between nodes are lost or delayed. In other words, the system handles the network being unreliable.

But you don’t really “pick” partition tolerance. Networks partition. It happens. Switches fail, cables get cut, cloud regions lose connectivity. If your system runs on more than one machine, partitions will occur. So partition tolerance isn’t a feature you choose - it’s a reality you accept.

That means the real choice is: when a partition happens, do you sacrifice consistency (keep serving requests that might be stale) or availability (refuse requests until the partition heals)?

CP and AP in practice

CP systems choose consistency over availability during partitions. When a node can’t confirm it has the latest data, it refuses to serve reads rather than risk returning something stale. HBase, Zookeeper, and etcd work this way. For a coordination service, this is the right call - a distributed lock that lies about who holds it is worse than a lock that’s temporarily unavailable.

AP systems choose availability over consistency during partitions. Every node keeps responding, even if it might be out of sync. Cassandra, DynamoDB, and DNS work this way. For a shopping cart or a social media feed, showing slightly stale data is way better than showing nothing at all.

But most real systems don’t make a single global choice. They make different choices for different operations. DynamoDB lets you choose per-read whether you want strong or eventual consistency. Cassandra lets you tune consistency levels per query. You might use strong consistency for a bank balance and eventual consistency for a user’s avatar, in the same database. The “pick two” framing makes it sound like a system-wide commitment. In practice, it’s a dial you can turn per-operation.

What CAP doesn’t tell you

CAP is a theorem about the theoretical impossibility of having everything. It doesn’t tell you much about what to actually build.

Daniel Abadi pointed this out with his PACELC model: even when there’s no partition (the normal case), there’s a tradeoff between latency and consistency. A system that replicates data across three data centers can be strongly consistent, but every write has to wait for acknowledgment from a majority. That’s physics - the speed of light between Virginia and Oregon is about 80ms round trip. PACELC says: during a Partition, choose Availability or Consistency; Else, choose Latency or Consistency.

That “else” clause is what you’re actually dealing with 99.9% of the time. Partitions are rare. Latency is constant.

Martin Kleppmann has argued that labeling systems as “CP” or “AP” is misleading, and I think he’s right. Real systems exist on a spectrum. They make nuanced tradeoffs that a binary label can’t capture. MongoDB is “CP” in some configurations and “AP” in others. Even calling something “consistent” requires asking: consistent with respect to what? Linearizable? Sequential? Causal? Read-your-writes?

Why it still matters

Despite its limitations, CAP taught me something I keep coming back to: you can’t avoid making tradeoffs in distributed systems. There’s no free lunch. If someone tells you their distributed database is consistent, available, and partition-tolerant, they’re either redefining one of those terms or they haven’t tested it during a real partition.

The useful question isn’t “is this system CP or AP?” - it’s the same one that comes up in ordering and synchronization: what does this system need to get right, and what can it afford to get wrong? CAP tells you that you can’t have everything. The engineering is in figuring out what you actually need.