Fault Tolerance

best-effort, reliable (eager/gossip), total order broadcast; consensus requirement; relationship to CRDTs and replication

Closed/open/half-open state machine for downstream resiliency

failure taxonomy: hardware, incorrect error handling (2014 study), configuration changes, SPOFs, gray failures, resource leaks, cascading/metastable failures; risk = probability × impact; airline/Black Friday/"Trampled" case studies; building-for-tests vs building-for-production; GC death spiral, queue management (LIFO/CoDel), deadline propagation, latency vs capacity cache

Raft, Paxos, Zab; FLP result; ZooKeeper/etcd; equivalence theorem (linearizable CAS = total order broadcast = locks = uniqueness)

Timeouts (fundamental limitation), pings vs heartbeats, when to use active detection; imperfect failure detection theorem

Raft state machine (follower/candidate/leader); election terms; CAS+lease practical approach; fencing tokens for mutual exclusion; leader as SPOF

State machine replication (Raft), chain replication (head/tail topology, failure modes, data/control plane split), leader-follower, Dynamo-style; replication lag anomalies; multi-leader conflict resolution

Transient failure recovery; exponential backoff with jitter; retry amplification in chains; idempotency prerequisite; retry queues

Link models (fair-loss/reliable/authenticated), process failure models (Byzantine/crash-recovery/crash-stop), timing models (sync/async/partial sync); default assumptions

Bounding wait time on every blocking call; sizing by P99.9; absent-timeout gotchas; relationship to circuit breaker and retry