Resiliency

Uptime/downtime definition; the nines table (90%–99.999%); techniques for high availability; dependency chaining effects; design-for-production philosophy; ROI of availability investment; MTBF/MTTR/RPO/RTO measurement framework; tyranny of the nines antipattern; Allspaw: MTTR > MTBF; SRE: 100% is always the wrong target

Resource partitioning, shuffle sharding, cellular architecture

definition, prerequisites, experiment design, injection types (Chaos Monkey/Latency Monkey/FIT), targeting strategy, automation and moderation, disaster simulations; theoretical foundations (drift into failure, regulator paradox, antifragility); complex systems foundations (accidental vs essential complexity, emergent failures, composability of safety); Netflix origin story; Chaos Maturity Model

*Chaos Engineering: System Resiliency in Practice* — Rosenthal & Jones (eds.); complex systems theory, Five Advanced Principles, Game Days, LDFI, Chaos Maturity Model, Continuous Verification, Security Chaos Engineering, organisational/human factors

Closed/open/half-open state machine for downstream resiliency

failure taxonomy: hardware, incorrect error handling (2014 study), configuration changes, SPOFs, gray failures, resource leaks, cascading/metastable failures; risk = probability × impact; airline/Black Friday/"Trampled" case studies; building-for-tests vs building-for-production; GC death spiral, queue management (LIFO/CoDel), deadline propagation, latency vs capacity cache

Timeouts (fundamental limitation), pings vs heartbeats, when to use active detection; imperfect failure detection theorem

Load shedding (503, priority/age ordering), load leveling (async channel + auto-scaling), rate limiting (sliding window buckets, distributed atomic increment, fail-open), constant work pattern (periodic full-state dump, antifragile, self-healing)

*Release It! Design and Deploy Production-Ready Software* — Michael Nygard

Transient failure recovery; exponential backoff with jitter; retry amplification in chains; idempotency prerequisite; retry queues

East–west traffic governance: mTLS, service authorisation, evolution from libraries to eBPF; sidecar as reuse mechanism for operational concerns; orthogonal coupling; data-sinking sidecar for EDM legacy integration

When to use synchronous vs asynchronous communication; temporal coupling as the core distinction; availability chaining risk; async concerns (idempotency, ordering, correlation, observability); mixed model (sync north-south, async east-west); decision guide

Bounding wait time on every blocking call; sizing by P99.9; absent-timeout gotchas; relationship to circuit breaker and retry

*Understanding Distributed Systems* — Roberto Vitillo