Operations

Automation vs autonomy; 5-level hierarchy (manual → autonomous); safety properties (rate limiting, idempotency); MySQL on Borg case study

Uptime/downtime definition; the nines table (90%–99.999%); techniques for high availability; dependency chaining effects; design-for-production philosophy; ROI of availability investment; MTBF/MTTR/RPO/RTO measurement framework; tyranny of the nines antipattern; Allspaw: MTTR > MTBF; SRE: 100% is always the wrong target

definition, prerequisites, experiment design, injection types (Chaos Monkey/Latency Monkey/FIT), targeting strategy, automation and moderation, disaster simulations; theoretical foundations (drift into failure, regulator paradox, antifragility); complex systems foundations (accidental vs essential complexity, emergent failures, composability of safety); Netflix origin story; Chaos Maturity Model

failure taxonomy: hardware, incorrect error handling (2014 study), configuration changes, SPOFs, gray failures, resource leaks, cascading/metastable failures; risk = probability × impact; airline/Black Friday/"Trampled" case studies; building-for-tests vs building-for-production; GC death spiral, queue management (LIFO/CoDel), deadline propagation, latency vs capacity cache

Data integrity vs availability distinction; replication ≠ recoverability; failure mode matrix (cause × scope × rate); defence in depth: soft deletion / tiered backups / out-of-band validation; point-in-time recovery; continuously test restore; Gmail and Google Music case studies

error budget = 1 − SLO target; resolves dev/ops conflict by aligning incentives; budget exhaustion triggers release freeze; burn rate alerting; 100% is wrong target argument

Hypothetico-deductive troubleshooting; ICS roles; "only Ops modifies"; blameless postmortems; outage tracking; ML incident response (harder detection, broader scope, fuzzy timeline, RPO/RTO for ML)

Dynamic configuration (config store, runtime re-read); feature flags (progressive rollout, A/B testing, kill switch, deployment-release decoupling); operational triad: monitor → observe → manage

Black-box vs white-box monitoring; metrics and pre-aggregation; SLIs/SLOs; burn rate alerting; three-layer ML monitoring taxonomy (golden signals/generic ML signals/domain-specific quality); four actuals cases; drift detection (PSI, KL divergence, Wasserstein); ML SLOs and privacy in monitoring

Observability as superset of monitoring; three telemetry sources (metrics, logs, traces); structured event logs (one event per work unit, request ID, sampling); distributed tracing (trace ID propagation, spans, Zipkin/X-Ray); metrics and traces as derived views of event logs; Newman: log aggregation first, correlation IDs, Jaeger, synthetic transactions

SRE as discipline: dev/ops conflict; error budgets; toil cap (50%); SLO-driven alerting; blameless postmortems; SRE vs DevOps distinction; applicability outside Google

Zero-MTTR via pre-production testing; traditional test hierarchy; production tests (config tests, stress tests, canary); production probes; barrier defence pattern; configuration file risk management