Monitoring — Software Architecture

AI Engineering Architecture concept

Five-step progressive architecture (context enhancement, guardrails, router/gateway, caching, agents); model drift detection; orchestration frameworks; user feedback systems (explicit vs implicit, edit-as-preference, degenerate feedback loops, sycophancy)

Data Distribution Shifts concept

Covariate shift/label shift/concept drift/feature change/label schema change taxonomy; degenerate feedback loops (mechanism, detection, correction via randomisation and positional features); detection methods (KS test, two-sample tests, feature validation, prediction monitoring, sliding vs cumulative statistics); proactive design; retraining strategies

Designing Machine Learning Systems source

*Designing Machine Learning Systems* — Chip Huyen

Failure Detection concept

Timeouts (fundamental limitation), pings vs heartbeats, when to use active detection; imperfect failure detection theorem

Monitoring concept

Black-box vs white-box monitoring; metrics and pre-aggregation; SLIs/SLOs; burn rate alerting; three-layer ML monitoring taxonomy (golden signals/generic ML signals/domain-specific quality); four actuals cases; drift detection (PSI, KL divergence, Wasserstein); ML SLOs and privacy in monitoring

Reliable Machine Learning source

*Reliable Machine Learning* — Chen, Murphy, Parisa, Sculley, Underwood

Site Reliability Engineering source

*Site Reliability Engineering* — Beyer, Jones, Petoff, Murphy