Machine Learning

AI engineering as a discipline vs ML engineering; three-layer stack (application dev, model dev, infrastructure); product planning; last-mile challenge; use case taxonomy

*AI Engineering* — Chip Huyen

Stateless retraining vs stateful training (fine-tuning); four stages of continual learning maturity; champion/challenger pattern; test in production methods (shadow deployment, A/B testing, canary release, interleaving experiments, bandits, contextual bandits); data freshness value; continuous ML challenges (feedback loops, emergency response, stable A/B baselines)

Covariate shift/label shift/concept drift/feature change/label schema change taxonomy; degenerate feedback loops (mechanism, detection, correction via randomisation and positional features); detection methods (KS test, two-sample tests, feature validation, prediction monitoring, sliding vs cumulative statistics); proactive design; retraining strategies

*Designing Machine Learning Systems* — Chip Huyen

Feature operations (missing value handling MNAR/MAR/MCAR, scaling, discretisation, categorical encoding/hashing trick, feature crossing, positional embeddings); data leakage causes and detection; feature importance (top 10 = 50% importance); feature generalisation (coverage × distribution overlap); best practices (split by time, scale after split)

What foundation models are; transformer architecture (prefill/decode, KV cache, MoE); training data constraints; Chinchilla scaling law; post-training (SFT, RLHF, DPO)

Hypothetico-deductive troubleshooting; ICS roles; "only Ops modifies"; blameless postmortems; outage tracking; ML incident response (harder detection, broader scope, fuzzy timeline, RPO/RTO for ML)

When to use ML (nine conditions); research vs production differences (silent failure, latency vs throughput, messy data, fairness, interpretability); four system requirements (reliability, scalability, maintainability, adaptability); business vs ML objective alignment; problem framing (task types, decoupling objectives); mind vs data debate

Four-layer MLOps stack (storage/compute, resource management, ML platform, dev environment); cloud repatriation; Docker + Kubernetes; workflow orchestration tools (Airflow/Argo/Prefect/Metaflow); model store (8 artifact types); feature store (management/computation/consistency, eliminates training-serving skew); build vs buy framework

Six model selection tips; four phases of ML adoption; ensembles (bagging/boosting/stacking); experiment tracking and data versioning challenges; ML debugging; distributed training (data/model/pipeline parallelism); AutoML (hyperparameter tuning, NAS, learned optimisers); offline evaluation baselines; evaluation methods (perturbation, invariance, directional expectation, calibration, confidence, slice-based/Simpson's paradox)

Black-box vs white-box monitoring; metrics and pre-aggregation; SLIs/SLOs; burn rate alerting; three-layer ML monitoring taxonomy (golden signals/generic ML signals/domain-specific quality); four actuals cases; drift detection (PSI, KL divergence, Wasserstein); ML SLOs and privacy in monitoring

*Reliable Machine Learning* — Chen, Murphy, Parisa, Sculley, Underwood