Evaluation

AI engineering as a discipline vs ML engineering; three-layer stack (application dev, model dev, infrastructure); product planning; last-mile challenge; use case taxonomy

Evaluation methodology: perplexity, pass@k, BLEU/ROUGE, embeddings, AI-as-a-judge, comparative evaluation (Elo/Bradley-Terry, LMSYS Arena); factual consistency, safety, instruction-following; model selection workflow; API vs self-hosting; evaluation pipeline design; benchmark contamination

Six model selection tips; four phases of ML adoption; ensembles (bagging/boosting/stacking); experiment tracking and data versioning challenges; ML debugging; distributed training (data/model/pipeline parallelism); AutoML (hyperparameter tuning, NAS, learned optimisers); offline evaluation baselines; evaluation methods (perturbation, invariance, directional expectation, calibration, confidence, slice-based/Simpson's paradox)