ML Handbook

Theme

Model Evaluation and Validation

Quick overview Model evaluation is the foundation of machine learning — bias-variance tradeoff, train/validation/test splits, cross-validation, confusion matrix, precision/recall/F1/AUC, handling class imbalance. This article provides a complete framework from metric selection to validation methodology.

Model Evaluation and Validation ​

Concept Definition: Without Evaluation, There's No Machine Learning ​

The entire value of machine learning lies in generalization — performing well on unseen data. And "how well" must be answered by evaluation. Evaluation answers three questions:

  1. How good is the model exactly? (metrics)
  2. How credible is this conclusion? (validation methods)
  3. Where should we improve? (error analysis)

ML projects without an evaluation system either have false confidence (99% on the training set) or fail in production (a mess on real data). Evaluation design is one of the most important engineering decisions in an ML project — it determines whether your iteration direction is correct.

The Bias-Variance Tradeoff: The Theoretical Foundation of Evaluation ​

Decompose the model's prediction error on unseen data into three parts:

E[(y - ĥ(x))²] = Bias²(ĥ) + Variance(ĥ) + σ² (irreducible noise)
  • Bias: the gap between the model's assumptions and the true patterns — linear model fitting nonlinear data = high bias (underfitting);
  • Variance: the model's sensitivity to fluctuations in the training set — deep network on small samples = high variance (overfitting);
  • Irreducible noise: the randomness inherent in the data that no model can eliminate.

The more complex the model, the lower the bias and the higher the variance. Tuning is fundamentally about finding the balance between the two. This framework explains almost all the "mysteries" in ML:

PhenomenonDiagnosisCountermeasure
High training error, high validation errorHigh bias (underfitting)Add features, increase model complexity, reduce regularization
Low training error, high validation errorHigh variance (overfitting)Add data, add regularization, simplify model, early stopping
Both errors highData/feature problemGo back to the data stage and check

Diagnostic rule of thumb

Underfitting: check the training set (if the model can't learn it), overfitting: check the validation set (the model learns it but can't generalize). Judge which type it is first, then apply the right remedy — random parameter tuning is the least efficient iteration method.

Data Splits: Train / Validation / Test ​

Responsibilities of the Three Datasets ​

DatasetPurposeDon'ts
Training setTrain model parametersCan't be used for hyperparameter selection (will overfit to the training set)
Validation setSelect hyperparameters, early stopping, model selectionCan't be used to report final scores
Test setFinal evaluation, report scoresCan't be touched during any iteration

The test set is "the exam paper": you've seen the answers in the training set, you've seen practice questions in the validation set, but only the test set is truly new material you've never encountered. Using test set data for any hyperparameter/feature selection decision is "looking at the answer ahead of time" — i.e., data leakage.

Pitfalls of Splitting ​

  • Random split: applies to i.i.d. data;
  • Time series must be split by time: use the first 80% for training and the last 20% for testing; random splitting lets the model "peek into the future";
  • Grouped data: samples from the same user/patient must go into the same split (GroupKFold), otherwise data leakage.

Cross-Validation: Squeezing Reliable Conclusions from Small Data ​

The holdout method (single split) is highly variable on small data — a good split inflates scores artificially. K-fold cross-validation: split the data into K folds, take 1 fold as validation and the remaining K-1 as training, loop through, and report the mean and variance of the K results. K=5 or 10 is the default choice.

python
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

scores = cross_val_score(model, X, y, cv=5, scoring='roc_auc')
print(f"AUC: {scores.mean():.3f} ± {scores.std():.3f}")

Variants: StratifiedKFold (equal class proportion in each fold, mandatory for classification), GroupKFold (split by group), TimeSeriesSplit (rolling validation for time series). The cost of cross-validation is training K times; on large datasets, a single holdout + large data volume is often used instead.

Cross-validation ≠ a silver bullet

Cross-validation assumes i.i.d. samples. Using regular K-fold on time series or user-correlated data gives an optimistic bias. Before choosing a validation method, think carefully about how the data was generated.

Metrics: Classification ​

Confusion Matrix: the Starting Point of All Classification Metrics ​

              Predicted positive   Predicted negative
Actual positive   TP (true positive)   FN (false negative)
Actual negative   FP (false positive)  TN (true negative)

Four cells that derive all common metrics:

MetricFormulaQuestion Answered
Accuracy(TP+TN)/totalOverall proportion predicted correctly (usable when classes are balanced)
PrecisionTP/(TP+FP)Of those predicted positive, how many are truly positive?
RecallTP/(TP+FN)Of all actual positives, how many were retrieved?
F12·P·R/(P+R)Harmonic mean of precision and recall
SpecificityTN/(TN+FP)Proportion of negatives correctly rejected
AUCArea under ROC curveProbability that a random positive scores higher than a random negative

Precision vs. Recall: the Business Perspective ​

Precision penalizes false positives; recall penalizes false negatives:

  • Spam filtering: better to miss a few spam emails than to flag important mail as spam → prioritize precision;
  • Cancer screening: better to check a few extra people than to miss a patient → prioritize recall.

Precision and recall are inherently in tension (threshold tuning can trade them), and which one to prioritize depends on "which type of error is more costly." F1 is the compromise when both errors are equally weighted; when business costs are asymmetric, directly optimize the threshold by cost using weighted Fβ.

Why Accuracy Can Be a Trap ​

When classes are imbalanced (fraud at 1:1000), a model that "predicts all negative" achieves 99.9% accuracy but is useless. In imbalanced scenarios, look at Precision/Recall/AUC/PR-AUC, not Accuracy. More class imbalance countermeasures are in Supervised Learning.

Understanding AUC Correctly ​

AUC = the probability that, given a random positive and a random negative sample, the model gives the positive a higher score. Properties: threshold-independent (pure ranking ability), relatively insensitive to class imbalance. But AUC obscures probability calibration issues (model scores of 0.9 might correspond to only 50% actual positives, yet AUC can still be high) — business scenarios needing probabilities (risk pricing) must also check calibration curves.

Metrics: Regression and Ranking ​

Regression Metrics ​

MetricFormulaCharacteristics
MSEmean((y-ŷ)²)Squared penalty, sensitive to outliers, well-differentiable
MAEmean(|y-ŷ|)Linear penalty, robust to outliers
R²1 - SS_res/SS_tot"How much variance the model explains," compared to the baseline (mean prediction)
MAPEmean(|y-ŷ|/y)Percentage error, explodes when y is near 0

R² is not accuracy; it can be negative (the model is worse than "guessing the mean"). Choosing regression metrics also comes back to the business: a 50k vs. 100k error in house price prediction is more intuitive with MAE; MSE is smoother for gradient optimization.

Ranking Metrics ​

  • NDCG: relevance-discounted cumulative gain weighted by position, the standard for search ranking;
  • MAP / MRR: mean reciprocal rank of the position where the correct result appears;
  • Pairwise loss: directly optimize "relative order" during training rather than absolute scores.

The difference between ranking and classification metrics: ranking only cares about relative order, not absolute scores. Optimizing ranking metrics directly (LTR) in recommendation/search produces better results than classifying first and then ranking.

The Full Evaluation Workflow ​

A qualified evaluation process looks like this:

text
1. Define business goal → 2. Choose metrics (offline + online) → 3. Design data splits (prevent leakage)
4. Establish baselines (majority class/linear) → 5. Cross-validate evaluation → 6. Test set final review
7. Error analysis (grouped by error type) → 8. Iterate (based on analysis, not random)

Steps ① and ⑦, the easiest to skip, are actually the most important. Evaluation isn't "calculating a score at the end"; it's a design that starts from the problem definition — the metrics you use determine the direction in which the model optimizes.

The power of error analysis

Group the validation set's error samples by type (which categories are frequently wrong, which missing features cause errors, which intervals have the most errors), and you'll find that improvement directions are often not in model hyperparameters but in data, features, and thresholds. One serious error analysis is worth ten blind tuning sessions.

Tradeoffs ​

  • Number of metrics: looking at only one metric makes it easy to "game the score," but looking at too many makes decision-making hard. Recommend one primary metric (aligned with business) + two or three secondary metrics (monitoring degradation).
  • Offline vs. online: high offline AUC doesn't mean high online conversion. The mature approach is offline evaluation + online A/B testing closed loop, see MLOps and Model Deployment.
  • Validation rigor vs. cost: larger K in cross-validation is more accurate but more expensive; on large datasets (millions of samples), K=5 or holdout is usually sufficient.
  • Evaluation bias vs. variance: single holdout is variable, K-fold is more stable; but K-fold has systematic bias on non-independent data — fix data independence first, then talk about validation rigor.

Further Reading ​

References ​