Skip to main content
Evaluation is a critical phase in the machine learning lifecycle that measures how well a model performs on a task and guides decisions about deployment, tuning, or redesign. Effective evaluation compares model predictions to ground truth, quantifies different error types, and helps select the best candidate among multiple models.
The image is a flowchart illustrating the process of inputting data into a model to produce predictions, with the question "Why Evaluate ML Models?" It emphasizes that evaluation helps determine a model's performance on a task.
In practice, we often run the same dataset through several candidate models (for example, Model A and Model B) to produce predictions. Raw predictions are not enough: you must evaluate them against ground truth and compare models using appropriate metrics that reflect your business goals and risk tolerance.
The image is a flowchart titled "Why Evaluate ML Models?" showing input data being processed by Model A and Model B, both producing predictions that are then compared for accuracy.
This comparison reveals which model generalizes better, whether further training or feature engineering is needed, and which candidate is safest for production.

​
Confusion Matrix (Classification Diagnostics)

A confusion matrix decomposes binary classification outcomes into four categories:
  • True Positive (TP): predicted positive, actual positive
  • False Negative (FN): predicted negative, actual positive
  • False Positive (FP): predicted positive, actual negative
  • True Negative (TN): predicted negative, actual negative
The confusion matrix not only gives overall accuracy but helps identify specific mistakes (e.g., many false positives vs. many false negatives), which is essential when different errors carry different business costs.
The image displays a confusion matrix with categories for true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN), highlighting the correct prediction of the positive class.
Key classification metrics derived from the confusion matrix:
  • Choose metrics that reflect business priorities: reducing false negatives vs. false positives has different operational consequences.
  • For highly imbalanced datasets, avoid relying solely on accuracy.
The image shows three performance metrics—Accuracy, Precision, and Recall (Sensitivity)—along with their respective formulas involving true positives, true negatives, false positives, and false negatives.

​
Receiver Operating Characteristic (ROC) and AUC

An ROC curve evaluates a classifier’s ability to separate positive and negative classes across all decision thresholds. It plots:
  • y-axis: True Positive Rate (TPR or Recall)
  • x-axis: False Positive Rate (FPR)
Formulas:
The Area Under the ROC Curve (AUC-ROC) is a single-number summary: closer to 1.0 is better; 0.5 indicates random performance. Use ROC-AUC to compare classifiers when class distribution remains similar between training and production.
The image displays a Receiver Operating Characteristic (ROC) curve, highlighting an Area Under the Curve (AUC) of 0.90, indicating model performance between true positive and false positive rates.
Tip: When positive class is rare, consider Precision-Recall curves (AUC-PR) as they can be more informative than ROC.

​
Learning Curves (Bias–Variance Diagnosis)

Learning curves plot model performance on training and validation (or cross-validation) sets as a function of training set size. Typical patterns:
  • High training score and much lower validation score => overfitting (high variance)
  • Both low scores => underfitting (high bias)
  • Validation score improves and plateaus as more data is added => model capacity may be adequate
Use learning curves to decide whether adding more data, regularization, or model complexity changes are needed.
The image shows a learning curve graph comparing training accuracy and cross-validation accuracy against the training set size. Each line has a shaded area representing variance or confidence intervals.

​
Regression Metrics (Continuous Targets)

Common metrics for regression tasks include:
  • MAE: average absolute deviations — robust to outliers compared with MSE.
  • MSE: penalizes larger errors more — useful as an optimization objective.
  • RMSE: same units as the target — easier to interpret.
  • R^2: fraction of variance explained; values near 1 are good, 0 indicates performance equivalent to predicting the mean, and negative values indicate worse than the mean predictor.
The image explains the concept of Mean Squared Error (MSE), including its formula and characteristics, such as penalizing larger errors more and being sensitive to outliers.

​
Validation Strategies (Holdout vs. Cross-Validation)

Holdout Validation:
  • Split the dataset into distinct subsets:
    • Training set: ~60–80% — used to train the model.
    • Test set: ~20–40% — used to evaluate final performance.
    • Optional validation set: held out from the training split for hyperparameter tuning.
  • Pros: simple and fast; works well on large datasets.
  • Cons: higher variance in estimates if dataset is small.
The image illustrates a flowchart of Holdout Validation, showing input data split into a training set (60-80%) to train a model, and a testing set (20-40%) to evaluate the model.
K-Fold Cross-Validation:
  • Split data into k folds (commonly k = 5 or k = 10).
  • For each fold: train on k-1 folds and validate on the remaining fold.
  • Average metrics across folds for a more robust estimate.
  • Reduces evaluation variance and is especially valuable with limited data.
When to choose:
  • Holdout: large datasets, need quick iteration.
  • Cross-validation: limited data or when you need stable, lower-variance estimates.
The image is a comparison table between Holdout Validation and Cross-Validation, listing differences in simplicity, bias/variance, and data efficiency characteristics.

​
Practical Tooling and Monitoring (AWS)

AWS offers services to help evaluate, monitor, and detect issues in models: Use monitoring to detect model drift, data-schema changes, and production performance regressions.
Choose evaluation metrics and validation strategies that match your business goals and the risks of each type of error. For imbalanced problems, prefer precision/recall or AUC-PR and consider cross-validation to get stable estimates.
References and Further Reading

Watch Video