Skip to main content
Question 12. When designing an experiment to test whether Model A is better than Model B for a specific task, which statistical approach provides the most rigorous evidence?
  • Calculating the average performance of each model on the test set?
  • Performing a paired statistical test on per-example performance differences?
  • Comparing the best performance each model achieves on any single example?
  • Measuring how quickly each model was trained?
Answer: Performing a paired statistical test on per-example performance differences. Why this is the best approach
  • It compares models on the same examples, controlling for example-level difficulty and increasing statistical power.
  • It preserves per-example variance instead of averaging it away, which helps detect consistent differences rather than extreme outliers.
  • It yields a formal probability (p-value) and can be paired with effect-size and confidence-interval estimates to quantify practical significance.
Recommended paired tests by outcome type Practical checklist for a rigorous comparison
  1. Define the evaluation metric at the example level (e.g., correct/incorrect, per-example score).
  2. Evaluate both models on the exact same test examples to enable paired comparisons.
  3. Compute per-example differences: diff_i = score_A(i) - score_B(i).
  4. Inspect the differences for distributional assumptions (histogram, Q-Q plot).
  5. Select the appropriate paired test (see table).
  6. Report: test statistic, p-value, effect size (e.g., Cohen’s d or median difference), and confidence interval.
  7. If you run multiple hypotheses (e.g., many tasks or metrics), correct for multiple comparisons (Bonferroni, Holm, or false discovery rate methods).
  8. Share code and the example-level results so others can reproduce the test.
Always report both statistical significance and effect size. A small p-value can be meaningless if the effect is tiny; conversely, a large effect with insufficient sample size can be inconclusive without a confidence interval.
The image is a multiple-choice question about the most rigorous statistical approach to compare Model A and Model B. The correct answer highlighted is using a paired statistical test on per-example performance differences.
Why the other options are insufficient
  • Simple averages: Ignore per-example variance and may mask consistent but small differences across many examples.
  • Best-case single-example performance: Extremely sensitive to outliers and not representative of overall behavior.
  • Training speed: Relevant for engineering trade-offs but unrelated to held-out evaluation performance on the target task.
Avoid unpaired comparisons or using only aggregate metrics for hypothesis testing. Always pair examples between models, check test assumptions, and guard against data leakage and multiple comparisons.
Links and references Table of related resources

Watch Video