Skip to main content
Welcome to the Data Analysis module from the NVIDIA Generative AI LLMs Associate Certification. Question 1 A data scientist compares two LLM models on a text classification task:
  • Model A: accuracy = 87%, F1 = 0.83
  • Model B: accuracy = 85%, F1 = 0.86
The dataset is imbalanced, with one class much more frequent than the other. Which model should be selected?
  • Model A, because it has higher accuracy.
  • Model B, because it has a higher F1 score.
  • Both models perform equally well, or neither model is suitable for imbalanced datasets.
Answer: Model B — it has the higher F1 score.
The image presents a decision scenario comparing two LLM models for a text classification task, highlighting their accuracy and F1 scores. It concludes that Model B should be selected due to its higher F1 score, which is more reliable for imbalanced datasets.
Summary
  • Although Model A shows higher overall accuracy, Model B is the better choice for imbalanced datasets because its higher F1 score (0.86) indicates a healthier balance between precision and recall on the minority class.
  • Accuracy can be misleading when class distribution is skewed: a trivial classifier that always predicts the majority class can still yield high accuracy while failing to identify the minority class.
Metric definitions and quick formulas Why F1 is preferable for imbalanced classes
  • F1 combines precision and recall, so it penalizes models that achieve high accuracy by only predicting the majority class.
  • When the minority class is important (e.g., fraud detection, disease diagnosis), missing positives (low recall) or producing many false positives (low precision) are both costly — F1 captures both types of errors.
  • Use macro-F1 to treat classes equally, and weighted-F1 to account for class frequency while still exposing poor minority-class performance.
When to prefer which metric
  • Prefer F1 / macro-F1 / weighted-F1: imbalanced datasets, minority-class importance, or class-wise performance matters.
  • Use PR-AUC (precision-recall AUC) for rare positive classes — PR curves emphasize performance on the positive class.
  • Use ROC-AUC only when classes are reasonably balanced or when you want a threshold-agnostic measure and false positive/negative costs are similar. In extreme imbalance, ROC-AUC can be overly optimistic.
  • Accuracy is acceptable when classes are balanced and the costs of different error types are similar.
Practical evaluation checklist
  • Inspect the confusion matrix and per-class precision/recall — don’t rely solely on a single scalar like accuracy.
  • Report multiple metrics: precision, recall, F1, macro-F1, weighted-F1, and PR-AUC where applicable.
  • Consider business impact: define whether false positives or false negatives are more costly and tune thresholds accordingly.
  • Improve minority-class performance if needed:
    • Class weighting in the loss function.
    • Resampling: oversample minority class (SMOTE, ADASYN) or undersample majority class.
    • Use ensemble methods or specialized algorithms for imbalanced data.
    • Calibrate classification thresholds based on precision/recall trade-offs.
Practical example (intuition)
  • If 90% of the data is class A and 10% is class B, a model that always predicts class A gets 90% accuracy but zero recall on class B. F1 for class B will be 0 — clearly showing the model is useless for the minority class despite high accuracy.
Recommendations for this scenario
  • Choose Model B because its higher F1 (0.86) signals better balance between precision and recall for the minority class, which is the key concern in imbalanced datasets.
  • Report per-class metrics and include macro-F1 / weighted-F1 in any model comparison to make decisions transparent.
When classes are imbalanced, favor F1-based metrics and per-class evaluation (precision, recall, confusion matrix). Model B is the better choice here because its higher F1 indicates more balanced performance across classes despite slightly lower accuracy.
Links and references

Watch Video