> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Strategies for Addressing Class Imbalance > Overview of methods to address class imbalance in machine learning, covering resampling like oversampling and SMOTE, algorithmic weighting and cost-sensitive learning, evaluation metrics and best practices Class imbalance occurs when one class (the majority) greatly outnumbers another (the minority). This is common in domains such as fraud detection, rare disease diagnosis, and anomaly detection. Imbalanced classes can bias models toward the majority class, producing deceptively high overall accuracy while failing to detect the minority class—the class you often care most about. The image is an introduction to the concept of class imbalance in data, illustrating it with colored circles and highlighting three issues: bias to the majority class, poor detection of the minority class, and effects on rare events like fraud and disease. Why this matters: if a dataset has 99% negatives and 1% positives, a model that always predicts negative achieves 99% accuracy yet is useless for detecting positives. For imbalanced datasets, accuracy is not a reliable metric—use targeted metrics that reflect minority-class performance. The image lists metrics better than accuracy for evaluating models: Precision, Recall, F1 Score, and AUC-ROC. Key evaluation metrics for imbalanced problems * Precision = TP / (TP + FP) — of predicted positives, how many are correct. * Recall (Sensitivity) = TP / (TP + FN) — of actual positives, how many we detected. * F1 Score = 2 \* (Precision \* Recall) / (Precision + Recall) — harmonic mean of precision and recall; useful when both false positives and false negatives matter. * ROC AUC — area under the ROC curve (TPR vs FPR) across thresholds; good for overall separability. * PR AUC (Precision-Recall AUC) — often more informative than ROC AUC when the positive class is rare. When the minority class detection is critical, prioritize precision, recall, F1 and PR AUC over plain accuracy. Use ROC AUC to compare separability and PR AUC to evaluate performance on rare positives. Approaches to handle class imbalance fall into two categories: * Data-level strategies (resampling): alter the training data distribution. * Algorithm-level strategies: change the learning algorithm or loss to account for imbalance. Data-level strategies (resampling) The image outlines three data-level strategies: Random Oversampling, Random Undersampling, and SMOTE, each focused on balancing class distributions in datasets. * Random oversampling: duplicate minority-class examples to rebalance classes. Pros: simple, effective for small imbalances. Cons: increases overfitting risk (exact duplicates). * Random undersampling: remove majority-class examples to balance classes. Pros: faster training, less storage. Cons: may discard useful information. * SMOTE (Synthetic Minority Oversampling Technique): synthesize new minority examples by interpolating between a sample and its k nearest minority neighbors. Pros: generates diverse minority examples, reduces overfitting compared with naive duplication. Cons: can create borderline/noisy samples and is sensitive to k and feature scaling. Important: Always split into train/test (preferably stratified) before applying oversampling. Applying oversampling to the entire dataset before splitting causes data leakage and inflated performance. Never oversample (including SMOTE) before splitting your data. Apply resampling only to the training set to avoid information leakage into your validation/test sets. How SMOTE works SMOTE generates a synthetic sample by selecting a minority-class sample, choosing one of its k nearest minority neighbors, and interpolating a new point along the line segment between them. This produces plausible, non-duplicate minority samples and can help classifiers generalize better to minority regions. The image outlines the advantages and disadvantages of the Synthetic Minority Oversampling Technique (SMOTE), highlighting that it creates diverse examples and reduces overfitting risk, but may generate borderline samples. Advantages of SMOTE: * Produces new, diverse minority examples (reduces overfitting compared with duplication). * Can improve classifier sensitivity to minority examples. Drawbacks and cautions: * May generate noisy/borderline samples that cross class boundaries if classes overlap. * Sensitive to the number of neighbors (k) and to feature scaling—standardize features first. * For structured data (tabular), SMOTE works well; for images/text, use domain-specific augmentation. Example: apply SMOTE only on the training set (using imbalanced-learn) ```python theme={null} from imblearn.over_sampling import SMOTE from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler # X, y are your features and labels X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, stratify=y, random_state=42 ) scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test) smote = SMOTE(random_state=42, k_neighbors=5) X_resampled, y_resampled = smote.fit_resample(X_train, y_train) # Train a classifier on X_resampled, y_resampled ``` References: [imbalanced-learn SMOTE documentation](https://imbalanced-learn.org/stable/over_sampling.html#smote) Algorithm-level strategies Algorithm-level methods modify how the model learns so it treats minority errors as more important without altering the dataset. * Class weighting: increase the loss contribution of the minority class by assigning larger weights to its examples. Many libraries (e.g., scikit-learn) support a `class_weight` parameter. * Cost-sensitive learning: implement a custom loss that penalizes different types of errors differently; useful when you know relative costs of false positives vs false negatives. * Ensemble and sampling-aware models: balanced random forests, boosting with class weights, or bagging combined with resampling can improve minority detection by aggregating multiple models and controlling sampling or weighting. Libraries to explore: [XGBoost](https://xgboost.readthedocs.io/en/stable/), [LightGBM](https://lightgbm.readthedocs.io/en/latest/). The image outlines an algorithm-level strategy for handling data by assigning class weights, giving high weight to the minority class and low weight to the majority class. Example: class weights in scikit-learn ```python theme={null} from sklearn.linear_model import LogisticRegression from sklearn.metrics import classification_report # Option 1: let scikit-learn compute balanced weights automatically clf = LogisticRegression(class_weight='balanced', random_state=42, max_iter=1000) clf.fit(X_train, y_train) y_pred = clf.predict(X_test) print(classification_report(y_test, y_pred)) # Option 2: provide manual weights, e.g., {0: 1.0, 1: 10.0} clf = LogisticRegression(class_weight={0: 1.0, 1: 10.0}, random_state=42, max_iter=1000) ``` Choosing a strategy: quick guidance * If you have plenty of majority examples and training time matters: consider undersampling or ensemble methods with sampling. * If discarding majority data is risky: use oversampling (SMOTE or variants) or class weighting. * If misclassification costs are known or asymmetric: use cost-sensitive models or manual class weights. * For image/text data: prefer domain-specific augmentation over SMOTE. Summary table of strategies | Strategy type | When to use | Pros | Cons | | ------------------------------------------------- | --------------------------------------------------- | ---------------------------------------------------- | -------------------------------------------------- | | Random oversampling | Small to moderate imbalance, limited majority data | Simple to apply, increases minority representation | Can overfit due to duplicates | | Random undersampling | Very large datasets where training time matters | Reduces training time and storage | May remove useful information | | SMOTE & variants | Tabular data where synthetic examples are plausible | Generates diverse minority data, reduces duplication | Can create noisy/borderline samples; needs scaling | | Class weighting / cost-sensitive | When you can alter loss or costs | No change to dataset; works with many models | May need careful tuning of weights | | Ensemble methods (balanced RF, weighted boosting) | Complex problems requiring robustness | Combines models, can balance sampling & weighting | More complex to tune and compute | Validation and best practices * Always split first: train/validation/test splits must be done before any resampling. Use stratified splitting to preserve class ratios in evaluation folds. * Use appropriate metrics: precision, recall, F1, PR AUC, and ROC AUC. Select the metric that reflects business impact (e.g., recall for capturing fraud). * Use stratified cross-validation when tuning hyperparameters to maintain class balance across folds. * Monitor overfitting: when using oversampling, compare performance on validation/test sets to detect over-optimistic training results. * For thresholded classifiers, tune decision thresholds using validation metrics that matter (e.g., maximize F1 or a weighted cost function). Further reading and references * [imbalanced-learn documentation (SMOTE and resampling)](https://imbalanced-learn.org/stable/) * [scikit-learn model weighting and metrics](https://scikit-learn.org/stable/) * [XGBoost documentation](https://xgboost.readthedocs.io/en/stable/) * [LightGBM documentation](https://lightgbm.readthedocs.io/en/latest/) In short: diagnose imbalance, pick metrics that reflect your objective, split data before resampling, and combine data-level and algorithm-level techniques as needed. Test thoroughly using stratified validation and the metrics that matter to your problem to avoid leakage and to get reliable performance estimates.