Regularization Techniques in ML

Underfitting happens when a model is too simple to capture the underlying patterns in the data, producing poor performance on both training and test sets. Overfitting occurs when a model learns the training data — including noise and outliers — so well that it fails to generalize to unseen data. The goal of regularization is to find a balanced model that generalizes well: not too simple (underfitting) and not overly complex (overfitting).

The image illustrates three graphs showing examples of underfitting, balanced fitting, and overfitting in data modeling. Each graph depicts a different type of line fitting a set of white data points.

Regularization is a group of techniques that discourage overly complex models by adding a complexity penalty to the loss function. This penalty nudges model parameters toward smaller values or sparsity, improving generalization on unseen data.

The image illustrates the concept of regularization in machine learning, showing a model that overfits on a training dataset leading to high in-sample accuracy but low accuracy on new data, suggesting poor generalization.

How regularization modifies the loss function:

Base loss: measures prediction error (e.g., MSE, cross-entropy).
Regularized loss: base loss + penalty term that grows with model complexity.
Strength: controlled by a hyperparameter (commonly λ or alpha) that scales the penalty.

Common penalty functions and when to use them:

Regularization	Penalty (concept)	Effect	Use case
L1 (Lasso)	Sum of absolute weights	Encourages sparsity (drives some coefficients to zero)	Feature selection, high-dimensional data
L2 (Ridge)	Sum of squared weights	Shrinks weights smoothly (reduces variance)	When many small features contribute
Elastic Net	Combination of L1 and L2	Balances sparsity and stability	When correlated features exist

The image depicts three types of regularization in machine learning: L1 (Lasso), L2 (Ridge), and Elastic Net, explaining their penalty calculations.

Scikit-learn examples: Ridge (L2) and Lasso (L1)

from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# X, y = your dataset
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# L2 regularization (Ridge)
ridge = Ridge(alpha=1.0)  # alpha scales the L2 penalty (λ)
ridge.fit(X_train, y_train)
y_pred_ridge = ridge.predict(X_test)
print("Ridge MSE:", mean_squared_error(y_test, y_pred_ridge))

# L1 regularization (Lasso)
lasso = Lasso(alpha=0.1)  # alpha scales the L1 penalty
lasso.fit(X_train, y_train)
y_pred_lasso = lasso.predict(X_test)
print("Lasso MSE:", mean_squared_error(y_test, y_pred_lasso))

Regularization beyond linear models

Dropout (deep learning): randomly disables a subset of neurons during each training step, preventing co-adaptation and reducing overfitting.

The image illustrates the dropout technique in neural networks, showing network nodes before and after dropout is applied, with some nodes randomly deactivated to prevent co-adaptation.

Early stopping: track validation loss or metrics and stop training when improvement stalls to prevent overfitting from excessive epochs.
Data augmentation (images): artificially expand training data with random transforms — flips, rotations, crops, brightness/gamma changes — to improve model robustness.

Applying regularization in AWS SageMaker AWS SageMaker built-in algorithms and frameworks expose hyperparameters for common regularization techniques. Use these to control model capacity and generalization when training at scale.

Linear Learner supports both L1 and L2:
- L1: absolute weight penalty, useful for sparsity and feature selection.
- L2: squared weight penalty, shrinks coefficients to reduce variance.

The image is a slide titled "Applying Regularization in SageMaker" describing L1 regularization, which adds an absolute weight penalty, creates sparse models, and is beneficial for feature selection.

Example: Linear Learner via the SageMaker SDK

from sagemaker import LinearLearner

linear = LinearLearner(
    role='SageMakerRole',
    instance_count=1,
    instance_type='ml.m5.large',
    predictor_type='binary_classifier',
    l1=0.01,   # L1 regularization strength
    l2=0.1,    # L2 regularization strength
    epochs=20
)

XGBoost regularization parameters (summary)

Parameter	Meaning	Typical effect
`alpha`	L1 regularization term on weights	Encourages sparsity in leaf weights
`lambda`	L2 regularization term on weights	Shrinks leaf weights to reduce overfitting
`gamma`	Minimum loss reduction for a split	Controls how easily new splits are created
`max_depth`	Maximum tree depth	Direct control of model complexity

Example: XGBoost estimator in SageMaker

import sagemaker

region = sagemaker.Session().boto_region_name
role = "SageMakerRole"

xgboost = sagemaker.estimator.Estimator(
    image_uri=sagemaker.image_uris.retrieve("xgboost", region, version="1.3-1"),
    role=role,
    instance_count=1,
    instance_type='ml.m5.large',
    hyperparameters={
        "objective": "binary:logistic",
        "alpha": 0.5,   # L1 regularization
        "lambda": 1.0,  # L2 regularization (note: key is a string)
        "gamma": 0,
        "max_depth": 6
    }
)

Other regularization-related training knobs in SageMaker and deep learning:

weight_decay: commonly used for L2 regularization.
num_layers: increasing layers raises capacity and overfitting risk.
mini_batch_size: affects gradient estimates and training stability.
epochs: more epochs increase risk of overfitting (use early stopping).
learning_rate: impacts optimization dynamics; lower rates can act like a regularizer.

ResNet-based image classification example (SageMaker built-in image classifier)

weight_decay implements L2 regularization.
learning_rate controls the optimizer behavior (not a regularization parameter, but relevant to convergence).

image_classifier.set_hyperparameters(
    num_layers=18,
    use_pretrained_model=1,
    image_shape='3,224,224',
    num_classes=2,
    mini_batch_size=32,
    epochs=10,
    learning_rate=0.001,  # optimizer learning rate (not L2)
    weight_decay=0.0001,  # L2 regularization (weight decay)
    top_k=2
)

Tuning regularization strengths

Use automated hyperparameter tuning (AWS SageMaker Automatic Model Tuning or other HPO frameworks) to search for optimal values of alpha, lambda, weight_decay, dropout rate, etc.
Start with reasonable defaults and sweep logarithmically (e.g., 1e-4 to 1e0) for penalty strengths.

When supplying XGBoost hyperparameters in a Python dict, use string keys for parameters like "lambda" because lambda is a reserved keyword in Python. For example: {"lambda": 1.0}.

Watch Video

Methods to Reduce Model Training Time

Hyperparameter Tuning Techniques

Prerequisites

Data Preparation for Machine Learning (ML)

ML Model Development

Deployment and Orchestration of ML Workflows

ML Solution Monitoring, Maintenance, and Security

Bringing it all together

Regularization Techniques in ML

Watch Video