> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Regularization Techniques in ML
> Overview of regularization methods to reduce overfitting and improve generalization in machine learning, covering L1 L2 Elastic Net dropout early stopping and SageMaker usage.
Underfitting happens when a model is too simple to capture the underlying patterns in the data, producing poor performance on both training and test sets. Overfitting occurs when a model learns the training data — including noise and outliers — so well that it fails to generalize to unseen data. The goal of regularization is to find a balanced model that generalizes well: not too simple (underfitting) and not overly complex (overfitting).
Regularization is a group of techniques that discourage overly complex models by adding a complexity penalty to the loss function. This penalty nudges model parameters toward smaller values or sparsity, improving generalization on unseen data.
How regularization modifies the loss function:
* Base loss: measures prediction error (e.g., MSE, cross-entropy).
* Regularized loss: base loss + penalty term that grows with model complexity.
* Strength: controlled by a hyperparameter (commonly λ or `alpha`) that scales the penalty.
Common penalty functions and when to use them:
| Regularization | Penalty (concept) | Effect | Use case |
| -------------: | ------------------------ | ------------------------------------------------------ | ---------------------------------------- |
| L1 (Lasso) | Sum of absolute weights | Encourages sparsity (drives some coefficients to zero) | Feature selection, high-dimensional data |
| L2 (Ridge) | Sum of squared weights | Shrinks weights smoothly (reduces variance) | When many small features contribute |
| Elastic Net | Combination of L1 and L2 | Balances sparsity and stability | When correlated features exist |
Scikit-learn examples: Ridge (L2) and Lasso (L1)
```python theme={null}
from sklearn.linear_model import Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
# X, y = your dataset
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
# L2 regularization (Ridge)
ridge = Ridge(alpha=1.0) # alpha scales the L2 penalty (λ)
ridge.fit(X_train, y_train)
y_pred_ridge = ridge.predict(X_test)
print("Ridge MSE:", mean_squared_error(y_test, y_pred_ridge))
# L1 regularization (Lasso)
lasso = Lasso(alpha=0.1) # alpha scales the L1 penalty
lasso.fit(X_train, y_train)
y_pred_lasso = lasso.predict(X_test)
print("Lasso MSE:", mean_squared_error(y_test, y_pred_lasso))
```
Regularization beyond linear models
* Dropout (deep learning): randomly disables a subset of neurons during each training step, preventing co-adaptation and reducing overfitting.
* Early stopping: track validation loss or metrics and stop training when improvement stalls to prevent overfitting from excessive epochs.
* Data augmentation (images): artificially expand training data with random transforms — flips, rotations, crops, brightness/gamma changes — to improve model robustness.
Applying regularization in AWS SageMaker
AWS SageMaker built-in algorithms and frameworks expose hyperparameters for common regularization techniques. Use these to control model capacity and generalization when training at scale.
* Linear Learner supports both L1 and L2:
* L1: absolute weight penalty, useful for sparsity and feature selection.
* L2: squared weight penalty, shrinks coefficients to reduce variance.
Example: Linear Learner via the SageMaker SDK
```python theme={null}
from sagemaker import LinearLearner
linear = LinearLearner(
role='SageMakerRole',
instance_count=1,
instance_type='ml.m5.large',
predictor_type='binary_classifier',
l1=0.01, # L1 regularization strength
l2=0.1, # L2 regularization strength
epochs=20
)
```
XGBoost regularization parameters (summary)
| Parameter | Meaning | Typical effect |
| ----------: | ---------------------------------- | ------------------------------------------ |
| `alpha` | L1 regularization term on weights | Encourages sparsity in leaf weights |
| `lambda` | L2 regularization term on weights | Shrinks leaf weights to reduce overfitting |
| `gamma` | Minimum loss reduction for a split | Controls how easily new splits are created |
| `max_depth` | Maximum tree depth | Direct control of model complexity |
Example: XGBoost estimator in SageMaker
```python theme={null}
import sagemaker
region = sagemaker.Session().boto_region_name
role = "SageMakerRole"
xgboost = sagemaker.estimator.Estimator(
image_uri=sagemaker.image_uris.retrieve("xgboost", region, version="1.3-1"),
role=role,
instance_count=1,
instance_type='ml.m5.large',
hyperparameters={
"objective": "binary:logistic",
"alpha": 0.5, # L1 regularization
"lambda": 1.0, # L2 regularization (note: key is a string)
"gamma": 0,
"max_depth": 6
}
)
```
Other regularization-related training knobs in SageMaker and deep learning:
* `weight_decay`: commonly used for L2 regularization.
* `num_layers`: increasing layers raises capacity and overfitting risk.
* `mini_batch_size`: affects gradient estimates and training stability.
* `epochs`: more epochs increase risk of overfitting (use early stopping).
* `learning_rate`: impacts optimization dynamics; lower rates can act like a regularizer.
ResNet-based image classification example (SageMaker built-in image classifier)
* `weight_decay` implements L2 regularization.
* `learning_rate` controls the optimizer behavior (not a regularization parameter, but relevant to convergence).
```python theme={null}
image_classifier.set_hyperparameters(
num_layers=18,
use_pretrained_model=1,
image_shape='3,224,224',
num_classes=2,
mini_batch_size=32,
epochs=10,
learning_rate=0.001, # optimizer learning rate (not L2)
weight_decay=0.0001, # L2 regularization (weight decay)
top_k=2
)
```
Tuning regularization strengths
* Use automated hyperparameter tuning (AWS SageMaker Automatic Model Tuning or other HPO frameworks) to search for optimal values of `alpha`, `lambda`, `weight_decay`, dropout rate, etc.
* Start with reasonable defaults and sweep logarithmically (e.g., `1e-4` to `1e0`) for penalty strengths.
When supplying XGBoost hyperparameters in a Python dict, use string keys for parameters like `"lambda"` because `lambda` is a reserved keyword in Python. For example: `{"lambda": 1.0}`.
Further reading and references
* [Scikit-learn documentation](https://scikit-learn.org/stable/)
* [XGBoost documentation](https://xgboost.readthedocs.io/en/latest/)
* [ResNet paper (He et al.)](https://arxiv.org/abs/1512.03385)
* AWS SageMaker resources and tutorials: [SageMaker documentation](https://aws.amazon.com/sagemaker/)
Use regularization as part of a broader model-validation strategy: combine cross-validation, monitoring of validation metrics, early stopping, and automated tuning to achieve robust models that generalize well.