Skip to main content
In machine learning, hyperparameters are configuration settings set before training begins. They differ from model parameters (weights, splits, etc.), which are learned from the data during training. Hyperparameters are chosen by engineers or by hyperparameter-optimization systems and directly affect model accuracy, training speed, and generalization.
A slide titled "What are Hyperparameters?" that compares two boxes: Model Parameters (learned automatically from data during training, e.g., split thresholds) versus Hyperparameters (configured before training by you or an optimizer, e.g., n_estimators, max_depth).
Small changes to hyperparameters can produce large differences in behavior. Systematic tuning — using a validation set, cross-validation, or automated search — is essential to build reliable, high-performing models.
Hyperparameter tuning balances model accuracy, training cost, and generalization. Start with reasonable defaults, run validation experiments, and iterate while monitoring compute and latency constraints.

​
Why hyperparameter tuning matters

  • Improves predictive performance and stability.
  • Controls underfitting vs. overfitting.
  • Directly impacts training time and resource usage.
  • Enables reproducibility by making configuration explicit.

​
Common tuning methods

  • Grid search — exhaustive search over a specified parameter grid.
  • Random search — randomized sampling over parameter distributions (often more efficient than grid for high-dimensional spaces).
  • Bayesian optimization — models the objective function to suggest promising hyperparameters.
  • Dedicated tools — Katib, Optuna, Ray Tune, or cloud-managed hyperparameter services.

​
Random Forest — Key Hyperparameters

Random forests have several hyperparameters that determine ensemble behavior. The most commonly tuned include: These hyperparameters interact — changing one can change the optimal values of others — so tuning typically considers several simultaneously.

​
The n_estimators trade-off

The n_estimators parameter controls the number of trees. Each tree contributes via averaging (regression) or majority voting (classification). Increasing this value tends to reduce model variance and improve stability, but it increases training time and memory usage. If compute is constrained, start smaller and increase only if validation performance improves meaningfully.
A slide titled "Understanding n_estimators" showing multiple model/tree icons across the top and two panels summarizing trade-offs. The left panel says "More Stability" (lower variance, more accurate predictions) and the right panel says "More Cost" (higher training time and greater compute cost).

​
The max_depth trade-off

The max_depth hyperparameter limits how deep each decision tree can grow. Deep trees can model complex relationships but may overfit; shallow trees may underfit. The right depth depends on the dataset and other regularization hyperparameters like min_samples_leaf and max_features. Use cross-validation or inspect learning curves to guide depth selection.
An infographic titled "Understanding max_depth" showing three decision-tree diagrams—Shallow, Balanced, and Deep—illustrating how tree depth trades off underfitting and overfitting. Each panel includes a simple tree sketch and a short caption about generalization, the "sweet spot," or overfitting.

​
Practical tuning workflow

  1. Choose a small set of important hyperparameters (e.g., n_estimators, max_depth, max_features, min_samples_leaf).
  2. Pick a tuning method (random search or Bayesian optimization for efficiency).
  3. Use cross-validation with a consistent scoring metric.
  4. Monitor training time and resource usage; prefer models that meet performance and cost constraints.
  5. Iterate — widen or refine the search space based on results and diagnostics.
Example: a simple grid for scikit-learn GridSearchCV
Careful: exhaustive grid searches can be very expensive for large grids or expensive models. Use random search or Bayesian methods to explore high-dimensional spaces efficiently, and always monitor compute and memory usage.

​
Tips for robust tuning

  • Use stratified splits for classification tasks to keep class balance in folds.
  • Log experiments with consistent naming and parameter serialization (e.g., MLflow, Weights & Biases).
  • Start with coarse search ranges and refine around promising regions.
  • Consider early stopping or incremental training for large datasets.

​
References and further reading

Overall, hyperparameter tuning is iterative: combine domain expertise, systematic search, and validation metrics to find configurations that deliver strong, generalizable performance within your resource constraints.

Watch Video