HPO for Random Forest with Ray Tune and cuML#

This notebook demonstrates how to perform hyperparameter optimization (HPO) for a Random Forest classifier using Ray Tune and cuML. We’ll use Ray Tune to efficiently search through hyperparameter combinations while leveraging cuML’s GPU-accelerated Random Forest implementation for faster training.

Problem Overview#

We’re solving a binary classification problem using the airline dataset, where we predict flight delays. The goal is to find the optimal hyperparameters (number of estimators, max depth, and max features) that maximize the model’s accuracy. Ray Tune will orchestrate multiple training trials in parallel, each testing different hyperparameter combinations, while cuML provides GPU acceleration for each individual model training.

Setup Instructions#

Brev#

See Documentation

For the purpose of this example, follow Option 1 (Setting up your Brev GPU Environment) in the Brev Instance Setup section:

  • Create a GPU environment with 4 L4 GPUs

  • Make sure to include Jupyter in your setup

  • Wait until the “Open Notebook” button is flashing

  • Open the Notebook and navigate to a Jupyter terminal

Visit the documentation >>

Environment Setup#

  1. Check Your CUDA Version in the Jupyter terminal

Before installing dependencies, verify your CUDA version (shown in the top right corner of the output):

nvidia-smi

  1. Create a file named pyproject.toml and copy the content below

Based on your CUDA version you have, modify the cuML package:

  • CUDA 12.x: Use cuml-cu12==26.2.*

  • CUDA 13.x: Change to cuml-cu13==26.2.*

The pyproject.toml file should look like this:

[project]
name = "ray-cuml"
version = "0.1.0"
requires-python = "==3.13.*"
dependencies = [
    "ray[default]==2.53.0",
    "ray[data]==2.53.0",
    "ray[train]==2.53.0",
    "ray[tune]==2.53.0",
    "cuml-cu12==26.2.*",  # Change cu12 to cu13 if you have CUDA 13.x
    "jupyterlab-nvdashboard",
    "ipykernel",
    "ipywidgets",
]

  1. Install Dependencies

uv sync

Enable Jupyter nvdashboard

We can use the jupyterlab-nvdashboard extension monitor GPU usage in Jupyter

To be able to enable the nvdashboard jupyter extension, installed in as part of the setup,

  1. Restart Jupyter: sudo systemctl restart jupyter.service

  2. Exit and reopen the notebook or refresh your browser

When installing libraries with conda each individual CUDA library can be installed as a conda package. So we don’t need to ensure any of the CUDA libraries already exist in /usr/local/cuda.

  1. Install JupyterLab nvdashboard Extension

Important: Even though you’re using conda for this setup, the JupyterLab nvdashboard extension must be installed using uv (which is already available in the system). This is because JupyterLab extensions need to be installed where the JupyterLab server runs, not where individual kernels run. In the current setup, the JupyterLab server runs from /home/ubuntu/.venv/ (system uv environment), so we need to install the extension using uv:

uv pip install jupyterlab_nvdashboard

sudo systemctl restart jupyter.service

Exit and reopen the notebook, and go back to a Jupyter terminal.

  1. Install Miniforge

If you prefer to use conda, you need to install it first:

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"

bash Miniforge3-$(uname)-$(uname -m).sh  # Follow the prompts and choose yes to update your shell profile to automatically initialize conda

Note

You’ll need to source your .bashrc to make conda available in your current shell:

source ~/.bashrc

  1. Check Your CUDA Version

Check the CUDA version available on your system:

nvidia-smi

  1. Create Environment File

Create a file named env.yaml and copy the content below. Modify the cuda-version to match your CUDA version (e.g., 12.8 or 13.0):

name: ray-cuml
channels:
  - rapidsai
  - conda-forge
dependencies:
  - python=3.13
  - "ray-default=2.53.0"
  - "ray-data=2.53.0"
  - "ray-train=2.53.0"
  - "ray-tune=2.53.0"
  - cuml=26.02
  - "cuda-version=12.8"  # Change to match your CUDA version (e.g., 12.8 or 13.0)
  - ipykernel
  - ipywidgets

  1. Create and Activate Conda Environment

Create a new conda environment using the env.yaml file:

conda env create -f env.yaml

conda activate ray-cuml

  1. Install Jupyter Kernel

Install the Jupyter kernel for this environment:

python -m ipykernel install --user --name ray-cuml --display-name "Python (ray-cuml)" --env PATH "$CONDA_PREFIX/bin:$PATH"

After running this, refresh your browser, open a new notebook and select the “Python (ray-cuml)” kernel.

Getting Started#

Download this notebook and the get_data.pyscript from the side panel and upload them to Jupyter, then run through the notebook.

You should now see a button on the left panel that looks like a GPU, which will give you several dashboards to choose from. For the sake of this example, we will look at GPU memory and GPU Utilization.

GPU Dashboard Button

Data Preparation#

Make sure the get_data.py script is the same directory that current jupyter working directory. We will use this script to get the airline dataset.

The script supports both a small dataset (for quick testing) and a full dataset (20M rows). By default, it downloads the small dataset. Use the --full-dataset flag for the complete dataset.

! python get_data.py --full-dataset ## for a smaller dataset remove --full-dataset

import pandas as pd
import ray
from cuml.ensemble import RandomForestClassifier
from cuml.metrics import accuracy_score
from ray import tune
from ray.tune import RunConfig, TuneConfig
from sklearn.model_selection import train_test_split

def train_rf(config, data_dict):
    """
    Training function for Ray Tune.

    Args:
        config: Dictionary of hyperparameters from Ray Tune
        data_dict: Dictionary containing training and test data (NumPy arrays)
    """
    # Extract data
    X_train = data_dict["X_train"]
    X_test = data_dict["X_test"]
    y_train = data_dict["y_train"]
    y_test = data_dict["y_test"]

    # Initialize cuML Random Forest with hyperparameters from config
    rf = RandomForestClassifier(
        n_estimators=config["n_estimators"],
        max_depth=config["max_depth"],
        max_features=config["max_features"],
        random_state=42,
    )

    # Train the model
    rf.fit(X_train, y_train)

    # Evaluate on test set
    predictions = rf.predict(X_test)

    # Calculate accuracy using cuML's metric function
    score = accuracy_score(y_test, predictions)

    # Report metrics back to Ray Tune
    return {"accuracy": score}