Ctrl+K

Getting Started with Optuna and RAPIDS for HPO#

Hyperparameter optimization (HPO) automates the process of picking values for the hyperparameters of a machine learning algorithm to improve model performance. This can help boost the model accuracy, but can be resource-intensive, as it may require training the model for hundreds of hyperparameter combinations. Let’s take a look at how we can use Optuna and RAPIDS to make HPO less time-consuming.

RAPIDS#

The RAPIDS framework provides a suite of libraries to execute end-to-end data science pipelines entirely on GPUs. One of the libraries in this framework is cuML, which implements common machine learning models with a scikit-learn-compatible API and a GPU-accelerated backend. You can learn more about RAPIDS here.

Optuna#

Optuna is a lightweight framework for automatic hyperparameter optimization. It provides a define-by-run API, which makes it easy to adapt to any already existing code that we have and enables high modularity along with the flexibility to construct hyperparameter spaces dynamically. By simply wrapping the objective function with Optuna, we can perform a parallel-distributed HPO search over a search space as we’ll see in this notebook.

In this notebook, we’ll use BNP Paribas Cardif Claims Management dataset from Kaggle to predict if a claim will receive accelerated approval or not. We’ll explore how to use Optuna with RAPIDS in combination with Dask to run multi-GPU HPO experiments that can yield results faster than CPU.

## Run this cell to install optuna
# !pip install optuna

import cudf
import cuml
import dask_cudf
import numpy as np
import optuna
import os
import dask

from cuml import LogisticRegression
from cuml.model_selection import train_test_split
from cuml.metrics import log_loss

from dask_cuda import LocalCUDACluster
from dask.distributed import Client, wait, performance_report

Set up CUDA Cluster#

We start a local cluster and keep it ready for running distributed tasks with dask. The dask scheduler can help leverage multiple nodes available on the cluster.

LocalCUDACluster launches one Dask worker for each GPU in the current systems. It’s developed as a part of the RAPIDS project. Learn More:

# This will use all GPUs on the local host by default
cluster = LocalCUDACluster(threads_per_worker=1, ip="", dashboard_address="8081")
c = Client(cluster)

# Query the client for all connected workers
workers = c.has_what().keys()
n_workers = len(workers)
c

2023-02-17 10:37:50,019 - distributed.preloading - INFO - Creating preload: dask_cuda.initialize
2023-02-17 10:37:50,019 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2023-02-17 10:37:50,042 - distributed.preloading - INFO - Creating preload: dask_cuda.initialize
2023-02-17 10:37:50,043 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2023-02-17 10:37:50,169 - distributed.preloading - INFO - Creating preload: dask_cuda.initialize
2023-02-17 10:37:50,169 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize
2023-02-17 10:37:50,169 - distributed.preloading - INFO - Creating preload: dask_cuda.initialize
2023-02-17 10:37:50,169 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

Client

Client-208a573f-aeaf-11ed-838c-98039b8174ae

Connection method: Cluster object Cluster type: dask_cuda.LocalCUDACluster
Dashboard: http://10.136.7.105:8081/status

Cluster Info

Loading the data#

Data Acquisition#

Dataset can be acquired from Kaggle: BNP Paribas Cardif Claims Management. To download the dataset:

  1. Create a data folder in the current directory. If you’re downloading it elsewhere, be sure to change data_dir to point to the approproate location.

  2. Follow the instructions here to: Set-up the Kaggle API

  3. Run the following command in the command line in the data folder.

    kaggle competitions download -c bnp-paribas-cardif-claims-management

This is an anonymized dataset containing categorical and numerical values for claims received by BNP Paribas Cardif. The “target” column in the train set is the variable to predict. It is equal to 1 for claims suitable for an accelerated approval. The task is to predict whether a claim will be suitable for accelerated approval or not. We’ll only use the train.csv file as test.csv does not have a target column.

import os

file_name = "train.csv"

data_dir = "data/"
INPUT_FILE = os.path.join(data_dir, file_name)

Select the N_TRIALS for the number of runs of HPO trials.

import pandas as pd

N_TRIALS = 150

df = cudf.read_csv(INPUT_FILE)

# Drop ID column
df = df.drop("ID", axis=1)

# Drop non-numerical data and fill NaNs before passing to cuML RF
CAT_COLS = list(df.select_dtypes("object").columns)
df = df.drop(CAT_COLS, axis=1)
df = df.fillna(0)

df = df.astype("float32")
X, y = df.drop(["target"], axis=1), df["target"].astype("int32")

study_name = "dask_optuna_lr_log_loss_tpe"

Training and Evaluation#

The train_and_eval function accepts the different parameters to try out. This function should look very similar to any ML workflow. We’ll use this function within the Optuna objective function to show how easily we can fit an existing workflow into the Optuna work.

def train_and_eval(
    X_param, y_param, penalty="l2", C=1.0, l1_ratio=None, fit_intercept=True
):
    """
    Splits the given data into train and test split to train and evaluate the model
    for the params parameters.

    Params
    ______

    X_param:  DataFrame.
              The data to use for training and testing.
    y_param:  Series.
              The label for training
    penalty, C, l1_ratio, fit_intercept: The parameter values for Logistic Regression.

    Returns
    score: log loss of the fitted model
    """
    X_train, X_valid, y_train, y_valid = train_test_split(
        X_param, y_param, random_state=42
    )
    classifier = LogisticRegression(
        penalty=penalty,
        C=C,
        l1_ratio=l1_ratio,
        fit_intercept=fit_intercept,
        max_iter=10000,
    )
    classifier.fit(X_train, y_train)
    y_pred = classifier.predict(X_valid)
    score = log_loss(y_valid, y_pred)
    return score

For a baseline number, let’s see what the default performance of the model is.

print("Score with default parameters : ", train_and_eval(X, y))

Score with default parameters :  8.206788138572605

Objective Function#

We will optimize the objective function using Optuna Study. The objective function tries out specified values for the parameters that we are tuning and returns the score obtained with those parameters. These results will be aggregated in study.trials_dataframes().

Let’s define the objective function for this HPO task by making use of the train_and_eval(). You can see that we simply choose a value for the parameters and call the train_and_eval method, making Optuna very easy to use in an existing workflow.

The objective function does not need to be changed when switching to different samplers, which are built-in options in Optuna to enable the selection of different sampling algorithms that optuna provides. Some of the available ones include - GridSampler, RandomSampler, TPESampler, etc. We’ll use TPESampler for this demo, but feel free to try different samplers to notice the changes in performance.

Tree-Structured Parzen Estimators or TPE works by fitting two Gaussian Mixture Model during each trial - one to the set of parameter values associated with the best objective values, and another to the remaining parameter values. It chooses the parameter value that maximizes the ratio between the two GMMs

def objective(trial, X_param, y_param):
    C = trial.suggest_float("C", 0.01, 100.0, log=True)
    penalty = trial.suggest_categorical("penalty", ["none", "l1", "l2"])
    fit_intercept = trial.suggest_categorical("fit_intercept", [True, False])

    score = train_and_eval(
        X_param, y_param, penalty=penalty, C=C, fit_intercept=fit_intercept
    )
    return score

HPO Trials and Study#

Optuna uses studies and trials to keep track of the HPO experiments. Put simply, a trial is a single call of the objective function while a set of trials make up a study. We will pick the best observed trial from a study to get the best parameters that were used in that run.

Here, DaskStorage class is used to set up a storage shared by all workers in the cluster. Learn more about what storages can be used here

optuna.create_study is used to set up the study. As you can see, it specifies the study name, sampler to be used, the direction of the study, and the storage. With just a few lines of code, we have set up a distributed HPO experiment.

storage = optuna.integration.DaskStorage()
study = optuna.create_study(
    sampler=optuna.samplers.TPESampler(seed=142),
    study_name=study_name,
    direction="minimize",
    storage=storage,
)
# Optimize in parallel on your Dask cluster
#
# Submit `n_workers` optimization tasks, where each task runs about 40 optimization trials
# for a total of about N_TRIALS trials in all
futures = [
    c.submit(
        study.optimize,
        lambda trial: objective(trial, X, y),
        n_trials=N_TRIALS // n_workers,
        pure=False,
    )
    for _ in range(n_workers)
]
wait(futures)
print(f"Best params: {study.best_params}")

print("Number of finished trials: ", len(study.trials))

/tmp/ipykernel_1049484/2672349610.py:1: ExperimentalWarning: DaskStorage is experimental (supported from v3.1.0). The interface can change in the future.
  storage = optuna.integration.DaskStorage()
[I 2023-02-17 10:37:57,870] A new study created in memory with name: dask_optuna_lr_log_loss_tpe
/nvme/0/thead/miniconda/envs/cloud-ml-deploy-optuna/lib/python3.10/site-packages/distributed/worker.py:2990: UserWarning: Large object of size 49.31 MiB detected in task graph: 
  [<function <listcomp>.<lambda> at 0x7fad203a3d90>]
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and 
keep data on workers

    future = client.submit(func, big_data)    # bad

    big_future = client.scatter(big_data)     # good
    future = client.submit(func, big_future)  # good
  warnings.warn(
/nvme/0/thead/miniconda/envs/cloud-ml-deploy-optuna/lib/python3.10/site-packages/distributed/protocol/pickle.py:73: ExperimentalWarning: DaskStorage is experimental (supported from v3.1.0). The interface can change in the future.
  return pickle.loads(x)
/nvme/0/thead/miniconda/envs/cloud-ml-deploy-optuna/lib/python3.10/site-packages/distributed/protocol/pickle.py:73: ExperimentalWarning: DaskStorage is experimental (supported from v3.1.0). The interface can change in the future.
  return pickle.loads(x)
/nvme/0/thead/miniconda/envs/cloud-ml-deploy-optuna/lib/python3.10/site-packages/distributed/protocol/pickle.py:73: ExperimentalWarning: DaskStorage is experimental (supported from v3.1.0). The interface can change in the future.
  return pickle.loads(x)
/nvme/0/thead/miniconda/envs/cloud-ml-deploy-optuna/lib/python3.10/site-packages/distributed/protocol/pickle.py:73: ExperimentalWarning: DaskStorage is experimental (supported from v3.1.0). The interface can change in the future.
  return pickle.loads(x)
[I 2023-02-17 10:38:07,033] Trial 0 finished with value: 8.247875378327683 and parameters: {'C': 40.573838784392514, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 0 with value: 8.247875378327683.
[I 2023-02-17 10:38:07,953] Trial 3 finished with value: 8.221289538703516 and parameters: {'C': 12.183061292316713, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:08,041] Trial 2 finished with value: 8.253917695992195 and parameters: {'C': 40.573838784392514, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:08,169] Trial 1 finished with value: 8.228540336686851 and parameters: {'C': 40.573838784392514, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:08,605] Trial 6 finished with value: 8.250292310988796 and parameters: {'C': 12.183061292316713, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:08,664] Trial 5 finished with value: 8.241833396381626 and parameters: {'C': 0.028870434982716877, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:09,044] Trial 4 finished with value: 8.245458781385024 and parameters: {'C': 40.573838784392514, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:09,099] Trial 8 finished with value: 8.25029236694187 and parameters: {'C': 0.028870434982716877, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:09,251] Trial 9 finished with value: 8.247875294398069 and parameters: {'C': 67.82426496348721, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:09,455] Trial 11 finished with value: 8.251500763331082 and parameters: {'C': 67.82426496348721, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:09,566] Trial 12 finished with value: 8.24304134514623 and parameters: {'C': 6.682974952480256, 'penalty': 'l2', 'fit_intercept': False}. Best is trial 3 with value: 8.221289538703516.
[I 2023-02-17 10:38:09,619] Trial 7 finished with value: 8.21403885262633 and parameters: {'C': 12.183061292316713, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:09,722] Trial 10 finished with value: 8.25029236694187 and parameters: {'C': 12.183061292316713, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:09,818] Trial 13 finished with value: 8.253917612062581 and parameters: {'C': 2.0030868464433764, 'penalty': 'none', 'fit_intercept': False}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:09,872] Trial 14 finished with value: 8.251500679401468 and parameters: {'C': 2.0030868464433764, 'penalty': 'none', 'fit_intercept': False}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:10,092] Trial 16 finished with value: 8.24787526642153 and parameters: {'C': 2.0030868464433764, 'penalty': 'none', 'fit_intercept': False}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:10,099] Trial 15 finished with value: 8.252709243649907 and parameters: {'C': 2.0030868464433764, 'penalty': 'none', 'fit_intercept': False}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:10,557] Trial 17 finished with value: 8.249084278294575 and parameters: {'C': 0.9459046834713561, 'penalty': 'l1', 'fit_intercept': False}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:10,581] Trial 18 finished with value: 8.247875769999213 and parameters: {'C': 0.4428829638044645, 'penalty': 'l1', 'fit_intercept': False}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:11,477] Trial 19 finished with value: 8.21524817224129 and parameters: {'C': 0.34118772696334027, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:11,684] Trial 20 finished with value: 8.214039719899006 and parameters: {'C': 0.34118772696334027, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:12,541] Trial 22 finished with value: 8.215248116288217 and parameters: {'C': 7.20323151375873, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:12,603] Trial 21 finished with value: 8.21524817224129 and parameters: {'C': 7.20323151375873, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:12,780] Trial 23 finished with value: 8.214039775852083 and parameters: {'C': 0.15587899286863757, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:13,189] Trial 24 finished with value: 8.214039775852083 and parameters: {'C': 0.15587899286863757, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:13,946] Trial 25 finished with value: 8.220082205422743 and parameters: {'C': 0.13331995152410292, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:14,425] Trial 26 finished with value: 8.218873697127382 and parameters: {'C': 0.13331995152410292, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 7 with value: 8.21403885262633.
[I 2023-02-17 10:38:14,506] Trial 27 finished with value: 8.211622731284818 and parameters: {'C': 0.13317405504008187, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 27 with value: 8.211622731284818.
[I 2023-02-17 10:38:14,754] Trial 28 finished with value: 8.218873753080457 and parameters: {'C': 0.09161974779967816, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 27 with value: 8.211622731284818.
[I 2023-02-17 10:38:14,932] Trial 30 finished with value: 8.217665356691247 and parameters: {'C': 0.01161870199450424, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 27 with value: 8.211622731284818.
[I 2023-02-17 10:38:15,511] Trial 29 finished with value: 8.218873781056995 and parameters: {'C': 0.0834294717814003, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 27 with value: 8.211622731284818.
[I 2023-02-17 10:38:15,538] Trial 31 finished with value: 8.217665496573938 and parameters: {'C': 0.01161870199450424, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 27 with value: 8.211622731284818.
[I 2023-02-17 10:38:15,802] Trial 32 finished with value: 8.21645704423165 and parameters: {'C': 0.01161870199450424, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 27 with value: 8.211622731284818.
[I 2023-02-17 10:38:16,028] Trial 33 finished with value: 8.210414362872145 and parameters: {'C': 0.5889161850687648, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:16,593] Trial 34 finished with value: 8.211622787237895 and parameters: {'C': 0.29039065153275884, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:17,556] Trial 36 finished with value: 8.214039635969392 and parameters: {'C': 0.29039065153275884, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:17,762] Trial 38 finished with value: 8.214039747875544 and parameters: {'C': 0.6356655062780381, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:17,906] Trial 37 finished with value: 8.214039719899006 and parameters: {'C': 0.40506074358435035, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:17,938] Trial 35 finished with value: 8.215248200217829 and parameters: {'C': 0.29039065153275884, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:18,033] Trial 39 finished with value: 8.247875574163448 and parameters: {'C': 0.7211398436523727, 'penalty': 'none', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:18,215] Trial 40 finished with value: 8.250292338965332 and parameters: {'C': 0.8988603430698667, 'penalty': 'none', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:18,357] Trial 42 finished with value: 8.252709215673368 and parameters: {'C': 0.8562976914659627, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:18,521] Trial 43 finished with value: 8.246667177774237 and parameters: {'C': 1.1735421529699095, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 33 with value: 8.210414362872145.
[I 2023-02-17 10:38:18,759] Trial 41 finished with value: 8.203162697616131 and parameters: {'C': 1.0180328243030141, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:18,875] Trial 45 finished with value: 8.249083998529198 and parameters: {'C': 1.470107035818969, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:19,078] Trial 44 finished with value: 8.209204903374491 and parameters: {'C': 1.215885094202452, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:19,122] Trial 47 finished with value: 8.244249853441593 and parameters: {'C': 3.1251012068006196, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:19,212] Trial 48 finished with value: 8.241832892803943 and parameters: {'C': 3.1251012068006196, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:19,607] Trial 49 finished with value: 8.249083802693432 and parameters: {'C': 0.5247083049626469, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:19,787] Trial 51 finished with value: 8.250292283012255 and parameters: {'C': 0.594672049508206, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:19,883] Trial 50 finished with value: 8.215246941273623 and parameters: {'C': 0.594672049508206, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:19,924] Trial 52 finished with value: 8.245458417690031 and parameters: {'C': 0.6982022429771052, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:20,141] Trial 53 finished with value: 8.247875322374606 and parameters: {'C': 0.6982022429771052, 'penalty': 'l2', 'fit_intercept': False}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:20,364] Trial 55 finished with value: 8.252709187696832 and parameters: {'C': 23.88860256720992, 'penalty': 'l2', 'fit_intercept': False}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:20,631] Trial 46 finished with value: 8.220082093516593 and parameters: {'C': 0.18943289702535804, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:20,707] Trial 56 finished with value: 8.243041568958532 and parameters: {'C': 23.88860256720992, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:21,054] Trial 59 finished with value: 8.249083886623046 and parameters: {'C': 0.2304567118696556, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:21,190] Trial 54 finished with value: 8.217664153700117 and parameters: {'C': 28.94484897140112, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:21,320] Trial 57 finished with value: 8.209205099210257 and parameters: {'C': 0.2304567118696556, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:21,667] Trial 61 finished with value: 8.247875602139985 and parameters: {'C': 1.3083983229057563, 'penalty': 'none', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:21,697] Trial 60 finished with value: 8.217664041793965 and parameters: {'C': 1.3350093383986137, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:22,213] Trial 64 finished with value: 8.25150079130762 and parameters: {'C': 0.44802419204313526, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 41 with value: 8.203162697616131.
[I 2023-02-17 10:38:22,283] Trial 62 finished with value: 8.198328832293907 and parameters: {'C': 0.46570264632805103, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:22,632] Trial 65 finished with value: 8.24908397055266 and parameters: {'C': 0.24456030311795268, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:22,636] Trial 58 finished with value: 8.205579714206857 and parameters: {'C': 0.22786149466321445, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:22,719] Trial 66 finished with value: 8.245458277807343 and parameters: {'C': 0.2562139315011522, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:22,864] Trial 63 finished with value: 8.210413383693316 and parameters: {'C': 0.42963908148132157, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,059] Trial 68 finished with value: 8.255126148334481 and parameters: {'C': 0.4325592525072435, 'penalty': 'none', 'fit_intercept': False}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,092] Trial 67 finished with value: 8.251500763331082 and parameters: {'C': 0.400363407043762, 'penalty': 'none', 'fit_intercept': False}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,111] Trial 69 finished with value: 8.250292283012257 and parameters: {'C': 0.45811082432017874, 'penalty': 'none', 'fit_intercept': False}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,202] Trial 70 finished with value: 8.252709215673368 and parameters: {'C': 0.3366950282604448, 'penalty': 'none', 'fit_intercept': False}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,447] Trial 73 finished with value: 8.24666687003232 and parameters: {'C': 1.001276433559266, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,497] Trial 72 finished with value: 8.249083830669969 and parameters: {'C': 0.9465587879326931, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,539] Trial 74 finished with value: 8.244249881418131 and parameters: {'C': 0.17498501865193736, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,762] Trial 75 finished with value: 8.234582010890993 and parameters: {'C': 0.09676716520222205, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,875] Trial 76 finished with value: 8.251500875237234 and parameters: {'C': 0.11796046210672348, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:23,882] Trial 71 finished with value: 8.252709243649907 and parameters: {'C': 0.35804471299010093, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:24,158] Trial 78 finished with value: 8.247875602139986 and parameters: {'C': 0.5360163523983655, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:24,554] Trial 77 finished with value: 8.214038628814027 and parameters: {'C': 0.11385001891151933, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:24,605] Trial 81 finished with value: 8.252709383532595 and parameters: {'C': 0.19199360495534568, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:24,616] Trial 80 finished with value: 8.247875378327683 and parameters: {'C': 0.2180889828523076, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:25,238] Trial 79 finished with value: 8.215247640687073 and parameters: {'C': 0.19199360495534568, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:25,394] Trial 82 finished with value: 8.222498746412327 and parameters: {'C': 0.20307546518876748, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:26,112] Trial 84 finished with value: 8.206788865962592 and parameters: {'C': 0.32564474352266887, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:26,431] Trial 83 finished with value: 8.215248116288217 and parameters: {'C': 0.2765570042119916, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:26,634] Trial 85 finished with value: 8.215248200217832 and parameters: {'C': 0.3226259801946211, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:26,877] Trial 86 finished with value: 8.214039719899006 and parameters: {'C': 0.2819874675157722, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:27,817] Trial 88 finished with value: 8.214039775852081 and parameters: {'C': 0.31458836735455475, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:27,949] Trial 87 finished with value: 8.217665104902405 and parameters: {'C': 0.3065189818970442, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:28,018] Trial 89 finished with value: 8.220082149469668 and parameters: {'C': 0.15565170050074784, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:28,294] Trial 90 finished with value: 8.215248228194367 and parameters: {'C': 0.7506376168835345, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:28,558] Trial 92 finished with value: 8.249083774716894 and parameters: {'C': 0.5051732837836762, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:28,952] Trial 91 finished with value: 8.21041427894253 and parameters: {'C': 0.8065457520463032, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:29,007] Trial 94 finished with value: 8.246667709328458 and parameters: {'C': 0.560374893711422, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:29,057] Trial 93 finished with value: 8.204371205911494 and parameters: {'C': 0.8065457520463032, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:29,665] Trial 98 finished with value: 8.250292338965332 and parameters: {'C': 0.7337233638654149, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:30,198] Trial 97 finished with value: 8.21283126755672 and parameters: {'C': 0.8412756142236871, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:30,235] Trial 96 finished with value: 8.211622815214431 and parameters: {'C': 0.562054482682929, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:30,414] Trial 95 finished with value: 8.216456652560117 and parameters: {'C': 0.5851560435116105, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:30,716] Trial 99 finished with value: 8.21283031635443 and parameters: {'C': 1.0461855157044002, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:30,837] Trial 102 finished with value: 8.244249853441593 and parameters: {'C': 1.1835595334965499, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:30,837] Trial 101 finished with value: 8.216455617428215 and parameters: {'C': 1.0461855157044002, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:31,278] Trial 103 finished with value: 8.222497655327349 and parameters: {'C': 1.4806853519511822, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:31,285] Trial 100 finished with value: 8.212830260401352 and parameters: {'C': 1.0867788142619192, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:31,549] Trial 105 finished with value: 8.22733194029764 and parameters: {'C': 0.3745168254537794, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:31,652] Trial 106 finished with value: 8.250292338965332 and parameters: {'C': 0.4276855478215523, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:32,137] Trial 107 finished with value: 8.207996590914894 and parameters: {'C': 0.385747857973403, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:32,157] Trial 104 finished with value: 8.216455701357829 and parameters: {'C': 1.4996076633665074, 'penalty': 'none', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:32,288] Trial 108 finished with value: 8.23216533001872 and parameters: {'C': 0.43235250787942997, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:32,990] Trial 109 finished with value: 8.215248256170906 and parameters: {'C': 0.7633220953391018, 'penalty': 'l1', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:33,010] Trial 112 finished with value: 8.233373950220235 and parameters: {'C': 0.6895329552551867, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:33,280] Trial 110 finished with value: 8.214038656790564 and parameters: {'C': 0.7956169312610232, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:33,317] Trial 113 finished with value: 8.238207507800542 and parameters: {'C': 0.6334264079574883, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:33,560] Trial 111 finished with value: 8.210413439646391 and parameters: {'C': 0.8065508826459779, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:33,755] Trial 116 finished with value: 8.255126232264095 and parameters: {'C': 0.2513108574682647, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:33,837] Trial 115 finished with value: 8.26237680643513 and parameters: {'C': 0.22712407979701488, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:33,850] Trial 114 finished with value: 8.215247249015542 and parameters: {'C': 0.2585858272339558, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:34,373] Trial 117 finished with value: 8.210413187857553 and parameters: {'C': 0.4718385150774379, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:34,413] Trial 120 finished with value: 8.246666786102704 and parameters: {'C': 0.37448244554979354, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:34,548] Trial 119 finished with value: 8.230956905652974 and parameters: {'C': 0.4900985680950606, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:34,730] Trial 122 finished with value: 8.246666786102704 and parameters: {'C': 0.49870893747744915, 'penalty': 'l2', 'fit_intercept': False}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:34,824] Trial 121 finished with value: 8.255126092381406 and parameters: {'C': 0.49284965645764406, 'penalty': 'l2', 'fit_intercept': False}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:34,895] Trial 118 finished with value: 8.21524727699208 and parameters: {'C': 0.4832324710781941, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:35,010] Trial 123 finished with value: 8.252709131743755 and parameters: {'C': 0.49547914281942596, 'penalty': 'l2', 'fit_intercept': False}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:35,307] Trial 124 finished with value: 8.249083942576123 and parameters: {'C': 0.8573213288225163, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:35,358] Trial 125 finished with value: 8.25029236694187 and parameters: {'C': 0.9150325602275309, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:35,399] Trial 126 finished with value: 8.251500875237234 and parameters: {'C': 0.8573213288225163, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:35,892] Trial 129 finished with value: 8.247875518210373 and parameters: {'C': 0.6422634169492649, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:35,906] Trial 127 finished with value: 8.211621696152916 and parameters: {'C': 0.9044711611615769, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:35,921] Trial 128 finished with value: 8.252709243649907 and parameters: {'C': 0.35612096813530547, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:36,336] Trial 131 finished with value: 8.247875518210373 and parameters: {'C': 0.6348982306281373, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:37,029] Trial 132 finished with value: 8.216455645404753 and parameters: {'C': 0.3879975753305631, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:37,137] Trial 130 finished with value: 8.205579658253782 and parameters: {'C': 1.7492428217117773, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:37,497] Trial 133 finished with value: 8.209205155163334 and parameters: {'C': 0.38584940993610345, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:37,966] Trial 134 finished with value: 8.216455393615911 and parameters: {'C': 1.270310588632334, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:38,063] Trial 136 finished with value: 8.244249993324283 and parameters: {'C': 1.6903129048405776, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:38,308] Trial 135 finished with value: 8.210413355716778 and parameters: {'C': 1.743501860802611, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:38,530] Trial 138 finished with value: 8.251500903213772 and parameters: {'C': 1.7228365993625243, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:39,186] Trial 139 finished with value: 8.244250049277358 and parameters: {'C': 2.004047082657374, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:39,847] Trial 137 finished with value: 8.211622003894833 and parameters: {'C': 1.6881329891557173, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:41,105] Trial 140 finished with value: 8.209204903374491 and parameters: {'C': 0.32273955285610206, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:41,614] Trial 141 finished with value: 8.247875574163448 and parameters: {'C': 0.32068692350825667, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:42,102] Trial 142 finished with value: 8.238207479824005 and parameters: {'C': 0.22677737647344867, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:42,744] Trial 143 finished with value: 8.221289175008522 and parameters: {'C': 0.33183468485179807, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:43,434] Trial 144 finished with value: 8.246666953961933 and parameters: {'C': 0.1648049730677063, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:44,646] Trial 145 finished with value: 8.216455393615913 and parameters: {'C': 0.3859354935049987, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.
[I 2023-02-17 10:38:45,849] Trial 146 finished with value: 8.207996451032203 and parameters: {'C': 0.2871110323573213, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.

Best params: {'C': 0.46570264632805103, 'penalty': 'none', 'fit_intercept': True}
Number of finished trials:  148

[I 2023-02-17 10:38:46,575] Trial 147 finished with value: 8.249083886623046 and parameters: {'C': 0.2843200519404071, 'penalty': 'l2', 'fit_intercept': True}. Best is trial 62 with value: 8.198328832293907.

Visualization#

Optuna provides an easy way to visualize the trials via builtin graphs. Read more about visualizations here.

Conluding Remarks#

This notebook shows how RAPIDS and Optuna can be used along with dask to run multi-GPU HPO jobs, and can be used as a starting point for anyone wanting to get started with the framework. We have seen how by just adding a few lines of code we were able to integrate the libraries for a muli-GPU HPO runs. This can also be scaled to multiple nodes.

Next Steps#

This is done on a small dataset, you are encouraged to test out on larger data with more range for the parameters too. These experiments can yield performance improvements. Refer to other examples in the rapidsai/cloud-ml-examples repository.

Resources#

Hyperparameter Tuning in Python

Overview of Hyperparameter tuning

How to make your model awesome with Optuna

On this page
Related files
Tags