Databricks

You can install RAPIDS on Databricks in a few different ways:

1. Accelerate machine learning workflows in a single-node GPU notebook environment on classic compute

2. Accelerate machine learning workflows in a single-node GPU notebook environment on serverless GPU compute

3. Spark users can install RAPIDS Accelerator for Apache Spark 3.x on Databricks

4. Install Dask alongside Spark and then use libraries like dask-cudf for multi-node workloads

Classic GPU compute

Create init-script

To get started, you must first configure an initialization script to install RAPIDS libraries and all other dependencies for your project.

Databricks recommends using cluster-scoped init scripts stored in the workspace files.

Navigate to the top-left Workspace tab and click on your Home directory then select Add > File from the menu. Create an init.sh script with contents:

#!/bin/bash
set -e

# Install RAPIDS libraries
pip install \
    --extra-index-url=https://pypi.nvidia.com \
    "cudf-cu13==26.08" "cuml-cu13==26.08"

Launch cluster

To get started, navigate to the All Purpose Compute tab of the Compute section in Databricks and select Create Compute. Name your cluster and choose “Single node”.

In order to launch a GPU node check the Machine Learning box and uncheck the Use Photon Acceleration box just below it, then select a runtime in the dropdown. For example you could select the 18 LTS (Scala 2.13, Spark 4.1.0) runtime version.

The “GPU accelerated” nodes should now be available in the Node type dropdown.

Then expand the Advanced Options section, open the Init Scripts tab and enter the file path to the init-script in your Workspace directory starting with /Users/<user-name>/<script-name>.sh and click “Add”.

Select Create Compute

Serverless GPU compute

Serverless GPU compute gives you a single-node GPU notebook with no cluster to create and no init script to maintain. Databricks provisions the GPU on demand when you attach a notebook to it.

Because there is no cluster to configure, the init script approach above does not apply. Install RAPIDS from inside the notebook instead.

Note

Serverless GPU compute is in public preview and is only available in certain regions.

Connect a notebook

Open a notebook, click the compute dropdown at the top and select Serverless GPU.

From the submenu select Configuration to open the configuration side panel. Under Hardware set Accelerator to 1xA10, and under Environment set Base environment to Standard v5. Then click Apply and Confirm.

Note

Choose the Standard environment rather than the AI environment. The AI environment preinstalls cupy-cuda12x, which conflicts with the cupy-cuda13x build that the RAPIDS CUDA 13 wheels depend on.

Install RAPIDS

Install the RAPIDS libraries into your notebook environment.

%pip install \
    --extra-index-url=https://pypi.nvidia.com \
    "cudf-cu13==26.08" "cuml-cu13==26.08"

Then restart the Python process so that the new packages are picked up.

%restart_python

Test RAPIDS

You can run the following code snippet to verify that the RAPIDS libraries are installed successfully on your choice of compute.

import cudf

gdf = cudf.DataFrame({"a":[1,2,3],"b":[4,5,6]})
gdf
    a   b
0   1   4
1   2   5
2   3   6

Quickstart with cuDF Pandas

RAPIDS recently introduced cuDF’s pandas accelerator mode to accelerate existing pandas workflows with zero changes to code.

Using cudf.pandas in Databricks on a single-node can offer significant performance improvements over traditional pandas when dealing with large datasets; operations are optimized to run on the GPU (cuDF) whenever possible, seamlessly falling back to the CPU (pandas) when necessary, with synchronization happening in the background.

Below is a quick example how to load the cudf.pandas extension in a Jupyter notebook:

%load_ext cudf.pandas

%%time

import pandas as pd

df = pd.read_parquet(
    "nyc_parking_violations_2022.parquet",
    columns=["Registration State", "Violation Description", "Vehicle Body Type", "Issue Date", "Summons Number"]
)

(df[["Registration State", "Violation Description"]]
 .value_counts()
 .groupby("Registration State")
 .head(1)
 .sort_index()
 .reset_index()
)

Upload the 10 Minutes to RAPIDS cuDF Pandas notebook into your Databricks workspace and run through the cells.