Dependency Management Documentation

Overview

Outlines how to test and change dependencies of the RAPIDS projects

Intended audience

Developers

Project Leads

Operations

Adding a Dependency

The RAPIDS Integration repository contains all the information needed for adding a dependency to the RAPIDS projects. Which meta.yaml changed is based on the purpose of the package being installed. The purpose of each meta-package is listed below:

Package Name Purpose
rapids-build-env Installs all conda build dependencies to build & test RAPIDS libraries from source
rapids-doc-env Installs all tools needed to build RAPIDS documentation
rapids-notebook-env Installs a Jupyter Notebook server and other dependencies to run RAPIDS example notebooks; used in the runtime stable and nightly RAPIDS containers.

Adding Versions

For new packages or those that do not have defined versions they need to be added.

Modifying Recipes

To add a package with versioning to the recipe a PACKAGE_NAME and the VERSIONING_NAME must be added to the file.

  • PACKAGE_NAME - is the conda package name
  • VERSIONING_NAME - is the conda package name with - replaced with _ and a suffix of _version added
    • For example
      • cupy would become cupy_version
      • scikit-learn would become scikit_learn_version

Once the PACKAGE_NAME and VERSIONING_NAME are ready, add them to the meta.yml as follows:

PACKAGE_NAME {{ VERSIONING_NAME }}
  • NOTE: The VERSIONING_NAME must be surrounded by the {{ }} for the substitution to work.

Using the examples of cupy and scikit-learn the entries in the meta.yaml would be:

cupy {{ cupy_version }}
scikit-learn {{ scikit_learn_version }}

Modifying Versions Files

There are two versions files that are in conda/recipes:

  • release-versions.yaml - These are versions used by the ci/axis/release.yaml for RELEASE builds
  • nightly-versions.yaml - These are versions used by the ci/axis/nightly.yaml for NIGHTLY builds

Both of these files will need a config added to specify the version for the newly created VERSIONING_NAME.

For each VERSIONING_NAME there must be a VERSION_SPEC. This can be any of the standard conda version specifiers:

>=1.8.0
>=0.48,<0.49
>=7.0,<8.0.0a0,!=7.1.0
=2.5.*

Combined together each of the versions files would add the following for each VERSIONING_NAME:

VERSIONING_NAME:
  - 'VERSION_SPEC'

Using the examples of cupy and scikit-learn the entries in the *-versions.yaml would be:

cupy_version:
  - '>=7,<8.0.0a0,!=7.1.0'
scikit_learn_version:
  - '=0.21.3'

Updating Versions

There are two versions files that are in conda/recipes:

  • release-versions.yaml - These are versions used by the ci/axis/release.yaml for RELEASE builds
  • nightly-versions.yaml - These are versions used by the ci/axis/nightly.yaml for NIGHTLY builds

Edit the files above and update the VERSION_SPEC as desired.

For more information see the Modifying Versions Files

Once done, submit a PR to the integration repository. The changes will be built and tested on a development docker image to confirm that it doesn’t conflict with other RAPIDS libraries.

Testing a Dependency

The build scripts located in the ci/gpu folder of the project can be easily modified to test a dependency. For example, the build script for cuDF may contain these lines to install packages required to build from source and test notebooks. To test a new dependency before adding to or updating the integration repo, follow these steps:

conda install "rmm=$MINOR_VERSION.*" "rapids-build-env=$MINOR_VERSION.*" \
              "rapids-notebook-env=$MINOR_VERSION.*"

rapids-build-env and rapids-notebook-env are meta-packages that are used to keep dependencies across the projects consistent. These packages have various packages with specific versions listed as its dependencies. To change these dependencies as a test on a PR, the meta-package should be removed. Removing it allows the installation of different versions of dependencies without causing conda conflicts. Here is what to add to the script to allow this:

conda install "rmm=$MINOR_VERSION.*" "rapids-build-env=$MINOR_VERSION.*" \
              "rapids-notebook-env=$MINOR_VERSION.*"

conda remove --force rapids-build-env rapids-notebook-env
conda install "your-pkg=1.0.0"

With this method, dependency changes can be tested inside a PR. However, this should only be used as a temporary test, and not a permanent change. Once it is confirmed that the change works, the new dependency should be updated or added to the integration repository.