GitHub Actions

Overview

The RAPIDS team is in the process of migrating from Jenkins to GitHub Actions for CI/CD. The page below outlines some helpful information pertaining to the implementation of GitHub Actions provided by the RAPIDS Ops team. The official GitHub documentation for GitHub Actions is also useful and can be viewed here.

Intended audience

Operations

Developers

Table of contents

1. Implementation
   1. Ignoring Pull Request Branches in git
   2. Downloading CI Artifacts
   3. Skipping CI for Commits
   4. Rerunning Failed GitHub Actions
2. Self-Hosted Runners
   1. CPU Labels
   2. GPU Labels
   3. Usage

Implementation

The RAPIDS Ops team provides GPU enabled self-hosted runners for use with GitHub Actions to the RAPIDS and other select GitHub organizations.

To ensure proper usage of these GPU enabled CI machines, the RAPIDS Ops team has adopted a strategy known as Marking code as trusted by pushing upstream which is described in this CircleCI blog post.

The gist of the strategy is that the source code from trusted pull requests can be copied to a prefixed branch (e.g. pull-request/<PR_NUMBER>) within the source repository and CI can be configured to test only those prefixed branches rather than the pull requests themselves.

Pull requests authored by members of the given GitHub organization are considered trusted and therefore are copied to a pull-request/* branch for testing automatically.

Pull requests from authors outside of the GitHub organization must first be reviewed by a repository member with write permissions (or greater) to ensure that the code changes are legitimate and benign. That reviewer must leave an /ok to test (or /okay to test) comment on the pull request before its code is copied to a pull-request/* branch for testing.

The /ok to test comment is only valid for a single commit. Subsequent commits must be re-reviewed and validated with another /ok to test comment.

Ignoring Pull Request Branches in git

One consequence of the strategy described above is that a lot of pull-request/* branches will be created and deleted in GitHub as pull requests are opened and closed. To avoid having these branches fetched locally, you can run the following git config command, where upstream in remote.upstream.fetch is the git remote name corresponding to the source repository:

git config \
  --global \
  --add "remote.upstream.fetch" \
  '^refs/heads/pull-request/*'

Note that this git configuration option requires git version 2.29 or greater to support negative refspecs (source).

Downloading CI Artifacts

For NVIDIA employees with VPN access, artifacts from both pull-requests and branch builds can be accessed on https://downloads.rapids.ai/.

There is a link provided at the end of every C++ and Python build job where the build artifacts for that particular workflow run can be accessed.

Skipping CI for Commits

See the GitHub Actions documentation page below on how to prevent GitHub Actions from running on certain commits. This is useful for preventing GitHub Actions from running on pull requests that are not fully complete. This also helps preserve the finite GPU resources provided by the RAPIDS Ops team.

With GitHub Actions, it is not possible to configure all commits for a pull request to be skipped. It must be specified at the commit level.

Link: https://docs.github.com/en/actions/managing-workflow-runs/skipping-workflow-runs

Rerunning Failed GitHub Actions

See the GitHub Actions documentation page below on how to rerun failed workflows. In addition to rerunning an entire workflow, GitHub Actions also provides the ability to rerun only the failed jobs in a workflow.

At this time there are no alternative ways to rerun tests with GitHub Actions beyond what is described in the documentation (e.g. there is no rerun tests comment for GitHub Actions).

Link: https://docs.github.com/en/actions/managing-workflow-runs/re-running-workflows-and-jobs

Self-Hosted Runners

The RAPIDS Ops team provides a set of self-hosted runners that can be used in GitHub Action workflows throughout supported organizations. The tables below outline the labels that can be utilized and their related specifications.

CPU Labels

The CPU labeled runners are backed by various EC2 instances and do not have any GPUs installed.

Additional specifications:

1. https://aws.amazon.com/ec2/instance-types/m5/
2. https://aws.amazon.com/ec2/instance-types/m6g/

The CPU label names consist of the following components:

linux-amd64-cpu4
^     ^     ^  ^
|     |     |  |
|     |     |  CPU Core Count
|     |     CPU Designator
|     Architecture
Operating System

GPU Labels

The GPU labeled runners are backed by lab machines and have the GPUs specified in the table below installed.

IMPORTANT: GPU jobs have two requirements. If these requirements aren’t met, the GitHub Actions job will fail. See the Usage section below for an example.

1. They must run in a container (i.e. nvidia/cuda:11.8.0-base-ubuntu22.04)
2. They must set the NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }} container environment variable

Due to our limited GPU capacity and the overhead associated with manually rotating self-hosted runner labels when GPU drivers are updated, there are no driver-specific self-hosted runner labels (e.g. linux-amd64-gpu-t4-525-1).

Instead, the driver-version designators earliest and latest are used. The values of these designators represent the GPU driver version that RAPIDS uses for testing.

The chart below will be kept up-to-date with the corresponding driver versions at any given time.

Supported organizations will be notified whenever these versions are scheduled to be updated.

The GPU label names consist of the following components:

linux-amd64-gpu-t4-latest-1
^     ^     ^   ^  ^      ^
|     |     |   |  |      |
|     |     |   |  |      Number of GPUs Available
|     |     |   |  GPU Driver Version
|     |     |   GPU Type
|     |     GPU Designator
|     Architecture
Operating System

Usage

The code snippet below shows how the labels above may be utilized in a GitHub Action workflow.

name: Test Self Hosted Runners
on: push
jobs:
  job1_cpu:
    runs-on: linux-amd64-cpu8
    steps:
      - name: hello
        run: echo "hello"
  job2_gpu:
    runs-on: linux-amd64-gpu-v100-latest-1
    container: # GPU jobs must run in a container
      image: nvidia/cuda:11.8.0-base-ubuntu22.04
      env:
        NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }} # GPU jobs must set this container env variable
    steps:
      - name: hello
        run: |
          echo "hello"
          nvidia-smi

For additional details on self-hosted runner usage, see the official GitHub Action documentation page here: https://docs.github.com/en/actions/hosting-your-own-runners/using-self-hosted-runners-in-a-workflow