The goal of the benchmarks in this repository is to measure the performance of various cuSpatial APIs.
Benchmarks in cuSpatial are written using the
pytest-benchmark plugin to the
pytest Python testing framework.
pytest-benchmark provides a seamless experience for developers familiar with
We include benchmarks of both public APIs and internal functions.
The former give us a macro view of our performance, especially vis-à-vis geopandas.
The latter help us quantify and minimize the overhead of our Python bindings.
Our current benchmarks focus entirely on measuring run time. However, minimizing memory footprint can be just as important for some cases. In the future, we may update our benchmarks to also include memory usage measurements.
At the top level benchmarks are divided into
API benchmarks are for public features that we expect users to consume.
Internal benchmarks capture the performance of cuSpatial internals that have no stability guarantees.
Within each directory, benchmarks are organized based on the type of function. Functions in cuSpatial generally fall into two groups:
Methods of classes like
Free functions operating on the above classes like
The former should be organized into files named
For example, benchmarks of
GeoDataFrame.sjoin belong in
Benchmarks should be written at the highest level of generality possible with respect to the class hierarchy.
For instance, all classes support the
take method, so those benchmarks belong in
pytest does not support having two benchmark files with the same name, even if they are in separate directories.
Therefore, benchmarks of internal methods of public classes go in files suffixed with
GeoDataFrame.polygons.xy, for instance, belong in
Free functions have more flexibility.
Broadly speaking, they should be grouped into benchmark files containing similar functionality.
For example, I/O benchmarks can all live in
For now those groupings are left to the discretion of developers.
By default, pytest discovers test files and functions prefixed with
For benchmarks, we configure
pytest to instead search using the
pytest-benchmark, running benchmarks is as simple as just running
When benchmarks are run, the default behavior is to output the results in a table to the terminal.
A common requirement is to then compare the performance of benchmarks before and after a change.
We can generate these comparisons by saving the output using the
--benchmark-autosave option to pytest.
When using this option, after the benchmarks are run the output will contain a line:
Saved benchmark data in: /path/to/XXXX_*.json
XXXX is a four-digit number identifying the benchmark.
If preferred, a user may also use the
which allows more control over the resulting filename.
Given two benchmark runs
YYYY, benchmarks can then be compared using
pytest-benchmark compare XXXX YYYY
Note that the comparison uses the
pytest-benchmark command rather than the
pytest-benchmark has a number of additional options that can be used to customize the output.
The next line contains one useful example, but developers should experiment to find a useful output
pytest-benchmark compare XXXX YYYY --sort="name" --columns=Mean --name=short --group-by=param
For more details, see the
Just as benchmarks should be written in terms of the highest level classes in the hierarchy, they should also assume as little as possible about the nature of the data.
Comparing to geopandas#
As the cuSpatial api matures, we’ll be comparing it performance-wise with matching geopandas functions.
Benchmarks need to be kept up to date with API changes in cuspatial. The current set of benchmarks are debug benchmarks on a small set of test data. Our CI testing takes advantage of this to ensure that benchmarks remain valid code.
Although not strictly part of our benchmarking suite, profiling is a common need so we provide some guidelines here. Here are two easy ways (there may be others) to profile benchmarks:
Using the former is as simple as adding the
--profile-svg) arguments to the
The latter requires instead invoking pytest from py-spy, like so:
py-spy record -- pytest bench_foo.py
Each tool has different strengths and provides somewhat different information. Developers should try both and see what works for a particular workflow. Developers are also encouraged to share useful alternatives that they discover.