Attention

The vector search and clustering algorithms in RAFT are being migrated to a new library dedicated to vector search called cuVS. We will continue to support the vector search algorithms in RAFT during this move, but will no longer update them after the RAPIDS 24.06 (June) release. We plan to complete the migration by RAPIDS 24.10 (October) release and they will be removed from RAFT altogether in the 24.12 (December) release.

Resources#

All resources which are specific to a computing environment like host or device are contained within, and managed by, raft::resources. This design simplifies the APIs and eases user burden by making the APIs opaque by default but allowing customization based on user preference.

Vocabulary#

#include <raft/core/resource/resource_types.hpp>

namespace raft::resource

enum resource_type#

Resource types can apply to any resource and don’t have to be host- or device-specific.

Values:

enumerator CUBLAS_HANDLE#
enumerator CUSOLVER_DN_HANDLE#
enumerator CUSOLVER_SP_HANDLE#
enumerator CUSPARSE_HANDLE#
enumerator CUDA_STREAM_VIEW#
enumerator CUDA_STREAM_POOL#
enumerator CUDA_STREAM_SYNC_EVENT#
enumerator COMMUNICATOR#
enumerator SUB_COMMUNICATOR#
enumerator DEVICE_PROPERTIES#
enumerator DEVICE_ID#
enumerator STREAM_VIEW#
enumerator THRUST_POLICY#
enumerator WORKSPACE_RESOURCE#
enumerator CUBLASLT_HANDLE#
enumerator CUSTOM#
enumerator LARGE_WORKSPACE_RESOURCE#
enumerator NCCL_CLIQUE#
enumerator LAST_KEY#
class resource#
#include <resource_types.hpp>

A resource constructs and contains an instance of some pre-determined object type and facades that object behind a common API.

Subclassed by raft::resource::comms_resource, raft::resource::cublas_resource, raft::resource::cublaslt_resource, raft::resource::cuda_event_resource, raft::resource::cuda_stream_pool_resource, raft::resource::cuda_stream_resource, raft::resource::cusolver_dn_resource, raft::resource::cusolver_sp_resource, raft::resource::cusparse_resource, raft::resource::custom_resource, raft::resource::device_id_resource, raft::resource::device_memory_resource, raft::resource::device_properties_resource, raft::resource::empty_resource, raft::resource::limiting_memory_resource, raft::resource::nccl_clique_resource, raft::resource::stream_view_resource, raft::resource::sub_comms_resource, raft::resource::thrust_policy_resource

class empty_resource : public raft::resource::resource#
#include <resource_types.hpp>
class resource_factory#
#include <resource_types.hpp>

A resource factory knows how to construct an instance of a specific raft::resource::resource.

Subclassed by raft::resource::comms_resource_factory, raft::resource::cublas_resource_factory, raft::resource::cublaslt_resource_factory, raft::resource::cuda_stream_pool_resource_factory, raft::resource::cuda_stream_resource_factory, raft::resource::cusolver_dn_resource_factory, raft::resource::cusolver_sp_resource_factory, raft::resource::cusparse_resource_factory, raft::resource::custom_resource_factory, raft::resource::device_id_resource_factory, raft::resource::device_properties_resource_factory, raft::resource::empty_resource_factory, raft::resource::large_workspace_resource_factory, raft::resource::nccl_clique_resource_factory, raft::resource::stream_view_resource_factory, raft::resource::sub_comms_resource_factory, raft::resource::thrust_policy_resource_factory, raft::resource::workspace_resource_factory

Public Functions

virtual resource_type get_resource_type() = 0#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

virtual resource *make_resource() = 0#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

class empty_resource_factory : public raft::resource::resource_factory#
#include <resource_types.hpp>

A resource factory knows how to construct an instance of a specific raft::resource::resource.

Public Functions

inline virtual resource_type get_resource_type() override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource() override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Device Resources#

raft::device_resources is a convenience over using raft::resources directly. It provides accessor methods to retrieve resources such as the CUDA stream, stream pool, and handles to the various CUDA math libraries like cuBLAS and cuSOLVER.

#include <raft/core/device_resources.hpp>

namespace raft::core

class device_resources : public raft::resources#

Main resource container object that stores all necessary resources used for calling necessary device functions, cuda kernels and/or libraries.

Public Functions

inline device_resources(rmm::cuda_stream_view stream_view = rmm::cuda_stream_per_thread, std::shared_ptr<rmm::cuda_stream_pool> stream_pool = {nullptr}, std::shared_ptr<rmm::mr::device_memory_resource> workspace_resource = {nullptr}, std::optional<std::size_t> allocation_limit = std::nullopt)#

Construct a resources instance with a stream view and stream pool.

Parameters:

  • stream_view – the default stream (which has the default per-thread stream if unspecified)

  • stream_pool – [in] the stream pool used (which has default of nullptr if unspecified)

  • workspace_resource – [in] an optional resource used by some functions for allocating temporary workspaces.

  • allocation_limit – [in] the total amount of memory in bytes available to the temporary workspace resources.

inline virtual ~device_resources()#

Destroys all held-up resources

inline void sync_stream(rmm::cuda_stream_view stream) const#

synchronize a stream on the current container

inline void sync_stream() const#

synchronize main stream on the current container

inline rmm::cuda_stream_view get_stream() const#

returns main stream on the current container

inline bool is_stream_pool_initialized() const#

returns whether stream pool was initialized on the current container

inline const rmm::cuda_stream_pool &get_stream_pool() const#

returns stream pool on the current container

inline rmm::cuda_stream_view get_stream_from_stream_pool() const#

return stream from pool

inline rmm::cuda_stream_view get_stream_from_stream_pool(std::size_t stream_idx) const#

return stream from pool at index

inline rmm::cuda_stream_view get_next_usable_stream() const#

return stream from pool if size > 0, else main stream on current container

inline rmm::cuda_stream_view get_next_usable_stream(std::size_t stream_idx) const#

return stream from pool at index if size > 0, else main stream on current container

Parameters:

stream_idx – [in] the required index of the stream in the stream pool if available

inline void sync_stream_pool() const#

synchronize the stream pool on the current container

inline void sync_stream_pool(const std::vector<std::size_t> stream_indices) const#

synchronize subset of stream pool

Parameters:

stream_indices – [in] the indices of the streams in the stream pool to synchronize

inline void wait_stream_pool_on_stream() const#

ask stream pool to wait on last event in main stream

Device Resources Manager#

While raft::device_resources provides a convenient way to access device-related resources for a sequence of RAFT calls, it is sometimes useful to be able to limit those resources across an entire application. For instance, in highly multi-threaded applications, it can be helpful to limit the total number of streams rather than relying on the default stream per thread. raft::device_resources_manager offers a way to access raft::device_resources instances that draw from a limited pool of underlying device resources.

#include <raft/core/device_resources_manager.hpp>

namespace raft::core

Warning

doxygenclass: Cannot find class “raft::device_resources_manager” in doxygen xml output for project “RAFT” from directory: ../../cpp/doxygen/_xml/

Resource Functions#

Comms#

#include <raft/core/resource/comms.hpp>

namespace raft::resource

inline bool comms_initialized(resources const &res)#
inline comms::comms_t const &get_comms(resources const &res)#
inline void set_comms(resources const &res, std::shared_ptr<comms::comms_t> communicator)#

cuBLAS Handle#

#include <raft/core/resource/cublas_handle.hpp>

namespace raft::resource

inline cublasHandle_t get_cublas_handle(resources const &res)#

Load a cublasHandle_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cublas handle

cuBLASLt Handle#

#include <raft/core/resource/cublaslt_handle.hpp>

namespace raft::resource

inline cublasLtHandle_t get_cublaslt_handle(resources const &res)#

Load a cublasLtHandle_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cublasLt handle

CUDA Stream#

#include <raft/core/resource/cuda_stream.hpp>

namespace raft::resource

inline rmm::cuda_stream_view get_cuda_stream(resources const &res)#

Load a rmm::cuda_stream_view from a resources instance (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

inline void set_cuda_stream(resources const &res, rmm::cuda_stream_view stream_view)#

Load a rmm::cuda_stream_view from a resources instance (and populate it on the res if needed).

Parameters:

  • res – [in] raft resources object for managing resources

  • stream_view – cuda stream view

inline void sync_stream(const resources &res, rmm::cuda_stream_view stream)#

synchronize a specific stream

Parameters:

  • res – [in] the raft resources object

  • stream – [in] stream to synchronize

inline void sync_stream(const resources &res)#

synchronize main stream on the resources instance

CUDA Stream Pool#

#include <raft/core/resource/cuda_stream_pool.hpp>

namespace raft::resource

inline const rmm::cuda_stream_pool &get_cuda_stream_pool(const resources &res)#

Load a cuda_stream_pool, and create a new one if it doesn’t already exist

Parameters:

res – raft res object for managing resources

Returns:

inline void set_cuda_stream_pool(const resources &res, std::shared_ptr<rmm::cuda_stream_pool> stream_pool)#

Explicitly set a stream pool on the current res. Note that this will overwrite an existing stream pool on the res.

Parameters:

  • res –

  • stream_pool –

inline std::size_t get_stream_pool_size(const resources &res)#
inline rmm::cuda_stream_view get_stream_from_stream_pool(const resources &res)#

return stream from pool

inline rmm::cuda_stream_view get_stream_from_stream_pool(const resources &res, std::size_t stream_idx)#

return stream from pool at index

inline rmm::cuda_stream_view get_next_usable_stream(const resources &res)#

return stream from pool if size > 0, else main stream on res

inline rmm::cuda_stream_view get_next_usable_stream(const resources &res, std::size_t stream_idx)#

return stream from pool at index if size > 0, else main stream on res

Parameters:

  • res – [in] the raft resources object

  • stream_idx – [in] the required index of the stream in the stream pool if available

inline void sync_stream_pool(const resources &res)#

synchronize the stream pool on the res

Parameters:

res – [in] the raft resources object

inline void sync_stream_pool(const resources &res, const std::vector<std::size_t> stream_indices)#

synchronize subset of stream pool

Parameters:

  • res – [in] the raft resources object

  • stream_indices – [in] the indices of the streams in the stream pool to synchronize

inline void wait_stream_pool_on_stream(const resources &res)#

ask stream pool to wait on last event in main stream

Parameters:

res – [in] the raft resources object

cuSolverDn Handle#

#include <raft/core/resource/cusolver_dn_handle.hpp> namespace raft::resource

inline cusolverDnHandle_t get_cusolver_dn_handle(resources const &res)#

Load a cusolverSpres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusolver dn handle

class cusolver_dn_resource_factory : public raft::resource::resource_factory#
#include <cusolver_dn_handle.hpp>

Factory that knows how to construct a specific raft::resource to populate the res_t.

Public Functions

inline virtual resource_type get_resource_type() override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource() override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

cuSolverSp Handle#

#include <raft/core/resource/cusolver_sp_handle.hpp>

namespace raft::resource

inline cusolverSpHandle_t get_cusolver_sp_handle(resources const &res)#

Load a cusolverSpres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusolver sp handle

cuSparse Handle#

#include <raft/core/resource/cusparse_handle.hpp>

namespace raft::resource

inline cusparseHandle_t get_cusparse_handle(resources const &res)#

Load a cusparseres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusparse handle

Device ID#

#include <raft/core/resource/device_id.hpp>

namespace raft::resource

inline int get_device_id(resources const &res)#

Load a device id from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

device id

Device Memory Resource#

#include <raft/core/resource/device_memory_resource.hpp>

namespace raft::resource

inline rmm::mr::limiting_resource_adaptor<rmm::mr::device_memory_resource> *get_workspace_resource(resources const &res)#

Load a temp workspace resource from a resources instance (and populate it on the res if needed).

Parameters:

res – raft resources object for managing resources

Returns:

device memory resource object

inline size_t get_workspace_total_bytes(resources const &res)#

Get the total size of the workspace resource.

inline size_t get_workspace_used_bytes(resources const &res)#

Get the already allocated size of the workspace resource.

inline size_t get_workspace_free_bytes(resources const &res)#

Get the available size of the workspace resource.

inline void set_workspace_resource(resources const &res, std::shared_ptr<rmm::mr::device_memory_resource> mr = {nullptr}, std::optional<std::size_t> allocation_limit = std::nullopt, std::optional<std::size_t> alignment = std::nullopt)#

Set a temporary workspace resource on a resources instance.

Parameters:

  • res – raft resources object for managing resources

  • mr – an optional RMM device_memory_resource

  • allocation_limit – the total amount of memory in bytes available to the temporary workspace resources.

  • alignment – optional alignment requirements passed to RMM allocations

inline void set_workspace_to_pool_resource(resources const &res, std::optional<std::size_t> allocation_limit = std::nullopt)#

Set the temporary workspace resource to a pool on top of the global memory resource (rmm::mr::get_current_device_resource().

Parameters:

  • res – raft resources object for managing resources

  • allocation_limit – the total amount of memory in bytes available to the temporary workspace resources; if not provided, a last used or default limit is used.

inline void set_workspace_to_global_resource(resources const &res, std::optional<std::size_t> allocation_limit = std::nullopt)#

Set the temporary workspace resource the same as the global memory resource (rmm::mr::get_current_device_resource().

Note, the workspace resource is always limited; the limit here defines how much of the global memory resource can be consumed by the workspace allocations.

Parameters:

  • res – raft resources object for managing resources

  • allocation_limit – the total amount of memory in bytes available to the temporary workspace resources.

inline rmm::mr::device_memory_resource *get_large_workspace_resource(resources const &res)#
inline void set_large_workspace_resource(resources const &res, std::shared_ptr<rmm::mr::device_memory_resource> mr = {nullptr})#
class device_memory_resource : public raft::resource::resource#
#include <device_memory_resource.hpp>
class limiting_memory_resource : public raft::resource::resource#
#include <device_memory_resource.hpp>
class large_workspace_resource_factory : public raft::resource::resource_factory#
#include <device_memory_resource.hpp>

Factory that knows how to construct a specific raft::resource to populate the resources instance.

Public Functions

inline virtual resource_type override get_resource_type ()

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *override make_resource ()

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

class workspace_resource_factory : public raft::resource::resource_factory#
#include <device_memory_resource.hpp>

Factory that knows how to construct a specific raft::resource to populate the resources instance.

Public Functions

inline virtual resource_type override get_resource_type ()

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *override make_resource ()

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Public Static Functions

static inline std::shared_ptr<rmm::mr::device_memory_resource> default_pool_resource(std::size_t limit)#

Construct a sensible default pool memory resource.

static inline std::shared_ptr<rmm::mr::device_memory_resource> default_plain_resource()#

Get the global memory resource wrapped into an unmanaged shared_ptr (with no deleter).

Note: the lifetime of the underlying rmm::mr::get_current_device_resource() is managed somewhere else, since it’s passed by a raw pointer. Hence, this shared_ptr wrapper is not allowed to delete the pointer on destruction.

Device Properties#

#include <raft/core/resource/device_properties.hpp>

namespace raft::resource

inline cudaDeviceProp &get_device_properties(resources const &res)#

Load a cudaDeviceProp from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

populated cuda device properties instance

class device_properties_resource_factory : public raft::resource::resource_factory#
#include <device_properties.hpp>

Factory that knows how to construct a specific raft::resource to populate the res_t.

Public Functions

inline virtual resource_type get_resource_type() override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource() override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Sub Communicators#

#include <raft/core/resource/sub_comms.hpp>

namespace raft::resource

inline const comms::comms_t &get_subcomm(const resources &res, std::string key)#
inline void set_subcomm(resources const &res, std::string key, std::shared_ptr<comms::comms_t> subcomm)#

Thrust Exec Policy#

#include <raft/core/resource/thrust_policy.hpp>

namespace raft::resource

inline rmm::exec_policy_nosync &get_thrust_policy(resources const &res)#

Load a thrust policy from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

thrust execution policy

Custom runtime-shared resources#

A custom resource is an arbitrary default-constructible C++ class. The consumer of the API can keep such a resource in the raft::resources handle. For example, consider a function that is expected to be called repeatedly and involves a costly kernel configuration. One can cache the kernel configuration in a custom resource. The cost of accessing it is one hashmap lookup.

#include <raft/core/resource/custom_resource.hpp>

namespace raft::resource

template<typename ResourceT>
ResourceT *get_custom_resource(resources const &res)#

Get the custom default-constructible resource if it exists, create it otherwise.

Note: in contrast to the other, hard-coded resources, there’s no information about the custom resources at compile time. Hence, custom resources are kept in a hashmap and looked-up at runtime. This leads to slightly slower access times.

Template Parameters:

ResourceT – the type of the resource; it must be complete and default-constructible.

Parameters:

res – [in] the raft resources object

Returns:

a pointer to the custom resource.