Memory Resource Adaptors

using failure_callback_t = std::function<bool(std::size_t, void*)>

Callback function type used by failure_callback_resource_adaptor.

The resource adaptor calls this function when a memory allocation throws a specified exception type. The function decides whether the resource adaptor should try to allocate the memory again or re-throw the exception.

The callback function signature is: bool failure_callback_t(std::size_t bytes, void* callback_arg)

The callback function is passed two parameters: bytes is the size of the failed memory allocation and arg is the extra argument passed to the constructor of the failure_callback_resource_adaptor. The callback function returns a bool where true means to retry the memory allocation and false means to re-throw the exception.

class aligned_resource_adaptor : public cuda::mr::shared_resource<detail::aligned_resource_adaptor_impl>

#include <aligned_resource_adaptor.hpp>

Resource that adapts an upstream resource to allocate memory with a specified alignment.

If the requested alignment is smaller than CUDA_ALLOCATION_ALIGNMENT (256 bytes) it is increased to CUDA_ALLOCATION_ALIGNMENT. An optional threshold controls the minimum size above which the custom alignment is applied.

This class is copyable and shares ownership of its internal state via cuda::mr::shared_resource.

Public Functions

explicit aligned_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT, std::size_t alignment_threshold = default_alignment_threshold)

Construct an aligned resource adaptor using upstream to satisfy allocation requests.

Throws:

rmm::logic_error – if alignment is not a power of 2

Parameters:

• upstream – The resource used for allocating/deallocating device memory.

• alignment – The size used for allocation alignment (raised to CUDA_ALLOCATION_ALIGNMENT if smaller).

• alignment_threshold – Only allocations >= this size are aligned to alignment.

~aligned_resource_adaptor() = default

device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

Public Static Attributes

static constexpr std::size_t default_alignment_threshold = detail::aligned_resource_adaptor_impl::default_alignment_threshold

The default alignment threshold used by the adaptor (0 = always align).

Friends

inline friend void get_property(aligned_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

template<typename ExceptionType = rmm::out_of_memory>
class failure_callback_resource_adaptor : public cuda::mr::shared_resource<detail::failure_callback_resource_adaptor_impl<rmm::out_of_memory>>

#include <failure_callback_resource_adaptor.hpp>

A device memory resource that calls a callback function when allocations throw a specified exception type.

An instance of this resource must be constructed with an existing, upstream resource in order to satisfy allocation requests.

The callback function takes an allocation size and a callback argument and returns a bool representing whether to retry the allocation (true) or re-throw the caught exception (false).

This class is copyable and shares ownership of its internal state via cuda::mr::shared_resource.

Template Parameters:

ExceptionType – The type of exception that this adaptor should respond to

Public Types

using exception_type = ExceptionType

The type of exception this object catches/throws.

Public Functions

inline failure_callback_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream, failure_callback_t callback, void *callback_arg)

Construct a new failure_callback_resource_adaptor using upstream to satisfy allocation requests.

See also

failure_callback_t

Parameters:

• upstream – The resource used for allocating/deallocating device memory

• callback – Callback function

• callback_arg – Extra argument passed to callback

~failure_callback_resource_adaptor() = default

inline device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

Friends

inline friend void get_property(failure_callback_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

class limiting_resource_adaptor : public cuda::mr::shared_resource<detail::limiting_resource_adaptor_impl>

#include <limiting_resource_adaptor.hpp>

Resource that uses an upstream resource to allocate memory and limits the total allocations possible.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests, but any existing allocations will be untracked. Atomics are used to make this thread-safe, but note that the get_allocated_bytes may not include in-flight allocations.

This class is copyable and shares ownership of its internal state via cuda::mr::shared_resource.

Public Functions

limiting_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream, std::size_t allocation_limit, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT)

Construct a new limiting resource adaptor using upstream to satisfy allocation requests and limiting the total allocation amount possible.

Parameters:

• upstream – The resource used for allocating/deallocating device memory

• allocation_limit – Maximum memory allowed for this allocator

• alignment – Alignment in bytes for the start of each allocated buffer

~limiting_resource_adaptor() = default

device_async_resource_ref get_upstream_resource() const noexcept

device_async_resource_ref to the upstream resource

Returns:

device_async_resource_ref to the upstream resource

std::size_t get_allocated_bytes() const

Query the number of bytes that have been allocated. Note that this can not be used to know how large of an allocation is possible due to both possible fragmentation and also internal page sizes and alignment that is not tracked by this allocator.

Returns:

std::size_t number of bytes that have been allocated through this allocator.

std::size_t get_allocation_limit() const

Query the maximum number of bytes that this allocator is allowed to allocate. This is the limit on the allocator and not a representation of the underlying device. The device may not be able to support this limit.

Returns:

std::size_t max number of bytes allowed for this allocator

Friends

inline friend void get_property(limiting_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

class logging_resource_adaptor : public cuda::mr::shared_resource<detail::logging_resource_adaptor_impl>

#include <logging_resource_adaptor.hpp>

Resource that uses an upstream resource to allocate memory and logs information about the requested allocation/deallocations.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests and log allocation/deallocation activity.

This class is copyable and shares ownership of its internal state, allowing multiple instances to safely reference the same underlying resource and logger.

Public Functions

logging_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream, std::string const &filename = get_default_filename(), bool auto_flush = false)

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the file specified by filename.

The logfile will be written using CSV formatting.

Clears the contents of filename if it already exists.

Creating multiple logging_resource_adaptors with the same filename will result in undefined behavior.

Throws:

spdlog::spdlog_ex – if opening filename failed

Parameters:

• upstream – The resource_ref used for allocating/deallocating device memory.

• filename – Name of file to write log info. If not specified, retrieves the file name from the environment variable “RMM_LOG_FILE”.

• auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

logging_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream, std::ostream &stream, bool auto_flush = false)

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the ostream specified by stream.

The logfile will be written using CSV formatting.

Parameters:

• upstream – The resource_ref used for allocating/deallocating device memory.

• stream – The ostream to write log info.

• auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

logging_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream, std::initializer_list<rapids_logger::sink_ptr> sinks, bool auto_flush = false)

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the sinks specified.

The logfile will be written using CSV formatting.

Parameters:

• upstream – The resource_ref used for allocating/deallocating device memory.

• sinks – A list of logging sinks to which log output will be written.

• auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

rmm::device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

void flush()

Flush logger contents.

std::string header() const

Return the CSV header string.

Returns:

CSV formatted header string of column names

Public Static Functions

static std::string get_default_filename()

Return the value of the environment variable RMM_LOG_FILE.

Throws:

rmm::logic_error – if RMM_LOG_FILE is not set.

Returns:

The value of RMM_LOG_FILE as std::string.

Friends

inline friend void get_property(logging_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

This property declares that a logging_resource_adaptor provides device accessible memory

class prefetch_resource_adaptor : public cuda::mr::shared_resource<detail::prefetch_resource_adaptor_impl>

#include <prefetch_resource_adaptor.hpp>

Resource that prefetches all memory allocations.

This class is copyable and shares ownership of its internal state via cuda::mr::shared_resource.

Public Functions

explicit prefetch_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream)

Construct a new prefetch resource adaptor using upstream to satisfy allocation requests.

Parameters:

upstream – The resource_ref used for allocating/deallocating device memory

~prefetch_resource_adaptor() = default

device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

Friends

inline friend void get_property(prefetch_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

class statistics_resource_adaptor : public cuda::mr::shared_resource<detail::statistics_resource_adaptor_impl>

#include <statistics_resource_adaptor.hpp>

Resource that uses an upstream resource to allocate memory and tracks allocation statistics (current, peak, total bytes and allocation counts).

Supports nested statistics via push_counters()/pop_counters(). Intended as a debug adaptor.

This class is copyable and shares ownership of its internal state via cuda::mr::shared_resource.

Public Types

using counter = detail::statistics_resource_adaptor_impl::counter

Counter type tracking current, peak, and total bytes or allocations.

using read_lock_t = detail::statistics_resource_adaptor_impl::read_lock_t

Shared-reader lock type used to protect the counter stack.

using write_lock_t = detail::statistics_resource_adaptor_impl::write_lock_t

Exclusive-writer lock type used to protect the counter stack.

Public Functions

explicit statistics_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream)

Construct a statistics resource adaptor using upstream to satisfy allocation requests.

Parameters:

upstream – The resource used for allocating/deallocating device memory.

~statistics_resource_adaptor() = default

device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

counter get_bytes_counter() const noexcept

Returns a counter struct for bytes allocated since construction (or last push).

Returns:

counter containing current, peak, and total byte counts

counter get_allocations_counter() const noexcept

Returns a counter struct for number of allocations since construction (or last push).

Returns:

counter containing current, peak, and total allocation counts

std::pair<counter, counter> push_counters()

Push a pair of zero counters — new counters start fresh.

Returns:

pair of counters (bytes, allocations) from the top before the push

std::pair<counter, counter> pop_counters()

Pop a pair of counters from the stack.

Throws:

std::out_of_range – if the counter stack has fewer than two entries

Returns:

pair of counters (bytes, allocations) from the top before the pop

Friends

inline friend void get_property(statistics_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

class thread_safe_resource_adaptor : public cuda::mr::shared_resource<detail::thread_safe_resource_adaptor_impl>

#include <thread_safe_resource_adaptor.hpp>

Resource that adapts an upstream resource to be thread safe.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests. This adaptor wraps allocations and deallocations from the upstream in a mutex lock.

This class is copyable and shares ownership of its internal state via cuda::mr::shared_resource.

Public Types

using lock_t = std::lock_guard<std::mutex>

Type of lock used to synchronize access.

Public Functions

explicit thread_safe_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream)

Construct a new thread safe resource adaptor using upstream to satisfy allocation requests.

Parameters:

upstream – The resource used for allocating/deallocating device memory.

~thread_safe_resource_adaptor() = default

device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

Friends

inline friend void get_property(thread_safe_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

template<typename T>
class thrust_allocator : public thrust::device_malloc_allocator<T>

#include <thrust_allocator_adaptor.hpp>

An allocator compatible with Thrust containers and algorithms using a device_async_resource_ref for memory (de)allocation.

Unlike a device_async_resource_ref, thrust_allocator is typed and bound to allocate objects of a specific type T, but can be freely rebound to other types.

The allocator records the current CUDA device and may only be used with a backing device_async_resource_ref valid for the same device.

Template Parameters:

T – The type of the objects that will be allocated by this allocator

Public Types

using Base = thrust::device_malloc_allocator<T>

The base type of this allocator.

using pointer = typename Base::pointer

The pointer type.

using size_type = typename Base::size_type

The size type.

Public Functions

inline thrust_allocator()

Default constructor creates an allocator using the default memory resource and default stream.

inline explicit thrust_allocator(cuda_stream_view stream)

Constructs a thrust_allocator using the default device memory resource and specified stream.

Parameters:

stream – The stream to be used for device memory (de)allocation

inline thrust_allocator(cuda_stream_view stream, cuda::mr::any_resource<cuda::mr::device_accessible> mr)

Constructs a thrust_allocator using a device memory resource and stream.

Parameters:

• mr – The resource to be used for device memory allocation

• stream – The stream to be used for device memory (de)allocation

inline thrust_allocator(thrust_allocator const &other)

Copy constructor. Copies the resource pointer and stream.

Parameters:

other – The thrust_allocator to copy

inline thrust_allocator(thrust_allocator &&other) noexcept

Move constructor. Moves the resource pointer and stream.

Parameters:

other – The thrust_allocator to move from

thrust_allocator &operator=(thrust_allocator const&) = default

Default copy assignment operator.

Returns:

thrust_allocator& Reference to the assigned object

thrust_allocator &operator=(thrust_allocator&&) noexcept = default

Default move assignment operator.

Returns:

thrust_allocator& Reference to the assigned object

template<typename U>
inline thrust_allocator(thrust_allocator<U> const &other)

Copy constructor from a thrust_allocator of a different type. Copies the resource pointer and stream.

Parameters:

other – The thrust_allocator to copy

inline pointer allocate(size_type num)

Allocate objects of type T

Parameters:

num – The number of elements of type T to allocate

Returns:

pointer Pointer to the newly allocated storage

inline void deallocate(pointer ptr, size_type num) noexcept

Deallocates objects of type T

Parameters:

• ptr – Pointer returned by a previous call to allocate

• num – number of elements, must be equal to the argument passed to the prior allocate call that produced ptr

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

inline cuda_stream_view stream() const noexcept

The stream used by this allocator.

Returns:

The stream used by this allocator

Friends

inline friend void get_property(thrust_allocator const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.

This property declares that a thrust_allocator provides device accessible memory

template<typename U>
struct rebind

#include <thrust_allocator_adaptor.hpp>

Provides the type of a thrust_allocator instantiated with another type.

Template Parameters:

U – the other type to use for instantiation

Public Types

using other = thrust_allocator<U>

The type to bind to.

class tracking_resource_adaptor : public cuda::mr::shared_resource<detail::tracking_resource_adaptor_impl>

#include <tracking_resource_adaptor.hpp>

Resource that uses an upstream resource to allocate memory and tracks allocations.

Tracks every allocation (size, pointer, and optionally stack trace). Intended as a debug adaptor; should not be used in performance-sensitive code.

This class is copyable and shares ownership of its internal state via cuda::mr::shared_resource.

Public Types

using allocation_info = detail::tracking_resource_adaptor_impl::allocation_info

Allocation info type (pointer, size, optional stack trace).

using read_lock_t = detail::tracking_resource_adaptor_impl::read_lock_t

Shared-reader lock type used to protect the allocations map.

using write_lock_t = detail::tracking_resource_adaptor_impl::write_lock_t

Exclusive-writer lock type used to protect the allocations map.

Public Functions

tracking_resource_adaptor(cuda::mr::any_resource<cuda::mr::device_accessible> upstream, bool capture_stacks = false)

Construct a tracking resource adaptor using upstream to satisfy allocation requests.

Parameters:

• upstream – The resource used for allocating/deallocating device memory.

• capture_stacks – If true, capture stacks for each allocation.

~tracking_resource_adaptor() = default

device_async_resource_ref get_upstream_resource() const noexcept

rmm::device_async_resource_ref to the upstream resource

Returns:

rmm::device_async_resource_ref to the upstream resource

std::map<void*, allocation_info> const &get_outstanding_allocations() const noexcept

Get the outstanding allocations map.

Returns:

map of outstanding allocations (pointer → allocation_info)

std::size_t get_allocated_bytes() const noexcept

Query the number of bytes currently allocated.

Returns:

std::size_t number of bytes currently allocated

std::string get_outstanding_allocations_str() const

Gets a string describing all outstanding allocations (pointer, size, optional stack).

Returns:

std::string describing outstanding allocations

void log_outstanding_allocations() const

Log any outstanding allocations via RMM_LOG_DEBUG.

Friends

inline friend void get_property(tracking_resource_adaptor const&, cuda::mr::device_accessible) noexcept

Enables the cuda::mr::device_accessible property.