Memory Resource Adaptors#

using failure_callback_t = std::function<bool(std::size_t, void*)>#

Callback function type used by failure_callback_resource_adaptor.

The resource adaptor calls this function when a memory allocation throws a specified exception type. The function decides whether the resource adaptor should try to allocate the memory again or re-throw the exception.

The callback function signature is: bool failure_callback_t(std::size_t bytes, void* callback_arg)

The callback function is passed two parameters: bytes is the size of the failed memory allocation and arg is the extra argument passed to the constructor of the failure_callback_resource_adaptor. The callback function returns a Boolean where true means to retry the memory allocation and false means to re-throw the exception.

template<template<typename...> class Resource, typename ...Upstreams, typename ...Args> auto make_owning_wrapper(std::tuple<std::shared_ptr<Upstreams>...> upstreams, Args&&... args)#

Constructs a resource of type Resource wrapped in an owning_wrapper using upstreams as the upstream resources and args as the additional parameters for the constructor of Resource.

template <typename Upstream1, typename Upstream2>
class example_resource{
  example_resource(Upstream1 * u1, Upstream2 * u2, int n, float f);
};

auto cuda_mr = std::make_shared<rmm::mr::cuda_memory_resource>();
auto cuda_upstreams = std::make_tuple(cuda_mr, cuda_mr);

// Constructs an `example_resource<rmm::mr::cuda_memory_resource, rmm::mr::cuda_memory_resource>`
// wrapped by an `owning_wrapper` taking shared ownership of `cuda_mr` and using it as both of
// `example_resource`s upstream resources. Forwards the  arguments `42` and `3.14` to the
// additional `n` and `f` arguments of `example_resource` constructor.
auto wrapped_example = rmm::mr::make_owning_wrapper<example_resource>(cuda_upstreams, 42, 3.14);

Template Parameters:

Resource – Template template parameter specifying the type of the wrapped resource to construct
Upstreams – Types of the upstream resources
Args – Types of the arguments used in Resources constructor

Parameters:

upstreams – Tuple of std::shared_ptrs to the upstreams used by the wrapped resource, in the same order as expected by Resources constructor.
args – Function parameter pack of arguments to forward to the wrapped resource’s constructor

Returns:

An owning_wrapper wrapping a newly constructed Resource<Upstreams...> and upstreams.

template<template<typename> class Resource, typename Upstream, typename ...Args> auto make_owning_wrapper(std::shared_ptr<Upstream> upstream, Args&&... args)#

Additional convenience factory for owning_wrapper when Resource has only a single upstream resource.

When a resource has only a single upstream, it can be inconvenient to construct a std::tuple of the upstream resource. This factory allows specifying the single upstream as just a std::shared_ptr.

Template Parameters:

Resource – Type of the wrapped resource to construct
Upstream – Type of the single upstream resource
Args – Types of the arguments used in Resources constructor

Parameters:

upstream – std::shared_ptr to the upstream resource
args – Function parameter pack of arguments to forward to the wrapped resource’s constructor

Returns:

An owning_wrapper wrapping a newly construct Resource<Upstream> and upstream.

template<typename Upstream> class aligned_resource_adaptor : public rmm::mr::device_memory_resource #

#include <aligned_resource_adaptor.hpp>

Resource that adapts Upstream memory resource to allocate memory in a specified alignment size.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests. This adaptor wraps allocations and deallocations from Upstream using the given alignment size.

By default, any address returned by one of the memory allocation routines from the CUDA driver or runtime API is always aligned to at least 256 bytes. For some use cases, such as GPUDirect Storage (GDS), allocations need to be aligned to a larger size (4 KiB for GDS) in order to avoid additional copies to bounce buffers.

If the requested alignment is smaller than CUDA_ALLOCATION_ALIGNMENT (256 bytes), the alignment is increased to CUDA_ALLOCATION_ALIGNMENT. This is to ensure that the allocation is always aligned to at least the alignment required for CUDA allocations.

Since a larger alignment size has some additional overhead, the user can specify a threshold size. If an allocation’s size falls below the threshold, it is aligned to the default size. Only allocations with a size above the threshold are aligned to the custom alignment size.

Template Parameters:: Upstream – Type of the upstream resource used for allocation/deallocation.

Public Functions

inline explicit aligned_resource_adaptor(device_async_resource_ref upstream, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT, std::size_t alignment_threshold = default_alignment_threshold)#

Construct an aligned resource adaptor using upstream to satisfy allocation requests.

Throws:

rmm::logic_error – if allocation_alignment is not a power of 2

Parameters:

upstream – The resource used for allocating/deallocating device memory.
alignment – The size used for allocation alignment. Values smaller than CUDA_ALLOCATION_ALIGNMENT are increased to CUDA_ALLOCATION_ALIGNMENT.
alignment_threshold – Only allocations with a size larger than or equal to this threshold are aligned.

inline explicit aligned_resource_adaptor(Upstream *upstream, std::size_t alignment = rmm::CUDA_ALLOCATION_ALIGNMENT, std::size_t alignment_threshold = default_alignment_threshold)#

Construct an aligned resource adaptor using upstream to satisfy allocation requests.

Throws:

rmm::logic_error – if upstream == nullptr
rmm::logic_error – if alignment is not a power of 2

Parameters:

upstream – The resource used for allocating/deallocating device memory.
alignment – The size used for allocation alignment. Values smaller than CUDA_ALLOCATION_ALIGNMENT are increased to CUDA_ALLOCATION_ALIGNMENT.
alignment_threshold – Only allocations with a size larger than or equal to this threshold are aligned.

aligned_resource_adaptor() = delete#

~aligned_resource_adaptor() override = default#

aligned_resource_adaptor(aligned_resource_adaptor const&) = delete#

aligned_resource_adaptor(aligned_resource_adaptor&&) = delete#

aligned_resource_adaptor &operator=(aligned_resource_adaptor const&) = delete#

aligned_resource_adaptor &operator=(aligned_resource_adaptor&&) = delete#

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

Public Static Attributes

static constexpr std::size_t default_alignment_threshold = 0#: The default alignment used by the adaptor.

template<typename Upstream, typename ExceptionType = rmm::out_of_memory> class failure_callback_resource_adaptor : public rmm::mr::device_memory_resource #

#include <failure_callback_resource_adaptor.hpp>

A device memory resource that calls a callback function when allocations throw a specified exception type.

An instance of this resource must be constructed with an existing, upstream resource in order to satisfy allocation requests.

The callback function takes an allocation size and a callback argument and returns a bool representing whether to retry the allocation (true) or re-throw the caught exception (false).

When implementing a callback function for allocation retry, care must be taken to avoid an infinite loop. The following example makes sure to only retry the allocation once:

using failure_callback_adaptor =
  rmm::mr::failure_callback_resource_adaptor<rmm::mr::device_memory_resource>;

bool failure_handler(std::size_t bytes, void* arg)
{
  bool& retried = *reinterpret_cast<bool*>(arg);
  if (!retried) {
    retried = true;
    return true;  // First time we request an allocation retry
  }
  return false;  // Second time we let the adaptor throw std::bad_alloc
}

int main()
{
  bool retried{false};
  failure_callback_adaptor mr{
    rmm::mr::get_current_device_resource_ref(), failure_handler, &retried
  };
  rmm::mr::set_current_device_resource_ref(mr);
}

Template Parameters:

Upstream – The type of the upstream resource used for allocation/deallocation.
ExceptionType – The type of exception that this adaptor should respond to

Public Types

using exception_type = ExceptionType #: The type of exception this object catches/throws.

Public Functions

inline failure_callback_resource_adaptor(device_async_resource_ref upstream, failure_callback_t callback, void *callback_arg)#

Construct a new failure_callback_resource_adaptor using upstream to satisfy allocation requests.

See also

failure_callback_t

Throws:

rmm::logic_error – if upstream == nullptr

Parameters:

upstream – The resource used for allocating/deallocating device memory
callback – Callback function
callback_arg – Extra argument passed to callback

failure_callback_resource_adaptor() = delete#

~failure_callback_resource_adaptor() override = default#

failure_callback_resource_adaptor(failure_callback_resource_adaptor const&) = delete#

failure_callback_resource_adaptor &operator=(failure_callback_resource_adaptor const&) = delete#

failure_callback_resource_adaptor(failure_callback_resource_adaptor&&) noexcept = default#: Default move constructor.

failure_callback_resource_adaptor &operator=(failure_callback_resource_adaptor&&) noexcept = default#

Default move assignment operator.

Returns:: failure_callback_resource_adaptor& Reference to the assigned object

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

template<typename Upstream> class limiting_resource_adaptor : public rmm::mr::device_memory_resource #

#include <limiting_resource_adaptor.hpp>

Resource that uses Upstream to allocate memory and limits the total allocations possible.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests, but any existing allocations will be untracked. Atomics are used to make this thread-safe, but note that the get_allocated_bytes may not include in-flight allocations.

Template Parameters:: Upstream – Type of the upstream resource used for allocation/deallocation.

Public Functions

inline limiting_resource_adaptor(device_async_resource_ref upstream, std::size_t allocation_limit, std::size_t alignment = CUDA_ALLOCATION_ALIGNMENT)#

Construct a new limiting resource adaptor using upstream to satisfy allocation requests and limiting the total allocation amount possible.

Parameters:

upstream – The resource used for allocating/deallocating device memory
allocation_limit – Maximum memory allowed for this allocator
alignment – Alignment in bytes for the start of each allocated buffer

inline limiting_resource_adaptor(Upstream *upstream, std::size_t allocation_limit, std::size_t alignment = CUDA_ALLOCATION_ALIGNMENT)#

Construct a new limiting resource adaptor using upstream to satisfy allocation requests and limiting the total allocation amount possible.

Throws:

rmm::logic_error – if upstream == nullptr

Parameters:

upstream – The resource used for allocating/deallocating device memory
allocation_limit – Maximum memory allowed for this allocator
alignment – Alignment in bytes for the start of each allocated buffer

limiting_resource_adaptor() = delete#

~limiting_resource_adaptor() override = default#

limiting_resource_adaptor(limiting_resource_adaptor const&) = delete#

limiting_resource_adaptor(limiting_resource_adaptor&&) noexcept = default#: Default move constructor.

limiting_resource_adaptor &operator=(limiting_resource_adaptor const&) = delete#

limiting_resource_adaptor &operator=(limiting_resource_adaptor&&) noexcept = default#

Default move assignment operator.

Returns:: limiting_resource_adaptor& Reference to the assigned object

inline device_async_resource_ref get_upstream_resource() const noexcept#

device_async_resource_ref to the upstream resource

Returns:: device_async_resource_ref to the upstream resource

inline std::size_t get_allocated_bytes() const#

Query the number of bytes that have been allocated. Note that this can not be used to know how large of an allocation is possible due to both possible fragmentation and also internal page sizes and alignment that is not tracked by this allocator.

Returns:: std::size_t number of bytes that have been allocated through this allocator.

inline std::size_t get_allocation_limit() const#

Query the maximum number of bytes that this allocator is allowed to allocate. This is the limit on the allocator and not a representation of the underlying device. The device may not be able to support this limit.

Returns:: std::size_t max number of bytes allowed for this allocator

template<typename Upstream> class logging_resource_adaptor : public rmm::mr::device_memory_resource #

#include <logging_resource_adaptor.hpp>

Resource that uses Upstream to allocate memory and logs information about the requested allocation/deallocations.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests and log allocation/deallocation activity.

Template Parameters:: Upstream – Type of the upstream resource used for allocation/deallocation.

Public Functions

inline logging_resource_adaptor(Upstream *upstream, std::string const &filename = get_default_filename(), bool auto_flush = false)#

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the file specified by filename.

The logfile will be written using CSV formatting.

Clears the contents of filename if it already exists.

Creating multiple logging_resource_adaptors with the same filename will result in undefined behavior.

Throws:

rmm::logic_error – if upstream == nullptr
spdlog::spdlog_ex – if opening filename failed

Parameters:

upstream – The resource used for allocating/deallocating device memory
filename – Name of file to write log info. If not specified, retrieves the file name from the environment variable “RMM_LOG_FILE”.
auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

inline logging_resource_adaptor(Upstream *upstream, std::ostream &stream, bool auto_flush = false)#

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the ostream specified by stream.

The logfile will be written using CSV formatting.

Throws:

rmm::logic_error – if upstream == nullptr

Parameters:

upstream – The resource used for allocating/deallocating device memory
stream – The ostream to write log info.
auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

inline logging_resource_adaptor(Upstream *upstream, std::initializer_list<rapids_logger::sink_ptr> sinks, bool auto_flush = false)#

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the ostream specified by stream.

The logfile will be written using CSV formatting.

Throws:

rmm::logic_error – if upstream == nullptr

Parameters:

upstream – The resource used for allocating/deallocating device memory
sinks – A list of logging sinks to which log output will be written.
auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

inline logging_resource_adaptor(device_async_resource_ref upstream, std::string const &filename = get_default_filename(), bool auto_flush = false)#

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the file specified by filename.

The logfile will be written using CSV formatting.

Clears the contents of filename if it already exists.

Creating multiple logging_resource_adaptors with the same filename will result in undefined behavior.

Throws:

spdlog::spdlog_ex – if opening filename failed

Parameters:

upstream – The resource_ref used for allocating/deallocating device memory.
filename – Name of file to write log info. If not specified, retrieves the file name from the environment variable “RMM_LOG_FILE”.
auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

inline logging_resource_adaptor(device_async_resource_ref upstream, std::ostream &stream, bool auto_flush = false)#

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the ostream specified by stream.

The logfile will be written using CSV formatting.

Parameters:

upstream – The resource_ref used for allocating/deallocating device memory.
stream – The ostream to write log info.
auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

inline logging_resource_adaptor(device_async_resource_ref upstream, std::initializer_list<rapids_logger::sink_ptr> sinks, bool auto_flush = false)#

Construct a new logging resource adaptor using upstream to satisfy allocation requests and logging information about each allocation/free to the ostream specified by stream.

The logfile will be written using CSV formatting.

Parameters:

upstream – The resource_ref used for allocating/deallocating device memory.
sinks – A list of logging sinks to which log output will be written.
auto_flush – If true, flushes the log for every (de)allocation. Warning, this will degrade performance.

logging_resource_adaptor() = delete#

~logging_resource_adaptor() override = default#

logging_resource_adaptor(logging_resource_adaptor const&) = delete#

logging_resource_adaptor &operator=(logging_resource_adaptor const&) = delete#

logging_resource_adaptor(logging_resource_adaptor&&) noexcept = default#: Default move constructor.

logging_resource_adaptor &operator=(logging_resource_adaptor&&) noexcept = default#

Default move assignment operator.

Returns:: logging_resource_adaptor& Reference to the assigned object

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

inline void flush()#: Flush logger contents.

inline std::string header() const#

Return the CSV header string.

Returns:: CSV formatted header string of column names

Public Static Functions

static inline std::string get_default_filename()#

Return the value of the environment variable RMM_LOG_FILE.

Throws:: rmm::logic_error – if RMM_LOG_FILE is not set.
Returns:: The value of RMM_LOG_FILE as std::string.

template<typename Resource, typename ...Upstreams> class owning_wrapper : public rmm::mr::device_memory_resource #

#include <owning_wrapper.hpp>

Resource adaptor that maintains the lifetime of upstream resources.

Many device_memory_resource derived types allocate memory from another “upstream” resource. E.g., pool_memory_resource allocates its pool from an upstream resource. Typically, a resource does not own its upstream, and therefore it is the user’s responsibility to maintain the lifetime of the upstream resource. This can be inconvenient and error prone, especially for resources with complex upstreams that may themselves also have an upstream.

owning_wrapper simplifies lifetime management of a resource, wrapped, by taking shared ownership of all upstream resources via a std::shared_ptr.

For convenience, it is recommended to use the make_owning_wrapper factory instead of constructing an owning_wrapper directly.

Example:

auto cuda = std::make_shared<rmm::mr::cuda_memory_resource>();
auto pool = rmm::mr::make_owning_wrapper<rmm::mr::pool_memory_resource>(cuda,initial_pool_size,
                                                                        max_pool_size);
// The `cuda` resource will be kept alive for the lifetime of `pool` and automatically be
// destroyed after `pool` is destroyed

Template Parameters:

Resource – Type of the wrapped resource
Upstreams – Template parameter pack of the types of the upstream resources used by Resource

Public Types

using upstream_tuple = std::tuple<std::shared_ptr<Upstreams>...>#: Tuple of upstream memory resources.

Public Functions

template<typename ...Args> inline owning_wrapper(upstream_tuple upstreams, Args&&... args)#

Constructs the wrapped resource using the provided upstreams and any additional arguments forwarded to the wrapped resources constructor.

Resource is required to have a constructor whose first argument(s) are raw pointers to its upstream resources in the same order as upstreams, followed by any additional arguments in the same order as args.

Example:

template <typename Upstream1, typename Upstream2>
class example_resource{
  example_resource(Upstream1 * u1, Upstream2 * u2, int n, float f);
};

using cuda = rmm::mr::cuda_memory_resource;
using example = example_resource<cuda, cuda>;
using wrapped_example = rmm::mr::owning_wrapper<example, cuda, cuda>;
auto cuda_mr = std::make_shared<cuda>();

// Constructs an `example_resource` wrapped by an `owning_wrapper` taking shared ownership of
//`cuda_mr` and using it as both of `example_resource`s upstream resources. Forwards the
// arguments `42` and `3.14` to the additional `n` and `f` arguments of `example_resources`
// constructor.
wrapped_example w{std::make_tuple(cuda_mr,cuda_mr), 42, 3.14};

Template Parameters:

Args – Template parameter pack to forward to the wrapped resource’s constructor

Parameters:

upstreams – Tuple of std::shared_ptrs to the upstreams used by the wrapped resource, in the same order as expected by Resources constructor.
args – Function parameter pack of arguments to forward to the wrapped resource’s constructor

inline Resource const &wrapped() const noexcept#

A constant reference to the wrapped resource.

Returns:: A constant reference to the wrapped resource

inline Resource &wrapped() noexcept#

A reference to the wrapped resource.

Returns:: A reference to the wrapped resource

template<typename Upstream> class prefetch_resource_adaptor : public rmm::mr::device_memory_resource #

#include <prefetch_resource_adaptor.hpp>

Resource that prefetches all memory allocations.

Template Parameters:: Upstream – Type of the upstream resource used for allocation/deallocation.

Public Functions

inline prefetch_resource_adaptor(device_async_resource_ref upstream)#

Construct a new prefetch resource adaptor using upstream to satisfy allocation requests.

Parameters:: upstream – The resource_ref used for allocating/deallocating device memory

inline prefetch_resource_adaptor(Upstream *upstream)#

Construct a new prefetch resource adaptor using upstream to satisfy allocation requests.

Throws:: rmm::logic_error – if upstream == nullptr
Parameters:: upstream – The resource used for allocating/deallocating device memory

prefetch_resource_adaptor() = delete#

~prefetch_resource_adaptor() override = default#

prefetch_resource_adaptor(prefetch_resource_adaptor const&) = delete#

prefetch_resource_adaptor &operator=(prefetch_resource_adaptor const&) = delete#

prefetch_resource_adaptor(prefetch_resource_adaptor&&) noexcept = default#: Default move constructor.

prefetch_resource_adaptor &operator=(prefetch_resource_adaptor&&) noexcept = default#

Default move assignment operator.

Returns:: prefetch_resource_adaptor& Reference to the assigned object

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

template<typename Upstream> class statistics_resource_adaptor : public rmm::mr::device_memory_resource #

#include <statistics_resource_adaptor.hpp>

Resource that uses Upstream to allocate memory and tracks statistics on memory allocations.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests, but any existing allocations will be untracked. Tracking statistics stores the current, peak and total memory allocations for both the number of bytes and number of calls to the memory resource.

This resource supports nested statistics, which makes it possible to track statistics of a code block. Use .push_counters() to start tracking statistics on a code block and use .pop_counters() to stop the tracking. The nested statistics are cascading such that the statistics tracked by a code block include the statistics tracked in all its tracked sub code blocks.

statistics_resource_adaptor is intended as a debug adaptor and shouldn’t be used in performance-sensitive code.

Template Parameters:: Upstream – Type of the upstream resource used for allocation/deallocation.

Public Types

using read_lock_t = std::shared_lock<std::shared_mutex>#: Type of lock used to synchronize read access.

using write_lock_t = std::unique_lock<std::shared_mutex>#: Type of lock used to synchronize write access.

Public Functions

inline statistics_resource_adaptor(device_async_resource_ref upstream)#

Construct a new statistics resource adaptor using upstream to satisfy allocation requests.

Parameters:: upstream – The resource_ref used for allocating/deallocating device memory.

inline statistics_resource_adaptor(Upstream *upstream)#

Construct a new statistics resource adaptor using upstream to satisfy allocation requests.

Throws:: rmm::logic_error – if upstream == nullptr
Parameters:: upstream – The resource used for allocating/deallocating device memory.

statistics_resource_adaptor() = delete#

~statistics_resource_adaptor() override = default#

statistics_resource_adaptor(statistics_resource_adaptor const&) = delete#

statistics_resource_adaptor &operator=(statistics_resource_adaptor const&) = delete#

statistics_resource_adaptor(statistics_resource_adaptor&&) noexcept = default#: Default move constructor.

statistics_resource_adaptor &operator=(statistics_resource_adaptor&&) noexcept = default#

Default move assignment operator.

Returns:: statistics_resource_adaptor& Reference to the assigned object

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

inline counter get_bytes_counter() const noexcept#

Returns a counter struct for this adaptor containing the current, peak, and total number of allocated bytes for this adaptor since it was created.

Returns:: counter struct containing bytes count

inline counter get_allocations_counter() const noexcept#

Returns a counter struct for this adaptor containing the current, peak, and total number of allocation counts for this adaptor since it was created.

Returns:: counter struct containing allocations count

inline std::pair<counter, counter> push_counters()#

Push a pair of zero counters on the stack, which becomes the new counters returned by get_bytes_counter() and get_allocations_counter()

Returns:: top pair of counters <bytes, allocations> from the stack before the push

inline std::pair<counter, counter> pop_counters()#

Pop a pair of counters from the stack.

Throws:: std::out_of_range – if the counter stack has fewer than two entries.
Returns:: top pair of counters <bytes, allocations> from the stack before the pop

struct counter#

#include <statistics_resource_adaptor.hpp>

Utility struct for counting the current, peak, and total value of a number.

Public Functions

inline counter &operator+=(int64_t val)#

Add val to the current value and update the peak value if necessary.

Parameters:: val – Value to add
Returns:: Reference to this object

inline counter &operator-=(int64_t val)#

Subtract val from the current value and update the peak value if necessary.

Parameters:: val – Value to subtract
Returns:: Reference to this object

inline void add_counters_from_tracked_sub_block(const counter &val)#

Add val to the current value and update the peak value if necessary.

When updating the peak value, we assume that val is tracking a code block inside the code block tracked by this. Because nested statistics are cascading, we have to convert val.peak to the peak it would have been if it was part of the statistics tracked by this. We do this by adding the current value that was active when val started tracking such that we get std::max(value + val.peak, peak).

Parameters:: val – Value to add

Public Members

int64_t value = {0}#: Current value.

int64_t peak = {0}#: Max value of value

int64_t total = {0}#: Sum of all added values.

template<typename Upstream> class thread_safe_resource_adaptor : public rmm::mr::device_memory_resource #

#include <thread_safe_resource_adaptor.hpp>

Resource that adapts Upstream memory resource adaptor to be thread safe.

An instance of this resource can be constructured with an existing, upstream resource in order to satisfy allocation requests. This adaptor wraps allocations and deallocations from Upstream in a mutex lock.

Template Parameters:: Upstream – Type of the upstream resource used for allocation/deallocation.

Public Types

using lock_t = std::lock_guard<std::mutex>#: Type of lock used to synchronize access.

Public Functions

inline thread_safe_resource_adaptor(device_async_resource_ref upstream)#

Construct a new thread safe resource adaptor using upstream to satisfy allocation requests.

All allocations and frees are protected by a mutex lock

Parameters:: upstream – The resource used for allocating/deallocating device memory.

inline thread_safe_resource_adaptor(Upstream *upstream)#

Construct a new thread safe resource adaptor using upstream to satisfy allocation requests.

All allocations and frees are protected by a mutex lock

Throws:: rmm::logic_error – if upstream == nullptr
Parameters:: upstream – The resource used for allocating/deallocating device memory.

thread_safe_resource_adaptor() = delete#

~thread_safe_resource_adaptor() override = default#

thread_safe_resource_adaptor(thread_safe_resource_adaptor const&) = delete#

thread_safe_resource_adaptor(thread_safe_resource_adaptor&&) = delete#

thread_safe_resource_adaptor &operator=(thread_safe_resource_adaptor const&) = delete#

thread_safe_resource_adaptor &operator=(thread_safe_resource_adaptor&&) = delete#

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

template<typename T> class thrust_allocator : public thrust::device_malloc_allocator<T>#

#include <thrust_allocator_adaptor.hpp>

An allocator compatible with Thrust containers and algorithms using a device_async_resource_ref for memory (de)allocation.

Unlike a device_async_resource_ref, thrust_allocator is typed and bound to allocate objects of a specific type T, but can be freely rebound to other types.

The allocator records the current CUDA device and may only be used with a backing device_async_resource_ref valid for the same device.

Template Parameters:: T – The type of the objects that will be allocated by this allocator

Public Types

using Base = thrust::device_malloc_allocator<T>#: The base type of this allocator.

using pointer = typename Base::pointer#: The pointer type.

using size_type = typename Base::size_type#: The size type.

Public Functions

thrust_allocator() = default#: Default constructor creates an allocator using the default memory resource and default stream.

inline explicit thrust_allocator(cuda_stream_view stream)#

Constructs a thrust_allocator using the default device memory resource and specified stream.

Parameters:: stream – The stream to be used for device memory (de)allocation

inline thrust_allocator(cuda_stream_view stream, rmm::device_async_resource_ref mr)#

Constructs a thrust_allocator using a device memory resource and stream.

Parameters:

mr – The resource to be used for device memory allocation
stream – The stream to be used for device memory (de)allocation

template<typename U> inline thrust_allocator(thrust_allocator<U> const &other)#

Copy constructor. Copies the resource pointer and stream.

Parameters:: other – The thrust_allocator to copy

inline pointer allocate(size_type num)#

Allocate objects of type T

Parameters:: num – The number of elements of type T to allocate
Returns:: pointer Pointer to the newly allocated storage

inline void deallocate(pointer ptr, size_type num) noexcept#

Deallocates objects of type T

Parameters:

ptr – Pointer returned by a previous call to allocate
num – number of elements, must be equal to the argument passed to the prior allocate call that produced p

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

inline cuda_stream_view stream() const noexcept#

The stream used by this allocator.

Returns:: The stream used by this allocator

Friends

inline friend void get_property(thrust_allocator const&, cuda::mr::device_accessible) noexcept#

Enables the cuda::mr::device_accessible property.

This property declares that a thrust_allocator provides device accessible memory

template<typename U> struct rebind#

#include <thrust_allocator_adaptor.hpp>

Provides the type of a thrust_allocator instantiated with another type.

Template Parameters:: U – the other type to use for instantiation

Public Types

using other = thrust_allocator<U>#: The type to bind to.

template<typename Upstream> class tracking_resource_adaptor : public rmm::mr::device_memory_resource #

#include <tracking_resource_adaptor.hpp>

Resource that uses Upstream to allocate memory and tracks allocations.

An instance of this resource can be constructed with an existing, upstream resource in order to satisfy allocation requests, but any existing allocations will be untracked. Tracking stores a size and pointer for every allocation, and a stack frame if capture_stacks is true, so it can add significant overhead. tracking_resource_adaptor is intended as a debug adaptor and shouldn’t be used in performance-sensitive code. Note that callstacks may not contain all symbols unless the project is linked with -rdynamic. This can be accomplished with add_link_options(-rdynamic) in cmake.

Template Parameters:: Upstream – Type of the upstream resource used for allocation/deallocation.

Public Types

using read_lock_t = std::shared_lock<std::shared_mutex>#: Type of lock used to synchronize read access.

using write_lock_t = std::unique_lock<std::shared_mutex>#: Type of lock used to synchronize write access.

Public Functions

inline tracking_resource_adaptor(device_async_resource_ref upstream, bool capture_stacks = false)#

Construct a new tracking resource adaptor using upstream to satisfy allocation requests.

Parameters:

upstream – The resource used for allocating/deallocating device memory
capture_stacks – If true, capture stacks for allocation calls

inline tracking_resource_adaptor(Upstream *upstream, bool capture_stacks = false)#

Construct a new tracking resource adaptor using upstream to satisfy allocation requests.

Throws:

rmm::logic_error – if upstream == nullptr

Parameters:

upstream – The resource used for allocating/deallocating device memory
capture_stacks – If true, capture stacks for allocation calls

tracking_resource_adaptor() = delete#

~tracking_resource_adaptor() override = default#

tracking_resource_adaptor(tracking_resource_adaptor const&) = delete#

tracking_resource_adaptor(tracking_resource_adaptor&&) noexcept = default#: Default move constructor.

tracking_resource_adaptor &operator=(tracking_resource_adaptor const&) = delete#

tracking_resource_adaptor &operator=(tracking_resource_adaptor&&) noexcept = default#

Default move assignment operator.

Returns:: tracking_resource_adaptor& Reference to the assigned object

inline rmm::device_async_resource_ref get_upstream_resource() const noexcept#

rmm::device_async_resource_ref to the upstream resource

Returns:: rmm::device_async_resource_ref to the upstream resource

inline std::map<void*, allocation_info> const &get_outstanding_allocations() const noexcept#

Get the outstanding allocations map.

Returns:: std::map<void*, allocation_info> const& of a map of allocations. The key is the allocated memory pointer and the data is the allocation_info structure, which contains size and, potentially, stack traces.

inline std::size_t get_allocated_bytes() const noexcept#

Returns:: std::size_t number of bytes that have been allocated through this allocator.

inline std::string get_outstanding_allocations_str() const#

Gets a string containing the outstanding allocation pointers, their size, and optionally the stack trace for when each pointer was allocated.

Stack traces are only included if this resource adaptor was created with capture_stack == true. Otherwise, outstanding allocation pointers will be shown with their size and empty stack traces.

Returns:: std::string Containing the outstanding allocation pointers.

inline void log_outstanding_allocations() const#: Log any outstanding allocations via RMM_LOG_DEBUG.

struct allocation_info#

#include <tracking_resource_adaptor.hpp>

Information stored about an allocation. Includes the size and a stack trace if the tracking_resource_adaptor was initialized to capture stacks.

Public Functions

allocation_info() = delete#

inline allocation_info(std::size_t size, bool capture_stack)#

Construct a new allocation info object.

Parameters:

size – Size of the allocation
capture_stack – If true, capture the stack trace for the allocation

Public Members

std::unique_ptr<rmm::detail::stack_trace> strace#: Stack trace of the allocation.

std::size_t allocation_size#: Size of the allocation.

Memory Resource Adaptors#

This Page