ai.rapids.cudf.nvcomp

Class BatchedCompressor

java.lang.Object

ai.rapids.cudf.nvcomp.BatchedCompressor

Direct Known Subclasses:

BatchedLZ4Compressor, BatchedZstdCompressor

public abstract class BatchedCompressor
extends Object

Multi-buffer compressor

Constructor Summary

Constructors

Constructor and Description
BatchedCompressor(long chunkSize, long maxOutputChunkSize, long maxIntermediateBufferSize) Construct a batched compressor instance

Method Summary

Modifier and Type	Method and Description
protected abstract void	batchedCompressAsync(long devInPtrs, long devInSizes, long chunkSize, long batchSize, long tempPtr, long tempSize, long devOutPtrs, long compressedSizesOutPtr, long stream) Asynchronously compress a batch of buffers.
protected abstract long	batchedCompressGetTempSize(long batchSize, long maxChunkSize) Get the temporary workspace size required to perform compression of an entire batch.
DeviceMemoryBuffer[]	compress(BaseDeviceMemoryBuffer[] origInputs, Cuda.Stream stream) Compress a batch of buffers.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchedCompressor

public BatchedCompressor(long chunkSize,
                         long maxOutputChunkSize,
                         long maxIntermediateBufferSize)

Construct a batched compressor instance

Parameters:

chunkSize - maximum amount of uncompressed data to compress as a single chunk. Inputs larger than this will be compressed in multiple chunks.

maxIntermediateBufferSize - desired maximum size of intermediate device buffers used during compression.

Method Detail

compress

public DeviceMemoryBuffer[] compress(BaseDeviceMemoryBuffer[] origInputs,
                                     Cuda.Stream stream)

Compress a batch of buffers. The input buffers will be closed.

Parameters:

origInputs - buffers to compress

stream - CUDA stream to use

Returns:

compressed buffers corresponding to the input buffers

batchedCompressGetTempSize

protected abstract long batchedCompressGetTempSize(long batchSize,
                                                   long maxChunkSize)

Get the temporary workspace size required to perform compression of an entire batch.

Parameters:

batchSize - number of chunks in the batch

maxChunkSize - maximum size of an uncompressed chunk in bytes

Returns:

The size of required temporary workspace in bytes to compress the batch.

batchedCompressAsync

protected abstract void batchedCompressAsync(long devInPtrs,
                                             long devInSizes,
                                             long chunkSize,
                                             long batchSize,
                                             long tempPtr,
                                             long tempSize,
                                             long devOutPtrs,
                                             long compressedSizesOutPtr,
                                             long stream)

Asynchronously compress a batch of buffers. Note that compressedSizesOutPtr must point to pinned memory for this operation to be asynchronous.

Parameters:

devInPtrs - device address of uncompressed buffer addresses vector

devInSizes - device address of uncompressed buffer sizes vector

chunkSize - maximum size of an uncompressed chunk in bytes

batchSize - number of chunks in the batch

tempPtr - device address of the temporary workspace buffer

tempSize - size of the temporary workspace buffer in bytes

devOutPtrs - device address of output buffer addresses vector

compressedSizesOutPtr - device address where to write the sizes of the compressed data written to the corresponding output buffers. Must point to a buffer with at least 8 bytes of memory per output buffer in the batch.

stream - CUDA stream to use