kvikio::FileHandle Class Reference

Handle of an open file registered with cufile. More...

#include <file_handle.hpp>

Public Member Functions
	FileHandle (const std::string &file_path, const std::string &flags="r", mode_t mode=m644, CompatMode compat_mode=defaults::compat_mode())
	Construct a file handle from a file path. More...
	FileHandle (const FileHandle &)=delete
	FileHandle support move semantic but isn't copyable.
FileHandle &	operator= (FileHandle const &)=delete
	FileHandle (FileHandle &&o) noexcept
FileHandle &	operator= (FileHandle &&o) noexcept
bool	closed () const noexcept
	Whether the file is closed according to its initialization status. More...
void	close () noexcept
	Deregister the file and close the two files.
CUfileHandle_t	handle ()
	Get the underlying cuFile file handle. More...
int	fd () const noexcept
	Get one of the file descriptors. More...
int	fd_open_flags () const
	Get the flags of one of the file descriptors (see open(2)) More...
std::size_t	nbytes () const
	Get the file size. More...
std::size_t	read (void *devPtr_base, std::size_t size, std::size_t file_offset, std::size_t devPtr_offset, bool sync_default_stream=true)
	Reads specified bytes from the file into the device memory. More...
std::size_t	write (const void *devPtr_base, std::size_t size, std::size_t file_offset, std::size_t devPtr_offset, bool sync_default_stream=true)
	Writes specified bytes from the device memory into the file. More...
std::future< std::size_t >	pread (void *buf, std::size_t size, std::size_t file_offset=0, std::size_t task_size=defaults::task_size(), std::size_t gds_threshold=defaults::gds_threshold(), bool sync_default_stream=true)
	Reads specified bytes from the file into the device or host memory in parallel. More...
std::future< std::size_t >	pwrite (const void *buf, std::size_t size, std::size_t file_offset=0, std::size_t task_size=defaults::task_size(), std::size_t gds_threshold=defaults::gds_threshold(), bool sync_default_stream=true)
	Writes specified bytes from device or host memory into the file in parallel. More...
void	read_async (void *devPtr_base, std::size_t *size_p, off_t *file_offset_p, off_t *devPtr_offset_p, ssize_t *bytes_read_p, CUstream stream)
	Reads specified bytes from the file into the device memory asynchronously. More...
StreamFuture	read_async (void *devPtr_base, std::size_t size, off_t file_offset=0, off_t devPtr_offset=0, CUstream stream=nullptr)
	Reads specified bytes from the file into the device memory asynchronously. More...
void	write_async (void *devPtr_base, std::size_t *size_p, off_t *file_offset_p, off_t *devPtr_offset_p, ssize_t *bytes_written_p, CUstream stream)
	Writes specified bytes from the device memory into the file asynchronously. More...
StreamFuture	write_async (void *devPtr_base, std::size_t size, off_t file_offset=0, off_t devPtr_offset=0, CUstream stream=nullptr)
	Writes specified bytes from the device memory into the file asynchronously. More...
bool	is_compat_mode_preferred () const noexcept
	Returns true if the compatibility mode is expected to be ON for this file. More...
bool	is_compat_mode_preferred_for_async () const noexcept
	Returns true if the compatibility mode is expected to be ON for the asynchronous I/O on this file. More...

Static Public Attributes
static constexpr mode_t	m644 = S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH

Detailed Description

Handle of an open file registered with cufile.

In order to utilize cufile and GDS, a file must be registered with cufile.

Definition at line 44 of file file_handle.hpp.

Constructor & Destructor Documentation

FileHandle()

kvikio::FileHandle::FileHandle	(	const std::string &	file_path,
		const std::string &	flags = "r",
		mode_t	mode = m644,
		CompatMode	compat_mode = defaults::compat_mode()
	)

Construct a file handle from a file path.

FileHandle opens the file twice and maintains two file descriptors. One file is opened with the specified flags and the other file is opened with the flags plus the O_DIRECT flag.

Parameters

file_path	File path to the file
flags	Open flags (see also fopen(3)): "r" -> "open for reading (default)" "w" -> "open for writing, truncating the file first" "a" -> "open for writing, appending to the end of file if it exists" "+" -> "open for updating (reading and writing)"
mode	Access modes (see open(2)).
compat_mode	Set KvikIO's compatibility mode for this file.

Member Function Documentation

closed()

bool kvikio::FileHandle::closed ( ) const

inlinenoexcept

Whether the file is closed according to its initialization status.

Returns

Boolean answer.

Definition at line 140 of file file_handle.hpp.

fd()

int kvikio::FileHandle::fd ( ) const

inlinenoexcept

Get one of the file descriptors.

Notice, FileHandle maintains two file descriptors - one opened with the O_DIRECT flag and one without. This function returns one of them but it is unspecified which one.

Returns

File descriptor

Definition at line 184 of file file_handle.hpp.

fd_open_flags()

int kvikio::FileHandle::fd_open_flags

(

)

const

Get the flags of one of the file descriptors (see open(2))

Notice, FileHandle maintains two file descriptors - one opened with the O_DIRECT flag and one without. This function returns the flags of one of them but it is unspecified which one.

Returns

File descriptor

handle()

CUfileHandle_t kvikio::FileHandle::handle ( )

inline

Get the underlying cuFile file handle.

The file handle must be open and not in compatibility mode i.e. both closed() and is_compat_mode_preferred() must be false.

Returns

cuFile's file handle

Definition at line 166 of file file_handle.hpp.

is_compat_mode_preferred()

bool kvikio::FileHandle::is_compat_mode_preferred ( ) const

inlinenoexcept

Returns true if the compatibility mode is expected to be ON for this file.

Compatibility mode can be explicitly enabled in object creation. The mode is also enabled automatically, if file cannot be opened with the O_DIRECT flag, or if the system does not meet the requirements for the cuFile library under the AUTO compatibility mode.

Returns

Boolean answer.

Definition at line 656 of file file_handle.hpp.

is_compat_mode_preferred_for_async()

bool kvikio::FileHandle::is_compat_mode_preferred_for_async ( ) const

inlinenoexcept

Returns true if the compatibility mode is expected to be ON for the asynchronous I/O on this file.

For asynchronous I/O, the compatibility mode can be automatically enabled if the cuFile batch and stream symbols are missing, or if the cuFile configuration file is missing, or if is_compat_mode_preferred() returns true.

Returns

Boolean answer.

Definition at line 671 of file file_handle.hpp.

nbytes()

std::size_t kvikio::FileHandle::nbytes

(

)

const

Get the file size.

The value are cached.

Returns

The number of bytes

pread()

std::future<std::size_t> kvikio::FileHandle::pread ( void *  buf, std::size_t  size, std::size_t  file_offset = 0, std::size_t  task_size = defaults::task_size(), std::size_t  gds_threshold = defaults::gds_threshold(), bool  sync_default_stream = true  )

inline

Reads specified bytes from the file into the device or host memory in parallel.

This API is a parallel async version of .read() that partition the operation into tasks of size task_size for execution in the default thread pool.

In order to improve performance of small buffers, when size < gds_threshold a shortcut that circumvent the threadpool and use the POSIX backend directly is used.

Note

For cuFile reads, the base address of the allocation buf is part of is used. This means that when registering buffers, use the base address of the allocation. This is what memory_register and memory_deregister do automatically.

Parameters

buf	Address to device or host memory.
size	Size in bytes to read.
file_offset	Offset in the file to read from.
task_size	Size of each task in bytes.
gds_threshold	Minimum buffer size to use GDS and the thread pool.
sync_default_stream	Synchronize the CUDA default (null) stream prior to calling cuFile. Contrary to most of the non-async CUDA API, cuFile does not have the semantic of being ordered with respect to other non-cuFile work in the default stream. By enabling sync_default_stream, KvikIO will synchronize the default stream and order the operation with respect to other work in the null stream. When in KvikIO's compatibility mode or when accessing host memory, the operation is always default stream ordered like the rest of the non-async CUDA API. In this case, the value of sync_default_stream is ignored.

Returns

Future that on completion returns the size of bytes that were successfully read.

Definition at line 340 of file file_handle.hpp.

pwrite()

std::future<std::size_t> kvikio::FileHandle::pwrite ( const void *  buf, std::size_t  size, std::size_t  file_offset = 0, std::size_t  task_size = defaults::task_size(), std::size_t  gds_threshold = defaults::gds_threshold(), bool  sync_default_stream = true  )

inline

Writes specified bytes from device or host memory into the file in parallel.

This API is a parallel async version of .write() that partition the operation into tasks of size task_size for execution in the default thread pool.

In order to improve performance of small buffers, when size < gds_threshold a shortcut that circumvent the threadpool and use the POSIX backend directly is used.

Note

For cuFile reads, the base address of the allocation buf is part of is used. This means that when registering buffers, use the base address of the allocation. This is what memory_register and memory_deregister do automatically.

Parameters

buf	Address to device or host memory.
size	Size in bytes to write.
file_offset	Offset in the file to write from.
task_size	Size of each task in bytes.
gds_threshold	Minimum buffer size to use GDS and the thread pool.
sync_default_stream	Synchronize the CUDA default (null) stream prior to calling cuFile. Contrary to most of the non-async CUDA API, cuFile does not have the semantic of being ordered with respect to other non-cuFile work in the default stream. By enabling sync_default_stream, KvikIO will synchronize the default stream and order the operation with respect to other work in the null stream. When in KvikIO's compatibility mode or when accessing host memory, the operation is always default stream ordered like the rest of the non-async CUDA API. In this case, the value of sync_default_stream is ignored.

Returns

Future that on completion returns the size of bytes that were successfully written.

Definition at line 417 of file file_handle.hpp.

read()

std::size_t kvikio::FileHandle::read ( void *  devPtr_base, std::size_t  size, std::size_t  file_offset, std::size_t  devPtr_offset, bool  sync_default_stream = true  )

inline

Reads specified bytes from the file into the device memory.

This API reads the data from the GPU memory to the file at a specified offset and size bytes by using GDS functionality. The API works correctly for unaligned offset and data sizes, although the performance is not on-par with aligned read. This is a synchronous call and will block until the IO is complete.

Note

For the devPtr_offset, if data will be read starting exactly from the devPtr_base that is registered with buffer_register, devPtr_offset should be set to 0. To read starting from an offset in the registered buffer range, the relative offset should be specified in the devPtr_offset, and the devPtr_base must remain set to the base address that was used in the buffer_register call.

Parameters

devPtr_base	Base address of buffer in device memory. For registered buffers, devPtr_base must remain set to the base address used in the buffer_register call.
size	Size in bytes to read.
file_offset	Offset in the file to read from.
devPtr_offset	Offset relative to the devPtr_base pointer to read into. This parameter should be used only with registered buffers.
sync_default_stream	Synchronize the CUDA default (null) stream prior to calling cuFile. Contrary to most of the non-async CUDA API, cuFile does not have the semantic of being ordered with respect to other non-cuFile work in the default stream. By enabling sync_default_stream, KvikIO will synchronize the default stream and order the operation with respect to other work in the null stream. When in KvikIO's compatibility mode or when accessing host memory, the operation is always default stream ordered like the rest of the non-async CUDA API. In this case, the value of sync_default_stream is ignored.

Returns

Size of bytes that were successfully read.

Definition at line 236 of file file_handle.hpp.

read_async() [1/2]

void kvikio::FileHandle::read_async ( void *  devPtr_base, std::size_t *  size_p, off_t *  file_offset_p, off_t *  devPtr_offset_p, ssize_t *  bytes_read_p, CUstream  stream  )

inline

Reads specified bytes from the file into the device memory asynchronously.

This is an asynchronous version of .read(), which will be executed in sequence for the specified stream.

When running CUDA v12.1 or older, this function falls back to use .read() after stream has been synchronized.

The arguments have the same meaning as in .read() but some of them are deferred. That is, the values pointed to by size_p, file_offset_p and devPtr_offset_p will not be evaluated until execution time. Notice, this behavior can be changed using cuFile's cuFileStreamRegister API.

Parameters

devPtr_base	Base address of buffer in device memory. For registered buffers, devPtr_base must remain set to the base address used in the buffer_register call.
size_p	Pointer to size in bytes to read. If the exact size is not known at the time of I/O submission, then you must set it to the maximum possible I/O size for that stream I/O. Later the actual size can be set prior to the stream I/O execution.
file_offset_p	Pointer to offset in the file from which to read. Unless otherwise set using cuFileStreamRegister API, this value will not be evaluated until execution time.
devPtr_offset_p	Pointer to the offset relative to the bufPtr_base from which to write. Unless otherwise set using cuFileStreamRegister API, this value will not be evaluated until execution time.
bytes_read_p	Pointer to the bytes read from file. This pointer should be a non-NULL value and *bytes_read_p set to 0. The bytes_read_p memory should be allocated with cuMemHostAlloc/malloc/mmap or registered with cuMemHostRegister. After successful execution of the operation in the stream, the value *bytes_read_p will contain either: The number of bytes successfully read. -1 on IO errors. All other errors return a negative integer value of the CUfileOpError enum value.
stream	CUDA stream in which to enqueue the operation. If NULL, make this operation synchronous.

Definition at line 502 of file file_handle.hpp.

read_async() [2/2]

StreamFuture kvikio::FileHandle::read_async ( void *  devPtr_base, std::size_t  size, off_t  file_offset = 0, off_t  devPtr_offset = 0, CUstream  stream = nullptr  )

inline

Reads specified bytes from the file into the device memory asynchronously.

This is an asynchronous version of .read(), which will be executed in sequence for the specified stream.

When running CUDA v12.1 or older, this function falls back to use .read() after stream has been synchronized.

The arguments have the same meaning as in .read() but returns a StreamFuture object that the caller must keep alive until all data has been read from disk. One way to do this, is by calling StreamFuture.check_bytes_done(), which will synchronize the associated stream and return the number of bytes read.

Parameters

devPtr_base	Base address of buffer in device memory. For registered buffers, devPtr_base must remain set to the base address used in the buffer_register call.
size	Size in bytes to read.
file_offset	Offset in the file to read from.
devPtr_offset	Offset relative to the devPtr_base pointer to read into. This parameter should be used only with registered buffers.
stream	CUDA stream in which to enqueue the operation. If NULL, make this operation synchronous.

Returns

A future object that must be kept alive until all data has been read to disk e.g. by synchronizing stream.

Definition at line 544 of file file_handle.hpp.

write()

std::size_t kvikio::FileHandle::write ( const void *  devPtr_base, std::size_t  size, std::size_t  file_offset, std::size_t  devPtr_offset, bool  sync_default_stream = true  )

inline

Writes specified bytes from the device memory into the file.

This API writes the data from the GPU memory to the file at a specified offset and size bytes by using GDS functionality. The API works correctly for unaligned offset and data sizes, although the performance is not on-par with aligned writes. This is a synchronous call and will block until the IO is complete.

Note

GDS functionality modified the standard file system metadata in SysMem. However, GDS functionality does not take any special responsibility for writing that metadata back to permanent storage. The data is not guaranteed to be present after a system crash unless the application uses an explicit fsync(2) call. If the file is opened with an O_SYNC flag, the metadata will be written to the disk before the call is complete. Refer to the note in read for more information about devPtr_offset.

Parameters

devPtr_base	Base address of buffer in device memory. For registered buffers, devPtr_base must remain set to the base address used in the buffer_register call.
size	Size in bytes to write.
file_offset	Offset in the file to write from.
devPtr_offset	Offset relative to the devPtr_base pointer to write from. This parameter should be used only with registered buffers.
sync_default_stream	Synchronize the CUDA default (null) stream prior to calling cuFile. Contrary to most of the non-async CUDA API, cuFile does not have the semantic of being ordered with respect to other non-cuFile work in the default stream. By enabling sync_default_stream, KvikIO will synchronize the default stream and order the operation with respect to other work in the null stream. When in KvikIO's compatibility mode or when accessing host memory, the operation is always default stream ordered like the rest of the non-async CUDA API. In this case, the value of sync_default_stream is ignored.

Returns

Size of bytes that were successfully written.

Definition at line 286 of file file_handle.hpp.

write_async() [1/2]

void kvikio::FileHandle::write_async ( void *  devPtr_base, std::size_t *  size_p, off_t *  file_offset_p, off_t *  devPtr_offset_p, ssize_t *  bytes_written_p, CUstream  stream  )

inline

Writes specified bytes from the device memory into the file asynchronously.

This is an asynchronous version of .write(), which will be executed in sequence for the specified stream.

When running CUDA v12.1 or older, this function falls back to use .read() after stream has been synchronized.

The arguments have the same meaning as in .write() but some of them are deferred. That is, the values pointed to by size_p, file_offset_p and devPtr_offset_p will not be evaluated until execution time. Notice, this behavior can be changed using cuFile's cuFileStreamRegister API.

Parameters

devPtr_base	Base address of buffer in device memory. For registered buffers, devPtr_base must remain set to the base address used in the buffer_register call.
size_p	Pointer to size in bytes to read. If the exact size is not known at the time of I/O submission, then you must set it to the maximum possible I/O size for that stream I/O. Later the actual size can be set prior to the stream I/O execution.
file_offset_p	Pointer to offset in the file from which to read. Unless otherwise set using cuFileStreamRegister API, this value will not be evaluated until execution time.
devPtr_offset_p	Pointer to the offset relative to the bufPtr_base from which to read. Unless otherwise set using cuFileStreamRegister API, this value will not be evaluated until execution time.
bytes_written_p	Pointer to the bytes read from file. This pointer should be a non-NULL value and *bytes_written_p set to 0. The bytes_written_p memory should be allocated with cuMemHostAlloc/malloc/mmap or registered with cuMemHostRegister. After successful execution of the operation in the stream, the value *bytes_written_p will contain either: The number of bytes successfully read. -1 on IO errors. All other errors return a negative integer value of the CUfileOpError enum value.
stream	CUDA stream in which to enqueue the operation. If NULL, make this operation synchronous.

Definition at line 592 of file file_handle.hpp.

write_async() [2/2]

StreamFuture kvikio::FileHandle::write_async ( void *  devPtr_base, std::size_t  size, off_t  file_offset = 0, off_t  devPtr_offset = 0, CUstream  stream = nullptr  )

inline

Writes specified bytes from the device memory into the file asynchronously.

This is an asynchronous version of .write(), which will be executed in sequence for the specified stream.

When running CUDA v12.1 or older, this function falls back to use .read() after stream has been synchronized.

The arguments have the same meaning as in .write() but returns a StreamFuture object that the caller must keep alive until all data has been written to disk. One way to do this, is by calling StreamFuture.check_bytes_done(), which will synchronize the associated stream and return the number of bytes written.

Parameters

devPtr_base	Base address of buffer in device memory. For registered buffers, devPtr_base must remain set to the base address used in the buffer_register call.
size	Size in bytes to write.
file_offset	Offset in the file to write from.
devPtr_offset	Offset relative to the devPtr_base pointer to write from. This parameter should be used only with registered buffers.
stream	CUDA stream in which to enqueue the operation. If NULL, make this operation synchronous.

Returns

A future object that must be kept alive until all data has been written to disk e.g. by synchronizing stream.

Definition at line 634 of file file_handle.hpp.

---

The documentation for this class was generated from the following file:

• file_handle.hpp