Namespaces | Classes | Typedefs | Enumerations | Functions | Variables
cudf::io Namespace Reference

IO interfaces. More...

Namespaces

 orc
 Orc I/O interfaces.
 
 parquet
 Parquet I/O interfaces.
 

Classes

class  arrow_io_source
 Implementation class for reading from an Apache Arrow file. The file could be a memory-mapped file or other implementation supported by Arrow. More...
 
class  avro_reader_options
 Settings to use for read_avro(). More...
 
class  avro_reader_options_builder
 Builder to build options for read_avro(). More...
 
class  csv_reader_options
 Settings to use for read_csv(). More...
 
class  csv_reader_options_builder
 Builder to build options for read_csv(). More...
 
class  csv_writer_options
 Settings to use for write_csv(). More...
 
class  csv_writer_options_builder
 Builder to build options for writer_csv() More...
 
class  data_sink
 Interface class for storing the output data from the writers. More...
 
class  datasource
 Interface class for providing input data to the readers. More...
 
struct  schema_element
 Allows specifying the target types for nested JSON data via json_reader_options' set_dtypes method. More...
 
class  json_reader_options
 Input arguments to the read_json interface. More...
 
class  json_reader_options_builder
 Builds settings to use for read_json(). More...
 
class  json_writer_options
 Settings to use for write_json(). More...
 
class  json_writer_options_builder
 Builder to build options for writer_json() More...
 
class  orc_reader_options
 Settings to use for read_orc(). More...
 
class  orc_reader_options_builder
 Builds settings to use for read_orc(). More...
 
class  chunked_orc_reader
 The chunked orc reader class to read an ORC file iteratively into a series of tables, chunk by chunk. More...
 
class  orc_writer_options
 Settings to use for write_orc(). More...
 
class  orc_writer_options_builder
 Builds settings to use for write_orc(). More...
 
class  chunked_orc_writer_options
 Settings to use for write_orc_chunked(). More...
 
class  chunked_orc_writer_options_builder
 Builds settings to use for write_orc_chunked(). More...
 
class  orc_chunked_writer
 Chunked orc writer class writes an ORC file in a chunked/stream form. More...
 
struct  raw_orc_statistics
 Holds column names and buffers containing raw file-level and stripe-level statistics. More...
 
struct  minmax_statistics
 Base class for column statistics that include optional minimum and maximum. More...
 
struct  sum_statistics
 Base class for column statistics that include an optional sum. More...
 
struct  integer_statistics
 Statistics for integral columns. More...
 
struct  double_statistics
 Statistics for floating point columns. More...
 
struct  string_statistics
 Statistics for string columns. More...
 
struct  bucket_statistics
 Statistics for boolean columns. More...
 
struct  decimal_statistics
 Statistics for decimal columns. More...
 
struct  timestamp_statistics
 Statistics for timestamp columns. More...
 
struct  column_statistics
 Contains per-column ORC statistics. More...
 
struct  parsed_orc_statistics
 Holds column names and parsed file-level and stripe-level statistics. More...
 
struct  orc_column_schema
 Schema of an ORC column, including the nested columns. More...
 
struct  orc_schema
 Schema of an ORC file. More...
 
class  orc_metadata
 Information about content of an ORC file. More...
 
class  parquet_reader_options
 Settings for read_parquet(). More...
 
class  parquet_reader_options_builder
 Builds parquet_reader_options to use for read_parquet(). More...
 
class  chunked_parquet_reader
 The chunked parquet reader class to read Parquet file iteratively in to a series of tables, chunk by chunk. More...
 
struct  sorting_column
 Struct used to describe column sorting metadata. More...
 
class  parquet_writer_options_base
 Base settings for write_parquet() and parquet_chunked_writer. More...
 
class  parquet_writer_options_builder_base
 Base class for Parquet options builders. More...
 
class  parquet_writer_options
 Settings for write_parquet(). More...
 
class  parquet_writer_options_builder
 Class to build parquet_writer_options. More...
 
class  chunked_parquet_writer_options
 Settings for parquet_chunked_writer. More...
 
class  chunked_parquet_writer_options_builder
 Class to build chunked_parquet_writer_options. More...
 
class  parquet_chunked_writer
 chunked parquet writer class to handle options and write tables in chunks. More...
 
struct  parquet_column_schema
 Schema of a parquet column, including the nested columns. More...
 
struct  parquet_schema
 Schema of a parquet file. More...
 
class  parquet_metadata
 Information about content of a parquet file. More...
 
class  writer_compression_statistics
 Statistics about compression performed by a writer. More...
 
struct  column_name_info
 Detailed name (and optionally nullability) information for output columns. More...
 
struct  table_metadata
 Table metadata returned by IO readers. More...
 
struct  table_with_metadata
 Table with table metadata used by io readers to return the metadata by value. More...
 
struct  host_buffer
 Non-owning view of a host memory buffer. More...
 
struct  source_info
 Source information for read interfaces. More...
 
struct  sink_info
 Destination information for write interfaces. More...
 
class  column_in_metadata
 Metadata for a column. More...
 
class  table_input_metadata
 Metadata for a table. More...
 
struct  partition_info
 Information used while writing partitioned datasets. More...
 
class  reader_column_schema
 schema element for reader More...
 

Typedefs

using no_statistics = std::monostate
 Monostate type alias for the statistics variant.
 
using date_statistics = minmax_statistics< int32_t >
 Statistics for date(time) columns.
 
using binary_statistics = sum_statistics< int64_t >
 Statistics for binary columns. More...
 
using statistics_type = std::variant< no_statistics, integer_statistics, double_statistics, string_statistics, bucket_statistics, decimal_statistics, date_statistics, binary_statistics, timestamp_statistics >
 Variant type for ORC type-specific column statistics. More...
 

Enumerations

enum class  json_recovery_mode_t { FAIL , RECOVER_WITH_NULL }
 Control the error recovery behavior of the json parser. More...
 
enum class  compression_type {
  NONE , AUTO , SNAPPY , GZIP ,
  BZIP2 , BROTLI , ZIP , XZ ,
  ZLIB , LZ4 , LZO , ZSTD
}
 Compression algorithms. More...
 
enum class  io_type {
  FILEPATH , HOST_BUFFER , DEVICE_BUFFER , VOID ,
  USER_IMPLEMENTED
}
 Data source or destination types. More...
 
enum class  quote_style { MINIMAL , ALL , NONNUMERIC , NONE }
 Behavior when handling quotations in field data. More...
 
enum  statistics_freq { STATISTICS_NONE = 0 , STATISTICS_ROWGROUP = 1 , STATISTICS_PAGE = 2 , STATISTICS_COLUMN = 3 }
 Column statistics granularity type for parquet/orc writers. More...
 
enum class  column_encoding {
  USE_DEFAULT = -1 , DICTIONARY , PLAIN , DELTA_BINARY_PACKED ,
  DELTA_LENGTH_BYTE_ARRAY , DELTA_BYTE_ARRAY , BYTE_STREAM_SPLIT , DIRECT ,
  DIRECT_V2 , DICTIONARY_V2
}
 Valid encodings for use with column_in_metadata::set_encoding() More...
 
enum  dictionary_policy { NEVER = 0 , ADAPTIVE = 1 , ALWAYS = 2 }
 Control use of dictionary encoding for parquet writer. More...
 

Functions

table_with_metadata read_avro (avro_reader_options const &options, rmm::device_async_resource_ref mr=rmm::mr::get_current_device_resource())
 Reads an Avro dataset into a set of columns. More...
 
table_with_metadata read_csv (csv_reader_options options, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=rmm::mr::get_current_device_resource())
 Reads a CSV dataset into a set of columns. More...
 
void write_csv (csv_writer_options const &options, rmm::cuda_stream_view stream=cudf::get_default_stream())
 Writes a set of columns to CSV format. More...
 
table_with_metadata read_json (json_reader_options options, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=rmm::mr::get_current_device_resource())
 Reads a JSON dataset into a set of columns. More...
 
void write_json (json_writer_options const &options, rmm::cuda_stream_view stream=cudf::get_default_stream())
 Writes a set of columns to JSON format. More...
 
table_with_metadata read_orc (orc_reader_options const &options, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=rmm::mr::get_current_device_resource())
 Reads an ORC dataset into a set of columns. More...
 
void write_orc (orc_writer_options const &options, rmm::cuda_stream_view stream=cudf::get_default_stream())
 Writes a set of columns to ORC format. More...
 
raw_orc_statistics read_raw_orc_statistics (source_info const &src_info, rmm::cuda_stream_view stream=cudf::get_default_stream())
 Reads file-level and stripe-level statistics of ORC dataset. More...
 
parsed_orc_statistics read_parsed_orc_statistics (source_info const &src_info, rmm::cuda_stream_view stream=cudf::get_default_stream())
 Reads file-level and stripe-level statistics of ORC dataset. More...
 
orc_metadata read_orc_metadata (source_info const &src_info, rmm::cuda_stream_view stream=cudf::get_default_stream())
 Reads metadata of ORC dataset. More...
 
table_with_metadata read_parquet (parquet_reader_options const &options, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=rmm::mr::get_current_device_resource())
 Reads a Parquet dataset into a set of columns. More...
 
std::unique_ptr< std::vector< uint8_t > > write_parquet (parquet_writer_options const &options, rmm::cuda_stream_view stream=cudf::get_default_stream())
 Writes a set of columns to parquet format. More...
 
std::unique_ptr< std::vector< uint8_t > > merge_row_group_metadata (std::vector< std::unique_ptr< std::vector< uint8_t >>> const &metadata_list)
 Merges multiple raw metadata blobs that were previously created by write_parquet into a single metadata blob. More...
 
parquet_metadata read_parquet_metadata (source_info const &src_info)
 Reads metadata of parquet dataset. More...
 
template<typename T >
constexpr auto is_byte_like_type ()
 Returns true if the type is byte-like, meaning it is reasonable to pass as a pointer to bytes. More...
 

Variables

constexpr size_t default_stripe_size_bytes = 64 * 1024 * 1024
 64MB default orc stripe size
 
constexpr size_type default_stripe_size_rows = 1000000
 1M rows default orc stripe rows
 
constexpr size_type default_row_index_stride = 10000
 10K rows default orc row index stride
 
constexpr size_t default_row_group_size_bytes = 128 * 1024 * 1024
 128MB per row group
 
constexpr size_type default_row_group_size_rows = 1000000
 1 million rows per row group
 
constexpr size_t default_max_page_size_bytes = 512 * 1024
 512KB per page
 
constexpr size_type default_max_page_size_rows = 20000
 20k rows per page
 
constexpr int32_t default_column_index_truncate_length = 64
 truncate to 64 bytes
 
constexpr size_t default_max_dictionary_size = 1024 * 1024
 1MB dictionary size
 
constexpr size_type default_max_page_fragment_size = 5000
 5000 rows per page fragment
 

Detailed Description

IO interfaces.