Parquet#
- class pylibcudf.io.parquet.ChunkedParquetReader(
- ParquetReaderOptions options,
- stream: CudaStreamLike | None = None,
- DeviceMemoryResource mr=None,
- size_t chunk_read_limit=0,
- size_t pass_read_limit=1024000000,
- parquet_metadatas: Sequence[FileMetaData] | None = None,
Reads chunks of a Parquet file into a
TableWithMetadata.For details, see
chunked_parquet_reader.- Parameters:
- optionsParquetReaderOptions
Settings for controlling reading behavior
- streamStream | None
CUDA stream used for device memory operations and kernel launches
- mrDeviceMemoryResource, optional
Device memory resource used to allocate the returned table’s device memory.
- chunk_read_limitsize_t, default 0
Limit on total number of bytes to be returned per read, or 0 if there is no limit.
- pass_read_limitsize_t, default 1024000000
Limit on the amount of memory used for reading and decompressing data or 0 if there is no limit.
- parquet_metadataslist[FileMetaData], optional
Pre-materialized parquet footer metadata, one for each source. If not provided, footers are read from the sources internally.
Methods
has_next(self)Returns True if there is another chunk in the Parquet file to be read.
read_chunk(self, DeviceMemoryResource mr=None)Read the next chunk into a
TableWithMetadata- has_next(self) bool#
Returns True if there is another chunk in the Parquet file to be read.
- Returns:
- True if we have not finished reading the file.
- read_chunk(
- self,
- DeviceMemoryResource mr=None,
Read the next chunk into a
TableWithMetadata- Parameters:
- mrDeviceMemoryResource, optional
Device memory resource used to allocate the returned table’s device memory.
- Returns:
- TableWithMetadata
The Table and its corresponding metadata (column names) that were read in.
- class pylibcudf.io.parquet.ChunkedParquetWriterOptions#
The settings to use for chunked Parquet writing.
Methods
builder(SinkInfo sink)Create builder to create ChunkedParquetWriterOptions.
set_dictionary_policy(self, ...)Sets the policy for dictionary use.
- static builder(
- SinkInfo sink,
Create builder to create ChunkedParquetWriterOptions.
- Parameters:
- sink: SinkInfo
The sink used for writer output
- Returns:
- ChunkedParquetWriterOptionsBuilder
- set_dictionary_policy(
- self,
- dictionary_policy policy,
Sets the policy for dictionary use.
- Parameters:
- policyDictionaryPolicy
Policy for dictionary use
- Returns:
- None
- class pylibcudf.io.parquet.ChunkedParquetWriterOptionsBuilder#
Builder to build options for chunked Parquet writing.
Methods
build(self)Create a ChunkedParquetWriterOptions object
compression(self, compression_type compression)Sets compression type.
key_value_metadata(self, metadata)Sets Key-Value footer metadata.
max_dictionary_size(self, size_t val)Sets the maximum dictionary size, in bytes.
max_page_size_bytes(self, size_t val)Sets the maximum uncompressed page size, in bytes.
max_page_size_rows(self, size_type val)Sets the maximum page size, in rows.
metadata(self, TableInputMetadata metadata)row_group_size_bytes(self, size_t val)Sets the maximum row group size, in bytes.
row_group_size_rows(self, size_type val)Sets the maximum row group size, in rows.
stats_level(self, statistics_freq sf)Sets the level of statistics.
write_arrow_schema(self, bool enabled)Set to true if arrow schema is to be written.
- build(
- self,
Create a ChunkedParquetWriterOptions object
- compression(
- self,
- compression_type compression,
Sets compression type.
- Parameters:
- compressionCompressionType
The compression type to use
- Returns:
- Self
- key_value_metadata( ) ChunkedParquetWriterOptionsBuilder#
Sets Key-Value footer metadata.
- Parameters:
- metadatalist[dict[str, str]]
Key-Value footer metadata
- Returns:
- Self
- max_dictionary_size(
- self,
- size_t val,
Sets the maximum dictionary size, in bytes.
- Parameters:
- valsize_t
Sets the maximum dictionary size, in bytes.
- Returns:
- Self
- max_page_size_bytes(
- self,
- size_t val,
Sets the maximum uncompressed page size, in bytes.
- Parameters:
- valsize_t
Maximum uncompressed page size, in bytes to set
- Returns:
- Self
- max_page_size_rows(
- self,
- size_type val,
Sets the maximum page size, in rows.
- Parameters:
- valsize_type
Maximum page size, in rows to set.
- Returns:
- Self
- metadata(
- self,
- TableInputMetadata metadata,
- row_group_size_bytes(
- self,
- size_t val,
Sets the maximum row group size, in bytes.
- Parameters:
- valsize_t
Maximum row group size, in bytes to set
- Returns:
- Self
- row_group_size_rows(
- self,
- size_type val,
Sets the maximum row group size, in rows.
- Parameters:
- valsize_type
Maximum row group size, in rows to set
- Returns:
- Self
- stats_level(
- self,
- statistics_freq sf,
Sets the level of statistics.
- Parameters:
- sfStatisticsFreq
Level of statistics requested in the output file
- Returns:
- Self
- write_arrow_schema(
- self,
- bool enabled,
Set to true if arrow schema is to be written.
- Parameters:
- enabledbool
Boolean value to enable/disable writing of arrow schema.
- Returns:
- Self
- class pylibcudf.io.parquet.ParquetReaderOptions#
The settings to use for
read_parquetFor details, seecudf::io::parquet_reader_optionsMethods
builder(SourceInfo source)Create a ParquetReaderOptionsBuilder object
enable_case_sensitive_names(self, bool val)Sets whether column names are matched case-sensitively.
Returns whether column name matching is case sensitive.
Returns whether to use JIT compilation for filtering.
set_column_field_ids(self, list column_field_ids)Sets Parquet field IDs of the columns/fields to be read.
set_column_indices(self, list col_indices)Sets indices of the top-level columns to be read.
set_column_names(self, list col_names)Sets names of the columns to be read.
set_columns(self, list col_names)Sets names of the columns to be read.
set_filter(self, Expression filter)Sets AST based filter for predicate pushdown.
set_num_rows(self, int64_t nrows)Sets number of rows to read.
set_row_groups(self, list row_groups)Sets list of individual row groups to read.
set_skip_rows(self, int64_t skip_rows)Sets number of rows to skip.
set_source(self, SourceInfo src)Set a new source info location.
- static builder(
- SourceInfo source,
Create a ParquetReaderOptionsBuilder object
For details, see
cudf::io::parquet_reader_options::builder()- Parameters:
- sinkSourceInfo
The source to read the Parquet file from.
- Returns:
- ParquetReaderOptionsBuilder
Builder to build ParquetReaderOptions
- enable_case_sensitive_names(
- self,
- bool val,
Sets whether column names are matched case-sensitively.
- Parameters:
- valbool
Enables case-sensitive matching
- Returns:
- None
- is_enabled_case_sensitive_names(self) bool#
Returns whether column name matching is case sensitive.
- Returns:
- bool
Whether column names are matched case-sensitively
- set_column_field_ids(self, list column_field_ids: list[int]) void#
Sets Parquet field IDs of the columns/fields to be read.
- Parameters:
- column_field_idslist
List of Parquet field IDs
- Returns:
- None
- set_column_indices(self, list col_indices: list[int]) void#
Sets indices of the top-level columns to be read.
- Parameters:
- col_indiceslist
List of top-level column indices
- Returns:
- None
- set_column_names(self, list col_names: list[str]) void#
Sets names of the columns to be read.
- Parameters:
- col_nameslist
List of column names
- Returns:
- None
- set_columns(self, list col_names: list[str]) void#
Sets names of the columns to be read. Deprecated and will be removed in a future version. Use set_column_names instead.
- Parameters:
- col_nameslist
List of column names
- Returns:
- None
- set_filter(self, Expression filter) void#
Sets AST based filter for predicate pushdown.
- Parameters:
- filterExpression
AST expression to use as filter
- Returns:
- None
- set_num_rows(self, int64_t nrows) void#
Sets number of rows to read.
- Parameters:
- nrowsint64_t
Number of rows to read after skip
- Returns:
- None
Notes
Although this allows one to request more than size_type::max() rows, if any single read would produce a table larger than this row limit, an error is thrown.
- set_row_groups(self, list row_groups: list[list[int]]) void#
Sets list of individual row groups to read.
- Parameters:
- row_groupslist[list[int]]
Row groups to read, one inner list per input source.
- Returns:
- None
Notes
Rows are emitted in input-source order; all rows selected from source 0 are emitted before rows selected from source 1, and so on. Within each source, row groups are read in the order provided; indices are not sorted or deduplicated, and repeated indices are emitted multiple times. Empty inner lists contribute no rows. When unset, all row groups are read in source order, then in on-disk order within each source. Predicate pushdown drops row groups in place; remaining row groups keep their relative order.
- set_skip_rows(self, int64_t skip_rows) void#
Sets number of rows to skip.
- Parameters:
- skip_rowsint64_t
Number of rows to skip from start
- Returns:
- None
- set_source(self, SourceInfo src) void#
Set a new source info location.
- Parameters:
- srcSourceInfo
New source information, replacing existing information.
- Returns:
- None
- class pylibcudf.io.parquet.ParquetReaderOptionsBuilder#
Builder to build options for
read_parquet.Methods
allow_mismatched_pq_schemas(self, bool val)Sets to enable/disable reading of matching projected and filter columns from mismatched Parquet sources.
build(self)Create a ParquetReaderOptions object
case_sensitive_names(self, bool val)Sets whether column name matching is case sensitive.
column_field_ids(self, list column_field_ids)Sets Parquet field IDs of the columns/fields to be read.
column_indices(self, list col_indices)Sets indices of the top-level columns to be read.
column_names(self, list col_names)Sets names of the columns to be read.
columns(self, list col_names)Sets names of the columns to be read.
convert_strings_to_categories(self, bool val)Sets enable/disable conversion of strings to categories.
decimal_width(self, type_id width)Sets the decimal width used to cast all decimal columns.
filter(self, Expression filter)Sets AST based filter for predicate pushdown.
ignore_missing_columns(self, bool val)Sets to enable/disable ignoring of non-existent projected columns while reading.
use_arrow_schema(self, bool val)Sets to enable/disable use of arrow schema to read.
use_jit_filter(self, bool use_jit_filter)Sets whether to use JIT compilation for filtering.
use_pandas_metadata(self, bool val)Sets to enable/disable use of pandas metadata to read.
- allow_mismatched_pq_schemas(
- self,
- bool val,
Sets to enable/disable reading of matching projected and filter columns from mismatched Parquet sources.
- Parameters:
- valbool
Boolean value whether to read matching projected and filter columns from mismatched Parquet sources.
- Returns:
- ParquetReaderOptionsBuilder
- build(self) ParquetReaderOptions#
Create a ParquetReaderOptions object
- case_sensitive_names(
- self,
- bool val,
Sets whether column name matching is case sensitive.
- Parameters:
- valbool
Trueto enable case-sensitive matching (default),Falsefor case-insensitive matching.
- Returns:
- ParquetReaderOptionsBuilder
- column_field_ids(self, list column_field_ids: list[int]) ParquetReaderOptionsBuilder#
Sets Parquet field IDs of the columns/fields to be read.
- Parameters:
- column_field_idslist[int]
List of Parquet field IDs
- Returns:
- ParquetReaderOptionsBuilder
- column_indices(self, list col_indices: list[int]) ParquetReaderOptionsBuilder#
Sets indices of the top-level columns to be read.
- Parameters:
- col_indiceslist[int]
List of top-level column indices
- Returns:
- ParquetReaderOptionsBuilder
- column_names(self, list col_names: list[str]) ParquetReaderOptionsBuilder#
Sets names of the columns to be read.
- Parameters:
- col_nameslist[str]
List of column names
- Returns:
- ParquetReaderOptionsBuilder
- columns(self, list col_names: list[str]) ParquetReaderOptionsBuilder#
Sets names of the columns to be read. Deprecated and will be removed in a future version. Use column_names instead.
- Parameters:
- col_nameslist[str]
List of column names
- Returns:
- ParquetReaderOptionsBuilder
- convert_strings_to_categories(
- self,
- bool val,
Sets enable/disable conversion of strings to categories.
- Parameters:
- valbool
Boolean value to enable/disable conversion of string columns to categories
- Returns:
- ParquetReaderOptionsBuilder
- decimal_width(
- self,
- type_id width,
Sets the decimal width used to cast all decimal columns.
- Parameters:
- widthTypeId
The decimal type_id (DECIMAL32, DECIMAL64, or DECIMAL128) to which all decimal columns should be cast.
- Returns:
- ParquetReaderOptionsBuilder
- filter(
- self,
- Expression filter,
Sets AST based filter for predicate pushdown.
- Parameters:
- filterExpression
AST expression to use as filter
- Returns:
- ParquetReaderOptionsBuilder
- ignore_missing_columns(
- self,
- bool val,
Sets to enable/disable ignoring of non-existent projected columns while reading.
- Parameters:
- valbool
Boolean indicating whether to ignore non-existent projected columns while reading.
- Returns:
- ParquetReaderOptionsBuilder
- use_arrow_schema(
- self,
- bool val,
Sets to enable/disable use of arrow schema to read.
- Parameters:
- valbool
Boolean value whether to use arrow schema
- Returns:
- ParquetReaderOptionsBuilder
- use_jit_filter(
- self,
- bool use_jit_filter,
Sets whether to use JIT compilation for filtering.
- Parameters:
- use_jit_filterbool
Boolean value whether to use JIT filter
- Returns:
- ParquetReaderOptionsBuilder
- use_pandas_metadata(
- self,
- bool val,
Sets to enable/disable use of pandas metadata to read.
- Parameters:
- valbool
Boolean value whether to use pandas metadata
- Returns:
- ParquetReaderOptionsBuilder
- class pylibcudf.io.parquet.ParquetWriterOptions#
The settings to use for
write_parquet.Methods
builder(SinkInfo sink, Table table)Create builder to create ParquetWriterOptionsBuilder.
set_column_chunks_file_paths(self, file_paths)Sets column chunks file path to be set in the raw output metadata.
set_max_dictionary_size(self, size_t size_bytes)Sets the maximum dictionary size, in bytes.
set_max_page_size_bytes(self, size_t size_bytes)Sets the maximum uncompressed page size, in bytes.
set_max_page_size_rows(self, size_type size_rows)Sets the maximum page size, in rows.
set_partitions(self, list partitions)Sets partitions.
set_row_group_size_bytes(self, size_t size_bytes)Sets the maximum row group size, in bytes.
set_row_group_size_rows(self, ...)Sets the maximum row group size, in rows.
- static builder(
- SinkInfo sink,
- Table table,
Create builder to create ParquetWriterOptionsBuilder.
- Parameters:
- sinkSinkInfo
The sink used for writer output
- tableTable
Table to be written to output
- Returns:
- ParquetWriterOptionsBuilder
- set_column_chunks_file_paths(
- self,
- file_paths: Sequence[str],
Sets column chunks file path to be set in the raw output metadata.
- Parameters:
- file_pathslist[str]
Vector of strings which indicate file paths.
- Returns:
- None
- set_max_dictionary_size(
- self,
- size_t size_bytes,
Sets the maximum dictionary size, in bytes.
- Parameters:
- size_bytesint
Sets the maximum dictionary size, in bytes.
- Returns:
- None
- set_max_page_size_bytes(
- self,
- size_t size_bytes,
Sets the maximum uncompressed page size, in bytes.
- Parameters:
- size_bytesint
Maximum uncompressed page size, in bytes to set
- Returns:
- None
- set_max_page_size_rows(
- self,
- size_type size_rows,
Sets the maximum page size, in rows.
- Parameters:
- size_rowsint
Maximum page size, in rows to set.
- Returns:
- None
- set_partitions(self, list partitions: list[PartitionInfo]) void#
Sets partitions.
- Parameters:
- partitionslist[Partitions]
Partitions of input table in {start_row, num_rows} pairs.
- Returns:
- None
- set_row_group_size_bytes(
- self,
- size_t size_bytes,
Sets the maximum row group size, in bytes.
- Parameters:
- size_bytesint
Maximum row group size, in bytes to set
- Returns:
- None
- set_row_group_size_rows(
- self,
- size_type size_rows,
Sets the maximum row group size, in rows.
- Parameters:
- size_rowsint
Maximum row group size, in rows to set
- Returns:
- None
- class pylibcudf.io.parquet.ParquetWriterOptionsBuilder#
Builder to build options for
write_parquet.Methods
build(self)Create a ParquetWriterOptions from the set options.
compression(self, compression_type compression)Sets compression type.
dictionary_policy(self, dictionary_policy val)Sets the policy for dictionary use.
int96_timestamps(self, bool enabled)Sets whether timestamps are written as int96 or timestamp micros.
key_value_metadata(self, metadata)Sets Key-Value footer metadata.
max_page_size_bytes(self, size_t val)Sets the maximum uncompressed page size, in bytes.
metadata(self, TableInputMetadata metadata)Sets metadata.
page_level_compression(self, bool enabled)Set to true to enable per-page compression decisions for V2 data pages.
row_group_size_rows(self, size_type val)Sets the maximum row group size, in rows.
stats_level(self, statistics_freq sf)Sets the level of statistics.
utc_timestamps(self, bool enabled)Set to true if timestamps are to be written as UTC.
write_arrow_schema(self, bool enabled)Set to true if arrow schema is to be written.
write_v2_headers(self, bool enabled)Set to true to write V2 page headers, otherwise false to write V1 page headers.
- build(self) ParquetWriterOptions#
Create a ParquetWriterOptions from the set options.
- Returns:
- ParquetWriterOptions
- compression(
- self,
- compression_type compression,
Sets compression type.
- Parameters:
- compressionCompressionType
The compression type to use
- Returns:
- Self
- dictionary_policy(
- self,
- dictionary_policy val,
Sets the policy for dictionary use.
- Parameters:
- valDictionaryPolicy
Policy for dictionary use.
- Returns:
- Self
- int96_timestamps(
- self,
- bool enabled,
Sets whether timestamps are written as int96 or timestamp micros.
- Parameters:
- enabledbool
Boolean value to enable/disable int96 timestamps
- Returns:
- Self
- key_value_metadata( ) ParquetWriterOptionsBuilder#
Sets Key-Value footer metadata.
- Parameters:
- metadatalist[dict[str, str]]
Key-Value footer metadata
- Returns:
- Self
- max_page_size_bytes(
- self,
- size_t val,
Sets the maximum uncompressed page size, in bytes.
- Parameters:
- valsize_t
Maximum uncompressed page size, in bytes to set
- Returns:
- Self
- metadata(
- self,
- TableInputMetadata metadata,
Sets metadata.
- Parameters:
- metadataTableInputMetadata
Associated metadata
- Returns:
- Self
- page_level_compression(
- self,
- bool enabled,
Set to true to enable per-page compression decisions for V2 data pages.
When enabled, each V2 data page independently decides whether to compress based on compression ratio. When disabled (default), all V2 data pages in a chunk follow the same compression decision as dictionary pages.
- Parameters:
- enabledbool
Boolean value to enable/disable per-page compression decisions.
- Returns:
- Self
- row_group_size_rows(
- self,
- size_type val,
Sets the maximum row group size, in rows.
- Parameters:
- valsize_type
Maximum row group size, in rows to set
- Returns:
- Self
- stats_level(
- self,
- statistics_freq sf,
Sets the level of statistics.
- Parameters:
- sfStatisticsFreq
Level of statistics requested in the output file
- Returns:
- Self
- utc_timestamps(
- self,
- bool enabled,
Set to true if timestamps are to be written as UTC.
- Parameters:
- enabledbool
Boolean value to enable/disable writing of timestamps as UTC.
- Returns:
- Self
- write_arrow_schema(
- self,
- bool enabled,
Set to true if arrow schema is to be written.
- Parameters:
- enabledbool
Boolean value to enable/disable writing of arrow schema.
- Returns:
- Self
- write_v2_headers(
- self,
- bool enabled,
Set to true to write V2 page headers, otherwise false to write V1 page headers.
- Parameters:
- enabledbool
Boolean value to enable/disable writing of V2 page headers.
- Returns:
- Self
- pylibcudf.io.parquet.is_supported_read_parquet(compression_type compression) bool#
Check if the compression type is supported for reading Parquet files.
For details, see
is_supported_read_parquet().- Parameters:
- compressionCompressionType
The compression type to check
- Returns:
- bool
True if the compression type is supported for reading Parquet files
- pylibcudf.io.parquet.is_supported_write_parquet(compression_type compression) bool#
Check if the compression type is supported for writing Parquet files.
For details, see
is_supported_write_parquet().- Parameters:
- compressionCompressionType
The compression type to check
- Returns:
- bool
True if the compression type is supported for writing Parquet files
- pylibcudf.io.parquet.merge_row_group_metadata(list metdata_list) memoryview#
Merges multiple raw metadata blobs that were previously created by write_parquet into a single metadata blob.
For details, see
merge_row_group_metadata().- Parameters:
- metdata_listlist
List of input file metadata
- Returns:
- memoryview
A parquet-compatible blob that contains the data for all row groups in the list
- pylibcudf.io.parquet.read_parquet(
- ParquetReaderOptions options,
- stream: CudaStreamLike | None = None,
- DeviceMemoryResource mr=None,
- parquet_metadatas: Sequence[FileMetaData] | None = None,
Read from Parquet format.
The source to read from and options are encapsulated by the options object.
For details, see
read_parquet().- Parameters:
- options: ParquetReaderOptions
Settings for controlling reading behavior
- streamStream | None
CUDA stream used for device memory operations and kernel launches
- mrDeviceMemoryResource, optional
Device memory resource used to allocate the returned table’s device memory.
- parquet_metadataslist[FileMetaData], optional
Pre-materialized parquet footer metadata, one for each source. If not provided, footers are read from the sources internally.
- pylibcudf.io.parquet.write_parquet(
- ParquetWriterOptions options,
- stream: CudaStreamLike | None = None,
Writes a set of columns to parquet format.
- Parameters:
- optionsParquetWriterOptions
Settings for controlling writing behavior
- streamStream | None
CUDA stream used for device memory operations and kernel launches
- Returns:
- memoryview
A blob that contains the file metadata (parquet FileMetadata thrift message) if requested in parquet_writer_options (empty blob otherwise).