Parquet Metadata#

class pylibcudf.io.parquet_metadata.ColumnChunk#

Metadata for a row group’s column chunk.

Attributes

`column_index_length`	ColumnChunk.column_index_length: int
`column_index_offset`	ColumnChunk.column_index_offset: int
`file_offset`	ColumnChunk.file_offset: int
`file_path`	ColumnChunk.file_path: str
`meta_data`	ColumnChunk.meta_data: ColumnChunkMetaData
`offset_index_length`	ColumnChunk.offset_index_length: int
`offset_index_offset`	ColumnChunk.offset_index_offset: int
`schema_idx`	ColumnChunk.schema_idx: int

column_index_length#

ColumnChunk.column_index_length: int

Size of the chunk’s ColumnIndex, in bytes.

column_index_offset#

ColumnChunk.column_index_offset: int

File offset of the chunk’s ColumnIndex.

file_offset#

ColumnChunk.file_offset: int

Deprecated byte offset to column metadata.

file_path#

ColumnChunk.file_path: str

Relative file path for this column chunk.

meta_data#

ColumnChunk.meta_data: ColumnChunkMetaData

Column metadata for this chunk.

offset_index_length#

ColumnChunk.offset_index_length: int

Size of the chunk’s OffsetIndex, in bytes.

offset_index_offset#

ColumnChunk.offset_index_offset: int

File offset of the chunk’s OffsetIndex.

schema_idx#

ColumnChunk.schema_idx: int

Derived index in the flattened schema.

class pylibcudf.io.parquet_metadata.ColumnChunkMetaData#

Metadata payload for a column chunk.

Attributes

`num_values`	ColumnChunkMetaData.num_values: int
`path_in_schema`	ColumnChunkMetaData.path_in_schema: list[str]
`statistics`	ColumnChunkMetaData.statistics: ColumnChunkStatistics
`total_compressed_size`	ColumnChunkMetaData.total_compressed_size: int
`total_uncompressed_size`	ColumnChunkMetaData.total_uncompressed_size: int

num_values#

ColumnChunkMetaData.num_values: int

Number of values in this chunk.

path_in_schema#

ColumnChunkMetaData.path_in_schema: list[str]

Column path components in the flattened schema.

statistics#

ColumnChunkMetaData.statistics: ColumnChunkStatistics

Column chunk statistics.

total_compressed_size#

ColumnChunkMetaData.total_compressed_size: int

Total compressed page bytes for this chunk.

total_uncompressed_size#

ColumnChunkMetaData.total_uncompressed_size: int

Total uncompressed page bytes for this chunk.

class pylibcudf.io.parquet_metadata.ColumnChunkStatistics#

Column chunk statistics.

Attributes

`distinct_count`	ColumnChunkStatistics.distinct_count: int \| None
`has_min_max`	ColumnChunkStatistics.has_min_max: bool
`is_max_value_exact`	ColumnChunkStatistics.is_max_value_exact: bool \| None
`is_min_value_exact`	ColumnChunkStatistics.is_min_value_exact: bool \| None
`max_encoded`	ColumnChunkStatistics.max_encoded: bytes \| None
`min_encoded`	ColumnChunkStatistics.min_encoded: bytes \| None
`null_count`	ColumnChunkStatistics.null_count: int \| None

distinct_count#

ColumnChunkStatistics.distinct_count: int | None

Number of distinct values in the column chunk.

has_min_max#

ColumnChunkStatistics.has_min_max: bool

Whether this column chunk has encoded minimum and maximum values.

is_max_value_exact#

ColumnChunkStatistics.is_max_value_exact: bool | None

Whether max_value is the exact column-chunk maximum.

is_min_value_exact#

ColumnChunkStatistics.is_min_value_exact: bool | None

Whether min_value is the exact column-chunk minimum.

max_encoded#

ColumnChunkStatistics.max_encoded: bytes | None

Encoded maximum value, preferring max_value over deprecated max.

The bytes are the raw Parquet statistics payload and must be interpreted using the column’s Parquet physical and logical type metadata.

min_encoded#

ColumnChunkStatistics.min_encoded: bytes | None

Encoded minimum value, preferring min_value over deprecated min.

The bytes are the raw Parquet statistics payload and must be interpreted using the column’s Parquet physical and logical type metadata.

null_count#

ColumnChunkStatistics.null_count: int | None

Number of null values in the column chunk.

class pylibcudf.io.parquet_metadata.FileMetaData#

Parquet file footer metadata.

For details, see cudf::io::parquet::FileMetaData

Attributes

`columnchunk_metadata`	FileMetaData.columnchunk_metadata: dict[str, list[int]]
`created_by`	FileMetaData.created_by: str
`num_rows`	FileMetaData.num_rows: int
`row_group_num_rows`	FileMetaData.row_group_num_rows: list[int]
`row_groups`	FileMetaData.row_groups: list[RowGroup]
`version`	FileMetaData.version: int

Methods

from_bytes(cls, const uint8_t[)

Build FileMetaData from parquet footer bytes.

See also

pylibcudf.io.parquet_metadata.read_parquet_footers: Read one FileMetaData per source directly from pylibcudf.io.types.SourceInfo.

columnchunk_metadata#

FileMetaData.columnchunk_metadata: dict[str, list[int]]

Get a map of dotted column paths to lists of total_uncompressed_size values from every column chunk in this file.

Returns:

dict[str, list[int]]: Map of dotted column paths (".".join(path_in_schema)) to lists of total_uncompressed_size metadata from all their column chunks.

Notes

Equivalent to, but faster than, walking each row group’s columns:

>>> result: dict[str, list[int]] = {}
>>> for rg in file_metadata.row_groups:
...     for col in rg.columns:
...         name = ".".join(col.meta_data.path_in_schema)
...         result.setdefault(name, []).append(
...             col.meta_data.total_uncompressed_size
...         )

created_by#

FileMetaData.created_by: str

Get the application that created the file.

classmethod from_bytes( cls, const uint8_t[::1] footer_bytes: Buffer, ) → FileMetaData#

Build FileMetaData from parquet footer bytes.

Parameters:

footer_bytesBuffer: A contiguous bytes-like object containing parquet footer bytes. The bytes are forwarded as-is to cudf::io::parquet::experimental::hybrid_scan_reader without Python-side preprocessing. This method does not strip the parquet footer suffix (4-byte footer length + PAR1 magic), so callers should generally pass only the footer region bytes.

Returns:

FileMetaData: Parsed parquet file footer metadata.

num_rows#

FileMetaData.num_rows: int

Get the total number of rows.

row_group_num_rows#

FileMetaData.row_group_num_rows: list[int]

Get row counts for each row group in this file.

Returns:

list: A list with the row count per row group in this file.

Notes

Equivalent to, but faster than, checking each row groups’ num_rows:

>>> [rg.num_rows for rg in file_metadata.row_groups]

row_groups#

FileMetaData.row_groups: list[RowGroup]

Get row group metadata in this file.

version#

FileMetaData.version: int

Get the file format version.

class pylibcudf.io.parquet_metadata.ParquetColumnSchema#

Schema of a parquet column, including the nested columns.

Parameters:

parquet_column_schema

Methods

`child`(self, int idx)	Returns schema of the child with the given index.
`children`(self)	Returns schemas of all child columns.
`cudf_type`(self)	Returns the cudf data type for this column.
`name`(self)	Returns parquet column name; can be empty.
`num_children`(self)	Returns the number of child columns.

child(self, int idx) → ParquetColumnSchema#

Returns schema of the child with the given index.

Parameters:

idxint: Child Index

Returns:

ParquetColumnSchema: Child schema

children(self) → list#

Returns schemas of all child columns.

Returns:

list[ParquetColumnSchema]: Child schemas.

cudf_type(self) → DataType#

Returns the cudf data type for this column.

This is the resolved cudf data type mapped from the parquet physical/logical types.

Returns:

DataType: cudf data type

name(self) → str#

Returns parquet column name; can be empty.

Returns:

str: Column name

num_children(self) → int#

Returns the number of child columns.

Returns:

int: Children count

class pylibcudf.io.parquet_metadata.ParquetMetadata#

Information about content of a parquet file.

Parameters:

parquet_metadata

Methods

`columnchunk_metadata`(self)	Returns a map of leaf column names to lists of total_uncompressed_size metadata from all column chunks in the file footer.
`metadata`(self)	Returns the key-value metadata in the file footer.
`num_rowgroups`(self)	Returns the total number of rowgroups in the file.
`num_rowgroups_per_file`(self)	Returns the number of rowgroups in each file.
`num_rows`(self)	Returns the number of rows of the root column.
`rowgroup_metadata`(self)	Returns the row group metadata in the file footer.
`schema`(self)	Returns the parquet schema.

columnchunk_metadata(self) → dict#

Returns a map of leaf column names to lists of total_uncompressed_size metadata from all column chunks in the file footer.

Returns:

dict[str, list[int]]: Map of leaf column names to lists of total_uncompressed_size metadata from all their column chunks.

metadata(self) → dict#

Returns the key-value metadata in the file footer.

Returns:

dict[str, str]: Key value metadata as a map.

num_rowgroups(self) → int#

Returns the total number of rowgroups in the file.

Returns:

int: Number of row groups.

num_rowgroups_per_file(self) → list#: Returns the number of rowgroups in each file.

num_rows(self) → int#

Returns the number of rows of the root column.

Returns:

int: Number of rows

rowgroup_metadata(self) → list#

Returns the row group metadata in the file footer.

Returns:

list[dict[str, int]]: Vector of row group metadata as maps.

schema(self) → ParquetSchema#

Returns the parquet schema.

Returns:

ParquetSchema: Parquet schema

class pylibcudf.io.parquet_metadata.ParquetSchema#

Schema of a parquet file.

Parameters:

parquet_schema

Methods

`column_types`(self)	Returns a dictionary mapping column names to their cudf data types.
`root`(self)	Returns the schema of the struct column that contains all columns as fields.

column_types(self) → dict#

Returns a dictionary mapping column names to their cudf data types.

Returns:

dict[str, DataType]: Dictionary mapping column names to DataType objects

root(self) → ParquetColumnSchema#

Returns the schema of the struct column that contains all columns as fields.

Returns:

ParquetColumnSchema: Root column schema

class pylibcudf.io.parquet_metadata.RowGroup#

Parquet row group metadata.

Attributes

`columns`	RowGroup.columns: list[ColumnChunk]
`file_offset`	RowGroup.file_offset: int \| None
`num_rows`	RowGroup.num_rows: int
`ordinal`	RowGroup.ordinal: int \| None
`sorting_columns`	RowGroup.sorting_columns: list[SortingColumn] \| None
`total_byte_size`	RowGroup.total_byte_size: int
`total_compressed_size`	RowGroup.total_compressed_size: int \| None

columns#

RowGroup.columns: list[ColumnChunk]

Column chunk metadata for each column in this row group.

file_offset#

RowGroup.file_offset: int | None

Optional byte offset to first page in this row group.

num_rows#

RowGroup.num_rows: int

Number of rows in this row group.

ordinal#

RowGroup.ordinal: int | None

Optional row group ordinal within the file.

sorting_columns#

RowGroup.sorting_columns: list[SortingColumn] | None

Optional row sort order metadata.

total_byte_size#

RowGroup.total_byte_size: int

Total uncompressed byte size in this row group.

total_compressed_size#

RowGroup.total_compressed_size: int | None

Optional total compressed bytes for this row group.

class pylibcudf.io.parquet_metadata.SortingColumn#

Sort metadata for a row group column.

Attributes

`column_idx`	SortingColumn.column_idx: int
`descending`	SortingColumn.descending: bool
`nulls_first`	SortingColumn.nulls_first: bool

column_idx#

SortingColumn.column_idx: int

Column index (within the row group).

descending#

SortingColumn.descending: bool

Whether this column is sorted in descending order.

nulls_first#

SortingColumn.nulls_first: bool

Whether null values are ordered before non-null values.

pylibcudf.io.parquet_metadata.read_parquet_column_chunk_bounds( file_metadatas, columns, stream=None, DeviceMemoryResource mr=None, ) → Table#

Decode parquet column-chunk min/max statistics for selected columns.

Missing min/max statistics are returned as nulls. Parquet min/max exactness flags are not interpreted by this function.

Parameters:

file_metadatasSequence[FileMetaData]: Parquet footer metadata objects, one per source.
columnsSequence[str]: Dotted leaf-column paths to decode statistics for.
streamCudaStreamLike, optional: CUDA stream used for device memory operations.
mrDeviceMemoryResource, optional: Device memory resource used for device memory allocation.

Returns:

Table: Table containing file indices in column 0, file-local row-group indices in column 1, and one (min, max) column pair per requested column after that. For columns[i], the minimum column is at 2 + 2 * i and the maximum column is at 3 + 2 * i.

pylibcudf.io.parquet_metadata.read_parquet_footers(SourceInfo src_info) → list#

Read parquet file footers as FileMetaData objects.

Parameters:

src_infoSourceInfo: Dataset source.

Returns:

list[FileMetaData]: One footer metadata object per input source.

pylibcudf.io.parquet_metadata.read_parquet_metadata(SourceInfo src_info) → ParquetMetadata#

Reads metadata of parquet dataset.

Parameters:

src_infoSourceInfo: Dataset source.

Returns:

ParquetMetadata: Parquet_metadata with parquet schema, number of rows, number of row groups and key-value metadata.