ORC

class pylibcudf.io.orc.ChunkedOrcWriterOptions

The settings to use for chunked ORC writing.

Methods

builder(SinkInfo sink)	Create builder to create ChunkedOrcWriterOptions.
set_row_index_stride(self, size_type stride)	Sets the row index stride.
set_stripe_size_bytes(self, size_t size_bytes)	Sets the maximum stripe size, in bytes.
set_stripe_size_rows(self, size_type size_rows)	Sets the maximum stripe size, in rows.

static builder(

SinkInfo sink,

) → ChunkedOrcWriterOptionsBuilder

Create builder to create ChunkedOrcWriterOptions.

Parameters:

sink: SinkInfo

The sink used for writer output

table: Table

Table to be written to output

Returns:

ChunkedOrcWriterOptionsBuilder

set_row_index_stride(

self,

size_type stride,

) → void

Sets the row index stride.

Rounded down to a multiple of 8.

Parameters:

size_bytes: size_type

Maximum stripe size, in rows to be set

Returns:

None

set_stripe_size_bytes(

self,

size_t size_bytes,

) → void

Sets the maximum stripe size, in bytes.

Parameters:

size_bytes: size_t

Sets the maximum stripe size, in bytes.

Returns:

None

set_stripe_size_rows(

self,

size_type size_rows,

) → void

Sets the maximum stripe size, in rows.

If the stripe size is smaller that the row group size, row group size will be reduced to math the stripe size.

Parameters:

size_bytes: size_type

Maximum stripe size, in rows to be set

Returns:

None

class pylibcudf.io.orc.ChunkedOrcWriterOptionsBuilder

Builder to build options for chunked ORC writing.

Methods

build(self)	Create a OrcWriterOptions object
compression(self, compression_type comp)	Sets compression type.
enable_statistics(self, statistics_freq val)	Choose granularity of column statistics to be written.
key_value_metadata(self, dict kvm, str])	Sets Key-Value footer metadata.
metadata(self, TableInputMetadata meta)	Sets associated metadata.

build(self) → ChunkedOrcWriterOptions

Create a OrcWriterOptions object

compression(

self,

compression_type comp,

) → ChunkedOrcWriterOptionsBuilder

Sets compression type.

Parameters:

comp: CompressionType

The compression type to use

Returns:

ChunkedOrcWriterOptionsBuilder

enable_statistics(

self,

statistics_freq val,

) → ChunkedOrcWriterOptionsBuilder

Choose granularity of column statistics to be written.

Parameters:

val: StatisticsFreq

Level of statistics collection

Returns:

ChunkedOrcWriterOptionsBuilder

key_value_metadata(self, dict kvm: dict[str, str]) → ChunkedOrcWriterOptionsBuilder

Sets Key-Value footer metadata.

Parameters:

kvm: dict

Key-Value footer metadata

Returns:

ChunkedOrcWriterOptionsBuilder

metadata(

self,

TableInputMetadata meta,

) → ChunkedOrcWriterOptionsBuilder

Sets associated metadata.

Parameters:

meta: TableInputMetadata

Associated metadata

Returns:

ChunkedOrcWriterOptionsBuilder

class pylibcudf.io.orc.OrcColumnStatistics

Attributes

has_null
number_of_values

Methods

get(self, item[, default])

get(self, item, default=None)

class pylibcudf.io.orc.OrcReaderOptions

The settings to use for read_orc

For details, see cudf::io::orc_reader_options

Methods

builder(SourceInfo source)	Create a OrcReaderOptionsBuilder object
set_columns(self, list col_names)	Sets names of the column to read.
set_decimal128_columns(self, list val)	Set columns that should be read as 128-bit Decimal.
set_num_rows(self, int64_t nrows)	Sets number of row to read.
set_skip_rows(self, int64_t skip_rows)	Sets number of rows to skip from the start.
set_source(self, SourceInfo src)	Set a new source info location.
set_stripes(self, list stripes)	Sets list of stripes to read for each input source.
set_timestamp_type(self, DataType type_)	Sets timestamp type to which timestamp column will be cast.

static builder(SourceInfo source) → OrcReaderOptionsBuilder

Create a OrcReaderOptionsBuilder object

For details, see cudf::io::orc_reader_options::builder()

Parameters:

sinkSourceInfo

The source to read the ORC file from.

Returns:

OrcReaderOptionsBuilder

Builder to build OrcReaderOptions

set_columns(self, list col_names: list[str]) → void

Sets names of the column to read.

Parameters:

col_names: list[str]

List of column names

Returns:

None

set_decimal128_columns(self, list val: list[str]) → void

Set columns that should be read as 128-bit Decimal.

Parameters:

val: list[str]

List of fully qualified column names

Returns:

None

set_num_rows(self, int64_t nrows) → void

Sets number of row to read.

Parameters:

nrows: int64_t

Number of rows

Returns:

None

set_skip_rows(self, int64_t skip_rows) → void

Sets number of rows to skip from the start.

Parameters:

skip_rows: int64_t

Number of rows

Returns:

None

set_source(self, SourceInfo src) → void

Set a new source info location.

Parameters:

srcSourceInfo

New source information, replacing existing information.

Returns:

None

set_stripes(self, list stripes: list[list[int]]) → void

Sets list of stripes to read for each input source.

Parameters:

stripes: list[list[size_type]]

List of lists, mapping stripes to read to input sources

Returns:

None

set_timestamp_type(self, DataType type_) → void

Sets timestamp type to which timestamp column will be cast.

Parameters:

type_: DataType

Type of timestamp

Returns:

None

class pylibcudf.io.orc.OrcReaderOptionsBuilder

Builder to build options for read_orc.

Methods

build(self)	Create a OrcReaderOptions object
use_index(self, bool use)	Enable/Disable use of row index to speed-up reading.

build(self) → OrcReaderOptions

Create a OrcReaderOptions object

use_index(

self,

bool use,

) → OrcReaderOptionsBuilder

Enable/Disable use of row index to speed-up reading.

Parameters:

usebool

Boolean value to enable/disable row index use

Returns:

OrcReaderOptionsBuilder

class pylibcudf.io.orc.OrcWriterOptions

The settings to use for write_orc.

Methods

builder(SinkInfo sink, Table table)	Create builder to create OrcWriterOptions.
set_row_index_stride(self, size_type stride)	Sets the row index stride.
set_stripe_size_bytes(self, size_t size_bytes)	Sets the maximum stripe size, in bytes.
set_stripe_size_rows(self, size_type size_rows)	Sets the maximum stripe size, in rows.

static builder(

SinkInfo sink,

Table table,

) → OrcWriterOptionsBuilder

Create builder to create OrcWriterOptions.

For details, see cudf::io::orc_writer_options::builder()

Parameters:

sink: SinkInfo

The sink used for writer output

table: Table

Table to be written to output

Returns:

OrcWriterOptionsBuilder

set_row_index_stride(self, size_type stride) → void

Sets the row index stride.

Rounded down to a multiple of 8.

For details, see cudf::io::orc_writer_options::set_row_index_stride()

Parameters:

size_bytes: size_type

Maximum stripe size, in rows to be set

Returns:

None

set_stripe_size_bytes(

self,

size_t size_bytes,

) → void

Sets the maximum stripe size, in bytes.

For details, see cudf::io::orc_writer_options::set_stripe_size_bytes()

Parameters:

size_bytes: size_t

Sets the maximum stripe size, in bytes.

Returns:

None

set_stripe_size_rows(

self,

size_type size_rows,

) → void

Sets the maximum stripe size, in rows.

If the stripe size is smaller that the row group size, row group size will be reduced to math the stripe size.

For details, see cudf::io::orc_writer_options::set_stripe_size_rows()

Parameters:

size_bytes: size_type

Maximum stripe size, in rows to be set

Returns:

None

class pylibcudf.io.orc.OrcWriterOptionsBuilder

Builder to build options for write_orc.

Methods

build(self)	Moves the ORC writer options builder
compression(self, compression_type comp)	Sets compression type.
enable_statistics(self, statistics_freq val)	Choose granularity of column statistics to be written.
key_value_metadata(self, dict kvm, str])	Sets Key-Value footer metadata.
metadata(self, TableInputMetadata meta)	Sets associated metadata.

build(self) → OrcWriterOptions

Moves the ORC writer options builder

compression(

self,

compression_type comp,

) → OrcWriterOptionsBuilder

Sets compression type.

For details, see cudf::io::orc_writer_options_builder::compression()

Parameters:

comp: CompressionType

The compression type to use

Returns:

OrcWriterOptionsBuilder

enable_statistics(

self,

statistics_freq val,

) → OrcWriterOptionsBuilder

Choose granularity of column statistics to be written.

For details, see cudf::io::orc_writer_options_builder::enable_statistics()

Parameters:

val: StatisticsFreq

Level of statistics collection

Returns:

OrcWriterOptionsBuilder

key_value_metadata(self, dict kvm: dict[str, str]) → OrcWriterOptionsBuilder

Sets Key-Value footer metadata.

Parameters:

kvm: dict

Key-Value footer metadata

Returns:

OrcWriterOptionsBuilder

metadata(

self,

TableInputMetadata meta,

) → OrcWriterOptionsBuilder

Sets associated metadata.

For details, see cudf::io::orc_writer_options_builder::metadata()

Parameters:

meta: TableInputMetadata

Associated metadata

Returns:

OrcWriterOptionsBuilder

class pylibcudf.io.orc.ParsedOrcStatistics

Holds column names and parsed file-level and stripe-level statistics.

For details, see cudf::io::parsed_orc_statistics

Attributes

column_names
file_stats
stripes_stats

pylibcudf.io.orc.is_supported_read_orc(compression_type compression) → bool

Check if the compression type is supported for reading ORC files.

For details, see is_supported_read_orc().

Parameters:

compressionCompressionType

The compression type to check

Returns:

bool

True if the compression type is supported for reading ORC files

pylibcudf.io.orc.is_supported_write_orc(compression_type compression) → bool

Check if the compression type is supported for writing ORC files.

For details, see is_supported_write_orc().

Parameters:

compressionCompressionType

The compression type to check

Returns:

bool

True if the compression type is supported for writing ORC files

pylibcudf.io.orc.read_orc(

OrcReaderOptions options,

stream: CudaStreamLike | None = None,

DeviceMemoryResource mr=None,

) → TableWithMetadata

Read from ORC format.

The source to read from and options are encapsulated by the options object.

For details, see read_orc().

Parameters:

options: OrcReaderOptions

Settings for controlling reading behavior

streamStream | None

CUDA stream used for device memory operations and kernel launches

mrDeviceMemoryResource, optional

Device memory resource used to allocate the returned table’s device memory.

pylibcudf.io.orc.read_parsed_orc_statistics(

SourceInfo source_info,

stream: CudaStreamLike | None = None,

) → ParsedOrcStatistics

Read ORC statistics from a source.

Parameters:

source_infoSourceInfo

The source to read statistics from.

streamStream | None

CUDA stream used for device memory operations and kernel launches.

Returns:

ParsedOrcStatistics

The parsed ORC statistics.

pylibcudf.io.orc.write_orc(

OrcWriterOptions options,

stream: CudaStreamLike | None = None,

) → void

Write to ORC format.

The table to write, output paths, and options are encapsulated by the options object.

For details, see write_orc().

Parameters:

options: OrcWriterOptions

Settings for controlling writing behavior

streamStream | None

CUDA stream used for device memory operations and kernel launches

Returns:

None