Base class for Parquet options builders. More...
#include <parquet.hpp>
Public Member Functions | |
| parquet_writer_options_builder_base ()=default | |
| Default constructor. More... | |
| BuilderT & | metadata (table_input_metadata metadata) |
| Sets metadata. More... | |
| BuilderT & | key_value_metadata (std::vector< std::map< std::string, std::string >> metadata) |
| Sets Key-Value footer metadata. More... | |
| BuilderT & | stats_level (statistics_freq sf) |
| Sets the level of statistics. More... | |
| BuilderT & | compression (compression_type compression) |
| Sets compression type. More... | |
| BuilderT & | row_group_size_bytes (size_t val) |
| Sets the maximum row group size, in bytes. More... | |
| BuilderT & | row_group_size_rows (size_type val) |
| Sets the maximum number of rows in output row groups. More... | |
| BuilderT & | max_page_size_bytes (size_t val) |
| Sets the maximum uncompressed page size, in bytes. More... | |
| BuilderT & | max_page_size_rows (size_type val) |
| Sets the maximum page size, in rows. Counts only top-level rows, ignoring any nesting. Cannot be larger than the row group size in rows, and will be adjusted to match if it is. More... | |
| BuilderT & | column_index_truncate_length (int32_t val) |
| Sets the desired maximum size in bytes for min and max values in the column index. More... | |
| BuilderT & | dictionary_policy (enum dictionary_policy val) |
| Sets the policy for dictionary use. More... | |
| BuilderT & | max_dictionary_size (size_t val) |
| Sets the maximum dictionary size, in bytes. More... | |
| BuilderT & | max_page_fragment_size (size_type val) |
| Sets the maximum page fragment size, in rows. More... | |
| BuilderT & | compression_statistics (std::shared_ptr< writer_compression_statistics > const &comp_stats) |
| Sets the pointer to the output compression statistics. More... | |
| BuilderT & | int96_timestamps (bool enabled) |
| Sets whether int96 timestamps are written or not. More... | |
| BuilderT & | utc_timestamps (bool enabled) |
| Set to true if timestamps are to be written as UTC. More... | |
| BuilderT & | write_arrow_schema (bool enabled) |
| Set to true if arrow schema is to be written. More... | |
| BuilderT & | write_v2_headers (bool enabled) |
| Set to true if V2 page headers are to be written. More... | |
| BuilderT & | sorting_columns (std::vector< sorting_column > sorting_columns) |
| Sets column sorting metadata. More... | |
| operator OptionsT && () | |
| move options member once it's built. | |
| OptionsT && | build () |
| move options member once it's built. More... | |
Protected Member Functions | |
| OptionsT & | get_options () |
| Return reference to the options object being built. More... | |
| parquet_writer_options_builder_base (OptionsT options) | |
| Constructor from options. More... | |
Base class for Parquet options builders.
Definition at line 928 of file parquet.hpp.
|
explicitprotected |
Constructor from options.
| options | Options object to build |
|
explicitdefault |
Default constructor.
This has been added since Cython requires a default constructor to create objects on stack.
| OptionsT&& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::build | ( | ) |
move options member once it's built.
This has been added since Cython does not support overloading of conversion operators.
parquet_writer_options object's r-value reference | BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::column_index_truncate_length | ( | int32_t | val | ) |
Sets the desired maximum size in bytes for min and max values in the column index.
Values exceeding this limit will be truncated, but modified such that they will still be valid lower and upper bounds. This only applies to variable length types, such as string. Maximum values will not be truncated if there is no suitable truncation that results in a valid upper bound.
Default value is 64.
| val | length min/max will be truncated to, with 0 indicating no truncation |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::compression | ( | compression_type | compression | ) |
Sets compression type.
| compression | The compression type to use |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::compression_statistics | ( | std::shared_ptr< writer_compression_statistics > const & | comp_stats | ) |
Sets the pointer to the output compression statistics.
| comp_stats | Pointer to compression statistics to be filled once writer is done |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::dictionary_policy | ( | enum dictionary_policy | val | ) |
Sets the policy for dictionary use.
Certain compression algorithms (e.g Zstandard) have limits on how large of a buffer can be compressed. In some circumstances, the dictionary can grow beyond this limit, which will prevent the column from being compressed. This setting controls how the writer should act in these circumstances. A setting of dictionary_policy::ADAPTIVE will disable dictionary encoding for columns where the dictionary exceeds the limit. A setting of dictionary_policy::NEVER will disable the use of dictionary encoding globally. A setting of dictionary_policy::ALWAYS will allow the use of dictionary encoding even if it will result in the disabling of compression for columns that would otherwise be compressed.
The default value is dictionary_policy::ADAPTIVE.
| val | policy for dictionary use |
|
inlineprotected |
Return reference to the options object being built.
Definition at line 937 of file parquet.hpp.
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::int96_timestamps | ( | bool | enabled | ) |
Sets whether int96 timestamps are written or not.
| enabled | Boolean value to enable/disable int96 timestamps |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::key_value_metadata | ( | std::vector< std::map< std::string, std::string >> | metadata | ) |
Sets Key-Value footer metadata.
| metadata | Key-Value footer metadata |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::max_dictionary_size | ( | size_t | val | ) |
Sets the maximum dictionary size, in bytes.
Disables dictionary encoding for any column chunk where the dictionary will exceed this limit. Only used when the dictionary_policy is set to 'ADAPTIVE'.
Default value is 1048576 (1MiB).
| val | maximum dictionary size |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::max_page_fragment_size | ( | size_type | val | ) |
Sets the maximum page fragment size, in rows.
Files with nested schemas or very long strings may need a page fragment size smaller than the default value of 5000 to ensure a single fragment will not exceed the desired maximum page size in bytes.
| val | maximum page fragment size |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::max_page_size_bytes | ( | size_t | val | ) |
Sets the maximum uncompressed page size, in bytes.
Serves as a hint to the writer, and can be exceeded under certain circumstances. Cannot be larger than the row group size in bytes, and will be adjusted to match if it is.
| val | maximum page size |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::max_page_size_rows | ( | size_type | val | ) |
Sets the maximum page size, in rows. Counts only top-level rows, ignoring any nesting. Cannot be larger than the row group size in rows, and will be adjusted to match if it is.
| val | maximum rows per page |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::metadata | ( | table_input_metadata | metadata | ) |
Sets metadata.
| metadata | Associated metadata |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::row_group_size_bytes | ( | size_t | val | ) |
Sets the maximum row group size, in bytes.
| val | maximum row group size |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::row_group_size_rows | ( | size_type | val | ) |
Sets the maximum number of rows in output row groups.
| val | maximum number or rows |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::sorting_columns | ( | std::vector< sorting_column > | sorting_columns | ) |
Sets column sorting metadata.
| sorting_columns | Column sort order metadata |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::stats_level | ( | statistics_freq | sf | ) |
Sets the level of statistics.
| sf | Level of statistics requested in the output file |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::utc_timestamps | ( | bool | enabled | ) |
Set to true if timestamps are to be written as UTC.
| enabled | Boolean value to enable/disable writing of timestamps as UTC. |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::write_arrow_schema | ( | bool | enabled | ) |
Set to true if arrow schema is to be written.
| enabled | Boolean value to enable/disable writing of arrow schema |
| BuilderT& cudf::io::parquet_writer_options_builder_base< BuilderT, OptionsT >::write_v2_headers | ( | bool | enabled | ) |
Set to true if V2 page headers are to be written.
| enabled | Boolean value to enable/disable writing of V2 page headers. |
1.9.1