cudf.io.parquet.ParquetDatasetWriter#

class cudf.io.parquet.ParquetDatasetWriter(path, partition_cols, index=None, compression=None, statistics='ROWGROUP', max_file_size=None, file_name_prefix=None, **kwargs)#

Write a parquet file or dataset incrementally

Parameters
pathstr

A local directory path or S3 URL. Will be used as root directory path while writing a partitioned dataset.

partition_colslist

Column names by which to partition the dataset Columns are partitioned in the order they are given

indexbool, default None

If True, include the dataframe’s index(es) in the file output. If False, they will not be written to the file. If None, index(es) other than RangeIndex will be saved as columns.

compression{‘snappy’, None}, default ‘snappy’

Name of the compression to use. Use None for no compression.

statistics{‘ROWGROUP’, ‘PAGE’, ‘NONE’}, default ‘ROWGROUP’

Level at which column statistics should be included in file.

max_file_sizeint or str, default None

A file size that cannot be exceeded by the writer. It is in bytes, if the input is int. Size can also be a str in form or “10 MB”, “1 GB”, etc. If this parameter is used, it is mandatory to pass file_name_prefix.

file_name_prefixstr

This is a prefix to file names generated only when max_file_size is specified.

Examples

Using a context

>>> df1 = cudf.DataFrame({"a": [1, 1, 2, 2, 1], "b": [9, 8, 7, 6, 5]})
>>> df2 = cudf.DataFrame({"a": [1, 3, 3, 1, 3], "b": [4, 3, 2, 1, 0]})
>>> with ParquetDatasetWriter("./dataset", partition_cols=["a"]) as cw:
...     cw.write_table(df1)
...     cw.write_table(df2)

By manually calling close()

>>> cw = ParquetDatasetWriter("./dataset", partition_cols=["a"])
>>> cw.write_table(df1)
>>> cw.write_table(df2)
>>> cw.close()

Both the methods will generate the same directory structure

dataset/
    a=1
        <filename>.parquet
    a=2
        <filename>.parquet
    a=3
        <filename>.parquet

Methods

close([return_metadata])

Close all open files and optionally return footer metadata as a binary blob

write_table(df)

Write a dataframe to the file/dataset