Files | |
file | aggregation.hpp |
Representation for specifying desired aggregations from aggregation-based APIs, e.g., groupby, reductions, rolling, etc. | |
Classes | |
class | cudf::aggregation |
Abstract base class for specifying the desired aggregation in an aggregation_request . More... | |
class | cudf::rolling_aggregation |
Derived class intended for rolling_window specific aggregation usage. More... | |
class | cudf::groupby_aggregation |
Derived class intended for groupby specific aggregation usage. More... | |
class | cudf::groupby_scan_aggregation |
Derived class intended for groupby specific scan usage. More... | |
class | cudf::reduce_aggregation |
Derived class intended for reduction usage. More... | |
class | cudf::scan_aggregation |
Derived class intended for scan usage. More... | |
class | cudf::segmented_reduce_aggregation |
Derived class intended for segmented reduction usage. More... | |
Enumerations | |
enum class | cudf::rank_percentage : int32_t { cudf::NONE , cudf::ZERO_NORMALIZED , cudf::ONE_NORMALIZED } |
Whether returned rank should be percentage or not and mention the type of percentage normalization. More... | |
enum class | cudf::udf_type : bool { CUDA , PTX } |
Type of code in the user defined function string. | |
enum class | cudf::correlation_type : int32_t { PEARSON , KENDALL , SPEARMAN } |
Type of correlation method. | |
enum class | cudf::ewm_history : int32_t { INFINITE , FINITE } |
Type of treatment of EWM input values' first value. | |
Functions | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_sum_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_product_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_min_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_max_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_count_aggregation (null_policy null_handling=null_policy::EXCLUDE) |
Factory to create a COUNT aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_any_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_all_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_histogram_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_sum_of_squares_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_mean_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_m2_aggregation () |
Factory to create a M2 aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_variance_aggregation (size_type ddof=1) |
Factory to create a VARIANCE aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_std_aggregation (size_type ddof=1) |
Factory to create a STD aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_median_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_quantile_aggregation (std::vector< double > const &quantiles, interpolation interp=interpolation::LINEAR) |
Factory to create a QUANTILE aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_argmax_aggregation () |
Factory to create an ARGMAX aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_argmin_aggregation () |
Factory to create an ARGMIN aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_nunique_aggregation (null_policy null_handling=null_policy::EXCLUDE) |
Factory to create a NUNIQUE aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_nth_element_aggregation (size_type n, null_policy null_handling=null_policy::INCLUDE) |
Factory to create a NTH_ELEMENT aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_row_number_aggregation () |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_ewma_aggregation (double const center_of_mass, ewm_history history) |
Factory to create an EWMA aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_rank_aggregation (rank_method method, order column_order=order::ASCENDING, null_policy null_handling=null_policy::EXCLUDE, null_order null_precedence=null_order::AFTER, rank_percentage percentage=rank_percentage::NONE) |
Factory to create a RANK aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_collect_list_aggregation (null_policy null_handling=null_policy::INCLUDE) |
Factory to create a COLLECT_LIST aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_collect_set_aggregation (null_policy null_handling=null_policy::INCLUDE, null_equality nulls_equal=null_equality::EQUAL, nan_equality nans_equal=nan_equality::ALL_EQUAL) |
Factory to create a COLLECT_SET aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_lag_aggregation (size_type offset) |
Factory to create a LAG aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_lead_aggregation (size_type offset) |
Factory to create a LEAD aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_udf_aggregation (udf_type type, std::string const &user_defined_aggregator, data_type output_type) |
Factory to create an aggregation base on UDF for PTX or CUDA. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_merge_lists_aggregation () |
Factory to create a MERGE_LISTS aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_merge_sets_aggregation (null_equality nulls_equal=null_equality::EQUAL, nan_equality nans_equal=nan_equality::ALL_EQUAL) |
Factory to create a MERGE_SETS aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_merge_m2_aggregation () |
Factory to create a MERGE_M2 aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_merge_histogram_aggregation () |
Factory to create a MERGE_HISTOGRAM aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_covariance_aggregation (size_type min_periods=1, size_type ddof=1) |
Factory to create a COVARIANCE aggregation. More... | |
template<typename Base = aggregation> | |
std::unique_ptr< Base > | cudf::make_correlation_aggregation (correlation_type type, size_type min_periods=1) |
Factory to create a CORRELATION aggregation. More... | |
template<typename Base > | |
std::unique_ptr< Base > | cudf::make_tdigest_aggregation (int max_centroids=1000) |
Factory to create a TDIGEST aggregation. More... | |
template<typename Base > | |
std::unique_ptr< Base > | cudf::make_merge_tdigest_aggregation (int max_centroids=1000) |
Factory to create a MERGE_TDIGEST aggregation. More... | |
|
strong |
Whether returned rank should be percentage or not and mention the type of percentage normalization.
Enumerator | |
---|---|
NONE | rank |
ZERO_NORMALIZED | rank / count |
ONE_NORMALIZED | (rank - 1) / (count - 1) |
Definition at line 67 of file aggregation.hpp.
std::unique_ptr<Base> cudf::make_all_aggregation | ( | ) |
Factory to create a ALL aggregation
std::unique_ptr<Base> cudf::make_any_aggregation | ( | ) |
Factory to create an ANY aggregation
std::unique_ptr<Base> cudf::make_argmax_aggregation | ( | ) |
Factory to create an ARGMAX aggregation.
ARGMAX returns the index of the maximum element.
std::unique_ptr<Base> cudf::make_argmin_aggregation | ( | ) |
Factory to create an ARGMIN aggregation.
argmin
returns the index of the minimum element.
std::unique_ptr<Base> cudf::make_collect_list_aggregation | ( | null_policy | null_handling = null_policy::INCLUDE | ) |
Factory to create a COLLECT_LIST aggregation.
COLLECT_LIST
returns a list column of all included elements in the group/series.
If null_handling
is set to EXCLUDE
, null elements are dropped from each of the list rows.
null_handling | Indicates whether to include/exclude nulls in list elements |
std::unique_ptr<Base> cudf::make_collect_set_aggregation | ( | null_policy | null_handling = null_policy::INCLUDE , |
null_equality | nulls_equal = null_equality::EQUAL , |
||
nan_equality | nans_equal = nan_equality::ALL_EQUAL |
||
) |
Factory to create a COLLECT_SET aggregation.
COLLECT_SET
returns a lists column of all included elements in the group/series. Within each list, the duplicated entries are dropped out such that each entry appears only once.
If null_handling
is set to EXCLUDE
, null elements are dropped from each of the list rows.
null_handling | Indicates whether to include/exclude nulls during collection |
nulls_equal | Flag to specify whether null entries within each list should be considered equal. |
nans_equal | Flag to specify whether NaN values in floating point column should be considered equal. |
std::unique_ptr<Base> cudf::make_correlation_aggregation | ( | correlation_type | type, |
size_type | min_periods = 1 |
||
) |
Factory to create a CORRELATION aggregation.
Compute correlation coefficient between two columns. The input columns are child columns of a non-nullable struct columns.
type | correlation_type |
min_periods | Minimum number of non-null observations required to produce a result |
std::unique_ptr<Base> cudf::make_count_aggregation | ( | null_policy | null_handling = null_policy::EXCLUDE | ) |
Factory to create a COUNT aggregation.
null_handling | Indicates if null values will be counted |
std::unique_ptr<Base> cudf::make_covariance_aggregation | ( | size_type | min_periods = 1 , |
size_type | ddof = 1 |
||
) |
Factory to create a COVARIANCE aggregation.
Compute covariance between two columns. The input columns are child columns of a non-nullable struct columns.
min_periods | Minimum number of non-null observations required to produce a result |
ddof | Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N is the number of non-null observations. |
std::unique_ptr<Base> cudf::make_ewma_aggregation | ( | double const | center_of_mass, |
ewm_history | history | ||
) |
Factory to create an EWMA aggregation.
EWMA
returns a non-nullable column with the same type as the input, whose values are the exponentially weighted moving average of the input sequence. Let these values be known as the y_i.
EWMA aggregations are parameterized by a center of mass (com
) which affects the contribution of the previous values (y_0 ... y_{i-1}) in computing the y_i.
EWMA aggregations are also parameterized by a history cudf::ewm_history
. Special considerations have to be given to the mathematical treatment of the first value of the input sequence. There are two approaches to this, one which considers the first value of the sequence to be the exponential weighted moving average of some infinite history of data, and one which takes the first value to be the only datapoint known. These assumptions lead to two different formulas for the y_i. ewm_history
selects which.
EWMA aggregations have special null handling. Nulls have two effects. The first is to propagate forward the last valid value as far as it has been computed. This could be thought of as the nulls not affecting the average in any way. The second effect changes the way the y_i are computed. Since a moving average is conceptually designed to weight contributing values by their recency, nulls ought to count as valid periods even though they do not change the average. For example, if the input sequence is {1, NULL, 3} then when computing y_2 one should weigh y_0 as if it occurs two periods before y_2 rather than just one.
center_of_mass | the center of mass. |
history | which assumption to make about the first value |
std::unique_ptr<Base> cudf::make_histogram_aggregation | ( | ) |
Factory to create a HISTOGRAM aggregation
std::unique_ptr<Base> cudf::make_lag_aggregation | ( | size_type | offset | ) |
Factory to create a LAG aggregation.
offset | The number of rows to lag the input |
std::unique_ptr<Base> cudf::make_lead_aggregation | ( | size_type | offset | ) |
Factory to create a LEAD aggregation.
offset | The number of rows to lead the input |
std::unique_ptr<Base> cudf::make_m2_aggregation | ( | ) |
Factory to create a M2 aggregation.
A M2 aggregation is sum of squares of differences from the mean. That is: M2 = SUM((x - MEAN) * (x - MEAN))
.
This aggregation produces the intermediate values that are used to compute variance and standard deviation across multiple discrete sets. See https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm
for more detail.
std::unique_ptr<Base> cudf::make_max_aggregation | ( | ) |
Factory to create a MAX aggregation
std::unique_ptr<Base> cudf::make_mean_aggregation | ( | ) |
Factory to create a MEAN aggregation
std::unique_ptr<Base> cudf::make_median_aggregation | ( | ) |
Factory to create a MEDIAN aggregation
std::unique_ptr<Base> cudf::make_merge_histogram_aggregation | ( | ) |
Factory to create a MERGE_HISTOGRAM aggregation.
Merges the results of HISTOGRAM
aggregations on independent sets into a new HISTOGRAM
value equivalent to if a single HISTOGRAM
aggregation was done across all of the sets at once.
std::unique_ptr<Base> cudf::make_merge_lists_aggregation | ( | ) |
Factory to create a MERGE_LISTS aggregation.
Given a lists column, this aggregation merges all the lists corresponding to the same key value into one list. It is designed specifically to merge the partial results of multiple (distributed) groupby COLLECT_LIST
aggregations into a final COLLECT_LIST
result. As such, it requires the input lists column to be non-nullable (the child column containing list entries is not subjected to this requirement).
std::unique_ptr<Base> cudf::make_merge_m2_aggregation | ( | ) |
Factory to create a MERGE_M2 aggregation.
Merges the results of M2
aggregations on independent sets into a new M2
value equivalent to if a single M2
aggregation was done across all of the sets at once. This aggregation is only valid on structs whose members are the result of the COUNT_VALID
, MEAN
, and M2
aggregations on the same sets. The output of this aggregation is a struct containing the merged COUNT_VALID
, MEAN
, and M2
aggregations.
The input M2
aggregation values are expected to be all non-negative numbers, since they were output from M2
aggregation.
std::unique_ptr<Base> cudf::make_merge_sets_aggregation | ( | null_equality | nulls_equal = null_equality::EQUAL , |
nan_equality | nans_equal = nan_equality::ALL_EQUAL |
||
) |
Factory to create a MERGE_SETS aggregation.
Given a lists column, this aggregation firstly merges all the lists corresponding to the same key value into one list, then it drops all the duplicate entries in each lists, producing a lists column containing non-repeated entries.
This aggregation is designed specifically to merge the partial results of multiple (distributed) groupby COLLECT_LIST
or COLLECT_SET
aggregations into a final COLLECT_SET
result. As such, it requires the input lists column to be non-nullable (the child column containing list entries is not subjected to this requirement).
In practice, the input (partial results) to this aggregation should be generated by (distributed) COLLECT_LIST
aggregations, not COLLECT_SET
, to avoid unnecessarily removing duplicate entries for the partial results.
nulls_equal | Flag to specify whether nulls within each list should be considered equal during dropping duplicate list entries. |
nans_equal | Flag to specify whether NaN values in floating point column should be considered equal during dropping duplicate list entries. |
std::unique_ptr<Base> cudf::make_merge_tdigest_aggregation | ( | int | max_centroids = 1000 | ) |
Factory to create a MERGE_TDIGEST aggregation.
Merges the results from a previous aggregation resulting from a make_tdigest_aggregation
or make_merge_tdigest_aggregation
to produce a new a tdigest (https://arxiv.org/pdf/1902.04023.pdf) column.
The tdigest column produced is of the following structure:
struct { // centroids for the digest list { struct { double // mean double // weight }, ... } // these are from the input stream, not the centroids. they are used // during the percentile_approx computation near the beginning or // end of the quantiles double // min double // max }
Each output row is a single tdigest. The length of the row is the "size" of the tdigest, each element of which represents a weighted centroid (mean, weight).
max_centroids | Parameter controlling compression level and accuracy on subsequent queries on the output tdigest data. max_centroids places an upper bound on the size of the computed tdigests: A value of 1000 will result in a tdigest containing no more than 1000 centroids (32 bytes each). Higher result in more accurate tdigest information. |
std::unique_ptr<Base> cudf::make_min_aggregation | ( | ) |
Factory to create a MIN aggregation
std::unique_ptr<Base> cudf::make_nth_element_aggregation | ( | size_type | n, |
null_policy | null_handling = null_policy::INCLUDE |
||
) |
Factory to create a NTH_ELEMENT aggregation.
NTH_ELEMENT returns the n'th element of the group/series.
If n
is not within the range [-group_size, group_size)
, the result of the respective group will be null. Negative indices [-group_size, -1]
corresponds to [0, group_size-1]
indices respectively where group_size
is the size of each group.
n | index of nth element in each group |
null_handling | Indicates to include/exclude nulls during indexing |
std::unique_ptr<Base> cudf::make_nunique_aggregation | ( | null_policy | null_handling = null_policy::EXCLUDE | ) |
Factory to create a NUNIQUE aggregation.
NUNIQUE returns the number of unique elements.
null_handling | Indicates if null values will be counted |
std::unique_ptr<Base> cudf::make_product_aggregation | ( | ) |
Factory to create a PRODUCT aggregation
std::unique_ptr<Base> cudf::make_quantile_aggregation | ( | std::vector< double > const & | quantiles, |
interpolation | interp = interpolation::LINEAR |
||
) |
Factory to create a QUANTILE aggregation.
quantiles | The desired quantiles |
interp | The desired interpolation |
std::unique_ptr<Base> cudf::make_rank_aggregation | ( | rank_method | method, |
order | column_order = order::ASCENDING , |
||
null_policy | null_handling = null_policy::EXCLUDE , |
||
null_order | null_precedence = null_order::AFTER , |
||
rank_percentage | percentage = rank_percentage::NONE |
||
) |
Factory to create a RANK aggregation.
RANK
returns a column of size_type or double "ranks" (see note 3 below for how the data type is determined) for a given rank method and column order. If nulls are excluded, the rank will be null for those rows, otherwise a non-nullable column is returned. Double precision column is returned only when percentage!=NONE and when rank method is average.
This aggregation only works with "scan" algorithms. The input column into the group or ungrouped scan is an orderby column that orders the rows that the aggregate function ranks. If rows are ordered by more than one column, the orderby input column should be a struct column containing the ordering columns.
Note:
RANK
aggregations are not compatible with exclusive scans.method | The ranking method used for tie breaking (same values) |
column_order | The desired sort order for ranking |
null_handling | flag to include nulls during ranking If nulls are not included, the corresponding rank will be null. |
null_precedence | The desired order of null compared to other elements for column |
percentage | enum to denote the type of conversion of ranks to percentage in range (0,1] |
std::unique_ptr<Base> cudf::make_row_number_aggregation | ( | ) |
Factory to create a ROW_NUMBER aggregation
std::unique_ptr<Base> cudf::make_std_aggregation | ( | size_type | ddof = 1 | ) |
Factory to create a STD aggregation.
ddof | Delta degrees of freedom. The divisor used in calculation of std is N - ddof , where N is the population size. |
cudf::logic_error | if input type is chrono or compound types. |
std::unique_ptr<Base> cudf::make_sum_aggregation | ( | ) |
Factory to create a SUM aggregation
std::unique_ptr<Base> cudf::make_sum_of_squares_aggregation | ( | ) |
Factory to create a SUM_OF_SQUARES aggregation
std::unique_ptr<Base> cudf::make_tdigest_aggregation | ( | int | max_centroids = 1000 | ) |
Factory to create a TDIGEST aggregation.
Produces a tdigest (https://arxiv.org/pdf/1902.04023.pdf) column from input values. The input aggregation values are expected to be fixed-width numeric types.
The tdigest column produced is of the following structure:
struct { // centroids for the digest list { struct { double // mean double // weight }, ... } // these are from the input stream, not the centroids. they are used // during the percentile_approx computation near the beginning or // end of the quantiles double // min double // max }
Each output row is a single tdigest. The length of the row is the "size" of the tdigest, each element of which represents a weighted centroid (mean, weight).
max_centroids | Parameter controlling compression level and accuracy on subsequent queries on the output tdigest data. max_centroids places an upper bound on the size of the computed tdigests: A value of 1000 will result in a tdigest containing no more than 1000 centroids (32 bytes each). Higher result in more accurate tdigest information. |
std::unique_ptr<Base> cudf::make_udf_aggregation | ( | udf_type | type, |
std::string const & | user_defined_aggregator, | ||
data_type | output_type | ||
) |
Factory to create an aggregation base on UDF for PTX or CUDA.
[in] | type | either udf_type::PTX or udf_type::CUDA |
[in] | user_defined_aggregator | A string containing the aggregator code |
[in] | output_type | expected output type |
std::unique_ptr<Base> cudf::make_variance_aggregation | ( | size_type | ddof = 1 | ) |
Factory to create a VARIANCE aggregation.
ddof | Delta degrees of freedom. The divisor used in calculation of variance is N - ddof , where N is the population size. |
cudf::logic_error | if input type is chrono or compound types. |