nvtext::unicode_normalizer Struct Reference

Normalizer object for Unicode TR15 normalization. More...

#include <nvtext/unicode_normalize.hpp>

Public Member Functions
	unicode_normalizer (cudf::table_view const &unicode_data, unicode_normalization_form form, cuda::stream_ref stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
	Construct a unicode_normalizer from UnicodeData.txt columns. More...

Detailed Description

Normalizer object for Unicode TR15 normalization.

The normalizer is constructed from the contents of the Unicode Character Database UnicodeData.txt file, loaded by the caller as a cudf table (e.g. via cudf::io::read_csv).

The file is published by the Unicode Consortium and can be downloaded from: https://unicode.org/Public/15.1.0/ucd/UnicodeData.txt It is a semicolon-delimited file with 15 fields per row. The three fields required by this API are field 0 (code point hex), field 3 (CCC), and field 5 (decomposition mapping).

The unicode_data table must contain exactly three columns in the following order, corresponding to the fields above:

• column[0]: STRING Code point values as uppercase hex strings (e.g. "00C9")
• column[1]: INT32 Canonical_Combining_Class (CCC) values in range [0, 254]
• column[2]: STRING Decomposition_Mapping field; empty string for identity mappings, optionally prefixed with a compatibility tag such as <compat>, <font>, <wide>, etc.

// typical workflow
cudf::io::csv_reader_options in_opts =
  cudf::io::csv_reader_options::builder(cudf::io::source_info("UnicodeData.txt"))
   .delimiter(';').header(-1)
   .use_cols_indexes({0, 3, 5})
   .dtypes({cudf::data_type{cudf::type_id::STRING}, cudf::data_type{cudf::type_id::INT32},
            cudf::data_type{cudf::type_id::STRING}});
auto const ud = cudf::io::read_csv(in_opts);
auto normalizer = nvtext::create_unicode_normalizer(ud.tbl->view(), NFKC);
// the following can be called repeatedly on different strings columns with the same normalizer
auto result = nvtext::normalize_unicode(input_strings, *normalizer);

Decomposition of Hangul syllables (U+AC00..U+D7A3) is performed algorithmically per the Unicode standard and does not require entries in the provided table.

Composition exclusions (singletons, non-starter decompositions, and the ~70 Unicode-specified explicit exclusions) are computed internally.

Definition at line 76 of file unicode_normalize.hpp.

Constructor & Destructor Documentation

unicode_normalizer()

nvtext::unicode_normalizer::unicode_normalizer	(	cudf::table_view const &	unicode_data,
		unicode_normalization_form	form,
		cuda::stream_ref	stream = cudf::get_default_stream(),
		rmm::device_async_resource_ref	mr = cudf::get_current_device_resource_ref()
	)

Construct a unicode_normalizer from UnicodeData.txt columns.

Parameters

unicode_data	Table with three columns as described above
form	Normalization form to apply
stream	CUDA stream used for device memory operations and kernel launches
mr	Device memory resource used to allocate internal tables

---

The documentation for this struct was generated from the following file:

• nvtext/unicode_normalize.hpp