Normalizing

Files
file	normalize.hpp
	APIs for normalizing whitespace and characters within strings columns.
file	unicode_normalize.hpp
	APIs for Unicode TR15 normalization of strings columns.

Classes
struct	nvtext::character_normalizer
	Normalizer object to be used with nvtext::normalize_characters. More...
struct	nvtext::unicode_normalizer
	Normalizer object for Unicode TR15 normalization. More...

Enumerations
enum class	nvtext::unicode_normalization_form : int32_t { nvtext::NFD , nvtext::NFC , nvtext::NFKD , nvtext::NFKC }
	Unicode normalization form per Unicode Standard Annex #15. More...

Functions
std::unique_ptr< cudf::column >	nvtext::normalize_spaces (cudf::strings_column_view const &input, cuda::stream_ref stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
	Returns a new strings column by normalizing the whitespace in each string in the input column. More...
std::unique_ptr< character_normalizer >	nvtext::create_character_normalizer (bool do_lower_case, cudf::strings_column_view const &special_tokens=cudf::strings_column_view(cudf::column_view{ cudf::data_type{cudf::type_id::STRING}, 0, nullptr, nullptr, 0}), cuda::stream_ref stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
	Create a normalizer object. More...
std::unique_ptr< cudf::column >	nvtext::normalize_characters (cudf::strings_column_view const &input, character_normalizer const &normalizer, cuda::stream_ref stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
	Normalizes the text in input strings column. More...
std::unique_ptr< unicode_normalizer >	nvtext::create_unicode_normalizer (cudf::table_view const &unicode_data, unicode_normalization_form form, cuda::stream_ref stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
	Create a unicode_normalizer object. More...
std::unique_ptr< cudf::column >	nvtext::normalize_unicode (cudf::strings_column_view const &input, unicode_normalizer const &normalizer, cuda::stream_ref stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
	Normalize a strings column using Unicode TR15 normalization. More...

Detailed Description

Enumeration Type Documentation

unicode_normalization_form

enum nvtext::unicode_normalization_form : int32_t

strong

#include <nvtext/unicode_normalize.hpp>

Unicode normalization form per Unicode Standard Annex #15.

https://unicode.org/reports/tr15/

Enumerator
NFD	Canonical Decomposition.
NFC	Canonical Decomposition followed by Canonical Composition.
NFKD	Compatibility Decomposition.
NFKC	Compatibility Decomposition followed by Canonical Composition.

Definition at line 27 of file unicode_normalize.hpp.

Function Documentation

create_character_normalizer()

std::unique_ptr<character_normalizer> nvtext::create_character_normalizer	(	bool	do_lower_case,
		cudf::strings_column_view const &	special_tokens = cudf::strings_column_view(cudf::column_view{ cudf::data_type{cudf::type_id::STRING}, 0, nullptr, nullptr, 0}),
		cuda::stream_ref	stream = cudf::get_default_stream(),
		rmm::device_async_resource_ref	mr = cudf::get_current_device_resource_ref()
	)

#include <nvtext/normalize.hpp>

Create a normalizer object.

Creates a normalizer object which can be reused on multiple calls to nvtext::normalize_characters

See also

nvtext::character_normalizer

Parameters

do_lower_case	If true, upper-case characters are converted to lower-case and accents are stripped from those characters. If false, accented and upper-case characters are not transformed.
special_tokens	Individual tokens including [] brackets. Default is no special tokens.
stream	CUDA stream used for device memory operations and kernel launches
mr	Device memory resource used to allocate the returned column's device memory

Returns

Object to be used with nvtext::normalize_characters

create_unicode_normalizer()

std::unique_ptr<unicode_normalizer> nvtext::create_unicode_normalizer	(	cudf::table_view const &	unicode_data,
		unicode_normalization_form	form,
		cuda::stream_ref	stream = cudf::get_default_stream(),
		rmm::device_async_resource_ref	mr = cudf::get_current_device_resource_ref()
	)

#include <nvtext/unicode_normalize.hpp>

Create a unicode_normalizer object.

See also

nvtext::unicode_normalizer

Parameters

unicode_data	Table with three columns parsed from UnicodeData.txt
form	Normalization form to apply
stream	CUDA stream used for device memory operations and kernel launches
mr	Device memory resource used to allocate internal tables

Returns

Normalizer object to be reused across calls to nvtext::normalize_unicode

normalize_characters()

std::unique_ptr<cudf::column> nvtext::normalize_characters	(	cudf::strings_column_view const &	input,
		character_normalizer const &	normalizer,
		cuda::stream_ref	stream = cudf::get_default_stream(),
		rmm::device_async_resource_ref	mr = cudf::get_current_device_resource_ref()
	)

#include <nvtext/normalize.hpp>

Normalizes the text in input strings column.

See also

nvtext::character_normalizer for details on the normalizer behavior

cn = create_character_normalizer(true)
s = ["éâîô\teaio", "ĂĆĖÑÜ", "ACENU", "$24.08", "[a,bb]"]
s1 = normalize_characters(s,cn)
s1 is now ["eaio eaio", "acenu", "acenu", " $ 24 . 08", " [ a , bb ] "]
 
cn = create_character_normalizer(false)
s2 = normalize_characters(s,cn)
s2 is now ["éâîô eaio", "ĂĆĖÑÜ", "ACENU", " $ 24 . 08", " [ a , bb ] "]

A null input element at row i produces a corresponding null entry for row i in the output column.

Parameters

input	The input strings to normalize
normalizer	Normalizer to use for this function
stream	CUDA stream used for device memory operations and kernel launches
mr	Memory resource to allocate any returned objects

Returns

Normalized strings column

normalize_spaces()

std::unique_ptr<cudf::column> nvtext::normalize_spaces	(	cudf::strings_column_view const &	input,
		cuda::stream_ref	stream = cudf::get_default_stream(),
		rmm::device_async_resource_ref	mr = cudf::get_current_device_resource_ref()
	)

#include <nvtext/normalize.hpp>

Returns a new strings column by normalizing the whitespace in each string in the input column.

Normalizing a string replaces any number of whitespace character (character code-point <= ' ') runs with a single space ' ' and trims whitespace from the beginning and end of the string.

Example:
s = ["a b", "  c  d\n", "e \t f "]
t = normalize_spaces(s)
t is now ["a b","c d","e f"]

A null input element at row i produces a corresponding null entry for row i in the output column.

Parameters

input	Strings column to normalize
mr	Device memory resource used to allocate the returned column's device memory
stream	CUDA stream used for device memory operations and kernel launches

Returns

New strings columns of normalized strings.

normalize_unicode()

std::unique_ptr<cudf::column> nvtext::normalize_unicode	(	cudf::strings_column_view const &	input,
		unicode_normalizer const &	normalizer,
		cuda::stream_ref	stream = cudf::get_default_stream(),
		rmm::device_async_resource_ref	mr = cudf::get_current_device_resource_ref()
	)

#include <nvtext/unicode_normalize.hpp>

Normalize a strings column using Unicode TR15 normalization.

Input is UTF-8 encoded and output is UTF-8 encoded. Each string is normalized independently. Null entries produce null output entries.

cn = create_unicode_normalizer(unicode_table, NFKC)
s  = ["é", "fi", "가", "hello"]
r  = normalize_unicode(s, cn)
// NFD:  ["é", "fi",   "가", "hello"]
// NFC:  ["é",       "fi",   "가",            "hello"]
// NFKD: ["é", "fi",  "가", "hello"]
// NFKC: ["é",       "fi",  "가",            "hello"]

During each call to nvtext::normalize_unicode, the amount of temporary memory required is approximately 16x the input data size.

Parameters

input	Strings column to normalize
normalizer	Normalizer object created by nvtext::create_unicode_normalizer
stream	CUDA stream used for device memory operations and kernel launches
mr	Device memory resource used to allocate the returned column's device memory

Returns

New strings column of normalized UTF-8 strings