Classes | Namespaces | Functions
subword_tokenize.hpp File Reference
#include <cudf/column/column.hpp>
#include <cudf/column/column_view.hpp>
#include <cudf/strings/strings_column_view.hpp>
#include <cudf/utilities/export.hpp>
#include <cudf/utilities/memory_resource.hpp>

Go to the source code of this file.

Classes

struct  nvtext::hashed_vocabulary
 The vocabulary data for use with the subword_tokenize function. More...
 
struct  nvtext::tokenizer_result
 Result object for the subword_tokenize functions. More...
 

Namespaces

 nvtext
 NVText APIs.
 

Functions

std::unique_ptr< hashed_vocabulary > nvtext::load_vocabulary_file (std::string const &filename_hashed_vocabulary, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
 Load the hashed vocabulary file into device memory. More...
 
tokenizer_result nvtext::subword_tokenize (cudf::strings_column_view const &strings, hashed_vocabulary const &vocabulary_table, uint32_t max_sequence_length, uint32_t stride, bool do_lower_case, bool do_truncate, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref())
 Creates a tokenizer that cleans the text, splits it into tokens and returns token-ids from an input vocabulary. More...