Go to the source code of this file.
|
std::unique_ptr< hashed_vocabulary > | nvtext::load_vocabulary_file (std::string const &filename_hashed_vocabulary, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref()) |
| Load the hashed vocabulary file into device memory. More...
|
|
tokenizer_result | nvtext::subword_tokenize (cudf::strings_column_view const &strings, hashed_vocabulary const &vocabulary_table, uint32_t max_sequence_length, uint32_t stride, bool do_lower_case, bool do_truncate, rmm::cuda_stream_view stream=cudf::get_default_stream(), rmm::device_async_resource_ref mr=cudf::get_current_device_resource_ref()) |
| Creates a tokenizer that cleans the text, splits it into tokens and returns token-ids from an input vocabulary. More...
|
|