Skip to main content

Ctrl+K

cuDF User Guide
cudf.pandas
Polars GPU engine
pylibcudf documentation
libcudf documentation
- Developer Guide

GitHub
Twitter

cudf

cucim cudf-java cudf cugraph cuml cuproj cuspatial cuvs cuxfilter dask-cuda dask-cudf kvikio libcudf libcuml libcuproj libcuspatial libkvikio librmm libucxx raft rapids-cmake rmm

nightly (25.04)

nightly (25.04)stable (25.02)legacy (24.12)

cuDF User Guide
cudf.pandas
Polars GPU engine
pylibcudf documentation
libcudf documentation
Developer Guide

GitHub
Twitter

Section Navigation

API reference
10 Minutes to cuDF and Dask cuDF
Comparison of cuDF and Pandas
Supported Data Types
Input / Output
- Input / Output
- Working with JSON data
Working with missing data
GroupBy
Overview of User Defined Functions with cuDF
Interoperability between cuDF and CuPy
Options
Performance comparisons
- Performance comparison
Pandas Compatibility Notes
Copy-on-write
Memory Profiling
Breaking changes for pandas 2 in cuDF 24.04+

cuDF User Guide
API reference
WordPieceTokenizer

WordPieceTokenizer#

Constructor#

`WordPieceVocabulary`(vocabulary)	A vocabulary object used to tokenize input text.
`WordPieceVocabulary.tokenize`(text[, ...])

previous

cudf.core.character_normalizer.CharacterNormalizer.normalize

next

cudf.core.wordpiece_tokenize.WordPieceVocabulary

On this page

Constructor

This Page

Show Source

© Copyright 2018-2025, NVIDIA Corporation.

Created using Sphinx 8.2.3.

Built with the PyData Sphinx Theme 0.16.1.