Articles → NLP → Tokenization In NLP

Tokenization In NLP

This article describes tokenization in NLP.

What Is Tokenization?

The process of breaking down the text into smaller units (tokens) is called tokenization. The tokens can be words, characters, or even sub-words.

Query/Feedback