Articles → NLP → Tokenization In NLP

Tokenization In NLP

This article describes tokenization in NLP.

What Is Tokenization?

The process of breaking down the text into smaller units (tokens) is called tokenization. The tokens can be words, characters, or even sub-words.

NLP supports the following tokenization types.

Tokenization Type	Purpose
Word tokenizer	Splits a piece of text into individual words.
Sentence tokenizer	Split the text into sentences.
Blank line tokenizer	Splits text into paragraphs based on blank lines.
Regexp tokenizer	Splits text into tokens using regular expression (regex) patterns.
Sub-Word tokenizer	Split text into smaller units, larger than characters but smaller than words.

Query/Feedback