Articles → NLP → Word Tokenizer In NLP

Word Tokenizer In NLP

This article describes the word tokenizer in NLP.

Purpose

A word tokenizer is a tool in Natural Language Processing (NLP) that splits a piece of text into individual words.

Consider the following example: -

from nltk.tokenize import word_tokenize

sentence = "I love AI and NLP!"
tokens = word_tokenize(sentence)
print(tokens)

Query/Feedback