Sub-Word Tokenizer In NLP

pip install transformers

from transformers import AutoTokenizer

# Load a pretrained tokenizer (e.g., BERT's WordPiece tokenizer)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "Playfulness makes NLP interesting"

# Tokenize
tokens = tokenizer.tokenize(text)
print("Subword Tokens:", tokens)

Query/Feedback