Hugging Face tokenizers is the Rust-backed library that turns text into model input ids — blazing fast, with full offset tracking back to the original string. It's what powers the "Fast" tokenizers in 🌐 Transformers. A tokenizer is a pipeline: normalizer → pre-tokenizer → model (BPE/WordPiece/Unigram) → post-processor → decoder. Use it to train a new vocabulary or run an existing one. This sheet targets tokenizers 0.2x.
Verified 2026-08-24 against the official docs at huggingface.co/docs/tokenizers and the huggingface/tokenizers repo. Python bindings over a Rust core; integrates with Transformers via PreTrainedTokenizerFast.
Most people either (a) load an existing tokenizer and encode, or (b) assemble the 5-stage pipeline and train a new one. Both below.
Load and encode — the common case.
pip install tokenizers★transformers. Prebuilt Rust wheels — no compiler needed.from tokenizers import Tokenizer
tok = Tokenizer.from_pretrained("bert-base-uncased")★Tokenizer.from_file("tokenizer.json") for a local one.tok.get_vocab_size() · tok.get_vocab() · tok.token_to_id("the")enc = tok.encode("Hello, world!")★Encoding — the rich result object, not just ids.enc.ids · enc.tokens · enc.attention_mask · enc.type_ids★enc.ids → the integer input; enc.tokens → the human-readable subwords.enc.offsets # [(start,end), ...] char spans in the original text★tok.decode(enc.ids) · tok.encode("A", "B") # sentence pairAssemble the pipeline from parts.
# 1 normalizer 2 pre_tokenizer 3 model 4 post_processor 5 decoder★from tokenizers import Tokenizer
from tokenizers.models import BPE
tok = Tokenizer(BPE(unk_token="[UNK]"))★tok.normalizer = ... · tok.pre_tokenizer = ... · tok.decoder = ...from tokenizers.models import BPE, WordPiece, Unigram, WordLevel★from tokenizers import normalizers
tok.normalizer = normalizers.Sequence([NFD(), Lowercase(), StripAccents()])★Sequence.from tokenizers import pre_tokenizers
tok.pre_tokenizer = pre_tokenizers.Whitespace() # or ByteLevel(), Metaspace()★ByteLevel (GPT-style, handles any byte) is common for BPE.pre_tokenizers.Sequence([Whitespace(), Digits(individual_digits=True)])Learn a vocabulary from your corpus.
from tokenizers.trainers import BpeTrainer
trainer = BpeTrainer(vocab_size=30000, special_tokens=["[UNK]","[CLS]","[SEP]","[PAD]","[MASK]"])★BpeTrainer, WordPieceTrainer, UnigramTrainer, WordLevelTrainer).tok.train(["corpus1.txt", "corpus2.txt"], trainer)★tok.train_from_iterator(text_generator, trainer, length=n)★datasets stream, a generator) — no need to dump to disk.BpeTrainer(..., min_frequency=2, show_progress=True)initial_alphabet / ByteLevel.alphabet() to guarantee full byte coverage.from tokenizers.processors import TemplateProcessing
tok.post_processor = TemplateProcessing(
single="[CLS] $A [SEP]", pair="[CLS] $A [SEP] $B:1 [SEP]:1",
special_tokens=[("[CLS]", cls_id), ("[SEP]", sep_id)])★[CLS]/[SEP] and set type ids. $A/$B are the sequences; :1 = type id.from tokenizers import decoders
tok.decoder = decoders.ByteLevel() # or WordPiece(), Metaspace()Batches, alignment, and handing off to Transformers.
tok.enable_padding(pad_id=0, pad_token="[PAD]")★length= for fixed). Set on the tokenizer, applies to all encodes.tok.enable_truncation(max_length=512)★strategy controls which sequence gets cut in pairs.encs = tok.encode_batch(["a", "b", "c"])★Encodings.tok.encode(text, add_special_tokens=False)enc.word_ids · enc.words # which word each token belongs to★enc.char_to_token(pos) · enc.token_to_chars(i)★enc.token_to_word(i) · enc.word_to_tokens(w)tok.save("tokenizer.json")★Tokenizer.from_file(...) to reload.from transformers import PreTrainedTokenizerFast
hf = PreTrainedTokenizerFast(tokenizer_object=tok)★pad_token etc.hf.save_pretrained("my-tok/"); hf.push_to_hub("me/my-tok")AutoTokenizer.from_pretrained alongside a model.decoder must match pre-tokenizerByteLevel pre-tokenizer needs a ByteLevel decoder (etc.) or decode() produces garbled spacing.add special tokens in the vocab AND post-processor[CLS]/[SEP] must be in the trainer's special_tokens and referenced by the post-processor's template.this library ≠ SentencePiece.model SentencePiece files aren't loaded directly — convert or retrain.offsets are on the normalized text unless configuredByteLevel(add_prefix_space) settings) so char offsets line up with your raw input.