Python · Natural Language Processing · Quick Reference

nltk · Natural Language Toolkit 3.10

The whole toolkit is one idea repeated: text flows through a pipeline, changing shape at each step — a raw str becomes a list of tokens, then a list of (word, tag) tuples, then a Tree. NLTK bundles the tokenizers, taggers, stemmers, parsers and dozens of corpora to move text along that line. The catch: almost every step needs its data downloaded first. Colour-coded by stage; ★ marks daily-use calls.

setup · download · corpora tokenize · text · count normalize · stem · lemmatize POS tag · chunk · NER wordnet · n-grams · similarity classify · sentiment · parse gotcha ★most common
Verified against a live nltk 3.10.0 install (every pipeline step + gotcha run, not remembered) · nltk.org docs & the NLTK Book · the current nltk_data resource names · GeeksforGeeks / Real Python / DataCamp cheat sheets.
THE NLP PIPELINE — TOKENIZE · NORMALIZE · TAG · PARSE · ANALYZE Raw text str a document Tokenize word_tokenize sent_tokenize Normalize stopwords·lower stem·lemmatize POS Tag pos_tag (word, tag) Chunk · Parse ne_chunk → Tree Analyze FreqDist · WordNet · collocations classify · sentiment THE DATA CHANGES SHAPE AT EACH STEP Every function expects the type the previous one produced — pass the wrong shape and it errors. 1 · raw string "The dog barks." type: str 2 · tokens ['The','dog','barks','.'] type: list[str] 3 · tagged [('The','DT'), ('dog','NN'),...] type: list[tuple] 4 · tree Tree('S',[NP, VP]) type: nltk.Tree tokenize pos_tag chunk first, get the data · the #1 error Corpora & models load lazily — nothing ships with pip. import nltk nltk.download('punkt_tab') # tokenize nltk.download('averaged_perceptron_ tagger_eng') # pos_tag nltk.download('stopwords', 'wordnet') shortcut: nltk.download('popular') Names changed in 3.9+: punkt_tab, not punkt; ..._tagger_eng, not ..._tagger. then the pipeline in 6 lines toks = word_tokenize(text.lower()) toks = [w for w in toks if w not in sw] tags = pos_tag(toks) lem = [wnl.lemmatize(w) for w,t in tags] fd = FreqDist(lem) fd.most_common(10) tokenize → drop stopwords → tag → lemmatize → count. The everyday NLTK shape.
01

Setup & download

  • $ pip install nltk — the library. It ships no data.
  • import nltk
    nltk.download('punkt_tab') — fetch one resource. Bare download() opens a GUI picker.
  • nltk.download('popular') — the common bundle in one go; 'all' grabs everything (~3 GB).
  • nltk.download(['stopwords', 'wordnet']) — pass a list to fetch several.
  • nltk.data.path — where it looks; set NLTK_DATA env var to relocate.
  • Corpora are lazy — importing nltk.corpus.brown loads nothing until you call .words(). The LookupError you'll hit means "download that resource".
02

Tokenization

  • from nltk.tokenize import word_tokenize, sent_tokenize
  • word_tokenize(text) → list[str]. Splits punctuation off words. Needs punkt_tab.
  • sent_tokenize(text) → list of sentence strings (handles "Mr." etc.).
  • wordpunct_tokenize(text) — splits on every punctuation mark; no model needed.
  • TweetTokenizer().tokenize(t) — keeps @handles, #hashtags, emoji intact.
  • RegexpTokenizer(r'\w+').tokenize(t) — roll your own; \w+ grabs words, drops punctuation.
  • Everything downstream expects a token list — tokenize first, always.
03

The Text object

  • text = nltk.Text(tokens) — wraps a token list with exploration tools.
  • text.concordance('word') — every occurrence in context (KWIC).
  • text.similar('word') — words appearing in similar contexts.
  • text.common_contexts(['a','b']) — contexts two words share.
  • text.collocations() — frequent multi-word phrases.
  • text.dispersion_plot([...]) — where words fall across the text (needs matplotlib).
  • text.count('word') · text.concordance_list(...)
04

Counting & frequency

  • fd = nltk.FreqDist(tokens) — a dict subclass: token → count.
  • fd.most_common(10) → top-10 (token, count) pairs.
  • fd['the'] · fd.freq('the') — raw count · relative frequency.
  • fd.hapaxes() — words that appear exactly once.
  • fd.plot(20, cumulative=True) — frequency curve.
  • cfd = nltk.ConditionalFreqDist((c, w) ...) — counts split by condition (e.g. genre → word).
  • Lexical diversity = len(set(t))/len(t) — unique-word ratio.
05

Stopwords & cleaning

  • from nltk.corpus import stopwords
  • sw = set(stopwords.words('english')) — ~179 common words. set() makes lookups fast.
  • [w for w in toks if w.lower() not in sw] — the standard filter.
  • [w for w in toks if w.isalpha()] — drop punctuation & numbers.
  • stopwords.fileids() — 20+ languages: 'spanish', 'german'…
  • Lowercase before matching stopwords — the list is all-lowercase.
06

Stemming

  • from nltk.stem import PorterStemmer
    ps = PorterStemmer()
  • ps.stem('studies') → 'studi'. Fast, rule-based chopping — not a real word.
  • SnowballStemmer('english') — Porter2; better, and supports many languages.
  • LancasterStemmer() — most aggressive (shortest stems).
  • RegexpStemmer('ing$|s$') — strip suffixes by pattern.
  • Use stemming for search/indexing where the exact word doesn't matter — it's fast and crude.
07

Lemmatization

  • from nltk.stem import WordNetLemmatizer
    wnl = WordNetLemmatizer() — needs the wordnet corpus.
  • wnl.lemmatize('studies') → 'study' — a real dictionary word.
  • wnl.lemmatize('running', pos='v') → 'run'. Default pos is noun — lemmatize('running') stays 'running'!
  • pos = 'n' | 'v' | 'a' | 'r' — noun, verb, adjective, adverb.
  • Best practice: map the POS tag → WordNet pos, then lemmatize with it. Slower than stemming, but linguistically correct.
08

POS tagging

  • nltk.pos_tag(tokens) → [(word, tag), ...]. Needs averaged_perceptron_tagger_eng. Pass a list, not a string.
  • [('The','DT'), ('dog','NN'), ('runs','VBZ')] — tags are Penn Treebank by default.
  • nltk.pos_tag(toks, tagset='universal') → simple NOUN/VERB/DET. Needs universal_tagset.
  • nltk.help.upenn_tagset('NN') — look up what a tag means.
  • The tagger uses context: "runs" alone may tag NNS, but "dog runs" gives VBZ.
09

Chunking & NER

  • tree = nltk.ne_chunk(pos_tag(toks)) — named-entity chunks: PERSON, GPE, ORGANIZATION. Needs maxent_ne_chunker_tab + words.
  • grammar = r'NP: {<DT>?<JJ>*<NN>}'
    cp = nltk.RegexpParser(grammar) — define phrase patterns over POS tags.
  • cp.parse(tagged) → a Tree; group tokens into noun phrases etc.
  • for st in tree: if hasattr(st, 'label'): ... — iterate to pull out labelled subtrees.
  • nltk.chunk.tree2conlltags(tree) — convert to IOB (B-/I-/O) tagging.
10

N-grams

  • from nltk import bigrams, trigrams, ngrams
  • list(bigrams(tokens)) → [(w1,w2), ...]. It's a generator — wrap in list().
  • list(ngrams(tokens, 3)) — any n; trigrams = ngrams(t, 3).
  • list(everygrams(tokens, 1, 3)) — all 1- to 3-grams at once.
  • ngrams(t, 2, pad_left=True, ...) — pad edges for language-model contexts.
  • N-grams feed language models & collocation scoring.
11

Collocations

  • from nltk.collocations import BigramCollocationFinder, BigramAssocMeasures
  • f = BigramCollocationFinder.from_words(tokens) — count all adjacent pairs.
  • f.apply_freq_filter(3) — drop rare pairs first (noise).
  • f.nbest(BigramAssocMeasures().pmi, 10) — top pairs by pointwise mutual information — "words that stick together".
  • ...likelihood_ratio · ...chi_sq — other association scores.
  • Trigram variants exist too. PMI finds phrases like "New York", "machine learning".
12

WordNet

  • from nltk.corpus import wordnet as wn — a lexical graph of word senses.
  • wn.synsets('dog') → all senses; each a Synset like dog.n.01 (word.pos.num).
  • ss = wn.synset('dog.n.01') — one specific sense.
  • ss.definition() · ss.examples() · ss.lemma_names() — gloss, usages, synonyms.
  • ss.hypernyms() → broader (dog → canine). hyponyms() → narrower (dog → poodle).
  • ss.lemmas()[0].antonyms() — opposites live on lemmas, not synsets.
  • POS letters: wn.NOUN='n', VERB='v', ADJ='a', ADV='r' (note: r, not "adv").
13

Similarity & distance

  • a.path_similarity(b) → 0–1 by graph distance between synsets; dog~cat = 0.2.
  • a.wup_similarity(b) — Wu-Palmer; uses depth of the common ancestor.
  • a.lowest_common_hypernyms(b) — the shared parent sense.
  • nltk.edit_distance('kitten', 'sitting') → 3. Levenshtein — for fuzzy matching / spell-check.
  • nltk.jaccard_distance(set(a), set(b)) — set overlap distance.
14

Corpora

  • from nltk.corpus import gutenberg, brown, reuters — download each first.
  • brown.words(categories='news') — tokens; also .sents(), .paras(), .raw(), .tagged_words().
  • gutenberg.fileids() → ['austen-emma.txt', ...]. .categories() for topic-split corpora.
  • movie_reviews.words(fileid) — labelled pos/neg — the classic classification dataset.
  • inaugural · reuters · wordnet · names — speeches, news, lexical, name lists.
  • Every corpus reader shares the same .words / .sents / .raw / .fileids API.
15

Classification

  • feats = ({'has(great)': True}, 'pos') — NLTK classifiers take (feature_dict, label) pairs.
  • clf = nltk.NaiveBayesClassifier.train(train_set) — the go-to baseline.
  • clf.classify(features) → predicted label for one item.
  • nltk.classify.accuracy(clf, test_set) → 0–1 score.
  • clf.show_most_informative_features(10) — which features drive predictions.
  • nltk.classify.SklearnClassifier(...) — wrap any scikit-learn model in the same API.
16

Sentiment · VADER

  • from nltk.sentiment import SentimentIntensityAnalyzer — needs vader_lexicon.
  • sia = SentimentIntensityAnalyzer()
    sia.polarity_scores('I love this!') → {'neg','neu','pos','compound'}.
  • scores['compound'] — one number in [-1, +1]. Rule of thumb: ≥ 0.05 positive, ≤ -0.05 negative.
  • VADER is rule-based & tuned for social media — no training needed. Understands caps, "!!!", and 😀.
17

Parsing & grammar

  • g = nltk.CFG.fromstring("""S -> NP VP ...""") — define a context-free grammar.
  • parser = nltk.ChartParser(g) — also RecursiveDescentParser, ShiftReduceParser.
  • for tree in parser.parse(tokens): ... — yields every valid parse Tree.
  • tree.draw() · tree.pretty_print() — visualize the structure.
  • tree.leaves() · tree.label() · tree.subtrees() — walk a Tree.
18

Preprocessing recipe

  • The canonical clean-up, top to bottom:
  • text = text.lower() — case-fold.
  • toks = word_tokenize(text) — split.
  • toks = [w for w in toks if w.isalpha()] — drop punctuation/digits.
  • toks = [w for w in toks if w not in sw] — remove stopwords.
  • toks = [wnl.lemmatize(w) for w in toks] — to base forms.
  • Order matters: lower → tokenize → filter → lemmatize. Now feed FreqDist, a classifier, or vectorizer.
19

Interop & next steps

  • tree.leaves(), FreqDist(...).most_common() — everything returns plain Python (lists, tuples, dicts) — easy to hand to pandas / sklearn.
  • SklearnClassifier · TfidfVectorizer — NLTK for linguistics, scikit-learn for the model.
  • NLTK vs spaCy: NLTK is a teaching/research toolbox (many algorithms, lots of corpora, WordNet). spaCy is faster & production-first with one opinionated pipeline. Use NLTK to learn & explore, spaCy to ship.
  • For embeddings/transformers reach for gensim or 🤗 transformers — NLTK stops at classical NLP.
★

Tagsets & download map

  • NN/NNS/NNP noun · plural · proper
  • VB/VBD/VBG/VBZ verb base · past · gerund · 3sg
  • JJ · RB · DT · IN · PRP · CC adj · adverb · determiner · prep · pronoun · conj
  • universal: NOUN VERB ADJ ADV PRON DET ADP CONJ — the 12-tag simple set
  • wn pos: n noun · v verb · a adj · r adverb
  • who needs what data:
  • punkt_tab → word_tokenize, sent_tokenize
  • averaged_perceptron_tagger_eng → pos_tag
  • wordnet → WordNetLemmatizer, WordNet
  • stopwords → stopwords · vader_lexicon → sentiment
  • maxent_ne_chunker_tab + words → ne_chunk

Four ideas that explain the rest

1 · Stemming vs lemmatization

Stem = crude chop (fast, fake word). Lemma = dictionary form (needs POS).

studies studying running studi PorterStemmer — not a word study running (pos=v→run) Lemmatizer — real word lemmatize('running') stays 'running' — until pos='v' gives 'run' default POS is noun · verified live on nltk 3.10

2 · WordNet is a graph of senses

Hypernyms go up (broader), hyponyms down (narrower).

animal.n.01 carnivore.n.01 canine.n.02 dog.n.01 poodle · spitz hypernyms ↑ hyponyms ↓ path_similarity walks these edges · dog~cat = 0.2

3 · Tag, then chunk into a tree

POS tags label words; a grammar groups them into phrases.

TheDT dogNN chasedVBD theDT catNN S NP VP NP The dog the cat chased RegexpParser grammar: NP: {<DT>?<JJ>*<NN>}

4 · Every step needs its data

The LookupError just means "download this first".

word_tokenize punkt_tab pos_tag averaged_perceptron_tagger_eng lemmatize wordnet ne_chunk maxent_ne_chunker_tab · words SentimentIntensity vader_lexicon or just once: nltk.download('popular')

Worth memorizing

The dozen facts behind most NLTK errors — all confirmed on a live nltk 3.10.0 install.
01
It ships no data. Almost every call needs nltk.download(...) first — the LookupError tells you which resource.
02
Names changed in 3.9+. punkt_tab not punkt; averaged_perceptron_tagger_eng not ..._tagger. Stale sheets break here.
03
The data changes type at each step: str → list[str] → list[tuple] → Tree. Pass the right shape.
04
pos_tag takes a list, not a string. Tokenize first, or every character gets tagged.
05
Lemmatizer defaults to noun. lemmatize('running') → 'running'; lemmatize('running','v') → 'run'.
06
Stems aren't words ('studies'→'studi'); lemmas are. Stem to index, lemmatize to read.
07
Tags are Penn Treebank by default (NN, VBD, JJ). Add tagset='universal' for simple ones.
08
Lowercase before stopword filtering — the list is all lowercase, or 'The' slips through.
09
bigrams/ngrams are generators — wrap in list() to see or reuse them.
10
WordNet POS letters: n v a r — adverb is r, not "adv".
11
FreqDist is a dict. Use .most_common(n); indexing missing keys gives 0, not KeyError.
12
NLTK teaches; spaCy ships. Reach for spaCy/🤗 when you need speed or transformers.