NLP Text Preprocessing Cheat Sheet

1Overview & installthe three tools

pip install contractions rake-nltk keybert★
contractions = normalize; rake-nltk = statistical keyphrases (no ML); KeyBERT = semantic keyphrases (BERT embeddings, pulls in sentence-transformers + torch).
import nltk nltk.download("stopwords"); nltk.download("punkt_tab")★
rake-nltk needs NLTK data — stopwords and the tokenizer (punkt_tab on NLTK 3.9+). Do this once before using RAKE.
# RAKE: fast, deterministic, no semantics # KeyBERT: semantic, needs a model (slower, better phrases)
Rule of thumb: reach for RAKE at scale or with no GPU; KeyBERT when phrase quality/meaning matters.

2Expand contractionscontractions 0.1.73

import contractions contractions.fix("I can't believe it's working") # -> "I cannot believe it is working"★
One function does it. Handles hundreds of forms, including compound ones like shouldn't've → should not have.
contractions.add("mfw", "my face when")★
Register your own expansions (slang, domain abbreviations) before calling fix.
# ambiguous forms expand to ONE fixed meaning contractions.fix("ain't") # -> "are not"
Limitation: it's not context-aware — ain't (am/are/is/has/have not) always becomes one form.

3Cleaning pipelineorder matters

text = contractions.fix(raw_text) text = text.strip() # then tokenize / remove stopwords / extract keywords★
Expand contractions early — before tokenizing or stopword removal — so "don't" doesn't split into a junk "n't" token.
# be gentle with lowercasing
Aggressive .lower() helps RAKE/count methods but can slightly hurt KeyBERT (its embeddings handle case). Normalize to match the extractor.
# combine with NLTK / spaCy for tokenize + lemmatize
These three cover normalize + keyphrases; use the NLTK/spaCy sheets for tokenization, POS, and lemmatization in the same pipeline.

4Rake setuprake-nltk 1.0.6

from rake_nltk import Rake r = Rake() # English stopwords + punctuation by default★
Import is rake_nltk (underscore). A default Rake() uses NLTK English stopwords as phrase delimiters.
r = Rake(stopwords=my_stopwords, language="german")
Override the stopword set or language. Stopwords + punctuation are the boundaries that split text into candidate phrases.
# purely statistical: word degree / frequency — no ML, no GPU
RAKE scores phrases from how often words co-occur and how "connected" they are — fully deterministic and instant.

5Extract & rankphrases out

r.extract_keywords_from_text(text)★
Run extraction (stores results on the instance). Or r.extract_keywords_from_sentences([...]) for a list of sentences.
r.get_ranked_phrases() # ['phrase', ...] high -> low★
The ranked keyphrases as plain strings.
r.get_ranked_phrases_with_scores() # [(9.0, 'phrase'), ...]★
Same list with numeric scores — use it to threshold or take the top-k.

6Tuningphrase length & metric

r = Rake(min_length=1, max_length=3)★
Bound phrase word-count — e.g. 1–3 words to avoid long run-on phrases.
from rake_nltk import Metric r = Rake(ranking_metric=Metric.DEGREE_TO_FREQUENCY_RATIO)
The default metric. Alternatives: Metric.WORD_DEGREE, Metric.WORD_FREQUENCY — trade off favoring longer vs more frequent phrases.
r = Rake(include_repeated_phrases=False)
Drop duplicate phrases. Also punctuations= to customize the delimiter set.

7Extract keywordsKeyBERT 0.9

from keybert import KeyBERT kw = KeyBERT() # default: all-MiniLM-L6-v2 (sentence-transformers)★
Loads a sentence-transformers model (downloaded on first use). It embeds the document and candidate n-grams, then ranks by cosine similarity.
kw.extract_keywords(doc, keyphrase_ngram_range=(1, 2), stop_words="english", top_n=10)★
The core call. Returns [(keyword, score)] sorted by similarity. keyphrase_ngram_range sets phrase length; top_n the count.
kw = KeyBERT(model="all-mpnet-base-v2")
Swap the embedding model — bigger models (mpnet) = better quality, slower. Pass any sentence-transformers name or model object.

8Diversity & modelsavoid near-duplicates

kw.extract_keywords(doc, use_mmr=True, diversity=0.7)★
Maximal Marginal Relevance spreads out the results so you don't get five phrasings of the same idea. Higher diversity = more varied.
kw.extract_keywords(doc, use_maxsum=True, nr_candidates=20)
Max Sum Similarity — picks the top-n whose pairwise similarity is lowest, from a candidate pool.
from keyphrase_vectorizers import KeyphraseCountVectorizer kw.extract_keywords(doc, vectorizer=KeyphraseCountVectorizer())
POS-aware candidates — a keyphrase vectorizer proposes grammatically valid noun phrases instead of raw n-grams. Also candidates= and seed_keywords= to steer.

!Gotchas & choosingread before shipping

rake-nltk: download NLTK data first
Missing stopwords/punkt_tab raises LookupError. On NLTK 3.9+ the tokenizer resource is punkt_tab (older code used punkt).
KeyBERT downloads a model (~90MB) + needs torch
First run fetches the sentence-transformers model and pulls in torch. Heavier and slower than RAKE; a GPU speeds up embedding.
RAKE vs KeyBERT = statistics vs meaning
RAKE is fast/deterministic but blind to synonyms and context; KeyBERT captures meaning but needs a model. Choose per corpus size and quality bar (or run both and compare).
return shapes differ
KeyBERT returns (phrase, score) tuples; RAKE's get_ranked_phrases() returns plain strings (use ..._with_scores() for scores). Don't assume the same shape.
contractions isn't context-aware
Ambiguous forms expand to one fixed meaning. Fine for normalization; don't treat it as disambiguation.
expand contractions BEFORE tokenizing
Run contractions.fix on raw text first, or apostrophe splitting produces junk tokens that pollute stopword removal and keyphrases.
KeyBERT stop_words filters candidates, not the doc
stop_words removes stopword n-grams from the candidate set; the document embedding still uses the full text.

NLP Text Preprocessing cheat sheet