Quick Reference · topic modelling & vector NLP for Python

gensim cheat sheet

Every gensim workflow is one pipeline: raw text → a Dictionary that maps each token to an id → bag-of-words vectors (that's your Corpus) → a Model that transforms those vectors (TF-IDF, LSI, LDA, or embeddings) → similarity queries. Corpora stream from disk one vector at a time, so nothing has to fit in RAM. Learn the four nouns once and the API stops being a list to memorise.

setup / import preprocess & dictionary corpus & vectors transforms & topics embeddings similarity & eval gotcha / removed ★ most common

Distilled & cross-checked against: radimrehurek.com/gensim (Core Concepts + API, gensim 4.4) · introspected from the installed gensim 4.4.0 package · machinelearningplus.com · cheatography (Gadpandey) · geeksforgeeks.org

The pipeline & the four nouns every command touches
THE PIPELINE — text in, ranked topics & neighbours out Raw text a Document = a str simple_preprocess() → [tokens] Dictionary token ↔ integer id {computer:0,  human:1, ...} Corpus bag-of-words vectors [(0,1),(1,1),(2,1)] one per document Model vector → vector TF-IDF · LSI · LDA Word2Vec · Doc2Vec Query index[q] most_ similar() topics build doc2bow fit model[·] a corpus is just an iterable of vectors — stream it from disk, never load all of RAM at once THE FOUR CORE NOUNS — the whole mental model Document some text — a str "Human machine interface" Corpus a collection of documents used to train a model Vector a math representation of one document Model transforms vectors from one representation to another Unsupervised throughout — gensim learns themes, topics & word meanings from plain text, no labels required. Apply any trained model with subscript syntax: tfidf[bow] · lsi[corpus] · lda[bow] — training and transforming are separate steps. Word/doc embeddings live on model.wv / model.dv — a KeyedVectors table you can detach, save, and query on its own.
quickstart.py — text → topics → similarity, end to end
from gensim import corpora, models, similarities
from gensim.utils import simple_preprocess

docs  = ["Human computer interaction", "A survey of user response time", "The EPS user interface system"]
texts = [simple_preprocess(d) for d in docs]     # tokenise + lowercase

dct    = corpora.Dictionary(texts)             # token ↔ id map
corpus = [dct.doc2bow(t) for t in texts]     # bag-of-words vectors

tfidf  = models.TfidfModel(corpus)            # learn a transform...
lsi    = models.LsiModel(tfidf[corpus], id2word=dct, num_topics=2)

index  = similarities.MatrixSimilarity(lsi[corpus])
q      = lsi[tfidf[dct.doc2bow(simple_preprocess("human interface"))]]
sims   = sorted(enumerate(index[q]), key=lambda x: -x[1])   # ranked docs
01Setup & Importonce per project
02Preprocess Textstr → [tokens]
03Dictionarytoken ↔ id
04Corpus & Bag-of-Wordsthe vectors
05TF-IDFreweight the BoW
06LSI / LSAlatent topics · SVD
07LDA Topic Modelprobabilistic topics
08More Topic Modelspick your algorithm
09Word2Vecword embeddings
10KeyedVectors & Word Maththe vector table
11Doc2Vecdocument embeddings
12FastTextsubword embeddings
13Phrases & Bigramsmultiword tokens
14Similarity Queriesrank documents
15Coherence & Evaluationis it any good?
16Pretrained · downloaderskip the training
17Save · Load · Persisttwo different formats

Four ideas worth a picture

The mental models behind the API — bag-of-words, topic factorisation, vector algebra, and the two training schemes.

doc2bow — text becomes a sparse vector

Tokens map to integer ids through the Dictionary; the document becomes a list of (id, count) pairs.

"human computer interface" human computer interface 1 0 2 dictionary: {computer:0, human:1, interface:2} [(0, 1), (1, 1), (2, 1)]

Topic model = matrix factorisation

LSI, LDA and NMF all approximate the big documents×words matrix as documents×topics times topics×words.

docs × words C ≈ docs × topics · topics × words (what each topic says) few topics ⇒ compression + latent structure = the "themes"

Word2Vec — meaning as geometry

Directions in the vector space encode relationships: king − man + woman lands near queen.

man king woman queen royalty gender wv.most_similar(positive=["king","woman"], negative=["man"])

CBOW vs skip-gram — the sg knob

CBOW (sg=0) predicts a word from its context; skip-gram (sg=1) predicts the context from the word.

CBOW · sg=0 the quick brown jumps fox? skip-gram · sg=1 fox the quick brown jumps skip-gram is slower but better on rare words & small data

Worth memorizing

same Dictionarydoc2bow silently drops words the dictionary never saw
a corpus streamsit's an iterable of vectors — gensim never needs all of RAM
gensim 4 renamessize→vector_size, iter→epochs on the embedding models
vocab accesswv.index_to_key / wv.key_to_index, not wv.vocab
vectors on .wvcall model.wv.most_similar, not model.most_similar
min_count=5default drops rare words — lower it for small corpora
apply with [ ]tfidf[bow], lda[bow] — train & transform are separate
Similarity vs MatrixSimilarity shards to disk; MatrixSimilarity holds all in RAM
c_v vs u_masscoherence c_v needs texts=, u_mass needs corpus=
FastText ≠ Word2VecFastText returns OOV vectors; Word2Vec raises KeyError
save ≠ save_..format.save() keeps training state; the w2v format keeps vectors only
num_topics is yoursgensim won't pick it — sweep k and compare coherence