Python · Natural Language Processing · Quick Reference

spaCy · Industrial-strength NLP 3.8

One idea drives the whole library: load a pipeline once, call it, get objects back. nlp = spacy.load(...) builds a pipeline; doc = nlp(text) runs every component in one pass and hands you a Doc — a rich container of Token and Span objects with POS, lemma, dependency and entity already computed. No lists of tuples: real objects with attributes. Colour-coded by domain; ★ marks daily-use calls.

setup · models · pipeline Doc · Token · Span POS · dep · lemma entities · spans matchers · rules scale · serialize · visualize gotcha ★most common
Verified against a live spacy 3.8.14 + en_core_web_sm 3.8.0 install (the whole object model + every gotcha run, not remembered) · spacy.io docs, API & the official cheat sheet · Real Python / DataCamp / Explosion course.
ONE CALL RUNS THE WHOLE PIPELINE — nlp(text) → Doc text str a document tokenizer → Token[] tagger .pos_ .tag_ .morph parser .dep_ .head .sents lemmatizer .lemma_ ner .ents Doc a sequence of Token / Span objects every annotation already attached RESULTS ARE OBJECTS, NOT STRINGS Everything hangs off the Doc. String attributes end in an underscore — that trips up everyone. the container model Doc Apple is looking at London . Token doc[0:3] → Span doc[4] → Token the underscore rule · the #1 gotcha token.pos_ → 'PROPN' the readable string token.pos → 96 the hash id (rarely wanted) Same for .lemma_ .dep_ .tag_ .ent_type_ .label_ — without the underscore you get an integer. quickstart import spacy nlp = spacy.load('en_core_web_sm') what one Token knows token = doc[2] # "looking" .text → 'looking' .lemma_ → 'look' .pos_ → 'VERB' .dep_ → 'ROOT' .tag_ → 'VBG' .is_stop → False .head → looking .children → [Apple, is, at, ...] named entities · doc.ents (a tuple of Span) for ent in doc.ents: ent.text, ent.label_ AppleORG LondonGPE $1 billionMONEY Each entity is a Span with .start_char .end_char. spacy.explain('GPE') → "Countries, cities, states".
01

Setup & models

  • $ pip install spacy — the library ships no models.
  • $ python -m spacy download en_core_web_sm — fetch a model; separate step from pip.
  • nlp = spacy.load('en_core_web_sm') — load a pipeline. The everyday first line.
  • en_core_web_sm · md · lg · trf — small→large; md/lg add word vectors, trf is transformer-based (most accurate, slowest).
  • # name = lang_core_genre_size en · core (general) · web (source) · sm.
  • nlp = spacy.blank('en') — tokenizer only, no annotations — a base to build on.
  • Other languages: de_, fr_, xx_ (multi) core models exist too.
02

The nlp object & Doc

  • doc = nlp(text) — runs the full pipeline once; returns a Doc.
  • for token in doc: ... — a Doc iterates as Tokens; len(doc) = token count.
  • doc.text — reconstruct the original string.
  • docs = nlp.pipe(texts) — batch processing; far faster than a Python loop over nlp().
  • doc = nlp.make_doc(text) — tokenize only, skip the pipeline (cheap).
  • You never build a Doc by hand — nlp() creates it, fully annotated.
03

Token attributes

  • token.text · .lemma_ · .pos_ · .tag_ · .dep_ — the core five. Trailing _ = string.
  • token.pos (no _) → an integer hash (96), not text. Almost always you want pos_.
  • token.is_stop · .is_alpha · .is_punct · .like_num — boolean flags for filtering.
  • token.shape_ → 'Xxxx', 'dd' — orthographic shape.
  • token.i · .idx — token index · character offset in the doc.
  • token.lower_ · .is_title · .is_digit — casing & string helpers.
04

POS & morphology

  • token.pos_ → coarse Universal POS (NOUN VERB ADJ).
  • token.tag_ → fine-grained Penn Treebank tag (NN VBG JJR).
  • token.morph → features like Tense=Past|Number=Sing; .morph.get('Tense').
  • spacy.explain('VBG') → "verb, gerund…". Works on any tag, dep or ent label.
  • POS & tag come from the tagger; a blank pipeline leaves them empty.
05

Dependency parsing

  • token.dep_ → syntactic relation to its head (nsubj, dobj, ROOT).
  • token.head → the parent Token. The root's head is itself.
  • token.children · .subtree · .ancestors — walk the parse tree.
  • doc.noun_chunks → base noun phrases as Spans ("the red car").
  • token.left_edge · .right_edge — span boundaries of a token's subtree.
  • Provided by the parser. Visualize with displacy (card 17).
06

Named entities

  • for ent in doc.ents: ... doc.ents is a tuple of Span, not a list.
  • ent.text · ent.label_ — surface text · type string (PERSON, ORG, GPE, DATE, MONEY).
  • ent.start_char · ent.end_char — character offsets into doc.text.
  • token.ent_type_ · .ent_iob_ — per-token entity type · B/I/O scheme.
  • spacy.explain('GPE') → "Countries, cities, states". Demystifies any label.
  • Entities come from the ner component; add your own with the EntityRuler (card 12).
07

Lemmatize & normalize

  • token.lemma_ → base form ('running'→'run'). Context-aware, real words.
  • [t.lemma_ for t in doc if not t.is_stop and t.is_alpha] — the standard clean-token list.
  • token.lower_ — lowercase string (an attribute, no call).
  • token.is_stop — stopword flag; edit nlp.Defaults.stop_words to customize.
  • Unlike NLTK's stemmer, spaCy only lemmatizes — always dictionary forms.
08

Spans & slicing

  • doc[2] → a Token.
  • doc[2:5] → a Span (a view, not a list). span.text, span[0].
  • span.label_ · span.root · span.sent — optional label · head token · containing sentence.
  • doc.char_span(0, 5, label='ORG') — build a Span from character offsets (returns None if it doesn't align to tokens).
  • span.as_doc() — promote a Span to a standalone Doc.
  • Slicing never returns a Python list — it's always a Span.
09

Sentence segmentation

  • for sent in doc.sents: ... → each sentence as a Span.
  • token.is_sent_start — boolean flag per token.
  • # needs the parser OR the senter doc.sents raises if neither is in the pipe — a blank model has neither.
  • nlp.add_pipe('senter') — a fast, parser-free sentence splitter.
  • nlp.add_pipe('sentencizer') — rule-based splitter for blank pipelines.
10

Similarity & vectors

  • doc1.similarity(doc2) → 0–1 cosine of averaged vectors. Also on Token / Span.
  • # sm has NO word vectors! Use en_core_web_md or lg (300-d) — on sm, similarity still runs but warns (W007) and is unreliable.
  • token.vector · .vector_norm — the raw array · its length.
  • token.has_vector · .is_oov — has a vector? · out of vocabulary?
  • For semantic search reach for sentence-transformers; spaCy vectors are word-level averages.
11

Token Matcher

  • from spacy.matcher import Matcher
    m = Matcher(nlp.vocab)
  • pattern = [{'LOWER': 'hello'}, {'POS': 'PROPN'}] — a list of dicts; keys are UPPERCASE token attributes.
  • m.add('RULE', [pattern]) — note the pattern is wrapped in a list.
  • matches = m(doc) → [(match_id, start, end), ...]; slice doc[start:end] for the Span.
  • {'OP': '?'} — quantifiers: ? 0-1 · * 0-many · + 1-many · ! negate.
  • {'LOWER': {'IN': ['a','b']}} · {'TEXT': {'REGEX': ...}} — rich value operators.
12

Phrases & entity rules

  • from spacy.matcher import PhraseMatcher — match against big terminology lists, fast.
  • pm.add('TECH', [nlp.make_doc(t) for t in terms]) — feed Docs, not strings.
  • PhraseMatcher(nlp.vocab, attr='LOWER') — case-insensitive matching.
  • ruler = nlp.add_pipe('entity_ruler')
    ruler.add_patterns([{'label':'ORG', 'pattern':'spaCy'}]) — inject rule-based entities into doc.ents.
  • Place the ruler before or after 'ner' to override or supplement the model.
13

The pipeline

  • nlp.pipe_names → ['tok2vec','tagger','parser','attribute_ruler','lemmatizer','ner'].
  • nlp.pipe(texts, disable=['parser','ner']) — skip components you don't need — big speedup.
  • nlp.add_pipe('sentencizer', first=True) — insert a component (first/last/before=/after=).
  • with nlp.select_pipes(disable=[...]): ... — temporarily turn components off.
  • nlp.analyze_pipes(pretty=True) — show what each component sets & needs.
  • spacy.load(m, exclude=['ner']) exclude = don't load at all; disable = load but skip.
14

Custom components

  • @Language.component('my_comp')
    def my_comp(doc): ...; return doc — a component takes a Doc, returns a Doc.
  • nlp.add_pipe('my_comp', last=True) — then it runs on every nlp() call.
  • Doc.set_extension('is_greeting', default=False) — custom attributes live under ._.
  • doc._.is_greeting · token._.foo · span._.bar — access them via the ._ namespace.
  • Extensions keep custom data on the objects without subclassing.
15

Processing at scale

  • docs = nlp.pipe(texts, batch_size=1000) — stream many texts efficiently (a generator).
  • nlp.pipe(texts, n_process=4) — multiprocessing across CPU cores.
  • nlp.pipe(pairs, as_tuples=True) — carry (text, context) pairs through; get (doc, context) back.
  • doc.to_array(['POS','DEP']) — dump attributes to a numpy array for bulk work.
  • Always prefer nlp.pipe() over a for-loop of nlp() for more than a handful of texts.
16

Save & load

  • from spacy.tokens import DocBin
    db = DocBin(); db.add(doc) — the efficient way to store many Docs.
  • db.to_disk('data.spacy') · DocBin().from_disk(...) — the training data format too.
  • doc.to_bytes() · Doc(vocab).from_bytes(b) — one doc to/from bytes.
  • nlp.to_disk('./model') · spacy.load('./model') — persist a whole pipeline.
  • Don't pickle a Doc directly — DocBin/to_bytes are smaller & version-safe.
17

Visualize · displaCy

  • from spacy import displacy
  • displacy.render(doc, style='dep') — draw the dependency arcs.
  • displacy.render(doc, style='ent') — highlight named entities inline.
  • displacy.render(doc, style='span') — overlapping spans (doc.spans).
  • displacy.serve(doc, ...) — open in a browser; add jupyter=True inside notebooks.
  • options={'compact':True, 'distance':90} — tune the layout.
18

Training (v3 config)

  • $ python -m spacy init config config.cfg --lang en --pipeline ner — v3 training is config-driven, not code.
  • $ python -m spacy train config.cfg --output ./out — reads .spacy DocBin data, writes a model.
  • from spacy.training import Example Example.from_dict(doc, {'entities': [...]}) — the annotation unit.
  • $ python -m spacy debug data config.cfg — sanity-check your training set first.
  • The old nlp.update() loop still works, but the CLI + config is the recommended path.
19

Interop & when to use

  • doc.to_json() — tokens, ents & spans as plain dicts for APIs.
  • [(t.text, t.pos_, t.dep_) for t in doc] — drop into a pandas DataFrame in one line.
  • pip install spacy[transformers] — en_core_web_trf wraps 🤗 transformers as a pipeline component.
  • spaCy vs NLTK: spaCy is fast, object-oriented & production-first with one opinionated pipeline; NLTK is a teaching toolbox of many algorithms + corpora. spaCy to ship, NLTK to explore.
  • For raw embeddings/LLMs pair spaCy with sentence-transformers or spacy-llm.
★

Attribute & label map

  • Token .text .lemma_ .pos_ .tag_ .dep_ .head .is_stop .ent_type_ .shape_ .i .idx .vector
  • Span .text .label_ .root .sent .start .end .start_char .ents
  • Doc .text .ents .sents .noun_chunks .vocab .cats ._
  • ent labels PERSON ORG GPE LOC DATE TIME MONEY PERCENT PRODUCT NORP FAC
  • POS (pos_) NOUN VERB ADJ ADV PROPN PRON DET ADP AUX NUM PUNCT
  • dep (dep_) nsubj dobj ROOT det amod prep pobj aux conj cc
  • models sm (no vectors) · md/lg (300-d vectors) · trf (transformer)
  • spacy.explain(x) — turns ANY tag / dep / label into plain English.

Four ideas that explain the rest

1 · One call runs the whole pipeline

Each component writes annotations onto the same Doc.

text token­izer tagger parser ner Doc Tokens + annotations nlp = spacy.load(...) builds it once; nlp(text) runs it disable components you don't need for a big speedup

2 · Doc · Token · Span · Vocab

A Doc holds Tokens; a Span is a slice; the Vocab stores shared strings.

Doc Apple is looking at London Token doc[0:3] = Span Vocab · StringStore — one shared table, hash ↔ string 'London'↔hash every token stores the hash, not a copy of the string

3 · The underscore rule

.pos is a hash int; .pos_ is the readable string.

token "Apple" .pos 96 a hash int .pos_ 'PROPN' the string you want StringStore Trailing _ means "give me the string". Same for .lemma_ .dep_ .tag_ .label_ .ent_type_

4 · A dependency parse (displaCy)

Arcs point from head → child; each is labelled with a relation.

ApplePROPN isAUX lookingVERB LondonPROPN nsubj aux prep "looking" is the ROOT — its head is itself token.head + token.children walk this tree

Worth memorizing

The dozen facts behind most spaCy confusion — all confirmed on a live spacy 3.8.14 + en_core_web_sm install.
01
Trailing underscore = string. token.pos_ → 'PROPN'; token.pos → 96 (a hash). Same for .lemma_ .dep_ .label_.
02
Results are objects, not strings. nlp(text) returns a Doc of Tokens with attributes — no lists of tuples.
03
One call runs everything. nlp(text) executes the whole pipeline; for many texts use nlp.pipe().
04
doc[i] is a Token, doc[i:j] is a Span — slicing never gives a Python list.
05
doc.ents is a tuple of Span, not a list you append to. Add rules via the EntityRuler.
06
sm has no word vectors. .similarity() still runs but warns (W007) and is poor — use md/lg/trf.
07
Download ≠ install. pip install spacy then python -m spacy download en_core_web_sm separately.
08
Sentences need the parser or senter. doc.sents errors on a blank pipeline with neither.
09
Matcher patterns are lists of dicts with UPPERCASE keys (LOWER, POS, OP); the pattern is wrapped in a list.
10
Disable to go faster. nlp.pipe(texts, disable=['parser','ner']) skips work you don't need.
11
spacy.explain(x) turns any tag, dep or label into plain English — keep it handy.
12
pos_ = coarse (Universal), tag_ = fine (Penn). NOUN vs NN; custom data lives under doc._.