BeautifulSoup (Python) Cheat Sheet

01Setup & Importinstall once

$ pip install beautifulsoup4
The package is beautifulsoup4; you import it as bs4.
$ pip install lxml html5lib
Optional faster / more-lenient parsers (see card 02).
from bs4 import BeautifulSoup★
The one class you need to start.
import bs4; bs4.__version__
Check the version, e.g. '4.14.3'.

02Make the Soup & Parserspick one

soup = BeautifulSoup(html, "html.parser")★
stdlib, no deps, decent — the safe default.
BeautifulSoup(html, "lxml")★
Fastest; needs pip install lxml. Great for big pages.
BeautifulSoup(html, "html5lib")
Browser-lenient, fixes broken markup like a browser; slowest.
BeautifulSoup(xml, "lxml-xml")
Parse XML (or pass features="xml").
BeautifulSoup(open("page.html"), "lxml")
Feed a file handle directly, or a str / bytes.
# always name the parsergotcha
Omitting it triggers GuessedAtParserWarning & can change behaviour across machines.

03The Four Object Typeswhat's in the tree

bs4.BeautifulSoup
The whole document / root. Behaves like a Tag.
bs4.Tag★
An element — has .name, .attrs, children. What you search & navigate.
bs4.NavigableString
Text inside a tag — a str subclass, so string methods work.
bs4.Comment
; a special NavigableString (also CData, Doctype).
isinstance(x, bs4.Tag)
Test node type when walking mixed children.

04Inspect a Tagname · attrs

tag.name★
The tag's name, e.g. "a". Assignable to rename.
tag.attrs
All attributes as a dict: {"href": "/x", "id": "c1"}.
tag["href"]★
One attribute value. KeyError if missing.
tag.get("href")★
Safe access — returns None if absent.
tag.contents · len(tag.contents)
Direct children as a list.
str(tag) · repr(tag)
The tag rendered back to HTML.

05find & find_allthe workhorse

soup.find_all("a")★
Every matching tag — a list (empty if none).
soup.find("a")★
First match only, or None. Same as soup.a.
soup("a")
Calling the soup is find_all — a handy shortcut.
find_all(["h1", "h2"])
A list matches any of several tag names.
find_all(re.compile("^h[1-6]$"))
A regex matches tag names by pattern.
find_all(True) · find_all(fn)
True matches every tag; a function is a custom filter.
find_all("a", limit=5, recursive=False)
Cap results; recursive=False = direct children only.

06Search by Attributes & Textfilter the match

find_all("div", class_="lead")★
class_ (trailing underscore) — class is reserved.
find_all(id="main")★
Any attribute works as a keyword arg.
find_all(attrs={"data-id": "7"})★
Use attrs= for hyphenated / reserved names.
find_all("a", href=re.compile("^/docs"))
Regex-match an attribute value.
find_all("a", string="Next")
string= filters by the tag's text (was text=).
class_="a b" vs class_="b a"gotcha
Multi-class match is order-sensitive; use select("div.a.b") instead.

07CSS Selectors · selectsoupsieve

soup.select("div.content a")★
CSS selector → list of tags. Familiar if you know CSS.
soup.select_one("h1#title")★
First match or None — the find of the CSS world.
select("ul > li") · select("a[href]")
Child combinator, attribute-presence — full CSS4 support.
select("p.lead.intro")
Multiple classes the easy way (vs class_).
select("li:nth-of-type(2)")
Pseudo-classes via the bundled soupsieve engine.
select(".sister", limit=2)
limit= caps results like find_all.

08Directional Searchfind near a tag

tag.find_parent("div")
Nearest ancestor matching; find_parents for all.
tag.find_next_sibling("td")★
Next sibling tag — skips whitespace strings (see card 10).
tag.find_previous_sibling(…)
The sibling before it.
tag.find_next("p") · find_all_next()
Anything later in the document (parse order, any depth).
tag.find_previous(…) · find_all_previous()
Anything earlier in the document.

09Navigate Downinto children

soup.title · soup.body.p★
Dotted access = first matching child tag by name.
tag.contents
Direct children as a list.
for c in tag.children:★
Direct children as an iterator.
for d in tag.descendants:
Every nested node, depth-first.
tag.string
The single child string — None if 0 or 2+ children.
tag.strings · tag.stripped_strings
Iterate all text pieces; stripped_ drops whitespace-only ones.

10Navigate Up & Sidewaysparents · siblings

tag.parent★
The tag that contains this one.
for p in tag.parents:
All ancestors, up to the document root.
tag.next_sibling · tag.previous_sibling
Adjacent node at the same level.
tag.next_siblings · tag.previous_siblings
Iterators over siblings in each direction.
next_sibling → often '\n'gotcha
Whitespace between tags is a NavigableString sibling — use find_next_sibling().
tag.next_element · tag.previous_element
Next node in parse order (descends into children) — not the same as siblings.

11Extract Textget the words out

tag.get_text()★
All text under the tag, concatenated. Alias: tag.text.
tag.get_text(separator=" ", strip=True)★
Join pieces with a separator & trim whitespace — clean output.
tag.string
Text only if the tag has exactly one string child, else None.
list(tag.stripped_strings)
Each non-empty text chunk as its own list item.
.string is None here?gotcha
Multiple children → .string is None. Reach for get_text().

12Extract Attributeshref · src · class

a["href"]★
Direct access — raises KeyError if the attr is absent.
a.get("href", "")★
Safe access with a default — the scraping-friendly form.
img["src"] · a["title"]
Any attribute by name.
tag["class"] → ["a", "b"]gotcha
class is multi-valued — you get a list, not a string.
"href" in a.attrs
Check presence before reading.

13Modify · Edit & Buildchange the tree

tag.name = "div"
Rename an element.
tag["class"] = "new"; del tag["id"]
Set or delete attributes.
tag.string = "new text"
Replace all contents with a single string.
new = soup.new_tag("a", href="/x")★
Build a fresh tag (then set .string / append it).
tag.append(new) · tag.insert(0, x)
Add a child at the end / at an index.
tag.insert_before(x) · tag.insert_after(x)
Place a node as a sibling.

14Modify · Delete & Replaceremove nodes

tag.decompose()★
Destroy the tag & its contents in place.
removed = tag.extract()
Remove from the tree but return it for reuse.
tag.clear()
Empty the tag's contents, keep the tag.
tag.replace_with(new)
Swap this node for another.
tag.unwrap()
Remove the tag, keep its children (strip a wrapper).
tag.wrap(soup.new_tag("div"))
Surround a node with a new parent.

15Output & Formattingtree → string

print(soup.prettify())★
Indented, human-readable HTML — great for debugging.
str(soup) · str(tag)★
Compact HTML for the whole tree or one node.
soup.encode("utf-8")
Serialize to bytes in a chosen encoding.
tag.decode(formatter="html")
Formatters: "minimal" (default), "html", "html5", None, or a function.
soup.prettify(formatter="minimal")
Control entity escaping in the output.

16Encoding & Unicodebytes in, str out

BeautifulSoup(resp.content, "lxml")★
Pass bytes so BS4 can sniff the encoding itself.
BeautifulSoup(data, from_encoding="latin-1")
Force an input encoding when detection is wrong.
soup.original_encoding
What BS4 decided the input encoding was.
from bs4 import UnicodeDammit
UnicodeDammit(data).unicode_markup — standalone encoding guesser.
soup.prettify("latin-1")
Choose the output encoding (default is UTF-8).

17Common Recipescopy-paste

[a["href"] for a in soup.select("a[href]")]★
Every link on the page.
[img["src"] for img in soup.find_all("img")]
All image sources.
soup.get_text("\n", strip=True)
All visible text, one line per chunk.
for tr in soup.select("table tr"):
Walk table rows; then tr.find_all(["td","th"]) for cells.
soup.find("label").find_next("input")["value"]
Hop from one tag to a related one, then read an attr.

18Performance & SoupStrainergo faster

# use lxml for speed on big docs★
The parser choice is the biggest lever on performance.
from bs4 import SoupStrainer
Parse only part of a document to save time & memory.
only = SoupStrainer("a")
Keep just <a> tags…
BeautifulSoup(html, "lxml", parse_only=only)
…the rest of the tree is never built.
str(navstr) # detach text
Call str() on a NavigableString you keep, or it pins the whole tree in memory.

BeautifulSoup cheat sheet · bs4

Four pictures that make BeautifulSoup click

1 · From markup to tree

2 · Moving around the family

3 · find_all vs select — same goal

4 · .string vs .get_text() vs .stripped_strings

Worth memorizing