BeautifulSoup (Python) Cheat Sheet

01Setup & Importinstall once

$ pip install beautifulsoup4
The package is beautifulsoup4; you import it as bs4.
$ pip install lxml html5lib
Optional faster / more-lenient parsers (see card 02).
from bs4 import BeautifulSoup★
The one class you need to start.
from bs4 import Tag, NavigableString, Comment
Import the node types for isinstance checks while walking.
import bs4; bs4.__version__
Check the version, e.g. '4.15.0'.

02Make the Soup & Parserspick one

soup = BeautifulSoup(html, "html.parser")★
stdlib, no deps, decent — the safe default.
BeautifulSoup(html, "lxml")★
Fastest; needs pip install lxml. Great for big pages.
BeautifulSoup(html, "html5lib")
Browser-lenient, fixes broken markup like a browser; slowest.
BeautifulSoup(xml, "lxml-xml")
Parse XML (or pass features="xml").
BeautifulSoup(open("page.html"), "lxml")
Feed a file handle directly, or a str / bytes.
# always name the parsergotcha
Omitting it triggers GuessedAtParserWarning & can change behaviour across machines.

03The Four Object Typeswhat's in the tree

bs4.BeautifulSoup
The whole document / root. Behaves like a Tag.
bs4.Tag★
An element — has .name, .attrs, children. What you search & navigate.
bs4.NavigableString
Text inside a tag — a str subclass, so string methods work.
bs4.Comment
; a special NavigableString (also CData, Doctype).
isinstance(x, bs4.Tag)
Test node type when walking mixed children.
x.__class__.__name__
All four share the PageElement base — hence common navigation.

04Inspect a Tagname · attrs

tag.name★
The tag's name, e.g. "a". Assignable to rename.
tag.attrs
All attributes as a dict: {"href": "/x", "id": "c1"}.
tag["href"]★
One attribute value. KeyError if missing.
tag.get("href")★
Safe access — returns None if absent.
tag.has_attr("href")
Boolean presence check — cleaner than in tag.attrs.
tag.contents · str(tag)
Direct children as a list; the tag rendered back to HTML.

05find & find_allthe workhorse

soup.find_all("a")★
Every matching tag — a list (empty if none).
soup.find("a")★
First match only, or None. Same as soup.a.
soup("a")
Calling the soup is find_all — a handy shortcut.
find_all(["h1", "h2"])
A list matches any of several tag names.
find_all(re.compile("^h[1-6]$"))
A regex matches tag names by pattern.
find_all(string=True)
Return NavigableStrings instead of Tags — grab raw text nodes.
find_all("a", limit=5, recursive=False)
Cap results; recursive=False = direct children only.

06Search by Attributes & Textfilter the match

find_all("div", class_="lead")★
class_ (trailing underscore) — class is reserved.
find_all("a", class_="btn", id="go")★
Stack keyword filters — all must match (AND).
find_all(attrs={"data-id": "7"})★
Use attrs= for hyphenated / reserved names.
find_all("a", href=re.compile("^/docs"))
Regex-match an attribute value.
find_all(string=re.compile("Sale"))
string= filters by text; takes a str, regex, list, or True.
class_="a b" vs class_="b a"gotcha
Multi-class match is order-sensitive; use select("div.a.b") instead.

07Function & Custom Filterswhen kwargs aren't enough

find_all(lambda t: t.name=="a" and
Pass a function — it receives each Tag, returns True to keep.
not t.has_attr("id"))★
e.g. links that lack an id — logic no kwarg can express.
find_all(lambda t: len(t.get("class", [])) > 2)
Match tags with more than two CSS classes.
def has_price(t): return "$" in t.get_text()
Named functions read better for complex rules.
find_all(True)
The True filter matches every tag — useful with a string/attr filter.
find_all(id=lambda v: v and v.startswith("nav"))
A function can also filter a single attribute's value.

08CSS Selectors · Basicssoupsieve

soup.select("div.content a")★
CSS selector → list of tags. Familiar if you know CSS.
soup.select_one("h1#title")★
First match or None — the find of the CSS world.
select("ul > li")
Child combinator (direct children only).
select("p.lead.intro")★
Multiple classes the easy way (vs order-sensitive class_).
select("li:nth-of-type(2)")
Pseudo-classes via the bundled soupsieve engine.
table.select("tr td")
Call select on any Tag to scope the search to its subtree.

09CSS Selectors · Attributes & Pseudothe scraping power tools

select("a[href]")★
Attribute presence — only <a> that actually have an href.
select('a[href^="https"]')★
^= starts-with — external links.
select('a[href$=".pdf"]')
$= ends-with — PDF links.
select('a[href*="/blog/"]')
*= contains — substring anywhere in the value.
select("h2 ~ p") · select("label + input")
General (~) & adjacent (+) sibling combinators.
select("div:has(> img)")
:has() — parents that contain a match.
select("p:-soup-contains('Sale')")
soupsieve's text selector (the old :contains).

10Directional Searchfind near a tag

tag.find_parent("div")
Nearest ancestor matching; find_parents for all.
tag.find_next_sibling("td")★
Next sibling tag — skips whitespace strings (see card 12).
tag.find_previous_sibling(…)
The sibling before it.
tag.find_next("p") · find_all_next()
Anything later in the document (parse order, any depth).
tag.find_previous(…) · find_all_previous()
Anything earlier in the document.

11Navigate Downinto children

soup.title · soup.body.p★
Dotted access = first matching child tag by name.
tag.contents
Direct children as a list.
for c in tag.children:★
Direct children as an iterator.
for d in tag.descendants:
Every nested node, depth-first.
tag.string
The single child string — None if 0 or 2+ children.
tag.strings · tag.stripped_strings
Iterate all text pieces; stripped_ drops whitespace-only ones.

12Navigate Up & Sidewaysparents · siblings

tag.parent★
The tag that contains this one.
for p in tag.parents:
All ancestors, up to the document root.
tag.next_sibling · tag.previous_sibling
Adjacent node at the same level.
tag.next_siblings · tag.previous_siblings
Iterators over siblings in each direction.
next_sibling → often '\n'gotcha
Whitespace between tags is a NavigableString sibling — use find_next_sibling().
tag.next_element · tag.previous_element
Next node in parse order (descends into children) — not the same as siblings.

13Extract Textget the words out

tag.get_text() · tag.text★
All text under the tag, concatenated. .text is the property alias.
tag.get_text(separator=" ", strip=True)★
Join pieces with a separator & trim whitespace — clean output.
tag.string
Text only if the tag has exactly one string child, else None.
list(tag.stripped_strings)
Each non-empty text chunk as its own list item.
tag.get_text(types=(NavigableString,))
Restrict which string types count — e.g. skip Comment / CData text.
.string is None here?gotcha
Multiple children → .string is None. Reach for get_text().

14Extract Attributeshref · src · class

a["href"]★
Direct access — raises KeyError if the attr is absent.
a.get("href", "")★
Safe access with a default — the scraping-friendly form.
img["src"] · a["data-id"]
Any attribute by name, including data-*.
tag["class"] → ["a", "b"]gotcha
class is multi-valued — you get a list, not a string. Also rel, accept-charset.
" ".join(tag.get("class", []))
Re-join a multi-valued attribute into a string.

15Modify · Edit & Buildchange the tree

tag.name = "div"; tag["class"] = "new"
Rename an element; set or (del tag["id"]) delete attributes.
tag.string = "new text"
Replace all contents with a single string.
new = soup.new_tag("a", href="/x")★
Build a fresh tag (then set .string / append it). 4.15: callable on any Tag, not just the BeautifulSoup object.
soup.new_string("hello")
Build a standalone text node. 4.15: also callable directly on any Tag/NavigableString.
child = tag.append(x) · tag.extend([a, b]) · tag.insert(0, x)
Add one / many children at the end, or at an index. 4.15: append now returns the appended element; extend/insert return the list of inserted items (was None).
tag.insert_before(x) · tag.insert_after(x)
Place a node as a sibling. 4.15: these return the inserted item(s) too.
tag.smooth()
Consolidate adjacent NavigableStrings after edits.

16Modify · Delete & Replaceremove nodes

tag.decompose()★
Destroy the tag & its contents in place. Check tag.decomposed.
removed = tag.extract()
Remove from the tree but return it for reuse.
tag.clear()
Empty the tag's contents, keep the tag.
tag.replace_with(new)
Swap this node for another.
tag.unwrap()
Remove the tag, keep its children (strip a wrapper).
tag.wrap(soup.new_tag("div"))
Surround a node with a new parent.

17Output & Formattingtree → string

print(soup.prettify())★
Indented, human-readable HTML — great for debugging.
str(soup) · str(tag)★
Compact HTML for the whole tree or one node.
soup.encode("utf-8")
Serialize to bytes in a chosen encoding.
tag.decode(formatter="html")
Formatters: "minimal" (default), "html", "html5", None, or a function.
from bs4.formatter import HTMLFormatter
prettify(formatter=HTMLFormatter(indent=2)) to control indent width.
formatter=Noneunsafe
Emits no entity escaping — risky if the HTML is untrusted.

18Encoding & Unicodebytes in, str out

BeautifulSoup(resp.content, "lxml")★
Pass bytes so BS4 can sniff the encoding itself.
BeautifulSoup(data, from_encoding="latin-1")
Force an input encoding when detection is wrong.
exclude_encodings=["utf-8"]
Rule out encodings you know are wrong to help the sniffer.
soup.original_encoding
What BS4 decided the input encoding was.
from bs4 import UnicodeDammit
UnicodeDammit(data).unicode_markup — standalone encoding guesser.

19Tables & Structured Datathe #1 scrape

rows = [[c.get_text(strip=True)
Walk a table into a list of row-lists…
for c in tr(["td","th"])] for tr in soup.select("table tr")]★
…tr([...]) is tr.find_all([...]).
head, *body = rows
Split header row from data rows.
[dict(zip(head, r)) for r in body]★
Zip each row against the header → list of dicts.
import json; s = soup.find("script",
Structured data is often embedded as JSON-LD…
type="application/ld+json"); json.loads(s.string)★
…pull & parse it in two lines.
import pandas; pandas.read_html(str(table))
For big tables, hand the HTML to pandas.

20Real-World Scrapingthe full loop

soup = BeautifulSoup(requests.get(url).content, "lxml")★
Fetch with requests, parse with BS4 — they're a pair.
from urllib.parse import urljoin★
urljoin(base, a["href"]) → turn relative links absolute.
el = soup.select_one(".price"); el.get_text() if el else None★
Always guard — missing elements return None, not empty.
for s in soup("script"): s.decompose()
Strip <script>/<style> before extracting clean page text.
# BS4 can't run JavaScriptkey
It sees only the initial HTML. For JS-rendered pages use Selenium / Playwright, then feed .page_source to BS4.
# be polite: rate-limit, honour robots.txt
Set a real User-Agent and add delays between requests.

21Common Recipescopy-paste

[a["href"] for a in soup.select("a[href]")]★
Every link on the page.
[img["src"] for img in soup.find_all("img")]
All image sources.
soup.get_text("\n", strip=True)
All visible text, one line per chunk.
"".join(p.find_all(string=True, recursive=False))
A tag's direct text only, ignoring nested tags.
soup.find("label").find_next("input")["value"]
Hop from one tag to a related one, then read an attr.

22Performance & SoupStrainergo faster

# use lxml for speed on big docs★
The parser choice is the biggest lever on performance.
from bs4 import SoupStrainer
Parse only part of a document to save time & memory.
only = SoupStrainer("a", href=True)
Keep just <a href> tags — strainers take the same filters as find…
BeautifulSoup(html, "lxml", parse_only=only)
…the rest of the tree is never built.
# pip install charset_normalizer
Speeds up & sharpens BS4's encoding detection.
str(navstr) # detach text
Call str() on a NavigableString you keep, or it pins the whole tree in memory.

BeautifulSoup cheat sheet · bs4 · expanded

Six pictures that make BeautifulSoup click

1 · From markup to tree

2 · Moving around the family

3 · find_all vs select — same goal

4 · .string vs .get_text() vs .stripped_strings

5 · Choosing a parser

6 · The find_* family, by direction

Worth memorizing