Hugging Face Datasets Cheat Sheet

1load_dataset & splitsfrom the Hub

pip install datasets from datasets import load_dataset★
Add extras as needed: datasets[audio], datasets[vision].
ds = load_dataset("stanfordnlp/imdb")★
Returns a DatasetDict keyed by split (train/test/...). Downloads + caches Parquet under HF_HOME.
train = load_dataset("imdb", split="train")★
One split → a Dataset. Slice at load: split="train[:1000]" or "train[:80%]".
load_dataset("user/repo", name="config", revision="main", token=...)
name= picks a config/subset; token= for gated/private repos; data_dir= for a repo subfolder.
no more loading scriptsv4
datasets 4.0 removed Python loading scripts — a repo with a .py loader now raises RuntimeError, and trust_remote_code is gone. Use Parquet-converted mirrors.

2From memory / local filesyour own data

from datasets import Dataset ds = Dataset.from_dict({"text": [...], "label": [...]})★
Also from_pandas(df), from_list([{...}]), from_generator(fn) (great for huge/lazy sources).
load_dataset("csv", data_files="data/*.csv")★
Built-in loaders: csv, json, parquet, text, arrow, imagefolder, audiofolder, webdataset.
load_dataset("json", data_files={"train":"tr.jsonl", "test":"te.jsonl"})
Map splits to files → a DatasetDict. JSON Lines just works.
ds.to_pandas() · ds.to_polars() · ds.to_dict()
Export back out when you need a DataFrame view.

3Inspect, slice, indexlook at rows

ds · ds.features · ds.column_names · ds.num_rows · len(ds)★
features is the typed schema; ds.shape, ds.info for metadata.
ds[0] # one row as a dict★
ds[:5] = dict of column→list (a batch), ds["text"] = whole column. Arrow-backed, so this is cheap.
ds.take(5) · ds.skip(10)
Head/offset without materializing; handy in notebooks and for streams.
ds.unique("label") · ds.features["label"].names
Distinct values; ClassLabel exposes .int2str/.str2int.

4select / filter / shuffle / splitsubset

ds.select(range(1000))★
Pick rows by index — returns a new (lazy) view, doesn't copy the data.
ds.filter(lambda x: x["label"] == 1)★
Row-wise predicate; batched=True + a mask list is much faster on big data.
ds.shuffle(seed=42) · ds.sort("len")
Reproducible shuffle; sorting helps length-bucketed batching.
ds.train_test_split(test_size=0.1, seed=42)★
Returns a DatasetDict with train/test; add stratify_by_column= for balanced splits.

5map() — transformthe workhorse

ds = ds.map(lambda x: {"n": len(x["text"])})★
Add/replace columns row-by-row. The returned dict is merged into each example.
ds.map(tok_fn, batched=True, batch_size=1000)★
Always batch heavy ops (tokenization) — the fn gets dict-of-lists and returns dict-of-lists; far faster. Default batch size 1000.
ds.map(fn, num_proc=8, remove_columns=ds.column_names)
num_proc parallelizes across cores; remove_columns drops originals so tensors stay clean.
ds.map(fn, cache_file_name=..., load_from_cache_file=True)
Results are cached to Arrow on disk — re-runs are instant unless the fn/args change.
map caches on fn fingerprintgotcha
If you edit a mapped fn but the cache looks stale-but-valid, pass load_from_cache_file=False or bump new_fingerprint.

6Features, columns, Audio/Imagetyped schema

from datasets import Features, Value, ClassLabel, Image, Audio★
The type system. Value("int32"), ClassLabel(names=[...]), Sequence(...), Image(), Audio(sampling_rate=16000).
ds = ds.cast_column("img", Image()) # decode on access★
Image/Audio columns lazily decode to PIL / arrays only when a row is read — storage stays compact.
ds.rename_column("old","new") · ds.remove_columns([...]) · ds.cast(features)
Reshape the schema; class_encode_column("label") turns strings into ClassLabel ints.
ds = ds.cast_column("audio", Audio(sampling_rate=16000))
Resample audio on the fly to match your model's expected rate.

7Formats & the training loopto tensors

ds = ds.with_format("torch")★
Rows now come out as torch tensors. Also "numpy", "pandas", "jax", "polars". set_format(...) mutates in place.
ds.with_format("torch", columns=["input_ids","label"])
Restrict which columns are tensor-ized; the rest stay Python and are hidden from the loader.
from torch.utils.data import DataLoader DataLoader(ds, batch_size=32, shuffle=True)★
A formatted Dataset is a drop-in PyTorch dataset. Or hand it straight to the 🌐 Transformers Trainer.
with ds.formatted_as("numpy"): ...
Temporary format via context manager — useful for a one-off numpy computation.

8Streaming (IterableDataset)no full download

ds = load_dataset("HuggingFaceFW/fineweb", split="train", streaming=True)★
Returns an IterableDataset — rows are fetched lazily as you iterate. Nothing is fully downloaded.
for ex in ds.map(fn).filter(pred).take(1000): ...★
Build a lazy pipeline. map(batched=True), filter, take/skip all work; with_format("torch") feeds a DataLoader.
ds.shuffle(buffer_size=10_000, seed=42)
Streaming shuffle uses a fixed-size buffer (approximate). ds.batch(1000) for chunked iteration.
IterableDataset.push_to_hub("me/streamed", num_proc=4)v4
v4 added pushing a streamed pipeline straight back to the Hub — end-to-end without landing it all on disk.

9Save, load, push to Hubpersist & publish

ds.save_to_disk("my_ds/") · load_from_disk("my_ds/")★
Round-trip the Arrow files locally (keeps features & splits).
ds.to_parquet("data.parquet") · ds.to_json(...) · ds.to_csv(...)
Export to a portable format — Parquet is the Hub-preferred one.
ds.push_to_hub("me/my-dataset", private=True)★
Uploads Parquet shards + a dataset card. Needs hf auth login / HF_TOKEN; num_proc= speeds big pushes.
ds.cleanup_cache_files()
Reclaim disk from intermediate Arrow caches created by map/filter.

!Common gotchasread before shipping

"Dataset scripts are no longer supported"v4
Old script-based datasets fail on v4. Use a Parquet-converted copy, or pin datasets<4 only as a last resort.
map without batched=True is slow
Per-row Python calls dominate. Batch heavy transforms and set num_proc for a big speedup.
IterableDataset has no len()
Streams are lazy & unsized — you can't index ds[0] or call len(); iterate, take, or set max_steps in training.
stale map cache
Editing a mapped fn may still hit the old cache; pass load_from_cache_file=False when in doubt.
columns must match the model
Remove text/extra columns before training — the collator/Trainer expects only tensorizable fields (input_ids, labels, ...).

datasets cheat sheet