fastparquet Cheat Sheet

1Install & statuspip / read this

pip install fastparquet★
Pulls numpy, pandas, cramjam, fsspec. Python 3.9+.
# fastparquet is winding down — pandas 3.0 requires pyarrowretiring
Active development has largely stopped. For new projects use pyarrow / df.to_parquet() (card 10). Keep using fastparquet only for existing pandas-2.x pipelines.
from fastparquet import write, ParquetFile★
The two names you need: write (out) and ParquetFile (in).
df.to_parquet("f.parq", engine="fastparquet") pd.read_parquet("f.parq", engine="fastparquet")★
Most people use fastparquet via pandas by naming the engine — no direct import needed.

2Write a DataFramewrite()

write("out.parq", df)★
Single-file Parquet from a pandas DataFrame. The index is written by default (write_index=False to drop it).
write("out.parq", df, compression="snappy", row_group_offsets=500_000)★
row_group_offsets = row boundaries for logical segments (an int = rows per group). Row groups are the unit of read parallelism & filtering.
write("dataset/", df, file_scheme="hive")★
file_scheme="hive" writes a multi-file dataset directory with a _metadata file (vs "simple" = one file).
write("out.parq", df, object_encoding="utf8", times="int64")
Control how object columns & timestamps are encoded when types are ambiguous.

3Read: ParquetFileto_pandas()

pf = ParquetFile("out.parq") df = pf.to_pandas()★
Open then materialize to a DataFrame. ParquetFile also accepts a directory (a dataset) or a list of files.
pf.columns · pf.dtypes · pf.count() · pf.info★
Inspect schema & row count without reading the data — Parquet stores this in the footer.
for df in pf.iter_row_groups(): process(df)★
Iterate one row-group DataFrame at a time — bounded memory over a big file.

4Column/row-group selectread less

pf.to_pandas(columns=["a", "b"])★
Column pushdown: read only the columns you need — the whole point of a columnar format.
pf.to_pandas(filters=[("year", "==", 2026)])★
Predicate/row-group pushdown: skip row groups (and partitions) whose stats can't match — big speedups on filtered reads.
pf.to_pandas(columns=["city"], categories=["city"])
Load low-cardinality string columns as pandas category dtype to save memory.
# filters prune whole row groups, not individual rowsnote
Row-group filtering is coarse — you still get all rows in a group that might match; re-filter the DataFrame for exact results.

5Partitioned datasetspartition_on

write("dataset/", df, partition_on=["year", "month"], file_scheme="hive")★
Hive partitioning: writes year=2026/month=8/….parq subdirs. Filtering on a partition column then skips whole directories.
pf = ParquetFile("dataset/") pf.to_pandas(filters=[("year", "==", 2026)])★
Reading a partitioned dataset restores the partition columns and prunes by them automatically.
# partition on LOW-cardinality columns onlytip
Partitioning on a high-cardinality column makes millions of tiny files — slow & wasteful. Partition on coarse buckets (date, region).

6Compression & optionscodecs

write(path, df, compression="zstd") # or snappy/gzip/lz4/brotli★
snappy = fast, decent ratio (common default); zstd = better ratio, still fast; gzip = smallest but slow. Codecs come from cramjam.
write(path, df, compression={"a":"zstd", "b":"snappy", "_default":"gzip"})
Per-column compression via a dict — use heavier codecs on big columns only.
write(path, df, stats=True)
Column statistics (min/max) enable the row-group filtering in card 4 — on by default; keep them for filterable data.

7Append & schemagrow a dataset

write("dataset/", new_df, append=True, file_scheme="hive")★
append=True adds row groups/files to an existing dataset — the schema must match.
# append requires file_scheme="hive" (multi-file), not "simple"gotcha
You can't append to a single-file (simple) Parquet — use a hive dataset directory if you plan to append.
pf = ParquetFile(["a.parq", "b.parq"]) # read many as one
Pass a list of files to read several as a single logical table (matching schemas).

8Metadata & inspectionthe footer

pf.schema · pf.row_groups · pf.statistics
Inspect the Parquet schema tree, row-group layout, and per-column min/max stats.
pf.key_value_metadata # custom + pandas metadata
Read the embedded key/value metadata (pandas stores its index/dtype info here).
pf.count(filters=[("year", "==", 2026)])
Count matching rows using stats/partitions without reading the data.

fastparquet cheat sheet