Data Archival & Compression Cheat Sheet

1tar — bundle filestape archive

tar -czf archive.tar.gz dir/★
Create, gzip, to file. tar bundles the tree into one stream; z pipes it through gzip. Mnemonic: "create ze file."
tar -xzf archive.tar.gz -C /dest★
xtract into /dest (auto-detects gzip). Modern tar sniffs the compressor, so -z is often optional on extract.
tar -tf archive.tar.gz # list without extracting
List contents first — verify before you unpack. Add -v for sizes/permissions.
tar -cJf a.tar.xz dir/ · tar --zstd -cf a.tar.zst dir/★
Swap the compressor: -J=xz, -j=bzip2, --zstd=zstd (GNU tar). ⚠️ tar czf re-compresses the whole set together (solid) — great ratio, but you can't extract one file cheaply.

2zip & the classicsgzip · bzip2

zip -r archive.zip dir/ · unzip archive.zip -d /dest★
zip compresses each file independently (random access to any entry) — the universal cross-platform choice, esp. on Windows. Weaker ratio than solid tarballs.
gzip file # → file.gz (removes original) gunzip file.gz · gzip -k file # -k keeps original★
gzip (DEFLATE) is the compatibility baseline — everywhere, single-file, streamable. ⚠️ It deletes the source by default; use -k to keep it.
bzip2 file # → file.bz2 · better ratio, slower than gzip
bzip2 (Burrows-Wheeler) beats gzip on ratio but is slow and largely superseded by zstd/xz today.

3zstd — the modern defaultZstandard

zstd file # → file.zst zstd -d file.zst # decompress (or: unzstd / zstd --decompress)★
Zstandard gives near-gzip speed with better-than-gzip ratios, and decompresses very fast regardless of level — the best all-rounder for most new work.
zstd -19 file · zstd --ultra -22 file # max ratio★
Levels 1–19 (up to 22 with --ultra). Higher = smaller & slower to compress; decompression stays fast. Negative levels (--fast) trade ratio for raw speed.
zstd -T0 --long -19 bigfile # all cores + long-range matching
-T0 uses every core; --long widens the match window for large/repetitive files. Decompressing --long data needs the same flag.
zstd --train samples/* -o dict · zstd -D dict small.json
Dictionaries dramatically improve compression of many small, similar records — zstd's signature feature. Train once, reuse.

4xz, lz4, brotlithe specialists

xz -9 file # → file.xz · LZMA2, highest ratio, slow★
xz/LZMA squeezes the most out of data — ideal for write-once archives & distro packages where compress time doesn't matter. Slow both ways, RAM-hungry at high levels.
lz4 file # extremely fast, modest ratio
LZ4 compresses/decompresses at GB/s — for hot paths, real-time streams, and DB internals where CPU, not size, is the constraint.
brotli -q 11 file # → file.br · great for text/web assets
Brotli excels on text/HTTP responses (built-in dictionary) — standard for Content-Encoding: br. Snappy (like LZ4) is the fast in-datastore choice for Parquet/Cassandra.
# already-compressed data (jpg, mp4, zip) barely shrinks — don't double-compressgotcha
Media & encrypted blobs are near-incompressible; re-compressing wastes CPU and can slightly grow the file. Store those as-is.

5gzip / bz2 / lzmastdlib compressors

import gzip with gzip.open("out.txt.gz", "wt", compresslevel=6) as f: f.write(text)★
gzip.open gives a file object — use "wt"/"rt" for text, "wb"/"rb" for bytes. Same API in bz2 and lzma.
gzip.compress(data, compresslevel=9) · gzip.decompress(blob)★
One-shot bytes→bytes helpers when the whole payload fits in memory. Present in gzip, bz2, lzma, and zlib.
import lzma lzma.open("a.xz", "wb", preset=9 | lzma.PRESET_EXTREME)
lzma writes .xz (highest ratio). bz2 writes .bz2. All three share the open/compress/decompress shape, so swapping is a one-line change.

6tarfile & zipfilebundles in Python

import tarfile with tarfile.open("a.tar.gz", "w:gz") as t: t.add("dir/", arcname="dir")★
Mode picks the codec: w:gz, w:bz2, w:xz, or w (uncompressed). Read with r:* (auto-detect).
t.extractall("/dest", filter="data")★
⚠️ Always pass a filter (e.g. "data") on extract — it blocks path-traversal / absolute-path entries ("tarbomb"). It's the safe default in modern Python and a hard requirement soon.
import zipfile with zipfile.ZipFile("a.zip", "w", zipfile.ZIP_DEFLATED) as z: z.write("file.csv")★
ZipFile with ZIP_DEFLATED (gzip-class), ZIP_BZIP2, or ZIP_LZMA. Per-entry compression = pull one file without unpacking the rest.

7compression.zstdPython 3.14 · PEP 784

from compression import zstd blob = zstd.compress(data, level=19) data = zstd.decompress(blob)★new 3.14
Zstandard in the stdlib at last (PEP 784) — no third-party package needed on 3.14+. One-shot compress/decompress plus a full streaming API.
with zstd.open("a.zst", "wb", level=19) as f: f.write(data)★
zstd.open / ZstdFile mirror the gzip/lzma file API, so existing code ports cleanly. tarfile also gains a w:zst mode on 3.14.
from compression import gzip, bz2, lzma, zlib # new home for all of them
3.14 groups every compressor under the compression package. The old top-level import gzip etc. still work — the new namespace is the forward-looking one.
c = zstd.ZstdCompressor(level=19, zstd_dict=d) # dictionaries too
Advanced knobs (CompressionParameter, dictionaries via train_dict) match the CLI. On <3.14, use the PyPI zstandard package instead.

8shutil archivesone-liners

import shutil shutil.make_archive("backup", "gztar", root_dir="data/")★
Whole-directory archive in one call — formats: zip, tar, gztar, bztar, xztar. Produces backup.tar.gz here.
shutil.unpack_archive("backup.tar.gz", "/dest")
The inverse — auto-detects the format from the extension. Great for quick backup/restore scripts.
# stream compress a huge file without loading it into RAM: with open("big","rb") as i, gzip.open("big.gz","wb") as o: shutil.copyfileobj(i, o)
copyfileobj streams in fixed-size chunks — the memory-safe pattern for files bigger than RAM.

9Parquet / ORC / Avrodata-lake formats

df.to_parquet("data.parquet", compression="zstd")★
Parquet stores data by column, so similar values sit together and compress far better than CSV/JSON — plus readers skip columns they don't need. The de-facto analytics archive format.
# per-column encodings: dictionary, RLE, delta — before the codec even runs
Columnar files apply lightweight encodings first, then a block codec (snappy/gzip/zstd). That two-stage scheme is why a Parquet file often dwarfs a gzipped CSV on ratio and query speed.
ORC # columnar, Hive/Spark ecosystem Avro # row-based, schema-carrying
ORC is Parquet's columnar cousin in the Hadoop/Hive world. Avro is row-oriented with an embedded schema — better for streaming/write-heavy & record-at-a-time, and strong schema evolution.
# open table formats add ACID + time-travel on top of Parquet: Apache Iceberg · Delta Lake · Apache Hudi
For evolving datasets, table formats wrap Parquet files with a transaction log — schema evolution, snapshots, and safe concurrent writes over an archive.

10Choosing an algorithmthe tradeoff

# ratio ↑ xz/lzma > bzip2 ≈ zstd(hi) > gzip/brotli > lz4/snappy★
Rough ordering of how small. It's always a trade against speed — the tightest packers are the slowest to compress.
# speed ↑ lz4/snappy > zstd > gzip > brotli > bzip2 > xz
Rough ordering of throughput. zstd's edge: strong ratio and fast — and its decompression speed barely changes with level.
# write-once/read-rarely → xz · hot/streamed → lz4 · everything else → zstd★
Pick by access pattern: max shrink for long-term cold archives; max speed for real-time paths; zstd as the balanced default. Use gzip only when compatibility demands it.
# always benchmark on YOUR data — text, logs, binaries all differ wildlytip
Ratios are data-dependent. Measure time + output size at a few levels on a representative sample before committing.

11Cold storage & lifecycletiering

# object-store tiers: Standard → Infrequent Access → Archive/Glacier★
Cloud stores tier by access frequency: hot (instant, pricey) → cool → cold/archive (cheap per-GB, but retrieval costs money & can take minutes-to-hours). Match the tier to how often you'll actually read it.
# lifecycle rule: auto-transition after N days, expire after M
Automate tiering & deletion with lifecycle policies (S3/GCS/Azure Blob) so old data ages into cheaper tiers without manual work.
# WORM / Object Lock for compliance retention
Immutable (write-once-read-many) locks prevent deletion/overwrite for a retention window — for legal/regulatory archives. ⚠️ Archive-tier retrieval latency & fees can surprise you; test a restore.

12Integrity & practicetrust the bytes

sha256sum archive.tar.zst > archive.sha256 sha256sum -c archive.sha256★
Store a checksum next to every archive and verify on restore — catches bit-rot and truncated transfers. SHA-256 for integrity/tamper; CRC32 only for accidental corruption.
zstd -t file.zst · gzip -t file.gz · tar -tf a.tar # test★
Most tools have a built-in integrity test (-t). Run it after writing an archive and before deleting the source.
# a backup you've never restored is not a backuprule
Periodically do a full test-restore. Also: keep the tool/spec to read the format (3-2-1 rule — 3 copies, 2 media, 1 offsite), and prefer open, documented formats for long-term survival.
# compress THEN encrypt — never the other way around
Encrypted data is random & incompressible, so encrypting first destroys the ratio. (Compressing secret-bearing data before encryption has known side-channel caveats — know your threat model.)

Data archival & compression

Outline

Formats & algorithms

Python APIs

Columnar & durability

Archive Formats & Algorithms

Python APIs

Columnar Storage & Durability

Worth memorizing