Data Archival & Compression Cheat Sheet

1tar — bundle filestape archive

tar -czf archive.tar.gz dir/★
Create, gzip, to file. tar bundles the tree into one stream; z pipes it through gzip. Mnemonic: "create ze file."
tar -xzf archive.tar.gz -C /dest · tar -tf archive.tar.gz★
xtract into /dest (auto-detects the compressor); -tf lists contents without extracting — verify before you unpack.
tar -cJf a.tar.xz dir/ · tar --zstd -cf a.tar.zst dir/★
Swap the compressor: -J=xz, -j=bzip2, --zstd=zstd (GNU tar). ⚠️ tar czf compresses the whole set together (solid) — great ratio, but you can't extract one file cheaply.
tar -czg snapshot.snar -f backup.tgz /data # see card 5
-g (--listed-incremental) turns tar into an incremental backup engine — covered in the Backup section.

2zip & the classicsgzip · bzip2

zip -r archive.zip dir/ · unzip archive.zip -d /dest★
zip compresses each file independently (random access to any entry) — the universal cross-platform choice, esp. on Windows. Weaker ratio than solid tarballs.
gzip file # → file.gz (removes original) gunzip file.gz · gzip -k file # -k keeps original★
gzip (DEFLATE) is the compatibility baseline — everywhere, single-file, streamable. ⚠️ It deletes the source by default; use -k to keep it.
bzip2 file # → file.bz2 · better ratio, slower than gzip
bzip2 (Burrows-Wheeler) beats gzip on ratio but is slow and largely superseded by zstd/xz today.

3zstd — the modern defaultZstandard 1.5.7

zstd file # → file.zst zstd -d file.zst # decompress (or: unzstd)★
Zstandard gives near-gzip speed with better-than-gzip ratios, and decompresses very fast regardless of level — the best all-rounder for new work. CLI now uses multiple threads by default (up to 4).
zstd -19 file · zstd --ultra -22 file · zstd --max file★1.5.7
Levels 1–19 (up to 22 with --ultra); negative levels (--fast) trade ratio for speed. The new --max maximizes ratio at the cost of time/memory. High levels (18+) got much faster in 1.5.7.
zstd -T0 --long -19 bigfile # all cores + long-range matching
-T0 uses every core; --long widens the match window for large/repetitive files. ⚠️ Decompressing --long data needs the same flag.
zstd --train samples/* -o dict · zstd -D dict small.json
Dictionaries dramatically improve compression of many small, similar records — zstd's signature feature. Train once, reuse (large dicts can be memory-mapped).

4xz, lz4, brotlithe specialists

xz -9 file # → file.xz · LZMA2, highest ratio, slow★
xz/LZMA squeezes the most out of data — ideal for write-once archives & distro packages where compress time doesn't matter. Slow both ways, RAM-hungry at high levels.
lz4 file # extremely fast, modest ratio
LZ4 compresses/decompresses at GB/s — for hot paths, real-time streams, and DB internals where CPU, not size, is the constraint.
brotli -q 11 file # → file.br · great for text/web assets
Brotli excels on text/HTTP responses (built-in dictionary) — standard for Content-Encoding: br. Snappy (like LZ4) is the fast in-datastore choice for Parquet/Cassandra.
# already-compressed data (jpg, mp4, zip) barely shrinks — don't double-compressgotcha
Media & encrypted blobs are near-incompressible; re-compressing wastes CPU and can slightly grow the file. Store those as-is.

5Incremental backuptar · rsync

tar -czg /snap.snar -f full.tgz /data # level 0 (full) tar -czg /snap.snar -f inc1.tgz /data # later: only changes★backup
--listed-incremental (-g) keeps a snapshot file (.snar) of prior state; each run archives only what changed since. Restore = extract the full, then each incremental in order.
rsync -av --link-dest=/bak/prev /src/ /bak/new/★
rsync snapshots: unchanged files become hard links to the previous backup, so each dated folder is a full, browsable tree but only new/changed files use disk. The classic space-cheap rotating backup.
rsync -a --delete /src/ /mirror/ # exact mirror (no history)
Plain rsync mirrors a tree efficiently (delta-transfer sends only changed blocks over the wire). Add --link-dest for history; --delete to prune removed files.
# incrementals chain — lose one and later ones can't fully restoregotcha
A tar incremental chain depends on every prior level. Take periodic new level-0 fulls, and test the whole restore path (card 16).

6Dedup backup toolsrestic · Borg

restic -r s3:s3.amazonaws.com/bucket init restic -r ... backup --compression max /srv restic -r ... snapshots · restore latest --target /r★dedup
restic: encrypted, deduplicated, incremental-forever backups. Speaks S3/object storage natively (nothing installed at the far end). zstd compression since 0.14 (--compression max). restic prune reclaims space.
borg init --encryption=repokey repo borg create --compression zstd,10 repo::'{now}' /srv borg prune --keep-daily=7 --keep-weekly=4 · borg mount★
BorgBackup: same idea, per-archive compression, and often finer dedup. ⚠️ Needs the borg binary on the repo host (a repo is served by a process, not a plain filesystem/API). borg mount browses a backup as a FUSE filesystem.
# both: content-defined chunking → store each unique chunk once
Files are split into content-defined chunks, hashed, and only unseen chunks are stored — so re-backing-up mostly-unchanged data is tiny and deduplicated across snapshots/machines.

7Delta & chunkingdiffs & CDC

zstd --patch-from=old.bin new.bin -o patch.zst zstd -d --patch-from=old.bin patch.zst -o new.bin★1.5.7
Differential compression: --patch-from encodes a file as a delta against an older version (like bsdiff) — tiny patches for near-identical files. 1.5.7 made high-level --patch-from 3–5× faster.
bsdiff old new patch · xdelta3 -e -s old new patch.vcdiff
Dedicated binary-diff tools for release/OTA updates — ship a small patch instead of the whole file. Great for versioned binaries & images.
# content-defined chunking (CDC): rolling-hash boundaries
CDC is the dedup primitive behind restic/Borg: a rolling hash sets chunk boundaries on content, so an insertion shifts only nearby chunks — unlike fixed-size blocks, most chunks stay identical and dedupe.

8gzip / bz2 / lzmastdlib compressors

import gzip with gzip.open("out.txt.gz", "wt", compresslevel=6) as f: f.write(text)★
gzip.open gives a file object — use "wt"/"rt" for text, "wb"/"rb" for bytes. Same API in bz2 and lzma.
gzip.compress(data, compresslevel=9) · gzip.decompress(blob)★
One-shot bytes→bytes helpers when the whole payload fits in memory. Present in gzip, bz2, lzma, and zlib.
import lzma lzma.open("a.xz", "wb", preset=9 | lzma.PRESET_EXTREME)
lzma writes .xz (highest ratio). bz2 writes .bz2. All three share the open/compress/decompress shape, so swapping is a one-line change.

9tarfile & zipfilebundles in Python

import tarfile with tarfile.open("a.tar.gz", "w:gz") as t: t.add("dir/", arcname="dir")★
Mode picks the codec: w:gz, w:bz2, w:xz, w:zst (3.14), or w (uncompressed). Read with r:* (auto-detect).
t.extractall("/dest", filter="data")★
⚠️ Always pass a filter (e.g. "data") on extract — it blocks path-traversal / absolute-path entries ("tarbomb"). It's the safe default in modern Python and a hard requirement soon.
import zipfile with zipfile.ZipFile("a.zip", "w", zipfile.ZIP_DEFLATED) as z: z.write("file.csv")★
ZipFile with ZIP_DEFLATED (gzip-class), ZIP_BZIP2, or ZIP_LZMA. Per-entry compression = pull one file without unpacking the rest.

10compression.zstdPython 3.14 · PEP 784

from compression import zstd blob = zstd.compress(data, level=19) data = zstd.decompress(blob)★new 3.14
Zstandard in the stdlib at last (PEP 784) — no third-party package on 3.14+. One-shot compress/decompress plus a full streaming API.
with zstd.open("a.zst", "wb", level=19) as f: f.write(data)★
zstd.open / ZstdFile mirror the gzip/lzma file API, so existing code ports cleanly.
from compression.zstd import ZstdCompressor, train_dict, CompressionParameter c = ZstdCompressor(level=19, zstd_dict=train_dict(samples, 100*1024))
Full advanced API: streaming ZstdCompressor/ZstdDecompressor, dictionaries (train_dict/finalize_dict), and tuning via CompressionParameter (window log, threads, ...).
from compression import gzip, bz2, lzma, zlib # new home for all
3.14 groups every compressor under the compression package (old top-level import gzip still works). On <3.14, use the PyPI zstandard package for zstd.

11shutil archivesone-liners

import shutil shutil.make_archive("backup", "gztar", root_dir="data/")★
Whole-directory archive in one call — formats: zip, tar, gztar, bztar, xztar. Produces backup.tar.gz here.
shutil.unpack_archive("backup.tar.gz", "/dest")
The inverse — auto-detects the format from the extension. Great for quick backup/restore scripts.
# stream compress a huge file without loading it into RAM: with open("big","rb") as i, gzip.open("big.gz","wb") as o: shutil.copyfileobj(i, o)
copyfileobj streams in fixed-size chunks — the memory-safe pattern for files bigger than RAM.

13Parquet internalswhy it's small & fast

# layout: file → row groups → column chunks → pages★deep
A Parquet file is split into row groups; within each, every column is a column chunk of pages. Tune with row_group_size — bigger groups compress better, smaller groups allow finer skipping.
# encodings run BEFORE the codec: dictionary · RLE/bit-pack · delta · byte-stream-split★
Lightweight per-column encodings (DICTIONARY, RLE, DELTA_BINARY_PACKED, BYTE_STREAM_SPLIT for floats) shrink data first; then a block codec (zstd/snappy/gzip) runs. That two-stage scheme is why Parquet dwarfs a gzipped CSV.
df.to_parquet(..., row_group_size=256_000, use_dictionary=True)
Per-column min/max statistics (+ optional page indexes / bloom filters) let engines do predicate & projection pushdown — skip whole row groups/pages that can't match a filter, reading far less.
# many tiny Parquet files kill read performance ("small files problem")gotcha
Aim for ~128MB–1GB files / sensible row groups. Compact small files (table-format OPTIMIZE/compaction) rather than writing thousands of tiny ones.

14Choosing an algorithmthe tradeoff

# ratio ↑ xz/lzma > bzip2 ≈ zstd(hi) > gzip/brotli > lz4/snappy★
Rough ordering of how small. It's always a trade against speed — the tightest packers are the slowest to compress.
# speed ↑ lz4/snappy > zstd > gzip > brotli > bzip2 > xz
Rough ordering of throughput. zstd's edge: strong ratio and fast — and its decompression speed barely changes with level.
# write-once/read-rarely → xz · hot/streamed → lz4 · everything else → zstd★
Pick by access pattern: max shrink for long-term cold archives; max speed for real-time paths; zstd as the balanced default. Use gzip only when compatibility demands it.
# always benchmark on YOUR data — text, logs, binaries all differ wildlytip
Ratios are data-dependent. Measure time + output size at a few levels on a representative sample before committing.

15Cold storage & lifecycletiering

# object-store tiers: Standard → Infrequent Access → Archive/Glacier★
Cloud stores tier by access frequency: hot (instant, pricey) → cool → cold/archive (cheap per-GB, but retrieval costs money & can take minutes-to-hours). Match the tier to how often you'll read it.
aws s3 cp big.tar.zst s3://bucket/ --storage-class DEEP_ARCHIVE★
Write straight to an archive class, or let a lifecycle rule auto-transition after N days & expire after M. Glacier restores come in expedited/standard/bulk speeds (different fees).
# WORM / Object Lock for compliance retention
Immutable (write-once-read-many) locks prevent deletion/overwrite for a retention window — for legal/regulatory archives. ⚠️ Archive-tier retrieval latency & fees surprise people; test a restore.

16Integrity & practicetrust the bytes

sha256sum archive.tar.zst > archive.sha256 · sha256sum -c archive.sha256★
Store a checksum next to every archive and verify on restore — catches bit-rot and truncated transfers. SHA-256 for integrity/tamper; CRC32 only for accidental corruption.
zstd -t file.zst · gzip -t file.gz · restic check · borg check★
Most tools have a built-in integrity test (-t); backup tools have check. Run after writing and before deleting the source.
# a backup you've never restored is not a backuprule
Periodically do a full test-restore. Keep the tool/spec to read the format (3-2-1 rule — 3 copies, 2 media, 1 offsite), and prefer open, documented formats for long-term survival.
# compress THEN encrypt — never the other way around
Encrypted data is random & incompressible, so encrypting first destroys the ratio. (Compressing secret-bearing data before encryption has known side-channel caveats — know your threat model.)

Data archival & compression

Outline

Formats, algorithms & backup

Python APIs

Columnar & durability

Archive Formats & Algorithms

Backup & Deduplication

Python APIs

Columnar Storage

Choose, Store & Verify

Worth memorizing