Quick Reference · HDF5 for Python · n-dimensional arrays on disk

h5py 3.16 · HDF5 2.0

Two ideas carry the whole library. First, an HDF5 file is a filesystem inside a file — Groups are directories, Datasets are n-dimensional arrays, Attributes are metadata stuck to either, and everything is addressed by POSIX-style path. Second, a Dataset is a lazy proxy, not a NumPy array: it has .shape and .dtype, but no bytes move until you slice it. Get those two right and the rest is detail.

install & File groups, links & attrs creating datasets reading & writing dtypes, strings & specials chunks, filters & advanced gotcha / hazard ★ most common

Distilled & cross-checked against: docs.h5py.org (Quick Start · File Objects · Groups · Datasets · Attributes · Dimension Scales · Special types · Strings in HDF5 · References · Parallel HDF5 · Multi-threading · SWMR · Virtual Datasets · FAQ) · the 3.16 / 3.15 / 3.14 release notes · github.com/h5py/h5py · PyPI · The HDF Group's HDF5 documentation

A filesystem in a file — and a proxy that reads nothing until asked
A · ONE FILE, A WHOLE DIRECTORY TREE INSIDE IT / root Group = f experiment/ Group = a directory run1/ Groups nest freely temperature Dataset · (1000, 3) float32 pressure Dataset · the actual array .attrs units="celsius" · sensor_id=7 Attributes attach to any object — groups and datasets alike. Small metadata only, not bulk data. ADDRESSED BY POSIX PATH f["experiment/ run1/temperature"] absolute or relative, like a shell B · NOTHING IS READ UNTIL YOU SLICE — THE DATASET IS A PROXY ds = f["temperature"] 0 bytes of data read ds.shape · ds.dtype metadata only — still nothing read ds[0:100, 0] reads only the chunks that overlap the slice ds[:] · ds[...] the ENTIRE dataset into RAM. Check .shape first. this is the whole point of HDF5 a 500 GB dataset on disk, a 4 MB slice in memory — provided the file is chunked ds is NOT an ndarray: np.asarray(ds) copies
quickstart — write a chunked, compressed, growable dataset and read part of it back
# pip install h5py            # wheels bundle HDF5 2.0 as of h5py 3.16
import h5py, numpy as np

with h5py.File("run.h5", "w") as f:            # 'w' TRUNCATES — 'a' to append
    g = f.create_group("experiment/run1")      # intermediate groups auto-created

    ds = g.create_dataset(
        "temperature",
        shape=(1000, 3),
        dtype="f4",                            # always name a dtype (3.16 deprecation)
        chunks=(100, 3),                       # required for compression AND resizing
        compression="gzip", compression_opts=4,
        shuffle=True,                          # big win on float data, costs nothing
        maxshape=(None, 3),                    # None = growable along axis 0
    )
    ds[:] = np.random.rand(1000, 3)
    ds.attrs["units"] = "celsius"              # metadata rides along
    ds.dims[0].label = "sample"

with h5py.File("run.h5") as f:                 # mode defaults to 'r' since h5py 3.0
    ds = f["experiment/run1/temperature"]      # lazy — no data read yet
    print(ds.shape, ds.dtype, ds.chunks)
    block = ds[0:100, 0]                       # NOW bytes move — just these chunks
    print(ds.attrs["units"])                   # attribute strings come back as str
01Install & Versionwheels bundle HDF5
02Opening Filessix modes
03File Options & Driversthe useful keywords
04Groups & Pathsdict-like, path-addressed
05Traversing the Treerecursive walks
06Attributesmetadata, kept small
07Linksthree kinds
08Delete, Move & Copyand the space they don't free
09Creating Datasetsthe central call
10Dataset Propertiesmetadata, free to read
11Reading & Slicingwhere I/O happens
12Writingassignment is the write
13Fancy Indexingnarrower than NumPy
14Chunkingthe performance decision
15Compression & Filtersper chunk, transparent
16Resizable Datasetsappend as you go
17DtypesNumPy in, NumPy out
18Stringsthe biggest gotcha
19Special Typesbeyond plain arrays
20Referencespointers inside the file
21Dimension Scalesaxes with meaning
22Efficient I/Ohabits that pay
23Virtual Datasetsmany files, one array
24SWMRwrite while others read
25Parallel HDF5MPI
26Locking & Threadswhere things go wrong
27Tools & Ecosystemaround the file

Four pictures that prevent most h5py bugs

Why chunk shape decides your read speed, what type a string comes back as, what the filter pipeline actually does, and who is allowed to open the file at once.

1 · chunk shape must match your reads

Same dataset, same query — ds[:, 0], one column. Storage layout decides whether that touches the whole file or a sixteenth of it.

query: ds[:, 0] — a single column contiguous (chunks=None) chunks=(2, 1) rows stored end to end — all 16 blocks are read each chunk read whole — only 2 of 8 chunks touched row-wise reads want chunks=(1, ncols) · column-wise want (nrows, 1) mixed or unknown access → roughly square chunks, 10 KB to 1 MB each and if one chunk exceeds the 1 MiB cache, every touch re-reads and re-decompresses it.

2 · what type is my string?

The question h5py is asked most. Datasets and attributes deliberately behave differently, and it surprises everyone once.

HOW YOU READ IT WHAT YOU GET ds[0] — vlen string dataset b"hello" → bytes ds[0] — fixed "S10" dataset b"hello" → bytes ds.asstr()[:] "hello" → str ds.astype("T")[:] NumPy StringDType ds.attrs["units"] "celsius" → str, automatically the asymmetry: datasets give bytes, attributes give str. Nobody expects this. Order matters: ds.astype("T")[:] is fast; ds[:].astype("T") detours via object arrays. Writing: h5py.string_dtype() for vlen UTF-8. Fixed length truncates without warning.

3 · the filter pipeline

Filters are applied to each chunk on the way out and reversed on the way in. That is why none of them work without chunking.

ON WRITE your array ds[:] = arr split into chunks shuffle reorder bytes compress gzip / lzf disk +fletcher32 ON READ — EXACTLY THE SAME, BACKWARDS Touch one element and its whole chunk is fetched, checksummed and decompressed. shuffle=True — groups equal-significance bytes. Nearly free, big gzip win on floats. scaleoffset=3 — lossy. Keeps three decimals, discards the rest permanently. compression=32001 — third-party filters by id; readers need hdf5plugin too. Ask for compression without chunks= and h5py picks a chunk shape for you.

4 · who may open the file at once

HDF5 is not a database. The rules are strict, the failure modes are quiet, and unable to lock file is the message you will meet first.

reader + reader + reader any number of 'r' opens, same or different processes ✓ one writer, alone 'r+', 'w' or 'a' with nothing else attached ✓ writer + plain reader OSError: unable to lock file — the classic ✗ writer + readers, in SWMR mode libver="latest", f.swmr_mode=True, readers pass swmr=True ✓ writer + writer silent corruption. There is no multi-writer mode outside MPI. ✗ many MPI ranks writing driver="mpio" — the only genuine parallel-write path ✓ Threads don't help either: h5py serialises on one global lock. Use processes. HDF5_USE_FILE_LOCKING=FALSE removes the guard rail, not the hazard.

Appendix — h5py and its neighbours

All three store chunked n-dimensional arrays. They differ in what the file is, and in who can write to it at once.

h5pya thin, faithful HDF5 wrapper

✚ reach for it when

  • The format matters — HDF5 is the lingua franca in science and instrumentation
  • You want NumPy semantics with nothing between you and the C library
  • You need references, dimension scales, committed types or the low-level API
  • The consumer is MATLAB, IDL, netCDF4, Julia or a C++ pipeline
  • One writer at a time, or true MPI parallel writes

⚠ it can't

  • Let several ordinary processes write the same file
  • Query or index — there is no where()
  • Reclaim space after a delete without h5repack
PyTablesHDF5 with a database accent

✚ reach for it when

  • You want table.where("(x > 3) & (y < 7)") evaluated out of core
  • Indexed columns and fast queries over very long tables
  • Built-in blosc compression without hunting for plugins
  • You think in rows and records rather than in arrays

⚠ trade-offs

  • A thicker abstraction — more of its own opinions on layout
  • Still one writer; the concurrency rules are HDF5's
  • Files are valid HDF5, but carry PyTables-specific attributes
Zarrchunks as objects, not one file

✚ reach for it when

  • The data lives on S3, GCS or Azure — each chunk is its own object
  • You need many processes writing concurrently, the thing HDF5 won't do
  • You're already in Dask or xarray and want the storage to match
  • You want the spec readable without a C library

⚠ it can't

  • Hand someone a single self-contained file
  • Be read by the decades of existing HDF5 tooling
  • Offer HDF5's references, region refs or committed datatypes

Worth memorizing

filesystem in a fileGroups are directories, Datasets are arrays, Attributes are metadata
a Dataset is a proxyit has .shape and .dtype, but holds no data
mode defaults to 'r'since h5py 3.0 — it used to be 'a'
'w' truncatesuse 'x' to create-or-fail, 'a' to append
ds[:] reads everythingcheck ds.nbytes first
dataset strings → bytesuse .asstr() or .astype("T") — before the slice
attribute strings → strthe asymmetry that catches everyone
fixed strings truncatesilently, and length is counted in bytes
no bool, no datetime64and NumPy 'U' dtypes are rejected outright
compression needs chunksso do resizing and SWMR
chunk shape = access patternthe highest-leverage decision you'll make
chunk cache is 1 MiBper dataset — raise rdcc_nbytes when chunks are big
maxshape=(None, …)is what makes a dataset growable
del doesn't shrink the filerun h5repack to actually reclaim it
keys() is alphabeticalunless you passed track_order=True
one writer or many readersnever two writers; SWMR needs libver="latest"
threads don't paralleliseh5py holds a global lock — use processes, and open after the fork
3.16 deprecationcreate_dataset with no data and no dtype becomes an error in 4.0