Arrow is one columnar memory format that many libraries and languages share. PyArrow gives you the pieces — Array → ChunkedArray → Table — plus compute kernels, fast file I/O, and datasets. The payoff: read only the columns you need, compute on them vectorized, and hand data to pandas / NumPy / Polars often with zero copies.
Distilled & cross-checked against: arrow.apache.org/docs/python (data model, compute, parquet, dataset, fs) · PyPI · community guides (Medium, DEV, Substack) — API for PyArrow ≈ 25.x
pip install pyarrow★import pyarrow as pa★import pyarrow.compute as pc★import pyarrow.parquet as pq★import pyarrow.dataset as dsimport pyarrow.csv, pyarrow.featherpyarrow.json, pyarrow.fs.pa.__version__'25.0.0'.pa.Array★pa.ChunkedArraypa.RecordBatchpa.Table★pa.Schema / Field / DataTypepa.array([1, 2, None])★pa.array([1, 2], type=pa.int32())pa.table({"n": [1,2], "s": ["a","b"]})★pa.Table.from_arrays([a, b], names=[...])pa.chunked_array([[1,2],[3,4]])pa.record_batch({...})pa.int64() # float64 · bool_ · string★pa.timestamp("ms") # date32 · decimal128pa.list_(pa.int32())pa.struct([...]).pa.dictionary(pa.int32(), pa.string())pa.schema([("n", pa.int64())])★pa.field(...) for metadata.table.cast(schema)table.schema★table.num_rows # .num_columnstable["col"] # → ChunkedArray★table.select(["a", "b"])★table.slice(0, 5)table.rename_columns([...])drop_columns, append_column.table.combine_chunks()pc.field("x") > 5★table.filter(expr)★(pc.field("a")>0) & (pc.field("b")<9)& | ~.pc.field("c").isin(["IN","US"])★table.take([0, 3, 7])arr.filter(mask)table.group_by("k").aggregate([("v","sum")])★.aggregate([("v","sum"),("v","mean")]).aggregate([([], "count_all")])pc.sum(col) # mean · min_max · stddev★table.sort_by("x")★[("x","descending")] for order.pc.value_counts(col) # uniquepc.add(col, 1) # multiply · subtractpc.cast(col, pa.int32())★pc.fill_null(col, 0)★pc.dictionary_encode(col)pc.utf8_upper(col) # match_substringpc.is_null(col) # is_validleft.join(right, keys="id")★join_type="inner""left outer", "full outer".join_type="left semi"join_type="left anti"left_suffix="_l", right_suffix="_r"pq.read_table("data.parquet")★pq.read_table(p, columns=[...], filters=[...])★pq.write_table(table, p, compression="zstd")★snappy (default) · zstd · gzip.pq.write_to_dataset(table, root, partition_cols=[...])pq.ParquetFile(p).schema_arrowpq.read_metadata(p)pa.csv.read_csv("f.csv")★pa.csv.write_csv(table, "f.csv")pa.json.read_json("f.json")pa.feather.write_feather(table, "f.arrow")★read_feather back.pa.ipc.new_stream(sink, schema)ds.dataset("dir/", format="parquet")★dataset.to_table()dataset.scanner(columns=c, filter=e).to_table()★dataset.filter(pc.field("x")>5).head(5).ds.write_dataset(table, base, partitioning=...)ds.dataset("s3://bucket/path/")★pa.fs.S3FileSystem(region="us-east-1")pa.fs.GcsFileSystem() # LocalFileSystempa.fs.HadoopFileSystem(host, port)filesystem=fspa.Table.from_pandas(df)★table.to_pandas()★table.to_pandas(types_mapper=pd.ArrowDtype)★arrow-backedarr.to_numpy(zero_copy_only=True)no copypa.array(np_arr)pl.from_arrow(table)table["x"][0] = 9immutableto_pandas() # object dtypecopiestypes_mapper=pd.ArrowDtype.chunked.to_numpy(zero_copy_only=True)failscombine_chunks().df.to_parquet() # whole fileno pushdownds.scanner to filter before reading.pa.Tablenot in specThe columnar data model, why it's fast, how nulls work, and the zero-copy hand-off. Based on the official Apache Arrow docs.
An Array is one column-chunk. Stack chunks → ChunkedArray (a column). Line up columns under a schema → Table.
Arrow stores each column contiguously. That means SIMD-friendly compute and reading only the columns a query touches.
Every array carries a separate 1-bit-per-value bitmap. No sentinel value, no wasted space — and null checks are just bit ops.
The same memory buffers back an Arrow Table, an Arrow-dtype pandas frame, NumPy, and other languages via the C Data Interface — no serialization.
ds.scanner(columns=, filter=) prunes before readingto_pandas(types_mapper=pd.ArrowDtype) stays Arrow-backedto_numpy()