Why Apache Arrow and Apache Parquet Are Not the Same Thing

Apache Parquet is how columnar data sits on disk. Apache Arrow is how columnar data sits in memory. They were designed by overlapping groups of people to work together, and the differences between them are all consequences of that one split.

Both are Apache projects. Both are columnar. Both turn up in the same sentence about analytics performance constantly. So the obvious question is why there are two of them, and the obvious guess, that one superseded the other, is wrong.

The One Difference Everything Follows From

A format on disk and a format in memory are optimising for opposite things.

On disk, bytes are expensive to move and cheap to think about. Reading a gigabyte from object storage costs real time and real money. Spending CPU cycles to make that gigabyte into 200 megabytes is an excellent trade. So a disk format compresses hard, encodes aggressively, and accepts that reading a single value requires decoding work.

In memory, bytes are cheap to move and expensive to think about. The data is already there. What costs you now is CPU: branches, pointer chasing, cache misses, and anything that stops the processor running the same instruction across many values at once. So a memory format stays uncompressed, keeps values at predictable offsets, and makes reading any single value a pointer arithmetic problem.

Parquet is the first. Arrow is the second. Every other difference is downstream of that.

What That Looks Like Concretely

                     Apache Parquet            Apache Arrow
Lives                on disk, object storage   in RAM, on the wire
Optimised for        bytes transferred         CPU cycles
Compression          Snappy, ZSTD, GZIP        none by default
Encoding             dictionary, RLE, delta    fixed-width, validity bitmap
Random access        decode the page first     O(1) offset arithmetic
Vectorisation        after decoding            directly on the buffers
Nulls                definition levels         separate validity bitmap
Typical lifetime     months or years           milliseconds to minutes
You interact via     a file path               a library in your process

Random access

In Arrow, an int64 column is a contiguous buffer. The millionth value is at byte offset eight million. Reading it is arithmetic.

In Parquet, that value is inside a page, which is compressed, inside a column chunk, inside a row group. Getting to it means locating the page, decompressing it and decoding it. Fine when you are scanning the whole column. Hopeless as a random access structure.

Nulls

Arrow keeps a separate validity bitmap, one bit per value, alongside a values buffer that stays fixed-width whether or not the value is null. Checking nullity never disturbs the values buffer, so vectorised code can stay vectorised.

Parquet does not store nulls at all. Nullity is carried in run-length encoded definition levels, which is dramatically more compact on disk and requires decoding before you can act on it.

Both are correct for their layer, and neither would work in the other's place.

Why Arrow Exists at All

The problem Arrow was created to solve was not storage. It was the tax every tool paid to talk to every other tool.

Before Arrow, moving a dataset from a database into pandas, or from Spark into a Python UDF, meant serialising it into some intermediate representation and deserialising it on the other side. Every pair of systems had its own conversion. The conversions were frequently the slowest part of a pipeline, and they produced nothing of value.

Arrow's proposition is that if every tool agrees on the same in-memory layout, that step disappears. Two processes can share a buffer rather than copy it. A library in C++ and a library in Rust can operate on the same bytes. The conversion cost does not get optimised, it stops existing.

That is why Arrow adoption shows up as a list of projects rather than a feature: pandas, Polars, DuckDB, Spark, Dremio, Snowflake connectors, BigQuery connectors and most of the Python data ecosystem.

How They Work Together

The normal path through an analytical query touches both, in order.

• Parquet files sit in object storage, compressed and encoded.
• A query engine reads the footer, skips what it can, and fetches only the surviving column chunks.
• Those bytes are decoded into Arrow buffers in memory.
• All filtering, joining and aggregation happens on the Arrow representation, vectorised.
• Results move to the client, often still as Arrow.

The decode step in the middle is the only conversion, and the formats were designed close enough together that it is efficient. Some encodings map almost directly: a dictionary-encoded Parquet column becomes an Arrow dictionary array with little work.

Notice what is missing. There is no row-oriented intermediate anywhere. The old path turned columnar storage into rows, handed the rows to a client, and let the client turn them back into columns. Both conversions were pure overhead.

Arrow Flight, and Why It Changes the Client Side

That last step matters more than people expect. A query can be fast and still feel slow because the result has to cross a protocol built for rows.

JDBC and ODBC are row-oriented interfaces. An engine holding a perfectly good Arrow result set must flatten it into rows, push them through the driver, and let the client reassemble columns on the other side. On a large result that cost can exceed the query itself.

Arrow Flight is a gRPC-based protocol that moves Arrow buffers directly, so no flattening happens at either end. Arrow Flight and the Flight SQL layer above it exist to remove exactly that tax.

When You Actually Touch Each One

You think about Parquet when

You are choosing compression codecs, setting row group sizes, wondering why statistics are not skipping anything, or deciding how to partition and sort data that will sit around for years. The file internals are the subject.

You think about Arrow when

You are moving data between processes or languages, building something that reads query results into Python, or wondering why a fast query takes a long time to arrive. The memory layout and the transport are the subject.

You think about neither when

You are writing SQL. Both formats are doing their jobs under an engine that hides them, and that is the intended experience. These become visible when something is slow in a way the query plan does not explain.

The Mistake Worth Avoiding

The one that costs real money is treating Arrow as a storage format.

Arrow has a file representation, the IPC format, and it is genuinely useful for handing a dataset to another process or caching an intermediate result. It is not a substitute for Parquet. Arrow IPC is uncompressed by default, carries no per-column statistics for skipping and no page index, and is far larger on disk for the same data.

Writing a large table as Arrow IPC instead of Parquet trades a one-off decode cost for a permanent storage and scan cost. The decode was never the expensive part.

The Short Version

Parquet makes data small and skippable so that reading it from storage is cheap. Arrow makes data fast to compute on and free to pass between systems. A good analytical stack reads Parquet, computes on Arrow, and ships Arrow to the client.

If someone asks you to pick one, they have misunderstood the question in the same way as asking whether to use a hard drive or RAM.

Frequently Asked Questions

Is Apache Arrow a replacement for Parquet?

No. Arrow is an in-memory format and Parquet is an on-disk format. A normal query reads Parquet from storage and decodes it into Arrow to compute on. They are consecutive steps, not alternatives.

Can I store data as Arrow files?

You can, using the Arrow IPC format, and it is useful for passing a dataset between processes or caching an intermediate result. It is a poor long-term storage choice: uncompressed by default, no column statistics for skipping, and much larger on disk.

Why is Arrow not compressed?

Because compression would defeat its purpose. Arrow is designed so a value can be read by offset arithmetic and so vectorised instructions can run straight over the buffers. Compressed data must be decompressed before either is possible.

Which is faster, Arrow or Parquet?

They are fast at different things. Parquet is faster to move from storage because it is far smaller. Arrow is faster to compute on because no decoding stands between the processor and the values.

What is Arrow Flight?

A gRPC-based protocol for moving Arrow data between systems without converting it. It matters because JDBC and ODBC are row-oriented, so an engine holding an Arrow result has to flatten it into rows and the client has to rebuild columns. Flight removes both steps.

Does Apache Iceberg use Arrow?

Not for storage. Iceberg tables hold Parquet, ORC or Avro files, and Iceberg's own metadata is Avro. Arrow appears when an engine reads those files into memory, which is an engine concern rather than a table format one.

Do I need to learn both?

Only when something is slow in a way the query plan does not explain. Parquet knowledge helps with storage layout, compression and skipping. Arrow knowledge helps with moving results between processes and languages.

Keep learning

For a full treatment of the table format, download Apache Iceberg: The Definitive Guide by Tomer Shiran, Jason Hughes, and Alex Merced, free from Dremio.

To see these ideas applied in practice, explore the Dremio documentation.