Five Apache projects sit underneath most open lakehouses running in production today. Four of them were there a year ago. The fifth, Apache Ossie, entered the Apache Incubator in June 2026 and has not cut its first release yet.
That is the fastest way to describe what changed this year. The stack did not get replaced. It got a new floor added on top, and the four existing floors spent 2026 doing the unglamorous work that formats and catalogs do once real money depends on them: versioning policies, conformance test suites, encryption key handling, credential refresh, and long arguments about how to change a spec without breaking the people who already shipped against it.
This piece covers each of the five projects as of early September 2026: where it stands, what shipped, what is being argued about, and what any of it changes for the tables you own.
The open lakehouse stack has a clean layering, which is a large part of why it won. Each project owns one job and refuses the others.
Apache Parquet owns bytes on disk. It decides how a column of doubles gets encoded, how a row group is framed, where the statistics live, and how a reader skips data it does not need. Parquet is thirteen years old and holds more analytical data than any other format on earth.
Apache Arrow owns bytes in memory and bytes on the wire. It defines a columnar layout that every language agrees on, so a Rust engine hands a result to a Python client without serializing anything. Arrow turned ten in February 2026.
Apache Iceberg owns tables. A pile of Parquet files in object storage is not a table. It has no schema history, no atomic commits, no snapshot isolation, and no safe way for two writers to touch it at once. Iceberg's metadata tree turns that pile into something with the guarantees a database gives you.
Apache Polaris owns the catalog. It answers "what tables exist," "who is allowed to read this one," and "what is the current metadata pointer." Polaris graduated to an Apache Top-Level Project on February 18, 2026, after incubating with contributions from Snowflake, Google, Microsoft, Confluent, Dremio, and many others.
Apache Ossie owns meaning. It is a YAML and JSON specification for expressing what a metric is, how dimensions relate, and which joins a business definition depends on, so that "revenue" resolves the same way in a BI tool, a notebook, and an AI agent.
The reason a fifth layer appeared in 2026 is agents. When a human wrote every query, semantic drift was an annoyance that got caught in a quarterly review. When an LLM writes the query, semantic drift becomes a wrong answer delivered with full confidence and no reviewer. Every layer below Ossie tells a machine what the data is. Ossie is the first layer that tells it what the data means.
| Layer | Project | Owns | Current state, September 2026 |
|---|---|---|---|
| Storage encoding | Parquet | Bytes in files | format 2.13.0 shipped, parquet-java 1.18.1 in progress, major-version policy voted in |
| Memory and transport | Arrow | Bytes in RAM and on the wire | 25.0.1 released August 10, 2026, security model published February 2026 |
| Table semantics | Iceberg | Schemas, snapshots, commits | 1.11.0 released May 19, 2026, V3 production-stable, V4 in design |
| Catalog and access | Polaris | Namespaces, identity, grants | 1.7.0 shipped, 1.8.0 targeted for early September 2026 |
| Business semantics | Ossie | Metrics, dimensions, joins | Incubating since June 2026, first source release pending |
The rest of this piece walks each layer in turn, then puts them back together.
Parquet spent 2026 solving a problem that only successful formats get to have. The community has more good ideas than it has safe ways to ship them.
The pipeline of proposed additions is long. ALP, the adaptive lossless floating-point encoding, compresses double columns far better than the general-purpose codecs by exploiting the fact that most real-world floats are decimals in disguise. FSST does something similar for short strings by learning a symbol table over common substrings. A FlatBuffer-based footer replaces the Thrift footer to cut metadata parse time on wide schemas. A FILE logical type lets a Parquet column reference or inline an external blob. And the Variant type with shredding, which shipped earlier, is now the production answer for semi-structured columns that used to live as JSON strings.
Each of those changes has the same shape of risk. A writer that emits a new encoding produces a file that an older reader cannot decode. Parquet's compatibility promise is the reason people trust it with data they plan to keep for a decade, so the community refused to trickle these out as optional features.
In August 2026 the Parquet PMC passed a versioning vote with five binding +1s. The policy formalizes major versions as the delivery vehicle for forward-incompatible changes. New encodings arrive bundled into a version, which means one compatibility conversation per version rather than one per feature. If you operate a shop where one engine writes and a different engine reads, that policy is the single most useful Parquet development of the year. It gives you a discrete thing to plan around instead of a moving surface.
The same week, the community shipped parquet-java 1.18.0, then opened a 1.18.1 thread after regressions turned up. Fokko Driesprong is tracking the patch milestone.
In late August, Alkis Evlogimenos withdrew a proposal and called a vote to remove self-references from the FILE logical type entirely. The reasoning is worth understanding because it shows how this community thinks.
FILE as originally merged let a value specify an offset and a size with no URI, addressing a byte range inside the containing Parquet file. That sounds harmless. It is not. Those bytes are owned by the Parquet writer, so specifying them properly means giving them a compression block, an encryption module, an AAD identity for authenticated encryption, and their own size accounting. In other words, a second page mechanism reachable only through one logical type.
The community concluded that mid-sized values belong in a separate non-contiguous pages proposal, where outside values live elsewhere in the file as ordinary pages and inherit compression, encryption, and uncompressed size handling for free. Non-contiguous pages composes with FILE instead of competing with it. The vote removes self-references, requires a URI whenever an offset is set, drops the prohibition on modular encryption, and allows inline bytes alongside locator fields.
The format has not shipped this in a release yet, which is exactly why the cut was made now. Russell Spitzer raised the one open wording question: the spec cannot enforce that inline bytes and located bytes are identical, so the language should say a reader treats them interchangeably rather than asserting they match.
If you are implementing FILE today, stop and wait for the vote to close. Writing against the merged-but-unreleased spec means rewriting next month.
Two shipped features deserve attention right now, independent of every debate above.
If semi-structured data lives in your tables as JSON strings, Variant with shredding is production-real across a growing engine set, and the read-side gains are large. Shredding pulls frequently-accessed fields out of the variant blob into typed subcolumns, so a predicate on one field reads one column instead of parsing every blob.
If spatial data lives in your stack as WKB blobs or in a separate system, the native geospatial types are the migration target as engine support fills in through the year.
The third item is not a feature at all. Most teams write Parquet with whatever defaults their engine picked in 2019. A deliberate pass over compression codec, dictionary behavior, row group size, and column statistics settings routinely recovers double-digit percentages of storage and meaningful scan time. Your files get read thousands of times more often than they get written, so write-side care is the cheapest performance available. It also gives you a measured baseline, which is what you need to evaluate ALP when it lands.
Arrow turned ten in February 2026. The first commit landed on February 5, 2016. The project's origin braided together the Apache Drill community's in-memory format work, Wes McKinney's frustration with pandas performance, and the Parquet founders wanting an in-memory complement to their on-disk format. I have a house interest here that is close to genetic: Arrow was co-created by Jacques Nadeau, Dremio's co-founder, who served as Arrow's original PMC chair, and Dremio's founding engineers built the company's engine Arrow-native from day one.
People who last checked on Arrow in 2019 think of it as a memory spec with a Python library. The 2026 project is a family of standards covering the full lifecycle of data in motion.
The format layer is still the core. The columnar specification has accumulated a decade of carefully added types: string view and binary view layouts that store short strings inline and long strings by reference, list view variants, run-end encoding for repetitive data, and smaller decimal widths. The C Data Interface lets two libraries in the same process share arrays with no copy and no dependency on each other. The IPC format handles the same data across process and network boundaries.
Above that sit Flight, a gRPC-based protocol for moving Arrow record batches at wire speed, Flight SQL, a database wire protocol built on Flight, and ADBC, a client-side API where an application asks for data and receives Arrow regardless of what the source speaks. ADBC deliberately complements Flight SQL rather than replacing it. ADBC is the API your code calls, Flight SQL is one protocol a driver speaks underneath, and a driver is free to speak a native protocol instead.
Releases through 2026: 23.0.0 in January, 23.0.1 in February with a security fix for the C++ IPC file reader, 24.0.0 in April, 25.0.0 in July, and 25.0.1 on August 10.
In February 2026 the Arrow project published a formal security model covering the columnar format, the C Data Interface, and the IPC format. It sets expectations and gives guidelines for handling Arrow data that arrives from an untrusted source.
That document exists because Arrow stopped being an internal detail. When Arrow crossed process boundaries only inside your own cluster, a malformed buffer was a bug. Now Arrow crosses organizational boundaries: a Flight endpoint serving external clients, an ADBC driver pointed at a partner's system, an IPC file downloaded from somewhere. A reader that trusts offsets and lengths in a hostile buffer is a memory safety problem. If you expose an Arrow endpoint to anyone outside your trust boundary, read that document before your next release.
Arrow's late-summer threads are less about features and more about capacity, which is a healthy sign for a mature project.
A canonical BigDecimal extension type vote is open and unfinished. Micah Kornfield pushed back on process rather than substance, asking that the chosen representation follow from clearly stated requirements. The same requirements-first pushback appeared on Parquet's decimal floating-point thread in the same weeks. Both formats are being asked to represent decimals wider than 38 digits, and both communities want the requirements settled before the bytes are. Whatever lands should land in both formats at once, and the people involved know it.
A second open item matters if you handle semi-structured data. Kosta Tarasov flagged that Arrow's Variant extension spec is inconsistent with Parquet's shredding spec. If you are building Variant support in an Arrow-native engine, that inconsistency is the kind of thing that produces subtle bugs precisely at the boundary where data crosses from Parquet into Arrow memory.
The third is a review-capacity story. Nic Crane proposed limiting non-committers to three concurrent open pull requests, and Antoine Pitrou agreed. LLMs made writing pull requests cheap. Reviewing them did not get cheaper. Arrow reached for a hard cap. Polaris, as you will see, reached for a different lever on the same problem.
Iceberg had the loudest year of the five, and the story splits cleanly in two. V3 became the thing you should be running. V4 became a real specification instead of a wish list.
Iceberg 1.11.0 released on May 19, 2026. It was the largest release the project has shipped outside the initial one, with 1,117 commits since 1.10.0 and enough of them that GitHub's auto-generated release notes hit a size limit.
The headline is that the V3 feature set moved from experimental to stable defaults. Deletion vectors replace position delete files with a compact per-data-file bitmap, so a merge-on-read table stops accumulating a long tail of tiny delete files that the reader has to open and merge. The Variant type with shredding arrived as the native answer for semi-structured columns. Native geospatial types landed, along with nanosecond timestamps.
The release also carried structural work. A new File Format API abstracts the engine's internal handling of file formats, which is the groundwork for adding format plugins without touching every read path. The REST catalog learned server-side scan planning, shifting metadata work off the query engine and onto the catalog. A partition statistics scan API gives optimizers a supported way to read a table's shape instead of reverse-engineering it. Built-in table encryption arrived with envelope encryption. And the planner now applies partition pruning before deletion vector validation, so a query that eliminates a partition skips validating the deletion vectors for every file inside it.
Spark 4.1 and Flink 2.1 became the default build targets in the same release.
If you run Iceberg on V2 tables, the practical instruction is simple. Pick the tables where merge-on-read hurts most, upgrade them to V3, and rewrite the position deletes as deletion vectors. V3 is the foundation V4 features get added on top of.
Iceberg Summit 2026 in San Francisco drew more than 600 people across two days and over 70 sessions, and not one talk argued that you should adopt Iceberg. Every session assumed you already run it and asked what to do about the limits that success created. Streaming pipelines commit every few seconds. Feature tables carry thousands of columns. Disaster recovery demands table portability. V4 is the answer to that list.
As of September 2026 V4 is not released and not finalized. It exists as design documents, Iceberg Improvement Proposals, GitHub issues, and long dev list threads. The threads are where the real decisions happen, and several closed over the summer.
Equality deletes are being deprecated. Huaxin Gao's vote to deprecate equality deletes in V4 and forbid new writes has a result, and the spec PR is in progress. Equality deletes were always the cheap path for streaming writers and the expensive path for readers, because resolving them means evaluating a predicate against every candidate row. V4 chose readers. If your Flink or Kafka Connect pipeline relies on equality deletes for upserts, you need a merge-on-read strategy built on deletion vectors or a compaction cadence that folds upserts into data files. The engines will follow over two or three release cycles, and the tables you create this quarter will outlive that.
Old position deletes with row data get a loud failure. Position deletes carrying row data were deprecated in 1.11. Hongyue Zhang proposed that maintenance actions fail with an explicit exception when they meet those files rather than silently dropping the row column. That is the right call and it has an operational consequence: a rewrite_position_delete job that ran clean last month starts failing after a library upgrade. Audit for those files before you upgrade.
Manifests are heading toward Parquet-only. Steven Wu proposed making V4 manifests Parquet-only, and early responses from Anoop Johnson, Russell Spitzer, and Manu Zhang pointed the same direction. If you rely on Avro tooling to inspect manifests, start planning for a world where that tooling does not apply.
The REST catalog gets a versioned loadTable endpoint. Dhruv Arya opened the question of whether V4 tables flow through the existing loadTable response with new optional fields or through a new versioned endpoint. Amogh Jahagirdar argued the first option is not really an option, since clients are not required to parse the format version before interpreting the rest of the payload, so pushing V4 structures into a v1 response breaks the compatibility promise. Daniel Weeks converged on a shape: the v1 endpoint stays frozen at the v1 through v3 structure, a v2 endpoint returns whatever a newer client reads, and a catalog that cannot serve a V4 table to a v1 client returns an explicit error instead of an unparseable payload. Catalogs advertise V4 support through the endpoints list.
If you run a catalog service, that is your homework. If you write clients, V4 support means calling a new endpoint, not just parsing new fields.
Manifest lists get byte totals. Hemanth Boyina proposed adding data-file size totals to manifest list entries. Manifest lists already summarize file counts and row counts per manifest but not bytes, so getting a table's live size today means opening and decompressing every manifest body. Three optional long fields fix that at the manifest list layer with no extra write cost.
The single most consequential Iceberg outcome of late August was a vote that created apache/iceberg-verification, a standalone home for language-neutral conformance fixtures. Neelesh Salian called it with Sung Yun and Andrei Tserakhau, and it passed with five binding +1s and twenty-two non-binding votes.
The turnout is the point. The non-binding list read like a roll call of the Rust, Python, Go, and Java maintainers. Iceberg has at least five serious implementations, and each one has carried its own test fixtures and its own reading of edge cases in the spec. When two implementations disagree about how to interpret a manifest list, users find out in production. Shared fixtures turn spec ambiguity into a failing test.
If you maintain any Iceberg client, consume those fixtures in CI as soon as the first batch lands. A client that skips them is a client whose users report spec disagreements for it.
PyIceberg 0.12.0 is close, with view support, geometry and geography types, Python 3.14 support, a new file format API, and commit retry support. A gap surfaced in the same window: 0.12 supports vended credentials but does not automatically refresh them, so long jobs against a REST catalog that vends short-lived credentials expire mid-run. Aaron Niskode-Dossett of Etsy raised it and there is an open PR for S3 refresh. Until it lands, keep individual jobs short or pass long-lived credentials outside the vending path.
Iceberg Rust is preparing 0.11 with Danny Jones of Amazon as release manager. The most interesting structural question there is whether the DataFusion integration moves out of iceberg-rust into its own repository. Matt Butrovich cross-posted the proposal to both dev lists. The motivations are concrete: feature PRs against the TableProvider go stale because few iceberg-rust committers use DataFusion, the project wants to stay engine-agnostic, and downstream projects like Comet get blocked waiting for Arrow and DataFusion version bumps. Shawn Chang agreed in principle and named the risk, which is governance. He wants any extracted repository to stay Apache-governed rather than drifting to the engine side.
Iceberg 1.12.0 is in flight, with a branch cut targeted on or after August 26 and Hilbert curve clustering on track.
Polaris graduated to Top-Level Project on February 18, 2026, and has been shipping on a roughly monthly cadence since. 1.7.0 landed with Kafka event publishing and GCS principal attribution, along with CVE-2026-64640, a low-severity flaw in the register endpoint. Jean-Baptiste Onofré proposed 1.8.0 for early September with a preview of one large feature, naming Directories, Tags, OpenLineage, and Data Sharing as candidates.
The reason to pay attention to Polaris is architectural rather than feature-driven. In the original lakehouse picture, the catalog was a lookup table. You asked it for the current metadata pointer and it gave you one. Everything interesting happened in the engine.
That is no longer where the interesting work is. The catalog is becoming the place where identity, authorization, credential vending, lineage, and policy live, because it is the only component every engine talks to. Once five engines read the same tables, per-engine access control stops being viable. The policy has to sit at the catalog or it does not sit anywhere.
Tags. EJ Wang opened a PR for the public API contract covering tag management, assignment and unassignment, direct and inherited reads, and reverse lookup. V1 scope is catalogs, namespaces, Iceberg and generic tables as whole objects, plus top-level Iceberg table columns. Views, nested fields, and tag-based authorization are deferred. Tags are the likeliest candidate to make the 1.8.0 preview cut. The follow-on that matters most is using tags to drive access decisions, which turns classification into policy.
A purge default that blocks view drops. Ayush Saxena surfaced a real bug. With the purge guard enabled and PURGE_VIEW_METADATA_ON_DROP also true by default, views cannot be dropped at all out of the box. The drop internally requests purge and the guard returns a 403. Onofré called it a category error: the purge guard protects table storage, and a view has no comparable storage risk. The direction is to scope the guard to tables only. If you run Polaris with views today, set DROP_WITH_PURGE_ENABLED to true or PURGE_VIEW_METADATA_ON_DROP to false depending on which risk you prefer.
Federation and who owns location policy. Jiajia Li of Alibaba raised the sharpest architectural question of the summer. A federated catalog in Polaris mints credentials from its own storage configuration. When the remote catalog owns the storage, there is no local configuration and every route fails. Li built an opt-in per-catalog flag that forwards the remote's credentials, read off the loaded table's FileIO. The catch is that Polaris stops being the location policy point on that path, because it cannot validate locations against storage it never configured. Li asked the list directly whether Polaris should take this on, noting that the alternative is engines bypassing federation and talking to remote catalogs directly. This decides whether Polaris federation is a proxy or a policy layer. If you federate to a catalog that owns its storage, read that thread and say something.
Persistence plumbing. Yufei Gu argued for a Polaris-owned JDBC schema-name property rather than relying on Quarkus datasource properties, on the grounds that the Quarkus route works for PostgreSQL but not the MySQL driver. Alexandre Dutra came around on single versioned DDL scripts after a PostgreSQL-specific fix forced a pointless H2 script version bump. Neither item is exciting. Both are the sort of thing that decides whether a catalog upgrade is a Tuesday or a weekend.
Yufei Gu opened a thread on PR review and committership with a short argument. LLMs make it easy to write code and open pull requests. Polaris has seen a flood of them, which is a good problem, but the bottleneck moved to reviewers. Gu said he will weight sustained, high-quality review more heavily than PR count when considering someone for committership, for three reasons: the community has to trust a committer to merge responsibly, more good reviewers raise everyone's quality, and a good committer knows when to ask someone with more context before merging. Onofré supported it.
Every Apache project is going to have that conversation. Polaris had it early and in public. If you contribute to any of these projects, reviewing three pull requests carefully is worth more right now than opening three more.
Apache Ossie started life in November 2025 as the Open Semantic Interchange. In June 2026 Jean-Baptiste Onofré brought the proposal to the Apache Incubator. The discussion ran on the public incubator list, the vote passed with binding support across the Incubator PMC, and the resources spun up within weeks. The rename to Ossie happened because "OSI" collides with the Open Source Initiative.
The disclosure here is direct. Onofré is a colleague of mine at Dremio, and Dremio is one of three companies named as core developers of the project alongside Snowflake and dbt Labs. The contributor list is broader than those three, and the reason it is at the ASF is precisely so no single company controls it.
Ossie is a specification, not software you run. It defines a YAML and JSON format for expressing business metrics, dimensions, and the relationships between datasets, so that any tool reading or writing semantic definitions has a common language.
The problem it addresses is familiar to anyone who has audited a company's dashboards. "Revenue" is defined in the BI tool, defined again in the transformation layer, defined again in a finance spreadsheet, and defined a fourth time inside whatever the data science team built. Three of the four definitions agree. The fourth excludes refunds. Nobody finds out until two executives bring different numbers to the same meeting.
That was expensive before agents. With agents it gets worse, because an LLM asked "what was revenue last quarter" picks whichever definition it finds and reports the answer in the same confident tone regardless of which one it picked. Grounding an agent in a semantic model is the difference between a system that is right and a system that is fluent.
Here is a minimal Ossie model, taken from the project's own example:
semantic_model:
- name: ecommerce_analytics
description: Sales and customer analytics
ai_context:
instructions: "Use for sales analysis"
datasets:
- name: orders
source: sales.public.orders
primary_key: [order_id]
fields:
- name: order_date
dimension:
is_time: true
metrics:
- name: total_revenue
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.amount)
Walk through what each part is doing, because the design decisions are visible in a file this small.
semantic_model is the top-level container. A model holds datasets, relationships, and metrics as one unit, which means the thing you exchange between tools is a complete, self-consistent definition rather than a loose metric that references tables you have to go find.
ai_context is the piece that dates this specification to 2026. It carries natural-language instructions aimed at a model rather than a human. An agent reading this file learns not only what total_revenue computes but when it is appropriate to use it. No previous semantic layer format had a field for that because no previous consumer needed one.
datasets maps a logical business entity onto a physical source. source: sales.public.orders is a three-part name that resolves in the engine's catalog, which is where Ossie hands off to Polaris and Iceberg. The primary_key declaration is what makes fan-out detection possible when a metric spans a join.
fields carries row-level attributes. Marking order_date with is_time: true tells any consumer that this is the column to use for time grain, period-over-period comparison, and date filters, without the consumer guessing from the column name.
metrics is where the value concentrates. A metric carries an expression, and the expression carries dialects. That plural is deliberate. The same metric holds an ANSI SQL form plus any number of engine-specific forms, so a definition that needs a vendor function on one engine still travels to a tool that speaks a different dialect. The alternative, which the industry has lived with for a decade, is N-to-N custom translation between every pair of tools.
The specification also defines relationships as foreign key constraints between datasets, supporting simple and composite keys, and dimensions as categorical attributes.
The honest status: no release yet. Onofré revived the first-release thread and proposed starting with a source-only distribution to verify the build and release process, which is the standard way an Apache podling proves it can produce a compliant artifact before it worries about content.
The live design argument is about release units. Onofré argued converters should ship on independent cycles rather than as part of one project release, since a fix to one converter should not force a re-release of everything. Yufei Gu asked whether the Python converters should consolidate into a single project with a shared build, a common converter interface, and a consistent CLI such as ossie-convert import databricks, with heavy dependencies like MetricFlow or sqlglot as optional extras. Both threads are the same question: what is the unit of release for a specification with many converters?
There is also a live disagreement about scope. Justin Talbot proposed a smaller SQL-with-measures standard as an alternative to the broader foundational semantics document under review. That is a real fork in the road, and it is the sort of argument a young specification needs to have in the open before adoption locks anything in.
Converter work is moving. Reference converters exist for dbt, GoodData, Polaris, and Salesforce, and Markus Weimer reported that Power BI converter work started in earnest. Ecosystem participants keep arriving, with Denodo joining in March and Kyvos in August. An ontology working group and a financial services semantic working group are both running.
The repository has over 1,800 stars, 227 forks, and roughly 240 commits, and Onofré welcomed Yong Zheng as a new committer in August.
Ossie is early. Treat it accordingly: read the spec, run a converter against a model you already own, and file an issue where it fails to express something your business actually needs. That kind of feedback is worth far more to a specification at this stage than adoption is.
Here is one question moving through all five layers, so the boundaries stop being abstract.
An agent gets asked: "what was total revenue by region last quarter?"
The agent reads an Ossie model and finds total_revenue with its ANSI SQL expression, the orders dataset, the relationship joining orders to customers, and order_date marked as the time dimension. It now knows which columns to touch and which definition of revenue to use. It generates SQL:
SELECT c.region, SUM(o.amount) AS total_revenue FROM sales.public.orders AS o JOIN sales.public.customers AS c ON o.customer_id = c.customer_id WHERE o.order_date >= DATE '2026-04-01' AND o.order_date < DATE '2026-07-01' GROUP BY c.region;
The engine hands the table names to Polaris. Polaris checks the caller's identity, evaluates grants on the sales.public namespace and both tables, refuses or allows, and on allow returns the current Iceberg metadata pointer for each table plus scoped, short-lived storage credentials. That credential vending is why the engine never holds a long-lived key to the bucket.
Iceberg takes over. It reads the metadata file, picks the current snapshot, walks the manifest list, and uses partition summaries and per-column bounds to eliminate manifests that cannot contain April through June rows. Hidden partitioning on order_date means the planner prunes without the query mentioning a partition column. What survives planning is a list of Parquet data files plus any deletion vectors that apply to them.
Parquet does the reading. For each surviving file the reader consults the footer, uses row group statistics and the page index to skip pages outside the date range, decodes only the four columns the query touches, and applies the dictionary and encoding schemes each column chunk declares.
Arrow carries the result. The engine materializes decoded columns into Arrow arrays, executes the join and aggregation over that layout, and ships the final batches to the client over Flight or through an ADBC driver with no row-by-row serialization anywhere in the path.
Five projects, five jobs, one query. The layering is why you can swap the engine without touching the tables, swap the catalog without rewriting the files, and point a second tool at the same semantic model without redefining revenue.
Every layer in that path has a characteristic failure, and most of them announce themselves before they hurt.
Small files at the Parquet layer. The symptom is planning time growing faster than data volume. A streaming writer committing every thirty seconds produces thousands of files a day, and each one costs a footer read at plan time. The warning sign is a query where planning takes longer than execution. The fix is compaction on a schedule, sized so target files land in the hundreds of megabytes.
Manifest bloat at the Iceberg layer. Every commit writes a manifest. Frequent commits without maintenance leave a manifest list with thousands of entries, and planning has to open and decompress each one. Watch the count of manifests per snapshot. Rewrite manifests as part of routine maintenance.
Sorted tables that stop getting faster. This one is subtle and worth naming, because Heekyung Kim's recent proposal exists precisely for it. rewrite_data_files selects files by size and delete count. A table whose files are all a healthy size but which overlap heavily on the sort key never gets picked for sort or z-order compaction. Every run reports success. Clustering never improves. Kim's PR adds a read-only compute_sort_order_stats procedure that reports per-partition overlap depth from manifest bounds, plus an opt-in min-overlap-depth option that rewrites the files behind that depth. Until it merges, you can approximate the diagnosis by reading manifest lower and upper bounds for the sort column and counting overlaps per partition.
Equality delete accumulation. A streaming upsert pipeline writing equality deletes puts the cost on every reader, forever, until compaction resolves them. The warning sign is read latency that degrades between compaction runs on a table whose data volume is flat. V4 forbids new equality delete writes, so this is a migration with a deadline attached.
Credential expiry mid-job. A long PyIceberg job against a REST catalog that vends short-lived credentials fails partway through with an authorization error that looks like a permissions problem and is actually a clock problem. The tell is that the same job succeeds when run against a smaller partition.
Catalog as single point of failure. Once the catalog owns identity, authorization, and metadata pointers, an outage there takes down every engine at once. That is the cost of the control plane consolidating, and it means catalog availability targets need to match or exceed the strictest engine SLA you support. Test failover for real.
Semantic drift. No error message. Two tools produce different numbers and both are internally consistent. The only detection mechanism is a reconciliation someone has to run deliberately, which is the argument for a shared semantic model in the first place.
A short list of things worth doing between now and the end of the year, ordered by how much time they save relative to effort.
Get your highest-churn tables onto Iceberg V3 and convert position deletes to deletion vectors. This is the single highest-value migration available today, and it also puts you on the base V4 builds on.
Audit for V2 position delete files that carry row data. Do it before your next Iceberg library upgrade, not after a maintenance job starts failing. The fix is either a V3 upgrade with a rewrite to deletion vectors or a data compaction pass on V2 that folds deletes into data files.
Plan the exit from equality deletes if any pipeline writes them. Identify which streaming jobs produce them, decide on a replacement strategy, and schedule it. The deprecation vote has closed.
Measure your Parquet write settings before ALP arrives. Record current compressed size, encoding distribution, and scan time per column type on a representative table. Without a baseline you cannot evaluate any new encoding, and with one the evaluation takes an afternoon.
Point your implementations at shared fixtures. When apache/iceberg-verification publishes its first batch, wire it into CI for every client you maintain. Meanwhile, parquet-testing example files already exist and cost nothing to add.
Check your Polaris purge configuration if you run views. The default combination blocks view drops entirely.
Read the Arrow security model if you expose a Flight endpoint, an ADBC driver, or IPC files across a trust boundary.
Try Ossie against one real model. Pick a metric that already has three definitions in your stack, express it once in Ossie, and run the relevant converter. Where the spec fails to express what your business means, file an issue. Early feedback shapes a specification in ways that later adoption never does.
And put someone on the mailing lists. Every decision described in this piece happened in public, weeks or months before it appeared in a release note. The teams that get surprised by format changes are the teams whose only channel is the release page.
Three directions are visible from where the projects stand today.
The first is agent-shaped read patterns reshaping the formats. Point lookups against lakehouse data used to be a misuse of a scan-optimized stack. Agents issue them constantly. Work like Spotify's random access Parquet exists because of that, and the Iceberg proposals around richer statistics and manifest-level byte totals point the same way. The formats are being tuned for a workload that barely existed when they were designed.
The second is the catalog absorbing responsibilities that used to live in engines. Fine-grained read restrictions, credential vending, lineage through OpenLineage, data sharing, tags, and directories are all catalog features in flight. The design conversation happening in Iceberg about column-level policies made one decision worth internalizing: the group rejected capability negotiation, on the grounds that if clients advertise versions and two representations of a policy exist, a client opts into the weaker one. TLS downgrade attacks are the precedent. The consequence, stated plainly in the notes, is that you upgrade clients before servers.
The third is semantics becoming infrastructure. Ossie is the visible piece, but the same pressure shows up as ai_context fields in a spec, as MCP servers exposing catalogs to agents, and as engines shipping semantic layers of their own. The layer where meaning gets defined is being standardized for the same reason storage and table formats were: because N-to-N translation between proprietary formats stops scaling once N gets large enough.
The counterweight to all three is process. Every project on this list spent 2026 building the machinery to change safely: Parquet's versioning policy, Iceberg's conformance repository, Arrow's security model and PR caps, Polaris's committership standards. That work is boring to read about and it is the reason the stack is worth building on.
The state of the open lakehouse in September 2026 is a stack that stopped arguing about whether it won and started dealing with the consequences of winning.
Parquet is figuring out how to add encodings without stranding readers. Arrow is a decade old, quietly inside almost everything, and now has a security model because it crosses trust boundaries. Iceberg made V3 the production target and is writing V4 in public, one dev list thread at a time. Polaris turned the catalog into the control plane and is working through what that means for federation, policy, and governance. Ossie is early, contested, and pointed at the one problem the other four never addressed.
None of these projects belongs to a vendor, and that is the whole point. Every decision described here happened on a public list, with a vote, in front of anyone who cared to watch. If you run this stack, that is not just a governance nicety. It is your ability to see what is coming before it arrives in your production cluster.
If this piece was useful, I have written a lot more on lakehouse architecture and the projects underneath it. Apache Iceberg: The Definitive Guide covers the table format from metadata layout through production operations, and Apache Polaris: The Definitive Guide does the same for the catalog layer. You can find every book I have written, across lakehouse architecture, Apache Iceberg, Apache Polaris, and AI, at books.alexmerced.com.
For the week-to-week version of everything above, follow along on LinkedIn. Dremio's LinkedIn page carries the product and open source news. I post breakdowns of these dev list threads as they happen at linkedin.com/in/alexmerced, and my colleague Will Martin, Dremio's evangelist, covers the same stack from the EMEA side with a heavy focus on hands-on engineering. Between the three feeds you get most of what matters in this ecosystem without reading five mailing lists yourself.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
We're always looking for ways to better handle and save money on our data. That's why the "data lakehouse" is becoming so popular. It offers a mix of the flexibility of data lakes and the ease of use and performance of data warehouses. The goal? Make data handling easier and cheaper. So, how do we […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern. Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, […]