Apache Paimon and Apache Iceberg are both open table formats for the lakehouse, and they were built for different jobs. Paimon suits streaming and change data capture (CDC) workloads with frequent primary-key updates. Iceberg suits broad analytical workloads that several engines read at the same time. Your write pattern and engine stack decide between them, and a data lakehouse for AI still needs a separate answer to a different question: what an agent should know about the table before it acts on it.
Iceberg is the more established format. Paimon is the newer one, built for streaming. Both fix limits of the older Apache Hive table layout, and both sit alongside alternatives such as Apache Hudi and Delta Lake. This guide compares Paimon and Iceberg on architecture, transactions, schema evolution, and interoperability, then looks at what neither format carries on its own.
Apache Paimon vs. Apache Iceberg: an overview
Both formats solve problems Hive left open, for different workloads. Iceberg was designed around large batch-style reads and writes. Paimon was designed for lakehouses that ingest streams.
Apache Iceberg: large-scale lakehouses
Hive tracked tables by directory, which made query planning slow and directory listing expensive. It also could not guarantee atomic, consistent writes across several engines at Netflix’s scale. Netflix built Iceberg to fix that, then contributed it to the Apache Software Foundation in 2018.
Iceberg commits changes as atomic snapshots with snapshot isolation. It avoids rewriting whole partitions when data changes, and it uses hidden partitioning and partition pruning to speed up reads and writes. Query engines including Apache Spark, Apache Flink, Trino, and Presto can read it, which is a large part of why teams adopted it over Hive.
Iceberg does support streaming. It was designed around large-scale ingestion, processing, and ETL, though, which left room for a format built around streaming-first workloads. The spec keeps moving, and reader support matters: Paimon’s deletion vectors, for example, only work through Iceberg readers on format v3, which the Apache Iceberg v3 guide covers.
Apache Paimon: streaming lakehouses
Paimon began in the Flink community as Flink Table Store, with Alibaba developers among its main contributors, and became an Apache top-level project in April 2024. Its design keeps that streaming lineage. On primary-key tables it uses a log-structured merge-tree (LSM) layout. Writers buffer records, sort them, and flush new files. Compaction later merges the files so reads stay cheap. That structure is why Paimon absorbs high-volume updates and CDC streams efficiently.
Paimon’s documented capabilities include primary-key tables built for updates, time travel, schema evolution, and changelog production for streaming consumers (Apache Paimon documentation). Flink and Spark get dedicated documentation; Hive, Trino, StarRocks, Doris, and Presto are listed as ecosystem integrations, and feature support varies by engine.
Teams that already run Iceberg readers do not have to choose all at once. Paimon can publish Iceberg-compatible metadata that points at its existing data files, so supported Iceberg readers such as Trino, Athena, and DuckDB can query a Paimon table without copying the data. The option is off by default. Paimon keeps managing writes and maintenance, and the Iceberg side is read-only. On primary-key tables, files at the highest LSM level only become visible to Iceberg readers after full compaction and metadata publication.
Other projects, such as Apache XTable and Delta Lake UniForm, attack the same interoperability problem from different directions.
Apache Paimon vs. Apache Iceberg: how do the features compare?
The two formats are both foundations for a lakehouse. They differ in which problem they optimize first.
| Comparing on | Apache Paimon | Apache Iceberg |
|---|---|---|
| Workload focus | Streaming, CDC, and real-time updates | Broad analytical workloads with transactional guarantees |
| Storage architecture | LSM-tree layout for primary-key tables: sorted runs that compaction merges | Snapshots and manifests that track data files |
| Concurrency | Optimistic concurrency control; with several writers on the same files, Paimon recommends a dedicated compaction job | Snapshot isolation with optimistic concurrency; supports rollback and time travel |
| Query planning | Primary-key lookups and partition pruning | Partition pruning and hidden partitioning |
| Partitioning | Supports partitioning; primary-key tables carry extra constraints on updates across partitions | Supports partition evolution, so specs can change without rewriting existing data |
| Schema evolution | Add, drop, rename, reorder, and change column types, subject to table and engine constraints (Paimon ALTER TABLE) | Add, drop, rename, reorder, and promote column types (Iceberg evolution docs) |
| Time travel | Snapshots and tags, with some constraints on primary-key tables | Snapshot-based time travel over table history |
| Interoperability | Can publish Iceberg-compatible metadata for supported readers | Broad engine support, plus a large catalog ecosystem |
| Best fit | A high-volume, real-time lakehouse with analytical and transactional workloads | Very large analytical tables read by several engines, with concurrent writers and time travel |
The call comes down to the workload. If Flink writes streaming updates or CDC into the lake and readers need fresh data fast, Paimon fits. If Spark, Trino, and other engines need to read the same analytical tables, Iceberg fits. If you need both, Paimon’s Iceberg-compatible metadata can remove the need for a separate conversion tool such as XTable in supported scenarios. Iceberg does not natively publish a Paimon representation.
What an agent still needs on top of either format
Both formats keep technical metadata: schemas, snapshots, partition specs, file lists. Their catalogs serve the format itself. None of that tells an agent what is classified as PII, which policies apply, who can access the table, or whether anyone certified it. Before an agent acts on data, it needs all four, which is the working core of AI agent governance.
A fast Paimon stream feeding an Iceberg reader makes that gap show up sooner, not later. Real-time data for AI agents arrives faster than any human can review it, so the checks have to travel with the table. Data lineage for AI shows where a table’s rows came from, and data quality for AI agents decides whether an agent should trust them. Data contracts and data observability for AI pipelines catch drift between a producer and the agents reading its tables. Handling PII in AI pipelines and agent access control cover the other two questions: what is sensitive and who may see it.
A data catalog built for AI is the usual place teams start, and a catalog alone covers only part of the job. The difference between a data catalog and a context layer is whether policies, lineage, and meaning travel with the data to the agent. Whether your estate is AI-ready depends on that, not on which table format holds the files. If you are choosing tooling for it, an AI readiness platform evaluation lays out the questions to ask.
How Atlan fits around Paimon and Iceberg tables
Atlan is the context layer for AI, the infrastructure that makes enterprise AI accurate, trustworthy, and scalable. It does not rank table formats, and it does not need you to pick one. Context Store, Atlan’s Iceberg-native storage layer, persists context as Apache Iceberg tables behind a Polaris REST catalog. Snowflake, Databricks, Spark, and Athena can all query it, with no proprietary export. That portability is the reason a context layer should stay model-agnostic and avoid single-stack lock-in.
Atlan also connects to Iceberg tables from the other direction. Its Iceberg connector catalogs assets from an Iceberg REST catalog, and as a launch partner for Apache Polaris, Atlan documents how Polaris and Atlan work together. Atlan stays neutral across engines. The guides for Snowflake and Databricks show how the context layer sits beside each, and a multi-cloud context layer covers estates that span both.
For the architecture in more depth, here is Atlan’s own session on keeping context in Apache Iceberg.
Whichever format wins, the work that remains is the same, and it is why enterprises need a context layer between data and AI. An agent that reads a Paimon-fed Iceberg table needs the table’s meaning, owner, classification, and certification at the moment it queries. Enterprise context layers exist to supply that, and a context layer glossary helps teams agree on the vocabulary first. The context catalog holds it, and MCP-connected catalogs serve it to agents at inference. Agents can also trace an answer through lineage over MCP when a number looks wrong.
FAQs about Apache Paimon and Apache Iceberg
1. What are the primary differences between Apache Paimon and Apache Iceberg?
Paimon uses an LSM tree to absorb streaming updates and CDC from Flink. Iceberg tracks snapshots and manifests so Spark, Flink, Trino, and Presto can read the same analytical tables. Paimon can also publish Iceberg-compatible metadata for Iceberg readers to query.
2. Which data processing engines are compatible with Apache Paimon and Apache Iceberg?
Iceberg works with Spark, Flink, Trino, and Presto, among others. Paimon integrates most deeply with Flink and Spark, and it lists Hive, Trino, StarRocks, Doris, and Presto as ecosystem integrations. Feature support varies by engine.
3. How do Apache Paimon and Apache Iceberg handle schema evolution?
Both support adding, dropping, renaming, and reordering columns, and both support changing a column’s type. Iceberg’s schema evolution works the same way across any engine that reads the table. Paimon’s ALTER TABLE operations vary by engine, and some changes, such as making a column NOT NULL, are supported only in Flink.
4. When should I choose Apache Paimon over Apache Iceberg for my data lakehouse?
Choose Paimon when Flink writes streaming updates or CDC into the lake and readers need fresh data fast. Choose Iceberg when several engines, such as Spark and Trino, need to read the same analytical tables.
5. How do Apache Paimon and Apache Iceberg support partition evolution and time travel?
Iceberg lets you change a table’s partition layout without rewriting existing data, and it supports time travel through snapshots. Paimon supports partitioning and time travel through snapshots and tags, with some constraints tied to primary-key tables.
Sources
- Apache Software Foundation, “ASF Project Spotlight: Apache Iceberg.” https://news.apache.org/foundation/entry/asf-project-spotlight-apache-iceberg
- Apache Paimon, “Primary Key Table.” https://paimon.apache.org/docs/master/primary-key-table/
- Apache Paimon, “Iceberg Compatibility.” https://paimon.apache.org/docs/master/iceberg/
- Apache Paimon, “Concurrency Control.” https://paimon.apache.org/docs/master/concepts/concurrency-control/
- Apache Paimon, “Flink SQL ALTER TABLE.” https://paimon.apache.org/docs/1.3/flink/sql-alter/
- Apache Iceberg, “Evolution.” https://iceberg.apache.org/docs/latest/evolution/
- Apache Paimon, “Documentation.” https://paimon.apache.org/docs/master/
- Apache Software Foundation, “Apache Software Foundation Announces New Top-Level Project Apache Paimon.” https://news.apache.org/foundation/entry/apache-software-foundation-announces-new-top-level-project-apache-paimon
