Apache Iceberg and Parquet: How the File Format Works

---

Parquet sets how bytes sit inside one file. Iceberg sets which Parquet files make up a table at any moment. Teams that mix the two hit the same problems: a rename that blanks a column, a codec that differs between writers, loose files nobody can roll back. Atlan is the Context Layer for AI, and its Context Lakehouse stores metadata the same way: Iceberg tables over Parquet files. This guide covers the file layout, the defaults Iceberg ships with, how to set both layers up, and how to move existing files across.

Type	Open source, column-oriented data file format, maintained as an Apache Software Foundation project
Physical layout	Row groups, then column chunks, then pages, with file metadata in a footer at the end of the file (file format)
Codecs Iceberg accepts	zstd, brotli, lz4, gzip, snappy, or uncompressed (Iceberg configuration)
Default in Iceberg	Yes, write.format.default is parquet; Avro and ORC are the alternatives
Iceberg write defaults	128 MB row groups, 1 MB pages, zstd codec for new tables
Implementations	Java, C++, Go, and Rust libraries, plus Python through PyArrow
Works without Iceberg?	Yes. Any Parquet reader, including Spark, DuckDB, pandas, and Polars, can open the files
Best fit	Read-heavy analytics, ML training datasets, lakehouse tables

---

What is Apache Parquet?

The Apache Parquet project defines Parquet as an open source, column-oriented data file format designed for efficient storage and retrieval, with compression and encoding schemes built for bulk data. The specification lives in the parquet-format repository, and libraries in several languages implement it.

A published standard means a file written by one engine can be read by another. That portability is why Parquet became the file layer under most open lakehouse table formats. The format specification defines four terms worth knowing:

• Row group: a logical horizontal slice of the rows. A row group holds one column chunk for each column.
• Column chunk: the values for one column inside one row group, stored contiguously in the file.
• Page: a column chunk is divided into pages, and a page is the indivisible unit for compression and encoding.
• Footer: the file metadata, written after the data so a writer can finish in a single pass. A reader opens the footer first, finds the column chunks it wants, and skips the rest.

---

Where is Parquet used?

Parquet sits under most large tabular data. Four representative settings:

• Cloud data warehouses: Snowflake supports Iceberg tables that use Parquet files, and its external volumes point at customer-managed S3, Google Cloud Storage, or Azure storage. Atlan’s notes on Apache Iceberg v3 in Snowflake cover what the newer table spec changes.
• Lakehouse engines: Spark, Trino, PrestoDB, Flink, Hive, and Impala all add Iceberg tables to their SQL surface, and Parquet is the default file format underneath. The same lakehouse pattern shows up in the context layer for Databricks and the context layer for Snowflake.
• AI and ML datasets: the Hugging Face dataset viewer converts public datasets under 5 GB to Parquet automatically, so pandas, Polars, DuckDB, and PySpark can query them. Whether a Parquet dataset counts as AI-ready data depends on the context around it, not the file format.
• Local analytics: PyArrow and DuckDB read Parquet files on a laptop with no cluster.

---

How does Iceberg use Parquet?

Parquet is the default data file format for Iceberg tables, and Iceberg adds rules for how those files are written and read as part of a table. Three of them matter day to day.

Field IDs decide which column is which

The Iceberg spec says columns in data files are selected by field ID. A table’s column names and order can change after a file is written, and projection still works because it runs on IDs. The spec’s own example: a file written with columns a, b, and c can be read through a schema that renames c to measurement, and the reader picks the right column anyway. Renames and reorders do not break older files.

Manifests carry file-level statistics

Iceberg records per-file metrics, including null counts and lower and upper bounds for each column, in manifest entries. The spec says these field statistics are used when filtering to select data files, so a planner can drop files whose value range cannot match a query. Parquet also keeps min and max statistics inside the file, at the column chunk and page levels.

Table properties set write behavior once

Codec, row group size, page size, and per-column bloom filters are table properties. Set them on the table and every writer follows them.

---

Which Parquet settings does Iceberg control, and what are the defaults?

The Iceberg configuration reference lists the Parquet write properties. The ones that change file layout:

Property	Default	What it does
write.format.default	parquet	Data file format for the table: parquet, avro, or orc
write.parquet.row-group-size-bytes	134217728 (128 MB)	Parquet row group size
write.parquet.page-size-bytes	1048576 (1 MB)	Parquet page size
write.parquet.compression-codec	zstd	Codec: zstd, brotli, lz4, gzip, snappy, or uncompressed
write.parquet.bloom-filter-enabled.column.<col>	not set	Writes a bloom filter for the named column

Two details trip people up. First, the zstd default came in with Iceberg 1.4.0, whose release notes say zstd is the Parquet default “in new tables.” A table created before that keeps the codec it was created with, so read the table’s own properties before assuming zstd.

Second, Iceberg’s 128 MB row group is smaller than what the Parquet project suggests. Its configuration guidance recommends large row groups of 512 MB to 1 GB for read performance. Both are documented defaults; the right number depends on your scan pattern and the memory your readers have.

---

Can Parquet be used standalone, or does it need Apache Iceberg?

Parquet works on its own. Any Parquet library can write a file and any compatible engine can read it with no table format involved. What it lacks is table-level coordination, because a Parquet file has no notion of a snapshot or a second writer. Iceberg provides serializable isolation by swapping one table metadata file for another atomically.

• Parquet alone is enough when: one process writes the data once, many readers query it, and rewriting files to change a schema is acceptable.
• Pair Parquet with Iceberg when: several writers or engines share a table, readers need earlier snapshots, or columns and partitions will change.
• Other formats exist: Iceberg can also write Avro and ORC, and Parquet files sit under Delta Lake tables and are one of the base file formats Apache Hudi supports.

The full comparison is in Atlan’s guide to Apache Parquet vs. Apache Iceberg. For how the table layers on top are organized, see Apache Iceberg architecture and the Apache Iceberg 101 primer.

---

How do you set up Parquet, standalone and inside an Iceberg table?

The right setup depends on whether you need loose files or a managed table.

Option 1: standalone Parquet with PyArrow

PyArrow’s Parquet module writes with write_table() and reads back only the columns you ask for with read_table(). PyArrow compresses with snappy by default, so pass the codec explicitly if you want files that match an Iceberg table’s zstd default.

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({
    "order_id": [1, 2, 3],
    "region": ["emea", "apac", "amer"],
    "amount": [120.5, 80.0, 45.25],
})

pq.write_table(table, "orders.parquet", compression="zstd")  # snappy is PyArrow's default

print(pq.read_table("orders.parquet", columns=["order_id", "amount"]))  # reads two columns only

meta = pq.read_metadata("orders.parquet")
print(meta.num_row_groups, meta.row_group(0).column(0).compression)
print(meta.row_group(0).column(0).statistics)

Option 2: Parquet inside an Iceberg table with Spark

Configure an Iceberg catalog in Spark first. Then create the table with USING iceberg, a PARTITIONED BY clause, and TBLPROPERTIES for the Parquet settings. The syntax follows the Iceberg Spark DDL docs, and prod below stands for whatever catalog name you configured.

CREATE TABLE prod.db.orders (
    order_id bigint,
    region string,
    amount double,
    order_ts timestamp)
USING iceberg
PARTITIONED BY (days(order_ts))
TBLPROPERTIES (
    'write.format.default' = 'parquet',
    'write.parquet.compression-codec' = 'zstd',
    'write.parquet.row-group-size-bytes' = '134217728',
    'write.parquet.bloom-filter-enabled.column.order_id' = 'true'
);

Properties set on the table apply to every engine that writes to it, so Spark, Flink, and Trino jobs all produce files with the same settings.

---

How do you move existing Parquet files into Iceberg?

Iceberg supports in-place metadata migration, which keeps your data files and adds Iceberg metadata on top. The Spark procedures are snapshot, which tests a migration without touching the source; migrate, which replaces a Hive or Spark table with an Iceberg one; and add_files, which registers files from a table or path in an existing Iceberg table. The procedure docs say add_files does not check that the files’ schema matches the table, and that migrate or snapshot should be used where possible.

Check field IDs before you migrate. When a data file carries none, the spec lets a table define a name mapping in the schema.name-mapping.default property, which supplies fallback IDs. Columns that cannot be resolved by ID or by name mapping come back as null. So for any Parquet written outside Iceberg, find out which tool wrote it and whether the table has a name mapping.

---

What does a Parquet file not tell an agent?

Parquet and Iceberg keep technical metadata: schemas, column statistics, snapshots, and file lists. They do not record who owns a table, which definition of a metric is approved, or whether anyone certified the data. An agent that reads the table sees columns and values with no meaning attached.

That missing layer is what a data catalog for AI or a context catalog supplies. The two are not the same thing, and Atlan’s comparison of a data catalog vs. a context layer shows where each one stops. Active metadata vs. a context layer draws a similar line. Agents then need the result in a form they can query, which is where a context graph and the core components of a context layer come in.

Lineage fills one more gap. A Parquet footer cannot say which upstream job produced a file. MCP for data lineage lets an agent ask that question directly, and training data lineage for LLMs covers the same trace for model inputs.

---

How does Parquet fit into Atlan’s enterprise context layer?

AI agents need open, versioned storage for context, the same way analytics teams needed it for data. The Context Lakehouse is built on that idea. Per the product documentation, it uses Apache Iceberg as the table format, Apache Polaris (incubating) as the Iceberg REST catalog, and Apache Parquet for the underlying data files.

The video above walks through the metadata lakehouse on Apache Iceberg and what it means for the context AI agents read.

That design gives context the properties Parquet and Iceberg give data:

• Open formats: the lakehouse architecture stores Iceberg metadata files and Parquet data files, so context is not locked into a proprietary format.
• Plain SQL from your own compute: any Iceberg REST-compatible client, including Snowflake, Trino, and Spark, can query the lakehouse.
• Table history: the product page lists time travel among the Iceberg capabilities it builds on.

A knowledge graph and vector search sit on top of that storage and serve context to agents through Atlan MCP and APIs, an agent context layer in practice. Parquet stays the layer underneath, so the engines your teams already run can read it. A unified context layer built this way keeps one governed copy of meaning, instead of one per tool, and a context layer for data engineering teams shows what that changes in daily work.

---

Moving forward with Iceberg and Parquet

Start by listing where Parquet files already live in your estate, who wrote them, and whether they carry field IDs. Set the codec, row group size, and bloom filter properties on the table so every writer behaves the same. Keep standalone Parquet for single-writer, read-heavy jobs, and move shared tables to Iceberg.

The tables are only half of what an agent needs. Owners, definitions, and lineage for those tables decide whether an answer can be trusted, and that is the work of metadata management for enterprise AI and tracing data lineage for AI. Which of your Parquet-backed tables could an agent describe today, without a person explaining them first?

---

FAQs about Iceberg and Parquet

1. Is Parquet the same thing as Iceberg?

No. Parquet is a file format that defines how data is stored inside a single file. Iceberg is a table format that tracks a collection of data files, usually Parquet, as one table with snapshots and schema history.

2. Does Apache Iceberg require Parquet?

No. Iceberg’s write.format.default property accepts parquet, avro, or orc. Parquet is the default, and the Iceberg spec lists Parquet, Avro, and ORC as the immutable file formats it manages.

3. Can existing Parquet files become an Iceberg table?

Yes. Iceberg’s in-place migration keeps the existing data files and adds Iceberg metadata on top, through the snapshot, migrate, and add_files Spark procedures. Files written without field IDs need a name mapping in the schema.name-mapping.default table property so readers match columns correctly.

4. Which compression codec should I use for Parquet?

Iceberg’s documented default is zstd for new tables created from Iceberg 1.4.0 onward, and it also accepts brotli, lz4, gzip, snappy, or uncompressed. PyArrow defaults to snappy, which its documentation says generally performs better, while gzip may produce smaller files. Test the codecs on your own data before standardizing.

5. Why is Parquet used for AI and machine learning datasets?

Training and analysis jobs often read a few columns across many rows, and the columnar layout lets a reader load only those column chunks. The Hugging Face dataset viewer converts public datasets under 5 GB to Parquet so pandas, Polars, DuckDB, and PySpark can query them directly.

6. Can I open a Parquet file like a CSV?

Not in a text editor, because Parquet is a binary format. PyArrow’s read_metadata() shows the schema, row groups, and column statistics, and DuckDB can query the file with SQL.

---

Sources

1. Overview, Apache Parquet Documentation, 2026. https://parquet.apache.org/docs/overview/
2. File Format, Apache Parquet Documentation, 2026. https://parquet.apache.org/docs/file-format/
3. Configurations, Apache Parquet Documentation, 2026. https://parquet.apache.org/docs/file-format/configurations/
4. parquet-format specification, Apache Parquet, GitHub, 2026. https://github.com/apache/parquet-format
5. Configuration, Apache Iceberg Documentation, 2026. https://iceberg.apache.org/docs/latest/configuration/
6. Iceberg Table Spec, Apache Iceberg, 2026. https://iceberg.apache.org/spec/
7. Releases (1.4.0), Apache Iceberg, 2026. https://iceberg.apache.org/releases/
8. Introduction, Apache Iceberg Documentation, 2026. https://iceberg.apache.org/docs/latest/
9. Spark Configuration, Apache Iceberg Documentation, 2026. https://iceberg.apache.org/docs/latest/spark-configuration/
10. Spark DDL, Apache Iceberg Documentation, 2026. https://iceberg.apache.org/docs/latest/spark-ddl/
11. Table Migration, Apache Iceberg Documentation, 2026. https://iceberg.apache.org/docs/latest/table-migration/
12. Spark Procedures, Apache Iceberg Documentation, 2026. https://iceberg.apache.org/docs/latest/spark-procedures/
13. Reading and Writing the Apache Parquet Format, Apache Arrow Python Documentation, 2026. https://arrow.apache.org/docs/python/parquet/parquet.html
14. Apache Iceberg tables, Snowflake Documentation, 2026. https://docs.snowflake.com/en/user-guide/tables-iceberg
15. Dataset viewer: Overview, Hugging Face Documentation, 2026. https://huggingface.co/docs/dataset-viewer/parquet_process
16. Delta Transaction Log Protocol, Delta Lake, GitHub, 2026. https://github.com/delta-io/delta/blob/master/PROTOCOL.md
17. Storage Layouts, Apache Hudi Documentation, 2026. https://hudi.apache.org/docs/storage_layouts/
18. What is Atlan Lakehouse, Atlan Documentation, 2026. https://docs.atlan.com/platform/lakehouse/concepts/what-is-atlan-lakehouse
19. Lakehouse architecture, Atlan Documentation, 2026. https://docs.atlan.com/platform/lakehouse/concepts/architecture
20. Context Lakehouse, Atlan, 2026. https://atlan.com/context-lakehouse/
21. Atlan’s Metadata Lakehouse on Apache Iceberg: Real-Time AI Context, atlanHQ, YouTube, 2026. https://www.youtube.com/watch?v=KTJ0OmsZBp0