.svg)
Key Takeaways
- Apache Iceberg architecture tracks a table through a metadata tree, catalog, metadata file, manifest list, manifest files, and data files, rather than a folder listing.
- Schema evolution works because Iceberg tracks columns by a permanent internal ID, not by name or position, so changes never rewrite existing data.
- Partition evolution lets a table's partitioning strategy change going forward without touching any already-written data files.
- Time travel works by keeping every snapshot in an immutable log, addressable by ID or timestamp, until a maintenance operation expires it.
- Multiple engines can safely write to the same table because every commit is a single atomic catalog pointer swap under optimistic concurrency control.
A folder listing cannot tell two concurrent Spark jobs whether they just corrupted the same table. Apache Iceberg architecture solves that by replacing directory scans with a strict, versioned metadata record, a catalog pointer, a metadata file, manifest lists, and manifest files that track exactly which data files belong to a table at any given moment.
That structure also makes schema evolution, partition evolution, and time travel natural consequences of the design, not separate features bolted on top.
What Makes Apache Iceberg's Table Architecture Different From a Data Lake File Listing?
Apache Iceberg differs from a traditional data lake by defining a table through a strict, versioned metadata record rather than inferring its state from raw directory contents.
This shift from folder scans to a definitive ledger solves the real problem of concurrent writes and partial reads. In a raw data lake, object stores lack multi-file atomic renames, making half-finished writes visible by default.
For example, if an engine writes 100 Parquet files to a directory and crashes at file 50, a concurrent query will process those 50 incomplete files as valid data. Iceberg prevents this by using atomic commits. The table's state updates only after the entire job finishes, ensuring readers never see a partial or corrupted dataset.
Because Iceberg stores per-column minimum and maximum values alongside each file reference, a query engine knows exactly what is inside a file before opening it.
If you query for December transactions, the engine instantly skips any file where the maximum date is in November. A standard directory listing offers no such intelligence, forcing the system to either read everything or rely entirely on folder structure to guess what it needs.
For a broader look at what this enables day-to-day, Iceberg tables: features, benefits and best practices covers the operational side. The rest of this piece stays at the internals level.
What Are the Layers of Apache Iceberg's Metadata Architecture?
Apache Iceberg organizes a table's state into four linked metadata layers, with the catalog above them as the entry point, and a query engine walks that chain top-down to decide what needs to be read. Each layer narrows the candidate set further before the engine ever touches a data file:
A fresh metadata file is written on every commit, and each snapshot gets its own manifest list, which is why a table with millions of files still stays fast to query: the engine narrows the candidate set through two layers of metadata before it touches any data.
Keeping that metadata accurate and observable across a growing estate is its own discipline, covered in how metadata management powers scalable lakehouse architectures.
How Does Apache Iceberg's Catalog Track the Current State of a Table?
Apache Iceberg's catalog tracks a table's current state by holding a single pointer to its current metadata file, and every commit swaps that pointer to a new metadata file in one atomic operation, so a reader always sees a complete, consistent table and never a half-written state.
Think of it as a library front desk: you ask for the exact location instead of wandering the aisles. Several backends can serve this role, so most teams reuse infrastructure they already operate:
- REST catalog services, which define catalog operations as an open protocol instead of a vendor-specific API, so a table can move between engines and platforms freely.
- AWS Glue, a managed catalog many teams already run alongside other AWS data services.
- Hive Metastore, the long-standing backend most Hadoop-era platforms already have in place.
- Nessie, which adds Git-like versioning semantics on top of catalog operations.
- JDBC-compatible databases, pointing the catalog at infrastructure a team already monitors.
That openness keeps the format's broader promise intact, a theme explored in how to build a data lake that doesn't lock you in. Acceldata's engineering team has tackled a related problem from the storage side, detailed in how Acceldata extended Apache Ranger governance to Amazon S3 by compiling policy into native bucket rules instead of an enforcement agent.
How Does Schema Evolution Work Internally in Apache Iceberg?
Schema evolution works internally in Apache Iceberg because every column gets a permanent internal ID at creation, and the engine tracks it by ID rather than name or position, so a schema change only updates the metadata file's schema record and never touches a data file.
That's why these operations complete at the same speed on a ten-row table and a ten-billion-row one:
- Adding a column assigns it a new ID and appends it to the schema record.
- Dropping a column marks its ID absent from the current schema, while older snapshots keep referencing it.
- Renaming a column changes the name attached to an existing ID, not the data underneath it.
- Reordering columns changes the order existing IDs appear in, without moving any bytes.
- Widening a column type updates the type against that ID, since Iceberg allows safe promotions like int to long.
Formats that resolve columns by name or position behave differently: reordering there can silently change which physical column a query reads, and a rename can orphan data unless every file is rewritten.
Iceberg sidesteps both by never letting file layout carry a column's identity. The upstream modeling choices behind painless evolution are a separate concern from the mechanics, covered in schema design best practices for scalability.
How Does Partition Evolution Work in Apache Iceberg's Architecture?
Partition evolution works in Apache Iceberg's architecture because the partition transform, for example, day() to group timestamps by date or bucket() to distribute data evenly, lives inside the metadata file rather than the physical directory layout, so changing how a table is partitioned creates a new spec version without rewriting existing data.
Older data keeps querying under the spec it was written with, new data adopts the new layout, and each manifest file records which spec version its files belong to, so the engine plans correctly even if the strategy changed a year ago.
Hidden partitioning runs on the same principle. Because partition values are derived automatically from raw column values through these transforms, users query the actual column directly instead of a separate partition column the engine maintains behind the scenes, which removes a common source of missed pruning: silently losing partition skipping just because someone filtered on a raw timestamp instead of a manually derived date string.
How Does Time Travel Work Internally in Apache Iceberg?
Time travel works internally in Apache Iceberg because every commit creates a new, immutable snapshot recorded in the metadata file's snapshot log, and querying by a specific snapshot ID or timestamp simply tells the engine which manifest list to read instead of the current one.
Snapshots and the snapshot log
Every commit appends a new snapshot to the log rather than overwriting the last one, and each points to the exact manifest list defining the table's contents at that moment. That log is what makes "read this table as of last Tuesday" possible: the engine looks up the matching snapshot and reads its manifest list instead of the current one.
Rollback
Rollback uses the same machinery in reverse: pointing the catalog's current-metadata pointer back at an earlier snapshot restores that version. No data is copied or rewritten, which is why rolling back a large table completes almost instantly.
Retention and cleanup
Snapshots aren't retained forever by default, so retention is an explicit planning decision, not free history. Expiring old snapshots doesn't reclaim storage on its own, since orphaned data files with no remaining reference need a separate cleanup step; teams that schedule neither watch metadata and storage grow unbounded.
How Does Apache Iceberg Handle Row-Level Deletes and Updates Internally?
Apache Iceberg handles row-level deletes and updates internally through two strategies, copy-on-write and merge-on-read, because data files are immutable and nothing edits a row in place.
Copy-on-write vs. merge-on-read
Deleting one row from a 500 MB file shows the trade-off plainly. Copy-on-write rewrites the whole file without that row, so the write is expensive, but every read stays clean.
Merge-on-read leaves the file alone and writes a small file naming which rows to skip, so the write is cheap, but every read applies that instruction first; copy-on-write suits tables that change rarely, merge-on-read suits ones under constant update.
Position deletes and equality deletes
Merge-on-read identifies skipped rows two ways. Position deletes point at a row's exact location, such as row 4,812 of file A, requiring the writer to locate it first, while equality deletes name a value instead, such as id = 5, so every matching row disappears wherever it lives, at the cost of more read-time work.
Deletion vectors in Iceberg v3
Position delete files created a matching problem: a reader opening one data file had to find every delete file referencing it and line up row numbers across them. Iceberg v3 replaces this with deletion vectors, a bitmap attached to one data file marking whether each row survives, so the reader checks its own vector instead.
Under v2, engines weren't required to compact delete files together at write time, so unmaintained tables built up too many of them, a failure mode v3's single-vector-per-file rule is built to close.
How Does Apache Iceberg's Architecture Stay Consistent When Multiple Engines Read and Write the Same Table?
Apache Iceberg's architecture stays consistent across engines because every commit is a single atomic catalog pointer swap, so Spark, Trino, Flink, and others can all commit under optimistic concurrency control: an engine reads the current snapshot, the catalog accepts its commit only if nothing else landed in between, and the loser retries against the new snapshot instead of corrupting the table.
That guarantee matters most at the scale where manual checks stop working. One Acceldata customer, a global data provider running quality checks across 30,000+ sources, was stuck resolving issues reactively while rule updates took over a month and processing ran for 9 days at a stretch. After moving to AI-powered, self-service rule management, processing dropped to 7 hours and rule deployment to under a day, a 30x acceleration.
The guarantee also stops at the table: Iceberg confirms Flink and Trino saw consistent state but stays silent on which upstream job wrote the bad value they agreed on, and that question belongs to lineage.
Acceldata's ADOC answers it by joining OpenLineage events into one graph spanning REST catalog and Glue-registered Iceberg tables, so an event published to Pub/Sub, processed through Glue, stored in Iceberg, and loaded into Redshift appears as one connected path, detailed in OpenLineage coverage across Redshift, Glue, Pub/Sub and Iceberg.
Operationalizing Apache Iceberg's Architecture With Acceldata
Understanding the metadata tree, the field-ID model, and the snapshot log is what lets a team trust Iceberg at the format level. At enterprise scale, the harder problem sits one layer up: metadata correctly structured on disk still has to stay observable and governed across every engine writing to it, and the format takes no position on that.
Holding that trust together comes down to a few habits:
- Standardize on the REST catalog specification instead of a proprietary catalog, so tables stay portable across engines and platforms.
- Monitor snapshot growth and schedule both expiration and orphan-file cleanup, so metadata and storage don't expand unchecked.
- Trace lineage through Iceberg as an intermediary layer rather than only at its edges, so a broken upstream job gets caught before it corrupts downstream tables.
Acceldata's ADOC platform correlates OpenLineage events across REST catalog and Glue-registered Iceberg tables into a single lineage graph spanning ingestion to analytics.
Book a demo and see how Acceldata traces lineage and governs metadata across every Iceberg table in the estate, no matter which engine wrote to it.
FAQs
Does Apache Iceberg require running a separate database to operate its catalog?
No. Apache Iceberg does not require a separate database for its catalog; it supports catalog implementations that can use services such as a REST endpoint, Hive Metastore, JDBC database, or cloud-native stores. The database or service requirement depends on the specific catalog implementation you choose.
What happens if two different query engines try to commit to the same Iceberg table at the same time?
If two query engines try to commit to the same Iceberg table simultaneously, Iceberg’s optimistic concurrency control detects the conflict and allows only a valid commit to succeed. The conflicting writer must then retry against the table’s updated metadata, preventing concurrent writes from silently overwriting each other.
How does Apache Iceberg avoid the small-file problem common in data lakes?
Apache Iceberg does not automatically prevent small files, but its table-management features support compaction and file rewriting to combine many small data files into larger ones. This reduces metadata overhead and improves query performance while preserving Iceberg’s transactional table semantics.
Do old snapshots and metadata files get deleted automatically in Apache Iceberg?
No, old snapshots and their associated metadata files are not automatically deleted because they are no longer current. Iceberg provides snapshot expiration and orphan-file cleanup procedures, which can be scheduled or triggered according to the retention policy an organization defines.
Can Apache Iceberg show what a table's schema looked like at a past point in time?
Yes. Apache Iceberg maintains schema versions in its table metadata, allowing queries against historical snapshots to reconstruct the schema associated with a table at a specific point in time. This makes it possible to examine how the table’s schema evolved alongside its data history.
