A Folder Is Not a Table
Data lakes spent a decade pretending a directory of Parquet files was a table. This chapter breaks one on purpose — three different answers to the same query during a single job — and defines what a table format adds.
Apache Iceberg from the Ground Up
The table format that made a folder of Parquet behave like a table — snapshots, schema and partition evolution, row-level deletes, and the maintenance nobody warns you about. Run-verified against Iceberg 1.11.0.
Download as an EPUB 21 chapters — read offline on any e-reader
-
-
Four Books and a Catalog
A working Iceberg table in about ten lines and one pip install — then the five things that quietly did not do what you meant, including the two rows that ended up sharing an id.
-
Four Layers and a Pointer
Open an Iceberg table and account for every file in it — why each commit adds exactly four, why the metadata is a tree rather than a list, and how a query answers from statistics without opening a single data file.
-
The Row That Owns the Warehouse
The whole of an Iceberg table's transactionality fits in one database row and one UPDATE statement. This chapter opens the catalog, watches the pointer move, races two writers into it, and hand-builds the filesystem catalog that Iceberg deliberately does not ship.
-
The Only Part of Iceberg You Can Ping
A catalog server is the one piece of Iceberg that listens on a port. This chapter stands one up, watches the compare-and-swap from chapter 4 become an HTTP 409, and moves a table between two catalogs without touching a single byte of data.
-
The Table Remembers
Every commit leaves a complete, readable copy of what the table was. This chapter reads yesterday's table, rolls one back, discovers that rollback deletes nothing, and finds out what PyIceberg's expire_snapshots does not do.
-
Field IDs, or How to Rename a Column Without Losing Your Data
Every column has a number, and the number is the column. This chapter renames, drops, re-adds and promotes columns on a live table, finds a table that lies about itself, and finds an off-by-one-word bug in PyIceberg's type-promotion rules.
-
The Partition Column That Isn't There
In Hive the query had to know the layout. Here the table knows it. This chapter runs the query that prunes ten files to one without naming a partition anywhere, then measures what happens when you pick the wrong transform.
-
Changing the Shape of a Table That Already Has Data
Repartitioning a Hive table meant rewriting it. Here it is a metadata edit that does not even record a snapshot — and this chapter measures exactly what that leaves behind, then finds a sort order that PyIceberg records and does not enforce.
-
Deleting a Row From an Immutable File
Parquet files cannot be edited, so Iceberg offers two ways to delete a row — rewrite the file, or write a note saying it is gone. Both measured, with the read cost each one moves around.
-
Twelve Thousand Rows, Fifteen Megabytes
A table that streams in a batch a minute is a table that quietly rots. This chapter wrecks one on purpose, measures the damage four different ways, and shows the query that tells you it is happening before your users do.
-
The Procedure That Reports Success and Does Nothing
rewrite_data_files is one call, and the first thing it did on my table was politely change nothing. This chapter pins the default that caused it, then measures bin-pack, sort and z-order compaction on the bookshop orders table.
-
Nothing Is Deleted Until Something Deletes It
Four maintenance procedures, two of which did nothing the first time I called them, and one of which will happily eat a file your table still needs. What each one actually reclaims, in what order, measured on the bookshop.
-
Adopting Files You Did Not Write
add_files, snapshot and migrate turn a directory of Parquet into an Iceberg table without rewriting it. Then expire_snapshots deleted the original Parquet, which is the part nobody warns you about.
-
Everybody Gets the Same Number
Four engines, one table, one catalog, no export step. PyIceberg, Polars, DuckDB and Spark all read the bookshop orders, all write to it, and all agree — right up to the point where they quietly disagree about what a timestamp means.
-
Nobody Sees It Until You Say So
MERGE INTO in one atomic snapshot, branches and tags that cost nothing, and write-audit-publish — plus the two Spark defaults that will delete your table or publish a batch you meant to stage.
-
The Firehose and the File Count
Streaming into an Iceberg table: 24 seconds of traffic produced 22 files and nine times more metadata than data. What Flink and Kafka Connect add, why streaming writers need equality deletes — and a clear line through the chapter marking what was executed and what was not.
-
Same Spec, Different Year
PyIceberg will create a format v3 table you cannot write a single row into. That is one of four version numbers that have to agree — spec, library, engine and catalog — and this chapter shows what each disagreement looks like when it fails.
-
Seven Additions and a One-Way Door
Format v3 went GA in 2026 and your tables did not notice. This chapter runs all seven of its additions against a real installation, shows which ones you can actually reach, and explains why turning the version up changes almost nothing by itself.
-
A Bitmap and a Serial Number
The two v3 additions that change a table's architecture rather than its type system: deletion vectors, which replace a Parquet file with 44 bytes of bitmap, and row lineage, which gives every row a stable identity and turns change data capture into a WHERE clause.
-
Fifty Thousand Orders, and Everything That Happens to Them
One table, one session: build it, evolve its schema and its layout, mutate it, wreck it with forty commits, then measure the repair — and get the same answer from three engines that never spoke to each other.