Congratulations, It's Yours Now
The capstone of the first book ended with three engines agreeing to the cent. Then a second writer showed up. Nothing about the format changed; everything about the job did.
Apache Iceberg in the Wild
Iceberg once it is shared infrastructure — object storage, production catalogs, conflicting writers, multi-engine correctness, compaction as policy, streaming tables, incidents, recovery and the bill. Every chapter opens with something broken.
Download as an EPUB 19 chapters — read offline on any e-reader
-
-
There Is No ls in Object Storage
Iceberg never lists a bucket, and the reason is a bill. A counting proxy in front of the object store shows what each operation really costs, what a copy of a table actually copies, and which store lied about a conditional write.
-
The Only Part That Can Say No
Two production catalogs on one object store, stood up, broken and restored. Credential vending works against a store that is not S3, one catalog refuses to register a table it does not own, and a table registered twice becomes two tables that share a past.
-
Found Conflicting Files
Two writers reach the catalog half a second apart. Which one is told no depends on what each was doing, on a table property, and on which client library made the call. Every class, run, with the exception text.
-
Compatible Is a Test Result
Five engines, two format versions, one catalog. Every engine writes, every engine reads every other's table, Spark deletes a row from each, and every cell of the matrix is generated rather than typed. Then the same rows are read in three time zones.
-
There Are No Tuning Numbers
Three bookshop workloads, each built two ways, measured under a protocol that survives the JVM. The layout that won the point lookup lost the full scan, and the one that fixed the scan lost the lookup again.
-
Forty-Two Percent in One Partition
A day-partitioned orders table diagnosed from its own metadata, asked the questions it was not built for, and redesigned two ways. Bucketing fixed the customer queries and taxed every day query; z-ordering did nothing at all, for a reason worth knowing.
-
Missing Required Files to Delete
Compaction as a policy rather than a procedure: what the planner will pick, what each strategy costs in bytes written, what a live writer does to a rewrite and a rewrite to a live writer, why v2's delete files outlive the data they point at, and a score that ranks a fleet from metadata.
-
Six Hundred Copies of the Snapshot List
Six hundred commits, counted at the store: the manifests merged themselves, the metadata file grew to 558 KB and left 171 MB of copies behind, rewrite_manifests bought nothing, retention deleted ninety of six hundred, and the statistics Spark wrote were read by Trino and ignored by Spark.
-
Eight Survivors of Forty-Four
Retention designed as policy and then proven: an expiry that kept two snapshots on main left a quarter-close tag and a five-deep branch untouched, a branch removed itself after a minute, a tag doubled a table's storage after one compaction, and an incremental reader that fell behind the window lost its place.
-
Seventy-Three Thousand Duplicates
A streaming writer that never stops, measured: one snapshot and one manifest per checkpoint with no merging, maintenance running alongside it, a killed TaskManager recovered to the exact row, a savepoint resumed twice and the duplicates that made, a column dropped under a running job, late data landing where nobody compacts, and an upsert stream's delete files removed by the right rewrite.
-
Sixty Thousand Deletes, Nothing Deleted
Change data from PostgreSQL through Debezium, Kafka and Flink into Iceberg, converged in seconds and proven by reconciliation; a replay that converges instead of duplicating; the landing table that cannot serve its own changes; and a merge-maintained v3 table whose row lineage gives consumers an exact delta.
-
The Rule Was Right, the List Was Wrong
A table-health pack of eighteen metrics derived from metadata, run against forty-three tables in forty-four seconds, and seven alert rules scored against what the lab had actually done to each table. Three of the pack's own queries were wrong before any rule was, and the rule that inferred a streaming table from its cadence fired on six batch tables.
-
The Count Was Right and the File Was Gone
Eight lakehouse incidents produced on the lab on purpose and worked to recovery: a stray metadata file, a data file removed by a lifecycle rule, an accidental overwrite, a dropped column, a stale catalog pointer, a corrupted metadata file, an expired recovery point, and the orphan cleanup that the procedure refuses to let you get wrong.
-
No Grant, One Key, Every File
The three authorisation planes of a lakehouse, tested against each other: a read-only catalog principal refused at commit after the files were already written, a static storage key that read every table with no catalog involved, vended credentials that held exactly to their table, an engine whose fixed catalog identity turned every user into root, and a view that was no policy at all.
-
Copying the Bucket Is Not Copying the Table
Disaster recovery on the lab: a namespace dropped from the catalog and re-registered from its own metadata in under half a second with history intact, a region lost and recovered on a second object store under a fresh catalog, a replica under a renamed bucket that registers and then cannot find its first manifest, and a copy that took data first and metadata last under a live writer, leaving four files of eight missing.
-
What the Service Takes Off Your Hands, and What It Takes Away
Managed Iceberg, held against the seventeen chapters before it: which maintenance jobs each of five services runs for you, which of the book's decisions each one takes away, where the catalog is the lock-in and the format is not, and an operating-model matrix for deciding when to accept the trade. Documentation-verified, dated, and labelled as such throughout.
-
The Bill Is a Table Too
The cost model of the bookshop's lakehouse, populated from the lab's own request and byte counts: what a commit costs, what a streaming writer costs a day, what compaction buys back, what the metadata file costs the catalog per load, and what the unreferenced files cost until they are cleaned. Then the operating model that owns each line, and the production review that puts the whole book on one page.
-
Appendix A: Version Four Is a Number, Not a Format
Appendix A. Iceberg 1.11.0's jar declares format version 4 supported, and Spark will create a v4 table today, but the v4 specification is unreleased and the table's metadata carries exactly v3's fields. What the Summit 2026 roadmap set out, what has been voted into the spec, what is still contested, which engines read the table Spark wrote, and why no production table should carry the number yet.