Congratulations, It's Yours Now
The capstone of the first book ended with three engines agreeing to the cent. Then a second writer showed up. Nothing about the format changed; everything about the job did.
Here is the first incident. It happened in the lab this book is written against, about forty seconds after the platform came up, and nobody did anything wrong.
Spark created a table through a REST catalog on object storage and loaded fifty thousand orders into it. PyIceberg, running in a separate process with no JVM, appended one row through the same catalog. DuckDB, in a third process, inserted another. Then each engine was asked how many rows the table held.
pyiceberg count: 50002
duckdb count : 50002
spark count : 50000Spark was not wrong about anything it had seen. It had loaded the table before the other two engines committed, and its REST client caches table metadata. cache-enabled defaults to true, and the session simply kept answering from the snapshot it already had. One REFRESH TABLE later, Spark said 50,002 like everyone else.
With one writer, this stale count would have been a curiosity: you controlled when the table changed. With several writers, agreement depends on the system around the table. The catalog every engine talks to, the storage every file lands in, the caches each client keeps and the maintenance schedule all matter. Iceberg’s format has not changed, but a correct snapshot is no longer enough to make the platform reliable. Somebody has to answer for how these pieces work together.
What you were running before, and what you are running now
Apache Iceberg from the Ground Up ended with a capstone: fifty thousand orders loaded, evolved, deliberately wrecked, compacted, expired, and read back by three engines that agreed to the cent. It ran on a laptop. The catalog was the Apache reference REST fixture with an in-memory SQLite behind it. The warehouse was a directory bind-mounted into that container. Every write came from one process at a time, and every process was started by the person reading the book.
That is a lakehouse in the same sense that a database on your laptop is a database. Every mechanism is real. None of the pressures are. Nobody else’s job depends on the table existing at nine o’clock. No two writers race for the same commit. No bucket lifecycle rule quietly deletes a data file the metadata still references. Nothing runs at a scale where a request count is a line item on a bill.
A platform is what you get when those pressures arrive, and the difference is easiest to see as a shift in what the word “broken” means.
Table correctness is what the first book was about. A commit is atomic or it is not. A snapshot is consistent or it is not. Two engines read the same metadata and get the same answer — or one of them has a bug. These are properties you can check by reading metadata, and the first book spent twenty-one chapters checking them.
Platform reliability is different. A table can be perfectly correct and still be useless. The catalog that resolves its name is down. The credentials that reach its files expired. The one engine that produces the nightly numbers cannot see the snapshot the other engine just committed. None of those is a defect in Iceberg. All of them are your problem now.
The reference architecture this book uses treats the platform as seven layers with contracts between them. They are separable, which is the whole point — you should be able to swap any one without rewriting the others.
| Layer | What it owns | What “down” means for everyone else |
|---|---|---|
| Storage | the bytes: data files, manifests, metadata files | nothing works, including reads of tables nobody is writing |
| Catalog | the pointer from a name to the current metadata file, and the atomic swap of that pointer | no engine can find or commit to any table, even though every byte is intact |
| Compute | the engines that plan scans, write files and commit | that engine’s users are down; every other engine is fine |
| Governance | who may read and write what, and how credentials reach the engines | reads fail closed or, worse, fail open |
| Orchestration | when writes and maintenance run, and in what order | tables go stale, maintenance stops, and nobody notices until the bill or the planning time says so |
| Observability | the metadata tables, turned into signals someone watches | incidents are discovered by users instead of by you |
| Recovery | catalog backups, metadata-consistent replicas, the practiced way back | the first real loss is permanent |
The failures in the third column have different reach. Losing the catalog affects every engine at once; losing storage also stops reads of tables nobody is writing. Compute is the only layer whose failure stays local — which is why platform teams are so relaxed about engines and so nervous about catalogs. The catalog’s single point of failure, introduced in chapter 4 of the first book, now needs an end-to-end test. Standing up the platform and breaking that dependency once gives us one.
Standing it up
Everything in this book runs from one compose file in the companion repository. Book 1’s convention holds: the environment grows only when a chapter needs it, and each growth is that chapter’s lesson. But the growth happened, so the platform is bigger than the first book’s. Here is what comes up, with the memory each service was measured holding after the chapter’s runs.
git clone https://github.com/book-companion/iceberg-bookshop
cd iceberg-bookshop && make venv
make -C book-2 upwaiting for catalog ready
waiting for polaris ready
waiting for lakekeeper ready
waiting for trino.. ready
waiting for flink ready
waiting for connect ready
book2-catalog Up 8 seconds (healthy)
book2-connect Up 13 seconds
book2-flink-jm Up 7 seconds
book2-flink-tm Up 7 seconds
book2-kafka Up 13 seconds
book2-lakekeeper Up 8 seconds (healthy)
book2-lakekeeper-s3-loopback Up 8 seconds
book2-polaris Up 8 seconds (healthy)
book2-postgres Up 13 seconds (healthy)
book2-seaweedfs Up 13 seconds (healthy)
book2-trino Up 2 seconds (health: starting)| Service | Role | Resident memory |
|---|---|---|
| SeaweedFS 4.45 | S3-compatible object store, single process, with IAM and STS | 190 MB |
| REST fixture 1.10.1 | the reference catalog from Book 1, now on object storage | 240 MB |
| PostgreSQL 18 | the database behind both production catalogs, and the bookshop’s operational source | 50 MB |
| Apache Polaris 1.7.0 | production catalog one | 480 MB |
| Lakekeeper 0.13.3 | production catalog two | 100 MB |
| Trino 483 | the query engine Book 1 never ran | 1.0 to 1.7 GB |
| Flink 2.1.3 | the streaming writer Book 1 could only describe | 2.6 GB across two containers |
| Kafka 4.3.1 and Debezium 3.6.2 | event streams and change data capture | 1.6 GB together |
The whole platform resident at once measured about 8.2 GB. The planning number before it was measured was 17 GB. Write that down as the first lesson of the operating model. Capacity estimates written from memory are wrong by a factor of two, in either direction — and the only cure is docker stats.
What each of these is, and why it is here
The first book ran on three components: PyIceberg, Spark and a reference catalog. This platform has eleven containers and three host-side clients, and half of them will be new to a reader who has only that book behind them. Here is what each one is, sorted into the layers of the reliability table, so that the rest of the book can name them without stopping to explain.
Storage: SeaweedFS. An object store that speaks the S3 API, running as one process. It plays the part S3, GCS or Azure Blob play in production: every data file, manifest and metadata file lands here, and nothing else on the platform holds a byte of table data. It is SeaweedFS rather than MinIO because MinIO stopped publishing Docker images in September 2025 and archived its repository the following April. Every Iceberg tutorial you have read uses MinIO — this book could not. Chapter 2 shows how the replacement was chosen, with a probe that also caught a store returning success for a write condition it did not honour. The store is a dependency like any other, and this one changed under the book while it was being written.
Catalogs, three of them. A catalog holds one row per table, the pointer to that table’s current metadata file, and swaps the pointer atomically. That is the whole contract, and the first book spent chapters 4 and 5 on it. The platform runs three implementations of the same REST protocol on purpose. The reference REST fixture is Book 1’s container, an Apache-provided test server with SQLite behind it, moved onto object storage. That way the first thing you run in this book is the last thing you ran in the previous one. It is also here so that it can fail in front of you — which it will, below. Apache Polaris is an Apache project: a REST catalog with OAuth2 authentication, catalog roles and the ability to hand engines short-lived storage credentials. PostgreSQL is its database. Lakekeeper is an independent REST catalog written in Rust, with the same protocol, the same credential vending and the same PostgreSQL behind it, organised around projects and warehouses. Two production catalogs rather than one because chapter 3 is about how they differ, and they differ in ways that will cost you a day each if you meet them unprepared.
The database: PostgreSQL 18. It has two jobs, and keeping them straight matters. It is the metadata store for both production catalogs, so a catalog’s “one row per table” is literally a row in this database. It is also the bookshop’s operational system — the sixty thousand orders of the SQL and dbt books. Chapter 12 captures changes from it. The same instance, two databases, two very different reasons to care whether it is up.
Engines, five of them. An engine is anything that reads or writes Iceberg tables through a catalog. Spark is the first book’s writer and the home of every maintenance procedure; it runs on the host, not in a container. PyIceberg is the JVM-free microscope, the library that reads metadata instantly and is used throughout for inspection. DuckDB is an in-process analytical database that attaches to a REST catalog directly, and it is the third engine in every cross-engine check. Trino is a distributed SQL query engine: a coordinator that plans queries and workers that execute them, with a native Iceberg connector. It is the engine large platforms put in front of their analysts. The first book could not fit it, and chapter 5 certifies it. Flink is a stream processor: it runs jobs that never finish, checkpoints their state on an interval, and commits to Iceberg once per checkpoint. It is the writer Book 1’s chapter 17 could only describe, and chapter 11 runs it.
Streams: Kafka, Kafka Connect and Debezium. Kafka is the message broker, a durable log of events organised into topics — one node here, in its KRaft mode with no ZooKeeper. Kafka Connect is Kafka’s framework for running connectors, processes that move data into or out of topics without custom code. Debezium is a family of change-data-capture connectors that run inside Connect. They read a database’s transaction log and publish every insert, update and delete as an event. Together they are how the bookshop’s PostgreSQL changes reach an Iceberg table in chapter 12, and they are the reason the platform’s memory table has a Kafka line at all.
That is the cast. The reliability table said which layer’s failure hurts whom; this list says which process is which layer. When chapter 3 says “the catalog is down”, it means one of three containers, and when chapter 5 says “an engine cannot see the snapshot”, it means one of five.
Every chapter prints its pins. Here are this one’s, as the companion’s verify script prints them.
== version manifest
trino 483
flink 2.1.3
lakekeeper 0.13.3
kafka connect 4.3.0
postgres 18.6 (Debian 18.6-1.pgdg13+2)
pyspark 4.1.3
pyiceberg 0.11.1
duckdb 1.5.5The Iceberg library is 1.11.0 everywhere it is a library: the Spark runtime, the Flink runtime, PyIceberg’s spec support, and the copy Trino 483 bundles. That last one was checked by listing the plugin directory rather than by assumption, because an engine’s Iceberg version is whatever the engine ships, not whatever you installed.
org.apache.iceberg_iceberg-api-1.11.0.jar
org.apache.iceberg_iceberg-aws-1.11.0.jar
org.apache.iceberg_iceberg-core-1.11.0.jarProving it end to end
The companion’s first verify script reproduces the opening incident deliberately. Spark writes, two more engines commit, and four engines count. A successful Spark write alone would not test the platform’s promise. The proof is that every engine reaches the same number through the same catalog and storage, including the Spark session that loaded the table before the other commits.
== spark wrote 50,000 rows into 9 files under s3://warehouse/ in 3.3s
== pyiceberg appended 1 row through the catalog
== duckdb inserted 1 row through the catalog
== spark saw 50000 before REFRESH TABLE (its REST client caches table metadata)
== counts: {'spark': 50002, 'pyiceberg': 50002, 'duckdb': 50002, 'trino': 50002}
== PASS: four engines, one catalog, one table, 50,002 rows, 3 snapshotsNine data files for fifty thousand rows, because the table is partitioned by day and the orders span nine days. Three snapshots, because three engines committed once each. Those two numbers are the first book’s arithmetic and they still hold. What is new is the fourth engine and the cache.
Trino is worth a paragraph because it is the engine the first book explicitly did not run. Chapter 15 of that book said why: the lab’s Docker allocation could not fit a coordinator that wants a multi-gigabyte heap next to everything else. That reason is gone, and this platform gives Trino its own catalog properties file per way of reaching the tables. The one used here is the plainest.
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://catalog:8181
iceberg.rest-catalog.warehouse=s3://warehouse/
fs.native-s3.enabled=true
s3.endpoint=http://seaweedfs:8333
s3.path-style-access=true
s3.region=us-east-1
s3.aws-access-key=lab-access-key
s3.aws-secret-key=lab-secret-keyThe storage endpoint is seaweedfs:8333 here and localhost:8333 in every host-side client. Trino runs in a container, so it reaches the same store through a different address. A catalog that supplies storage configuration to both kinds of client has to account for that difference. Chapters 2 and 3 follow the problem through the two production catalogs; it also explains the small sidecar next to Lakekeeper in the compose file.
Now the cache, properly. Spark’s Iceberg REST catalog wraps every loaded table in a caching layer, and a session that loaded a table keeps serving that table’s metadata until the entry expires or something refreshes it. The relevant properties on the catalog are cache-enabled and cache.expiration-interval-ms, and the operational choice is genuinely a choice.
- Leave the cache on, and a long-lived session — the kind a notebook server or a query service holds — will report stale results for tables other engines write to. The staleness is bounded by the expiration interval, which defaults to thirty seconds, and unbounded if that interval is set to a negative number, which disables expiry.
- Turn it off with
cache-enabled=false, and every table reference costs a catalog round trip. On the fixture that is a millisecond. On a production catalog under load it is a real cost, and chapter 3 measures it.
The rule this book uses has two halves. Sessions that only read and live for seconds keep the default. Anything long-lived that shares tables with another writer either disables the cache or refreshes explicitly before a decision is made on the numbers. The exception is a session that also owns the table’s only writer — which cannot be stale about its own commits.
Breaking it, once
The reference catalog survived the four-engine proof. It did not survive the next thing this book tried, which was a streaming writer committing every ten seconds while a batch job ran alongside it. Here is what every client saw from that moment on — Spark, PyIceberg, DuckDB and Trino alike.
org.apache.iceberg.exceptions.RESTException: Unhandled error:
ErrorResponse(code=500, type=UncheckedSQLException,
message=Failed to get table stage4.orders_upsert from catalog rest_backend)And here is the cause, from the catalog’s own log.
Caused by: org.sqlite.SQLiteException: [SQLITE_ERROR]
SQL error or missing database (no such table: iceberg_tables)Book 1 ran the fixture with the JDBC URI it ships with: jdbc:sqlite:file:/tmp/iceberg_rest?mode=memory. That URI describes an in-memory SQLite database — and in-memory SQLite databases are per connection. As long as the catalog’s connection pool handed every request the same connection, there was one database and it had the iceberg_tables table in it. The moment concurrent commits made the pool open a second connection, that connection got a brand-new, empty, in-memory database with no tables at all, and answered every query with the error above. It never recovered. A restart cleared it, and a restart of an in-memory database also forgets every table ever registered.
Three lessons from the first incident, in the order you will need them.
First, the catalog failed for every engine at once — which is what the reliability table said would happen. Storage was fine. Every data file, every manifest, every metadata file was exactly where the metadata said it was. Nobody could reach any of them, because nobody could resolve a name to a pointer.
Second, the failure was silent until it was total. There was no warning that the pool had grown and no log line at the moment the second connection opened. From the client side there was no way to tell which of the two databases a request would land on. Chapter 13 is about turning “the catalog is answering” into a signal — and this is the incident that justifies it.
Third, the fix was a configuration line, and the configuration was Book 1’s. The platform now runs the fixture on a file-backed SQLite, jdbc:sqlite:file:/catalog/fixture.db, on a bind mount. Not a named Docker volume — the image runs as a non-root user and could not open a database file on a root-owned volume (SQLITE_CANTOPEN). That is the kind of detail that costs an hour on a Tuesday. File-backed SQLite handles concurrent connections the way SQLite always has, with a lock. It is still SQLite behind a Jetty server. Chapter 3 replaces it with Polaris and Lakekeeper, both on PostgreSQL, and measures what that buys.
Book 1’s chapter 5 introduced this fixture as a test server. The concurrent workload has now exposed why that distinction matters.
What to decide, and why
A platform team’s first artifact is not a dashboard or a runbook. It is a sentence per layer saying who owns it — and the sentences are harder to write than they look. Here is the bookshop’s, filled in for the platform you just stood up. Yours will have different names in the right-hand columns and the same shape.
| Layer | Owner | Change control | What they promise |
|---|---|---|---|
| Storage | platform | bucket policy and lifecycle rules reviewed; no lifecycle rule touches a warehouse prefix | durability; no deletion outside Iceberg’s own maintenance |
| Catalog | platform | schema migrations of the catalog database are the platform’s release, not the engine teams’ | name resolution and atomic commit, with a stated availability |
| Compute | each engine’s team | engine upgrades certified against the platform’s tables (chapter 5) before rollout | their users’ queries |
| Governance | security, executed by platform | policy changes reviewed by both | the same answer through every engine |
| Orchestration | data engineering | maintenance is scheduled by the platform, pipelines by their owners | freshness per table |
| Observability | platform | signals derived from metadata tables, not from engine logs | incidents found before users find them |
| Recovery | platform | restores are practiced, not documented | a stated recovery point and time |
Two ownership decisions in this table follow directly from the failures above. The catalog and storage belong to the same team because a usable table depends on both. Split that responsibility and you can end up with each team’s dashboard green while no engine can read the table.
Maintenance is scheduled by the platform — not by whoever wrote the pipeline. The first book showed that a compaction can conflict with a concurrent delete and that expiry deletes files a rollback would have needed. Those are platform decisions with table-owner consequences, and Part II of this book is about making them as policy rather than as favours.
The rest is judgment, and this book will say so when it is. Whether engine teams can run their own maintenance, whether governance sits in the catalog or in front of it, what availability the catalog promises: each of those has a chapter. Each chapter runs what can be run and labels what it cannot.
The reliability model, in one paragraph
Every later incident in this book is described against the same model, so here it is once. A table is available when a named engine can resolve it through the catalog and read its current snapshot from storage. A table is fresh when its current snapshot is younger than its owner’s stated freshness. A table is healthy when its metadata says nothing is wrong: no delete-file ratio climbing, no snapshot count growing without expiry, no planning time that has stopped being proportional to the data. Availability is the catalog’s and storage’s promise. Freshness is orchestration’s. Health is maintenance’s, and it is the one that erodes silently — which is why chapter 13 turns it into numbers.
What was not run
The Kafka Connect Iceberg sink runs, creates its table, and has never committed; that is open in the lab’s record and chapter 11 says so again. Every memory figure above is resident memory at one instant on an idle platform, not under load. And the reference catalog’s behaviour under concurrent commits from more than one engine was not tested here; chapters 8 and 11 found its limit, and the later chapters moved to Polaris because of it.
Exercises
Every Iceberg state change is visible in metadata, so these check themselves.
1. Find the cache. Open two Spark sessions against the platform. In the first, load platform.orders and count it. In the second, insert one row. Count again in the first session, without REFRESH TABLE, and keep counting once every ten seconds until the number changes. How long did it take, and which catalog property explains the number you got?
Show answer
The first session reports the old count until its cache entry expires, and with the default cache.expiration-interval-ms of 30000 that is up to thirty seconds. Setting the property to -1 on the catalog disables expiry and the first session never sees the row without an explicit refresh. Setting cache-enabled=false makes every reference a catalog round trip and the row appears on the next count.
2. Make the fixture fail on purpose, then not. Edit the companion’s compose file to put the fixture back on jdbc:sqlite:file:/tmp/iceberg_rest?mode=memory, restart it, and run the streaming job from chapter 11’s SQL script alongside the verify script. When it fails, read the catalog’s log for the SQLite error, then look at the metadata files on storage. Are any of them missing?
Show answer
None. Every data file, manifest and metadata file the commits wrote is present on the object store; the catalog lost only its pointer table, which lived in a per-connection in-memory database. That is the whole difference between a storage failure and a catalog failure, and it is why chapter 16’s recovery procedure for a lost catalog starts by registering the newest metadata.json it can find.
Final thoughts
The first book taught a mechanism per chapter, and every mechanism still holds. Snapshots are still immutable, the catalog is still one conditional swap, a manifest is still a list of files with statistics. What this chapter added is the sentence that goes in front of all of them now. The platform is the thing that lets more than one process rely on those mechanisms at the same time. It is nobody’s until somebody says it is theirs.
You have stood the platform up and watched its weakest component fail in the way the reliability table predicted. The next chapter goes one layer down, to the storage every file landed in. It asks a question the first book never had to — how many requests did that cost, and which of them would you pay for?
Comments