The Only Part That Can Say No
Two production catalogs on one object store, stood up, broken and restored. Credential vending works against a store that is not S3, one catalog refuses to register a table it does not own, and a table registered twice becomes two tables that share a past.
Take the current metadata.json of a table that lives in Polaris and register it, the same file at the same path, in a second catalog under a different name. Both catalogs now say the table exists. Both will accept a write.
rows before: polaris 1003, fixture twin 1003
INSERT one row through POLARIS: OK
INSERT one row through the FIXTURE twin: OK
rows after : polaris 1004, fixture twin 1004
polaris points at 00005-3bdd1cf7-d6a3-4b66-9289-f5c1d497a611.metadata.json
fixture points at 00005-81920c91-d773-4e67-9952-c9fd8215a228.metadata.json
same file: FalseTwo commits, two successes, two metadata files numbered 00005, both written under s3://warehouse/wild/stage2/orders/, each recording a history that shares everything up to 00004 and nothing after. Neither catalog knows the other exists. Run expire_snapshots through either and it will delete data files the other still references — and it will report success doing it.
A conditional swap protects a table only when its writers agree on which catalog owns the pointer. The reference server from chapter 5 of the first book does not enforce that ownership. Apache Polaris and Lakekeeper add controls around the same basic contract, and their differences affect deployment, recovery and what clients may do. One of those controls stopped the registration above when tried in the opposite direction. To understand that refusal, we need to see what each catalog adds to the pointer.
What a production catalog adds to the pointer
The contract has not changed. A catalog maps a table name to the location of its current metadata file and swaps that location atomically, and every engine on the platform trusts it to do exactly that. What the reference fixture lacks is everything around the contract. A production catalog adds four things.
Identity. The fixture accepts any request from anyone. Polaris and Lakekeeper both speak OAuth2: a client presents a credential, receives a token, and sends the token with every request. The catalog knows who is asking.
Authorisation. Knowing who is asking is only useful if the answer can be no. Polaris has catalog roles and principal roles with grants between them; Lakekeeper has projects, warehouses and a permission model that can be backed by an external authoriser. Both can refuse a table load — which the fixture cannot.
Credential vending. This is the one that changes the architecture. An engine that reads Iceberg needs credentials for storage as well as for the catalog. Historically it carried its own, so every engine on the platform held a key to the whole bucket. A vending catalog hands each client short-lived storage credentials scoped to the table it just loaded. The catalog’s decision about the table is then enforced at the storage layer too. Chapter 15 is about the three authorisation planes; this is the mechanism that lets one plane govern the other two.
A database, and therefore a backup. Both catalogs keep their rows in PostgreSQL. That is where their availability comes from, where their durability comes from, and — as the outage experiments show — where their failures come from too.
The landscape is wider than two. Project Nessie adds git-like branching of the catalog itself. Apache Gravitino federates several catalogs behind one. AWS Glue, Unity Catalog and Snowflake’s Open Catalog are vendor services that speak the same REST protocol with their own identity systems in front. None of those was run for this book, and the chapter says so here rather than pretending. The two that were run were chosen because they are open source, deployable from one compose file, and different enough from each other that the difference teaches something.
Standing them up
Both catalogs need a database and a bootstrap step, and the two bootstrap steps are a good first look at each project’s character.
Polaris bootstraps a realm into its database with a separate admin tool, run once before the server starts. The server is then told the same root credential so that it can serve the realm.
polaris-bootstrap:
image: apache/polaris-admin-tool:1.7.0
command: ["bootstrap", "--realm=POLARIS", "--credential=POLARIS,root,s3cr3t"]
polaris:
image: apache/polaris:1.7.0
environment:
POLARIS_PERSISTENCE_TYPE: relational-jdbc
QUARKUS_DATASOURCE_JDBC_URL: jdbc:postgresql://postgres:5432/polaris
POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3tRealm 'POLARIS' successfully bootstrapped.After that, everything is an HTTP call with a bearer token. Fetch a token with the root credential, create a catalog with a storage configuration, grant the catalog’s admin role the right to manage content.
token: HTTP 200
create catalog 'wild': HTTP 201
grant CATALOG_MANAGE_CONTENT on 'wild': HTTP 201Lakekeeper runs its own migrations, then serves. Its bootstrap is one request that accepts the terms of use, followed by one that creates a warehouse with a storage profile and the credentials the catalog itself will use to reach the store.
info: HTTP 200 {"version":"0.13.3", "bootstrapped":false, ...}
bootstrap: HTTP 204
create warehouse 'wild': HTTP 201 {"warehouse-id":"0a208018-…"}The companion does both from a script rather than from a curl container in the compose file, so that every call is visible. Resident memory afterwards: Polaris 480 MB, Lakekeeper 100 MB, PostgreSQL 50 MB. Polaris is a Java service on Quarkus, Lakekeeper is a Rust binary — and that is most of the difference.
Every chapter prints its pins. This one’s are Polaris 1.7.0, Lakekeeper 0.13.3, PostgreSQL 18.6, and SeaweedFS 4.45 as the store; clients as in chapter 1.
Credential vending, run against a store that is not S3
The point of vending is that the engine never holds a storage key. The test of vending is therefore a process with no storage credentials in it at all: no environment variables, no properties, nothing. Spark, started that way, with three catalogs configured.
== Spark 4.1.3 (host, local mode, NO ambient AWS credentials)
spark via lk: OK count=1000 path=s3://warehouse/lakekeeper/01a074af-…/data/00000-0-….parquet
spark via po: OK count=1000 path=s3://warehouse/wild/stage2/orders/data/00000-7-….parquet
spark via po_sts: OK count=1000 path=s3://warehouse/wild_sts/stage2/orders/data/00000-14-….parquetThree writes of a thousand rows, three successes — and the storage keys came from three different places. lk is Lakekeeper vending through the store’s STS endpoint, with the client sending X-Iceberg-Access-Delegation: vended-credentials. po is a Polaris catalog configured with stsUnavailable: true, which means Polaris cannot vend and the client brought static credentials as catalog properties. po_sts is a second Polaris catalog configured with a roleArn and an stsEndpoint pointing at SeaweedFS’s STS, and Polaris vended.
That last one is worth a sentence, because it is not something the documentation promises. Credential vending is described everywhere in terms of AWS IAM roles. What it actually needs is an STS endpoint that answers AssumeRole, and SeaweedFS’s IAM configuration provides one. The store the book runs on is not S3 and vending works against it, from both catalogs.
Then the clients that were not Spark.
pyiceberg via lk (vended): OK 1000
pyiceberg via po (oauth2 + static s3): FAILED BadRequestError: Credential vending was requested
for table stage2.orders, but no credentials are available
pyiceberg via po_sts (vended): OK 1000
duckdb attach po (oauth2) + count: FAILED HTTP 400 "Credential vending was requested for
table stage2.orders, but no credentials are available"PyIceberg and DuckDB ask for vended credentials by default. They send the delegation header without being told to — and a Polaris catalog that cannot vend answers the request with a 400 instead of ignoring it. Spark does not ask unless configured to, which is why Spark worked against the same catalog with static keys. The fix is per client and the vocabulary is different in each.
pyiceberg, header.X-Iceberg-Access-Delegation='none': OK 1000
duckdb ATTACH ... ACCESS_DELEGATION_MODE 'none': OK 1000
duckdb ATTACH ... ACCESS_DELEGATION_MODE 'remote_signing': FAILED Unrecognized access mode
'remote_signing'. Supported options are 'vended_credentials' and 'none'The client’s vending request has to match the catalog’s capability. A catalog that cannot vend needs clients configured not to ask; one that can vend can satisfy the delegation header. The mismatches above fail loudly — but a successful Spark test would have missed both of them. That is why chapter 5’s certification harness runs every client against every catalog before rollout.
Trino is the odd one out, and it took a stack trace to see how.
polaris_sts.stage2.orders (vended): FAILED Error processing metadata for table stage2.orders
lakekeeper.stage2.orders (vended): FAILED Failed to get status for file: s3://warehouse/lakekeeper/…/00001-….gz.metadata.jsonTrino accepts vended credentials and ignores the vended endpoint. With iceberg.rest-catalog.vended-credentials-enabled=true it takes the key, secret and session token the catalog hands it, and then tries to use them against whatever s3.endpoint its own properties say, which was nothing. Adding the store’s in-network address to the catalog properties fixed both, and Trino then read, inserted and counted through every catalog on the platform. That is a third behaviour for the same feature across four clients, and none of them is wrong. They are just different, and a platform team has to know which one each engine has.
The address problem, and the two ways to solve it
Chapter 2 ended on a question: what address should a client use to reach the bytes? On this platform the store is seaweedfs:8333 to a container and localhost:8333 to the host, and in production it is a private endpoint from inside the VPC and a public one from a laptop. The two catalogs answer the question differently, and the difference is a design decision each project made.
Polaris keeps two addresses. Its S3 storage configuration has an endpoint, which is what it hands to clients, and an endpointInternal, which is what the server itself uses to write metadata and to talk to STS. A catalog serving host-side clients and container-side servers just fills in both.
Lakekeeper keeps one. Its storage profile has a single endpoint, and that address is what it vends. A client that receives http://seaweedfs:8333 on the host cannot resolve it. The obvious workaround does not work:
pyiceberg vended + own s3.endpoint=localhost:8333: FAILED Could not resolve hostname
pyiceberg NO vending + static creds + own endpoint: FAILED Could not resolve hostname
duckdb vended, secret scoped to s3://warehouse/ with endpoint localhost: FAILED Could not resolve hostname
duckdb ACCESS_DELEGATION_MODE 'none' + static secret: OK 1000The table configuration a catalog returns beats the client’s own s3.endpoint, in PyIceberg in either mode and in DuckDB when vending. Only a DuckDB client that has switched vending off entirely uses its own address. So a Lakekeeper warehouse has exactly one endpoint — and it must be one every client can reach.
The platform solves that with a piece of plumbing that is also the chapter’s best operational lesson. A tiny socat container forwards its own 127.0.0.1:8333 to the store, and Lakekeeper runs inside that container’s network namespace. So http://localhost:8333 is a valid store address for the catalog and for every host client at once. The first version of that arrangement had it the other way round, with the catalog owning the namespace and the forwarder joining it, and it worked until the first time the catalog was restarted.
load_table after restore: FAILED ShortTermCredentialError: Failed to assume role via STS:
tcp connect error, 127.0.0.1:8333A container that joins another’s network namespace dies with it. Every catalog restart silently killed credential vending until someone restarted the forwarder too. Inverting the ownership — so that the forwarder owns the namespace and the catalog joins it — made a catalog restart free. The compose file says so in a comment long enough to be a warning. A reader with an /etc/hosts file and no interest in container namespaces can add one line instead. Either way, the lesson is that the storage address a catalog vends is a piece of infrastructure, and it needs an owner.
One more address to get wrong, and it is the catalog’s own. Lakekeeper has a LAKEKEEPER__BASE_URI setting, and whatever it holds is returned to every client in the REST configuration response as overrides.uri. The protocol says clients honour overrides — and they do.
Trino: SHOW SCHEMAS FROM lakekeeper
Caused by: HttpHostConnectException: Connect to http://localhost:8381 failed: Connection refusedTrino, a container, asked the catalog at lakekeeper:8181, was told the catalog’s real address was localhost:8381, believed it — and connected to itself. With the setting removed, Lakekeeper answers each caller with the address that caller used, and the platform has been fine since. Trino also caches that configuration at startup, so a catalog-side fix needs a Trino restart to take.
Latency, and what the client cache is worth
Every query begins with a table load, so its cost is the useful starting point for comparing these catalogs. Authentication and credential vending now sit on that path alongside the metadata lookup; the laptop SQLite catalog in Book 1 had neither to measure.
| Catalog | load_table p50 | p90 | Backing |
|---|---|---|---|
| Reference fixture | 4.9 ms | 5.9 ms | SQLite, no auth |
| Lakekeeper 0.13.3 | 4.2 ms | 7.3 ms | PostgreSQL, vending on |
| Polaris 1.7.0 | 17.1 ms | 19.8 ms | PostgreSQL, OAuth2 |
Thirty warm loads each from PyIceberg on the same host, so the numbers are the catalogs’ own overhead and nothing else. Lakekeeper is as fast as the fixture. Polaris costs about twelve milliseconds more per load, most of which is a Java service validating a bearer token and checking grants on every request, which is the price of the authorisation model. Neither number is a problem for a query that takes seconds. Both are a problem for a service that loads a thousand tables to plan a job, which chapter 9 comes back to.
Whether Spark’s client-side cache helps depends on what the query does with the table.
Spark, 20 x SELECT count(*) on Polaris (ms)
cache on p50 74.7 p90 120.9
cache-enabled=false p50 72.7 p90 87.7The cache made no measurable difference to this query. A count(*) through Polaris costs about seventy milliseconds either way; the catalog round trip accounts for seventeen, with planning and execution making up the rest. The cache that made chapter 1’s Spark session report a stale count buys nothing measurable on this query. For a session that runs many short queries against many tables the arithmetic changes, and chapter 9 does that arithmetic. For anything else the rule from chapter 1 stands, and this measurement is why it can afford to.
When the database goes away
The reliability table in chapter 1 said the catalog fails for every engine at once. Here is what that looks like in practice, with one twist — a production catalog’s database is a second thing that can fail, and it fails differently.
Stop PostgreSQL. The fixture, which does not use it, carries on. The two production catalogs do not, and the error texts are worth reading closely.
fixture load_table: OK
polaris load_table: FAILED ServerError: Failed to retrieve principal secrets for clientId: root
lakekeeper load_table: FAILED ServiceUnavailableError: error returned from database:
terminating connection due to administrator command
spark count via polaris (cached table): OK 1002
spark INSERT via polaris: FAILED ServiceUnavailableException: Unable to fetch principal entityPolaris did not fail at the table. It failed at authentication — the principals and their secrets live in the same database as the tables. With PostgreSQL down, nobody can even log in. Lakekeeper failed at the table lookup with the database’s own error passed through. And Spark’s cached table kept answering reads for as long as its cache entry lived, while the first commit failed. A commit needs the catalog; a cached read does not. That is the cache from chapter 1 again, now on the right side of the ledger: during a short catalog outage, sessions that already hold a table keep reading.
Start PostgreSQL again and both catalogs recover on their own — no restart needed. The pending insert went through on the next attempt.
Now stop Polaris itself, with the database healthy.
spark count via polaris, table cached from before: OK 1003
spark count via polaris, cache-enabled=false: FAILED RESTException: Error occurred while processing GET request
pyiceberg load_table via polaris: FAILED ConnectionError: Connection refused
the same table through LAKEKEEPER is unaffected: OK 1000Storage untouched, every byte intact, and no client can resolve a name through Polaris — exactly as the reliability table predicted. Lakekeeper, on the same store and the same database, is unaffected. That is the argument for the compute layer being able to reach tables through more than one route in a disaster, and the reason chapter 16 exists.
Then the twist. Start Polaris again — and the session that was reading happily before the outage cannot read at all.
spark count via polaris after restart, SAME session: FAILED NotAuthorizedException: Not authorized
pyiceberg via polaris after restart, NEW session: OKThe Spark session’s bearer token was issued by the Polaris process that no longer exists. The new process does not recognise it. Spark’s REST client did not fetch a new one on the 401 — it reported the error and kept the dead token. A fresh client that authenticated after the restart was fine. The operational consequence is blunt: a catalog restart invalidates every session that authenticated before it, unless the catalog’s signing key is persisted across restarts, and it is not by default here. A runbook for a catalog restart has to end with “and then restart the engines”, for this reason.
Backup and restore
Both catalogs keep their state in PostgreSQL, so a catalog backup is a database backup. The thing to prove is that a restored catalog points at the same metadata files as before, without any data file having been touched.
pg_dump rc=0
before: 00001-01a074b1-a0d1-7961-861f-d95a69f9ac31.gz.metadata.json
drop_table(purge=False) in Lakekeeper: OK
load_table after drop: FAILED NoSuchTableError: Error getting tabular from catalog
restore: rc=0
load_table after restore: OK s3://warehouse/lakekeeper-local/…/metadata/00001-01a074b1-….gz.metadata.json
same metadata file as before the drop: Truepg_dump of the catalog’s database. A drop_table with purge=False, so that only the catalog’s row goes and the files stay. A restore over a freshly created database. The table loads again at exactly the metadata file it had. Nothing on the object store was read or written by the restore. That is the whole point of separating the pointer from the bytes. It is why chapter 16’s disaster-recovery plan treats the catalog database as the thing to back up on a schedule and the object store as the thing to replicate.
Two details from the run. The restore needed the catalog stopped, because PostgreSQL will not drop a database with connections open, and a catalog holds a pool of them. And purge=False is the difference between a recoverable mistake and an unrecoverable one: Lakekeeper’s drop_table with the default arguments would have deleted the files too, and no database restore brings those back.
Moving a table between catalogs
register_table creates a catalog entry from a table name and the path of an existing metadata.json. It leaves the files where they are, which makes it useful for moving a table between catalogs — and dangerous if both entries remain writable. The opening experiment first tried registering the fixture’s table into Polaris.
polaris.register_table(stage2.orders_from_fixture, <file under s3://warehouse/stage5/>):
FAILED ForbiddenException: Invalid locations '[s3://warehouse/stage5/orders/metadata/00009-….metadata.json]'
for identifier 'stage2.orders_from_fixture': s3://warehouse/stage5/ordersPolaris refused. Every Polaris catalog has a set of allowed locations, and it will not register, create or point at a table outside them. The fixture’s table lives under s3://warehouse/stage5/, the Polaris catalog owns s3://warehouse/wild/ — and the answer is no. That is authorisation applied to the storage layout, and it is the only reason the opening experiment had to be run the other way round.
So the other way round: Polaris’s table registered into the fixture, which enforces nothing. Both catalogs pointed at the same 00004 file, both accepted a one-row insert, and afterwards they pointed at two different 00005 files under the same table location. Each catalog believes it owns the table. Each will run maintenance on it. Whichever runs expire_snapshots first will delete files the other still references. The other will discover that the next time a query reads a column, not when it counts rows, which chapter 2 explained.
The rule has two halves. A table lives in one catalog. Moving it is register-then-drop, in that order and without a gap in which both exist, and the drop is purge=False. And when the target catalog enforces locations, the table’s files must already be under a root it allows, which means the move is chapter 2’s rewrite_table_path first and the registration second. Polaris’s refusal is the safety mechanism. The fixture’s acceptance is the bug it prevents.
Two smaller differences that matter for the same reason. Lakekeeper lays tables out under a UUID rather than the table name, s3://warehouse/lakekeeper/01a074af-…/, so a rename touches nothing on storage and a bucket listing tells a human nothing. Polaris lays them out by name. Lakekeeper also writes gzip-compressed metadata files, 00001-….gz.metadata.json, which every engine on the platform read without comment and which a script that pattern-matches on .metadata.json will miss.
What to decide, and why
The catalog’s database is the platform’s single point of failure, one level down. Its availability is the catalog’s availability, and its backup is the catalog’s backup. Run it the way you would run the database behind anything that matters, with replication and a tested restore. Treat “the catalog is up but its database is not” as a full outage — because for Polaris it is one at the login prompt.
Choose vending, and then choose clients that match. Static storage keys in every engine are the state this chapter started from, and a vending catalog is the way out. The exception is a client that cannot be told to stop asking against a catalog that cannot vend. On this platform there was none, and chapter 5’s harness is where you find out whether there is one on yours.
Give the vended storage address an owner. If the catalog vends one address, make it one every client can reach. Treat the forwarding or DNS that makes that true as part of the catalog’s deployment, with the restart order written down.
Restart order is: catalog, then engines. A restarted catalog invalidates tokens; engines holding them do not refresh on their own.
Which of the two to run is a judgment and is labelled as one. Polaris costs more per request and more memory, and buys a richer role model and the allowed-locations enforcement that stopped this chapter’s foot-gun. Lakekeeper is as fast as the fixture, lighter, and vends through a single endpoint that has to be reachable from everywhere. A platform with many teams and an audit requirement leans to the first; one with a small number of engines inside one network leans to the second. Both ran everything in this book.
What was not run
Every outage here was one container stopped and started. The catalogs were not tested behind a load balancer, on a replicated database, or under concurrent commits from more than one engine. Chapters 8 and 11 do that, and it is where the reference fixture failed. Token expiry inside a long-running Spark session was observed once, on a restart, and not timed. And Lakekeeper’s vending through the port forwarder was run on one table at a time.
Exercises
1. Watch the token die. Open a PyIceberg session against Polaris, load a table, restart the Polaris container, and load the table again from the same session. Then create a second session and load it. Which call fails, with what exception, and what does that tell you about where a session should keep its credential rather than its token?
Show answer
The first session’s second load fails with NotAuthorizedException because its bearer token was issued by the previous Polaris process. The second session succeeds because it authenticates fresh. A client should hold the credential and re-run the token exchange on a 401. PyIceberg 0.11.1 and Spark’s REST client do not do that on their own, so the operational fix is to restart engines after the catalog.
2. Register, then count the references. Register a Lakekeeper table’s metadata file into the fixture under a new name, write one row through each, and then run remove_orphan_files with dry_run => true and prefix_listing => true through the fixture. What does it list, and what would running it for real do to the Lakekeeper table?
Show answer
It lists the data file the Lakekeeper-side write added after the split, because the fixture’s history does not include that commit and the file sits under the shared location. Running it for real deletes that file, and the Lakekeeper table’s next column read fails with NotFoundException. Two catalogs on one location is a maintenance job away from data loss.
Final thoughts
Book 1 called the catalog the only part of Iceberg you can ping. This chapter’s version is that it is the only part that can say no. It says no to a client without a token, to a table outside its root, to a storage key that should never have been in an engine’s hands. Everything a production catalog adds to the pointer is a way of refusing something — and the platform is safer for each refusal.
It is also the component whose outage takes every engine down at once, whose database is the failure behind the failure, and whose restart quietly logs everyone out. The next chapter stays on the commit path and asks what happens when two writers reach the catalog at the same time and it has to say no to one of them.
Next: Found Conflicting Files
Comments