Copying the Bucket Is Not Copying the Table

A table being written to every few hundred milliseconds, and a bucket replication that copied its data files first and its metadata files last, with two commits landing in between.

  data first, metadata last:   copied 4 + 13 objects with 2 commits in between
     replica's newest metadata: 4 snapshots, references 8 data files, 4 MISSING;  scan -> FAILED: Path does not exist
  metadata first, data last:   copied 13 + 12 objects with 2 commits in between
     replica's newest metadata: 4 snapshots, references 8 data files, 0 MISSING;  scan -> 4,000 rows

Same table, same writer, same replica, and nearly the same number of objects copied: seventeen and twenty-five, the difference being the two commits’ worth of files that the second half of each copy did or did not include. One order produced a replica whose newest metadata names four data files the replica does not have, and a scan that fails on the first of them. The other produced a replica that is a consistent, slightly older table, which is what a replica is supposed to be. Bucket replication copies objects. A table is a metadata file that names objects, and the naming is what has to be consistent — which no object copy knows about.

A readable replica is the starting point for the two recovery drills below: losing the catalog and losing the region. Each was produced in the lab, recovered and timed. Each recovery also validates history: rows, snapshot counts, and the tags and branches from chapter 10, before and after. A recovered table that has the rows and not the tags has lost the audit holds — and a file count would not have said so.

The catalog is gone

Every table in a namespace dropped from the catalog with its files untouched, which is what a catalog database loss looks like from the tables’ side.

  8 tables; before:  audit 12,000 rows, 4 snaps, refs [main, month_end, week_end];  dev 12,000, 10, [feature, main];
                     held 20,000, 3, [hold, main];  orders 40,000, 8, [dev, main, q2_close]; …
  load a dropped table: FAILED -> does not exist: stage13.audit
  re-registered 8 tables from the newest metadata.json under each prefix in 0.4s (0.05s per table)
  history validated: 8/8 identical (rows, snapshot count, refs, schema)

The recovery is a listing and a loop: for each table prefix, find the metadata files, register the newest. Fifty milliseconds a table, and every tag, branch and snapshot came back, because all of it lives in the metadata file and none of it lives in the catalog. The catalog holds one pointer per table, and everything else is on the store. A catalog backup is therefore a backup of a list of pointers. The list can be rebuilt from the store in the time it takes to list it — which is why this chapter does not back the catalog up at all. It backs up the ability to enumerate table prefixes, which is a namespace listing, and it keeps the store.

To check that the recovered tables retained their history, the drill needed more than a successful read. Before the loss, each table was reduced to a record: row count, snapshot count, the sorted list of references with their snapshot ids, the number of schema fields, and the metadata file’s name. After the recovery the same record was taken again and compared as a whole. The orders table came back with its q2_close tag on the same snapshot id and its dev branch on the same head. The audit table came back with both tags and their lifetimes. The development table came back with a branch ten snapshots deep. None of that is visible in a file count, and all of it would have been lost by a recovery that registered a metadata file from before the tags were made.

The trap is in the word “newest”.

  stage17.t5: metadata files [00000, 00001, 00002, 00003-02053a, 00003-f2b549, 00004, 00005];  pointer 00005;  newest by name 00005 -> same
  stage17.t1: metadata files [00000, 00001, 00002, 00003, 00004, 00005-3e14e0, 00006-3e14e0];  pointer 00005;  newest by name 00006 -> DIFFERENT

Chapter 14’s forked table has two files at version three and the newest by name is still the right one. Chapter 14’s stray — a metadata file a failed commit left behind — sorts after the pointer, and a recovery that registers the newest by name registers the stray. In this run the stray was a copy and the state would have been the same. In a real failed commit it is the state that was refused, which may be the overwrite chapter 14 rolled back. The newest metadata file on the store is not the table’s state; the catalog’s pointer was. A recovery that has lost the pointer needs a second source for it. The previous catalog’s own log is one. The metadata file’s metadata-log field is another. Every file lists its predecessors, so a file that no other file lists as a predecessor is either the true head or a stray. Two such files under one prefix means one of them is a stray. The third is a policy that a failed commit’s file is cleaned up before the next recovery could see it, which is chapter 9’s orphan cleanup on a schedule shorter than the recovery drill’s.

The region is gone

A second object store standing in for the replica region, the namespace’s objects copied to it, and a fresh catalog on the replica side built from the copy.

  replicated 570 objects, 5.6 MB, in 2.8s (data first, metadata last)
  DR catalog: 8 tables registered and read from the replica in 0.4s;  history identical to the primary's pre-loss state: 8/8

The replica catalog is a PyIceberg SQL catalog with its storage endpoint pointed at the second store. It was populated the same way as the first recovery: list the prefixes, register the newest metadata file. Its entire state after the recovery is eight rows in a SQLite file, one per table, which is what a catalog’s state is once the metadata files are somewhere safe. Eight tables, every tag and branch, in less than half a second, and every row readable from the replica. The DR catalog is not a replica of the primary catalog — it is a new catalog built from the replica’s metadata files. That is the design that makes the recovery time a function of the number of tables and nothing else. It is only possible because the copy above happened to be consistent, which is what the rest of the chapter is about.

This successful copy used data first, metadata last — the order that left four files missing in the opening. Here nothing was writing to the namespace during the copy. That quiet interval explains the different result; it does not establish that the order is safe under a live writer.

The bucket has a different name

The same replica, into a bucket called warehouse-dr, with nothing under the original name on the replica side.

  objects under s3://warehouse/stage13/audit/ on the replica: 0;  under s3://warehouse-dr/stage13/audit/: 44
  register from s3://warehouse-dr/…/00015-….metadata.json:   OK
  scan:   FAILED -> does not exist 'warehouse/stage13/audit/metadata/snap-8001950488593599572-1-….avro'
  the metadata file's own paths:  s3://warehouse/stage13/audit/metadata/sn…

The registration succeeds, because the catalog reads the one file it was pointed at. The first scan fails on the first manifest list, because every path inside the metadata file is absolute and names the original bucket. Chapter 2 found this on a plain copy, and the repair is the same. rewrite_table_path, run on the healthy source before the disaster, produces a staged copy of the metadata with the paths rewritten. A replica in a differently named bucket is not readable until its metadata is rewritten, and the rewrite has to be part of the replication, not part of the recovery. The recovery may not have a healthy source to run it against. The first run of this test passed by accident: the replica’s own warehouse bucket still held the same objects, and the absolute paths resolved against them. A test that passes for the wrong reason is worse than one that fails — and it took a second run with the original prefix emptied to see it.

Bucket replication is not table replication

The opening failure follows from what a metadata file contains: paths to data files and manifests. A copy that moves the data files first and the metadata files last, with writes continuing, copies a metadata file that names data files committed after the data copy finished. The replica’s newest state points at objects it does not have, and the newest state is what a recovery will register.

The other order is safe for the same reason. A metadata file copied first names only files that existed when it was copied, and a data copy that finishes later is a superset of those. The replica’s newest metadata is older than the primary’s by the copy’s duration — and it is consistent. Copy metadata first and data last, and the replica is a consistent table as of the metadata copy. Copy the other way and it is a set of objects.

Three things follow. Object-store replication services copy in whatever order they like. A replica built by one has to be validated the way the probe validated it: register the newest metadata, list the files it references, check each exists. Every uncommitted file on the primary, from chapter 14’s failed commits and chapter 11’s restored checkpoints, is replicated too. It is as much an orphan on the replica as on the primary. And the replica’s consistency is per table: the check is a loop over tables, and one table failing it does not mean the others did.

RPO and RTO, with the numbers

The recovery point is the age of the replica’s newest consistent metadata file. With metadata copied first, that is the copy interval plus the copy duration. With the other order, it is undefined until the validation finds the newest metadata whose files all exist — which may be several commits back. On the lab the copy took 2.8 seconds for 570 objects; on a platform the interval dominates.

The recovery time is three parts, two of them measured. Discovery, which is a listing of table prefixes. Registration, at fifty milliseconds a table, which is four minutes for five thousand tables on one client and can be parallelised. And the engines, which have to be pointed at the replica’s catalog and the replica’s endpoint. Chapter 11’s Polaris override showed that the endpoint is one configuration in the catalog and not one per engine. The catalog’s own state is not on the recovery path at all.

What is not measurable on this lab, and is labelled as judgment, is whether the replica region runs a warm catalog or a cold one. A warm catalog registers continuously as the replication lands, so that failover is a DNS change. A cold one is built at recovery time from the listing. The warm one has a recovery time of seconds and a second catalog to keep consistent. The cold one has a recovery time of the registration loop and nothing to keep consistent. The bookshop, with a few hundred tables, takes the cold one.

The game-day script

The game-day sequence records the expected state before the loss, then checks the recovered tables against it. The final step deliberately uses the wrong copy order to test whether the validation catches an incomplete replica.

1. record   for each table: rows, snapshot count, refs, schema, metadata location   (the pre-loss state)
2. lose     drop the namespace from the catalog / point the engines at the replica endpoint
3. discover list <warehouse>/<ns>/<table>/metadata/*.metadata.json per prefix
4. choose   the newest file whose predecessors match the metadata-log chain; never simply the newest by name
5. register register_table(ns.table, that file) in the recovery catalog
6. validate for each table: every referenced file exists; rows, snapshots, refs, schema equal step 1
7. time     steps 3–6 per table; that is the RTO
8. drill    repeat with the replica copied in the wrong order, and confirm step 6 catches it

Step 6 is the one that distinguishes a drill from a demo. Without the reference comparison, eight tables could return the right rows while losing their tags — and the drill would report success despite losing the audit holds.

What to decide

Do not back up the catalog — back up the ability to rebuild it. One pointer per table, rebuilt from the store in fifty milliseconds each.

Replicate metadata first and data last, or validate every replica by referenced-file existence before trusting it. Never register the newest metadata file by name alone.

Rewrite paths as part of replication when the bucket name changes. The recovery has no healthy source to run the rewrite against.

Validate history, not counts. Rows, snapshots, refs, schema, per table, against a record taken before the loss.

Run the wrong-order drill. The replication that passed was the one that was not being written to.

What was not run

A replication service’s own ordering, rather than a copy loop whose order the probe chose. A warm replica catalog kept in step with replication. A recovery under a writer that keeps writing to the primary during failover — which is the split-brain that chapter 14’s forked history is the small version of. And the credential-vending dependency on the replica side: the replica catalog here used static keys, and a vending catalog needs its token service replicated too, which chapter 15 flagged.

Exercises

1. Validate a replica you did not build. Take any replicated bucket and, for one table, register its newest metadata file in a scratch catalog and list the files it references against the bucket.

Show answer

If any referenced file is missing, the replication copied metadata after data under writes, and the next-newest metadata file whose files all exist is the recovery point. The number of metadata files between that one and the newest is how many commits the replica is behind in the worst case.

2. Find the stray before the drill does. For every table, compare the newest metadata file by name with the catalog’s pointer.

Show answer

A mismatch is a failed commit’s file, and a recovery by newest-by-name would register it. The orphan cleanup removes them after a day; a recovery drill run inside that day will find them, which is the argument for the drill’s step 4 reading the metadata-log chain rather than the listing.

Final thoughts

The catalog turned out to be the easy part. One pointer per table, rebuilt from the store in the time it takes to list it, with every tag and branch intact because none of them ever lived in the catalog. The store turned out to be the hard part — and not because copying it is difficult. Copying it is trivial, and a trivial copy in the wrong order under a live writer produced a replica that registered cleanly and could not read its own newest state. The recovery point of a lakehouse is the age of its newest consistent metadata file, and consistency is a property of the copy order that no object store enforces.

A managed service changes who operates this infrastructure, but you still need to know what a recovery preserves. The next chapter compares the jobs those services take on with the decisions and operations they leave available.

Next: What the Service Takes Off Your Hands, and What It Takes Away