Eight Survivors of Forty-Four

Forty-four snapshots on one table, and an expiry told to keep two.

  expire_snapshots(older_than => 2030-01-01, retain_last => 2):
     deleted 0 data files, 36 manifest lists
  survivors: 8 snapshots
     q2_close   TAG      1 snapshot    retained 3650 days
     dev        BRANCH   5 snapshots   retention: 5 snapshots
     main       BRANCH   2 snapshots   retain_last => 2

Eight survived, and every one of them survived because something in the table’s own metadata said it should. The quarter-close tag held its snapshot. The development branch held the five its retention names. Main kept the two the call asked for. The other thirty-six are gone, with their manifest lists, and the metadata file is a fifth of its size. Then the three reads that matter: the tag answers with 10,000 rows, the branch with 34,000, main with 40,000 — all after the expiry.

The references from chapter 16 of the first book give a platform separate ways to retain recovery history, audit states and work in progress. Table properties then let a scheduled expiry apply those decisions without carrying them in its arguments. The eight survivors are the proof: each required state still reads correctly after the unwanted history is gone.

Three questions, three references

Retention on a lakehouse is not one number. It is three questions with different owners.

How far back can main be rolled? That is recovery, and it belongs to the platform team. The answer is main’s own retention: a minimum count and a maximum age, from which chapter 14’s rollback draws.

Which states must stay addressable, and for how long? That is reproducibility and audit, and it belongs to whoever signs the numbers. The answer is a tag: a named, immovable reference with its own lifetime, that expiry is not allowed to touch.

Which work in progress needs its own history? That is development and write-audit-publish, and it belongs to the team doing the work. The answer is a branch with its own retention and its own age, so that it keeps what it needs and disappears when it is done.

The defaults from the jar do not distinguish those owners’ needs. Main gets five days and a one-snapshot minimum, while branches and tags can live indefinitely — a combination worth changing deliberately.

Property	Default	What it decides
`history.expire.max-snapshot-age-ms`	5 days	how old a snapshot may be before a no-argument expiry removes it
`history.expire.min-snapshots-to-keep`	1	how many main keeps regardless of age
`history.expire.max-ref-age-ms`	unbounded	how long a branch or tag lives
tag `RETAIN n DAYS`	unbounded	the tag’s own lifetime
branch `WITH SNAPSHOT RETENTION n SNAPSHOTS`	main’s rule	the branch’s own count
branch `RETAIN n DAYS`	unbounded	the branch’s own lifetime

Five days and one snapshot is a recovery window that closes over a long weekend — and references that never expire are holds that never release. Every table on the platform gets these three answers set explicitly, and the rest of this chapter is what happens when they are.

The policy is stored in the table

Before changing retention, refs shows which named states the table must preserve. Reading it again after expiry checks whether those references survived with the same settings.

  name       type     snapshot              max_ref_age_ms   min_snapshots_to_keep
  main       BRANCH   7968585649391623250   —                —
  q2_close   TAG       457693268547859144   315,360,000,000  —
  dev        BRANCH   4061115003485435561   —                5

The tag’s RETAIN 3650 DAYS is stored as milliseconds; the branch’s count is stored as min_snapshots_to_keep on the ref, not on the table. After the expiry that opened this chapter the same three rows were still there, unchanged. What changed was the snapshots table, which went from forty-four rows to eight. It is worth one warning that Book 1 raised and this chapter confirms: snapshots spans every branch. Its count is not main’s history — it is the union of every reachable snapshot from every reference. A snapshot count on a dashboard means nothing until it is split by ref, and chapter 13 splits it.

The opening expiry deleted thirty-six manifest lists but zero data files, despite removing most of the snapshots. Every one of the forty commits was an append, so main’s head still references every data file any of them wrote. Expiring the history behind an append-only table frees metadata — and nothing else. Data files are only freed when the expired history contains a snapshot that removed them, which means a compaction, a delete, or an overwrite. That is why the cost of a hold, below, had to be measured through a compaction, and why chapter 9’s storage numbers were all under metadata/.

The branch’s five survivors are also worth counting. The branch was cut from commit 30 and had four commits of its own, so its ancestry is five deep, and five is what its retention keeps. Had it been cut from main’s head instead, its retained ancestry would have overlapped main’s and the two references would have held some of the same snapshots. That is exactly what happened in the development class at the end of this chapter.

References expire themselves

A branch created with a lifetime of one minute, and an expiry run twice.

  ALTER TABLE orders CREATE BRANCH short_lived RETAIN 1 MINUTES
  refs: short_lived (BRANCH, max_ref_age_ms 60,000)
  expire immediately:   refs now [main, short_lived, q2_close, dev]
  expire after 70s:     refs now [q2_close, dev, main]

The reference outlived the first expiry because it was younger than its own limit, and the second removed it, along with any snapshot only it had been holding. A reference’s age is policy the reference carries. A write-audit-publish branch created with RETAIN 1 DAYS cleans itself up when the job that made it is forgotten. An audit tag created with ten years does not depend on anyone remembering not to delete it. The one thing to check is that the platform’s scheduled expiry runs at all — because a reference’s age is only enforced when something enforces it.

What a hold costs

A tag holds a snapshot, and a snapshot holds every data file it references. Two identical twenty-file tables, one tagged at its head, both compacted and then expired down to one snapshot.

  held   with a tag:  data/ before compaction 20 files 172 KB   after compaction + expiry 40 files 344 KB   (expiry deleted 0)
  free   no tag:      data/ before compaction 20 files 172 KB   after compaction + expiry 20 files 172 KB   (expiry deleted 20)

The tagged table doubled. Compaction wrote a new copy of every file; expiry could delete none of the old ones, because the tag’s snapshot still names them. The untagged table’s expiry deleted all twenty. A tag costs the full size of the table as it was at the tag, for as long as the tag lives. It costs it again after every rewrite of those files. That is a fair price for a quarter close on an orders table. It is not a fair price for a daily tag on a table that chapter 8 compacts nightly, where each tag pins a layout the next night’s compaction replaces. Tags go on audit boundaries. Daily reproducibility is main’s retention — not a tag a day.

Reproducibility and recovery, after the cleanup

The retained tag still gives a name to its snapshot after expiry. These reads and rollback attempts check both sides of that boundary: the states kept by a reference and the history already removed.

  VERSION AS OF 'hold'                                   OK -> 20,000 rows
  TIMESTAMP AS OF '2026-01-01' (before any survivor)      FAILED: Cannot find a snapshot older than 2026-01-01T00:00:00+00:00
  one more commit on main, then
  rollback_to_snapshot(the tag's snapshot)               OK: previous 2577380811136104542, current 4116973491680761675
  main after rollback: 20,000 rows; refs [main, hold]
  rollback_to_snapshot(an expired id)                    FAILED: ValidationException: Cannot roll back to unknown snapshot id: 1234567890

A tag is addressable by name for its lifetime, and rolling main back to it works even after later commits. The commit that was rolled past becomes an unreferenced snapshot for the next expiry. The procedure’s return row names both: the snapshot main pointed at before, and the one it points at now. That row is the audit trail of the rollback, and chapter 14 records it. A timestamp before the oldest survivor fails, and the message is clear about why. An expired snapshot id is unknown to the table — not an old state, not a state at all. The recovery window is exactly main’s retention. A rollback in chapter 14 can reach as far as min-snapshots-to-keep and max-snapshot-age-ms allow, and not one commit further. That is the argument for setting them from the recovery requirement rather than from storage cost.

Downstream readers keep a position you can expire

An incremental consumer, of the kind chapter 12 builds, holds a snapshot id as its position. The table’s retention does not know that.

  incremental read from a surviving start snapshot     OK -> 2,000 rows
  incremental read from an expired start snapshot      FAILED: Starting snapshot (exclusive) 7362979822115305788
                                                       is not a parent ancestor of end snapshot 5013190851639854671

A changelog view built on the same expired start is created without complaint. It fails with the same message the first time it is read, so the check is at read time — not at definition. A reader that falls further behind than the retention window does not get its delta late. It gets an error, because its start snapshot is no longer an ancestor of anything, and its only recovery is a full re-read. Main’s retention has to exceed the longest consumer lag the platform will tolerate, plus the time it takes to notice a consumer has stalled. A consumer that reads hourly against a table retaining five days has four days of slack — the same consumer against a table on the aggressive policy above has none. Chapter 12 registers every incremental consumer against its source’s retention for exactly this reason.

The scheduled job takes no arguments

Chapter 13 of Book 1 found that expire_snapshots(table) with no arguments does nothing, and blamed the five-day default. That is the mechanism a platform wants, once the properties are set per table.

  no args, defaults (5 days):                                   0 deleted; 10 snapshots remain
  no args, max-snapshot-age-ms=1 + min-snapshots-to-keep=3:     7 deleted;  3 snapshots remain
  + branch keep_more (5 snapshots), 6 commits on it, no args:   8 snapshots remain: 3 on main, 5 on the branch

The same call, with nothing but a table name, produced three different results, because the policy is in the table and the refs, not in the call. That is the operating model: one scheduled job per table, expire_snapshots(table => …), no arguments, and every retention decision expressed as a property or a reference. A job that carries older_than and retain_last in its own arguments has moved the policy out of the table and into a scheduler. The next person to read the table’s metadata will not find it there.

The three classes, expired together

Three tables, three policies, one no-argument expiry each.

Class	Properties and references	Before	After	What survived and why
hot	`max-snapshot-age-ms` short, `min-snapshots-to-keep` 2	12	2	main’s two most recent; the recovery window
audit	hot’s properties, plus tags `month_end` (2555 days) and `week_end` (30 days)	12	4	main’s two, plus one per tag; both tags still read their rows
dev	hot’s properties, plus branch `feature RETAIN 7 DAYS WITH SNAPSHOT RETENTION 10 SNAPSHOTS`	18	10	the branch’s ten, which include main’s two; the branch removes itself in a week

  audit tags after expiry:  month_end 6,000 rows   week_end 10,000 rows   main 12,000 rows
  dev after expiry:         main 12,000 rows       feature 18,000 rows

These three classes provide a retention template for the bookshop. Every table is in a class, a class is a property set, and anything a class does not cover is a tag or a branch with its own lifetime. The bookshop’s orders is audit-class with a tag per month end. Its events stream is hot-class, because chapter 9 showed what six hundred retained snapshots do to the metadata file. Its customers dimension is dev-class while the write-audit-publish job that maintains it is running, and hot-class between runs.

The two audit tags are two different instruments. The seven-year tag is a records requirement, and it will outlive the table’s owners. The thirty-day tag is a hold with an end date: a week-end state kept long enough for a reconciliation to be re-run, and released by the expiry job on the thirty-first day without anyone deciding to release it. Legal holds on a lakehouse are tags with lifetimes — and the lifetime is the release. A branch is the wrong instrument for either, because a branch is a place to write, and the audit state must not be writable.

The development class carries Book 1’s two write-audit-publish footguns with it, both run there and not re-run here. While spark.wap.branch is set, reads in that session go to the branch too, so a validation query in the same session validates the branch and not main. And without write.wap.enabled on the table, both WAP settings write straight to main with no error. The dev class therefore sets write.wap.enabled in its property set, so that the branch is the only place a WAP session can land.

What to decide

Set the three properties on every table at creation, from the recovery requirement for main and the consumer lag for anything read incrementally. The defaults are a five-day window and a one-snapshot floor.

Tags for audit boundaries, priced. A tag pins a full copy of the table through every later compaction. Month ends on orders — not days.

Branches with both a count and an age. The count is what the work needs; the age is when the branch stops needing anyone to remember it.

One no-argument expiry per table, scheduled, and after it the orphan cleanup that chapter 9 found is not optional. A reference’s lifetime is only enforced when the job runs.

Read refs before changing anything and after expiring anything. It is the policy as stored — and the only place it is stored.

What was not run

Snapshot age with real time. Every expiry here used a far-future older_than or a one-millisecond property, so the five-day default’s behaviour on a real calendar is Book 1’s finding, not re-measured. A branch’s own max-snapshot-age-ms, which the syntax allows, was not exercised; only its count and its ref age were. The cost of a tag on a table large enough for the doubling to matter is arithmetic from the small case. And a rollback’s orphaned snapshot was left for the next expiry rather than expired and counted.

Exercises

1. Find the unowned references. Run SELECT name, type, max_reference_age_in_ms FROM t.refs on every table in a namespace and list the references with a null age that are not main. Each one is a hold with no expiry and, unless it is an audit tag someone signed for, a leak.

Show answer

Every branch a write-audit-publish job created without RETAIN and every tag created “just in case” shows up with a null age. The fix is ALTER TABLE … REPLACE BRANCH name RETAIN n DAYS for the ones that should live a while and DROP BRANCH for the ones nobody can name an owner for. The storage they were pinning returns at the next expiry.

2. Measure your recovery window. For a hot-class table, record the oldest snapshot’s timestamp from snapshots each hour for a day. The gap between it and now, at its smallest, is the furthest back a rollback could have reached at the worst moment.

Show answer

With min-snapshots-to-keep at two on a table committing every five minutes, the window at its narrowest is ten minutes: the two newest snapshots. If the recovery requirement is “any commit from the last day”, the count has to be the commit rate times a day, or the age has to be a day. The age is the one to set, because the count changes with the commit rate.

Final thoughts

Retention looked like one number, a snapshot age. It turned out to be three references with three owners and three lifetimes, stored in the table itself and enforced by a job that takes no arguments. The proof was the cheap part. The tag survived and read its rows. The branch kept its five and removed itself on schedule. The stall-prone reader failed loudly instead of silently. And the cost of a hold turned out to be the whole table — twice.

The next chapter is the table that never stops being written to. A streaming writer commits on every checkpoint and uses the fast append that skips chapter 9’s manifest merge. It produces delete files for real. And it needs every maintenance job in this part of the book scheduled against a writer that will not pause for them.

Next: Seventy-Three Thousand Duplicates