Missing Required Files to Delete

A rewrite of thirty partitions, and a delete on one of them, started half a second apart.

  delete       committed at 1.5s
  compaction   FAILED at 2.0s: ValidationException: Missing required files to delete:
               s3://warehouse/stage11/c1/data/ordered_at_day=2026-06-15/00000-4145-….parquet,
               s3://warehouse/stage11/c1/data/ordered_at_day=2026-06-15/00000-3995-….parquet,
               … three more
  snapshots +1, 146 data files, rows 5,900,000

The delete rewrote five files on the sale day and committed. The compaction had already rewritten the same five, along with a hundred and forty-five others, and when it went to commit, the five it meant to replace were gone. Iceberg refused the whole thing. Thirty groups of finished work — twenty-nine of them untouched by the delete — were discarded together and their output files cleaned up. The table is left with the delete applied and the fragmentation intact.

Book 1’s chapter 12 warned that compaction commits can conflict. The discarded work here shows why a platform needs more than that warning. Running the procedure thousands of times means deciding which tables and partitions to rewrite, which strategy to use, how much work to commit together, which writers can overlap, and what to verify afterwards. Every one of those was measured. Two of the measurements differ from what the documentation would lead you to expect.

The defaults, read from the jar

The policy is written in the options, so start with what they are when you write nothing. These are the constants in iceberg-spark-runtime-4.1_2.13-1.11.0.jar, read with javap -constants — not from a page.

Option	Default	What it decides
target-file-size-bytes	the table’s write.target-file-size-bytes, 512 MB if unset	the output size
min-file-size-bytes / max-file-size-bytes	0.75 × and 1.8 × the target	which files are candidates
min-input-files	5	the smallest group worth a commit
rewrite-all	false	ignore the candidate rules and rewrite everything
partial-progress.enabled	false	one commit, or several
partial-progress.max-commits	10	how many, if several
max-concurrent-file-group-rewrites	5	groups rewritten in parallel
max-file-group-size-bytes	100 GB	the largest single group
use-starting-sequence-number	true	what sequence number rewritten files carry
delete-file-threshold	Integer.MAX_VALUE	rewrite a file with this many delete files, whatever its size
delete-ratio-threshold	0.3	rewrite a file with this fraction of rows deleted, whatever its size
remove-dangling-deletes	false	drop delete files that no longer apply

Two of those matter more than their names suggest. use-starting-sequence-number decides whether v2 delete files survive a rewrite, which is the second half of this chapter. And delete-ratio-threshold is in this release’s jar with a default of 0.3. A file with a third of its rows deleted is a compaction candidate on its own, with no option set.

The table a stream leaves behind

The workload is chapter 11’s, arrived early: a writer appending ten thousand orders at a time, one day partition per batch, five batches a day for a month.

  stream: 150 files, 8.9 MB, min 58 KB, median 58 KB, 150 snapshots, ingested in 31s
  one day                  GETs   15   0.2 MB   114 ms
  one retail customer      GETs  450   2.4 MB   430 ms

A hundred and fifty files of fifty-eight kilobytes, and a hundred and fifty snapshots. The customer query makes four hundred and fifty requests to read two and a half megabytes. Three requests per file is the reader’s fixed price: a footer, a column index, a column chunk.

The first policy question is what the planner would do to this if run with no options, and the answer is predictable from metadata before the job starts. A candidate is a file under 75% of the target, grouped by partition, in a group of at least five. That is a query.

SELECT partition, count(*) AS candidates
FROM   ice.stage11.stream.files
WHERE  file_size_in_bytes < 0.75 * 536870912
GROUP  BY partition HAVING count(*) >= 5;

  metadata says: 30 partitions have >= 5 files under 75% of target -> 150 candidate files
  binpack, no options: 150 -> 30 files, 8.5 MB written (30 PUTs), 1.0s

The prediction matched all 150 candidate files — enough to decide whether this table needed a job before starting one. A zero result would have meant no compaction to schedule. The next check is whether the rewrite helped the queries, measured with interleaved, pre-warmed runs against the uncompacted copy.

  customer query   150 files: median 430 ms   30 files: median 193 ms
  day query        150 files: median 114 ms   30 files: median  86 ms
  customer query   before: GETs 450, 2.4 MB   after: GETs 100, 7.6 MB

The latency halved, the requests fell by four and a half times — and the bytes tripled. The small files were so small that the reader fetched only a footer and one column chunk from each. The compacted files hold a whole day in one row group. The customer predicate cannot skip inside it, so the reader fetches the two columns it needs in full. On a store that prices bytes and requests separately, chapter 2’s table decides whether that is a win. On this one it was, by latency and by requests, and it would not have been by bytes alone.

Candidate selection is the policy

The same fragmented table can justify a narrow rewrite or no job at all, depending on which partitions and file counts the policy selects. These three runs vary that selection and the target size.

  where => one day:    rewrote 5 -> 1 files, 0.3 MB, in 0.5s; table now 146 files
  min-input-files=50:  rewrote 0 -> 0 files (no partition has 50 files: nothing to do)
  target 4 MB:         rewrote 145 -> 29 files in 1.1s; median file 275 KB

A where on the partition column scopes the rewrite to the partitions that are still being written, which is what a nightly job should do. History does not re-fragment, and Book 1 said so. min-input-files is the sensitivity dial, and the no-op it produces is silent — the return row is all zeros and the job exits green. The target size decides the output. On a table whose partitions hold a few hundred kilobytes, a 512 MB target means “one file per partition” whatever it is set to. The 4 MB run produced the same twenty-nine files, with the threshold lowered to two.

Compaction policy is a where, a threshold, and a schedule. The where follows the writer. The threshold is the prediction query above, its result compared against a number you chose. The schedule is how often the prediction is re-run.

Three strategies, priced

Identical copies of the fragmented table, compacted three ways.

Strategy	Time	Bytes written	Customer query	Day query
binpack	1.0 s	8.5 MB	100 requests, 7.6 MB, 144 ms	3 requests, 62 ms
sort by customer_id	1.3 s	9.9 MB	100 requests, 8.9 MB, 149 ms	3 requests, 60 ms
z-order (customer_id, ordered_at)	1.8 s	11.4 MB	100 requests, 10.3 MB, 149 ms	3 requests, 65 ms

The sorted and z-ordered copies cost more to write and are larger on disk — the opposite of chapter 6’s halving. The difference is what the rows looked like before: the ingest wrote them in order-id order, which is timestamp order, and Parquet’s encoders had already made the most of that. Sorting by customer destroyed it. Whether a sort shrinks a table depends on what order it was in, and the only way to know is to write it both ways.

And the query columns are identical across all three, for the reason chapter 7 found. With one file per partition and one row group per file, there is nothing for an order to help a reader skip. Sorting is a strategy for partitions big enough to hold several files or several row groups. On a table of small partitions it is a more expensive binpack.

Binpack unless a predicate has been proven, on this table, to benefit. The proof is the request and byte counts — not the strategy’s name.

Partial progress, and what it is for

  partial-progress off:               150 -> 30 files, 1 commit,  1.0s
  partial-progress on, max-commits 5: 150 -> 30 files, 5 commits, 1.0s
  the last five snapshots: replace(+6/-30) x 5

The same work in the same second, committed as five snapshots of six files each instead of one of thirty. The cost is four extra snapshots, which chapter 10’s expiry will remove. The benefit is the opening of this chapter — run again.

Writers that never stop

A writer appending two thousand rows every three hundred milliseconds, and compaction started in the middle of it.

  compaction: 153 -> 30 files in 1.0s while the writer landed 8 appends (0 writer errors)
  table after: 35 files, 1,516,000 rows = 1,500,000 + 8 x 2,000 -> consistent

Appends and rewrites do not conflict. The rewrite replaces files the appends never touch, the appends add files the rewrite never saw, and both commit. The five files the writer added during the rewrite are the next run’s candidates. Chapter 11’s streaming table lives on this fact.

Deletes are different, because a copy-on-write delete replaces files and so does compaction. The opening of this chapter was the race on the reference catalog, where the delete committed first. The same race was run on Polaris four times, with the delete issued half a second into the rewrite each time.

  run 1, plain rewrite:      compaction committed 150 -> 30 at 2.4s;  delete FAILED at 3.3s
  run 2, plain rewrite:      delete committed at 1.6s;  compaction FAILED at 2.1s;  146 files
  run 1, partial progress:   delete FAILED at 1.2s;  compaction committed 150 -> 30 in 10 commits
  run 2, partial progress:   delete committed at 1.6s;  compaction committed 135 -> 27,
                             failed groups 0;  38 files
  delete at 1.3s, plain:     compaction committed at 1.9s;  delete FAILED at 2.0s:
                             ValidationException: Missing required files to delete: …/ordered_at_day=2026-06-15/…
  delete after the commit:   both committed; snapshots: replace(+30/-150), overwrite(+1/-1)

Whichever side commits second loses, with the same exception, and which side that is comes down to milliseconds. The same catalog, the same delay — and the rewrite won one run and lost the next. When the delete loses it fails on the same Missing required files to delete, naming the sale-day files the rewrite replaced under it. There is no arbitration and no priority. A plain rewrite that loses discards every group, and the message the procedure raises says so in its own words:

  ValidationException or CommitFailedException. This usually means that this rewrite has
  conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts,
  set partial-progress.enabled which will break up the rewrite into multiple smaller commits

A partial-progress rewrite that loses keeps every group committed before the collision. In the second run it kept twenty-seven of thirty and lost the batch of three that held the sale day. And it reported those three as zero failed groups. The return row said 135 files rewritten, 27 added, failed_data_files_count 0; the table had thirty-eight files where a clean run leaves thirty. The counter is for groups that fail to execute — not for commits that are refused. A scheduled job that trusts it will believe the table is compacted when a partition is not.

The writer that loses sees a ValidationException and does not retry it. The commit retry loop is for a CommitFailedException — a concurrent change to the table’s pointer — not for a validation that found the files gone.

The policy that follows: compaction and row-level writers on the same partitions need a schedule that keeps them apart, or a writer that expects to retry from a fresh scan. Partial progress bounds the compaction’s loss; nothing bounds the writer’s except its own retry.

The catalog that could not take it

The partial-progress race was run first on the reference REST fixture, before Polaris, and the fixture is why the numbers above are Polaris’s.

  fixture, partial progress, delete at 0.5s:
     delete       FAILED at 1.3s: ServiceFailureException: Service failed: 500: Unknown failure
     compaction   returned 150 -> 30 ... 105 -> 21, failed groups 0, at 4.9s
     snapshots +7; table now 66 data files
  catalog log:  SQLiteException: [SQLITE_BUSY] The database file is locked
  driver log:   CommitStateUnknownException ... cannot clean up files because they may
                have been committed successfully  (three times, nine groups)

Ten commits in a second from the rewrite plus one from the delete were more than the fixture’s SQLite would take. It returned 500s. The delete failed outright. Nine of the rewrite’s thirty groups ended in CommitStateUnknownException, which is the state chapter 4 described. The driver does not know whether the catalog applied the commit, so it cannot delete the output files. It does not count the groups as failed either. The return row said seven commits, a hundred and five files rewritten, zero failed groups. The table had sixty-six files where a clean run leaves thirty. The nine groups’ output files are orphans until remove_orphan_files finds them, and the fixture itself did not recover — it took a restart.

Three lessons, each cheap once known. Partial progress is a burst of commits, and the catalog has to be sized for the burst, which chapter 3 warned about in the abstract. The procedure’s return row does not report refused or unknown-state groups. The snapshot log and the file count do, so a scheduled compaction verifies its result from metadata, not from its own counters. And a 500 from the catalog mid-commit produces orphans by design, which is why chapter 10’s orphan cleanup is not optional on a platform with any compaction at all.

Delete files: v2 and v3 are different tables

Two merge-on-read tables of six hundred thousand rows, sixty files, ten rounds of a one-percent delete each.

  v2: after 10 rounds: 60 data files, 60 delete files (PARQUET), 60,000 positions
      count(*) costs GETs 300, 0.2 MB
  v3: after 10 rounds: 60 data files, 60 delete files (PUFFIN),  60,000 positions
      count(*) costs GETs 180, 0.1 MB

Sixty delete files after ten rounds — not six hundred. Spark rewrites a data file’s position deletes each time it adds to them, so the count stays at one per data file in both versions. The reader still pays for every one: five requests per data file on v2, three on v3. A deletion vector is one contiguous fetch rather than a Parquet file with a footer.

rewrite_position_delete_files on either table did nothing, because there was nothing to merge. What removes the deletes is rewriting the data files that carry them. delete-file-threshold is how the planner is told that a file with any deletes at all is a candidate, whatever its size.

  v2: rewrite_data_files delete-file-threshold=1: 60 -> 30 data files, 0 delete files removed
      now 30 data files, 60 delete files still listed; count(*) costs GETs 0
  v3: rewrite_data_files delete-file-threshold=1: 60 -> 30 data files, 60 delete files removed
      now 30 data files, 0 delete files

On v3 the rewrite took the deletion vectors with it. On v2 the sixty position-delete files are still listed after the files they point at are gone. They no longer cost a reader anything, because a position delete applies by file path and the paths no longer exist. But they are in the manifests, they are in every metadata query, and they will be in the next planner’s input. Setting remove-dangling-deletes on the same call removed none of them, and the reason is in the sequence numbers.

  after the rewrite: data files at sequence number 70, delete files at sequence number 70

A position delete applies to data files whose sequence number is at or below its own, and the dangling test uses that rule. With use-starting-sequence-number at its default of true, the rewritten data files carry the sequence number the rewrite started from, which is the last delete’s. By the number, the deletes still apply and are not dangling. Two things fix it, both measured.

  rewrite_position_delete_files with rewrite-all:            60 -> 0 delete files
  rewrite_data_files with use-starting-sequence-number=false: new files at 71, 60 delete files removed

The first is the cleanup to schedule after any v2 data rewrite. The second changes what the rewritten files’ sequence number means for every other delete that might be in flight against them. That interaction was not measured, so the first is the safer policy. And the ratio threshold does what the jar says: forty percent of one day deleted, and a rewrite with no options rewrote that day’s two files and nothing else.

A v2 table needs a delete-file cleanup step in its policy that a v3 table does not. That is a reason to be on v3 — and chapter 5’s certification matrix is the reason some tables cannot be.

Return on cost, and a score for a fleet

The binpack run wrote 8.5 MB in thirty requests and one second. Against it, the customer query saves 350 requests per run and the day query twelve. At chapter 2’s prices and the bookshop’s query rate, that is the payback period — and it is minutes. The general form is bytes and requests written, against requests saved per query times queries per day. A compaction that does not pay back within its own schedule interval is one to run less often.

The prioritisation question across a fleet is which tables to spend the compaction budget on, and the answer comes from the three metadata tables this chapter has been reading. Files beyond one per partition, delete positions per data file, and snapshots since the last replace, which is compaction’s own operation name in the snapshot log.

SELECT (SELECT count(*) - count(DISTINCT partition) FROM t.data_files)          AS excess_files,
       (SELECT coalesce(sum(record_count), 0) FROM t.delete_files)
         / greatest((SELECT count(*) FROM t.data_files), 1)                   AS deletes_per_file,
       (SELECT count(*) FROM t.snapshots
         WHERE committed_at > coalesce((SELECT max(committed_at) FROM t.snapshots
                                        WHERE operation = 'replace'), timestamp'1970-01-01')) AS since_compaction;

  table                      files parts excess dels del/file since  score
  stage11.stream               150    30    120    0        0   150     270
  stage10.orders_hourly        936   720    216    0        0    10     226
  stage11.c2                    66    30     36    0        0     0      36
  stage11.d2r                   59    30     29    2      136     0      30
  stage11.mor2                  30    30      0   60     2000     0      20
  stage11.live                  35    30      5    0        0     3       8

The score is the sum of the three, the second scaled by a hundred, and its purpose is ranking rather than measurement. The stream table that opened this chapter is at the top. The hourly table from chapter 7 is second, the sixty-six-file leftover of the catalog failure is third, and the tables that were just compacted are at the bottom. A first draft of this score used the small-file ratio against the 512 MB target instead of files per partition. It rated every table in the lab as 100% small and ranked them by file count alone. Files beyond one per partition is the number compaction can change — distance from a target it will never reach is not.

Chapter 13 puts these three columns on a dashboard and gives them thresholds. Book 3’s table-health agent runs the same query, which is why it is deterministic SQL and not a judgment.

The decision tree

For each table, in order:

1. Is there anything to do? Run the candidate query. Zero candidates, no job. Do not schedule a compaction that the prediction says is a no-op; it will exit green and tell you nothing.
2. Scope it. A where on the partitions the writer is still filling. History is compacted once.
3. Binpack, unless a measured predicate on this table justifies a sort, and the partitions are large enough that a sort has something to give a reader to skip.
4. Partial progress on, with max-commits sized to what the catalog can take in a burst. Verify the result from the snapshot log and the file count, never from the return row.
5. Deletes: on v2, delete-file-threshold for the data rewrite and a rewrite_position_delete_files with rewrite-all after it; on v3, the data rewrite alone.
6. Schedule against the writers. Appends can overlap. Row-level writers on the same partitions cannot, unless they retry from a fresh scan.
7. Expire and clean orphans after, which is chapter 10, and the second is not optional once a catalog has ever returned a 500.

What was not run

The strategies were compared on partitions of a few hundred kilobytes. A sort or z-order on a partition holding many row groups was not measured here, and chapter 7 says what to expect. The use-starting-sequence-number=false rewrite was measured for its effect on dangling deletes only, not for its interaction with a concurrent delete. Whether a Spark copy-on-write delete can be made to retry a ValidationException was not tested. And the catalog failure was the fixture’s SQLite; Polaris and Lakekeeper took the same burst without complaint, but neither was pushed to find its own limit.

Exercises

1. Predict before you run. Take a table you own, run the candidate query with its actual write.target-file-size-bytes, and write down the number of files it says a default compaction will rewrite. Then run rewrite_data_files with no options and compare rewritten_data_files_count. If they differ, find the option that explains it.

Show answer

They match when the table’s files are all in one spec and no file has deletes. A file rewritten that the query did not predict is usually a delete-ratio candidate. The jar’s delete-ratio-threshold of 0.3 makes a file with a third of its rows deleted a candidate at any size, and the size-only query does not see it.

2. Find the unknown-state commits. After any compaction run with partial progress, count the replace snapshots it added and compare against the run’s added_data_files_count divided by the files per commit. Then compare the table’s data-file count against the count the plan predicted.

Show answer

A run whose snapshot count and file count agree with its return row completed cleanly. Fewer snapshots than commits reported, or more files than the plan should have left, means groups ended in an unknown state, and their output files are orphans until the next remove_orphan_files with an older_than that has passed them.

Final thoughts

Compaction on a platform is not a procedure you call — it is a policy you own, and the policy has more parts than the procedure has options. Which partitions, decided by a where. Whether at all, decided by a query against metadata that predicts the plan. Which strategy, decided by measurement rather than by the name of the strategy. How many commits, decided by what the catalog can absorb. When, decided by who else is writing. What to verify afterwards, decided by the knowledge that the return row is not the truth. And on v2, one more step that v3 does not need.

The next chapter is what compaction does not touch: the metadata. Manifests accumulate at a rate the data files do not, the planning half of a query’s latency is paid in them, and there is a rewrite for those too.

Next: Six Hundred Copies of the Snapshot List