Found Conflicting Files

Writer A is Spark, deleting every even-numbered German order from a ten-million-row table. Copy-on-write, so it has to rewrite every file in the DE partition — which takes about two seconds. Writer B is PyIceberg, and half a second into A’s rewrite it appends two German orders with even numbers.

   B: committed 2 row(s), snapshot 1127152807837912356 at t+0.5s
   A: DELETE WHERE country='DE' AND order_id % 2 = 0: FAILED after 2.4s:
      ValidationException: Found conflicting files that can contain records matching
      ref(name="country") == "DE": [s3://warehouse/stage7/c2/data/country=DE/00000-0-….parquet]
   B's rows present: 2 | even DE rows left: 1666668

A did all the work and was refused at the last moment. B’s two rows are in the table; none of the 1,666,668 rows A meant to delete are gone. Now the same race on a table that differs by one property, write.delete.isolation-level = snapshot.

   B: committed 2 row(s), snapshot 6514933618262427568 at t+0.5s
   A: DELETE WHERE country='DE' AND order_id % 2 = 0: committed after 2.0s
   B's rows present: 2 | even DE rows left: 1

A committed, leaving B’s two rows in place — including the even-numbered German order covered by A’s DELETE. The first run required the delete to account for a concurrent matching append; the second required it to apply only to the snapshot A read. Both outcomes satisfy their isolation setting. With one writer, the distinction never surfaced. With two writers touching the same partition, it decides whether the delete commits.

What a commit actually checks

Book 1’s chapter 4 reduced Iceberg’s transactionality to one conditional swap. A writer reads the current metadata pointer, does its work, and asks the catalog to move the pointer from the value it read to a new one. If the pointer moved in the meantime, the swap fails. That is still the whole mechanism. What this chapter adds is what happens after the swap fails, because the answer is not “the writer fails”.

A writer’s commit carries two things. It carries the changes it wants to make, as a list of updates. And it carries requirements: assertions about the state the table must be in for the changes to make sense. The most basic of them is “the main branch still points at the snapshot I started from.” The REST protocol sends both in the same request, and the catalog rejects the request if any requirement fails. Book 1 chapter 5 showed the requirements array on the wire; this is what it is for.

A failed requirement does not yet tell you whether the write must fail. The client can respond in three ways — depending on whether the work remains valid against the new table state.

1. Refresh and retry. Reload the table, re-apply the same logical change on top of the new snapshot, and try the swap again. This is right when the change does not depend on what the table looked like: an append is an append regardless of what else was appended.
2. Refresh and validate. Reload the table, look at what changed since the snapshot the operation read, and decide whether the operation is still correct. A delete that scanned files which have since been rewritten, or a compaction whose input files have since been deleted, is not.
3. Refuse. Report the failure and do nothing. This is what a client does when it has no retry logic, and it is what one of the clients on this platform does.

The Java library — which is what Spark, Flink and Trino use — does the first two automatically and picks between them by operation type. Four table properties control how many times and how patiently, and their defaults were read from the 1.11.0 jar rather than the documentation: commit.retry.num-retries is 4, commit.retry.min-wait-ms is 100, commit.retry.max-wait-ms is 60,000, and commit.retry.total-timeout-ms is 1,800,000, thirty minutes. The validation is per operation, and whether a given concurrent change counts as a conflict is what the rest of this chapter runs.

The requirement on the wire

The installed PyIceberg 0.11.1 defines exactly eight requirement types, and the list is worth having in front of you, because every refusal in this chapter is one of them failing.

assert-create                     the table must not exist yet
assert-table-uuid                 the table is the one I think it is
assert-ref-snapshot-id            branch X still points at snapshot Y
assert-current-schema-id          nobody changed the schema under me
assert-last-assigned-field-id     nobody added a column under me
assert-default-spec-id            nobody changed the partition spec under me
assert-last-assigned-partition-id nobody added a partition field under me
assert-default-sort-order-id      nobody changed the sort order under me

An append attaches one of them: assert-ref-snapshot-id for main, with the snapshot id the client read. When the fixture answers Requirement failed: branch main has changed: expected id … != …, that is the server’s message for that one requirement, and it is the same text whichever client sent it. A schema change attaches the schema assertions instead, which is why a concurrent ALTER TABLE and an append do not conflict with each other — they assert different things.

What differs between clients is what happens next. The Java library catches the failure, refreshes, and re-applies. PyIceberg’s REST client has a retry decorator on its calls, and the installed source says exactly what it retries: expired and unauthorised tokens, two attempts, nothing else. A failed requirement is passed straight to the caller.

The conflict classes, run

The lab for this chapter is deliberately artificial in one way. Real concurrent writers arrive whenever they arrive. Here, writer A runs on a thread and writer B is released at a fixed moment, so that each class can be produced on demand and its exception text pasted rather than described. The table is ten million orders in three country partitions, copy-on-write deletes, Spark 4.1.3 for A and PyIceberg 0.11.1 for B, both through the reference REST catalog on object storage.

A is doing	B does meanwhile	Isolation	Outcome
append	append	any	both commit; A retried on top of B
delete, copy-on-write	append rows the delete predicate matches	serializable (default)	A refused: ValidationException: Found conflicting files
delete, copy-on-write	append rows the delete predicate matches	snapshot	both commit; B’s matching rows survive the delete
delete, copy-on-write	append rows the predicate does not match	serializable	both commit
compaction	delete a row in a file being compacted	any	compaction refused; the delete survives
append from a stale snapshot	(A already committed)	any	PyIceberg refuses: CommitFailedException: Requirement failed: branch main has changed

Each row of that table is a run. The append race shows why a stale snapshot need not invalidate a writer’s work: A loaded the table, B appended a row, and A then appended another from its stale view.

   B committed 1 row(s), snapshot 156528149968054319
  A: INSERT from the stale snapshot: OK
   rows now: 3002 (3000 + B's 1 + A's 1)

Both rows landed. Spark’s first swap failed the branch requirement, the library refreshed, re-applied the append on top of B’s snapshot, and the second swap succeeded. Nothing about an append depends on the rest of the table, so nothing needs validating. A pipeline of independent appenders never conflicts in any way that costs more than a retry, which is why streaming writers can share a table at all.

A delete that overlaps an append is the case the isolation level decides, and it is the chapter’s opening. Under the default, serializable, A’s commit validates that no file was added since A’s scan that could contain rows matching the delete predicate. B’s file could, so A is refused, and the refusal names the file. Note what the validation checks: not whether B’s rows do match, which would require reading them, but whether the file’s partition and statistics say they might. It is conservative — and it is conservative on purpose.

Under snapshot isolation A’s commit only checks that the files it rewrote were not themselves changed. B’s file was new, not rewritten, so A commits, and B’s even-numbered German order survives a DELETE whose predicate covers it. That is not a bug. Snapshot isolation promises that A’s delete applied to the snapshot A read, and it did. It does not promise that the table afterwards contains no row matching the predicate — and it does not.

Which one you want depends entirely on what the delete means. “Remove these customers’ data because they asked” needs serializable, and needs the refusal, because a late row slipping through is a compliance failure. “Clear out yesterday’s staging rows” is fine with snapshot, and better with it, because a refused delete on a busy table is a delete that never completes. The property is per table and per operation, write.delete.isolation-level here, and the same pair exists for UPDATE and MERGE under write.update.isolation-level and write.merge.isolation-level. Only the delete pair was run; the other two are documented as behaving the same way and are flagged here as unrun.

A delete that does not overlap the append commits under either isolation level. B appended a US row while A was deleting German ones.

   B: committed 1 row(s) at t+0.5s
   A: DELETE WHERE country='DE' AND order_id % 2 = 0: committed after 2.0s
   B's rows present: 1 | even DE rows left: 0

The validation is partition-aware. B’s file lives under country=US, the predicate says country = 'DE', and the file cannot contain a match, so serializable validation passes. That is the practical reason to partition by the columns your deletes filter on — not only does the delete read fewer files, it conflicts with fewer writers.

Compaction against a concurrent delete is a conflict, and it is refused with a hint. A held a view of seven small files in the DE partition. B deleted one row, which under copy-on-write rewrote one of those seven files. A then asked to compact all seven.

  A: rewrite_data_files on the stale view: FAILED
     ValidationException or CommitFailedException. This usually means that this rewrite has
     conflicted with another concurrent Iceberg operation. To reduce the likelihood of conflicts,
     set partial-progress.enabled which will break up the rewrite into multiple smaller commits …
   order 95001 present? 0 (0 = B's delete survived)

The compaction’s commit validates that every file it is replacing still exists in the current snapshot. One of them did not — because B’s delete had replaced it. Committing would have resurrected the deleted row from the compaction’s copy of the old file, so the refusal is the correct outcome and the delete stands. The hint in the message is real and chapter 8 takes it up. partial-progress.enabled makes a compaction commit in several smaller groups, so a conflict costs one group rather than the whole job. Book 1’s chapter 12 met this refusal once by accident. This is what it was protecting.

PyIceberg does not retry, and its refusal is exact. B loaded the table, A committed an append, and B appended from its now-stale table object.

  B: append from the stale table object: FAILED
     CommitFailedException: Requirement failed: branch main has changed:
     expected id 6590538206344769727 != 6959608700459847087
  B: reload, then append: OK

That is the requirement from the top of the chapter, failing as designed, and PyIceberg 0.11.1 passes the failure straight to the caller. There is no automatic refresh and no retry. The caller reloads the table and appends again, and it works — because an append cannot conflict. A PyIceberg writer sharing a table with anything else needs a retry loop in its own code. Book 1’s chapter 4 said so with a smaller table; this is the exception it will see.

The retry that duplicates

A retry is only safe if the operation being retried did not already happen. The failure that violates that is not a conflict. It is a commit that succeeded and whose response was lost, and it is the one the catalog cannot tell you about, because from the catalog’s side nothing went wrong.

   first attempt raised: simulated: the response was lost after the catalog applied the commit
   naive retry appended again; rows with order_id 91000: 2

The client appended a row, the catalog applied the commit, and the client’s connection dropped before the response arrived. The client saw an error, retried, and the row is now in the table twice. Nothing in Iceberg prevents this, and nothing should: two appends of the same row are two valid commits. The library’s own retries do not cause it, because they only fire on a failed requirement, and this requirement did not fail. It is the application’s retry — the one in the orchestrator or the pipeline framework — that does the damage.

The fix is an idempotency key, and Iceberg has a natural place for one. Every snapshot carries a summary map, writers can put arbitrary properties in it, and a retrying client can read it before deciding whether to write.

tbl.append(rows, snapshot_properties={"batch-id": "orders-2026-06-12-run-7"})

   idempotent retry: 2 snapshot(s) already carry batch-id=orders-2026-06-12-run-7 -> skip the append

Before retrying, list the snapshots, look for one whose summary carries the batch’s key, and skip the write if it is there. Spark writers set the same property through spark.sql.catalog.<name>.snapshot-property.<key> or per-write options, and Book 1’s chapter 16 used the same mechanism for wap.id. That gives a writer retried by an orchestrator a way to check whether its earlier attempt succeeded. It stamps each commit and checks the stamp before retrying. A pipeline that is at-least-once at the orchestration layer becomes exactly-once at the table only through that check.

One of the two production catalogs on this platform advertises something better. Lakekeeper’s configuration response carries idempotency-key-lifetime: PT30M, meaning it can recognise a repeated commit request by a client-supplied key for thirty minutes and answer it with the original result instead of applying it again. That was observed in the configuration and not exercised, and the chapter says so. A client library that sends the key has to exist first, and the snapshot-summary check works against every catalog today.

What was not run

Two things in this chapter’s outline stayed on paper, and a book that prices requests should say which.

A retry storm under a slow catalog. The retry policy’s backoff exists for the case where a hundred writers all fail a requirement at once. Each refreshes and retries, and the catalog that was slow enough to cause the first round is now slower. Producing that honestly needs a catalog with injected latency and a fleet of writers, and neither was built for this chapter. Chapter 13 puts the symptom on the table’s dashboard and commit latency and retry counts on the writer’s, and chapter 11 has the writer that retries.

Flink and Trino as the second writer. Every race here was Spark against PyIceberg. Flink’s commits are made by its job manager once per checkpoint through the same Java library, so the same retry and validation rules apply. Chapter 11 runs a streaming writer against a concurrent compaction. Trino’s write path was not raced against anything. Both are the Java library underneath — which is the reason to expect the same texts, and not the same thing as having seen them.

What to decide, and why

Set the isolation level per table, on purpose, and write down why. Serializable is the default and the safe one: a delete, update or merge that overlaps a concurrent write is refused rather than applied to a table it did not read. Choose snapshot only for tables where a late row surviving a delete is acceptable and a refused delete is not. In practice that means append-heavy tables cleaned by predicate rather than tables corrected for a reason. The exception: a table with a single writer can run either and will never see the difference.

The division of responsibility depends on the client: the Java library handles failed requirements, while PyIceberg leaves that retry to its caller. Neither removes the need to recognise a batch an orchestrator has already submitted.

Client	Library retry on a failed requirement	What you add
Spark, Trino, Flink (Java)	yes: 4 attempts, 100 ms to 60 s backoff, 30 min cap	a batch key in the snapshot summary, checked before any orchestrator-level re-run
PyIceberg 0.11.1	none; only expired tokens are retried, twice	a reload-and-retry loop around every commit, plus the batch key
DuckDB 1.5.5	not measured in this chapter	treat as PyIceberg until chapter 5 says otherwise

Every client that shares a table has a retry, and every retry has a key. Spark, Flink and Trino retry through the library. PyIceberg does not, so a PyIceberg writer wraps its commit in a reload-and-retry loop of its own. And any writer that an orchestrator can re-run stamps its commits with a batch key and checks for the stamp first.

Partition by what you delete on. The validation is partition-aware, so a delete conflicts only with writers in the partitions it touches. A table deleted by country and partitioned by day conflicts with everything.

Maintenance is a writer too. Compaction validates that its inputs still exist, and loses to any concurrent delete or rewrite of them. Chapter 8 makes that a policy: partial progress on, and compaction scheduled against the writers’ cadence rather than into it.

Exercises

1. Make the delete win. Re-run the opening race with the append released two seconds after the delete instead of half a second. Which writer is refused now, and what does its exception say? Then set commit.retry.num-retries to 0 on the table and run the append-against-append case. What changes?

Show answer

With B released after A’s commit, B’s PyIceberg append fails the branch requirement (Requirement failed: branch main has changed) and A’s delete is untouched; PyIceberg does not retry, so B must reload and append again. With retries set to zero, Spark’s append from a stale snapshot fails the same requirement and Spark surfaces it as a CommitFailedException instead of retrying; the row does not land.

2. Find the duplicate without counting. After the naive retry in the idempotency section, the table holds order 91000 twice. Using only the snapshots metadata table and each snapshot’s summary, identify the two commits that wrote it, without querying the data. Which summary field distinguishes them?

Show answer

Both snapshots carry batch-id=orders-2026-06-12-run-7 in their summary, operation=append, and added-records=1; they differ only in snapshot-id and committed_at. The key is what identifies the duplicate; the timestamps are what tell you which one the retry produced. Rolling back to the earlier snapshot removes the second row without reading a data file.

Final thoughts

Book 1 said the catalog is the transaction boundary, and chapter 3 of this book said it is the only part that can say no. This chapter is what “no” sounds like in the common case. Not an outage, not a permission — but a second writer, half a second earlier, whose file could contain a row yours was about to touch. The refusal is the format working. The retry is the library working. The duplicate is the one failure neither can see, and the idempotency key is yours to add.

Every race here was between two engines that use the same Java library or a Python one that does not retry at all. The next chapter widens that to every engine on the platform and asks a plainer question. When five engines write the same table, does every one of them read what the others wrote?

Next: Compatible Is a Test Result