A Bitmap and a Serial Number
The two v3 additions that change a table's architecture rather than its type system: deletion vectors, which replace a Parquet file with 44 bytes of bitmap, and row lineage, which gives every row a stable identity and turns change data capture into a WHERE clause.
Five of format v3’s seven additions are types. You use them or you do not, and the previous chapter established which ones you can currently reach.
The other two are different in kind. Deletion vectors change how a mutation is written to storage. Row lineage adds something to the table that no version of Iceberg had before: a per-row identity that survives being updated. Neither is a type you opt into per column. Both apply to the whole table — and both change what the table is capable of.
Row lineage has the larger consequence: it means an Iceberg table can answer which rows changed since this point without a change data capture pipeline attached to it. No Debezium, no log reader, no separate stream. The answer is a WHERE clause against a column that Iceberg maintains for you.
The runs below show both features and measure deletion-vector storage. The lineage query also exposes a limit in August 2026: it finds inserts and updates, but a deleted row is absent from its result.
Pinned: Apache Iceberg 1.11.0 (iceberg-spark-runtime-4.1_2.13:1.11.0) on Spark 4.1.3, Java 21, against the 1.10.1 REST catalog fixture. PyIceberg 0.11.1 and DuckDB 1.5.5 read the same tables. Everything below was executed unless the sentence says otherwise.
The file that tells you which one it is
Chapter 10 established merge-on-read. Instead of rewriting a data file to remove a row, the writer leaves the data file alone and records the deletion in a separate file. Under v2 that separate file is a Parquet file listing (file_path, position) pairs.
Under v3 it is this:
00000-33-12eff7ed-8bb9-4301-a52e-20aee32f24e3-00001-deletes.puffinNot .parquet. .puffin. That extension is the whole visible difference, and it is the fastest way to tell by eye which encoding a table is actually using. Which turns out to matter: as the previous chapter showed, a v3 table produces neither until you ask it to.
Here is a bookshop orders table set up to produce them. It is a cut-down version: four columns rather than the eight chapter 3 dissected. Deletion vectors are indifferent to how wide a row is, and a narrower table makes the Puffin file easier to read by hand:
CREATE TABLE ice.ch20.orders (order_id BIGINT, customer STRING, status STRING, amount DOUBLE)
USING iceberg TBLPROPERTIES (
'format-version'='3',
'write.delete.mode'='merge-on-read',
'write.update.mode'='merge-on-read',
'write.merge.mode'='merge-on-read');
INSERT INTO ice.ch20.orders VALUES
(1,'ada','placed',30.0),(2,'grace','placed',12.5),(3,'alan','shipped',99.0),
(4,'kay','placed',7.25),(5,'edsger','shipped',44.0);
DELETE FROM ice.ch20.orders WHERE order_id = 2;SELECT content, record_count, file_size_in_bytes, file_path
FROM ice.ch20.orders.all_delete_files;+-------+------------+------------------+---------------------------------------------+
|content|record_count|file_size_in_bytes|file_path |
+-------+------------+------------------+---------------------------------------------+
|1 |1 |478 |...-00001-deletes.puffin |
+-------+------------+------------------+---------------------------------------------+content = 1 is Iceberg’s marker for a position delete, the same value a v2 position delete file carries. The metadata schema did not change — the encoding did.
What is actually inside 478 bytes
Puffin is Iceberg’s own small container format for auxiliary blobs. Open one: the structure explains the performance argument better than a description of it does.
Take the 480-byte file produced by a later mutation, and read its first four and last four bytes:
size 480
head magic b'PFA1'
tail magic b'PFA1'A magic number at both ends — so a reader can find the footer from the end of the file without scanning. The footer is JSON:
{
"blobs": [
{
"type": "deletion-vector-v1",
"fields": [ 2147483645 ],
"snapshot-id": -1,
"sequence-number": -1,
"offset": 4,
"length": 44,
"properties": {
"referenced-data-file": ".../00000-27-daa520b3-...-0-00001.parquet",
"cardinality": "2"
}
}
],
"properties": {
"created-by": "Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"
}
}The footer tells a reader how to interpret the blob, where to find it and which data file it affects.
"type": "deletion-vector-v1" is the blob type. Puffin is a general container, and Iceberg already uses it for other things, notably table statistics. The type string is how a reader knows which of those it is holding.
fields: [2147483645] is the field ID the blob is about. That number is Integer.MAX_VALUE - 2, one of Iceberg’s reserved metadata field IDs. It means _pos, the row’s ordinal position within its data file. So the blob is a set of positions, exactly as a v2 position delete file is.
offset: 4, length: 44 locates the payload. The entire deletion vector for this data file is 44 bytes, in a 480-byte file. Everything else is the container and the footer.
referenced-data-file and cardinality are the two things a query planner wants before it reads anything — which data file this applies to, and how many rows it removes.
The payload itself is a serialised Roaring bitmap:
blob bytes: 00000024 d1d33964 0100000000000000 000000003a3000000100000000000100...
^length ^magicA four-byte length, a four-byte magic number, then the bitmap. That is the substantive change. A v2 position delete file stores a row per deleted position; a deletion vector stores a bitmap. Bitmaps compress runs of deletions to nearly nothing. And a reader applies one with a membership test on an integer, rather than by merging a sorted stream of positions against the data file it is scanning.
Exactly one per data file
There is a rule in v3 that v2 does not have: at most one deletion vector may apply to a data file. It is the constraint that makes the encoding useful — and it is directly observable.
Continuing on the same table, the delete above produced a DV over the first data file with record_count = 1. Now update a row in that same file:
UPDATE ice.ch20.orders SET status = 'shipped' WHERE order_id = 1;delete files now:
+-------+------------+---------------------------------------------------------+
|content|record_count|file_path |
+-------+------------+---------------------------------------------------------+
|1 |1 |00000-33-12eff7ed-...-00001-deletes.puffin |
|1 |2 |00000-37-e00d4dc3-...-00001-deletes.puffin |
+-------+------------+---------------------------------------------------------+A second puffin, with record_count = 2. It did not add a delete to the existing vector; it wrote a new vector containing both positions — and the old one is now historical. Now delete once more, against a different data file, and ask for the live set. You get exactly two files, one per data file:
LIVE delete files (current snapshot only):
+-------+------------+---------------------------------------------------------+
|content|record_count|file_path |
+-------+------------+---------------------------------------------------------+
|1 |1 |00000-42-1fa323fb-...-00001-deletes.puffin |
|1 |2 |00000-37-e00d4dc3-...-00001-deletes.puffin |
+-------+------------+---------------------------------------------------------+Three deletions across two data files, two vectors, one each. The all_delete_files view shows all three that were ever written, with referenced_data_file making the pairing explicit. The live view shows the two that count.
This is where the reading cost goes. Under a rule of one vector per data file, a scan’s worst case is one extra small read per data file it touches. And the planner knows the cardinality before opening anything. Without that rule, delete files accumulate against a data file and every scan pays for all of them.
A caveat on that framing, because I checked and the obvious version of the comparison is wrong. The claim you will read is that v2 accumulates delete files while v3 does not. In Iceberg 1.11.0 that is not what Spark does. Ten successive single-row deletes against a v2 merge-on-read table leave exactly one live delete file. The writer rewrites it each time, just as the v3 writer rewrites the vector. The difference between them is not file count. It is size — and the guarantee.
Measuring the difference
Two identical bookshop tables, 1,000 rows each, two data files each, one at v2 and one at v3, both on merge-on-read. Ten separate DELETE statements against each.
--- v2 table, 2 data file(s), 1000 rows ---
after delete # 1: live delete files=1 positions=1 bytes=1596
after delete # 2: live delete files=1 positions=2 bytes=1647
after delete # 3: live delete files=1 positions=3 bytes=1651
after delete # 5: live delete files=1 positions=5 bytes=1654
after delete #10: live delete files=1 positions=10 bytes=1681
rows remaining: 990
files on disk by extension: {'parquet': 12}
total bytes in data/: 20480
--- v3 table, 2 data file(s), 1000 rows ---
after delete # 1: live delete files=1 positions=1 bytes=478
after delete # 2: live delete files=1 positions=2 bytes=480
after delete # 3: live delete files=1 positions=3 bytes=482
after delete # 5: live delete files=1 positions=5 bytes=486
after delete #10: live delete files=1 positions=10 bytes=497
rows remaining: 990
files on disk by extension: {'puffin': 10, 'parquet': 2}
total bytes in data/: 8804Both tables end with 990 rows and one live delete file. The v2 delete file is 1,681 bytes and the v3 vector is 497, a factor of about 3.4. Across everything left on disk, including the superseded versions that a later expiry will collect, 20,480 bytes against 8,804.
Look at the growth rates — because they are the real story. Ten deletes moved the v2 file from 1,596 to 1,681 bytes, and the v3 vector from 478 to 497. Roughly 9 bytes per deleted row against roughly 2. The v2 floor of 1,596 bytes is Parquet overhead — a schema, a footer, row group metadata, all to describe a single integer. Puffin’s floor is 478 bytes for a 42-byte bitmap and a JSON footer with a file path in it.
These are small absolute numbers on a deliberately small table, and I want to be careful about what they do and do not show. They demonstrate the shape of the difference, not its magnitude at scale. What I did not measure is query latency with vectors versus position deletes, on either table, at any size. The bitmap-versus-merge argument is a structural one and I have not put a stopwatch on it.
Two knobs that stop mattering
Two things you may have configured for v2 behave differently here.
write.delete.granularity is inert. Under v2 it chooses between one delete file per data file and one per partition. Under v3 the one-vector-per-data-file rule settles it. Two identical partitioned tables, one at each setting, one DELETE hitting two data files:
--- write.delete.granularity = file ---
delete-file entries: 2 | distinct puffin files: 1
...deletes.puffin -> refs 00000-2-4dc9ec8b... | records 1 | offset 4 | blob bytes 42
...deletes.puffin -> refs 00000-5-fd2cfa91... | records 1 | offset 46 | blob bytes 42
puffin files on disk: 1 [836]
--- write.delete.granularity = partition ---
delete-file entries: 2 | distinct puffin files: 1
...deletes.puffin -> refs 00000-16-cb12a355... | records 1 | offset 4 | blob bytes 42
...deletes.puffin -> refs 00000-13-4e503acb... | records 1 | offset 46 | blob bytes 42
puffin files on disk: 1 [848]Identical structure at both settings. Notice what one puffin file is doing here: it holds two deletion vectors, at offsets 4 and 46, 42 bytes each, one per data file. The rule is one vector per data file, not one file per vector, and a single commit packs its vectors together. That is why delete_files shows two rows with the same file_path and different referenced_data_file values. It looks like a contradiction until you have seen the footer.
Mixed encodings coexist, and both are applied. A table that was v2, took a merge-on-read delete, and was then upgraded does not lose its Parquet delete file:
LIVE delete files at v2:
|content|record_count|...-00001-deletes.parquet|
LIVE delete files after upgrading and deleting again:
|content|record_count|...-00001-deletes.puffin |
|content|record_count|...-00001-deletes.parquet|rows: 2
+--------+--------+
|order_id|customer|
+--------+--------+
|1 |ada |
|4 |kay |
+--------+--------+Four rows, two deletes, one recorded as Parquet and one as a bitmap, and the reader applies both correctly. The upgrade does not migrate existing delete files — and it does not have to; the v3 specification keeps position delete files readable. What it does not allow is writing a new one.
What deletion vectors do not replace
The new bitmap encoding still needs a known row position. Chapter 10 described two kinds of delete file. A position delete says “row 47 of that file is gone”. An equality delete says “any row where order_id = 5 is gone”, without knowing where such rows live. Position deletes are what a batch engine writes — because it just scanned the data and knows the positions. Equality deletes are what a streaming writer produces, because it has a key and no intention of scanning anything.
Deletion vectors replace the position encoding only. The specification is explicit:
Position deletes are encoded in a position delete file (V2) or deletion vector (V3 or above). Equality deletes — Mark a row deleted by one or more column values, like
id = 5. Equality deletes are encoded in equality delete file.
So the v2-to-v3 delete story is:
| v2 | v3 | |
|---|---|---|
| Position deletes | position delete file (Parquet) | deletion vector (Puffin) |
| Equality deletes | equality delete file | equality delete file, unchanged |
Equality deletes are alive and well in v3. If you write to Iceberg from Flink or Kafka Connect, that is still your mechanism and nothing about it changed.
I could not produce an equality delete to show you. Spark SQL does not write them, and this lab has no Flink or Kafka Connect, which chapter 17 already flagged. What I can offer instead is corroboration from the 1.11.0 runtime jar, which is evidence rather than execution. ClusteredEqualityDeleteWriter, DeleteFileIndex$EqualityDeletes, ConvertEqualityDeleteFiles and ConvertEqualityDeleteStrategy are all present. And a scan of the shaded classes turns up no “not supported in v3” validation for them.
That third class name is the interesting one. ConvertEqualityDeleteFiles is a maintenance action that rewrites equality deletes into position deletes, which is the right thing to want. Equality deletes are cheap to write and expensive to read, because applying one means evaluating a predicate against every candidate row. Converting them on a schedule moves that cost off the read path. Spark 4.1 exposes no SQL procedure for it. The procedures shipped in that runtime are rewrite_data_files, rewrite_position_delete_files, rewrite_manifests, expire_snapshots, remove_orphan_files, add_files, migrate, snapshot, and the snapshot-management set; there is no convert_equality_deletes among them. Whether the action is reachable from another engine, I did not test.
Maintenance, with vectors in the way
Chapter 12 built the compaction story on a v2 table. It holds on v3, with one addition worth showing.
rewrite_position_delete_files runs on a v3 table without complaint and, on a table that already has exactly one compact vector per data file, does nothing:
v3: [Row(rewritten_delete_files_count=0, added_delete_files_count=0,
rewritten_bytes_count=0, added_bytes_count=0)]Which is the correct answer and also a reminder — a procedure returning zeroes is not the same as a procedure failing. The v2 table returned the same zeroes for the same reason.
Compaction is where the deletes actually get resolved. Asking rewrite_data_files to rewrite any file carrying at least one delete:
CALL ice.system.rewrite_data_files(
table => 'ch20.amp_v3',
options => map('delete-file-threshold','1'));[Row(rewritten_data_files_count=2, added_data_files_count=1, rewritten_bytes_count=3933,
failed_data_files_count=0, removed_delete_files_count=1)]
live delete files after compaction: Row(count(1)=0, sum(record_count)=None)
rows: 990Two data files became one, the deletion vector was consumed, and the table now has zero delete files and the same 990 rows. The deletions have been materialised into the data. This is the operation that stops merge-on-read read costs from growing without bound. Nothing about v3 changes when you need to run it.
What v3 adds is a property that survives it, and this is the bridge to the second half of the chapter:
+--------+-------+-----------------------------+
|order_id|_row_id|_last_updated_sequence_number|
+--------+-------+-----------------------------+
|1 |0 |1 |
|36 |35 |1 |
|38 |37 |1 |
+--------+-------+-----------------------------+Order 38 kept _row_id = 37 through a physical rewrite that changed which file it lives in and which position it occupies. Note that 37 is not its new position; order 37 was deleted, so order 38 sits at offset 36 in the compacted file. The identity was carried — not recomputed. If it were recomputed from position, compaction would silently renumber every row in the table, and everything in the rest of this chapter would be worthless.
Row lineage
Two hidden columns appear on a v3 table and on no other kind:
_row_id— a numeric identity for the row, assigned once and preserved across updates._last_updated_sequence_number— the sequence number of the commit that last changed the row.
Ask for them on a v2 table and there is nothing there:
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name
`_row_id` cannot be resolved. Did you mean one of the following? [`id`].On the v3 bookshop table, straight after the initial insert:
+--------+--------+-------+-------+-----------------------------+
|order_id|customer|status |_row_id|_last_updated_sequence_number|
+--------+--------+-------+-------+-----------------------------+
|1 |ada |placed |0 |1 |
|2 |grace |placed |1 |1 |
|3 |alan |shipped|2 |1 |
|4 |kay |placed |3 |1 |
|5 |edsger |shipped|4 |1 |
+--------+--------+-------+-------+-----------------------------+Five rows, ids 0 through 4, all last updated at sequence number 1, which is the first commit.
The mechanism is cheap — which is why it can be on by default. Nothing stores a row id per row. Each data file is stamped with a first_row_id, and the table metadata keeps a next-row-id counter. A row’s id is its file’s first_row_id plus its position within the file. That is why the counter is the only new key v3 adds to metadata.json. And it is why turning lineage on costs one integer per file.
Now mutate the table and watch what is preserved.
DELETE FROM ice.ch20.orders WHERE order_id = 2;+--------+-------+-----------------------------+
|order_id|_row_id|_last_updated_sequence_number|
+--------+-------+-----------------------------+
|1 |0 |1 |
|3 |2 |1 |
|4 |3 |1 |
|5 |4 |1 |
+--------+-------+-----------------------------+Row 2’s id is simply absent. Nothing renumbered. Order 3 is still id 2, order 5 is still id 4 — and their sequence numbers did not move because those rows did not change.
UPDATE ice.ch20.orders SET status = 'shipped' WHERE order_id = 1;+--------+--------+-------+-------+-----------------------------+
|order_id|customer|status |_row_id|_last_updated_sequence_number|
+--------+--------+-------+-------+-----------------------------+
|1 |ada |shipped|0 |3 |
|3 |alan |shipped|2 |1 |
|4 |kay |placed |3 |1 |
|5 |edsger |shipped|4 |1 |
+--------+--------+-------+-------+-----------------------------+This is the whole feature in one row. Order 1 was physically deleted and rewritten, since that is what an update is on a merge-on-read table. Its _row_id is still 0. Its _last_updated_sequence_number moved from 1 to 3.
An identity that survives a rewrite, and a marker saying when it last moved. That is the pair of properties change data capture needs, and until v3 an Iceberg table had neither.
Change data capture, without a pipeline
The sequence number is not an opaque token. It is the sequence number of a snapshot — and you can join it back:
SELECT latest_snapshot_id, latest_sequence_number FROM ice.ch20.cdc.metadata_log_entries;+-------------------+----------------------+
|latest_snapshot_id |latest_sequence_number|
+-------------------+----------------------+
|NULL |NULL |
|4376255550193412477|1 |
|7479481975871161618|2 |
|2179919659063503891|3 |
|8390287567131515192|4 |
+-------------------+----------------------+(The snapshots metadata table does not expose the sequence number; metadata_log_entries does. It is not obvious and it costs a few minutes to discover.)
So: take a bookshop table at sequence number 1, apply an update, a delete and an insert, and ask what changed.
SELECT order_id, customer, status, _row_id, _last_updated_sequence_number
FROM ice.ch20.cdc
WHERE _last_updated_sequence_number > 1
ORDER BY order_id;+--------+--------+-------+-------+-----------------------------+
|order_id|customer|status |_row_id|_last_updated_sequence_number|
+--------+--------+-------+-------+-----------------------------+
|1 |ada |shipped|0 |2 |
|4 |kay |placed |4 |4 |
+--------+--------+-------+-------+-----------------------------+Order 1, updated at sequence 2. Order 4, inserted at sequence 4. Both correct, both identified by a predicate on an ordinary column — planned like any other filter. No connector, no log tail, no external state. Record the high-water sequence number your consumer has seen, and next time ask for everything above it.
The query identifies the surviving changed rows. It does not yet provide a complete change feed, for four reasons.
A deleted row does not appear. Order 2 was deleted and it is not in that result, because lineage describes rows that exist — there is no tombstone. If your consumer needs to know about deletions, and most do, lineage alone does not tell you. You would have to diff against the previous state or read the delete files directly.
Row ids have gaps, and they carry no ordering. Order 4 was inserted third and got id 4, not 3. Ids are allocated to files in ranges rather than to rows one at a time. Treat _row_id as an opaque identifier. It is not a sequence, not a count — and comparing two of them tells you nothing.
The built-in changelog procedure does not work with this. Spark’s Iceberg runtime ships create_changelog_view, which is the obvious tool for the job. It produces a view with insert, delete and update-before/update-after markers. On a table with delete files it refuses:
CALL ice.system.create_changelog_view(table => 'ch20.cdc', ...)UnsupportedOperationException: Delete files are currently not supported in changelog scansWhich is to say it does not work on any merge-on-read table, which is every table you would want deletion vectors on. This is the sharpest edge in the chapter. v3 gives you the raw material for CDC, and the tool built to consume it cannot yet read a table that uses v3’s other headline feature.
Incremental reads see appends only. The other pre-existing approach, scanning between two snapshots, has the same shape of problem:
(spark.read.format("iceberg")
.option("start-snapshot-id", snap1)
.option("end-snapshot-id", snap2)
.load("ice.ch20.cdc").show())+--------+--------+------+
|order_id|customer|status|
+--------+--------+------+
|4 |kay |placed|
+--------+--------+------+One row. The update to order 1 and the deletion of order 2 both happened in that range and neither is visible, because an incremental scan reads appended data files.
So the honest summary is: row lineage is the substrate for CDC, and in Iceberg 1.11.0 you write the query yourself. The predicate is simple and it works. What it gives you is updates and inserts, and you need a separate answer for deletes.
The upgrade case, which is stranger than you expect
The earlier lineage runs used tables born at v3. An upgraded table starts with files that have no row identities, so a consumer needs to account for the first commit after the upgrade.
Three rows written at v2, then upgraded:
+------------+------------+---------------------------------------+
|record_count|first_row_id|file_path |
+------------+------------+---------------------------------------+
|1 |NULL |00000-12-32497487-...-0-00001.parquet |
|2 |NULL |00001-13-32497487-...-0-00001.parquet |
+------------+------------+---------------------------------------+
+---+-------+
|id |_row_id|
+---+-------+
|1 |NULL |
|2 |NULL |
|3 |NULL |
+---+-------+No first_row_id on the files, no _row_id on the rows. The previous chapter showed this and left it there. Here is what happens next. Commit anything at all, in this case an unrelated insert of a single row:
INSERT INTO ice.ch20.backfill VALUES (99);+------------+------------+---------------------------------------+
|record_count|first_row_id|file_path |
+------------+------------+---------------------------------------+
|1 |0 |00000-16-5ea78af4-...-0-00001.parquet |
|1 |1 |00000-12-32497487-...-0-00001.parquet |
|2 |2 |00001-13-32497487-...-0-00001.parquet |
+------------+------------+---------------------------------------+
+---+-------+
|id |_row_id|
+---+-------+
|1 |1 |
|2 |2 |
|3 |3 |
|99 |0 |
+---+-------+The three pre-existing rows were assigned ids, retroactively, by a commit that had nothing to do with them. And the newest row got id 0 while the older rows got 1, 2 and 3. Why the new file got the first range rather than the last, I did not chase into the manifest-writing code. What I can report is the order the ids came out in, and it is not the one you would guess.
Nothing here is wrong — ids are opaque, and the specification does not promise that a lower id means an older row. But if you had built anything on the assumption that ids reflect insertion order, this is where it breaks. And it breaks quietly, on a table you upgraded rather than created.
The practical consequence has two halves. On an upgraded table, do not start a CDC consumer until after the first post-upgrade commit. And do not treat the sequence numbers on back-filled rows as meaningful history. They record when the row was given an identity, not when it was written.
Where the engines are
Deletion vectors and row lineage are supported unevenly, and the unevenness does not follow engine maturity. Against one v3 table with both features in use:
| Reads v3 | Applies the deletion vector | _row_id | |
|---|---|---|---|
| Spark 4.1.3 + Iceberg 1.11.0 | yes | yes | yes |
| DuckDB 1.5.5 | yes | yes | yes |
| PyIceberg 0.11.1 | yes | yes | no |
PyIceberg reads the table, applies the vector correctly, sees the puffin files through inspect.all_delete_files(), and reports next-row-id: 6 from the metadata. It has no lineage columns:
_row_id via PyIceberg FAILED: ValueError: Could not find column: '_row_id'DuckDB, attached to the same REST catalog, returns ids that match Spark’s exactly:
duckdb _row_id: [(3, 2), (5, 4), (1, 0)]Beyond this lab I am relying on reported support rather than measurement, and it should be treated accordingly. As of mid-2026, Trino implements deletion vectors for read and write but not full row lineage, and its compaction is not vector-aware. Flink has deletion vectors and vector-aware compaction, with row lineage in progress. I have run neither. Check your engine’s release notes against the specific operation you need, because “supports v3” is not a claim that means anything at this level of granularity.
When not to do this
Merge-on-read with deletion vectors is not a default you should reach for because it is new.
If your table is refreshed wholesale, none of this applies. A table you overwrite nightly has no deletes to encode — and no rows whose identity persists. Copy-on-write is simpler and reads faster.
If you mutate rarely and read constantly, copy-on-write is still probably right. Merge-on-read moves cost from the writer to every subsequent reader, and the vector shrinks that cost without removing it. The arithmetic in chapter 10 is unchanged by v3; the constant got smaller.
If you adopt merge-on-read, you have adopted compaction. The rewrite_data_files call above is not optional maintenance, it is the other half of the design. A merge-on-read table nobody compacts gets slower every day, and it does so without a single error.
And do not build a CDC pipeline on row lineage this quarter without prototyping the delete path first. Updates and inserts work today, cleanly, in a WHERE clause. Deletes do not, and create_changelog_view cannot see a table with delete files. Both are the kind of gap that closes in a release or two. Find out where they are on your versions before you design around them.
Final thoughts
Deletion vectors are the smaller of the two changes and the easier one to justify. They are a better encoding of something Iceberg already did: 44 bytes of bitmap instead of a Parquet file, one per data file by rule rather than by luck. The cardinality and the referenced file are legible from the footer before you open anything. Turn them on with write.delete.mode, keep compacting, and the table behaves as it did before, more cheaply.
Row lineage is the one that changes what an Iceberg table is. A row can now be updated, moved to a different file by compaction, and still answer to the same identity, with a marker saying when it last changed. That is a property databases have and data lakes did not. It removes the argument for bolting a change data capture system onto the side of a table that already knows what changed.
There is still work to do around that identity. A WHERE clause finds updates and inserts. Deletes are invisible to it. The built-in changelog view cannot read a merge-on-read table. And an upgraded table hands out its first ids in an order that will surprise you. All three are the kind of thing a book written from the specification would not mention, because the specification does not have them. They are what the installed version does in August 2026.
The capstone puts the mechanisms together on one bookshop lakehouse: loading, evolution, deliberate fragmentation, maintenance and queries from four engines. Watching the state change across that workflow is the final check on how the parts fit.
Comments