Compatible Is a Test Result
Five engines, two format versions, one catalog. Every engine writes, every engine reads every other's table, Spark deletes a row from each, and every cell of the matrix is generated rather than typed. Then the same rows are read in three time zones.
Here is a table written by DuckDB and upgraded to format version 3 by Spark. Spark then deleted one row from it as a deletion vector, and all five engines on the platform read it back.
duckdb_v3 | delete by Spark: OK | spark OK | duckdb OK | trino OK | pyiceberg OK | flink OKAnd here is the same catalog, the same session, a view created by Spark over one of those tables, read by the others.
spark: 2
trino: ERR Cannot read unsupported dialect 'spark' for view 'cert.v_spark'
duckdb: ERR Table with name v_spark does not exist! Did you mean "spark_v3"?
pyiceberg: True (view_exists; it cannot execute one)The first result is the multi-engine promise kept in full. A row deleted by one engine with the newest delete mechanism the format has is invisible to four others that share no code with it. The second is the same promise, made about a view instead of a table, kept by exactly one engine — the one that wrote it. Nothing in the first result would have told you the second. That is the whole argument of this chapter: format compatibility is not operational compatibility. The specification says what a table is. Whether the engine in front of you can read the one you have, write the version you need, apply the delete files it finds, and agree with its neighbours about what time it is, are separate questions. The only trustworthy answer to any of them is a test you ran on the versions you run.
What the specification promises, and what it does not
Book 1 spent chapter 15 on four engines reading one table and chapter 18 on the version skew between the spec, the library, the engine and the catalog. Both chapters ended at the same place: an engine’s Iceberg support is whatever that engine’s bundled implementation does, and the spec is the contract they are all trying to meet.
The contract has layers, and an engine can meet some and not others.
Table format version. A v2 table has position and equality delete files; a v3 table adds deletion vectors, row lineage, new types and default values. An engine may read v3 and not write it. Book 1’s chapter 18 found exactly that in PyIceberg — and this chapter finds it again on purpose.
Row-level operations. Reading a table with delete files means merging them on read. There are three kinds now, and an engine can support any subset: position deletes as Parquet files, equality deletes as Parquet files, and v3’s deletion vectors as Puffin files. Chapter 11 shows PyIceberg refusing a table with equality deletes; this chapter shows every engine applying deletion vectors.
Types. The format defines a type system. Each engine maps it onto its own — and the mapping is where timestamps go wrong. A timestamp without a zone and a timestamp with one are different Iceberg types, and an engine’s session time zone decides what each of them prints.
Views and dialects. An Iceberg view stores SQL text tagged with the dialect that wrote it. The catalog will happily store a Spark view and hand it to Trino. What Trino does with it is Trino’s decision, and the refusal above is that decision.
None of these is a defect in Iceberg, and none of them is stable across releases. The pins for this chapter are the platform’s: Iceberg 1.11.0 as the library under Spark 4.1.3, Flink 2.1.3 and Trino 483; PyIceberg 0.11.1; DuckDB 1.5.5; the reference REST catalog on SeaweedFS. Change any one and the matrix below is a hypothesis again.
The harness
The companion keeps the test results in a Markdown file under book-2/compatibility/. A script regenerates it on every version bump; nobody edits the cells by hand. The script tests four parts of the contract that a successful table read alone would leave unanswered.
- Every engine that can create a table creates one per format version, in its own SQL dialect, and writes the same three rows. Each row has an id, a naive timestamp, a zoned timestamp, a date, a decimal, a string and a boolean. Three rows with a decimal sum of sixty, every timestamp
2026-06-01 12:00:00, every session pinned to UTC. - Every engine reads every table and reports the row count, the decimal sum, and the string each timestamp column rendered as. A cell is
OKonly if all four match. - Spark, the platform’s maintenance engine, deletes one row from every table in merge-on-read mode, which produces a position delete file on v2 and a deletion vector on v3. Every engine reads every table again.
- Spark creates a view. Every engine tries to read it.
Five engines and two versions is ten writers, nine tables that exist, and forty-five reads before the delete and forty-five after. Every cell below came out of the script.
Writers
| Writer | v2 | v3 |
|---|---|---|
| Spark 4.1.3 | OK | OK |
| DuckDB 1.5.5 | OK | OK, after Spark upgraded the table |
| Trino 483 | OK | OK, WITH (format_version = 3) |
| Flink 2.1.3 | OK | OK, 'format-version'='3' |
| PyIceberg 0.11.1 | OK | ValueError: Cannot write manifest for table version: 3 |
PyIceberg creates a v3 table without complaint and then refuses to append to it, just as Book 1’s chapter 18 found. Everything else writes both versions. That distinction is easy to miss if certification stops at CREATE TABLE — the harness has to write rows before it can call a writer compatible.
Two things about the writers did not fit in the table. DuckDB’s CREATE TABLE against a REST catalog takes no format-version option. So its v3 table was created by DuckDB, upgraded by Spark with ALTER TABLE … SET TBLPROPERTIES ('format-version'='3'), and then written by DuckDB — which is the question that matters. And Flink’s first two runs failed with a parse error at column 92. A column named dec is DEC, a reserved synonym for DECIMAL in Flink SQL, and needs quoting there and nowhere else. That is a dialect difference of the dullest kind — and it is exactly what a harness exists to find before a pipeline does.
One schema, five dialects
The harness writes the same seven Iceberg types from five SQL dialects, and the DDL it had to generate is itself a compatibility finding. Every name below was run.
| Iceberg type | Spark | Trino | Flink | DuckDB | PyIceberg |
|---|---|---|---|---|---|
long | BIGINT | BIGINT | BIGINT | BIGINT | LongType |
timestamp (no zone) | TIMESTAMP_NTZ | TIMESTAMP(6) | TIMESTAMP(6) | TIMESTAMP | TimestampType |
timestamptz | TIMESTAMP | TIMESTAMP(6) WITH TIME ZONE | TIMESTAMP_LTZ(6) | TIMESTAMPTZ | TimestamptzType |
date | DATE | DATE | DATE | DATE | DateType |
decimal(10,2) | DECIMAL(10,2) | DECIMAL(10,2) | `dec` DECIMAL(10,2) | DECIMAL(10,2) | DecimalType(10, 2) |
string | STRING | VARCHAR | STRING | VARCHAR | StringType |
boolean | BOOLEAN | BOOLEAN | BOOLEAN | BOOLEAN | BooleanType |
The timestamp mappings deserve particular care. Spark’s plain TIMESTAMP is the zoned Iceberg type and its TIMESTAMP_NTZ is the naive one. Trino and Flink spell the zoned one out. DuckDB’s plain TIMESTAMP is the naive one. The same word means the opposite thing in two of the five dialects. A table created by hand in one and read in another carries the type the creator’s dialect chose — not the one the author had in mind. The section on time zones below is what that costs.
Readers, before the delete
| Table | Spark | DuckDB | Trino | PyIceberg | Flink |
|---|---|---|---|---|---|
spark_v2 | OK | OK | OK | OK | OK |
spark_v3 | OK | OK | OK | OK | OK |
duckdb_v2 | OK | OK | OK | OK | OK |
duckdb_v3 | OK | OK | OK | OK | OK |
trino_v2 | OK | OK | OK | OK | OK |
trino_v3 | OK | OK | OK | OK | OK |
flink_v2 | OK | OK | OK | OK | OK |
flink_v3 | OK | OK | OK | OK | OK |
pyiceberg_v2 | OK | OK | OK | OK | OK |
Forty-five cells, forty-five OK. Every engine reads every other engine’s table, at both versions, with the count, the sum and both timestamps agreeing. That includes PyIceberg reading v3 tables it cannot write — and Flink reading tables written by engines it has never heard of.
Readers, after Spark deleted one row from each
| Table | Delete by Spark | Spark | DuckDB | Trino | PyIceberg | Flink |
|---|---|---|---|---|---|---|
| all nine | OK | OK | OK | OK | OK | OK |
Forty-five more OKs, and the reason is in the delete files Spark wrote.
spark_v2: content=1 x1 …-00001-deletes.parquet
spark_v3: content=1 x1 …-00001-deletes.puffin
duckdb_v3: content=1 x1 …-00001-deletes.puffin
trino_v3: content=1 x1 …-00001-deletes.puffin
flink_v3: content=1 x1 …-00001-deletes.puffinOn every v2 table the delete is a position delete in a Parquet file. On every v3 table it is a deletion vector in a Puffin file, content=1 still — because a deletion vector is a position delete in a different container. And every engine on the platform applied both kinds and returned two rows. Book 1’s chapters 18 and 20 found that Spark, DuckDB and PyIceberg all read deletion vectors; this is the same fact for five engines on nine tables, generated.
Which engine can produce each kind of delete is a different question from which can read it, and the platform’s runs so far answer part of it.
| Engine | Copy-on-write delete | Position delete file (v2) | Deletion vector (v3) | Equality delete |
|---|---|---|---|---|
| Spark 4.1.3 | yes (chapter 4) | yes | yes | no: Spark SQL cannot write them |
| Flink 2.1.3 | yes, from an upsert table (chapter 11) | |||
| PyIceberg 0.11.1 | yes (chapter 4’s delete) | cannot write v3 | no | |
| Trino 483, DuckDB 1.5.5 | not run | not run | not run | not run |
The blank and “not run” cells are the honest part of that table, and they are where the next revision of the harness goes.
What the harness did not test is worth stating with the same precision. It did not test equality deletes, because Spark cannot write them; chapter 11’s Flink run did, and PyIceberg refused that table with PyIceberg does not yet support equality deletes. A reader that is OK on every row above can still be the reader that fails on the one table a streaming writer produced. The certification matrix needs a row for that table, and the companion’s next revision adds it from the Flink job.
Views
A successful table read says nothing about whether the same engine can execute a stored view. The Spark view makes that gap visible.
spark: 2
trino: ERR Cannot read unsupported dialect 'spark' for view 'cert.v_spark'
duckdb: ERR Table with name v_spark does not exist! Did you mean "spark_v3"?
pyiceberg: TrueSpark created a view over one of its tables and stored it in the catalog with dialect = spark. Spark reads it. Trino found it, read the dialect tag, and refused with a message that names the problem. DuckDB does not list views from a REST catalog at all, so the name does not exist as far as it is concerned. PyIceberg can confirm the view exists — and do nothing else with it.
Book 1’s chapter 5 said that a view is a weaker guarantee than a table because the stored SQL is tagged with the dialect that wrote it. This is what “weaker” looks like across four engines. A table written by any of them is read by all of them. A view written by any of them is read by itself. The catalog stored it faithfully. Interoperability was never the catalog’s to promise.
The rule: views are per engine until a harness says otherwise. Store them in the catalog for the engine that uses them, and do not build a cross-engine contract on one.
Timestamps, in three time zones
Every session in the harness was pinned to UTC, and every timestamp cell said 2026-06-01 12:00:00. Book 1’s chapter 15 got three different instants from the same wall-clock string across three engines. Book 1’s chapter 12 found the same trap deciding which partition a rewrite scoped to. The harness’s second script is the mechanism, isolated.
table cert.spark_v2, written with ts = tstz = 2026-06-01 12:00:00 in a UTC session
session UTC spark ts=2026-06-01 12:00:00 tstz=2026-06-01 12:00:00
duckdb ts=2026-06-01 12:00:00 tstz=2026-06-01 12:00:00+00
session Europe/Berlin spark ts=2026-06-01 12:00:00 tstz=2026-06-01 14:00:00
duckdb ts=2026-06-01 12:00:00 tstz=2026-06-01 14:00:00+02
session America/Los_Angeles spark ts=2026-06-01 12:00:00 tstz=2026-06-01 05:00:00
duckdb ts=2026-06-01 12:00:00 tstz=2026-06-01 05:00:00-07Same bytes, three sessions. The ts column is Iceberg’s timestamp type, a wall-clock value with no zone, and every session renders it as written. The tstz column is timestamptz, an instant, stored as UTC and rendered in whatever zone the session has. Both engines agree in every row — because both are doing the right thing for the type. Neither result is wrong. A dashboard reading tstz in Berlin and a pipeline reading it in Los Angeles are looking at the same instant and different strings.
Reading the zoned value changes its presentation. Writing a naive literal into that column can change the instant stored, because the writer’s session supplies the missing zone.
now WRITE a naive literal '2026-06-01 12:00:00' into tstz from a Berlin session, read it back in UTC
written in Berlin, read in UTC: ts=2026-06-01 12:00:00 tstz=2026-06-01 10:00:00A naive literal written into a timestamptz column is interpreted in the writing session’s zone. Berlin’s noon is ten in the morning in UTC, and that is the instant the table now holds. Nothing about the literal said Berlin — the session did. That is Book 1’s three instants, explained. Three engines with three default session zones, one naive string, three interpretations, and a row that landed in a different day partition depending on who wrote it.
The rule, and every engine on the platform has the knob for it:
- Instants go in
timestamptz; wall clocks go intimestamp. A row that means “this happened at this moment” is an instant. A row that means “the shop opens at nine” is a wall clock. - Every session that writes is pinned to UTC. Spark:
spark.sql.session.timeZone. DuckDB:SET TimeZone. Trino: thetime_zone_idsession property. Flink:table.local-time-zone. The harness sets all four, and a writer that does not is a writer whose output depends on the machine it ran on. - Never write a naive literal into
timestamptz. Attach the zone, or convert explicitly. The engines will each accept the naive form and each do something defensible with it — and defensible is not the same as the same.
What to decide, and why
Certify before you upgrade, not after. The matrix is a function of the versions in it, and the only cheap moment to run it is before a new engine version touches a production table. The companion’s harness runs in under three minutes on this platform. An upgrade that skips it is an upgrade that finds out from a pipeline.
Certify the operations you use, not the ones the vendor lists. The harness checks creation, insert, read, delete files of both kinds, views and timestamps because those are what this platform does. If yours uses equality deletes from a streaming writer, nested types, or a column type the harness does not carry, add the row. A green matrix that omits your workload is a green matrix about someone else’s.
The maintenance engine’s writes are everyone’s reads. Compaction, expiry and row-level deletes are done by one engine, Spark here, and every other engine has to read what it produces. That is why the delete step exists — it is not testing Spark, it is testing whether four other engines can consume Spark’s maintenance.
Views are not a compatibility layer. Use them inside one engine. The exception is a catalog and engine pair whose harness row says the dialect is read — and this platform has no such pair.
Keep the result, not the conclusion. The file under compatibility/ is dated, generated, and kept. When an engine version changes and a cell flips, the diff is the finding. A remembered conclusion that the engines were compatible cannot tell you that.
What was not run
Nested types, UUIDs, nanosecond timestamps and the v3 variant type were not in the row. Book 1’s chapter 19 found that neither Spark nor DuckDB could query a table with the three spatial and nanosecond types. This harness stayed on the types every engine claims, which is the right baseline and not the whole story. Flink reading the Spark view was not exercised. The Kafka Connect sink, a 1.9.2 Iceberg client on this 1.11.0 platform, was not in the matrix, and chapter 11 says why. And every read here went through the reference catalog; chapters 2 and 3 covered how the production catalogs change the address a reader uses, and nothing else.
Exercises
1. Add a row. Run the Flink upsert script from chapter 11 to produce a table with equality deletes, then add it to the harness as a tenth table and rerun the reader matrix. Which cells change, and what does each engine say?
Show answer
PyIceberg’s cell fails with PyIceberg does not yet support equality deletes; Spark, Trino, DuckDB and Flink read the table with the deletes applied. The matrix now has a column-independent failure that the previous ninety cells could not show, which is the point of adding rows for the writers you actually run.
2. Break the timezone rule and find the row. Write one order through Spark with the session zone set to Asia/Kolkata and a naive timestamp literal into a timestamptz column, into a table partitioned by day. Read the partitions metadata table. Which partition did it land in, and why is that the wrong one?
Show answer
The naive literal is interpreted as Kolkata local time, five and a half hours ahead of UTC, so an evening timestamp lands in the previous UTC day’s partition. The partitions table shows a row in a day the writer did not intend. Pinning the session to UTC, or attaching the zone to the literal, puts it where every other engine will look for it.
Final thoughts
Ninety green cells is the most reassuring thing this book has printed, and the two red ones beside them are the reason to have printed any of it. PyIceberg cannot write v3 and cannot read equality deletes; views are read by their author and nobody else; a naive literal in a timestamptz column means whatever the session says it means. None of those is a surprise to someone who ran the harness — and every one of them is an incident for someone who did not.
These results give the platform a tested set of engine versions and operations. Keeping those tables usable also requires decisions about layout, compaction, metadata and retention. The next chapter starts with the query workload, because the same layout can help one reader and penalise another.
Comments