Everybody Gets the Same Number

Chapter 1 ended with a single line of output that was doing a lot of work:

duckdb over the iceberg table: [(300, 3000.0)]

Three hundred rows, out of an engine that had no part in writing any of them. No export, no load, no copy. That was the promise the format exists to make, and I deferred collecting on it properly for fourteen chapters.

The fuller test uses one bookshop table and one catalog with four engines: PyIceberg, Polars, DuckDB and Spark. Each reads and writes the table; all end up agreeing on the same eleven rows.

They also disagree, in three specific ways, and the disagreements are the more useful half of the chapter. Interoperability is not a boolean. It is a set of behaviours that overlap, and knowing exactly where the overlap ends is the difference between a lakehouse and a support ticket.

Everything below was run against Iceberg 1.11.0, Spark 4.1.3, PyIceberg 0.11.1, DuckDB 1.5.5 and Polars 1.44.1, on Python 3.13 and Java 21. The catalog is the apache/iceberg-rest-fixture:1.10.1 REST catalog that chapter 5 stood up. One engine is missing from that list, and I will say plainly at the end which one and why.

What engine neutrality actually buys you

“Iceberg works with everything” leaves out the conditions that determine whether a client can use your table. The useful claim is narrower:

An engine that implements the Iceberg spec at the version your table is written in, and that can reach your catalog, can read and write your table with no coordination with whatever wrote it.

Every clause there is load-bearing, and each one is a failure mode. The version clause is chapter 18. The catalog clause is why chapters 4 and 5 came first. Without a shared catalog you do not have one table that four engines see; you have four engines guessing at a directory. The last clause is the payoff: no schema registration, no export job, no nightly sync, no “the Spark team owns that table.”

What that buys, concretely, is the ability to stop moving data between systems to use different tools on it. The bookshop’s orders live in one place. Spark does the heavy nightly merge, because it is the engine that can. A Python service reads the same table with PyIceberg to answer a support query, without a JVM anywhere in its container. An analyst points DuckDB at it from a laptop. None of these is a copy — which is the whole point, because a copy is a thing that drifts.

Be equally clear about when this is not worth having. If exactly one engine ever touches your data, engine neutrality is a cost with no benefit. You are paying for a metadata layer and a catalog to solve a problem you do not have. The same goes for tables small enough that “export a Parquet file” is a five-second operation nobody minds repeating. Multi-engine access earns its keep when the alternative is a pipeline whose entire job is copying rows between systems — and where the copy is the thing that breaks at 3am.

Three ways an engine reaches an Iceberg table

The four engines reach the table by different routes. That determines whose Iceberg capabilities — and limits — each one inherits.

A native implementation. The engine has its own code for parsing table metadata, planning a scan, and writing manifests. Spark has one, through the iceberg-spark-runtime jar. DuckDB has one, in its iceberg extension. Trino, Flink, Snowflake and BigQuery each have one. A native implementation can be ahead of or behind the reference libraries on any given feature, which is exactly why chapter 18 exists.

A library, wrapped. The engine does not implement Iceberg at all. It calls PyIceberg, gets Arrow back, and works on that. This is what Polars does. It is also what a great deal of Python code does without noticing. The consequence: Polars inherits PyIceberg’s capabilities exactly, including its limits.

A handoff. Nothing implements Iceberg on the engine’s side. Something else reads the table, produces Arrow, and the engine consumes the Arrow. This is what chapter 1 did when it pointed DuckDB at a PyIceberg scan. It always works, at the cost of materialising the result somewhere first. So it does not scale to tables that do not fit in memory.

Keep the middle one in mind. “Polars reads Iceberg” is true, and what it means is “PyIceberg reads Iceberg and hands Polars the rows.”

	Reaches the table by	Needs a JVM	In this chapter
Spark 4.1.3	native (iceberg-spark-runtime)	yes	reads and writes
DuckDB 1.5.5	native (iceberg extension)	no	reads and writes
PyIceberg 0.11.1	it is the library	no	reads and writes
Polars 1.44.1	wraps PyIceberg	no	reads and writes
Trino	native	yes	not run (last section)

The table

Spark creates it, because in this book Spark is the writer that can do everything. Chapter 10 is where that stopped being optional. The bookshop’s orders, partitioned by day. The columns are a narrower set than chapter 3’s eight, and amount is a decimal rather than a double on purpose. The point of this chapter is whether four engines agree on types, and a decimal is where they most often do not:

CREATE TABLE ice.ch15.orders (
    order_id BIGINT, customer STRING, book STRING, status STRING,
    amount DECIMAL(8,2), ordered_at TIMESTAMP)
  USING iceberg
  PARTITIONED BY (days(ordered_at));

Two inserts, six rows then two more. Here is the snapshot chain afterwards, with the summary abbreviated:

operation  added-data-files  added-records  added-files-size  total-records
append     3                 6              5755              6
append     1                 2              1799              8

And the files:

partition      record_count
{2026-07-01}   3
{2026-07-02}   2
{2026-07-03}   2
{2026-07-03}   1

Eight rows, four data files, three days of partitions from two statements. If the partition assignment looks slightly wrong to you, hold that thought. It is wrong in an interesting way, and it is the last technical section of this chapter.

PyIceberg: the microscope

PyIceberg has been in this book since chapter 2, and its role has quietly changed. It is no longer the engine. It is the instrument you reach for when you want to know what just happened to a table, because it starts in a hundred milliseconds and needs no JVM.

from pyiceberg.catalog.rest import RestCatalog
cat = RestCatalog("ice", uri="http://localhost:8181", warehouse="warehouse")
t = cat.load_table("ch15.orders")

format-version : 2
schema         : ['order_id', 'customer', 'book', 'status', 'amount', 'ordered_at']
current snapshot: 4724950649707989917
rows           : 8
shipped only   : 4
projection     : ['order_id', 'amount']
sum(amount)    : 377.59

The shipped only and projection lines are what a scan is actually for. t.scan(row_filter="status = 'shipped'") pushes the predicate down to file pruning and then into Parquet row groups. selected_fields=("order_id","amount") reads two columns off disk instead of six. Neither is Python looping over rows. The work happens in Arrow and in the Parquet reader.

What comes out the other end is Arrow, and PyIceberg will hand that Arrow to whatever you like:

scan outputs: ['to_arrow', 'to_arrow_batch_reader', 'to_duckdb', 'to_pandas', 'to_polars', 'to_ray']

Six exits, and exactly one of them is Iceberg-specific. This is the handoff pattern from the previous section, packaged. t.scan().to_duckdb("orders") returns a live DuckDB connection with the scan already registered as a table:

to_duckdb(): [(10,)]

That is a legitimate way to use DuckDB against Iceberg and it was the only way for a long time. It is no longer the best way, which we get to shortly.

The one to notice in that list is to_arrow_batch_reader, which streams batches instead of materialising the whole scan. Everything else builds the full result in memory first. On a table of any size, the difference between to_arrow and to_arrow_batch_reader is the difference between a working script and a dead kernel. The microscope is a microscope — not a cluster, and not a substitute for one.

Polars: PyIceberg wearing a dataframe

Polars reads Iceberg through pl.scan_iceberg, which takes the PyIceberg table object directly:

import polars as pl
lf = pl.scan_iceberg(t)
df = (lf.filter(pl.col("status") == "shipped")
        .group_by("customer").agg(pl.col("amount").sum().alias("total"))
        .sort("customer").collect())

shape: (4, 2)
┌──────────┬───────────────┐
│ customer ┆ total         │
│ ---      ┆ ---           │
│ str      ┆ decimal[38,2] │
╞══════════╪═══════════════╡
│ ada      ┆ 42.00         │
│ alan     ┆ 38.25         │
│ edsger   ┆ 31.00         │
│ grace    ┆ 42.00         │
└──────────┴───────────────┘

It is a LazyFrame, so the filter and the projection are known before anything is read, and Polars pushes both into the PyIceberg scan. The developer experience is excellent. Remember the mechanism underneath it: there is no Polars implementation of the Iceberg spec. Everything Polars can do with an Iceberg table, PyIceberg can do — because PyIceberg is the thing doing it.

Notice the decimal width in that output. Polars widened decimal(8,2) to decimal[38,2] on the way in. Harmless for a sum, and not harmless at all later in this chapter.

DuckDB: attached to the catalog, not to the files

The DuckDB book in this catalog got as far as pointing DuckDB at an Iceberg table directory, and recorded two findings worth reading before you try this. iceberg_scan on a PyIceberg-written directory fails outright. And COPY … TO … (FORMAT iceberg, append true) silently replaces the table rather than appending to it, because DuckDB accepts unknown COPY options without complaint. A hundred rows in, fifty rows out, exit zero. That book also left an open item: ATTACH … (TYPE ICEBERG) against a REST catalog stopped at an error demanding an OAuth secret.

That open item has a one-clause answer, and the error message contains it. The missing key is AUTHORIZATION_TYPE:

INSTALL iceberg; LOAD iceberg;
ATTACH 'warehouse' AS ice (TYPE ICEBERG,
                           ENDPOINT 'http://localhost:8181',
                           AUTHORIZATION_TYPE 'none');

With that one clause, DuckDB stops being a file reader and becomes a catalog client. The difference is total:

┌──────────┬───────┬───────────────┐
│  status  │   n   │     total     │
│ varchar  │ int64 │ decimal(38,2) │
├──────────┼───────┼───────────────┤
│ placed   │     3 │        199.35 │
│ returned │     1 │         24.99 │
│ shipped  │     4 │        153.25 │
└──────────┴───────┴───────────────┘

Eight rows, and the three group totals sum to PyIceberg’s 377.59 exactly. It reads the snapshot chain too:

[(2, 4724950649707989917, '.../snap-4724950649707989917-1-3357bee5-....avro'),
 (1, 5689416249657599062, '.../snap-5689416249657599062-1-accc83c9-....avro')]

Sequence numbers, snapshot ids and manifest list paths, from iceberg_snapshots('ice.ch15.orders'). This is not a Parquet reader that has been taught to glob a directory. It is an Iceberg client that happens to be 40 megabytes and start instantly.

AUTHORIZATION_TYPE 'none' is right for a local fixture and wrong for anything with users. Polaris, Lakekeeper and Glue all want real credentials, and chapter 5 covered what those look like. The extension speaks the REST protocol; the auth mode is a parameter, not a limitation.

Spark

Nothing surprising here, which is itself the point. Spark reads the table it wrote:

spark     : 8 | shipped: 4

And prunes on the hidden partition without being told the partition column exists:

SELECT count(*) FROM ice.ch15.orders WHERE ordered_at >= TIMESTAMP '2026-07-04 00:00:00';

2

That query names a timestamp, not a day bucket, and chapter 8 is the reason it still prunes. It bears restating here for one reason. Every engine in this chapter gets that behaviour from the table’s own metadata — not from anything the query author knew, and not from a convention the four of them agreed on.

Four writers, not four readers

The usual framing gives Spark the writes and everyone else the reads. On these versions, all four engines can commit to this table. I ran each against it in turn, adding orders 9, 10 and 11 after Spark’s initial load.

PyIceberg appends order 9:

t.append(pa.Table.from_pylist([{...}], schema=t.schema().as_arrow()))

rows after pyiceberg append: 9

DuckDB inserts order 10, through the catalog:

INSERT INTO ice.ch15.orders VALUES
  (10,'lynn','Programming Pearls','shipped',34.50, TIMESTAMP '2026-07-04 15:45:00');

duckdb rows: [(10,)]

That is a real commit. It grows the table rather than replacing it, and it produces a genuine snapshot in the chain. Which inverts the DuckDB book’s directory-COPY finding completely. Pointed at a directory, DuckDB’s Iceberg write is a bulk export wearing an append’s clothes. Pointed at a catalog, DuckDB is a transactional Iceberg writer.

Polars writes order 11, and gets refused:

df.write_iceberg(t, mode="append")

ValueError | Mismatch in fields:
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃    ┃ Table field                        ┃ Dataframe field                    ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ✅ │ 1: order_id: optional long         │ 1: order_id: optional long         │
│ ✅ │ 2: customer: optional string       │ 2: customer: optional string       │
│ ✅ │ 3: book: optional string           │ 3: book: optional string           │
│ ✅ │ 4: status: optional string         │ 4: status: optional string         │
│ ❌ │ 5: amount: optional decimal(8, 2)  │ 5: amount: optional decimal(38, 2) │
│ ❌ │ 6: ordered_at: optional            │ 6: ordered_at: optional timestamp  │
│    │ timestamptz                        │                                    │
└────┴────────────────────────────────────┴────────────────────────────────────┘

That refusal is the format working. A default Polars dataframe carries decimal(38,2) and a naive timestamp; the table wants decimal(8,2) and a timestamptz. Neither side is being unusual. Nothing about the write was unreasonable, and it was rejected anyway — with a field-by-field diff, field ids attached, and a tick or a cross against every column.

Put that next to chapter 1. There, a directory of Parquet accepted a renamed column without a word, and left one logical column split across two physical ones for somebody to find weeks later. The schema is the table’s, the writer answers to it, and the error arrives at write time rather than at query time.

Declare the types and it goes through:

schema={"amount": pl.Decimal(8,2), "ordered_at": pl.Datetime("us", time_zone="UTC"), ...}

polars append -> rows now 11

PyIceberg made the commit on Polars’ behalf. That is also why the rejected write came back as a PyIceberg schema comparison: the wrapper inherits the library’s validation as well as its write path.

And all four agree on the result:

pyiceberg : 11 | shipped: 5
polars    : 11 | shipped: 5
duckdb    : 11 | shipped: 5
spark     : 11 | shipped: 5

Four engines, three separate implementations of the spec, five commits, one answer. That is the thesis of the format in six lines of output.

Who wrote what, and why the table half-remembers

Every snapshot carries a summary map, a plain string-to-string dictionary, and engines put whatever they like in it. Ask the table who wrote each commit:

seq=1 append    added=6   engine=spark
seq=2 append    added=2   engine=spark
seq=3 append    added=1   engine=(not recorded)
seq=4 append    added=1   engine=(not recorded)
seq=5 append    added=1   engine=(not recorded)

Spark records its provenance in detail. Here is a full Spark summary:

{'spark.app.id': 'local-1787902561956', 'manifests-created': '1', 'manifests-kept': '1',
 'manifests-replaced': '0', 'added-data-files': '1', 'added-records': '2',
 'added-files-size': '1799', 'changed-partition-count': '1', 'total-records': '8',
 'total-files-size': '7554', 'total-data-files': '4', 'total-delete-files': '0',
 'total-position-deletes': '0', 'total-equality-deletes': '0',
 'engine-version': '4.1.3', 'app-id': 'local-1787902561956', 'engine-name': 'spark',
 'iceberg-version': 'Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)',
 'app-name': 'ch15_setup'}

Engine name, engine version, the exact Iceberg build down to its commit hash, and the Spark application id. If a bad row shows up in this table, that is enough to find the job that wrote it. More than most warehouses can say.

PyIceberg records the file and record counters and nothing else:

{'added-files-size': '2499', 'added-data-files': '1', 'added-records': '1',
 'changed-partition-count': '1', 'total-data-files': '5', 'total-delete-files': '0',
 'total-records': '9', 'total-files-size': '10053', 'total-position-deletes': '0',
 'total-equality-deletes': '0'}

DuckDB records less still — and a slightly different set:

{'total-position-deletes': '0', 'total-delete-files': '0', 'total-records': '10',
 'total-data-files': '6', 'added-data-files': '1', 'deleted-data-files': '0',
 'deleted-records': '0', 'added-records': '1'}

No added-files-size, no changed-partition-count, but it does volunteer deleted-data-files: 0.

The lesson is not that DuckDB and PyIceberg are sloppy. It is that snapshot summary keys are advisory, not a schema. Only a small set is required, and provenance is not in it. So do not build an audit story on engine-name being present. And do not write a dashboard that groups commits by writer unless you have checked every writer you have. In a multi-engine table you will get nulls, and they will come from the engine you were least expecting.

There is a corollary for capacity planning. total-files-size is optional too, and DuckDB omitted it. Any monitoring that reads table size out of the latest snapshot summary will show a hole exactly when a non-Spark engine made the most recent commit. Read it from the files metadata table instead, which is derived from the manifests and cannot be left out.

Where they actually disagree

Three engines gave the same eleven rows. Now look at what they each wrote into one column.

ordered_at is an Iceberg timestamptz, which stores an instant. Ask PyIceberg to print the raw stored values:

arrow type: timestamp[us, tz=UTC]
    1 2026-07-01 16:14:00+00:00     <- Spark wrote '2026-07-01 09:14:00'
    6 2026-07-03 02:51:00+00:00     <- Spark wrote '2026-07-02 19:51:00'
    9 2026-07-04 10:15:00+00:00     <- PyIceberg wrote datetime(2026,7,4,10,15)
   10 2026-07-04 22:45:00+00:00     <- DuckDB wrote '2026-07-04 15:45:00'

Spark and DuckDB both applied the session timezone, which on this machine is PDT, seven hours behind UTC. 09:14 local became 16:14 UTC; 15:45 local became 22:45 UTC. That is correct behaviour for a timestamp-with-timezone column, and both engines did the same thing.

PyIceberg did not. A naive Python datetime(2026, 7, 4, 10, 15) went in as 10:15 UTC — no conversion, because there was nothing to convert from.

So the same wall-clock string, written through three clients on one machine, lands as three different instants. And nobody here is wrong. timestamptz means “an instant”, and a naive datetime is not an instant. PyIceberg resolves the ambiguity by assuming UTC; the SQL engines resolve it by assuming the session. This is the most likely correctness bug in a multi-engine table, and it is completely silent: no warning, no error, and every engine reads back exactly what was stored.

Now look at what it did to the physical layout. Order 6 was written as 2026-07-02 19:51:00 and lives in partition 2026-07-03:

partition      record_count
{2026-07-01}   3
{2026-07-02}   2
{2026-07-03}   2
{2026-07-03}   1

The partition transform runs on the stored instant, which is UTC. So an evening order on the 2nd is a 3rd-of-July order as far as the table’s layout is concerned. This is not a bug either, and hidden partitioning did not invent it: it is what a UTC-stored timestamp has always done. But this is the moment it becomes visible. In chapter 8 the partition value stopped being a column somebody wrote and became a value the table derives.

Three rules fall out, and all three are cheap:

1. Write timezone-aware datetimes from Python. datetime(..., tzinfo=timezone.utc) costs nothing and removes the ambiguity permanently.
2. Decide once whether your table’s day boundaries are UTC days. If the bookshop reports on Pacific days, partitioning a timestamptz on days() puts every Pacific evening into the next day’s partition, forever. Your daily counts will be quietly wrong at the edges.
3. Reach for timestamp rather than timestamptz when the value genuinely has no timezone, such as a local business date. Then every engine stores what you gave it and none of them convert.

The engine that is not in this chapter

Trino was not run. Anything I could tell you about Trino’s Iceberg connector would be documentation I read rather than behaviour I observed. So this chapter does not tell you any of it.

The reason is boring and specific, and worth showing rather than asserting. This lab’s Docker VM has 2 CPUs and 4.10 GB of memory, and it was already holding the REST catalog plus an unrelated Kubernetes lab:

icerest                 163.3MiB / 3.823GiB
k8s-lab-control-plane   1.149GiB / 3.823GiB
k8s-lab-worker          398.2MiB / 3.823GiB
k8s-lab-worker2         1.056GiB / 3.823GiB
cadence-db               64MiB   / 3.823GiB

Roughly a gigabyte free, against a coordinator that wants a multi-gigabyte heap before it will agree to start. I could have torn down the other containers to make room, and chose not to. The honest version of this chapter is worth more than a fifth engine. The four here were all genuinely executed, and Trino is named as a gap rather than described from a manual.

One thing I will say, because it is structural rather than behavioural: where Trino sits in the taxonomy at the top of this chapter. It is a native implementation, in the same category as Spark and DuckDB. It is what people reach for when they want ANSI SQL over a warehouse without running Spark. Its capabilities on any given Iceberg feature are its own — not Spark’s, and not DuckDB’s. Chapter 18’s version-skew material applies to it exactly as it applies to everything here. Which is the practical point: if you are choosing engines, test yours. Do not inherit the results in this chapter, and do not inherit anyone else’s.

What this chapter did not test

Beyond Trino, four gaps, all deliberate:

• Object storage. Every engine here reached a file:// warehouse. Credential vending, S3 path styles and the failure modes of a signed URL that expires mid-scan are all real, and none of them appears above.
• Concurrent writers across engines. Each write happened after the previous one finished. Two engines committing at the same instant is chapter 4’s compare-and-swap. I did not stage a cross-engine race.
• Scale. Eleven rows. Nothing here measures how these four engines differ on a table where scan planning costs real time, and they differ a great deal.
• Views. The REST catalog advertises view endpoints, and no engine here created one.

Final thoughts

The interoperability promise turns out to be real, and slightly narrower than advertised, in a way that is easy to state.

The data is portable. The behaviour is not. Four independent implementations agreed on eleven rows and five shipped orders. That is the hard part, and the part that used to require an export job and a scheduler. They then disagreed about what to record in a snapshot summary, how wide a decimal is, and what a datetime without a timezone means. None of those disagreements corrupted anything. All of them would have stayed invisible until something downstream produced a number that looked entirely plausible.

So the practical shape of a multi-engine table is four things. One schema that everything answers to. One catalog that everything points at. Explicit types at every write boundary. And a test that reads the table with every client you actually use. That last one takes an afternoon, and it is the only way you find out which of your engines is the one that will surprise you.

Agreement across engines answers the read-and-write question. Publishing a whole batch only after checking it requires a different set of operations: MERGE INTO, branches and tags, and the staged writes in the next chapter.

Next: Nobody Sees It Until You Say So