The Firehose and the File Count

Streaming is where an Iceberg table stops being something you load and starts being something that is always moving. It is also the one chapter in this book where the lab could not reach the whole subject. So before anything else, here is exactly which parts of what follows were executed.

What was run, and what was not

This chapter is not uniformly verified — and I would rather you knew that in the first paragraph than worked it out in the fourth.

Executed on this machine, against Iceberg 1.11.0, Spark 4.1.3 and the apache/iceberg-rest-fixture:1.10.1 REST catalog: a real streaming write from Spark Structured Streaming, the file and metadata counts it produced, and the compaction that fixed them. Also executed: a streaming read out of an Iceberg table, including the error it raises on a table that has ever been merged into.

Not executed, and marked as such at every claim: everything about Apache Flink and everything about the Kafka Connect Iceberg sink. This lab has neither. Standing up a Flink cluster next to the existing containers was not possible inside a Docker VM with 2 CPUs and about a gigabyte free. What you get on those two is architecture, mechanism, and the reasoning behind their design. There is no pasted output, because there is no output to paste.

Also not executed: an equality delete file. That is the mechanism streaming upserts are built on, and it is the single most important idea in this chapter. Spark SQL does not produce equality deletes, so I could not make one. What I could do is show three things: that the machinery for them is present in the Iceberg runtime, that every table already counts them, and that one specific operation you would want is missing. That section says exactly which parts are jar inspection rather than execution.

The reason for the separation is the standard this book holds itself to, and streaming is where that standard costs something. A chapter that is mostly architectural is fine. A chapter that looks executed and is not would be worse than no chapter at all. A reader cannot calibrate against a claim they were never told was untested.

What streaming into a table format even means

A stream may deliver rows continuously, but the table cannot expose each row as it arrives. A commit to an Iceberg table is an atomic swap of a metadata pointer, and swapping a pointer per row would be absurd. What a streaming writer actually does is micro-batch and commit: accumulate records for some interval, write them as data files, make one commit. The “stream” is a sequence of ordinary Iceberg commits arriving quickly.

Accumulating rows between commits determines the latency and storage costs:

• Your latency floor is your commit interval. Data is invisible until its commit lands. A two-second trigger means readers are at least two seconds behind, and no amount of tuning changes that. The alternative is a table with no atomic version, which is the thing chapter 1 spent itself arguing against.
• Your file count is your commit rate times your parallelism. Each commit writes at least one file per writing task — and that number does not depend on how much data actually arrived.
• Your metadata volume is your commit count. Every commit writes a new metadata.json, a new manifest list and at least one manifest, regardless of whether it carried a million rows or nothing.

Chapter 11 called this the small-files problem and chapter 12 fixed it. Streaming is where it stops being a scenario you can construct and becomes a property of your architecture. So the useful question is not “can Iceberg do streaming”. It obviously can; that is what a commit is. The question is what a table looks like after a day of it, and what has to be running to keep it usable.

The short Spark run below makes those costs visible before extrapolating them to a day.

Twenty-four seconds of traffic

This section was executed. Spark Structured Streaming has an Iceberg sink in the same runtime jar the rest of this book uses. No extra dependency, no separate service.

src = spark.readStream.format("rate").option("rowsPerSecond", 20).load()
q = (src.selectExpr("value AS order_id",
                    "concat('cust-', CAST(value % 7 AS STRING)) AS customer",
                    "timestamp AS ordered_at")
       .writeStream.format("iceberg").outputMode("append")
       .trigger(processingTime="2 seconds")
       .option("checkpointLocation", "/tmp/ck17")
       .toTable("ice.ch17.orders_stream"))

Twenty bookshop orders a second, committed every two seconds. I let it run for roughly twenty-four seconds and stopped it. This is a deliberately gentle load. Twenty rows per second is nothing, and a 2-second trigger is relaxed by streaming standards, which is exactly what makes the numbers below worth looking at.

rows      : 440
snapshots : 12
data files: 22

Every commit in the chain:

operation  added  added-data-files  added-files-size  total-records
append     NULL   NULL              NULL              0
append     40     2                 2478              40
append     40     2                 2488              80
append     40     2                 2467              120
append     40     2                 2468              160
append     40     2                 2467              200
append     40     2                 2467              240
append     40     2                 2491              280
append     40     2                 2474              320
append     40     2                 2474              360
append     40     2                 2494              400
append     40     2                 2474              440

Read the first row. The stream committed an empty snapshot before any data arrived: an append operation with zero records and no files. The stream’s first trigger fired before the rate source had produced anything, and Iceberg dutifully recorded the fact. It is harmless, and it is also a real snapshot in the chain. It occupies a metadata.json and a manifest list of its own, exactly like one that carried data.

Every subsequent batch is identical in shape: 40 records, 2 data files, about 2,470 bytes. Two files because the session runs local[2], so two tasks each wrote their share. Hold on to that relationship. Files per commit is a function of write parallelism, not of data volume. A cluster with 200 executors writing every two seconds produces 200 files every two seconds, whether the batch holds four hundred rows or four.

Here is what those files look like individually:

record_count  file_size_in_bytes
20            1233
20            1233
20            1233
...           ...
20            1247

Twenty-two files, every one of them twenty rows and roughly 1.2 kilobytes. At that size the word “file” is doing more work than the contents. Average 1,238 bytes. Total data on disk: 27,242 bytes.

The number that should worry you

Now count the metadata.

data     : 22 files,  27,242 bytes
metadata : 36 files, 243,879 bytes

The metadata is nine times the size of the data. Twenty-two Parquet files holding 440 rows, against thirty-six metadata files describing them. That is thirteen metadata.json documents and twenty-three Avro manifests and manifest lists.

Chapter 1 measured a version of this ratio on a three-hundred-row toy table and warned you not to over-read it. Metadata size tracks files and snapshots rather than rows. That warning still applies, and streaming is precisely the case where it stops being reassuring. A batch job that appends once an hour writes 24 commits a day. This stream, at its gentle two-second trigger, writes 43,200 commits a day.

That last figure is arithmetic and not a measurement — I ran the stream for 24 seconds, not for 24 hours. But the arithmetic is not complicated, and it is the whole reason chapter 12 is not optional on a table like this:

Per day, at a 2-second trigger and 2 writing tasks
Commits (and therefore snapshots)	43,200
metadata.json files written	43,200
Manifest lists written	43,200
Data files written	86,400
Rows	1,728,000
Data bytes, at the measured 1,238 per file	~107 MB

A hundred megabytes of orders, in eighty-six thousand files, described by a hundred and thirty thousand metadata files. Every query against that table has to plan over the current manifest set before it reads a single row. And every one of those files is a separate object-store request, billed and latency-bearing. This is how a table that is technically correct becomes operationally unusable inside a week.

None of that is an Iceberg defect. It is what committing every two seconds means in any system. The format is at least honest enough to let you count it in advance rather than discover it.

Compaction, on the same table

This was executed. Chapter 12’s procedure, run against the streaming table exactly as it was left:

CALL ice.system.rewrite_data_files(table => 'ch17.orders_stream',
                                   options => map('min-input-files','2'));

{'rewritten_data_files_count': 22, 'added_data_files_count': 1,
 'rewritten_bytes_count': 27242, 'failed_data_files_count': 0,
 'removed_delete_files_count': 0}

data files after: 1
rows after      : 440
file sizes after: [(440, 3397)]

Twenty-two files became one, and the row count is unchanged — which is the only correctness claim that matters here.

The byte count is the interesting part. Those 22 files held 27,242 bytes of data; the single compacted file holds 3,397 bytes. The same 440 rows, in an eighth of the space. Nothing was dropped. That is Parquet’s per-file overhead, paid twenty-two times over. Footers, schema, column metadata and dictionary pages cost roughly the same whether a file holds twenty rows or two million. So a table of tiny files is mostly not data.

One thing not to misread. Immediately after compaction the directory has grown:

data on disk after compaction: 23 files, 30,639 bytes

The 22 old files are still on disk. They are no longer in the table and no query will read them. They remain because older snapshots still reference them. Reclaiming that space is expire_snapshots, from chapter 13, and it is a separate decision with its own retention trade-off. Compaction changes what the table points at; expiry deletes what nothing points at any more.

For a streaming table both have to be scheduled, forever. They are a routine part of operating the pipeline, not something you reach for when a query gets slow. A streaming ingest without a maintenance job is not a finished design. It is a design with a deadline.

Flink

Nothing in this section was run. There is no Flink in this lab — no cluster, no job, no output. What follows is the architecture and the reasoning, kept deliberately to structural claims rather than API details. API details written from documentation are exactly what this book’s verification protocol exists to keep out.

Flink is the engine most Iceberg streaming deployments actually use, and the reason is a genuine architectural fit rather than fashion. Spark Structured Streaming is a micro-batch system that presents a streaming API. Flink is a true streaming dataflow engine with checkpointing built into its core. The two things line up neatly: a Flink checkpoint is a natural commit boundary. Flink already needs to periodically snapshot the state of the whole job for fault tolerance, and Iceberg already needs an atomic commit point. Wiring the second to the first gives exactly-once semantics almost for free. A checkpoint that succeeds commits its files. A checkpoint that fails re-runs, leaving its uncommitted files as orphans for chapter 13 to sweep up.

The structural consequences that follow are the ones worth carrying, and they hold regardless of API:

• Your commit interval is your checkpoint interval. Everything in the arithmetic above applies, with the checkpoint interval in place of the trigger. Flink does not have a different relationship to file count than Spark does — it has the same one, reached by a different route.
• A single Flink job commits from one coordinator, not from every parallel writer independently. Iceberg commits are compare-and-swap against a catalog, and N parallel writers racing on one table would spend their time losing to each other. The parallel tasks write files; one place commits them. Any streaming Iceberg writer, on any engine, has to do something of this shape.
• Flink is the writer that produces equality deletes, which is the next section and the reason it can do upserts that Spark SQL cannot.

I am not going to tell you what the Flink connector’s option names are, or which Flink and Iceberg versions pair with which, because I did not run it. Chapter 18 is about what happens to people who take version compatibility from a page instead of a build. Flink’s Iceberg support has moved in every recent release.

Equality deletes, and why streaming needs them

This is the most important mechanism in the chapter, and it is also the one I could not produce. Here is the honest split, stated before the explanation.

Established by execution: nothing. I did not create an equality delete file, because the only writers that emit them are streaming writers and this lab has none.

Established by inspecting the Iceberg 1.11.0 Spark runtime jar: the classes exist and are shipped.

Established by reading the spec: the definitions and the v2-versus-v3 behaviour.

Chapter 10 introduced position deletes: a delete file that says row 47 of file X is gone. That encoding is efficient, and it requires something a streaming writer simply does not have. To write “row 47 of file X”, you must first know that the row you are deleting is at position 47 of file X. Which means finding it, which means reading the table.

A streaming upsert cannot do that. An update for order 12345 arrives, and the writer has no idea which of eighty thousand files holds order 12345. Looking it up would turn a stream into a series of table scans. So Iceberg provides the other encoding — the one that names no file at all:

An equality delete marks rows deleted by value rather than by position. It says “any row where order_id = 12345 is deleted”, and it does not name a file at all.

That is writable without reading anything. A streaming upsert becomes three cheap steps: write an equality delete for the key, write the new version of the row, commit both. The reader is left to reconcile it.

And reconciling it is the cost. A position delete is a cheap bitmap intersection against one known file. An equality delete is a predicate that must be evaluated against every data file the delete could apply to. Sequence numbers bound that set: a delete only applies to data written before it. So a table with many equality deletes has readers doing anti-joins on every scan. And the penalty grows with the number of accumulated delete files rather than with the number of deleted rows, which is a much less forgiving quantity.

That read-time cost raises two questions: what v3 changes, and what maintenance the runtime actually exposes.

One: v3 did not remove equality deletes. This surprises people — and it matters for chapter 19. Deletion vectors replace the position delete encoding only. The spec is explicit that a row can be marked deleted by position, “encoded in a position delete file (V2) or deletion vector (V3 or above)”, or by value, “encoded in equality delete file”. There is no version qualifier on the second. Equality deletes are unchanged in v3, and streaming upsert remains the reason they exist.

	v2	v3
Position deletes	position delete file (Parquet)	deletion vector (Puffin)
Equality deletes	equality delete file	equality delete file, unchanged

Two: the machinery is present, and one obvious operation is not. Listing the classes in iceberg-spark-runtime-4.1_2.13:1.11.0 shows the full writer stack for equality deletes:

org/apache/iceberg/deletes/EqualityDeleteWriter.class
org/apache/iceberg/io/ClusteredEqualityDeleteWriter.class
org/apache/iceberg/io/RollingEqualityDeleteWriter.class
org/apache/iceberg/DeleteFileIndex$EqualityDeletes.class
org/apache/iceberg/spark/source/EqualityDeleteRowReader.class

So a reader for them ships with Spark, along with metrics classes that count them (AddedEqualityDeletes, TotalEqualityDeletes). And every snapshot summary in this entire book has been carrying the counter already, quietly. Including the ones from a plain batch append:

'total-equality-deletes': '0'

That field is in every table because the format expects some writer, somewhere, to be producing them.

Now the gap. There is an action interface for converting equality deletes into position deletes, which is the maintenance operation you would obviously want on a table an upsert stream has been writing to:

org/apache/iceberg/actions/ConvertEqualityDeleteFiles.class
org/apache/iceberg/actions/ConvertEqualityDeleteStrategy.class

Both are in the jar. Neither has a Spark implementation behind it. The spark/actions/ package holds RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction, with no equivalent for equality deletes. The 20 stored procedures listed in chapter 16 include rewrite_position_delete_files and nothing that converts equality deletes.

That is a jar listing, not a test. I did not attempt the conversion and get an error — I looked for an implementation and did not find one. So take it as a strong hint, not as a statement about what Spark will do when you call it. Check your own engine’s capabilities before designing a maintenance plan around that step.

What is not in doubt is the read cost. If a streaming upsert is writing equality deletes into your table, compaction is not a housekeeping task. It is the thing that converts an unbounded read penalty back into a bounded one. Its schedule is a correctness-adjacent decision about query latency, not a disk-space one.

Kafka Connect

Nothing in this section was run either. The Iceberg project ships a Kafka Connect sink connector. It is the right answer for a specific and common shape of problem: topic in, table out, no transformation, no engine.

The reason it exists as its own connector rather than as “use Kafka’s generic sink” is worth understanding, because it is the same problem Flink solves with checkpoints. Kafka Connect distributes a sink across worker tasks, each consuming its own partitions. If every task committed to Iceberg independently, they would collide on the catalog’s compare-and-swap and most of them would lose. So the connector runs a coordinator. Workers write data files and report them; one coordinator collects those reports and makes a single Iceberg commit per interval. Consumer offsets are committed alongside, so a restart resumes at the right place and does not double-write.

That structure gives the same three consequences as everything else in this chapter. Commit interval sets latency. Commit rate times task count sets file count. And the table needs the same maintenance as any other streaming target.

Kafka Connect is the low-operational-cost option when the work genuinely is “land this topic in this table”. It is the wrong tool the moment you want joins, windows or enrichment, because those are Flink’s job and the connector does not do them. Since I have not run it, treat the architecture above as the design intent. Then go and verify the specifics — connector version, Iceberg version, catalog configuration — against your own cluster.

Streaming out of a table

This section was executed, and it produced the sharpest result in the chapter.

An Iceberg table can be a streaming source, not just a sink. Spark reads the snapshot chain incrementally, treating each new snapshot as a new batch. That turns any Iceberg table into a change feed for free.

Against the append-only streaming table from earlier:

spark.readStream.format("iceberg").load("ice.ch17.orders_stream")

streaming read from the append-only table -> rows: 440

All 440 rows, streamed out of the table they were streamed into. Now try the same thing against the table chapter 16 ran MERGE INTO on:

java.lang.IllegalStateException: Cannot process overwrite snapshot: 7584038833222273539,
to ignore overwrites, set streaming-skip-overwrite-snapshots=true

The query terminated. Not degraded and not incomplete — terminated with an exception. And the table is completely healthy: every other engine in chapter 15 reads it without a murmur.

That is a real architectural limit rather than a rough edge. An incremental streaming read wants to emit new rows. A snapshot produced by MERGE INTO or by a copy-on-write UPDATE is an overwrite: it contains rewritten copies of rows that already existed. Emitting them would duplicate records downstream. Skipping them, which is what the suggested option does, means the downstream consumer silently never sees those updates.

Neither behaviour is correct, so Spark refuses to choose and stops instead. Put that next to chapter 16’s batch incremental read, where the identical situation produces the opposite policy. There, the overwrite snapshot is silently skipped and the query returns a partial answer with no error at all. Same engine, same table shape, two different decisions — and the loud one is the one you want. The error message offers you the second option, and you should read it as a warning rather than a fix: streaming-skip-overwrite-snapshots=true does not make the stream handle updates. It makes the stream ignore them.

A downstream stream therefore constrains what writes its source table can accept:

A table that is a streaming source should be append-only. If it is also merged into, the stream is either broken or lying.

If you need both, split them: a raw append-only table that streams downstream, and a merged table built from it. That is a familiar shape, and the same reason a warehouse keeps its landing tables separate from its curated ones. Iceberg does not exempt you from it.

Putting it together

The measured costs connect five pipeline decisions:

1. Pick the commit interval from the latency you need, and then accept its file count. These are the same number. Ten seconds instead of two is five times fewer files and eight more seconds of lag. No setting anywhere gives you both ends.
2. Count your writing tasks. Files per commit is parallelism, and it is usually the larger of the two multipliers. Reducing writer parallelism is often a bigger win than lengthening the interval.
3. Schedule compaction from day one, not from the first slow query. On a streaming table it is load-bearing infrastructure. Chapter 12 has the procedure and chapter 13 has the expiry that reclaims the space.
4. Keep streaming sources append-only, per the previous section.
5. If you are doing upserts, find out what your engine can do about the equality deletes it is accumulating before you are in production with them. The read cost grows with the delete files rather than with the deleted rows.

What this chapter did not run, collected in one place

The untested parts remain outside the evidence from the Spark run:

• Flink. No cluster, no job, no output. The architecture section is reasoning about design, not a report of behaviour.
• The Kafka Connect Iceberg sink. Same.
• Producing an equality delete file. Never created one. The class listing is jar inspection; the v2-versus-v3 table is the spec; the read cost is the mechanism, not a benchmark.
• Converting equality deletes to position deletes. I established that no Spark implementation appears in the jar. I did not call the action and observe a failure.
• Exactly-once semantics under failure. Nothing was killed mid-checkpoint, so the recovery story is described and untested.
• Catalog contention with many concurrent streaming writers. One stream, one table, no race.
• A day of anything. The stream ran for twenty-four seconds. Every per-day figure in this chapter is arithmetic on measured per-batch numbers, and is labelled as such where it appears.

Final thoughts

The thing that surprises people about streaming into a table format is that there is no streaming machinery in it at all. There is no special write path, no append-only log, no separate real-time tier. A streaming writer is a batch writer that commits very often — and every property of the result follows from that one sentence.

Which is why the numbers in this chapter are less about Iceberg than about the shape of the commitment you are making. Twenty-four seconds of a modest stream left a table where the metadata outweighed the data nine to one. A single compaction then put the same 440 rows into an eighth of the bytes. Nothing went wrong in that experiment. That is simply what a two-second commit interval costs. The format’s contribution is that you can measure it precisely in advance instead of discovering it in a postmortem.

So the honest summary is that Iceberg makes streaming ingest possible and does not make it free. What you get is a table that concurrent readers can query correctly while it is being written to, with atomic visibility and full history. What you owe is a maintenance job that never stops — because the same commit that gives you atomicity also gives you a file.

The next chapter is about the other thing this book has been quietly accumulating: the places where the version of the spec, the library, the engine and the catalog do not agree. And what each disagreement looks like when it fails.

Next: Same Spec, Different Year