There Is No ls in Object Storage

The table from chapter 1 holds 50,004 rows. Here it is read two ways by the same engine, DuckDB, against the same object store, with every request passing through a proxy that counts them.

duckdb: count via catalog            9 requests  {'GET metadata': 9}                 50004
duckdb: count via read_parquet glob  36 requests  {'GET data': 35, 'LIST': 1}      194455

The second query is the one every data lake ran for a decade: point the engine at the directory, read every Parquet file under it. It made one LIST call, fetched thirty-five data files, and returned a number nearly four times too large. The directory also holds the inputs to a compaction and the leftovers of a few expired snapshots that maintenance has not yet deleted. The first query made nine requests, read no data files at all, and was right.

The bucket prefix contains more than the current table, and listing it costs a priced API call before returning an answer off by 144,451 rows. That is the production consequence of A Folder Is Not a Table, the first chapter of the first book. Understanding the object store beneath Iceberg explains both the extra requests and the wrong count. Every number here was counted — not estimated.

What a bucket is not

A filesystem has directories, and a directory is a thing: it exists, it can be empty, it can be listed cheaply, and renaming it moves everything under it in one operation. An object store has keys. A key is a string — and the slashes in it are a convention. The “directory” you see in a console is the result of a LIST call with a prefix and a delimiter, computed at request time.

Four consequences follow, and every one of them shows up in Iceberg’s design.

Listing is a request, and it is paginated. ListObjectsV2 returns up to a thousand keys per call. A prefix with a million objects under it costs a thousand calls to enumerate, every time. There is no cached directory entry to consult.

There is no rename. Moving an object is a copy followed by a delete, and moving a prefix is one copy per object. A table layout that depends on renaming a directory into place — which is how Hive’s INSERT OVERWRITE worked — is a table layout that does not work here.

Writes are whole objects. There is no append and no in-place update. A file is written completely or not at all. Above a size threshold the write is a multipart upload: an initiation call, one call per part, and a completion call that makes the object visible.

Every call is priced. On AWS S3 at the time of writing, PUT, COPY, POST and LIST requests are listed at roughly ten times the price of GET and HEAD requests, per thousand. This book did not run against S3, and the tariff is quoted from AWS’s published price list rather than from a bill; chapter 18 puts real numbers on it. What matters here is the shape: writes and listings are the expensive calls, reads are cheap, and none of them is free.

Iceberg’s answer to all four is the same answer: the metadata is the listing. A manifest records every data file’s full path, size and statistics. A manifest list records every manifest. A metadata file records the manifest list of every snapshot. A reader that starts from the current metadata file never has to ask the store what exists — because the table already told it. That single design choice is why the catalog read above made nine metadata requests and zero listings, and why the directory read made a listing and got the wrong answer.

Counting what each operation costs

An nginx reverse proxy between every client and the object store records one line per request: method, key, status, bytes and conditional headers. Each Iceberg operation was run with the log snapshotted before and after, and the difference classified. The resulting counts below are for Spark 4.1.3 with S3FileIO against the reference REST catalog. They let us separate the fixed work of a commit from the requests caused by its data files.

Operation	Requests	Of which	Result
`CREATE TABLE`, empty, partitioned by day	0		the catalog wrote the first metadata file itself
`INSERT` 50,000 rows across 9 day partitions	13	9 `PUT` data, 2 `PUT` metadata, 1 `GET` metadata, 1 `HEAD`	9 files
`SELECT count(*)`	3	2 `GET` metadata, 1 `HEAD`	50,000, no data read
`SELECT` one day, partition pruned	7	4 `GET` data, 2 `GET` metadata, 1 `HEAD`	5,556 rows from one file
`SELECT` one `order_id`	39	36 `GET` data, 2 `GET` metadata, 1 `HEAD`	one row, every file opened
`DELETE` one row, copy-on-write	42	30 `GET` data, 1 `PUT` data, 3 `PUT` metadata, 6 `GET` metadata, 2 `HEAD`	one file rewritten
`INSERT` one row, five times	7 each	1 `PUT` data, 2 `PUT` metadata, 2 `GET` metadata, 2 `HEAD`	five files, five snapshots
`rewrite_data_files`	44	18 `GET` data, 1 `PUT` data, 8 `PUT` metadata, 15 `GET` metadata, 2 `HEAD`	6 files became 1
`expire_snapshots`, all but current	81	41 `GET` metadata, 37 `HEAD`, 3 `POST` multi-delete	7 data files deleted
`remove_orphan_files`, dry run	15	10 `GET` metadata, 4 `HEAD`, 1 `LIST`	0 orphans

A commit has a fixed footprint. Every one-row insert cost exactly seven requests: one data file, one manifest, one manifest list, and the reads needed to know what the current metadata was. The fifty-thousand-row insert cost thirteen, because it wrote nine data files instead of one. The first book measured this as four files per commit on a local disk; this is the same fact priced per request. A pipeline that commits once a minute pays seven requests a minute whether it writes one row or one million — which is why chapter 11 cares so much about commit rate.

CREATE TABLE cost the client nothing, because with a REST catalog the server writes the initial metadata file. That is worth noticing for what it implies. The catalog has its own credentials and its own path to storage, and when something is wrong with either, CREATE TABLE is the first thing that breaks. The reference fixture writes directly; the production catalogs in chapter 3 do too, and their storage configuration is the reason each one has a separate “internal” endpoint.

A count(*) reads no data. Three requests, all metadata. The row count is in the manifests, per file, and the engine summed them. This is the cheapest query an Iceberg table can answer and it is also — as the next section shows — the one query that cannot notice a missing file.

Pruning is only as good as the statistics. The partition-pruned query opened one file. The order_id lookup opened all nine, because the order IDs are interleaved across the day partitions and every file’s minimum and maximum bracket the value being looked for. Column statistics cannot prune what the layout has spread evenly, and no amount of catalog cleverness changes the request count. Chapter 7 is about laying data out so that it can.

Maintenance is where the requests are. Compaction cost 44 and expiry cost 81. Expiry’s cost was almost entirely reading: 41 metadata GETs and 37 HEADs to work out which files no surviving snapshot references, then three bulk deletes. The first book established that expiry is what actually removes bytes. This table establishes that it is also what actually costs requests, and that the cost scales with the number of snapshots and manifests being examined — not with the data. A table with ten thousand snapshots pays for all ten thousand every time it expires.

The only LIST in the table is orphan cleanup, and it is there for the reason orphan cleanup exists. The whole point of the procedure is to find files the metadata does not know about, and the only way to do that is to ask the store. It is also the one operation in the table that did not work the first time.

remove_orphan_files (dry run, Hadoop FS)   FAILED
  org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
remove_orphan_files (dry run, prefix_listing)   15 requests  {'GET metadata': 10, 'HEAD': 4, 'LIST': 1}

The table’s reads and writes go through Iceberg’s own S3FileIO. By default, remove_orphan_files does not. It lists the table location through Hadoop’s filesystem layer, which on this platform has no S3 implementation installed, and the procedure fails with an error that says nothing about orphans. The prefix_listing => true option asks it to list through the table’s FileIO instead, at which point it worked and cost one LIST for a table whose location holds thirty-odd objects. This is the third time in two books that a maintenance procedure has turned out to use a different code path from the table it maintains — and it is not the last.

What the directory read actually did

The directory read at the start made thirty-six requests: one LIST to discover the keys and thirty-five GETs to read files. Most of those files are absent from the table’s current snapshot. The listing was cheap because SeaweedFS returned every key in one page. On a bucket with a million-object prefix it would have been a thousand calls before the first byte of data, and the answer would still have been wrong.

Paths are absolute, and a copy is not a copy

A manifest records each data file as a full URI, such as s3://warehouse/stage5/paths_src/data/ordered_at_day=2026-06-01/00000-4-….parquet. That path is absolute, so copying the manifest to another prefix does not change where it points. On object storage — where moving a directory already means copying every object — this is an easy way to pay for a copy that still depends on the original. The following experiment makes that dependency visible.

Copy every object under a table’s prefix to a new prefix with server-side CopyObject: thirteen objects for a fresh fifty-thousand-row table. Then register the copied metadata.json as a new table and ask where its data files are.

== copied 13 objects to stage5copy/orders/ (server-side CopyObject)
  register_table on the COPIED metadata.json: Row(current_snapshot_id=…, total_records_count=50000, total_data_files_count=9)
  where the copy's data files point: ['s3://warehouse/stage5/paths_src']
  count via the copy: 50000
  sum(amount) via the copy: 3696428.73

The copy registered without complaint, counts correctly, sums correctly — and every one of its data files is the original’s. The copied metadata is a second pointer to the first table’s bytes. Nothing under stage5copy/orders/data/ is referenced by anything. You have a table that looks independent and is not, and the way you find out is by deleting a file from the original.

== deleted one original data file: …/ordered_at_day=2026-06-01/00000-4-27f69d71-ee59-44…
  count(*) via the copy, after the delete (stats only): 50000
  sum(amount) via the copy, after the delete: FAILED
    org.apache.iceberg.exceptions.NotFoundException: Location does not exist:
    s3://warehouse/stage5/paths_src/data/ordered_at_day=2026-06-01/00000-4-27f69d71-….parquet
  sum(amount) via the ORIGINAL, after the delete: FAILED  (the same file)

Deleting the original file broke both tables — but count(*) still reported 50,000 rows. The request counts explain why: that query reads no data files, so it could not notice that 5,556 rows were now missing. A metadata-only query cannot detect a storage-level loss. Chapter 14 adds a periodic column read to chapter 13’s health pack for exactly this reason, and its incident list has a data file missing because a lifecycle rule ran.

The supported way to move a table is rewrite_table_path. It is a Spark procedure that rewrites every absolute path in a table’s metadata from one prefix to another and stages the result, without touching the data. Run on the still-healthy original, before the deletion:

  rewrite_table_path(source -> target): Row(latest_version='00001-024639a0-….metadata.json',
    file_list_location='s3://warehouse/stage5/paths_src/metadata/copy-table-staging-…/file-list',
    rewritten_manifest_file_paths_count=1, …)
     the file list has 13 lines; first: s3://warehouse/stage5/paths_src/metadata/copy-table-staging-…/metadata/00000-….metadata.json<TAB>s3://warehouse/stage5moved/orders/metadata/00000-….metadata.json
     copied 13 listed objects to their target paths
  register the REWRITTEN metadata: Row(current_snapshot_id=…, total_records_count=50000, total_data_files_count=9)
  where the moved table's data files point: ['s3://warehouse/stage5moved/orders']
  sum(amount) via the moved table: 3696428.73

The procedure does three things. It writes a rewritten copy of every metadata file, manifest list and manifest under a staging prefix, with every path translated. It writes a file list: one line per object, source path, tab, target path, covering the staged metadata and the untouched data files. And it returns the name of the newest rewritten metadata file. Your job is the copy the list describes, with whatever tool moves objects in your environment, then a register_table on the rewritten metadata at its new home. After the original’s data file was deleted, the moved table still summed to the cent, because its files are its own.

  sum(amount) via the MOVED table, after the delete: 3696428.73

The rule: never register a copied metadata.json. A table’s metadata is a set of absolute paths, and copying it makes a second reference, not a second table. Chapter 16’s replication procedure needs rewrite_table_path whenever the replica’s bucket has a different name, for this reason, and chapter 4 of the migration book is a whole chapter of it.

There is a second kind of absolute path worth knowing about before chapter 3. Lakekeeper lays tables out under a UUID, s3://warehouse/lakekeeper/01a074af-…/, rather than under the table’s name, so renaming a table renames nothing on storage. Polaris lays them out by name. Neither is wrong, and both are absolute; the difference is what a listing of the bucket tells a human, which is nothing in one case and everything in the other.

Hash prefixes, and why the layout is a property

Object stores scale throughput per key prefix. The details are each vendor’s and change over time, but the historical shape is that many writes to keys sharing a long common prefix contend, and spreading keys across prefixes does not. Iceberg has a table property for it.

CREATE TABLE ice.stage5.orders_hashed (...) USING iceberg
  PARTITIONED BY (days(ordered_at))
  TBLPROPERTIES ('write.object-storage.enabled'='true')

stage5/orders_hashed/data/1010/0110/0010/11101010/ordered_at_day=2026-06-01/00000-29-….parquet
stage5/orders_hashed/data/0110/0011/1000/10010100/ordered_at_day=2026-06-02/00000-29-….parquet
stage5/orders_hashed/data/0111/1011/1010/10010000/ordered_at_day=2026-06-03/00000-29-….parquet

Each data file’s key now begins with a hash, written as four groups of binary digits, before the partition directory. Three files in three partitions landed under three unrelated prefixes. The partition value is still in the path, so a human reading a listing can still tell what a file holds — but a listing is no longer a partition. The table’s manifests are the only thing that knows which files belong to 2026-06-02, and nothing about that changed, because nothing about how Iceberg reads a table involves the path.

That last sentence is the reason this property can be flipped on a table with data in it. Old files keep their old keys, new files get hashed keys, the manifests record both — and no reader can tell. It is also why the request table earlier did not change when this table was queried. The number of GETs is the number of files the manifests select, and the manifests do not care what the keys look like.

Streaming writers and many parallel writers concentrate writes on one prefix, so those are the tables for which this book recommends hashed keys. A table written by one batch job does not have that contention pattern. If people also inspect its layout in a console, keeping the readable keys preserves something useful without giving up a benefit that workload needed.

Conditional writes, and a store that lied

The first book, in chapter 4, flagged one thing about object storage it could not run. S3 gained conditional writes in 2024, PutObject with If-None-Match: *, which is the create-if-absent primitive a catalog-less commit protocol would need. This platform runs on an S3-compatible store, so the question became testable, and then it became two questions.

The first is whether the store honours the condition. Four candidate stores were probed with the same ten calls: bucket creation, put and get, a prefixed listing, a multipart upload, a server-side copy, and four conditional writes.

  Put If-None-Match:* on EXISTING key (want 412)   412   PreconditionFailed   PASS
  Put If-None-Match:* on NEW key (want OK)         OK                         PASS
  Put If-Match wrong etag (want 412)               412   PreconditionFailed   PASS
  Put If-Match right etag (want OK)                OK                         PASS

SeaweedFS 4.45, RustFS 1.0.0-rc.5 and the community fork of MinIO all returned that. Garage 2.3.0 returned this.

  Put If-None-Match:* on EXISTING key (want 412)   OK    {'ResponseMetadata': {'HTTPStatusCode': 200 …   FAIL (want 412)
  Put If-Match wrong etag (want 412)               OK    {'ResponseMetadata': {'HTTPStatusCode': 200 …   FAIL (want 412)

Garage accepted the conditional headers, ignored them, overwrote the object, and returned success. That is the single worst thing a store can do to a table format — because it is indistinguishable from correctness until two writers race. A commit protocol built on “create this key unless it exists” would have both writers succeed, and the second would silently overwrite the first’s metadata file. The store is otherwise capable, actively maintained, and fine for a great many uses. It is not fine for this one, and no documentation would have told you, because the documentation describes what the store intends to support. The probe took two minutes and it is in the companion.

The second question is whether Iceberg uses conditional writes at all on this platform, and the proxy log answers it: the client never wrote a metadata.json. Zero PUTs of a metadata file passed through the proxy in the entire session. With a REST catalog, the server performs the commit, writes the metadata file with its own credentials through its own path to storage, and swaps the pointer in its own database. Whether that write is conditional is the catalog’s implementation detail, and the transaction boundary is where the first book said it was: one conditional swap, in the catalog. Conditional writes on the store make catalog-less designs possible. They do not make the catalog optional for anyone who has one.

One more thing the probe did not see, stated so that it is not assumed — no multipart upload. Every file this platform wrote was under S3FileIO’s multipart threshold, so the request table has no POST initiations or per-part PUTs in it. On a table whose data files are hundreds of megabytes, each file is several requests. The fixed footprint of a commit grows with file size in a way this lab is too small to show.

What was not run

No request in this chapter was made to a cloud store: every count is the proxy’s, against SeaweedFS, and the prices are a published list read once. The four stores’ conditional-write behaviour was probed with single objects, not under contention. And a copy of a table at a size where the copy itself takes minutes, which is where a replication’s ordering starts to matter, is chapter 16’s.

Exercises

1. Price a pipeline. Using the request table, estimate the requests per day for a pipeline that commits one small file every minute to a table that is compacted hourly and expired daily. Then re-estimate with the pipeline committing every ten minutes. Which term dominates, and what does the table say about compaction’s cost as the number of small files grows?

Show answer

Every minute: 1,440 commits at 7 requests is 10,080, plus 24 compactions and one expiry. Every ten minutes: 144 commits is 1,008. Commits dominate at either rate, but the compaction term is not constant: it read 18 data files to rewrite 6, and expiry read 41 metadata objects for 7 snapshots. Both scale with what the commit rate produced. Halving commit rate roughly halves everything, which is chapter 11’s argument in arithmetic.

2. Find the second reference. Register a copied metadata.json as a table the way this chapter did, then run remove_orphan_files on the original with a dry run and prefix_listing => true. Does the procedure list the copy’s data files as orphans? Then run it on the copy. What does each result mean for a real cleanup?

Show answer

The original’s cleanup lists nothing under the copy’s prefix, because orphan cleanup only examines the table’s own location. The copy’s cleanup lists every data file under stage5copy/orders/data/ as an orphan, because the copy’s metadata references the original’s files, not its own. Running that cleanup for real would delete the copy’s data, which nothing referenced anyway, and leave the copy still pointing at the original. The copy is not a table — it is a set of pointers wearing a table’s name.

Final thoughts

Every fact in this chapter comes from the same source: the metadata is the listing. That is why a count reads no data and why a commit has a fixed footprint. It is why the only LIST in the request table belongs to the one procedure whose job is to find what the metadata does not know. It is why a copied metadata file is a second pointer rather than a second table, and why the key layout can change under a table without a reader noticing.

It is also why the catalog matters as much as it does. The metadata is the listing, but the catalog is what says which metadata file is current — and on a REST catalog it is the catalog that writes that file. The next chapter stands up two production catalogs beside the fixture that failed in chapter 1 and hands each of them the same object store. They disagree about the most basic question this chapter raised: what address should a client use to reach the bytes?

Next: The Only Part That Can Say No