The Database That Lives Inside Your Program
Why an in-process analytics engine exists at all — what it does that SQLite and Postgres structurally cannot, measured rather than asserted, and the honest list of jobs you should not give it.
DuckDB from the Ground Up
The analytics database that runs inside your process — files as tables, a vectorized engine, nested types, and the embedding story a server-based warehouse cannot tell. Run-verified against DuckDB 1.5.5.
Download as an EPUB 18 chapters — read offline on any e-reader
-
-
Two Installs and a Shell Worth Knowing
Getting DuckDB running in both places you'll use it — the Python package and the standalone CLI — plus the shell's output modes, the pipe-friendly flags, and the one composition rule that catches everyone.
-
One File, One Writer, No Exceptions
The database as an ordinary file — when it actually gets written, how ATTACH turns several files into one query, the lock rule that refuses even a read-only peek, and what survives when the engine version moves under it.
-
The Sniffer Is Usually Right, and That's the Problem
Getting CSV, JSON and Parquet into DuckDB — what type inference decides for you, how to override one column without describing all of them, and the ignore_errors behaviour where count(*) and a SELECT disagree about how many rows exist.
-
A Directory Is a Table
Querying half a million rows of Parquet without loading any of it — globs, hive partitioning that turns a path into a column, reading metadata without touching rows, and the pruning you can watch in the plan.
-
Three Answers to a Directory That Can't Be Trusted
Iceberg, Delta and DuckLake — what a table format adds on top of Parquet, running a REST catalog in one container, reading a format v3 deletion vector, and the append option that silently threw away a hundred rows.
-
What the Engine Refuses to Read
Columnar storage, vectorized execution and morsel parallelism, measured — including a query whose EXPLAIN ANALYZE plan is completely empty, and an index that the planner declined to use in every single case.
-
It Spills Before It Fails
What actually happens at the memory limit — measured spill volumes, why a query that OOMs on 16 threads succeeds on 8, and how to read the error message that tells you it ran out of disk rather than RAM.
-
Four Things Are in the Binary
How DuckDB stays small — what ships statically linked, what downloads silently on first use, the one core extension that refuses to autoload, and a community package that installed a binary for the wrong CPU.
-
The Parts You Stop Writing
DuckDB's own additions to SQL — EXCLUDE and REPLACE on a star, GROUP BY ALL, COLUMNS, QUALIFY, PIVOT, list comprehensions and FROM-first syntax, plus the one that is a performance feature in disguise.
-
Stop Paying the Flattening Tax
LIST, STRUCT and MAP as real types — how to choose between a struct and a map by their failure modes, the zero-index that silently returns NULL, and proof that reading a nested field costs the same as reading a flat column.
-
The Shape You Cannot Write Down
VARIANT, JSON and struct widening — one column holding five different types, why a missing key returns NULL here and a binder error in a struct, and the measurement showing VARIANT is not the fast option.
-
Your Variable Name Is a Table Name
Replacement scans, measured — joining a pandas DataFrame to an Arrow Table to a Polars DataFrame in one query, the live-rebinding behaviour nobody documents, and why fetchall() is fifteen times slower than every other way out.
-
Queries Without Query Strings
The relational API — building a query as Python objects, reading back the SQL it generated, the alias you must set before a join will bind, and the fact that a relation is a plan rather than a result.
-
When the Database Is a Library
Running DuckDB as a component rather than a tool — the thread-safety failure that reports itself as a Python TypeError, cursors, parameter binding, and the resource settings a shared process forces you to choose.
-
One Writer, and What to Build Around It
The single-writer lock measured across real processes — four readers fine, a reader refused while a writer holds the file, and a blocked writer that fails in five milliseconds rather than waiting its turn.
-
The Chapter That Undoes the First One
Three ways past the single-writer lock — a DuckLake whose catalog choice decides everything, DuckDB actually serving queries over TCP to another process, and the managed service this book could not run.
-
Files to Answers
A complete workflow on 7.4 million rows of real, dirty data nobody cleaned for you — queried over HTTP before it was downloaded, including the file named for April 2019 that contains trips from 2008 and 2033.