When the Payload Is Bigger Than the Heap

Everything works on the sample file. The sample file is four orders. Then a partner sends a nightly export with two million line items, the transform that ran in fifty milliseconds now runs for a minute and then dies with an out-of-memory error, and you learn the hard way how DataWeave reads data by default.

By default it reads all of it. The reader parses the entire payload into the in-memory data model, your expression runs over that fully-materialized structure, and the writer serialises the result. For a four-order file that’s invisible. For a multi-gigabyte one it’s the whole problem. Streaming is how you avoid materialising, and it’s less a switch you flip than a set of constraints you agree to keep.

Deferred, not eager

Start with why streaming is even possible. DataWeave evaluates lazily — an expression’s value isn’t computed until something needs it, and a large collection isn’t held in memory all at once if it can be produced and consumed one element at a time. map over an array doesn’t have to build the whole output array before the writer starts writing; it can hand elements downstream as it produces them, and the reader can parse input elements as they’re demanded.

When that pipeline lines up end to end — a streaming reader feeding a single-pass transform feeding a deferred writer — data flows through in a bounded window instead of piling up. Nothing between the input bytes and the output bytes is ever fully in memory. That’s the whole game. Everything below is about keeping that pipeline unbroken.

Turning on the reader

Streaming starts at the reader, and you ask for it with a reader property on the input. For a large JSON array:

%dw 2.0
input payload application/json { streaming: true }
output application/json { deferred: true }
---
payload map (order) -> {
  id: order.orderId,
  buyer: order.customer,
  total: order.total
}

streaming: true tells the JSON reader to parse incrementally rather than build the entire tree up front. deferred: true tells the writer to emit output as it’s produced instead of buffering the whole result. Between them sits a map — and map is streaming-friendly, because it looks at each element once, in order, and never needs to see the rest.

JSON streaming has a shape requirement worth stating plainly: it works when the payload is a top-level array (or an array reached by a simple path) that you walk sequentially. XML and CSV have their own streaming readers with the same spirit — read forward, don’t hold the past.

@StreamCapable, so your functions don’t break it

The moment you factor logic into a module (as in the last post), you introduce a way to accidentally kill streaming: a function that consumes its argument in a way that forces the whole thing into memory. The @StreamCapable annotation is how you declare — and let the engine check — that a parameter is meant to be consumed as a stream:

%dw 2.0
fun summarise(@StreamCapable orders) =
  orders map (o) -> { id: o.orderId, total: o.total }

The annotation is a contract. It says “this parameter may be a stream; treat it as one, and warn me if I do something that would materialise it.” It doesn’t make code stream — it protects code that’s supposed to.

What quietly breaks streaming

Here’s the honest core of the topic. Streaming survives only as long as every step is single-pass and forward-only. A surprising number of everyday operations aren’t, and each one silently materialises the collection:

• Random or indexed access — payload[500], or anything that reaches for an element by position, needs the elements before it to exist, so it buffers.
• sizeOf — you can’t know the length of a stream without reaching the end, so asking for the size consumes and holds the whole thing.
• orderBy — sorting is inherently multi-pass; you can’t emit the first sorted element until you’ve seen the last input element. Same for anything global — distinctBy, groupBy.
• Consuming a stream twice — reference the same input in two places and something has to remember it for the second reader.

map and filter keep the stream alive because they’re one-element-at-a-time. orderBy and sizeOf end it because they can’t be. If your big transform sorts the whole payload or counts it before mapping, it isn’t streaming, whatever reader property you set — and it’s worth knowing that rather than being surprised by the heap graph.

Repeatable streams, and the runtime asterisk

This is where I have to be honest that the language is only half the story. In the Mule runtime, streams are repeatable by default — the runtime records what flows past so a stream can be consumed more than once, spilling to disk when it grows beyond an in-memory buffer. That’s why “consuming a stream twice” doesn’t simply throw; the runtime quietly makes it work, sometimes by writing to a temp file, sometimes by holding it in memory. Whether a given transform truly streams, and how much memory it takes when it doesn’t, depends on the runtime version, the configured buffer sizes, and the repeatable-streaming strategy, not on DataWeave alone.

So treat the language-level rules above as necessary, not sufficient. Getting the reader property and a single-pass transform right is what lets streaming happen; the runtime configuration is what determines whether it actually does. When it matters, measure — watch heap and throughput against a realistically large input rather than trusting the sample file.

Rules of thumb for big payloads

A few habits keep large transforms out of trouble. When work is inherently global — sorting, deduplication, totals across the whole set — push it upstream to the database or downstream to a step that sees only a bounded slice, rather than forcing it over the full payload in memory. And if you must aggregate inside the transform, do it incrementally with a single reduce rather than sorting first.

Final thoughts

Streaming in DataWeave isn’t a mode you switch on and forget — it’s an agreement between the reader, your expression, and the writer that nobody will ask to see the whole collection at once. Honour it and a two-gigabyte feed flows through in a small, steady memory footprint. Break it with one stray orderBy and you’re back to loading the file into the heap. The language gives you the tools and the constraints; the runtime decides the rest, so when the numbers matter, measure them.

Next: One Ugly XML Feed, One Clean Report — the capstone, where every piece of both series lands on one genuinely messy order feed.