DataWeave Grows Teeth

Beyond map and Transform Message: functions, reduce, groupBy, and a real flatten-group-sum transform that turns nested orders into a category revenue report.

In the foundations series, DataWeave was the friendly one — %dw 2.0, a header, a ---, and a map over an array. That’s enough to reshape a payload. It is not enough to earn DataWeave its reputation: once you’re fluent, you reach for it before you reach for a <flow>.

This post is where it grows teeth. Named functions, higher-order combinators, local scopes, and a worked transform that would be a genuinely annoying loop in Java — collapsed into a dozen readable lines. If one post in this series repays slow reading, it’s this one.

Functions and variables in the header

The header isn’t just for output. You can declare reusable variables with var and named functions with fun, and the body reads like it’s calling a small library you wrote inline:

%dw 2.0
output application/json

var taxRate = 0.08

fun withTax(amount: Number) = amount * (1 + taxRate)
fun initials(name: String) = (name splitBy " ") map (upper($[0])) joinBy ""
---
{
  price: withTax(100),        // 108
  who:   initials("ada lovelace")  // "AL"
}

A fun is a first-class value, so it composes with everything below. $ is the shorthand for a single lambda parameter — map (upper($[0])) means “for each word, uppercase its first character.” Pull the logic you’d otherwise repeat into a named function once, and the transform body stays about what, not how.

The higher-order toolkit

map is the one everyone learns. The rest of the family is where DataWeave stops being a mapper and starts being a language:

  • filter — keep the elements matching a predicate.
  • reduce — fold a collection into a single value, carrying an accumulator.
  • groupBy — turn an array into an object keyed by whatever you return.
  • pluck — the inverse-ish of groupBy: walk an object’s entries and emit an array, giving you the key and the value.
  • orderBy — sort by a computed criterion.
  • distinctBy — dedupe on a computed key.

reduce is the one people flinch at, so pin it down. The lambda takes the current item and the accumulator, and its result becomes the next accumulator; the = 0 sets the seed:

[10, 20, 30] reduce ((item, total = 0) -> total + item)  // 60

That’s a sum. Swap the body and it’s a max, a concatenation, or a hand-built object. Every aggregation you’d write as a loop-with-a-running-variable is a reduce.

For objects there’s mapObject, which rebuilds an object key-by-key, handy for renaming or filtering fields:

{ firstName: "Ada", lastName: "Lovelace" }
  mapObject (value, key) -> { (upper(key as String)): value }
// { "FIRSTNAME": "Ada", "LASTNAME": "Lovelace" }

Branching, scopes, and the standard library

Conditionals come in two shapes. Inline if/else for a single fork, and match/case when you’re switching on a value or a pattern:

fun band(score: Number) = score match {
  case s if (s >= 90) -> "A"
  case s if (s >= 80) -> "B"
  case s if (s >= 70) -> "C"
  else -> "F"
}

When an expression needs a few local names to stay readable, wrap it in a do scope — a block that can declare its own vars and returns its final expression, without leaking those names to the rest of the script:

fun discountedTotal(order) = do {
    var subtotal = order.items reduce ((i, acc = 0) -> acc + i.price)
    var discount = if (subtotal > 500) 0.1 else 0
    ---
    subtotal * (1 - discount)
}

And you’re not limited to the built-in operators. DataWeave ships modules you import for the specialized stuff — dw::core::Strings for casing and padding, dw::core::Arrays for partition/countBy, dw::util::Values, and more:

import capitalize from dw::core::Strings
---
capitalize("order shipped")  // "Order Shipped"

Reach for a module before you hand-roll string surgery; the core libraries have already solved most of it.

The worked transform: nested orders to a category report

Now the payoff. Here’s the input, orders each carrying a list of line items, each item tagged with a category:

[
  { "orderId": "ord-1", "customer": "Acme",
    "items": [
      { "sku": "A1", "category": "Hardware", "qty": 2, "price": 50 },
      { "sku": "B2", "category": "Software", "qty": 1, "price": 300 }
    ]
  },
  { "orderId": "ord-2", "customer": "Globex",
    "items": [
      { "sku": "A1", "category": "Hardware", "qty": 5, "price": 50 },
      { "sku": "C3", "category": "Hardware", "qty": 1, "price": 120 }
    ]
  }
]

The goal: a report of revenue and units per category, sorted by revenue, ignoring the order boundaries entirely. That’s a flatten, then a group, then a sum inside each group — three operations that read almost like the sentence describing them:

%dw 2.0
output application/json

// pull every line item out of every order into one flat list
var lineItems = payload flatMap ((order) -> order.items)
---
(lineItems groupBy ((item) -> item.category)
  pluck ((items, category) -> {
    category: category,
    units:   items reduce ((item, acc = 0) -> acc + item.qty),
    revenue: items reduce ((item, acc = 0) -> acc + (item.qty * item.price))
  }))
  orderBy ((row) -> -row.revenue)

Walk it once. flatMap unnests: it maps each order to its items array and flattens the results into one list, so the nested structure disappears and you’re left with every line item side by side. groupBy turns that flat list into an object keyed by category: { Hardware: [...], Software: [...] }. pluck walks that object, handing each branch both its items array and its category key, and inside, two reduces do the arithmetic, one summing quantities, one summing qty * price. Finally orderBy on the negated revenue sorts biggest-first. The output:

[
  { "category": "Hardware", "units": 8, "revenue": 470 },
  { "category": "Software", "units": 1, "revenue": 300 }
]

The equivalent in imperative code is a nested loop, a HashMap<String, Accumulator>, a null-check on first insert for each category, and a sort at the end. Here it’s four combinators, top to bottom, no mutable state. That density is the reason DataWeave earns a Transform Message component of its own instead of a scattering of Java.

Null safety and formatting

Real payloads have holes, so navigate defensively. The ? after a selector returns null instead of erroring when a key is absent, and default supplies a fallback:

{
  city:  payload.address.city default "unknown",
  phone: payload.contact.phone? default "n/a"
}

Formatting is coercion with a format schema attached. Dates and numbers both take one:

{
  when:  (payload.placedAt as Date) as String { format: "dd MMM yyyy" },
  total: payload.total as String { format: "#,##0.00" }
}

Note the shape — value as Type { format: "..." }. Get the target type right and the format string is just the pattern.

Final thoughts

DataWeave rewards the move from “reshape this one payload” to “express this transformation.” The tools that make that jump — fun, reduce, groupBy, pluck, do, and the module system — are the same ones that let you delete flows, because logic that used to sprawl across Choice routers and For Each loops collapses into a single Transform Message.

The worked example is the mental model to keep: flatten, group, fold, sort — four verbs that dissolve a nested loop. When you next find yourself reaching for a For Each with a variable accumulating outside it, stop. There’s almost certainly a reduce that says it in one line.

Next: batch processing — because some datasets are too big to hold in a single payload, no matter how sharp your DataWeave is.

Comments