Queues That Don't Drop the Ball

VM queues, the Async scope, Until Successful, and reconnection strategies — the Mule building blocks that let you accept work fast and finish it reliably, even when a downstream is having a bad day.

A synchronous flow is a promise that everything downstream stays up. The caller waits, you call the database, you call the partner API, you respond — and if the partner API takes eight seconds, so does your caller. If it’s down, you fail. That promise is fine for a lookup. It’s a liability when you’re accepting an order that must not be lost just because a warehouse system is rebooting.

Reliability in Mule is mostly about breaking that promise on purpose: accept the work quickly, acknowledge it, and finish it on your own schedule — with retries where the network is flaky and a durable queue underneath so nothing evaporates if a worker restarts. The pieces that do this are the VM connector, the Async scope, Until Successful, and reconnection strategies. They compose.

Accept fast, process later: VM queues

The VM connector gives you in-app queues — a way for one flow to hand work to another without a network hop. One flow publishes to a queue and returns; a separate flow listens on that queue and does the slow work. You’ve decoupled the fast part (accepting the request) from the slow part (fulfilling it).

<vm:config name="vmConfig">
    <vm:queues>
        <vm:queue queueName="orders" queueType="PERSISTENT"/>
    </vm:queues>
</vm:config>

<flow name="acceptOrder">
    <http:listener config-ref="httpListener" path="/orders"/>
    <vm:publish config-ref="vmConfig" queueName="orders"/>
    <set-payload value='#[{ status: "accepted" }]'/>
    <set-variable variableName="httpStatus" value="202"/>
</flow>

<flow name="processOrder">
    <vm:listener config-ref="vmConfig" queueName="orders"/>
    <flow-ref name="fulfillOrder"/>
</flow>

The listener returns 202 Accepted the instant the order lands on the queue — the caller is done in milliseconds and never waits on fulfillment. The processOrder flow drains the queue in the background.

The word doing the heavy lifting is PERSISTENT. A persistent VM queue is written to disk (and, on CloudHub, backed by a managed store shared across workers), so a message that’s been published survives a worker crash or redeploy — when the worker comes back, the message is still on the queue waiting to be processed. That’s the foundation of at-least-once delivery: once you’ve accepted the work, you won’t lose it. The trade is that “at least once” can mean more than once — if a worker dies mid-processing, the message may be redelivered — which is why the flows behind a queue should be idempotent. Design the fulfillment so that processing the same order twice produces the same result as processing it once, usually by keying on a business id and checking whether you’ve already handled it.

Fire-and-forget: the Async scope

Sometimes you don’t need a durable queue — you just don’t want the caller waiting on a side effect. The Async scope runs its contents on a separate thread and lets the main flow continue immediately.

<flow name="createUser">
    <http:listener config-ref="httpListener" path="/users"/>
    <db:insert config-ref="db"> ... </db:insert>
    <async>
        <http:request config-ref="analytics" path="/track" method="POST"/>
    </async>
    <set-payload value='#[{ id: payload.id }]'/>
</flow>

The user is created and the response goes back without waiting on the analytics call. Async is the right tool when the work is genuinely optional to the response — logging, metrics, a nice-to-have notification. What it does not give you is durability: an Async block lives in memory, so if the worker dies before it finishes, that work is gone. Async trades reliability for simplicity. When “gone” is unacceptable, reach for a persistent VM queue instead.

Keep trying: Until Successful

Transient failures are the normal weather of integration — a connection reset, a 503 while the target scales, a lock timeout. The Until Successful scope retries its body until it succeeds or exhausts its attempts.

<until-successful maxRetries="5" millisBetweenRetries="2000">
    <http:request config-ref="warehouse" path="/reserve" method="POST"/>
</until-successful>

Five attempts, two seconds apart, before it gives up and throws. Wrap the calls that fail transiently — where trying again in a moment is likely to work. Don’t wrap a call that fails because the request itself is malformed; retrying a 400 five times just fails five times slower. Pair Until Successful with a persistent queue and you get a strong shape: the queue guarantees the work isn’t lost, and Until Successful rides out the target’s bad minutes without dropping the record.

Reconnection strategies

Until Successful retries a message processor. Reconnection strategies operate a layer down — they govern what a connector does when it can’t establish its connection in the first place. A database that’s down at startup, an SFTP host that isn’t reachable yet: without a reconnection strategy, the connector fails immediately.

<db:config name="db">
    <db:my-sql-connection host="..." user="..." password="..."/>
    <reconnection>
        <reconnect frequency="4000" count="5"/>
    </reconnection>
</db:config>

<reconnect frequency="4000" count="5"/> tries five times, four seconds apart. There’s also <reconnect-forever frequency="..."/> for a connector that should keep trying indefinitely — appropriate for a listener-side connection you want to self-heal after an outage without a redeploy. Set these on the config, not the operation: reconnection is about the connection, retries are about the message.

Putting it together

The reliable shape for “accept an order and fulfill it” stacks all four:

  • The HTTP listener publishes to a persistent VM queue and returns 202 — fast acceptance, durable handoff.
  • A separate flow listens on the queue and fulfills — decoupled from the caller.
  • The fulfillment call sits inside Until Successful — it survives the warehouse’s transient blips.
  • The warehouse connector carries a reconnection strategy — it self-heals through a full outage.
  • The fulfillment logic is idempotent — redelivery after a crash doesn’t double-charge anyone.

Backpressure falls out of this naturally: if fulfillment is slower than acceptance, work piles up on the queue rather than overwhelming the target or the caller. The queue is the shock absorber. You size the processing flow’s concurrency to what the target can take, and the queue holds the rest.

Final thoughts

Reliability isn’t a single switch — it’s a set of small decisions about where you’re willing to wait, what you’re willing to lose, and how you recover. VM queues decide durability, Async decides whether the caller waits, Until Successful decides how hard you try, and reconnection decides how the connector heals. None of them is exotic; the skill is composing them so that a downstream outage becomes a delay your system rides out rather than an error your customer sees.

Next: Policies at the Gate

Comments