Durable Workflow Engines vs Message Queues: Choosing the Right Tool for Background and Long-Running Work
Background jobs, message queues and durable workflow engines compared: idempotency, retries, dead-letter queues, sagas and when to use each.
Queue or durable workflow engine?
Use a message queue with workers for short, independent background jobs, making each job idempotent, retrying with backoff and sending repeated failures to a dead-letter queue. Use a durable workflow engine when a business process has several dependent steps, waits for hours or days (approvals, timers, external callbacks), needs compensation if a later step fails, and must resume reliably after crashes. Many systems use both.
This guide is for engineers and technical leads. For ecommerce examples, see ecommerce queue architecture; for AI agents, durable execution for AI agents.
Key takeaways
- Every background job must be safe to run twice.
- Retries need backoff, limits and dead-letter queues.
- Long, multi-step processes are easier in workflow engines.
- Use sagas and compensations for cross-service consistency.
- Monitor queue depth, failures and stuck workflows.
Background job essentials
- Idempotency: use job keys so duplicates do not double-charge or double-send.
- Retries: exponential backoff with jitter and a maximum attempts limit.
- Dead-letter queue: park messages that keep failing, alert and allow replay after fixes.
- Timeouts and visibility: make sure slow jobs are not processed twice by accident.
- Ordering: design for out-of-order processing unless the queue guarantees order.
- See API integration reliability for patterns with external APIs.
Comparison
| Factor | Message queue + workers | Durable workflow engine |
|---|---|---|
| Best for | Independent, short tasks | Multi-step, long-running processes |
| State | You store it in your database | Engine records workflow state and history |
| Waiting | Awkward (scheduled messages, polling) | Built-in timers and signals |
| Failure handling | Retries, DLQ per message | Retries per step, compensations, resume after crash |
| Visibility | Queue metrics, logs | Per-workflow history and status |
| Operational cost | Low to moderate | New platform or managed service to learn and run |
Durable workflow engines
Engines such as Temporal and AWS Step Functions record each step so a workflow resumes after failures. Limits differ: AWS documents a maximum execution time of one year for Standard workflows and five minutes for Express workflows (AWS). Keep workflow code deterministic where the engine requires it, put side effects in activities or tasks, and version workflows carefully because running instances may last a long time.
Sagas and compensations
For a process such as 'reserve stock, charge card, book courier', define a compensation for each step: release stock, refund the charge, cancel the booking. If a later step fails, run compensations in reverse. Workflow engines make sagas explicit and observable; with queues you must track saga state yourself. See integration architecture patterns.
Hypothetical example: an onboarding process
This example is hypothetical. A B2B SaaS's onboarding involved creating an account, waiting up to five days for a signed agreement, provisioning resources, syncing to the CRM and billing, and sending reminders. It ran as a tangle of cron jobs and queue messages that sometimes stalled silently. The team moved the process to a durable workflow with timers and compensations, kept email sending and CRM syncs as queue jobs, and could finally see where each customer's onboarding stood.
Common mistakes
- Non-idempotent jobs that double-process on retry.
- Infinite retries with no dead-letter queue.
- Long processes held together by cron and flags.
- Workflow code with side effects outside activities.
- No alerts for stuck or failed workflows.
Related guides
- AI agent orchestration for agent workflows.
- B2B customer onboarding automation for a common long-running process.
- Monitoring and observability for queues and workflows.
- Modular monolith vs microservices for service boundaries.
Conclusion
Queues and durable workflow engines solve different problems. Use queues with idempotent, retried jobs for independent work, and workflow engines for multi-step, long-running processes that need state, timers and compensations. Monitor both.
Background processes failing silently?
ZSpace Labs is an India-based, remote-first technology studio working with Australian and international businesses on reliable backend systems and integrations and workflow automation. If it would help to choose between queues and workflow engines, we are happy to talk it through.
Common questions.
For independent, short tasks such as sending emails, resizing images, syncing a record or processing a webhook. Each message is handled by a worker with retries, idempotency and a dead-letter queue for failures.