Durable workflows: retries, timeouts, and human approvals

Some of the most important logic in a business application isn't a request and a response. It's a process: a document is uploaded, extracted, checked, sent for approval, and filed. An order is paid, fulfilled, shipped, and invoiced. A new customer is created, provisioned, emailed, and followed up a week later.

These processes span minutes to weeks, touch several external systems, and include people. They're also where a surprising number of production incidents come from.

How it usually starts, and goes wrong

The first version is often a status column and a cron job:

SELECT * FROM documents WHERE status = 'extracted' AND approved_at IS NULL;

It works until it doesn't. A step fails halfway and the status is never updated. A retry sends the same email twice. Two cron workers pick up the same row. Nobody can answer "where is this document stuck, and why?" without reading logs across three services.

The underlying problem is that the workflow's state is scattered across columns, queues, and timers, and each piece has its own failure modes.

What durable execution changes

Durable execution engines such as Temporal, Inngest, Restate, and AWS Step Functions take a different approach. You write the workflow as ordinary sequential code, and the engine persists its progress after every step. If a worker crashes, the workflow resumes from the last completed step instead of starting over. Retries, timeouts, and waiting, even for days, are built in.

The APIs differ between tools, but the shape is similar. In pseudocode:

workflow("document-approval", async ({ document, step }) => {
  const extracted = await step.run("extract", () =>
    extractFields(document.id)
  )

  const issues = await step.run("validate", () => validateFields(extracted))

  if (issues.length > 0) {
    const decision = await step.waitForEvent("reviewer-decision", {
      match: { documentId: document.id },
      timeout: "3d"
    })

    if (!decision) {
      await step.run("escalate", () => notifyManager(document.id))
      return
    }

    if (decision.status === "rejected") {
      await step.run("notify-rejection", () => notifyUploader(document.id))
      return
    }
  }

  await step.run("file", () => fileDocument(document.id, extracted))
})

Each step.run is checkpointed. Waiting three days for a reviewer costs nothing while it waits, and the timeout path is explicit instead of being a forgotten edge case.

The rules that make it reliable

Durable execution doesn't remove the need for careful design. These rules matter regardless of which tool you use.

Make every step idempotent

The engine guarantees a step completes, which means it may run more than once if it fails partway through. Every step must be safe to repeat. Use idempotency keys for payments and emails, upserts instead of inserts, and check-before-act for external systems that lack native idempotency.

Keep steps small and side effects isolated

One side effect per step is a good default. A step that charges a card and sends an email can't be retried cleanly if the email fails after the charge succeeded.

Make waiting and timeouts explicit

Every wait for a human or an external system should have a timeout and a defined consequence: a reminder, an escalation, or a cancellation. "Wait forever" is how processes silently stall.

Treat human approval as a first-class step

Approvals aren't an exception to the workflow. They are the workflow. Model them as events the workflow waits for, with clear identities, audit records of who decided what and when, and a way to reassign them when someone is on holiday.

Version deliberately

Long-running workflows will still be running when you deploy new code. Most engines need you to handle changes to in-flight workflows explicitly, so learn how yours does it before the first production deploy, not after.

When you don't need an engine

Not every background job needs durable execution. A single-step job like resizing an image or sending a notification is fine on a plain queue with retries. A Postgres-backed queue with idempotent handlers covers a lot of ground for small teams.

The signal that you need more is multi-step processes with waits, compensation logic, and questions from the business like "how many documents are waiting for approval, and for how long?" At that point, a workflow engine replaces a pile of status columns and cron jobs with something you can actually reason about.

Observability comes for free, almost

One underrated benefit: every workflow run has a history. You can see which step it's on, what each step returned, how many retries happened, and how long it has been waiting. Support and operations teams can answer "where is my document?" without an engineer. That alone often justifies the switch.