Durable execution and workflow engines

An onboarding flow waits up to 7 days for a person to review a merchant’s documents, and in that week the worker running it is restarted and redeployed many times. Durable executiondurable execution: Running a long, multi-step process as ordinary code that survives crashes: an engine records each step's result in an event history and, after a crash, runs the code again from the start, reusing the recorded results. runs such a long, multi-step process as ordinary code that survives crashes. The engine records the result of every step in an event historyevent history (workflow): The durable log a workflow engine keeps for one execution: its input and the result of every step that has completed, such as activity results, timers that fired and signals received.; when the process running the code dies, another one runs the same code from the start, takes each finished step’s result from the history instead of repeating the step, and carries on where the first one stopped.

Workflow engines built on this idea let a workflow wait days for a timer or a person without holding a thread. The rule that makes replay work is determinism: workflow code must make the same decisions from the same history, so clock reads, random numbers and calls to other systems go through the engine, which records their results. For a short job of two steps, a queue and a status table are cheaper.

Replay rebuilds a workflow from its event history

Temporal’s documentation calls the event history “the durable log behind Durable Execution”. After a failure, a fresh worker loads that log and runs the workflow code over it once more, rebuilding the state the lost worker held in memory, which only works if the code makes the same choices every time (Temporal). Work that touches the outside world, such as API calls, database queries and file access, runs in activitiesactivity (workflow): A step of a workflow that touches the outside world, such as an API call or a database query. The engine records its result when it completes and reuses that result when the workflow code is replayed.: an activity’s result is recorded when it completes, and during replay the recorded result is reused, not recomputed (Temporal). Step through the diagram’s execution, one recorded result at a time:

The drawing shows the whole scene; its 7 steps are listed below.

Every step as text

1. Check the documents. Worker 1 runs the workflow code. Its first activity checks the documents, and the engine records the result in the execution's history.
2. Wait for a review. The code waits for the review signal. While it waits, no thread is held; when the signal arrives, the engine records it too.
3. Create the account. The account is created, and its result, account acct-77, is recorded.
4. The worker crashes. Worker 1 dies before the next step. Its local variables are gone; the history is not, because the engine keeps it durably.
5. The same code again. Worker 2 loads the history and runs the same workflow code from its first line.
6. Replayed, not repeated. Each of the first three steps asks for a result the history already holds, so worker 2 takes the recorded result instead of running the step. No document is checked twice and no second account is created.
7. The next step runs. The fourth step, send welcome, has no recorded result, so it really runs and is recorded. The execution has carried on exactly where worker 1 stopped.

Three workflow engines show the range of designs:

• Temporal records an event history per workflow and replays the workflow code against it after a failure.
• Azure Durable Functions runs orchestrator functions by event sourcing and replays them, so it imposes the same determinism rules (Azure).
• AWS Step Functions describes a workflow as a state machine written in the Amazon States Language, and its Standard workflows can run for up to a year (AWS).

Histories have limits, and long workflows meet them. Temporal caps an execution’s event history at 51,200 events or 50 MB and warns at 10,240 events or 10 MB (Temporal). A workflow that wakes once a minute to poll something adds at least two events per wake-up, a timer started and a timer fired, so it reaches the cap in under 18 days: 51,200 ÷ (2 × 1,440). Long-lived workflows are therefore written to wait for events rather than poll, or to start over with a fresh history at a safe point.

A replay engine, run: a crash, a clock and a deploy Pro: opens with a Pro, Premium or Ultimate pass

In this part:

• The determinism rules, and how engines enforce them
• Shipping a change to workflows that are still running
• Activities run at least once
• A refund with a timer and an approval, as a state machine
• Engine or status table: choosing, and how answers are judged
• Practise: make a workflow deterministic
  • Remove the three replay bugs from a refund workflow
• Interview questions
  • What does a durable execution engine store, and how does it bring a workflow back after the worker running it crashes?
  • Design merchant onboarding: verify documents, wait up to 7 days for a human review with reminders every 2 days, then create the account and send credentials. Would you use a workflow engine or a queue with a status table? Justify it with the failure cases.
  • Right after a deploy, hundreds of running workflows fail with non-determinism errors, while new ones run fine. What happened, how do you recover the failed ones, and how should the change have been shipped?
  • A workflow's "charge card" activity charged a customer twice, although the engine records every activity's result. How is that possible, and how do you prevent it?
• Key takeaways

2 diagrams · 3 runnable examples (JavaScript, Python) · 1 exercise with sample tests · 3 interview answers · about 45 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

• Pro pass · 1 week · ₹79 · $2.99
• 30 days · ₹179 · $5.99
• 1 year · ₹1,699 · $45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a passI have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Read a complete sample first: Design a unique ID generator

Terms in this lesson

activity (workflow)

A step of a workflow that touches the outside world, such as an API call or a database query. The engine records its result when it completes and reuses that result when the workflow code is replayed.

deterministic workflow code

Workflow code that makes the same calls in the same order from the same history, so a replay rebuilds the same state; reading the clock, drawing random numbers and calling other systems go through the engine.

durable execution

Running a long, multi-step process as ordinary code that survives crashes: an engine records each step's result in an event history and, after a crash, runs the code again from the start, reusing the recorded results.

event history (workflow)

The durable log a workflow engine keeps for one execution: its input and the result of every step that has completed, such as activity results, timers that fired and signals received.