System Design (High-Level Design) Module 12 – Messaging and stream processing
Delivery guarantees, retries and dead letters
At-most-once, at-least-once and exactly-once delivery: how acknowledgements and visibility timeouts cause repeats, and when to retry or dead-letter.
What you will learn
- Explain how the order of the work and the acknowledgement makes delivery at most once, at least once or effectively once
- Predict when a visibility timeout shorter than the work repeats a job without any crash
- Decide between a retry and the dead-letter queue for a message that has just failed
- Design retry policies with backoff, jitter and attempt limits sized from processing times
- Build idempotent consumers, and run a dead-letter queue with alarms, retention and redrive
Before you start
What you’ll learn · Before you start
What you will learn
- Explain how the order of the work and the acknowledgement makes delivery at most once, at least once or effectively once
- Predict when a visibility timeout shorter than the work repeats a job without any crash
- Decide between a retry and the dead-letter queue for a message that has just failed
- Design retry policies with backoff, jitter and attempt limits sized from processing times
- Build idempotent consumers, and run a dead-letter queue with alarms, retention and redrive
Before you start
On this page
A payment consumer charges a card and crashes before it acknowledges the message, and 30 seconds later the queue hands the same message to another consumer, which charges the card again. Delivery guarantees describe what such a failure can do to a message: with at-most-once delivery: A delivery guarantee under which a message is never processed twice but may be lost, for example because the consumer acknowledged it before doing the work and then crashed. a message may be lost but is never processed twice, with at-least-once delivery: A delivery guarantee under which every message arrives but some may arrive more than once, after a retry, so the receiver must recognise and drop the copies. More in “First-hand case: how MySmartCoPilot is built” it is never lost but may be processed twice, and with exactly-once processing: A message's effect happens once although the message may be delivered more than once: the work and the record that it was done commit together, so a repeated message is recognised and skipped. its effect happens once.
The choice comes down to the order of two steps. A consumer that acknowledges a message before doing the work can lose it; one that acknowledges after the work can repeat it. Most brokers default to at least once, so consumers must be safe to run twice. Retries with backoff recover from transient failures, such as a timeout. A message that fails every time, a poison message: A message that fails every time it is processed, such as one whose payload does not parse, so no retry can ever succeed and it has to be set aside., is moved after a few attempts to a dead-letter queue: A queue that holds the messages that used up their attempts or can never succeed, set aside from the main queue so that someone can inspect them and replay them once the cause is fixed., where dead letters wait for someone to inspect and replay them.
Where a message is lost or repeated
Every consumer runs three steps: receive the message, do the work, acknowledge. The broker keeps the message until the acknowledgement arrives, so the order of the last two steps decides what a crash costs.
Where a crash loses or repeats a message
Text description of the diagram
The diagram has three panels, one above the other, each with a consumer's steps from top to bottom.
- At most once: receive, then acknowledge, so the broker forgets the message, then do the work. A crash between the acknowledgement and the work loses the message for good.
- At least once: receive, then do the work, then acknowledge. A crash between the work and the acknowledgement means the message comes back and the work is done again.
- Effectively once: receive, then do the work and record the message id in one transaction, then acknowledge. A message whose id is already recorded is skipped, so a crash before the acknowledgement makes the repeat harmless: it finds the id and does nothing.
The diagram shows the three orders side by side. Step through one payment message under each of them, with the crash coming at the worst moment every time:
The drawing shows the whole scene; its 9 steps are listed below.
Every step as text
- The message arrives. The queue delivers payment message m1 to the consumer and keeps it until the consumer acknowledges it.
- At least once, work first. The consumer does the work first and adds m1's payment to the ledger.
- A crash before the ack. It crashes before it acknowledges m1. The queue never learns that the payment was added.
- m1 comes back. When m1's visibility timeout ends, the queue delivers it again, to this consumer after its restart or to another one.
- The work repeats. The payment is added a second time. Nothing was lost, and at least once never promised more than that.
- At most once, ack first. Swap the last two steps. The consumer acknowledges m1 as soon as it arrives, and the queue forgets it.
- A crash before the work. A crash now, before the ledger write, loses the payment for good. The queue has forgotten m1, and the ledger never got it.
- Effectively once. The third design writes the payment and m1's id in one transaction, and only then acknowledges.
- The repeat is skipped. If a crash comes before the acknowledgement, m1 is delivered again, the consumer finds its id in the ledger and skips the work. One payment, however often m1 arrives.
Kafka’s design documentation describes the same choice for a log: a consumer that saves its position before processing gets at-most-once semantics, one that processes first gets at least once, and at least once is Kafka’s default. Exactly once needs more than an order: the work and the record that it was done must commit together, so a repeated message is recognised and skipped. All three delivery guarantees follow from where the acknowledgement sits.
| Criterion | At most once | At least once | Exactly once, in effect |
|---|---|---|---|
| The consumer acknowledges | Before the work | After the work | After the work and its record commit together |
| A crash at the wrong moment | Loses the message | Repeats the work | Repeats a message, which is then skipped |
| What it costs | Nothing extra | Consumers that are safe to repeat | A record of processed messages in the same store, or broker transactions |
| Typical use | Samples and positions that the next message replaces | Most jobs and events | Payments, balances, counts that are money |
| When to choose | Losing a few messages is harmless | The default for almost everything | A repeated effect costs money or trust |
Checkpoint
Check: Where the acknowledgement sits
Visibility timeouts decide when a message comes back
A broker cannot tell a slow consumer from a dead one, so it waits for an acknowledgement for a while and then hands the message to someone else. RabbitMQ waits for the connection: a delivery that was never acknowledged is requeued when its channel or connection closes, and marked as redelivered. Amazon SQS uses a clock: a received message is hidden for its visibility timeout: How long a queue hides a message it has handed to a consumer. If the consumer has not deleted it when the time is up, the message becomes visible and is delivered again; Amazon SQS uses 30 seconds by default., 30 seconds by default and at most 12 hours, and becomes visible again if nobody has deleted it by then.
A timeout shorter than the work creates duplicates without any crash. This model runs twelve jobs on four consumers, two of them slow, under four timeouts:
"""A queue with a visibility timeout, four consumers and twelve jobs, two of them slow.
A received message stays in the queue, hidden for the visibility timeout. If the consumer has not deleted it when the
timeout ends, it becomes visible again and another consumer receives it, while the first one is still working. The
clock moves one second at a time, so every run prints the same.
"""
DURATIONS = [12, 18, 25, 9, 45, 14, 22, 70, 16, 11, 28, 19] # seconds of work for jobs j1 .. j12
CONSUMERS = 4
def run(timeout, trace_job=None):
queue = [f"j{i + 1}" for i in range(len(DURATIONS))] # visible messages, next to be received first
hidden = {} # message -> the second at which it becomes visible again
working = {} # consumer -> (message, the second its work ends)
done = {m: 0 for m in queue} # how many times each job's work was carried out
deleted = set()
deliveries = busy = t = 0
events = []
while queue or hidden or working:
for m, at in list(hidden.items()): # a timeout ended: the message is visible again
if at == t:
del hidden[m]
queue.insert(0, m)
if m == trace_job:
events.append(f"t={t:>3}s {m} visible again: its timeout ended before anyone deleted it")
for c, (m, end) in sorted(working.items()): # finished work: the side effect happens, then the delete
if end == t:
del working[c]
done[m] += 1
if m in deleted:
if m == trace_job:
events.append(f"t={t:>3}s consumer {c} finishes {m} again; the message is already gone")
continue
deleted.add(m)
hidden.pop(m, None)
if m in queue:
queue.remove(m)
if m == trace_job:
events.append(f"t={t:>3}s consumer {c} finishes {m} and deletes it")
for c in range(1, CONSUMERS + 1): # idle consumers receive the next visible message
if c not in working and queue:
m = queue.pop(0)
hidden[m] = t + timeout
working[c] = (m, t + DURATIONS[int(m[1:]) - 1])
deliveries += 1
if m == trace_job:
events.append(f"t={t:>3}s consumer {c} receives {m}")
busy += len(working)
t += 1
repeated = {m: n for m, n in done.items() if n > 1}
extra = busy - sum(DURATIONS)
return deliveries, repeated, extra, t - 1, events
print("timeout deliveries jobs done more than once extra consumer-seconds all done at")
for timeout in (20, 30, 60, 90):
deliveries, repeated, extra, end, _ = run(timeout)
twice = ", ".join(f"{m} x{n}" for m, n in repeated.items()) or "none"
print(f"{timeout:>5} s {deliveries:>10} {twice:<24} {extra:>22} {end:>9} s")
print()
print("Job j8 (70 s of work) with the default 30-second timeout:")
for line in run(30, trace_job="j8")[4]:
print(" ", line)Output
timeout deliveries jobs done more than once extra consumer-seconds all done at 20 s 15 j5 x2, j8 x3 185 152 s 30 s 14 j5 x2, j8 x2 115 143 s 60 s 13 j8 x2 70 155 s 90 s 12 none 0 95 s Job j8 (70 s of work) with the default 30-second timeout: t= 25s consumer 3 receives j8 t= 55s j8 visible again: its timeout ended before anyone deleted it t= 73s consumer 4 receives j8 t= 95s consumer 3 finishes j8 and deletes it t=143s consumer 4 finishes j8 again; the message is already gone
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 visibility_sim.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
At the default 30 seconds both slow jobs ran twice. The repeats cost 115 consumer-seconds and moved the end of the run from 95 to 143 seconds. Only the 90-second timeout, longer than the slowest job, did every job once. The trace at the end of the output follows j8, and you can step through it:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- 25 s, consumer 3 starts. At 25 seconds consumer 3 receives j8, a job of 70 seconds. The queue hides j8 for its visibility timeout, 30 seconds.
- 55 s, j8 visible again. At 55 seconds the timeout ends. Consumer 3 is still working, but it has not deleted j8, so the queue makes j8 visible again.
- 73 s, consumer 4 starts. At 73 seconds consumer 4 is idle and receives j8. Two consumers now do the same 70 seconds of work.
- 95 s, consumer 3 deletes j8. At 95 seconds consumer 3 finishes and deletes j8. The message is gone, but consumer 4 does not know that.
- 143 s, done a second time. At 143 seconds consumer 4 finishes j8 again and finds the message already deleted. The repeat cost 70 consumer-seconds and nothing crashed.
- The timeout that fits. Of the four timeouts the program tried, only 90 seconds, longer than the slowest job, did every job once.
j8 became visible at 55 seconds while consumer 3 was still working on it, so consumer 4 started it again. A long timeout has its own cost: when a consumer really dies, its message waits the whole timeout before anyone retries it.
Checkpoint
Check: Visibility timeouts and repeats
Retries for transient failures, dead letters for the rest
A transient failure, such as a timeout from a busy service, deserves a retry; a permanent one, such as a payload that does not parse, deserves none. Without a limit, a poison message is retried forever. The pieces that handle this, each covered in depth below:
- Backoff waits longer after each failure, so a struggling service gets room to recover.
- Jitter randomises each wait, so messages that failed together do not retry together.
- An attempt limit caps how often one message is tried before it is set aside.
- A dead-letter queue holds messages that used up their attempts.
- Redrive moves dead letters back to the main queue once the cause is fixed.
- An idempotent consumer makes a repeated message harmless by recognising work it has already done.
Brokers build the limit in. An SQS redrive policy
moves a message to its dead-letter queue after maxReceiveCount receives. Since RabbitMQ 4.0,
quorum queues have a delivery limit of 20 by default: a message
redelivered more often is dropped or dead-lettered. Retries in the consumer and retries in the broker multiply, so
decide where each one lives. Here is one message that keeps timing out, under the policy of the exercise below:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- A failure that can clear. Delivery 1 fails because the inventory service timed out. The consumer has to decide what happens to the message next.
- Can a retry succeed?. A timeout can clear, so a retry may succeed. A payload that does not parse never will, and would go to the dead-letter queue at once.
- Attempts left. This was attempt 1 of the 5 the policy allows, so the message gets another try.
- Back off before the retry. The message becomes visible again after a delay that starts at the 1-second base and doubles after each failure, never above the 8-second cap.
- Attempts 2 to 5 fail. The cause does not clear. Attempts 2, 3 and 4 fail too, and each is followed by a longer wait, 2, 4 and then 8 seconds. Attempt 5, the last one allowed, fails as well.
- Into the dead-letter queue. No attempts are left, so the message leaves the main queue for the dead-letter queue, where it stops using attempts and waits for a person.
- Redrive. Once the cause is fixed, the dead letters are redriven, moved back to the main queue to be processed again.
Checkpoint
Check: Retries and dead letters
Practise: retry or dead-letter
Write the rule a consumer applies after a failure; the policy of its sample tests is the one in the flow above. The self-check after the exercise asks about acknowledgements, timeouts and retries, starting where your answers are weakest.
Exercise · Easy · JavaScript
Decide between a retry and the dead-letter queue
Write nextStep(attempt, error, policy) in next_step.mjs. A consumer calls it when processing a message has just failed, and it decides what happens to the message next.
attemptis how many times the message has been delivered so far, counting this delivery: 1 the first time.error.permanentistruefor failures that can never succeed on a retry, such as a payload that does not parse or an order for an account that does not exist.policyhasmaxAttempts,baseMsandcapMs.
Return { action: 'dead-letter' } when the error is permanent, or when attempt has reached maxAttempts. Otherwise return { action: 'retry', delayMs }, where delayMs doubles with every attempt, starting at baseMs after the first failure, and never exceeds capMs.
Starter code · next_step.mjs
/**
* What happens to a message whose processing just failed.
* attempt: deliveries so far, counting this one (1 the first time); error: { permanent };
* policy: { maxAttempts, baseMs, capMs }.
* Returns { action: 'dead-letter' } or { action: 'retry', delayMs }.
*/
export function nextStep(attempt, error, policy) {
// Replace this line with your code.
return { action: 'retry', delayMs: 0 };
}The sample tests · next_step.test.mjs
import { test, assert } from 'mysmartcopilot:test';
import { nextStep } from './next_step.mjs';
const policy = { maxAttempts: 5, baseMs: 1000, capMs: 8000 };
const transient = { permanent: false };
test('the first transient failure retries after baseMs', () => {
assert.deepEqual(nextStep(1, transient, policy), { action: 'retry', delayMs: 1000 });
});
test('the delay doubles with every attempt', () => {
assert.deepEqual([2, 3].map((a) => nextStep(a, transient, policy).delayMs), [2000, 4000]);
});
test('the delay never exceeds capMs', () => {
assert.deepEqual(nextStep(4, transient, { ...policy, capMs: 5000 }), { action: 'retry', delayMs: 5000 });
});
test('the last allowed attempt goes to the dead-letter queue', () => {
assert.deepEqual(nextStep(5, transient, policy), { action: 'dead-letter' });
assert.deepEqual(nextStep(4, transient, policy).action, 'retry');
});
test('a permanent error goes to the dead-letter queue at once', () => {
assert.deepEqual(nextStep(1, { permanent: true }, policy), { action: 'dead-letter' });
});A hint
Check the two dead-letter cases first; everything else is a retry. After failure number attempt, the uncapped delay is baseMs * 2 ** (attempt - 1): baseMs after the first failure, twice that after the second. Math.min applies the cap.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (QuickJS): nothing is sent to mysmartcopilot.com. The first run downloads JavaScript (about 0.6 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Practice that adapts to you
Practise this lesson
10 questions about 11 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Retry policies that recover, and ones that cause outages Pro: opens with a Pro, Premium or Ultimate pass
A retry policy is a handful of numbers and one judgement: which errors can ever succeed. This part runs three policies against the same poison message, works out the backoff numbers, and sizes the timeout and the attempt limit from real processing times.
In this part:
- A poison message reproduced: three policies on one queue
- Backoff with jitter and a cap, worked in numbers
- Head-of-line blocking in ordered queues
- Sizing the visibility timeout and the attempt limit from processing times
1 diagram · 1 runnable example (JavaScript) · about 10 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Idempotent consumers and a dead-letter queue you can run Pro: opens with a Pro, Premium or Ultimate pass
At-least-once delivery is safe only when a repeat does no harm, and a dead-letter queue is useful only when someone watches it and can replay it. This part measures four consumer designs against the same failures, then sets up the dead-letter queue’s alarm, retention and redrive.
In this part:
- Four consumers through the same failures, measured
- What exactly-once means in practice
- Running the dead-letter queue: alarms, retention and redrive
- Guarantees and limits for five workloads
2 diagrams · 1 runnable example · about 15 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Interview questions
Decide each brief before you open the questions below it; the feedback says what each choice gets right or misses.
Checkpoint
Your first decision
-
Fresher to mid · Warm-up
What does at-least-once delivery promise, and what does it not promise?
It promises that a message the broker has accepted is not lost: the broker keeps delivering it until a consumer acknowledges it. It does not promise that the message is processed once, because a consumer can do the work and crash before its acknowledgement arrives, or a timeout can end while the work is still running, and the message is delivered again. It says nothing about order once retries and several consumers are involved, and nothing about the producer: a producer that retries a send can publish the same message twice. Consumers under at-least-once delivery must therefore be idempotent, for example by recording the ids of processed messages in the same transaction as their effects.
-
Mid to senior
A payment consumer charges a card and then acknowledges the message. After a deploy, 40 customers had been charged twice. Explain how that happened and how you would make it impossible.
-
Mid to senior
One malformed message makes your consumer crash. For an hour the consumer restarts in a loop while the backlog grows. How do you stop it now, and how do you design so that it cannot happen again?
-
Mid to senior
Jobs take 20 seconds on average but up to 4 minutes for the largest inputs, and a few consumers crash every day. How do you choose the visibility timeout and the maximum receive count?
Key takeaways
- Acknowledge before the work and a crash loses the message; acknowledge after it and a crash repeats it. Most brokers deliver at least once by default.
- A visibility timeout shorter than the work repeats it with no crash at all.
- Retry transient failures, with backoff, jitter and an attempt limit; send permanent failures and exhausted messages to a dead-letter queue, and redrive them after the fix.
- Exactly once, in effect, is at-least-once delivery plus a consumer that recognises work it has already done.
Terms in this lesson
- at-most-once delivery
- A delivery guarantee under which a message is never processed twice but may be lost, for example because the consumer acknowledged it before doing the work and then crashed.
- dead-letter queue
- A queue that holds the messages that used up their attempts or can never succeed, set aside from the main queue so that someone can inspect them and replay them once the cause is fixed.
- exactly-once processing
- A message's effect happens once although the message may be delivered more than once: the work and the record that it was done commit together, so a repeated message is recognised and skipped.
- idempotent consumer
- A consumer for which processing the same message twice has the same effect as processing it once, because it recognises work it has already done.
- poison message
- A message that fails every time it is processed, such as one whose payload does not parse, so no retry can ever succeed and it has to be set aside.
- redrive
- Moving messages from a dead-letter queue back to their main queue once the cause of their failures has been fixed.
- visibility timeout
- How long a queue hides a message it has handed to a consumer. If the consumer has not deleted it when the time is up, the message becomes visible and is delivered again; Amazon SQS uses 30 seconds by default.
References
- Amazon SQS visibility timeout (Amazon Web Services)
- Using dead-letter queues in Amazon SQS (Amazon Web Services)
- Amazon SQS message quotas (Amazon Web Services)
- FIFO queue delivery logic in Amazon SQS (Amazon Web Services)
- Exactly-once processing in Amazon SQS (Amazon Web Services)
- Creating and configuring an Amazon SQS event source mapping (AWS Lambda) (Amazon Web Services)
- Consumer acknowledgements and publisher confirms (RabbitMQ documentation) (Broadcom (RabbitMQ))
- Quorum queues (RabbitMQ documentation) (Broadcom (RabbitMQ))
- Apache Kafka 4.3 documentation: Design (The Apache Software Foundation)
- Apache Kafka 4.3 documentation: Producer configs (The Apache Software Foundation)
- Subscription overview (Google Cloud Pub/Sub documentation) (Google Cloud)
- Timeouts, retries, and backoff with jitter (Amazon Builders' Library) (Amazon Web Services)
- Avoiding insurmountable queue backlogs (Amazon Builders' Library) (Amazon Web Services)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress