Design a pub-sub system with fan-out

An “order placed” event on a company’s event platform must reach billing, email, search, analytics and 30 other services, and the busiest topic already has 300 subscriptions. A pub-subpublish-subscribe: A messaging style in which a producer publishes an event and every subscriber receives its own copy and acts on it in its own time, without the producer knowing who listens. More in “Synchronous vs asynchronous communication” service lets the publisher send each message once and delivers a copy to every subscription, so the publisher never needs to know who is listening.

Each subscription tracks what its subscribers have acknowledged and delivers the rest again, by letting workers pull or by pushing to an endpoint. You will estimate what fan-out does to bandwidth and storage, step one event through a first design, and then work out where messages live, how long a subscriber may hold one, how ordering keys fence off a slow key, and what happens to a subscriber that disappears for days.

Fan-out turns 50 MB a second of publishing into 500 MB of delivery

Rates, fan-out, delivery and retention each change the design, so ask about them first. This platform’s answers:

Functional requirements: create topics and subscriptions; publish batches; pull, acknowledge and extend deadlines; push to endpoints with retries; ordering keys; filters; dead-letter topics; replay from a time. Non-functional requirements: no message lost once publish succeeds, delivery within about 100 ms for healthy subscribers, and a slow subscriber never slows the publisher or other subscribers.

The first estimate multiplies those answers:

# A first estimate for the pub-sub service of this lesson. The inputs are assumptions to replace with measurements.
PEAK_PUBLISH, AVG_PUBLISH = 50_000, 20_000   # messages published a second, across all topics
MSG_BYTES = 1_000
FAN_OUT = 10                                 # subscriptions per topic, on average
RETENTION_DAYS, COPIES = 7, 3                # kept for replay, and replicated three times
MB, TB = 10**6, 10**12

kept = AVG_PUBLISH * MSG_BYTES * RETENTION_DAYS * 86_400 * COPIES
print(f"publish at peak: {PEAK_PUBLISH:,} messages/s × {MSG_BYTES:,} bytes = {PEAK_PUBLISH * MSG_BYTES / MB:,.0f} MB/s in")
print(f"deliveries at peak with an average fan-out of {FAN_OUT}: {PEAK_PUBLISH * FAN_OUT:,} messages/s, "
      f"{PEAK_PUBLISH * FAN_OUT * MSG_BYTES / MB:,.0f} MB/s out")
print(f"{RETENTION_DAYS} days kept for replay, {COPIES} copies: one shared log {kept / TB:,.1f} TB; "
      f"a copy for every subscription {kept * FAN_OUT / TB:,.0f} TB")

publish at peak: 50,000 messages/s × 1,000 bytes = 50 MB/s in
deliveries at peak with an average fan-out of 10: 500,000 messages/s, 500 MB/s out
7 days kept for replay, 3 copies: one shared log 36.3 TB; a copy for every subscription 363 TB

Fan-out multiplies everything that happens after the publish: 50 MB a second in becomes 500,000 deliveries and 500 MB a second out. Storage depends on where messages live: 36.3 TB for one shared log, ten times as much if every subscription keeps its own copy.

The first design: one log per topic, a small state per subscription

Publishers write each message once to the topic’s log. Each subscription reads the log from its own cursor and hands messages out under a lease, pull and push dispatchers deliver them, and messages that fail too often go to a dead-letter topic.

Step one event through it, from the publish to both acknowledgements:

Six choices are still open, each in one sentence:

• A shared log with cursors: each message is stored once, and every subscription records how far it has read.
• A copy per subscription: each subscription keeps its own queue of the messages it has not acknowledged.
• Pull or push: workers ask for messages when they are ready, or the service sends each message to an endpoint.
• Acknowledgement deadlines: how long a delivered message may stay unacknowledged before it is delivered again.
• Ordering keys: messages with the same key reach a subscription in the order they were published.
• Filters and dead-letter topics: a filter passes only messages whose attributes match, and a dead-letter topic takes messages that failed too often.

The deep dives that decide whether the design works:

1. Where are messages stored when one topic has 300 subscriptions, and what does each subscription keep?
2. How long should an acknowledgement deadline be, and what happens when work takes longer?
3. When should delivery be push, and when pull?
4. How do ordering keys keep order without letting one slow key stall everything?
5. Where do filters run, and what do they do to cost and backlog?
6. What happens to a message that always fails, and to a subscriber that is away for two days?

An answer is judged on requirements and estimates, storage and fan-out, delivery and acknowledgement, ordering, failure handling and operations.

A subscription is a cursor, the acknowledgements above it and a lease table Pro: opens with a Pro, Premium or Ultimate pass