Design a chat app like WhatsApp: groups and E2EE

Groups of up to 1,000 members, photos and end-to-end encryption change a chat design in three places. A group message must reach every device of every member, so the server either adds a small entry to each device’s mailbox or, for large groups, keeps one group log that devices read from their own cursors. A photo is encrypted on the phone and uploaded once; messages carry only its id, its key and its hash, and a CDN serves the ciphertext to whoever fetches it. And with end-to-end encryptionEncryption in which only the communicating devices hold the keys: the servers that store and route the messages see ciphertext and metadata, never the content. the server no longer reads messages at all. Devices hold the keys. The server publishes public keys and lists of devices, stores and routes ciphertext, and can no longer search, preview or filter what people write. Presence and typing stay soft state that expires within seconds. The design builds on the delivery lesson and keeps its rules.

Groups multiply deliveries, and media carries the bytes

Ask these first; the answer this lesson assumes follows each question:

• How large can a group be? Up to 1,000 members, though most are far smaller. Broadcast channels with millions of followers are another product.
• Which media? Photos, voice notes and videos, compressed on the phone before they are sent.
• What is end-to-end encrypted? Everything: one-to-one chats, groups, media, and the history copied to a new device. The server still sees who writes to whom, and when.
• How many devices? A phone and up to four linked laptops or browsers per person, each with its own keys.
• What about presence? “Online” and “typing” in one-to-one chats; in groups, typing and a count of members online.

People create groups, add and remove members, send text and media to them, forward media, link a new device and see who is typing. The design is judged by four promises: a removed member cannot read what is sent after the removal; the server never holds a key that decrypts a message; a group message reaches online members within a second; and a forward never uploads the media again.

# The estimate sheet for groups and media (lesson system-design-hld/design-chat-groups-media-e2ee), on top of the
# one-to-one service of the delivery lesson: 200 million people a day. Every constant is an assumption with a name, so
# you can put your own numbers in and run it again. Sizes are decimal (1 PB = 10**15 bytes).
DAY = 86_400
PEAK = 3  # the evening peak against the daily average

GROUP_MESSAGES = 3_000_000_000  # group messages a day
DEVICES_PER_MESSAGE = 66.2  # devices a group message must reach, averaged over a mix of groups of 8 to 1,000
CIPHERTEXT_BYTES = 300  # one encrypted group message with its headers

MEDIA_ITEMS = 1_000_000_000  # photos, voice notes and videos sent a day, in chats and groups
MEDIA_MIX = [("photo", 0.85, 120_000), ("voice note", 0.12, 30_000), ("video", 0.03, 6_000_000)]  # share, bytes
DOWNLOADS_EACH = 8  # devices that download each media item, on average
KEEP_DAYS = 30  # a media blob is kept 30 days after it was last sent or forwarded

avg_bytes = sum(share * size for _, share, size in MEDIA_MIX)
deliveries = GROUP_MESSAGES * DEVICES_PER_MESSAGE
gbit = lambda bytes_a_day, peak=1: bytes_a_day * 8 * peak / DAY / 10**9

print("Groups and media: the estimate sheet")
print(f"  Group messages  {GROUP_MESSAGES / DAY:,.0f}/s on average, {GROUP_MESSAGES / DAY * PEAK:,.0f}/s at the peak")
print(f"  Deliveries      {deliveries / 10**9:,.0f} billion device deliveries a day, {deliveries / DAY * PEAK / 10**6:.1f} million/s at the peak")
print(f"  Group traffic   {gbit(deliveries * CIPHERTEXT_BYTES, PEAK):.1f} Gbit/s out at the peak, text only")
print(f"  Media           {avg_bytes / 1000:,.1f} KB per item on average; videos are {0.03 * 6_000_000 / avg_bytes:.0%} of the bytes")
up = MEDIA_ITEMS * avg_bytes
print(f"  Uploads         {up / 10**12:,.1f} TB a day: {gbit(up):,.1f} Gbit/s on average, {gbit(up, PEAK):,.1f} at the peak")
print(f"  Downloads       {up * DOWNLOADS_EACH / 10**15:,.2f} PB a day: {gbit(up * DOWNLOADS_EACH, PEAK):,.0f} Gbit/s at the peak")
print(f"  Media stored    {up * KEEP_DAYS / 10**15:,.1f} PB for {KEEP_DAYS} days of uploads, before replicas or erasure coding")

Groups and media: the estimate sheet
  Group messages  34,722/s on average, 104,167/s at the peak
  Deliveries      199 billion device deliveries a day, 6.9 million/s at the peak
  Group traffic   16.6 Gbit/s out at the peak, text only
  Media           285.6 KB per item on average; videos are 63% of the bytes
  Uploads         285.6 TB a day: 26.4 Gbit/s on average, 79.3 at the peak
  Downloads       2.28 PB a day: 635 Gbit/s at the peak
  Media stored    8.6 PB for 30 days of uploads, before replicas or erasure coding

The sheet sets the shape of the design. At the peak, 104,167 group messages a second become 6.9 million device deliveries a second, and the text alone is small: 16.6 Gbit/s. Media is the bulk of the traffic. Uploads reach 79 Gbit/s at the peak and downloads 635 Gbit/s, which only a CDN carries, and videos make 63 % of the bytes. Keeping blobs for 30 days after they were last sent or forwarded holds 8.6 PB before replication.

The first design

Follow one group message through it. The sender’s device fetches keys for any device it has not written to before, encrypts the message, and sends ciphertext through its gateway. The message service numbers it in the group’s sequence, stores it once per device or once per group, and pushes it to connected devices. Media takes its own path: the device uploads ciphertext with a signed token, and members download it from the CDN with tokens of their own. Step through it:

The questions this design has to answer:

• When should a group message go into every device’s mailbox, and when into one group log?
• How does a photo reach 1,000 members with one upload, fetched only by people who have its key?
• What can the server still do once it cannot read any message?
• How does a sender reach a member with five devices, and what if its list of devices is out of date?
• What does removing one member cost with sender keys, and what with MLS, the IETF’s group protocol?
• What do presence and typing cost in a group of 1,000?
• What breaks first at ten times the traffic?

Two choices decide most of the rest, one about delivery and one about keys. A device mailbox receives a small entry for each message its device must get. A group log stores each message once, and devices read it from their own cursors. Sender keys let a member encrypt each group message once, after sending its key to every member’s device in pairs. MLS is the IETF’s standard protocol for group keys, built for large groups whose members change.

Device mailboxes or one group log, with the writes counted Pro: opens with a Pro, Premium or Ultimate pass