System Design (High-Level Design) Module 19 – Building blocks: traffic, storage and messaging
Design a content delivery network
Design a CDN from requirements and estimates to points of presence, cache tiers, request collapsing, purges that reach every PoP and origin protection.
What you will learn
- Estimate a CDN's egress, servers per point of presence and origin load from its requirements
- Design points of presence, request routing and a cache hierarchy in front of an origin
- Measure what owner servers, cache tiers and request collapsing do to hit ratios and origin load
- Design purges that reach every point of presence quickly and safely, including unreachable ones
- Plan certificates, configuration, failure drills and growth to ten times the traffic
Before you start
What you’ll learn · Before you start
What you will learn
- Estimate a CDN's egress, servers per point of presence and origin load from its requirements
- Design points of presence, request routing and a cache hierarchy in front of an origin
- Measure what owner servers, cache tiers and request collapsing do to hit ratios and origin load
- Design purges that reach every point of presence quickly and safely, including unreachable ones
- Plan certificates, configuration, failure drills and growth to ten times the traffic
Before you start
On this page
At 3 million requests a second, 4.8 Tbit/s, a site’s files cannot all come from its own servers. A content delivery network (CDN) answers them from caching servers in many cities, called point of presence (PoP): One of a content delivery network's sites: a group of caching servers in a city close to users, which answers their requests from its caches and goes to the origin only on a miss. (PoPs), and goes to the site’s own servers, the origin, only on a miss.
Designing one means five decisions. Routing sends each user to a nearby PoP, by anycast or by DNS steering. A PoP of many servers must act as one cache, not many small ones. Tiers above the PoPs, a regional tier and a shield, catch misses before they reach the origin. Request collapsing turns many simultaneous misses for one object into a single fetch. And a change at the origin must reach every PoP, through purges or through new addresses.
Requirements and estimate: the busiest PoP needs 16 servers
Ask first what is served (files, images, video segments, cacheable API answers), for how many customer domains, how fast a purge must work and where the users are. This design takes these answers:
- Functional: serve cached
GETandHEADresponses for many customer domains, each with its own certificate, cache rules and origin; follow HTTP caching (RFC 9111); purge by URL and by tag; deliver each customer’s logs. - Non-functional: 3 million requests a second at peak, 200 KB per response on average; 95 % answered by the PoP; a purge applied at 99 % of PoPs within 2 seconds; keep serving when a PoP or the origin fails.
| Quantity | Working | Result |
|---|---|---|
| Egress | 3 million requests a second × 200 KB | 600 GB/s, 4.8 Tbit/s |
| Busiest PoP | 5 % of the traffic | 150,000 requests a second, 240 Gbit/s |
| Its servers | 25 Gbit/s cards run at 60 % | 16 servers |
| PoP misses | 5 % of 3 million | 150,000 a second |
| Origin requests | a shield answers 80 % of the misses | 30,000 a second |
The same estimate as a program, to change and run again:
# The first estimate for a CDN. Every input is an assumption, written down so that you can change it and run the
# script again.
import math
PEAK = 3_000_000 # requests a second at peak, every PoP together
RESPONSE_BYTES = 200_000 # average response, 200 KB
POPS = 60
BUSIEST_PERCENT = 5 # the busiest PoP's share of the traffic
CARD_GBIT = 25 # network card of one cache server
CARD_PERCENT = 60 # run the cards at 60 % at peak
POP_HIT_PERCENT = 95 # requests the PoPs answer themselves
SHIELD_HIT_PERCENT = 80 # PoP misses that the shield answers
gbit = PEAK * RESPONSE_BYTES * 8 / 1e9
print("Egress")
print(f" {PEAK:,} requests a second x 200 KB = {gbit / 8:,.0f} GB/s, {gbit / 1000:.1f} Tbit/s")
print(f" over {POPS} PoPs: {gbit / POPS:.0f} Gbit/s each on average, but traffic is never even")
busiest = PEAK * BUSIEST_PERCENT // 100
busiest_gbit = busiest * RESPONSE_BYTES * 8 / 1e9
per_server = CARD_GBIT * CARD_PERCENT / 100
servers = math.ceil(busiest_gbit / per_server)
print("The busiest PoP")
print(f" {BUSIEST_PERCENT} % of the traffic: {busiest:,} requests a second, {busiest_gbit:.0f} Gbit/s")
print(f" {CARD_GBIT} Gbit/s cards at {CARD_PERCENT} %: {busiest_gbit:.0f} / {per_server:.0f} = {servers} servers")
misses = PEAK * (100 - POP_HIT_PERCENT) // 100
origin = misses * (100 - SHIELD_HIT_PERCENT) // 100
print("The origin")
print(f" PoP misses: {100 - POP_HIT_PERCENT} % of {PEAK:,} = {misses:,} a second")
print(f" past a shield that answers {SHIELD_HIT_PERCENT} % of them: {origin:,} a second")
print(f" without the shield it would see all {misses:,}")Output
Egress 3,000,000 requests a second x 200 KB = 600 GB/s, 4.8 Tbit/s over 60 PoPs: 80 Gbit/s each on average, but traffic is never even The busiest PoP 5 % of the traffic: 150,000 requests a second, 240 Gbit/s 25 Gbit/s cards at 60 %: 240 / 15 = 16 servers The origin PoP misses: 5 % of 3,000,000 = 150,000 a second past a shield that answers 80 % of them: 30,000 a second without the shield it would see all 150,000
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 cdn_estimate.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Checkpoint
Check the estimate
The first design: one cache per PoP, and two tiers above it
The first complete design: a PoP, two tiers above it, and the control plane
Text description of the diagram
Users (1) reach a nearby point of presence, one of 60, by anycast or by DNS steering.
Inside the point of presence, layer-4 load balancers spread connections over 16 cache servers. The server that receives a request terminates TLS and has the request served by the object's owner, the one server in the point of presence that stores that object.
On a miss (2), the request goes to the regional tier of its region. On a miss there (3), it goes to the shield next to the origin, and only on a miss at the shield (4) to the customer's origin. Concurrent misses for the same object are collapsed into one fetch.
A control plane pushes configuration, purges and certificates to every point of presence and collects their logs.
Along the request path: anycast or DNS steering brings a user to a nearby PoP, where layer-4 load balancers spread connections over its 16 cache servers. The server that terminates TLS builds the cache key: The value a cache files a stored response under: at least the method and the full URL, plus the request headers the response names in Vary. Requests with the same key get the same stored response. More in “CDNs and edge caching” and has the object’s owner in the PoP serve it. A miss goes to the regional tier, then to a shield next to the origin, and only then to the origin, with concurrent misses for one object collapsed into one fetch. A control plane pushes configuration, purges and certificates to every PoP and collects their logs. Follow one request that misses everywhere:
The drawing shows the whole scene; its 7 steps are listed below.
Every step as text
- To a nearby PoP. A user asks for a product image. Anycast or DNS steering brings the request to a nearby point of presence, one of 60.
- The server that owns it. A layer-4 load balancer hands the connection to one of the 16 cache servers, which ends TLS, builds the cache key and asks the key's owner, the one server of the PoP that stores this object.
- A miss at the PoP. The owner has no fresh copy. The PoPs answer 95 % of requests themselves, and this one is among the 5 % that go on, 150,000 a second across the CDN.
- The regional tier. The miss goes to the regional tier of the PoP's region, which serves all the PoPs there. It has no copy either.
- The shield. Next comes the shield next to the origin. In the estimate a shield answers 80 % of the PoPs' misses; this request is one of the rest.
- The origin answers. Only now does the request reach the origin, which sees 30,000 requests a second instead of 150,000. On the way back each tier can keep a copy for the next user.
- Misses collapsed. Other users who missed the same object at the same owner while this fetch was in flight did not start fetches of their own. They waited for this one, so concurrent misses cost the origin a single fetch.
The deep dives this design must answer:
- Anycast or DNS steering: anycast announces one address from every PoP; DNS steering answers each resolver with a chosen PoP.
- Owners inside a PoP: hashing each cache key to one server turns 16 small caches into one large one.
- Tiered caching: a regional tier and a shield answer misses before the origin sees them.
- Request collapsing: concurrent misses for one object wait for a single fetch.
- Stale-while-revalidate and stale-if-error: a cache serves a stale copy while it refreshes, or while the origin fails.
- Purge fan-out: every PoP must apply each purge and confirm it, even one that was unreachable.
- Surrogate keys: tags on cached responses let one purge remove every object that carries the tag.
- Certificate automation: ACME issues and renews the certificates of every customer domain.
- Cache poisoning: a harmful response stored in a shared cache reaches every user of that cache.
Two or three of these usually fill the second half of a round. The rubric looks at the questions asked, the numbers, the request path, the chosen deep dives, the failures handled and the plan for growth.
Checkpoint
Check the first design
Owners by hashing turn sixteen caches into one Pro: opens with a Pro, Premium or Ultimate pass
In this part:
- A regional tier and a shield cut origin requests 2.3 times
- Collapsing turns 625 fetches into one, and stale copies remove the wait
- A purge counts only when every PoP has applied it
- Anycast or DNS steering, and the headroom to drain a PoP
- Certificates renew 0.75 times a second, and logs reach 104 TB a day
- Failure drills: seven ways the design is tested
- At 10× the ports give out first, at 100× the control plane
- What each level is expected to cover
- A sample answer in ten minutes
- A 45-minute self-mock
- Practise: collapse the fetches
- Count origin fetches with request collapsing
- Interview questions
- What is an origin shield in a CDN, and what does it cost?
- A legal notice requires an image to be gone from every PoP within 10 minutes. The purge reached 59 of your 60 PoPs in a second, but one PoP has been unreachable from the control plane for 20 minutes while still serving users. What do you do now, and what would you build so this is never a scramble?
- A game publisher releases a 2 GB patch at a fixed time. Within a minute of the release its origin falls over, although your CDN reports a 97 % hit ratio for the customer. Explain what happened and how the CDN should protect the origin next time.
- Users of one PoP report that a customer's homepage loads scripts from a domain nobody recognises. The origin builds absolute script addresses from the X-Forwarded-Host request header. Explain how that page got into the cache and how you fix it in the CDN and at the origin.
- Key takeaways
2 diagrams · 4 runnable examples (JavaScript, Python) · 1 exercise with sample tests · 3 interview answers · about 44 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Terms in this lesson
- point of presence (PoP)
- One of a content delivery network's sites: a group of caching servers in a city close to users, which answers their requests from its caches and goes to the origin only on a miss.
References
- RFC 9111: HTTP Caching (IETF)
- RFC 5861: HTTP Cache-Control Extensions for Stale Content (IETF)
- RFC 4786: Operation of Anycast Services (IETF)
- RFC 7871: Client Subnet in DNS Queries (IETF)
- RFC 8555: Automatic Certificate Management Environment (ACME) (IETF)
- Ballot SC-081v3: Introduce schedule of reducing validity and data reuse periods (CA/Browser Forum)
- Request collapsing (Fastly documentation)
- Shielding (Fastly documentation)
- Purging (Fastly documentation)
- Caching overview (Cloud CDN) (Google Cloud)
- Invalidate files to remove content (Amazon CloudFront Developer Guide) (Amazon Web Services)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress