Growing a service from one server to many regions

Almost every large service started small. The design that fits a hundred users would collapse under ten million, and the design for ten million would bury a small team serving a hundred. This lesson follows one service through the stages services usually pass on the way up, says what runs out first at each stage and which component fixes it, and then turns the picture around: every component you add has a price, and paying it too early is a design mistake.

Stage 1: one server

Everything runs on one machine: the app, the database and the files. It is cheap, simple to deploy and easy to debug, because there is only one place to look. For a prototype, an internal tool or a young product it is often the right design, and for longer than people expect. What breaks first is that everything competes for the same CPU, memory and disk, and that the machine is a single point of failure: a crash, a bad deploy or a full disk takes the whole service down, and a backup is the only protection for the data.

Each stage fixes the bottleneck of the one before

Each stage adds one thing, because of one bottleneck, and has a price:

2. A separate database server, because the app and the database compete for one machine. It costs a network hop on every query and a second machine to run.
3. A load balancer and several stateless app servers, because one app server’s CPU is full and every deploy takes the site down. State must leave the servers, the load balancer needs health checks, and it is one more component.
4. A CDN for static files, because the servers spend their bandwidth sending the same images and scripts. Old files stay cached far away, so file names carry versions, and there is one more provider.
5. A cache in front of the database, because the database answers the same reads again and again. It brings stale data, invalidation, and a cold cache after a restart.
6. Read replicas, because reads still outgrow one database. Replication lag means a user may not see their own change at once.
7. A queue and workers for slow work, because emails, thumbnails and reports make requests slow, or make them fail when a provider is slow. That work now finishes later, may be retried or done twice, and leaves a backlog to watch.
8. Sharding, because writes or the size of the data outgrow one primary database. Joins and transactions across shards stop being easy, and data has to move as shards fill.
9. More regions, because distant users wait on every request and one region’s outage takes everything down. Data must be copied between regions, writes conflict or slow down, and running the service gets much harder.

Each stage has a source you can check. Shared caches in front of a service exist to cut response time and bandwidth by reusing stored responses (the HTTP caching standard, RFC 9111). With the cache-aside pattern, the app loads data into the cache on demand and has to decide how long an entry may live: too short and the database still does the work, too long and readers see stale data. A PostgreSQL standby answers read-only queries, and because changes reach it after a delay, the same query on the primary and the standby can return different answers. A queue between a task and a slow service lets the service work at its own pace and be sized for the average load instead of the peak. And a single database server eventually runs out of storage, computing power or network bandwidth, which is when the data is divided into shards that each hold part of it.

By stage 8 the same service, still in one region, looks like this:

Stateless servers make scaling out easy

Stage 3 is the one that unlocks most of the rest. A stateless app server keeps nothing between requests that another server would need: sessions live in a shared store or a signed token (the session trade-off of an earlier lesson), uploads go to object storage, and anything that must last goes to a database. Then any server can answer any request, so the load balancer can send traffic anywhere, a failed server can simply be replaced, and new versions can be deployed one server at a time while the others keep serving.

The Twelve-Factor App states the rule bluntly: its processes are stateless and share nothing, anything that must persist lives in a backing service such as a database, and sticky sessions are not allowed. It also explains why the rule pays: one machine can only grow so large, but share-nothing processes can be multiplied. AWS’s reliability guidance makes the same point from the failure side: replace one large resource with several small ones, so that losing one has a small effect.

Watch it grow: a toy model

This program raises the traffic of two different services tenfold at a time. At each step it works out how busy each resource is, fixes the busiest one that is above 70% (the rest is headroom for peaks and for losing a machine), and repeats until everything fits. Every capacity at the top is an assumption, not a measurement.

"""A toy model of a service that grows: raise the traffic tenfold at a time, find the resource that is busiest
past the headroom limit, apply the usual fix for it, and repeat until everything fits again.
Every capacity below is an assumption for the sketch, not a measurement: change them and run it again."""

import math


def count(n, word):
    """'1 write', '2 writes', '0.05 writes'."""
    return f"{n} {word}" if n == 1 else f"{n} {word}s"

HEADROOM = 0.7  # keep every resource under 70% busy, to absorb peaks and the loss of one machine
APP_SERVER = 500  # requests a second one app server handles
DB_READS = 3000  # simple reads a second one database server answers
DB_WRITES = 1000  # writes a second one database server takes
NETWORK = 1250  # MB a second one load balancer can send (10 Gbit/s)
SHARED = 0.5  # app and database on one machine: each gets about half of it
CACHE_HITS = 0.9  # share of reads a cache answers, once there is one

WORKLOADS = {
    "A read-heavy site": {"reads": 2, "writes": 0.05, "kb": 100, "static_kb": 80},
    "A write-heavy tracker": {"reads": 0.2, "writes": 1, "kb": 2, "static_kb": 0},
}


def busy(rate, w, s):
    """How busy each resource is (1.0 = full) at `rate` requests a second."""
    share = 1 if s["own_db"] else SHARED
    reads = rate * w["reads"] * (1 - CACHE_HITS if s["cache"] else 1)
    sent_kb = w["kb"] - (w["static_kb"] if s["cdn"] else 0)
    return {
        "app servers": rate / (APP_SERVER * share * s["app"]),
        "database reads": reads / (DB_READS * share * (s["shards"] + s["replicas"])),
        "database writes": rate * w["writes"] / (DB_WRITES * share * s["shards"]),
        "network": rate * sent_kb / 1000 / (NETWORK * s["balancers"]),
    }


def fix(resource, rate, w, s):
    """Change the design so that `resource` has room again; say what was done."""
    if resource != "network" and not s["own_db"]:
        s["own_db"] = True
        return "move the database to a server of its own"
    if resource == "app servers":
        s["app"] = math.ceil(rate / (APP_SERVER * HEADROOM))
        return f"run {s['app']} stateless app servers behind a load balancer"
    if resource == "database reads" and not s["cache"]:
        s["cache"] = True
        return "put a cache in front of the database"
    if resource == "database reads":
        reads = rate * w["reads"] * (1 - CACHE_HITS)
        s["replicas"] = math.ceil(reads / (DB_READS * HEADROOM)) - s["shards"]
        return f"add read replicas: {s['replicas']} in all"
    if resource == "database writes":
        before = s["shards"]
        s["shards"] = math.ceil(rate * w["writes"] / (DB_WRITES * HEADROOM))
        s["replicas"] = max(0, s["replicas"] - (s["shards"] - before))  # each new shard serves reads too
        return f"split the data by key into {s['shards']} shards"
    if not s["cdn"]:
        s["cdn"] = True
        return "serve images, scripts and styles from a CDN"
    s["balancers"] = math.ceil(rate * (w["kb"] - w["static_kb"]) / 1000 / (NETWORK * HEADROOM))
    return f"spread traffic over {s['balancers']} load balancers"


for name, w in WORKLOADS.items():
    print(f"{name}: per request {count(w['reads'], 'read')}, {count(w['writes'], 'write')} and {w['kb']} KB sent")
    s = {"own_db": False, "app": 1, "cache": False, "replicas": 0, "shards": 1, "cdn": False, "balancers": 1}
    for rate in (10, 100, 1_000, 10_000, 100_000):
        steps = []
        while True:
            load = busy(rate, w, s)
            resource = max(load, key=load.get)
            if load[resource] <= HEADROOM:
                break
            steps.append(f"{resource} at {load[resource]:.0%}: {fix(resource, rate, w, s)}")
        busiest = max(load, key=load.get)
        print(f"  {rate:,} requests a second, busiest {busiest} at {load[busiest]:.0%}")
        for step in steps:
            print(f"    {step}")
    parts = [count(s["app"], "app server"), count(s["shards"], "database shard"), count(s["replicas"], "read replica")]
    parts += [label for label, used in (("a cache", s["cache"]), ("a CDN", s["cdn"])) if used]
    print(f"  In the end: {', '.join(parts)} and {count(s['balancers'], 'load balancer')}")
    print()

A read-heavy site: per request 2 reads, 0.05 writes and 100 KB sent
  10 requests a second, busiest app servers at 4%
  100 requests a second, busiest app servers at 40%
  1,000 requests a second, busiest app servers at 67%
    app servers at 400%: move the database to a server of its own
    app servers at 200%: run 3 stateless app servers behind a load balancer
  10,000 requests a second, busiest app servers at 69%
    app servers at 667%: run 29 stateless app servers behind a load balancer
    database reads at 667%: put a cache in front of the database
    network at 80%: serve images, scripts and styles from a CDN
  100,000 requests a second, busiest app servers at 70%
    app servers at 690%: run 286 stateless app servers behind a load balancer
    database reads at 667%: add read replicas: 9 in all
    database writes at 500%: split the data by key into 8 shards
    network at 160%: spread traffic over 3 load balancers
  In the end: 286 app servers, 8 database shards, 2 read replicas, a cache, a CDN and 3 load balancers

A write-heavy tracker: per request 0.2 reads, 1 write and 2 KB sent
  10 requests a second, busiest app servers at 4%
  100 requests a second, busiest app servers at 40%
  1,000 requests a second, busiest app servers at 67%
    app servers at 400%: move the database to a server of its own
    app servers at 200%: run 3 stateless app servers behind a load balancer
    database writes at 100%: split the data by key into 2 shards
  10,000 requests a second, busiest app servers at 69%
    app servers at 667%: run 29 stateless app servers behind a load balancer
    database writes at 500%: split the data by key into 15 shards
  100,000 requests a second, busiest app servers at 70%
    app servers at 690%: run 286 stateless app servers behind a load balancer
    database writes at 667%: split the data by key into 143 shards
  In the end: 286 app servers, 143 database shards, 0 read replicas and 1 load balancer

Read the first service, a read-heavy site, from the top:

• At 10 and 100 requests a second nothing is added. One server is enough, and anything more would be cost and work for nothing.
• At 1,000, the shared machine runs out first, so the database moves to its own server; then the app needs three servers behind a load balancer.
• At 10,000, the database would be answering the same reads 20,000 times a second, so a cache goes in front of it, and the network fills with images and scripts, so they move to a CDN.
• At 100,000, even the reads that miss the cache outgrow one database, writes outgrow one primary, and the data is split into shards with read replicas.

The second service, a tracker that mostly writes small records, grows differently. It never needs a cache or a CDN, but the database’s writes run out at 1,000 requests a second, and by 100,000 the model wants 143 shards. That number is the real lesson: when the only fix you know is “more shards”, a write-heavy workload needs a different design, such as batching writes, putting them through a log, or a store built for heavy writes, which later modules cover. The order of the stages is not a law. It follows the workload, which is why the estimates of a design round come before the diagram.

The model is a sketch for reasoning, with one fix per resource; it knows nothing about latency, failures or cost. Change the capacities or the workloads at the top and run it again to see how the order moves.

Do not add parts before the load needs them

Every stage on that list adds failure modes, work for whoever is on call, and money. Azure’s guidance on reliability puts it plainly: each component added to make a system resilient also makes it more complex to run. In concrete terms:

• A cache brings invalidation bugs, stale reads, and a stampede on the database when it restarts empty.
• A read replica brings users who do not see the change they just made.
• A queue brings work that happens later, twice, or not at all, and a backlog someone must watch.
• Shards take away joins and transactions across shards and bring the work of moving data when one fills up.
• More regions bring conflicting or slower writes and double the infrastructure.

So add a component when a measured or estimated bottleneck needs it, and write down the trigger with it: “add a read replica when the database’s CPU stays above 60% at the daily peak”. In a design round, a design for a few thousand users with a message log, a cache cluster, sixteen shards and three regions is not a strong answer. A strong answer says what the design looks like now, and which component it would add at which number.

Many regions

The last stage deserves its own warning. Going to several regions makes sense when users far away pay for the distance on every request (no hardware makes a signal cross an ocean faster), when the service must survive the loss of a whole region, or when rules say where data must stay. The hard part is always the data: which region accepts writes, and how they reach the others. The write-path trade-off of the quality-attributes lesson applies with large numbers: synchronous copies between regions make every write wait for an ocean crossing, asynchronous ones can lose the last writes or let two regions accept conflicting ones. The two common shapes are active-passive, where one region takes the writes while another stands ready, and active-active, where every region serves users, which is faster for them and much harder to keep correct. A later module of this track covers both in depth.

Key takeaways

• Services grow in stages: one server, a separate database, stateless app servers behind a load balancer, a CDN, a cache, read replicas, queues for slow work, shards, then more regions.
• Each stage removes the bottleneck of the one before and adds failure modes, operational work and cost.
• Stateless app servers are what make scaling out, replacing servers and rolling deploys easy.
• The order depends on the workload: read-heavy services lean on caches and CDNs, write-heavy ones on the write path.
• Add a component when a measured or estimated bottleneck needs it, and write down the number that triggers it.