Cells, deployment stamps and blast radius

A service for a million customers that runs as one deployment shares every failure with all of them: one bad deploy stops everyone at once. A cell is a complete, independent copy of the service, with its own compute and data, serving only a subset of the customers; a thin cell router in front sends each request to its customer’s cell, chosen by a partition keypartition key: The field of a request or a record, such as a tenant id or a user id, that decides which partition, shard or cell owns it. such as the tenant id.

Cells share nothing, so a failure or a bad deploy stays inside one of them: AWS’s guidance gives the example of 10 cells serving 100 requests, where one failed cell leaves 90 % of them unaffected. Azure calls the same idea deployment stamps, also scale units, and the share of customers one failure can reach is the blast radius. Use cells when a single shared failure is unacceptable and the workload divides cleanly by customer, and expect to pay for it: more copies to run, a router simpler than anything behind it, and real work whenever a customer moves.

One service cut into cells, each a full copy

The AWS Well-Architected guidance names three parts: cells, each a complete workload; a cell router, which it calls the thinnest possible layer, whose only job is to send requests to the right cell; and a control plane that creates cells, removes them and migrates customers. Cells need not mean more machines: thirty hosts can stay thirty hosts, grouped into cells. Azure’s deployment stamps give each stamp a predefined number of tenants and add stamps to grow almost linearly, and they also use stamps for staged updates and for keeping a tenant’s data in one region.

The rest of this lesson takes each design decision in turn:

• Partition key: the field, usually a customer or tenant id, that decides which cell owns a request.
• Cell router: a mapping from that key to a cell, kept so simple that it rarely changes.
• Cell size: the most load a cell may carry, set from a load test rather than hope.
• Tenant migration: moving one customer’s data and traffic to another cell while it keeps working.
• Staged rollout: deploying to one cell first and growing the waves only after each has baked.
• Shuffle sharding: giving every customer its own small random set of workers instead of a fixed cell.

The model below puts numbers on the blast radius of a failed cell and of a bad build, for one, four and sixteen cells, when every rollout goes one cell first and then in growing waves:

"""Customers hit by one failure, for a service in 1, 4 or 16 cells of equal size.
A failed cell takes down only its own customers. A bad build is rolled out cell by cell in waves of 1, 1, 2, 4, 8
cells, baking 30 minutes after each wave. A fast bug is noticed 10 minutes after the first cell gets it, a slow bug
50 minutes after; rolling back every cell that has it then takes 5 minutes. Deploying to everything at once is the
1-cell row. Every number is an assumption of the model."""

CUSTOMERS = 1_000_000
BAKE_MIN = 30
ROLLBACK_MIN = 5


def waves(cells):
    """Cells per wave: one, one, then doubling, until every cell has the build."""
    plan, done = [], 0
    for size in (1, 1, 2, 4, 8, 16, 32):
        if done >= cells:
            break
        size = min(size, cells - done)
        plan.append(size)
        done += size
    return plan


def bad_build(cells, noticed_after_min):
    """Customers who got the bad build, and customer-minutes of impact until it is rolled back everywhere."""
    per_cell = CUSTOMERS / cells
    rolled_back_at = noticed_after_min + ROLLBACK_MIN
    hit = minutes = 0.0
    for i, size in enumerate(waves(cells)):
        starts = i * BAKE_MIN
        if starts >= noticed_after_min:  # the rollout stops once the bug is noticed
            break
        hit += size * per_cell
        minutes += size * per_cell * (rolled_back_at - starts)
    return hit, minutes


print(f"{'cells':>5}{'one cell fails':>16}{'fast bug: hit':>15}{'customer-minutes':>18}{'slow bug: hit':>15}{'customer-minutes':>18}{'rollout':>9}")
for cells in (1, 4, 16):
    fast_hit, fast_min = bad_build(cells, 10)
    slow_hit, slow_min = bad_build(cells, 50)
    rollout = (len(waves(cells)) - 1) * BAKE_MIN
    print(f"{cells:>5}{1 / cells:>16.2%}{fast_hit:>15,.0f}{fast_min:>18,.0f}{slow_hit:>15,.0f}{slow_min:>18,.0f}{rollout:>6} min")

cells  one cell fails  fast bug: hit  customer-minutes  slow bug: hit  customer-minutes  rollout
    1         100.00%      1,000,000        15,000,000      1,000,000        55,000,000     0 min
    4          25.00%        250,000         3,750,000        500,000        20,000,000    60 min
   16           6.25%         62,500           937,500        125,000         5,000,000   120 min

Step through the 16-cell row with the fast bug, wave by wave:

With 16 cells, a bug noticed ten minutes after the first deploy reaches 62,500 customers instead of a million. The price is time: rolling a good build out to all 16 cells in waves takes two hours instead of one deploy.

The router never sends a tenant to another cell Pro: opens with a Pro, Premium or Ultimate pass