Serverless and edge computing: when they fit

An API that answers 2,000 requests a second in 120 ms keeps 240 copies of its code busy on a serverless platform, and 4,000 if a slow dependency stretches each request to 2 seconds. Serverless functions run your code in response to a request or an event, in short-lived environments that the provider creates and scales, and you pay for the time they run. Edge computing runs small pieces of code at a content delivery network’s locations near users, before a request travels to your region.

Both take servers off your list of worries and add limits instead. A serverless function usually handles one request per environment at a time, runs for at most 15 minutes per invocation on AWS Lambda, sometimes starts cold, and keeps nothing between invocations you can count on. Use functions for spiky, event-driven work, use edge computing for quick decisions about the request itself, such as redirects and token checks, and give every piece of state one home outside both.

How functions run, and what the edge adds

For each request in flight, a Lambda function in its default mode gets its own execution environmentexecution environment: The isolated runtime a function platform creates to run one copy of a function's code. It handles one request at a time and may be kept and reused for later requests., so the number of environments it needs is the request rate times the request duration (Lambda concurrency). That is how the slow dependency above turns 240 environments into 4,000, four times the default account limit of 1,000 per Region, and the rest fail with a throttling error. Step through that arithmetic:

Edge code lives with tighter limits: CloudFront Functions run in under a millisecond with 2 MB of memory and no network access, while Lambda@Edge may call other services and run for up to 30 seconds (edge functions compared). The picture shows where each kind of code sits:

What the rest of this lesson works through, one line each:

• Cold starts: a new environment runs its setup code before it answers, so lighter setup shortens each one.
• Snapshots: new environments resume from a snapshot of one already set up.
• Provisioned concurrency: environments kept ready, for a charge, so requests up to that number start warm.
• A queue in front: requests wait instead of failing while the function adds environments.
• Reserved concurrency: a cap and a guarantee of environments for one function.
• Connection poolers: each environment holds its own database connection, so a burst can exhaust the database’s connections. A pooler lets thousands share a few.
• Single-writer objects: one owner per key that applies updates one at a time.

The AWS documentation says cold starts typically occur in under 1 % of invocations and take from under 100 ms to over a second. The model below shows what that does to the tail when the handler alone takes 40 ms at the median:

"""Latency of a function when some invocations start cold.
A warm invocation runs only the handler; a cold one first creates the execution environment and runs the
initialisation code. The AWS Lambda documentation says cold starts typically occur in under 1 % of
invocations and take from under 100 ms to over 1 second. The durations below are assumptions inside that range:
   - handler: median 40 ms, p99 about 130 ms (lognormal);
   - extra time of a cold start: median 600 ms, p99 about 1,900 ms (lognormal)."""

import math
import random
import statistics

random.seed(11)
REQUESTS = 200_000


def handler_ms():
    return random.lognormvariate(math.log(40), 0.5)


def cold_extra_ms():
    return random.lognormvariate(math.log(600), 0.5)


def latencies(cold_share):
    return [handler_ms() + (cold_extra_ms() if random.random() < cold_share else 0.0) for _ in range(REQUESTS)]


print("cold starts     p50 ms   p95 ms   p99 ms   p99.9 ms")
for share in (0.0, 0.001, 0.01, 0.05):
    q = statistics.quantiles(latencies(share), n=1000)
    print(f"{share:>10.1%} {q[499]:>10.0f} {q[949]:>8.0f} {q[989]:>8.0f} {q[998]:>10.0f}")

cold starts     p50 ms   p95 ms   p99 ms   p99.9 ms
      0.0%         40       91      128        191
      0.1%         40       91      129        275
      1.0%         40       96      230       1206
      5.0%         41      212      958       1707

At 1 % the median does not move, but the p99p99 (99th percentile): The latency within which 99 % of requests finish, so the slowest 1 % take longer. Tail latency is read from such high percentiles, because an average hides the slowest requests. More in “Latency, throughput and percentiles” goes from 128 ms to 230 ms and the p99.9 passes a second. Cold starts are a tail problem, and a busy service’s users meet its tail every day.

Cold starts live in the tail; fix them cheapest first Pro: opens with a Pro, Premium or Ultimate pass