Load testing, capacity planning and chaos tests

A load testload test: Sending realistic traffic to a system at rising rates to measure what it really sustains, such as the requests a second one server serves within its latency goal, instead of guessing it. More in “Little's law, utilisation and server counts” reported a p99 of 2.5 ms for a server that had stopped answering for a second and a half, because the test waited instead of sending. Load testing measures how a system behaves under a defined load: how many requests it completes, how slowly and with how many errors as the load rises. Capacity planning turns a forecast of demand into the resources that serve it, with a margin for failures, and load testing supplies the number it depends on most: what one instance carries while meeting its objective. Chaos tests check the plan under failure: they inject one realistic fault, such as a lost zone or a slow dependency, and compare the result with a hypothesis about normal behaviour. Together they answer one question before users do: will the system meet its objective at the next peak, even when something breaks?

Two mistakes make load tests flatter, and a plan needs three numbers

Two mistakes make load testing report better numbers than users will see. A closed-model test runs a fixed number of virtual users, each waiting for its response before sending again, so when the system slows down the test slows its own arrival rate; an open-model test starts requests at a fixed rate whatever happens, as independent users do (k6: open and closed models). And a load generator that waits for slow responses never sends the requests that would have queued behind them, so the worst moments vanish from its percentiles. That second mistake is coordinated omissioncoordinated omission: The gap in a load test whose tester waits for each reply: it sends nothing while the system is slow, so the slow requests users would have made never appear, and its percentiles look better than they are. More in “Latency, throughput and percentiles” (wrk2). An open-model test avoids it, and a log that has it can be corrected by timing each request from when it was due, or by adding the missing samples back.

Capacity planning needs three numbers: the peak to serve, the measured limit of one instance, and the failure the plan must survive. A worked plan for a service whose weekly peak grew from 6,100 to 6,640 requests a second over its last eight weekly peaks, about 1.22% a week:

Step	Value
Forecast peak 26 weeks ahead	9,099 requests a second
Measured limit of one instance with a p99 within 300 ms	450 requests a second
Planned load per instance, 70% of that limit	315 requests a second
Instances per zone if one of three zones is lost	9,099 ÷ 2 ÷ 315 = 14.4, so 15
Instances in all	45, at about 45% of their limit with every zone up

Each number in the table comes from a test or a decision. The rest of the lesson builds them:

• Open and closed models, measured against the same server.
• Coordinated omission, reproduced and corrected.
• Arrival-rate executors and the virtual users they need.
• Capacity plans for steady growth, planned events and a lost zone.
• Recovery tests that check a system comes back after overload.
• Chaos tests with a hypothesis, a small blast radius and an automatic stop.

A closed test can only offer what the server completes Pro: opens with a Pro, Premium or Ultimate pass

In this part:

• Coordinated omission, reproduced and corrected
• An arrival-rate test needs rate times latency in virtual users
• A capacity plan from a forecast, a measured limit and a lost zone
• Recovery tests: after the overload, does it come back?
• Chaos tests with a hypothesis, a blast radius and a stop button
• Practise: correct a latency log for coordinated omission
  • Correct a closed-loop latency log for coordinated omission
• Interview questions
  • What is coordinated omission in a load test, and why does it make latency look better than it is?
  • Your load test with 200 virtual users reports a p99 of 120 ms at the target rate, yet production reaches a p99 of 2 s at the same request rate. What could the test be doing wrong, and how do you fix it?
  • A shop expects a sale in eight weeks with three times its normal peak of 6,000 requests a second. Plan the capacity: what do you measure, what do you compute, and what do you check before the day?
  • Design a chaos experiment that checks whether checkout survives losing one of its three zones at the peak. What is the hypothesis, how do you limit the blast radius, and when do you stop?
• Key takeaways

3 diagrams · 4 runnable examples (JavaScript, Python) · 1 exercise with sample tests · 3 interview answers · about 49 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

• Pro pass · 1 week · ₹79 · $2.99
• 30 days · ₹179 · $5.99
• 1 year · ₹1,699 · $45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a passI have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Read a complete sample first: Design a unique ID generator