System Design (High-Level Design) Module 29 – Case studies: platforms, analytics and developer tools

Design an online judge like LeetCode

Run untrusted code against hidden tests with strict limits, absorb contest bursts with queues and fair scheduling, and return fair, repeatable verdicts.

Advanced
20 minutes
Examples run with Python 3.14.8, Pyodide 314.0.7, Node.js 24.21.0 and quickjs 0.32.0
By MySmartCoPilot

Free: Ten languages, weekly contests and a verdict within 10 seconds · The estimate: 167 submissions a second keep 1,190 cores busy · Submissions are accepted at once and judged later · 3 more sections. With a pass: 3 in-depth parts and 3 interview answers, about 13 minutes.

What you will learn

Estimate the cores a contest burst needs from its arrival rate and the work per submission
Run untrusted code safely with strict limits on CPU, memory, processes and output
Scale judging for contest bursts with queues that are fair between users
Protect hidden tests and make verdicts fair and repeatable

Before you start

What you’ll learn · Before you start

What you will learn

Estimate the cores a contest burst needs from its arrival rate and the work per submission
Run untrusted code safely with strict limits on CPU, memory, processes and output
Scale judging for contest bursts with queues that are fair between users
Protect hidden tests and make verdicts fair and repeatable

Before you start

On this page

Not affiliated

Based on public information about LeetCode. MySmartCoPilot is not affiliated with LeetCode, and LeetCode’s real architecture may differ from what this lesson describes.

A contest of 40,000 people opens at a fixed time, and for its first minutes an online judge must compile and run 167 submissions a second, code from strangers, against tests its authors have never seen. Each one gets a verdict: whether it passed, with the time and memory it used.

The core idea is to treat every submission as hostile and every contest start as a flood. Each run happens in a fresh sandbox with hard limits on CPU time, wall-clock time, memory, processes and output, and with no network. Submissions go through a queue to a pool of judge workers that is sized, and started, before the contest begins, and the queue shares those workers fairly between users. Hidden tests stay on the judging side; only verdicts and per-test numbers come back. Security, bursts and fairness shape everything else.

Ten languages, weekly contests and a verdict within 10 seconds

Ask these questions first; the second column holds the answers assumed here:

Question	Assumed here
Which languages must run? Compiled ones add a compile step with its own limits	About ten
Do contests happen, and how many people join one?	Weekly, 40,000 people
How quickly must a verdict arrive?	Within 10 seconds outside contests
Is a submission judged on all tests, or does it stop at the first failure?	Every test, with a result per test

Functional:

Submit code for a problem in a chosen language.
Compile it, run it against the problem’s sample and hidden tests, and return a verdict: Accepted, Wrong Answer, Time Limit Exceeded, Run-Time Error, or Compile Error, with time and memory per test.
Show progress while judging runs, and the verdict when it ends.
Run contests with a scoreboard, and judge a whole problem again when its tests change.

Non-functional:

Isolated: no submission can read the tests, reach the network, see another submission or harm the host.
Fair: the same code gets the same verdict, measured on the same hardware with the same limits.
Fast: 95 % of verdicts within 10 seconds outside contests and within 60 seconds in a contest’s first minutes.
Durable: an accepted submission is never lost, and every verdict can be explained later.

Checkpoint

Check: What the judge must guarantee

One question Which of these does the non-functional requirement "isolated" rule out? Choose every one.
Choose every answer that is right.

A submission reaching the network A submission taking 3 seconds of CPU A submission printing a wrong answer A submission seeing another submission A submission reading the hidden tests
Show the answer: One question
Answer:
- A submission reaching the network
- A submission seeing another submission
- A submission reading the hidden tests
Isolated means no submission can read the tests, reach the network, see another submission or harm the host. Running long or answering wrongly are verdicts, not breaches.

The estimate: 167 submissions a second keep 1,190 cores busy

Assume 1.2 million practice submissions a day, a contest whose 40,000 people send 100,000 submissions in the first 30 minutes with the busiest minutes at three times that pace, and about 5 CPU-seconds per submission to compile and run some 40 tests. The script turns that into the numbers that shape the design:

The estimate sheet, from its assumptions Python · estimates/estimates.py

"""The online judge's estimates from stated assumptions: change any of them and run it again."""
DAY = 86_400  # seconds

practice_per_day = 1_200_000  # submissions outside contests
practice_peak = 3  # the busiest hour against the day's average
contest_people = 40_000
contest_submissions = 100_000  # in the first 30 minutes
contest_window = 30 * 60  # seconds
contest_peak = 3  # the busiest minutes against the contest's average
cpu_seconds = 5  # per submission: compiling and about 40 tests
busy_share = 0.7  # keep cores at most 70 % busy, so that timings stay steady
cores_per_machine = 8
code_bytes = 2_000  # per submission
result_bytes = 2_000  # per submission: time and memory for each test
problems = 3_000
test_data_mb = 20  # per problem

practice_avg = practice_per_day / DAY
contest_avg = contest_submissions / contest_window
contest_top = contest_avg * contest_peak
busy = contest_top * cpu_seconds
cores = busy / busy_share
stored = practice_per_day * (code_bytes + result_bytes) * 365

print(f"Practice: {practice_avg:.1f} submissions a second on average, {practice_avg * practice_peak:.1f} at its peak")
print(f"Contest: {contest_avg:.1f} a second over the first 30 minutes, {contest_top:.0f} in the busiest minutes")
print(f"Busy cores in those minutes: {contest_top:.0f} x {cpu_seconds} CPU-seconds = {busy:,.0f}")
print(f"Cores to have running at {busy_share:.0%} busy: {cores:,.0f} ({cores / cores_per_machine:,.0f} machines of {cores_per_machine})")
print(f"Code and results kept for a year of practice: {stored / 1e12:.2f} TB")
print(f"Test data for {problems:,} problems: {problems * test_data_mb / 1e3:.0f} GB")

Output

Practice: 13.9 submissions a second on average, 41.7 at its peak
Contest: 55.6 a second over the first 30 minutes, 167 in the busiest minutes
Busy cores in those minutes: 167 x 5 CPU-seconds = 833
Cores to have running at 70% busy: 1,190 (149 machines of 8)
Code and results kept for a year of practice: 1.75 TB
Test data for 3,000 problems: 60 GB

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 estimates.py

Compute, not storage, shapes this system. 167 submissions a second at 5 CPU-seconds each keep 833 cores busy, the arrival rate times the time each one takes (Little's law: In a stable system, the average number of requests inside equals the arrival rate times the average time each one spends inside, L = λW. At 100 requests a second and 50 ms each, about 5 are in progress. More in “Little's law, utilisation and server counts”), and running cores at most 70 % busy, so that one run does not slow another, means about 1,190 cores, or 149 machines of 8 cores, during the busiest minutes. Practice traffic at its own peak needs about a quarter of that. Storage is minor: about 1.75 TB a year of code and results, and 60 GB of test data in total.

Checkpoint

Check the estimate

Question 1 of 2 A contest receives 100,000 submissions in its first 30 minutes, and the busiest minutes run at 3 times that average. How many submissions a second arrive in those minutes?
Type a number.
Your answer submissions a second

A hint

30 minutes are 1,800 seconds.

Show the answer: Question 1 of 2

Answer: 167 submissions a second (anything from 150 to 184 counts)

100,000 ÷ 1,800 seconds ≈ 55.6 a second on average, and 3 times that is about 167. Estimating from the contest's total alone would miss that the first minutes carry most of the load.
Question 2 of 2 Each submission needs about 5 CPU-seconds to compile and run its tests. How many cores are busy at 167 submissions a second?
Type a number.
Your answer cores

A hint

Work in progress is the arrival rate times the time each item takes.

Show the answer: Question 2 of 2

Answer: 833 cores (anything from 750 to 916 counts)

167 × 5 ≈ 833 cores busy at every moment of those minutes, by the same reasoning as Little's law: the work in progress is the arrival rate times the time each item takes.

Submissions are accepted at once and judged later

POST /submissions with the problem, the language and the code: it answers 202 Accepted with a submission id, because judging takes seconds.
GET /submissions/{id}: the status, and once judged, the verdict and the time and memory of each test.
GET /submissions/{id}/events: the same progress as a stream, test by test.
GET /contests/{id}/scoreboard: the standings, cached for a few seconds.

Record	Holds
Submission	Its id, user, problem, language, status, verdict, worst time and peak memory, and a pointer to the code in object storage
Result, one per test	The test number, verdict, time and memory
Problem	Pointers to versioned test sets, each identified by the hash of its files, so a verdict always records which tests it was judged against

Checkpoint

Check: Accepted now, judged later

One question Why does POST /submissions answer 202 Accepted with an id instead of waiting for the verdict?
Choose one answer.

The verdict is never stored Judging takes seconds, so the API stores and queues the submission and the browser follows its progress by id 202 is the only status a POST may return Browsers cannot wait for a response longer than a second

Show the answer: One question

Answer: Judging takes seconds, so the API stores and queues the submission and the browser follows its progress by id

202 Accepted says the request was accepted for processing that has not finished. The submission id is the handle for GET /submissions/{id} and its event stream.

The high-level design, walked along one submission

The online judge's high-level design

Text description of the diagram

The diagram runs from top to bottom.

In the browser, a user writes code and submits it with a POST to the submission API.
The submission API checks the user, the size of the code and the user's rate, stores the submission in the submissions database (the code itself in object storage), and puts the submission id on the judge queue.
The judge queue is fair between users and serves contest work first.
Judge workers pull one submission at a time. They hold no credentials and have no network access.
For each run, a worker starts a sandbox, a microVM or a user-space kernel with cgroup and seccomp limits, and in it compiles the code and runs each test.
The test data is mounted into the sandbox read-only, as copies identified by version. Hidden tests are never sent out of the judging side.
The worker sends the result of each test to the results service, which records the verdict, the time and the memory of every test, updates the scoreboard and stores everything in the database.
A dashed arrow goes from the results service back to the browser: the user sees progress and then the verdict.

The write path. The API checks the user, the code’s size and how often this user has submitted, stores the code and the row, and puts the submission id on the judge queue. Nothing judges inside the API, so a burst only makes the queue longer.

The judging path. A worker takes one submission, starts a fresh sandbox, copies the code in and mounts the test set read-only. It compiles the code under its own limits, then runs each test with limits on CPU, wall-clock time, memory, processes and output, and compares the output with the expected answer. Results go to the results service test by test, so the user sees progress.

The read paths. Status comes from the database, progress from the event stream, and problem pages and the scoreboard from caches, so none of them competes with judging. Step through one made-up submission, from the browser to its verdict:

Interactive · message by message One submission, from the browser to its verdict Step through one submission. Each step says who acts, what is stored and what the user sees.

The drawing shows the whole scene; its 8 steps are listed below.

Every step as text

Submit. A user submits a C++ solution to a problem. The API checks who they are, the size of the code and how often they have submitted in the last minute.
Accepted at once. It stores the code and the row, puts the submission id on the judge queue and answers 202 Accepted with the id, s-77. Nothing is judged inside the API, so a burst only makes the queue longer.
Taken from the queue. A judge worker takes the next submission from the queue, which shares the workers fairly between users.
A fresh sandbox. The worker starts a fresh sandbox, copies the code in and mounts the problem's test set read-only, by the hash of its files. It compiles the code under limits of its own.
Test by test. Each test runs with limits on CPU time, wall-clock time, memory, processes and output, and its result goes back as soon as it is known.
Progress on screen. The browser follows the results on the event stream, test by test.
The verdict. After the last test, the worker reports the verdict with the worst time and the peak memory. The hidden tests never left the judging side; only verdicts and numbers did.
Explained later. The verdict is stored with the hash of the test set it was judged against, so it can be explained, or judged again, if the tests ever change.

Checkpoint

Check: The write, judging and read paths

Question 1 of 2 The first minute of a contest brings far more submissions than the workers can judge. Where does the burst land?
Choose one answer.

On the scoreboard cache On the API servers, which slow down because they judge too On the database, which must store every test result at once On the judge queue, which grows; the API keeps accepting quickly because nothing judges inside it

Show the answer: Question 1 of 2

Answer: On the judge queue, which grows; the API keeps accepting quickly because nothing judges inside it

Separating accepting from judging is what lets a burst wait instead of fail. The queue's length is then the thing to watch, and capacity is planned to keep it short.
Question 2 of 2 Why can checks that run in the learner's own browser never keep their tests hidden?
Choose one answer.

Browsers block every program that reads input The tests must be sent to the browser to run there, so anyone can read them Browsers cannot run programs at all Browser code is always slower, so the tests time out

Show the answer: Question 2 of 2

Answer: The tests must be sent to the browser to run there, so anyone can read them

Whatever runs on the learner's device must be delivered to it, and anything delivered can be read, so browser checks give fast feedback but can never prove that a solution passes tests nobody has seen.

Practise: the first design, one question at a time

Practise the four ideas above, the weakest first:

The deep dives this design must answer

Three groups of choices remain, and each has a short list of options:

Isolation: containers with seccomp and cgroups share the host’s kernel and start fastest. gVisor puts a kernel written in Go between the program and the host. Firecracker microVMs give each submission its own virtual machine.
Queueing: first come, first served is the simplest order. Round robin between users takes one submission from each waiting user in turn. A per-user cap refuses new submissions while a user already has several waiting.
Progress: polling asks for the status every second or two. Server-sent events keep one HTTP response open and push each result (HTML standard). WebSockets carry messages both ways.

MySmartCoPilot’s own exercises are a useful contrast. They run learners’ code in the learners’ browsers, on a separate sandbox origin, so each run uses the visitor’s own machine. The price is that the tests travel to the browser, where anyone can read them, so those checks are feedback, not proof: only a server-side judge can keep tests hidden.

The deep dives answer six questions: how to run hostile code without letting it out, which limits each layer must enforce, how to absorb the first minutes of a contest, how to keep one user from delaying everyone, how to make verdicts fair and repeatable, and how to keep hidden tests hidden.

Sandboxing untrusted code, layer by layer, with the limits reproduced Pro: opens with a Pro, Premium or Ultimate pass

No single mechanism makes hostile code safe, so a judge stacks layers that each stop something different. This part compares the isolation choices with their published costs and reproduces the limits a run must enforce.

In this part:

What each isolation layer stops, and its trade-offs in numbers
Limits that hold: CPU, wall clock, memory, processes and output

1 runnable example (Python) · about 6 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Read a complete sample first: Design a real-time leaderboard

Contest bursts: fair queues, warm workers and capacity, with numbers Pro: opens with a Pro, Premium or Ultimate pass

A contest is the one event whose start time is known in advance, so the judge can be ready before the flood instead of reacting to it. This part measures three queue policies on the same burst and works out how many workers must be warm when the contest opens.

In this part:

Why first come, first served fails at a contest start, measured
Workers warm before the start, worked through
Contest first, practice later, and limits on resubmission
Bottlenecks at ten and a hundred times the load

1 runnable example (Python) · about 6 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Fair verdicts, hidden tests and failure drills Pro: opens with a Pro, Premium or Ultimate pass

A verdict is only worth something if the same code gets the same answer every time and the tests stay unseen. This part reproduces the comparison and timing problems that make verdicts unfair, then drills the failures a judge must survive.

In this part:

Comparing output: exact bytes, tokens and tolerances
Time limits from reference solutions, and flaky verdicts reproduced
Keeping hidden tests hidden
Failure drills: a dead worker, changed tests and a backlog
What each level is expected to cover

2 runnable examples (Python) · about 10 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Interview questions

Fresher to mid · Warm-up

Why does an online judge limit both CPU time and wall-clock time for every run?

CPU time measures the work a program does, so it is the fair limit: it barely changes when the machine is busy with other runs. But a program that sleeps, waits for input that never comes or blocks on a lock uses almost no CPU, and under a CPU limit alone it would hold its worker forever. A wall-clock limit, usually a few times the CPU limit, catches those runs. The kernel enforces the CPU limit, through RLIMIT_CPU or a cgroup's CPU accounting, and the judge enforces the wall-clock deadline by killing the run. Either one ends in Time Limit Exceeded.
Mid to senior

A contest of 40,000 people opens at a fixed time, and its first minutes bring 167 submissions a second at about 5 CPU-seconds each. How many workers do you need, when must they be running, and how do you stop a few heavy users from delaying everyone else?

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Senior to staff

You must run untrusted C++, Java and Python submissions. Choose between containers with seccomp and cgroups, gVisor and microVMs, and defend the choice with what each layer stops and what it costs per submission.

Model answer, Pro. First step and follow-ups
Mid to senior

A participant reports that their solution got Time Limit Exceeded on test 17, while an identical resubmission a minute later was accepted. How do you investigate, and how do you make verdicts repeatable?

Model answer, Pro. First step and follow-ups

Key takeaways

An online judge is a queue in front of sandboxes: the API stores and queues, workers judge, and results come back test by test.
Contest load is compute: 167 submissions a second at 5 CPU-seconds each keep 833 cores busy, about 1,190 at 70 %.
Every run needs limits on CPU time, wall-clock time, memory, processes and output, and no network.
Tests that run in a learner’s browser are feedback, never proof; hidden tests must stay on the judging side.

References

gVisor documentation (The gVisor Authors)
Firecracker (The Firecracker project)
Control Group v2 (Linux kernel documentation) (The Linux kernel documentation)
Seccomp BPF, SECure COMPuting with filters (Linux kernel documentation) (The Linux kernel documentation)
resource, resource usage information (Python documentation) (Python Software Foundation)
Problem Package Format, version 2025-09 (ICPC)
Analysis and simulation of a fair queueing algorithm (Demers, Keshav and Shenker) (ACM SIGCOMM)
HTML Living Standard: server-sent events (WHATWG)
RFC 6585: Additional HTTP Status Codes (429 Too Many Requests) (IETF)

Report a problem with this lesson

In this module: Case studies: platforms, analytics and developer tools

Design an ad click aggregator Free core, Pro depth: 7 of 20 minutes free
Design a real-time leaderboard Free: 35 minutes
Design an online judge like LeetCode Free core, Pro depth: 7 of 20 minutes free
Design a calendar service Free intro, Pro: 4 of 20 minutes free

Your country

Design an online judge like LeetCode

What you will learn

Before you start

What you will learn

Before you start

Ten languages, weekly contests and a verdict within 10 seconds

The estimate: 167 submissions a second keep 1,190 cores busy

Submissions are accepted at once and judged later

The high-level design, walked along one submission

Practise: the first design, one question at a time

The deep dives this design must answer

Sandboxing untrusted code, layer by layer, with the limits reproduced Pro: opens with a Pro, Premium or Ultimate pass

Contest bursts: fair queues, warm workers and capacity, with numbers Pro: opens with a Pro, Premium or Ultimate pass

Fair verdicts, hidden tests and failure drills Pro: opens with a Pro, Premium or Ultimate pass

Interview questions

Why does an online judge limit both CPU time and wall-clock time for every run?

A contest of 40,000 people opens at a fixed time, and its first minutes bring 167 submissions a second at about 5 CPU-seconds each. How many workers do you need, when must they be running, and how do you stop a few heavy users from delaying everyone else?

You must run untrusted C++, Java and Python submissions. Choose between containers with seccomp and cgroups, gVisor and microVMs, and defend the choice with what each layer stops and what it costs per submission.

A participant reports that their solution got Time Limit Exceeded on test 17, while an identical resubmission a minute later was accepted. How do you investigate, and how do you make verdicts repeatable?

Key takeaways

References

Related tools