Web Security (OWASP Top 10:2025) Module 17 – Secure code review and routine

Review clinic: find and fix planted flaws

Review two small apps with twenty planted flaws, fix each with a test that fails on the original, rank the findings and write the review report.

Advanced
35 minutes
Examples run with Node.js 24.21.0, quickjs 0.32.0, Python 3.14.8, Pyodide 314.0.7 and sql.js 1.14.2
By MySmartCoPilot

Free: Run the clinic in six stages, and spend the hours where flaws hide · Score a review by what it found, what it missed and what it invented · A fix is finished when its test fails on the original code. With a pass: 3 in-depth parts and 3 interview answers, about 24 minutes.

What you will learn

Find and classify twenty planted weaknesses in a Node service and a Python service, from A01 to A10
Write a minimal fix for each flaw, with a test that fails on the original code
Rank the findings by likelihood and impact, and write the review report
Compare your findings with a reference list, and turn each miss into a check

Before you start

Secure code review: a repeatable method

What you’ll learn · Before you start

What you will learn

Find and classify twenty planted weaknesses in a Node service and a Python service, from A01 to A10
Write a minimal fix for each flaw, with a test that fails on the original code
Rank the findings by likelihood and impact, and write the review report
Compare your findings with a reference list, and turn each miss into a check

Before you start

Secure code review: a repeatable method

On this page

A scan of the shop’s vulnerable build raised 15 alerts, three of them High, and none about who may read whose order. Read by a person, two small apps hold twenty planted flaws, ten in a Node service and ten in a Python one, with at least one in every Top 10:2025 category from A01 to A10.

The clinic runs the review of the last lesson as an exercise. You follow a time plan, score a review by what it found, missed and invented, close each flaw with a test that fails on the original code, rank the findings, and turn every miss into a check for the next review. Programs that ran score a practice slice, check ten fixes in each app and rank all twenty findings.

Run the clinic in six stages, and spend the hours where flaws hide

A review that opens the first file and reads on spends its hours on whatever it meets first. The clinic runs in six stages and 165 minutes, with the most time on following inputs and on fixes, because those two find and close the most. The stages are the seven passes of the last lesson, grouped in the order that wastes least. Step through them:

Interactive · stage by stage The clinic in six stages and 165 minutes Step through the stages. Each one says what to do and what to hand on, and the last step shows where the review's misses go.

The drawing shows the whole scene; its 8 steps are listed below.

Every step as text

Two apps, twenty flaws. The clinic is two small apps with ten planted flaws each. A plan of 165 minutes sends the hours to where flaws hide. The two longest stages are following inputs and fixing, and the shortest is the report.
Orient. List the routes, the files that handle them and the inputs each route reads, and note which routes sit behind a sign-in. Nothing is a finding yet. The list tells you where the next 45 minutes go.
Trace inputs. Follow each input to its sinks, as in the seven passes. A search finds where to read and the reading decides. Most of what a person can find by following data is found in this stage.
Access and logic. Ask who may do what on every route, and which fields a client may set that it should not. A scanner is blind here, because it does not know your rules. A check that is missing is a finding.
Config and errors. Read the secrets, the settings, the packages and the way failures are reported. These are quick to read and easy to forget when the code looks right.
Fix and test. Fix the findings with the highest rank first, each with a test that fails on the original code and passes on the fix. A fix without such a test is a guess, and it can come back.
Report. Rank the findings by likelihood and impact, write each with its place, CWE, ASVS requirement, fix and test, and say what the review did not cover. A reader should know what to do first.
Misses become checks. Compare your list with the reference list. Each miss gets a line in your checklist or a test of its own, so that the next review catches it. A review that teaches nothing to the next one has to be done again.

Two rules keep the plan honest. Stop a stage when its time is up and write down what it did not reach, since an unfinished stage is information for the report. And fix nothing until the findings are ranked, because the minutes for fixes are few and belong to the flaws that matter most.

Checkpoint

Check: The six stages of the clinic

Question 1 of 2 Which two stages of the clinic get the most minutes, 45 each?
Choose one answer.

Orient, and report Orient, and access and logic Trace inputs, and fix and test Access and logic, and config and errors

Show the answer: Question 1 of 2

Answer: Trace inputs, and fix and test

The plan gives 10, 45, 30, 20, 45 and 15 minutes to its six stages, 165 in all. Tracing inputs and fixing take the most.
Question 2 of 2 A route skips the sign-in check entirely. Which stage of the plan is the one that finds it?
Choose one answer.

Orient Access and logic Config and errors Report

Show the answer: Question 2 of 2

Answer: Access and logic

A check that is not there leaves no line to search for. Stage 3 asks the question of every route, in turn, and that is how it is found.

Score a review by what it found, what it missed and what it invented

A review is scored by a finding matcher: The rule that scores a review against the planted flaws. A finding is a hit when it names the same family of weakness within a few lines of a planted flaw; otherwise it is a duplicate or a false positive.. A finding is a line and a CWE. The matcher treats two CWEs as the same when they name one family of weakness, such as the two cookie flags, and accepts a line up to two away from the planted one. Each finding is then a hit, a duplicate of a flaw already found, or a false positive that points at nothing planted. The planted flaw: A weakness put into a training app on purpose, at a known line and with a known CWE, so that a review can be scored on what it finds, misses and invents. that nobody found are the misses. A hit earns 10 points and a false positive costs 5, so a long list of guesses does worse than a short list of facts.

The program below holds a payments file of 30 lines with five planted flaws, and the matcher scoring one review of it. The file is a warm-up and not one of the twenty:

Five planted flaws, one review and its score JavaScript · practice/slice_score.mjs

// A practice slice: 30 lines of a payments route file with five planted flaws, and the score of one review of it.
// A finding is a hit when it names the same family of weakness within 2 lines of a planted flaw, a second finding for a
// flaw that was already found is a duplicate, and any other finding is a false positive. A hit earns 10 points and a
// false positive costs 5. Press Edit, change the REVIEW list, and run it again.
const SLICE = `const express = require('express');
const { exec } = require('node:child_process');
const https = require('node:https');
const jwt = require('jsonwebtoken');
const router = express.Router();
const gateway = new https.Agent({ rejectUnauthorized: false });

router.post('/login', async (req, res) => {
  console.log('login attempt', req.body);
  const user = await checkPassword(req.body.email, req.body.password);
  if (!user) return res.status(401).send('Wrong email or password');
  res.cookie('sid', await newSession(user.id));
  res.send('Signed in');
});

function whoIs(req) {
  return jwt.decode(req.cookies.token);
}

router.get('/receipt.png', requireUser, (req, res) => {
  exec('convert receipts/' + req.query.id + '.pdf out.png', () => res.sendFile('out.png'));
});

router.post('/pay', requireUser, async (req, res) => {
  const user = whoIs(req);
  const paid = await charge(gateway, user.id, req.body.cents);
  res.json({ paid });
});

module.exports = router;
`;

const PLANTED = [
  { line: 6, cwe: 'CWE-295' },
  { line: 9, cwe: 'CWE-532' },
  { line: 12, cwe: 'CWE-1004' },
  { line: 17, cwe: 'CWE-347' },
  { line: 21, cwe: 'CWE-78' },
];

// CWE identifiers that name the same weakness for a reviewer, grouped in a family. A CWE that is not listed is its own family.
const FAMILY = {
  'CWE-78': 'injection', 'CWE-89': 'injection', 'CWE-79': 'injection', 'CWE-94': 'injection',
  'CWE-295': 'transport', 'CWE-319': 'transport',
  'CWE-347': 'token', 'CWE-345': 'token',
  'CWE-532': 'logging', 'CWE-117': 'logging',
  'CWE-1004': 'cookie', 'CWE-614': 'cookie',
  'CWE-639': 'access', 'CWE-862': 'access', 'CWE-863': 'access', 'CWE-285': 'access',
};
const TOLERANCE = 2;

const REVIEW = [
  { line: 21, cwe: 'CWE-78' },
  { line: 20, cwe: 'CWE-78' },
  { line: 12, cwe: 'CWE-614' },
  { line: 6, cwe: 'CWE-295' },
  { line: 10, cwe: 'CWE-89' },
  { line: 17, cwe: 'CWE-639' },
];

const family = (cwe) => FAMILY[cwe] ?? cwe;

function score(planted, findings) {
  const taken = new Set();
  const rows = [];
  for (const f of [...findings].sort((a, b) => a.line - b.line)) {
    const near = planted
      .map((p, i) => ({ p, i, gap: Math.abs(p.line - f.line) }))
      .filter(({ p, gap }) => family(p.cwe) === family(f.cwe) && gap <= TOLERANCE);
    const open = near.filter(({ i }) => !taken.has(i)).sort((a, b) => a.gap - b.gap || a.p.line - b.p.line);
    if (near.length === 0) rows.push({ f, verdict: 'false positive' });
    else if (open.length === 0) rows.push({ f, verdict: 'duplicate' });
    else {
      taken.add(open[0].i);
      rows.push({ f, verdict: 'hit' });
    }
  }
  const count = (verdict) => rows.filter((r) => r.verdict === verdict).length;
  const hits = count('hit');
  const falsePositives = count('false positive');
  return { rows, hits, falsePositives, duplicates: count('duplicate'), missed: planted.filter((_, i) => !taken.has(i)), points: Math.max(0, 10 * hits - 5 * falsePositives) };
}

const lines = SLICE.split('\n');
const result = score(PLANTED, REVIEW);
console.log(`the payments slice: ${lines.length - 1} lines, ${PLANTED.length} planted flaws, ${REVIEW.length} findings`);
for (const { f, verdict } of result.rows) {
  console.log(`line ${String(f.line).padStart(2)}  ${f.cwe.padEnd(9)} ${verdict.padEnd(14)} ${lines[f.line - 1].trim().slice(0, 52)}`);
}
console.log(`found ${result.hits} of ${PLANTED.length}; ${result.falsePositives} false positives, ${result.duplicates} duplicate`);
console.log(`missed: ${result.missed.map((p) => `line ${p.line} (${family(p.cwe)})`).join(', ') || 'nothing'}`);
console.log(`points: ${result.hits} x 10 - ${result.falsePositives} x 5 = ${result.points}`);

Output

the payments slice: 30 lines, 5 planted flaws, 6 findings
line  6  CWE-295   hit            const gateway = new https.Agent({ rejectUnauthorized
line 10  CWE-89    false positive const user = await checkPassword(req.body.email, req
line 12  CWE-614   hit            res.cookie('sid', await newSession(user.id));
line 17  CWE-639   false positive return jwt.decode(req.cookies.token);
line 20  CWE-78    hit            router.get('/receipt.png', requireUser, (req, res) =
line 21  CWE-78    duplicate      exec('convert receipts/' + req.query.id + '.pdf out.
found 3 of 5; 2 false positives, 1 duplicate
missed: line 9 (logging), line 17 (token)
points: 3 x 10 - 2 x 5 = 20

Recorded with Node.js 24.21.0 on macOS 26 arm64. To run it yourself: mise exec node@24.21.0 -- node slice_score.mjs

The review found 3 of the 5 and scored 20. Another CWE of the cookie family counted, and the command injection reported twice cost nothing. It missed the password written to the log on line 9 and the token decoded and never verified on line 17, and its two false positives, a SQL injection on line 10 where no query is visible and an access flaw on line 17, took away 10 points.

Find the other two

Press Edit and add { line: 9, cwe: 'CWE-532' } and { line: 17, cwe: 'CWE-347' } to REVIEW, delete the two false positives, and run it again. All five are found and the score is 50.

Checkpoint

Check: Scoring a review by hits, misses and false positives

Question 1 of 2 One flaw is planted on line 41. A review reports the same weakness twice, on lines 40 and 41, in the same family. How does the matcher score the two findings?
Choose one answer.

One hit and one duplicate Two hits Two false positives One hit and one false positive

Show the answer: Question 1 of 2

Answer: One hit and one duplicate

The matcher reads findings in order of line. The one on line 40 finds the flaw, and the one on line 41 repeats it.
Question 2 of 2 A review has 4 hits, 3 false positives and 1 duplicate. How many points does it score?
Type a number.
Your answer points

A hint

A hit earns 10 points, a false positive costs 5, and a duplicate scores nothing.

Show the answer: Question 2 of 2

Answer: 25 points

4 hits earn 40 points and 3 false positives cost 15, so the review scores 25. The duplicate changes nothing.

A fix is finished when its test fails on the original code

Finding a flaw is half of the work; the other half is closing it so that it stays closed. Each fix in the clinic comes with a canary test: A regression test for one flaw that sends a harmless canary input to the handler. It fails on the vulnerable code, passes on the fix and on the case that was always valid, and stays in the suite.: a regression test: A test kept after a fix so that the weakness cannot come back unnoticed. In a security lab it fails on the weak version and passes on the fixed one, and it runs again on every later change. More in “Start here: learn web security safely and legally” that sends a harmless canary input: A harmless test value holding characters that matter to an interpreter, such as an apostrophe for SQL or quotes and angle brackets for HTML. A test fails if the value changes the query's or the page's structure. More in “SQL injection and parameterised queries” to the handler. It checks both sides, because a fix that only makes the canary pass can break the behaviour it was meant to protect:

The code	The input to refuse	The case that was always valid, such as your own order
Vulnerable	The test fails, so it notices the flaw	The test passes
Fixed	The test passes	The test passes

A test that passes on the vulnerable code tests nothing.

The test that was never red

Wrong: make the fix, write a test, see it pass and move on. Right: run the test on the original first and see it fail, then make the fix and see it pass. A test you never saw fail may not look at the flaw at all.

Checkpoint

Check: A fix is finished when its test fails on the original

Question 1 of 2 What must be true of a canary test before the fix is made?
Choose one answer.

It is written in the same commit as the fix It sends a real attack to a live system It passes on the vulnerable code It fails on the vulnerable code

Show the answer: Question 1 of 2

Answer: It fails on the vulnerable code

A canary has two duties. It fails on the vulnerable code and passes on the fix, and the first is what makes the second mean anything.
Question 2 of 2 The canary for a discount-code field sends a code of a thousand characters and expects it to be refused. What else should the same canary check, so that a bad fix cannot pass it?
Choose one answer.

That the refusal comes in under a second Nothing, because a refusal proves the flaw is closed That the refusal repeats the long code back to the sender That an ordinary code still gets its discount

Show the answer: Question 2 of 2

Answer: That an ordinary code still gets its discount

A canary tests the refusal and the case that must still work. Without the second, the easiest fix is to refuse everything.

ShopNest's ten flaws: where each sits in the standards, how it is fixed, and the canary that guards it Pro: opens with a Pro, Premium or Ultimate pass

ShopNest is a small shop service in Node. This part lists its ten planted flaws with their CWE, Top 10 category and ASVS requirement, shows each fix beside the vulnerable code, and runs a canary for every flaw on both builds.

In this part:

Ten flaws, and where each one sits in the standards
The fixes, and the canary that guards each one

1 runnable example (JavaScript, fixed.mjs, weak.mjs) · about 4 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

NoteBox's ten flaws in Python, four that no response shows, and a canary that also runs in a browser Pro: opens with a Pro, Premium or Ultimate pass

NoteBox is a small notes service in Flask. This part lists its ten planted flaws, shows why four of them leave no mark in any response, and runs a canary for each on both builds, in Python that also runs in your browser.

In this part:

Ten more flaws, and four that no response shows
The fixes, in a canary that also runs in a browser

1 runnable example (Python, fixed.py, weak.py) · about 4 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Rank twenty findings, prove each fix against spoiled copies, write the report, and turn each miss into a check Pro: opens with a Pro, Premium or Ultimate pass

Twenty findings and 45 minutes for fixes need an order. This part ranks the findings with OWASP’s likelihood and impact, proves a fix against spoiled copies of itself, lays out a report that a team can act on, and turns each miss into a line of the next review.

In this part:

Rank twenty findings by likelihood and impact
A fix is proven when no spoiled copy of it passes
A report a team can act on
From a miss to a check

1 diagram · 2 runnable examples (JavaScript, Python) · about 9 minutes

Opening this part with your pass…

This part opens with any MySmartCoPilot pass.

Pro pass · 1 week₹79$2.99
30 days₹179$5.99
1 year₹1,699$45

One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.

Choose a pass I have a licence key

It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).

Practise: score a review, then close a flaw

The matcher scores a review, and the receipt fix closes the command injection that the practice slice plants. The sample tests run in your browser, in JavaScript or Python:

Code lab · 2 steps

Score a review, then fix what it found

Write the matcher that scores a review of the practice slice, then fix its command injection and prove the fix with canaries.

Score a review against the planted flaws To do
Fix the receipt id and prove the fix with canaries To do

Exercise · Medium · JavaScript, Python

Score a review against the planted flaws

Write scoreReview(planted, findings, family) (Python: score_review), the matcher of the clinic. planted is a list of {line, cwe}: the flaws that were planted. findings is a list of {line, cwe}: what a reviewer reported. family maps a CWE to a family name, such as 'CWE-614' to 'cookie'; a CWE that is not in it is a family of its own.

Go through the findings in order of line number (findings on the same line keep the order they were given). A planted flaw is near a finding when it has the same family and its line differs by at most 2, and each finding is one of these:

a false positive when no planted flaw is near it;
a duplicate when planted flaws are near it, but all of them were already found by earlier findings;
a hit otherwise. It finds one planted flaw: of the flaws near it that were not found yet, the one whose line is closest, and the lower line when two are equally close.

Return an object with hits, misses (the planted flaws that no finding found), falsePositives (Python: false_positives), duplicates and points. A hit earns 10 points and a false positive costs 5; the points never fall below 0. A duplicate scores nothing and costs nothing.

The starter counts every finding as a hit.

JavaScript · Starter code · score.mjs

/** Scores a review against the planted flaws (see the prompt). */
export function scoreReview(planted, findings, family = {}) {
  // Starter: every finding counts as a hit.
  const hits = findings.length;
  return { hits, misses: Math.max(0, planted.length - hits), falsePositives: 0, duplicates: 0, points: 10 * hits };
}

The sample tests · score.test.mjs

import { test, assert } from 'mysmartcopilot:test';
import { scoreReview } from './score.mjs';

const FAMILY = { 'CWE-614': 'cookie', 'CWE-1004': 'cookie', 'CWE-78': 'injection', 'CWE-89': 'injection' };
const PLANTED = [{ line: 6, cwe: 'CWE-295' }, { line: 12, cwe: 'CWE-1004' }, { line: 21, cwe: 'CWE-78' }];
const counts = (r) => [r.hits, r.misses, r.falsePositives, r.duplicates, r.points];

test('a review that finds every flaw and nothing else scores 10 for each', () => {
  const review = PLANTED.map((p) => ({ ...p }));
  assert.deepEqual(counts(scoreReview(PLANTED, review, FAMILY)), [3, 0, 0, 0, 30]);
});

test('an empty review finds nothing and scores 0', () => {
  assert.deepEqual(counts(scoreReview(PLANTED, [], FAMILY)), [0, 3, 0, 0, 0]);
});

test('a line at most 2 away counts, and 3 away does not', () => {
  assert.deepEqual(counts(scoreReview(PLANTED, [{ line: 8, cwe: 'CWE-295' }], FAMILY)), [1, 2, 0, 0, 10]);
  assert.deepEqual(counts(scoreReview(PLANTED, [{ line: 9, cwe: 'CWE-295' }], FAMILY)), [0, 3, 1, 0, 0]);
  assert.deepEqual(counts(scoreReview(PLANTED, [{ line: 4, cwe: 'CWE-295' }], FAMILY)), [1, 2, 0, 0, 10]);
});

test('a CWE of the same family counts, and a CWE of another family at the same line does not', () => {
  assert.deepEqual(counts(scoreReview(PLANTED, [{ line: 12, cwe: 'CWE-614' }], FAMILY)), [1, 2, 0, 0, 10]);
  assert.deepEqual(counts(scoreReview(PLANTED, [{ line: 12, cwe: 'CWE-89' }], FAMILY)), [0, 3, 1, 0, 0]);
  assert.deepEqual(counts(scoreReview(PLANTED, [{ line: 12, cwe: 'CWE-614' }], {})), [0, 3, 1, 0, 0]);
});

test('a second finding for a flaw that was found is a duplicate and costs nothing', () => {
  const review = [{ line: 21, cwe: 'CWE-78' }, { line: 20, cwe: 'CWE-89' }];
  assert.deepEqual(counts(scoreReview(PLANTED, review, FAMILY)), [1, 2, 0, 1, 10]);
});

test('each flaw is found once: two close flaws take the two findings, the lower line first on a tie', () => {
  const close = [{ line: 10, cwe: 'CWE-78' }, { line: 12, cwe: 'CWE-78' }];
  const review = [{ line: 11, cwe: 'CWE-78' }, { line: 11, cwe: 'CWE-78' }, { line: 11, cwe: 'CWE-78' }];
  assert.deepEqual(counts(scoreReview(close, review, FAMILY)), [2, 0, 0, 1, 20]);
});

test('findings are read in order of line, so an early finding does not take the flaw that a later one needs', () => {
  const far = [{ line: 10, cwe: 'CWE-78' }, { line: 14, cwe: 'CWE-78' }];
  const review = [{ line: 12, cwe: 'CWE-78' }, { line: 10, cwe: 'CWE-78' }];
  assert.deepEqual(counts(scoreReview(far, review, FAMILY)), [2, 0, 0, 0, 20]);
  assert.deepEqual(review.map((f) => f.line), [12, 10]);
});

test('a finding takes the closest flaw that was not found yet, even if a later finding then has none left', () => {
  const close = [{ line: 10, cwe: 'CWE-78' }, { line: 13, cwe: 'CWE-78' }];
  assert.deepEqual(counts(scoreReview(close, [{ line: 11, cwe: 'CWE-78' }, { line: 13, cwe: 'CWE-78' }], FAMILY)), [2, 0, 0, 0, 20]);
  assert.deepEqual(counts(scoreReview(close, [{ line: 12, cwe: 'CWE-78' }, { line: 13, cwe: 'CWE-78' }], FAMILY)), [1, 1, 0, 1, 10]);
});

test('on a tie the lower line is taken, however the planted flaws are listed', () => {
  const review = [{ line: 11, cwe: 'CWE-78' }, { line: 13, cwe: 'CWE-78' }];
  for (const lines of [[12, 10], [10, 12]]) {
    const planted = lines.map((line) => ({ line, cwe: 'CWE-78' }));
    assert.deepEqual(counts(scoreReview(planted, review, FAMILY)), [2, 0, 0, 0, 20], lines.join(' '));
  }
});

test('three findings given out of order are read in order of line', () => {
  const planted = [{ line: 10, cwe: 'CWE-78' }, { line: 14, cwe: 'CWE-78' }, { line: 30, cwe: 'CWE-78' }];
  const review = [{ line: 30, cwe: 'CWE-78' }, { line: 12, cwe: 'CWE-78' }, { line: 10, cwe: 'CWE-78' }];
  assert.deepEqual(counts(scoreReview(planted, review, FAMILY)), [3, 0, 0, 0, 30]);
});

test('false positives cost 5 each, and the points never fall below 0', () => {
  const review = [{ line: 6, cwe: 'CWE-295' }, { line: 30, cwe: 'CWE-89' }];
  assert.deepEqual(counts(scoreReview(PLANTED, review, FAMILY)), [1, 2, 1, 0, 5]);
  const noisy = [{ line: 6, cwe: 'CWE-295' }, ...[30, 31, 32, 33].map((line) => ({ line, cwe: 'CWE-89' }))];
  assert.deepEqual(counts(scoreReview(PLANTED, noisy, FAMILY)), [1, 2, 4, 0, 0]);
});

Python · Starter code · score.py

def score_review(planted, findings, family=None):
    """Scores a review against the planted flaws (see the prompt)."""
    # Starter: every finding counts as a hit.
    hits = len(findings)
    return {"hits": hits, "misses": max(0, len(planted) - hits), "false_positives": 0, "duplicates": 0, "points": 10 * hits}

The sample tests · test_score.py

from score import score_review

FAMILY = {"CWE-614": "cookie", "CWE-1004": "cookie", "CWE-78": "injection", "CWE-89": "injection"}
PLANTED = [{"line": 6, "cwe": "CWE-295"}, {"line": 12, "cwe": "CWE-1004"}, {"line": 21, "cwe": "CWE-78"}]


def counts(r):
    return [r["hits"], r["misses"], r["false_positives"], r["duplicates"], r["points"]]


def test_a_review_that_finds_every_flaw_and_nothing_else_scores_10_for_each():
    review = [dict(p) for p in PLANTED]
    assert counts(score_review(PLANTED, review, FAMILY)) == [3, 0, 0, 0, 30]


def test_an_empty_review_finds_nothing_and_scores_0():
    assert counts(score_review(PLANTED, [], FAMILY)) == [0, 3, 0, 0, 0]


def test_a_line_at_most_2_away_counts_and_3_away_does_not():
    assert counts(score_review(PLANTED, [{"line": 8, "cwe": "CWE-295"}], FAMILY)) == [1, 2, 0, 0, 10]
    assert counts(score_review(PLANTED, [{"line": 9, "cwe": "CWE-295"}], FAMILY)) == [0, 3, 1, 0, 0]
    assert counts(score_review(PLANTED, [{"line": 4, "cwe": "CWE-295"}], FAMILY)) == [1, 2, 0, 0, 10]


def test_a_cwe_of_the_same_family_counts_and_a_cwe_of_another_family_at_the_same_line_does_not():
    assert counts(score_review(PLANTED, [{"line": 12, "cwe": "CWE-614"}], FAMILY)) == [1, 2, 0, 0, 10]
    assert counts(score_review(PLANTED, [{"line": 12, "cwe": "CWE-89"}], FAMILY)) == [0, 3, 1, 0, 0]
    assert counts(score_review(PLANTED, [{"line": 12, "cwe": "CWE-614"}], {})) == [0, 3, 1, 0, 0]


def test_a_second_finding_for_a_flaw_that_was_found_is_a_duplicate_and_costs_nothing():
    review = [{"line": 21, "cwe": "CWE-78"}, {"line": 20, "cwe": "CWE-89"}]
    assert counts(score_review(PLANTED, review, FAMILY)) == [1, 2, 0, 1, 10]


def test_each_flaw_is_found_once_two_close_flaws_take_the_two_findings_the_lower_line_first_on_a_tie():
    close = [{"line": 10, "cwe": "CWE-78"}, {"line": 12, "cwe": "CWE-78"}]
    review = [{"line": 11, "cwe": "CWE-78"}, {"line": 11, "cwe": "CWE-78"}, {"line": 11, "cwe": "CWE-78"}]
    assert counts(score_review(close, review, FAMILY)) == [2, 0, 0, 1, 20]


def test_findings_are_read_in_order_of_line_so_an_early_finding_does_not_take_the_flaw_that_a_later_one_needs():
    far = [{"line": 10, "cwe": "CWE-78"}, {"line": 14, "cwe": "CWE-78"}]
    review = [{"line": 12, "cwe": "CWE-78"}, {"line": 10, "cwe": "CWE-78"}]
    assert counts(score_review(far, review, FAMILY)) == [2, 0, 0, 0, 20]
    assert [f["line"] for f in review] == [12, 10]


def test_a_finding_takes_the_closest_flaw_that_was_not_found_yet_even_if_a_later_finding_then_has_none_left():
    close = [{"line": 10, "cwe": "CWE-78"}, {"line": 13, "cwe": "CWE-78"}]
    assert counts(score_review(close, [{"line": 11, "cwe": "CWE-78"}, {"line": 13, "cwe": "CWE-78"}], FAMILY)) == [2, 0, 0, 0, 20]
    assert counts(score_review(close, [{"line": 12, "cwe": "CWE-78"}, {"line": 13, "cwe": "CWE-78"}], FAMILY)) == [1, 1, 0, 1, 10]


def test_on_a_tie_the_lower_line_is_taken_however_the_planted_flaws_are_listed():
    review = [{"line": 11, "cwe": "CWE-78"}, {"line": 13, "cwe": "CWE-78"}]
    for lines in ([12, 10], [10, 12]):
        planted = [{"line": line, "cwe": "CWE-78"} for line in lines]
        assert counts(score_review(planted, review, FAMILY)) == [2, 0, 0, 0, 20], lines


def test_three_findings_given_out_of_order_are_read_in_order_of_line():
    planted = [{"line": 10, "cwe": "CWE-78"}, {"line": 14, "cwe": "CWE-78"}, {"line": 30, "cwe": "CWE-78"}]
    review = [{"line": 30, "cwe": "CWE-78"}, {"line": 12, "cwe": "CWE-78"}, {"line": 10, "cwe": "CWE-78"}]
    assert counts(score_review(planted, review, FAMILY)) == [3, 0, 0, 0, 30]


def test_false_positives_cost_5_each_and_the_points_never_fall_below_0():
    review = [{"line": 6, "cwe": "CWE-295"}, {"line": 30, "cwe": "CWE-89"}]
    assert counts(score_review(PLANTED, review, FAMILY)) == [1, 2, 1, 0, 5]
    noisy = [{"line": 6, "cwe": "CWE-295"}] + [{"line": line, "cwe": "CWE-89"} for line in (30, 31, 32, 33)]
    assert counts(score_review(PLANTED, noisy, FAMILY)) == [1, 2, 4, 0, 0]

A hint

Keep a set of the indexes of the planted flaws that were found. For each finding, build the list of planted flaws that are near it (same family, lines at most 2 apart) before you decide anything: an empty list is a false positive, a list in which every index is already in the set is a duplicate, and otherwise you sort the unfound ones by distance and then by line and take the first. Sort the findings by line before the loop, on a copy, so that the caller's list does not change.

The sample tests run on this device, in your browser (QuickJS, Pyodide): nothing is sent to mysmartcopilot.com. The first run of each language downloads it: JavaScript (about 0.6 MB) or Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.

Exercise · Easy · JavaScript, Python

Fix the receipt id and prove the fix with canaries

The receipt route of the practice slice builds a command from the id in the query string, as 'convert receipts/' + id + '.pdf out.png'. Write receiptCommand(id) (Python: receipt_command(id)) so that it returns the argument list to run, never one string: ['convert', 'receipts/<id>.pdf', 'out.png'].

An id is a text of 1 to 9 digits, 0 to 9 only, and the whole text must be digits: leading zeros are fine, anything else is refused. For a refused id, throw an Error (Python: raise ValueError). A value that is not text is refused too.

The sample tests are canaries: they fail on the starter, which pastes the id into one string and checks nothing, and they pass on the fix.

JavaScript · Starter code · receipt.mjs

/** The command to run for a receipt id, as an argument list (see the prompt). */
export function receiptCommand(id) {
  // Starter: the id is pasted into one string and nothing is checked.
  return [`convert receipts/${id}.pdf out.png`];
}

The sample tests · receipt.test.mjs

import { test, assert } from 'mysmartcopilot:test';
import { receiptCommand } from './receipt.mjs';

const refused = (id) => assert.throws(() => receiptCommand(id), Error, JSON.stringify(id));

test('an ordinary id gives an argument list of three items', () => {
  assert.deepEqual(receiptCommand('1042'), ['convert', 'receipts/1042.pdf', 'out.png']);
});

test('the valid case still works at its edges: one digit, nine digits and leading zeros', () => {
  assert.deepEqual(receiptCommand('7'), ['convert', 'receipts/7.pdf', 'out.png']);
  assert.deepEqual(receiptCommand('123456789'), ['convert', 'receipts/123456789.pdf', 'out.png']);
  assert.deepEqual(receiptCommand('007'), ['convert', 'receipts/007.pdf', 'out.png']);
});

test('a canary with a separator or a space in it is refused', () => {
  for (const id of ['12;34', '12 34', '1042 && 5', '$(5)', '12`5`']) refused(id);
});

test('a canary that leaves the folder is refused', () => {
  for (const id of ['../1042', '1042/../7', '..']) refused(id);
});

test('an empty id, a final newline and ten digits are refused', () => {
  for (const id of ['', '1042\n', '\n1042', '1234567890']) refused(id);
});

test('digits of other scripts are refused: only 0 to 9 count', () => {
  refused('١٢٣');
  refused('１２');
});

test('a value that is not text is refused', () => {
  for (const id of [1042, null, undefined, ['1042']]) refused(id);
});

Python · Starter code · receipt.py

def receipt_command(id):
    """The command to run for a receipt id, as an argument list (see the prompt)."""
    # Starter: the id is pasted into one string and nothing is checked.
    return [f"convert receipts/{id}.pdf out.png"]

The sample tests · test_receipt.py

from receipt import receipt_command


def refused(id):
    try:
        receipt_command(id)
    except ValueError:
        return
    raise AssertionError(f"{id!r} should be refused")


def test_an_ordinary_id_gives_an_argument_list_of_three_items():
    assert receipt_command("1042") == ["convert", "receipts/1042.pdf", "out.png"]


def test_the_valid_case_still_works_at_its_edges_one_digit_nine_digits_and_leading_zeros():
    assert receipt_command("7") == ["convert", "receipts/7.pdf", "out.png"]
    assert receipt_command("123456789") == ["convert", "receipts/123456789.pdf", "out.png"]
    assert receipt_command("007") == ["convert", "receipts/007.pdf", "out.png"]


def test_a_canary_with_a_separator_or_a_space_in_it_is_refused():
    for id in ("12;34", "12 34", "1042 && 5", "$(5)", "12`5`"):
        refused(id)


def test_a_canary_that_leaves_the_folder_is_refused():
    for id in ("../1042", "1042/../7", ".."):
        refused(id)


def test_an_empty_id_a_final_newline_and_ten_digits_are_refused():
    for id in ("", "1042\n", "\n1042", "1234567890"):
        refused(id)


def test_digits_of_other_scripts_are_refused_only_0_to_9_count():
    refused("١٢٣")
    refused("１２")


def test_a_value_that_is_not_text_is_refused():
    for id in (1042, None, ["1042"], b"1042"):
        refused(id)

A hint

Check the id before you build anything. Use a pattern that matches the whole text and nothing else: in JavaScript ^[0-9]{1,9}$ is exact, but in Python use re.fullmatch, because $ also matches before a final newline. Write the digits as [0-9], since in Python \d accepts digits of other scripts. Build the list from the checked id last.

Then check yourself on the three ideas above, weakest first:

Interview questions

Warm-up: what is a canary test, and what must it do on the vulnerable and on the fixed code?

A canary test is a regression test for one flaw that sends a harmless input to the handler. It must fail on the vulnerable code, which shows that it notices the flaw, and pass on the fixed code and on the case that was always valid, so that a fix which refuses everything cannot satisfy it. It stays in the suite and runs on every change.

Mid to staff

A scan of your team's two test apps raises 15 alerts and misses most of the 20 flaws that a review knows about. A manager asks why you still review code and write tests. What do you answer, and what do you change in the pipeline?

Model answer, Pro. Model answers open with any pass; prices are in the notice on this page. First step and follow-ups
Fresher to senior

A teammate fixes an open redirect and adds one test: an address on another site is sent to the home page. The test passes. Is the fix done, and what do you ask for?

Model answer, Pro. First step and follow-ups
Mid to staff

Your review has twenty findings, the release is on Friday, and you have two hours for fixes. How do you decide what to fix, what to defer and what to tell the release owner?

Model answer, Pro. First step and follow-ups

Flashcards · 6 cards

Recall it, then check

Say the answer to yourself before you look: recalling it is what makes it stick.

How does the review clinic spend its time, and which two rules keep the plan honest?

Six stages in 165 minutes, with the most time on following inputs and on fixes, because those find and close the most. Stop a stage when its time is up and write down what it did not reach, since an unfinished stage is information for the report. And fix nothing until the findings are ranked, because the minutes for fixes belong to the flaws that matter most.
How does the clinic's matcher score a review?

A finding is a line and a CWE. The matcher treats two CWEs as the same when they name one family of weakness, such as the two cookie flags, and accepts a line up to two away from the planted one. A hit earns 10 points, a false positive costs 5, and a duplicate of a flaw already found scores nothing; the planted flaws nobody found are the misses.
Why does a long list of guesses do worse than a short list of facts?

A false positive costs 5 points and a hit earns only 10. In the warm-up payments file a review found 3 of 5 planted flaws and scored 20: its two false positives took away 10 points, and it missed the password written to the log and the token decoded and never verified.
What is a canary test, and what must it do on the vulnerable code and on the fixed code?

A regression test for one flaw that sends a harmless input to the handler. It must fail on the vulnerable code, which shows that it notices the flaw, and pass on the fixed code and on the case that was always valid, so a fix that refuses everything cannot satisfy it. It stays in the suite and runs on every change.
Why must you see a new test fail on the original code first?

A test that passes on the vulnerable code tests nothing, and a test you never saw fail may not be looking at the flaw in the first place. Run the test on the original first, then make the fix and see it pass.
What do the clinic's two apps hold?

Twenty planted flaws, ten in a Node service and ten in a Python one, with at least one in every Top 10:2025 category from A01 to A10.

Key takeaways

The clinic gives 165 minutes to six stages, most of them to tracing inputs and fixing, and says what a stage did not reach.
A matcher scores a review: a hit earns 10 points, a false positive costs 5 and a duplicate scores nothing.
A fix is finished when its canary fails on the original code and passes on the fix, the valid case included.

Terms in this lesson

canary test: A regression test for one flaw that sends a harmless canary input to the handler. It fails on the vulnerable code, passes on the fix and on the case that was always valid, and stays in the suite.
finding matcher: The rule that scores a review against the planted flaws. A finding is a hit when it names the same family of weakness within a few lines of a planted flaw; otherwise it is a duplicate or a false positive.
mutant: A copy of a fix with one small change, such as a condition removed or turned round. A mutant that still passes every test is a survivor, and it names a test that is missing.
planted flaw: A weakness put into a training app on purpose, at a known line and with a known CWE, so that a review can be scored on what it finds, misses and invents.

References

Secure Code Review Cheat Sheet (OWASP Foundation)
ASVS 5.0.0, requirements cited in the clinic (chapters V1, V3, V5, V6, V8, V11, V13, V15 and V16) (OWASP Foundation)
A01:2025 Broken Access Control (OWASP Foundation)
A02:2025 Security Misconfiguration (OWASP Foundation)
A03:2025 Software Supply Chain Failures (OWASP Foundation)
A04:2025 Cryptographic Failures (OWASP Foundation)
A05:2025 Injection (OWASP Foundation)
A06:2025 Insecure Design (OWASP Foundation)
A07:2025 Authentication Failures (OWASP Foundation)
A08:2025 Software or Data Integrity Failures (OWASP Foundation)
A09:2025 Security Logging and Alerting Failures (OWASP Foundation)
A10:2025 Mishandling of Exceptional Conditions (OWASP Foundation)
OWASP Risk Rating Methodology (OWASP Foundation)
Password Storage Cheat Sheet (OWASP Foundation)

Report a problem with this lesson

Your country

Review clinic: find and fix planted flaws

What you will learn

Before you start

What you will learn

Before you start

Run the clinic in six stages, and spend the hours where flaws hide

Score a review by what it found, what it missed and what it invented

A fix is finished when its test fails on the original code

ShopNest's ten flaws: where each sits in the standards, how it is fixed, and the canary that guards it Pro: opens with a Pro, Premium or Ultimate pass

NoteBox's ten flaws in Python, four that no response shows, and a canary that also runs in a browser Pro: opens with a Pro, Premium or Ultimate pass

Rank twenty findings, prove each fix against spoiled copies, write the report, and turn each miss into a check Pro: opens with a Pro, Premium or Ultimate pass

Practise: score a review, then close a flaw

Score a review against the planted flaws

Results of the sample tests

Fix the receipt id and prove the fix with canaries

Results of the sample tests

Interview questions

A scan of your team's two test apps raises 15 alerts and misses most of the 20 flaws that a review knows about. A manager asks why you still review code and write tests. What do you answer, and what do you change in the pipeline?

A teammate fixes an open redirect and adds one test: an address on another site is sent to the home page. The test passes. Is the fix done, and what do you ask for?

Your review has twenty findings, the release is on Friday, and you have two hours for fixes. How do you decide what to fix, what to defer and what to tell the release owner?

Key takeaways

References

Related tools