Web Security (OWASP Top 10:2025) Module 15 – Security testing and OWASP ZAP in CI
Security testing toolbox: SAST, DAST, SCA, secrets
See what SAST, DAST, IAST, SCA, secret scanning and fuzzing find and miss, build a small-team pipeline, and prove every fix with a test that failed first.
What you will learn
- Explain what SAST, DAST, IAST, SCA, secret scanning and fuzzing find and what they miss
- Choose a minimal pipeline for a small team and set gates that do not drown developers
- Write security regression tests for every fixed vulnerability
- Use fuzzing and property-based tests on parsers and validators
- Classify findings as true or false positives, fix them and prove each fix with a test
Before you start
What you’ll learn · Before you start
What you will learn
- Explain what SAST, DAST, IAST, SCA, secret scanning and fuzzing find and what they miss
- Choose a minimal pipeline for a small team and set gates that do not drown developers
- Write security regression tests for every fixed vulnerability
- Use fuzzing and property-based tests on parsers and validators
- Classify findings as true or false positives, fix them and prove each fix with a test
Before you start
On this page
A small team fixes a path traversal bug on Tuesday, and on Friday a refactor brings it back, because the fix came with no test and nothing else knew to look. In this lesson’s first program, a rule set that searches the text of code finds 13 of 14 real problems but buries them among 8 wrong findings, while a rule set that reads the syntax tree makes 2 mistakes and misses 2 problems.
You will say what each kind of security test finds and misses, choose a small pipeline with gates that people accept, write a regression test for every fix that fails before the fix, find bugs in a parser with random inputs, and turn an alert into a fix.
Each kind of security test answers one question, and none reads intent
SAST: Static application security testing: tools that read source code, without running it, for unsafe calls and patterns. They work early and point at a line, and they cannot see deployed settings or what the code is meant to do. reads source code for risky patterns, DAST: Dynamic application security testing: tools that send requests to a running copy of an application and read the responses. They see what the server really sends, and only the pages and parameters that their crawl reaches. probes a running application from outside, IAST: Interactive application security testing: a sensor inside a running application watches data reach dangerous calls while tests or people use it. It sees only the code that actually runs during that time. watches a running application from inside while tests exercise it, and SCA: Software composition analysis: tools that match the packages a project uses against lists of known flaws. They report vulnerable versions, and they usually cannot tell whether your code calls the flawed part. matches the packages you install against lists of known flaws. secret scanning: Reading code, history, logs or images for values that look like secrets, by provider formats, names, entropy or all three. A scanner trades missed leaks against false alarms, and an allowlist records the decoys. More in “Secrets and keys: storage, rotation, scanning” looks for keys and tokens in the code and its history. fuzzing: Testing by feeding a program unexpected, malformed or random input and watching for crashes and broken rules. It is aimed at parsers and validators, and it is run only against code you own. feeds a program unexpected or malformed input and watches for trouble. The OWASP DevSecOps Guideline describes the first four as steps of vulnerability scanning, and OWASP’s A05:2025 page says that source code review together with automated testing, fuzzing included, is the best way to find injection flaws.
| Kind of test | What it reads | What it finds | What it misses |
|---|---|---|---|
| SAST | Source code | Unsafe calls and patterns, some data flows | Deployed settings, and the intent of the code |
| SCA | The lockfile and a list of advisories | Package versions with a known flaw | Whether your code reaches the flaw |
| Secret scanning | Files and git history | Keys and tokens, by rule or by randomness | Secrets that look ordinary |
| DAST | A running copy, from outside | Headers, cookies, reflected input, some injection | Code the crawl never reaches, and logic flaws |
| IAST | A running copy, from inside | Data reaching a sink while tests run | What the tests do not run |
| Fuzzing and property tests | A function fed generated inputs | Crashes and broken invariants in parsers and validators | Anything the invariants do not say |
These are kinds of test, not products. Bandit for Python and CodeQL, which powers GitHub’s code scanning, are SAST tools. OSV-Scanner is an SCA tool, Gitleaks is a secret scanner, and Hypothesis and fast-check are property-testing libraries. Run them only on code that you own.
What none of them reads is intent. They do not know who may open an invoice, whether a coupon can be applied twice, or whether the design lets one customer act as another. Those weaknesses need a threat model, review, and tests you write, which is why A06:2025 asks for unit and integration tests of critical flows against the threat model. The Web Security Testing Guide, version 4.2, lists what a tester covers in twelve groups, from information gathering through authorisation and session management to business logic, client-side and API testing; no single tool covers all twelve. Answer three questions about a project of your own, and the table shows which kinds of test fit:
Answer the questions above: the table then marks what fits your case and says why the rest does not.
| Kind of test | Flaws in your own code | Vulnerable packages | Leaked secrets | Served headers and cookies | Access control and logic |
|---|---|---|---|---|---|
| SAST Reads: your source code | Good unsafe calls in source: 12 of 14 real problems by tree rules, 13 of 14 by text search | Poor does not read your lockfile | Mixed a few rules; random-looking strings make it noisy | Poor cannot see what the deployed server sends | Poor does not know who may do what |
| SCA Reads: your lockfile and an advisory list | Poor reads versions and not your code | Good matches versions to advisories, not whether you call the flaw | Poor does not read file contents | Poor does not see the running site | Poor does not know who may do what |
| Secret scanning Reads: your code and its history | Poor looks for keys and not for flaws | Poor does not read your lockfile | Good rules and entropy over the code and its history | Poor does not see the running site | Poor does not know who may do what |
| DAST Reads: a running copy, from outside | Mixed reflected XSS and SQL injection on pages it reaches | Poor sees pages and not packages | Poor a key in a response is luck | Good headers and cookies exactly as the server sends them | Poor rarely sees that user A can read user B's data |
| IAST Reads: a running copy, from inside | Good sees data reach a sink while tests run; needs an agent | Mixed some agents list the libraries in use | Mixed can see plain-text keys at run time | Mixed sees the settings the agent reads | Poor does not know who may do what |
| Fuzzing and property tests Reads: a function fed random inputs | Mixed crashes and broken rules in parsers and validators | Poor does not read your lockfile | Poor not what it is for | Poor does not see the running site | Poor does not know who may do what |
How the answers decide
- “Is there a running copy of the application that you may test?” No, only the code: rules out DAST and IAST. DAST and IAST need the application running. Without a test copy of it, start with the tools that read source.
- “Does an automated test suite exercise most of the application?” No, hardly: rules out IAST. An IAST agent sees only what runs while it watches. Without tests that exercise the application, there is little for it to see.
- “What worries you most?” Packages I install: adds a caveat to SAST, DAST, IAST, Fuzzing and property tests and Secret scanning. None of these reads your lockfile. SCA is the tool that matches package versions to advisories.
- “What worries you most?” Keys in the repository: adds a caveat to SAST, SCA, DAST, IAST and Fuzzing and property tests. Secret scanning reads the repository and its history for keys. The others are not built for it.
- “What worries you most?” What the live site sends: rules out SCA and Secret scanning and adds a caveat to SAST. What the live site sends is visible from outside, which is where DAST reads it. Source tools cannot see the deployed settings.
- “What worries you most?” My own code: adds a caveat to SCA, Secret scanning and DAST. SAST reads your code, and fuzzing probes the parsers and validators in it. The others look at packages, keys or the running site.
Checkpoint
Check: What each kind of test finds
Static rules trade precision for recall, so measure them on labelled code
A static rule is a hypothesis about code, and it can be wrong in two ways. It can report code that is fine, which is a
false positive: An alert raised for something that is not an attack. Too many of them bury the real alerts and teach people to ignore the pager, so each rule is scored against labelled cases and tuned. More in “Alerting and first-hour incident response”, and it can miss code that is not. precision: The share of a rule's findings that are right: 12 right findings out of 14 is a precision of about 86 %. Low precision means noise, which teaches people to ignore the tool. is the share
of its findings that are right, and recall: The share of the real problems that a rule finds: 12 found out of 14 real ones is a recall of about 86 %. Low recall means gaps that the tool's quiet report hides. is the share of the real problems that it finds. You cannot know
either from the tool’s documentation; you measure them on code where you know the answer. The program holds six rules for dangerous
Python, such as shell=True, yaml.load without a safe loader, eval, pickle, verify=False and a fast hash of a password. It
writes each rule twice, as a search of the text and as a walk over the syntax tree, as Bandit does, and scores both on 28 labelled
snippets: 14 with a real problem and 14 that look like one or are safe.
"""Six rules for dangerous Python, written twice and scored on the same labelled code.
The first version searches the text of each snippet, as a grep would. The second reads the syntax tree with Python's ast
module, so a comment, a string or a different spelling no longer fools it. The corpus has 28 small snippets: 14 that hold one
of the six problems and 14 that look like them or are safe. A rule is scored by what it found, how many of those were right,
and how many real problems it missed.
"""
import ast
import re
RULES = ["shell-true", "yaml-load", "eval-exec", "pickle-loads", "verify-false", "md5-password"]
# (id, the problem it holds or None, the code)
CORPUS = [
("ping built into a string", "shell-true", 'subprocess.run(f"ping -c 1 {host}", shell=True)'),
("ls built into a string", "shell-true", 'out = subprocess.check_output("ls " + folder, shell=True)'),
("yaml.load, no loader", "yaml-load", "cfg = yaml.load(open(path))"),
("yaml.load, full loader", "yaml-load", "data = yaml.load(text, Loader=yaml.Loader)"),
("eval of a variable", "eval-exec", "result = eval(user_expression)"),
("exec of a variable", "eval-exec", "exec(code_from_form)"),
("pickle.loads of a blob", "pickle-loads", "obj = pickle.loads(blob)"),
("pickle.load of a file", "pickle-loads", 'state = pickle.load(open(session_file, "rb"))'),
("verify=False on get", "verify-false", "requests.get(url, verify=False)"),
("verify=False on post", "verify-false", "session.post(url, data=body, verify=False, timeout=5)"),
("md5 of a password", "md5-password", "digest = hashlib.md5(password.encode()).hexdigest()"),
("sha1 of a password", "md5-password", "h = hashlib.sha1(user_pwd.encode())"),
("shell=True after an import", "shell-true", "from subprocess import run\nrun(cmd, shell=True)"),
("eval with a built-up name", "eval-exec", 'getattr(builtins, "ev" + "al")(expr)'),
("ping as a list", None, 'subprocess.run(["ping", "-c", "1", host], shell=False)'),
("ls as a list", None, 'subprocess.run(["ls", folder])'),
("yaml.safe_load", None, "cfg = yaml.safe_load(open(path))"),
("yaml.load, safe loader", None, "data = yaml.load(text, Loader=yaml.SafeLoader)"),
("ast.literal_eval", None, "value = ast.literal_eval(text)"),
("eval in a comment", None, "# result = eval(user_expression) (removed)"),
("eval in a string", None, 'message = "never call eval(x) or pickle.loads(y) on input"'),
("json.loads", None, "payload = json.loads(blob)"),
("verify=True", None, "requests.get(url, verify=True)"),
("verify with a CA file", None, 'requests.get(url, verify="/etc/ssl/internal-ca.pem")'),
("md5 of a file", None, "etag = hashlib.md5(file_bytes).hexdigest()"),
("pickle of our own data", None, "blob = pickle.dumps(entry)\nrestored = pickle.loads(blob)"),
("eval of a constant", None, 'answer = eval("2 + 2")'),
("sha256 of a file", None, "digest = hashlib.sha256(file_bytes).hexdigest()"),
]
GREP = {
"shell-true": r"shell=True",
"yaml-load": r"yaml\.load\(",
"eval-exec": r"eval\(|exec\(",
"pickle-loads": r"pickle\.loads?\(",
"verify-false": r"verify=False",
"md5-password": r"md5\(|sha1\(",
}
def grep_rules(code):
return {rule for rule, pattern in GREP.items() if re.search(pattern, code)}
SUBPROCESS = {f"subprocess.{f}" for f in ("run", "call", "Popen", "check_output", "check_call")}
def dotted(node):
parts = []
while isinstance(node, ast.Attribute):
parts.append(node.attr)
node = node.value
if isinstance(node, ast.Name):
return ".".join([node.id, *reversed(parts)])
return None
def keyword(call, name):
return next((k.value for k in call.keywords if k.arg == name), None)
def constant(node, value):
return isinstance(node, ast.Constant) and node.value is value
def tree_rules(code):
found = set()
for node in ast.walk(ast.parse(code)):
if not isinstance(node, ast.Call):
continue
target = dotted(node.func)
if target in SUBPROCESS and constant(keyword(node, "shell"), True):
found.add("shell-true")
if target == "yaml.load" and dotted(keyword(node, "Loader") or ast.Constant(None)) not in ("yaml.SafeLoader", "SafeLoader"):
found.add("yaml-load")
if target in ("eval", "exec"):
found.add("eval-exec")
if target in ("pickle.load", "pickle.loads"):
found.add("pickle-loads")
if constant(keyword(node, "verify"), False):
found.add("verify-false")
if target in ("hashlib.md5", "hashlib.sha1"):
names = [n.id for a in node.args for n in ast.walk(a) if isinstance(n, ast.Name)]
if any(re.search(r"pass|pwd", name) for name in names):
found.add("md5-password")
return found
def score(engine):
table = {rule: {"found": 0, "right": 0, "missed": 0} for rule in RULES}
for _, truth, code in CORPUS:
found = engine(code)
for rule in RULES:
if rule in found:
table[rule]["found"] += 1
table[rule]["right"] += rule == truth
elif rule == truth:
table[rule]["missed"] += 1
return table
def percent(part, whole):
return f"{100 * part / whole:.0f} %" if whole else "-"
grep, tree = score(grep_rules), score(tree_rules)
print(f"{'rule':<14}{'search the text':<28}read the syntax tree")
print(f"{'':<14}{'found right missed':<28}found right missed")
for rule in RULES:
g, t = grep[rule], tree[rule]
print(f"{rule:<14}{g['found']:>5}{g['right']:>6}{g['missed']:>7}{'':<10}{t['found']:>5}{t['right']:>6}{t['missed']:>7}")
print()
for name, table in (("Searching the text", grep), ("Reading the syntax tree", tree)):
found = sum(r["found"] for r in table.values())
right = sum(r["right"] for r in table.values())
real = sum(1 for _, truth, _ in CORPUS if truth)
print(f"{name}: {found} findings, {right} right ({percent(right, found)} of findings), {right} of {real} real problems found ({percent(right, real)})")
print()
for label, truth, code in CORPUS:
g, t = grep_rules(code), tree_rules(code)
if (g == {truth} or (not g and truth is None)) != (t == {truth} or (not t and truth is None)):
print(f"differ: {label:<28} truth {truth or '-':<13} text {sorted(g) or '-'} tree {sorted(t) or '-'}")Output
rule search the text read the syntax tree
found right missed found right missed
shell-true 3 3 0 2 2 1
yaml-load 3 2 0 2 2 0
eval-exec 6 2 1 3 2 1
pickle-loads 4 2 0 3 2 0
verify-false 2 2 0 2 2 0
md5-password 3 2 0 2 2 0
Searching the text: 21 findings, 13 right (62 % of findings), 13 of 14 real problems found (93 %)
Reading the syntax tree: 14 findings, 12 right (86 % of findings), 12 of 14 real problems found (86 %)
differ: shell=True after an import truth shell-true text ['shell-true'] tree -
differ: yaml.load, safe loader truth - text ['yaml-load'] tree -
differ: ast.literal_eval truth - text ['eval-exec'] tree -
differ: eval in a comment truth - text ['eval-exec'] tree -
differ: eval in a string truth - text ['eval-exec', 'pickle-loads'] tree -
differ: md5 of a file truth - text ['md5-password'] tree -
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 ast_rules.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Searching the text gives 21 findings, of which 13 are right, a precision of 62 %, and it finds 13 of the 14 real problems. It flags
a comment, a string, ast.literal_eval, yaml.load with a safe loader and the MD5 of a file. Reading the syntax tree gives 14
findings, of which 12 are right, a precision of 86 %, and it finds 12 of the 14. It misses run(cmd, shell=True) after
from subprocess import run, which the text search found, because its rule looks only for subprocess.run. Both miss a call to eval
that is spelled out of pieces, and both wrongly flag a pickle.loads of the program’s own data and an eval of a constant. No
rule knows where its input came from, and that gap is the reason a person still triages every finding.
Key idea
Judge a static rule by its precision and its recall on code where you know the answer, because a rule with noise is ignored and a rule with gaps gives false comfort.
Checkpoint
Check: Precision and recall
A small team needs few gates, placed where they cost least
A pipeline of security tests is a trade of time for confidence, and the cheapest place to find a problem is the earliest. A secret scan runs on the change before it is committed and again on every push, because a key that reaches the history stays there. SAST and SCA run on a pull request, where the diff is small and the author is still looking. Scans of images and infrastructure files run when those files change. A passive baseline scan of a test copy runs at night, and slower scans run before a release. Walk through the stages:
- Pre-commit
A secret scan of the change seconds
A secret scanner reads the staged change before it is committed, and the same scan runs on every push. A key that reaches the history stays there, so the cheapest place to catch it is the first one.
- Pull request
Static rules, packages and tests minutes
SAST runs on the changed files and SCA on the lockfile, next to the unit tests and the security regression tests. Gate on new high-severity findings only, and keep a baseline for the old ones.
- Merge to main
Build, image and file scans minutes
Scans of container images and infrastructure files run when those files change, and the build that passed is the one that is promoted.
- Nightly
A baseline scan of staging tens of minutes
A passive DAST baseline of a test copy runs every night. It follows links and reads responses, and it attacks nothing, so it suits a schedule.
- Weekly or release
Deeper scans and fuzz runs hours
A fuller scan, an authenticated API scan and longer fuzzing of parsers run before a release, against disposable data only and never against production without written agreement.
The timeline has 5 events, from Pre-commit to Weekly or release.
Four habits keep the gates from drowning people. Gate on new findings of high severity, and keep a baseline of the old ones so that a legacy of 300 alerts does not block a change that adds none. Fail fast and in plain words: a finding that names a file, a rule and a fix is read, and one that prints a screen of JSON is not. Review suppressions, each with a reason and an expiry. And measure the tools with the numbers of the last section: the share of findings that were real, and the time from alert to fix. A09:2025 adds a point that is easy to forget. Scans that raise no alert in your monitoring are a failure of the monitoring, and a scan whose findings nobody reads is a failure of its own.
Scan only what you own or may test
Run active scanners against systems you own, or against ones whose owner has allowed that test in writing, and use made-up data. The practice rules of the first lesson of the track apply to every tool in this lesson.
Checkpoint
Check: Gates for a small team
Every fixed vulnerability gets a test that failed first
A fix without a test is a loan. The refactor of the opening story is the ordinary way a fixed bug returns, and the cure is a
regression test: A test kept after a fix so that the weakness cannot come back unnoticed. In a security lab it fails on the weak version and passes on the fixed one, and it runs again on every later change. More in “Start here: learn web security safely and legally” written for the fix: it fails on the old code and passes on the new. Keep the
inputs that attack the weakness in one shared fixture of canary input: A harmless test value holding characters that matter to an interpreter, such as an apostrophe for SQL or quotes and angle brackets for HTML. A test fails if the value changes the query's or the page's structure. More in “SQL injection and parameterised queries”, harmless strings such as
../secret/keys.txt and a script marker, and run them through every place that handles them. The program tests two targets with one
fixture, a function that finds a file under a base folder and a function that shows a comment, each in four versions: the weak one,
a half fix, a fix that is too strict, and the correct one.
// One shared fixture of canary inputs, run through two targets (a function that finds a file under a base folder, and a function
// that shows a comment on a page), each in several versions. The test for a fixed vulnerability must fail on the weak version
// and on a half fix, and pass on the fix and on nothing that breaks ordinary inputs. The canaries are plain strings: no real
// payloads, and the "files" are entries of a set.
const CANARIES = {
traversal: ['../secret/keys.txt', 'docs/../../secret/keys.txt', 'a/../../secret/keys.txt'],
html: ['<script>markHit(1)</script>', '"><img src=x onerror=markHit(2)>', 'a & b > c'],
fine: ['logo.png', 'docs/guide.txt', 'docs/v1.2.txt'],
};
const FILES = new Set(['public/logo.png', 'public/docs/guide.txt', 'public/docs/v1.2.txt', 'secret/keys.txt']);
// The path target. normalize() collapses "." and ".." the way a file system does.
function normalize(path) {
const out = [];
for (const part of path.split('/')) {
if (part === '' || part === '.') continue;
if (part === '..') out.pop();
else out.push(part);
}
return out.join('/');
}
const refuse = () => { throw new Error('refused'); };
const PATH_VERSIONS = {
weak: (name) => normalize(`public/${name}`),
'half fix': (name) => (name.startsWith('../') ? refuse() : normalize(`public/${name}`)),
'too strict': (name) => (name.includes('.') ? refuse() : normalize(`public/${name}`)),
fixed: (name) => {
const path = normalize(`public/${name}`);
return path === 'public' || path.startsWith('public/') ? path : refuse();
},
};
// The page target.
const ESCAPES = { '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' };
const HTML_VERSIONS = {
weak: (text) => text,
'half fix': (text) => text.replace(/</g, '<'),
'too strict': () => '[comment hidden]',
fixed: (text) => text.replace(/[&<>"']/g, (c) => ESCAPES[c]),
};
// The two tests share the fixture. Each returns what went wrong, as a list.
function pathTest(serve) {
const problems = [];
for (const name of CANARIES.traversal) {
let got;
try { got = serve(name); } catch { got = 'refused'; }
if (got !== 'refused' && !got.startsWith('public')) problems.push(`${name} reached ${got}`);
}
for (const name of CANARIES.fine) {
let got;
try { got = serve(name); } catch { got = 'refused'; }
if (!FILES.has(got)) problems.push(`${name} was not served`);
}
return problems;
}
function htmlTest(show) {
const problems = [];
for (const text of CANARIES.html) if (/[<>"]/.test(show(text))) problems.push(`${text} kept a raw bracket or quote`);
if (show('Great shop') !== 'Great shop') problems.push('plain text was changed');
return problems;
}
console.log('target version problems found by the test');
for (const [target, versions, test] of [['path', PATH_VERSIONS, pathTest], ['page', HTML_VERSIONS, htmlTest]]) {
for (const [name, version] of Object.entries(versions)) {
const problems = test(version);
console.log(`${target.padEnd(8)}${name.padEnd(12)}${problems.length ? `${problems.length}: ${problems[0]}` : 'none, the test passes'}`);
}
}Output
target version problems found by the test path weak 3: ../secret/keys.txt reached secret/keys.txt path half fix 2: docs/../../secret/keys.txt reached secret/keys.txt path too strict 3: logo.png was not served path fixed none, the test passes page weak 3: <script>markHit(1)</script> kept a raw bracket or quote page half fix 3: <script>markHit(1)</script> kept a raw bracket or quote page too strict 1: plain text was changed page fixed none, the test passes
Recorded with Node.js 24.21.0 on macOS 26 arm64. To run it yourself: mise exec node@24.21.0 -- node regression_test_pattern.mjs
Runs on this device, in your browser. The first run downloads JavaScript (about 0.6 MB), which is kept for the next runs.
Your run, in this browser
The path test finds 3 problems in the weak version, 2 in the half fix that refuses only names starting with ../, 3 in the version
that refuses every name with a dot, and none in the fix. The page test finds 3 in the weak version, 3 in the half fix that
escapes only <, 1 in the version that hides every comment, and none in the fix. The half fixes matter most. A test that cannot
tell a half fix from the fix is not strong enough, and a test that passes a version that breaks ordinary names is not checking
function. So a regression test has three parts: attacking inputs that fail on every weak version, ordinary inputs that fail on
every over-strict one, and the same fixture reused for every target.
Checkpoint
Check: Regression tests that fail first
Fuzzing and property tests find what chosen examples miss
An example-based test checks the inputs its author thought of. A property-based test: A test that states a rule which must hold for every input, generates many inputs and reports one that breaks it. A good tool then shrinks the input to the smallest one that still fails. names a rule that all inputs must obey, generates many inputs, and reports one that breaks the rule. Fuzzing is the same idea at the edge of a program, feeding malformed input to parsers, headers, cookies, JSON and XML in a safe environment and watching for crashes. Libraries such as Hypothesis for Python and fast-check for JavaScript do this with strategies and shrinking; the program builds the parts by hand so that you can see them: a seeded generator, a rule, and a shrinker that cuts a failing input to the smallest one that still fails. It tests an HTML encoder against two rules, that its output never holds a raw angle bracket or quote and that it decodes back to the input, and a path function against the rule that its result never leaves the base.
"""A property-based test written by hand: random inputs from a seeded generator, an invariant that must hold for all of
them, and a shrinker that cuts a failing input down to the smallest one that still fails.
Two functions are tested, each in a weak and a fixed version: an HTML encoder, whose output must never hold a raw angle
bracket or quote and must decode back to the input, and a function that resolves a name under a base folder, whose result
must never leave it. The generator is a small xorshift, so the same seed gives the same cases in every Python.
"""
SEED = 7
CASES = 500
PIECES = ["a", "b", "z", "0", "-", "_", " ", ".", "..", "/", "../", "%2e", "<", ">", "&", '"', "'", "<"]
class Rng:
def __init__(self, seed):
self.state = seed or 1
def below(self, n):
x = self.state
x ^= (x << 13) & 0xFFFFFFFF
x ^= x >> 17
x ^= (x << 5) & 0xFFFFFFFF
self.state = x
return x % n
def cases(count):
rng = Rng(SEED)
for _ in range(count):
yield [PIECES[rng.below(len(PIECES))] for _ in range(1 + rng.below(8))]
# ---- the HTML encoder ------------------------------------------------------------------------------------------------
def encode_forgets_gt(text):
return text.replace("&", "&").replace("<", "<").replace('"', """)
def encode_ampersand_last(text):
return text.replace("<", "<").replace(">", ">").replace('"', """).replace("&", "&")
def encode_fixed(text):
return text.replace("&", "&").replace("<", "<").replace(">", ">").replace('"', """)
def decode(text):
return text.replace(""", '"').replace(">", ">").replace("<", "<").replace("&", "&")
def no_raw_brackets(encode, text):
out = encode(text)
return not any(c in out for c in '<>"')
def round_trip(encode, text):
return decode(encode(text)) == text
# ---- resolving a name under a base folder -----------------------------------------------------------------------------
def normalize(path):
out = []
for part in path.split("/"):
if part in ("", "."):
continue
if part == "..":
if out:
out.pop()
else:
out.append(part)
return "/".join(out)
def resolve_weak(name):
return normalize("base/" + name)
def resolve_half_fixed(name):
if name.split("/")[0] == "..":
raise ValueError("refused")
return normalize("base/" + name)
def resolve_fixed(name):
path = normalize("base/" + name)
if path != "base" and not path.startswith("base/"):
raise ValueError("refused")
return path
def stays_inside(resolve, name):
try:
path = resolve(name)
except ValueError:
return True
return path == "base" or path.startswith("base/")
# ---- the runner and the shrinker --------------------------------------------------------------------------------------
def shrink(pieces, holds):
"""Removes one piece at a time while the property still fails; what is left is a smallest failing input."""
shrunk = True
while shrunk:
shrunk = False
for i in range(len(pieces)):
smaller = pieces[:i] + pieces[i + 1:]
if smaller and not holds("".join(smaller)):
pieces, shrunk = smaller, True
break
return pieces
def run(label, version, holds):
for count, pieces in enumerate(cases(CASES), start=1):
if not holds("".join(pieces)):
small = shrink(pieces, holds)
print(f"{label:<26}{version:<18}failed on case {count}: {''.join(pieces)!r}, shrunk to {''.join(small)!r}")
return
print(f"{label:<26}{version:<18}held for {CASES} cases")
print(f"seed {SEED}, {CASES} cases for each property")
print(f"{'property':<26}{'version':<18}result")
for name, encode in (("forgets >", encode_forgets_gt), ("ampersand last", encode_ampersand_last), ("fixed", encode_fixed)):
run("html: no raw < > or \"", name, lambda t, e=encode: no_raw_brackets(e, t))
run("html: decodes back", name, lambda t, e=encode: round_trip(e, t))
for name, resolve in (("weak", resolve_weak), ("half fix", resolve_half_fixed), ("fixed", resolve_fixed)):
run("path: stays inside base", name, lambda t, r=resolve: stays_inside(r, t))Output
seed 7, 500 cases for each property property version result html: no raw < > or " forgets > failed on case 1: '.>0> ../.%2e', shrunk to '>' html: decodes back forgets > held for 500 cases html: no raw < > or " ampersand last held for 500 cases html: decodes back ampersand last failed on case 1: '.>0> ../.%2e', shrunk to '>' html: no raw < > or " fixed held for 500 cases html: decodes back fixed held for 500 cases path: stays inside base weak failed on case 5: '..', shrunk to '..' path: stays inside base half fix failed on case 94: '/../', shrunk to '/..' path: stays inside base fixed held for 500 cases
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 property_fuzz.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
An encoder that forgets > fails the first rule on the very first case, and the shrinker cuts the failing input to >. An encoder
that escapes & last passes that rule and fails the second, because > becomes &gt; and decodes back to >, again shrunk
to >. A path function that refuses names starting with .. holds for 93 cases and fails on case 94, shrunk to /... The correct
versions hold for all 500. The seed is printed, so a failing run is repeated exactly, and the counterexample becomes the canary of
the next regression test. A property can only say what you state, so an invariant that is too weak passes anything.
Checkpoint
Check: Properties, fuzzing and shrinking
Triage turns an alert into a fix and a test
A finding is a claim from a tool, and triage is how you turn it into a decision. Step through the life of one finding:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- Alert. A tool reports a rule, a place and a severity. Do not act on the severity alone. The tool does not know your data, or who can reach the code.
- Reproduce. Make it happen in a copy you own, with a request, a test or a small program. A finding that you cannot reproduce is not yet a finding.
- Classify. Decide whether it is a true positive, a false positive or an accepted risk. Name the weakness with its CWE and the Top 10 category, so that the fix pattern is known.
- Fix. Change the code at its cause. If two alerts have one cause, such as a query string echoed into a page, fix it once.
- Prove. Add a regression test that fails on the old code and passes on the new, then run the scan again and watch the alert go.
- Suppress. Only a finding that is still false after all this gets a suppression, with the reason, an owner and an expiry. A silent suppression is how a scanner goes quiet while the code stays unsafe.
Reproduce it first, in a copy you own, because a finding you cannot reproduce is not yet a finding. Then classify it as a true positive, a false positive or an accepted risk, and name the weakness with its CWE, so that the fix pattern is known. Fix it at the cause: two alerts with one cause, such as a query string echoed into a page, are one fix. Then prove it. The regression test that failed first is the proof, and the scan run again is the second. Only a finding that is still false after all of that is suppressed, and a suppression carries its reason, its owner and a date or a release at which it is looked at again.
Closing the alert and not the cause
Wrong: mark the alert as fixed because the scanner went quiet after a change of the page. Right: change the code at the cause, add the regression test, and let the next scan confirm it. A scanner can be quiet because it no longer reaches the page.
Checkpoint
Check: Triage
Practise: write the check that catches a half fix
Write the regression check for a path function yourself. It has to reject the weak version, two half fixes and a version that crashes, and accept two correct ones that work differently. The sample tests give you all seven:
Exercise · Medium · Python
Write the regression check for a fixed path function
A function resolve(name) serves files from the folder base. It takes a relative file name and returns the path inside the folder, after . and .. have been collapsed, as text such as base/docs/guide.txt. A name that would end up outside base must raise ValueError. A name that stays inside is served: docs/../logo.png returns base/logo.png, the empty name returns base, and a name with dots such as docs/v1.2.txt is an ordinary name.
Write check(resolve) in check.py. It is the regression test for the fix: it runs resolve on inputs of your choice and returns True when resolve behaves as described for all of them, and False as soon as it does not. A resolve that raises anything other than ValueError is wrong too.
The sample tests give check seven versions of resolve: the original weak one, a fix that stops at names starting with .., a fix that tests the prefix base instead of base/, a version that refuses every name with a dot, one that crashes on the empty name, and two correct ones that work differently. A good check returns False for the first five and True for the last two. It needs inputs that make the half fixes fail and ordinary inputs that make the over-strict one fail, and it needs to know the right answer for each, so write a small oracle of your own: do not call any of the versions to get it.
Starter code · check.py
def check(resolve):
"""True when resolve(name) behaves as the prompt describes for every input this check tries."""
return TrueThe sample tests · test_check.py
from check import check
def normalize(path):
out = []
for part in path.split("/"):
if part in ("", "."):
continue
if part == "..":
if out:
out.pop()
else:
out.append(part)
return "/".join(out)
def weak(name):
return normalize("base/" + name)
def half_fixed(name):
if name.split("/")[0] == "..":
raise ValueError("refused")
return normalize("base/" + name)
def prefix_bug(name):
path = normalize("base/" + name)
if not path.startswith("base"):
raise ValueError("refused")
return path
def too_strict(name):
if "." in name:
raise ValueError("refused")
return normalize("base/" + name)
def crashes_on_empty(name):
path = normalize("base/" + name)
if path.split("/")[1] != "":
pass
return path
def fixed(name):
path = normalize("base/" + name)
if path != "base" and not path.startswith("base/"):
raise ValueError("refused")
return path
def fixed_by_segments(name):
stack = ["base"]
for part in name.split("/"):
if part in ("", "."):
continue
if part == "..":
stack.pop()
if not stack:
raise ValueError("refused")
else:
stack.append(part)
return "/".join(stack)
def test_the_check_rejects_the_original_and_the_fix_that_only_looks_at_the_start():
assert check(weak) is False
assert check(half_fixed) is False
def test_the_check_rejects_a_fix_that_tests_the_wrong_prefix():
assert check(prefix_bug) is False
def test_the_check_rejects_a_fix_that_refuses_ordinary_names():
assert check(too_strict) is False
def test_the_check_rejects_a_version_that_crashes_and_does_not_crash_itself():
assert check(crashes_on_empty) is False
def test_the_check_accepts_two_correct_versions_that_work_differently():
assert check(fixed) is True
assert check(fixed_by_segments) is TrueA hint
Write your own normalize, then an oracle expected(name) that returns the path or None. Run one list of names that try to leave the folder, including .., a name that climbs back out after going in, and a sibling folder whose name starts with base, and one list of ordinary names. For each name, compare what resolve returned (None for a ValueError) with the oracle, and return False for a difference or any other exception.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Then work through the questions on the six ideas of the lesson. They begin with the ones you have answered least well:
Practice that adapts to you
Practise this lesson
17 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up: what is the difference between SAST and DAST, and why use both?
SAST reads the source code, or a model of it, without running the program, so it finds unsafe calls and patterns early, on a pull request, and it can point at the line. It does not see the deployed configuration, and it reports code that looks risky whatever its input. DAST sends requests to a running copy and reads the responses, so it sees what the server really sends, such as missing headers and reflected input, and it reports only what a request can reach. It knows nothing of the code behind. Used together they cover different questions, and neither reads intent, so access control and logic flaws still need review and tests that you write.
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
A team of five has no security testing and wants a pipeline this month. What do you add first, where does each test run, and what gates do you set so that developers do not learn to ignore them?
-
Mid to senior
A static analysis tool reports 300 findings on its first run over a ten-year-old codebase. What do you do in the first week, and how do you decide which findings to fix and which to suppress?
Flashcards · 8 cards
Recall it, then check
Say the answer to yourself before you look: recalling it is what makes it stick.
-
What does each kind of security test read, and what does it miss?
SAST reads source code and misses deployed settings and intent. SCA reads the lockfile against advisories and misses whether your code reaches the flaw. Secret scanning reads files and history and misses secrets that look ordinary. DAST probes a running copy from outside and misses code the crawl never reaches and logic flaws. IAST watches a running copy from inside and misses what the tests do not run. Fuzzing and property tests feed generated inputs to a function and miss anything the invariants do not say.
-
What do none of the security tools read?
Intent: who may open an invoice, whether a coupon can be applied twice, whether the design lets one customer act as another. Those need a threat model, review and tests you write; A06:2025 asks for tests of critical flows against the threat model. The Web Security Testing Guide 4.2 lists twelve groups of tests, and no single tool covers all twelve.
-
What are precision and recall for a static rule, and how do you find them?
Precision is the share of its findings that are right, and recall is the share of the real problems it finds. You cannot know either from documentation, so measure them on labelled code. In the lesson a text search found 13 of 14 real problems at a precision of 62 %, and a syntax-tree rule found 12 of 14 at 86 %, missing run(cmd, shell=True) after from subprocess import run.
-
Where in a pipeline does each kind of security test run?
A secret scan before the commit and again on each push, since a key that gets into the history never leaves it. SAST and SCA on a pull request, where the diff is small. Scans of images and infrastructure files when those files change. At night, a passive baseline scan of a copy used for testing, and slower scans before a release.
-
Which four habits keep security gates from drowning people?
Gate on new findings of high severity and keep a baseline of the old ones; fail fast and in plain words, naming a file, a rule and a fix; review suppressions, each with a reason and an expiry; and measure the tools by how many of their findings were real and how long it takes to go from alert to fix.
-
What makes a regression test for a security fix strong?
It fails on the code before the fix and passes on the code after it, and has three parts: attacking inputs that fail on every weak version, ordinary inputs that fail on every over-strict one, and one shared fixture of harmless canary inputs reused for every target. In the lesson a half fix that refused only names starting with ../ still had 2 path problems, and a fix that refused every name with a dot had 3.
-
How do property-based tests and fuzzing find what chosen examples miss?
A property names a rule that all inputs must obey, many inputs are generated from a printed seed, and a shrinker cuts a failing input to the smallest that still fails. An encoder that forgot > failed on the first case, shrunk to >; a path function that refused names starting with .. held for 93 cases and failed on case 94, shrunk to /.. . The counterexample becomes the canary of the next regression test, and a property can only say what you state.
-
What is the order of triage for a scanner finding?
Reproduce it in a copy you own; decide whether it is a true positive, false positive or accepted risk, and name its CWE; fix the cause, because two alerts that share a cause need one fix; prove it with the regression test that failed first and a new scan; and suppress only a finding that is still false, with a reason, an owner and a date or release for review. A scanner can be quiet because it no longer reaches the page.
Key takeaways
- Each kind of test answers one question: SAST reads code, SCA reads packages, secret scanning reads history, DAST and IAST read a running copy, and fuzzing feeds a function generated inputs. None reads intent.
- Measure a static rule on labelled code: searching the text found 13 of 14 real problems with a precision of 62 %, and reading the syntax tree found 12 of 14 with a precision of 86 %.
- Run the cheapest test first, gate on new high-severity findings, keep a baseline for the old ones, and give every suppression a reason, an owner and an expiry.
- Give every fixed vulnerability a regression test that failed first, with attacking inputs from one shared fixture and ordinary inputs that catch an over-strict fix.
- State rules that must hold for every input, generate many inputs, and shrink a failure to the smallest input that still fails.
- Triage is reproduce, classify, fix at the cause, prove with a test, and only then suppress.
Terms in this lesson
- DAST
- Dynamic application security testing: tools that send requests to a running copy of an application and read the responses. They see what the server really sends, and only the pages and parameters that their crawl reaches.
- fuzzing
- Testing by feeding a program unexpected, malformed or random input and watching for crashes and broken rules. It is aimed at parsers and validators, and it is run only against code you own.
- IAST
- Interactive application security testing: a sensor inside a running application watches data reach dangerous calls while tests or people use it. It sees only the code that actually runs during that time.
- precision
- The share of a rule's findings that are right: 12 right findings out of 14 is a precision of about 86 %. Low precision means noise, which teaches people to ignore the tool.
- property-based test
- A test that states a rule which must hold for every input, generates many inputs and reports one that breaks it. A good tool then shrinks the input to the smallest one that still fails.
- recall
- The share of the real problems that a rule finds: 12 found out of 14 real ones is a recall of about 86 %. Low recall means gaps that the tool's quiet report hides.
- SAST
- Static application security testing: tools that read source code, without running it, for unsafe calls and patterns. They work early and point at a line, and they cannot see deployed settings or what the code is meant to do.
- SCA
- Software composition analysis: tools that match the packages a project uses against lists of known flaws. They report vulnerable versions, and they usually cannot tell whether your code calls the flawed part.
References
- A05:2025 Injection (OWASP Foundation)
- A06:2025 Insecure Design (OWASP Foundation)
- A09:2025 Security Logging and Alerting Failures (OWASP Foundation)
- OWASP Web Security Testing Guide, version 4.2 (OWASP Foundation)
- OWASP DevSecOps Guideline, Static Application Security Testing (OWASP Foundation)
- OWASP DevSecOps Guideline, Dynamic Application Security Testing (OWASP Foundation)
- OWASP DevSecOps Guideline, Interactive Application Security Testing (OWASP Foundation)
- OWASP DevSecOps Guideline, Software Composition Analysis (OWASP Foundation)
- Fuzzing (OWASP Foundation)
- Code scanning with CodeQL (GitHub Docs)
- Bandit documentation (Bandit project (PyCQA))
- OSV-Scanner (OSV project (Google))
- Gitleaks (Gitleaks project)
- Hypothesis documentation (Hypothesis project)
- fast-check documentation (fast-check project)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress