Integrity checks, SRI and canonical hashing

A build that downloads a tool and its SHA-256 checksum from the same mirror accepts a replaced tool without a word: whoever could change one file could change both. A digest proves that bytes arrived as somebody intended, never who that somebody was, and most mistakes with hashes in real code come from forgetting the difference.

This lesson puts hashes to work and marks their limits. You separate integrity from authenticity, pin a script to its digest with Subresource Integrity, hash structured data so that one digest means exactly one message, see on messages made here why XOR sums and CRC-32 protect nothing against an attacker, and choose the right hash or check for a job. A two-step code lab at the end makes SRI values and unambiguous digests.

A matching checksum proves integrity, not who made the file

A digest from SHA-256 tells you whether bytes changed since the digest was made. That is data integritydata integrity: The guarantee that data has not been changed without authorisation since it was created, sent or stored. A MAC, an authenticated cipher's tag or a signature detects deliberate changes; a checksum catches only accidents. More in “What cryptography can and cannot do”. It says nothing about who made the bytes or the digest, which is source authenticationsource authentication: Assurance about who created or sent a piece of data, also called authenticity. A MAC shows that a holder of the shared key made it; a signature shows which private key made it. More in “What cryptography can and cannot do”, because anyone can compute a digest for any file: a hash has no key. Step through a build that trusts a checksum from the same place as its download:

The check passed on a changed file because both answers came from the same place. It started protecting the build only when the expected digest came from a source the mirror could not change. In practice that is one of three: the project’s own site over HTTPS or a lock file committed to your repository, which fix the digest you expect; a MAC with a key you share with the sender; or a digital signature, which anyone can check against the sender’s public key. The next module builds MACs, and the public-key module builds signatures.

HTTP has the same split. RFC 9530 defines the Content-Digest and Repr-Digest fields, which carry a body’s digest as in Content-Digest: sha-256=:…:, and says plainly that they are not meant as general protection against deliberate tampering: someone who can change the body in transit can replace the field too. For that it points to TLS or to HTTP Message Signatures, RFC 9421, which the api-design track covers.

Subresource Integrity pins a script to its digest

A page that loads a script from another server runs whatever that server sends. Subresource IntegritySubresource Integrity (SRI): A W3C mechanism that lets a page pin a script or stylesheet to a SHA-256, SHA-384 or SHA-512 digest in its integrity attribute. A response with another digest is refused as a network error., a W3C specification, lets the page name the exact bytes it expects: the integrity attribute of a script or link element holds an algorithm, a hyphen and the Base64 of a digest, and browsers must support SHA-256, SHA-384 and SHA-512. Step through what the browser does with it:

The program computes the value for the lesson’s small script with Web Crypto, compares it with the value its page pins, then changes one byte of the script, as a changed copy on the other server would:

// Subresource Integrity: compute the integrity value of a script with Web Crypto, check it against the value the page
// carries, and show what one changed byte does. The page and the script are files beside this program.
import { readFileSync } from 'node:fs';

async function integrityOf(bytes, algorithm = 'SHA-384') {
  const digest = await crypto.subtle.digest(algorithm, bytes);
  return `${algorithm.toLowerCase().replace('-', '')}-${Buffer.from(digest).toString('base64')}`;
}

const script = readFileSync('app.js');
const page = readFileSync('page.html', 'utf8');
const pinned = /integrity="([^"]+)"/.exec(page)[1];

const value = await integrityOf(script);
console.log(`integrity of app.js: ${value}`);
console.log(`the page pins:       ${pinned}`);
console.log(`a browser runs it:   ${value === pinned}`);

const changed = Buffer.from(script);
changed[changed.indexOf('ran')] = 'R'.charCodeAt(0); // one byte changed on the other server
const after = await integrityOf(changed);
console.log(`after one changed byte: ${after.slice(0, 20)}…`);
console.log(`a browser runs it:   ${after === pinned} (it reports a network error instead)`);
console.log(`the value for SHA-256 would be ${(await integrityOf(script, 'SHA-256')).length} characters, for SHA-512 ${(await integrityOf(script, 'SHA-512')).length}`);

integrity of app.js: sha384-rDst8WSLk6rpAjGeULUERiKE1i87H21JKbSP+JD3rdKoPDqMQkx+fZxtxMql3bkE
the page pins:       sha384-rDst8WSLk6rpAjGeULUERiKE1i87H21JKbSP+JD3rdKoPDqMQkx+fZxtxMql3bkE
a browser runs it:   true
after one changed byte: sha384-6lr6I+D8uIoIu…
a browser runs it:   false (it reports a network error instead)
the value for SHA-256 would be 51 characters, for SHA-512 95

The value matched the page, so a browser would run the script. After one changed byte the SHA-384 digest shares nothing with the pinned one, and the specification has the browser refuse the response as a network error, with an error event the page can use to load a fallback. SHA-384 values take 71 characters, SHA-256 values 51 and SHA-512 values 95. Because the script comes from another origin, the element needs crossorigin="anonymous": the specification notes that integrity checks need CORS.

One digest should mean one message: framing and canonical forms

A digest is only unambiguous if the bytes it covers are. Code that hashes several fields by joining them loses the boundaries between them, so two different requests can hash alike. This program builds a download-link digest from a user name and a path, first naively, then with each part framed by its length:

"""Hashing fields joined together is ambiguous; length prefixes and a purpose label make one digest mean one message."""
import hashlib


def naive(fields):
    return hashlib.sha256("".join(fields).encode()).hexdigest()


def framed(fields, label):
    """SHA-256 over the label and every field, each preceded by its length in bytes as 4 bytes, big-endian."""
    h = hashlib.sha256()
    for part in [label, *fields]:
        data = part.encode()
        h.update(len(data).to_bytes(4, "big"))
        h.update(data)
    return h.hexdigest()


a = ["user 12", "3/report.pdf"]  # user "user 12" asks for "3/report.pdf"
b = ["user 123", "/report.pdf"]  # user "user 123" asks for "/report.pdf"
print(f"joined:   {''.join(a)!r} and {''.join(b)!r}")
print(f"naive digests equal:  {naive(a) == naive(b)}")
print(f"framed digests equal: {framed(a, 'download-link/v1') == framed(b, 'download-link/v1')}")

print(f"two empty fields vs one: naive equal {naive(['', '']) == naive([''])}, framed equal {framed(['', ''], 'x') == framed([''], 'x')}")
same = ["user 12", "3/report.pdf"]
print(f"same fields, labels 'download-link/v1' and 'audit-log/v1': equal {framed(same, 'download-link/v1') == framed(same, 'audit-log/v1')}")
print(f"'é' counts as {len('é'.encode())} bytes in its prefix, not 1 character")

joined:   'user 123/report.pdf' and 'user 123/report.pdf'
naive digests equal:  True
framed digests equal: False
two empty fields vs one: naive equal True, framed equal False
same fields, labels 'download-link/v1' and 'audit-log/v1': equal False
'é' counts as 2 bytes in its prefix, not 1 character

user 12 with 3/report.pdf and user 123 with /report.pdf join to the same text, so the naive digests are equal, and a link signed for one user would work for the other. Putting each part’s length in bytes in front of it, as 4 bytes, makes the encoding reversible, so different lists can no longer meet; it also keeps two empty fields apart from one. The label in front, download-link/v1, adds domain separationdomain separation: Making the purpose part of what is hashed or signed, for example with a label such as download-link/v1 in front of the fields, so that a value made for one purpose is never accepted for another.: the same fields hashed for the audit log under audit-log/v1 give another digest, so a value made for one purpose is never accepted for another. Change the fields in the program and try to make the framed digests meet.

Structured formats bring a second problem: the same data can be written in many texts. JSON allows any key order, any spacing and several spellings of one number, so two systems that agree on the data can still disagree on the bytes. RFC 8785, the JSON Canonicalization Scheme, fixes one text per value: no whitespace, object keys sorted by their UTF-16 code units at every level, and numbers and strings written by the ECMAScript rules. This program applies those rules to the JSON it uses:

// The JSON Canonicalization Scheme (RFC 8785) for the JSON this example uses: object keys sorted by UTF-16 code units,
// no whitespace, numbers and strings written as JSON.stringify writes them (the ECMAScript rules RFC 8785 adopts).
// A minimal canonicaliser for the example; production code uses a tested library.
import { createHash } from 'node:crypto';

function canonical(value) {
  if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
  if (value !== null && typeof value === 'object') {
    const keys = Object.keys(value).sort(); // default sort compares UTF-16 code units
    return `{${keys.map((k) => `${JSON.stringify(k)}:${canonical(value[k])}`).join(',')}}`;
  }
  if (typeof value === 'number' && !Number.isFinite(value)) throw new Error('NaN and Infinity are not JSON');
  return JSON.stringify(value);
}

const first = '{"amount": 500.00, "to": "Asha", "currency": "INR", "tags": ["rent", "monthly"]}';
const second = '{\n  "tags": ["rent", "monthly"],\n  "currency": "INR",\n  "to": "Asha",\n  "amount": 5e2\n}';
const sha256 = (text) => createHash('sha256').update(text).digest('hex');

console.log(`the two texts are equal as text: ${first === second}`);
console.log(`their SHA-256 digests are equal: ${sha256(first) === sha256(second)}`);
const a = canonical(JSON.parse(first));
const b = canonical(JSON.parse(second));
console.log(`canonical form: ${a}`);
console.log(`canonical forms equal: ${a === b}; digest ${sha256(a).slice(0, 16)}… for both`);

the two texts are equal as text: false
their SHA-256 digests are equal: false
canonical form: {"amount":500,"currency":"INR","tags":["rent","monthly"],"to":"Asha"}
canonical forms equal: true; digest bcde00c092895c39… for both

The two texts differ in key order, spacing and in 500.00 against 5e2, so their digests differ. Both canonicalise to the same 69-character text, which has one digest. Hash or sign the canonical form, and send the JSON as it is.

XOR sums and CRC-32 catch accidents, never attackers

A fixed-size output and speed are not what makes a hash cryptographic. The Handbook of Applied Cryptography gives the counterexamples: checksums such as a running XOR of 32-bit words or a CRC detect transmission errors, but a data string with any chosen checksum is easy to build, for the XOR sum by appending one block that corrects the total. CRCs are linear codes, the Handbook says, with one-wayness comparable to an XOR sum. This program runs those checks, and the same checks on SHA-256 and HMAC-SHA-256, on messages made here:

"""Textbook simple hashes and CRC-32 put to the checks an attacker would make, on messages made here, and the same
checks on SHA-256 and HMAC-SHA-256. "weak: True" means the check found the weakness."""
import hashlib
import hmac
import random
import zlib

BLOCK = 8
MASK = 2**64 - 1


def blocks(message):
    message = message + bytes(-len(message) % BLOCK)  # pad with zeros to whole 8-byte blocks
    return [int.from_bytes(message[i : i + BLOCK], "big") for i in range(0, len(message), BLOCK)]


def rotate(h):
    return ((h << 1) | (h >> 63)) & MASK


def xor_hash(message):
    h = 0
    for b in blocks(message):
        h ^= b
    return h


def rotate_xor_hash(message):
    h = 0
    for b in blocks(message):
        h = rotate(h) ^ b  # rotate the running value one bit left, then XOR the block in
    return h


results = []


def check(name, weak, expected):
    results.append(weak == expected)
    print(f"weak: {str(weak):<6}{name}")


original = b"ship 12 boxes to the warehouse in Pune"
other = b"ship 99 boxes to the yard in Kochi".ljust(40)  # a different message of five whole blocks
print(f"original: {original.decode()!r}\n")

reordered = b"".join(x.to_bytes(BLOCK, "big") for x in reversed(blocks(original)))
check("XOR hash: the blocks in reverse order keep the value", xor_hash(reordered) == xor_hash(original), True)

fix = xor_hash(original)
for b in blocks(other):
    fix ^= b
check("XOR hash: one chosen block gives another message the same value", xor_hash(other + fix.to_bytes(BLOCK, "big")) == xor_hash(original), True)

running = 0
for b in blocks(other):
    running = rotate(running) ^ b
last = rotate(running) ^ rotate_xor_hash(original)
check("rotate-XOR hash: the same works", rotate_xor_hash(other + last.to_bytes(BLOCK, "big")) == rotate_xor_hash(original), True)

check("SHA-256: the blocks in reverse order keep the digest", hashlib.sha256(reordered).digest() == hashlib.sha256(original).digest(), False)

print(f"\nCRC-32 check value of '123456789': {zlib.crc32(b'123456789'):08x}")
rng = random.Random(7)  # seeded, so the 1,000 cases repeat on every run
affine = 0
for _ in range(1000):
    size = rng.randrange(1, 200)
    m, d = rng.randbytes(size), rng.randbytes(size)
    flipped = bytes(x ^ y for x, y in zip(m, d))
    affine += zlib.crc32(flipped) == zlib.crc32(m) ^ zlib.crc32(d) ^ zlib.crc32(bytes(size))
check(f"CRC-32: crc(m ^ d) = crc(m) ^ crc(d) ^ crc(zeros) held in {affine:,} of 1,000 cases", affine == 1000, True)

d = bytearray(len(original))
d[5], d[6] = ord("1") ^ ord("9"), ord("2") ^ ord("9")  # the change that turns "12" into "99"
changed = bytes(x ^ y for x, y in zip(original, d))
patched = zlib.crc32(original) ^ zlib.crc32(bytes(d)) ^ zlib.crc32(bytes(len(original)))
check(f"CRC-32: the checksum of {changed[:12].decode()!r}… follows without reading it", patched == zlib.crc32(changed), True)

key = b"a key that only the sender and the receiver hold"
tag = lambda message: int.from_bytes(hmac.new(key, message, "sha256").digest(), "big")
check("HMAC-SHA-256: the same patch gives a valid tag", tag(original) ^ tag(bytes(d)) ^ tag(bytes(len(original))) == tag(changed), False)

print(f"\n{sum(results)} of {len(results)} checks came out as the lesson says")

original: 'ship 12 boxes to the warehouse in Pune'

weak: True  XOR hash: the blocks in reverse order keep the value
weak: True  XOR hash: one chosen block gives another message the same value
weak: True  rotate-XOR hash: the same works
weak: False SHA-256: the blocks in reverse order keep the digest

CRC-32 check value of '123456789': cbf43926
weak: True  CRC-32: crc(m ^ d) = crc(m) ^ crc(d) ^ crc(zeros) held in 1,000 of 1,000 cases
weak: True  CRC-32: the checksum of 'ship 99 boxe'… follows without reading it
weak: False HMAC-SHA-256: the same patch gives a valid tag

7 of 7 checks came out as the lesson says

The XOR hash keeps its value when the message’s blocks are reversed, and one chosen block gave a different message the original’s value; rotating the running value before each XOR did not help. CRC-32 reproduced its standard check value cbf43926 for 123456789 and showed the property that sinks it: for messages of equal length, crc(m ⊕ d) = crc(m) ⊕ crc(d) ⊕ crc(zeros), in 1,000 of 1,000 random cases. So the checksum of a changed message follows from the old checksum and the change alone, without reading the message. Against HMAC-SHA-256 the same patch fails, because a tag cannot be computed without the key.

This is not a toy problem. The original Wi-Fi encryption, WEP, put a CRC-32 inside its encrypted frames as the integrity check. The 2001 analysis by Borisov, Goldberg and Wagner showed that the checksum’s linearity let frames be modified without detection, and named a keyed MAC such as HMAC as the remedy.

Choosing a hash or a check for the job

The right choice follows from two questions: could anyone change the data on purpose, and must the value show who made it? A hash table adds a third, because a table that hashes keys chosen by users can be flooded with keys that collide. Python met that problem in its own dictionaries: PEP 456 moved the hashing of strings and bytes to SipHash, a keyed function with a 128-bit seed, to stop hash-collision denial of service, and the interpreter reports it:

"""Python's own dictionaries hash str and bytes with a keyed function, so that crafted keys cannot be made to collide."""
import sys

print(f"sys.hash_info.algorithm: {sys.hash_info.algorithm}")
print(f"its key (seed) is {sys.hash_info.seed_bits} bits, its output {sys.hash_info.hash_bits} bits")

sys.hash_info.algorithm: siphash13
its key (seed) is 128 bits, its output 64 bits

The recorded run reports siphash13, with a 128-bit key and a 64-bit output. Answer the three questions in the table to see which hash or check fits your case:

In short: a checksum such as CRC-32 for accidents, a keyed hash such as SipHash inside hash tables, SHA-256 for integrity when the expected digest arrives over a trusted channel, HMAC when both sides share a key, and a signature when a third party must be able to check who made the data. ASVS 5.0’s Appendix C lists every CRC as disallowed for cryptographic use, whatever its length.

Practise: SRI values and unambiguous digests

Two exercises, with sample tests that run on this page, turn the lesson into build tools. The first makes and checks Subresource Integrity values as a browser decides, strongest algorithm first, and refuses attributes that would check nothing. The second hashes a list of fields under a purpose label with length prefixes, held to the exact bytes of the encoding:

import hashlib


def sri_for(content, algorithm="sha384"):
    """The integrity value of content: '<algorithm>-<Base64 of the digest>'."""
    return f"{algorithm}-{hashlib.new(algorithm, content).hexdigest()}"  # hex, not Base64


def check_sri(content, integrity):
    """True if content matches the integrity attribute, as a browser decides."""
    first = integrity.split()[0]  # only the first value is looked at
    algorithm = first.split("-", 1)[0]
    return sri_for(content, algorithm) == first

import base64
import hashlib

from sri import check_sri, sri_for

SCRIPT = b"// A small script that a page loads from another server.\ndocument.querySelector('#status').textContent = 'The script ran.';\n"
PINNED = "sha384-rDst8WSLk6rpAjGeULUERiKE1i87H21JKbSP+JD3rdKoPDqMQkx+fZxtxMql3bkE"  # the lesson's page


def raises_value_error(call):
    try:
        call()
    except ValueError:
        return True
    return False


def test_the_lessons_value():
    """the lesson's app.js gives the value its page pins"""
    assert sri_for(SCRIPT) == PINNED


def test_other_algorithms():
    """sha256 and sha512 values, with Base64 padding"""
    assert sri_for(b"abc", "sha256") == "sha256-ungWv48Bz+pBQUDeXa4iI7ADYaOWF3qctBD/YfIAFa0="
    value = sri_for(b"abc", "sha512")
    assert value.startswith("sha512-") and len(value) == 7 + 88


def test_only_three_algorithms():
    """md5, sha1 and sha3_256 are refused when making a value"""
    for name in ["md5", "sha1", "sha3_256"]:
        assert raises_value_error(lambda: sri_for(b"abc", name)), name


def test_check_matches_and_mismatches():
    """the pinned script passes, one changed byte fails"""
    assert check_sri(SCRIPT, PINNED)
    assert not check_sri(SCRIPT.replace(b"ran", b"Ran"), PINNED)


def test_strongest_algorithm_decides():
    """a right sha256 value does not help when a sha512 value is present and wrong"""
    wrong512 = "sha512-" + base64.b64encode(hashlib.sha512(b"other").digest()).decode()
    assert not check_sri(SCRIPT, f"{sri_for(SCRIPT, 'sha256')} {wrong512}")
    assert check_sri(SCRIPT, f"{wrong512} {sri_for(SCRIPT, 'sha512')}")


def test_two_versions_of_one_algorithm():
    """either of two sha384 values may match, as during an update"""
    other = sri_for(b"the next version", "sha384")
    assert check_sri(SCRIPT, f"{other} {PINNED}")


def test_nothing_usable():
    """an attribute with only md5 or unknown values is refused"""
    for attribute in ["md5-kAFQmDzST7DWlj99KOF/cg==", "sha1-qZk+NkcGgWq6PiVxeFDCbJzQ2J0=", ""]:
        assert raises_value_error(lambda: check_sri(SCRIPT, attribute)), attribute

import hashlib


def tuple_digest(fields, label):
    """SHA-256 of a list of text fields under a purpose label, with every part length-prefixed."""
    return hashlib.sha256("".join(fields).encode()).hexdigest()  # ambiguous, and the label is ignored

import hashlib

from tuple_digest import tuple_digest


def raises_value_error(call):
    try:
        call()
    except ValueError:
        return True
    return False


def test_the_encoding_exactly():
    """label 'v1' and fields 'ab', 'c': 4-byte lengths in front of each part, then SHA-256"""
    encoded = b"\x00\x00\x00\x02v1" + b"\x00\x00\x00\x02ab" + b"\x00\x00\x00\x01c"
    assert tuple_digest(["ab", "c"], "v1") == hashlib.sha256(encoded).hexdigest()


def test_shifted_boundaries_differ():
    """['ab', 'c'] and ['a', 'bc']; the lesson's 'user 12' + '3/report.pdf' and 'user 123' + '/report.pdf'"""
    assert tuple_digest(["ab", "c"], "v1") != tuple_digest(["a", "bc"], "v1")
    assert tuple_digest(["user 12", "3/report.pdf"], "v1") != tuple_digest(["user 123", "/report.pdf"], "v1")


def test_empty_fields_count():
    """two empty fields, one empty field and no field are three different lists"""
    digests = {tuple_digest(fields, "v1") for fields in [["", ""], [""], []]}
    assert len(digests) == 3


def test_labels_separate_purposes():
    """the same fields under two labels give two digests"""
    assert tuple_digest(["user 12"], "download-link/v1") != tuple_digest(["user 12"], "audit-log/v1")


def test_lengths_count_bytes():
    """'é' is 2 bytes in UTF-8, so its prefix says 2"""
    encoded = b"\x00\x00\x00\x02v1" + b"\x00\x00\x00\x02" + "é".encode()
    assert tuple_digest(["é"], "v1") == hashlib.sha256(encoded).hexdigest()


def test_empty_label_is_refused():
    """a label is required"""
    assert raises_value_error(lambda: tuple_digest(["a"], ""))

The self-check below covers the five ideas. It begins with the idea you know least, raises the level as your answers come out right, and brings back what you miss:

Interview questions

Warm-up (fresher to mid level): how do integrity and authenticity differ, and which tools give each?

Integrity means the data has not changed since some reference point; authenticity means you know who produced it. A checksum or a hash such as SHA-256 gives integrity against accidents, and against deliberate changes only when the expected digest reaches you over a channel the attacker cannot touch, because anyone can compute a digest for any data. Authenticity needs a key: a MAC such as HMAC-SHA-256 shows that someone holding the shared key made the data, and a digital signature shows which private key made it, in a way a third party can check. So a checksum downloaded from the same mirror as the file proves almost nothing, while a signature from the project’s key, or a digest from its own site over HTTPS, does.

Key takeaways

• A matching digest proves integrity, not authenticity: a checksum from the same place as the file can be changed with it, so the expected digest must come from a trusted channel, a MAC or a signature.
• Subresource Integrity pins a script or stylesheet to a SHA-256, SHA-384 or SHA-512 digest; one changed byte makes the browser refuse it, and an attribute with no usable value checks nothing.
• Hash structured data through an encoding with one reading: lengths in front of fields, a label for the purpose, and RFC 8785’s canonical form for JSON.
• XOR sums and CRC-32 catch accidents only: CRC-32 is affine, so a change and its new checksum follow without the data, while the same patch fails against HMAC-SHA-256.
• Choose by the threat: checksums for accidents, SipHash in hash tables, SHA-256 with a trusted digest, HMAC with a shared key, and signatures when others must check who made the data.