Encrypting files, streams and passphrase data

crypto.subtle.encrypt takes a whole message at once, and Web Crypto has no streaming AEAD, so a 2 GB video cannot be one call on a phone. Cut it into 64 KiB chunks and seal each one, and a new problem appears: a file whose chunks are sealed independently still decrypts after its last chunk is dropped, two chunks are swapped or a chunk is added.

Encrypting files safely takes a chunk layout that refuses all three, keys derived from passphrases with their settings kept in the header, and no unverified plaintextunverified plaintext: Decrypted bytes handed out before the authentication tag has been checked. They may belong to a changed message, so they must not be written, shown or acted on until the check succeeds. in the output. This lesson builds each of them and then opens real age test files with Node.js’s standard library.

Files and streams are sealed in chunks that know their place

Encrypting files of any size therefore means encrypting them piece by piece, so that neither side holds the whole file in memory. The age format, a C2SP specification, shows the layout: a random file keyfile key: A random key made for one file and used for nothing else. It protects the file's contents, and is itself wrapped in the file's header, by a passphrase or by each recipient's key. for this file alone, a 16-byte stream nonce, a payload key from HKDFHKDF: The HMAC-based key derivation function of RFC 5869. HKDF-Extract turns input keying material and a salt into a pseudorandom key; HKDF-Expand turns that key and an info label into as many bytes as asked for, up to 255 hash blocks. More in “Deriving keys with HKDF” over both, and chunks of 64 KiB sealed with ChaCha20-Poly1305ChaCha20-Poly1305: The AEAD of RFC 8439: the ChaCha20 stream cipher with the Poly1305 authenticator, a 256-bit key, a 96-bit nonce and a 128-bit tag. It runs fast in plain software, without AES instructions. More in “Authenticated encryption: AES-GCM and ChaCha20”. Each chunk’s 12-byte nonce is an 11-byte chunk counter and a flag that is 1 only for the last chunk. The counter stops reordering; the flag stops a file from being cut short or extended. The age specification calls its payload a variant of STREAMSTREAM construction: A way to encrypt a long message in chunks with an AEAD in which each chunk's nonce holds its position and a flag for the last chunk, so that a reader detects chunks that are dropped, reordered, repeated or added. The age format uses a variant., a construction from the research literature, and libsodium’s encrypted streams make the same promise with a final tag. Step through a 200,000-byte file:

The program computes the layout for three sizes, with every chunk’s nonce:

# How a file is cut into authenticated chunks in the payload layout of the age format: a 16-byte random nonce in front,
# chunks of 64 KiB each sealed with a 16-byte tag, and a 12-byte nonce per chunk made of an 11-byte big-endian chunk
# counter and a last-chunk flag. A 200,000-byte file, an empty one and one of exactly two chunks.
CHUNK = 64 * 1024
TAG = 16
NONCE = 16


def chunk_nonce(counter, last):
    return counter.to_bytes(11, "big") + (b"\x01" if last else b"\x00")


def layout(size):
    """(plaintext bytes, chunk nonce) for every chunk; an empty file is one empty final chunk."""
    count = max(1, -(-size // CHUNK))
    return [(min(CHUNK, size - i * CHUNK), chunk_nonce(i, i == count - 1)) for i in range(count)]


for size in (200_000, 0, 131_072):
    chunks = layout(size)
    total = NONCE + sum(length + TAG for length, _ in chunks)
    print(f"{size:,} bytes: {len(chunks)} chunk{'s' if len(chunks) > 1 else ''}, {total:,} bytes stored, {total - size} of them overhead")
    for length, nonce in chunks:
        print(f"  {length:>6,} bytes   nonce {nonce.hex()}")

200,000 bytes: 4 chunks, 200,080 bytes stored, 80 of them overhead
  65,536 bytes   nonce 000000000000000000000000
  65,536 bytes   nonce 000000000000000000000100
  65,536 bytes   nonce 000000000000000000000200
   3,392 bytes   nonce 000000000000000000000301
0 bytes: 1 chunk, 32 bytes stored, 32 of them overhead
       0 bytes   nonce 000000000000000000000001
131,072 bytes: 2 chunks, 131,120 bytes stored, 48 of them overhead
  65,536 bytes   nonce 000000000000000000000000
  65,536 bytes   nonce 000000000000000000000101

The 200,000-byte file becomes three full chunks and one of 3,392 bytes, 80 bytes larger in all: the 16-byte stream nonce and a 16-byte tag per chunk. An empty file is still one empty final chunk, so its end is checked too, and a file of exactly two chunks ends with a full final chunk, never an empty extra one.

The same layout serves streams that never sit on a disk, such as an upload or a backup sent over the network, and those streams raise one more question: when to let plaintext out. Four other choices complete a design. Sealing chunks independently with random nonces protects each chunk but not their order or their number. Passphrase data needs a slow key-derivation function, such as scrypt or PBKDF2 at 600,000 iterations, before any key exists. The salt and cost of that function, such as an age file’s scrypt work factorwork factor: The cost setting of a password-based key derivation, such as scrypt's log₂ N in the age format, stored with the encrypted data so that each file opens with the cost it was made with., belong in the header of the passphrase data, where a reader finds them. And a streaming reader must not write or show plaintext until its tag, and for the whole file the final chunk, has checked.

Chunks that refuse truncation, reordering and additions Pro: opens with a Pro, Premium or Ultimate pass