Bytes, hex and Base64 in crypto code

Every crypto function reads bytes, never text: the word café can be 5 bytes or 6, and its SHA-256 digest changes with them. A webhook check that parses the JSON before verifying it rejects valid messages, and a Base64 decoder that skips junk turns Zm9v!!YmFy into foobar without a word. Bugs of this kind fail quietly, often only for some users.

This lesson keeps bytes exact from end to end. You turn text into UTF-8, write bytes as hex, Base64 and Base64url by the rules of RFC 4648, see why a MAC must check the bytes exactly as they arrived, decode untrusted Base64 strictly, and move the same bytes between Web Crypto, Node.js and Python, with a two-step code lab at the end.

Crypto functions read bytes, so text must become bytes first

hashlib.sha256() takes bytes, and crypto.subtle.digest() takes an ArrayBuffer or a typed array. Text has to be encoded first, and in crypto code that encoding is UTF-8UTF-8: The standard way to store Unicode text as bytes: 1 byte for ASCII characters and 2 to 4 bytes for every other code point. TextEncoder and Python's str.encode() produce it.: in JavaScript TextEncoder always produces UTF-8, and in Python str.encode() uses it by default. What looks like one string can still become different bytes. Unicode can write é as one code pointA number that Unicode assigns to one character or mark, written U+00E9 for é. UTF-8 stores each code point in 1 to 4 bytes., U+00E9, or as e followed by a combining accent, U+0301. The normalisation formsUnicode normalisation: Rewriting text into one standard form, so that text that looks the same has the same code points. NFC composes characters such as é into one code point; NFD splits them into a letter and combining marks. of Unicode Annex 15 name the two: NFC composes, NFD decomposes. Run the program to see what that does to the bytes and to a digest, then add your own name, or a word in your own language, to samples:

"""Text becomes bytes before any crypto function sees it, and the same-looking text can become different bytes."""
import hashlib
import unicodedata

samples = [
    ("hello", "hello"),
    ("café, NFC", unicodedata.normalize("NFC", "café")),
    ("café, NFD", unicodedata.normalize("NFD", "café")),
    ("नमस्ते", "नमस्ते"),
    ("😀", "😀"),
]
print(f"{'Text':<11}{'len()':<7}{'UTF-8 bytes':<13}Hex")
for label, text in samples:
    data = text.encode("utf-8")
    print(f"{label:<11}{len(text):<7}{len(data):<13}{data.hex()}")

nfc = unicodedata.normalize("NFC", "café")
nfd = unicodedata.normalize("NFD", "café")
print()
print("Equal as Python strings:", nfc == nfd)
print("SHA-256 of the NFC bytes:", hashlib.sha256(nfc.encode()).hexdigest()[:16], "...")
print("SHA-256 of the NFD bytes:", hashlib.sha256(nfd.encode()).hexdigest()[:16], "...")
same = unicodedata.normalize("NFC", nfd) == nfc
print("Equal after normalising both to NFC:", same)
print("Back to text:", bytes.fromhex("e0a4a8e0a4aee0a4b8e0a58de0a4a4e0a587").decode("utf-8"))

Text       len()  UTF-8 bytes  Hex
hello      5      5            68656c6c6f
café, NFC  4      5            636166c3a9
café, NFD  5      6            63616665cc81
नमस्ते     6      18           e0a4a8e0a4aee0a4b8e0a58de0a4a4e0a587
😀          1      4            f09f9880

Equal as Python strings: False
SHA-256 of the NFC bytes: 850f7dc43910ff89 ...
SHA-256 of the NFD bytes: 81ef060bcd98adc7 ...
Equal after normalising both to NFC: True
Back to text: नमस्ते

The two forms of café print the same on screen, yet one is 5 bytes and the other 6, and their SHA-256 digests share nothing. A password typed on one keyboard as NFC and on another as NFD would fail to match, which is why NIST’s SP 800-63B-4 recommends normalising passwords to NFC before they are hashed. Length is no help either: नमस्ते is 6 characters and 18 bytes, and the emoji is 1 character in Python, 2 in JavaScript and 4 bytes in both. The same program in JavaScript shows that difference in its .length column:

// The same text in JavaScript: TextEncoder always produces UTF-8, and .length counts UTF-16 code units.
const hex = (bytes) => Array.from(bytes, (b) => b.toString(16).padStart(2, '0')).join('');
const encoder = new TextEncoder();

const samples = [
  ['hello', 'hello'],
  ['café, NFC', 'café'.normalize('NFC')],
  ['café, NFD', 'café'.normalize('NFD')],
  ['नमस्ते', 'नमस्ते'],
  ['😀', '😀'],
];
console.log(`${'Text'.padEnd(11)}${'.length'.padEnd(9)}${'UTF-8 bytes'.padEnd(13)}Hex`);
for (const [label, text] of samples) {
  const data = encoder.encode(text);
  console.log(`${label.padEnd(11)}${String(text.length).padEnd(9)}${String(data.length).padEnd(13)}${hex(data)}`);
}

const nfc = 'café'.normalize('NFC');
const nfd = 'café'.normalize('NFD');
const sha256 = async (text) => hex(new Uint8Array(await crypto.subtle.digest('SHA-256', encoder.encode(text))));
console.log();
console.log('Equal as JavaScript strings:', nfc === nfd);
console.log('SHA-256 of the NFC bytes:', (await sha256(nfc)).slice(0, 16), '...');
console.log('SHA-256 of the NFD bytes:', (await sha256(nfd)).slice(0, 16), '...');
console.log('Equal after normalising both to NFC:', nfd.normalize('NFC') === nfc);

Text       .length  UTF-8 bytes  Hex
hello      5        5            68656c6c6f
café, NFC  4        5            636166c3a9
café, NFD  5        6            63616665cc81
नमस्ते     6        18           e0a4a8e0a4aee0a4b8e0a58de0a4a4e0a587
😀         2        4            f09f9880

Equal as JavaScript strings: false
SHA-256 of the NFC bytes: 850f7dc43910ff89 ...
SHA-256 of the NFD bytes: 81ef060bcd98adc7 ...
Equal after normalising both to NFC: true

Hex and Base64 write bytes as text, and change nothing else

Bytes often have to travel through places made for text: JSON, a URL, an HTTP header, a configuration file. RFC 4648 defines the standard ways to write them. Hexhex (Base16): Writing each byte as two characters from 0-9 and a-f, as RFC 4648 defines Base16. It doubles the size of the data and is case-insensitive., its Base16, uses two characters per byte. Base64Base64: The encoding of RFC 4648 that writes every 3 bytes as 4 characters from A-Z, a-z, 0-9, + and /, with = as padding. It carries bytes through text channels and hides nothing. packs 3 bytes into 4 characters from an alphabet of 64, so it grows data by a third; Base64urlBase64url: Base64 with - and _ in place of + and /, safe in URLs and file names (RFC 4648 section 5). Tokens and JSON Web Signatures usually write it without = padding. is the same with - and _ instead of + and /, safe in URLs and file names; Base32 uses only capital letters and the digits 2 to 7, for places that may lose the difference between capital and small letters; the one-time password lesson of this track meets it again. Here is how Base64 turns the two bytes of fo into four characters:

The final = is paddingpadding (Base64): The = characters that fill the last group of four when the data does not divide into groups of 3 bytes: one = for 2 leftover bytes, two for 1.: it stands for the byte that the last group of three lacks. RFC 7515, the JSON Web Signature standard, writes Base64url with every trailing = left out, and so do most tokens. The program below computes the test vectors of RFC 4648 section 10 in all four forms; compare them with the standard:

"""The test vectors of RFC 4648 section 10, computed here, plus the unpadded Base64url form."""
import base64

print(f"{'Input':<9}{'Base16 (hex)':<14}{'Base32':<18}{'Base64':<10}Base64url, unpadded")
for n in range(7):
    data = b"foobar"[:n]
    b16 = base64.b16encode(data).decode()
    b32 = base64.b32encode(data).decode()
    b64 = base64.b64encode(data).decode()
    b64url = base64.urlsafe_b64encode(data).decode().rstrip("=")
    print(f"{repr(data.decode()):<9}{b16:<14}{b32:<18}{b64:<10}{b64url}".rstrip())

two = bytes([0xFB, 0xFF])
print()
print("Bytes fb ff: Base64", base64.b64encode(two).decode(), "| Base64url", base64.urlsafe_b64encode(two).decode())
key = bytes(range(32))
print("A 32-byte key: hex", len(key.hex()), "characters, Base64", len(base64.b64encode(key)), "characters")

Input    Base16 (hex)  Base32            Base64    Base64url, unpadded
''
'f'      66            MY======          Zg==      Zg
'fo'     666F          MZXQ====          Zm8=      Zm8
'foo'    666F6F        MZXW6===          Zm9v      Zm9v
'foob'   666F6F62      MZXW6YQ=          Zm9vYg==  Zm9vYg
'fooba'  666F6F6261    MZXW6YTB          Zm9vYmE=  Zm9vYmE
'foobar' 666F6F626172  MZXW6YTBOI======  Zm9vYmFy  Zm9vYmFy

Bytes fb ff: Base64 +/8= | Base64url -_8=
A 32-byte key: hex 64 characters, Base64 44 characters

Every value matches the RFC, from Zg== for f to Zm9vYmFy for foobar, and a 32-byte key takes 64 hex characters or 44 in Base64. All four forms can be reversed by anyone with one standard function, which is the point: an encoding carries bytes, it does not protect them, as the first lesson of this module showed for Base64.

Signatures and MACs cover exact bytes

A MAC or a signature is computed over bytes, and one changed byte, even a space, gives a different result. That creates a classic bug. A webhook arrives, the handler parses the JSON, then serialises it again to check the tag. The data is the same, but the bytes are not:

"""A MAC covers exact bytes: parse first and verify later, and a valid message fails."""
import base64
import hashlib
import hmac
import json

# A fixed demo key, so the output repeats. Real keys come from a secrets store.
KEY = b"key-for-demo-only-not-a-real-key"
received = b'{"order":4471,"amount":500}'  # the bytes exactly as they arrived
tag = hmac.new(KEY, received, hashlib.sha256).digest()


def check(body):
    return hmac.compare_digest(hmac.new(KEY, body, hashlib.sha256).digest(), tag)


reserialised = json.dumps(json.loads(received)).encode()
print("Received bytes:    ", received.decode())
print("After parse + dump:", reserialised.decode())
print("Same data, same bytes?", json.loads(received) == json.loads(reserialised), received == reserialised)
print("Tag check on the received bytes:", check(received))
print("Tag check on the re-serialised bytes:", check(reserialised))
print()
# Two different Base64 strings, one byte: the unused low bits of the last character are ignored.
for text in ["YQ==", "YR=="]:
    print(f"b64decode({text!r}) = {base64.b64decode(text)!r}")
print("Equal as text:", "YQ==" == "YR==", "| equal as bytes:", base64.b64decode("YQ==") == base64.b64decode("YR=="))
# Hex is case-insensitive, so one tag has many spellings.
upper, lower = tag.hex().upper(), tag.hex()
print("Hex tag, two spellings equal as text:", upper == lower, "| as bytes:", bytes.fromhex(upper) == bytes.fromhex(lower))

Received bytes:     {"order":4471,"amount":500}
After parse + dump: {"order": 4471, "amount": 500}
Same data, same bytes? True False
Tag check on the received bytes: True
Tag check on the re-serialised bytes: False

b64decode('YQ==') = b'a'
b64decode('YR==') = b'a'
Equal as text: False | equal as bytes: True
Hex tag, two spellings equal as text: False | as bytes: True

The re-serialised body gained three spaces, so the tag check failed on a genuine message. The rule is to verify the raw bytes exactly as they arrived, and to parse only after the check passes. The rest of the output shows the opposite problem: two different strings that stand for the same bytes. YQ== and YR== both decode to a, because the last four bits of their second character are padding bits that a lenient decoder ignores, and a tag written in upper-case hex is the same 32 bytes as in lower case. Code that compares, stores or blocks values as text sees two values where the bytes hold one, so compare decoded bytes, and for tags use a constant-time comparison such as hmac.compare_digest.

Decode untrusted Base64 strictly

RFC 4648 is strict about input: implementations must reject encoded data that contains characters outside the alphabet unless the specification that uses it says otherwise, and decoders may reject an encoding whose unused pad bits are not zero. Its security section explains why: a decoder that skips characters lets data hide in what it skips, and makes equality checks on strings unreliable. Most decoders are more forgiving than that. This program gives Node’s Buffer, atob() and a strict decoder of a few lines six malformed inputs:

// Six malformed Base64 inputs and one good one, given to Node's Buffer, to atob() and to a strict decoder.
const CANONICAL = /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/;

function strictDecode(text) {
  if (!CANONICAL.test(text)) throw new Error('bad characters, length or padding');
  const bytes = Buffer.from(text, 'base64');
  if (bytes.toString('base64') !== text) throw new Error('padding bits are not zero');
  return bytes;
}

// Printable ASCII as it is, any other byte as \xNN.
const show = (bytes) => Array.from(bytes, (b) => (b >= 0x20 && b < 0x7f ? String.fromCharCode(b) : `\\x${b.toString(16).padStart(2, '0')}`)).join('');
const attempt = (decode) => {
  try {
    return show(decode());
  } catch (err) {
    return `error: ${err.name === 'Error' ? err.message : err.name}`;
  }
};

const inputs = [
  ['Zm9vYmFy', 'good'],
  ['Zm9v!!YmFy', 'junk characters'],
  ['Zm9v YmFy', 'a space'],
  ['Zm9vYmE', 'padding missing'],
  ['Zm9vYmE===', 'padding extra'],
  ['Zm9vYmF=', 'last bits not zero'],
  ['Zm9v-_8=', 'URL-safe letters'],
];
for (const [text, what] of inputs) {
  console.log(`${JSON.stringify(text)} (${what})`);
  console.log(`  Buffer.from: ${attempt(() => Buffer.from(text, 'base64'))}`);
  console.log(`  atob:        ${attempt(() => Uint8Array.from(atob(text), (c) => c.charCodeAt(0)))}`);
  console.log(`  strict:      ${attempt(() => strictDecode(text))}`);
}

"Zm9vYmFy" (good)
  Buffer.from: foobar
  atob:        foobar
  strict:      foobar
"Zm9v!!YmFy" (junk characters)
  Buffer.from: foobar
  atob:        error: InvalidCharacterError
  strict:      error: bad characters, length or padding
"Zm9v YmFy" (a space)
  Buffer.from: foobar
  atob:        foobar
  strict:      error: bad characters, length or padding
"Zm9vYmE" (padding missing)
  Buffer.from: fooba
  atob:        fooba
  strict:      error: bad characters, length or padding
"Zm9vYmE===" (padding extra)
  Buffer.from: fooba
  atob:        error: InvalidCharacterError
  strict:      error: bad characters, length or padding
"Zm9vYmF=" (last bits not zero)
  Buffer.from: fooba
  atob:        fooba
  strict:      error: padding bits are not zero
"Zm9v-_8=" (URL-safe letters)
  Buffer.from: foo\xfb\xff
  atob:        error: InvalidCharacterError
  strict:      error: bad characters, length or padding

Buffer.from accepted all six, including the URL-safe letters inside standard Base64. atob() refused three and accepted the space, the missing padding and the non-zero last bits, as the WHATWG’s forgiving-base64 algorithm prescribes. Here are Python’s decoders on the same inputs. Add your own malformed strings to inputs and run it again:

"""The same seven inputs given to Python's b64decode, with and without validate=True, and to a strict decoder."""
import base64
import binascii
import re

CANONICAL = re.compile(r"(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?")


def strict_decode(text):
    if not CANONICAL.fullmatch(text):
        raise ValueError("bad characters, length or padding")
    data = base64.b64decode(text, validate=True)
    if base64.b64encode(data).decode() != text:
        raise ValueError("padding bits are not zero")
    return data


def show(data):
    """Printable ASCII as it is, any other byte as \\xNN."""
    return "".join(chr(b) if 0x20 <= b < 0x7F else f"\\x{b:02x}" for b in data)


def attempt(decode):
    try:
        return show(decode())
    except (ValueError, binascii.Error) as err:
        return f"error: {err}"


inputs = [
    ("Zm9vYmFy", "good"),
    ("Zm9v!!YmFy", "junk characters"),
    ("Zm9v YmFy", "a space"),
    ("Zm9vYmE", "padding missing"),
    ("Zm9vYmE===", "padding extra"),
    ("Zm9vYmF=", "last bits not zero"),
    ("Zm9v-_8=", "URL-safe letters"),
]
for text, what in inputs:
    print(f'"{text}" ({what})')
    print(f"  b64decode:          {attempt(lambda: base64.b64decode(text))}")
    print(f"  validate=True:      {attempt(lambda: base64.b64decode(text, validate=True))}")
    print(f"  strict:             {attempt(lambda: strict_decode(text))}")

"Zm9vYmFy" (good)
  b64decode:          foobar
  validate=True:      foobar
  strict:             foobar
"Zm9v!!YmFy" (junk characters)
  b64decode:          foobar
  validate=True:      error: Only base64 data is allowed
  strict:             error: bad characters, length or padding
"Zm9v YmFy" (a space)
  b64decode:          foobar
  validate=True:      error: Only base64 data is allowed
  strict:             error: bad characters, length or padding
"Zm9vYmE" (padding missing)
  b64decode:          error: Incorrect padding
  validate=True:      error: Incorrect padding
  strict:             error: bad characters, length or padding
"Zm9vYmE===" (padding extra)
  b64decode:          fooba
  validate=True:      error: Excess padding not allowed
  strict:             error: bad characters, length or padding
"Zm9vYmF=" (last bits not zero)
  b64decode:          fooba
  validate=True:      fooba
  strict:             error: padding bits are not zero
"Zm9v-_8=" (URL-safe letters)
  b64decode:          error: Invalid base64-encoded string: number of data characters (5) cannot be 1 more than a multiple of 4
  validate=True:      error: Only base64 data is allowed
  strict:             error: bad characters, length or padding

"Zm9vYmFy" (good)
  b64decode:          foobar
  validate=True:      foobar
  strict:             foobar
"Zm9v!!YmFy" (junk characters)
  b64decode:          foobar
  validate=True:      error: Only base64 data is allowed
  strict:             error: bad characters, length or padding
"Zm9v YmFy" (a space)
  b64decode:          foobar
  validate=True:      error: Only base64 data is allowed
  strict:             error: bad characters, length or padding
"Zm9vYmE" (padding missing)
  b64decode:          error: Incorrect padding
  validate=True:      error: Incorrect padding
  strict:             error: bad characters, length or padding
"Zm9vYmE===" (padding extra)
  b64decode:          fooba
  validate=True:      error: Excess data after padding
  strict:             error: bad characters, length or padding
"Zm9vYmF=" (last bits not zero)
  b64decode:          fooba
  validate=True:      fooba
  strict:             error: padding bits are not zero
"Zm9v-_8=" (URL-safe letters)
  b64decode:          error: Invalid base64-encoded string: number of data characters (5) cannot be 1 more than a multiple of 4
  validate=True:      error: Only base64 data is allowed
  strict:             error: bad characters, length or padding

b64decode() with its default settings discarded the junk and the space and let the extra padding through; validate=True refused five of the six but still took Zm9vYmF= for fooba. Only the strict decoder refused all six. It does two things: a pattern that allows only the alphabet and the two legal endings, then a round trip that encodes the result again and compares it with the input, which catches non-zero last bits. Which decoder your code can use depends on where the text comes from and what you do with it. Answer the questions in the table to see:

In short: for anything that arrives from outside, decode strictly, and whenever the text itself is compared, stored as a key or signed, insist on the canonical formcanonical encoding: The one spelling an encoder writes for a value. For Base64 it means the unused bits of the last character are zero, so that every value has exactly one valid string. as well.

Moving bytes between Web Crypto, Buffer and Python

Each runtime has its own type for bytes. Web Crypto returns an ArrayBuffer, a block of memory with no methods to read it, and you read it through a Uint8Array, a view of that memory. Node.js adds Buffer, a subclass of Uint8Array with conversions built in. Python has one immutable type, bytes, and a mutable bytearray. Follow one digest from JavaScript to Python:

Integers need a byte order too. HOTP one-time passwords put their counter into HMAC as 8 bytes, high-order byte first (RFC 4226, sections 5.1 and 5.2); this is called big-endianbig-endian: A byte order that puts the most significant byte of a number first. HOTP counters, TLS record numbers and most network protocols use it.. JavaScript’s DataView.setBigUint64 writes big-endian unless its third argument asks for little-endian, and Python’s struct says which with a > or a <. The two programs below print the same digest, the same Base64url and the same counter:

// One digest, one counter and their encodings in JavaScript. across.py prints the same first six lines.
const hex = (bytes) => Array.from(bytes, (b) => b.toString(16).padStart(2, '0')).join('');
// Node.js 24 has no Uint8Array.prototype.toBase64 yet, so a small helper that works everywhere:
const base64url = (bytes) => btoa(String.fromCharCode(...bytes)).replaceAll('+', '-').replaceAll('/', '_').replace(/=+$/, '');

const data = new TextEncoder().encode('नमस्ते'); // Uint8Array of UTF-8 bytes
const digest = await crypto.subtle.digest('SHA-256', data); // an ArrayBuffer: no methods to read it
const bytes = new Uint8Array(digest); // a view of the same memory, to read the bytes
console.log('UTF-8 bytes:      ', data.length);
console.log('SHA-256, hex:     ', hex(bytes));
console.log('SHA-256, base64url:', base64url(bytes));

// An 8-byte counter, high-order byte first, as HOTP (RFC 4226) wants it.
const counter = new Uint8Array(8);
new DataView(counter.buffer).setBigUint64(0, 1n); // big-endian unless littleEndian is true
const little = new Uint8Array(8);
new DataView(little.buffer).setBigUint64(0, 1n, true);
console.log('Counter 1, big-endian:   ', hex(counter));
console.log('Counter 1, little-endian:', hex(little));
console.log('Back to a number:', new DataView(counter.buffer).getBigUint64(0).toString());

// Node's Buffer is a Uint8Array. Buffer.from(arrayBuffer) shares its memory; Buffer.from(uint8array) copies.
const shared = Buffer.from(digest);
const copied = Buffer.from(bytes);
bytes[0] ^= 0xff;
console.log('After changing bytes[0]: shared view follows', shared[0] === bytes[0], '| copy follows', copied[0] === bytes[0]);
console.log('Uint8Array.prototype.toBase64 in this Node.js:', typeof Uint8Array.prototype.toBase64);

UTF-8 bytes:       18
SHA-256, hex:      ddb08d77c2d511947652161b35987022711aa216387b041dd459a03eb66a8304
SHA-256, base64url: 3bCNd8LVEZR2UhYbNZhwInEaohY4ewQd1FmgPrZqgwQ
Counter 1, big-endian:    0000000000000001
Counter 1, little-endian: 0100000000000000
Back to a number: 1
After changing bytes[0]: shared view follows true | copy follows false
Uint8Array.prototype.toBase64 in this Node.js: undefined

"""One digest, one counter and their encodings in Python. across.mjs prints the same first six lines."""
import base64
import hashlib
import struct

data = "नमस्ते".encode("utf-8")  # bytes
digest = hashlib.sha256(data).digest()  # bytes as well: Python has one type for both
print("UTF-8 bytes:      ", len(data))
print("SHA-256, hex:     ", digest.hex())
print("SHA-256, base64url:", base64.urlsafe_b64encode(digest).rstrip(b"=").decode())

# An 8-byte counter, high-order byte first, as HOTP (RFC 4226) wants it.
counter = struct.pack(">Q", 1)  # ">" is big-endian, "Q" an unsigned 64-bit integer
print("Counter 1, big-endian:   ", counter.hex())
print("Counter 1, little-endian:", struct.pack("<Q", 1).hex())
print("Back to a number:", int.from_bytes(counter, "big"))

UTF-8 bytes:       18
SHA-256, hex:      ddb08d77c2d511947652161b35987022711aa216387b041dd459a03eb66a8304
SHA-256, base64url: 3bCNd8LVEZR2UhYbNZhwInEaohY4ewQd1FmgPrZqgwQ
Counter 1, big-endian:    0000000000000001
Counter 1, little-endian: 0100000000000000
Back to a number: 1

The JavaScript run ends with two lines of its own. Buffer.from(arrayBuffer) shares the memory it was given, so changing the Uint8Array changed the Buffer too, while Buffer.from(uint8array) made a copy. A key or a buffer you meant to keep can change under you that way, so copy deliberately when it matters.

Practise: a strict Base64url codec and a counter

Two exercises build the two conversions that crypto code needs most, and their sample tests run on this page. Step one is an unpadded Base64url codec whose decoder refuses everything its encoder would never write, held to the RFC 4648 vectors. Step two packs a 64-bit counter big-endian, held in Python to the official HOTP values of RFC 4226:

import base64


def base64url_encode(data):
    """data (bytes) as Base64url text, without '=' padding."""
    return base64.urlsafe_b64encode(data).decode()  # still writes the padding


def base64url_decode(text):
    """The bytes of unpadded Base64url text; raises ValueError for anything else."""
    return base64.urlsafe_b64decode(text + "==")  # lenient: accepts far too much

from base64url import base64url_decode, base64url_encode

# RFC 4648 section 10, without the padding.
VECTORS = [(b"", ""), (b"f", "Zg"), (b"fo", "Zm8"), (b"foo", "Zm9v"), (b"foob", "Zm9vYg"), (b"fooba", "Zm9vYmE"), (b"foobar", "Zm9vYmFy")]


def rejected(text):
    try:
        base64url_decode(text)
    except ValueError:
        return True
    return False


def test_rfc_4648_vectors_encode():
    """the RFC 4648 section 10 vectors, unpadded"""
    for data, text in VECTORS:
        assert base64url_encode(data) == text, f"{data!r} should encode to {text!r}"


def test_rfc_4648_vectors_decode():
    """the same vectors decode back to their bytes"""
    for data, text in VECTORS:
        assert base64url_decode(text) == data, f"{text!r} should decode to {data!r}"


def test_url_safe_alphabet():
    """bytes fb ff use - and _ instead of + and /"""
    assert base64url_encode(bytes([0xFB, 0xFF])) == "-_8"
    assert base64url_decode("-_8") == bytes([0xFB, 0xFF])


def test_round_trip():
    """every length from 0 to 40 bytes, and every byte value, survive a round trip"""
    for n in range(41):
        data = bytes((7 * i + n) % 256 for i in range(n))
        assert base64url_decode(base64url_encode(data)) == data
    every = bytes(range(256))
    assert base64url_decode(base64url_encode(every)) == every


def test_rejects_what_the_encoder_never_writes():
    """padding, the standard alphabet, whitespace, other characters, impossible lengths, non-zero last bits"""
    for bad in ["Zg==", "Zm8=", "+_8", "-/8", "Zm9 v", "Zm9v\n", "Zm9v!", "Z", "Zm9vY", "Zh", "Zm9", "Zm9vYmF"]:
        assert rejected(bad), f"{bad!r} should raise ValueError"

const ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';

/** bytes (a Uint8Array) as Base64url text, without "=" padding. */
export function base64urlEncode(bytes) {
  // Standard Base64 with padding: change the alphabet and drop the padding.
  let out = '';
  for (let i = 0; i < bytes.length; i += 3) {
    const n = (bytes[i] << 16) | ((bytes[i + 1] ?? 0) << 8) | (bytes[i + 2] ?? 0);
    out += ALPHABET[(n >> 18) & 63] + ALPHABET[(n >> 12) & 63];
    out += i + 1 < bytes.length ? ALPHABET[(n >> 6) & 63] : '=';
    out += i + 2 < bytes.length ? ALPHABET[n & 63] : '=';
  }
  return out;
}

/** The bytes of unpadded Base64url text; throws for anything else. */
export function base64urlDecode(text) {
  // Lenient: skips every character it does not know. Make it strict.
  const out = [];
  let value = 0;
  let bits = 0;
  for (const ch of text) {
    const v = ALPHABET.indexOf(ch);
    if (v < 0) continue;
    value = ((value << 6) | v) & 0xffffff;
    bits += 6;
    if (bits >= 8) {
      bits -= 8;
      out.push((value >> bits) & 255);
    }
  }
  return Uint8Array.from(out);
}

import { test, assert } from 'mysmartcopilot:test';
import { base64urlDecode, base64urlEncode } from './base64url.mjs';

const ascii = (text) => Uint8Array.from(text, (c) => c.charCodeAt(0));
// RFC 4648 section 10, without the padding.
const VECTORS = [['', ''], ['f', 'Zg'], ['fo', 'Zm8'], ['foo', 'Zm9v'], ['foob', 'Zm9vYg'], ['fooba', 'Zm9vYmE'], ['foobar', 'Zm9vYmFy']];

test('the RFC 4648 section 10 vectors, unpadded', () => {
  for (const [data, text] of VECTORS) assert.equal(base64urlEncode(ascii(data)), text, `"${data}" should encode to "${text}"`);
});

test('the same vectors decode back to their bytes', () => {
  for (const [data, text] of VECTORS) assert.deepEqual(base64urlDecode(text), ascii(data), `"${text}" should decode to "${data}"`);
});

test('bytes fb ff use - and _ instead of + and /', () => {
  assert.equal(base64urlEncode(Uint8Array.from([0xfb, 0xff])), '-_8');
  assert.deepEqual(base64urlDecode('-_8'), Uint8Array.from([0xfb, 0xff]));
});

test('every length from 0 to 40 bytes, and every byte value, survive a round trip', () => {
  for (let n = 0; n <= 40; n++) {
    const data = Uint8Array.from({ length: n }, (_, i) => (7 * i + n) % 256);
    assert.deepEqual(base64urlDecode(base64urlEncode(data)), data, `${n} bytes`);
  }
  const every = Uint8Array.from({ length: 256 }, (_, i) => i);
  assert.deepEqual(base64urlDecode(base64urlEncode(every)), every);
});

test('padding, the standard alphabet, whitespace, other characters, impossible lengths, non-zero last bits', () => {
  for (const bad of ['Zg==', 'Zm8=', '+_8', '-/8', 'Zm9 v', 'Zm9v\n', 'Zm9v!', 'Z', 'Zm9vY', 'Zh', 'Zm9', 'Zm9vYmF']) {
    assert.throws(() => base64urlDecode(bad), `"${bad}" should throw`);
  }
});

def counter_to_bytes(n):
    """n as 8 bytes, high-order byte first; ValueError unless 0 <= n < 2**64."""
    return n.to_bytes(8, "little")  # the wrong byte order


def counter_from_bytes(data):
    """The number in 8 big-endian bytes; ValueError for any other length."""
    return int.from_bytes(data, "little")

import hashlib
import hmac

from counter import counter_from_bytes, counter_to_bytes


def raises_value_error(call):
    try:
        call()
    except ValueError:
        return True
    return False


def test_rfc_4226_appendix_d():
    """HMAC-SHA-1 over your counter bytes gives the official HOTP values for counts 0, 1 and 2"""
    secret = b"12345678901234567890"
    expected = ["cc93cf18508d94934c64b65d8ba7667fb7cde4b0", "75a48a19d4cbe100644e8ac1397eea747a2d33ab", "0bacb7fa082fef30782211938bc1c5e70416ff44"]
    for count, value in enumerate(expected):
        assert hmac.new(secret, counter_to_bytes(count), hashlib.sha1).hexdigest() == value, f"count {count}"


def test_big_endian():
    """the high-order byte comes first"""
    assert counter_to_bytes(1) == bytes.fromhex("0000000000000001")
    assert counter_to_bytes(0x0102030405060708) == bytes([1, 2, 3, 4, 5, 6, 7, 8])
    assert counter_to_bytes(2**64 - 1) == b"\xff" * 8


def test_out_of_range():
    """below 0 and from 2**64 up are refused"""
    for bad in [-1, 2**64, 2**70]:
        assert raises_value_error(lambda: counter_to_bytes(bad)), f"{bad} should raise ValueError"


def test_round_trip():
    """numbers come back unchanged, up to the largest 64-bit one"""
    for n in [0, 1, 255, 256, 2**32, 2**53 + 1, 2**64 - 1]:
        assert counter_from_bytes(counter_to_bytes(n)) == n, n


def test_from_bytes_needs_eight():
    """7 or 9 bytes are refused"""
    for bad in [bytes(7), bytes(9)]:
        assert raises_value_error(lambda: counter_from_bytes(bad)), f"{len(bad)} bytes should raise ValueError"

/** n (a BigInt or a safe integer) as 8 bytes, high-order byte first; throws unless 0 <= n < 2**64. */
export function counterToBytes(n) {
  const out = new Uint8Array(8);
  new DataView(out.buffer).setUint32(4, Number(n), true); // only 32 bits, and the wrong byte order
  return out;
}

/** The BigInt in 8 big-endian bytes; throws for any other length. */
export function counterFromBytes(bytes) {
  return BigInt(new DataView(bytes.buffer, bytes.byteOffset, 8).getUint32(4, true));
}

import { test, assert } from 'mysmartcopilot:test';
import { counterFromBytes, counterToBytes } from './counter.mjs';

test('the high-order byte comes first', () => {
  assert.deepEqual(counterToBytes(1), Uint8Array.from([0, 0, 0, 0, 0, 0, 0, 1]));
  assert.deepEqual(counterToBytes(0x0102030405060708n), Uint8Array.from([1, 2, 3, 4, 5, 6, 7, 8]));
  assert.deepEqual(counterToBytes(2n ** 64n - 1n), Uint8Array.from([255, 255, 255, 255, 255, 255, 255, 255]));
});

test('below 0, from 2**64 up and unsafe numbers are refused', () => {
  for (const bad of [-1n, -1, 2n ** 64n, 2 ** 53, 1.5]) assert.throws(() => counterToBytes(bad), `${String(bad)} should throw`);
});

test('numbers come back unchanged, up to the largest 64-bit one', () => {
  for (const n of [0n, 1n, 255n, 256n, 2n ** 32n, 2n ** 53n + 1n, 2n ** 64n - 1n]) assert.equal(counterFromBytes(counterToBytes(n)), n, String(n));
  assert.equal(counterFromBytes(counterToBytes(4294967296)), 4294967296n);
});

test('7 or 9 bytes are refused', () => {
  for (const bad of [new Uint8Array(7), new Uint8Array(9)]) assert.throws(() => counterFromBytes(bad), `${bad.length} bytes should throw`);
});

To finish, practise across the five ideas: questions on your weakest idea come first, each right answer earns a harder one, and a miss returns before the session ends:

Interview questions

Warm-up (fresher to mid level): what is the difference between Base64 and Base64url, and when do you use each?

Both write 3 bytes as 4 characters from an alphabet of 64, as RFC 4648 defines them. Base64 uses + and / for the last two values, and both mean something in URLs and file names, so a Base64 string in a query parameter can arrive changed. Base64url uses - and _ instead, and is usually written without the = padding, which is how JSON Web Tokens and many APIs carry signatures and keys. So use Base64 where the standard you follow says so, as in PEM files, MIME email and many data formats, and Base64url for anything that travels in a URL, a cookie, a token or a file name. Never mix them in one decoder: decode each with its own strict decoder, so that a value has one spelling only.

Key takeaways

• Crypto functions read bytes: encode text as UTF-8, normalise it (NFC) where people type it, and measure bytes, not characters.
• Hex, Base64, Base64url and Base32 write bytes as text by the rules of RFC 4648, can be reversed by anyone, and so protect nothing.
• A MAC or a signature covers exact bytes: verify the body as it arrived, before parsing, and compare decoded bytes rather than text.
• Decoders are lenient by default: for untrusted input, check the alphabet and padding and re-encode to reject non-canonical text.
• Know each runtime’s byte type and byte order: ArrayBuffer behind a Uint8Array view, Buffer’s shared memory, Python’s bytes, and big-endian counters for HOTP.