Text encoding, bytes and Base64

Text is stored and sent as bytes, and a character encoding is the rule that turns characters into bytes and back. UTF-8 is the encoding of the web, of JSON and of most files, and in JavaScript TextEncoder and TextDecoder convert between strings and UTF-8 bytes held in a Uint8Array. Base64 solves a different problem: it writes any bytes, an image or a key or encrypted data, as text made of 64 safe characters, so the bytes can travel through JSON, a URL or an email. ECMAScript 2026 gives Uint8Array its own toBase64 and fromBase64 methods, with toHex and fromHex beside them. They replace the old btoa and atob, which accept only characters up to U+00FF and so throw on नमस्ते and on most of the world’s text. This lesson converts text to bytes and back, takes Base64 apart bit by bit, and ends with a fallback for runtimes that do not have the new methods yet.

Strings, code units and UTF-8 bytes

A JavaScript string is a sequence of UTF-16 code units, and its length counts them. Most characters are one code unit; a character above U+FFFF, such as most emoji, is two, a surrogate pair. UTF-8 is a different way to write the same code points: one to four bytes each, depending on how large the code point is.

// TextEncoder turns a string into UTF-8 bytes: one to four bytes for each code point.
const encoder = new TextEncoder();

for (const ch of ['A', 'é', '₹', 'न', '😀']) {
  const codePoint = `U+${ch.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')}`;
  const bytes = encoder.encode(ch);
  console.log(`${ch}  ${codePoint.padEnd(7)}  ${ch.length} code unit(s)  ${bytes.length} byte(s): ${bytes.toHex()}`);
}

// A string's length counts UTF-16 code units, not characters and not bytes.
const text = 'Aé😀न';
const bytes = encoder.encode(text);
console.log(`${text}: length ${text.length}, code points ${[...text].length}, UTF-8 bytes ${bytes.length}`);
console.log([...bytes].map((b) => b.toString(16).padStart(2, '0')).join(' '));

// A lone surrogate (half of an emoji) is not a character: it is encoded as U+FFFD, the replacement character.
console.log(encoder.encode('\uD83D').toHex(), '|', '\uD83D'.isWellFormed());

A  U+0041   1 code unit(s)  1 byte(s): 41
é  U+00E9   1 code unit(s)  2 byte(s): c3a9
₹  U+20B9   1 code unit(s)  3 byte(s): e282b9
न  U+0928   1 code unit(s)  3 byte(s): e0a4a8
😀  U+1F600  2 code unit(s)  4 byte(s): f09f9880
Aé😀न: length 5, code points 4, UTF-8 bytes 10
41 c3 a9 f0 9f 98 80 e0 a4 a8
efbfbd | false

The same four characters, Aé😀न, are 5 code units, 4 code points and 10 bytes, and each of those numbers matters somewhere: length for string positions, code points for what a person sees as characters (most of the time), and bytes for what a database column, an HTTP header or a file actually stores. TextEncoder has no options, because it only ever produces UTF-8. It also cleans up one problem on the way: a lone surrogate, half of an emoji left over after a careless slice, is not a character at all, so it becomes the three bytes of U+FFFD, the replacement character. isWellFormed() tells you beforehand whether a string contains one.

From bytes back to text: TextDecoder

TextDecoder does the opposite, and it is more careful than it looks. Bytes come from outside the program, from a file, a network response or a device, and they are not always valid UTF-8.

// TextDecoder turns bytes back into a string. 'utf-8' is the default encoding.
const utf8 = new TextDecoder();
console.log(utf8.decode(Uint8Array.fromHex('e0a4a8e0a4ae'))); // नम

// Bytes that are not valid UTF-8 become U+FFFD (�), unless fatal: true asks for an error.
const damaged = Uint8Array.fromHex('4142ff43'); // A, B, an invalid byte, C
console.log(JSON.stringify(utf8.decode(damaged)));
try {
  new TextDecoder('utf-8', { fatal: true }).decode(damaged);
} catch (e) {
  console.log('fatal:', e.name);
}

// Text read in pieces can split a character. stream: true keeps the unfinished bytes for the next call.
const pieces = [Uint8Array.fromHex('e0a4'), Uint8Array.fromHex('a8e0a4ae')]; // न is cut after its second byte
console.log('piece by piece:', JSON.stringify(pieces.map((p) => new TextDecoder().decode(p)).join('')));
const streaming = new TextDecoder();
console.log('streaming:', JSON.stringify(pieces.map((p, i) => streaming.decode(p, { stream: i < pieces.length - 1 })).join('')));

// TextDecoder also reads older encodings by their labels.
console.log(new TextDecoder('windows-1252').decode(Uint8Array.fromHex('93636166e994'))); // a file saved by old Windows software
console.log(new TextDecoder('utf-16le').decode(Uint8Array.fromHex('28092e09'))); // UTF-16, little-endian

नम
"AB�C"
fatal: TypeError
piece by piece: "��म"
streaming: "नम"
“café”
नम

By default an invalid byte becomes U+FFFD, shown as �, so the rest of the text still reads. 0xFF can never appear in UTF-8, which is why it is replaced. Where a damaged file must be rejected rather than repaired, { fatal: true } makes decode throw a TypeError instead. Text that arrives in pieces needs { stream: true }: here the bytes of न are split between two pieces, and decoding each piece on its own turns both halves into replacement characters. A streaming decoder keeps the unfinished bytes and completes the character with the next piece; the last call, without stream, flushes it.

TextDecoder also reads the older encodings named in the Encoding Standard, such as windows-1252, shift_jis or utf-16le, by their labels. The labels have surprises: latin1 and iso-8859-1 both decode as windows-1252, which gives printable characters such as € and the curly quotes to most of the bytes from 0x80 to 0x9F, where ISO-8859-1 has control codes. A label the standard does not know throws a RangeError. A UTF-8 byte order mark at the start is removed, unless you pass ignoreBOM: true.

Base64: bytes as safe text

Many channels carry text but not arbitrary bytes: JSON strings, URLs, HTTP headers, email bodies. Base64 maps every group of 3 bytes, 24 bits, onto 4 characters of 6 bits each, chosen from an alphabet of 64 plain ASCII characters: A to Z, a to z, 0 to 9, + and /.

// Base64 by hand: every 3 bytes (24 bits) become 4 characters of 6 bits each.
const ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';

function explain(bytes) {
  const bits = bytes.map((b) => b.toString(2).padStart(8, '0')).join('');
  const sixes = bits.padEnd(Math.ceil(bits.length / 6) * 6, '0').match(/.{6}/g);
  const chars = sixes.map((six) => ALPHABET[parseInt(six, 2)]).join('');
  return `${bytes.join(' ').padEnd(11)} ${bits.padEnd(24)} → ${sixes.join(' ').padEnd(27)} → ${chars.padEnd(4, '=')}`;
}

console.log(explain([224, 164, 168])); // the three UTF-8 bytes of न
console.log(explain([77, 97, 110])); // 'Man'
console.log(explain([77, 97])); // two bytes: one = of padding
console.log(explain([77])); // one byte: two = of padding

// The size rule: 4 characters for every 3 bytes, rounded up.
const grouped = (n) => String(n).replace(/\B(?=(\d{3})+$)/g, ','); // 1000000 → 1,000,000
for (const n of [1, 2, 3, 10, 1_000_000]) {
  console.log(`${grouped(n)} ${n === 1 ? "byte" : "bytes"} → ${grouped(4 * Math.ceil(n / 3))} characters`);
}

224 164 168 111000001010010010101000 → 111000 001010 010010 101000 → 4KSo
77 97 110   010011010110000101101110 → 010011 010110 000101 101110 → TWFu
77 97       0100110101100001         → 010011 010110 000100        → TWE=
77          01001101                 → 010011 010000               → TQ==
1 byte → 4 characters
2 bytes → 4 characters
3 bytes → 4 characters
10 bytes → 16 characters
1,000,000 bytes → 1,333,336 characters

When the number of bytes is not a multiple of three, the last group is padded with zero bits, and = characters stand in for the missing characters: one = after two bytes, two after one byte. That is why every Base64 text of standard form has a length divisible by four, and why encoding makes data about a third larger: 1,000,000 bytes become 1,333,336 characters. Base64 is an encoding, not encryption. Anyone can decode it, so it hides nothing, and a password or token written in Base64 is as exposed as the plain text.

The variant called base64url replaces + and / with - and _, because + and / mean something in a URL, and usually leaves out the = padding, which a URL would have to escape. RFC 4648 defines both alphabets, and the three parts of a JSON Web Token are base64url without padding, as RFC 7515 specifies.

Uint8Array.toBase64 and fromBase64

ECMAScript 2026 adds four methods to Uint8Array: toBase64() and toHex() on an array, and the static Uint8Array.fromBase64() and Uint8Array.fromHex() that build a new one.

// ES2026: Uint8Array has Base64 and hex methods of its own.
const text = 'नमस्ते';
const bytes = new TextEncoder().encode(text);

const b64 = bytes.toBase64();
console.log(`${text} → ${bytes.length} bytes → ${b64}`);
const back = new TextDecoder().decode(Uint8Array.fromBase64(b64));
console.log(`${b64} → ${back} (same text: ${back === text})`);

// The URL-safe alphabet uses - and _ instead of + and /; tokens and URLs usually leave out the = padding.
const id = new Uint8Array([251, 255, 191, 251, 255]);
console.log(id.toBase64());
console.log(id.toBase64({ alphabet: 'base64url' }));
console.log(id.toBase64({ alphabet: 'base64url', omitPadding: true }));
console.log([...Uint8Array.fromBase64('-_-_-_8', { alphabet: 'base64url' })].join(' '));

// Hex works the same way.
console.log(bytes.toHex(), [...Uint8Array.fromHex('cafe')].join(' '));

// Invalid input throws a SyntaxError instead of guessing.
for (const bad of ['-_8', 'abc$', 'TWFuA']) {
  try {
    Uint8Array.fromBase64(bad);
  } catch (e) {
    console.log(`${JSON.stringify(bad)}: ${e.name}`);
  }
}

नमस्ते → 18 bytes → 4KSo4KSu4KS44KWN4KSk4KWH
4KSo4KSu4KS44KWN4KSk4KWH → नमस्ते (same text: true)
+/+/+/8=
-_-_-_8=
-_-_-_8
251 255 191 251 255
e0a4a8e0a4aee0a4b8e0a58de0a4a4e0a587 202 254
"-_8": SyntaxError
"abc$": SyntaxError
"TWFuA": SyntaxError

The text goes through UTF-8 first: TextEncoder makes the 18 bytes of नमस्ते, and toBase64 encodes the bytes, never the string itself. The options cover the common variants. alphabet: 'base64url' switches to - and _, and omitPadding: true drops the = signs. On the way back, fromBase64 takes the same alphabet option, and lastChunkHandling decides what happens with an incomplete last group: 'loose', the default, accepts it with or without padding, 'strict' demands correct padding and zero bits, and 'stop-before-partial' decodes only the complete groups. Spaces and line breaks in the input are skipped. Anything else that is not valid input throws a SyntaxError rather than producing garbage: here, characters of the other alphabet, a stray $ and a single character left over at the end. To fill an existing array instead of creating one, setFromBase64 and setFromHex write into it and return how many characters they read and how many bytes they wrote.

Why btoa fails on most text

btoa and atob date from the early browsers, and they work on “binary strings”: strings in which every character stands for one byte and therefore must be between U+0000 and U+00FF. The HTML standard specifies that btoa throws an InvalidCharacterError for anything else.

// btoa and atob work on "binary strings": every character must be below U+0100 and stands for one byte.
console.log(btoa('hello'), atob('aGVsbG8='));

try {
  btoa('नमस्ते');
} catch (e) {
  console.log(e.name); // न is U+0928: it is not a byte
}

// Worse, text with characters below U+0100 is accepted and encoded as Latin-1, not UTF-8.
console.log(`btoa('café') = ${btoa('café')}, the UTF-8 bytes of café = ${new TextEncoder().encode('café').toBase64()}`);

// The fallback for runtimes without toBase64: UTF-8 bytes → one character per byte → btoa.
function toBase64Fallback(bytes) {
  let binary = '';
  for (const b of bytes) binary += String.fromCharCode(b); // a loop, because spreading a large array overflows the call stack
  return btoa(binary);
}
function fromBase64Fallback(b64) {
  const binary = atob(b64);
  return Uint8Array.from(binary, (ch) => ch.charCodeAt(0));
}

const bytes = new TextEncoder().encode('नमस्ते');
console.log(toBase64Fallback(bytes), toBase64Fallback(bytes) === bytes.toBase64());
console.log(new TextDecoder().decode(fromBase64Fallback('4KSo4KSu4KS44KWN4KSk4KWH')));

aGVsbG8= hello
InvalidCharacterError
btoa('café') = Y2Fm6Q==, the UTF-8 bytes of café = Y2Fmw6k=
4KSo4KSu4KS44KWN4KSk4KWH true
नमस्ते

btoa('नमस्ते') throws, because न is U+0928, far beyond one byte. The quieter failure is worse: btoa('café') works, but it encodes é as the single Latin-1 byte E9, so the result, Y2Fm6Q==, is not the Base64 of the UTF-8 text a server expects, Y2Fmw6k=. The cure in both cases is the same: encode the text to UTF-8 bytes first, then encode the bytes. Where toBase64 is missing, the bytes can be turned into a binary string, one character per byte, and passed to btoa; atob and charCodeAt reverse it. Build the binary string in a loop: String.fromCharCode(...bytes) looks shorter, but it passes every byte as a separate argument and throws a RangeError for large arrays; in Node.js 26 it already fails at 200,000 bytes. Node.js marks btoa and atob as legacy and recommends Buffer in Node code: Buffer.from(bytes) has toString('base64') and toString('base64url'), and Buffer.from(text, 'base64') decodes.

Exercise

/** The bytes as base64url text (alphabet with - and _, no = padding). */
export function toBase64Url(bytes) {
  // Replace this with your code.
  return '';
}

/** The bytes of base64url text, with or without = padding; a SyntaxError for anything else. */
export function fromBase64Url(text) {
  // Replace this with your code.
  return new Uint8Array(0);
}

import { test, assert } from 'mysmartcopilot:test';
import { fromBase64Url, toBase64Url } from './base64url.mjs';

const bytesOf = (...values) => new Uint8Array(values);

test('toBase64Url uses - and _ and leaves out the padding', () => {
  assert.equal(toBase64Url(bytesOf(251, 255)), '-_8');
  assert.equal(toBase64Url(bytesOf(251, 255, 191)), '-_-_');
});

test('toBase64Url: groups of three bytes, and the shorter last groups', () => {
  assert.equal(toBase64Url(bytesOf(77, 97, 110)), 'TWFu');
  assert.equal(toBase64Url(bytesOf(77, 97)), 'TWE');
  assert.equal(toBase64Url(bytesOf(77)), 'TQ');
  assert.equal(toBase64Url(bytesOf()), '');
  assert.equal(toBase64Url(bytesOf(224, 164, 168)), '4KSo');
});

test('fromBase64Url reads text with and without padding', () => {
  assert.deepEqual([...fromBase64Url('-_8')], [251, 255]);
  assert.deepEqual([...fromBase64Url('TWE')], [77, 97]);
  assert.deepEqual([...fromBase64Url('TWE=')], [77, 97]);
  assert.deepEqual([...fromBase64Url('')], []);
  assert.ok(fromBase64Url('TWFu') instanceof Uint8Array, 'returns a Uint8Array');
});

test('every byte value survives the round trip', () => {
  const all = Uint8Array.from({ length: 256 }, (_, i) => i);
  assert.deepEqual([...fromBase64Url(toBase64Url(all))], [...all]);
});

test('fromBase64Url throws a SyntaxError for characters outside the alphabet and impossible lengths', () => {
  for (const bad of ['+/8', 'TW E', 'TW*u', 'TWFuA']) assert.throws(() => fromBase64Url(bad), SyntaxError, JSON.stringify(bad));
});

Interview questions

Why does btoa(‘नमस्ते’) throw, and how do you Base64-encode any text correctly? btoa encodes a “binary string”, in which every character stands for one byte, so it accepts only characters from U+0000 to U+00FF and throws an InvalidCharacterError for anything else; न is U+0928. Base64 encodes bytes, not characters, so the text has to be turned into bytes first, almost always UTF-8, with new TextEncoder().encode(text). The bytes then go to bytes.toBase64() where ECMAScript 2026 is available, or, as a fallback, into a binary string built one byte per character in a loop and passed to btoa. Decoding reverses the steps: Uint8Array.fromBase64 or atob gives the bytes, and new TextDecoder().decode(bytes) the text. Calling btoa directly on text with é in it does not throw, but it silently produces the Base64 of Latin-1 bytes rather than of UTF-8.

Key takeaways

• A string’s length counts UTF-16 code units; UTF-8 needs 1 to 4 bytes per code point, so नमस्ते is 6 code units and 18 bytes.
• new TextEncoder().encode(text) always gives UTF-8 bytes; new TextDecoder(label, options).decode(bytes) reads UTF-8 or an older encoding, with fatal to reject bad bytes and stream for text in pieces.
• Base64 writes 3 bytes as 4 characters from a 64-character alphabet, with = padding; it is about a third larger and it is not encryption.
• base64url uses - and _ instead of + and / and usually has no padding.
• In ECMAScript 2026, bytes.toBase64(), Uint8Array.fromBase64(), toHex() and fromHex() do the work, with options for the alphabet, the padding and the last group, and a SyntaxError for bad input.
• btoa and atob only handle characters up to U+00FF: encode text to UTF-8 bytes first, and keep them for old runtimes only.