Regular expressions in JavaScript

A regular expression, or regex, is a short pattern that describes text: /^\d{6}$/ means “six digits and nothing else”. In JavaScript a regex is a RegExp object. You write it as a literal between two slashes, with optional flags such as g or i after the closing slash, or you build it while the program runs with new RegExp(pattern, flags). Two groups of methods use it: the regex’s own test and exec, and the string methods match, matchAll, replace, replaceAll, split and search. Together they check that input has the right shape, pull fields out of a log line, rewrite text and cut it into pieces. This lesson shows how to create a regex, what each flag changes, which method fits which job, and the lastIndex trap: a regex with the g or y flag remembers where its last match ended, so the same call with the same kind of input can answer differently the second time.

A literal or the RegExp constructor

A literal is the pattern itself between slashes, /\d{6}/. It is read together with the rest of the code, so a broken pattern, such as a bracket that is never closed, stops the script with a SyntaxError before any line of it runs. Each time the literal is evaluated, for example on every call of a function that contains it, it creates a new RegExp object.

new RegExp(pattern, flags) takes the pattern as a string. That is what you need when part of the pattern is known only at run time: a field name read from a configuration file, a word someone searches for. The cost of the string is the backslash. In a string literal '\d' is simply the letter d, so the backslash that belongs to the pattern has to be written twice, '\\d'.

// The same pattern written two ways: six digits in a row.
const literal = /\d{6}/;
const built = new RegExp('\\d{6}'); // inside a string, the pattern's backslash is written twice

console.log(literal.source, '|', built.source);
console.log(literal.test('Pune 411001'), built.test('Pune 411001'));

// A pattern that is only known when the program runs: build it from a string.
const field = 'pin';
const fieldLine = new RegExp(`^${field}=(\\d+)$`, 'm');
console.log(fieldLine.source, '| flags:', fieldLine.flags);
console.log(fieldLine.exec('name=Asha\npin=411001\ncity=Pune')[1]);

// A copy with other flags: pass a regex and the new flags. The original keeps its own.
const everyMatch = new RegExp(literal, 'g');
console.log(everyMatch.source, '| flags:', everyMatch.flags, '| original flags:', JSON.stringify(literal.flags));

// The usual mistake: '\d' in a string is just 'd', so this pattern looks for six letters d.
const wrong = new RegExp('\d{6}');
console.log(wrong.source, wrong.test('411001'), wrong.test('dddddd'));

\d{6} | \d{6}
true true
^pin=(\d+)$ | flags: m
411001
\d{6} | flags: g | original flags: ""
d{6} false true

Both ways produce the same kind of object: source holds the pattern and flags the flags. Passing an existing regex and new flags to the constructor makes a copy with those flags, and the original keeps its own. The last line is the mistake almost everyone makes once: new RegExp('\d{6}') received the string d{6}, so it rejects six digits and accepts six letters d, without any error. A template literal is a handy way to assemble a pattern from pieces, but a piece that comes from outside the program, such as a search term, must be escaped first, or its ., ( and + turn into pattern syntax. RegExp.escape, which does that safely, is in the next lesson of this module.

The pattern language, briefly

The building blocks below cover most patterns in everyday code. Characters without a special meaning match themselves, and the rest combine freely.

Lookahead and lookbehind, Unicode property escapes such as \p{L} and the syntax added in the newest editions are covered in the next lesson.

Flags change how the whole pattern matches

Flags come after the closing slash, /x/gi, or as the second argument of the constructor. Each one has a read-only property with a longer name.

• d (hasIndices): each result also gives the start and end of every group, in indices.
• g (global): look for every match; test and exec start where the last match ended.
• i (ignoreCase): capital and small letters match each other.
• m (multiline): ^ and $ match at the start and end of every line, not only of the text.
• s (dotAll): . matches line breaks as well.
• u (unicode): the text is read by code point, so an emoji is one character; the syntax is stricter, and \p{…} escapes become available.
• v (unicodeSets): everything u does, plus set operations inside character classes.
• y (sticky): a match must start exactly at lastIndex.

const notes = 'Line one\nline two\nLINE THREE';

// i ignores case; m makes ^ and $ match at every line; g finds every match.
console.log(JSON.stringify(notes.match(/^line \w+/gim)));
console.log(JSON.stringify(notes.match(/^line \w+/gi)));

// s lets . match a line break too.
console.log(/one.line/.test(notes), /one.line/s.test(notes));

// u reads the pattern and the text by code point, so one emoji is one character.
console.log('🚆'.length, /^.$/.test('🚆'), /^.$/u.test('🚆'));

// The flags property lists the flags in a fixed order, whatever order you wrote them in.
const re = new RegExp('x', 'yigm');
console.log(re.flags, re.global, re.ignoreCase, re.multiline, re.sticky);

// Unknown, repeated or clashing flags are refused when the regex is made.
for (const flags of ['gg', 'q', 'uv']) {
  try {
    new RegExp('x', flags);
  } catch (e) {
    console.log(JSON.stringify(flags), e.name);
  }
}

["Line one","line two","LINE THREE"]
["Line one"]
false true
2 false true
gimy true true true true
"gg" SyntaxError
"q" SyntaxError
"uv" SyntaxError

With i and m together, ^line \w+ finds a match at the start of each of the three lines whatever their case; without m, ^ means only the start of the whole text, so only the first line matches. A dot refuses the line break between “one” and “line” until the s flag allows it. The train emoji is two UTF-16 code units, so /^.$/ sees two characters, while the u flag reads it as the one code point it is. Whatever order you type the flags in, flags lists them in one fixed order, dgimsuvy. A repeated flag, an unknown one, or u together with v is refused with a SyntaxError when the regex is created.

Which method for which job

The regex methods and the string methods overlap. Pick by what you need back:

search and split ignore the g flag and lastIndex, and leave them as they found them; the others are covered one by one below.

Checking that a whole string has a shape

test answers yes or no. To check a value rather than search inside it, the pattern must be anchored: ^ at the start and $ at the end, so that nothing may come before or after the shape.

// The shape of an IFSC: 4 letters for the bank, the digit 0, then 6 letters or digits for the branch.
const IFSC = /^[A-Z]{4}0[A-Z0-9]{6}$/;

function ifscShape(input) {
  const code = input.trim().toUpperCase();
  return IFSC.test(code) ? `${code} has the shape of an IFSC` : `${JSON.stringify(input)} is not an IFSC`;
}

for (const input of ['ABCD0123456', ' wxyz0a1b2c3 ', 'ABCD0I23456', 'ABCDO123456', 'ABCD012345', 'ABCD01234567', 'AB1D0123456']) {
  console.log(ifscShape(input));
}

// Without ^ and $, test() is true when the shape appears anywhere in the text.
const unanchored = /[A-Z]{4}0[A-Z0-9]{6}/;
console.log(unanchored.test('XABCD0123456789'), IFSC.test('XABCD0123456789'));

ABCD0123456 has the shape of an IFSC
WXYZ0A1B2C3 has the shape of an IFSC
ABCD0I23456 has the shape of an IFSC
"ABCDO123456" is not an IFSC
"ABCD012345" is not an IFSC
"ABCD01234567" is not an IFSC
"AB1D0123456" is not an IFSC
true false

The function cleans the input before testing it: trim() removes the spaces a form often adds, and toUpperCase() lets someone type the code in small letters while the pattern stays strict about the stored value. Two lines of the output deserve a second look. ABCD0I23456 passes, because a capital I is a valid branch character: a regex checks the shape of a value, never whether a branch with that code exists. ABCDO123456, with the letter O in place of the zero, is rejected, which is exactly what the fifth position is for. The last line shows why the anchors matter: without them, test is happy to find a valid-looking code inside a longer, invalid string.

Getting the parts: exec and match

exec returns the first match as an array: the whole matched text at index 0, then one entry per numbered group, plus a few properties. index is where the match starts, input is the text that was searched, and groups is an object with one property per named group (it is undefined when the pattern has no named groups). When nothing matches, the result is null.

const sms = 'Order 4521: 2 x masala chai, ready. Order 4522: 1 x samosa, on its way.';
const ORDER = /Order (?<id>\d+): (?<qty>\d+) x (?<item>[a-z ]+)/;

// exec returns the first match with its groups and position, or null.
const m = ORDER.exec(sms);
console.log(m[0]);
console.log(m[1], m.groups.item, 'at index', m.index);

// match without the g flag returns the same kind of result as exec.
const { id, qty } = sms.match(ORDER).groups;
console.log(id, Number(qty) * 20);

// With the g flag, match returns only the matched texts: no groups, no positions.
console.log(JSON.stringify(sms.match(/Order \d+/g)));

// No match gives null, so read .length or a group only after checking.
console.log(sms.match(/refund/));
console.log((sms.match(/refund/g) ?? []).length);

Order 4521: 2 x masala chai
4521 masala chai at index 0
4521 40
["Order 4521","Order 4522"]
null
0

Without the g flag, text.match(re) gives back the same kind of result as re.exec(text), which makes destructuring the groups convenient. With g, match changes its answer: an array of the matched texts only, with no groups and no positions. In both forms a text with no match gives null, not an empty array, so code such as text.match(re).length throws a TypeError on the first message that does not match; ?? [] turns the null into an empty array.

Every match with its groups: matchAll

matchAll returns an iterator that produces one full match array per match, each with its groups and index. It is the method for reading many records out of one text, such as the lines of a log.

const log = `GET /api/trains/12951 200 38ms
POST /api/bookings 201 120ms
GET /api/pnr/4521 404 12ms
GET /api/trains/12002 200 41ms
POST /api/bookings 500 1503ms
GET /health 200 2ms`;

// One named group per field; m makes ^ and $ work per line, g lets matchAll find every line.
const LINE = /^(?<method>GET|POST) (?<path>\S+) (?<status>\d{3}) (?<ms>\d+)ms$/gm;

const perStatus = new Map();
const slow = [];
for (const { groups, index } of log.matchAll(LINE)) {
  perStatus.set(groups.status, (perStatus.get(groups.status) ?? 0) + 1);
  if (Number(groups.ms) >= 100) slow.push(`${groups.method} ${groups.path}, ${groups.ms} ms, at character ${index}`);
}
console.log([...perStatus].map(([status, n]) => `${status}: ${n}`).join(', '));
console.log(slow.join('\n'));

// matchAll works on a copy of the regex, so LINE.lastIndex is still 0 afterwards.
console.log('lastIndex', LINE.lastIndex);

// Without the g flag, matchAll refuses to start.
try {
  log.matchAll(/GET/);
} catch (e) {
  console.log(e.name);
}

200: 3, 201: 1, 404: 1, 500: 1
POST /api/bookings, 120 ms, at character 31
POST /api/bookings, 1503 ms, at character 118
lastIndex 0
TypeError

The pattern has the m flag so that ^ and $ hold at each line, and the g flag because matchAll refuses to start without it: it throws a TypeError rather than quietly return a single match. The loop destructures groups and index straight from each match. matchAll copies the regex before it starts, so the lastIndex of LINE is still 0 afterwards and the same regex can be used again safely. The iterator is lazy and finds the next match only when the loop asks for it; [...text.matchAll(re)] turns it into an array when you need all of them at once.

Replacing text

replace returns a new string and leaves the original as it was. What it replaces depends on the pattern: a string pattern replaces its first occurrence only, a regex without g its first match, and a regex with g every match. replaceAll replaces every occurrence of a string, and with a regex it insists on the g flag.

const note = 'Platform 4 for train 12951, platform 4 again for train 12953';

// A string pattern replaces the first occurrence only; replaceAll replaces every one.
console.log(note.replace('4', 'four'));
console.log(note.replaceAll('4', 'four'));

// A regex without g replaces the first match; with g, every match.
console.log(note.replace(/platform/i, 'Track'));
console.log(note.replace(/platform/gi, 'Track'));

// In the replacement text, $& is the whole match and $<name> a named group.
console.log(note.replace(/\b\d{5}\b/g, '[$&]'));
console.log('Rao, Asha'.replace(/^(?<last>\w+), (?<first>\w+)$/, '$<first> $<last>'));

// A function receives the match and its groups and returns the replacement.
const camel = 'seat_number_and_coach'.replace(/_([a-z])/g, (match, letter) => letter.toUpperCase());
console.log(camel);
const masked = 'Account 123456789012, IFSC ABCD0123456'.replace(/\b\d{8,}\b/g, (digits) => '•'.repeat(digits.length - 4) + digits.slice(-4));
console.log(masked);

// replaceAll with a regex demands the g flag.
try {
  note.replaceAll(/platform/i, 'Track');
} catch (e) {
  console.log(e.name);
}

Platform four for train 12951, platform 4 again for train 12953
Platform four for train 12951, platform four again for train 12953
Track 4 for train 12951, platform 4 again for train 12953
Track 4 for train 12951, Track 4 again for train 12953
Platform 4 for train [12951], platform 4 again for train [12953]
Asha Rao
seatNumberAndCoach
Account ••••••••9012, IFSC ABCD0123456
TypeError

The replacement text may refer to the match:

When the new text needs logic, pass a function instead of a string. It is called once per match with the matched text, then the text of each numbered group, the position and the whole string (and an object of named groups, when the pattern has any); whatever it returns is inserted. The example converts snake_case to camelCase that way and masks an account number so that only its last four digits remain readable.

Splitting with a regex

split accepts a regex as the separator, which handles separators that vary, such as a comma or a semicolon with any number of spaces around it.

// Split on a run of separators with optional spaces around them.
console.log(JSON.stringify('chai, samosa;vada pav ,  idli'.split(/\s*[,;]\s*/)));

// A capture group in the separator puts the separators into the result too.
console.log(JSON.stringify('12+30-5'.split(/([+-])/)));

// A separator at the start gives an empty first piece.
console.log(JSON.stringify(',a,,b'.split(/,/)));

// The second argument limits how many pieces come back.
console.log(JSON.stringify('a1b2c3d'.split(/\d/, 2)));

["chai","samosa","vada pav","idli"]
["12","+","30","-","5"]
["","a","","b"]
["a","b"]

A capture group in the separator puts the separators themselves into the result, between the pieces, which is a quick way to keep the operators of a sum. A separator at the very start produces an empty first piece, and two separators in a row produce an empty piece between them. The second argument caps the number of pieces returned.

The lastIndex trap

A regex with the g flag or the y flag is stateful. test and exec start reading at the regex’s lastIndex property, set it to the end of the match when they find one, and put it back to 0 when they find none. That is what lets a loop of exec calls walk through a text match by match. It is also a classic bug when one global regex is shared and used with test on different strings.

// The g flag makes a regex remember where its last match ended (lastIndex).
const DIGITS = /^\d+$/g;

for (const code of ['411001', '560001', '110001', '600001']) {
  const result = DIGITS.test(code);
  console.log(code, result, 'lastIndex is now', DIGITS.lastIndex);
}

// The same bug inside filter: half of the valid codes are dropped.
const codes = ['411001', '560001', '110001', '600001'];
console.log(codes.filter((c) => DIGITS.test(c)).length, 'of', codes.length, 'kept');

// Fix 1: a regex that is only used with test() needs no g flag.
const ALL_DIGITS = /^\d+$/;
console.log(codes.filter((c) => ALL_DIGITS.test(c)).length, 'of', codes.length, 'kept');

// Fix 2: when a g regex has to be reused, set lastIndex back to 0 before every call.
const again = codes.map((c) => {
  DIGITS.lastIndex = 0;
  return DIGITS.test(c);
});
console.log(again.join(' '));

411001 true lastIndex is now 6
560001 false lastIndex is now 0
110001 true lastIndex is now 6
600001 false lastIndex is now 0
2 of 4 kept
4 of 4 kept
true true true true

The second call starts at index 6, which is already the end of the next six-digit code, so it fails and resets lastIndex to 0; the third call succeeds again. Inside filter, that drops half of the valid codes, and the bug depends on what was checked before, which makes it hard to reproduce. The fixes, in order of preference:

• Leave out g when a regex is only used with test or for a single exec. It brings nothing there.
• Create the regex inside the function that uses it: a literal makes a new object, with lastIndex 0, every time it is evaluated.
• When one global regex has to be reused, set lastIndex = 0 before each call.
• Prefer match, matchAll, replace and split for global work. match and replace start from 0 and leave lastIndex at 0, matchAll works on a copy, and split and search ignore it.

The y flag uses the same property on purpose. A sticky regex matches only if the match starts exactly at lastIndex, never further along, which is what a tokenizer needs: every character must belong to a token, and an unknown one must stop it rather than be skipped.

// y (sticky): every match must start exactly at lastIndex, so nothing can be skipped.
const TOKEN = /\s*(?:(?<number>\d+(?:\.\d+)?)|(?<op>[-+*/()]))\s*/y;

function tokenize(source) {
  const tokens = [];
  TOKEN.lastIndex = 0;
  while (TOKEN.lastIndex < source.length) {
    const at = TOKEN.lastIndex;
    const m = TOKEN.exec(source);
    if (m === null) throw new SyntaxError(`unexpected ${JSON.stringify(source[at])} at position ${at}`);
    tokens.push(m.groups.number ?? m.groups.op);
  }
  return tokens;
}

console.log(tokenize('12 + 3.5*(4 - 1)').join(' | '));
console.log(tokenize(' 7/2 ').join(' | '));
try {
  tokenize('2 + x * 3');
} catch (e) {
  console.log(`${e.name}: ${e.message}`);
}

// With g instead of y, exec searches on from lastIndex: it jumps over the x and finds the 3.
const searching = /\d+/g;
searching.lastIndex = 4;
console.log(searching.exec('2 + x * 3')[0], 'found at', searching.lastIndex - 1);

12 | + | 3.5 | * | ( | 4 | - | 1 | )
7 | / | 2
SyntaxError: unexpected "x" at position 4
3 found at 8

Each exec call reads one token where the previous one ended, so the loop stops with a precise error at the x. The last two lines show the difference the flag makes: with g, exec searches onward from lastIndex and silently passes over the x to find the 3.

Exercise

extractAmounts('Paid ₹1,234.50 to Asha and ₹99 to Ravi')  →  [123450, 9900]
extractAmounts('Refund of Rs. 12,34,567.5 approved')       →  [123456750]
extractAmounts('No amounts here, only 42 and ₹ alone')     →  []

/** Every rupee amount in the text, in paise (100 paise = 1 rupee), in the order they appear. */
export function extractAmounts(text) {
  // Replace these lines with your code. This version stops reading each amount at its first comma.
  const found = text.match(/₹\s?[\d,.]+/g) ?? [];
  return found.map((amount) => Math.round(parseFloat(amount.slice(1)) * 100));
}

import { test, assert } from 'mysmartcopilot:test';
import { extractAmounts } from './amounts.mjs';

test('the first example of the prompt', () => {
  assert.deepEqual(extractAmounts('Paid ₹1,234.50 to Asha and ₹99 to Ravi'), [123450, 9900]);
});

test('Indian digit grouping and one digit of paise', () => {
  assert.deepEqual(extractAmounts('Refund of Rs. 12,34,567.5 approved'), [123456750]);
});

test('Rs in any case, with or without a full stop and a space', () => {
  assert.deepEqual(extractAmounts('rs 40 for tea, RS.15 for biscuits, Rs.7.25 for the bus'), [4000, 1500, 725]);
});

test('numbers without a currency mark, and a mark without digits, are not amounts', () => {
  assert.deepEqual(extractAmounts('No amounts here, only 42 and ₹ alone'), []);
  assert.deepEqual(extractAmounts(''), []);
});

test('the same result every time the function is called', () => {
  const text = '₹10 now and ₹20 later';
  assert.deepEqual(extractAmounts(text), [1000, 2000]);
  assert.deepEqual(extractAmounts(text), [1000, 2000]);
});

Interview questions

Why does calling test() twice on a regex with the g flag return true and then false? With the g or y flag a regex keeps state in its lastIndex property. test and exec start reading at lastIndex, and after a match they set it to the position where the match ended. The second call therefore starts part of the way into the next string, or past its end, finds nothing, returns false and sets lastIndex back to 0, so a third call succeeds again. A regex that is only used with test should not have the g flag. If a global regex must be shared, reset lastIndex to 0 before each call, or create the regex inside the function that uses it. match, matchAll, replace and split do not have this problem.

Key takeaways

• Write a regex as a literal when the pattern is fixed, and with new RegExp(string, flags) when it is built at run time; in a string, the pattern’s backslashes are written twice.
• Flags change the whole match: i for case, m for line anchors, s for dots across lines, u or v for Unicode, d for group positions, and g or y for searching on from lastIndex.
• Anchor a validation pattern with ^ and $, and remember that a regex checks a shape, not that the value exists.
• exec and match without g give the first match with its groups; match with g gives only texts; matchAll gives every match with its groups.
• replace with g, or replaceAll, changes every match; $&, $1 and $<name> reuse the match, and a function can compute each replacement.
• A shared regex with g or y carries lastIndex from call to call: leave the flag out for test, create the regex where it is used, or reset lastIndex.