Regex in PCRE2, PHP and Perl

PHP 8.5.11 runs its patterns through PCRE2 10.44, the pcre2test and pcre2grep programs on the same Mac are PCRE2 10.49, and Perl runs an engine of its own that PCRE2 imitates. Three front doors lead to nearly one dialectflavour: A particular regular-expression syntax and feature set, such as ECMAScript, Python re, PCRE2 or RE2; a pattern is written for one of them. More in “What a regular expression is and where it runs”, and each has its own way to write a pattern, to set its options and to build a replacement: a missing modifier makes \d skip the Devanagari digit ७, and one stray character turns a call into false.

This lesson goes through the doors in turn: PHP delimiters and modifiers, how to read a failed preg_* call, the replacements that only some doors can build, and the two shell programs that show what the library is doing. By the end you can write a PHP pattern with the right delimiter and letters, diagnose a false, rewrite a replacement for each door, and read a match limit.

One dialect, three front doors

The previous lesson compared flavours across many languages; this one stays in one family and looks at its doors. Start with what is installed. The run below prints the version of PHP and the version of the PCRE2 library inside it:

<?php
// Two version numbers: the PHP that runs the script and the PCRE2 library inside it.
// PCRE_VERSION also carries a release date after the number; only the number is printed.
echo "PHP_VERSION  ", PHP_VERSION, "\n";
echo "PCRE2 library ", explode(' ', PCRE_VERSION)[0], "\n";

PHP_VERSION  8.5.11
PCRE2 library 10.44

The library inside PHP 8.5.11 is 10.44, and the first line of every pcre2test transcript later in this lesson says 10.49. They are two releases of one library, so a feature added in between exists at one door and not at the other. The PHP manual calls the dialect compatible with Perl’s, and the pcre2pattern manual says PCRE2 tries to match Perl’s syntax and semantics as closely as it can. Perl itself is the odd door out: it keeps its own engine, so a pattern that works there is a close relative of one that works in PHP, not the same program.

The figure follows a pattern through the four stages PCRE2 gives every user and marks where each door takes hold of it. Step through it before reading the table:

A PHP pattern is a string: delimiter first, modifiers last

A PHP pattern is an ordinary string, not a literal the language parses, so PHP finds the pattern’s edges from its first character. That character is the delimiterpattern delimiter: The character that opens and closes a pattern written as a string, as in PHP's preg functions. It is the string's first character and must end the body; modifier letters follow the closing one.: the manual allows any single-byte character that is not alphanumeric, not a backslash, not NUL and not white space, and the same character must end the body. The bracket pairs (), {}, [] and <> work too, with the closing bracket ending the body. Three spellings of one path test follow, and a fourth string whose first character was meant as an anchor:

<?php
// The first character of a PHP pattern string is its delimiter. Three spellings of the same path test,
// and one string whose first character was meant as an anchor.
$path = '/usr/local/share';
$patterns = [
    '/^\/usr\/local\//',   // slash delimiter: every slash in the body is escaped
    '#^/usr/local/#',      // hash delimiter: nothing to escape
    '{^/usr/local/}',      // bracket delimiters: the pair { }
    '^/usr/local/',        // no delimiter: PHP takes ^ as one and looks for a second ^
];
foreach ($patterns as $pattern) {
    $result = @preg_match($pattern, $path);
    printf("%-18s %s\n", $pattern, $result === false ? 'false: ' . error_get_last()['message'] : "returned $result");
}

/^\/usr\/local\//  returned 1
#^/usr/local/#     returned 1
{^/usr/local/}     returned 1
^/usr/local/       false: preg_match(): No ending delimiter '^' found

The slash version needs a backslash before every slash of the path; the hash and the braces need none. The last string has no closing ^, so PHP warns that it found no ending delimiter and the call returns false. Which delimiter to pick is a readability question only, since none of them changes what the body matches. Read a whole pattern from left to right in the next figure:

Everything after the closing delimiter is a modifierpattern modifier: A letter written after the closing delimiter of a pattern that sets an option for the whole pattern, such as i for case-insensitive matching, m for line anchors or u for UTF-8 and Unicode classes., one letter each. The eight that change results most often are in the table, and the run after it tries seven of them with and without the letter (u has a run of its own further down):

S is still accepted but has done nothing since PHP 7.3, and n (PHP 8.2) makes plain groups non-capturing.

<?php
// One modifier at a time: the same subject with and without the letter after the closing delimiter.
$rows = [
    ['i', '/error/',       '/error/i',       'ERROR: disk full'],
    ['m', '/^b$/',         '/^b$/m',         "a\nb\nc"],
    ['s', '/a.b/',         '/a.b/s',         "a\nb"],
    ['x', '/\d{3} \d{3}/', '/\d{3} \d{3}/x', '411001'],
    ['U', '/<.+>/',        '/<.+>/U',        '<b>bold</b>'],
    ['A', '/b/',           '/b/A',           'ab'],
    ['D', '/^a$/',         '/^a$/D',         "a\n"],
];
foreach ($rows as [$letter, $without, $with, $subject]) {
    echo $letter, ' on ', json_encode($subject, JSON_UNESCAPED_SLASHES), "\n";
    foreach ([$without, $with] as $pattern) {
        $found = preg_match($pattern, $subject, $m) ? json_encode($m[0], JSON_UNESCAPED_SLASHES) : 'no match';
        printf("    %-16s %s\n", $pattern, $found);
    }
}

i on "ERROR: disk full"
    /error/          no match
    /error/i         "ERROR"
m on "a\nb\nc"
    /^b$/            no match
    /^b$/m           "b"
s on "a\nb"
    /a.b/            no match
    /a.b/s           "a\nb"
x on "411001"
    /\d{3} \d{3}/    no match
    /\d{3} \d{3}/x   "411001"
U on "<b>bold</b>"
    /<.+>/           "<b>bold</b>"
    /<.+>/U          "<b>"
A on "ab"
    /b/              "b"
    /b/A             no match
D on "a\n"
    /^a$/            "a"
    /^a$/D           no match

Two rows deserve a second look. With U, which turns on ungreedy modeungreedy mode: An option of PCRE2, PHP (the U modifier) and Go, also set with (?U) in a pattern, that swaps the two forms: a quantifier without a question mark is lazy and one with it is greedy. JavaScript and Python have no such mode. More in “Greedy and lazy quantifiers”, /<.+>/ stops at the first > and returns <b>, the reverse of what a reader of the pattern expects. And D rejects "a\n" where plain /^a$/ accepts it, the trailing newline traptrailing-newline trap: A validation bug: a pattern such as ^abc$ accepts abc followed by a line feed in engines whose $ also holds before a final newline, so the checked value still carries the extra character. More in “Anchors: start, end and multiline mode”: a value that came from a form or a file keeps its final newline, and a bare $ lets it through.

The u modifier does two jobs, and the next run shows both. It makes \d take ७, as the shorthand classes lesson explains, and it makes PHP check that the text is valid UTF-8: a stray byte makes the call fail instead of matching bytes.

<?php
// The u modifier in PHP: UTF-8 text, Unicode properties for \d, and a check that the subject is valid UTF-8.
$text = "\u{96D}9"; // the Devanagari digit seven, then the ASCII digit 9
foreach (['/\d/', '/\d/u'] as $pattern) {
    preg_match_all($pattern, $text, $found);
    printf("%-6s %s\n", $pattern, json_encode($found[0], JSON_UNESCAPED_UNICODE));
}

// A subject that is not valid UTF-8 (one stray byte): the u pattern refuses to run.
$bytes = "ab\xFF";
foreach (['/./', '/./u'] as $pattern) {
    $count = preg_match_all($pattern, $bytes);
    printf("%-6s %s\n", $pattern, $count === false ? 'false: ' . preg_last_error_msg() : "$count matches");
}

/\d/   ["9"]
/\d/u  ["७","9"]
/./    3 matches
/./u   false: Malformed UTF-8 characters, possibly incorrectly encoded

A failed preg call returns false, not zero

preg_match has three answers: 1 for a match, 0 for none, and false when the call fails. Failure has two causes, and they report differently. A pattern that does not compile raises a PHP warning with the library’s own description, and a run that goes wrong afterwards, such as bad UTF-8 or too much work, is named by preg_last_error_msg(). The run shows a compile failure and why the return value must be tested strictly:

<?php
// A pattern that does not compile: preg_match() returns false, PHP raises a warning,
// and preg_last_error_msg() has only a general word for it.
$bad = @preg_match('/(/', 'x');
echo "warning: ", error_get_last()['message'], "\n";
echo "last error: ", preg_last_error(), ' = ', preg_last_error_msg(), "\n";

// A good pattern that finds nothing returns 0, which is not false.
$none = preg_match('/zz/', 'x');
var_dump($bad, $none);
var_dump($bad == $none);   // loose comparison: false == 0
var_dump($bad === $none);  // strict comparison tells them apart

foreach ([$bad, $none, preg_match('/x/', 'x')] as $result) {
    echo match (true) {
        $result === false => "error: the pattern or the run failed\n",
        $result === 0     => "no match\n",
        default           => "match\n",
    };
}

warning: preg_match(): Compilation failed: missing closing parenthesis at offset 1
last error: 1 = Internal error
bool(false)
int(0)
bool(true)
bool(false)
error: the pattern or the run failed
no match
match

The warning says missing closing parenthesis at offset 1, but preg_last_error_msg() says only Internal error, because the pattern never ran and so there is no run error to describe. The detail lives in the warning, which the @ here hid from the screen but error_get_last() still holds. The loose comparison false == 0 is true, so a check written as if (!$result) treats a broken pattern as an ordinary non-match. Step through the decision once:

A second kind of false comes from the run itself. PHP stops a match after pcre.backtrack_limit steps, set to 1,000,000 here. A pattern with a repeat inside a repeat can need far more than that on text that almost matches, and the call gives up with false. The limit is a safety net, not a setting to tune:

<?php
// The backtrack limit is a safety net: when a run exceeds it, preg_match() returns false
// and preg_last_error_msg() says why. The fix is the pattern, not a bigger limit.
echo "pcre.backtrack_limit = ", ini_get('pcre.backtrack_limit'), "\n";

$subject = str_repeat('a', 26) . '!';
foreach (['/^(a+)+$/' => 'nested repeat', '/^a+$/' => 'one repeat'] as $pattern => $label) {
    $result = preg_match($pattern, $subject);
    printf("%-12s %-14s %s\n", $pattern, $label, $result === false ? 'false: ' . preg_last_error_msg() : "returned $result");
}

pcre.backtrack_limit = 1000000
/^(a+)+$/    nested repeat  false: Backtrack limit exhausted
/^a+$/       one repeat     returned 0

The nested (a+)+ runs into the limit on 26 letters a followed by !, while the single repeat a+ returns 0 straight away. Both accept exactly the same strings, so the fix is the pattern: remove the repeat inside a repeat, and do not raise the limit. Why such patterns blow up, and how to test for them, belongs to the performance and security module later in this track.

Replacements: what only some doors can do

Take one job: turn hello world into World HELLO!, with the second word first and capitalised and the first word in capitals. PHP’s preg_replace reads only three things in a replacement: \n, $n and ${n}, where n is a group number. A group’s name is not one of them, and a digit after $1 is read as part of the number:

<?php
// preg_replace() reads \n, $n and ${n} in a replacement. Nothing else: a group name is not a reference.
echo preg_replace('/(?<a>\w+)/', '${a}!', 'hi'), "\n";
echo preg_replace('/(\w+)/', '$1!', 'hi'), "\n";
echo preg_replace('/(\w+)/', '${1}!', 'hi'), "\n";

// A digit after $1 is read as part of the group number: $10 is group 10, which does not exist.
echo '[', preg_replace('/(a)/', '$10', 'a'), "]\n";
echo '[', preg_replace('/(a)/', '${1}0', 'a'), "]\n";

// Names and case changes need a function: preg_replace_callback() hands over the groups by name.
echo preg_replace_callback('/(?<w>\w+)/', fn($m) => strtoupper($m['w']) . '!', 'hello world'), "\n";

${a}!
hi!
hi!
[]
[a0]
HELLO! WORLD!

The first line shows the name ${a} copied as it stands. $10 means group 10, which does not exist, so it inserts nothing; ${1}0 is group 1 and then a zero. Anything beyond numbered groups goes to preg_replace_callback, which hands a function the groups by name, as the last line does to upper-case both words.

PCRE2 itself can do the whole job. In pcre2test, the replace= modifier gives the replacement text, and substitute_extended makes the library read it fully. Then case conversion escapescase conversion escape: A backslash escape in a replacement that changes the case of inserted text: \u and \l change the next letter, \U and \L change everything up to \E. PCRE2 reads them with substitute_extended, and Perl always. work: \u and \l change the next letter, \U and \L change everything up to \E, and ${name} reads a named group. The first block below is the same replacement without the option:

"""Case-conversion escapes in a PCRE2 replacement, through pcre2test.

The replace= modifier gives the replacement string. Without substitute_extended, the library copies the backslash
sequences as they stand; with it, \\u and \\l change the next letter, \\U and \\L change everything up to \\E, and
${name} reads a named group. The line after the pattern shows the subject; " 1: " is the number of replacements
followed by the new text.
"""

from pcre2_session import run

run(r'''
# the replacement as plain text: no modifier, no case change
/(\w+)\s(\w+)/replace=\u$2 \U$1\E!
    hello world

# the same replacement with substitute_extended
/(\w+)\s(\w+)/replace=\u$2 \U$1\E!,substitute_extended
    hello world

# a named group in the replacement, and \L lowering a shouted word
/(?<city>[A-Z]+), (?<pin>\d{6})/replace=${pin} \L${city}\E,substitute_extended
    PUNE, 411001
''')

$ pcre2test -q patterns.txt   (PCRE2 10.49)
# the replacement as plain text: no modifier, no case change
/(\w+)\s(\w+)/replace=\u$2 \U$1\E!
    hello world
 1: \uworld \Uhello\E!

# the same replacement with substitute_extended
/(\w+)\s(\w+)/replace=\u$2 \U$1\E!,substitute_extended
    hello world
 1: World HELLO!

# a named group in the replacement, and \L lowering a shouted word
/(?<city>[A-Z]+), (?<pin>\d{6})/replace=${pin} \L${city}\E,substitute_extended
    PUNE, 411001
 1: 411001 pune

"""Runs a script of patterns and subjects through pcre2test, the test program of the PCRE2 10.49 library, and prints
what it printed.

pcre2test reads a pattern line such as /a.b/ (with modifiers after the closing slash, for example /a.b/dotall,utf),
then one subject per indented line, and prints each subject with " 0:" and the matched text (" 1:" and on for groups)
or "No match". A line starting with # is a comment and is printed as it is. Escapes in a subject such as \\r, \\n or
\\x{2028} stand for those characters (pcre2test's own subject escapes), so invisible characters can be tested.

The script is given in the example file itself, so the example's run record changes whenever the script does:

    from pcre2_session import run
    run(r'''
    /a.b/
        a\\rb
    ''')

The version banner and its release date are not printed (pcre2test -q); the session checks the version itself and
stops if this computer's pcre2test is not PCRE2 10.49, so a record never shows another version's behaviour.
"""

import re
import subprocess
import sys
import tempfile
from pathlib import Path

VERSION = "10.49"
_CANDIDATES = ("/opt/homebrew/bin/pcre2test", "/usr/local/bin/pcre2test")
_ENV = {"PATH": "/usr/bin:/bin", "LC_ALL": "C.UTF-8", "LANG": "C.UTF-8", "TZ": "UTC"}


def _tool() -> str:
    for candidate in _CANDIDATES:
        if Path(candidate).is_file():
            return candidate
    sys.exit("pcre2test (PCRE2 10.49) was not found")


def _check_version(tool: str) -> None:
    first = subprocess.run([tool, "-C"], capture_output=True, text=True, env=_ENV).stdout.splitlines()[:1]
    found = re.match(r"PCRE2 version (\S+)", first[0]) if first else None
    if not found or found.group(1) != VERSION:
        sys.exit(f"this session needs PCRE2 {VERSION}, found: {first[0] if first else 'nothing'}")


def run(script: str) -> None:
    """Prints the pcre2test transcript of the script (patterns, subjects and results)."""
    tool = _tool()
    _check_version(tool)
    lines = script.strip("\n").splitlines()
    text = "\n".join(_dedent(lines)) + "\n"
    with tempfile.TemporaryDirectory() as folder:
        source = Path(folder) / "patterns.txt"
        source.write_text(text, encoding="utf-8")
        done = subprocess.run([tool, "-q", str(source)], capture_output=True, text=True, encoding="utf-8", env=_ENV)
    print(f"$ pcre2test -q patterns.txt   (PCRE2 {VERSION})")
    sys.stdout.write(done.stdout)
    if done.stderr:
        sys.stdout.write(done.stderr)
    if done.returncode != 0:
        print(f"[exit status {done.returncode}]")
    sys.stdout.flush()


def _dedent(lines: list[str]) -> list[str]:
    """Removes the indentation the whole script shares, so it can sit indented inside a Python call."""
    widths = [len(line) - len(line.lstrip(" ")) for line in lines if line.strip()]
    cut = min(widths) if widths else 0
    return [line[cut:] for line in lines]

Without the option the library copies the backslashes: \uworld \Uhello\E!. With it the line reads World HELLO!, and the third block swaps two named groups and lowers one. The manual adds that a \u straight before a \L makes a title case, so a group holding heLLo comes out as Hello. The case escapes act on everything inserted after them, literal text included.

Perl builds the same result without any option. The replacement part of s/// is an interpolated string, so the case escapes of perlop work in it, and $+{name} reads a named group (perlre). The /r modifier returns the changed copy and leaves the original string alone. In Perl, s/(\w+)\s(\w+)/\u$2 \U$1\E!/r on hello world gives World HELLO!, and s/(\w+)/\u$1/g gives Hello World. \u is documented as title-casing the next character, which for a lone letter is the capital.

pcre2grep and pcre2test show what the library does

pcre2grep filters files the way grep does, with the PCRE2 dialect. By default it prints each line that matches. With -o it prints only the matched part, with -o3 only what capturing group 3 captured, and it prints nothing for a line that does not match. pcre2test is the workbench behind it: it reads patterns and subjects from a script and prints the match and every group for each subject. The run below models a pcre2grep search of four synthetic access-log lines, keeping only failed requests (status 4xx or 5xx), with private addresses:

"""What pcre2grep prints for failed requests, modelled with pcre2test on four synthetic access-log lines.

pcre2grep -o shows only the part of a line that matched, -o1 only what group 1 captured, and a line that does not match
prints nothing. pcre2test prints the same pieces numbered: " 0:" is the whole match (what -o shows), " 1:" and on are
the groups (-o1, -o2 ...), and "No match" is a line grep would skip. A subject modifier get=name, written after \\=,
prints one group by its name. Addresses are private ones.
"""

from pcre2_session import run

run(r'''
# lines of the form: address method path status; keep the 4xx and 5xx ones
/^(?<ip>\d+\.\d+\.\d+\.\d+) (?<method>[A-Z]+) (?<path>\S+) (?<status>[45]\d\d)$/
    10.0.4.17 GET /learn/regex/ 200
    10.0.4.17 GET /learn/missing 404
    192.168.1.9 POST /api/login 500
    192.168.1.9 GET /feed 503

# the same pattern, asking for two of the named groups by name
/^(?<ip>\d+\.\d+\.\d+\.\d+) (?<method>[A-Z]+) (?<path>\S+) (?<status>[45]\d\d)$/
    192.168.1.9 POST /api/login 500\=get=path,get=status
''')

$ pcre2test -q patterns.txt   (PCRE2 10.49)
# lines of the form: address method path status; keep the 4xx and 5xx ones
/^(?<ip>\d+\.\d+\.\d+\.\d+) (?<method>[A-Z]+) (?<path>\S+) (?<status>[45]\d\d)$/
    10.0.4.17 GET /learn/regex/ 200
No match
    10.0.4.17 GET /learn/missing 404
 0: 10.0.4.17 GET /learn/missing 404
 1: 10.0.4.17
 2: GET
 3: /learn/missing
 4: 404
    192.168.1.9 POST /api/login 500
 0: 192.168.1.9 POST /api/login 500
 1: 192.168.1.9
 2: POST
 3: /api/login
 4: 500
    192.168.1.9 GET /feed 503
 0: 192.168.1.9 GET /feed 503
 1: 192.168.1.9
 2: GET
 3: /feed
 4: 503

# the same pattern, asking for two of the named groups by name
/^(?<ip>\d+\.\d+\.\d+\.\d+) (?<method>[A-Z]+) (?<path>\S+) (?<status>[45]\d\d)$/
    192.168.1.9 POST /api/login 500\=get=path,get=status
 0: 192.168.1.9 POST /api/login 500
 1: 192.168.1.9
 2: POST
 3: /api/login
 4: 500
  G /api/login (10) path (group 3)
  G 500 (3) status (group 4)

"""Runs a script of patterns and subjects through pcre2test, the test program of the PCRE2 10.49 library, and prints
what it printed.

pcre2test reads a pattern line such as /a.b/ (with modifiers after the closing slash, for example /a.b/dotall,utf),
then one subject per indented line, and prints each subject with " 0:" and the matched text (" 1:" and on for groups)
or "No match". A line starting with # is a comment and is printed as it is. Escapes in a subject such as \\r, \\n or
\\x{2028} stand for those characters (pcre2test's own subject escapes), so invisible characters can be tested.

The script is given in the example file itself, so the example's run record changes whenever the script does:

    from pcre2_session import run
    run(r'''
    /a.b/
        a\\rb
    ''')

The version banner and its release date are not printed (pcre2test -q); the session checks the version itself and
stops if this computer's pcre2test is not PCRE2 10.49, so a record never shows another version's behaviour.
"""

import re
import subprocess
import sys
import tempfile
from pathlib import Path

VERSION = "10.49"
_CANDIDATES = ("/opt/homebrew/bin/pcre2test", "/usr/local/bin/pcre2test")
_ENV = {"PATH": "/usr/bin:/bin", "LC_ALL": "C.UTF-8", "LANG": "C.UTF-8", "TZ": "UTC"}


def _tool() -> str:
    for candidate in _CANDIDATES:
        if Path(candidate).is_file():
            return candidate
    sys.exit("pcre2test (PCRE2 10.49) was not found")


def _check_version(tool: str) -> None:
    first = subprocess.run([tool, "-C"], capture_output=True, text=True, env=_ENV).stdout.splitlines()[:1]
    found = re.match(r"PCRE2 version (\S+)", first[0]) if first else None
    if not found or found.group(1) != VERSION:
        sys.exit(f"this session needs PCRE2 {VERSION}, found: {first[0] if first else 'nothing'}")


def run(script: str) -> None:
    """Prints the pcre2test transcript of the script (patterns, subjects and results)."""
    tool = _tool()
    _check_version(tool)
    lines = script.strip("\n").splitlines()
    text = "\n".join(_dedent(lines)) + "\n"
    with tempfile.TemporaryDirectory() as folder:
        source = Path(folder) / "patterns.txt"
        source.write_text(text, encoding="utf-8")
        done = subprocess.run([tool, "-q", str(source)], capture_output=True, text=True, encoding="utf-8", env=_ENV)
    print(f"$ pcre2test -q patterns.txt   (PCRE2 {VERSION})")
    sys.stdout.write(done.stdout)
    if done.stderr:
        sys.stdout.write(done.stderr)
    if done.returncode != 0:
        print(f"[exit status {done.returncode}]")
    sys.stdout.flush()


def _dedent(lines: list[str]) -> list[str]:
    """Removes the indentation the whole script shares, so it can sit indented inside a Python call."""
    widths = [len(line) - len(line.lstrip(" ")) for line in lines if line.strip()]
    cut = min(widths) if widths else 0
    return [line[cut:] for line in lines]

The first log line has status 200, so pcre2test prints No match, where pcre2grep would stay silent. For the other three, 0: is the whole match, which is what -o prints, and 1: to 4: are the four groups that -o1 to -o4 pick. A subject modifier after \=, here get=path,get=status, prints groups by their names.

The same program measures how much work a pattern costs, which is the match limitmatch limit: A cap on the steps of the matching loop of PCRE2 that turns a runaway match into an error instead of an endless run. pcre2test reports error -47 when it is reached, and PHP returns false.. PCRE2 counts the steps of its main matching loop, and the limit stops a match when the count is reached. find_limits asks pcre2test for the smallest limit that still lets a match finish, and match_limit=N sets one:

"""The match limit of PCRE2, measured and enforced through pcre2test.

find_limits and match_limit=N are modifiers of a subject line, written after \\= at its end. find_limits makes the
library search for the smallest limits that still let the match finish; match_limit=N stops a match after N steps of
its main loop and reports error -47. The pattern is the nested repeat of the lesson; every subject ends in a character
that makes each attempt fail. The first three subjects have 14, 18 and 22 letters a; the fourth has the 26 letters of
the PHP run.
"""

from pcre2_session import run

run(r'''
/^(a+)+$/
    aaaaaaaaaaaaaa!\=find_limits_noheap
    aaaaaaaaaaaaaaaaaa!\=find_limits_noheap
    aaaaaaaaaaaaaaaaaaaaaa!\=find_limits_noheap
    aaaaaaaaaaaaaaaaaaaaaaaaaa!\=match_limit=1000

# one repeat: the same subject fails at once
/^a+$/
    aaaaaaaaaaaaaaaaaaaaaaaaaa!\=find_limits_noheap
''')

$ pcre2test -q patterns.txt   (PCRE2 10.49)
/^(a+)+$/
    aaaaaaaaaaaaaa!\=find_limits_noheap
Minimum match limit = 40960
Minimum depth limit = 31
No match
    aaaaaaaaaaaaaaaaaa!\=find_limits_noheap
Minimum match limit = 655360
Minimum depth limit = 39
No match
    aaaaaaaaaaaaaaaaaaaaaa!\=find_limits_noheap
Minimum match limit = 10485760
Minimum depth limit = 47
No match
    aaaaaaaaaaaaaaaaaaaaaaaaaa!\=match_limit=1000
Failed: error -47: match limit exceeded

# one repeat: the same subject fails at once
/^a+$/
    aaaaaaaaaaaaaaaaaaaaaaaaaa!\=find_limits_noheap
Minimum match limit = 2
Minimum depth limit = 2
No match

"""Runs a script of patterns and subjects through pcre2test, the test program of the PCRE2 10.49 library, and prints
what it printed.

pcre2test reads a pattern line such as /a.b/ (with modifiers after the closing slash, for example /a.b/dotall,utf),
then one subject per indented line, and prints each subject with " 0:" and the matched text (" 1:" and on for groups)
or "No match". A line starting with # is a comment and is printed as it is. Escapes in a subject such as \\r, \\n or
\\x{2028} stand for those characters (pcre2test's own subject escapes), so invisible characters can be tested.

The script is given in the example file itself, so the example's run record changes whenever the script does:

    from pcre2_session import run
    run(r'''
    /a.b/
        a\\rb
    ''')

The version banner and its release date are not printed (pcre2test -q); the session checks the version itself and
stops if this computer's pcre2test is not PCRE2 10.49, so a record never shows another version's behaviour.
"""

import re
import subprocess
import sys
import tempfile
from pathlib import Path

VERSION = "10.49"
_CANDIDATES = ("/opt/homebrew/bin/pcre2test", "/usr/local/bin/pcre2test")
_ENV = {"PATH": "/usr/bin:/bin", "LC_ALL": "C.UTF-8", "LANG": "C.UTF-8", "TZ": "UTC"}


def _tool() -> str:
    for candidate in _CANDIDATES:
        if Path(candidate).is_file():
            return candidate
    sys.exit("pcre2test (PCRE2 10.49) was not found")


def _check_version(tool: str) -> None:
    first = subprocess.run([tool, "-C"], capture_output=True, text=True, env=_ENV).stdout.splitlines()[:1]
    found = re.match(r"PCRE2 version (\S+)", first[0]) if first else None
    if not found or found.group(1) != VERSION:
        sys.exit(f"this session needs PCRE2 {VERSION}, found: {first[0] if first else 'nothing'}")


def run(script: str) -> None:
    """Prints the pcre2test transcript of the script (patterns, subjects and results)."""
    tool = _tool()
    _check_version(tool)
    lines = script.strip("\n").splitlines()
    text = "\n".join(_dedent(lines)) + "\n"
    with tempfile.TemporaryDirectory() as folder:
        source = Path(folder) / "patterns.txt"
        source.write_text(text, encoding="utf-8")
        done = subprocess.run([tool, "-q", str(source)], capture_output=True, text=True, encoding="utf-8", env=_ENV)
    print(f"$ pcre2test -q patterns.txt   (PCRE2 {VERSION})")
    sys.stdout.write(done.stdout)
    if done.stderr:
        sys.stdout.write(done.stderr)
    if done.returncode != 0:
        print(f"[exit status {done.returncode}]")
    sys.stdout.flush()


def _dedent(lines: list[str]) -> list[str]:
    """Removes the indentation the whole script shares, so it can sit indented inside a Python call."""
    widths = [len(line) - len(line.lstrip(" ")) for line in lines if line.strip()]
    cut = min(widths) if widths else 0
    return [line[cut:] for line in lines]

For ^(a+)+$ the minimum is 40,960 steps with 14 letters, 655,360 with 18 and 10,485,760 with 22: four more letters multiply the work by 16. With a limit of 1000 the 26-letter subject of the PHP run ends as error -47, a failure that is not the same as No match. The single repeat ^a+$ needs 2. pcre2grep takes --match-limit for the same purpose, and PHP’s pcre.backtrack_limit is the same kind of guard: they turn a very long run into an error, and the pattern still needs fixing.

Choosing a front door for the job

The choice depends on where the pattern has to run and on what the replacement must hold. Answer the two questions in the table and watch which doors are ruled out, and why:

Each answer rules out a door only for the reason the table gives, and a ruled-out door can still be the right one for a different job: the same person often keeps pcre2test open beside a PHP editor.

In short: use pcre2test and pcre2grep to see what a pattern does and to filter files, use PHP’s functions inside a PHP program and keep to numbered groups or a callback in replacements, and use Perl for a rewrite that needs case escapes or names in one string. Test the pattern in the engine that will run it, since Perl and PCRE2 are close and not the same.

Practise: validators that explain a bad pattern

The exercise rebuilds the lesson’s habits in one small module: three field validators that anchor the whole text, take only ASCII digits and letters and refuse a final newline, and a pattern compiler that reports a bad pattern with its position instead of raising. It is written in Python, the language its tests run in, in your browser; the ideas carry over, and re.fullmatch does the work of PHP’s anchors and D.

check_field("postal", "411 001")   # {"ok": True, "value": "411001"}
check_field("branch", "abcd1234567")  # {"ok": True, "value": "ABCD1234567"}
compile_checked("(")               # (None, "pattern error at position 0: missing ), unterminated subpattern")

import re


def check_field(kind, text):
    """Validates one form field and returns {"ok": True, "value": ...} or {"ok": False, "error": ...}."""
    if kind == "postal":
        # Bug: `$` also matches before a final newline, and `\d` takes digits of other scripts.
        if re.match(r"^\d{6}$", text):
            return {"ok": True, "value": text}
        return {"ok": False, "error": "not a valid postal code"}
    return {"ok": False, "error": "unknown kind: " + str(kind)}


def compile_checked(pattern):
    """Compiles a user's pattern and returns (compiled, None) or (None, "pattern error at position N: ...")."""
    return re.compile(pattern), None

from validators import check_field, compile_checked


def test_postal_plain_and_spaced():
    """six digits, or three, one space and three; the value has no space"""
    assert check_field("postal", "411001") == {"ok": True, "value": "411001"}
    assert check_field("postal", "411 001") == {"ok": True, "value": "411001"}


def test_postal_shapes_that_fail():
    """a leading zero, a short code and two spaces are refused with the kind's message"""
    for text in ("011001", "41100", "411  001", "4110011"):
        assert check_field("postal", text) == {"ok": False, "error": "not a valid postal code"}


def test_postal_trailing_newline():
    """the whole text must fit the shape: a final newline is not allowed"""
    assert check_field("postal", "411001\n")["ok"] is False


def test_postal_ascii_digits_only():
    """Devanagari digits are digits to Python's \\d but are not a postal code here"""
    assert check_field("postal", "४११००१")["ok"] is False


def test_branch_code():
    """four letters and seven digits; lower case in, upper case out"""
    assert check_field("branch", "ABCD1234567") == {"ok": True, "value": "ABCD1234567"}
    assert check_field("branch", "abcd1234567") == {"ok": True, "value": "ABCD1234567"}
    for text in ("ABCD123456", "ABC12345678", "ABCD12345678", "ABCD123456X", "ABCD1234567\n"):
        assert check_field("branch", text) == {"ok": False, "error": "not a valid branch code"}


def test_mobile_number():
    """ten digits starting 6 to 9, with an optional space after the fifth"""
    assert check_field("mobile", "9876543210") == {"ok": True, "value": "9876543210"}
    assert check_field("mobile", "98765 43210") == {"ok": True, "value": "9876543210"}
    for text in ("5876543210", "987654321", "98765  43210", "98765-43210"):
        assert check_field("mobile", text) == {"ok": False, "error": "not a valid mobile number"}


def test_unknown_kind():
    """a kind the function does not know is reported, not raised"""
    assert check_field("pan", "x") == {"ok": False, "error": "unknown kind: pan"}


def test_compile_checked_good_pattern():
    """a good pattern comes back compiled, with no message"""
    compiled, error = compile_checked("[0-9]{6}")
    assert error is None
    assert compiled.fullmatch("411001")


def test_compile_checked_bad_pattern():
    """a bad pattern gives no compiled object and a message with the position"""
    compiled, error = compile_checked("(")
    assert compiled is None
    assert error.startswith("pattern error at position 0: ")
    compiled, error = compile_checked("ab)")
    assert compiled is None
    assert error.startswith("pattern error at position 2: ")

Last, a round of questions on all six ideas. It opens on the one you handled least well and tells you why each answer is right or wrong the moment you pick it:

Interview questions

Warm-up: in PHP, which delimiter would you choose for a pattern that matches paths such as /learn/regex/, and where do the modifier letters go?

Choose a delimiter that does not occur in the pattern. A path pattern is full of slashes, so / would need a backslash before each one, as in /^\/learn\/regex\//. A hash avoids that: #^/learn/regex/#. Braces work as a pair too: {^/learn/regex/}. The first character of the string is the delimiter, and the same character, or the closing bracket of a pair, ends the body. Everything after it is read as modifier letters, so a case-insensitive version is #^/learn/regex/#i. A delimiter that stays inside the body must be escaped, and a letter, a digit, a backslash or white space is refused. When the delimiter comes from a variable, preg_quote($text, '#') escapes it for the body. The delimiter never changes what the pattern matches, only how readable it is.

Two harder questions follow, one on a failed call that reports almost nothing and one on a replacement PHP cannot read. Decide what you would check first in each, then open it:

Key takeaways

• PHP, pcre2test and pcre2grep share PCRE2’s pattern language, but they carry different releases of it (10.44 and 10.49 here), and Perl runs an engine of its own that PCRE2 only imitates.
• A PHP pattern is a string whose first character is the delimiter, and the letters after the closing delimiter are modifiers, such as u for UTF-8 and Unicode digits and D for a $ that ignores a final newline.
• preg_match returns 1, 0 or false, so test it with === false; read a compile problem in the warning and a run problem in preg_last_error_msg().
• A replacement for preg_replace can hold only \n, $n and ${n}, while PCRE2 with substitute_extended and Perl also read case escapes and group names, and PHP does the same work in preg_replace_callback.
• The match limit counts matching steps and turns a runaway pattern into an error, false in PHP and error -47 in pcre2test; the cure is a pattern without a repeat inside a repeat.