Emails, URLs and when to use a parser

Run 20 invented email addresses past three patterns, and the patterns disagree on 12 of them: \S+@\S+ accepts 18, a typical hand-written pattern accepts 7, and the rule a browser applies to type="email" accepts 9. None of the three can tell whether a mailbox exists. Links are harder still: give https://pay.test\@shop.example/cart to the URL parserURL parser: Code that reads a URL by the rules of a standard and returns its parts: scheme, user name, host, port, path, query and fragment. new URL() follows the WHATWG URL Standard; Python's urllib.parse mixes parts of it with RFC 3986. of a browser and to Python’s urllib.parse, and the first reads the host pay.test, the second shop.example.

A regular expression is the right tool for half of this work. It finds text that might be an address or a link, quickly and in any amount of text; a parser, or a rule written for the format, then decides what each candidate is. On the way you check addresses the way browsers do, compare what three URL readers make of the same strings, build an allowlist that one parser decides, see why tags inside tags defeat every regular expression, and chain a loose pattern with a strict check.

Browsers check email addresses with a deliberate rule, not RFC 5322

The program below runs three patterns over the same 20 addresses, each as a full matchfull match: A match that must cover the whole text, from its first character to its last. Python's re.fullmatch asks for it; match and search do not. More in “How an engine finds a match, step by step” from the first character to the last. The first is the naive \S+@\S+. The second is a typical hand-written pattern: letters, digits and ._%+- before the @, and a top-level domain of two letters or more at the end. The third is the HTML Standard’s own regular expression for a valid email address, the check a browser runs on <input type="email">. The comment beside each address says what RFC 5322, the standard for the format of mail messages, makes of it. Press Run, then Edit, add an address of your own to ADDRESSES and run it again:

// Three patterns judge the same 20 invented addresses, each as a full match (anchored at both ends).
//
// naive: anything without spaces on both sides of an @.
// hand:  a hand-written pattern of the kind many sign-up forms copy.
// html:  the HTML Standard's "valid email address" rule, written out as the regular expression the standard gives
//        for it (html.spec.whatwg.org/multipage/input.html#valid-e-mail-address). It is what type="email" checks.
const PATTERNS = {
  naive: /^\S+@\S+$/,
  hand: /^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$/,
  html: /^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/,
};

// The comment after each address says what RFC 5322's grammar makes of it.
const ADDRESSES = [
  'meera.k@example.com', // an ordinary dot-atom
  'dev+alerts@example.org', // + is an ordinary character before the @
  "o'brien@example.com", // so is the apostrophe
  'user/dept=it@example.org', // and / and =
  'ravi@mail.shop.example', // three labels after the @
  'ravi@xn--bcher-kva.example', // an international domain, in its ASCII form
  'ravi@192.0.2.7', // labels of digits only
  'ravi..k@example.com', // two dots in a row: RFC 5322 refuses it unless quoted
  '.ravi@example.com', // a leading dot: RFC 5322 refuses it too
  'ravi@example..com', // an empty label in the domain
  'ravi@shop.example.', // a dot at the very end
  'ravi@@example.com', // two @ signs
  '"ravi kumar"@example.com', // a quoted string, space and all: valid in RFC 5322
  '"ravi"@example.com', // a quoted string without a space: valid too
  'ravi(sales)@example.com', // a comment before the @: valid in RFC 5322
  'ravi@[192.0.2.7]', // a domain literal, an address in brackets: valid in RFC 5322
  'zoë@example.org', // a letter outside ASCII before the @
  'ravi@bücher.example', // an international domain, as people type it
  'ravi@exam_ple.test', // an underscore in the domain
  'ravi kumar@example.com', // a space without quotes
];

const names = Object.keys(PATTERNS);
const width = Math.max(...ADDRESSES.map((a) => a.length)) + 2;
console.log(('address'.padEnd(width) + names.map((n) => n.padEnd(7)).join('')).trimEnd());

const accepted = { naive: 0, hand: 0, html: 0 };
for (const address of ADDRESSES) {
  const marks = names.map((name) => {
    const ok = PATTERNS[name].test(address);
    if (ok) accepted[name] += 1;
    return (ok ? 'yes' : '-').padEnd(7);
  });
  console.log((address.padEnd(width) + marks.join('')).trimEnd());
}
console.log(('accepted'.padEnd(width) + names.map((n) => String(accepted[n]).padEnd(7)).join('')).trimEnd());

// The addresses on which at least two of the three patterns give different answers.
const split = ADDRESSES.filter((a) => new Set(names.map((n) => PATTERNS[n].test(a))).size > 1);
console.log(`the three patterns disagree on ${split.length} of ${ADDRESSES.length}`);

// Where the hand-written pattern and the HTML rule disagree.
const disagree = ADDRESSES.filter((a) => PATTERNS.hand.test(a) !== PATTERNS.html.test(a));
console.log(`hand and html disagree on ${disagree.length}: ${disagree.join('  ')}`);

address                     naive  hand   html
meera.k@example.com         yes    yes    yes
dev+alerts@example.org      yes    yes    yes
o'brien@example.com         yes    -      yes
user/dept=it@example.org    yes    -      yes
ravi@mail.shop.example      yes    yes    yes
ravi@xn--bcher-kva.example  yes    yes    yes
ravi@192.0.2.7              yes    -      yes
ravi..k@example.com         yes    yes    yes
.ravi@example.com           yes    yes    yes
ravi@example..com           yes    yes    -
ravi@shop.example.          yes    -      -
ravi@@example.com           yes    -      -
"ravi kumar"@example.com    -      -      -
"ravi"@example.com          yes    -      -
ravi(sales)@example.com     yes    -      -
ravi@[192.0.2.7]            yes    -      -
zoë@example.org             yes    -      -
ravi@bücher.example         yes    -      -
ravi@exam_ple.test          yes    -      -
ravi kumar@example.com      -      -      -
accepted                    18     7      9
the three patterns disagree on 12 of 20
hand and html disagree on 4: o'brien@example.com  user/dept=it@example.org  ravi@192.0.2.7  ravi@example..com

The naive pattern accepts 18 of the 20, because all it asks for is something on both sides of an @: two @ signs, a dot at the very end and a letter outside ASCII in the domain all pass. The hand-written one accepts 7 and is wrong in both directions. It turns away o'brien@example.com and user/dept=it@example.org, which RFC 5322 allows, and it accepts ravi@example..com, whose domain has an empty label. The run’s last line lists the four addresses on which it and the HTML rule part ways, and the line above it counts the 12 addresses on which the three patterns do not all agree.

The HTML rule accepts 9, and it is not RFC 5322 either. The standard says so on purpose: it calls its definition a willful violation of RFC 5322, whose syntax it judges too strict, too vague and too lax for a form field at the same time. Rows of the run show each of the three complaints:

Before the @, RFC 5322 refuses empty parts that the HTML rule lets through. After it, the RFC’s domain grammar allows names no host can have, and the RFC itself calls that syntax liberal and leaves the real rules to the standards of DNS and mail transport. And it allows comments, spaces inside quotes and bracketed addresses that people rarely type. A complete RFC 5322 checker would even have to count brackets, because the RFC’s comments may nest, which a later section shows no regular expression can do. Neither grammar covers the ë of zoë@example.org: RFC 6531 adds UTF-8 to the characters of an address, for mail systems that support internationalized email. Labels and text input types shows how browsers treat such a field.

No rule of either kind answers the question a sign-up form cares about. ravi.kumar.1987@example.com is well formed; whether a mailbox of that name exists, and whether the person typing it reads that mailbox, only the mail system at the domain knows. RFC 5322 leaves the part before the @ to the receiving host, which reads it as the name of one of its mailboxes. The test is a message: OWASP’s cheat sheet on email verification asks for random tokens that work once and expire, and for no active account until the link in the message has been used.

The confirmation step also covers what the HTML rule lets through on purpose: ravi..k@example.com passes its shape check, and only the mail system at the domain can say whether a mailbox of that name exists.

A URL regex splits the text; a URL parser decides the host

RFC 3986, the generic syntax of URIs, gives a regular expression of its own in its Appendix B. It splits a well-formed URI reference into five parts: scheme, authority, path, query and fragment. It checks nothing, and its authority is all the text between // and the next /, ? or #, user name and port included. The program sets it beside new URL(), which in Node.js follows the WHATWG URL Standard, the rules browsers use, on six URLs:

// Six invented URLs, read two ways in Node.js:
//   regex:     the regular expression of RFC 3986, Appendix B, which splits a well-formed URI reference into its
//              parts. Group 4 is the authority: everything between // and the next /, ? or #.
//   new URL(): the WHATWG URL parser, the one browsers use. It also says what it made of the user name and port.
const SPLIT = /^(([^:/?#]+):)?(\/\/([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?/;

const URLS = [
  'https://shop.example/cart',
  'https://shop.example:8443/cart',
  'https://shop.example@pay.test/cart',
  'https://pay.test\\@shop.example/cart', // one backslash in the URL itself
  'https://BÜCHER.example/',
  'https://shop.example#@pay.test/',
];

for (const text of URLS) {
  const authority = SPLIT.exec(text)[4];
  const url = new URL(text);
  console.log(text);
  console.log(`  regex authority:   ${authority}`);
  console.log(`  new URL hostname:  ${url.hostname}`);
  const extras = [];
  if (url.username) extras.push(`username ${url.username}`);
  if (url.port) extras.push(`port ${url.port}`);
  if (url.pathname !== '/') extras.push(`path ${url.pathname}`);
  if (url.hash) extras.push(`hash ${url.hash}`);
  if (extras.length) console.log(`  new URL also:      ${extras.join(', ')}`);
}

https://shop.example/cart
  regex authority:   shop.example
  new URL hostname:  shop.example
  new URL also:      path /cart
https://shop.example:8443/cart
  regex authority:   shop.example:8443
  new URL hostname:  shop.example
  new URL also:      port 8443, path /cart
https://shop.example@pay.test/cart
  regex authority:   shop.example@pay.test
  new URL hostname:  pay.test
  new URL also:      username shop.example, path /cart
https://pay.test\@shop.example/cart
  regex authority:   pay.test\@shop.example
  new URL hostname:  pay.test
  new URL also:      path /@shop.example/cart
https://BÜCHER.example/
  regex authority:   BÜCHER.example
  new URL hostname:  xn--bcher-kva.example
https://shop.example#@pay.test/
  regex authority:   shop.example
  new URL hostname:  shop.example
  new URL also:      hash #@pay.test/

Where the authority holds nothing but a host, as in the first URL, the regex and new URL() agree. Elsewhere the regex hands back raw text: shop.example:8443 with its port, shop.example@pay.test with a user name in front of the real host, and BÜCHER.example as typed, where the parser gives the ASCII form xn--bcher-kva.example that DNS uses. The regex has done its job, which is to cut; what the cut text means is the job of a URL parser. Python’s urllib.parse is one too. Run its urlsplit() on the same six URLs, then press Edit and add a URL of your own:

"""The same six invented URLs, split by Python's urllib.parse.

urlsplit() cuts the netloc (user name, host and port together) out of the URL, and its hostname attribute reads the
host from the netloc: the text after the last @, without the port, in lower case.
"""

from urllib.parse import urlsplit

URLS = [
    "https://shop.example/cart",
    "https://shop.example:8443/cart",
    "https://shop.example@pay.test/cart",
    "https://pay.test\\@shop.example/cart",  # one backslash in the URL itself
    "https://BÜCHER.example/",
    "https://shop.example#@pay.test/",
]

for text in URLS:
    parts = urlsplit(text)
    print(text)
    print(f"  netloc:    {parts.netloc}")
    print(f"  hostname:  {parts.hostname}")
    extras = []
    if parts.username:
        extras.append(f"username {parts.username}")
    if parts.port:
        extras.append(f"port {parts.port}")
    if parts.path not in ("", "/"):
        extras.append(f"path {parts.path}")
    if parts.fragment:
        extras.append(f"fragment {parts.fragment}")
    if extras:
        print(f"  also:      {', '.join(extras)}")

https://shop.example/cart
  netloc:    shop.example
  hostname:  shop.example
  also:      path /cart
https://shop.example:8443/cart
  netloc:    shop.example:8443
  hostname:  shop.example
  also:      port 8443, path /cart
https://shop.example@pay.test/cart
  netloc:    shop.example@pay.test
  hostname:  pay.test
  also:      username shop.example, path /cart
https://pay.test\@shop.example/cart
  netloc:    pay.test\@shop.example
  hostname:  shop.example
  also:      username pay.test\, path /cart
https://BÜCHER.example/
  netloc:    BÜCHER.example
  hostname:  bücher.example
https://shop.example#@pay.test/
  netloc:    shop.example
  hostname:  shop.example
  also:      fragment @pay.test/

Side by side, the two parsers report these hosts:

They agree on four hosts. On the international name they agree on the host but not on its spelling, since urlsplit() only lowers the case, so a comparison has to bring both sides to one form first. On the backslash they read two different hosts. Python’s documentation is frank about why: urlsplit() and urlparse() do not validate their input, and they take parts of both the WHATWG rules and RFC 3986 and cannot be called compliant with either. The WHATWG standard, for its part, sets out to replace RFC 3986 with the rules browsers really follow. Step through the backslash URL and watch where each parser ends the host:

Two readings of one string are a parser differentialparser differential: A case where two parsers read the same input differently, such as two URL parsers that report different hosts for one string. A check made with one of them then says nothing about what the other will do.. Node’s documentation warns about the same thing in its older url.parse(), which it calls lenient and prone to host name spoofing, and it sends readers to the WHATWG API instead. Running several parsers on the same inputs and comparing what they return is a differential testdifferential test: A test that runs the same cases through two implementations, such as two regex engines, and reports every case where their results differ. Each case holds a pattern, a subject and the result the job should give. More in “Flavours compared: porting a pattern safely”, the method the porting lesson uses for regex engines.

An allowlist compares the parsed host, never the raw string

An image proxy of a shop may fetch pictures only from img.shop.example, over HTTPS. Three versions of its check run on the same test tabletest table: A list of sample strings kept beside a pattern, each with the answer the pattern must give, run on every change. It documents the decisions of the pattern and catches edits that break them. More in “Readable patterns: free-spacing and comments” of six URLs: a test of how the raw string starts, a test of the host that urlsplit() returns, and a fixed check. Run it, then press Edit and add a case of your own to CASES, with the answer you expect:

"""An image proxy may fetch only from img.shop.example over HTTPS. Three checks of the requested URL run on the
same test table: a prefix check on the raw string, a check of the host that urlsplit() reads, and a check that
parses once, refuses anything two parsers could read differently, and hands on a URL rebuilt from what it checked.
"""

from urllib.parse import urlsplit, urlunsplit

ALLOWED_HOSTS = {"img.shop.example"}


def prefix_check(text):
    """Weak: compares the raw string."""
    return text if text.startswith("https://img.shop.example") else None


def host_only_check(text):
    """Weak: parses, but checks nothing except the host this parser reads."""
    return text if urlsplit(text).hostname in ALLOWED_HOSTS else None


def fixed_check(text):
    """Parses once, refuses ambiguous input, and returns the URL it checked."""
    if "\\" in text:  # parsers disagree on what a backslash means
        return None
    parts = urlsplit(text)
    if parts.scheme != "https" or "@" in parts.netloc:  # no other scheme, no user name before the host
        return None
    try:
        port = parts.port  # reading it raises ValueError for a port that is not a number in range
    except ValueError:
        return None
    if port not in (None, 443) or parts.hostname not in ALLOWED_HOSTS:
        return None
    return urlunsplit(("https", parts.hostname, parts.path, parts.query, ""))


# Each case: the URL, and whether the proxy may fetch it.
CASES = [
    ("https://img.shop.example/p/42.jpg", True),
    ("https://img.shop.example.pay.test/p/42.jpg", False),
    ("https://img.shop.example@pay.test/p/42.jpg", False),
    ("https://pay.test\\@img.shop.example/p/42.jpg", False),
    ("https://IMG.Shop.Example/p/42.jpg", True),
    ("http://img.shop.example/p/42.jpg", False),
]

CHECKS = [("prefix", prefix_check), ("host only", host_only_check), ("fixed", fixed_check)]

wrong = {name: 0 for name, _ in CHECKS}
for text, allowed in CASES:
    print(text)
    print(f"  expected: {'fetch' if allowed else 'refuse'}")
    for name, check in CHECKS:
        result = check(text)
        right = (result is not None) == allowed
        if not right:
            wrong[name] += 1
        verdict = "fetch " + result if result else "refuse"
        print(f"  {name:<10}{verdict}{'' if right else '   <- wrong'}")

print("wrong answers: " + ", ".join(f"{name} {count}" for name, count in wrong.items()))

https://img.shop.example/p/42.jpg
  expected: fetch
  prefix    fetch https://img.shop.example/p/42.jpg
  host only fetch https://img.shop.example/p/42.jpg
  fixed     fetch https://img.shop.example/p/42.jpg
https://img.shop.example.pay.test/p/42.jpg
  expected: refuse
  prefix    fetch https://img.shop.example.pay.test/p/42.jpg   <- wrong
  host only refuse
  fixed     refuse
https://img.shop.example@pay.test/p/42.jpg
  expected: refuse
  prefix    fetch https://img.shop.example@pay.test/p/42.jpg   <- wrong
  host only refuse
  fixed     refuse
https://pay.test\@img.shop.example/p/42.jpg
  expected: refuse
  prefix    refuse
  host only fetch https://pay.test\@img.shop.example/p/42.jpg   <- wrong
  fixed     refuse
https://IMG.Shop.Example/p/42.jpg
  expected: fetch
  prefix    refuse   <- wrong
  host only fetch https://IMG.Shop.Example/p/42.jpg
  fixed     fetch https://img.shop.example/p/42.jpg
http://img.shop.example/p/42.jpg
  expected: refuse
  prefix    refuse
  host only fetch http://img.shop.example/p/42.jpg   <- wrong
  fixed     refuse
wrong answers: prefix 3, host only 2, fixed 0

The prefix test gets three of the six wrong. https://img.shop.example.pay.test/ starts with the right letters and names another host; https://img.shop.example@pay.test/ puts the allowed name where the user name goes; and https://IMG.Shop.Example/ is the allowed host in capitals, which host names ignore and a string comparison does not. The host-only test gets two wrong: it lets plain http through, and it accepts the backslash URL, because urlsplit() reads the host img.shop.example there, while new URL() read pay.test from the same shape of URL in the previous section. Each check, with the cases it got wrong:

The fixed check follows OWASP’s cheat sheets on redirects and on server-side requests. It parses once and compares the parsed scheme, host and port with the allowlist. It refuses a user name, and it treats input that two parsers could read differently as a refusal, with no attempt to work out which reading is meant. It then returns a URL rebuilt from the parts it checked, so whatever fetches the picture receives exactly what passed, never the text it was given.

Every row of the table is a case the check must keep getting right, so it stays in the file beside the check and runs on every change.

Nested tags need a count that no regular expression keeps

Here is a page with one div inside another and a second div after them: <div>Plan<div>new</div>details</div><div>help</div>. Which text belongs to the outer div? A lazy quantifierlazy quantifier: A quantifier followed by a question mark, such as *? or {2,5}?, which repeats its atom as few times as it can and adds one repetition at a time only when the rest of the pattern cannot match otherwise. Also called non-greedy or reluctant. More in “Greedy and lazy quantifiers” looks like the answer. The program shows what it returns, beside a greedy pattern and Python’s html.parser:

"""One div inside another, then a second div after them. Which text belongs to the first, outer div?

A lazy and a greedy pattern each give an answer, and Python's html.parser gives a third by counting how deep it is.
"""

import re
from html.parser import HTMLParser

PAGE = "<div>Plan<div>new</div>details</div><div>help</div>"

print(PAGE)
print("lazy   <div>.*?</div> :", re.search(r"<div>.*?</div>", PAGE).group())
print("greedy <div>.*</div>  :", re.search(r"<div>.*</div>", PAGE).group())
print("lazy, every match     :", re.findall(r"<div>.*?</div>", PAGE))


class DivDepth(HTMLParser):
    """Counts open divs and remembers where each one starts, so it can tell which close tag ends which div."""

    def __init__(self):
        super().__init__()
        self.depth = 0
        self.starts = []  # where each div that is still open began
        self.outer = []  # the complete divs at depth 1, as (start, end)

    def handle_starttag(self, tag, attrs):
        if tag == "div":
            self.depth += 1
            self.starts.append(self.getpos()[1])
            print(f"  at {self.getpos()[1]:>2}  <div>   depth {self.depth}")

    def handle_endtag(self, tag):
        if tag == "div":
            start = self.starts.pop()
            end = self.getpos()[1] + len("</div>")
            print(f"  at {self.getpos()[1]:>2}  </div>  depth {self.depth} -> {self.depth - 1}")
            if self.depth == 1:
                self.outer.append((start, end))
            self.depth -= 1


print("html.parser, tag by tag:")
parser = DivDepth()
parser.feed(PAGE)
parser.close()
for start, end in parser.outer:
    print(f"outer div from {start} to {end}: {PAGE[start:end]}")

<div>Plan<div>new</div>details</div><div>help</div>
lazy   <div>.*?</div> : <div>Plan<div>new</div>
greedy <div>.*</div>  : <div>Plan<div>new</div>details</div><div>help</div>
lazy, every match     : ['<div>Plan<div>new</div>', '<div>help</div>']
html.parser, tag by tag:
  at  0  <div>   depth 1
  at  9  <div>   depth 2
  at 17  </div>  depth 2 -> 1
  at 30  </div>  depth 1 -> 0
  at 36  <div>   depth 1
  at 45  </div>  depth 1 -> 0
outer div from 0 to 36: <div>Plan<div>new</div>details</div>
outer div from 36 to 51: <div>help</div>

The lazy <div>.*?</div> stops at the first </div>, the inner one, and returns <div>Plan<div>new</div>: the start of the outer div and all of the inner one. The greedy form runs on to the last </div> of the page and swallows the second div as well. html.parser calls a method for every start tag and every end tag, and the class in the example adds one to a count at each <div> and takes one away at each </div>. The close tag that brings the count back to 0 ends the outer div, which runs from 0 to 36. Step through the page and compare the two readings:

The parser wins because it counts. Russ Cox describes a regular languageregular language: A set of strings that can be checked in one pass over the text with a fixed amount of memory. Regular expressions in the strict sense, without backreferences or recursion, describe these sets, so they cannot count how deeply things nest., loosely, as a set of strings that a matcher can check in one pass over the text with a fixed amount of memory, and a finite automatonfinite automaton: A machine that is always in one of a fixed set of states and moves between them as it reads the text. Engines such as Go's regexp simulate one instead of backtracking. More in “How an engine finds a match, step by step” that recognises one has a fixed number of states. Nesting needs a count with no upper bound: every extra level is one more thing to remember. PCRE2’s documentation says the same about nested parentheses: without recursionrecursion: A call written (?R) or (?0) that runs the whole pattern again at the current position, inside the match in progress, so a pattern can match nested structures such as balanced brackets. More in “Conditionals, recursion and PCRE2 extras”, a pattern can match nesting only up to a depth fixed in advance, never to any depth whatever. The RFC 5322 comments of the first section fall under the same limit.

PCRE2 does offer a way past the limit. (?R) runs the whole pattern again at the point where it stands, so an inner div is matched by a nested call that must reach its own </div> first. One recorded run shows what that gets right and what it hides:

"""PCRE2 10.49 on the same nested divs: a lazy pattern, then a recursive one.

In the recursive pattern, (?R) runs the whole pattern again where it stands, so an inner <div> is matched by a nested
call that must reach its own </div> before the outer match goes on. [^<]++ takes text up to the next tag, and
<(?!/?div>) takes a < that starts no div tag. The g modifier asks for every match, and the second subject has a
<div> that is never closed.
"""

from pcre2_session import run

run(r'''
/<div>.*?<\/div>/g
    <div>Plan<div>new</div>details</div><div>help</div>

/<div>(?:[^<]++|<(?!\/?div>)|(?R))*<\/div>/g
    <div>Plan<div>new</div>details</div><div>help</div>
    <div>Plan<div>new</div>details
''')

$ pcre2test -q patterns.txt   (PCRE2 10.49)
/<div>.*?<\/div>/g
    <div>Plan<div>new</div>details</div><div>help</div>
 0: <div>Plan<div>new</div>
 0: <div>help</div>

/<div>(?:[^<]++|<(?!\/?div>)|(?R))*<\/div>/g
    <div>Plan<div>new</div>details</div><div>help</div>
 0: <div>Plan<div>new</div>details</div>
 0: <div>help</div>
    <div>Plan<div>new</div>details
 0: <div>new</div>

"""Runs a script of patterns and subjects through pcre2test, the test program of the PCRE2 10.49 library, and prints
what it printed.

pcre2test reads a pattern line such as /a.b/ (with modifiers after the closing slash, for example /a.b/dotall,utf),
then one subject per indented line, and prints each subject with " 0:" and the matched text (" 1:" and on for groups)
or "No match". A line starting with # is a comment and is printed as it is. Escapes in a subject such as \\r, \\n or
\\x{2028} stand for those characters (pcre2test's own subject escapes), so invisible characters can be tested.

The script is given in the example file itself, so the example's run record changes whenever the script does:

    from pcre2_session import run
    run(r'''
    /a.b/
        a\\rb
    ''')

The version banner and its release date are not printed (pcre2test -q); the session checks the version itself and
stops if this computer's pcre2test is not PCRE2 10.49, so a record never shows another version's behaviour.
"""

import re
import subprocess
import sys
import tempfile
from pathlib import Path

VERSION = "10.49"
_CANDIDATES = ("/opt/homebrew/bin/pcre2test", "/usr/local/bin/pcre2test")
_ENV = {"PATH": "/usr/bin:/bin", "LC_ALL": "C.UTF-8", "LANG": "C.UTF-8", "TZ": "UTC"}


def _tool() -> str:
    for candidate in _CANDIDATES:
        if Path(candidate).is_file():
            return candidate
    sys.exit("pcre2test (PCRE2 10.49) was not found")


def _check_version(tool: str) -> None:
    first = subprocess.run([tool, "-C"], capture_output=True, text=True, env=_ENV).stdout.splitlines()[:1]
    found = re.match(r"PCRE2 version (\S+)", first[0]) if first else None
    if not found or found.group(1) != VERSION:
        sys.exit(f"this session needs PCRE2 {VERSION}, found: {first[0] if first else 'nothing'}")


def run(script: str) -> None:
    """Prints the pcre2test transcript of the script (patterns, subjects and results)."""
    tool = _tool()
    _check_version(tool)
    lines = script.strip("\n").splitlines()
    text = "\n".join(_dedent(lines)) + "\n"
    with tempfile.TemporaryDirectory() as folder:
        source = Path(folder) / "patterns.txt"
        source.write_text(text, encoding="utf-8")
        done = subprocess.run([tool, "-q", str(source)], capture_output=True, text=True, encoding="utf-8", env=_ENV)
    print(f"$ pcre2test -q patterns.txt   (PCRE2 {VERSION})")
    sys.stdout.write(done.stdout)
    if done.stderr:
        sys.stdout.write(done.stderr)
    if done.returncode != 0:
        print(f"[exit status {done.returncode}]")
    sys.stdout.flush()


def _dedent(lines: list[str]) -> list[str]:
    """Removes the indentation the whole script shares, so it can sit indented inside a Python call."""
    widths = [len(line) - len(line.lstrip(" ")) for line in lines if line.strip()]
    cut = min(widths) if widths else 0
    return [line[cut:] for line in lines]

The recursive pattern returns the outer div whole, details and all. On the second subject, whose outer div is never closed, it reports only <div>new</div> and says nothing about the div left open, where code built on a parser holds the open tags and their positions and can name the one that never closed. JavaScript and Python’s re refuse the syntax altogether, as the recursion lesson records, and that lesson covers recursion in depth. For markup, a parser is the shorter road.

A loose pattern finds candidates, a strict check decides

A pre-filterpre-filter: A cheap first pass, often a loose regular expression, that keeps everything that might be what you are looking for, so that a slower, strict check only has to judge the candidates it found. turns the limits of the last four sections into a method. A loose pattern runs over all of the text and keeps everything that might be an address or a link, junk included; a strict check then judges each candidate. The paragraph in this program mixes both kinds with ordinary sentence punctuation:

"""A loose pattern finds everything that might be an address or a link; a stricter check then decides.

Stage 1 is cheap and generous: any run of non-spaces around an @, any run of non-spaces after scheme://, with the
punctuation that ends a sentence trimmed off. Stage 2 decides: an email address must fully match the HTML Standard's
valid email address rule, and a link must survive urlsplit() with a web scheme, a host, no user name, no backslash
and a port in range.
"""

import re
from urllib.parse import urlsplit

TEXT = (
    "Questions? Write to meera.k@example.com or help@shop.example. The old list said ravi@@example.com, "
    "and the banner file is logo@2x.png. Setup notes live at https://docs.shop.example/setup, the cart moved "
    "to https://shop.example:99999/cart, and a mirror is listed as https://docs.shop.example\\@pay.test/setup. "
    "Files: ftp://files.example.org/manual.pdf and https://shop.example/help?topic=email#reset."
)

EMAIL_LIKE = re.compile(r"\S+@\S+")
URL_LIKE = re.compile(r"[A-Za-z][A-Za-z0-9+.-]*://\S+")
END_PUNCTUATION = ".,;:!?"

# The HTML Standard's rule, the same pattern the standard gives (a slash needs no backslash in Python).
HTML_EMAIL = re.compile(
    r"[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?"
    r"(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*"
)


def email_verdict(candidate):
    return None if HTML_EMAIL.fullmatch(candidate) else "the HTML rule refuses it"


def url_verdict(candidate):
    if "\\" in candidate:
        return "a backslash: parsers disagree on it"
    parts = urlsplit(candidate)
    if parts.scheme not in ("http", "https"):
        return f"scheme {parts.scheme}, not a web link"
    if not parts.hostname or "@" in parts.netloc:
        return "no plain host"
    try:
        parts.port
    except ValueError:
        return "the port is out of range"
    return None


def run_stage(title, pattern, verdict):
    candidates = [found.rstrip(END_PUNCTUATION) for found in pattern.findall(TEXT)]
    print(f"{title}: {len(candidates)} candidates")
    kept = 0
    for candidate in candidates:
        reason = verdict(candidate)
        if reason is None:
            kept += 1
            print(f"  keep  {candidate}")
        else:
            print(f"  drop  {candidate}  ({reason})")
    print(f"  kept {kept} of {len(candidates)}")
    return len(candidates), kept


emails = run_stage("email-like", EMAIL_LIKE, email_verdict)
urls = run_stage("URL-like", URL_LIKE, url_verdict)
print(f"in all: {emails[0] + urls[0]} candidates, {emails[1] + urls[1]} kept")

email-like: 5 candidates
  keep  meera.k@example.com
  keep  help@shop.example
  drop  ravi@@example.com  (the HTML rule refuses it)
  keep  logo@2x.png
  drop  https://docs.shop.example\@pay.test/setup  (the HTML rule refuses it)
  kept 3 of 5
URL-like: 5 candidates
  keep  https://docs.shop.example/setup
  drop  https://shop.example:99999/cart  (the port is out of range)
  drop  https://docs.shop.example\@pay.test/setup  (a backslash: parsers disagree on it)
  drop  ftp://files.example.org/manual.pdf  (scheme ftp, not a web link)
  keep  https://shop.example/help?topic=email#reset
  kept 2 of 5
in all: 10 candidates, 5 kept

Ten candidates come out of the first stage and five survive the second, and every drop has its reason: a doubled @, a link that the email pattern caught because it held an @, a port above 65535, an ftp link and a backslash. One kept address is no address at all. logo@2x.png is a file name, and the HTML rule accepts it, because labels of letters and digits are all it asks for. The second stage is only as strict as its rule. Follow the two stages:

MySmartCoPilot’s extractors are built the same way. The URL extractor looks for :// and www., trims the end of each link by the autolink rules of GitHub Flavored Markdown, then lets new URL() decide, and lists what it could not parse as text that is not a valid address. The email extractor reads outwards from each @ instead of running one large pattern, and its option Only real top-level domains drops names such as logo@2x.png, the junk this run kept, since png is no top-level domain.

Why two stages instead of one strict pattern? The loose pattern can afford to be generous because it decides nothing, and it is cheap enough to run over every character of the text. The strict check runs only on the candidates, so it can be a real parser with the real rules, and every reason it gives can be logged and counted. One pattern would have to do both jobs, and the sections before this one show it cannot be the judge of an address or a link.

Choose a regex for flat text and a parser for structure

The four approaches of this lesson each fit a different input. A field that holds one value of fixed shape, such as an order code, needs one anchored pattern; the validating lesson adds the checks in code that dates and check digits need after it. Addresses and links somewhere in ordinary text need a pre-filter and a check, and anything that nests needs a parser. Answer the two questions of the table for an input of your own and see which rows still fit:

In short: one anchored pattern for a value of fixed shape, a loose pattern followed by a rule or a parser for items inside text, and a parser for anything that nests. PCRE2 recursion is the exception for nested brackets, when the check only ever runs in PCRE2, PHP or Perl and a match without a report on errors is enough.

Practise: extract, then parse

The exercise asks for both stages, for addresses and for links, in Python. The loose patterns and the HTML rule are written for you; the trimming, the strict checks, the rebuilt links and the duplicates are yours, and the sample tests reuse strings this lesson worked through beside a few new ones.

"""extract.py: the email addresses and links in text that people wrote, in two stages each.

Stage 1 is a loose pattern that finds every candidate. Stage 2 is the strict check that decides. The starter has
stage 1 only, so it returns junk along with the real addresses and links.
"""

import re
from urllib.parse import urlsplit, urlunsplit


def extract_emails(text):
    """The valid email addresses in text, each once, in the order they first appear."""
    return EMAIL_LIKE.findall(text)


def extract_urls(text):
    """The valid http and https links in text, rebuilt from their parts, each once, in order."""
    return URL_LIKE.findall(text)


# Stage 1: loose patterns that miss nothing.
EMAIL_LIKE = re.compile(r"\S+@\S+")
URL_LIKE = re.compile(r"https?://\S+", re.IGNORECASE)

# Stage 2 for addresses: the HTML Standard's valid email address rule, written as the standard's own regular
# expression. Use it with fullmatch.
HTML_EMAIL = re.compile(
    r"[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?"
    r"(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*"
)

from extract import extract_emails, extract_urls


def test_emails_lose_their_punctuation():
    """extract_emails: a full stop, a comma or brackets around an address are not part of it"""
    text = "Write to meera.k@example.com, or to <dev+alerts@example.org>."
    assert extract_emails(text) == ["meera.k@example.com", "dev+alerts@example.org"]


def test_emails_follow_the_html_rule():
    """extract_emails: the HTML rule keeps an apostrophe and drops a doubled @ and a quoted name"""
    text = "o'brien@example.com ravi@@example.com \"ravi\"@example.com"
    assert extract_emails(text) == ["o'brien@example.com"]


def test_emails_need_a_dot_after_the_at():
    """extract_emails: a domain without a dot is dropped"""
    assert extract_emails("asha@localhost and asha@shop.test") == ["asha@shop.test"]


def test_emails_lower_the_domain_only():
    """extract_emails: the domain goes to lower case, the part before the @ stays as written, duplicates go"""
    text = "Help@Shop.Example, help@shop.example and Help@SHOP.example"
    assert extract_emails(text) == ["Help@shop.example", "help@shop.example"]


def test_emails_keep_their_order():
    """extract_emails: the order of first appearance"""
    text = "zed@shop.example, amy@shop.example, zed@shop.example"
    assert extract_emails(text) == ["zed@shop.example", "amy@shop.example"]


def test_urls_lose_their_punctuation():
    """extract_urls: punctuation at the end of a link belongs to the sentence"""
    text = "See https://docs.shop.example/setup, then (https://shop.example/help)."
    assert extract_urls(text) == ["https://docs.shop.example/setup", "https://shop.example/help"]


def test_urls_drop_what_a_parser_doubts():
    """extract_urls: a bad port, a backslash, a user name and a missing host are all dropped"""
    text = (
        "https://shop.example:99999/cart https://pay.test\\@shop.example/x "
        "https://shop.example@pay.test/ http://"
    )
    assert extract_urls(text) == []


def test_urls_are_rebuilt_from_their_parts():
    """extract_urls: lower-case scheme and host, the default port gone, the rest as written"""
    text = "HTTPS://Docs.Shop.Example:443/Setup?x=1#top and https://docs.shop.example/Setup?x=1#top"
    assert extract_urls(text) == ["https://docs.shop.example/Setup?x=1#top"]


def test_urls_keep_a_port_that_is_not_the_default():
    """extract_urls: port 8080 stays, port 80 of http goes"""
    text = "http://shop.example:8080/ and http://shop.example:80/ and https://shop.example:80/"
    assert extract_urls(text) == ["http://shop.example:8080/", "http://shop.example/", "https://shop.example:80/"]

Last, the self-check. It covers the six ideas from the HTML email rule to choosing between a pattern and a parser, puts the idea you have shown least first, and explains every option, the wrong ones included:

Interview questions

Warm-up: why can no regular expression prove that an email address works?

A regular expression reads only the characters of the address, so the most it can show is that they have the shape of one. Whether a mailbox of that name exists, accepts mail and belongs to the person who typed it is decided by the mail system at the domain, which no pattern can ask. A made-up address such as nobody.here@example.com is perfectly well formed, so every shape check accepts it. RFC 5322 itself treats the part before the @ as a name that only the receiving host can interpret. That is why a sign-up flow checks the shape with a forgiving rule such as the HTML Standard’s, then sends a message with a link that works once and expires, and treats the address as confirmed only after the link is opened.

Two harder situations carry these ideas into real systems: a sign-in page that sends people on to the page they asked for, and a nightly job that keeps the links customers paste into support messages. Each row below opens on a first step; write down your own before you read it.

Key takeaways

• The HTML Standard’s valid email address rule departs from RFC 5322 on purpose, and on the 20 addresses it accepted 9 where a naive pattern accepted 18 and a hand-written one 7; a pattern checks only the shape, and a confirmation message shows whether someone receives mail at the address.
• RFC 3986’s regular expression splits a URL into parts without checking them, and only a parser says which host a URL names: new URL() and urlsplit() agreed on four of six hosts and read two different hosts from the URL with a backslash.
• An allowlist decides with one parser, refuses input that parsers could read differently, compares the parsed scheme, host and port, and passes on the URL it rebuilt, where the prefix test and the host-only test got three and two of the six cases wrong.
• Nested structures need a count that grows with the depth, which no regular language can keep, so the lazy pattern stopped at the inner </div> while a depth count over html.parser found the outer div; PCRE2 recursion matched it but said nothing about an unclosed tag.
• A loose pattern makes a good pre-filter because it misses nothing and decides nothing, and the strict check after it is only as good as its rule, as logo@2x.png showed.