Regular expressions with source generation

The log line 18:30:05 WARN [payments] retry 3 of 5 has four parts, and one regular expression with named groups pulls all four out in a single match. Written with [GeneratedRegex], the same pattern becomes C# code when the program is built, while a carelessly written pattern can keep the engine busy for hours on a short input.

You will write .NET regular expressions with groups and named captures, generate them at build time, and recognise the patterns whose running time can explode, with the two ways .NET offers to bound it.

Named groups pull the parts out of a match

A regular expressionregular expression: A pattern that describes text, such as \d{2}:\d{2} for a time: .NET's Regex class finds, checks, splits or replaces the parts of a string that match it. is a small language for describing text: \d is a digit, [A-Z] a capital letter, + one or more of what comes before it, {2} exactly two, ^ the start of the text and $ its end or the point just before a final line break. Parentheses make a group, and (?<name>…) a named groupnamed group: A part of a regular expression written (?<name>…): the text it matches is kept in the match under that name, as Match.Groups["name"].Value., whose text the Match keeps under that name:

using System.Text.RegularExpressions;

// Three log lines, and one pattern whose named groups pull out the parts.
string[] lines =
[
    "18:30:05 WARN  [payments] retry 3 of 5",
    "18:30:07 INFO  [orders] order A-17 placed",
    "not a log line at all",
];

var pattern = new Regex(@"^(?<time>\d{2}:\d{2}:\d{2}) (?<level>[A-Z]+) +\[(?<area>[a-z]+)\] (?<message>.+)$");

foreach (string line in lines)
{
    Match m = pattern.Match(line);
    if (!m.Success)
    {
        Console.WriteLine($"no match: \"{line}\"");
        continue;
    }
    Console.WriteLine($"time {m.Groups["time"].Value}  level {m.Groups["level"].Value,-5} area {m.Groups["area"].Value,-9} message \"{m.Groups["message"].Value}\"");
}

// Every number in a text, with where each one starts.
foreach (Match n in Regex.Matches("retry 3 of 5, waited 250 ms", @"\d+"))
{
    Console.WriteLine($"number {n.Value} at index {n.Index}");
}

// Replace with a group: put the area first.
Console.WriteLine(pattern.Replace(lines[0], "${area}: ${message} (${level})"));

time 18:30:05  level WARN  area payments  message "retry 3 of 5"
time 18:30:07  level INFO  area orders    message "order A-17 placed"
no match: "not a log line at all"
number 3 at index 6
number 5 at index 11
number 250 at index 21
payments: retry 3 of 5 (WARN)

Step through how the pattern consumed the first line:

m.Groups["level"].Value gave WARN without counting positions, and the line that is not a log line came back with Success false instead of an exception. Regex.Matches found every number in a sentence with its index, and Replace reused the groups by name with ${area}. Brackets, dots and the other characters that mean something in a pattern are escaped with a backslash when you mean the character itself, as \[ and \] were.

[GeneratedRegex] turns the pattern into code at build time

new Regex(pattern) parses the pattern when the program runs and then interprets it. Since .NET 7, the source generatorsource generator: A component that runs inside the C# compiler and adds code to the program while it is built. The regex source generator turns each [GeneratedRegex] pattern into C# matching code. for regular expressions does that work while the program is built: mark a partial method, or since .NET 9 and C# 13 a partial property, with [GeneratedRegex(pattern)], and the build writes a class with C# code that matches exactly that pattern:

using System.Text.RegularExpressions;

// The same pattern as logs.cs, generated at build time by the regex source generator.
string line = "18:30:05 WARN  [payments] retry 3 of 5";
Match m = Patterns.LogLine.Match(line);
Console.WriteLine($"level {m.Groups["level"].Value}, area {m.Groups["area"].Value}");
Console.WriteLine($"IsMatch on a span, no string made: {Patterns.LogLine.IsMatch(line.AsSpan())}");
Console.WriteLine($"The generated matcher is a class of its own, derived from {Patterns.LogLine.GetType().BaseType?.Name}");

static partial class Patterns
{
    // C# 13 and later: a partial property; a partial method returning Regex works too.
    [GeneratedRegex(@"^(?<time>\d{2}:\d{2}:\d{2}) (?<level>[A-Z]+) +\[(?<area>[a-z]+)\] (?<message>.+)$")]
    public static partial Regex LogLine { get; }
}

level WARN, area payments
IsMatch on a span, no string made: True
The generated matcher is a class of its own, derived from Regex

The generated matcher found the same groups, and IsMatch on a span of the line matched without making a new string. The property returns one cached instance of a class derived from Regex, so there is nothing to store in a field. Microsoft’s guide to the generator lists what it buys: no parsing at start-up, the speed of RegexOptions.Compiled without compiling at run time, code that works with Native AOT and trimming, and generated code you can read and step through. The analyzer SYSLIB1045 points at Regex calls that could be converted.

Some patterns take exponential time on inputs that almost match

.NET’s default engines backtrackbacktracking: How .NET's default regex engines search: when part of a pattern fails, they go back and try another way to match the text before it. Nested quantifiers can make the number of tries exponential.: when part of a pattern fails, they go back and try another way to match what came before. For most patterns that costs little, but a quantifier inside a quantifier, such as (a+)+$, lets a run of a be split into groups in exponentially many ways, and an input that almost matches makes the engine try them all. Step through ^(a+)+$ on aaa!, three letters and a char it never accepts:

The engine tried every split before it gave up: 4 for three letters, 8 for four, and each extra a doubles the count, so a run of 20 has 524,288. Microsoft’s guide calls the result catastrophic backtracking.

Two settings bound the damage. A match timeout makes a match that runs too long throw RegexMatchTimeoutException. RegexOptions.NonBacktracking runs in time linear in the length of the input, but it supports no backreferences and no lookarounds. Often the pattern can simply be rewritten without the nesting: ^a+$ matches the same texts as ^(a+)+$ and has nothing to try twice.

Catastrophic backtracking, reproduced and bounded Pro: opens with a Pro, Premium or Ultimate pass

Validating input with an allowlist pattern, with its traps Pro: opens with a Pro, Premium or Ultimate pass

SYSLIB1045 and the code the generator writes Pro: opens with a Pro, Premium or Ultimate pass

Practise: parse a log line with a generated pattern

These questions go back over the log pattern, the generated property and the nested quantifier. Your least certain idea is asked first, and each choice explains what it gets right or misses:

Interview questions

Fresher · What does a named group give you in a .NET regular expression?

A name for a part of the match. (?<level>[A-Z]+) captures what that part matched and keeps it under the name, so code reads m.Groups["level"].Value instead of counting parentheses to find group 2. The name keeps working when someone adds a group earlier in the pattern, which would shift every number, and it can be reused in a replacement as ${level}. Check m.Success before reading groups: a failed match has empty groups rather than an exception. A pattern with many groups reads better with names, and RegexOptions.ExplicitCapture makes the unnamed parentheses plain groupings that capture nothing.

Key takeaways

• A named group (?<name>…) keeps its part of the text in Match.Groups["name"], and a failed match returns Success false rather than throwing.
• [GeneratedRegex] on a partial method or property turns a pattern into C# code at build time: no parsing at start-up, and code that works with Native AOT.
• A quantifier inside a quantifier can make the backtracking engines take exponential time; a match timeout or RegexOptions.NonBacktracking bounds it, and a simpler pattern removes it.