Log Pattern Analyzer (Group Millions of Log Lines into Templates)

About the Log Pattern Analyzer (Group Millions of Log Lines into Templates)

A production log has millions of lines but only a few hundred kinds of message. Load the log and the analyzer reads timestamps and levels, then groups the lines into templates — Connection to <IP> refused, GET /api/orders/<NUM> 200 <NUM>ms — with the Drain algorithm, turning IDs, numbers, IP addresses and times into placeholders. Templates are listed errors first, with their count, share, first and last time seen and real examples.

A timeline shows errors and warnings over time and marks the spikes, with the templates behind each one. Compare two logs (before and after a deploy, or a good day and a bad one) to see the patterns that are new, the ones that vanished and the ones whose rate changed threefold or more.

Plain text, JSON lines (also journalctl -o json and Docker’s json-file logs), syslog (RFC 5424 and RFC 3164), journalctl output and Kubernetes container logs are recognised. Files are read as a stream in a background worker, so multi-gigabyte logs and .gz files work; nothing is uploaded.

Without a pass the result is a free preview: every count, the first templates of each list (up to 10) and the timeline under a mark. The full lists, the CSV and copying unlock with a Premium pass.

Common uses

• Finding the handful of errors that matter in a log of millions of lines.
• Checking what a release changed: new error messages, messages that stopped, rates that jumped.
• Finding the minute an incident started and the messages that came with it.
• Choosing which messages to alert on, or which noisy messages to silence.

How the grouping works

Variable parts are replaced first: numbers by <NUM>, IPv4 and IPv6 addresses by <IP>, UUIDs by <UUID>, hex ids by <HEX>, times by <TIME>, e-mail addresses and URLs by <EMAIL> and <URL>.

Then Drain (He, Zhu, Zheng and Lyu, “Drain: An Online Log Parsing Approach with Fixed Depth Tree”, IEEE ICWS) looks the message up in a tree: first by its number of words, then by its first word (a word with a digit takes a wildcard branch); each level of depth above 4 adds the next word. In that leaf it joins the most similar template when the share of equal words reaches the similarity (0.4 by default); the words that differ become <*>. Otherwise the message starts a new template. The grouping follows Drain3, with its defaults: similarity 0.4, depth 4 and 100 children per node.

Because the first word chooses the branch, two lines whose first word differs (alice logged in, bob logged in) stay apart; mask such parts, or set the depth to 3, where only the number of words and the similarity count.

Spikes and comparisons

• The timeline divides the time span into at most 72 bars of a round width (a second up to a week). A bar is a spike when it is at least three times the median bar, at least 5 entries, and above the median by six times the median absolute deviation; the same is done for errors alone.
• In a comparison, rates are per 10,000 entries of each log, so logs of different lengths can be compared. “Much more or less frequent” lists templates whose rate changed by a factor of three or more (with at least 5 lines in one of the logs).
• Times are taken as written in the log; times without a zone are read as UTC, and syslog times without a year are shown without one.

Limitations

• Files are read as UTF-8; logs in UTF-16 or another encoding should be converted first. Only .gz compression is read; unpack .zip or .bz2 archives first.
• Grouping is a heuristic: some templates may join two kinds of message or split one. Adjust the similarity and the placeholders, and check the examples.
• Lines longer than 20,000 characters are cut for grouping. Up to 3 examples are kept per template.
• Speed depends on the device and on the lines: a laptop reads about two hundred thousand lines a second or more, so a gigabyte of logs takes roughly half a minute to a minute, and a phone takes longer.

Privacy

Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot.