Grok Pattern Debugger (Logstash and Ingest Pipelines)
Build and debug grok patterns against real log lines, with the full Logstash library.
Free preview.
- Free preview: your pattern run on every sample line with its fields and failures under a MySmartCoPilot mark, and the first lines of each export (up to 20).
- Locked until you unlock it: download and copy.
- Unlock: Pro pass, ₹179 for 30 days, a one-time payment that never renews.
Ways to unlock shows how to get the full result.
Printing this result is locked in the free preview.
%{PATTERN:field}, %{PATTERN:field:int} or (?<field>regex). Field names may be references such as [source][address].
One event per line (up to 500 lines are tested).
Custom patterns
They work like pattern_definitions: they can use the library and replace a built-in pattern of the same name.
Pattern library
Results
Event of the first matching line
Use it
Converted for the browser
The browser’s JavaScript engine runs the pattern here; these Oniguruma features were rewritten so they match the same text.
Locked in the free preview. Opens the ways to unlock this result.
Locked in the free preview. Batch runs unlock with a pass.
Locked in the free preview. Query results unlock with a pass.
About the Grok Pattern Debugger (Logstash and Ingest Pipelines)
Write a grok pattern, paste a few log lines and see at once what each line becomes: every named field with its value and type, highlighted in the line. A line that does not match shows how far the pattern got and the part where it stopped, so you fix the right piece instead of guessing.
The complete pattern library of Logstash (logstash-patterns-core) is built in, in its legacy and ECS v1 versions: COMBINEDAPACHELOG, SYSLOGLINE, TIMESTAMP_ISO8601, HAPROXYHTTP, JAVASTACKTRACEPART and the rest. Add your own patterns, let Suggest a pattern propose a start from your lines, and export the result as a Logstash grok filter, an Elasticsearch ingest pipeline or a _simulate request (with a Pro pass).
Everything runs in your browser: your log lines are never uploaded, which matters when they hold IP addresses, user names or tokens.
Without a pass the result is a free preview: the pattern runs on every line in full under a mark, and each export shows its first lines (up to 20). The whole exports, copying and downloading unlock with a Pro pass.
How to use it
- Paste a few sample lines from your log, one event per line (or load a log file: its first lines are used).
- Type the grok pattern, or press Suggest a pattern: it recognises known formats such as Apache or syslog, or builds a pattern from timestamps, IP addresses, levels, numbers and words.
- Read the results: matching lines list their fields; failing lines show the matching part in green, then the part of the pattern that did not match. Add
:intor:floatto a field to convert it. - Choose Legacy or ECS v1 to match your Logstash or Elasticsearch setting. The Logstash filter, the ingest pipeline and the _simulate test copy and download with a Pro pass; without one the page shows their free preview.
Examples
Pattern: %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes:int} %{NUMBER:duration:float}
Line: 55.3.244.1 GET /index.html 15824 0.043{
"client": "55.3.244.1",
"method": "GET",
"request": "/index.html",
"bytes": 15824,
"duration": 0.043
}Same pattern Line: 10.0.0.3 GET index.html 200 0.010
No match. The first 4 of 9 parts match (up to character 13). The next part, %{URIPATHPARAM:request}, does not match at “index.html 200 0.010”.URIPATHPARAM needs a leading slash: the client logged a relative path. %{NOTSPACE:request} accepts both.
Common uses
- Writing the grok filter of a new Logstash pipeline, or the grok processor of an Elasticsearch ingest pipeline, against real lines first.
- Finding out why events are tagged
_grokparsefailureafter a log format changed. - Moving a pipeline from legacy field names to ECS (
clientipbecomes[source][address]). - Turning application logs into structured fields before they reach a log platform.
Grok syntax in short
%{PATTERN}matches a library pattern without keeping it;%{PATTERN:field}keeps it as a field.%{PATTERN:field:int}and:floatconvert the value in Logstash; Elasticsearch’s processor also knows:long,:doubleand:boolean.- Fields can be references to nested fields:
%{IP:[source][address]}(Logstash), writtensource.addressin an ingest pipeline. (?<field>regex)captures with a plain regular expression, without a library pattern.- Custom patterns use the pattern-file format:
NAME regex, one per line, as in Logstash’spattern_definitionsorpatterns_dir.
The Logstash grok filter reference and the Elasticsearch grok processor reference describe every option.
Oniguruma patterns in a browser
Grok patterns are regular expressions in Oniguruma syntax (Ruby’s), which the browser’s JavaScript engine reads a little differently. The page converts them so they match the same text, and lists what it converted:
- atomic groups
(?>…)and possessive quantifiers (*+,++,?+), used byQUOTEDSTRINGandBASE10NUM, are rebuilt with a lookahead, which never gives text back; \h,\A,\z,\Z,\R,\x{…}, POSIX classes such as[[:alnum:]]and nested classes are written out;^and$match at every line end, as in Ruby;(?m)makes.match line breaks (Ruby’s meaning of m).
What JavaScript cannot do at all (\G, \K, conditionals, recursion) is refused with an explanation instead of giving different results.
Legacy or ECS patterns?
The library has two versions with the same pattern names. Legacy patterns name fields the old way (clientip, verb, response); ECS v1 patterns use the Elastic Common Schema ([source][address], [http][request][method], [http][response][status_code], typed as numbers). Logstash uses ECS patterns when ecs_compatibility is v1 or v8, and the Elasticsearch grok processor uses legacy patterns unless its ecs_compatibility is v1. The library is that of logstash-patterns-core (Apache License 2.0).
Limitations
- One pattern is tested at a time. Logstash and Elasticsearch accept a list of patterns and use the first one that matches; test each of them here.
- Matching uses the browser’s regular-expression engine with the conversions above;
\sand\wthere follow JavaScript (for example,\salso matches Unicode spaces). - A pattern that runs longer than 2 seconds is stopped, like Logstash’s own timeout, but sooner.
- Up to 500 lines are tested and 100 are listed; Suggest looks at the first 50.
Privacy
Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot.
Frequently asked questions
What do I get without a pass?
Without a pass, Grok Pattern Debugger (Logstash and Ingest Pipelines) shows your pattern run on every sample line with its fields and failures under a MySmartCoPilot mark, and the first lines of each export (up to 20). Until you unlock it, the result can’t be downloaded or copied. A Pro, Premium or Ultimate pass, a one-time payment that never renews, unlocks the full result. The pricing page lists the passes and their prices.
Why does my pattern work here but not in Logstash, or the other way round?
Check three things: the pattern set (legacy or ECS v1, which changes field names and types), the escaping of quotes in the Logstash config (the exported filter handles it), and custom patterns your pipeline loads from patterns_dir that are not added here. Elasticsearch also fails a whole document when a :int value is not a whole number; choose Elasticsearch under Convert types like to see that.
What does _grokparsefailure mean?
Logstash adds the tag _grokparsefailure (and Elasticsearch fails the processor) when none of the patterns matches an event. Paste such lines here: the result shows the part of the pattern that stops matching.
Should I use DATA or GREEDYDATA?
GREEDYDATA (.*) takes as much as it can and suits the end of a line. DATA (.*?) takes as little as it can and suits a part followed by a fixed separator. Prefer specific patterns (NOTSPACE, WORD, INT) where you can: they are faster and fail clearly.
Does the debugger upload my logs?
No. Patterns, lines and the library all stay in your browser; matching runs in a background worker on your device.
Why is a field an array?
Logstash keeps every value of a field that is captured more than once in one match, as an array: %{WORD:w} %{WORD:w} on “alpha beta” gives w: ["alpha", "beta"]. Give the two parts different names to keep them apart.