Test Report Analyzer: Flaky and Slow Tests
Drop CI test results and see the flaky, slow and failing tests across runs.
Try before you buy.
- Free preview: every count of every run, with the first rows of each table (up to 10) — flaky, slowest and failing tests — and the trends under a MySmartCoPilot mark.
- Locked until you unlock it: download.
- Unlock: Pro pass, ₹179 for 30 days or ₹79 for 1 week, a one-time payment that never renews. Unlock: Pro pass, $5.99 for 30 days or $2.99 for 1 week, a one-time payment that never renews.
Ways to unlock shows how to get the full result.
Printing this result is locked in the free preview.
Test result files
Runs
| # | Run | Commit or label | Tests | Passed | Failed | Skipped | Pass rate | Duration |
|---|
Trends
Flaky tests
| Test | Flake rate | History | Why |
|---|
Failures by message
Slowest tests
| Test | Average | Slowest | Runs |
|---|
Slowest suites
| Suite | Tests | Time per run | Failures |
|---|
Since the previous run
New failures
Fixed
Locked in the free preview. Opens the ways to unlock this result.
Locked in the free preview. Batch runs unlock with a pass.
Locked in the free preview. Query results unlock with a pass.
About the Test Report Analyzer: Flaky and Slow Tests
Drop the test result files from one CI run or many. The analyzer reads JUnit XML (Maven Surefire, Gradle, pytest, Go, Jest-junit, Playwright and most CI tools), .NET TRX, NUnit XML, Playwright and Jest or Vitest JSON, and Cypress’s mochawesome JSON, and shows each run’s pass rate, the slowest tests and suites, failures grouped by their message, and the flaky tests, ranked by how often they misbehave.
A test counts as flaky when it passed and failed on the same commit, passed only after a retry, or kept changing between passed and failed across runs; a test that broke once and stayed broken is listed as a new failure instead. Trends show the failures and duration of each run. Files are read by a background worker, a broken XML file is reported with its name and line, and nothing is uploaded.
Without a pass the result is a free preview: every count of every run, and the first rows of each table. The full tables and the HTML, PDF and CSV reports unlock with a Pro pass.
How to use it
- Choose or drop the result files: several files of one run (one per test class, as Surefire writes them) are joined automatically, and each repeat of the same tests starts the next run.
- Check the runs table: the order, the files of each run and the commit (read from the file when it names one; type it in when it does not, to find tests that fail and pass on the same commit).
- Read the flaky tests, the failures grouped by message, the slowest tests and suites, and what changed since the previous run.
- Download the HTML or PDF report and the CSV of flaky tests to quarantine them, with a Pro pass; the free preview shows the counts and the first rows.
Examples
<testcase name="retriesNetwork" classname="com.acme.CartTest" time="0.3"> <flakyFailure message="timeout after 5000 ms"/> </testcase>
Passed, and flaky: passed only after a retry in 1 run
6 JUnit XML files of an end-to-end suite on four commits
2 flaky tests (keepsCartAfterLogin, paysByCard), 1 new failure in the last run (filtersByPrice); failures grouped by 5 messages, “expected total <n> but was <n>” first
Common uses
- Finding the tests that make CI red at random, to fix or quarantine them first.
- Seeing which tests and suites take the most time in a slow pipeline.
- Spotting one root cause behind many failures: the same message in many tests.
- Checking what a merge broke or fixed since the previous run.
How tests are matched and when they count as flaky
A test is the same test in every run when its suite (class name, file or describe path, and the Playwright project) and its name match. Paths are compared without the folder that all test files share, so runs of one checkout meet; when runs come from two machines and share no folder at all, each run’s own shared folder is left out instead.
A test is flaky when at least one of these holds:
- it passed and failed on the same commit (the commit comes from Playwright’s git metadata or a JUnit property such as
git.commit, or you type it); - it passed only after a retry within a run (Playwright’s “flaky” status, Surefire’s
flakyFailure, Jest’s retries); - its result changed between passed and failed at least twice across the runs in order.
The flake rate is the share of the runs that ran the test in which it behaved flakily: a retry pass, a change from the run before, or a failure on a commit where it also passed. Skipped results do not count. A test that changed once (broke, or was fixed) is not flaky: it shows under Since the previous run when that happened in the last run.
Grouping failures
The first line of each failure message is reduced to its pattern: numbers become <n>, file paths <path>, ids and hashes <uuid> and <hex>, timestamps <time>. Failures with the same pattern are grouped, so “Timeout 5000ms exceeded.” and “Timeout 30000ms exceeded.” in different tests show as one cause, with the number of failures, tests and runs.
Formats
- JUnit XML:
<testsuites>or<testsuite>with<testcase>,<failure>,<error>,<skipped>; Surefire’sflakyFailure,flakyError,rerunFailure,rerunError; times in seconds. - TRX (Visual Studio,
dotnet test --logger trx):UnitTestResultwith its outcome and duration, including data-driven inner results. - NUnit 3 (
<test-run>) and 2 (<test-results>). - Playwright JSON reporter (reporters): suites, specs, projects and retries.
- Jest
--json(CLI) and Vitest’s JSON reporter, which writes the same shape. - mochawesome JSON, as Cypress writes it.
Limitations
- Runs are put in the order of the start times their files give, or of their file names; files without either are taken in the order chosen.
- Tests renamed or moved between runs count as two tests. Runs from two machines (a CI checkout and a local one) meet when each run holds the same test folders.
- Failure messages are kept to their first lines; full stack traces are not stored.
- Up to 200 MB of result files at a time; a large XML file takes a while on a phone.
Privacy
On this device: nothing is sent
Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot.
Frequently asked questions
What do I get without a pass?
Without a pass, Test Report Analyzer: Flaky and Slow Tests shows every count of every run, with the first rows of each table (up to 10) — flaky, slowest and failing tests — and the trends under a MySmartCoPilot mark. Until you unlock it, the result can’t be downloaded. A Pro, Premium or Ultimate pass, a one-time payment that never renews, unlocks the full result. The pricing page lists the passes and their prices.
Which test tools write files this analyzer reads?
Anything that writes JUnit XML (Maven Surefire and Failsafe, Gradle, pytest with --junitxml, Go with go-junit-report or gotestsum, PHPUnit, Jest with jest-junit, Playwright’s junit reporter, most CI systems), .NET TRX and NUnit XML, Playwright’s JSON reporter, jest --json, Vitest’s JSON reporter and Cypress with mochawesome.
Why is a test that failed once not listed as flaky?
One change between runs is usually a real break or a fix, not flakiness. It is listed under Since the previous run when it happened in the last run. A test becomes flaky when it passes and fails on the same commit, passes only after a retry, or keeps changing back and forth.
How does it know the commit of a run?
From Playwright’s JSON (its git metadata), or from a JUnit <property> whose name says commit, sha or revision and whose value is a commit hash. Otherwise type the commit, build number or any label into the runs table: runs with the same label count as the same code.
Are my results uploaded?
No. The files are read by a background worker in your browser; test names, messages and stack traces never leave your device.