How to use AI to surface the root cause of intermittent failures while keeping the first‑failure evidence intact and never letting a self‑healing step hide a real product defect.
1. Introduction
Flaky tests—tests that sometimes pass and sometimes fail without any code change—are a silent productivity killer. In a CI pipeline they can turn a genuine defect into a “green” build, eroding trust in automation and forcing developers to ignore test results. The Playwright test runner offers built‑in retry support and rich tracing capabilities, but those features are often used as a blunt “make‑it‑green” button.
This guide shows how to turn Playwright’s retry and trace mechanisms into a diagnostic signal instead of a status‑masking shortcut, and how to augment the data with AI‑driven analysis while keeping a human reviewer in the loop. The workflow is risk‑aware, evidence‑based, and compatible with everyday CI configurations.
2. Why Flaky Tests Are Dangerous
- Intermittent failures hide real defects. When a test fails once and then passes on a retry, the CI job finishes green, but the original failure may have been caused by a genuine bug (e.g., a race condition in the UI).
- Teams lose confidence in the test suite. If developers see “flaky” warnings repeatedly, they start treating all failures as noise and stop investigating.
- Flakiness inflates maintenance cost. Every flaky test needs a triage ticket, a root‑cause analysis, and possibly a redesign of the locator or test data setup.
The most common sources of flakiness in Playwright are documented in the Playwright community and align with the findings of Hari Gokul’s article:
| Category | Typical symptom | Example in Playwright |
|---|---|---|
| Timing / race conditions | Element not ready, network latency spikes | await page.click('#submit') fails intermittently |
| Brittle selectors | DOM changes break CSS/XPath locators | page.locator('text=Login') drifts after UI redesign |
| Network instability | External API timeouts | Test fails when a third‑party service is slow |
| Shared state | Tests pollute a global DB row or storage state | Two parallel workers write to the same file |
| Environment variance | Different OS fonts, headless vs headed rendering | Visual regression passes locally but fails on CI |
If the CI pipeline simply retries the test and marks it as passed, the first‑failure evidence (browser console, network logs, screenshots) disappears, making it impossible to decide whether the failure was a harmless glitch or a real defect.
3. Playwright Retries – From “Make‑It‑Green” to Diagnostic Signal
Playwright retries are disabled by default. When retries are enabled, the runner classifies a test as passed when it succeeds on the first attempt, flaky when it fails first and later passes, and failed when all attempts fail. That classification must remain visible in CI; a retry must never silently convert a flaky result into an ordinary pass.
Trace collection is a deliberate trade-off:
| Trace mode | What it records | Triage use |
|---|---|---|
retain-on-failure | Records each attempt and keeps traces for failed attempts | Strongest choice when the initial failure is the primary evidence |
on-first-retry | Records the first retry, not the original attempt | Useful for examining the fresh retry context |
on-all-retries | Records every retry attempt | Useful for repeated comparison, but creates more artifacts |
No single mode captures both the original attempt and every successful retry. Choose the evidence you need, and supplement it with screenshots, video, console logs, network records and the HTML report.
Recommended baseline configuration for CI
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: process.env.CI ? 2 : 0,
reporter: [['list'], ['html', { open: 'never' }]],
use: {
trace: 'retain-on-failure',
screenshot: 'only-on-failure',
video: 'retain-on-failure',
},
workers: process.env.CI ? 1 : undefined,
maxFailures: 20,
});This configuration preserves failed-attempt evidence and keeps retries visible in the report. The values 2, 1 and 20 are operational examples, not universal Playwright recommendations; teams should tune them using runtime, infrastructure and risk data.
4. Preserving the First‑Failure Evidence
Diagnostics must be enabled before the test runs. Starting a trace in afterEach is too late to reconstruct earlier browser actions. Built-in screenshots, videos and traces cover most cases; a small automatic fixture can add console evidence:
// fixtures.ts
import { test as base } from '@playwright/test';
export const test = base.extend<{ consoleEvidence: void }>({
consoleEvidence: [async ({ page }, use, testInfo) => {
const messages: string[] = [];
page.on('console', message => {
messages.push(`[${message.type()}] ${message.text()}`);
});
await use();
if (testInfo.status !== testInfo.expectedStatus) {
await testInfo.attach('console.log', {
body: Buffer.from(messages.join('\n'), 'utf-8'),
contentType: 'text/plain',
});
}
}, { auto: true }],
});
export { expect } from '@playwright/test';With trace: 'retain-on-failure', the first failed attempt keeps its trace. The screenshot, video, console attachment and reporter output form one evidence bundle. If the team needs a trace of the retry instead, use on-first-retry and document that the original attempt will be represented by the other artifacts.
5. AI‑Assisted Triage Workflow
Below is a risk‑based, human‑in‑the‑loop workflow that leverages AI to surface the most likely root cause while ensuring that no real defect is automatically quarantined.
┌─────────────────────┐
│ Test finishes (CI) │
└───────┬─────────────┘
│
▼
┌─────────────────────┐
│ Did the test retry? │───── No ──► Mark as PASS/FAIL (no AI needed)
└───────┬─────────────┘
│ Yes
▼
┌─────────────────────┐
│ Collect artifacts: │
│ – retained failure trace│
│ – retry result/report │
│ – screenshots, logs │
│ – network HAR (if enabled)│
└───────┬─────────────┘
│
▼
┌─────────────────────┐
│ Feed artifacts to AI│
│ (LLM or specialized│
│ log‑analysis model)│
└───────┬─────────────┘
│
▼
┌─────────────────────┐
│ AI returns: │
│ • Probable cause │
│ • Confidence score│
│ • Suggested action│
└───────┬─────────────┘
│
▼
┌─────────────────────┐
│ Human reviewer checks│
│ – confidence threshold │
│ – evidence checklist │
│ – risk impact │
└───────┬─────────────┘
│
┌────┴─────┐
│ │
▼ ▼
Quarantine Fix / Refactor
(only if (only after human sign‑off)
human confirms)5.1 Evidence Checklist for Human Review
| Item | Why it matters |
|---|---|
| Retained failed-attempt trace | Shows browser actions and state for the failed attempt when retain-on-failure is used. |
| Retry result and report entry | Confirms whether a later attempt passed and keeps the result classified as flaky. |
| Console & network logs | Identify timeout errors, 5xx responses, or JS exceptions. |
| Screenshot diff (if visual comparison is enabled) | Detects rendering differences caused by headless vs headed mode. |
| Historical run data (last 10 executions) | Helps the AI compute a flakiness score (e.g., % of runs that needed a retry). |
| Test metadata (retry count, worker index) | Rules out shared‑state problems. |
The reviewer must answer “Do we have enough evidence that the failure was caused by test‑side brittleness rather than a product defect?” before any self‑healing or selector change is applied.
5.2 Measurable Flakiness Indicators
| Indicator | Calculation | Interpretation |
|---|---|---|
| Flake rate | # tests classified as flaky / # tests executed | Track by suite, branch and environment; investigate sustained increases. |
| Retry-pass ratio | # tests that passed on retry / # tests retried | A rising ratio may indicate growing dependence on retries. |
| Time-to-failure distribution | Duration from test start to failure across runs | Compare patterns; duration alone does not prove a root cause. |
| AI confidence | Model-specific score plus cited evidence | Use only to prioritize human review, never as permission to auto-quarantine. |
These metrics can be plotted in CI dashboards to spot trends early.
6. Risk‑Based Triage Decision Matrix
| Risk level | Evidence pattern | Required action |
|---|---|---|
| Low | Reproducible test-side issue with clear evidence and no product impact | Open a maintenance task; a reviewer may approve a test-only fix. |
| Medium | Mixed evidence, recurring retries, or environment dependence | Create a triage ticket, retain artifacts and reproduce under controlled conditions; do not auto-quarantine. |
| High | Possible product defect, security/data impact, or unexplained intermittent behavior | Treat as a defect until disproved; block automated self-healing and require domain-owner review. |
The matrix intentionally contains no universal percentage or confidence threshold. Each team should define thresholds from its own baseline, but uncertainty and high impact always move the decision toward human investigation.
7. Safe Automation Boundaries
| Boundary | Rationale |
|---|---|
| Retries are explicitly bounded | More retries increase cost and can normalize instability; the exact limit is a team decision. |
| Trace mode is chosen deliberately | retain-on-failure prioritizes the failed attempt; retry-only modes answer a different question. |
| Worker count is controlled | Running with one worker can help diagnose resource pressure or shared-state coupling, but it does not fix isolation defects. |
| No automatic selector rewriting | Self-healing must be reviewed; otherwise a broken UI could go unnoticed. |
| Artifact retention is policy-driven | Keep evidence long enough for triage while respecting storage, privacy and security constraints. |
8. Concrete Playwright Configuration Examples
8.1 CI‑only configuration (environment variable CI=true)
// playwright.config.ci.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: 2,
workers: 1,
maxFailures: 20,
use: {
trace: 'retain-on-failure',
screenshot: 'only-on-failure',
video: 'retain-on-failure',
launchOptions: { headless: true },
},
reporter: [
['list'],
['html', { open: 'never', outputFolder: 'playwright-report' }],
],
});8.2 Local developer configuration (no retries, full tracing)
// playwright.config.local.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: 0,
workers: undefined, // default based on CPU cores
use: {
trace: 'on', // full local trace; expensive for CI
screenshot: 'on',
video: 'on',
},
});Switching between the two files can be done with an npm script:
{
"scripts": {
"test:ci": "playwright test --config=playwright.config.ci.ts",
"test:local": "playwright test --config=playwright.config.local.ts"
}
}9. Human‑In‑The‑Loop Guardrails
- Never merge a PR that contains a quarantined test without a reviewer’s explicit approval.
- Require a “triage comment” that cites at least one artifact (trace, log, screenshot) supporting the decision.
- Log every AI suggestion in the CI job summary so the team can audit false positives/negatives.
- Schedule a periodic review (e.g., every sprint) of the flake‑rate dashboard to adjust retry limits or improve test design.
These steps keep the process transparent and prevent the “self‑healing” trap where every flaky test is silently patched by an AI without human validation.
10. Five Concrete Questions When Evaluating an AI Agent for Flaky‑Test Triage
- What provenance does the AI use for its conclusions? Verify whether it reads raw Playwright traces, console logs, network evidence and historical runs or only a pre-processed summary.
- How does the model express uncertainty? Require calibrated uncertainty plus cited evidence; a confidence number alone is not proof.
- Can the AI identify the exact step that differed between failure and retry? The answer should reference a concrete action or artifact, not merely say “timing issue.”
- What is the audit trail for a suggested self-healing action? Record the original locator or behavior, proposed change and supporting evidence.
- How does the AI handle unknown cases? It must route them to manual review rather than defaulting to quarantine or an automatic fix.
These questions help QA leaders ensure that the AI behaves as a decision‑support tool, not a decision‑maker.
11. Where Automation Helps
- Rapid data collection – Playwright’s built‑in tracing, screenshots, and HAR capture provide a deterministic snapshot that would be tedious to gather manually.
- Pattern recognition – Large language models excel at correlating error messages across many runs, surfacing recurring network timeouts or selector drift that humans might miss.
- Prioritization – By scoring flake probability, AI can surface the most dangerous flaky tests first, allowing the team to focus limited manual effort where it matters most.
- Consistent reporting – Automated CI jobs produce uniform artifact bundles, making post‑mortem analysis reproducible.
Automation is therefore a force multiplier: it supplies the evidence, the AI helps interpret it, and the human reviewer makes the final call.
12. Conclusion
Flaky Playwright tests do not have to become a hidden threat to product quality. By configuring retries deliberately, capturing the first‑failure state, and feeding all artifacts to an AI‑assisted triage pipeline, teams can distinguish true defects from test‑code brittleness. The workflow described here preserves the critical evidence, quantifies flakiness, and enforces a risk‑based decision matrix that always requires human approval before any self‑healing or quarantine action. When used responsibly, automation and AI become powerful allies in maintaining a trustworthy test suite while keeping manual testing and exploratory insight at the heart of quality assurance.
Sources
- Hari Gokul — Why Flaky Tests Are More Dangerous Than Failed Tests
- Playwright documentation — Retries
- Playwright documentation — Trace Viewer
- Playwright documentation — Reporters
Only the seed article and official Playwright documentation are used for factual claims.
Leave a comment
Share your thoughts about the article. No account is required.
Comments
Loading comments…