How to use AI to surface the root cause of intermittent failures while keeping the first‑failure evidence intact and never letting a self‑healing step hide a real product defect.


1. Introduction

Flaky tests—tests that sometimes pass and sometimes fail without any code change—are a silent productivity killer. In a CI pipeline they can turn a genuine defect into a “green” build, eroding trust in automation and forcing developers to ignore test results. The Playwright test runner offers built‑in retry support and rich tracing capabilities, but those features are often used as a blunt “make‑it‑green” button.

This guide shows how to turn Playwright’s retry and trace mechanisms into a diagnostic signal instead of a status‑masking shortcut, and how to augment the data with AI‑driven analysis while keeping a human reviewer in the loop. The workflow is risk‑aware, evidence‑based, and compatible with everyday CI configurations.


2. Why Flaky Tests Are Dangerous

The most common sources of flakiness in Playwright are documented in the Playwright community and align with the findings of Hari Gokul’s article:

CategoryTypical symptomExample in Playwright
Timing / race conditionsElement not ready, network latency spikesawait page.click('#submit') fails intermittently
Brittle selectorsDOM changes break CSS/XPath locatorspage.locator('text=Login') drifts after UI redesign
Network instabilityExternal API timeoutsTest fails when a third‑party service is slow
Shared stateTests pollute a global DB row or storage stateTwo parallel workers write to the same file
Environment varianceDifferent OS fonts, headless vs headed renderingVisual regression passes locally but fails on CI

If the CI pipeline simply retries the test and marks it as passed, the first‑failure evidence (browser console, network logs, screenshots) disappears, making it impossible to decide whether the failure was a harmless glitch or a real defect.


3. Playwright Retries – From “Make‑It‑Green” to Diagnostic Signal

Playwright retries are disabled by default. When retries are enabled, the runner classifies a test as passed when it succeeds on the first attempt, flaky when it fails first and later passes, and failed when all attempts fail. That classification must remain visible in CI; a retry must never silently convert a flaky result into an ordinary pass.

Trace collection is a deliberate trade-off:

Trace modeWhat it recordsTriage use
retain-on-failureRecords each attempt and keeps traces for failed attemptsStrongest choice when the initial failure is the primary evidence
on-first-retryRecords the first retry, not the original attemptUseful for examining the fresh retry context
on-all-retriesRecords every retry attemptUseful for repeated comparison, but creates more artifacts

No single mode captures both the original attempt and every successful retry. Choose the evidence you need, and supplement it with screenshots, video, console logs, network records and the HTML report.

Recommended baseline configuration for CI

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [['list'], ['html', { open: 'never' }]],
  use: {
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  workers: process.env.CI ? 1 : undefined,
  maxFailures: 20,
});

This configuration preserves failed-attempt evidence and keeps retries visible in the report. The values 2, 1 and 20 are operational examples, not universal Playwright recommendations; teams should tune them using runtime, infrastructure and risk data.


4. Preserving the First‑Failure Evidence

Diagnostics must be enabled before the test runs. Starting a trace in afterEach is too late to reconstruct earlier browser actions. Built-in screenshots, videos and traces cover most cases; a small automatic fixture can add console evidence:

// fixtures.ts
import { test as base } from '@playwright/test';

export const test = base.extend<{ consoleEvidence: void }>({
  consoleEvidence: [async ({ page }, use, testInfo) => {
    const messages: string[] = [];
    page.on('console', message => {
      messages.push(`[${message.type()}] ${message.text()}`);
    });

    await use();

    if (testInfo.status !== testInfo.expectedStatus) {
      await testInfo.attach('console.log', {
        body: Buffer.from(messages.join('\n'), 'utf-8'),
        contentType: 'text/plain',
      });
    }
  }, { auto: true }],
});

export { expect } from '@playwright/test';

With trace: 'retain-on-failure', the first failed attempt keeps its trace. The screenshot, video, console attachment and reporter output form one evidence bundle. If the team needs a trace of the retry instead, use on-first-retry and document that the original attempt will be represented by the other artifacts.


5. AI‑Assisted Triage Workflow

Below is a risk‑based, human‑in‑the‑loop workflow that leverages AI to surface the most likely root cause while ensuring that no real defect is automatically quarantined.

┌─────────────────────┐
│ Test finishes (CI) │
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐
│ Did the test retry? │───── No ──► Mark as PASS/FAIL (no AI needed)
└───────┬─────────────┘
        │ Yes
        ▼
┌─────────────────────┐
│ Collect artifacts:  │
│  – retained failure trace│
│  – retry result/report   │
│  – screenshots, logs    │
│  – network HAR (if enabled)│
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐
│ Feed artifacts to AI│
│ (LLM or specialized│
│  log‑analysis model)│
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐
│ AI returns:         │
│  • Probable cause   │
│  • Confidence score│
│  • Suggested action│
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐
│ Human reviewer checks│
│  – confidence threshold │
│  – evidence checklist   │
│  – risk impact          │
└───────┬─────────────┘
        │
   ┌────┴─────┐
   │          │
   ▼          ▼
Quarantine   Fix / Refactor
(only if   (only after human sign‑off)
human confirms)

5.1 Evidence Checklist for Human Review

ItemWhy it matters
Retained failed-attempt traceShows browser actions and state for the failed attempt when retain-on-failure is used.
Retry result and report entryConfirms whether a later attempt passed and keeps the result classified as flaky.
Console & network logsIdentify timeout errors, 5xx responses, or JS exceptions.
Screenshot diff (if visual comparison is enabled)Detects rendering differences caused by headless vs headed mode.
Historical run data (last 10 executions)Helps the AI compute a flakiness score (e.g., % of runs that needed a retry).
Test metadata (retry count, worker index)Rules out shared‑state problems.

The reviewer must answer “Do we have enough evidence that the failure was caused by test‑side brittleness rather than a product defect?” before any self‑healing or selector change is applied.

5.2 Measurable Flakiness Indicators

IndicatorCalculationInterpretation
Flake rate# tests classified as flaky / # tests executedTrack by suite, branch and environment; investigate sustained increases.
Retry-pass ratio# tests that passed on retry / # tests retriedA rising ratio may indicate growing dependence on retries.
Time-to-failure distributionDuration from test start to failure across runsCompare patterns; duration alone does not prove a root cause.
AI confidenceModel-specific score plus cited evidenceUse only to prioritize human review, never as permission to auto-quarantine.

These metrics can be plotted in CI dashboards to spot trends early.


6. Risk‑Based Triage Decision Matrix

Risk levelEvidence patternRequired action
LowReproducible test-side issue with clear evidence and no product impactOpen a maintenance task; a reviewer may approve a test-only fix.
MediumMixed evidence, recurring retries, or environment dependenceCreate a triage ticket, retain artifacts and reproduce under controlled conditions; do not auto-quarantine.
HighPossible product defect, security/data impact, or unexplained intermittent behaviorTreat as a defect until disproved; block automated self-healing and require domain-owner review.

The matrix intentionally contains no universal percentage or confidence threshold. Each team should define thresholds from its own baseline, but uncertainty and high impact always move the decision toward human investigation.


7. Safe Automation Boundaries

BoundaryRationale
Retries are explicitly boundedMore retries increase cost and can normalize instability; the exact limit is a team decision.
Trace mode is chosen deliberatelyretain-on-failure prioritizes the failed attempt; retry-only modes answer a different question.
Worker count is controlledRunning with one worker can help diagnose resource pressure or shared-state coupling, but it does not fix isolation defects.
No automatic selector rewritingSelf-healing must be reviewed; otherwise a broken UI could go unnoticed.
Artifact retention is policy-drivenKeep evidence long enough for triage while respecting storage, privacy and security constraints.

8. Concrete Playwright Configuration Examples

8.1 CI‑only configuration (environment variable CI=true)

// playwright.config.ci.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 2,
  workers: 1,
  maxFailures: 20,
  use: {
    trace: 'retain-on-failure',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    launchOptions: { headless: true },
  },
  reporter: [
    ['list'],
    ['html', { open: 'never', outputFolder: 'playwright-report' }],
  ],
});

8.2 Local developer configuration (no retries, full tracing)

// playwright.config.local.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 0,
  workers: undefined, // default based on CPU cores
  use: {
    trace: 'on', // full local trace; expensive for CI
    screenshot: 'on',
    video: 'on',
  },
});

Switching between the two files can be done with an npm script:

{
  "scripts": {
    "test:ci": "playwright test --config=playwright.config.ci.ts",
    "test:local": "playwright test --config=playwright.config.local.ts"
  }
}

9. Human‑In‑The‑Loop Guardrails

  1. Never merge a PR that contains a quarantined test without a reviewer’s explicit approval.
  2. Require a “triage comment” that cites at least one artifact (trace, log, screenshot) supporting the decision.
  3. Log every AI suggestion in the CI job summary so the team can audit false positives/negatives.
  4. Schedule a periodic review (e.g., every sprint) of the flake‑rate dashboard to adjust retry limits or improve test design.

These steps keep the process transparent and prevent the “self‑healing” trap where every flaky test is silently patched by an AI without human validation.


10. Five Concrete Questions When Evaluating an AI Agent for Flaky‑Test Triage

  1. What provenance does the AI use for its conclusions? Verify whether it reads raw Playwright traces, console logs, network evidence and historical runs or only a pre-processed summary.
  2. How does the model express uncertainty? Require calibrated uncertainty plus cited evidence; a confidence number alone is not proof.
  3. Can the AI identify the exact step that differed between failure and retry? The answer should reference a concrete action or artifact, not merely say “timing issue.”
  4. What is the audit trail for a suggested self-healing action? Record the original locator or behavior, proposed change and supporting evidence.
  5. How does the AI handle unknown cases? It must route them to manual review rather than defaulting to quarantine or an automatic fix.

These questions help QA leaders ensure that the AI behaves as a decision‑support tool, not a decision‑maker.


11. Where Automation Helps

Automation is therefore a force multiplier: it supplies the evidence, the AI helps interpret it, and the human reviewer makes the final call.


12. Conclusion

Flaky Playwright tests do not have to become a hidden threat to product quality. By configuring retries deliberately, capturing the first‑failure state, and feeding all artifacts to an AI‑assisted triage pipeline, teams can distinguish true defects from test‑code brittleness. The workflow described here preserves the critical evidence, quantifies flakiness, and enforces a risk‑based decision matrix that always requires human approval before any self‑healing or quarantine action. When used responsibly, automation and AI become powerful allies in maintaining a trustworthy test suite while keeping manual testing and exploratory insight at the heart of quality assurance.


Sources

Only the seed article and official Playwright documentation are used for factual claims.