How to turn raw Playwright CI data into a reliable first‑pass classification before involving an AI assistant.


1. Why a deterministic triage layer matters

Flaky tests waste developer time, hide real regressions, and erode confidence in the CI pipeline. The temptation is to hand every ambiguous failure over to an AI model that suggests locator changes or automatically disables the test. While AI can be a useful assistant, it must never replace a disciplined evidence‑gathering step. A deterministic triage layer guarantees that the decision to involve a human—or an AI—rests on concrete, reproducible signals rather than on speculation.

Playwright’s built‑in retry support, trace viewer, screenshots, and artifact publishing (as described in the official documentation) give us exactly the data needed for that first pass. By extracting the retry count, the outcome of each attempt, the type of assertion that failed, and the environment metadata, we can separate high‑confidence infrastructure or product failures from cases that remain ambiguous.


2. A concrete CI scenario

Consider a GitHub Actions workflow that runs Playwright tests on a matrix of browsers. The workflow publishes the following artifacts for every run:

When a test fails, the workflow automatically uploads these artifacts as part of the run summary, making them instantly available for downstream analysis.


3. Step‑by‑step deterministic triage

StepWhat we captureWhy it matters
1. Detect failurePlaywright marks the test as failed in the JSON report.Entry point for triage logic.
2. Record attempt numbertestInfo.retry (0 for first attempt, 1 for first retry, …).Determines whether the failure is a first‑attempt problem or a retry‑only anomaly.
3. Capture retry outcomeSuccess or failure of the subsequent attempt(s).A retry that passes does not guarantee the original failure was harmless; it only shows that the test can succeed under slightly different timing or resource conditions.
4. Classify assertionIdentify the exception type (AssertionError, TimeoutError, network‑related error, etc.).Certain categories (e.g., network timeouts) are more likely to be infrastructure‑related, while others (e.g., mismatched UI text) may indicate a product change.
5. Attach trace & screenshotUse testInfo.attachments to store the trace file and the failure screenshot.Provides deterministic visual evidence that can be inspected without rerunning the test.
6. Record environment metadataOS, browser version, CI runner ID, commit SHA, and any feature flags.Helps spot patterns such as “fails only on Windows Chrome 115”.
7. Apply deterministic rulesA small, unit‑testable function evaluates the collected data and returns one of three labels: Infrastructure, Product, Ambiguous.The rule set is transparent, version‑controlled, and can be reviewed by the team.
8. Route the resultInfrastructure → auto‑create a ticket for the infra team. <br>Product → auto‑create a defect ticket with the trace attached. <br>Ambiguous → flag for human review or optional AI assistance.Guarantees that only high‑confidence cases are acted upon automatically.

The flow stops at step 7 for Infrastructure and Product classifications. Only when the rule set cannot reach a confident decision do we involve a human or an AI assistant.


4. TypeScript example – attaching deterministic evidence

import { test, expect, TestInfo } from '@playwright/test';

// Helper that builds a structured evidence object
function buildEvidence(info: TestInfo, error: Error) {
  const evidence = {
    retry: info.retry,                     // attempt number
    outcome: info.status,                  // "failed" or "passed"
    assertion: error.name,                 // e.g., "AssertionError"
    message: error.message,
    tracePath: info.attachments.find(a => a.name === 'trace')?.path,
    screenshotPath: info.attachments.find(a => a.name === 'screenshot')?.path,
    env: {
      os: process.platform,
      browser: info.project.name,
      commit: process.env.GITHUB_SHA,
      runId: process.env.GITHUB_RUN_ID,
    },
  };
  // Persist as JSON artifact for later rule evaluation
  info.attach('triage-evidence', {
    body: JSON.stringify(evidence, null, 2),
    contentType: 'application/json',
  });
}

// Example test that records evidence on failure
test('checkout flow shows correct total', async ({ page }, testInfo) => {
  try {
    await page.goto('https://example.com/checkout');
    await expect(page.locator('#total')).toHaveText('$42.00');
  } catch (err) {
    // Capture trace and screenshot before rethrowing
    await testInfo.attach('trace', { path: testInfo.outputPath('trace.zip'), contentType: 'application/zip' });
    await testInfo.attach('screenshot', { path: testInfo.outputPath('failure.png'), contentType: 'image/png' });
    buildEvidence(testInfo, err as Error);
    throw err; // Let Playwright mark the test as failed
  }
});

Key points


5. Deterministic classification rules (unit‑testable)

type Evidence = {
  retry: number;
  outcome: 'failed' | 'passed';
  assertion: string;
  env: { os: string; browser: string };
};

export function classify(e: Evidence): 'Infrastructure' | 'Product' | 'Ambiguous' {
  // Rule 1 – Infrastructure timeout on first attempt
  if (e.retry === 0 && e.assertion === 'TimeoutError') {
    return 'Infrastructure';
  }

  // Rule 2 – Assertion error that passes on retry → ambiguous
  if (e.retry > 0 && e.assertion === 'AssertionError' && e.outcome === 'passed') {
    return 'Ambiguous';
  }

  // Rule 3 – Consistent assertion error across all attempts → product defect
  if (e.retry === 0 && e.assertion === 'AssertionError') {
    return 'Product';
  }

  // Rule 4 – Known flaky pattern (e.g., Chrome on Linux) – treat as infrastructure
  if (e.env.browser.includes('chrome') && e.env.os === 'linux' && e.assertion === 'NetworkError') {
    return 'Infrastructure';
  }

  // Default – we cannot decide with confidence
  return 'Ambiguous';
}

Each rule is deliberately simple, making it easy to write unit tests that verify the expected label for a given evidence payload. The rule set can evolve as the team discovers new patterns, but any change must be reviewed and version‑controlled.


6. Why a “retry‑passed” result is not proof of harmless flakiness

Playwright’s retry mechanism reruns the whole test (or the failing step, depending on configuration). A passed retry shows that the test can succeed under a slightly different timing or resource snapshot, but it does not explain why the first attempt failed. Common hidden causes include:

If we automatically label such cases as “harmless”, we risk missing a regression that only appears under a specific load pattern. The deterministic evidence (trace, screenshot, retry count) lets a human verify whether the retry success is truly benign or whether it masks a deeper issue.


7. Why an AssertionError is not automatically a product defect

An AssertionError simply tells us that an expectation in the test did not match the observed value. The mismatch can stem from:

Only when the evidence shows a consistent mismatch across all attempts and environments should we treat it as a product defect. Otherwise, the failure belongs to the infrastructure or flaky‑test bucket, and the appropriate response is a retry‑policy adjustment or a test‑stability improvement, not a defect ticket.


8. Safe escalation to AI – what an assistant may (and may not) do

When the deterministic classifier returns Ambiguous, a team may choose to involve an AI assistant for suggestions. The scope of that assistance must be tightly bounded:

Allowed AI actionsProhibited AI actions
Propose a new locator based on the trace screenshot.Directly modify the test file without human approval.
Summarize the trace steps that led to the failure.Suppress the flaky test or mark it as “skip”.
Highlight missing network requests that could explain a timeout.Generate a defect description that is automatically filed without review.

The rationale is simple: an AI model does not have the contextual knowledge of product intent, release schedule, or risk tolerance. It can surface clues, but the final decision—whether to change a selector, adjust a timeout, or open a defect—must remain a human judgment.


9. Where automation helps

Automation stops short of decision‑making; it supplies the facts that enable a quick, reliable human or AI‑assisted review.


10. Review questions

  1. Which Playwright property tells you the current retry attempt, and why is it essential for deterministic triage?
  2. Explain why a test that passes on a retry should still be examined before being declared “flaky but harmless”.
  3. List two assertion categories that are more likely to indicate infrastructure problems rather than product defects.
  4. What are the three possible classification outcomes of the deterministic rule engine, and what action does each trigger?
  5. Identify one AI‑suggested action that is permissible in the “Ambiguous” bucket and one that must be prohibited.

11. Conclusion

Flaky tests are inevitable, but the way we handle them determines whether they become a productivity drain or a manageable signal. By leveraging Playwright’s retry metadata, trace artifacts, and structured attachments, we can build a deterministic triage layer that reliably separates high‑confidence infrastructure or product failures from truly ambiguous cases. Only the latter should be escalated to a human reviewer or an AI assistant, and even then the AI’s role must be limited to suggestion, not execution. This disciplined approach preserves test integrity, keeps the CI pipeline trustworthy, and ensures that automation remains a helper rather than an unchecked decision‑maker.


Sources