Skip to main content

Retries

Status: Stable

A test that does not pass is captured and compared again. The runner opens a fresh page, captures the story, and compares the new capture with the baseline; it never compares the same two images twice, because pixelmatch is deterministic and returns the same answer. A result is final when it passes. A dimension mismatch, a capture failure, an HTTP error and a page crash are captured again. A comparison that completed and differs is captured again only when retries.onDiff is on.

The loop stops as soon as two attempts write the same bytes. The two captures reproduce, so the difference is a real change and the remaining attempts return the same answer. A deterministic difference therefore costs one extra capture, not retries.default of them.

Configuration​

// .designTests.js
module.exports = {
  retries: {
    default: 2, // retries after the first attempt; 0 switches retries off
    delay: 0, // milliseconds to wait before each retry
    onDiff: false, // capture again after a comparison that differs
  },
};

retries.default is 2 and retries.onDiff is false when the block is absent. A manual page sets its own count beside expectStatus:

pages: [{ name: "Startseiten/Dach/Homepage DE", path: "/de/", retries: 0 }];

A Storybook story declares parameters.designTests.retries for the same purpose. The Storybook index carries no parameters, so the runner does not read this value from a Storybook today.

playwright.retries is rejected with a message naming retries.default. The library does not use Playwright Test, and the key named a count the parallel runner never read.

Cost​

A run in which every test passes takes one capture per test, because the loop returns after the first attempt. A retry costs one capture, so the extra work is proportional to the number of tests that did not pass.

With retries.onDiff on, a test whose difference is real costs one extra capture per run until its baseline is updated, because the second capture reproduces the first. A capture that varies between attempts costs up to retries.default extra captures and is reported as passed instead of as a difference.

Output​

One line per retry names the attempt and the outcome that caused it. The result line shows the retries the result used:

📸 🔄 [W04/32] [#125/1368] Retry 1/2 Desktop Seiten/Startseite "Interactive Experience Startseite"
   ↳ Diff: 147 pixels (0.49%)
📸 ✅  OK  [W08/32] [#125/1368] Desktop Seiten/Startseite "Interactive Experience Startseite" (7321ms, 1 retry)

A test that passed after at least one retry is flaky. Its baseline is confirmed; its capture is not reproducible on the first attempt, so it is a candidate for a stabilisation fix. The summary line adds N flaky when N is above 0, and the URL list after the summary lists every flaky test with its public URL under Flaky (passed after retry).

manifest.json carries retryCount on every captured item and summary.flaky. Flaky items stay in passed.

Exit codes​

A flaky test is a passed test. With --fail-on-changes, a difference that remains after the last attempt exits 1 and a capture failure that remains exits 2. Without the flag the run exits 0 once the report is written.

Error screenshots​

The parallel runner writes an error screenshot after the last attempt of a capture that never produced a screenshot, not after every attempt. The screenshot carries no timestamp, so the same error produces the same PNG (src/error-handling/error-screenshot.ts).

Implementation​

  • src/cli/retry.ts — runWithRetries, isFinal, resolveRetryPolicy, fingerprintActual
  • src/cli/executor.ts — both execution paths call runWithRetries with an attempt that opens a fresh page
  • src/config/schema.ts — retries, the rejected playwright.retries, pages[].retries
  • src/core/test-matrix.ts — copies parameters.designTests.retries into spec.maxRetries
  • src/report/manifest.ts — retryCount, summary.flaky