Stabilization playbook

Status: Stable

Techniques that made a visual-regression suite reproducible, with the evidence behind each one. Every entry names the consumer it was proven in and the measurement that proved it. Read this before adding a mask, an ignore-list entry, or a wait.

The first question: does the fix change pixels?

Split every candidate stabiliser into two classes before considering it.

A timing stabiliser makes the capture wait longer for the page to reach the state it was always going to reach. Waiting for image decode, waiting for network idle, waiting for fonts. It moves the capture toward the settled render, but a longer wait also lets an animation the page never stops reach a later frame. A timing change therefore needs one run against the existing baseline before it ships.

A pixel-altering stabiliser makes the page render differently from the way a user sees it. image-rendering: pixelated, hiding an element, swapping an asset. It invalidates every existing baseline and puts a lie in the reference images, so it needs evidence that the instability is real and that no timing fix reaches it.

The library shipped one of each in consecutive candidates and conflated them. rc.6 set image-rendering: pixelated on every image to stop a scaler non-determinism that does not exist in Chromium 1055: six sequential captures of a 1600-to-1425 downscale, six of a 1220-to-740 downscale, six with the decode margin removed and eight concurrent full-page captures were all byte-identical. The property itself changed 367,580 pixels, 28.4% of a capture, at pixelmatch threshold 0. It was reverted in rc.7.

Measure before you attribute

A diff report names a file, not a cause. Three attributions in a component Storybook's investigation were wrong because they were read off filenames.

Localise the diff spatially first. Compute the bounding box of the differing pixels, then ask the DOM which element occupies that box. On pages-detail--report-page the box was x=194..1037 y=1034..1088, inside acme-highlight-box-grid — which identified the counter row in one step, after two wrong guesses from the story name.

Correlate across the whole run. A component Storybook's build 17 reported 3,658 failures in 12,288 captures. All 3,627 diff markers sat in image-bearing components (Detail page 430, Page intro 384, Home page 330, Image slideshow 265, Image 160); 2,080 captures of text-only components produced zero (Form components 0/1248, Table 0/160, CTA button 0/352, Accordion 0/224). A defect that partitions that cleanly has one cause, not eleven.

Check your instrument before believing a negative result. A throwaway shadow-DOM walk that stopped at depth 6 reported two distinct renderings where the shipped depth-10 walk found one. A local repro of the scaler claim used a browser that was not the CI browser, because Chromium 1055 does not launch on an arm64 Mac (spawn error -88) — run the experiment inside the CI image.

Compare against what the pipeline actually does. A determinism sweep that omits the config's maskSelectors measures raw rendering, not the suite. The same sweep reported one unstable story with masks omitted and nothing once they were applied.

Reduced motion reaches CSS, not JavaScript

The library sets the preference on every context and kills CSS animation unconditionally, in src/capture/screenshot.ts (preparePageForScreenshot) and at each context creation in src/core/browser-pool.ts and src/cli/executor.ts:

await page.addStyleTag({ content: `*, *::before, *::after {
  animation-duration: 0s !important; animation-delay: 0s !important;
  transition-duration: 0s !important; transition-delay: 0s !important; }` });
await page.emulateMedia({ reducedMotion: "reduce" });

That covers CSS animations and transitions and nothing else. A counter driven by requestAnimationFrame, an autoplaying video, a drag inertia loop and a SMIL animation inside a background-image SVG all ignore it, because none of them is a CSS animation.

The app has to read the preference for those. A CMS site does it with one helper and gates the JS at each site:

// frontend/source/assets/scripts/lib/prefers-reduced-motion.js
export const prefersReducedMotion = () => matchMedia('(prefers-reduced-motion)')?.matches || false;

// c-hero.js
this.playing = this.config.autoplay && !prefersReducedMotion();
startVideo() { if (prefersReducedMotion()) return; }

This is the preferred fix, because it is the accessibility behaviour the preference already asks for: the test runner gets determinism as a side effect of the app being correct, and no baseline records something a user never sees.

Two cases it does not reach. An animation isolated inside a background-image SVG cannot be gated from the host document. A CMS site swapped in a static per-theme SVG that shows a frozen frame, which keeps the affordance visible. And a component that freezes mid-animation rather than completing needs the end state forced, not the animation stopped: use-count-up with isCounting={onScreen} halts wherever it was when the element left the viewport, so three counters on one page of a component Storybook recorded 360/12/40 on one load and 65/2/7 on another — the same ~18% of each animation, frozen at a timing-dependent point. Waiting longer made it worse (4000ms settle gave 3 distinct captures of 4) because extra time only adds variance to when the freeze happened.

Masks

A Playwright mask paints over an element's bounding box. It stabilises content that changes inside a fixed box. It does not stabilise a box whose own size changes, because then the painted rectangle differs between runs and the masks become the diff.

Test the claim before recording it. A component Storybook's config carried a comment that said these masks did not work, over .report-number__counter; masking that selector gives one distinct capture in five, in every tenant theme tried. The evidence behind that comment concerned a different selector, .report-number-map__counter, whose box does change.

A CMS site made the same error in the other direction: masked regions came out magenta once the selectors reached the capture, so the option was never ignored — the selector list was being dropped upstream, which looked the same from outside.

When a mask appears not to work, confirm the selectors reach the capture before concluding anything about masking. Masked regions render magenta; if nothing is magenta, the list was dropped.

Layout and viewport

Width selects behaviour, not only type scale. A CMS site captured at 601px, which sat 0.02px above a CSS bucket edge and produced up to 46% diff against 600px; and at 768px, where the search-results component's initial-view check matched mq-xsmall through an indexOf('small') test and forced list view. Widths moved to 390 / 1024 / 1440 / 1920, each well inside its bucket. A component Storybook's config documents the margin beside each width for the same reason.

Images without intrinsic size shift the page during scroll. A CMS site added imgWidth/imgHeight to 121 image objects across 11 stories and rescoped a selector that had left <img> at height: 0 until load; target drift during scroll went from 2,807px to 0.

Per-run state

Clear web storage in an init script, before navigation, not after. A CMS site's search-results component persists grid-or-list under acme-navigation-history and reads it during its own init; with 32 parallel workers the story order varies per run, so the same story rendered grid in one run and list in the next — 52% diff, no code change.

Strip variable text out of error placeholders. Timestamps, retry counters, UUIDs and file positions in a failure image make every identical failure a fresh diff.

Pointer

Playwright starts the pointer at (0, 0) on every new page, on the element in the top-left corner. Headless Chromium applies no :hover state until a mouse event arrives. In Chromium 112 (Playwright 1.32.3), on Linux and on macOS, the top-left element of a test page stayed unhovered after navigation, a scroll down and back up, a viewport change and an element screenshot, in three runs of each. Eight captures through the CLI showed no hover either. One page.mouse.move(0, 0) hovered the element every time, and a move to (-1, -1) after it removed the hover every time.

A mouse event at the start position therefore hovers the top-left element. The runner moves the pointer to (-1, -1) after navigation and before the waits, so no element is under it, and the waits absorb the repaint that the move causes. In a headless run that sends no other mouse event, the move changes no pixels. A headed run was not measured: there the real pointer of the screen can also reach the page. Set stability.parkPointer: false to leave the pointer at (0, 0).

A hover state that flips between headless runs has a cause other than the pointer. One pointer hovers one chain of elements only, so a hover style on several sibling cards at once comes from somewhere else. Check these sources:

• A class that a scroll listener sets. The capture scrolls down and back up, and a header that turns opaque on scroll can stay opaque if its listener runs late (see stickyClassSelectors).
• A media query that gates the hover style, such as (hover: hover) and (pointer: fine). Log matchMedia(...).matches at capture time for each viewport.
• Focus inside the element, which :focus-within and :focus-visible styles show.

Retry

A retry captures the page again and compares the new capture; it never compares the same two images twice, because pixelmatch is deterministic. Timeouts, HTTP errors, capture failures and size mismatches are retried. A completed pixel comparison that differs is retried only where the project sets retries.onDiff, because a CMS site measured the cost of retrying every difference at 49 deterministic diffs times two retries times about 12 seconds. A component Storybook's Pages/Home "Interactive Home" differs on a different set of tenants in every run while two Storybook builds produce byte-identical captures of it, so the capture is the variable there and only a second capture settles it. The loop stops as soon as two attempts write the same bytes, so a deterministic difference costs one extra capture. A test that passes after a retry is reported as flaky, with its URL, so the fix starts from the report.

Artifact footprint

Archiving is part of stability, because a full controller fails builds that captured perfectly. A component Storybook's build 18 completed all 12,288 captures and then failed with No space left on device during archiveArtifacts: a run archives about 9GB onto a 40GB volume shared with every other job on the instance. Bound retention per job, and prefer serving images from somewhere other than the controller.