Baseline Management

Status: Stable

This guide explains how to update, recreate, and manage baseline images for visual regression testing.

Understanding Baselines

Baseline images are reference screenshots stored in your repository. When you run tests, current screenshots are compared against these baselines to detect visual changes.

design-tests/
├── baseline-images/    # ✓ Committed to git (reference images + BASELINE.json)
├── actual-images/      # Generated (current screenshots)
├── diff-output/        # Generated (visual differences)
└── report/             # Generated (HTML report, reg.json, manifest.json, CURRENT.json)

design-tests run never writes into baseline-images/. A test without a baseline is reported as NEW and its screenshot stays in actual-images/; a changed test is reported as DIFF beside its diff image. Only design-tests baseline-update moves a run's screenshots into the baseline.

BASELINE.json sits beside the images and records what they are: the deployment reference they were captured from, and the capture platform — os, arch, browserVersion, playwrightVersion. baseline-update writes it from the run's CURRENT.json; run reads it and warns when the platforms differ.

Updating Baselines (After Intentional Changes)

When you make intentional design changes, you need to update the baseline to reflect the new expected state.

1. Review the Visual Differences

Run tests to see what changed:

pnpm exec design-tests run --config .designTests.js --all

Open the HTML report at design-tests/report/index.html to review all visual differences:

• Red areas: Pixels that changed
• Side-by-side view: Compare baseline vs. actual
• Slider view: Interactive comparison

2. Verify Changes are Intentional

Make sure the differences are expected:

• ✅ Intentional design updates
• ✅ New features or components
• ✅ Fixed bugs that changed appearance
• ❌ Unintended regressions
• ❌ Environmental differences (fonts, rendering) — the run prints ⚠️ Capture platform differs from the baseline when this is the cause

3. Update Baseline

If changes are intentional, promote the run's screenshots:

# See what would change
pnpm exec design-tests baseline-update --config .designTests.js --dry-run

# Promote the new and changed screenshots, write BASELINE.json
pnpm exec design-tests baseline-update --config .designTests.js

# Review what changed
git status
git diff --stat design-tests/baseline-images/

# Commit
git add design-tests/baseline-images/
git commit -m "Update baseline after button redesign"
git push

baseline-update reads report/manifest.json and copies its new and failed items. It skips errors items, the tests that produced no screenshot of the page, so an error screenshot cannot become a baseline. Baseline images the run reported as deleted stay in place unless --prune is passed.

A passed item keeps its baseline file, also when it passed after a retry. A pixel counts as changed only beyond the per-pixel tolerance (comparison.pixelmatch.threshold, default 0.1), and a capture passes while its changed pixels stay within the rate threshold (comparison.threshold). The file of a passing screenshot can therefore differ from its baseline file. In one nightly of the component Storybook, 12 passed captures had 6 to 52,513 pixels that differed from the baseline by at most 18 of 255 in a colour channel, all within the per-pixel tolerance. baseline-update leaves those baseline files unchanged, so the baseline changes only where the report shows a change.

4. CI/CD Workflow (Automatic PR)

In CI environments (like Jenkins), the pipeline:

1. Runs the tests
2. Reads reg.json or manifest.json and marks the build UNSTABLE when items are failed or new
3. Runs design-tests baseline-update --prune (an --all run, so pruning is safe), commits baseline-images/ to a baseline-update/<BUILD_NUMBER> branch and pushes it

Review and merge the PR to accept the changes. The pipeline runs git; baseline-update copies the files and checks the platform.

Do not empty baseline-images/ and copy actual-images/ over it. The run writes a screenshot for every capture, so such a copy changes more than the report lists:

• It changes every passed capture whose file differs from its baseline file. A whole-branch merge then carries changes the report calls passed, and the branch no longer matches the report.
• It promotes the error screenshots of errors items.
• It deletes the baselines of every project the run did not cover.
• It deletes BASELINE.json, so the next run cannot compare its platform with the baseline's.

Recreating Baselines

When to Recreate

Recreate baselines when:

• Upgrading Playwright (font rendering changes)
• Major browser version updates (Chrome, Firefox)
• Migrating from another testing framework
• Switching OS/architecture (macOS → Linux)
• Starting fresh after major refactoring

Each of these changes the capture platform recorded in BASELINE.json. baseline-update refuses to promote a run from a different platform or browser unless --allow-platform-change is passed. Pass it when recreating a baseline on a new platform.

How to Recreate

For a single project:

# Delete the project's baseline images
rm -rf design-tests/baseline-images/myproject/

# Run tests - every test of the project is reported as NEW
pnpm exec design-tests run --config .designTests.js --project myproject

# Promote the new screenshots
pnpm exec design-tests baseline-update --config .designTests.js --project myproject

# Review and commit
git add design-tests/baseline-images/
git commit -m "Recreate baseline for myproject"

For all projects:

# Delete all baselines
rm -rf design-tests/baseline-images/

# Run tests - all tests are reported as NEW
pnpm exec design-tests run --config .designTests.js --all

# Promote them; the platform record starts fresh
pnpm exec design-tests baseline-update --config .designTests.js

# Commit
git add design-tests/baseline-images/
git commit -m "Recreate all baselines"

Docker for Consistent Baselines

⚠️ Important: Always use Docker for generating/updating baselines to ensure consistency with CI.

Why Docker?

Font rendering and anti-aliasing differ between operating systems:

• macOS: Uses Apple's font rendering
• Linux (CI): Uses FreeType/fontconfig
• Windows: Uses ClearType

These differences cause pixel-level changes that fail tests.

What the CLI Enforces

• run prints 🖥️ Capture platform: … at the start and warns when BASELINE.json names another platform or browser build. The diffs that follow such a warning include rendering differences.
• baseline-update refuses a run whose platform differs from the baseline's, naming each differing field, unless --allow-platform-change is passed.
• A baseline without a platform record (written before this existed) accepts any run and gains the record on the next baseline-update.

Docker Usage

# Generate baseline in Docker (Linux x86_64)
docker compose run --rm design-tests

# Local debugging only (not for baseline generation)
pnpm exec design-tests run --config .designTests.js --all

Good workflow:

# Make changes
vim src/components/Button.tsx

# Capture in Docker, then promote
docker compose run --rm design-tests
docker compose run --rm design-tests pnpm exec design-tests baseline-update --config .designTests.js

# Commit
git add design-tests/baseline-images/
git commit -m "Update Button baseline"

Bad workflow:

# Capture natively on macOS
pnpm exec design-tests run --config .designTests.js --all
pnpm exec design-tests baseline-update --config .designTests.js
# 🚫 Baseline update refused: os: linux (baseline) → darwin (this run)

Partial Updates

Update only specific projects or images:

By Project

pnpm exec design-tests run --config .designTests.js --project myproject
pnpm exec design-tests baseline-update --config .designTests.js --project myproject

By Story Pattern

Not directly supported - use manual file operations:

# Run all tests
pnpm exec design-tests run --config .designTests.js --all

# Copy only button-related screenshots
find design-tests/actual-images -name "*button*" -exec \
  cp --parents {} design-tests/baseline-images/ \;

A manual copy leaves BASELINE.json untouched; the platform record still describes the images it was written for.

Handling Merge Conflicts

When multiple developers update baselines:

# Fetch latest
git pull origin develop

# Conflict in baseline images
# CONFLICT: design-tests/baseline-images/project/button-desktop.png

# Accept theirs or yours
git checkout --theirs design-tests/baseline-images/project/button-desktop.png

# Or regenerate
rm -rf design-tests/baseline-images/project/
pnpm exec design-tests run --config .designTests.js --project project
pnpm exec design-tests baseline-update --config .designTests.js --project project

# Commit resolution
git add design-tests/baseline-images/
git commit -m "Resolve baseline conflicts - regenerate"

Baseline Auditing

View Baseline History

See when and why baselines changed:

# Show baseline changes over time
git log --oneline design-tests/baseline-images/

# Show who changed specific baseline
git log --follow design-tests/baseline-images/project/button-desktop.png

# Show visual diff of baseline change
git difftool HEAD~1 HEAD -- design-tests/baseline-images/

# Show what the baseline was captured from and on
cat design-tests/baseline-images/BASELINE.json

Find Large Baselines

Identify oversized screenshots:

# Find baselines > 500KB
find design-tests/baseline-images/ -type f -size +500k

# Total baseline size
du -sh design-tests/baseline-images/

Consider:

• Reducing viewport sizes
• Using ignoreList to exclude large stories
• Optimizing page content (smaller images, fewer elements)

Troubleshooting

Every test is NEW

No baseline exists under the paths the run computes. Either the project is new, or snapshotsDirFn changed and moved the paths. Check git log on the baseline directory before promoting.

Every test is DIFF after a run on another machine

The run prints ⚠️ Capture platform differs from the baseline with the differing fields. Capture in the container CI uses; do not promote.

"Baseline update refused"

baseline-update found no manifest.json (run first), or the run's platform differs from the baseline's. The message names the differing fields. Pass --allow-platform-change to move the baseline to the new platform.

Persistent Pixel Differences

Differences that persist even after updating baseline:

Causes:

• Not using Docker for baseline generation
• Animations still running (adjust waitBeforeScreenshot)
• Unstable network requests (use waitForNetworkIdle)
• Dynamic content (timestamps, random data)

Solution:

stability: {
  waitForNetworkIdle: true,
  networkIdleTimeout: 30000,
  waitBeforeScreenshot: 2000, // Increase wait time
  waitForPageBusy: true,
}

Baseline Too Large

Repository size growing due to baseline images:

Solutions:

• Use Git LFS for large binaries
• Exclude non-critical stories
• Reduce number of viewports
• Optimize page weight

Next Steps

• Getting Started - Initial setup
• Adding Projects - Add more projects
• CLI Reference - Command options