Skip to main content

Baseline Management

Status: Stable This guide explains how to update, recreate, and manage baseline images for visual regression testing.

Understanding Baselines​

Baseline images are reference screenshots stored in your repository. When you run tests, current screenshots are compared against these baselines to detect visual changes.

design-tests/
├── baseline-images/        # ✓ Committed to git (reference images)
├── actual-images/      # Generated (current screenshots)
├── diff-output/        # Generated (visual differences)
└── report/             # Generated (HTML report)

Updating Baselines (After Intentional Changes)​

When you make intentional design changes, you need to update the baseline to reflect the new expected state.

1. Review the Visual Differences​

Run tests to see what changed:

npx design-tests run

Open the HTML report at design-tests/report/index.html to review all visual differences:

  • Red areas: Pixels that changed
  • Side-by-side view: Compare baseline vs. actual
  • Slider view: Interactive comparison

2. Verify Changes are Intentional​

Make sure the differences are expected:

  • ✅ Intentional design updates
  • ✅ New features or components
  • ✅ Fixed bugs that changed appearance
  • ❌ Unintended regressions
  • ❌ Environmental differences (fonts, rendering)

3. Update Baseline​

If changes are intentional, copy actual screenshots to baseline:

# Copy all changed screenshots
cp -r design-tests/actual-images/* design-tests/baseline-images/

# Review what changed
git status
git diff design-tests/baseline-images/

# Commit
git add design-tests/baseline-images/
git commit -m "Update baseline after button redesign"
git push

4. CI/CD Workflow (Automatic PR)​

In CI environments (like Jenkins), the pipeline automatically:

  1. Detects visual differences
  2. Creates a PR with updated baseline
  3. Marks build as UNSTABLE

Jenkinsfile snippet:

if (failedCount > 0) {
    currentBuild.result = 'UNSTABLE'
    // Creates: baseline-update/<BUILD_NUMBER>
    // Commit: "update BFH design test baseline <BUILD_NUMBER>"
}

Review and merge the PR to accept the changes.

Recreating Baselines​

When to Recreate​

Recreate baselines when:

  • Upgrading Playwright (font rendering changes)
  • Major browser version updates (Chrome, Firefox)
  • Migrating from another testing framework
  • Switching OS/architecture (macOS → Linux)
  • Starting fresh after major refactoring

How to Recreate​

For a single project:

# Delete the project's baseline directory
rm -rf design-tests/baseline-images/myproject/

# Run tests - baseline will be auto-created
npx design-tests run --project myproject

# Review and commit
git add design-tests/baseline-images/myproject/
git commit -m "Recreate baseline for myproject"

For all projects:

# Delete all baselines
rm -rf design-tests/baseline-images/

# Run tests - all baselines will be auto-created
npx design-tests run --all

# Commit
git add design-tests/baseline-images/
git commit -m "Recreate all baselines"

Auto-Detection Behavior​

The system automatically detects missing baselines:

📝 Project "myproject": Creating baseline (no existing baseline found)
...
📊 Test Results: 0 visual differences, 150 new baselines detected

In CI, the commit message will be:

"create BFH design test baseline <BUILD_NUMBER>"

Docker for Consistent Baselines​

⚠️ Important: Always use Docker for generating/updating baselines to ensure consistency with CI.

Why Docker?​

Font rendering and anti-aliasing differ between operating systems:

  • macOS: Uses Apple's font rendering
  • Linux (CI): Uses FreeType/fontconfig
  • Windows: Uses ClearType

These differences cause pixel-level changes that fail tests.

Docker Usage​

# Generate baseline in Docker (Linux x86_64)
docker compose run --rm design-tests

# Local debugging only (not for baseline generation)
npx design-tests run

Good workflow:

# Make changes
vim src/components/Button.tsx

# Generate new baseline (Docker)
docker compose run --rm design-tests

# Commit
git add design-tests/baseline-images/
git commit -m "Update Button baseline"

Bad workflow:

# Generate baseline locally (macOS/Windows)
npx design-tests run  # ❌ Will differ from CI

# CI fails with pixel differences
# Now you have to regenerate in Docker anyway

Partial Updates​

Update only specific stories or viewports:

By Project​

npx design-tests run --project myproject
# Copy only changed screenshots
cp -r design-tests/actual-images/myproject/* \
     design-tests/baseline-images/myproject/

By Story Pattern​

Not directly supported - use manual file operations:

# Run all tests
npx design-tests run

# Copy only button-related screenshots
find design-tests/actual-images -name "*button*" -exec \
  cp --parents {} design-tests/baseline-images/ \;

Handling Merge Conflicts​

When multiple developers update baselines:

# Fetch latest
git pull origin develop

# Conflict in baseline images
# CONFLICT: design-tests/baseline-images/project/button-desktop.png

# Accept theirs or yours
git checkout --theirs design-tests/baseline-images/project/button-desktop.png

# Or regenerate
rm -rf design-tests/baseline-images/project/
npx design-tests run --project project

# Commit resolution
git add design-tests/baseline-images/
git commit -m "Resolve baseline conflicts - regenerate"

Baseline Auditing​

View Baseline History​

See when and why baselines changed:

# Show baseline changes over time
git log --oneline design-tests/baseline-images/

# Show who changed specific baseline
git log --follow design-tests/baseline-images/project/button-desktop.png

# Show visual diff of baseline change
git difftool HEAD~1 HEAD -- design-tests/baseline-images/

Find Large Baselines​

Identify oversized screenshots:

# Find baselines > 500KB
find design-tests/baseline-images/ -type f -size +500k

# Total baseline size
du -sh design-tests/baseline-images/

Consider:

  • Reducing viewport sizes
  • Using ignoreList to exclude large stories
  • Optimizing page content (smaller images, fewer elements)

Troubleshooting​

"Baseline not found" errors​

❌ Baseline not found: design-tests/baseline-images/project/story.png

Solution: Run tests to create baseline:

npx design-tests run --project project

Persistent Pixel Differences​

Differences that persist even after updating baseline:

Causes:

  • Not using Docker for baseline generation
  • Animations still running (adjust waitBeforeScreenshot)
  • Unstable network requests (use waitForNetworkIdle)
  • Dynamic content (timestamps, random data)

Solution:

stability: {
  waitForNetworkIdle: true,
  networkIdleTimeout: 30000,
  waitBeforeScreenshot: 2000, // Increase wait time
  waitForPageBusy: true,
}

Baseline Too Large​

Repository size growing due to baseline images:

Solutions:

  • Use Git LFS for large binaries
  • Exclude non-critical stories
  • Reduce number of viewports
  • Optimize page weight

Next Steps​