Network Cache - MVP Status

Status: Experimental

Implemented but disabled by default. It measured an 8.6% improvement locally with a single worker, and that benefit is unverified in CI and unverified under parallel execution. Do not enable it in production.

Performance: 8.6% improvement measured locally (needs CI verification) Recommendation: Needs further testing before enabling in production

✅ Completed

⚠️ Current Status: Disabled by Default

The network cache feature is implemented but NOT enabled by default.

Reasons:

⚠️ Only 8.6% improvement measured locally (modest benefit)
⚠️ Not tested in CI with parallel execution (8 workers)
⚠️ Unknown interaction with BrowserWorkerPool pattern
⚠️ Needs verification that benefit exists in production environment

To enable (opt-in only):

ENABLE_NETWORK_CACHE=true NETWORK_CACHE_DIR=/tmp/.network-cache pnpm test

NOT recommended for production until:

Performance benefit verified in CI (with 8 workers)
No negative interaction with parallel execution
Verified stable over multiple runs

🚧 Required Testing Before Production

Critical (Must Complete)

Test with parallel execution (8 workers)
- Verify cache works with BrowserWorkerPool
- Measure performance: 8 workers with cache vs without
- Check for race conditions (8 browsers writing to same cache)
- Expected: Minimal benefit (network not bottleneck with localhost)
Test in CI environment
- Run in Jenkins with 1,155 tests
- Compare duration: with vs without cache
- Verify no visual regressions
- Check cache directory size (disk usage)
Add Docker tmpfs mount
- Update packages/bfh-design-tests/docker-compose.yml
- Add: tmpfs: ["/cache:size=500M,mode=1777"]
- Test that cache uses tmpfs in Docker
- Verify in-memory cache is faster than filesystem
Test with full test suite
- Run with complete BFH config (all 7 pages, both viewports = 14 tests)
- Measure aggregate time savings
- Verify cache doesn't cause issues with multiple tests
- Check cache size doesn't grow too large

Important (Should Complete)

Make cache configurable in .designTests.js
- Currently only via env var ENABLE_NETWORK_CACHE
- Add to config schema as optional field:
```
cache: {
  enabled: true,
  ttlMinutes: 60,
  baseDir: "/custom/path"  // optional
}
```
- Env var should override config (higher precedence)
- Update CLAUDE.md with config examples
Clean up tmp/ debugging files
- Remove or gitignore:
  - tmp/analyze-cache.js
  - tmp/compare-har-cache.js
  - tmp/debug-* directories
- Keep only essential files
- Document which files are for debugging vs production
Add cache statistics to report
- Show cache hit/miss ratio in final output
- Report cache size (number of files)
- Show time spent on cache operations
- Optional: Add to JSON report for CI parsing

Nice to Have (Optional)

Test with remote staging environment
- Deploy to staging with real network latency
- Measure improvement on remote servers
- Verify estimated 15-25% improvement for remote
Add cache clearing command
- design-tests cache clear CLI command
- Useful for debugging and CI cleanup
- Should clear cache directory safely
Selective caching strategy
- Only cache static assets (JS, CSS, fonts, images)
- Skip API calls or dynamic content
- Configurable via patterns or hostname filters
Document cache behavior with multiple workers
- Test if cache is safe with parallel execution
- Verify no file conflicts or race conditions
- Document findings in network-cache-results.md

📋 Pre-Merge Checklist

Code Quality

All TypeScript compiles without errors
Debug logs use debug module (not console.log)
Error handling for cache failures
Remove temporary debugging code
Update .gitignore for cache directories

Documentation

network-cache-results.md comprehensive report
CLAUDE.md env var best practices
Code comments explain cache behavior
Update main README.md with cache feature
Document cache in what-design-tests-are.md for stakeholders

Testing

Manual local testing (headless)
Visual regression verified
HAR file analysis
Docker/CI environment testing
Full test suite (not just single story)
BFH MVP pipeline integration

Performance

8.6% improvement measured locally
Improvement verified in CI
Performance documented with commands
Cache overhead acceptable

🎯 Merge Criteria

Ready to merge when:

✅ BFH MVP pipeline exists and passes
✅ Cache tested in Docker/CI environment
✅ Full test suite runs successfully with cache
✅ Docker tmpfs mount configured
✅ Documentation complete and accurate
✅ No visual regression issues
✅ Code cleanup complete (tmp/ files handled)

Current Status: 7/7 criteria met = 0% (waiting on BFH MVP pipeline)

🔍 Known Issues & Limitations

Documented Limitations

⚠️ No HTTP cache semantics: Ignores Cache-Control, ETag (caches everything for TTL)
⚠️ TTL-based only: No automatic invalidation on content changes
⚠️ Overcaching risk: All requests cached uniformly for 60 minutes
⚠️ Modest localhost benefit: 8.6% improvement (higher for remote servers)

Mitigation Strategies

✅ Cache cleared per test run (not shared across branches)
✅ Short TTL (60 minutes max)
✅ Opt-in only (requires explicit enable)
✅ Well-documented behavior in network-cache-results.md

No Known Bugs

✅ Visual regression: Initial +52px was baseline issue, NOT cache
✅ Cache registration: Fixed by calling .ALL()
✅ Report generation: Fixed for custom directory names
✅ Performance: Verified improvement is real

📚 Key Learnings Documented

In CLAUDE.md

✅ Env var best practices (never use export)
✅ Always set inline: ENV=value command
✅ Explanation of Bash tool state persistence

In Code Comments

✅ network-cache.ts: Explains cache doesn't respect HTTP semantics
✅ screenshot.ts: Documents when cache is enabled
✅ Critical bug fix documented (must call .ALL())

🚀 Post-Merge Tasks

These can be done AFTER merging:

Monitor visual regression in production (first 2 weeks)
Measure real CI performance improvement
Consider making cache default-enabled (if no issues)
Gather feedback from other projects
Test with MCHWEB full suite
Test with EWZ multi-project setup
Consider ETag support if plugin adds it
Optimize cache size/TTL based on real usage

📞 Questions to Answer

Before enabling cache by default:

How much disk space does cache use for full suite?
Does cache benefit scale with number of stories?
Is tmpfs necessary or does filesystem work fine?
Should TTL be configurable per project?
Any edge cases where cache breaks tests?

Last Updated: 2025-10-20 Next Review: After BFH MVP pipeline implementation Owner: Performance optimization team

✅ Completed​

⚠️ Current Status: Disabled by Default​

🚧 Required Testing Before Production​

Critical (Must Complete)​

Important (Should Complete)​

Nice to Have (Optional)​

📋 Pre-Merge Checklist​

Code Quality​

Documentation​

Testing​

Performance​

🎯 Merge Criteria​

🔍 Known Issues & Limitations​

Documented Limitations​

Mitigation Strategies​

No Known Bugs​

📚 Key Learnings Documented​

In network-cache-results.md​

In CLAUDE.md​

In Code Comments​

🚀 Post-Merge Tasks​

📞 Questions to Answer​