Diffing two runs

shotsweep capture --sitemap https://example.com/sitemap.xml --out ./before
shotsweep capture --sitemap https://example.com/sitemap.xml --out ./after
shotsweep diff ./before/manifest.json ./after/manifest.json --out ./diff --zip

Old capture
     ↓
New capture
     ↓
ShotSweep diff
     ↓
Changed images + report

shotsweep diff matches screenshots between the two manifests by URL and viewport, then pixel-diffs each pair.

Every changed page gets its own folder under --out, containing:

• old.png
• new.png
• diff.png

plus a top-level diff-report.json with the full summary and per-page results.

--zip bundles the whole diff output into a single archive.

Why pixel comparison?

shotsweep diff uses pixel comparison rather than ML or visual guessing. This makes the result deterministic and suitable for local development and CI.

Threshold

--threshold controls how much of an image must differ before a matched page is classified as changed.

It accepts three forms:

shotsweep diff old/manifest.json new/manifest.json --threshold 0.005
shotsweep diff old/manifest.json new/manifest.json --threshold "0.5%"
shotsweep diff old/manifest.json new/manifest.json --threshold loose

The threshold controls pixel matching. It is separate from the regression policy.

In other words:

• Threshold answers: “How much visual difference counts as a changed image?”
• Regression policy answers: “Which result statuses should fail the command?”

All threshold forms work from the CLI or config file.

See Config File.

Result statuses

Each matched or unmatched screenshot resolves to one status:

Regression policy

ShotSweep distinguishes between a visual regression and a structural change.

By default, the following policy is used:

A changed result means the pixels changed beyond the configured threshold. A size-changed result means the screenshot dimensions changed. An error result means a screenshot listed in a manifest was missing or unreadable, so the page couldn’t be compared. It is reported for that page and fails the run: a page that couldn’t be checked is not a page that passed.

Added and removed pages are reported in diff-report.json but do not fail the diff by default. This allows intentional additions and removals to be reviewed without making every site-structure change a visual-regression failure.

For example:

Visual regression summary
=========================

2 regressions

  Changed   : 1
  Resized   : 1
  Unchanged : 24
  Added     : 0
  Removed   : 0

In this example, the diff exits with code 1 because there is one changed page and one resized page. When a page couldn’t be compared, the summary adds an Errors : N (could not be compared) line, and those pages count toward the regression total.

A run containing only added or removed pages exits successfully:

Visual regression summary
=========================

0 regressions

  Changed   : 0
  Resized   : 0
  Unchanged : 25
  Added     : 0
  Removed   : 1

This policy is intended to work well both locally and in CI/CD pipelines. Teams can inspect added or removed pages while still using changed and size-changed as the default merge-blocking visual failures.

Environment drift

A pixel difference isn’t always a page change: a different Chromium version, operating system or font renderer can shift text by a pixel. When both manifests have a run-record.json next to them (written by capture unless you pass --no-record), diff compares the tool, Playwright, Chromium and Node versions and the OS and architecture of the two runs, and warns if any differ:

⚠ Environment drift detected between baseline and current run — this diff's verdict may reflect
environment changes, not just page changes:
    - Chromium version differs: 152.0.0.0 (baseline) vs 153.0.0.0 (current)

The same information is in diff-report.json (a provenance block, plus a decision block with the verdict and regression count, and a SHA-256 hash of each old.png, new.png and diff.png). A manifest with no run-record.json diffs exactly as before, just without the drift check.

Comparing runs from different machines

file paths in a manifest are recorded relative to where the capture ran. diff finds the screenshots even when a manifest has been moved to another folder, copied from another machine, or written on Windows and read on Linux. Pages are matched by URL and viewport, so https://example.com and https://example.com/ are the same page.

Re-baseline after upgrading to 1.3

ShotSweep 1.3 scrolls pages before capturing, so scroll-reveal content now appears where earlier versions captured blank areas. A baseline captured with an older version will therefore differ from a new capture of the same unchanged page. Capture a fresh baseline with the new version before comparing. See Upgrading to 1.3.

Summary output

Use --summary for a human-readable terminal summary:

shotsweep diff old/manifest.json new/manifest.json --summary

Example:

Visual regression summary
=========================

2 regressions

  Changed   : 1
  Resized   : 1
  Unchanged : 24
  Added     : 0
  Removed   : 0

Report: diff\diff-report.json

Text output

Use --text to save the human-readable summary to a file.

The option accepts a destination path, which makes it useful for CI artifacts:

shotsweep diff \
  old/manifest.json \
  new/manifest.json \
  --text ./reports/visual-summary.txt

Parent directories are created automatically when necessary.

You can combine it with --summary:

shotsweep diff \
  old/manifest.json \
  new/manifest.json \
  --summary \
  --text ./reports/visual-summary.txt

Use JSON when another program needs structured data, and text when the report is primarily for humans.

See JSON Output.

Exit code

diff exits non-zero when the run contains any changed, size-changed or error result.

added and removed results are reported but do not fail the diff by default.

This makes the default CI policy:

• changed → fail
• size-changed → fail
• error → fail
• unchanged → pass
• added → pass
• removed → pass

See CI/CD for a full workflow example.