Visu­al Regres­sion Test­ing for Exter­nal URLs With Playwright

We’ve all been there: You write a bit of CSS, check whether everything looks right. You deploy. Then someone sends you a screenshot: the mobile navigation is broken. And why is the size of those headings just a bit off? And where has that button gone?

Especially when you are working on a larger codebase together or you are refactoring your CSS or consolidating redundant styles, seemingly small changes in one corner of your CSS (or JavaScript) can have repercussions in a seemingly unrelated component. If you ever changed your base typography styles, you know what I mean. The Cascade and inheritance are powerful features, but as your codebase grows, tracking possible interdependencies can become a challenge.

This is when visual regression testing can be useful. Instead of manually checking every page after each CSS change, you capture screenshots automatically and compare them against a baseline – the gold standard. When something shifts, breaks, disappears, or turns lightgoldenrodyellow, you’ll know immediately.

In a recent client project, we had that exact challenge: we needed to check a large amount of pages – more than 1500 – for how well they digested a rework of the HTML structure and the CSS of all headings in all frontend components. And we needed to compare the changes deployed to a staging system to the production site.

But How? #

A first search surfaced a great article by Monica Dinculescu from 2017 in which she explained how to setup visual diffing with Puppeteer. At the same time, I found another interesting solution: the testing framework Vitest can now run native visual regression testing out of the box.

While this seems to be a fantastic solution for testing components or pages within your own local (or cloud) test environment, it is designed for the experimental browser mode feature in the Vitest API, which means it can’t navigate to external URLs. So if, like in our case, you want to test a staging environment against a production site, you need a different approach.

At first, I thought of combining Vitest and Playwright, but looking into the docs, I discovered that Playwright Test includes the ability to produce and visually compare screenshots using expect(page).toHaveScreenshot(). And because Playwright is built for browser testing, it can visit external URLs – and even interact with pages, if needed. 🎉

Setup #

The initial setup turned out to be easier than expected. All you basically need is a new project and install Playwright Test. So you could start a new project with

npm init

using playwright test as the test command in the project setup dialog and then install Playwright Test with

npm install -D @playwright/test

Alternatively, you can also scaffold a new Playwright project with

npm init playwright@latest

This will give you a package.jsonfile similar to this one – I added a few more useful test commands already:

{
  "name": "playwright-visual-regression",
  "version": "1.0.0",
  "type": "module",
  "description": "Little demo project for visual diff testing with Playwright",
  "scripts": {
    "test": "playwright test",
    "test:update": "playwright test --update-snapshots",
    "test:debug": "playwright test --debug"
  },
  "author": "Matthias Ott",
  "license": "MIT",
  "devDependencies": {
    "@playwright/test": "^1.56.0"
  }
}

The Script #

Now the juicy part, the test script. Which also turned out to be a lot shorter than expected. I saved it as ./tests/visual-regression.test.js. Playwright automatically executes all files in a tests (or test) folder that end in *.spec.ts/*.spec.js or *.test.ts/*.test.js.

We start by importing Playwright’s test runner and defining a config object for a few basic settings:

import { test, expect } from '@playwright/test';

/**
 * Configuration:
 * - baseUrl: The base URL of the site to test.
 * - routes: The routes to test (relative to baseUrl).
 * - viewports: Different viewport sizes to test (width, height, icon).
 */
const CONFIG = {
	
  baseUrl: 'https://matthiasott.com',
  routes: [
    '', 
    'notes', 
    'articles', 
    'workshops', 
    'links', 
    'about'
  ],
  viewports: {
    wide: { width: 1280, height: 800, icon: '🖥️' },
    narrow: { width: 375, height: 667, icon: '📱' },
  },
};

And now off to the actual tests – read the comments to see what each section does:

/**
 * Tests
 */
 
// We loop through each viewport configuration
Object.entries(CONFIG.viewports).forEach(([viewport, { width, height, icon }]) => {
  
  // Create a test group for this viewport
  // test.describe() groups related tests 
  // together in the console output
  test.describe(`${icon} ${viewport}`, () => {
    
    // Configure Playwright to use this viewport size
    test.use({ viewport: { width, height } });

    // Now we loop through each route we want to test …
    CONFIG.routes.forEach(route => {
      
      // … and define an individual test for this route. 
      // With the `page` object, Playwright controls the browser
      test(`📸 ${route || ''}`, async ({ page }) => {
        
        // Build the complete URL        
        const url = `${CONFIG.baseUrl}/${route}`;
        
        // Navigate to the URL
        // (waitUntil: 'networkidle' waits for the page to finish loading)
        await page.goto(url, { waitUntil: 'networkidle' });
        
        // Now we can take a screenshot and compare it to the baseline
        await expect(page).toHaveScreenshot({
          // Only capture the viewport, not entire scrollable page
          fullPage: false,
          // How much difference do we allow between the screenshots?
          maxDiffPixels: 100,   
        });
      });
    });
  });
});

And that’s it! The brilliant thing about using Playwright is that we don’t have to generate diff images that show the difference between the baseline and a new screenshot separately. Playwright does this automatically for us and puts the respective diff images into the image folder alongside the baseline and the screenshots to compare against.

The last step is to run the test:

npm test

On the first run, Playwright will create the baseline screenshots. On subsequent runs, it will take new screenshots and compare them against the original gold standard versions.

There are a few Playwright commands that can make this script even more useful, for example:

# Update baseline screenshots, for example after updating your design
npm run test:update

# Step-by-step debugging of failing tests with Playwright Inspector
npm run test:debug

A really useful last adjustment can be to generate an HTML report of the test results. To do this, we need to slightly adjust the test script section in our package.json and add the --reporter=html flag to the playwright test command, so that a “reporter” is specified. Then, we can add another script command to show the results ("test:report": "playwright show-report"):

"scripts": {
    "test": "playwright test --reporter=html",
    "test:update": "playwright test --update-snapshots",
    "test:debug": "playwright test --debug"
    "test:report": "playwright show-report",
  },

Now Playwright will serve an HTML report if there are any errors. To show the results of our test we can also use this command:

# View failed test results with visual diff viewer
npm run test:report

Of course, this is just the first version of this script which could be improved further. You could add a Playwright config file that changes the output directory or move the config options from the top of the script into the config file to make it easier to reuse the testing script in different projects, for example. Playwright also allows you to install different browser engines, like chromium, firefox, or webkit – all default engines can be installed with the command npx playwright install. So you could even use a specific browser to make your screenshots. You could also add authentication, in case your staging environment is password-protected, integrate the visual regression script into a CI workflow, and, and, and …

Now I’m curious: are you using any type of visual regression testing – or any testing at all? Let me know, for example on Mastodon, Bluesky, or by email

❦

This is post 6 of Blogtober 2025.

~