Generating Accessibility Test Results for a Whole Website With Pa11y CI

This week, I’m doing an accessibility audit for a client. One of the first steps is to have a general look at the site. You can – and should – do that manually for sure, but another very useful way to get a good first impression of how good or bad things are is to use an automated tool like Pa11y.

Pa11y is a set of free and open-source tools with which you can test a website for all kinds of accessibility (a11y) issues like insufficient color contrast, missing alt texts, missing labels, wrong heading levels, problems regarding ARIA roles, or general HTML validation issues. There are different tools available: pa11y itself is a command-line interface to run a one-off accessibility test against a web page. Pa11y Dashboard is a web interface for Pa11y to constantly keep an eye on the current a11y status of one or more websites. It is powered by Pa11y Webservice. And then, there is Pa11y CI, an accessibility test runner built using Pa11y that lets you run tests as part of a Continuous Integration environment. This is especially useful if you want to make sure that no issues make it to the live site.

Installing Pa11y

Node.js is a prerequisite – you might want to install it via nvm first. Then, install Pa11y globally via the command line:

npm install -g pa11y

Now, you can run tests against a URL…


…or a file:

pa11y ./path/to/your/file.html

Pa11y will run the test and display the results in the terminal.

Pa11y output on the command line

There is much more you can do with Pa11y, in particular the JavaScript Interface, but this would be beyond the scope of this post. So have a look at the GitHub repository in case you want to learn more.

Pa11y CI

Running a test against a URL or a file is already super helpful, but wouldn’t it be nice to run a test for a whole site and, let’s say, get a report of the results in HTML?

This is can be done with Pa11y CI. While Pa11y CI is best used to run automated tests, you can run it manually as a command-line tool as well. And it comes with a neat feature: you can tell it to run a test against a whole sitemap. Let’s try this. First, we install Pa11y CI globally:

npm install -g pa11y-ci

Now, you could create a config file that includes all the URLs that Pa11y CI should check. But you can also feed it with an XML sitemap with the --sitemap option:

pa11y-ci --sitemap
Pa11y CI output of 15 tested pages

Pick Your Reporter

We’ve seen that when we run a test with either pa11y or pa11y-ci, the results will be output to the command line by default. But this can be changed by switching the so-called “reporter” from cli to something else. You could use csvor json, for example, but you can also output the test results as HTML. Pa11y now has an HTML reporter built in. The reporter can be set via the --reporter option. So when you are using Pa11y, you can write:

pa11y --reporter html

But in our case, we want to run a test against a whole sitemap with Pa11y CI. Here, we have to install one additional rpm package that adds an HTML reporter: Pa11y CI HTML Reporter.

You install it via npm again:

npm install pa11y-ci-reporter-html

Now, this reporter can again be set via the --reporter option:

pa11y-ci --reporter=pa11y-ci-reporter-html

Pa11y CI will now run its test(s) and output the results in the current working directory as HTML. Nice! In the example above, we ran the test against a single URL, so the last step is to run the tests against a sitemap using the HTML reporter:

pa11y-ci --reporter=pa11y-ci-reporter-html --sitemap

The result:

Pa11y CI Report in Safari

Et voilà! We have generated a nice HTML overview of the test results of the whole website we want to test. Now, we can go on and dig deeper, document existing issues in more detail, or even – if we have access to the codebase – immediately fix a few things to reduce the number of issues we see.

Again, an automated tool like Pa11y is not a substitute for a proper accessibility audit by an actual human being – particularly one using a screen reader. But it does give you a good general overview and helps you to quickly address many of the things that are wrong in your codebase.


23 Webmentions

Photo of Guillaume Deblock
Guillaume Deblock
Generating #Accessibility Test Results for a Whole Website With Pa11y CI, by @m_ott…
Photo of Matthias Ott
Matthias Ott
Wow, nice! 👏; I’ll have a look at it! 😊;
Photo of Darek Kay
Darek Kay
Pa11y is great, but some limitations motivated me to create a similar tool. Evaluatory runs at multiple breakpoints (e.g. issues on mobile only) and is able to check different color schemes (e.g. contrast ratio issues only in dark mode):
Photo of Pablo Lara H
Pablo Lara H
🧪; Generating Accessibility Test Results for a Whole Website With Pa11y CI @pa11yorg by Matthias Ott @m_ott #accessibility #Pa11y…
Photo of FX
Generating Accessibility Test Results for a Whole Website With Pa11y CI, by @m_ott…
Photo of Web Axe
Web Axe
Generating Accessibility Test Results for a Whole Website With Pa11y CI… #a11y #webdev #testing
Photo of Frontend Dogma
Frontend Dogma
Generating Accessibility Test Results for a Whole Website with Pa11y CI, by @m_ott:…
Photo of Friday Front-End
Friday Front-End
Generating Accessibility Test Results for a Whole Website With Pa11y CI, by @m_ott…