Gen­er­at­ing Acces­si­bil­i­ty Test Results for a Whole Web­site With Pa11y CI

Posted by

59 Webmentions

This week, I’m doing an accessibility audit for a client. One of the first steps is to have a general look at the site. You can – and should – do that manually for sure, but another very useful way to get a good first impression of how good or bad things are is to use an automated tool like Pa11y.

Pa11y is a set of free and open-source tools with which you can test a website for all kinds of accessibility (a11y) issues like insufficient color contrast, missing alt texts, missing labels, wrong heading levels, problems regarding ARIA roles, or general HTML validation issues. There are different tools available: pa11y itself is a command-line interface to run a one-off accessibility test against a web page. Pa11y Dashboard is a web interface for Pa11y to constantly keep an eye on the current a11y status of one or more websites. It is powered by Pa11y Webservice. And then, there is Pa11y CI, an accessibility test runner built using Pa11y that lets you run tests as part of a Continuous Integration environment. This is especially useful if you want to make sure that no issues make it to the live site.

Installing Pa11y #

Node.js is a prerequisite – you might want to install it via nvm first. Then, install Pa11y globally via the command line:

npm install -g pa11y

Now, you can run tests against a URL…

pa11y https://example.com

…or a file:

pa11y ./path/to/your/file.html

Pa11y will run the test and display the results in the terminal.

Pa11y output on the command line

There is much more you can do with Pa11y, in particular the JavaScript Interface, but this would be beyond the scope of this post. So have a look at the GitHub repository in case you want to learn more.

Pa11y CI #

Running a test against a URL or a file is already super helpful, but wouldn’t it be nice to run a test for a whole site and, let’s say, get a report of the results in HTML?

This is can be done with Pa11y CI. While Pa11y CI is best used to run automated tests, you can run it manually as a command-line tool as well. And it comes with a neat feature: you can tell it to run a test against a whole sitemap. Let’s try this. First, we install Pa11y CI globally:

npm install -g pa11y-ci

Now, you could create a config file that includes all the URLs that Pa11y CI should check. But you can also feed it with an XML sitemap with the --sitemap option:

pa11y-ci --sitemap https://pa11y.org/sitemap.xml
Pa11y CI output of 15 tested pages

Pick Your Reporter #

We’ve seen that when we run a test with either pa11y or pa11y-ci, the results will be output to the command line by default. But this can be changed by switching the so-called “reporter” from cli to something else. You could use csvor json, for example, but you can also output the test results as HTML. Pa11y now has an HTML reporter built in. The reporter can be set via the --reporter option. So when you are using Pa11y, you can write:

pa11y --reporter html https://example.com

But in our case, we want to run a test against a whole sitemap with Pa11y CI. Here, we have to install one additional rpm package that adds an HTML reporter: Pa11y CI HTML Reporter.

You install it via npm again:

npm install pa11y-ci-reporter-html

Now, this reporter can again be set via the --reporter option:

pa11y-ci --reporter=pa11y-ci-reporter-html https://pa11y.org/

Pa11y CI will now run its test(s) and output the results in the current working directory as HTML. Nice! In the example above, we ran the test against a single URL, so the last step is to run the tests against a sitemap using the HTML reporter:

pa11y-ci --reporter=pa11y-ci-reporter-html --sitemap https://pa11y.org/sitemap.xml

The result:

Pa11y CI Report in Safari

Et voilà! We have generated a nice HTML overview of the test results of the whole website we want to test. Now, we can go on and dig deeper, document existing issues in more detail, or even – if we have access to the codebase – immediately fix a few things to reduce the number of issues we see.

Again, an automated tool like Pa11y is not a substitute for a proper accessibility audit by an actual human being – particularly one using a screen reader. But it does give you a good general overview and helps you to quickly address many of the things that are wrong in your codebase.

~

59 Webmentions

  1. Photo of @matthiasott @matthiasott
    @matthiasott this is my top reason for having a personal blog!!
  2. Photo of @matthiasott @matthiasott
    @matthiasott The best argument for writing according to my Swiss Cheese-memory brain 😂
  3. Photo of Matthias Ott Matthias Ott
    Aaand another link from the archives on checking your site for the correct heading levels: Level Up Your Headings Game #accessibility #a11y #testing #html https://matthiasott.com/notes/level-up-your-headings-game a11y accessibility html testing Level Up Your Headings Game · Matthias Ott – User Experience Designer
  5. Photo of The A11Y Project The A11Y Project
    "Et voilà! We have generated a nice HTML overview of the test results of the whole website we want to test." @m_ott matthiasott.com/notes/generati…
  6. Photo of Friday Front-End Friday Front-End
    Generating Accessibility Test Results for a Whole Website With Pa11y CI, by @m_ott matthiasott.com/notes/generati…
  7. Photo of Frontend Dogma Frontend Dogma
    Generating Accessibility Test Results for a Whole Website with Pa11y CI, by @m_ott: matthiasott.com/notes/generati…
  8. Photo of Web Axe Web Axe
    Generating Accessibility Test Results for a Whole Website With Pa11y CI matthiasott.com/notes/generati… #a11y #webdev #testing
  9. Photo of FX FX
    Generating Accessibility Test Results for a Whole Website With Pa11y CI, by @m_ott matthiasott.com/notes/generati…
  10. Photo of Pablo Lara H Pablo Lara H
    🧪 Generating Accessibility Test Results for a Whole Website With Pa11y CI @pa11yorg by Matthias Ott @m_ott #accessibility #Pa11y matthiasott.com/notes/generati…
  11. Photo of Darek Kay Darek Kay
    Pa11y is great, but some limitations motivated me to create a similar tool. Evaluatory runs at multiple breakpoints (e.g. issues on mobile only) and is able to check different color schemes (e.g. contrast ratio issues only in dark mode): darekkay.com/evaluatory/
  13. Photo of Guillaume Deblock Guillaume Deblock
    Generating #Accessibility Test Results for a Whole Website With Pa11y CI, by @m_ott matthiasott.com/notes/generati…

10 Reposts

ⓘ Webmentions are a way to notify other websites when you link to them, and to receive notifications when others link to you. Learn more about Webmentions.

More Notes