Series: Drupal Docker
tl;dr: Pa11y and Pa11y-ci are helpful accessibility testing tools, as part of your CI workflow or running locally.
This series has now covered all the core functionality of having a local development environment and deploying updates. The majority of what's left has to do with automated testing, including this one using Pa11y and Pa11y-CI.
Pa11y and Pa11y-CI, as the names suggest, are very similar with the same kinds of tests it can run, but with a few differences in exactly which features are available. The biggest one of interest to me is that Pa11y-CI includes an extremely simple command out of the box to check the accessibility of all pages in a sitemap.
One notable difference compared to the Playwright tests from the last post is that there are no other interactions or mock devices involved here. Whereas with Playwright I could say things like "using an iPhone X in dark mode, click on this button and then check the accessibility of the page after" here it is much more simply scanning every page as they are initially loaded. That gives it an advantage in ease of use, but doesn't cover every scenario you may want to test.
CI/CD Tests
Building A Dedicated Testing Image
A dedicated pa11y testing image is available in my GitLab. That can be used for running tests like one against the sitemap, which will work externally as long as the site you're trying to test is publicly accessible.
The Dockerfile installs some needed packages, sets up the necessary file structure, and installs pa11y-ci including with the HTML report that is the most friendly to read for a large site.
FROM debian:12-slim
RUN apt-get update && \
apt-get install -y --no-install-recommends \
nodejs \
npm \
ca-certificates \
fonts-liberation \
libasound2 \
libatk-bridge2.0-0 \
libatk1.0-0 \
libatspi2.0-0 \
libc6 \
libcairo2 \
libcups2 \
libdbus-1-3 \
libdrm2 \
libexpat1 \
libgbm1 \
libglib2.0-0 \
libgtk-3-0 \
libnspr4 \
libnss3 \
libpango-1.0-0 \
libpangocairo-1.0-0 \
libstdc++6 \
libudev1 \
libuuid1 \
libx11-6 \
libx11-xcb1 \
libxcb-dri3-0 \
libxcomposite1 \
libxcursor1 \
libxdamage1 \
libxext6 \
libxfixes3 \
libxi6 \
libxkbcommon0 \
libxrandr2 \
libxrender1 \
libxshmfence1 \
libxss1 \
libxtst6 \
wget \
xdg-utils \
&& apt-get clean && \
rm -rf /var/lib/apt/lists/*
# mkdir for the application tests
RUN mkdir -p /opt/pa11y
WORKDIR /opt/pa11y
# Install pa11y-ci and the HTML reporter
RUN npm install -g pa11y-ci pa11y-ci-reporter-html
# Copy config file
COPY / /opt/pa11y/As always in this Drupal Docker series, the gitlab-ci job then extends my general purpose Docker build job that I've already gotten into.
It also has a configuration file that defines how pa11y-ci should run, including that it will report to both the command line and to the HTML reporter. The most important here is the --no-sandbox option. None of the tests will run without that.
{
"defaults": {
"timeout": 120000,
"reporters": [
"cli",
"pa11y-ci-reporter-html"
],
"chromeLaunchConfig": {
"args": [
"--no-sandbox"
]
},
"runners": [
"axe",
"htmlcs"
],
"ignore": [
"frame-tested",
"frame-title"
],
"level": "warning"
}
}Running the Tests
When I want a specific project to be able to run tests, the CI job in that project would look like:
# Includes general CI jobs #
include:
- project: "ryan-l-robinson/gitlab-ci"
ref: main
file: test.yml
### Checks a sitemap using Pa11y. ###
pa11y_test:
extends: .pa11y_test
stage: content_test
environment: Production
variables:
SITEMAP_URL: https://tech.ryanrobinson.ca/sitemap.xml
only:
refs:
- mainThat is extending my general purpose job which uses the image I've previously built, checking all the pages of a sitemap, and keeping the report for 5 months:
## Runs accessibility sitemap tests using pa11y ##
.pa11y_test:
stage: test
image: registry.gitlab.com/ryan-l-robinson/pa11y-docker-gitlab/pa11y
variables:
SITEMAP_URL: ""
script:
# Use pa11y-ci's built-in function for scanning from a sitemap. #
- pa11y-ci --sitemap $SITEMAP_URL --config /opt/pa11y/config.json
# Show as warning but don't stop execution of later jobs if failure. #
allow_failure: true
# Save the reports for 5 months, past our next release. #
artifacts:
when: always
paths:
- pa11y-ci-report
expire_in: 5 monthsRunning Tests Locally
Installation Requirements
The pa11y packages are being added in the postCreateCommand discussed in a previous post.
Running Tests
These tests can only be run from the command line, like:
pa11yOr using pa11y-ci to check an entire sitemap:
pa11y-ci --sitemap https://tech.ryanrobinson.caHTML Reporter
Pa11y can dump the results into reports in a few different formats, including HTML. This then allows browsing the results one at a time in a very easy to read format. Some of the other formats like a JSON file aren't bad if there are few results, but get hard to work with on larger sites.
Configuration
Unlike Playwright, there are no tests to write yourself here, but you may need to adjust some of the configuration files. Pa11y and Pa11y-ci have different files. I'm not going to pretend to know why they have slightly different files, but pa11y defaults to a configuration file position of pa11y.json, and pa11y-ci defaults to .pa11yci.
In my case, here's my pa11y.json:
{
"chromeLaunchConfig": {
"args": [
"--no-sandbox",
"--ignore-certificate-errors",
"--allow-insecure-localhost"
],
"ignoreHTTPSErrors": true
},
"defaults": {
"timeout": 120000,
"reporters": ["cli", "pa11y-ci-reporter-html"],
"runners": ["axe", "htmlcs"],
"ignore": ["frame-tested", "frame-title"],
"level": "warning"
}
}And here's my .pa11yci:
{
"defaults": {
"chromeLaunchConfig": {
"args": [
"--no-sandbox",
"--ignore-certificate-errors",
"--allow-insecure-localhost"
],
"ignoreHTTPSErrors": true
},
"timeout": 120000,
"reporters": [
"cli",
"pa11y-ci-reporter-html"
],
"runners": [
"axe",
"htmlcs"
],
"ignore": [
"frame-tested",
"frame-title"
],
"level": "warning"
}
}Previous: Playwright Testing
Next: Database Lifecycle