GitLab Pipelines & Netlify Deploy Previews
While this technique still works, I’ve since switched to building the site myself, removing the need to wait on Deploy Previews.
Something I’ve had in my mind for a while is running tests on each merge request (for broken links, accessibility issues, and so on). Netlify has a plugin to check links and another plugin for pa11y, but, as ever, I want control, and I want to avoid using Netlify build minutes. Instead, I want to run these checks in GitLab, which even has ways to integrate test reports and show accessibility issues within merge requests.
I’m used to building pipelines for regular repositories, but Netlify doesn’t have any proper
integration with GitLab. Part of deploying a site from a Git repository is registering a
webhook to notify it any time
you push code. Netlify then uses GitLab’s API (in this case) to add an external pipeline
stage after the fact, which can’t be controlled at all from GitLab. On merge requests, this stage
tracks building Deploy
Previews: the
complete site, but built from the code in the merge request and hosted at a temporary domain.
Because GitLab doesn’t know about this job, you can’t directly wait for the Deploy Preview to be
available, and so can’t run checks on it once
it’s ready. (I did try declaring an external
stage in my pipeline, but it was ignored.) I decided
to find a way to do it.
Hacking Netlify integration the wrong way
One of the comments in the GitLab issue above showed how to build the URI of the Deploy Preview. I created a single-job stage in my pipeline that would block using wait-for-it until the URI was available. This worked, in a sense, but because the URI never changes for a particular branch, the pipeline would only ever wait the first time a branch was pushed to. After that, the URI would always be available, even if a new version was being built in the background and there would be nothing to wait for.
So I had a way to reach the Deploy Preview, but I needed to wait until it was ready. My next thought
was to use Netlify’s API to get a list of
deploys and watch for some information to
change. I could repeatedly fetch the list until the newest deploy had a state
of building
, then
repeatedly fetch it again until the newest one had a state of ready
. This is what the code looked
like:
YAMLwait_for_deploy:
stage: wait
image:
name: alpine:3.13.5
only:
- merge_requests
timeout: 2m
before_script:
- apk add -q curl jq
script:
- |-
until curl -sH "Authorization: Bearer $NETLIFY_AUTHORIZATION" $DEPLOY_LIST_URL | jq -e '.[0].state == "building"'; do
sleep 1s
done
- |-
until curl -sH "Authorization: Bearer $NETLIFY_AUTHORIZATION" $DEPLOY_LIST_URL | jq -e '.[0].state == "ready"'; do
sleep 1s
done
environment:
name: review/$CI_COMMIT_REF_NAME
url: $DEPLOY_PREVIEW_URL
interruptible: yes
(I configured NETLIFY_AUTHORIZATION
in my CI/CD settings and placed the other
variables in the global block. I set a timeout of two minutes because the deploy should be complete
in less than that. I also globally set GIT_STRATEGY
to none
since none of my tasks needs the raw
code: all of them interact with the Deploy Preview.)
This worked. I was able to push to a branch, wait for it to deploy, run code, push changes, and repeat the process. I set my sights on the next task: the code to run.
Setting up the checks
After looking around, I settled on broken-links-inspector to check my links. muffet was promising too, but the former had more useful features, including JUnit-style reports which GitLab could read. I experimented with it locally to arrive at the right incantations, then added it to my pipeline. I did have to make some concessions: use --single-threaded to prevent it from being rate-limited, increase the timeout a little, and add a single retry attempt. I also had to manually exclude numerous pages: a subreddit that doesn’t exist, a deliberately broken link to showcase the 404 page, a crates.io page that for some reason kept failing, TypeKit’s domains, and, funnily enough, shivjm.blog, because, as I realized after a while, the tool had added 9,000 views to my live site by following links in the Deploy Preview’s Atom feed. With those exclusions added and a handful of legitimately incorrect links fixed, the checker reported success.
I next wanted to add accessibility checking using pa11y.[1] GitLab’s documented solution is entirely à la carte. It forces you to have fixed stages, jobs, and parameters. I dug deeper, adapting the contents of the referenced template to my specific case. That should have been enough, but I unwittingly overestimated GitLab’s variable expansion. I had set up my pipeline like this:
YAMLvariables:
SITE_NAME: netlify-site-1234
DEPLOY_PREVIEW_DOMAIN: $SITE_NAME.netlify.app
DEPLOY_PREVIEW_URL: https://deploy-preview-$CI_MERGE_REQUEST_IID--$DEPLOY_PREVIEW_DOMAIN/
check-accessibility:
script:
- do-something-with $DEPLOY_PREVIEW_URL
But the job kept failing because DEPLOY_PREVIEW_URL
wasn’t receiving a last expansion pass, so
what the script saw was the literal string
https://deploy-preview-$CI_MERGE_REQUEST_IID--$DEPLOY_PREVIEW_DOMAIN/. I only found this
out when I delved deeper, into the source for the Docker image the template
used, and adapted the shell script to my
specific case.
I did end up customizing the script for my case anyway, so it wasn’t a wasted effort. Instead of
node:latest
, I wanted to run the task on Alpine Linux; Puppeteer took offence at this, which
reminded me that I have a lot of experience running Puppeteer on Alpine Linux, which reminded me
that I built an image specifically to run Puppeteer on Alpine
Linux… I switched and set the flags I needed
to. Then I set up the configuration files like GitLab’s shell script did. Then I replaced the
URIs with the ones I wanted. This exposed the problem I mentioned earlier. I had to reduce
the amount of recursive expansion I relied on.
By the end of it, my task looked like this:
YAMLcheck_accessibility:
stage: test
variables:
PA11YCI_CONFIG: |-
{
"defaults": {
"chromeLaunchConfig": {
"args": [
"--no-sandbox"
],
"hideElements": "pre > code"
}
}
}
PA11YJSON_CONFIG: |-
{
"chromeLaunchConfig": {
"args": [
"--no-sandbox"
]
},
"includeWarnings": true,
"reporter": "html"
}
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: 1
PUPPETEER_EXECUTABLE_PATH: /usr/bin/chromium-browser
urls: "/ colophon/multiple-simultaneous-immutable-responsive-images/"
image:
name: shivjm/node-chromium-alpine:node14.16.1-chromium81.0.4044.113
before_script:
- npm i pa11y@5.3.0 pa11y-reporter-html@1.0.0 pa11y-ci@2.3.0
- echo $PA11YCI_CONFIG > .pa11yci
- echo $PA11YJSON_CONFIG > pa11y.json
- mkdir reports
script:
- export URLS="$DEPLOY_PREVIEW_URL ${DEPLOY_PREVIEW_URL}colophon/multiple-simultaneous-immutable-responsive-images/"
- npx pa11y-ci -j --config .pa11yci $URLS > reports/gl-accessibility.json || true # ensure the job continues
- |-
for url in $URLS
do
filename="reports/$(echo $url | sed -E 's/^https?:\/\///' | sed -E 's/\//-/g')-accessibility.html"
npx pa11y --config pa11y.json $url | tee $filename > /dev/null
done
allow_failure: true
artifacts:
when: always
expose_as: 'Accessibility Reports'
paths: ['reports/']
reports:
accessibility: reports/gl-accessibility.json
only:
- merge_requests
I had to configure pa11y to ignore pre > code
because, as I have now discovered, the ostensibly
accessible PrismJS themes I’m using do not in fact have sufficient contrast. I also had to add || true
to the npx pa11y-ci
call to make sure the individual page reports were generated regardless
of whether it succeeded. The post I chose to test is one that’s a little lengthy and exercises
several features.
The hack breaks immediately
My pipelines started failing. It took me a few attempts to understand that timing was the issue: the
wait-for-deploy
job assumed it would run concurrently with the Netlify deploy, but GitLab laughed
at my naïveté and took minutes to schedule the jobs, which would therefore time out
without finding an in-progress Netlify deploy and block the pipeline.
I couldn’t see a way to safely and repeatably identify the status of the deployment. I gave up,
removed the wait_for_deploy
job, and turned the test jobs into manual
jobs.
Hacking Netlify integration the right way
Soon after I merged my changes, it struck me that the Netlify API was showering me with data I was ignoring. In fact, I could use a critical piece of information to synchronize GitLab and Netlify: the commit revision, which would necessarily be different every time I pushed the code. To put it in simple terms, all I had to do was wait until the first deploy with the right commit revision was ready. Phrasing it in a way that jq, YAML, and ash would all understand was rather more complex, but what I arrived at, eventually, was:
YAMLwait_for_deploy:
stage: wait
image:
name: alpine:3.13.5
only:
- merge_requests
timeout: 2m
before_script:
- apk add -q curl jq
script:
- |-
until curl -sH "Authorization: Bearer $NETLIFY_AUTHORIZATION" $DEPLOY_LIST_URL | jq -e "[.[] | select(.state == \"ready\" and .commit_ref == \"$CI_COMMIT_SHA\" and .branch != \"master\")] | length | . > 0"; do
sleep 1s
done
environment:
name: review/$CI_COMMIT_REF_NAME
url: $DEPLOY_PREVIEW_URL
interruptible: yes
This works reliably! (The check for the branch isn’t necessary, but it makes me feel better.)
Thus it is that, a total of 11 commits and 51 pipelines later, I have a solution that accounts for all the hurdles I observed. All I need to do is push changes and GitLab will test my links and verify the basics of accessibility in the new Deploy Preview, no matter whether the jobs run before, during, or after its build. I hope more robust support for Netlify arrives soon, but this should suffice for the moment.
- I understand that automated tools are no panacea. pa11y merely provides a baseline.↩