GitLab Pipelines & Netlify Deploy Previews

Something I’ve had in my mind for a while is running tests on each merge request (for broken links, accessibility issues, and so on). Netlify has a plugin to check links and another plugin for pa11y, but, as ever, I want control, and I want to avoid using Netlify build minutes. Instead, I want to run these checks in GitLab, which even has ways to integrate test reports and show accessibility issues within merge requests.

I’m used to building pipelines for regular repositories, but Netlify doesn’t have any proper integration with GitLab. Part of deploying a site from a Git repository is registering a webhook to notify it any time you push code. Netlify then uses GitLab’s API (in this case) to add an external pipeline stage after the fact, which can’t be controlled at all from GitLab. On merge requests, this stage tracks building Deploy Previews: the complete site, but built from the code in the merge request and hosted at a temporary domain. Because GitLab doesn’t know about this job, you can’t directly wait for the Deploy Preview to be available, and so can’t run checks on it once it’s ready. (I did try declaring an external stage in my pipeline, but it was ignored.) I decided to find a way to do it.

Hacking Netlify integration the wrong way

One of the comments in the GitLab issue above showed how to build the URI of the Deploy Preview. I created a single-job stage in my pipeline that would block using wait-for-it until the URI was available. This worked, in a sense, but because the URI never changes for a particular branch, the pipeline would only ever wait the first time a branch was pushed to. After that, the URI would always be available, even if a new version was being built in the background and there would be nothing to wait for.

So I had a way to reach the Deploy Preview, but I needed to wait until it was ready. My next thought was to use Netlify’s API to get a list of deploys and watch for some information to change. I could repeatedly fetch the list until the newest deploy had a state of building, then repeatedly fetch it again until the newest one had a state of ready. This is what the code looked like:

wait_for_deploy:
  stage: wait
  image:
    name: alpine:3.13.5
  only:
    - merge_requests
  timeout: 2m
  before_script:
    - apk add -q curl jq
  script:
    - |-
        until curl -sH "Authorization: Bearer $NETLIFY_AUTHORIZATION" $DEPLOY_LIST_URL | jq -e '.[0].state == "building"'; do
          sleep 1s
        done
    - |-
        until curl -sH "Authorization: Bearer $NETLIFY_AUTHORIZATION" $DEPLOY_LIST_URL | jq -e '.[0].state == "ready"'; do
          sleep 1s
        done
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: $DEPLOY_PREVIEW_URL
  interruptible: yes

(I configured NETLIFY_AUTHORIZATION in my CI/CD settings and placed the other variables in the global block. I set a timeout of two minutes because the deploy should be complete in less than that. I also globally set GIT_STRATEGY to none since none of my tasks needs the raw code: all of them interact with the Deploy Preview.)

This worked. I was able to push to a branch, wait for it to deploy, run code, push changes, and repeat the process. I set my sights on the next task: the code to run.

Setting up the checks

After looking around, I settled on broken-links-inspector to check my links. muffet was promising too, but the former had more useful features, including JUnit-style reports which GitLab could read. I experimented with it locally to arrive at the right incantations, then added it to my pipeline. I did have to make some concessions: use --single-threaded to prevent it from being rate-limited, increase the timeout a little, and add a single retry attempt. I also had to manually exclude numerous pages: a subreddit that doesn’t exist, a deliberately broken link to showcase the 404 page, a crates.io page that for some reason kept failing, TypeKit’s domains, and, funnily enough, shivjm.blog, because, as I realized after a while, the tool had added 9,000 views to my live site by following links in the Deploy Preview’s Atom feed. With those exclusions added and a handful of legitimately incorrect links fixed, the checker reported success.

I next wanted to add accessibility checking using pa11y.^[1] GitLab’s documented solution is entirely à la carte. It forces you to have fixed stages, jobs, and parameters. I dug deeper, adapting the contents of the referenced template to my specific case. That should have been enough, but I unwittingly overestimated GitLab’s variable expansion. I had set up my pipeline like this:

variables:
  SITE_NAME: netlify-site-1234
  DEPLOY_PREVIEW_DOMAIN: $SITE_NAME.netlify.app
  DEPLOY_PREVIEW_URL: https://deploy-preview-$CI_MERGE_REQUEST_IID--$DEPLOY_PREVIEW_DOMAIN/

check-accessibility:
  script:
    - do-something-with $DEPLOY_PREVIEW_URL

But the job kept failing because DEPLOY_PREVIEW_URL wasn’t receiving a last expansion pass, so what the script saw was the literal string https://deploy-preview-$CI_MERGE_REQUEST_IID--$DEPLOY_PREVIEW_DOMAIN/. I only found this out when I delved deeper, into the source for the Docker image the template used, and adapted the shell script to my specific case.

I did end up customizing the script for my case anyway, so it wasn’t a wasted effort. Instead of node:latest, I wanted to run the task on Alpine Linux; Puppeteer took offence at this, which reminded me that I have a lot of experience running Puppeteer on Alpine Linux, which reminded me that I built an image specifically to run Puppeteer on Alpine Linux… I switched and set the flags I needed to. Then I set up the configuration files like GitLab’s shell script did. Then I replaced the URIs with the ones I wanted. This exposed the problem I mentioned earlier. I had to reduce the amount of recursive expansion I relied on.

By the end of it, my task looked like this:

check_accessibility:
  stage: test
  variables:
    PA11YCI_CONFIG: |-
      {
      	"defaults": {
      		"chromeLaunchConfig": {
      			"args": [
      				"--no-sandbox"
      			],
            "hideElements": "pre > code"
      		}
      	}
      }
    PA11YJSON_CONFIG: |-
      {
      	"chromeLaunchConfig": {
      		"args": [
      			"--no-sandbox"
      		]
      	},
      	"includeWarnings": true,
      	"reporter": "html"
      }
    PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: 1
    PUPPETEER_EXECUTABLE_PATH: /usr/bin/chromium-browser
    urls: "/ colophon/multiple-simultaneous-immutable-responsive-images/"
  image:
    name: shivjm/node-chromium-alpine:node14.16.1-chromium81.0.4044.113
  before_script:
    - npm i pa11y@5.3.0 pa11y-reporter-html@1.0.0 pa11y-ci@2.3.0
    - echo $PA11YCI_CONFIG > .pa11yci
    - echo $PA11YJSON_CONFIG > pa11y.json
    - mkdir reports
  script:
    - export URLS="$DEPLOY_PREVIEW_URL ${DEPLOY_PREVIEW_URL}colophon/multiple-simultaneous-immutable-responsive-images/"
    - npx pa11y-ci -j --config .pa11yci $URLS > reports/gl-accessibility.json || true # ensure the job continues
    - |-
      for url in $URLS
      do
        filename="reports/$(echo $url | sed -E 's/^https?:\/\///' | sed -E 's/\//-/g')-accessibility.html"
        npx pa11y --config pa11y.json $url | tee $filename > /dev/null
      done
  allow_failure: true
  artifacts:
    when: always
    expose_as: 'Accessibility Reports'
    paths: ['reports/']
    reports:
      accessibility: reports/gl-accessibility.json
  only:
    - merge_requests

I had to configure pa11y to ignore pre > code because, as I have now discovered, the ostensibly accessible PrismJS themes I’m using do not in fact have sufficient contrast. I also had to add || true to the npx pa11y-ci call to make sure the individual page reports were generated regardless of whether it succeeded. The post I chose to test is one that’s a little lengthy and exercises several features.

The hack breaks immediately

My pipelines started failing. It took me a few attempts to understand that timing was the issue: the wait-for-deploy job assumed it would run concurrently with the Netlify deploy, but GitLab laughed at my naïveté and took minutes to schedule the jobs, which would therefore time out without finding an in-progress Netlify deploy and block the pipeline.

I couldn’t see a way to safely and repeatably identify the status of the deployment. I gave up, removed the wait_for_deploy job, and turned the test jobs into manual jobs.

Hacking Netlify integration the right way

Soon after I merged my changes, it struck me that the Netlify API was showering me with data I was ignoring. In fact, I could use a critical piece of information to synchronize GitLab and Netlify: the commit revision, which would necessarily be different every time I pushed the code. To put it in simple terms, all I had to do was wait until the first deploy with the right commit revision was ready. Phrasing it in a way that jq, YAML, and ash would all understand was rather more complex, but what I arrived at, eventually, was:

wait_for_deploy:
  stage: wait
  image:
    name: alpine:3.13.5
  only:
    - merge_requests
  timeout: 2m
  before_script:
    - apk add -q curl jq
  script:
    - |-
        until curl -sH "Authorization: Bearer $NETLIFY_AUTHORIZATION" $DEPLOY_LIST_URL | jq -e "[.[] | select(.state == \"ready\" and .commit_ref == \"$CI_COMMIT_SHA\" and .branch != \"master\")] | length | . > 0"; do
          sleep 1s
        done
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: $DEPLOY_PREVIEW_URL
  interruptible: yes

This works reliably! (The check for the branch isn’t necessary, but it makes me feel better.)

Thus it is that, a total of 11 commits and 51 pipelines later, I have a solution that accounts for all the hurdles I observed. All I need to do is push changes and GitLab will test my links and verify the basics of accessibility in the new Deploy Preview, no matter whether the jobs run before, during, or after its build. I hope more robust support for Netlify arrives soon, but this should suffice for the moment.

Next in series: Better Icons & URIs(#9 in Colophon: Finding A Place For My Head)