Self-Hosted Netlify Builds with GitLab CI
Once I had (mostly) recovered from a bungled Git LFS migration, I had no trouble switching from Netlify Large Media to eleventy-img. At least, I could do it in my local working tree. The next step was to make it work as part of my build. That was when the trouble began.
I wanted to move from having Netlify build the site to building it myself with GitLab CI on my own
runners in Kubernetes. (This would provide more control over the build environment and avoid
devouring the limited build minutes Netlify provides for free.) netlify
deploy will deploy any files under the output directory
as a draft; --alias foo will add
foo to the draft URL and --prod will deploy it to
the production site.
However, I wasn’t able to build the site in a Linux Docker container. I got an error referring to
dir in the esm library. esm allows CommonJS modules and ECMAScript Modules
or ESM to interoperate; the original library is no
longer maintained, so I use an also-no-longer-maintained fork.
Eleventy doesn’t support ESM but many modern
libraries are only available in that format.
Poring over the output of npm ls and even examining the esm library itself revealed no differences between my (working) Windows environment and the Linux environment other than the presence or absence of the fsevents module. However, I did find that the Windows version could only be built with that specific, existing node_modules directory; deleting and replacing the directory would stop the build from working.
I spent some time trying—as I have done twice before—to convert my blog entirely to ESM (which Node natively supports). The stumbling block continued to be Eleventy. I reverted to the esm workaround.
Through trial and error, I realized it would fail to build in Node v14 and wouldn’t run at all in v18, but the esm errors went away with v16 under both Windows and Linux. The next step was to get sharp (the image processing library powering eleventy-img) working on Linux: Windows worked with a prebuilt binary, but the version of libvips in the Linux image was too old. sharp couldn’t successfully compile a custom version, either.
I ended up creating a Docker image with Node and libvips preinstalled. That solved the problem, although I wasted a great deal of time between some silly issues caused by using a tmpfs mount and forgetting entirely that I was using LFS.
I had to add a dedicated CI machine to my Kubernetes pool because the build was chewing up 20 GB or more of RAM with the image processing… certainly not normal, considering only 10 images are being processed at a time, but I have no idea how to fix my code right now. (On my local machine, it stays under 2 GB in the host Windows but hits 20 GB under Linux in Docker.) At any rate, the first successful run took 45 minutes. After that, the cache brought it down to two.
Here are the relevant parts of my GitLab pipelines:
variables: LIBVIPS_IMAGE: shivjm/node-libvips:node16-libvips8.14.1 draft: stage: draft image: $LIBVIPS_IMAGE variables: GIT_DEPTH: 3 cache: paths: - dist/assets/images/resized - node_modules key: draft when: on_success script: - npm install - NODE_ENV=production npm run build - npx netlify deploy --alias $CI_COMMIT_REF_SLUG environment: name: draft/$CI_COMMIT_REF_SLUG url: https://$CI_COMMIT_REF_SLUG--$NETLIFY_SITE_ID.netlify.app/ only: - merge_requests deploy: stage: deploy image: $LIBVIPS_IMAGE variables: GIT_DEPTH: 3 CONTEXT: production cache: paths: - dist/assets/images/resized - node_modules key: deploy when: on_success script: - npm install - NODE_ENV=production npm run build - npx netlify deploy --prod environment: name: production url: https://shivjm.blog/ only: - main
NETLIFY_SITE_ID are defined as variables elsewhere. I set
a low number since I don’t need any history. I configure eleventy-img to produce images at
dist/assets/images/resized, so that’s what I cache.
Note that GitLab ‘stops’ an environment for a merge request once it’s merged. I could add an
on_stop job to delete the corresponding deployment from Netlify, but I’d rather keep them around
for as long as possible (only 90 days, as it happens).
Next in series: From broken-links-inspector to muffet(#34 in Colophon: Finding A Place For My Head)