Published in Technology & Work on A Place For My Head. Tagged , , & .

I discovered Rust in 2016 and I’ve been eager to use it on a major project ever since, but I never found the right opportunity until I was brought onto a friend’s side project a few months ago. My first stab at the (very simple) backend—let’s call it v0.1—in Rust was unsuccessful; I rewrote it in Python—let’s call this v1.0—in the span of two days; and then, because nothing much was happening, I spent the next two months working on the Rust version again until it was functional. This v2.0 is better in almost every way.

Along the way, I’d come across a few references to building completely statically-linked binaries for Rust programs, such as ‘Use Multi-Stage Docker Builds For Statically-Linked Rust Binaries’. They made it sound quite simple, so I thought I’d do that for backend v2.0 too. The only necessary changes to my otherwise-simple Dockerfile seemed to be:

  1. Add the MUSL target.
  2. Compile against the MUSL target.
  3. Copy the MUSL target into a scratch image.

Easy! I made the changes, built the image, and tried it out:

# docker run -it --rm backend
standard_init_linux.go:211: exec user process caused "no such file or directory"

That was a strange error, coming as it did from Docker itself. I double-checked my Dockerfile, rebuilt it, and tried again, but the result was the same. Google wasn’t of much help either. I asked on the official Rust Discord; no answer.

I spent a week trying to reproduce the problem in a small example crate, but I was never quite able to narrow it down that way. Eventually, I stumbled upon a very informative Stack Overflow answer:

The problem was that for each crate providing a native dependency – say OpenSSL – there is the build.rs build script that is in charge of communicating the build and linking options to Cargo and to rustc. (For example: they print out something like cargo:rustc-link-lib=static=ssl which Cargo then reads and acts accordingly.)

So just setting the "standard" GCC environmental variables is hardly going to have any effect. You must check each and every build.rs separately to know how to coerce that exact crate to convey cargo its options. For OpenSSL, its env vars like OPENSSL_DIR, OPENSSL_STATIC etc.

This made sense in conjunction with other things I’d read about the error relating to missing dynamic libraries. I wasn’t directly using any crates with native dependencies, but had a transitive dependency on openssl-sys, and perhaps others as well, and what I needed to do was compile those statically when compiling my program.

The Solution

I shall elide a lot of the missteps along the way. This is the Dockerfile I arrived at:

FROM ekidd/rust-musl-builder:1.46.0 AS builder

ENV CARGO_INCREMENTAL=0
ENV USER=rust

WORKDIR /home/rust/src
RUN cargo new backend
WORKDIR /home/rust/src/backend
COPY Cargo.toml Cargo.lock ./
RUN OPENSSL_LIB_DIR=/usr/local/musl/lib/ OPENSSL_INCLUDE_DIR=/usr/local/musl/include OPENSSL_STATIC=1 cargo build --target x86_64-unknown-linux-musl --release --locked

COPY src ./src
RUN OPENSSL_LIB_DIR=/usr/local/musl/lib/ OPENSSL_INCLUDE_DIR=/usr/local/musl/include OPENSSL_STATIC=1 cargo build --target x86_64-unknown-linux-musl --release --frozen --offline

FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ENV SSL_CERT_DIR=/etc/ssl/certs/
COPY --from=builder /home/rust/src/backend/target/x86_64-unknown-linux-musl/release/backend /usr/app/backend
USER 1000
CMD ["/usr/app/backend"]

Et voilà ! A single, working, statically-linked binary. This one uses the rust-musl-builder image, which contains a static version of OpenSSL, along with a few other things. Note that, unlike in the dev.to article that I linked at the beginning, I’m not using cargo install when I build my actual code. Instead, I’m repeating cargo build --release. cargo install only appears to be useful when you need to put the result in a predictable location; it recompiles the dependencies, too, rendering our previous step pointless. With cargo build, no dependencies need to be recompiled, and I copy the binary right from the ‘target’ directory.

Since the application relies on OpenSSL, I also copy the certificates from the previous step.

The Result (According to GitLab)

Before: 783.69 MiB
After: 5.09 MiB

Update (2021-04-10)

Rust 1.51.0 was released on the 25th of March. rust-musl-builder hasn’t yet been updated and shows no signs of life, so I was stuck on 1.49.0. I didn’t really need to use the newer version, but it irked me that I was being held back by a builder image. Given that I had to explicitly switch to rustls when I updated sqlx recently, I thought I would investigate the possibility of removing rust-musl-builder from my build. What I ended up needing to do was this:

  1. Switch to the rust:1.51-alpine image.

  2. Add the musl-dev and perl packages.

  3. Spend half an hour trying to figure out why it was still trying to compile OpenSSL. I couldn’t find openssl in the output from cargo tree, so I had no idea where it was coming from. I asked on Discord, where someone finally pointed out that I wasn’t seeing it because I was running the command on Windows, where the openssl crate would link schannel. I could find it by specifying the same target as in the Docker build.

    Meanwhile, by the time they told me this, I had already found that rusoto-core had features to control which SSL implementation it used. I had to enable rustls there and in rusoto_s3 to remove OpenSSL from my build.

  4. Remove any references to the rust user that rust-musl-builder expects you to work as.

That fixed the main binary. Next, I had to tackle the image I use for testing, which runs my Movine migrations, thereby requiring native-tls. I spent a while flipping between the documentation, examples, and source code for postgres (the synchronous client), tokio-postgres (the asynchronous client), postgres-native-tls (which provides native TLS support for both clients), and tokio-postgres-rustls (which only mentions the tokio-postgres asynchronous PostgreSQL client library). It was unclear to me whether I could use rustls with the synchronous client. All I found on the subject was a 3-year-old, archived repository called rust-postgres-rustls.

When I looked at the tokio-postgres-rustls examples, I realized it didn’t look all that different in use from postgres-native-tls, despite the former ostensibly being aimed at asynchronous code and the latter at synchronous code. I decided to try just mashing everything together. I moved the native-tls–dependent code in Movine into a new feature that was enabled by default and created a new rustls feature pulling in tokio-postgres-rustls and rustls, and created a feature-gated function for each case. At first, I got errors about incompatible types. I asked on Tokio Discord, but there was no response. Out of desperation, I tried simply updating the libraries… et voilà encore ! Everything just worked. I opened a merge request that was merged within a day, and that was the last of the OpenSSL in my code.