Published in Technology on A Place For My Head. Tagged , , , , , , , , , , , & . #6 in the The Death and Rebirth of a Cluster series.

Before I could install Linkerd as part of rebuilding my Kubernetes cluster, I needed to generate a root TLS certificate. For some reason, I struggled to do this with the hashicorp/tls Terraform provider, so I temporarily gave up on that approach. Instead, I created a certificate myself using step, as suggested, and proceeded to acquaint myself with Mozilla SOPS for secrets management slightly earlier than anticipated.

I was briefly confused because I thought SOPS might require some sort of complicated configuration. However, it has a simple model: it’s a stateless command-line program that encrypts and decrypts data with a predictable format. It can use cloud-based tools like KMS, but also local tools like GPG… which, as I’d conveniently forgotten, was why I’d selected it in the first place.

At any rate, using it with GPG requires the SOPS executable, the GPG executable, and the key. After creating a custom Argo CD image that added the first two, I wrapped the TLS certificate from earlier in a Kubernetes Secret (see below), encrypted it with SOPS locally (ditto), and committed it to my repository. Then I needed to provide the GPG key to Argo CD:

  1. Create a Secret containing the private key.
  2. Mount the Secret in argocd-repo-server using a projected volume.
  3. Use the last 16 characters of the key’s ID as the filename.

However, even though the logs said the key had been imported:

Outputtime="2021-08-13T12:12:58Z" level=info msg="Loaded 1 (and removed 0) keys from keyring"

…I couldn’t see it in the Argo UI, it didn’t show up using kubectl exec -it -n argocd deploy/argocd-repo-server -- gpg --list-secret-keys, and it wasn’t available to decrypt secrets:

Outputrpc error: code = Unknown desc = Manifest generation error (cached): `kustomize build /tmp/https:__foo/base --enable-alpha-plugins` failed exit status 1: trouble decrypting file Error getting data key: 0 successful groups required, got 0Error: failure in plugin configured via /tmp/kust-plugin-config-066333420; exit status 1: exit status 1

jannfis on the CNCF Slack thought I needed a custom plugin, but I found a simpler answer on GitHub: import the key in an init container and use GNUPGHOME. I promptly implemented this.

I later migrated from the antiquated and insecure GPG to age, a modern encryption tool with sensible defaults. As I was only accessing secrets through the kustomize-sops plugin (KSOPS) that my image included, all I had to do was:

  1. Replace GPG with age in the image.
  2. Re-encrypt the secrets locally (see below).
  3. On the Argo side: replace the file and remove the extra configuration.

Useful commands

Create a Kubernetes Secret as YAML:

Bashkubectl create secret generic -o yaml --dry-run=client -n linkerd linkerd-root --from-literal=tls.crt=$PUBLIC_KEY --from-literal=tls.key=$PRIVATE_KEY > linkerd-secret.yaml

Generate age keys:

Bashage-keygen > keys

# Repeat to create as many keys as you need, which you can safely
# store in the same file:
age-keygen >> keys

Encrypt a file with SOPS:

Bashsops -e --pgp $PUBLIC_KEY1,$PUBLIC_KEY2 linkerd-secret.yaml > linkerd-secret.enc.yaml

Rotate keys with SOPS:

BashSOPS_AGE_KEY_FILE=/path/to/age/keys
sops -r -i --rm-pgp $PUBLIC_KEY1,$PUBLIC_KEY2 --add-age $AGE_PUBLIC_KEY1,$AGE_PUBLIC_KEY2 base/secrets/external-dns.yaml

Next in series: (#7 in The Death and Rebirth of a Cluster)