Published in Technology on A Place For My Head. Tagged , , , & . #7 in the The Death and Rebirth of a Cluster series.

Now that I had Argo CD working, I could start setting up the applications I wanted to put in my rebuilt Kubernetes cluster. Adding the app-of-apps to Argo at the time of installation resulted in problems and broke Sync Waves. Deleting and recreating it afterwards worked.

On the advice of Frode Egeland on Slack, I switched to the newer ApplicationSet resource, which resolved my problems. It required adding the controller and converting my manifest, as well as defining the cluster in a JSON file for the Git generator.

Along the way, I ended up publicly releasing my custom image and updating Argo to v2.1.0. Editing the upstream manifests directly as I was doing until then made it hard to manage, so I refactored my code to Kustomize patches, separating my personal tweaks from the original Argo CD files.

I noticed that deleting Applications is problematic. kubectl delete seems to hang forever. The underlying cause is that the resources inside those Applications often misbehave and take a long time to fully remove. Deletion in the UI with the background delete policy works (inconsistently) by not waiting for it to complete. Adding the Argo CD finalizer is highly recommended, though it won’t make this issue go away.

I separately learnt to add the SkipDryRunOnMissingResource annotation to Custom Resources when installing them alongside their Custom Resource Definitions (CRDs). Without the annotation, Argo’s dry run fails because of the CRD not having been installed yet and syncing becomes impossible.

Next in series: (#8 in The Death and Rebirth of a Cluster)