Grafana, Persistence, and Argo CD
Although I’d already set up Prometheus and Grafana as part of rebuilding my Kubernetes cluster, I didn’t fully understand how to deal with persistence. kube-grafana recreates everything with every deployment. That includes reverting to a single admin user and the default settings. Provisioning only applies to data sources and dashboards so far.
I’d already set the default theme to light, but it didn’t seem to work. I thought I’d have to
create a Job that uses the Grafana API to change the
theme. If I wanted to use
kubectl to watch for the Grafana Pod to be ready first, I’d need a ServiceAccount, Role, and
RoleBinding, after which I’d be able to run my actual requests. Or I could dispense with
authentication and check the Grafana health
endpoint directly.
Fortunately, all this was unnecessary. I found I hadn’t put the default theme in the right place. Once I’d fixed that, I no longer needed the complex Job machinery.
I also intended to look for an Argo CD dashboard at some point, so I enabled scraping the metrics by adding the ServiceMonitors from the documentation. (I submitted a tiny correction to get it right.)