Grafana, Persistence, and Argo CD
Although I’d already set up Prometheus and Grafana as part of rebuilding my Kubernetes cluster, I didn’t fully understand how to deal with persistence. kube-grafana recreates everything with every deployment. That includes reverting to a single admin user and the default settings. Provisioning only applies to data sources and dashboards so far.
I’d already set the default theme to light
, but it didn’t seem to work. I thought I’d have to
create a Job that uses the Grafana API to change the
theme. If I wanted to use
kubectl to watch for the Grafana Pod to be ready first, I’d need a ServiceAccount, Role, and
RoleBinding, after which I’d be able to run my actual requests. Or I could dispense with
authentication and check the Grafana health
endpoint directly.
Fortunately, all this was unnecessary. I found I hadn’t put the default theme in the right place. Once I’d fixed that, I no longer needed the complex Job machinery.
I also intended to look for an Argo CD dashboard at some point, so I enabled scraping the metrics by adding the ServiceMonitors from the documentation. (I submitted a tiny correction to get it right.)