Published in Technology on A Place For My Head. Tagged , , , , , , & . #18 in the The Death and Rebirth of a Cluster series.

Although I’d already set up Prometheus and Grafana as part of rebuilding my Kubernetes cluster, I didn’t fully understand how to deal with persistence. kube-grafana recreates everything with every deployment. That includes reverting to a single admin user and the default settings. Provisioning only applies to data sources and dashboards so far.

I’d already set the default theme to light, but it didn’t seem to work. I thought I’d have to create a Job that uses the Grafana API to change the theme. If I wanted to use kubectl to watch for the Grafana Pod to be ready first, I’d need a ServiceAccount, Role, and RoleBinding, after which I’d be able to run my actual requests. Or I could dispense with authentication and check the Grafana health endpoint directly.

Fortunately, all this was unnecessary. I found I hadn’t put the default theme in the right place. Once I’d fixed that, I no longer needed the complex Job machinery.

I also intended to look for an Argo CD dashboard at some point, so I enabled scraping the metrics by adding the ServiceMonitors from the documentation. (I submitted a tiny correction to get it right.)