Canceling the Apocalypse with Jaeger
With Prometheus for metrics and Loki for logs both in place as part of rebuilding my Kubernetes cluster, the last piece of the observability puzzle was tracing, for which I’d previously used Jaeger. It’s easy enough to deploy in and of itself, but it requires ElasticSearch for storage and Kafka for ingestion, both of which are a nuisance to configure and maintain. I’d heard about a promising alternative from Grafana called Tempo. The use of object storage and replication instead of heavyweight, stateful dependencies appealed to me, as did the simplicity of the application, but search and other features seemed too immature at the time,[1] and I was too inexperienced to safely experiment.
I deployed Jaeger in the same way as before through Jsonnet (remembering just in time not to give ES a password with spaces) and added it as a data source in Grafana. Initially, Argo CD kept deleting the Secrets I created, preventing ES from starting. I was forced to manually sync those. I’m not sure what ultimately fixed the problem. All I knew was that I’d got it working somehow and I couldn’t wait to switch to Tempo someday.
- The search was improved eventually, though with performance caveats. More on that in a future entry.↩