Improve ClickHouse Keeper Grafana Dashboard

[discostur]

Fix ClickHouse Keeper Grafana Dashboard

fix #1844

Problem

The ClickHouse Keeper dashboard was not working correctly due to:

1. Incorrect Prometheus label names (pod_name vs pod, container_name vs container)
2. Template variables using incorrect label filters
3. Hardcoded metric names in template variable queries

Changes

Label Fixes

• Changed pod_name → pod in all queries and template variables
  • Prometheus/Mimir exports Kubernetes pod labels as pod, not pod_name
• Removed container_name="clickhouse-keeper" filters from all panel queries
  • Label should be container, not container_name
  • Filter not necessary as queries are already scoped to Keeper-specific metrics
• Updated legend formats from {{pod_name}} to {{pod}} across all panels

Template Variable Improvements

Before:

label_values(up{container_name="clickhouse-keeper"}, namespace)
label_values(up{container_name="clickhouse-keeper", namespace=~"$namespace"}, pod_name)

After:

label_values({__name__=~"ClickHouse.*Keeper.*",pod=~"chk-clickhouse-keeper.*"}, namespace)
label_values({__name__=~"ClickHouse.*Keeper.*",pod=~"chk-clickhouse-keeper.*",namespace=~"$namespace"}, pod)

Benefits:

• Uses generic metric selector instead of up metric with incorrect label
• Explicit pod name filter (chk-clickhouse-keeper.*) ensures only Keeper pods are shown
• Works with any ClickHouse Keeper metric, not dependent on specific metric availability

[Slach]

thanks for contribution, really appreciated to sync metrics names

look to https://github.com/Altinity/clickhouse-operator/blob/master/deploy/prometheus/prometheus-template.yaml#L52-L54

in our prometheus scraping config we use pod_name, not pod

please use following commands to check your dashboard changes

minikube delete
minikube start
kubectl create ns test
helm install -n test clickhouse-operator ./deploy/helm/clickhouse-operator/

NO_WAIT=1 bash -xe ./deploy/prometheus/create-prometheus.sh
NO_WAIT=1 bash -xe ./deploy/grafana/grafana-with-grafana-operator/install-grafana-operator.sh
NO_WAIT=1 bash -xe ./deploy/grafana/grafana-with-grafana-operator/install-grafana-with-operator.sh

kubectl apply -f ./docs/chk-examples/01-chi-simple-with-keeper.yaml 

[discostur]

@Slach ah i did miss the relabel config ... ok with that it works, correct. However i think this is kind of outdated and all other projects these days just use pod and container as labels instead of pod_name and container_name. So my suggestion would be to use the updated dashboard i provided and adjust the relabel config to be more streamlined to other metrics. In addition we would not "break" other setups because the relabel config would relabel all pod labels which have the annotation "prometheus_io_scrape=true".

[discostur]

I provided the corresponding changes and some minor fixes to the test setup itself you recommended ;)

I tried it with minikube and all metrics are working now:

[Slach]

too many changes which not related to dashboard improvements
thanks to adopt script to MacOS

But please don't break name conversion, look to initial prometheus scraping config and rollback to pod_name and container_name

[Slach]

i don't understand what exactly you trying to improve here? I don't see improvements, only breaks old name conversions

up{container_name='clickhouse-keeper'} will have the same result with {__name__=~\"ClickHouse.*Keeper.*\",pod=~\"chk-.*\"

both could be broken

chk-.* could be broken with chk.spec.templates.podTemplates.generateName
and clickhouse-keeper container name could be broken with chk.spec.templates.podTemplates.spec.containers

[Slach]

up is metrics which generated from prometheus itself after Kubernetes pod discovery.

[discostur]

@Slach i agree; I just added a fix for the dashboard to load the prometheus datasource correct; It works now with the relabel config you pointed me to.