feat(container): make dashboard portable and ready to use

[DrummyFloyd]

Description

Please describe the changes made by your PR. This description should be at a high-level but detailed enough that a reviewer understands the scope of the fix or enhancement and can easily judge the PRs validity at addressing the stated issue/feature. Please fully describe any new or changed feature and whether said change is user-facing or not:

• Fix missing cluster_name relabel configuration for Crunchy containers (used by many dashboards)
• Replaced all hardcoded datasource references with the template variable

Benefits

• Dashboards now work with any Prometheus datasource name
• Easier import process - users can map to their existing datasources
• Follows Grafana best practices for reusable dashboards
• Maintains full backward compatibility

Please indicate what kind of change your PR includes (multiple selections are acceptable):

• Bugfix
• Enhancement
• Breaking Change
• Documentation

PRs should be against existing issues, so please list each issue using a separate 'closes' line:

closes #482

If you have an external dependency (packages, portal updates, etc), add the 'BLOCKED' tag to your PR.

Testing

None of the testing listed below is optional.

Tested with IAC configurations and manually over my grafana instance (import + datasource selected after import)

IAC method

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  #   # INFO:https://github.com/CrunchyData/postgres-operator-examples/pull/285
  #   - https://github.com/bilbof/postgres-operator-examples/kustomize/monitoring/grafana/dashboards?ref=main
  - ./dashboards.yaml
  - ./podmonitor.yaml

configMapGenerator:
  - name: grafana-dashboards
    files:
      - https://raw.githubusercontent.com/DrummyFloyd/pgmonitor/refs/heads/fix-uid-prom/grafana/containers/pgbackrest.json
      - https://raw.githubusercontent.com/DrummyFloyd/pgmonitor/refs/heads/fix-uid-prom/grafana/containers/pgbouncer_direct.json
      - https://raw.githubusercontent.com/DrummyFloyd/pgmonitor/refs/heads/fix-uid-prom/grafana/containers/pod_details.json
      - https://raw.githubusercontent.com/DrummyFloyd/pgmonitor/refs/heads/fix-uid-prom/grafana/containers/postgresql_details.json
      - https://raw.githubusercontent.com/DrummyFloyd/pgmonitor/refs/heads/fix-uid-prom/grafana/containers/postgresql_overview.json
      - https://raw.githubusercontent.com/DrummyFloyd/pgmonitor/refs/heads/fix-uid-prom/grafana/containers/postgresql_service_health.json
      - https://raw.githubusercontent.com/DrummyFloyd/pgmonitor/refs/heads/fix-uid-prom/grafana/containers/query_statistics.json
generatorOptions:
  disableNameSuffixHash: true

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: pgbackrest
spec:
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      grafana: "home"
  folder: Databases
  resyncPeriod: 10m
  configMapRef:
    name: grafana-dashboards
    key: pgbackrest.json
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "prometheus"
---
# PGBouncer Direct Dashboard
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: pgbouncer-direct
spec:
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      grafana: "home"
  folder: Databases
  resyncPeriod: 10m
  configMapRef:
    name: grafana-dashboards
    key: pgbouncer_direct.json
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "prometheus"
---
# pod details
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: pod-details
spec:
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      grafana: "home"
  folder: Databases
  resyncPeriod: 10m
  configMapRef:
    name: grafana-dashboards
    key: pod_details.json
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "prometheus"
---
# # PostgreSQL Details Dashboard
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: postgresql-details
spec:
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      grafana: "home"
  folder: Databases
  resyncPeriod: 10m
  configMapRef:
    name: grafana-dashboards
    key: postgresql_details.json
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "prometheus"
#
---
# PostgreSQL Overview Dashboard
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: postgresql-overview
spec:
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      grafana: "home"
  folder: Databases
  resyncPeriod: 10m
  configMapRef:
    name: grafana-dashboards
    key: postgresql_overview.json
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "prometheus"

---
# PostgreSQL Service Health Dashboard
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: postgresql-service-health
spec:
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      grafana: "home"
  folder: Databases
  resyncPeriod: 10m
  configMapRef:
    name: grafana-dashboards
    key: postgresql_service_health.json
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "prometheus"
#
---
# # Query Statistics Dashboard
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: query-statistics
spec:
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      grafana: "home"
  folder: Databases
  resyncPeriod: 10m
  configMapRef:
    name: grafana-dashboards
    key: query_statistics.json
  datasources:
    - inputName: "DS_PROMETHEUS"
      datasourceName: "prometheus"
---

# INFO: https://github.com/CrunchyData/postgres-operator/issues/2199#issuecomment-2412062711
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: crunchy-postgres-exporter
spec:
  jobLabel: crunchy-postgres-exporter
  selector:
    matchLabels:
      postgres-operator.crunchydata.com/crunchy-postgres-exporter: "true"
  podMetricsEndpoints:
    - interval: 15s
      port: exporter
      scrapeTimeout: 10s
      relabelings:
        # Keep exporter port and drop all others
        # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod
        - sourceLabels: [__meta_kubernetes_pod_container_port_number]
          action: keep
          regex: "9187"
        # Set label for namespace
        - sourceLabels: [__meta_kubernetes_namespace]
          targetLabel: kubernetes_namespace
        # Set label for pod name
        - sourceLabels: [__meta_kubernetes_pod_name]
          targetLabel: pod
        # Convert namespace and cluster name to pg_cluster=namespace:cluster
        - sourceLabels:
            [
              __meta_kubernetes_namespace,
              __meta_kubernetes_pod_label_postgres_operator_crunchydata_com_cluster,
            ]
          targetLabel: pg_cluster
          separator: ":"
          replacement: "$1$2"
        # Convert kubernetes pod ip to ip
        - sourceLabels: [__meta_kubernetes_pod_ip]
          targetLabel: ip
        # Convert postgres-operator.crunchydata.com/instance to deployment
        - sourceLabels:
            [
              __meta_kubernetes_pod_label_postgres_operator_crunchydata_com_instance,
            ]
          targetLabel: deployment
        # Convert postgres-operator.crunchydata.com/role to role
        - sourceLabels:
            [__meta_kubernetes_pod_label_postgres_operator_crunchydata_com_role]
          targetLabel: role
        # Missing Cluster name for dashbaord
        - sourceLabels:
            [
              __meta_kubernetes_pod_label_postgres_operator_crunchydata_com_cluster,
            ]
          targetLabel: cluster_name
---

• Installation method:
  • Binary install from source, version:
  • OS package repository, distro, and version: kubernetes + Operator
  • Local package server, version:
  • Custom-built package, version:
  • Other:
• PostgreSQL, Specify version(s): 17
• docs tested with hugo version(s):

Code testing

Have you tested your changes against:

• RedHat/CentOS
• Ubuntu
• Not applicable

If your code touches sql_exporter, have you:

• Tested against all versions of PostgreSQL affected
• Ensure that exporter runs with no scrape errors
• Not applicable

If your code touches node_exporter, have you:

• Ensure that exporter runs with no scrape errors
• Not applicable

If your code touches Prometheus, have you:

• Ensured all configuration changes pass promtool check config
• Ensured all alert rule changes pass promtool check rules
• Prometheus runs without issue
• Alertmanager runs without issue
• [] Not applicable

If your code touches Grafana, have you:

• Ensured Grafana runs without issue
• Ensured relevant dashboards load without issue
• Not applicable

Checklist:

• I have made corresponding changes to:
  • the documentation
  • the release notes
  • the upgrade doc

[keithf4]

@andrewlecuyer
Could you review and test these to make sure this still works ok with your container installs? We'd done this same thing to the core dashboards and it worked great.

[benjaminjb]

I've tried this and it didn't work in our installer; also as has been pointed out both internally and externally, our container dashboards are a little out of date.

I have some thoughts about how to advance at this moment, but need to check with the team and other maintainers.

[DrummyFloyd]

I've tried this and it didn't work in our installer; also as has been pointed out both internally and externally, our container dashboards are a little out of date.

I have some thoughts about how to advance at this moment, but need to check with the team and other maintainers.

it's because all dashboards from postgres-operator-examples are out of date and not usable , that i came here .. and propose this fix ..

maybe be use an https resources like i did from the description in IAC method

NB: on my side, all dashboard work as expected ..

[benjaminjb]

hey @DrummyFloyd, just wanted to let you know that we have a ticket in the backlog to update and I'm glad this isn't blocking you. Out of curiosity, are you using the postgres_exporter to get metrics from your clusters?

[keithf4]

@DrummyFloyd What version of Grafana are you using?

[DrummyFloyd]

hey @DrummyFloyd, just wanted to let you know that we have a ticket in the backlog to update and I'm glad this isn't blocking you. Out of curiosity, are you using the postgres_exporter to get metrics from your clusters?

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: databases-home:monitoring.coreos.com/PodMonitor:databases/crunchy-postgres-exporter
  creationTimestamp: "2025-08-24T21:16:35Z"
  generation: 1
  name: crunchy-postgres-exporter
  namespace: databases
spec:
  jobLabel: crunchy-postgres-exporter
  podMetricsEndpoints:
  - interval: 15s
    port: exporter
    relabelings:
    - action: keep
      regex: "9187"
      sourceLabels:
      - __meta_kubernetes_pod_container_port_number
    - action: replace
      sourceLabels:
      - __meta_kubernetes_namespace
      targetLabel: kubernetes_namespace
    - action: replace
      sourceLabels:
      - __meta_kubernetes_pod_name
      targetLabel: pod
    - action: replace
      replacement: $1$2
      separator: ':'
      sourceLabels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_pod_label_postgres_operator_crunchydata_com_cluster
      targetLabel: pg_cluster
    - action: replace
      sourceLabels:
      - __meta_kubernetes_pod_ip
      targetLabel: ip
    - action: replace
      sourceLabels:
      - __meta_kubernetes_pod_label_postgres_operator_crunchydata_com_instance
      targetLabel: deployment
    - action: replace
      sourceLabels:
      - __meta_kubernetes_pod_label_postgres_operator_crunchydata_com_role
      targetLabel: role
    - action: replace
      sourceLabels:
      - __meta_kubernetes_pod_label_postgres_operator_crunchydata_com_cluster
      targetLabel: cluster_name
    scrapeTimeout: 10s
  selector:
    matchLabels:
      postgres-operator.crunchydata.com/crunchy-postgres-exporter: "true"

[DrummyFloyd]

@DrummyFloyd What version of Grafana are you using?

grafana-operator v5.19.4
grafana-image : 12.1.0

[keithf4]

Waiting on changes from CPK team to update dashboards for more recent versions of Grafana

[DrummyFloyd]

any news ?

[andrewlecuyer]

@DrummyFloyd we recently updated our Dashboards in the Postgres Operator examples repo:

https://github.com/CrunchyData/postgres-operator-examples/tree/main/kustomize/monitoring

When you have a chance, can you verify that these changes work for you?

[DrummyFloyd]

@DrummyFloyd we recently updated our Dashboards in the Postgres Operator examples repo:

CrunchyData/postgres-operator-examples@main/kustomize/monitoring

When you have a chance, can you verify that these changes work for you?

this is still not okay for IaC configuration (or with manual import with grafana UI)

with same kind: GrafanaDashboard

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: databases-home:grafana.integreatly.org/GrafanaDashboard:databases/pgbackrest
  name: pgbackrest
  namespace: databases
spec:
  allowCrossNamespaceImport: true
  configMapRef:
    key: pgbackrest.json
    name: grafana-dashboards
  datasources:
  - datasourceName: prometheus
    inputName: DS_PROMETHEUS
  folder: Databases
  instanceSelector:
    matchLabels:
      grafana: home
  resyncPeriod: 10m

because uid
is sitill hardcoded....

      "datasource": {
        "type": "prometheus",
        "uid": "PROMETHEUS"
      },