Monitoring

Grafana

Grafana is the visualization layer for metrics, logs and traces. It is managed by the grafana-operator: the Grafana instance, its datasources and every dashboard are Kubernetes custom resources. Users sign in through Kanidm OIDC.

Layout

kubernetes/apps/pitower/monitoring/grafana-operator/ is a single ArgoCD application with two kustomizations, ordered by sync waves:

flowchart LR
    subgraph monitoring
        OP[grafana-operator]
        G[Grafana CR\nlabel dashboards: grafana]
        DS[GrafanaDatasource CRs]
        GD[GrafanaDashboard CRs]
    end
    subgraph other namespaces
        GD2[GrafanaDashboard CRs\nrook-ceph, ai, ...]
    end
    OP -->|reconciles| G
    DS -->|instanceSelector| G
    GD -->|instanceSelector| G
    GD2 -->|instanceSelector\nallowCrossNamespaceImport| G

Every datasource and dashboard selects the instance with instanceSelector.matchLabels: {dashboards: grafana}.

Data Sources

Tempo links traces to logs in VictoriaLogs and to metrics/service maps in Prometheus.

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: victoriametrics
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana
  datasource:
    name: VictoriaMetrics
    uid: victoriametrics
    type: prometheus
    access: proxy
    url: http://victoria-metrics-server.monitoring.svc.cluster.local:8428

Dashboards

Dashboards are GrafanaDashboard CRs. Each sets a folder and takes its JSON from one of three sources:

kube-prometheus-stack also emits its bundled dashboards as GrafanaDashboard CRs (grafana.operator.dashboardsConfigMapRefEnabled: true) into the Kubernetes folder.

Folders in use: AI, CI, Home Assistant, Infrastructure, Kubernetes, Networking, Observability, Status, Storage.

Adding a dashboard

Drop the JSON next to the app, generate a ConfigMap, and point a CR at it:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  folder: Infrastructure
  allowCrossNamespaceImport: true
  instanceSelector:
    matchLabels:
      dashboards: grafana
  configMapRef:
    name: my-app-dashboard
    key: my-app.json

allowCrossNamespaceImport: true is required outside the monitoring namespace.

Authentication

Grafana authenticates users via Kanidm (idm.wibrow.dev) with OpenID Connect and auto-login. Kanidm groups map to Grafana roles:

auth.generic_oauth:
  enabled: "true"
  name: Kanidm
  client_id: grafana
  scopes: openid profile email groups
  auth_url: https://idm.wibrow.dev/ui/oauth2
  token_url: https://idm.wibrow.dev/oauth2/token
  api_url: https://idm.wibrow.dev/oauth2/openid/grafana/userinfo
  use_pkce: "true"
  role_attribute_path: contains(groups[*], 'admins@idm.wibrow.dev') && 'Admin' || contains(groups[*], 'people@idm.wibrow.dev') && 'Viewer'

Secrets Management

All secrets come from Infisical through ExternalSecrets:

Plugins

Plugins are installed through GF_INSTALL_PLUGINS on the Grafana container:

Storage and Rendering

• Persistence: a 5Gi ceph-block PVC holds Grafana's SQLite DB, so sessions and state survive pod rebuilds (the operator default is an emptyDir). The Deployment uses Recreate.
• Image renderer: a separate grafana-operator-image-renderer Deployment (grafana/grafana-image-renderer) serves PNG rendering.

Access

Grafana is exposed at https://grafana.wibrow.dev through envoy-external, reachable from the internet via the towonel tunnel. Gatus checks /api/health through an HTTPRoute annotation.

Configuration Reference