Monitoring

Monitoring

The observability stack runs in the monitoring namespace, each component its own ArgoCD application under kubernetes/apps/pitower/monitoring/. Prometheus scrapes metrics and remote-writes them to VictoriaMetrics, Fluent Bit ships container logs to VictoriaLogs, Tempo stores traces, and Grafana (run by the grafana-operator) queries all of them. Alertmanager routes alerts to Discord and ntfy.

Overview

flowchart LR
    subgraph Sources
        Apps[Applications\n/metrics]
        Logs[Container logs]
        Ext[External hosts\nOTLP]
        BB[Blackbox / smartctl /\nNUT / UniFi exporters]
    end

    subgraph Metrics
        Prom[Prometheus\n10d]
        VM[VictoriaMetrics\n2y]
    end

    subgraph Logs and Traces
        FB[Fluent Bit]
        VL[VictoriaLogs\n14d]
        OC[otel-collector]
        Tempo[Tempo\n72h]
    end

    AM[Alertmanager]
    Grafana[Grafana]
    Notify[Discord / ntfy]

    Apps & BB -->|ServiceMonitor / PodMonitor / Probe| Prom
    Prom -->|remote_write| VM
    Logs --> FB --> VL
    Ext -->|otlp.wibrow.dev| OC
    OC -->|traces| Tempo
    OC -->|logs| VL
    OC -->|metrics :8889| Prom
    Prom -->|alerts| AM --> Notify
    Prom & VM & VL & Tempo --> Grafana

Observability Strategy

• Metrics: applications expose /metrics; ServiceMonitors, PodMonitors, Probes and ScrapeConfigs tell Prometheus what to scrape. Prometheus keeps 10 days locally and remote-writes everything to VictoriaMetrics, which keeps 2 years.
• Logs: Fluent Bit runs on every node, tails /var/log/containers/, and pushes to VictoriaLogs (14 days), queried with LogsQL.
• Traces: apps send OTLP to the OpenTelemetry Collector, which forwards traces to Tempo. The OpenTelemetry Operator can auto-instrument pods via the monitoring/otel-instrumentation Instrumentation.
• Dashboards: GrafanaDashboard CRs, in any namespace, select the Grafana instance by label.
• Alerting: Alertmanager sends everything to Discord; alerts labelled notify="ntfy" also go to ntfy as push notifications.
• Uptime: Gatus checks every HTTPRoute it discovers; the public status page is a Cloudflare Worker.

Prometheus, VictoriaMetrics, VictoriaLogs and Tempo run on worker-07 (the only wibrow.dev/compute=true node) on openebs-hostpath-fast, the fast/extra ZFS dataset.

Components

Namespace Configuration

The monitoring namespace (created by the kube-prometheus-stack app) runs with privileged Pod Security Standards for node-exporter, Fluent Bit and smartctl-exporter, which need host access:

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged

Key Endpoints

Alert Silences

Before a cluster upgrade or other disruptive work (draining or rebooting nodes, storage/ZFS changes, gateway or CNI changes), silence Alertmanager so ntfy is not flooded, and expire the silence once the work is verified. See Prometheus Stack → Silences.

Key Design Decisions

• VictoriaMetrics for history -- Prometheus stays small (10d) while VictoriaMetrics holds 2 years of everything Prometheus scrapes.
• VictoriaLogs over Loki -- single binary, LogsQL, and fed directly by Fluent Bit's HTTP output.
• Operator-managed Grafana -- datasources and dashboards are CRs, so any app can ship its own dashboard with a GrafanaDashboard.
• Fluent Bit over Promtail -- low resource footprint and a flexible filtering pipeline.
• Status page outside the cluster -- a Cloudflare Worker can still report when the cluster is down.