Monitoring

OpenTelemetry Collector

The OpenTelemetry Collector (otel-collector, deployed via the OpenTelemetry Operator) is pitower's single ingest point for telemetry that doesn't come from in-cluster Prometheus scraping or Fluent Bit -- most notably, hosts that live outside the cluster entirely, like the ovh-vps box running towonel, the tunnel hub *.wibrow.dev traffic goes through. It fans traces, metrics, and logs out to Tempo, VictoriaMetrics, and VictoriaLogs respectively.

Architecture

flowchart LR
    subgraph "ovh-vps (public internet)"
        Agent["otel-agent\n(otelcol-contrib)"]
        HM["hostmetrics"]
        JD["journald\n(towonel-hub, fail2ban, ssh)"]
        HM --> Agent
        JD --> Agent
    end

    subgraph "pitower cluster"
        EG["Envoy Gateway\nenvoy-external\notlp.wibrow.dev"]
        OC["otel-collector"]
        Tempo["Tempo"]
        VM["VictoriaMetrics\n(via kube-prometheus-stack\nscrape + remote_write)"]
        VL["VictoriaLogs"]
    end

    Agent -->|"OTLP/HTTP\nX-API-Key"| EG
    EG --> OC
    OC -->|traces| Tempo
    OC -->|"prometheus exporter :8889\nscraped by ServiceMonitor"| VM
    OC -->|"otlphttp\n/insert/opentelemetry/v1/logs"| VL

External OTLP Ingest

Any external client -- currently just ovh-vps -- can push telemetry in over https://otlp.wibrow.dev without a VPN or private network path:

A client only needs the standard OTLP env vars:

OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.wibrow.dev
OTEL_EXPORTER_OTLP_HEADERS=X-API-Key=<key>

The same OTLP/HTTP port serves all three signals (/v1/traces, /v1/metrics, /v1/logs), so adding a new signal to the collector's pipeline never requires a HTTPRoute/SecurityPolicy change -- only a pipeline change in the OpenTelemetryCollector config.

Pipelines

ovh-vps: otel-agent

ovh-vps is a public OVH VPS running towonel-hub, outside the cluster and outside the home VLAN. It ships its own telemetry in via the external OTLP ingest above, using the otel-agent Ansible role (ansible/roles/otel-agent/, applied by just ansible deploy-ovh-vps).

otel-agent runs otelcol-contrib as a native systemd service -- a pinned, checksum-verified GitHub release binary (same pattern as the oha role), not another docker run. Two receivers need the real host, not a container:

• hostmetrics -- cpu/memory/disk/filesystem/network/load, scraped every 60s.
• journald -- scoped to towonel-hub.service, fail2ban.service, and ssh.service rather than the whole journal firehose. towonel-hub.service's unit runs docker run --rm in the foreground (no -d), so towonel + Caddy's own stdout/stderr already land in the journal under that unit -- no separate docker-log plumbing needed. The receiver execs the host's journalctl directly, which only works running natively.
• A loopback-only otlp receiver (127.0.0.1:4317/4318) is also wired up so any future local process can emit spans without its own collector. Nothing on the host is instrumented yet, so the traces pipeline is currently idle by design.

The service runs as a dedicated otel-agent system user (no shell, no home directory), added only to the systemd-journal group -- never as root.

Gotchas Found Running This in Production

Two real bugs surfaced while wiring ovh-vps up, both worth knowing if you touch this stack again.

Verifying Data Is Flowing

# On ovh-vps
systemctl status otel-agent
journalctl -u otel-agent -f

# In pitower: metrics
kubectl -n monitoring exec victoria-metrics-server-0 -- \
  wget -qO- 'http://127.0.0.1:8428/api/v1/label/host_name/values'

# In pitower: logs
kubectl -n monitoring port-forward victoria-logs-0 9428:9428 &
curl -s 'http://localhost:9428/select/logsql/query' -d 'query=host.name:ovh-vps | limit 5'

Reference