Loki¶

Loki is a horizontally scalable log aggregation system inspired by Prometheus. In the cluster, Loki runs in SingleBinary mode and receives logs from Fluent Bit, storing them on local disk via OpenEBS.

Architecture¶

flowchart LR
    subgraph Nodes
        FB1[Fluent Bit\nNode 1]
        FB2[Fluent Bit\nNode 2]
        FB3[Fluent Bit\nNode N]
    end

    subgraph monitoring namespace
        Loki[Loki\nSingleBinary]
        PV[(OpenEBS\n20Gi PVC)]
    end

    Grafana[Grafana]

    FB1 & FB2 & FB3 -->|push logs| Loki
    Loki -->|persist| PV
    Grafana -->|LogQL queries| Loki

Deployment Mode¶

Loki is deployed in SingleBinary mode with a single replica. This keeps the deployment simple and resource-efficient for a home lab environment. All Loki components (ingester, querier, compactor, ruler) run in a single process.

deploymentMode: SingleBinary

singleBinary:
  replicas: 1
  persistence:
    enabled: true
    storageClass: openebs-hostpath
    size: 20Gi

All distributed mode components are explicitly disabled:

backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0
gateway:
  replicas: 0
chunksCache:
  enabled: false
resultsCache:
  enabled: false
lokiCanary:
  enabled: false

Storage¶

Loki uses filesystem storage backed by an OpenEBS hostpath PVC. Logs are stored locally on the node where Loki is scheduled.

loki:
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        index:
          prefix: loki_index_
          period: 24h
        object_store: filesystem
        schema: v13
  storage:
    type: filesystem

Retention¶

Loki is configured with a 14-day retention period. The compactor runs in the background to delete expired log chunks:

loki:
  compactor:
    working_directory: /var/loki/compactor/retention
    delete_request_store: filesystem
    retention_enabled: true
  limits_config:
    retention_period: 14d

Configuration¶

Key Loki settings:

Querying with LogQL¶

Loki uses LogQL, a log query language inspired by PromQL. Queries can be run in the Grafana Explore view with the Loki data source selected.

Common Queries¶

All logs from a specific namespace:

{namespace="networking"}

Logs from a specific pod:

{namespace="monitoring", pod=~"loki.*"}

Filter logs containing an error:

{namespace="networking"} |= "error"

Parse JSON logs and filter by field:

{namespace="networking", job="fluent-bit"} | json | level="error"

Count log lines per namespace over time:

sum by (namespace) (rate({job="fluent-bit"}[5m]))

Top 10 pods by log volume:

topk(10, sum by (pod) (rate({job="fluent-bit"}[1h])))

Integration with Grafana¶

Loki is configured as a data source in Grafana at the cluster-internal URL:

- name: Loki
  type: loki
  access: proxy
  url: http://loki-headless.monitoring.svc.cluster.local:3100

This enables:

• Explore -- ad-hoc log querying with LogQL
• Dashboard panels -- embedding log panels alongside metric panels
• Annotations -- overlaying log events on time-series graphs
• Alerting -- creating alert rules based on log patterns (when Alertmanager is enabled)

Sidecar Rules¶

The Loki Helm chart includes a sidecar that watches all namespaces for recording and alerting rules:

sidecar:
  rules:
    searchNamespace: ALL

This allows applications in any namespace to define Loki recording rules via ConfigMaps.

Helm Chart Reference¶