Observability

Three-pillar observability: metrics (Prometheus + Grafana), traces (OpenTelemetry + Tempo), and logs (Loki + Promtail).

Metrics — Prometheus + Grafana

Instrumentation

The API exposes a Prometheus /metrics endpoint at every pod, implemented with the metrics + metrics-exporter-prometheus crates.

Scraping in Kubernetes

When serviceMonitor.enabled: true, the chart deploys a ServiceMonitor CRD that tells the Prometheus Operator to scrape the API pod's /metrics every 30 seconds.

values.yaml
serviceMonitor:
  enabled: true
  interval: 30s
  namespace: monitoring

Grafana dashboard

The chart deploys a ConfigMap with label grafana_dashboard: "1". Grafana's sidecar discovers it automatically — no manual import needed.

API row

• Request rate by status (req/s)
• 5xx error rate (%, thresholds: green < 1%, yellow < 5%, red ≥ 5%)
• Latency p50 / p95 / p99 (seconds)

Push Delivery row

• NATS consumer lag on PUSH_BATCHES (messages pending per consumer)
• Worker pods available (stat — red < 1, yellow < 2, green ≥ 2)
• API pods available

Infrastructure row

• CPU usage by container (cores)
• Memory working set by container (bytes)
• Analytics PVC utilisation (gauge — yellow > 80%, red > 95%)

Logs row

• API container logs (streaming from Loki)
• Worker container logs

Database row

• PostgreSQL up/down
• Active DB connections by state
• Replication lag (seconds)

Alerts — Alertmanager

PrometheusRule — 14 alert rules

Deployed when serviceMonitor.enabled: true. Five groups:

visionpush.availability

visionpush.http

visionpush.delivery

visionpush.storage

visionpush.database

AlertmanagerConfig routing

values.yaml
alerting:
  enabled: true
  email:
    to:         ops@yourdomain.com
    from:       alerts@yourdomain.com
    smarthost:  smtp.resend.com:587
    requireTLS: true
  slack:
    enabled: true
    channel: "#visionpush-alerts"

Add Vault secrets:

Shell
vault kv put secret/visionpush/production \
  ALERTMANAGER_SMTP_PASSWORD="your-smtp-password" \
  ALERTMANAGER_SLACK_WEBHOOK_URL="https://hooks.slack.com/..."

Routing behaviour: warning alerts → email only. Critical alerts → email + Slack. Warning alerts are suppressed when critical fires for the same alertname (InhibitRule).

Traces — OpenTelemetry + Tempo

Instrumentation

Both API and worker emit OTLP spans via tracing-opentelemetry + opentelemetry-otlp. Every HTTP request, database query, NATS publish, and push delivery is a span.

.env
OTLP_ENDPOINT=http://tempo:4317   # Docker Compose
OTLP_ENDPOINT=http://tempo:4317   # Kubernetes (same)

Finding a trace

1. Find a trace ID in API logs: trace_id=abc123def456
2. Open Grafana → Explore → select Tempo datasource
3. Search by trace ID

Logs — Loki + Promtail

Architecture

Flow
Pod stdout/stderr
    → Promtail (DaemonSet)  – scrapes /var/log/containers/
    → Loki (single-binary, filesystem storage)
    → Grafana (log panels, LogQL queries)

Promtail automatically adds labels: namespace, pod, container, app.

LogQL queries

LogQL
# All API logs
{namespace="visionpush", container="api"}

# Error logs only
{namespace="visionpush", container="api"} |= "ERROR"

# Worker throttle events
{namespace="visionpush", container="worker"} |= "throttled"

# Logs for a specific campaign
{namespace="visionpush"} |= "campaign_id=abc-123"

Log format

Both API and worker emit structured JSON via the tracing crate with JSON formatter:

JSON log line
{
  "timestamp":   "2026-05-19T12:00:00Z",
  "level":       "INFO",
  "target":      "visionpush_api::routes::push",
  "span":        { "name": "push_send", "trace_id": "..." },
  "message":     "Campaign dispatched",
  "campaign_id": "...",
  "org_id":      "...",
  "device_count": 12345
}

.env — log levels
RUST_LOG=info                              # default
RUST_LOG=visionpush_api=debug,sqlx=warn   # verbose API, quiet sqlx
RUST_LOG=debug                             # everything (very noisy)

Kubernetes port-forwards

Shell
# Grafana
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring

# Prometheus
kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n monitoring

# Alertmanager
kubectl port-forward svc/kube-prometheus-stack-alertmanager 9093:9093 -n monitoring

# Loki
kubectl port-forward svc/loki 3100:3100 -n monitoring

Docker Compose stack