Operations Runbook

Day-to-day operations, scaling, backup/restore, rollbacks, and incident response for VisionPush in production.

Health checks

Shell — quick status
# All VisionPush pods
kubectl get pods -n visionpush

# Recent pod events
kubectl get events -n visionpush --sort-by='.lastTimestamp' | tail -20

# API health
curl https://api.yourdomain.com/healthz

# Worker health (in-cluster)
kubectl exec -n visionpush deploy/visionpush-worker -- \
  wget -qO- http://localhost:9090/healthz

# NATS JetStream lag
kubectl exec -n visionpush deploy/visionpush-worker -- \
  wget -qO- http://visionpush-nats:8222/jsz | jq '.streams[] | {name, messages}'

Open the VisionPush Grafana dashboard and check: Worker pods available (green ≥ 2), NATS consumer lag (near zero outside campaigns), 5xx error rate (green < 1%), Postgres up (UP).

Scaling

API — HPA

Shell
helm upgrade visionpush ./infra/k8s/charts/visionpush -n visionpush \
  --set api.hpa.maxReplicas=20 \
  --set api.hpa.targetCPUUtilizationPercentage=60

Worker — KEDA

For large campaigns (e.g. 10 million devices), temporarily raise the max replicas:

Shell
helm upgrade visionpush ./infra/k8s/charts/visionpush -n visionpush \
  --set keda.worker.maxReplicas=50 \
  --set keda.worker.lagThreshold="200"

PostgreSQL — add read replica

Shell
helm upgrade visionpush ./infra/k8s/charts/visionpush -n visionpush \
  --set postgres.instances=2

CloudNativePG handles failover automatically. Primary: visionpush-postgres-rw, replicas: visionpush-postgres-r.

Rollout and rollback

Shell — deploy via git tag (recommended)
git tag v1.2.3 && git push origin v1.2.3
# ArgoCD Image Updater commits the tag and ArgoCD syncs automatically

Shell — rollback
helm history visionpush -n visionpush
helm rollback visionpush -n visionpush       # previous release
helm rollback visionpush 3 -n visionpush     # specific revision

Shell — force pod restart
kubectl rollout restart \
  deployment/visionpush-api \
  deployment/visionpush-worker \
  -n visionpush

Database operations

Manual migration

Shell
kubectl run -it --rm migrate \
  --image=ghcr.io/visionlab-de/visionpush:latest \
  --env="DATABASE_URL=$(kubectl get secret visionpush-secrets -n visionpush \
    -o jsonpath='{.data.DATABASE_URL}' | base64 -d)" \
  --env="MIGRATE_ONLY=true" \
  --restart=Never \
  -n visionpush

Connect to the database

Shell
# Direct psql inside the pod
kubectl exec -it visionpush-postgres-1 -n visionpush \
  -- psql -U visionpush -d visionpush

# Via port-forward from local machine
kubectl port-forward svc/visionpush-postgres-rw 5432:5432 -n visionpush
psql "postgres://visionpush:PASSWORD@localhost:5432/visionpush"

Manual backup

Shell
kubectl annotate cluster visionpush-postgres \
  cnpg.io/immediateBackup=true -n visionpush

# Check status
kubectl get backup -n visionpush

Point-in-time restore

restore-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: visionpush-postgres-restored
  namespace: visionpush
spec:
  instances: 1
  bootstrap:
    recovery:
      source: visionpush-postgres
      recoveryTarget:
        targetTime: "2026-05-19T03:00:00Z"
  externalClusters:
    - name: visionpush-postgres
      barmanObjectStore:
        destinationPath: "s3://your-bucket/visionpush/"
        s3Credentials:
          accessKeyId:
            name: visionpush-secrets
            key: AWS_ACCESS_KEY_ID
          secretAccessKey:
            name: visionpush-secrets
            key: AWS_SECRET_ACCESS_KEY

Shell
kubectl apply -f restore-cluster.yaml
kubectl get cluster visionpush-postgres-restored -n visionpush -w

Analytics maintenance

Parquet compaction

Nightly compaction runs at 03:00 UTC automatically. To expand the analytics PVC when usage exceeds 80%:

Shell
# Check current usage
kubectl exec -n visionpush deploy/visionpush-worker \
  -- df -h /data/analytics

# Expand PVC
kubectl patch pvc visionpush-analytics -n visionpush \
  -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

Incident response

API returning 5xx

1. Check pod status: kubectl get pods -n visionpush
2. Check logs: kubectl logs deploy/visionpush-api -n visionpush --tail=100
3. Check DB connectivity — API logs sqlx errors if DB is unreachable
4. Check NATS: kubectl exec -n visionpush deploy/visionpush-api -- wget -qO- http://visionpush-nats:8222/healthz
5. If crash-looping: kubectl describe pod -n visionpush <pod-name> to see OOM or probe failures
6. Force restart: kubectl rollout restart deployment/visionpush-api -n visionpush

Push notifications not delivering

1. Check NATS consumer lag in Grafana → Push Delivery row
2. Check worker logs: kubectl logs deploy/visionpush-worker -n visionpush --tail=200
3. Look for throttled or auth error messages
4. If credentials are invalid, the campaign shows failed in stats — check platform config in admin panel
5. Emergency scale: kubectl scale deployment visionpush-worker --replicas=5 -n visionpush

Worker pods OOM

Shell
helm upgrade visionpush ./infra/k8s/charts/visionpush -n visionpush \
  --set worker.resources.limits.memory=2Gi \
  --set worker.resources.requests.memory=512Mi

Certificate expired

Shell
kubectl get certificate -n visionpush
kubectl describe certificate visionpush-tls -n visionpush

# Force renewal
kubectl delete certificate visionpush-tls -n visionpush
# Helm upgrade recreates it

Secret rotation (JWT / encryption key)

Shell
vault kv put secret/visionpush/production \
  JWT_SECRET="$(openssl rand -base64 32)" \
  CREDENTIAL_ENCRYPTION_KEY="$(openssl rand -base64 32)"

kubectl annotate externalsecret visionpush-secrets \
  force-sync=$(date +%s) -n visionpush

kubectl rollout restart \
  deployment/visionpush-api \
  deployment/visionpush-worker \
  -n visionpush

PostgreSQL failover

CloudNativePG handles failover automatically (~10 seconds). To check the current primary:

Shell
kubectl get cluster visionpush-postgres -n visionpush \
  -o jsonpath='{.status.currentPrimary}'

# Manual switchover (maintenance)
kubectl annotate cluster visionpush-postgres \
  cnpg.io/switchoverCandidate=visionpush-postgres-2 -n visionpush

Maintenance mode

Shell
# Enable (API returns 503)
kubectl scale deployment visionpush-api --replicas=0 -n visionpush

# Restore
kubectl scale deployment visionpush-api --replicas=2 -n visionpush

The worker continues processing the NATS queue during API maintenance.

Monitoring checklist

Daily

Weekly