Ai

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Namespace: ai                          │
│                                                             │
│  ┌─────────────────┐    ┌──────────────────────────────┐   │
│  │  Hermes Agent   │───▶│  LiteLLM Proxy               │   │
│  │  model=free     │    │  port 4000                   │   │
│  │  OPENAI_API_BASE│    │  fallback: free→free2→cheap  │   │
│  └─────────────────┘    └─────────────┬────────────────┘   │
│                                       │ HTTPS:443           │
│                                       ▼                     │
│                               OpenRouter API                │
│                               (external)                    │
│  PVC: hermes-data (/opt/data)                               │
│  Secret: litellm-secrets (OPENROUTER_API_KEY)               │
│  Secret: hermes-secrets (OPENROUTER_API_KEY + bot tokens)   │
└─────────────────────────────────────────────────────────────┘

Namespace: registry
  registry:2 pod ← kaniko pushes here ← Kaniko job (namespace: kaniko)
  registries.yaml on K3s nodes → mirror registry.registry:5000 → ClusterIP

Why Custom Build?

Hermes Agent official Docker image (nousresearch/hermes-agent) is amd64-only. For ARM64 clusters (Raspberry Pi CM4), we build in-cluster using kaniko.

Build process:

1. Kaniko job clones https://github.com/NousResearch/hermes-agent
2. Builds ARM64 image using custom Dockerfile (--snapshot-mode=redo for low memory)
3. Pushes to local registry: registry.registry:5000/ai/hermes-agent:0.7.0
4. Hermes deployment uses the local image

Build time: ~60 min on Raspberry Pi CM4 (heavy: debian + nodejs + pip deps + ffmpeg)

Kaniko gotchas:

• --snapshot-mode=redo — uses mtime for change detection (much less memory than default)
• Node affinity → control-plane node (more disk space than agent)
• backoffLimit: 3 — OOM on agent node caused earlier failures
• Wait timeout: 3600s

LiteLLM Proxy — Model Routing

Hermes does NOT call OpenRouter directly. It calls the in-cluster LiteLLM proxy:

OPENAI_API_BASE=http://litellm-proxy.ai.svc.cluster.local:4000
OPENAI_API_KEY=sk-hermes-internal  (LiteLLM master key)
HERMES_MODEL=free

LiteLLM config (roles/install-litellm-proxy/tasks/main.yml):

Hermes MCP lessons learned

• Kubernetes MCP works reliably when exposed as HTTP and configured with url: http://127.0.0.1:8080/mcp.
• type: sse was not enough in practice; the HTTP endpoint had to be explicit.
• Removing the terminal block did not force MCP usage by itself.
• Adding a strong system prompt helped, but Hermes can still prefer shell fallbacks unless the MCP transport is correct.
• kubectl/oc inside the Hermes container were not required for MCP to work once the client connected correctly.
• The working static manifest uses a sidecar pattern: Hermes agent + kubernetes-mcp-server sidecar + /opt/data + serviceAccountName + mcp_servers.kubernetes.url=/mcp.
• For cluster metrics, Hermes still needs either metrics-server or a custom bridge; Prometheus alone is not enough for pods_top / nodes_top.
• Telegram privacy is enforced with TELEGRAM_ALLOWED_USERS and the gateway platform allowed_users list. Keep those aligned to a single user ID when you want a private bot.

Fallback chain: free → free2 → cheap (automatic, transparent to Hermes). Use cheap or strong directly when you want to skip free tiers.

LiteLLM Metrics & Grafana Dashboard

LiteLLM exposes AI traffic and billing metrics to Prometheus when success_callback: ["prometheus"] is set.

The install-litellm-proxy role automates observability:

1. ServiceMonitor: Deploys a ServiceMonitor looking for labels app: litellm-proxy (the LiteLLM service must have these labels in its metadata or Prometheus will silently ignore it).
2. Grafana Dashboard: Injects a custom JSON dashboard via ConfigMap labeled grafana_dashboard: "1" into the monitoring namespace. The Grafana sidecar automatically provisions it.

Key Metrics Tracked:

• litellm_requests_metric_total: Total successful and failed API requests.
• litellm_request_total_latency_seconds_bucket: Request latency histograms.
• litellm_tokens_metric_total: Prompt and completion tokens processed.
• litellm_spend_metric_total: Estimated USD cost of the inference based on model pricing.
• litellm_deployment_successful_fallbacks_total: Counter for when fallback chains trigger successfully.

Installation

Step 1: Install registry (fast)

make ai-registry

Step 2: Build ARM64 images (60 min + 1 min)

make ai-hermes-build        # hermes-agent (~60 min on CM4)
make ai-kubernetes-mcp-build  # kubernetes-mcp-server sidecar (~1 min)
# Monitor with:
kubectl get jobs -n kaniko
kubectl logs -n kaniko job/build-hermes-arm64 -f | grep -v "npm WARN"
kubectl logs -n kaniko job/build-kubernetes-mcp-server-arm64 -f

Both images are required before Hermes deploy — hermes-agent-mcp pod has 2 containers.

Step 3: Deploy LiteLLM proxy + Hermes (2 min)

make ai-hermes-deploy

All at once

make ai  # registry + hermes-build + kubernetes-mcp-build + hermes-deploy (~70 min total)

Configuration

API key (required)

Create roles/install-litellm-proxy/defaults/secrets.yml (gitignored):

hermes_openrouter_api_key: "sk-or-v1-..."
hermes_telegram_token: ""   # optional
hermes_discord_token: ""    # optional

LiteLLM proxy loads this same file automatically.

Change default model tier

Edit roles/install-hermes-agent/defaults/main.yml:

hermes_model: "free"    # default — uses LiteLLM fallback chain
hermes_model: "cheap"   # skip free tiers entirely

Resources (CM4-friendly defaults)

Access

# Hermes web UI (requires ingress stack)