cluster-narrator

LLM-powered, read-only Kubernetes console. Open the page, see what's happening across your cluster, click a button, and Claude writes a one-paragraph diagnosis in plain English with kubectl commands you can copy-paste.

Built to demonstrate the pattern: production-shaped LLM tooling for SREs. No API key required to run the demo — the backend serves a high-quality canned response when ANTHROPIC_API_KEY is unset.

No live cluster required either — the backend serves deterministic mock data when no kubeconfig is available.

What it does

Five cards on a single page:

1. Cluster overview — node count, namespace count, pod count, pods-not-running, warning-events-in-the-last-hour. The dashboard you'd glance at every morning.
2. Diagnose a namespace — pick any namespace, Claude reads pod state + recent events and produces a one-paragraph health report with kubectl commands to investigate.
3. What changed recently — themes from the last 15min / 1h / 6h of events, grouped by what's happening (scheduling pressure, image pull failures, crash loops) instead of by namespace.
4. Explain why a pod is failing — drill-down for any pod. Claude reads its status, recent events, and the last 100 log lines, then proposes a root cause with concrete remediation steps.
5. Ask anything (chat) — plain-English questions about cluster state. Claude has read tools (list_pods, list_events, get_pod_logs, get_pod_details, list_namespaces) it can call autonomously to answer. It also has one write tool, propose_restart_pod, which never executes directly — it surfaces a confirm modal in the UI; the pod is only restarted after you click Confirm.

The safety story

This is the most important paragraph in the README. Reads are direct, writes always require human confirmation. That's enforced at three layers:

1. The HTTP API never executes a write inside a single request. Read endpoints (GET /api/cluster/overview, GET /api/pods, etc.) hit the cluster directly. The chat endpoint can propose a write — but the proposal travels back to the UI and a separate POST /api/chat/confirm call is required to actually run it. The model can ask for a write a million times in a tool-use loop; nothing happens until a human clicks the button.
2. The ServiceAccount RBAC only allows: get/list/watch on pods/services/events/namespaces/nodes/deployments/etc., plus get on pods/log, plus exactly one write verb — delete on pods. That's the underlying mechanism for "restart": deleting a pod causes its Deployment/StatefulSet to recreate it. There is intentionally no create, update, patch, deletecollection, *, or any verb on secrets/roles/exec in the ClusterRole. Even if the Python code were exploited, the K8s API server would reject anything else.
3. The pod runs as a non-root user with all capabilities dropped, no privilege escalation, no host mounts.

Result: the model can read everything and propose a single specific write (restart). The human is always the one who actually executes the change. This is the right boundary for an LLM-driven ops console.

The chat box's tool design makes this concrete: read tools (list_pods, get_pod_logs, etc.) execute immediately, but propose_restart_pod is the only "write" tool — and even it is structurally a proposal generator, not a writer.

Stack

Layer	Choice	Why
Backend	FastAPI · Python 3.12	Same stack as compliance-analyzer, fast iteration, typed responses
K8s client	kubernetes (official Python lib)	First-party, supports in-cluster + kubeconfig auto-detection
LLM	Anthropic Claude (Sonnet 4.5)	Three tightly-scoped system prompts, one per endpoint type
Frontend	React 19 · Vite · Tailwind 4 · Framer Motion	Same palette as the rest of the portfolio
Local dev	Docker Compose	Single command to run the whole app
Production	EKS · Nginx · shared ALB	Same cluster as the rest of the portfolio, ~$0 added

Quick start (local)

# Optional — without it the backend uses canned LLM responses
cp .env.example .env
# then edit .env and set ANTHROPIC_API_KEY=sk-ant-xxx

# Bring up both services. Defaults to mock cluster data — no kubeconfig needed.
docker compose up --build

Open http://localhost:8080. You'll see:

• mode: mock in the top-right (because no kubeconfig was mounted)
• claude: on if you set the API key, claude: canned if you didn't

Either way, every button works. Try Diagnose a namespace on trading — the mock dataset has two planted anomalies (a CrashLoopBackOff and an Unschedulable pod) that Claude will pick up on.

Pointing at your real cluster

Uncomment the volume mount in docker-compose.yaml:

volumes:
  - ${HOME}/.kube/config:/home/app/.kube/config:ro

Then docker compose up --build again. The backend will auto-detect the kubeconfig and switch from mode: mock to mode: kubeconfig. The frontend doesn't change — the same four cards now operate on your real cluster.

Quick start (frontend dev with hot reload)

# Terminal 1 — backend in docker
docker compose up backend

# Terminal 2 — frontend in dev mode (proxies /api to backend on :8000)
cd frontend
npm install
npm run dev

Then open http://localhost:5173.

Architecture

┌──────────────────────┐                ┌─────────────────────────┐
│ React frontend       │                │ FastAPI backend         │
│   4 cards, fetch /api │ ─── nginx ──> │   /api/cluster/overview │
│                       │                │   /api/diagnose/...     │
│                       │                │   /api/events/...       │
└──────────────────────┘                └────────┬────────┬───────┘
                                                 │         │
                                                 │         │
                                          ┌──────▼──┐  ┌───▼─────────┐
                                          │ K8s API │  │ Anthropic   │
                                          │  (RO SA)│  │ Claude API  │
                                          └─────────┘  └─────────────┘

The backend is the only component talking to either the Kubernetes API or Anthropic. The frontend is a static SPA that hits /api/*.

Endpoints

Method	Path	Body	Returns
GET	/api/health	–	{status, kube_mode, claude_enabled}
GET	/api/cluster/overview	–	Cluster-wide stats + per-namespace summary
GET	/api/pods?namespace=X	–	Pods in namespace X with phase/restarts
GET	/api/events/recent?since_minutes=60&namespace=X	–	Recent events
POST	/api/diagnose/namespace	{namespace}	Claude markdown diagnosis
POST	/api/events/summarize	{since_minutes, namespace?}	Claude themed summary
POST	/api/diagnose/pod	{namespace, pod}	Claude root-cause analysis
POST	/api/chat/message	{messages}	Chat turn — runs Claude with cluster tools, returns text + optional proposal
POST	/api/chat/confirm	{action, args}	Execute a proposal previously surfaced by /api/chat/message

Deploy to EKS

Prereqs

• ECR repos: cluster-narrator-backend, cluster-narrator-frontend
• Route 53 ALIAS record for cluster.fabiorollin.com → existing shared ALB
• Wildcard ACM cert (*.fabiorollin.com) — already covers it

One-time

aws ecr create-repository --repository-name cluster-narrator-backend  --region us-east-1
aws ecr create-repository --repository-name cluster-narrator-frontend --region us-east-1

# Optional: create the Anthropic API key secret
kubectl create namespace cluster-narrator
kubectl create secret generic anthropic-api -n cluster-narrator \
  --from-literal=api-key=sk-ant-xxxxx

Per-deploy

ECR=807291695385.dkr.ecr.us-east-1.amazonaws.com
TAG=$(date +%Y%m%d-%H%M)

aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS --password-stdin $ECR

docker build -t $ECR/cluster-narrator-backend:$TAG  ./backend
docker push  $ECR/cluster-narrator-backend:$TAG
docker build -t $ECR/cluster-narrator-frontend:$TAG ./frontend
docker push  $ECR/cluster-narrator-frontend:$TAG

# First deploy only
kubectl apply -f k8s/cluster-narrator.yaml

# Roll the new images
kubectl set image deploy/backend  "backend=$ECR/cluster-narrator-backend:$TAG"   -n cluster-narrator
kubectl set image deploy/frontend "frontend=$ECR/cluster-narrator-frontend:$TAG" -n cluster-narrator
kubectl rollout status deploy/backend  -n cluster-narrator
kubectl rollout status deploy/frontend -n cluster-narrator

Cost

• 2 small pods (backend ~50m/128Mi, frontend ~25m/32Mi) — fits inside existing EKS Auto capacity
• Shared ALB via alb.ingress.kubernetes.io/group.name: fabiorollin-shared — no new ELB charge
• Anthropic API: ~$0.005 per diagnosis call with Claude Sonnet — even ~50 calls/day = ~$8/month
• ECR storage: <$1/month for both images

Net: ~$1–10/month depending on how much it gets used.

File layout

cluster-narrator/
├── backend/
│   ├── Dockerfile
│   ├── main.py              # FastAPI app
│   ├── kube_client.py       # Read-only K8s wrapper, 3 connection modes
│   ├── claude_client.py     # Anthropic SDK wrapper, canned fallback
│   ├── mock_data.py         # Deterministic fake cluster for demo mode
│   └── requirements.txt
├── frontend/
│   ├── Dockerfile
│   ├── nginx.conf           # SPA fallback + /api proxy
│   ├── index.html
│   ├── package.json
│   ├── vite.config.js
│   └── src/
│       ├── App.jsx          # Single-component app, 4 cards
│       ├── main.jsx
│       └── index.css
├── k8s/
│   └── cluster-narrator.yaml  # Namespace + SA + ClusterRole (RO) + Deployments + Services + Ingress
├── .env.example
├── .gitignore
├── docker-compose.yaml
├── LICENSE
└── README.md

What this project demonstrates

For a Solutions Engineer / platform engineer / AI-curious portfolio:

• LLM tooling on real infrastructure — not a chat-with-your-docs demo, this reads from the actual Kubernetes API
• Tight system-prompt discipline — three different prompts, each scoped to one task with explicit output structure
• The right LLM/human boundary for ops tooling — model writes English, human runs the command. No agentic write operations.
• Defense in depth on RBAC — read-only enforced at the API layer, the ServiceAccount layer, and the container security context
• Mock-friendly architecture — the same code path runs against a real cluster, a kubeconfig, or planted demo data. Demos always work.
• End-to-end app: backend + frontend + container + K8s — ships, doesn't just demo

Roadmap

• More safe writes: scale deployment, drain node, restart deployment — each behind the same propose-then-confirm modal that restart pod uses today.
• Streaming responses: Server-Sent Events for the diagnose endpoints + chat so longer responses appear word-by-word.
• Multi-cluster: drop-down to switch between kubeconfig contexts.
• Audit log: every confirmed write recorded with timestamp + user + the chat history that produced it. Useful for incident retros.
• Slack integration: post a diagnosis to a channel with one click; receive @-mentions from Slack and answer there.