Autonomous SRE Investigation Agent
Receives alerts → runs a multi-step agentic investigation across your entire observability stack → posts a structured RCA to Slack with a PDF report. No human needed.
The argus topology scales Vishwakarma horizontally across clouds and adds
code-level root-causing:
- Two run modes — all-in-one
vk serve(zero deps, SQLite — start here) or orchestrator + per-cloud executor pools over Redis Streams (vk serve-orchestrator/vk serve-executor --cloud aws|gcp). - Durable investigations — every step checkpointed; pod dies → another executor resumes from the last step.
- Code intelligence —
code_analyst(blame / deploy diff / stacktrace→ source) + conversational OpenCode sessions for deep tracing and (gated) fix-writing in isolated worktrees. - Two Slack bots — Sage chats; Argus triggers a full RCA whenever
the team tags
@mre(zero workflow change) or@Argus. - DB-backed runbooks with hybrid retrieval (exact-map + keyword + vector → RRF → rerank), self-curating via ✅/❌ feedback.
- Web console at
/console— live investigations (SSE), incident history, runbook studio, fleet view. Admin/reader RBAC.
Deep dive: docs/ARCHITECTURE.md · security model: docs/SECURITY.md · k8s manifests: k8s/argus/ · build status: PLAN.md
graph TB
subgraph IN[Alert Sources]
AM[AlertManager Webhook]
CW[CloudWatch via Amazon Q Slack]
SNS[CloudWatch SNS via Lambda]
SLK[Slack at-sage debug]
end
subgraph SRV[Vishwakarma Server port 5050]
API[POST /api/alertmanager]
DEDUP{Dedup Check}
subgraph ENRICH[Pre-Enrichment - runs in parallel]
E1[kubectl pod status and events]
E2[SQLite prior incidents]
E3[fast_model entity extraction]
E4[Runbook keyword match]
end
subgraph LOOP[Agentic Loop severity-scaled steps]
LLM[LLM open-large]
EXEC[Execute tools in parallel up to 16]
CHK[Checkpoint at step 20]
end
end
subgraph TOOLS[Toolsets]
T1[bash - kubectl aws stern]
T2[prometheus - VictoriaMetrics]
T3[elasticsearch - logs]
T4[http - internet]
end
subgraph OUT[Outputs]
PDF[PDF Report]
POST[Slack RCA post with PDF and feedback buttons]
DB[(SQLite /data/vishwakarma.db)]
FB[Feedback loop to Learnings]
end
AM --> API
SNS --> API
CW -->|parse and forward| API
SLK -->|direct| LOOP
API --> DEDUP
DEDUP -->|duplicate| SKIP[skip]
DEDUP -->|new| ENRICH
ENRICH --> LOOP
LLM -->|tool calls| EXEC
EXEC -->|results| LLM
EXEC <--> TOOLS
LLM -->|step 20| CHK
CHK -->|continue| LLM
LLM -->|done| OUT
┌─────────────────────────────────────────────────────────────────────┐
│ ALERT SOURCES │
│ │
│ 🔔 AlertManager ──────────────────────────────────────────────┐ │
│ ☁️ CloudWatch SNS → Lambda ──────────────────────────────────┐ │ │
│ 📣 Amazon Q Slack msg ──► parse_cloudwatch_slack_message() ──┤ │ │
│ 💬 @sage debug <question> ──────────────────────────────┐ │ │ │
└───────────────────────────────────────────────────────────┼──┼─┼───┘
│ │ │
┌───────────────────────▼──▼─▼───┐
│ POST /api/alertmanager │
│ ┌──────────────────────────┐ │
│ │ Dedup Check │ │
│ │ fingerprint = hash( │ │
│ │ alertname + │ │
│ │ namespace + service) │ │
│ └────────┬────────┬────────┘ │
│ new│ dup──►skip │
└───────────┼────────────────────-┘
│
┌───────────────────────────▼─────────────────────────┐
│ PRE-ENRICHMENT (all 4 run in parallel) │
│ │
│ 🔍 kubectl get pods + events + replicasets │
│ 📚 SQLite: last 3 investigations of this alert │
│ ⚡ fast_model: extract service / namespace / impact │
│ 📖 Runbook match (DB): exact+keyword+vector → rerank │
│ 🧠 Learnings: for_alert() pre-injects relevant facts │
└───────────────────────────┬─────────────────────────┘
│ merged into extra_system_prompt
┌───────────────────────────▼─────────────────────────┐
│ AGENTIC LOOP (severity-scaled: 20–60 steps) │
│ │
│ Step 1: todo_write() plan + first tool calls │
│ fired simultaneously in same response │
│ Step 2+: LLM calls tools in parallel (up to 16) │
│ ├── bash (kubectl, aws, stern, jq) │
│ ├── prometheus_query_range │
│ ├── elasticsearch_search │
│ └── http_get │
│ Step 20: CHECKPOINT — RCA now or state gap │
│ Step N: LLM returns answer (no tool_calls) │
└───────────────────────────┬─────────────────────────┘
│
┌───────────────────────────▼─────────────────────────┐
│ OUTPUTS │
│ │
│ 📄 generate_pdf() ──► /tmp/rca-{uuid}.pdf │
│ 💬 Slack: post RCA summary + upload PDF in thread │
│ └── ✅ Correct / ❌ Wrong feedback buttons │
│ └── LLM distills fact → learnings │
│ 🗄️ SQLite: save_incident() → /data/vishwakarma.db │
└──────────────────────────────────────────────────────┘
Incoming Slack event
│
├─── app_mention / DM ──────────────────────────────────────────────┐
│ │
│ strip @mention → clean_question (first line only) │
│ │ │
│ ├── "help" ──► post help text │
│ ├── "status" ──► query SQLite stats │
│ │ │
│ ├── "oracle [question]" ──────────────────────────┐ │
│ │ │ Multi-turn investigation session │ │
│ │ │ per-thread context, follow-up memory │ │
│ │ └──► engine.stream_investigate() │ │
│ │ live tool call status updates ◄────┘
│ │ │
│ ├── "debug <question>" ─────────────────────────────┐ │
│ │ │ │ │
│ │ ├── in a thread? ──► fetch parent message │ │
│ │ │ extract alarm context │ │
│ │ └──► run_investigation() │ │
│ │ │ │ │
│ │ ├── engine.investigate() │ │
│ │ ├── generate_pdf() │ │
│ │ ├── post to Slack │ │
│ │ └── save_incident() │ │
│ │ ◄──┘
│ └── anything else ──► _simple_chat() (fast_model, no tools)
│ │
│ detect user tone
│ ├── casual → casual reply
│ └── formal → formal reply
│
└─── channel message (bot) ─────────────────────────────────────────┐
│ │
├── contains "CloudWatch Alarm"? │
│ │ │
│ ├── yes ──► parse_cloudwatch_slack_message() │
│ │ │ │
│ │ ├── is_firing=true │
│ │ │ └──► POST /api/alertmanager │
│ │ └── resolved / OK ──► drop │
│ └── no ──► ignore │
└────────────────────────────────────────────────────────────┘
engine.investigate(question, runbooks, extra_system_prompt)
│
▼
┌──────────────────────────────────────────────────────┐
│ Build prompt │
│ ┌─────────────────────────────────────────────┐ │
│ │ SYSTEM │ │
│ │ • Investigation protocol (todo→recon→RCA) │ │
│ │ • Runbook (alert-specific workflow) │ │
│ │ • Site knowledge base (infra mappings) │ │
│ │ • Pre-enrichment context │ │
│ └─────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────┐ │
│ │ USER: the alert question │ │
│ └─────────────────────────────────────────────┘ │
└──────────────────────────┬───────────────────────────┘
│
┌────────────▼────────────┐
│ LOOP step ≤ N │◄──────────────────┐
│ (N: critical=60, │
│ high=50, warning=40, │
│ low=25, info=20) │
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ │
│ Context > 80% window? │ │
│ yes → compact with │ │
│ fast_model │ │
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ │
│ Call LLM │ │
└────────────┬────────────┘ │
│ │
┌────────────▼────────────┐ │
│ tool_calls in response?│ │
└──┬──────────────────┬───┘ │
│ no │ yes │
▼ ▼ │
┌─────────┐ ┌──────────────────────┐ │
│ DONE │ │ Loop guard: same │ │
│ return │ │ tool+params before? │ │
│ RCA │ └──┬───────────────┬────┘ │
└─────────┘ │ blocked │ allowed │
▼ ▼ │
skip Execute in parallel │
up to 16 tools │
│ │
output > 8KB? │
yes → summarise │
with fast_model │
│ │
step == 20? │
yes → inject │
checkpoint msg │
│ │
└───────────────┘
Runbooks live in the runbooks table (DB). Recall runs three mechanisms in
parallel, merges them, and reranks once — never a short-circuit cascade.
Alert name: "RDS_HighCPU_app-db-r1" (cloud=aws)
│
▼ facet pre-filter: cloud_type ∈ {aws, both, any}
┌─────────────────────────────────────────────────────┐
│ Parallel recall over the runbooks table │
│ ┌──────────────┬───────────────┬─────────────────┐ │
│ │ exact-map │ keyword/FTS │ vector (pgvector)│ │
│ │ alert_runbook│ keywords[] ∩ │ embed(alert) │ │
│ │ _map lookup │ "rds,cpu" ✓ │ <=> embedding │ │
│ └──────────────┴───────────────┴─────────────────┘ │
│ → RRF merge (~10–15 candidates) │
└───────────────────────┬──────────────────────────────┘
│ one fast_model rerank
▼
top 1–3 runbooks (e.g. rds-investigation)
│
▼ inject into system prompt
LLM follows step-by-step; can also call
runbook_search(query) mid-investigation
Self-improving: a ✅-confirmed RCA bumps hit_count and inserts the exact-map
row, so repeat alerts hit the fast path; repeated ❌ demotes a runbook.
Configure which toolsets are enabled in
config.yaml. The agent uses only what's enabled.
| Toolset | Description |
|---|---|
🐚 bash |
Shell commands — kubectl, aws, stern, jq, grep. Allowlist/blocklist per deployment. Primary tool for K8s and AWS investigation. |
☸️ kubernetes |
Native K8s API (pod status, events, logs). Disabled by default — bash + kubectl preferred. |
📋 kubernetes_logs |
Pod logs via K8s API. Disabled by default — use stern via bash. |
🐳 docker |
Inspect containers, images, logs, resource usage. |
⛵ helm |
Helm release history, chart values, diff. |
🔄 argocd |
ArgoCD sync status, health, rollout history. |
🌐 cilium |
Cilium CNI — endpoint health, network policies, Hubble flows. |
☁️ aks |
Azure Kubernetes Service via az CLI. |
| Toolset | Description |
|---|---|
📈 prometheus |
Prometheus/VictoriaMetrics instant + range queries. Always use this — never http_get for metrics. |
📉 grafana |
Grafana dashboards, panels, annotations (also Loki). |
🐕 datadog |
Datadog metrics and monitors. |
🔮 newrelic |
New Relic metrics and alerts. |
🪵 coralogix |
Coralogix logs via DataPrime or Lucene syntax. |
| Toolset | Description |
|---|---|
🔎 elasticsearch |
Elasticsearch/OpenSearch Query DSL. Used for Istio access logs, app error logs, traces. |
🌍 http |
HTTP GET to external URLs. Only for external endpoints. |
📡 internet |
DNS lookup, ping, network diagnostics. |
| Toolset | Description |
|---|---|
🐘 database |
Read-only SQL against PostgreSQL or MySQL. |
🍃 mongodb |
MongoDB collection queries (read-only). |
📨 kafka |
Kafka topics, consumer groups, and lag inspection. |
| Toolset | Description |
|---|---|
🎫 servicenow_tables |
ServiceNow incidents and CMDB records. |
🔗 mcp |
Model Context Protocol — plug in any MCP-compatible tool server. |
✅ todo |
Internal task planner — agent writes a plan before touching any tool. |
📚 learnings |
Read/list accumulated incident knowledge. Agent calls learnings_list then learnings_read(category) at the start of every investigation to load relevant facts. |
📊 Metrics / PromQL ──► prometheus_query_range (NEVER http_get)
📜 Log search ──► elasticsearch_search (NEVER http_get)
☸️ K8s / AWS / CLI ──► bash tool
🌐 External URLs ──► http_get
| Feature | Description |
|---|---|
| 🗺️ Runbook routing | DB-only runbooks; hybrid recall (exact-map + keyword + pgvector) → RRF → one rerank |
| ⚡ Pre-enrichment | K8s pod status, warning events, prior incidents, and entity extraction — all in parallel before the loop starts |
| 📚 Site knowledge base | /data/knowledge.md on PVC, injected into every investigation — edit without rebuilding the image |
| 🧠 Learnings system | Accumulated incident knowledge organized by category (rds, redis, drainer…). Relevant facts are pre-injected before every investigation via for_alert() — no wasted tool steps. Auto-compacts when a category exceeds 50 facts or 5KB. |
| 🔁 Feedback loop | After every RCA, Slack posts ✅ Correct / ❌ Wrong buttons. On feedback, LLM distills the root cause into a clean one-sentence fact, deduplicates against existing learnings, and appends to the right category automatically. |
| ⚖️ Severity-scaled steps | Investigation depth scales with alert severity — critical alerts get up to 60 steps, info alerts get 20. Prevents wasted compute on low-priority alerts. |
| 🖥️ Web UI | Built-in dashboard at /ui — browse incidents, view investigation tool traces, manage learnings per category |
| 🔁 Incident history | SQLite stores every investigation; prior findings for recurring alerts are injected as context |
| 🚀 Parallel tools | Up to 16 tools run simultaneously per step |
| 🛑 Step-20 checkpoint | Forces LLM to evaluate evidence and write RCA or declare what's still missing |
| 🛡️ Safeguards | Identical tool+params blocked from re-running; bash non-zero exits not retried |
| 🗜️ Context compaction | Long investigations auto-compress with fast_model to stay within the context window |
| 📄 PDF reports | Branded RCA with evidence chain, uploaded to Slack thread |
| 🤖 Slack bot | @sage debug for on-demand investigations; casual questions answered with auto tone-matching |
- Kubernetes cluster with
kubectlaccess - LLM API key (OpenAI-compatible — set
api_basefor self-hosted) - Slack app with Bot Token (
xoxb-) and App Token (xapp-) for Socket Mode - AWS IRSA or env var credentials for CloudWatch/RDS queries
cp config.example.yaml config.yamlllm:
model: openai/gpt-4o # any OpenAI-compatible model
api_base: https://... # omit for OpenAI default
api_key: sk-... # or set VK_API_KEY env var
fast_model: openai/gpt-4o-mini # summarisation + extraction + compaction
cluster_name: my-cluster # shown to LLM in every investigation
max_steps: 40 # default max steps (overridden per-alert by severity)
dedup_window: 300 # seconds to suppress duplicate alerts
toolsets:
prometheus:
enabled: true
config:
url: http://prometheus.monitoring.svc.cluster.local:9090
elasticsearch:
enabled: true
config:
url: http://elasticsearch.logging.svc.cluster.local:9200
bash:
enabled: true
config:
allow: [kubectl, aws, stern, jq, grep, awk, head, tail, sort, uniq]
block: [rm, curl, wget, env, python]Runbooks are stored in the database (the runbooks table) — not in the repo. Each one tells the agent exactly how to investigate a specific alert type — which metrics to query, which logs to check, how to pivot between symptoms, and the fix (including a code-fix path). You author and edit them in the console (Runbook studio) or via storage/runbooks.py; no code deploy needed, and they persist across restarts.
Why DB-only: anyone can author/edit a runbook without a deploy; runbooks can be cloud-scoped (aws|gcp|both|any); and matching uses hybrid retrieval (exact-map + keyword + pgvector) that scales to a large corpus. The repo ships with none — you seed your own.
Adding a runbook (console, or programmatically):
from vishwakarma.storage.runbooks import save_runbook, map_alert
save_runbook(
runbook_id="my-alert-investigation",
title="Investigate MyAlert",
content_md="## 0 Match …\n## 1 Metrics …\n## 4 Fix …\n## 5 Code fix …",
cloud_type="any", # aws | gcp | both | any
keywords=["myalert", "keyword2"],
services=["my-svc"],
)
map_alert("MyAlert", "my-alert-investigation") # optional exact-match fast pathThe embedding is computed on upsert (for vector recall). The match self-improves:
a ✅-confirmed RCA bumps the runbook's hit_count and auto-inserts the
(alert, runbook) map row so repeat alerts hit the fast path.
Runbook design tips:
- Use
<placeholder>for anything cluster-specific (namespaces, service names, instance IDs) and tell the agent to "use the value from the Site Knowledge Base" - Put the actual values in
knowledge.md— not in the runbook - Put patterns discovered from real incidents in Learnings — not in the runbook
- Add a Code fix step (locate handler →
code_session→propose_fix→ draft PR) when a code change can resolve it
Example runbook coverage (you author these in the DB — RDS CPU/connections, ElastiCache evictions, ALB/LB 5xx via the istio→request-id→app-log pivot, drainer lag, login-success drops, ride-to-search ratio, config-parse failures, etc.). Each follows: §0 Match → §1 Metrics → §2 Logs → §3 Diagnosis → §4 Fix → §5 Code fix → §6 RCA.
Create /data/knowledge.md on your PVC — cluster-specific context injected into every investigation:
## RDS Instances (region: us-east-1)
my-app-writer (writer), my-app-reader-1 (reader) — aurora-postgresql 14
## Alert → Instance Mapping
"my-app-high-cpu" → check my-app-writer AND my-app-reader-1 simultaneously
## Redis Clusters
main-cache — primary app cache
## Known IAM Gaps (always fail — skip, don't retry)
aws rds describe-db-clusters → no permission
## Proven Commands
for i in my-app-writer my-app-reader-1; do
aws cloudwatch get-metric-statistics --namespace AWS/RDS \
--metric-name CPUUtilization \
--dimensions Name=DBInstanceIdentifier,Value=$i \
--start-time START --end-time END --period 300 \
--statistics Average Maximum --region us-east-1 \
--output json | jq -r '.Datapoints|sort_by(.Timestamp)[]|
"\(.Timestamp): avg=\(.Average|floor)%"'
doneUpdate without rebuilding:
kubectl cp ./knowledge.md <namespace>/<pod>:/data/knowledge.md
kubectl rollout restart deployment/vishwakarma -n <namespace>Learnings are facts your team accumulates from real incidents — patterns, gotchas, and context that aren't obvious from infrastructure topology alone.
Three layers of knowledge:
| Layer | What goes here | Where |
|---|---|---|
| Site Knowledge Base | Static infra facts: instance IDs, namespaces, metric names, proven commands | knowledge.md on PVC |
| Runbooks | How to investigate each alert type: which tools, which order, what to look for | runbooks table (DB) |
| Learnings | What you've learned from incidents: quirks, patterns, past RCAs | /data/learnings/*.md on PVC |
Teaching the bot via Slack:
@sage learn rds Atlas customer RDS CPU spikes always correlate with drainer batch inserts at :00 and :30
@sage learn redis location-redis evictions spike during 6-9 PM IST peak - not an alert worth waking for
@sage forget rds <fact-keyword>
Managing learnings via the UI:
Open http://<vishwakarma>:5050/ui → Learnings tab. Browse categories, edit facts, create new categories.
How the agent uses learnings:
Relevant learnings are pre-injected before every investigation starts — for_alert(alert_name) keyword-matches the alert to categories and injects the facts directly into the system prompt. No tool steps wasted, context arrives ready.
Feedback loop — auto-learning from RCAs:
After every investigation, Slack posts feedback buttons in the thread:
- ✅ Correct — LLM distills the full RCA into one actionable sentence (root cause + confirmation + what to check next time), deduplicates against existing facts, appends to the right category.
- ❌ Wrong — Opens a modal. User types the real cause. LLM cleans typos, distills into one sentence, deduplicates, appends.
- Duplicate detected — Buttons are replaced with "already captured" — no duplicate stored.
Auto-compaction:
When a category grows beyond 50 facts or 5KB, LLM consolidates it — merging duplicates, removing vague entries, keeping the most actionable facts.
Default categories: rds, redis, drainer, kubernetes, networking, general. Add your own via the UI or @sage learn <new-category> <fact>.
# RBAC (ServiceAccount + ClusterRole)
kubectl apply -f k8s/rbac.yaml
# PVC + ConfigMap + Deployment + Service
kubectl apply -f k8s/deployment.yaml
# Verify
kubectl rollout status deployment/vishwakarma -n monitoringOption A — AlertManager / VMAlertManager
receivers:
- name: vishwakarma
webhook_configs:
- url: http://vishwakarma.monitoring.svc.cluster.local:5050/api/alertmanager
send_resolved: falseOption B — CloudWatch → Amazon Q → Slack
Add the bot to the channel where Amazon Q posts CloudWatch alarms. It auto-detects the format and forwards to the agentic loop automatically.
Option C — CloudWatch → SNS → Lambda
Deploy lambda/handler.py — converts SNS CloudWatch events to AlertManager format and POSTs to /api/alertmanager.
vishwakarma/
├── 🧠 core/
│ ├── engine.py Agentic loop — tool calling, parallelism, checkpointing
│ ├── prompt.py System prompt builder (composable sections)
│ ├── tools.py Tool definitions + executor
│ ├── toolset_manager.py Loads and manages toolsets
│ └── models.py Pydantic data models
│
├── 🔌 plugins/
│ ├── toolsets/ bash, prometheus, elasticsearch, grafana, aws ...
│ │ (runbooks live in the DB, not here)
│ ├── channels/
│ │ └── alertmanager/ AlertManager webhook parser
│ └── relays/
│ └── slack/ Slack result poster (RCA + PDF)
│
├── 🤖 bot/
│ ├── slack.py Slack Socket Mode bot + CloudWatch detection
│ ├── cloudwatch.py CloudWatch alarm parser (Amazon Q + SNS formats)
│ └── pdf.py Branded PDF RCA report generation
│
├── 🗄️ storage/
│ └── db.py SQLite incident storage + full-text search
│
├── server.py FastAPI server + pre-enrichment + alert routing
└── config.py Config loader (YAML + env vars)
k8s/
├── deployment.yaml PVC + ConfigMap + Deployment + Service
└── rbac.yaml ServiceAccount + ClusterRole + Binding
lambda/
└── handler.py CloudWatch SNS → AlertManager forwarder
| Method | Endpoint | Description |
|---|---|---|
POST |
/api/alertmanager |
AlertManager / CloudWatch webhook — triggers investigation |
POST |
/api/investigate |
Ad-hoc investigation (synchronous) |
POST |
/api/investigate/stream |
Ad-hoc investigation (SSE streaming) |
GET |
/api/incidents |
List past investigations |
GET |
/api/incidents/{id} |
Full investigation with tool outputs |
GET |
/api/stats |
Investigation statistics |
GET |
/api/toolsets |
Toolset health and enabled status |
GET |
/healthz |
Liveness probe |
GET |
/readyz |
Readiness probe |
The built-in dashboard lets you browse investigations, inspect tool traces, and manage learnings.
Port-forward from your machine:
kubectl port-forward -n monitoring svc/vishwakarma 5050:5050Then open: http://localhost:5050/ui
Pages:
- Investigations — list of all past RCAs, click to see full tool trace and analysis
- Learnings — browse/edit facts by category, create new categories
- Toolsets — health check for each enabled toolset
The bot ships with the Sage persona — an SRE character used at ExampleApp. To use your own bot name and identity, edit bot/slack.py → _simple_chat() system prompt:
"content": (
"You are <YourBotName>, an SRE at <YourCompany>. "
"If anyone asks who you are, say: 'I'm <YourBotName>.' "
"If anyone asks who made you, say: 'I was made by <your-name>.' "
"..."
),Also update the @sage references in the help text (_help_text()) and the DM handler to match your Slack bot's user ID.
Slack commands:
@yourbot <question> — casual chat (no tools, fast reply, tone-matched)
@yourbot debug <question> — full investigation + PDF report in thread
@yourbot oracle <question> — multi-turn investigation session with memory
@yourbot oracle stop — end the oracle session in this thread
@yourbot learn <category> <fact> — teach the bot a new fact
@yourbot forget <category> <keyword> — remove a fact matching keyword
@yourbot status — show incident stats
@yourbot help — show help
debugvsoracle: Usedebugfor one-shot alert investigations — it produces a PDF and saves to incident history. Useoraclefor interactive troubleshooting where you want to ask follow-up questions in the same thread — the bot remembers context across turns.
| What to change | Where |
|---|---|
| LLM provider + model | config.yaml → llm.model, llm.api_base |
| Fast model (summarisation + extraction) | config.yaml → llm.fast_model |
| Cluster name shown to LLM | config.yaml → cluster_name |
| Prometheus / ES / Grafana URLs | config.yaml → toolsets.* |
| Which tools are available | config.yaml → toolsets.*.enabled |
| Bash allowlist | config.yaml → toolsets.bash.config.allow |
| Investigation workflow for your alerts | runbooks table (DB) — console / storage/runbooks.py |
| Alert → runbook routing | alert_runbook_map table (DB), auto-populated on ✅-confirmed RCAs |
| Infra-specific context (instances, namespaces, metrics) | /data/knowledge.md on PVC |
| Incident patterns and gotchas discovered over time | /data/learnings/*.md via @sage learn or the UI |
| Slack bot name + persona | bot/slack.py → _simple_chat system prompt (change "Sage" to your own bot name/persona) |