Skip to content

Commit efcb9ca

Browse files
chipiclaude
andcommitted
mini-metrics+homepage: per-vertical OpenRouter spend — collector, cards, ritual
openrouter-spend.sh (launchd, 10m): polls /auth/key per vertical key (pi/opencode/gateway/podcast; keys in gitignored openrouter-verticals.env, chmod 600) → openrouter_vertical_usd{vertical,window}. Homepage row 'LLM spend by vertical' (month + lifetime per card). Weekly ritual gains the three-ledger reconciliation step. First reading already flags a gateway discrepancy to chase: OR bills $10.16 MTD vs LiteLLM metered ~$0.45. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WThYGP2nTpVk4c44jcbBoZ
1 parent 41b7537 commit efcb9ca

5 files changed

Lines changed: 81 additions & 0 deletions

File tree

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,4 @@ workstation/config/opencode/opencode.json
6161
*.bak*
6262
*-bak
6363
bugfix-fleet/bakeoff/advisor-eval/archive/
64+
infra/mini-metrics/openrouter-verticals.env

docs/wip/operator-playbook-fleets.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,13 @@ The one thing no dashboard can verify: were the dismissals RIGHT?
115115
the fleet's cited evidence. A wrong dismissal = say it in session — it
116116
becomes a frozen fixture and a prompt/gate fix.
117117
4. Glance "fleetd day spend vs $2 cap" on the dashboard while you're there.
118+
5. **Spend reconciliation (added 2026-08-05, keys per vertical now live):**
119+
compare three ledgers that must roughly agree — homepage "LLM spend by
120+
vertical" cards (OpenRouter billing truth, per key), Grafana "LLM
121+
Gateway" per-key table (LiteLLM metering), and the fleet dashboards'
122+
self-reported spend. Drift >10% between any pair = an accounting bug
123+
(this check caught a 4x self-report undercount and is currently chasing
124+
a gateway-key discrepancy). One glance, three numbers.
118125

119126
## 6. Kill switches & rollback (when something feels wrong)
120127

infra/homelab-home/gen.sh

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,14 @@ table.ctbl td.u{color:#8888aa}table.ctbl code{font-size:12px}
9797
<a class=card href="$DASH_DELIVERY"><h3>Pending</h3><div id=dl_pending>&hellip;</div></a>
9898
<a class=card href="$DASH_DELIVERY"><h3>Cursor age</h3><div id=dl_cursor>&hellip;</div></a>
9999
</div>
100+
101+
<h2><a href="$G/d/llm-gateway">LLM spend by vertical &middot; OpenRouter billing &rarr;</a></h2>
102+
<div class=charts style="max-width:1140px">
103+
<a class=card href="$G/d/llm-gateway"><h3>Fleet gateway</h3><div id=v_gateway>&hellip;</div></a>
104+
<a class=card href="$G/d/llm-gateway"><h3>Podcast</h3><div id=v_podcast>&hellip;</div></a>
105+
<a class=card href="$G/d/llm-gateway"><h3>Lab (pi)</h3><div id=v_pi>&hellip;</div></a>
106+
<a class=card href="$G/d/llm-gateway"><h3>opencode</h3><div id=v_opencode>&hellip;</div></a>
107+
</div>
100108
</div>
101109
<div class=cols>
102110
<div class=col>
@@ -305,6 +313,13 @@ async function fleet(){
305313
try{const r=await(await fetch('https://api.github.com/search/issues?q=owner%3Achipi+is%3Aissue+is%3Aopen+label%3A%22triage-fleet%2Factionable%22')).json();
306314
set('b_route',cv(typeof r.total_count==='number'?String(r.total_count):'&rarr;'));}
307315
catch(e){set('b_route',cv('&rarr;'));}
316+
// per-vertical OpenRouter billing (openrouter-spend.sh collector, 10m cadence):
317+
// big number = month-to-date as OpenRouter bills it; small = lifetime total
318+
for (const vt of ['gateway','podcast','pi','opencode']) {
319+
const vm=await g1('last_over_time(openrouter_vertical_usd{vertical="'+vt+'",window="month"}[2h])');
320+
const vtot=await g1('last_over_time(openrouter_vertical_usd{vertical="'+vt+'",window="total"}[2h])');
321+
set('v_'+vt, cv($(vm))+'<div class=muted style="font-size:12px">'+$(vtot)+' total</div>');
322+
}
308323
}
309324
async function delivery(){
310325
const set=(id,html)=>{const e=document.getElementById(id);if(e)e.innerHTML=html;};
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3+
<plist version="1.0"><dict>
4+
<key>Label</key><string>com.homelab.openrouter-spend</string>
5+
<key>ProgramArguments</key><array><string>/bin/bash</string><string>/Users/markodragoljevic/agentic-ai-homelab/infra/mini-metrics/openrouter-spend.sh</string></array>
6+
<key>RunAtLoad</key><true/><key>KeepAlive</key><true/>
7+
<key>StandardOutPath</key><string>/tmp/openrouter-spend.log</string>
8+
<key>StandardErrorPath</key><string>/tmp/openrouter-spend.log</string>
9+
</dict></plist>
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
# Per-vertical OpenRouter spend -> VictoriaMetrics (homepage cards + weekly
3+
# reconciliation ritual). One OpenRouter key per vertical (verified separate
4+
# 2026-08-05): pi lab / opencode / fleet gateway / podcast workspace.
5+
#
6+
# Keys live in openrouter-verticals.env next to this script (gitignored; the
7+
# operator home dir is chmod 700 so the claude workbench user can't read it):
8+
# OR_KEY_PI=sk-or-...
9+
# OR_KEY_OPENCODE=sk-or-...
10+
# OR_KEY_GATEWAY=sk-or-... (same value as infra/litellm OPENROUTER_API_KEY)
11+
# OR_KEY_PODCAST=sk-or-... (same value as OPENROUTER_API_KEY_PODCAST)
12+
#
13+
# /auth/key returns usage for THE KEY MAKING THE CALL — that's why every key
14+
# must be present here; there is no list-all endpoint on a normal key.
15+
# Metrics (gauges, absolute USD as OpenRouter reports them):
16+
# openrouter_vertical_usd{vertical, window="day|week|month|total"}
17+
VM=http://localhost:8428/api/v1/import/prometheus
18+
HERE="$(cd "$(dirname "$0")" && pwd)"
19+
ENVF="$HERE/openrouter-verticals.env"
20+
21+
poll() {
22+
[ -f "$ENVF" ] || { echo "no $ENVF — nothing to do"; return; }
23+
# shellcheck disable=SC1090
24+
. "$ENVF"
25+
local LINES=""
26+
for V in pi:$OR_KEY_PI opencode:$OR_KEY_OPENCODE gateway:$OR_KEY_GATEWAY podcast:$OR_KEY_PODCAST; do
27+
local NAME="${V%%:*}" KEY="${V#*:}"
28+
[ -n "$KEY" ] || continue
29+
local J
30+
J=$(curl -s -m 15 https://openrouter.ai/api/v1/auth/key -H "Authorization: Bearer $KEY")
31+
local T D W M
32+
T=$(echo "$J" | /usr/bin/python3 -c "import json,sys; d=json.load(sys.stdin).get('data',{}); print(d.get('usage') or 0)" 2>/dev/null)
33+
D=$(echo "$J" | /usr/bin/python3 -c "import json,sys; d=json.load(sys.stdin).get('data',{}); print(d.get('usage_daily') or 0)" 2>/dev/null)
34+
W=$(echo "$J" | /usr/bin/python3 -c "import json,sys; d=json.load(sys.stdin).get('data',{}); print(d.get('usage_weekly') or 0)" 2>/dev/null)
35+
M=$(echo "$J" | /usr/bin/python3 -c "import json,sys; d=json.load(sys.stdin).get('data',{}); print(d.get('usage_monthly') or 0)" 2>/dev/null)
36+
[ -n "$T" ] || continue # auth/network failure: skip silently, keep last sample
37+
LINES+="openrouter_vertical_usd{vertical=\"$NAME\",window=\"total\",service=\"openrouter-spend\",environment=\"operations\"} $T
38+
openrouter_vertical_usd{vertical=\"$NAME\",window=\"day\",service=\"openrouter-spend\",environment=\"operations\"} $D
39+
openrouter_vertical_usd{vertical=\"$NAME\",window=\"week\",service=\"openrouter-spend\",environment=\"operations\"} $W
40+
openrouter_vertical_usd{vertical=\"$NAME\",window=\"month\",service=\"openrouter-spend\",environment=\"operations\"} $M
41+
"
42+
done
43+
[ -n "$LINES" ] && printf '%s' "$LINES" | curl -s -o /dev/null --data-binary @- "$VM"
44+
}
45+
46+
while true; do
47+
poll
48+
sleep 600
49+
done

0 commit comments

Comments
 (0)