Skip to content

Commit af7ed8c

Browse files
walm00claude
andauthored
Framework hardening: security, data integrity, schema registry, wiki dead-letter (#47)
Security - auto_save_session.sh: stop interpolating hook stdin / transcript path into a python -c string. Pipe via stdin and pass paths via env var. Closes a shell-injection-class hole on every assistant Stop hook. Publish safety - publish.sh: replace silent --theirs conflict resolution with a prompt that lists conflicting files and an explicit y/N. New --force-overwrite flag preserves headless flows. Single-squash-commit UX preserved. Data integrity (self-learning ladder) - gitignore + git rm --cached the four ecosystem runtime files (resolutions.jsonl, learned-rules.json, learning-blocklist.json, state.json). Append-only event logs / derived state — git tracking caused merge-conflict-driven data loss. - install.sh: new seed_if_missing helper writes empty defaults for the four files on fresh installs. - _jsonl_safe.py: safe_load_jsonl() returns rows AND a CorruptionReport so loaders surface dropped malformed lines as findings instead of silently skipping them. Migrated auto_fix_audit + promote_resolutions. Schema-version registry - _schema_versions.py: single source of truth for derived-artifact schemas. ArtifactSchema rows declare current/min_supported/policy. Replaces five different per-script policies for "what does this artifact's shape mean on upgrade?". - digest_sidecar.py: migrated to use the registry. Wiki failed-ingest dead-letter - wiki_failed_ingest.py: scanner for stuck queue.md URLs (>14d), source-summary files missing provenance.source, and drafts with malformed YAML. Auto-fix intentionally not whitelisted. - references/job-wiki-failed-ingest.md + schedule-config entry. Authority drift - CLAUDE.md: authority hierarchy now leads with _collections/ external-evidence (the artifact IS the truth). - document-standards.md: removed duplicate Mechanical Index Facets section. Skill ecosystem - context-ingest: dispatch front-door table — URL → /wiki run, conversation export → context-mine, loose notes → stay. - bcos-wiki + context-mine: prepend "see also: ingest" notes. - schedule-tune: soften DO-NOT-USE-for-onboarding wording. Onboarding - context-onboarding Step 6: add 6e (verify scheduled task exists) and 6f (offer cadence tuning via schedule-tune). Dispatcher - schedule-config.template.json: every job now defaults to "daily" (over-monitor early, auto-tuner suggests reductions after 5 green runs). Added auto_commit_branches allowlist (main/master/dev/develop). tuning.suggest_reduce_after_green_runs 3 → 5. - schedule-dispatcher SKILL.md: Step 2.5 — regenerate context-index.json once per run; jobs read cached via load_context_index_cached(). Lifecycle routing - lifecycle-routing.yml rule 5 (research-dump): destination bucket changed to "auto" — bcos-wiki picks pages/ vs source-summary/ at promote time. README - Lead the install section with "ask Claude to install from URL". Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 7e1ead6 commit af7ed8c

24 files changed

Lines changed: 832 additions & 148 deletions

File tree

.claude/hooks/auto_save_session.sh

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,14 @@ mkdir -p "$STATE_DIR" "$SESSIONS_DIR"
2727
# ---------------------------------------------------------------------------
2828
INPUT=$(cat)
2929

30-
# Extract fields using python (available everywhere, no jq dependency)
31-
read -r STOP_HOOK_ACTIVE SESSION_ID TRANSCRIPT_PATH <<< "$(python3 -c "
30+
# Extract fields using python (available everywhere, no jq dependency).
31+
# Pipe $INPUT to stdin — never interpolate untrusted hook input into a -c string.
32+
read -r STOP_HOOK_ACTIVE SESSION_ID TRANSCRIPT_PATH <<< "$(printf '%s' "$INPUT" | python3 -c "
3233
import json, sys
33-
data = json.loads('''$INPUT''')
34+
try:
35+
data = json.loads(sys.stdin.read())
36+
except Exception:
37+
data = {}
3438
active = str(data.get('stop_hook_active', False)).lower()
3539
sid = data.get('session_id', 'unknown')
3640
tp = data.get('transcript_path', '')
@@ -53,11 +57,13 @@ if [ -z "$TRANSCRIPT_PATH" ] || [ ! -f "$TRANSCRIPT_PATH" ]; then
5357
exit 0
5458
fi
5559

56-
EXCHANGE_COUNT=$(python3 -c "
57-
import json, sys
60+
# Pass transcript path via env var — never interpolate into source.
61+
EXCHANGE_COUNT=$(BCOS_TRANSCRIPT_PATH="$TRANSCRIPT_PATH" python3 -c "
62+
import json, os
5863
count = 0
64+
path = os.environ.get('BCOS_TRANSCRIPT_PATH', '')
5965
try:
60-
with open('$TRANSCRIPT_PATH', 'r') as f:
66+
with open(path, 'r') as f:
6167
for line in f:
6268
line = line.strip()
6369
if not line:
@@ -66,7 +72,6 @@ try:
6672
entry = json.loads(line)
6773
if entry.get('role') == 'user':
6874
content = str(entry.get('content', ''))
69-
# Skip slash commands and system messages
7075
if '<command-message>' not in content:
7176
count += 1
7277
except json.JSONDecodeError:

.claude/quality/ecosystem/state.json

Lines changed: 0 additions & 65 deletions
This file was deleted.

.claude/quality/lifecycle-routing.yml

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -233,16 +233,19 @@ rules:
233233
description: >
234234
Research documents with external URLs that have aged past the declared
235235
route_to_wiki_after_days threshold. Sweep confirms an external URL exists
236-
in the body (otherwise it's not source-summary material), then calls
237-
/wiki promote to produce a source-summary page.
236+
in the body and then delegates to /wiki promote — the bcos-wiki skill
237+
chooses the destination bucket (pages/ vs source-summary/) based on the
238+
document's shape, so the sweep MUST NOT encode the bucket itself. The
239+
`destination.bucket: auto` value documents this delegation; lifecycle_sweep
240+
surfaces a "delegate to /wiki promote" finding rather than moving files.
238241
match:
239242
zones: [active]
240243
lifecycle-triggers:
241244
- route_to_wiki_after_days: any
242245
cluster-hints: [market-intelligence, research, competitive, reference]
243246
destination:
244247
zone: wiki
245-
bucket: source-summary
248+
bucket: auto # bcos-wiki picks pages/ vs source-summary/ on promote
246249
body-markers:
247250
- pattern: "https?://"
248251
role: require

.claude/quality/schedule-config.template.json

Lines changed: 31 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -9,62 +9,67 @@
99
"index-health": {
1010
"enabled": true,
1111
"schedule": "daily",
12-
"_about": "Rebuild document index + scan for frontmatter, metadata, and cross-reference issues. Cheap — always run daily."
12+
"_about": "Run cadence: starts daily, tune down once stable. Rebuild document index + scan for frontmatter, metadata, and cross-reference issues. Cheap — fine to leave daily long-term."
1313
},
1414
"daydream-lessons": {
1515
"enabled": true,
16-
"schedule": "mon",
17-
"_about": "Strategic reflection + lessons capture. Once a week is usually enough."
16+
"schedule": "daily",
17+
"_about": "Run cadence: starts daily, tune down once stable. Steady state ~weekly (e.g. 'mon') — strategic reflection + lessons capture rarely needs more than that once context settles."
1818
},
1919
"daydream-deep": {
2020
"enabled": true,
21-
"schedule": "wed",
22-
"_about": "Deeper mid-week daydream — gaps, structural change. Skip in new projects where context is still forming."
21+
"schedule": "daily",
22+
"_about": "Run cadence: starts daily, tune down once stable. Steady state ~weekly mid-week (e.g. 'wed') — gaps and structural change. In mature projects, biweekly is also reasonable."
2323
},
2424
"audit-inbox": {
2525
"enabled": true,
26-
"schedule": "fri",
27-
"_about": "Deep CLEAR audit + inbox processing. Weekly by default; bump to twice-weekly if the inbox grows faster than you can triage."
26+
"schedule": "daily",
27+
"_about": "Run cadence: starts daily, tune down once stable. Steady state ~weekly (e.g. 'fri') — deep CLEAR audit + inbox processing. Bump back up to twice-weekly if the inbox grows faster than you can triage."
2828
},
2929
"architecture-review": {
3030
"enabled": true,
31-
"schedule": "1st",
32-
"_about": "Monthly full architecture + ecosystem pass. Quarterly is fine once context is mature — just change 'schedule' to a cron like '0 0 1 */3 *' or ask Claude to do it."
31+
"schedule": "daily",
32+
"_about": "Run cadence: starts daily, tune down once stable. Steady state monthly (e.g. '1st') or quarterly ('0 0 1 */3 *') once context is mature — full architecture + ecosystem pass."
3333
},
3434
"wiki-stale-propagation": {
3535
"enabled": true,
3636
"schedule": "daily",
37-
"_about": "Scan wiki pages whose builds-on sources changed after last-reviewed. Cheap metadata-only check; emits action items for human review."
37+
"_about": "Run cadence: starts daily, tune down once stable. Cheap metadata-only check — fine to leave daily long-term. Scans wiki pages whose builds-on sources changed after last-reviewed and emits action items."
3838
},
3939
"wiki-source-refresh": {
4040
"enabled": true,
41-
"schedule": "mon",
42-
"_about": "Two-tier source-summary refresh: HEAD-only quick checks at stale_threshold_days/4, full refresh-must-rediscover at stale_threshold_days."
41+
"schedule": "daily",
42+
"_about": "Run cadence: starts daily, tune down once stable. Steady state ~weekly (e.g. 'mon') — two-tier source-summary refresh: HEAD-only quick checks at stale_threshold_days/4, full refresh-must-rediscover at stale_threshold_days."
4343
},
4444
"wiki-graveyard": {
4545
"enabled": true,
46-
"schedule": "1st",
47-
"_about": "Monthly archive-candidate scan for stale/orphaned/expired wiki pages. Suggests archive actions; only whitelisted post-mortem expiry is auto-fixable."
46+
"schedule": "daily",
47+
"_about": "Run cadence: starts daily, tune down once stable. Steady state monthly (e.g. '1st') — archive-candidate scan for stale/orphaned/expired wiki pages. Only whitelisted post-mortem expiry is auto-fixable."
4848
},
4949
"wiki-coverage-audit": {
5050
"enabled": true,
51-
"schedule": "0 0 1 */3 *",
52-
"_about": "Quarterly cross-zone coverage scan. Surfaces active data points without wiki explainers and wiki clusters not present in document-index as INFO."
51+
"schedule": "daily",
52+
"_about": "Run cadence: starts daily, tune down once stable. Steady state quarterly ('0 0 1 */3 *') — cross-zone coverage scan. Surfaces active data points without wiki explainers and wiki clusters missing from document-index as INFO."
5353
},
5454
"auto-fix-audit": {
5555
"enabled": true,
56-
"schedule": "fri",
57-
"_about": "Friday weekly safety brake on the self-learning ladder. Surfaces amber cards when a learned rule reversal rate crosses 5% in 7 days. Recommend-only at v0.1."
56+
"schedule": "daily",
57+
"_about": "Run cadence: starts daily, tune down once stable. Steady state ~weekly (e.g. 'fri') — safety brake on the self-learning ladder. Surfaces amber cards when a learned rule reversal rate crosses 5% in 7 days. Recommend-only at v0.1."
5858
},
5959
"lifecycle-sweep": {
6060
"enabled": true,
61-
"schedule": "fri",
62-
"_about": "Friday weekly classifier of active-zone docs against lifecycle-routing.yml. Surface-only for the first 2 weeks (default); flip lifecycle-routing.yml > global.surface_only=false after 0 false-positives confirmed to enable auto-routing."
61+
"schedule": "daily",
62+
"_about": "Run cadence: starts daily, tune down once stable. Steady state ~weekly (e.g. 'fri') — classifier of active-zone docs against lifecycle-routing.yml. Surface-only for the first 2 weeks (default); flip lifecycle-routing.yml > global.surface_only=false after 0 false-positives confirmed to enable auto-routing."
6363
},
6464
"wiki-canonical-drift": {
6565
"enabled": true,
6666
"schedule": "daily",
67-
"_about": "Daily mechanical scan: wiki captures whose numeric facts diverge from canonical docs whose last-updated is > 180 days. Emits wiki-canonical-drift-suggestion findings; never edits canonical docs. Schema 1.2 Class D — see job-wiki-canonical-drift.md."
67+
"_about": "Run cadence: starts daily, tune down once stable. Mechanical scan — fine to leave daily long-term. Wiki captures whose numeric facts diverge from canonical docs whose last-updated is > 180 days. Emits wiki-canonical-drift-suggestion findings; never edits canonical docs. Schema 1.2 Class D — see job-wiki-canonical-drift.md."
68+
},
69+
"wiki-failed-ingest": {
70+
"enabled": true,
71+
"schedule": "daily",
72+
"_about": "Run cadence: starts daily, tune down once stable. Cheap dead-letter scan — fine to leave daily long-term. Surfaces stuck queue.md URLs (default >14 days), source-summary files missing provenance.source, and wiki drafts with malformed/missing YAML frontmatter. Mechanical-only; never auto-fixes. See job-wiki-failed-ingest.md."
6873
}
6974
},
7075
"auto_fix": {
@@ -89,12 +94,14 @@
8994
"write_file": true,
9095
"path": "docs/_inbox/daily-digest.md",
9196
"auto_commit": false,
92-
"_about": "Overwrites each run. Full history lives in .claude/hook_state/schedule-diary.jsonl. Set auto_commit: true to commit generated artifacts (digest, index, diary, wake-up context) at the end of each run — but ONLY if the working tree has no other changes. Never pushes, never branches. Safe default is false; flip to true once you trust a week of clean runs.",
97+
"_about": "Overwrites each run. Full history lives in .claude/hook_state/schedule-diary.jsonl. Set auto_commit: true to commit generated artifacts (digest, index, diary, wake-up context) at the end of each run — but ONLY if the working tree has no other changes AND the current branch is in auto_commit_branches. Never pushes, never branches. Safe default is false; flip to true once you trust a week of clean runs.",
9398
"auto_commit_block_on_red": true,
94-
"_about_block_on_red": "When true (default), auto_commit is skipped on red verdict runs even if the tree is otherwise clean. Surfaces an Auto-commit paused chip in the cockpit. Set to false to opt back into committing on red days."
99+
"_about_block_on_red": "When true (default), auto_commit is skipped on red verdict runs even if the tree is otherwise clean. Surfaces an Auto-commit paused chip in the cockpit. Set to false to opt back into committing on red days.",
100+
"auto_commit_branches": ["main", "master", "dev", "develop"],
101+
"_about_branches": "Allowlist of long-lived branches where auto-commit may land. On any other branch (short-lived feature branches like 'feat/foo'), the dispatcher writes the digest locally but skips the commit so unrelated daily-maintenance commits don't pollute feature PRs. Add 'trunk', 'production', 'staging', or your team's gitflow names if needed."
95102
},
96103
"tuning": {
97-
"suggest_reduce_after_green_runs": 3,
104+
"suggest_reduce_after_green_runs": 5,
98105
"suggest_increase_if_findings_trending_up": true,
99106
"_about": "Adaptive tuning is suggestion-only. The dispatcher never changes this config on its own."
100107
}

.claude/scripts/_jsonl_safe.py

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
"""Safe JSONL loader with corruption tracking.
2+
3+
Most BCOS loaders read JSONL with `try: json.loads(line) except: continue`,
4+
which silently drops malformed lines. On Windows, concurrent appends to a
5+
JSONL file are NOT atomic — a race can produce one mangled line, which then
6+
silently disappears from the auditor's denominator and skews the
7+
self-learning ladder's reversal-rate computation.
8+
9+
This module replaces the pattern with one that counts drops, so the
10+
dispatcher can surface a `data-corruption-detected` finding when a loader
11+
quietly threw away rows. No behaviour change on clean data.
12+
13+
Migration: replace
14+
15+
out = []
16+
with p.open(encoding="utf-8") as fh:
17+
for line in fh:
18+
line = line.strip()
19+
if not line:
20+
continue
21+
try:
22+
out.append(json.loads(line))
23+
except Exception:
24+
continue
25+
return out
26+
27+
with
28+
29+
rows, report = safe_load_jsonl(p)
30+
if report.dropped:
31+
log_corruption(report) # or: include in dispatcher findings
32+
return rows
33+
"""
34+
35+
from __future__ import annotations
36+
37+
import json
38+
import os
39+
from dataclasses import dataclass
40+
from pathlib import Path
41+
42+
43+
@dataclass
44+
class CorruptionReport:
45+
path: str
46+
dropped: int
47+
sample_offsets: list[int] # byte offsets of the first few bad lines
48+
49+
def as_finding(self) -> dict:
50+
"""Shape compatible with the dispatcher's `actions_needed` finding format."""
51+
return {
52+
"type": "data-corruption-detected",
53+
"severity": "amber" if self.dropped < 5 else "red",
54+
"path": self.path,
55+
"dropped_lines": self.dropped,
56+
"sample_offsets": self.sample_offsets[:3],
57+
"note": (
58+
f"{self.dropped} malformed JSONL line(s) dropped from {self.path}. "
59+
"Likely cause: concurrent append on Windows (no atomic O_APPEND) "
60+
"or an interrupted write. Inspect the file at the listed byte offsets."
61+
),
62+
}
63+
64+
65+
def safe_load_jsonl(path: Path) -> tuple[list[dict], CorruptionReport]:
66+
"""Load a JSONL file, returning rows and a corruption report.
67+
68+
Empty/missing files return empty results with `dropped=0` (not an error).
69+
"""
70+
rows: list[dict] = []
71+
bad_offsets: list[int] = []
72+
if not path.is_file():
73+
return rows, CorruptionReport(str(path), 0, [])
74+
offset = 0
75+
with path.open("rb") as fh:
76+
for raw in fh:
77+
line = raw.decode("utf-8", errors="replace").strip()
78+
if not line:
79+
offset += len(raw)
80+
continue
81+
try:
82+
rows.append(json.loads(line))
83+
except Exception:
84+
bad_offsets.append(offset)
85+
offset += len(raw)
86+
return rows, CorruptionReport(str(path), len(bad_offsets), bad_offsets)
87+
88+
89+
def safe_load_jsonl_rows(path: Path) -> list[dict]:
90+
"""Backwards-compatible drop-in: same signature as the old `_load_rows`.
91+
92+
Drops the corruption report; loaders that don't yet surface findings can
93+
use this and migrate later.
94+
"""
95+
rows, _ = safe_load_jsonl(path)
96+
return rows

0 commit comments

Comments
 (0)