VulnScout v3.2 Product Overhaul

Status: Planned, not started Target version: 3.2.0 Audience: Engineer or autonomous coding agent executing the plan Working directory: /Users/shayaunnejad/vibe-code/vuln-scout

Part 1 — Engineering Brief

What this is

A repackaging and trust-model upgrade for VulnScout, the whitebox security review plugin for Claude Code and Kuzushi. The plugin's runtime works well; its product shape doesn't. This overhaul fixes identity, install ergonomics, host parity, trust signals, and adoption surface — without rewriting the scanner core.

Why we're doing it

A recent product review surfaced four shipping-blockers:

Split brand. The repo presents itself as VulnScout, whitebox-pentest, and vuln-scout across manifests, command prefix, marketplace entry, and npm package. Users see three names for one thing.
Install friction. The default install path is a manual symlink into .claude/plugins/. Claude's official --plugin-dir flow and marketplace install aren't documented as first-class options.
Host parity gap. The Kuzushi wrapper (kuzushi-module.js) returns free-form text and only supports 3 of the 5 report formats the Python runtime produces. Claude users get HTML + bundle outputs; Kuzushi users don't.
Trust model under-claims. The findings.json schema has verdict/confidence/verification_level but lacks explicit provenance, exploitability_status, false_positive_risk, and confidence_reason. Reports can't show why a finding is trusted, only that it is.

Two secondary issues also need addressing:

Top-level metadata leads with breadth claims ("9 languages, 27 skills, 8 agents") while the honest stable/beta/experimental split is buried in docs/feature-maturity.md.
Agent handoffs (app-mapper → threat-modeler, code-reviewer → false-positive-verifier) live in agent Markdown prose rather than as deterministic runtime contracts. Behavior drifts between runs.

What changes (and what doesn't)

Doesn't change:

Scanner engines (Semgrep, Joern, CodeQL, Slither, Trivy, Checkov integrations untouched)
4-finding demo output
Quick scan determinism
Existing findings.json artifacts (schema bump is additive + has migration helper)
The 13 commands keep working

Does change:

Slash command prefix: /whitebox-pentest:* → /vuln-scout:* (with deprecation shims for one release)
Plugin directory: whitebox-pentest/ → vuln-scout/
Plugin/marketplace manifest names and descriptions
Schema version: 1.1.0 → 1.2.0 (adds optional trust_metadata nested object)
All five report renderers gain trust-label rendering
5 new task-shaped skills under vuln-scout/skills/tasks/
3 new deterministic agent handoff hooks
Kuzushi return shape becomes structured ({ ok, output, artifacts, maturity, toolName })
Kuzushi report format enum expands to include html and bundle

Breaking changes (called out explicitly)

Change	Who feels it	Mitigation
`/whitebox-pentest:` → `/vuln-scout:`	Anyone with the old commands memorized or scripted	Shim files for one release print deprecation notice
Marketplace plugin name `whitebox-pentest` → `vuln-scout`	Anyone with `claude plugin install` scripts	`docs/migration-3.x-to-3.2.md` migration guide; no shim possible at this layer
`--no-claude-analysis` flag renamed to `--no-semantic-analysis`	CI configs passing the old flag	Old flag remains as deprecated alias until v3.3.0

Goals (definition of done)

One canonical product name (vuln-scout) across every surface
claude --plugin-dir ./vuln-scout and marketplace install both work and are documented as primary paths
Kuzushi tools return structured { ok, output, artifacts, maturity, toolName } with html and bundle formats supported
Plugin manifest, marketplace entry, and README hero lead with the stable promise; tag beta/experimental honestly
Schema v1.2.0 ships with trust_metadata object; migration is idempotent; all five renderers surface trust labels
5 task-shaped skills (start-audit, review-pr, verify-finding, package-evidence, scope-repo) live under vuln-scout/skills/tasks/ and trigger correctly
3 deterministic handoff hooks replace prose-based agent coordination
Report-quality evals enforce trust label rendering, bundle completeness, PR-comment trust legend
Org-memory compiler turns ratified suppressions + confirmed findings into a reusable rule pack
CI is green end-to-end after every phase

Non-goals

Rewriting scanner integrations
Adding new vulnerability detection capability
Changing the offline quick scan profile's deterministic output
Rewriting the existing 27 knowledge skills
Removing any of the 13 commands

Constraints

check_consistency.py runs in CI and validates manifests, counts, and required strings. Mid-migration states must keep it green.
findings.json artifacts on disk in user repos must keep validating.
The PR-comment renderer has a 55 KB output budget.
--privacy defaults for the org-memory feature must be safe (data never leaks without explicit opt-in).
All changes must work on both Python 3.9 and 3.12 (current CI matrix).

Estimated scope

~30 files modified
~12 new files
1 directory rename (whitebox-pentest/ → vuln-scout/)
4 phases (P0 → P3); P0 is the load-bearing one

Part 2 — Executable Specification

Binding decisions

These are already approved and not up for re-litigation:

Namespace: Full collapse to vuln-scout. Rename plugin directory whitebox-pentest/ → vuln-scout/ and slash prefix /whitebox-pentest:* → /vuln-scout:*. Ship shim command files for one release as a deprecation alias.
Commands: Keep all 13. Promote canonical 5 (full-audit, verify, report, scope, diff) in all docs; demote the 8 advanced ones (scan, threats, sinks, trace, propagate, create-rule, mutate, auto-fix) to an "Advanced" section with maturity tags.
Skills: Add 5 first-class task-shaped skills (start-audit, review-pr, verify-finding, package-evidence, scope-repo) in vuln-scout/skills/tasks/. They orchestrate commands + agents + hooks, not thin wrappers.
Schema: Bump 1.1.0 → 1.2.0. Add trust_metadata nested object with provenance, exploitability_status, false_positive_risk, confidence_reason. Deterministic migration helper backfills from claude_analysis, fp_indicator, source_tool.

Phase P0 — Identity, install, parity, trust skeleton

P0.A — Identity collapse

P0.A.1 — Patch consistency check first (MUST land before directory move).

Edit whitebox-pentest/scripts/check_consistency.py:

Replace hardcoded PLUGIN_ROOT = ROOT / "whitebox-pentest" (line 11) with:

PLUGIN_DIR = os.environ.get("VULNSCOUT_PLUGIN_DIR")
if PLUGIN_DIR:
    PLUGIN_ROOT = ROOT / PLUGIN_DIR
elif (ROOT / "vuln-scout").is_dir():
    PLUGIN_ROOT = ROOT / "vuln-scout"
else:
    PLUGIN_ROOT = ROOT / "whitebox-pentest"

Replace hardcoded _count_dirs(PLUGIN_ROOT / "skills") with sum(1 for p in (PLUGIN_ROOT/"skills").rglob("SKILL.md")) so skills/tasks/ subdirectory is counted correctly. Expected count: 32 (27 knowledge + 5 task skills).
Add check that report tool definition in kuzushi-module.js includes html and bundle in its format enum.
Preserve required string assertions: "hotspot-aware findings", "Claude-first", "audit-plan.md", "review-ledger.json".

P0.A.2 — Atomic directory move. Single commit:

git mv whitebox-pentest vuln-scout
Rewrite all 64 files referencing whitebox-pentest/:
- package.json (lines 18-30 globs)
- .claude-plugin/marketplace.json (source: "./vuln-scout")
- kuzushi-module.js (line 11 path)
- .github/workflows/ci.yml (lines 36, 39, 42, 52-54)
- tests/test_*.py (8 files)
- docs/**, CLAUDE.md, AGENTS.md, AUDIT.md, CHANGELOG.md
- demo/vulnerable-app/README.md

P0.A.3 — Slash prefix rewrite. Mechanical replace /whitebox-pentest: → /vuln-scout: in:

All 13 command bodies under vuln-scout/commands/*.md
READMEs (root + plugin)
vuln-scout/evals/trigger_evals.json and workflow_evals.json (12 hits — query fields only; expected_targets use bare strings)
Hooks vuln-scout/hooks/session-init.md, suggest-next-phase.md
Agent prose in vuln-scout/agents/app-mapper.md, threat-modeler.md

P0.A.4 — Manifest field rewrites.

vuln-scout/.claude-plugin/plugin.json: "name": "vuln-scout", bump version to 3.2.0, new description (P0.B.1)
.claude-plugin/marketplace.json: plugin entry "name": "vuln-scout", "source": "./vuln-scout", new description (P0.B.2)
module.manifest.json: bump version to 3.2.0. id and displayName already canonical.
package.json: bump version to 3.2.0. Name @kuzushi/vuln-scout unchanged.

P0.A.5 — Deprecation aliases. Add 13 shim files at whitebox-pentest/commands/<name>.md (kept outside the renamed plugin directory):

---
description: "[deprecated] Renamed to /vuln-scout:<name>"
argument-hint: "<same as canonical>"
deprecated: true
---
This command was renamed. Run `/vuln-scout:<name>` instead.

The deprecated: true frontmatter is skipped by check_consistency.py's count logic. Removed in v4.0.0.

P0.B — Maturity-honest metadata

P0.B.1 — vuln-scout/.claude-plugin/plugin.json.description:

"Claude Code plugin for whitebox security review. Stable: deterministic offline quick scan, shared findings.json with stable keys and hotspot-aware findings, SARIF/Markdown/HTML/bundle reports, suppressions, CI fail-on gate, Kuzushi parity. Beta: deep profile with Joern/CodeQL/Slither/Trivy/Checkov when installed. Experimental: auto-fix, PoC, mutation. Writes audit-plan.md and review-ledger.json for reviewer-driven workflows."

P0.B.2 — .claude-plugin/marketplace.json.description:

"AI-assisted whitebox security review with a stable offline quick scan, evidence-backed verification, suppressions, and SARIF/HTML/bundle reports. Optional deep analyzers when installed. Five canonical workflows: full-audit, verify, report, scope, diff."

P0.B.3 — Root README hero rewrite (README.md lines 1-20):

VulnScout tagline
Maturity row (shields.io static badges linking docs/feature-maturity.md)
One-liner install + one-liner canonical command
Stable promise summary (offline quick scan, findings.json, reports, CI gate)

P0.B.4 — Command description prefixes. Add [stable] / [beta] / [experimental] prefix to each command's YAML description field:

[stable]: full-audit, verify, report, scope, diff
[beta]: scan, threats, sinks, trace, propagate
[experimental]: create-rule, mutate, auto-fix

P0.B.5 — Maturity surface points.

Slash autocomplete: prefixes above
vuln-scout/scripts/doctor.py and scan_orchestrator.py banners print "Profile maturity: quick=stable | deep=beta | audit=beta"
Generated reports: scripts/report.py adds a "Tool maturity" section pulling from new vuln-scout/references/maturity.json
Plugin manager: covered by P0.B.1 / P0.B.2

P0.C — Install + onboarding

P0.C.1 — Create docs/install.md (replaces symlink-first onboarding):

Marketplace install (canonical): claude plugin install vuln-scout

Local zip / dir testing with --plugin-dir:

git clone https://github.com/allsmog/vuln-scout
claude --plugin-dir ./vuln-scout/vuln-scout

Kuzushi runtime: npm install @kuzushi/vuln-scout
Symlink (contributor workflow, demoted to last)
Verify: python3 vuln-scout/scripts/doctor.py --strict then /vuln-scout:full-audit demo/vulnerable-app

P0.C.2 — Create workflow docs.

docs/workflows/first-run.md: 5-minute path → doctor → /vuln-scout:full-audit demo/vulnerable-app → report.html. Asserts 4 expected demo findings.
docs/workflows/pr-review.md: /vuln-scout:diff origin/main → /vuln-scout:verify loop → evidence bundle for PR comment
docs/workflows/ci.md: GitHub Actions template with --fail-on high gate and bundle artifact upload
docs/migration-3.x-to-3.2.md: rename guide for users with hardcoded /whitebox-pentest:* scripts

P0.C.3 — First-run smoke test. Create vuln-scout/scripts/first_run_smoke.py:

doctor.py --json --strict (must exit 0 with Semgrep present)
scan_orchestrator.py demo/vulnerable-app --profile quick --output /tmp/first-run.json
Assert exactly 4 findings, severities [high, high, medium, medium]
report.py /tmp/first-run.json --format html and --format bundle
Assert bundle zip contains findings.json, findings.sarif, vex.json, attestation.json, report.html, README.md

P0.C.4 — CI updates in .github/workflows/ci.yml:

All whitebox-pentest/ → vuln-scout/
New job step: python3 vuln-scout/scripts/first_run_smoke.py
New step: Kuzushi parity test (P0.D.2)

P0.D — Kuzushi parity

P0.D.1 — Edit kuzushi-module.js:

Replace format enum (line 135):

format: { type: "string", enum: ["sarif", "md", "json", "html", "bundle"], description: "Report format. bundle = zip of findings.json + report.html + audit-plan.md + review-ledger.json + vex.json + attestation.json." }

Add structured-return helper at top of file:

function collectArtifacts(target, params) {
  const claudeDir = join(target, ".claude");
  const artifacts = {};
  for (const [key, rel] of [
    ["findings", "findings.json"],
    ["audit_plan", "audit-plan.md"],
    ["review_ledger", "review-ledger.json"],
    ["threat_model", "threat-model.md"],
  ]) {
    const p = join(claudeDir, rel);
    if (existsSync(p)) artifacts[key] = p;
  }
  if (params.output && existsSync(params.output)) artifacts.report = params.output;
  return artifacts;
}

Rewrite success return (replace { ok: true, output: text }):

return {
  ok: true,
  output: text || "Analysis complete.",
  artifacts: collectArtifacts(target, params),
  maturity: TOOL_MATURITY[toolName],
  toolName,
};

Add TOOL_MATURITY constant mapping the 13 tool names to "stable" | "beta" | "experimental" (mirrors P0.B.4).

P0.D.2 — Parity eval. Create tests/test_kuzushi_parity.py:

Import module via node -e subprocess
Assert 13 tools exported with expected names
Assert report tool's format enum contains all 5 values
Assert each tool definition has headless: true and inputSchema.required
Wire into CI as a new step after existing Kuzushi import check

P0.E — Schema v1.2.0 + migration

P0.E.1 — Edit vuln-scout/references/findings.schema.json:

schema_version.enum: extend to ["1.0.0", "1.1.0", "1.2.0"]. Older artifacts continue to validate.
Add to properties.findings.items.properties (optional — preserves old-artifact compatibility):

"trust_metadata": {
  "type": "object",
  "description": "Trust-model signals introduced in schema v1.2.0",
  "properties": {
    "provenance": {
      "type": "object",
      "required": ["origin"],
      "properties": {
        "origin": { "type": "string", "enum": ["deterministic_tool","llm_analysis","dynamic_verified","human_review","mixed"] },
        "tool": { "type": "string" },
        "contributors": { "type": "array", "items": { "type": "string", "enum": ["deterministic_tool","llm_analysis","dynamic_verified","human_review"] } }
      }
    },
    "exploitability_status": { "type": "string", "enum": ["confirmed","plausible","blocked_by_control","requires_auth","unreachable","unknown"] },
    "false_positive_risk": {
      "type": "object",
      "required": ["level"],
      "properties": {
        "level": { "type": "string", "enum": ["low","medium","high","unknown"] },
        "reason": { "type": "string" }
      }
    },
    "confidence_reason": { "type": "string", "maxLength": 280 }
  }
}

P0.E.2 — Create vuln-scout/scripts/migrate_artifact.py:

Pure function migrate_to_1_2_0(artifact: dict) -> dict + CLI wrapper. Idempotent: if schema_version == "1.2.0" and every finding has trust_metadata, return unchanged.

Backfill rules (deterministic):

New field	Source	Rule
`provenance.origin`	`claude_analysis` present AND `verification_level >= 3`	`"mixed"`
	`claude_analysis` present, no other signal	`"llm_analysis"`
	`dynamic_verified == true`	`"dynamic_verified"`
	`verification_level >= 2`	`"deterministic_tool"`
	else	`"deterministic_tool"`
`provenance.tool`	`claude_analysis` present → `"claude"`; else `source_tool`
`exploitability_status`	`claude_analysis.exploitable == true` OR `dynamic_verified`	`"confirmed"`
	`verdict == "false_positive"` AND `fp_indicator` matches `sanitize`/`control`	`"blocked_by_control"`
	`verdict == "false_positive"` AND `fp_indicator` matches `unreachable`/`dead`	`"unreachable"`
	`verdict == "needs_review"`	`"plausible"`
	else	`"unknown"`
`false_positive_risk.level`	`fp_indicator` present	`"high"`
	`verdict == "verified"` AND `confidence in ("verified","high")`	`"low"`
	`verdict == "needs_review"`	`"medium"`
	`confidence == "low"`	`"high"`
	else	`"unknown"`
`false_positive_risk.reason`	`fp_indicator` verbatim if present, else first sentence of `claude_analysis.reasoning` (≤200 chars)
`confidence_reason`	First sentence of `claude_analysis.reasoning`, else `f"Derived from {source_tool} at L{verification_level}"`

P0.E.3 — Validator extension in vuln-scout/scripts/prompt_artifacts.py:

Add TRUST_METADATA_PROVENANCE_VALUES, TRUST_METADATA_EXPLOIT_VALUES, TRUST_METADATA_FP_LEVELS constants
Add validate_trust_metadata(finding) -> list[str]
Update validate_findings_artifact: when schema_version == "1.2.0", each finding MUST have trust_metadata. Older versions: optional.

P0.E.4 — Update producers. apply_claude_analysis.py and auto_triage.py must populate trust_metadata on the same write path that sets claude_analysis / fp_indicator — so freshly-generated artifacts skip migration. Migration helper remains the safety net.

P0.F — Renderer updates (consumes P0.E)

All renderers must degrade gracefully when trust_metadata is absent.

P0.F.1 — vuln-scout/scripts/markdown_report.py:

Add _trust_badge(finding) helper rendering single line: Trust: <provenance.origin> | FP-risk: <level> | Exploitability: <status> with shields.io badges
Insert after the Verdict/Confidence/ID line
Add _confidence_reason_block(finding) — italicized one-liner when present
Append _trust_legend() section before _next_actions

P0.F.2 — vuln-scout/scripts/html_report.py:

Add TRUST_COLORS dict (next to SEVERITY_COLORS):
- provenance: deterministic_tool=#0e7490, llm_analysis=#7c3aed, dynamic_verified=#16a34a, human_review=#0891b2, mixed=#475569
- fp_risk: low=#16a34a, medium=#ca8a04, high=#dc2626, unknown=#6b7280
- exploitability: confirmed=#dc2626, plausible=#ea580c, blocked_by_control=#16a34a, requires_auth=#0e7490, unreachable=#6b7280, unknown=#9ca3af
Add "Trust" column to _findings_table with three coloured chips; chip carries title="{confidence_reason or fp_risk.reason}" for tooltip
Add _trust_legend_section() after the chains graph
Inline CSS additions in _html_head

P0.F.3 — vuln-scout/scripts/pr_comment.py (55 KB budget):

Add _trust_marker(finding) helper returning compact [T:LLM|FP:M|X:plausible] (≤22 chars). Stable letter codes: provenance → DET/LLM/DYN/HR/MIX; fp risk → L/M/H/?; exploitability → first lowercase token.
Add one-time _trust_legend() block between _header_table and _new_in_pr (~180 bytes once, not per-finding)
Truncation function MUST keep the legend outside the <details> block

P0.F.4 — vuln-scout/scripts/evidence_bundle.py:

Extend _vex_state(finding) to prefer trust_metadata.exploitability_status:
- confirmed → "affected"
- blocked_by_control, unreachable → "not_affected"
- plausible, requires_auth, unknown → "under_investigation"
New _vex_justification(finding) returning CycloneDX analysis.justification enum when state is not_affected:
- blocked_by_control → "protected_by_mitigating_control"
- unreachable → "code_not_reachable"
- requires_auth → "requires_environment"

Extend vulnerability["analysis"] in build_vex:

"analysis": {
  "state": state,
  "justification": justification,
  "detail": "; ".join([confidence_reason, evidence_summary]).strip("; "),
  "response": []
}

Add three new CycloneDX properties: vuln-scout:provenance, vuln-scout:fp_risk, vuln-scout:exploitability_status
_readme gains a "Trust Model" section
build_attestation gains trust_model_summary block (counts grouped by provenance origin and fp_risk level)

P0.F.5 — vuln-scout/scripts/report.py dispatcher:

After load_artifact and before suppression apply: detect older schema version and auto-migrate via migrate_artifact.migrate_to_1_2_0(artifact)
Add --no-migrate CLI flag for debugging
Log single stderr line on migration: "info: migrated artifact from 1.1.0 to 1.2.0 in memory"

P0.G — Audit-profile naming fix

P0.G.1 — vuln-scout/scripts/scan_orchestrator.py:

Line 931 argparse: rename --no-claude-analysis to --no-semantic-analysis. Keep old flag as deprecated alias via dest=.
Lines 646-650 log message: replace with "info: --no-semantic-analysis acknowledged; scan_orchestrator.py runs only deterministic analyzers. Semantic analysis is performed by /vuln-scout:verify or /vuln-scout:full-audit post-scan."
Profile description for audit: change "Claude-assisted review baseline" → "Deterministic baseline for Claude-driven review (semgrep + joern + codeql + secrets, offline)"

P0.G.2 — Command doc updates. Grep for --no-claude-analysis across vuln-scout/commands/*.md, replace with new flag name + one-line note that audit profile does not invoke Claude during scan phase.

P0.G.3 — Deprecation timeline. Keep --no-claude-analysis alias for one minor version (until v3.3.0). CHANGELOG.md gets "Deprecated" entry under v3.2.0 release.

Phase P1 — Front-door skills + workflow ergonomics

P1.A — Five task-shaped skills

Create directory vuln-scout/skills/tasks/ with five SKILL.md files:

Skill	Description (≤120 chars)	Triggers (action verbs)	Orchestrates	Produces
`start-audit`	Guided first-run security audit: doctor, scope, threats, scan, verify, report.	"start audit", "audit this repo", "security review of", "review this codebase for vulns"	`/vuln-scout:scope` → `/vuln-scout:threats` → `/vuln-scout:scan` → `/vuln-scout:verify` (per finding) → `/vuln-scout:report --format bundle`. Calls agents: app-mapper, threat-modeler, code-reviewer. Triggers hooks: session-init, large-codebase-check.	`.claude/audit-plan.md`, `.claude/review-ledger.json`, `.claude/findings.json`, `report.html`, `bundle.zip`
`review-pr`	Diff-aware PR security review with verified findings and PR comment payload.	"review this PR", "scan PR", "diff scan", "check pull request"	`/vuln-scout:diff <base>` → `/vuln-scout:verify` (only new/changed) → `/vuln-scout:report --format md`. Calls false-positive-verifier, code-reviewer. Hook: poc-safety-check.	`.claude/diff-findings.json`, `pr-comment.md`, optional `bundle.zip`
`verify-finding`	Drive a single finding through CPG verification and false-positive triage.	"verify VSCOUT-", "is this exploitable", "confirm this finding"	`/vuln-scout:trace` → `/vuln-scout:verify` → optional `/vuln-scout:propagate`. Calls false-positive-verifier, local-tester.	Updated review-ledger.json entry, optional trace artifact
`package-evidence`	Bundle findings + reports + audit-plan + ledger into a single distributable evidence zip.	"package evidence", "export bundle", "create deliverable", "share findings"	`/vuln-scout:report --format bundle` + `--format sarif` + `--format html`. No agents.	`evidence-bundle.zip`
`scope-repo`	Decide audit boundaries for large/monorepo targets and write audit-plan.md.	"scope this repo", "where should I focus", "what should I audit first", "large codebase"	`/vuln-scout:scope` → app-mapper agent → writes `.claude/audit-plan.md`. Hook: large-codebase-check.	`.claude/audit-plan.md`, workspace list, scope name

Each SKILL.md includes a "When NOT to trigger" section listing knowledge-skill keywords (e.g., start-audit must NOT preempt the threat-modeling knowledge skill).

P1.B — Trigger eval coverage

Add 5 positive + 5 negative cases to vuln-scout/evals/trigger_evals.json:

Positive: each task skill triggers on its primary verb pattern
Negative: knowledge-skill keywords (e.g., "STRIDE threat model") must trigger threat-modeling skill, NOT start-audit

P1.C — README structural rewrite

Root README.md:

Hero (logo, tagline, maturity row, install + canonical command)
5-Minute Demo
Stable Promise (link to docs/feature-maturity.md)
Install (short, link to docs/install.md)
Canonical Workflows (5) — each links to docs/workflows/*.md
Advanced Commands (collapsed <details> — the other 8)
Feature Maturity table
Kuzushi integration (mention structured artifacts return)
Project structure
License

vuln-scout/README.md (renamed plugin README) — same shape, plugin-internal focus:

Plugin manifest summary
5 canonical commands (full flag tables)
8 advanced commands (table only)
5 task skills + 27 knowledge skills (one table each)
8 agents
4 hooks
Shared findings.json contract (link to schema)
Local development

Phase P2 — Agent rails + report-quality evals

P2.A — Deterministic agent handoffs

Convert prose-based handoffs into structured hooks. Each new hook writes an atomic typed payload that the receiving agent reads.

P2.A.1 — vuln-scout/hooks/handoff-app-mapper.md (event: SubagentStop, match_subagent: app-mapper):

Verify .claude/app-understanding.md exists; fail-soft with warning if missing
Validate required sections (P2.A.4)
Write .claude/handoff-app-mapper.json with extracted structured data: entry points, trust boundaries, framework list, identified high-risk modules
Append subjects[] entry to .claude/review-ledger.json with subject_type: "app-understanding" (new value — add to VALID_REVIEW_SUBJECT_TYPES in prompt_artifacts.py)
Suggest /vuln-scout:threats as the next command (single sentence)

P2.A.2 — vuln-scout/hooks/handoff-code-reviewer.md (replaces prose in suggest-next-phase.md):

Read .claude/findings.json post-SubagentStop
Compute counts: critical, high, verified, needs_review
Deterministic branching:
- critical > 0 OR high > 0 → invoke false-positive-verifier on top-N findings, write .claude/handoff-code-reviewer.json with prioritized stable_keys
- only medium/low → write suggested_action: "expand-scope"
- none → write suggested_action: "report"

P2.A.3 — vuln-scout/hooks/handoff-local-tester.md (match_subagent: local-tester):

Read .claude/local-test-results.json
Graduate findings with successful PoCs to verification_level=4, set trust_metadata.provenance.origin = "dynamic_verified" and exploitability_status = "confirmed"
Write .claude/handoff-local-tester.json containing stable_keys ready for poc-developer

P2.A.4 — App-understanding contract in vuln-scout/scripts/prompt_artifacts.py:

APP_UNDERSTANDING_REQUIRED_SECTIONS = (
    "Application Overview",
    "Trust Boundaries",
    "Entry Points",
    "Frameworks and Dependencies",
    "High-Risk Modules",
)

Plus validate_app_understanding(text) -> list[str].

P2.A.5 — Agent prose rewrites.

vuln-scout/agents/app-mapper.md: rewrite to "Produce .claude/app-understanding.md with sections listed in APP_UNDERSTANDING_REQUIRED_SECTIONS. Contract lives in code."
vuln-scout/agents/threat-modeler.md: replace prose-based expectations with "Read .claude/handoff-app-mapper.json (typed) and .claude/app-understanding.md (narrative). Cite findings using entry-point IDs from the typed payload."

P2.A.6 — Atomicity. Every handoff hook starts by deleting any prior handoff file for its agent, writes atomically (temp file + rename). Prevents stale data from failed runs.

P2.B — Report-quality evals

P2.B.1 — Create vuln-scout/evals/report_quality_evals.json:

[
  {
    "id": "trust-label-md-render",
    "input_fixture": "tests/fixtures/artifacts/sample-findings-v1_2_0.json",
    "renderer": "markdown",
    "must_contain": ["Trust:", "FP-risk:", "Exploitability:", "## Trust Legend"],
    "must_not_contain": ["KeyError", "undefined"]
  },
  {
    "id": "bundle-completeness",
    "input_fixture": "tests/fixtures/artifacts/sample-findings-v1_2_0.json",
    "renderer": "bundle",
    "expected_bundle_files": ["findings.json","findings.sarif","vex.json","attestation.json","README.md"],
    "expected_vex_states": ["affected","not_affected","under_investigation"],
    "attestation_must_contain_keys": ["trust_model_summary"]
  },
  {
    "id": "pr-comment-trust-legend",
    "input_fixture": "tests/fixtures/artifacts/sample-findings-v1_2_0.json",
    "renderer": "pr_comment",
    "must_contain": ["[T:", "FP:", "X:"],
    "max_bytes": 56320
  },
  {
    "id": "hotspot-to-finding-graduation",
    "input_fixture": "tests/fixtures/artifacts/sample-graduation.json",
    "assertion": "hotspot_with_verification_level_ge_3_becomes_finding"
  },
  {
    "id": "migration-backfill-idempotent",
    "input_fixture": "tests/fixtures/artifacts/sample-findings.json",
    "assertion": "migrate_then_migrate_equal"
  }
]

P2.B.2 — New fixtures under tests/fixtures/artifacts/:

sample-findings-v1_2_0.json — 12-15 findings covering every provenance/exploitability/fp_risk combination, attack chains, in_diff flags, hotspots, and one suppressed entry
sample-graduation.json — hotspot pre/post graduation (two embedded states)

P2.B.3 — validate_evals.py extensions:

Add validate_report_quality_cases(cases)
Glob *_evals.json rather than hardcoding two filenames

P2.B.4 — Runner extension. Extend vuln-scout/scripts/run_prompt_evals.py with --suite report-quality mode that loads fixtures, runs each renderer, asserts constraints. Non-zero exit on failure.

P2.B.5 — CI wiring. Add .github/workflows/ci.yml steps:

python3 vuln-scout/scripts/run_prompt_evals.py --suite report-quality
Separate step verifying migrate_artifact.py is no-op on already-migrated fixtures

Phase P3 — Org memory moat

P3.A — Directory layout

Inside target repo:

.vuln-scout/org-memory/
  accepted-suppressions.yaml   # ratified suppressions with rationale + reviewer
  confirmed-findings.yaml      # canonical patterns the org cares about
  custom-rules/
    semgrep/<rule-id>.yaml
    joern/<rule-id>.scala
  review-patterns.yaml         # human reviewer heuristics
  manifest.json                # version, last-updated, hashes, privacy mode

YAML over JSON: hand-editable, supports comments for audit trail.

P3.B — Compiler script

vuln-scout/scripts/org_memory_compiler.py:

Inputs:

.claude/scan-history/*.json (existing via feedback_collector.py)
.claude/rule-stats.json (existing)
.claude/review-ledger.json
Findings where verdict in {"verified","false_positive"} AND trust_metadata.provenance.origin == "human_review"

Rule generation policy (deterministic):

confirmed-findings.yaml: rule_id with verified >= 3 AND verified / total >= 0.5 graduates to confirmed pattern with CWE, sample paths (hashed if --privacy strict), message template
accepted-suppressions.yaml: stable_key suppressed manually 2+ times across scans graduates with reason verbatim + provenance: human_review
custom-rules/semgrep/: derived rules for confirmed Semgrep patterns, tightened by observed file/function context. Schema vuln-scout.org.v1
review-patterns.yaml: cluster claude_analysis.reasoning strings by rule_id when verdict is consistently false_positive; emit "demote-if-matches" hints

P3.C — Integration with existing feedback path

In scan_orchestrator.py (around lines 725-727 and 831):

Extend FeedbackCollector.get_auto_suppressions() to union with .vuln-scout/org-memory/accepted-suppressions.yaml
Add feedback.apply_org_memory_rules(findings) invoked just after apply_rule_calibration(findings); stamps trust_metadata.provenance.origin = "human_review" on matching findings, boosts confidence to "high"

P3.D — Privacy controls

org_memory_compiler.py --privacy {open,hashed,strict}:

open (default): file paths + code excerpts verbatim
hashed: SHA-256 of path + excerpt; preserves match counts
strict: no excerpts, no paths — only rule_id + verdict counts + CWE

manifest.json records chosen mode. Compiler refuses to overwrite strict with open without --force.

P3.E — Invocation surface

New command vuln-scout/commands/org-memory-compile.md
Add to check_consistency.py command count (becomes 14, or 13+1 advanced)
Default .gitignore entry for .vuln-scout/org-memory/ shipped with plugin; explicit --allow-commit required to remove
MIN_SAMPLES_FOR_DEMOTE reused from feedback_collector (already 10) prevents single-cluster overfit
--dry-run mode prints proposed rules without writing

Sequencing dependencies

P0.A.1 (consistency-check patch)
  └─> P0.A.2 (directory move) ─┬─> P0.A.3 (slash rewrite)
                                ├─> P0.A.4 (manifests)
                                └─> P0.D.1 (Kuzushi return shape)
                                      └─> P0.D.2 (parity eval)

P0.E.1 (schema bump) ─┬─> P0.E.2 (migration helper)
                      ├─> P0.E.3 (validator)
                      ├─> P0.E.4 (producer updates)
                      └─> P0.F.* (renderer updates)
                              └─> P2.B (report-quality evals)

P0.G (audit-profile naming) — independent, lands anytime in P0
P0.B (metadata) — after P0.A.4
P0.C (install docs) — after P0.A.* complete
P1 — after P0 complete
P2.A (agent rails) — can land in parallel with P1
P3 — after P2.A (consumes human_review provenance signals)

Critical files

Identity/UX:

whitebox-pentest/scripts/check_consistency.py (patch first)
.claude-plugin/marketplace.json
whitebox-pentest/.claude-plugin/plugin.json (becomes vuln-scout/.claude-plugin/plugin.json)
kuzushi-module.js
package.json, module.manifest.json
.github/workflows/ci.yml
README.md, whitebox-pentest/README.md
All 13 command files, 8 agent files, 4 hook files

Schema/runtime:

vuln-scout/references/findings.schema.json
vuln-scout/scripts/migrate_artifact.py (new)
vuln-scout/scripts/report.py
vuln-scout/scripts/markdown_report.py
vuln-scout/scripts/html_report.py
vuln-scout/scripts/pr_comment.py
vuln-scout/scripts/evidence_bundle.py
vuln-scout/scripts/scan_orchestrator.py
vuln-scout/scripts/prompt_artifacts.py
vuln-scout/scripts/apply_claude_analysis.py, auto_triage.py
vuln-scout/scripts/first_run_smoke.py (new)

Agent rails:

vuln-scout/hooks/handoff-app-mapper.md (new)
vuln-scout/hooks/handoff-code-reviewer.md (new)
vuln-scout/hooks/handoff-local-tester.md (new)
vuln-scout/agents/app-mapper.md, threat-modeler.md (rewrites)

Skills:

vuln-scout/skills/tasks/{start-audit,review-pr,verify-finding,package-evidence,scope-repo}/SKILL.md (new)

Evals:

vuln-scout/evals/trigger_evals.json (extend with 5 task skill cases)
vuln-scout/evals/report_quality_evals.json (new)
vuln-scout/scripts/validate_evals.py (extend)
vuln-scout/scripts/run_prompt_evals.py (extend)
tests/fixtures/artifacts/sample-findings-v1_2_0.json (new)
tests/fixtures/artifacts/sample-graduation.json (new)
tests/test_kuzushi_parity.py (new)

Org memory:

vuln-scout/scripts/org_memory_compiler.py (new)
vuln-scout/commands/org-memory-compile.md (new)

Docs:

docs/install.md (new)
docs/workflows/first-run.md, pr-review.md, ci.md (new)
docs/migration-3.x-to-3.2.md (new)
docs/feature-maturity.md (existing — link from hero)
vuln-scout/references/maturity.json (new, machine-readable maturity by tool)

Verification

End-to-end test sequence (run after each phase):

Consistency: python3 vuln-scout/scripts/check_consistency.py exits 0
Schema validation: python3 vuln-scout/scripts/validate_evals.py exits 0 (covers trigger, workflow, report_quality suites)
Migration round-trip: python3 vuln-scout/scripts/migrate_artifact.py tests/fixtures/artifacts/sample-findings.json --in-place then re-run; second invocation prints "already at 1.2.0" and is byte-identical to first output
First-run smoke: python3 vuln-scout/scripts/first_run_smoke.py exits 0 with 4 expected findings + bundle contents present
Renderers: For each format (md, html, sarif, bundle), python3 vuln-scout/scripts/report.py tests/fixtures/artifacts/sample-findings-v1_2_0.json --format <fmt> --output /tmp/out.<ext> succeeds and output contains trust badges/markers
Kuzushi parity: pytest tests/test_kuzushi_parity.py exits 0
Report-quality evals: python3 vuln-scout/scripts/run_prompt_evals.py --suite report-quality exits 0
Trigger evals: python3 vuln-scout/scripts/run_prompt_evals.py --suite triggers shows ≥0.67 activation rate on positive cases for new task skills
Install dry-run: claude --plugin-dir ./vuln-scout/vuln-scout --check succeeds (skip if Claude CLI unavailable in CI; continue-on-error: true)
CI green: .github/workflows/ci.yml passes end-to-end on both Python 3.9 and 3.12
Migration alias: /whitebox-pentest:full-audit shim file renders the deprecation notice and points to /vuln-scout:full-audit
PR-comment budget: synthetic 1000-finding artifact runs through pr_comment.py and output is ≤55 KB with legend still present

Risk register

check_consistency.py mid-migration: P0.A.1 patch MUST land before P0.A.2 directory move, else CI fails on the merge that renames
Eval references: expected_targets arrays in trigger_evals.json use bare strings like "full-audit" which survive the rename; only query fields need rewriting
Shim files on Windows: shim files (not symlinks) are portable. Symlink-based aliasing rejected for this reason
Schema strictness: validator gates trust_metadata requirement on schema_version == "1.2.0" so older fixtures keep working
Producer drift: if apply_claude_analysis.py / auto_triage.py are not updated alongside the migration helper, fresh artifacts may temporarily lack trust_metadata until migration on read fires
PR-comment truncation: legend MUST live outside the <details> block. Unit test covers 100-finding and 1000-finding cases
Privacy footgun: .vuln-scout/org-memory/ gitignored by default; --allow-commit required to remove
Marketplace breaking change: plugin name change from whitebox-pentest to vuln-scout breaks anyone with claude plugin install vuln-scout/whitebox-pentest in scripts. Called out in docs/migration-3.x-to-3.2.md
Demo rule IDs: vuln-scout.local.* prefix already canonical — no rule-ID rename needed

Execution notes for the agent

Work on a feature branch (feat/v3.2-product-overhaul recommended), not main
Commit at phase boundaries; run python3 whitebox-pentest/scripts/check_consistency.py after each commit to verify green state
Before P0.A.2 (directory move), run python3 whitebox-pentest/scripts/scan_orchestrator.py demo/vulnerable-app --profile quick once to capture baseline output for regression comparison
After P0.E (schema bump), regenerate tests/fixtures/artifacts/sample-findings.json through the migration helper and verify byte-equality on second invocation
Do not skip the deprecation alias shim files (P0.A.5) — they're the only thing keeping existing user workflows from hard-breaking
Do not amend commits across phase boundaries; each phase's commit history is the migration audit trail

FilesExpand file tree

V3.2-OVERHAUL.md

Latest commit

History