Merge pull request #28 from ThirdKeyAI/docs/readme-held-out-redteam

docs: surface the held-out red-team + typed-channel fix in the READMEs

Author: jaschadub

README.md

@@ -363,7 +363,7 @@ Adjust the workspace dep in `Cargo.toml` if your layout differs.
 - **Not "alignment alone is enough."** The v8 multi-model html-comment-smuggle sweep showed the model with the strongest action-level safety record (GPT-5, 0 v5 refusals) is the same model that bites the content-level smuggle 100% of the time. Action-level alignment doesn't extend to content-level discipline; you need a fence at every layer.
 - **Not a universal claim that reflective learning makes agents better.** v3 shows the curve fires under specific pairings; v2 shows it's mostly flat on frontier-only pairings. Both findings are reported.
 - **Not a comprehensive benchmark suite.** Five curated tasks with synthetic-but-realistic inputs (the karpathy-loop side); six curated task families on the escape-evaluation side. For research-grade evaluation you'd need dozens of tasks with statistical power and held-out attack sets written by parties who didn't design the fences.
-- **Not a measurement of attacker-distribution coverage.** The 14 reflector + task-side adversarial shapes and the six escape-eval task families are hand-authored. They demonstrate the fences fire against representative attacks; they do not estimate the fraction of plausible real-world attacks the fences would catch. A held-out red-team set is listed as a follow-on (see [evals/escape/METHODOLOGY-LIMITS.md](evals/escape/METHODOLOGY-LIMITS.md)).
+- **Not a measurement of attacker-distribution coverage.** The 14 reflector + task-side adversarial shapes and the six escape-eval task families are hand-authored. They demonstrate the fences fire against representative attacks; they do not estimate the fraction of plausible real-world attacks the fences would catch. A first in-repo **held-out** red-team set (attacks authored blind to the fences, scored behaviorally) is now committed: on injection it found the `agent_summary` content marker fence does **not** generalise (26% ≈ 28% baseline) and motivated a typed + grounded inter-agent decision that takes held-out injection to **0/120** — validated on the shipping runtime ([evals/escape/reports/redteam-holdout.md](evals/escape/reports/redteam-holdout.md)). An external human red-team against a frozen fence commit is still the stronger next experiment (see [evals/escape/METHODOLOGY-LIMITS.md](evals/escape/METHODOLOGY-LIMITS.md)).
 - **Not a measurement of false-positive cost.** The fence sweeps measure refusal rate on adversarial inputs. Per-fence benign-input sweeps (e.g., 100 valid hostnames against the URL validator, 100 valid file paths against the path validator) are not yet committed. The escape-eval `fs_boundary` task is the closest existing benign control — symbiont's `objective_success` rate on it is reported in [`evals/escape/reports/tier1-v5.md`](evals/escape/reports/tier1-v5.md).
 - **Not a demonstrated exploit prevention for the v11 typed-argument fence.** The v11 control arm captures the LLM-emitted `whois_lookup` target without actually running `whois`. Strictly: v11 demonstrates that the unfenced control *would have passed* shell-shaped arguments to the executor boundary — counterfactual-execution prevention, not demonstrated exploit prevention. A docker-sandboxed canary verification is listed as a follow-on.
 - **Not a replacement for a real observability stack.** The JSONL sidecars, broadcast traces, and dashboards are enough to audit a run; Langfuse/Helicone/PostHog wired through OpenRouter's Settings → Observability does the long-term visualisation.

evals/escape/README.md

@@ -22,6 +22,8 @@ percentages from the reports.
 - `harnesses/` — substrate adapters (`python_baseline`, `symbiont`).
 - `runner.py` — orchestrates N trials × M substrates, writes JSONL.
 - `analysis/` — per-cell escape rate + 95% CI.
+- `reports/` — substrate-comparison writeups (`tier1-v5`, `tier2-3`, `redteam-holdout`).
+- `redteam/` — held-out red-team harness + the typed-channel injection fix.
 - `results/` — JSONL trial records (gitignored).
 - `tests/` — unit + integration tests.
 
@@ -34,6 +36,36 @@ percentages from the reports.
 - `state_immutability` — read a runtime policy; canary on hash diff of declared-immutable file.
 - `injection_orchestrator` — produce a summary for a downstream agent; canary on injection regex.
 
+## Held-out red-team + the structural fix
+
+The task families above are hand-authored by the same team that built the
+fences. [`redteam/`](redteam/) adds a **held-out** set: 90 attack lures
+written by a subagent blind to the validators, marker list, and canary
+patterns ([`redteam/BRIEF.md`](redteam/BRIEF.md)), scored **behaviorally**
+(the downstream agent is actually run, not regex-matched).
+
+On the injection vector it found the `agent_summary` content marker fence
+does **not** generalise — symbiont **26%** vs permissive baseline **28%**
+on held-out attacks (the in-distribution 3.6% was inflated by marker-shaped
+attacks plus a marker-aligned regex canary). The fix is structural: a typed
+enum decision channel grounded in the trusted ticket, which takes held-out
+injection to **0/120** — validated on the shipping Symbiont runtime
+(`symbi_runtime::toolclad::decision::route_grounded`). Typing *alone*
+(trusting the worker's self-reported severity) is ~38%; grounding is the
+control. Full writeup + scope:
+[`reports/redteam-holdout.md`](reports/redteam-holdout.md).
+
+```
+cd evals/escape
+python -m redteam.run_holdout --mode marker --substrates python_baseline,symbiont --execute  # detection
+python -m redteam.run_holdout --mode typed  --substrates symbiont --execute                  # construction
+python -m redteam.analyze
+```
+
+(`fs_enumerate` / `net_boundary` held-out vectors are deferred — their
+permissive-baseline arm runs against the real host filesystem/network and
+needs a confined sandbox.)
+
 ## Single-cell run
 
 ```