Merge pull request #32 from ThirdKeyAI/test/sanitizer-jailbreak-regre… #72
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: ci | |
| # v7 #5 — turns the v6-shipped tooling into enforcement. | |
| # | |
| # Three gates: | |
| # | |
| # 1. safety-gates — pure-Python audit + lint. No Rust toolchain | |
| # required, runs in seconds. Catches sanitiser-escapes in any | |
| # committed knowledge.db, homoglyph identifiers in any .cedar | |
| # policy file, invisible control chars in any policy file. | |
| # | |
| # 2. rust-tests — full cargo test --workspace. Includes the | |
| # exhaustive sanitiser fuzz (`sanitize_covers_every_declared_range`, | |
| # 243 code points × every forbidden range), the new HTML-comment | |
| # strip tests, the no-shell-out static guard against MCP-shape | |
| # RCE, and the marketplace-poisoning trust-boundary test on the | |
| # delegator. Requires a sibling `symbiont/` checkout because the | |
| # workspace path-deps on `../symbiont/crates/runtime`. | |
| # | |
| # 3. clippy — `cargo clippy --all-targets -- -D warnings` on | |
| # the demo crates only (the upstream runtime emits its own | |
| # warnings outside our control). | |
| # | |
| # Trigger on push and PR so a sanitiser regression, a homoglyph | |
| # `Action::"…"` literal, or a process-spawning executor is caught | |
| # before merge instead of after the next sweep. | |
| on: | |
| push: | |
| branches: [main] | |
| pull_request: | |
| branches: [main] | |
| concurrency: | |
| group: ci-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| safety-gates: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Set up Python | |
| uses: actions/setup-python@v5 | |
| with: | |
| python-version: "3.12" | |
| - name: Cedar policy linter (homoglyph + invisible-char fence) | |
| run: scripts/lint-cedar-policies.py | |
| - name: DSL linter (homoglyph + invisible-char fence — v10) | |
| # v10 — symmetric to the Cedar linter. Catches a PR that | |
| # adds `tool "answеr"` (Cyrillic `е`) to agents/*.symbi, | |
| # reflector/*.symbi, or delegator/*.symbi (`.dsl` accepted | |
| # as a legacy fallback). | |
| run: scripts/lint-dsl-files.py | |
| - name: Knowledge-store audit (sanitiser-escape fence) | |
| # --strict respects .audit-allowlist (gpt5-ms historical pre-patch | |
| # escapes from v5; documented in MODEL-SWEEP-REPORT-v6.md). Any | |
| # new tag with escapes — e.g. a regression in a fresh sweep — fails. | |
| run: scripts/audit-knowledge-stores.py --strict | |
| - name: Paper-claims verifier (escape evals — tier1-v5 + tier2-3 + benign-control) | |
| # Recomputes every headline number in the escape-eval reports | |
| # against the committed redacted aggregate | |
| # (evals/escape/results/aggregate-summary.json — counts only, no | |
| # exploit content). Stdlib-only and runs in well under a second: | |
| # no PyYAML, no per-trial JSONL (which stays gitignored). Fails | |
| # the build if a published count drifts from the aggregate, so the | |
| # reports can't silently desync from the data again. See | |
| # evals/escape/METHODOLOGY-LIMITS.md "Reproducing the headline numbers". | |
| run: python3 evals/escape/scripts/verify_paper_claims.py | |
| rust-tests: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Check out this repo (under <parent>/symbiont-orga-demo) | |
| uses: actions/checkout@v4 | |
| with: | |
| path: symbiont-orga-demo | |
| # `Cargo.toml` declares the runtime as | |
| # `path = "../symbiont/crates/runtime"`. CI must place the | |
| # upstream side-by-side or the build fails on resolve. We pin | |
| # to a specific Symbiont SHA in the public-release branch so a | |
| # main-branch refactor in the runtime cannot break this CI | |
| # without a deliberate bump here. | |
| - name: Check out Symbiont runtime side-by-side | |
| uses: actions/checkout@v4 | |
| with: | |
| repository: ThirdKeyAI/symbiont | |
| path: symbiont | |
| ref: 57a8847c3f14d855086edaebb2d725babdc3e9d4 | |
| # v11+ — bridge crate path-deps on `../ToolClad/rust`. Without | |
| # this checkout, the workspace fails to resolve at cargo test | |
| # time. Pinned to a SHA so a main-branch change in ToolClad | |
| # cannot break this CI without a deliberate bump here. | |
| - name: Check out ToolClad side-by-side | |
| uses: actions/checkout@v4 | |
| with: | |
| repository: ThirdKeyAI/ToolClad | |
| path: ToolClad | |
| # includes the scope_target + url egress hardening that | |
| # idn_confusable_probe asserts (block_internal, obfuscated-IP | |
| # rejection, per-label length, always-block cloud metadata). | |
| # NOTE: ToolClad PR #8 head — re-pin to the merge commit once merged. | |
| ref: 519d7002121246eca875d77ce38eacfd3512482a | |
| - name: Install Rust toolchain (pinned — see rust-toolchain.toml) | |
| # Pinned, not @stable: the typestate trybuild compile-fail proofs | |
| # assert exact compiler diagnostics, which drift across rustc | |
| # releases (1.96 inlines single-candidate "method was found for" | |
| # notes; the blessed .stderr target 1.95). Pinning also stops a | |
| # new clippy lint in a future stable from breaking `-D warnings`. | |
| # Bump deliberately (and re-bless the .stderr) rather than floating. | |
| uses: dtolnay/rust-toolchain@1.95.0 | |
| - name: Cache cargo registry + target | |
| uses: actions/cache@v4 | |
| with: | |
| path: | | |
| ~/.cargo/registry | |
| ~/.cargo/git | |
| symbiont-orga-demo/target | |
| key: cargo-${{ runner.os }}-${{ hashFiles('symbiont-orga-demo/Cargo.lock') }} | |
| - name: cargo test --workspace | |
| working-directory: symbiont-orga-demo | |
| run: cargo test --release --workspace | |
| - name: Targeted — exhaustive sanitiser fuzz must run | |
| working-directory: symbiont-orga-demo | |
| # Belt-and-suspenders against `cargo test --workspace` ever | |
| # accidentally filtering this test out (e.g. via a target | |
| # selector). The 243-code-point sweep is the central evidence | |
| # for the v6 sanitiser claim. | |
| run: | | |
| cargo test --release --lib -p demo-karpathy-loop \ | |
| sanitize_covers_every_declared_range -- --nocapture | |
| - name: Targeted — process-spawn guard must run | |
| working-directory: symbiont-orga-demo | |
| # v7 #3 — proves no executor file imports `Command::new`. The | |
| # MCP STDIO RCE class lands here; this gate refuses it pre-merge. | |
| run: | | |
| cargo test --release --test no_shell_out -p demo-karpathy-loop -- --nocapture | |
| - name: Targeted — typestate compile-fail proofs (v9 §2) | |
| working-directory: symbiont-orga-demo | |
| # Belt-and-suspenders against a refactor that weakens | |
| # AgentLoop<Phase> from a typestate to a runtime check. Each | |
| # of the five files under tests/compile-fail/ asserts a | |
| # specific illegal phase transition is rejected by rustc with | |
| # the expected error code (E0599 / E0382 / E0603). | |
| run: | | |
| cargo test --release -p symbi-kloop-bench \ | |
| --test typestate_compile_fail -- --nocapture | |
| clippy: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| path: symbiont-orga-demo | |
| - uses: actions/checkout@v4 | |
| with: | |
| repository: ThirdKeyAI/symbiont | |
| path: symbiont | |
| ref: 57a8847c3f14d855086edaebb2d725babdc3e9d4 | |
| # v11+ — bridge crate path-deps on `../ToolClad/rust`. Same SHA | |
| # pin as the rust-tests job above. | |
| - uses: actions/checkout@v4 | |
| with: | |
| repository: ThirdKeyAI/ToolClad | |
| path: ToolClad | |
| # includes the scope_target + url egress hardening that | |
| # idn_confusable_probe asserts (block_internal, obfuscated-IP | |
| # rejection, per-label length, always-block cloud metadata). | |
| # NOTE: ToolClad PR #8 head — re-pin to the merge commit once merged. | |
| ref: 519d7002121246eca875d77ce38eacfd3512482a | |
| - uses: dtolnay/rust-toolchain@1.95.0 # pinned — see rust-toolchain.toml + rust-tests job | |
| with: | |
| components: clippy | |
| - name: Cache cargo registry + target | |
| uses: actions/cache@v4 | |
| with: | |
| path: | | |
| ~/.cargo/registry | |
| ~/.cargo/git | |
| symbiont-orga-demo/target | |
| key: clippy-${{ runner.os }}-${{ hashFiles('symbiont-orga-demo/Cargo.lock') }} | |
| - name: clippy on demo crates (upstream warnings excluded) | |
| working-directory: symbiont-orga-demo | |
| run: | | |
| cargo clippy --release --all-targets \ | |
| -p demo-karpathy-loop -p symbi-kloop-bench \ | |
| -- -D warnings |