Skip to content

Commit 7db4e85

Browse files
frawarefraware
committed
Add PCS rendering benchmarks and formal check release comparison.
1 parent 8e6f1b6 commit 7db4e85

134 files changed

Lines changed: 8449 additions & 713 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ Thumbs.db
8686
portal/public/generated/
8787
*.local.*
8888
benchmarks/reports/
89+
benchmark_runs/
8990
docs/verso/_build/
9091
tmp/
9192

‎JUSTFILE‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,13 @@ benchmark:
116116
benchmark-smoke:
117117
uv run --project pipeline python -m sm_pipeline.cli benchmark
118118

119+
# PCS rendering/query benchmark (Scientific Memory evidence layer; report for pcs-bench)
120+
pcs-benchmark-rendering CASES="benchmarks/rendering/labtrust_qc_release" OUT="benchmark_runs/labtrust_rendering":
121+
uv run --project pipeline python -m sm_pipeline.benchmark.rendering --cases {{CASES}} --out {{OUT}}
122+
123+
pcs-benchmark-rendering-all OUT="benchmark_runs/pcs_rendering":
124+
uv run --project pipeline python -m sm_pipeline.benchmark.rendering --cases benchmarks/rendering --out {{OUT}}
125+
119126
# Recipe dependencies avoid nested `just` subprocesses (fixes PATH on Windows).
120127
check: fmt lint validate test build
121128
@echo "==> check complete"

‎benchmarks/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ The run writes `benchmarks/reports/latest.json` and compares results against
1212

1313
## Folder guide
1414

15+
- [`rendering/`](rendering/README.md): PCS import/render/query benchmarks for the
16+
Scientific Memory evidence layer (`just pcs-benchmark-rendering`)
1517
- [`tasks/`](tasks/README.md): deterministic benchmark scorers
1618
- [`gold/`](gold/README.md): human-reviewed gold labels per paper
1719
- [`llm_eval/`](llm_eval/README.md): reviewed LLM reference bundles used as

‎benchmarks/rendering/README.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# PCS rendering benchmarks
2+
3+
Scientific Memory evidence-layer benchmarks for PCS release import, portal read-model
4+
rendering, query correctness, release comparison, and failed-release interpretability.
5+
6+
## Layout
7+
8+
Each case directory contains:
9+
10+
- `case.json` — case metadata and fixture pointer
11+
- `release_manifest.v0.json` — manifest alias (artifacts resolve via `fixture_dir`)
12+
- `expected_sections.json` — required interpretability sections
13+
- `expected_queries.json` — CLI query expectations
14+
- `expected_lineage.json` — lineage fields after import
15+
- `expected_staleness.json` — staleness expectations
16+
- `expected_compare.json` — release comparison expectations (when applicable)
17+
- `expected_failure.json` — failure-evidence checks (failed cases only)
18+
19+
### Success-path cases
20+
21+
- `labtrust_qc_release/`
22+
- `tool_use_safety/`
23+
- `computation_reproducibility/`
24+
25+
### Failed-release cases
26+
27+
Under `failed/`:
28+
29+
- `rejected_certificate/` — computation witness rejected
30+
- `stale_release/` — stale lineage after bundle drift
31+
- `failed_lean_check/` — formal kernel failed obligation (read-model evidence)
32+
- `failed_pf_verification/` — PF explain on failed formal check
33+
- `missing_registry_metadata/` — deferred registry checks in read model
34+
35+
## Run
36+
37+
```bash
38+
just pcs-benchmark-rendering CASES=benchmarks/rendering/labtrust_qc_release OUT=benchmark_runs/labtrust_rendering
39+
40+
just pcs-benchmark-rendering-all OUT=benchmark_runs/pcs_rendering
41+
42+
python -m sm_pipeline.benchmark.rendering --cases benchmarks/rendering --out benchmark_runs/pcs_rendering
43+
```
44+
45+
Reports are written to:
46+
47+
- `rendering_benchmark_report.json` — full case results (`PcsRenderingBenchmarkReport.v0`)
48+
- `pcs_bench_payload.json` — flattened metrics for **pcs-bench**
49+
- `rendering_benchmark_summary.md` — human-readable summary
50+
51+
Regression floors: `baseline_thresholds.json` (enforced by default via `--check-regression`).
52+
53+
## Regenerate expectations
54+
55+
After fixture or import changes:
56+
57+
```bash
58+
python scripts/bootstrap_pcs_rendering_benchmarks.py
59+
```

‎benchmarks/rendering/baseline_thresholds.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"description": "Minimum aggregate metrics for PCS rendering benchmarks (Scientific Memory evidence layer).",
3+
"metrics": {
4+
"required_sections_rendered": 1.0,
5+
"registry_metadata_rendered": 1.0,
6+
"query_responses_correct": 1.0,
7+
"release_comparison_correct": 1.0,
8+
"failure_evidence_rendered": 1.0,
9+
"lineage_records_created": 1.0
10+
}
11+
}

‎benchmarks/rendering/computation_reproducibility/case.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"case_id": "computation_reproducibility",
3+
"fixture_dir": "tests/pcs/fixtures/computation-release",
4+
"manifest_filename": "release_manifest.v0.json",
5+
"claim_id": "claim-computation-release-v0.1",
6+
"release_id": "release-pcs-v0.1-scientific-computation-reproducibility",
7+
"workflow_id": "scientific_computation.reproducibility_v0"
8+
}

‎benchmarks/rendering/computation_reproducibility/expected_lineage.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"claim_id": "claim-computation-release-v0.1",
3+
"release_id": "release-pcs-v0.1-scientific-computation-reproducibility",
4+
"has_signed_bundle_hash": true,
5+
"certificate_id": "witness-computation-demo-v0.1"
6+
}

‎benchmarks/rendering/computation_reproducibility/expected_queries.json‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
{
2+
"queries": [
3+
{
4+
"id": "list_claims",
5+
"type": "list_claims",
6+
"params": {},
7+
"expected_claim_ids": [
8+
"claim-computation-release-v0.1"
9+
]
10+
},
11+
{
12+
"id": "by_release",
13+
"type": "by_release",
14+
"params": {
15+
"release_id": "release-pcs-v0.1-scientific-computation-reproducibility"
16+
},
17+
"expected_claim_ids": [
18+
"claim-computation-release-v0.1"
19+
]
20+
},
21+
{
22+
"id": "by_workflow",
23+
"type": "by_workflow",
24+
"params": {
25+
"workflow_id": "scientific_computation.reproducibility_v0"
26+
},
27+
"expected_claim_ids": [
28+
"claim-computation-release-v0.1"
29+
]
30+
},
31+
{
32+
"id": "by_certificate",
33+
"type": "by_certificate",
34+
"params": {
35+
"certificate_id": "witness-computation-demo-v0.1"
36+
},
37+
"expected_claim_ids": [
38+
"claim-computation-release-v0.1"
39+
]
40+
},
41+
{
42+
"id": "by_source_commit",
43+
"type": "by_source_commit",
44+
"params": {
45+
"commit": "4c5439ae358733f9a4c4a58e33fdaed1ab0d29de"
46+
},
47+
"expected_claim_ids": [
48+
"claim-computation-release-v0.1"
49+
]
50+
},
51+
{
52+
"id": "by_dataset",
53+
"type": "by_dataset",
54+
"params": {
55+
"dataset_id": "dataset-demo-measurements-v0.1"
56+
},
57+
"expected_claim_ids": [
58+
"claim-computation-release-v0.1"
59+
]
60+
},
61+
{
62+
"id": "by_result_hash",
63+
"type": "by_result_hash",
64+
"params": {
65+
"hash": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"
66+
},
67+
"expected_claim_ids": [
68+
"claim-computation-release-v0.1"
69+
]
70+
},
71+
{
72+
"id": "by_lean_theorem",
73+
"type": "by_lean_theorem",
74+
"params": {
75+
"theorem": "PCS.SignedBundleAdmissible"
76+
},
77+
"expected_claim_ids": [
78+
"claim-computation-release-v0.1"
79+
]
80+
}
81+
]
82+
}

‎benchmarks/rendering/computation_reproducibility/expected_sections.json‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
{
2+
"required_sections": [
3+
"Claim",
4+
"Workflow Profile",
5+
"Assumptions",
6+
"Runtime Evidence",
7+
"Certificate or Witness",
8+
"Verification Result",
9+
"Formal Trust Kernel",
10+
"Release Manifest",
11+
"Release Chain Validation",
12+
"Artifact Registry",
13+
"Handoff Manifests",
14+
"Artifact Dependency Graph",
15+
"Lineage",
16+
"Staleness",
17+
"Artifact Hashes",
18+
"Source Repositories",
19+
"Reproduce / Verify",
20+
"Limitations"
21+
],
22+
"present_sections": [
23+
"Claim",
24+
"Workflow Profile",
25+
"Assumptions",
26+
"Runtime Evidence",
27+
"Certificate or Witness",
28+
"Verification Result",
29+
"Formal Trust Kernel",
30+
"Release Manifest",
31+
"Release Chain Validation",
32+
"Artifact Registry",
33+
"Handoff Manifests",
34+
"Artifact Dependency Graph",
35+
"Lineage",
36+
"Staleness",
37+
"Artifact Hashes",
38+
"Source Repositories",
39+
"Reproduce / Verify",
40+
"Limitations"
41+
]
42+
}

‎benchmarks/rendering/computation_reproducibility/expected_staleness.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"stale": false
3+
}

0 commit comments

Comments
 (0)