feat(experiment): paper-ready reproducibility archive (experiment bundle)#73
Open
NicolasSchuler wants to merge 5 commits into
Open
feat(experiment): paper-ready reproducibility archive (experiment bundle)#73NicolasSchuler wants to merge 5 commits into
experiment bundle)#73NicolasSchuler wants to merge 5 commits into
Conversation
Promote `append_path_to_tar` and `hash_file` to `pub(crate)`, extract the gzip tar builder open/finish into `create_tar_builder`/`finish_tar_builder`, and add `write_tree_tarball` for archiving an entire directory tree with deterministic sorted entries. `write_bundle_tarball` now reuses the shared helpers instead of inlining the encoder plumbing. These back the upcoming `experiment bundle` command without duplicating the artifact-export format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`hpc-compose experiment bundle [JOB_ID]` emits a citeable, paper-ready archive for one tracked run: the compose spec, the resolved config snapshot (secret-redacted), the rendered sbatch (re-rendered from the plan and marked "reconstructed" when the on-disk script is gone), the full submission record, provenance (git SHA + dirty + per-service image references), the sweep manifest with seeds for sweep trials, metrics (stats.csv + raw *.jsonl), checkpoint attempt history, and a generated README.md methods appendix, alongside a MANIFEST.json with per-file sha256 and a missing[] ledger. A spec/spec-drift.diff is included only when the current spec differs from the snapshot. Image entries are references as recorded at submit time, not content digests, and are never resolved against a registry (stated plainly in the appendix). Output defaults to experiment-bundle-<job_id>.tar.gz; --dir writes an unpacked directory, --strict fails (after reporting) when any ingredient is missing, and --format json is pinned by the new `experiment-bundle` output schema. Defaults to the latest tracked run and contacts the scheduler only as much as `stats` does. Bundle is a subcommand, so MAX_TOP_LEVEL_COMMANDS is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exercises the real binary end to end (extracting the tarball with `tar`): happy path with a full record (asserts every staged file, MANIFEST sha256 ledger, empty missing[], README caveat/identity, CSV header); degrade path with no snapshot or metrics (exit 0, warnings, missing[] entries, README still generated); --strict failure naming the missing ingredient; a sweep-trial bundle carrying sweep/manifest.json and the trial seed in the README; --format json (schema_version present, files array matches disk); --dir layout (no tarball written); and a drift case (spec-drift.diff present, drift_detected true, README flags drift). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a CHANGELOG [Unreleased] entry, a "Bundle a run for reproducibility" section in runtime-observability.md with the full layout table and the references-not-digests caveat, and a cli-reference.md row (plus an updated `experiment` parent-command summary). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a new hpc-compose experiment bundle subcommand that emits a “paper-ready” reproducibility bundle (tar.gz or directory) for a tracked run, including spec + snapshot, sbatch script (with reconstruction fallback), provenance, sweep seed join, metrics, checkpoint history, and a generated README/manifest, plus a pinned JSON output schema.
Changes:
- Implement
experiment bundlewith best-effort ingredient collection,missing[]degradation, optional--strict, and text/JSON output. - Promote/reuse tar + hashing helpers and register a new
experiment-bundleoutput schema contract. - Add end-to-end CLI tests plus documentation/manpage/changelog updates covering the new command and bundle layout.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/cli_runtime.rs | Adds E2E coverage for bundle creation (tar + dir), strict/degraded modes, sweep join, JSON output, and drift detection. |
| src/output/contract.rs | Registers experiment-bundle in the output schema contract and test allowlist. |
| src/job/mod.rs | Re-exports crate-internal tar/hash helpers for reuse by bundling logic. |
| src/job/artifacts.rs | Promotes hash_file, factors tar builder helpers, and adds write_tree_tarball for bundling. |
| src/commands/runtime/experiment.rs | Implements experiment bundle end-to-end: staging, ingredient capture, README + MANIFEST generation, tar/directory finalize, strict failure behavior. |
| src/commands/mod.rs | Wires the new CLI subcommand to the runtime implementation. |
| src/cli/help.rs | Adds help text for experiment bundle. |
| src/cli/commands.rs | Defines the experiment bundle CLI surface (flags, docs, help wiring). |
| schema/outputs/experiment-bundle.schema.json | Adds the pinned JSON schema for --format json output. |
| man/man1/hpc-compose-experiment.1 | Adds bundle to the experiment subcommand list. |
| man/man1/hpc-compose-experiment-tag.1 | Updates SEE ALSO to include the new bundle command. |
| man/man1/hpc-compose-experiment-show.1 | Updates SEE ALSO to include the new bundle command. |
| man/man1/hpc-compose-experiment-note.1 | Updates SEE ALSO to include the new bundle command. |
| man/man1/hpc-compose-experiment-bundle.1 | New manpage for hpc-compose experiment bundle. |
| docs/src/runtime-observability.md | Documents bundle use cases, layout, degradation policy, and semantics. |
| docs/src/cli-reference.md | Adds experiment bundle to the CLI reference table and description. |
| CHANGELOG.md | Notes the new command, defaults, bundle contents, and strict/degradation behavior. |
Comment on lines
+850
to
+858
| fn stage_write(staged_dir: &Path, relative: &str, bytes: &[u8]) -> Result<()> { | ||
| let path = staged_dir.join(relative); | ||
| if let Some(parent) = path.parent() { | ||
| fs::create_dir_all(parent) | ||
| .with_context(|| format!("failed to create {}", parent.display()))?; | ||
| } | ||
| crate::secure_io::write_atomic(&path, bytes, false) | ||
| .with_context(|| format!("failed to write {}", path.display())) | ||
| } |
Comment on lines
+506
to
+513
| fn create_tar_builder(tarball_path: &Path) -> Result<Builder<GzEncoder<File>>> { | ||
| if let Some(parent) = tarball_path.parent() { | ||
| fs::create_dir_all(parent).context(format!("failed to create {}", parent.display()))?; | ||
| } | ||
| let file = File::create(tarball_path) | ||
| .context(format!("failed to create {}", tarball_path.display()))?; | ||
| let encoder = GzEncoder::new(file, Compression::default()); | ||
| let mut builder = Builder::new(encoder); | ||
| for relative in copied_relative_paths { | ||
| let source = bundle_root.join(relative); | ||
| if !source.exists() && fs::symlink_metadata(&source).is_err() { | ||
| continue; | ||
| } | ||
| append_path_to_tar(&mut builder, &source, relative)?; | ||
| } | ||
| Ok(Builder::new(encoder)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap item 1.3: one command emits a citeable reproducibility tarball for a tracked run.
Approved sketch (implemented as designed unless noted)
Bundle layout (staged dir, then deterministic tar.gz;
--dirkeeps the directory instead):README methods appendix: run identity, submit timestamp, tool version, git SHA + dirty + branch, image references table, resources from the config snapshot, sweep id/trial/variables/seed, attempt history, metrics summary, tags/notes, artifact inventory, and a "Reproducing this run" section.
Degradation policy: every missing ingredient (no snapshot on run-style/legacy records, no metrics, script gone, no provenance, …) warns and lands in
MANIFEST.json missing[]; the command still succeeds.--strictfails (after finalizing the archive) when anything is missing.Design decisions (approved by Nicolas):
sweep_id.diff --against-specmachinery (including the sweep-trial interpolation overlay) and is best-effort: an unloadable current spec becomes amissing[]entry, not a failure.Implementation notes
append_path_to_tar,hash_file) promoted topub(crate)insrc/job/artifacts.rs+ newwrite_tree_tarball; deterministic sorted entries.ExperimentBundleOutputDTO follows the experiment-family precedent (hand-rolledschema_version, notflatten_envelope!); registered as"experiment-bundle";schema/outputs/experiment-bundle.schema.jsonblessed and committed.write_stats_snapshot_csvwas already generic overimpl Write— rendered into a buffer, no refactor needed.ArtifactManifestcarries no per-file sha256, so the README artifact inventory lists names/paths only.filesledger includesMANIFEST.jsonitself; the on-diskMANIFEST.jsonfiles[]necessarily excludes itself (cannot hash its own final bytes).Tests
Seven e2e tests (happy path with tar extraction, degraded,
--strict, sweep seed join,--format json,--dir, drift) plus the schema gates. Full suite green on the branch (1445 tests, 22 targets); fmt/clippy/typos clean; manpages regenerated by the integrator and--checkverified.Per the fast-mode agreement: no pre-PR adversarial review; batch review at the end of the push.
🤖 Generated with Claude Code