fix(slurm): tear down inference-only nodes after clean trainer exit

[eligotts]

Summary

srun --kill-on-bad-exit=1 only reaps peer tasks on non-zero exits, so in separate train/inference multi-node RL a trainer that finished cleanly left the inference-only nodes running vLLM until the allocation hit its time limit — the job stayed RUNNING with idle GPUs after RL trainer finished!.

This adds a run-completion sentinel ($OUTPUT_DIR/control/slurm_run_complete) for deployments with inference-only nodes:

• The trainer task wrapper touches the sentinel on exit 0 and propagates its exit code either way, so failure teardown via --kill-on-bad-exit is unchanged.
• Inference-only nodes run a watcher that exits cleanly once the sentinel appears; an inference component exiting without the sentinel is forced to exit 1 so premature vLLM/router death still tears the job down.
• The orchestrator-hosting task re-waits when the orchestrator exits cleanly before the trainer finishes — otherwise its wait -n would return early and kill the trainer wrapper before it could touch the sentinel.

Single-group deployments (num_infer_nodes == 0) render without any of this.

Validation

• Rendered the template with num_infer_nodes 0 and 2; sentinel machinery present only in the latter; bash -n passes on both rendered scripts.
• Pending: a live SLURM smoke in both directions — clean trainer exit tears down inference nodes, and trainer failure still propagates non-zero. Keeping as draft until that runs.

🤖 Generated with Claude Code