fix(orchestrator): propagate dispatcher and watcher failures

[morluto]

Summary

The orchestrator runs its dispatcher and weight watcher as background tasks. If
either task raises, the exception remains stored on the task while the main loop
continues waiting for rollouts.

Inspect these component tasks from the main loop and propagate failures with the
component name and original exception preserved.

Failure mode

The weight watcher has a reachable uncaught failure path:

1. WeightWatcher.apply_policy_update() awaits
   InferencePool.update_weights().
2. If the weight update raises, WeightWatcher.start() terminates because it
   only handles cancellation.
3. The orchestrator never awaits or inspects the failed task.
4. Once the training lead exceeds TARGET_LAG, dispatch_allowed is cleared.
   Without another policy update, the gate cannot reopen.

If the dispatcher task raises, no loop remains to collect completed rollout
tasks and put them on out_q. The main loop instead continues retrying its
queue timeout.

Individual rollout-task failures are handled separately by converting them into
errored rollouts. That behavior is unchanged.

Fix

• Inspect the dispatcher and watcher tasks on each main-loop iteration.
• Raise a named RuntimeError when either task contains an exception.
• Preserve the component exception as the error cause.
• Skip pending and cancelled tasks.

When the rollout queue is idle, failures are detected within its existing
0.5-second polling interval.

Verification

• Added test_raise_for_failed_component_tasks_preserves_cause.
• The test models a watcher failure with a named asyncio task and verifies that
  its ConnectionError is preserved as the cause of the orchestrator error.
• tests/unit/orchestrator/test_orchestrator_setup.py: 3 passed.
• ruff check passed.
• ruff format --check passed.
• git diff --check passed.

---

Note

Medium Risk
Changes orchestrator liveness by failing fast when core pipeline tasks die, which is desirable but alters long-running run behavior on weight-update or dispatch errors.

Overview
Background dispatcher and weight watcher tasks could exit with an exception while the orchestrator main loop kept polling the rollout queue, leading to silent stalls (e.g. dispatch gate closed after a failed weight update with no recovery).

Each main-loop iteration now calls raise_for_failed_component_tasks on component_tasks, which raises a RuntimeError naming the failed task and chains the original exception as __cause__. Pending and cancelled tasks are ignored.

A unit test verifies the named error and preserved cause for a failed "watcher" task.

^{Reviewed by Cursor Bugbot for commit d2f58c8. Bugbot is set up for automated code reviews on this repo. Configure here.}