feat: TOP-D algorithm (trust region policy distillation)

[hallerite]

Implements Trust Region Policy Distillation (TOP-D, arXiv:2607.04751) as a new algorithm type topd.

What the paper does

Standard OPD's per-token reward is the log-ratio log(π_teacher/π_student) on student-sampled tokens — unbounded below, so a token the teacher rejects sends the reward to −∞ and destabilizes training (worst when the capacity gap is large). TOP-D distills toward a proximal teacher, the probability-space interpolation α·π_teacher + (1−α)·π_student, which never needs materializing: the reward reduces algebraically to r̃ = log(α·ρ + 1−α), floored at log(1−α). Credit is token-level: return = immediate reward + mean of future rewards (length-normalized, Eq. 8), z-normalized across the group's tokens, optimized with an importance-ratio trust region against the behavior policy.

How it maps onto prime-rl

The paper's "internal trust region iterations" already exist here — the rl loss is importance-ratio corrected against sampling logprobs with trust-region masking, and async training decouples behavior from target. So unlike opd (teacher logprobs shipped to the trainer for the ref_kl loss), topd compiles entirely orchestrator-side into an ordinary per-token advantage stream:

• score_rollout — own-context prefill under the frozen teacher (identical to opd's path)
• score_group — bounded rewards via logaddexp (overflow-safe) → length-normalized future returns per sample → token-level group z-normalization → per-token advantage stream; ref_logprobs are consumed and cleared, never shipped
• action_loss_type = "rl" — advantage-based filters apply (a zero-spread group degrades to all-zero advantages, like a uniform GRPO group)

[orchestrator.algo]
type = "topd"          # alpha = 0.2 default (paper's vetted range: 0.1–0.2)

[orchestrator.algo.teacher]
name = "Qwen/Qwen3-30B-A3B-Instruct-2507"
base_url = ["http://teacher-host:8000/v1"]

Changes

• src/prime_rl/orchestrator/algo/topd.py — TOPDAlgorithm (new)
• packages/prime-rl-configs/.../algorithm.py — TOPDAlgoConfig in the discriminated union
• algo/__init__.py — registry + exports
• docs/algorithms.md, skills/configs/SKILL.md — kept in sync
• tests/unit/orchestrator/test_algorithms.py — vetted-defaults row, frozen-teacher validation generalized to opd/topd, and a numeric test asserting hand-computed advantages for a two-rollout group

Testing

uv run pytest tests/unit/orchestrator/test_algorithms.py — 28 passed. Not yet smoke-tested against a live teacher endpoint; the teacher I/O path is line-for-line opd's existing path.

🤖 Generated with Claude Code