fix(orchestrator): use raw reward for singleton MaxRL groups

[morluto]

Summary

MaxRL currently assigns zero advantage to every singleton rollout. Because
group_size defaults to 1, the default MaxRL configuration produces no
policy-gradient signal.

Use the raw reward for singleton groups so the order-1 estimator recovers
REINFORCE. The existing mean-normalized estimator remains unchanged for larger
groups.

Problem

For groups larger than one, MaxRL computes:

$$ A_i = \frac{r_i - \bar r}{\bar r} $$

Applying this control variate to a singleton success gives:

$$ A_1 = \frac{1 - 1}{1} = 0 $$

A singleton failure also receives zero. As a result, all singleton rollouts
carry zero policy-gradient weight and are removed by the default post-batch
zero-advantage filter.

The MaxRL paper defines $T=1$ as standard
RL. For a single rollout, that requires retaining its reward-weighted score
rather than centering it against itself.

Fix

• Use advantage = reward when a MaxRL group contains one rollout.
• Preserve the existing estimator for groups larger than one.
• Add regression coverage for singleton success and failure.
• Update the algorithm and configuration documentation.

This changes [1] -> [0] to [1] -> [1], while existing multi-rollout behavior
such as [1, 0, 0, 0] -> [3, -1, -1, -1] remains unchanged.

Verification

• Added test_max_rl_singleton_uses_reward_as_reinforce_advantage.
• Verified the production MaxRLAlgorithm.score_group path directly with
  CPU-only PyTorch for singleton success, singleton failure, mixed-reward
  groups, and all-success groups.
• ruff check passed.
• ruff format --check passed.
• Python compilation and git diff --check passed.

Related

Follow-up to #2778.

---

Note

Low Risk
Localized change to MaxRL credit assignment with regression tests; default MaxRL configs gain non-zero PG signal on successes where they previously had none.

Overview
MaxRL no longer assigns zero advantage to every group_size = 1 rollout. MaxRLAlgorithm.score_group now sets advantage = raw reward when the group has a single rollout, so the order-1 estimator matches REINFORCE instead of self-centering to zero.

For groups larger than one, behavior is unchanged: (reward − group mean) / group mean, with zero advantages when the group mean is ≤ 0.

Docs and MaxRLAlgoConfig describe the singleton exception (vs GRPO-style group_size = 1 zeroing). A unit test covers singleton success and failure.

^{Reviewed by Cursor Bugbot for commit 701998c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.