Race condition in child workflow cancellation

[shijiesheng]

Describe the bug
This is not a classic code non-determinism (e.g. a map iteration or a new activity added without GetVersion). The binary checksum is identical across all the failing decision tasks (uDeploy:3ee6dcdc8cb6e695ecd6ffea241e9a06e6dd4057), ruling that out.

The panic is triggered by a double-event race on a child workflow cancellation that the Cadence SDK's decision state machine cannot handle.

What the history shows

Here's the key sequence around the cancellation:

┌─────────┬─────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Event │ Type │ Detail │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 793 │ WorkflowExecutionCancelRequested │ Parent requests cancel │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 796 │ DecisionTaskCompleted │ Workflow reacts: cancels children │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 798 │ RequestCancelExternalWorkflowExecutionInitiated │ Cancel sent for d4207e73… (RolloutMonitor, initiatedEvt=387) │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 799 │ RequestCancelExternalWorkflowExecutionInitiated │ Cancel sent for f69b4074-..._46 (RolloutMonitorFinal, initiatedEvt=420) — childWorkflowOnly: true │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 801 │ ChildWorkflowExecutionCanceled │ f69b4074-..._46 successfully cancelled (initiatedEvt=420) — decision removed from state machine │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 804–806 │ DecisionTask… │ Workflow replays, processes event 801, decision is gone │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 807 │ RequestCancelExternalWorkflowExecutionFailed │ Cancel for f69b4074-..._46 (from event 799) also fails: UNKNOWN_EXTERNAL_WORKFLOW_EXECUTION │
├─────────┼─────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 811 │ DecisionTaskFailed │ PANIC: unknown decision DecisionType: ChildWorkflow, ID: f69b4074-..._46 │
└─────────┴─────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘

Why both events 801 and 807 fire for the same child

The RolloutMonitorFinal child (f69b4074-..._46) was already in the process of cancelling — or had just cancelled — independently of the parent's request at event 796. The Cadence server therefore generates two responses to the cancellation:

1. ChildWorkflowExecutionCanceled (event 801): the child's own cancellation arrived and was processed. The SDK's handleChildWorkflowExecutionCanceled looks up the ChildWorkflow decision for f69b4074-..._46 and removes it from the decisionsHelper state machine.
2. RequestCancelExternalWorkflowExecutionFailed (event 807): the explicit cancel request from event 799 returned UNKNOWN_EXTERNAL_WORKFLOW_EXECUTION — the child was already gone. The SDK's handleRequestCancelExternalWorkflowExecutionFailed tries to call getDecision(ChildWorkflow, f69b4074-..._46)… but the decision is already gone → panicIllegalState.

Why RolloutMonitorFinal specifically

The pattern for every other cluster tier in the history is:
StartChildWorkflow (RolloutClusterBatch) → ChildWorkflowCompleted
StartChildWorkflow (RolloutMonitor) → ChildWorkflowCompleted

The RolloutMonitorFinal (event 420) is a different child workflow type used only once, at the end. It was started and then the parent received signals (rollout_root_stop_pause_observing, rollout_root_restart_pause_observing) that may have caused the child to independently enter a cancel/complete path before the parent's own cancellation arrived at event 796. The RolloutMonitor child at event 387 hit the same double-event pattern (event 808 + event 812).

The fix

The panic is not in your workflow logic — it's the SDK reacting to a server-side race it doesn't defend against. Two options:

1. Prevent the double-cancel: Investigate whether RolloutMonitorFinal and the last RolloutMonitor can complete/cancel themselves via the signal path (rollout_root_stop_pause_observing) while the parent is also cancelling them. If the child signals its own termination, the parent's context cancel should be guarded — e.g. check whether the child future has already resolved before the parent cancellation fires.
2. SDK-level workaround: Wrap the child future in a select with a workflow.NewTimer or use a workflow.Selector that marks a local flag when the child completes, so by the time the parent cancel propagates, the child future is already resolved and the SDK never issues a new RequestCancelExternalWorkflow for it.

The childWorkflowOnly: true on events 798/799 confirms these are SDK-generated internal cancels (not explicit RequestCancelExternalWorkflow calls from your code), so the fix belongs in how the child's completion is tracked before the parent's cancel context fires.

To Reproduce
Is the issue reproducible?

• [Yes | No]

Steps to reproduce the behavior:
A clear and concise description of the reproduce steps.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here, E.g. Stackstace, workflow history.