[Scheduler] Add a new flag to allow only migration of schedules without running workflows

[lina-temporal]

What changed?

• Added a new dynamic config flag to control whether V1 schedules will migrate with running workflows.

Why?

• When V1 schedules migrate to V2 with running workflows, V2 attaches a completion callback to the running workflow (as V2 exclusively uses Nexus completion callbacks to monitor workflow status). Certain third-party SDKs are known to have issues with one of the events written to the history of the running workflow, which can cause them to panic (Coinbase SDK). This allows us to dial up migration for these customers without triggering that edge case.

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

Potential risks

• Slower migration cadence, potentially

[chaptersix]

if !s.State.PendingMigration && s.tweakables.EnableCHASMMigration &&
			(s.tweakables.MigrateWithRunningWorkflows || len(s.Info.RunningWorkflows) { 
			s.State.PendingMigration = true
		}
if s.State.PendingMigration {
err := s.executeMigration()
...
}

what do you thin about this?

[davidporter-id-au]

What about plumbing all these conditionals into the activity, so that we don't have to worry about nondeterminitism?

[chaptersix]

Changes to all of these variables are deterministic because we read them through MutableSideEffect (tweakables), or Signal into the workflow.

this suggested change is mostly to prevent changes in behavior given the same input and simplify it (at least in my head)

[chaptersix]

it would also still support migrations via signal.

[davidporter-id-au]

My thought about doing the eval in the activity is that it allows for instantaneous update, rather than capturing a side-effect in history earlier, which then must be honoured, irrespective of dynamic config's current state. Ie, it'd allow for a faster response time if DC changed value.

But re the logic, I'm not sure I understand it, to Alex's point, I'm not sure why EnableCHASMMigration is being evaluated on line 337 and 331?

[chaptersix]

why EnableCHASMMigration is being evaluated on line 337 and 331?
it would allow the migration flag going from true to false to to halt the migration of a schedule that is already in progress.

I'm happy with the changes @lina-temporal just pushed.

re using an activity: I'd rather read through a mutable side effect because it's an existing pattern in this workflow and we've test it quite extensively already.