Skip to content

Background steps execution engine#4476

Open
lokesh755 wants to merge 1 commit into
mainfrom
lokesh755-background-steps-engine
Open

Background steps execution engine#4476
lokesh755 wants to merge 1 commit into
mainfrom
lokesh755-background-steps-engine

Conversation

@lokesh755
Copy link
Copy Markdown
Contributor

@lokesh755 lokesh755 commented Jun 4, 2026

Summary

Adds the core background step execution engine to the runner, enabling concurrent step execution within a single job. Background steps (background: true) run asynchronously while the foreground step loop continues. Control-flow steps (wait, wait-all, cancel) coordinate synchronization and lifecycle management.

Changes

New files

  • BackgroundStepCoordinator.cs — Coordinates background step execution, waiting, cancellation, and deferred state flushing. Manages a concurrency semaphore (default max 10).
  • ControlFlowStepData.cs — Data class for wait/wait-all/cancel steps.
  • BackgroundStepsL0.cs — Unit tests for background step coordination, waiting, cancellation, and failure propagation.

Modified files

  • StepsRunner.cs — Integrates BackgroundStepCoordinator; background steps are queued without blocking the main loop; control-flow steps are dispatched directly; implicit wait-all runs before post-hooks as a safety net.
  • JobExtension.cs — Initializes control-flow steps from BackgroundStepControl pipeline type; maps logical step IDs to external IDs for UI reference; creates implicit wait-all for uncovered background steps.
  • ExecutionContext.cs — Adds background step metadata as optional parameters to CreateChild

Key design decisions

  • Runner owns concurrency — run-service remains agnostic to step execution order
  • Non-blocking semaphore — foreground steps execute immediately even when all background slots are occupied
  • Implicit wait-all — safety net before post-hooks waits for any uncovered background steps

@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch from 955429e to 13594a5 Compare June 4, 2026 03:11
@lokesh755 lokesh755 force-pushed the lokesh755-thread-safety-primitives-background-stepps branch from becd2c1 to b88b481 Compare June 4, 2026 03:25
@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch 2 times, most recently from 4fbc66b to 95ab648 Compare June 4, 2026 15:26
@lokesh755 lokesh755 marked this pull request as ready for review June 4, 2026 15:26
@lokesh755 lokesh755 requested a review from a team as a code owner June 4, 2026 15:26
Comment thread src/Runner.Worker/BackgroundStepCoordinator.cs Outdated
@lokesh755 lokesh755 force-pushed the lokesh755-thread-safety-primitives-background-stepps branch from b88b481 to 5007995 Compare June 4, 2026 17:52
Base automatically changed from lokesh755-thread-safety-primitives-background-stepps to main June 4, 2026 18:08
Copilot AI review requested due to automatic review settings June 4, 2026 18:27
@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch from 95ab648 to 5a09827 Compare June 4, 2026 18:27
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch from 5a09827 to 265b98d Compare June 4, 2026 18:28
Comment thread src/Runner.Worker/JobRunner.cs
Comment thread src/Runner.Worker/JobExtension.cs Outdated
Comment thread src/Runner.Worker/JobExtension.cs Outdated
@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch from 265b98d to c89cec5 Compare June 4, 2026 18:37
Comment thread src/Runner.Worker/JobExtension.cs Outdated
Comment thread src/Runner.Worker/JobExtension.cs Outdated
Comment thread src/Runner.Worker/BackgroundStepContext.cs Outdated
Comment thread src/Runner.Worker/BackgroundStepContext.cs Outdated
Comment thread src/Runner.Worker/ControlFlowStepData.cs Outdated
Comment thread src/Runner.Worker/FileCommandManager.cs
Comment thread src/Runner.Worker/ExecutionContext.cs Outdated
@lokesh755 lokesh755 changed the title Background steps engine Background steps execution engine Jun 4, 2026
Comment thread src/Runner.Worker/ControlFlowStepData.cs
Comment thread src/Runner.Worker/BackgroundStepCoordinator.cs Outdated
@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch 2 times, most recently from 50a2671 to 369bbe9 Compare June 4, 2026 22:14
@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch 3 times, most recently from 8a90a7d to 80ebb21 Compare June 5, 2026 01:05
@lokesh755 lokesh755 force-pushed the lokesh755-background-steps-engine branch from 80ebb21 to b30b777 Compare June 5, 2026 01:11

public sealed class StepsRunner : RunnerService, IStepsRunner
{
private IBackgroundStepCoordinator _bgCoordinator;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need this anymore, can you have just do var _bgCoordinator = HostContext.GetService<IBackgroundStepCoordinator>();

Comment on lines +47 to +50
_bgCoordinator = HostContext.GetService<IBackgroundStepCoordinator>();
var maxBgSteps = jobContext.Global.Variables.GetInt("system.runner.maxbackgroundsteps");
var maxConcurrent = (maxBgSteps.HasValue && maxBgSteps.Value > 0) ? maxBgSteps.Value : 10;
_bgCoordinator.InitializeCoordinator(maxConcurrent);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would suggest we move these initialization into jobextension after we realize we have any background step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants