Skip to content

docs(deployments): explain Docker vs Kubernetes runner internals and trade-offs#20054

Open
joeduffy wants to merge 2 commits into
masterfrom
fix_issue_13218_customer_managed_agents
Open

docs(deployments): explain Docker vs Kubernetes runner internals and trade-offs#20054
joeduffy wants to merge 2 commits into
masterfrom
fix_issue_13218_customer_managed_agents

Conversation

@joeduffy

@joeduffy joeduffy commented Jul 2, 2026

Copy link
Copy Markdown
Member

This adds the missing internals and trade-off documentation requested in #13218: a customer had a lot of questions about how the Docker (DinD) and Kubernetes customer-managed workflow runner configurations differ under the hood, and the docs didn't explain either one's architecture or the trade-offs between them.

The new section, "How Docker and Kubernetes deploy targets execute jobs," lives on the customer-managed runners concepts page right after the feature overview and before the credentials/configuration reference, since it explains the architecture that the configuration options below it control. It covers:

  • What actually happens when a runner claims a job under deploy_target: docker — a long-running process talks to a Docker daemon over the local Docker socket and launches a dedicated workflow-runner-embeddable container per job — and why running the runner process itself inside a container typically requires either a mounted host Docker socket or a nested Docker-in-Docker (DinD) daemon.
  • What happens under deploy_target: kubernetes — the runner creates a Pod per job through the in-cluster Kubernetes API, needing only standard Pod-creation RBAC, with no Docker socket or privileged containers involved.
  • A comparison table across isolation model, privileged access, best fit, and ephemeral-runner support.
  • A cross-link to the existing "Scaling and concurrency" section for the 1:1 runner-to-job relationship, rather than duplicating that content.

I verified the technical claims against the existing configuration reference on the same page (the deploy_target, working_directory, and env_forward_allowlist/DOCKER_HOST comments already describe the Docker-socket and runner-container mechanics) and against the "Kubernetes-native Option for Pulumi Deployments Customer-Managed Agents" blog post, which states the Kubernetes option doesn't require Docker-in-Docker or privileged execution.


🧠 This PR was created by workprentice on behalf of @joeduffy.

…trade-offs

Adds a new section to the customer-managed runners concepts page that
explains what actually happens when a workflow runner claims a job under
each deploy_target: the Docker target launches a runner container over
the local Docker socket (and needs a mounted socket or Docker-in-Docker
if the runner process itself is containerized), while the Kubernetes
target creates a Pod through the in-cluster API and needs no privileged
access. Adds a compact comparison table covering isolation, privileged
access, best fit, and ephemeral-runner support, and cross-links to the
existing Scaling and concurrency section for the 1:1 runner-to-job
relationship.

Fixes #13218
@github-actions github-actions Bot added review:triaging Claude Triage is currently classifying the PR domain:docs PR touches technical docs review:in-progress Claude review is currently running and removed review:triaging Claude Triage is currently classifying the PR labels Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Pre-merge Review — Last updated 2026-07-02T15:17:53Z

Tip

Summary: This PR adds a new "How Docker and Kubernetes deploy targets execute jobs" section to the customer-managed runners concept page (content/docs/deployments/concepts/customer-managed-runners.md), explaining how the deploy_target: docker and deploy_target: kubernetes settings turn a claimed job into a running process, their isolation and privilege trade-offs, and when to choose each — capped with a comparison table. The one blocking concern from the initial review — the section overstated the runner-to-job concurrency model as strict single-job exclusivity, when the linked Scaling and concurrency guide allows a parallel Insights scan or policy evaluation — is fixed in commit 9000356, which scopes the claim to deployments and names the exception; the flagged weasel word ("usually") was also removed. Investigative passes: re-verification of the revised L60/L69 claims against the same sources, a fresh Vale run (0 findings), and a repo-wide sweep confirming no other occurrence of the old over-claim phrasing remains.

Review confidence:

Dimension Level Notes
mechanics HIGH
facts HIGH 17 of 18 claims verified, 1 not-a-claim, 0 contradicted, 0 unverifiable.
cross-sibling consistency MEDIUM Read the 2 topically-overlapping peers (settings.md, pulumi-managed-runners.md) + the same-file config reference; no contradictions. The other 6 concept peers cover unrelated topics.
Investigation log
  • Cross-sibling reads: 2 of 8 siblings (the topically-overlapping peers; no contradictions found)
  • External claim verification: 17 of 18 claims verified (0 unverifiable, 0 contradicted) · 4 specialists (numerical, cross-reference, capability, framing); 0 cross-specialist corroborations · routed: 0 inline, 16 Pass 1, 0 Pass 2, 2 Pass 3 (verified 2, contradicted 0, unverifiable 0).
  • Cited-claim spot-checks: not run (no cited claims)
  • Frontmatter sweep: ran on body + meta_desc
  • Temporal-trigger sweep: ran (recency words present in diff; spot-check complete, no issues)
  • Code execution: not run (no static/programs/ change)
  • Code-examples checks: not run (no fenced code blocks in content files)
  • Editorial-balance pass: not run (not under content/blog/)
🚨 Outstanding ⚠️ Low-confidence 💡 Pre-existing ✅ Resolved
0 0 0 1

🔍 Verification trail

18 claims extracted · 17 verified · 0 unverifiable · 0 contradicted
  • L37 in content/docs/deployments/concepts/customer-managed-runners.md "Customer-Managed Workflow Runners are available on the Business Critical edition of Pulumi Cloud." → ✅ verified (evidence: The doc itself states in an info callout: "Customer-Managed Workflow Runners are available on the Business Critical edition of Pulumi Cloud. Contact sales if you are interested and want to enable Customer-Managed Workflow Runners."; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L40 in content/docs/deployments/concepts/customer-managed-runners.md "The Customer-Managed Workflow Runners guide at /docs/deployments/guides/customer-managed-workflow-runners/ covers how to set up, scale, and assign a customer-m…" → ✅ verified (evidence: The guide at content/docs/deployments/guides/customer-managed-workflow-runners.md exists with sections "Using customer-managed workflow runners" (setup steps), "### Scaling and concurrency" (scaling), and "### Setting an organization defau…; source: repo:content/docs/deployments/guides/customer-managed-workflow-runners.md)
  • L44 in content/docs/deployments/concepts/customer-managed-runners.md "The deploy_target setting controls how a workflow runner turns a claimed job into a running process." → 🤝 matches (evidence: The same document's configuration reference confirms: "Where workflow jobs are executed. One of: docker, kubernetes. - docker: the agent launches runner containers via the local Docker socket. - kubernetes: the agent launches runner Pods v…; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L48 in content/docs/deployments/concepts/customer-managed-runners.md "When the runner claims a job in the Docker deploy target, it launches a dedicated runner container from the workflow-runner-embeddable image to execute that…" → ✅ verified (evidence: The same document is internally consistent: line 48 states the Docker deploy target "launches a dedicated runner container from the workflow-runner-embeddable image," and the configuration reference later in the file confirms it; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L48 in content/docs/deployments/concepts/customer-managed-runners.md "With deploy_target: docker, the workflow runner is a long-running process, typically installed on a virtual machine or bare-metal host, that talks to a Docke…" → ➖ not-a-claim (evidence: This is the docs author's own description of the Pulumi Deployments customer-managed runner architecture, consistent with the sibling guide page; source: repo:content/docs/deployments/guides/customer-managed-workflow-runners.md)
  • L50 in content/docs/deployments/concepts/customer-managed-runners.md "When the workflow runner is installed directly on a host that already runs Docker, the runner uses that host's Docker daemon directly and no additional privile…" → ✅ verified (evidence: The claim quotes the doc almost verbatim; source: repo:content/docs/deployments/concepts/customer-managed-runners.md (lines 48-65))
  • L50 in content/docs/deployments/concepts/customer-managed-runners.md "Both docker-outside-of-docker and Docker-in-Docker require the runner's container to run with elevated privileges relative to a typical application container." → ✅ verified (evidence: The doc itself states this verbatim; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L50 in content/docs/deployments/concepts/customer-managed-runners.md "Mounting the host's Docker socket into the runner's container (docker-outside-of-docker) and giving the runner container its own nested Docker daemon (Docker-i…" → ✅ verified (evidence: Source doc states verbatim: "The common patterns are mounting the host's Docker socket into the runner's container (docker-outside-of-docker) or giving the runner container its own nested Docker daemon (Docker-in-Docker, or DinD)."; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L50 in content/docs/deployments/concepts/customer-managed-runners.md "Mounting the host's Docker socket into the runner's container (docker-outside-of-docker) is a common pattern for enabling the runner container to create siblin…" → ✅ verified (evidence: The doc itself states this exact claim in context; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L54 in content/docs/deployments/concepts/customer-managed-runners.md "With deploy_target: kubernetes, the workflow runner runs as a Kubernetes Deployment in the customer's cluster, using the in-cluster Kubernetes API rather tha…" → ✅ verified (evidence: The claim is a verbatim restatement of the doc's own text; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L54 in content/docs/deployments/concepts/customer-managed-runners.md "With deploy_target: kubernetes, when the runner claims a job, it creates a Pod, scoped to run that single job's Pulumi program, through the Kubernetes API." → ✅ verified (evidence: The claim describes Pulumi's own product behavior; internally consistent with the sibling guide; source: repo:content/docs/deployments/concepts/customer-managed-runners.md and repo:content/docs/deployments/guides/customer-managed-workflow-runners.md)
  • L56 in content/docs/deployments/concepts/customer-managed-runners.md "The Kubernetes deploy target composes naturally with the single_run: true setting and Job/CronJob patterns described in the Scaling and concurrency section o…" → ✅ verified (evidence: The source doc's Kubernetes deploy target section states this verbatim; source: repo:content/docs/deployments/concepts/customer-managed-runners.md and repo:content/docs/deployments/guides/customer-managed-workflow-runners.md)
  • L60 in content/docs/deployments/concepts/customer-managed-runners.md "Both deploy targets share the same concurrency model: a runner process runs one deployment at a time — optionally alongside a single Insights scan or policy e…" → ✅ verified (resolved in 9000356; evidence: matches the "Scaling and concurrency" guide verbatim: "Each workflow runner process runs one deployment at a time, plus optionally one Insights scan or policy evaluation in parallel, and has no internal worker pool to configure."; source: repo:content/docs/deployments/guides/customer-managed-workflow-runners.md)
  • L62-67 in content/docs/deployments/concepts/customer-managed-runners.md "With the Docker deploy target, runner containers share the host's Docker daemon, and running the runner itself in a container typically requires a mounted Dock…" → 🤝 matches (evidence: The table row (L64) and prose (L50) in the same doc both state this; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L64 in content/docs/deployments/concepts/customer-managed-runners.md "In the Docker deploy target, runner containers share the host's Docker daemon, and running the runner itself in a container typically requires a mounted Docker…" → 🤝 matches (evidence: Table cell at L64 restates prose already established earlier in the same document; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L66 in content/docs/deployments/concepts/customer-managed-runners.md "The Kubernetes deploy target is best fit for environments where Kubernetes is already run and workflows should use existing cluster capacity, autoscaling, and…" → ✅ verified (framing: strengthened — the claim's specific rationale (existing cluster capacity, autoscaling, RBAC) is a plausible editorial elaboration of the source's broader statement; evidence: Pulumi docs confirm the Kubernetes deploy target launches runner Pods via the in-cluster API and is documented as an option for Kubernetes-native installations; source: https://www.pulumi.com/docs/deployments/concepts/customer-managed-runners/)
  • L67 in content/docs/deployments/concepts/customer-managed-runners.md "Ephemeral one-shot runners are achievable in the Kubernetes deploy target with single_run: true plus a native Kubernetes Job or CronJob." → 🤝 matches (evidence: The same file's config reference and the Kubernetes deploy target section both describe this pattern; source: repo:content/docs/deployments/concepts/customer-managed-runners.md)
  • L69 in content/docs/deployments/concepts/customer-managed-runners.md "If you're already running Kubernetes, the Kubernetes deploy target is the better default because it avoids the privileged-access trade-offs of Docker-in-Docker…" → ✅ verified (framing: strengthened — claim narrows the documented technical fact (no Docker-in-Docker/privileged execution needed, native Pod scheduling) into a positioning recommendation; evidence: Pulumi's own blog confirms the Kubernetes deploy target avoids the privileged Docker-in-Docker trade-off: "This integrates natively with Kubernetes pods and does not require Docker in Docker or privileged execution."; source: https://www.pulumi.com/blog/customer-managed-agents-kubernetes/)

🚨 Outstanding in this PR

No outstanding findings in this PR.

⚠️ Low-confidence

No low-confidence findings to report.

💡 Pre-existing issues in touched files (optional)

No pre-existing issues in touched files.

✅ Resolved since last review

  • [L60] content/docs/deployments/concepts/customer-managed-runners.md — The strict 1:1 runner-to-job exclusivity claim now reads "a runner process runs one deployment at a time — optionally alongside a single Insights scan or policy evaluation — and only claims its next deployment once the current one finishes (or fails)," matching the Scaling and concurrency guide's own framing. Resolved in 9000356.

📜 Review history

  • 2026-07-02T14:45:50Z — New deploy-target section verifies clean except one concurrency claim (L60) that overstates the runner-to-job relationship vs. the Scaling and concurrency guide. (3731019)
  • 2026-07-02T15:17:53Z — Re-reviewed after fix push (1 new commit, 9000356): L60 concurrency framing now matches the Scaling and concurrency guide, and the L69 weasel word ("usually") is gone (fresh Vale run: 0 findings). 0 outstanding remain.

Need a re-review? Want to dispute a finding? Mention @claude and include #update-review.
(For ad-hoc questions or fixes, just @claude — no hashtag.)

@github-actions github-actions Bot added review:outstanding-issues Claude review completed; outstanding has author-actionable findings and removed review:in-progress Claude review is currently running labels Jul 2, 2026
@pulumi-bot

pulumi-bot commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

…g and concurrency

Addresses pre-merge review feedback: the new section overstated the
1:1 runner-to-job relationship as strict single-job exclusivity, but
the linked Scaling and concurrency guide is more precise — a runner
runs one deployment at a time, optionally alongside a single Insights
scan or policy evaluation in parallel. Scopes the claim to deployments
and names that exception. Also drops a weasel word flagged by prose
linting.
@github-actions github-actions Bot added review:stale New commits since last Claude review; refresh on next ready-transition or @claude mention and removed review:outstanding-issues Claude review completed; outstanding has author-actionable findings labels Jul 2, 2026
@joeduffy

joeduffy commented Jul 2, 2026

Copy link
Copy Markdown
Member Author

@claude #update-review — pushed a fix for the L60 concurrency-framing finding (scoped the claim to match the Scaling and concurrency guide's one-deployment-plus-optional-parallel-task model) and removed the weasel word on line 69.

@github-actions github-actions Bot added review:in-progress Claude review is currently running and removed review:stale New commits since last Claude review; refresh on next ready-transition or @claude mention labels Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

🤖 Review updated on @joeduffy's request.

@github-actions github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels Jul 2, 2026

@CamSoper CamSoper left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated rubber stamp approval applied :shipit:


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:docs PR touches technical docs review:no-blockers Claude review completed cleanly; outstanding is empty

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants