fix: pull helm charts atomically to survive concurrent kustomize builds

[KR-Ravindra]

Problem

Running several kustomize build --enable-helm processes at the same time against overlays that share a base with helmCharts (the usual CI or Argo CD setup) fails intermittently with:

Error: failed to untar: a file or directory with the name <chartHome>/<chart>-<version>/<chart> already exists
: unable to run: 'helm pull --untar --untardir <chartHome>/<chart>-<version> --repo <repo> <chart> --version <version>'

Only the first process succeeds; every other process that decided to pull the same chart is failed by helm pull, even though the chart it wanted is now present and complete.

Root cause

plugin/builtin/helmchartinflationgenerator/HelmChartInflationGenerator.go:281-289 checks chartExistsLocally() and, when the chart directory is absent, runs helm pull --untar --untardir <absChartHome> (pullCommand(), line 336). Nothing ties the check to the pull: two processes sharing a chartHome both observe "missing", both run helm pull into the same directory, and helm refuses to untar into a chart directory that another process has meanwhile created (helm/helm#12315 documents that helm considers this the caller's responsibility). The pull error is returned as-is, so the whole build fails.

Simply ignoring the helm error is not safe either: the winning process may still be extracting the archive when the loser re-checks, so the loser could run helm template against a half-extracted chart.

Fix

pullChart() now pulls into a private temporary directory created inside the chart home (os.MkdirTemp(chartHome, ".kustomize-pull-")) and then moves the extracted chart into place with os.Rename. Every process extracts into its own directory, so helm never sees a collision, and the rename is atomic, so <chartHome>/<chart> is either absent or complete. If the rename fails because another process already placed the chart there, that chart is used and no error is raised. The temporary directory is removed either way.

pullCommand() takes the untar directory as a parameter. No change to the on-disk layout of chartHome, so existing pulled charts and local charts keep working.

The generated copy in api/internal/builtins/HelmChartInflationGenerator.go was regenerated with make api/internal/builtins/HelmChartInflationGenerator.go.

api/testutils/kusttest.HarnessEnhanced gains LoadGenerator(config), which loads and configures a generator without running it; LoadAndRunGeneratorWithBuildAnnotations is now built on top of it. This lets the test run several configured generators concurrently.

How tested

New test TestHelmChartInflationGeneratorConcurrentPull (plugin/builtin/helmchartinflationgenerator/HelmChartInflationGenerator_test.go) packages testdata/charts/test-chart, serves it from an httptest helm repository, and runs eight generators that share one chartHome concurrently. It asserts that every generator succeeds with identical output and that only the chart directory remains in the chart home (no leftover temporary directories). It shells out to helmV3 like the neighbouring tests and skips when helm is unavailable.

Before the fix (go test -run TestHelmChartInflationGeneratorConcurrentPull in the plugin module):

--- FAIL: TestHelmChartInflationGeneratorConcurrentPull (0.76s)
    HelmChartInflationGenerator_test.go:1095:
        Error:  Received unexpected error:
                Error: failed to untar: a file or directory with the name /tmp/kust-testing-2854014185/charts/test-chart-1.0.0/test-chart already exists
                : unable to run: 'helmV3 pull --untar --untardir /tmp/kust-testing-2854014185/charts/test-chart-1.0.0 --repo <local httptest repo> test-chart --version 1.0.0' ... exit status 1
        Messages: generator 0
FAIL

After the fix:

$ go test -count=5 -run 'TestHelmChartInflationGeneratorConcurrentPull$' .
ok  	sigs.k8s.io/kustomize/plugin/builtin/helmchartinflationgenerator	2.244s
$ go test -count=1 .
ok  	sigs.k8s.io/kustomize/plugin/builtin/helmchartinflationgenerator	10.056s
$ (cd api && go build ./... && go test -run Helm ./krusty/)
ok  	sigs.k8s.io/kustomize/api/krusty	3.759s

gofmt, go vet and golangci-lint (v1.64.8, repo config) are clean for the plugin module and the changed api packages. Verified with helm v3.21.4.

Links

• Fixes Concurrency issues with Helm generator - archive already exists #5271
• Helm's position that the caller must serialise pulls: Support pull untar overwrite or ignore mode helm/helm#12315
• Helm chart inflation docs: https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/

This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.

[kubernetes-prow]

Welcome @KR-Ravindra!

It looks like this is your first PR to kubernetes-sigs/kustomize 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kustomize has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: KR-Ravindra / name: KR Ravindra (f88ef73)

[kubernetes-prow]

Hi @KR-Ravindra. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: KR-Ravindra
Once this PR has been reviewed and has the lgtm label, please assign koba1t for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[KR-Ravindra]

/easycla

[KR-Ravindra]

/easycla

[KR-Ravindra]

Round 1 self-review (AI agent operated by me; I review every revision). Staying a draft until this is fixed:

1. EasyCLA reports "Missing CLA Authorization" most likely because the commit carries a Co-Authored-By: ... <noreply@anthropic.com> trailer (plus a Claude-Session: trailer); EasyCLA also checks co-author emails, and that address has no CLA. Reword the commit to drop both trailers and force-push. The same author email passes EasyCLA on refactor(provider): drop AWS SDK dependency from ZoneTypeFilter external-dns#6707.

Verified against master: pullCommand() untars straight into absChartHome() and chartExistsLocally() only stats <chartHome>/<name>, so two processes that both observe "missing" both run helm pull into the same directory, as the body describes. Pull-into-temp-then-rename is sound: os.Rename onto an existing non-empty directory fails and the fallback re-checks chartExistsLocally(), so a half-extracted chart is never used. No other PR references #5271.

[KR-Ravindra]

/easycla

[KR-Ravindra]

Self-review before marking ready. The pull now extracts into a per-process temporary directory inside the chart home and moves the chart into place atomically, so concurrent builds sharing a chartHome never collide and a half-extracted chart is never used; the new test runs eight generators concurrently against a local Helm repository and fails on master with the exact error from the issue.

[KR-Ravindra]

@koba1t @sarab97 @varshaprasad96 when one of you has a moment, could this get /ok-to-test? It is a small fix for concurrent kustomize build runs racing on the Helm chart pull (issue linked), with a unit test that fails on master. Happy to adjust anything.