capmox VMID collision under concurrent multi-cluster rolls

[Leonorus]

What steps did you take and what happened:

A single cluster-api-provider-proxmox (capmox) v0.9.0 controller manages three workload clusters (cluster1/2/3) whose VMs all live on the same three-node Proxmox cluster (shared VMID namespace). We triggered a rolling node replacement (Talos/K8s upgrade) on all three clusters at roughly the same time, so many ProxmoxMachine objects were being provisioned concurrently.

capmox assigned the same VMID (104) to two different ProxmoxMachine objects in two different CAPI clusters, created ~6 seconds apart:

• cluster2-worker-ks7wv-qfpxb — virtualMachineID: 104, created 18:40:32Z — won the race, VM 104 cloned/started on node hv23, became Running.
• cluster3-worker-p5w4v-hs8jz — virtualMachineID: 104, created 18:40:26Z, target node hv25 — lost the race and got permanently wedged.

The losing ProxmoxMachine reconcile loop:

E find.go:54  "unable to find vm" err="cannot find vm with id 104: 500 Configuration file 'nodes/hv25/qemu-server/104.conf' does not exist"
E proxmoxmachine_controller.go:224 "error reconciling VM" err="error trying to locate vm: expected VM name to match \"cluster2-worker-ks7wv-qfpxb\" but it was \"cluster3-worker-p5w4v-hs8jz\""
E controller.go:474 "Reconciler error" err="failed to reconcile VM: error trying to locate vm: expected VM name to match \"cluster2-worker-ks7wv-qfpxb\" but it was \"cluster3-worker-p5w4v-hs8jz\""
I proxmoxmachine_controller.go:182 "Error state detected, skipping reconciliation"

The ProxmoxMachine then sits in VirtualMachineProvisioned=False / reason=VMProvisionFailed and "Error state detected, skipping reconciliation" forever — it never re-selects a free VMID or retries. Because the parent MachineDeployment uses maxUnavailable: 0, the surge replacement never becomes Ready, so the whole rollout stalls indefinitely (in our case ~50 minutes until manual intervention). No orphaned VM or disk was created for the losing machine — the VM at 104 belongs to the other cluster, and there was no config or backing volume on the loser's target node hv25.

We recovered by deleting the wedged Machine; CAPI recreated it and (with the other cluster's roll now finished) it was assigned a fresh, non-colliding VMID and provisioned normally.

What did you expect to happen:

VMID selection should be safe under concurrent reconciles that share one Proxmox VMID namespace — capmox should never hand the same VMID to two ProxmoxMachine objects. Either:

1. Reserve the chosen VMID atomically before other reconciles can pick it (so two concurrent machines cannot select the same "next free" ID), and/or
2. When a VMID collision is detected at provision time (existing VM name does not match the expected name), treat it as a recoverable condition — release/re-select a new VMID and requeue — instead of latching a terminal Error state, skipping reconciliation that blocks the machine (and the whole MachineDeployment) forever.

Anything else you would like to add:

• Root cause appears to be a TOCTOU race in next-free-VMID selection: concurrent reconciles each compute "next free ID" against a Proxmox view that hasn't yet accounted for the other in-flight clone, so both land on 104. It was likely aggravated by an old VM that still held VMID 104 being torn down at the same moment — we also saw cannot delete vm with id 104: 500 VM 104 is running - destroy failed for the outgoing cluster2-worker-gkzfd-k9r4j seconds earlier, so 104 was in a churning state during the allocation window.
• The expected VM name to match ... guard itself is correct and desirable (it prevents cross-cluster VM adoption / accidental takeover). The bug is (a) that a duplicate VMID is handed out at all, and (b) that hitting the guard produces a non-retryable terminal state with no self-recovery.
• Impact is amplified by MachineDeployment.strategy.rollingUpdate.maxUnavailable: 0: one wedged surge machine blocks the entire rollout, not just itself.
• Reproduction is timing-dependent but reliable when multiple clusters (or many machines) sharing one Proxmox cluster are provisioned concurrently. Manual workaround: kubectl delete machine <wedged-machine> and let CAPI recreate it once contention has cleared.
• VMID allocation config in use: capmox default next-id selection (no static vmIDRange/explicit per-cluster VMID partitioning configured across the three clusters).

Environment:

• Proxmox version: pve-manager/9.1.1 (running kernel 6.17.2-1-pve)
• Cluster-api-provider-proxmox version: v0.9.0 (ghcr.io/ionos-cloud/cluster-api-provider-proxmox:v0.9.0)
• Kubernetes version (use kubectl version): v1.35.3 (management and workload clusters)
• OS (e.g. from /etc/os-release): Talos Linux v1.13.5 (kernel 6.18.36-talos, containerd 2.2.5)

[Leonorus]

I'll work on fix in two ways. 1. auto-heal and 2. prevention - claim vmid before clone