Investigate slow inter-node throughput between same-DC OVH KS-5 nodes (~256 Mbps)

[agentydragon]

Priority: low ("eventually").

Observation

During the SeaweedFS volume evacuation of hdd-0 (ovh-ns103711 → ovh-ns102453/ovh-ns103656, all OVH KS-5 boxes in the hil DC) on 2026-07-05, throughput was ~32 MB/s (~256 Mbps) — ~90 GB moved in ~47 min, one volume at a time.

Same-DC machines should do gigabit+; 256 Mbps is far too slow even accounting for Nebula AES encryption. Traffic path: pod → Cilium VXLAN → nebula1 → eno1.

Hypotheses

1. Nebula single-stream AES-GCM capped on one CPU core (userspace UDP) — most likely.
2. Traffic routed over a relay / public path instead of the local DC subnet — compare against the kimsufi_eno1_peer_route direct path.
3. MTU/fragmentation on the VXLAN-over-nebula stack.
4. SeaweedFS volume.move is single-threaded + fsync-heavy on 7200rpm HDD → disk-bound, not network.

Follow-up

• iperf3 node-to-node over eno1 direct vs nebula1.
• Watch nebula process CPU during a transfer (is one core pinned?).
• Benchmark a raw volume.move in isolation.

Surfaced while executing the OVH data-disk mount rename (cluster/docs/plans/ovh_storage_tiering.md).

[agentydragon]

Update 2026-07-05: buffer/routines hypothesis DISPROVEN by measurement

Tried the nebula tuning in #2926 (listen.read_buffer/write_buffer = 10 MiB, routines = 2), applied it to both ovh-ns103711 and ovh-ns102453 (targeted Talos config applies), and re-measured iperf3 between them (the KS-5↔KS-5 path — the one the slow SeaweedFS moves used).

Result: no improvement. Retransmits are still huge — ~10,800 in 5s ≈ ~5-6% packet loss, throughput ~450 Mbps. Same order as before the change.

Why the buffer idea was wrong:

• net.core.rmem_max = 4 MiB on the nodes, so nebula's 10 MiB request is capped to 4 MiB regardless.
• But that doesn't matter: at same-DC sub-millisecond RTT the bandwidth-delay product is ~30-60 KB. 4 MiB is already ~60× more buffer than the link can use. The socket buffer was never the bottleneck, so enlarging it cannot help.
• routines = 2 only helps aggregate multi-flow throughput; a single TCP stream hashes to one routine, so it does nothing for the single-stream case (SeaweedFS volume.move, git, etc.).

So the ~5-6% loss comes from somewhere else — not the UDP socket buffer. #2926 is left in place (harmless) but is not a fix for this.

Process note: I should have measured the buffer hypothesis before merging the change. Lesson re-learned: diagnose with real data first.

Next (measure the loss source before any more config):

• UDP-mode iperf3 (-u) to quantify one-way loss without TCP's own retransmits confounding it.
• tun device drop counters on nebula1 (tx-queue overflow is a separate drop path from the socket buffer — tun.tx_queue unset/default).
• Interface + /proc/net/snmp (UDP InErrors/RcvbufErrors) on both ends.
• Rule out genuine OVH-underlay loss between these specific hosts (nebula-off eno1-direct iperf if feasible), and whether the path is direct-punched vs relayed.

[agentydragon]

Update 2026-07-07: live OVH specs explain the apparent Nebula cap

Follow-up investigation is in PR #2973 and debug/nebula_inter_node_perf/.

Main result: this is not primarily a Nebula networking bug. The direct public eno1 path is already near the observed Nebula/pod-overlay ceiling.

Live OVH API /dedicated/server/{serviceName}/specifications/network reports:

• KS-5 nodes (102453, 103656, 103711): OvhToOvh=500 Mbps, OvhToInternet=500 Mbps, InternetToOvh=500 Mbps, connection=1000 Mbps, no vRack bandwidth.
• KS-GAME nodes (104952, 104963): OvhToOvh=300 Mbps, OvhToInternet=300 Mbps, InternetToOvh=300 Mbps, connection=1000 Mbps, no vRack bandwidth.

Representative measurements:

• KS-5 102453 -> 103711: public host path 489.8 Mbps / 36.1k retransmits; Nebula host path 468.8 Mbps / 33.2k retransmits; pod overlay 453.4 Mbps / 29.8k retransmits.
• KS-GAME 104963 -> 104952: public host path 270.9 Mbps / 23.3k retransmits; Nebula host path 261.4 Mbps / 16.4k retransmits; pod overlay 255.5 Mbps / 17.0k retransmits.
• Benchmark-source 103656 -> 104952: public host path 492.3 Mbps / 26.4k retransmits; Nebula host path 390.2 Mbps / 11.6k retransmits; pod overlay 439.6 Mbps / 17.9k retransmits.

Conclusion: same-DC traffic on these Eco/Kimsufi servers is still constrained by the included public OvhToOvh bandwidth class; it is not an unmetered/private 1 Gbps path. Nebula adds overhead, but there is no evidence for a hidden 1 Gbps same-DC path that Nebula is failing to use.

Storage benchmark implication: the SSD SeaweedFS result was topology-confounded. Client pods were on HDD node ovh-ns103656; SSD volume servers are remote on KS-GAME nodes with 300 Mbps OvhToOvh, while HDD could hit a local HDD volume server.

Action in #2973: document the evidence, stop calling the previous Nebula buffer/routines tweak the primary single-stream fix, and update the storage-tiering note to call this an OVH inter-node throughput/topology limit.