NVMe layer: target subsystem linked under the wrong nvmet port when another port shares the same address

[Tydus]

Version info:

• LINSTOR controller (linstor controller version): 1.33.1 (build 95da7940d6efb6a39ea303c5f37b03478a6fab0b, identical to tag v1.33.1)
• LINSTOR client (linstor --version): 1.27.1
• Satellites: 1.33.1
• Kernel: 6.8 (Debian 12 / Proxmox VE), kernel nvmet via configfs
• Transport: NVMe-oF over RDMA (RoCEv2)

Affected versions: verified in 1.33.1 and on current master (c4ba920d, "prepare release v1.34.1"). The affected code is byte-identical between the two apart from cosmetic refactoring (explicit UTF-8 charset), so all 1.33.x and 1.34.x releases are affected.

Description:

NvmeUtils.getPortIdx() resolves the nvmet port for a target resource by shelling out to

grep -r -H --color=never <ip> /sys/kernel/config/nvmet/ports/

and substring-parsing the first output line (getPortIdx, grep exec at L1038-L1042, first-line parse at L1047-L1051).

The bare IP is matched — with no addr_trsvcid or addr_trtype in the pattern — against every file under ports/. As soon as any other nvmet port with the same addr_traddr exists on the target node (e.g. an admin-created manual export on a different service id), which port "wins" is decided by configfs readdir order, which is unspecified. On our cluster the foreign port won consistently, and every new LINSTOR NVMe resource creation fails while it exists; deletion of existing NVMe resources fails as well (details below).

Reproduction steps:

1. Target node already has a LINSTOR-managed nvmet port, e.g.:

   /sys/kernel/config/nvmet/ports/1: addr_traddr=169.254.1.9 addr_trsvcid=4420 addr_trtype=rdma
   

2. Create an unrelated second nvmet port on the same address, different service id (as an admin would for a temporary manual export):

   mkdir /sys/kernel/config/nvmet/ports/2
   echo ipv4        > /sys/kernel/config/nvmet/ports/2/addr_adrfam
   echo rdma        > /sys/kernel/config/nvmet/ports/2/addr_trtype
   echo 169.254.1.9 > /sys/kernel/config/nvmet/ports/2/addr_traddr
   echo 4499        > /sys/kernel/config/nvmet/ports/2/addr_trsvcid
   

3. Create any new NVMe resource with the target on this node, plus an initiator:

   linstor resource-group spawn-resources <nvme-rg> test-nvme01 1G
   linstor resource create --nvme-initiator <initiator-node> test-nvme01
   

4. Observed live (LINSTOR 1.33.1, kernel 6.8): the target satellite creates the subsystem LS-NVMe_test-nvme01 and its namespace normally, but symlinks it under ports/2 (verified: ls /sys/kernel/config/nvmet/ports/2/subsystems/ briefly shows LS-NVMe_test-nvme01). The initiator satellite then runs nvme discover against the configured port (4420); nvmet's discovery log only reports subsystems linked to the port the discovery connection arrived on, so the subsystem is not found and the create fails with:

   Failed to discover subsystem name 'LS-NVMe_test-nvme01'!
   

   (StorageException thrown at NvmeUtils.java:429 in 1.33.1, L430 on master). The controller rolls the resource back within about a second.

Root cause:

An nvmet port is defined by the (trtype, traddr, trsvcid) tuple, but getPortIdx() matches only the address — as an unanchored, unescaped regex, recursively over all of ports/. The result is consumed by both:

• create: L198-L199, which then symlinks the subsystem under the resolved port (L244-L254)
• delete: L306, which removes the symlink from the resolved port (L314-L317)

Further defects in the same lookup (verified in code):

1. Substring/regex matching: the IP is not anchored or escaped, so 10.0.0.9 also matches ports with addr_traddr 10.0.0.90, 10.0.0.99, or 110.0.0.9; . matches any character. grep -r also scans referral entries (ports/*/referrals/*/addr_*), which carry their own addr_traddr, so a discovery referral containing the address under any port matches too. No second port needed for a mismatch in these cases.
2. Multi-line grep output is silently truncated: when several files match, the parse takes the port index from the first line and ignores the rest — no warning, no disambiguation.
3. Delete fail-stop while the condition persists: for a resource whose subsystem is linked under ports/1 while grep resolves ports/2, File.delete() on the nonexistent symlink returns false and deleteTargetRsc throws Failed to remove symbolic link! (L314-L317) before any namespace/subsystem teardown. The resource sticks in DELETE with device-manager retries. Not destructive: nothing foreign is touched. In particular, the port-directory cleanup at L319-L327 is gated on ports/<idx>/subsystems not existing — but nvmet creates subsystems/ as a configfs default group for the port's entire lifetime, so that rmdir is dead code: LINSTOR can never delete a foreign port, and also never garbage-collects its own empty port directories (the intended check was presumably "subsystems dir is empty").
4. Adjacent, out of scope of the fix: the next-port-index computation (L206-L217) sorts directory names lexicographically, so ports {9, 10} yield "next" index 10, colliding with the existing port. Flagging for completeness; unlikely with LINSTOR's single-port usage.

Impact:

Any environment where a second nvmet port shares (or superstring-matches) the LINSTOR target address — a very ordinary situation on a storage node also serving manual exports — loses the ability to create new NVMe resources (create + rollback within ~1 s) and to delete existing ones (fail-stop, retried) until the foreign port is removed. Existing, already-linked resources keep serving I/O; the adjust path never consults ports/.

Suggested fix (PR prepared):

• Create path: match ports by the exact addr_traddr/addr_trsvcid/addr_trtype triple via plain Java directory iteration (trimming the configfs trailing newline; comparing traddr and trtype case-insensitively), instead of the grep shell-out. This makes a port what the kernel says it is.
• Delete path: locate the port by which ports/*/subsystems/<subsystemName> symlink actually exists. This stays correct when NVMe/Port or NVMe/TRType changed after creation, and also heals subsystems that the buggy version linked under a foreign port.

A PR implementing this fix is in preparation, together with fixes for #482 and #483 — NVMe-layer reports from the same environment.

Disclosure (ref. #478): the analysis was AI-assisted. All findings above were verified by me against the source at c4ba920d, and the failure was reproduced live on our production 1.33.1 cluster before filing.