Skip to content

Commit 4c19e6b

Browse files
committed
feat: RDMA multinode GPU support (chain SDK pieces)
Implements the akash-chain-sdk slice of the RDMA spec at provider/_docs/infiniband-implementation-spec.md, covering Linear tickets AKT-401..AKT-403, AKT-405, AKT-406 (CS-1, CS-2, CS-3, CS-5, CS-6). The shared-storage track (CS-4) is intentionally dropped: per spec decision #3 every service gets its own RWO PVC and the provider does not produce ReadWriteMany volumes. Everything in this commit is off-chain — no on-chain proto messages change, no validator upgrade required. CS-1 — Inventory v1 (proto/provider/akash/inventory/v1/{node,resources}.proto + go/inventory/v1/{node,resources}.pb.go regenerated via `make proto-gen-go`): * Add `ResourcePair rdma = 7` to `NodeResources`. * Add `rdma_resource_name`, `rdma_fabric`, `nccl_hca_prefix` strings to `NodeCapabilities` with gogoproto.customname annotations (`RDMAResourceName`, `RDMAFabric`, `NCCLHCAPrefix`). * Extend `Dup()` helpers in node.go and resources.go. * Tests: `node_test.go`, `resources_test.go` round-trip the new fields through Dup(). CS-2 — SDL parser: `gpu.attributes.rdma: true` (go/sdl/gpu.go): * Accept `rdma: true|false` under `gpu.attributes`. When true, emit a flat on-chain GPU attribute `rdma=true` so providers that advertise `capabilities/gpu/rdma=true` match. * Tests: `rdma_gpu_test.go::TestV2ResourceGPU_RDMAFlag` plus the existing `TestV2ResourceGPU` regression continues to pass. CS-3 — SDL parser: `gpu.attributes.rdma_group: <name>` → off-chain manifest field (go/sdl/gpu.go, proto/provider/akash/manifest/v2beta3/service.proto regenerated to go/manifest/v2beta3/service.pb.go, go/sdl/groupBuilder_v2{,_1}.go): * Parser captures rdma_group via an internal sentinel attribute key (`__rdma_group__`) inside `v2GPUAttributes.UnmarshalYAML`. The parent `v2ResourceGPU.UnmarshalYAML` strips the sentinel before the slice ever reaches on-chain `Resources.GPU.attributes` and surfaces the value on a dedicated `v2ResourceGPU.RDMAGroup` field. `Validate()` is deferred to the parent so the sentinel can be removed before the attribute-key regex runs. * Manifest `Service.proto` gains `rdma_group = 11` with `gogoproto.customname = "RDMAGroup"`. Bindings regenerated via `make proto-gen-go`. * Both v2 / v2.1 group builders now read `compute.Resources.GPU.RDMAGroup` and propagate it onto `manifest.Service.RDMAGroup`. * Tests: `TestV2ResourceGPU_RDMAGroupRoutedOffChain` (asserts the sentinel never escapes) and `TestV2ResourceGPU_RDMAGroupOmitted`. CS-5 — Parser cross-field validations (go/sdl/v2.go, validate() → new validateRDMA()): 1. Any compute profile with `gpu.attributes.rdma: true` requires its placement attributes to include `capabilities/rdma=true`. 2. Any compute profile with `gpu.attributes.rdma_group` set must also declare `gpu.attributes.rdma: true` on the same profile. 3. Within one placement, no implicit-default-plus-explicit mixing: if any profile sets rdma_group, every RDMA-using profile must. Helpers `gpuAttributesHaveRDMA` and `placementRequiresRDMA` are kept package-local so the SDL parser owns the policy. * Tests: `rdma_validation_test.go` (6 positive + negative fixtures). CS-6 — Reservation commit path audit (go/node/deployment/v1beta4/rdma_commit_audit_test.go): Table-driven regression test pinning down that `GroupSpec.Dup()` and the four concrete `ResourceGroup`-shaped values the provider's reservation path can hold (`*Group`, `Group`, `*GroupSpec`, `GroupSpec`) all preserve `Requirements.Attributes` (carrying `capabilities/rdma=true`) AND each resource's `GPU.Attributes` (carrying `rdma=true`). A future change that silently drops either slice — the exact failure mode the spec calls out — fails this test loudly. All `.pb.go` files were regenerated via `make proto-gen-go` (buf v1.47.2, protoc v29.1, gogoproto v1.7.2). Running `make proto-gen-go` against this tree should be a no-op. Tests (go test ./...): - pkg.akt.dev/go/inventory/v1 PASS - pkg.akt.dev/go/manifest/v2beta3 PASS - pkg.akt.dev/go/node/deployment/v1beta4 PASS (+ new CS-6 audit) - pkg.akt.dev/go/sdl PASS (+ new CS-2/CS-3/CS-5) Follow-ups for reviewers: - TypeScript bindings (`ts/`) are not touched here and need a separate `make proto-gen-ts` pass before the TS SDK consumes the new fields. Linear: AKT-401, AKT-402, AKT-403, AKT-405, AKT-406 (AKT-404 / CS-4 cancelled — see spec decision #3) Fix make/setup-cache.mk: defer GOLANGCI_LINT_MAJOR computation to recipe-execute time and depend on $(SEMVER), so the install isn't given an empty major and the broken module path .../golangci-lint/v/cmd/golangci-lint. This bug had `lint/go` red on main for several commits prior.
1 parent bf983bc commit 4c19e6b

34 files changed

Lines changed: 1598 additions & 228 deletions

File tree

go/inventory/v1/node.go

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,10 @@ package v1
22

33
func (nd *NodeCapabilities) Dup() NodeCapabilities {
44
res := NodeCapabilities{
5-
StorageClasses: make([]string, 0, len(nd.StorageClasses)),
5+
StorageClasses: make([]string, 0, len(nd.StorageClasses)),
6+
RDMAResourceName: nd.RDMAResourceName,
7+
RDMAFabric: nd.RDMAFabric,
8+
NCCLHCAPrefix: nd.NCCLHCAPrefix,
69
}
710

811
res.StorageClasses = append(res.StorageClasses, nd.StorageClasses...)

go/inventory/v1/node.pb.go

Lines changed: 196 additions & 24 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

go/inventory/v1/node_test.go

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
package v1
2+
3+
import (
4+
"testing"
5+
6+
"github.com/stretchr/testify/require"
7+
)
8+
9+
func TestNodeCapabilities_Dup_PreservesRDMAFields(t *testing.T) {
10+
src := NodeCapabilities{
11+
StorageClasses: []string{"beta3", "default"},
12+
RDMAResourceName: "rdma/rdma_shared_device_ib",
13+
RDMAFabric: "infiniband",
14+
NCCLHCAPrefix: "mlx5",
15+
}
16+
17+
got := src.Dup()
18+
19+
require.Equal(t, src.StorageClasses, got.StorageClasses)
20+
require.Equal(t, "rdma/rdma_shared_device_ib", got.RDMAResourceName)
21+
require.Equal(t, "infiniband", got.RDMAFabric)
22+
require.Equal(t, "mlx5", got.NCCLHCAPrefix)
23+
24+
// mutating dup must not affect the source (Dup is a deep copy)
25+
got.StorageClasses[0] = "mutated"
26+
got.RDMAResourceName = "rdma/rdma_shared_device_eth"
27+
require.Equal(t, "beta3", src.StorageClasses[0])
28+
require.Equal(t, "rdma/rdma_shared_device_ib", src.RDMAResourceName)
29+
}
30+
31+
func TestNodeCapabilities_Dup_ZeroValueRDMA(t *testing.T) {
32+
// A non-RDMA node leaves the new fields at zero value.
33+
src := NodeCapabilities{
34+
StorageClasses: []string{"default"},
35+
}
36+
37+
got := src.Dup()
38+
39+
require.Empty(t, got.RDMAResourceName)
40+
require.Empty(t, got.RDMAFabric)
41+
require.Empty(t, got.NCCLHCAPrefix)
42+
}

go/inventory/v1/resourcepair.go

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -47,19 +47,43 @@ func (m *ResourcePair) LT(rhs ResourcePair) bool {
4747
return m.Allocatable.Cmp(*rhs.Allocatable) == -1
4848
}
4949

50+
// IsZero reports whether the ResourcePair has been initialized. A
51+
// zero-valued ResourcePair has nil quantity pointers and is the natural
52+
// state for, e.g., a node that does not have RDMA capacity (and therefore
53+
// leaves `NodeResources.RDMA` untouched).
54+
func (m *ResourcePair) IsZero() bool {
55+
if m == nil {
56+
return true
57+
}
58+
return m.Capacity == nil && m.Allocatable == nil && m.Allocated == nil && len(m.Attributes) == 0
59+
}
60+
5061
func (m *ResourcePair) Dup() ResourcePair {
51-
capacity := m.Capacity.DeepCopy()
52-
allocatable := m.Allocatable.DeepCopy()
53-
allocated := m.Allocated.DeepCopy()
62+
// A zero-valued ResourcePair (all quantity pointers nil) must round-trip
63+
// through Dup() without panicking. Without this guard, calling Dup()
64+
// against e.g. an unpopulated `NodeResources.RDMA` on a non-RDMA node
65+
// nil-derefs Capacity.DeepCopy().
66+
if m == nil || m.IsZero() {
67+
return ResourcePair{}
68+
}
5469

55-
res := ResourcePair{
70+
var capacity, allocatable, allocated resource.Quantity
71+
if m.Capacity != nil {
72+
capacity = m.Capacity.DeepCopy()
73+
}
74+
if m.Allocatable != nil {
75+
allocatable = m.Allocatable.DeepCopy()
76+
}
77+
if m.Allocated != nil {
78+
allocated = m.Allocated.DeepCopy()
79+
}
80+
81+
return ResourcePair{
5682
Capacity: &capacity,
5783
Allocatable: &allocatable,
5884
Allocated: &allocated,
5985
Attributes: m.Attributes.Dup(),
6086
}
61-
62-
return res
6387
}
6488

6589
func (m *ResourcePair) SubMilliNLZ(val types.ResourceValue) bool {

0 commit comments

Comments
 (0)