Skip to content

Commit 28ab62e

Browse files
authored
[app-endpoint 5/6] snapshot-agent: add integration test harness (#100)
* snapshot-agent: add integration test harness End-to-end tests for all backends (CUDA, vLLM, SGLang) on real GPU hardware, in both deployment modes. The suite runs inside the cluster: run.sh deploys a test-runner pod, copies the repo source into it, and executes 'go test -tags=integration' there. All snapshot/restore calls go through the Python client (timeslice.snapshot_agent via agentctl.py), so the entire client layer is covered. agentctl.py builds BackendConfig protos in Python from primitive flags, the same way a real workload constructs them. The Go harness owns everything that is not the client: it deploys the agent and engine pods via client-go — one engine at a time, so a single free GPU is enough — waits for model load, measures VRAM via in-pod exec, and checks inference output before/after restore. All GPU pods are pinned to one node (the agent uses hostPID + cuda-checkpoint, which is node-local). Adding a test is a t.Run inside the engine group that provides the pods it needs; a new engine is an EngineSpec in engines.go. go.mod/go.sum gain three indirect entries (gorilla/websocket, moby/spdystream, mxk/go-flowrate) pulled in by client-go's remotecommand. Includes Dockerfile.snapshot-agent-runtime and a parameterized cloudbuild-image.yaml for building the agent image from the working tree. Signed-off-by: Aishu Kamal <aishuk@google.com> * snapshot-agent tests: address review feedback - agentctl.py: clear error when --backend app is used without --app. - engines.go: pin the cuda-checkpoint download to an immutable commit. - harness.go: bound execPod's exec stream with opTimeout. Signed-off-by: Aishu Kamal <aishuk@google.com> * snapshot-agent tests: scope runner pod CRUD/exec to namespace Role Cluster-wide access is now read-only (nodes, pod listing for GPU discovery); pod create/delete, exec, and logs move to a Role in the test namespace. Also declare the README options block language. Signed-off-by: Aishu Kamal <aishuk@google.com> --------- Signed-off-by: Aishu Kamal <aishuk@google.com>
1 parent 11a661f commit 28ab62e

12 files changed

Lines changed: 1131 additions & 0 deletions

File tree

Dockerfile.snapshot-agent-runtime

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
FROM golang:1.25-bookworm AS builder
2+
3+
RUN apt-get update && apt-get install -y protobuf-compiler && rm -rf /var/lib/apt/lists/*
4+
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
5+
RUN go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
6+
7+
WORKDIR /workspace
8+
COPY . .
9+
10+
RUN cd pkg/snapshot-agent/api/v1alpha1 && \
11+
protoc --go_out=. --go_opt=paths=source_relative \
12+
--go-grpc_out=. --go-grpc_opt=paths=source_relative \
13+
snapshot_agent.proto
14+
15+
RUN CGO_ENABLED=1 GOOS=linux go build -ldflags="-s -w" -o /workspace/snapshot-agent ./cmd/snapshot-agent
16+
17+
FROM debian:bookworm-slim
18+
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
19+
COPY --from=builder /workspace/snapshot-agent /snapshot-agent
20+
ENTRYPOINT ["/snapshot-agent"]

cloudbuild-image.yaml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
steps:
2+
- name: 'gcr.io/cloud-builders/docker'
3+
args:
4+
- 'build'
5+
- '-f'
6+
- 'Dockerfile.snapshot-agent-runtime'
7+
- '-t'
8+
- '${_IMAGE}'
9+
- '.'
10+
images:
11+
- '${_IMAGE}'
12+
substitutions:
13+
_IMAGE: 'gcr.io/${PROJECT_ID}/snapshot-agent:dev'
14+
timeout: '600s'
15+
options:
16+
machineType: 'E2_HIGHCPU_8'

go.mod

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,14 +27,17 @@ require (
2727
github.com/gogo/protobuf v1.3.2 // indirect
2828
github.com/google/gnostic-models v0.7.0 // indirect
2929
github.com/google/go-cmp v0.7.0 // indirect
30+
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect
3031
github.com/inconshreveable/mousetrap v1.1.0 // indirect
3132
github.com/josharian/intern v1.0.0 // indirect
3233
github.com/json-iterator/go v1.1.12 // indirect
3334
github.com/kylelemons/godebug v1.1.0 // indirect
3435
github.com/mailru/easyjson v0.7.7 // indirect
36+
github.com/moby/spdystream v0.5.1 // indirect
3537
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
3638
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
3739
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
40+
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
3841
github.com/pmezard/go-difflib v1.0.0 // indirect
3942
github.com/prometheus/client_model v0.6.2 // indirect
4043
github.com/prometheus/common v0.66.1 // indirect

go.sum

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
github.com/NVIDIA/go-nvml v0.13.3-1 h1:P76U2h88OZSiMtdhRsJjSF5DXyXUqHIXKeDicVAaae0=
22
github.com/NVIDIA/go-nvml v0.13.3-1/go.mod h1:ahi2psRYoa+wYUBIrZPRO+wJs9lcvMhxSSkjjvsJJNQ=
3+
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
4+
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
35
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
46
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
57
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
@@ -37,6 +39,8 @@ github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX
3739
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
3840
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
3941
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
42+
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 h1:JeSE6pjso5THxAzdVpqr6/geYxZytqFMBCOtn/ujyeo=
43+
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674/go.mod h1:r4w70xmWCQKmi1ONH4KIaBptdivuRPyosB9RmPlGEwA=
4044
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
4145
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
4246
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
@@ -58,6 +62,8 @@ github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0
5862
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
5963
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
6064
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
65+
github.com/moby/spdystream v0.5.1 h1:9sNYeYZUcci9R6/w7KDaFWEWeV4LStVG78Mpyq/Zm/Y=
66+
github.com/moby/spdystream v0.5.1/go.mod h1:xBAYlnt/ay+11ShkdFKNAG7LsyK/tmNBVvVOwrfMgdI=
6167
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
6268
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
6369
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -66,6 +72,8 @@ github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee h1:W5t00kpgFd
6672
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
6773
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
6874
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
75+
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f h1:y5//uYreIhSUg3J1GEMiLbxo1LJaP8RfCpH6pymGZus=
76+
github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
6977
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
7078
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
7179
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Snapshot Agent Integration Tests
2+
3+
End-to-end tests for snapshot-agent backends on real GPU hardware, in both standalone and K8s deployment modes.
4+
5+
The test suite is written in Go and runs inside the cluster: `run.sh` deploys a test-runner pod, copies the repo source into it, and executes `go test` there. The Go harness deploys the snapshot-agent and inference engine pods itself — one engine at a time, so a single free GPU is enough.
6+
7+
All snapshot/restore calls go through the **Python client** (`timeslice.snapshot_agent`, invoked via `agentctl.py`), so the entire client layer is covered.
8+
9+
- `run.sh` — launcher (deploy runner, copy source, install the Python client, `go test`, cleanup)
10+
- `runner.yaml` — test-runner pod + RBAC
11+
- `harness.go` / `engines.go` — harness: pod lifecycle, exec/HTTP helpers, pod specs
12+
- `agentctl.py` — thin CLI over the Python client; builds `BackendConfig` protos in Python from primitive flags
13+
- `standalone_test.go` / `k8s_test.go` — the test cases
14+
15+
## Adding a test
16+
17+
Add a `t.Run(...)` inside the engine group that provides the pods it needs, using the harness helpers:
18+
19+
```go
20+
h.WithEngine(t, VLLM, func(t *testing.T, e *Engine) {
21+
t.Run("MyNewTest", func(t *testing.T) {
22+
before := h.Inference(t, e) // deterministic completion
23+
h.SnapshotOK(t, "my-job", vllmSleepConfig(e.Endpoint(), 1))
24+
vram := h.VRAMMiB(t, e) // GPU memory in use
25+
h.RestoreOK(t, "my-job", vllmWakeConfig(e.Endpoint()))
26+
RequireFreedAndCorrect(t, vram, before, h.Inference(t, e))
27+
})
28+
})
29+
```
30+
31+
A new engine is an `EngineSpec` in `engines.go`.
32+
33+
## Prerequisites
34+
35+
- A GKE cluster with at least 1 free GPU
36+
- `gcloud` and `kubectl` on the machine running the tests
37+
(Go and everything else run inside the cluster)
38+
- A snapshot-agent image. Build one from the repo root with:
39+
40+
```bash
41+
gcloud builds submit --config=cloudbuild-image.yaml \
42+
--substitutions=_IMAGE=gcr.io/<your-project>/snapshot-agent:dev .
43+
```
44+
45+
This builds from your working directory, so local modifications are included — no commit needed. Requires the Cloud Build API (`gcloud services enable cloudbuild.googleapis.com`) and permission to push to the project's registry; GKE nodes in the same project can pull from `gcr.io/<project>` by default.
46+
47+
## Running
48+
49+
```bash
50+
./tests/integration/snapshot-agent/run.sh \
51+
--image gcr.io/<your-project>/snapshot-agent:dev \
52+
--project <your-project> \
53+
--cluster <your-cluster> \
54+
--zone <your-zone>
55+
```
56+
57+
## Options
58+
59+
```text
60+
--image IMAGE Snapshot-agent container image (required)
61+
--project PROJECT GCP project (runs gcloud get-credentials)
62+
--cluster CLUSTER GKE cluster name
63+
--zone ZONE GKE cluster zone
64+
--model MODEL Model to load (default: Qwen/Qwen2.5-0.5B)
65+
--phase PHASE "standalone", "k8s", or "both" (default: both)
66+
--skip-cleanup Leave the test-runner pod running for debugging
67+
```
68+
69+
## Exit code
70+
71+
`go test`'s exit code (0 = all passed).
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
#!/usr/bin/env python3
2+
"""CLI used by the integration tests to drive the snapshot-agent.
3+
4+
All snapshot/restore calls in the test suite go through this script so that
5+
the tests exercise the real production path: the Python client
6+
(timeslice.snapshot_agent) talking gRPC to the Go agent. Configs are
7+
constructed here, in Python, the same way a real workload would build them.
8+
9+
Usage:
10+
agentctl.py --agent HOST:PORT snapshot|restore --job-id ID --backend cuda|app
11+
[--pids 1,2,3] (cuda)
12+
[--app vllm|sglang] [--endpoints URL,URL]
13+
[--mode offload|discard] [--tags a,b] (app)
14+
15+
Exits 0 when the operation completes, 1 otherwise.
16+
"""
17+
18+
import argparse
19+
import sys
20+
21+
from timeslice.snapshot_agent import SnapshotAgentClient, snapshot_agent_pb2
22+
23+
APPS = {
24+
"vllm": snapshot_agent_pb2.APP_VLLM,
25+
"sglang": snapshot_agent_pb2.APP_SGLANG,
26+
}
27+
28+
MODES = {
29+
"": snapshot_agent_pb2.SUSPEND_MODE_UNSPECIFIED,
30+
"offload": snapshot_agent_pb2.SUSPEND_MODE_OFFLOAD,
31+
"discard": snapshot_agent_pb2.SUSPEND_MODE_DISCARD,
32+
}
33+
34+
35+
def build_config(args: argparse.Namespace) -> snapshot_agent_pb2.BackendConfig:
36+
if args.backend == "cuda":
37+
cuda = snapshot_agent_pb2.CudaBackendConfig()
38+
if args.pids:
39+
cuda.explicit_target.pids.extend(int(p) for p in args.pids.split(","))
40+
return snapshot_agent_pb2.BackendConfig(cuda=cuda)
41+
42+
if args.backend == "app":
43+
if args.app not in APPS:
44+
raise ValueError(f"--app is required for --backend app (one of {sorted(APPS)})")
45+
app_endpoint = snapshot_agent_pb2.AppEndpointConfig(
46+
app=APPS[args.app],
47+
endpoints=args.endpoints.split(",") if args.endpoints else [],
48+
mode=MODES[args.mode],
49+
)
50+
if args.tags:
51+
app_endpoint.tags.extend(args.tags.split(","))
52+
return snapshot_agent_pb2.BackendConfig(app_endpoint=app_endpoint)
53+
54+
raise ValueError(f"unknown backend {args.backend!r}")
55+
56+
57+
def main() -> int:
58+
parser = argparse.ArgumentParser(description=__doc__)
59+
parser.add_argument("action", choices=["snapshot", "restore"])
60+
parser.add_argument("--agent", required=True, help="agent endpoint HOST:PORT")
61+
parser.add_argument("--job-id", required=True)
62+
parser.add_argument("--group", default="test")
63+
parser.add_argument("--backend", required=True, choices=["cuda", "app"])
64+
parser.add_argument("--pids", default="", help="comma-separated PIDs (cuda)")
65+
parser.add_argument("--app", default="", choices=["", "vllm", "sglang"], help="application (app backend)")
66+
parser.add_argument("--endpoints", default="", help="comma-separated application URLs")
67+
parser.add_argument("--mode", default="", choices=["", "offload", "discard"], help="suspend mode")
68+
parser.add_argument("--tags", default="", help="comma-separated region tags")
69+
args = parser.parse_args()
70+
71+
config = build_config(args)
72+
73+
with SnapshotAgentClient(args.agent) as client:
74+
if args.action == "snapshot":
75+
result = client.snapshot_and_wait(args.job_id, args.group, backend_config=config)
76+
else:
77+
result = client.restore_and_wait(args.job_id, args.group, backend_config=config)
78+
79+
if result.status != "OPERATION_STATUS_COMPLETE":
80+
print(
81+
f"{args.action} {args.job_id} finished with {result.status}: {result.error}",
82+
file=sys.stderr,
83+
)
84+
return 1
85+
86+
print(f"{args.action} {args.job_id} complete in {result.elapsed_ms}ms")
87+
return 0
88+
89+
90+
if __name__ == "__main__":
91+
sys.exit(main())

0 commit comments

Comments
 (0)