feat(inference): HiSparse vLLM wheel for GLM-5.x PD decode instances

[faresobeid]

Enables HiSparse host-resident sparse-MLA decode (vllm-project/vllm#46326) for GLM-5.2 deployments. Three commits:

1. Wheel pin: x86_64 vLLM moves to the fork wheel (hisparse-c6f2044, v0.24.0 release base). aarch64 stays upstream; runtime patches/plugin apply unchanged.
2. kv_cache_offload.roles: scopes the Mooncake store to a subset of P/D roles. Out-of-scope nodes skip the store client and get a NIXL-only transfer config — decode-node RAM goes to HiSparse pinned pools (ranks × host_pool_gib) instead of a store segment.
3. Graceful teardown + hardening in the sbatch templates: TERM traps with a bounded drain window (decode ranks unpin multi-hundred-GiB pinned pools in-process; a SIGKILLed unpin runs in uninterruptible kernel context, outlives SLURM's KillWait, and drains the node), VLLM_WORKER_SHUTDOWN_TIMEOUT_SECONDS=600, component-crash escalation (scancel --signal=TERM + CRASH_<host> marker — an engine death can no longer leave a silently wedged deployment), envoy --use-dynamic-base-id (a stale /dev/shm/envoy_shared_memory_0 from another user's crashed run SIGABRTs the default base-id; reproduced and fix-validated on a poisoned node), orphan-proof mooncake store launcher, per-rank uv run --no-sync, decode on auto-detected multi-rail UCX_NET_DEVICES.

Why

A GLM-5.2 wide-EP decode rank has O(10 GiB) of GPU KV after weights and sparse-stack workspaces — a handful of long-context requests per engine. HiSparse keeps the full sparse-MLA KV in pinned host memory and serves decode from per-request GPU hot buffers of the indexer top-k.

Measured (GLM-5.2-FP8, this branch's wheel)

16-node stack (8 prefill + 8 decode DP64, llm-d, NIXL, mooncake), A/B vs plain GPU-KV decode — identical topology, traffic, and 50-min saturated window:

decode side	GPU-KV	HiSparse
generation throughput	10,653 tok/s	12,951 tok/s (+22%)
requests completed	84,026	101,653 (+21%)
preemptions	5,174	0
TTFT p99	99.9 s	2.5 s
KV usage peak — GPU KV vs host pool	100.0% (pinned)	57.5%

Pushed to seat saturation (c=6500): 26,056 decode tok/s sustained, 99% peak seat occupancy, zero preemptions — 3.3× the c=1000 operating point; the binding constraint is max_num_seqs, not memory.

Agentic RL workload (r2e-gym through the router, c=1000, 4P+2D): baseline pinned at 99.9% KV with 1,533 preemptions, 5 of 155 rollouts solved; HiSparse: zero preemptions, 585 solved, +124% steady-state decode throughput.

Deployment notes

• Enable via decode overrides: attention_config = { hisparse_config = { host_pool_gib = <N> } } per rank; budget node RAM for ranks × pool (+ any co-tenants — hence roles = ["prefill"] on the store).
• Cap decode max_num_seqs (e.g. 96): hot buffers are eager (~370 MB/request across GLM-5.2's 78 sparse layers) and the seat count is the admission control — size it to sustained decode compute at your context lengths, or rely on orchestrator inflight caps.
• Decode gpu_memory_utilization can go to 0.9 (the GPU KV budget mostly idles under HiSparse).
• Cancel HiSparse jobs with the two-step TERM-then-scancel flow (documented in the monitor-run skill) so pools unpin in-process.

🤖 Generated with Claude Code