Serve inference endpoints over Nebius managed HTTPS URLs with token auth (drop public IPs)

[vertix]

Problem

workflows/nebius/serve.sh creates every endpoint with --public and no authentication: a bare public IP serving plain HTTP on :8000. PolicyServer (positronic/offboard/server.py) has no auth of any kind, so anyone who finds the IP can list checkpoints (GET /api/v1/models) and open an inference session (WS /api/v1/session). Robot observations and action streams cross the internet in cleartext.

Nebius has since made this unnecessary. Per Managing endpoints:

• every HTTP container port is reachable over a managed https:// URL, and --public is explicitly not required to reach an endpoint from the internet;
• --auth token gates the endpoint on Authorization: Bearer <token>, with the token supplied by --token-secret — a secret selector whose payload key is AUTH_TOKEN.

We should move every served endpoint onto that: no public IP, TLS by default, token at the ingress.

Prior art: PR #451 (open, stale since 2026-06-25)

#451 built authenticated serving and is the right starting point, but it was shaped around constraints that no longer hold, and main has moved under it.

What it landed

• In-process bearer auth in VendorServer: AUTH_TOKEN from the environment, constant-time compare on the HTTP and both WS routes, rejecting before accept(); absent ⇒ open, set-but-empty ⇒ fail closed at startup.
• serve.sh injecting the token via --env-secret AUTH_TOKEN=<secret> (NEBIUS_AUTH_TOKEN_SECRET, default positronic-serverless-inference-token), with an opt-out for e2e.sh.
• .secure_remote / .modal_remote policy configs building auth headers from env vars.
• README + remote-training skill updates, including token rotation.

Why it stalled, and what changed

• "Nebius --auth token gates HTTP but drops the inference WebSocket upgrade" (verified live: /api/v1/session → 404 with a valid token) — this is why auth went in-process. Endpoints are now fronted by a managed HTTPS ingress rather than a raw IP, so this must be re-tested first; it decides the whole design.
• "Bearer token over cleartext" — the one unresolved review thread, left open pending TLS fronting. The managed HTTPS URL is that TLS fronting; the objection resolves itself.
• The AUTH_TOKEN-keyed secret the PR creates is already in --token-secret's expected shape, so ingress auth needs no new secret.

Why it can't just be rebased

• positronic/offboard/vendor_server.py was replaced by positronic/offboard/server.py / PolicyServer in Define a policy as one pipe terminated by a ModelSource #497; the PR's auth hooks and tests target the old file.
• positronic/cfg/policy.py is now remote = cfn.Config(RemotePolicy, url='localhost:8000') — the host/port/secure/headers signature .secure_remote overrode is gone.
• Client transport already landed independently: InferenceClient accepts https/wss URLs and a headers kwarg, RemotePolicy forwards it. --policy.url=https://<managed-url> covers scheme and port on its own, so only credential loading is missing.
• .modal_remote was rejected in review as unrelated to Nebius; drop it unless Modal is still wanted.

Proposed work

1. Verify the WebSocket path first — wss://<managed-url>/api/v1/session against a live --auth token endpoint, with and without the bearer header. Everything below branches on the result.
2. serve.sh: drop --public; add --auth token --token-secret <secret>; read the URL as .status.public_endpoints[] | select(startswith("https://")) instead of [0], and print it as-is (the banner currently prepends http:// to whatever that field holds). Fail loudly if no https:// entry appears rather than falling back to an IP.
3. Auth: if the ingress passes authenticated WS upgrades, that is the whole mechanism — no server-side auth, and PolicyServer stays untouched. If it still drops them, port Authenticate inference endpoints in-process; add secure_remote/modal_remote configs #451's bearer check into PolicyServer (one place, all vendors inherit) keeping the fail-closed-on-empty behaviour, and say plainly in the README why auth is duplicated.
4. Client: a config that attaches Authorization: Bearer from the environment and raises if unset (.secure_remote shape from Authenticate inference endpoints in-process; add secure_remote/modal_remote configs #451, adapted to URL-based remote).
5. Secrets: keep positronic-serverless-inference-token with payload key AUTH_TOKEN; check whether our CLI version still spells it mysterybox (docs now say SecretStash) and align the README. Keep the rotation caveat — an endpoint validates the token it was created with, so re-serve after rotating.
6. Update stop.sh (no IP to release), e2e.sh (smoke check now needs the token unless we keep an open mode), eval.sh, workflows/nebius/README.md, .claude/skills/remote-training/SKILL.md, and the <endpoint-ip>:8000 examples in docs/inference.md / docs/evaluation.md.
7. Migrate the endpoints currently running on public IPs; confirm nothing is left with --public.

Open questions

• Does the managed HTTPS ingress proxy WebSocket upgrades, authenticated and unauthenticated? The docs only cover request/response HTTP. This is the deciding test.
• Is the managed URL stable across stop/start, and across delete/recreate under the same name?
• Added latency vs. the direct IP, per inference round trip — and whether the ingress imposes an idle or request timeout that interacts with our long-lived sessions and the --idle_timeout_min watchdog.
• Does e2e.sh keep an unauthenticated mode, or does the smoke check get a token too?
• Does --token-secret accept the same selector syntax our --env-secret lines use, on our pinned CLI version?
• Do we keep in-process auth as a fallback at all if the ingress handles WS, and does .modal_remote survive?

Acceptance criteria

• No endpoint created by workflows/nebius/* passes --public or is reachable without a token.
• An unauthenticated request fails on both /api/v1/models and the WS session route; an authenticated positronic-inference run against a live endpoint succeeds end to end over wss://.
• Credentials never appear in a URL or in any doc example.
• Docs and the remote-training skill describe only the managed-URL path.