Skip to content

cloud-deployment.md

Latest commit

 

History

History
437 lines (351 loc) · 18.4 KB

cloud-deployment.md

File metadata and controls

437 lines (351 loc) · 18.4 KB
Raw

Cloud deployment

This document is the operator runbook for the cloud-mode topology (iterion server + iterion runner + Mongo + NATS JetStream + S3). It covers prerequisites, secret + token lifecycle, NetworkPolicy egress, observability, resume, and migration from a filesystem store.

The Helm chart is published to GHCR as an OCI artifact at oci://ghcr.io/socialgouv/charts/iterion (sources in charts/iterion/). It renders the full stack; values-dev.yaml bundles in-cluster Mongo / NATS / MinIO for smoke tests, values-prod.yaml expects external dependencies.

Install

helm install iterion oci://ghcr.io/socialgouv/charts/iterion \
  --version <semver> \
  --namespace iterion --create-namespace \
  -f values.yaml

Pick <semver> from the iterion releases; the chart version is kept in lock-step with the binary appVersion, so helm install --version 0.5.3 deploys the iterion 0.5.3 image.

OCI registries do not expose a helm search repo index; to inspect a chart before installing, pull it explicitly:

helm pull oci://ghcr.io/socialgouv/charts/iterion --version <semver>
tar -tzf iterion-<semver>.tgz | head

For chart hacking against unreleased changes, install from a checkout: helm install iterion ./charts/iterion -f values.yaml. task chart:kind exercises this path end-to-end on a kind cluster.

Topology

flowchart LR
  CLIENT["client"] -- "POST /api/runs" --> SERVER["server<br/>(cloud)"]
  SERVER -- "publish<br/>(NATS JetStream)" --> NATS[("NATS JetStream")]
  NATS --> RUNNER["runner<br/>(pool)"]
  RUNNER -- "AppendEvent" --> MONGO[("Mongo")]
  RUNNER -- "NATS-KV lease<br/>+ S3 artifacts" --> S3[("S3")]
  MONGO -- "change-stream<br/>events" --> WSCLIENT["WS client"]
Loading

For the fuller control-plane / data-plane view, see cloud-architecture.md.

  • server publishes RunMessages onto JetStream and serves the studio + run console (REST + WebSocket).
  • runner pulls RunMessages, claims a NATS-KV lease, executes the workflow, and writes events + artifacts to Mongo + S3.

Prerequisites

Component Requirement
Kubernetes 1.28+ for context.WithoutCancel semantics + native Probe.gRPC (optional)
CNI NetworkPolicy enforcement enabled (Calico, Cilium, Antrea) when networkPolicy.enabled=true
MongoDB 6.0+ with replica set (change-streams require an oplog)
NATS 2.10+ with JetStream enabled
S3-compatible bucket pre-created with s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject for the IAM principal
Valkey / Redis (optional) required only for multi-replica servers — shares per-pod state across replicas (see below); a single server replica runs without it
KEDA (optional) 2.13+ if runner.keda.enabled=true
Prometheus Operator (optional) for metrics.podMonitor.enabled=true

Auth bundle and access tokens

Every cloud server requires an auth bundle at boot:

Env var Purpose Generate with
ITERION_JWT_SECRET Server-side HS256 signing key for short-lived access JWTs (at least 32 random bytes) openssl rand -base64 48
ITERION_SECRETS_KEY AES-256-GCM master key for sealing BYOK, OAuth, and run-scoped credentials (exactly 32 bytes before base64) openssl rand -base64 32

Without those values, cloud-mode validation aborts with an explicit error (use ITERION_DISABLE_AUTH=true only for local smoke tests, not for shared deployments). The server pods need ITERION_JWT_SECRET; both server and runner pods must agree on ITERION_SECRETS_KEY so runners can unseal the credential bundle attached to each run.

Generate + apply the Secret:

kubectl create secret generic iterion-auth \
  --from-literal=ITERION_JWT_SECRET="$(openssl rand -base64 48)" \
  --from-literal=ITERION_SECRETS_KEY="$(openssl rand -base64 32)" \
  --from-literal=ITERION_BOOTSTRAP_ADMIN_EMAIL=ops@example.com \
  --namespace iterion

Reference it from values-prod.yaml:

secrets:
  auth:
    existingSecret: iterion-auth

On the first boot of an empty users collection, ITERION_BOOTSTRAP_ADMIN_EMAIL creates a super-admin account with a one-time password printed in the server logs. Capture that password, sign in, change it, and remove the bootstrap env var on the next deploy.

API clients do not send a static deployment token. They authenticate with an access JWT issued by login/refresh, passed as Authorization: Bearer <access-jwt> or via the iterion_auth cookie. WebSocket clients that cannot set headers may pass the same access JWT as ?t=<access-jwt> on /api/ws/*. Health probes, server info, and auth bootstrap routes remain public.

For rotation details, including JWT signing-key rotation and ITERION_SECRETS_KEY impact, see cloud-admin.md.

Queue connection (NATS JetStream)

Cloud mode routes runs through a NATS JetStream queue: the server publishes RunMessages, the runner pool pulls them. ITERION_NATS_URL is required when ITERION_MODE=cloud — the server refuses to start with ITERION_NATS_URL required when mode=cloud otherwise. The stream / bucket / DLQ names and the JetStream tuning knobs have working defaults (pkg/queue/nats/nats.go); override them only to match an existing cluster (a 0 on a numeric/duration knob inherits the default).

Env var Purpose
ITERION_NATS_URL JetStream connection string (nats://[user:pass@]host:4222) — required in cloud mode
ITERION_NATS_STREAM Runs stream name (default ITERION_RUNS)
ITERION_NATS_KV_BUCKET Per-run distributed-lease KV bucket (default iterion-run-locks)
ITERION_NATS_DLQ_STREAM Dead-letter stream for max-deliver-exhausted messages (default ITERION_RUNS_DLQ)
ITERION_NATS_MAX_ACK_PENDING Fleet-wide in-flight (delivered-unacked) ceiling on the durable consumer
ITERION_NATS_MAX_DELIVER Redelivery budget before a message parks on the DLQ (default 8)
ITERION_NATS_ACK_WAIT Per-message ack deadline, refreshed by runner heartbeats
ITERION_NATS_MAX_AGE / ITERION_NATS_DLQ_MAX_AGE Runs-stream / DLQ retention
ITERION_NATS_MAX_PAYLOAD Max message size

The queue's internal semantics (the MaxAckPending fleet ceiling, AckWait heartbeats, DLQ parking, the orphan sweeper) are covered in cloud-architecture.md § Queue internals.

Shared replica state (Valkey / Redis)

Some server state is per-pod and must be shared when you run more than one server replica: forge OAuth/CSRF/manifest-install state, board-MCP run tokens, and auth rate-limit buckets. Configure a Valkey/Redis backend and every replica reads/writes the same store; leave it unset and the server falls back to in-memory implementations (correct for a single replica, but an OAuth callback or rate-limit check can then land on a pod that never saw the paired request).

Env var Purpose
ITERION_REDIS_URL Single-node connection string (redis://[:pass@]host:port[/db]) — dev/local topology
ITERION_REDIS_SENTINEL_ADDRS Comma-separated Sentinel endpoints for the HA failover topology (wins over ITERION_REDIS_URL when set)
ITERION_REDIS_MASTER_NAME Sentinel-monitored master name (required with ITERION_REDIS_SENTINEL_ADDRS)
ITERION_REDIS_PASSWORD Password for the data nodes
ITERION_REDIS_SENTINEL_PASSWORD Password for the Sentinels (defaults to ITERION_REDIS_PASSWORD)

A Valkey outage degrades gracefully — each operation is bounded by a short round-trip timeout rather than blocking the request path.

NetworkPolicy egress

values-prod.yaml ships with networkPolicy.enabled=true + an empty networkPolicy.egress.extraAllow so the cluster default-denies egress except DNS. Add explicit rules for Mongo, NATS, S3, and the LLM provider (the allowlist is nested under egress):

networkPolicy:
  enabled: true
  egress:
    extraAllow:
      # In-cluster Mongo (same namespace)
      - to:
          - podSelector:
              matchLabels:
                app.kubernetes.io/name: mongodb
        ports:
          - protocol: TCP
            port: 27017
      # External LLM provider (Anthropic)
      - to:
          - ipBlock:
              cidr: 0.0.0.0/0
        ports:
          - protocol: TCP
            port: 443

The chart synthesises a single egress block from the union of defaults + egress.extraAllow. There is no auto-detection of bundled sub-charts; if you also bundle Mongo via mongodb.enabled, add the matching egress.extraAllow entry.

NATS monitoring endpoint (KEDA)

KEDA's NATS JetStream scaler scrapes /jsz on the monitoring port (8222 by default), not the client URL. The chart helper iterion.nats.monitoringEndpoint resolves to:

  1. .Values.config.nats.monitoringEndpoint if set, else
  2. <release>-nats:8222 for bundled NATS, else fails.

For external NATS:

config:
  nats:
    url: nats://nats.shared:4222          # JetStream client port
    monitoringEndpoint: nats.shared:8222  # /jsz scrape

Metrics & dashboards

The server + runner expose /metrics on :9090 (configurable via server.metricsPort / runner.metricsPort, or the ITERION_METRICS_PORT env var). Counters/gauges are documented at pkg/cloud/metrics/metrics.go and populated at runtime:

Metric Pod Meaning
iterion_runs_created_total{status} server Every Launch/Resume publish
iterion_runs_active{status="running"} runner Sum across pods = in-flight runs
iterion_run_duration_seconds{status} runner Histogram, terminal status
iterion_ws_connections server Live run-console subscribers
iterion_mongo_change_stream_lag_seconds server Set on each delivered event
iterion_nats_pending_messages runner Polled every 15s from JetStream consumer
iterion_llm_tokens_total{backend,model,direction} runner input/output/cache_read/cache_write
iterion_llm_cost_usd_total{backend,model} runner Added per claw-priced call; unknown models leave the counter untouched
iterion_runner_heartbeat_errors_total runner Each KV lease refresh failure

Wire a Prometheus PodMonitor:

metrics:
  podMonitor:
    enabled: true
    interval: 30s

/metrics is ClusterIP-only by design — no ingress should expose it publicly.

Tracing

The server + runner emit OpenTelemetry spans:

  • iterion.api.launch_run, iterion.api.resume_run (server)
  • iterion.runner.process_one (runner, root span per run)
  • iterion.node.execute (engine, child span per node)

Trace context propagates through the W3C traceparent header on the NATS RunMessage so a single trace covers client → server → queue → runner → node graph.

Configure the OTLP exporter via standard env vars:

config:
  extraEnv:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://tempo.observability:4318"
    OTEL_SERVICE_NAMESPACE: "iterion"
    OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=prod"

When OTEL_EXPORTER_OTLP_ENDPOINT is unset, spans are dropped and the W3C propagator-only path is installed (inbound trace context still respected, but no export).

Pod turnover & in-flight runs

A runner pod goes away for three reasons, and a deploy is the rarest of them:

  • autoscaling — with KEDA enabled the pool scales down whenever the queue drains. That is continuous, it happens under normal operation, and the ReplicaSet picks a victim without knowing which pods are busy. Measured in production on 2026-08-01: SuccessfulRescale … reason: All metrics below target killed a pod 21 minutes into a campaign, with idle pods available. A PodDisruptionBudget does not cover this — a scale-down deletes pods directly rather than evicting them, so it never consults the PDB. The drain below is what does.
  • node turnover — a drain, an upgrade, a spot reclaim.
  • a deploy — a rolling restart of the runner Deployment.

All three arrive as the same signal, SIGTERM, so one mechanism covers them all. What happens to a run a runner is executing is governed by config.runner.drainMode:

  • complete (default — lame-duck): on SIGTERM the runner stops claiming new runs but lets its in-flight run finish before exiting. New pods (already up) serve new runs; the draining pod holds its NATS KV lease so nothing double-claims. A deploy interrupts nothing — the run runs to completion, even if it takes hours. The bound is runner.terminationGracePeriodSeconds (the k8s hard stop before SIGKILL) and config.runner.drainTimeout (the internal ceiling, default 8h): a run exceeding it is capped — checkpointed and auto-resumed on another pod.
  • interrupt: on SIGTERM the runner cancels its in-flight run immediately, checkpoints it, and it auto-resumes on a healthy pod from the last completed node. The fast path for an urgent (e.g. security) deploy that must not wait for long runs.

Either way an interrupted run (lame-duck cap, interrupt mode, lost heartbeat, or an eviction the grace window can't cover) is promoted to failed_resumable and redelivered — it auto-resumes with no operator action. Only an operator cancel stays terminal cancelled, and it wins the race: a resumable failure never overwrites it.

Two consequences worth knowing before relying on this:

  • Resume restarts the interrupted node, not the one after it. A node whose side effects had already landed (a push, a posted comment) runs again. That is the standing resume contract (docs/resume.md), now reached automatically rather than by an operator's decision — so a node that must not repeat needs to be idempotent.
  • The runner suppresses its own completion notification for an interrupted run, but the usernotify reconciliation sweep re-derives outcomes from the persisted status every 2 minutes and reads failed_resumable as a failure. If the redelivery is slow (a rolling restart is exactly when it would be), a user can still get one "run failed" push for a run that then resumes silently.

Invariants when raising the lame-duck window:

  • runner.terminationGracePeriodSeconds ≥ config.runner.drainTimeout + a couple of minutes of checkpoint margin (else k8s SIGKILLs a capped run before it checkpoints — it still recovers via the orphan sweeper, just ~10 min slower).
  • runner.progressDeadlineSeconds above the grace period: a lame-duck rollout stays Progressing until the last old pod drains (possibly hours), and the default 600s deadline would otherwise mark the Deployment degraded while the new pods already serve. ArgoCD users: the sync will likewise show Progressing until the drain completes.

Verified in production, 2026-08-01. The same signal, before and after: a KEDA scale-down killed run 019fbd98 21 minutes into a campaign and it came back cancelled; a kubectl delete pod on the runner holding run 019fbdec left the pod Running with a deletionTimestamp 2h05 out — it finished its run (240.8s, status finished) and only then exited.

What a long window costs. A terminating pod keeps its node slot and its resource requests for the whole drain, so:

  • a scale-down→scale-up cycle inside the window (KEDA's cooldown is 60s) can hold well above maxReplicas worth of footprint for hours;
  • kubectl drain and cluster-autoscaler node removal block for up to the grace period per runner pod;
  • two runner generations talk to one Mongo and one NATS for the length of the window, so a change to run-doc shape, checkpoint semantics or event payloads must stay compatible across it — hours, not the 90 seconds this used to be;
  • a genuinely broken rollout takes grace + progressDeadline to be flagged.

Shorten config.runner.drainTimeout (and the grace with it) if any of those matter more than never interrupting a long run.

The mechanism lives in the runner's Shutdown (pkg/runner/loop.go); the run's context is decoupled from the fetch-loop context so stopping intake never cancels a live run.

Resume from a paused / failed run

Cloud-mode resume goes through the same NATS path as launch. The client passes the inline source of the workflow because the server pod has no operator filesystem:

curl -X POST https://iterion.example.com/api/runs/$RUN_ID/resume \
  -H "Authorization: Bearer $ITERION_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source": "'"$(jq -Rs . workflow.bot)"'",
    "answers": {"approved": true},
    "force": false
  }'

force=true bypasses the workflow-hash mismatch guard (useful after a local fix). The runner reads the flag from the RunMessage and applies it to runtime.New(WithForceResume).

Migration from filesystem store

iterion migrate to-cloud uploads runs from a local .iterion/ directory into Mongo + S3. Idempotent (Mongo upserts + S3 PUT overwrites):

ITERION_MONGO_URI=mongodb://...?replicaSet=rs0 \
ITERION_MONGO_DB=iterion \
ITERION_S3_ENDPOINT=https://s3.amazonaws.com \
ITERION_S3_BUCKET=iterion-prod \
ITERION_S3_REGION=eu-west-3 \
  iterion migrate to-cloud --store-dir ./.iterion --concurrency 4 --tenant <tenant-id> --owner <user-id>

Migration flags:

Flag Description
--store-dir <path> Filesystem .iterion/ store to migrate from (default .iterion).
--config <path> YAML config file for Mongo/S3 settings; environment variables take precedence.
--dry-run Print what would be uploaded without writing to Mongo or S3.
--concurrency <n> Number of parallel run uploads (default 4).
--tenant <id> Tenant ID assigned to migrated runs; required for multitenant cloud deployments.
--owner <id> Optional owner user ID attributed to migrated runs.

Re-run safely if interrupted; runs already in Mongo are no-ops.

Smoke test (task chart:kind)

devbox run -- task chart:kind

Renders + lints the chart, checks appVersion matches package.json. For a real install + workflow exec, see the cloud-e2e CI job in .github/workflows/tests.yml.

Health endpoints

Path Behaviour
/healthz 200 if the HTTP listener is up — covers liveness probe
/readyz Pings Mongo + NATS + S3 with 1s sub-deadline each, 503 on any failure — covers readiness probe

The /readyz JSON response details which dependency is failing so the operator can debug from kubectl describe pod.