Surface allocated resources to self-sizing task commands (nf-seqera)

[pditommaso]

What & why

Companion client change to seqeralabs/sched#493 — together they fix the qr/v2 request-vs-allocation OOM (seqeralabs/sched#492).

When the Seqera scheduler's optimizer reduces a task's memory allocation below its request, tools that self-size from the requested memory over-commit and get OOM-killed. The motivating case: nf-core BCFTOOLS_SORT renders --max-mem 33177.6M (= task.memory × 0.90) from a 36,864 MiB request, but the container cgroup enforced only 8,704 MiB — bcftools tried to reserve ~29.8 GB in ~8.5 GB and died with exit 255. Affects any self-sizing tool (bcftools/samtools --max-mem, JVM -Xmx, STAR --limitBAMsortRAM, skesa --memory, …).

The command is rendered before the scheduler decides the allocation, so it can't be patched afterward. This change captures the resource-derived command values and turns them into shell placeholders; the scheduler (sched#493) fills them with the allocated value at launch — keeping the memory cost savings while eliminating the OOM.

How it works (Seqera executor only; no re-execution of the user script)

At task materialization:

1. TaskRun retains the live command GString (before it's flattened to a string), on the plain scriptlet path only — a minimal hook so the executor can inspect which command values were interpolated.
2. ResourceInterpolator (new) locates resource-derived values from three signals — the live command GString (exact positions, recursing into nested GStrings), the raw body.source (the interpolation expressions), and the AST valRefs (a task-referenced gate). A value is memory/cpus-derived when its expression references task.memory/task.cpus directly or through one level of local-variable indirection. Each such numeric value is replaced with ${SEQERA_TASK_MEM_n} / ${SEQERA_TASK_CPUS_n}.
3. SeqeraTaskHandler rewrites task.script before the launcher builds .command.sh, ships the bindings on the task, and seeds each placeholder with its as-requested value as an environment default.

The placeholder is a plain shell variable expanded by bash at runtime — .command.sh is never rewritten by the scheduler.

Safety (never worse than today)

The parsed command template is validated against the live GString (literal text must match, ignoring whitespace/escaping) before any attribution, so a mis-identified template cannot produce wrong bindings. The original command is returned unchanged on: a placeholder that would land inside shell single-quotes (where it wouldn't expand), a joint memory+cpus expression, a non-numeric value, or any parse/validation failure. And because the client seeds the placeholders with the request value, backends that don't resolve bindings (or an older scheduler) still produce a valid command identical to today's behavior.

Dependency / merge order

• nf-seqera is bumped to io.seqera:sched-client:0.60.0-SNAPSHOT, which provides the ResourceBinding API added in seqeralabs/sched#493. This PR builds once that artifact is published, so it depends on sched#493 landing first. Draft until then.

Tests

• ResourceInterpolatorTest — 21 cases: direct self-sizing (bcftools, -Xmx giga, STAR bytes), cpus arithmetic, multi-line commands with continuations, two memory refs, double-quote expansion, local-variable indirection (skesa ternary), joint mem+cpus left untouched, and the safety fallbacks (single-quote, template mismatch, unrecognized form, anchored task.memoryUsage).
• SeqeraTaskHandlerTest — prepareLauncher rewrites task.script and captures bindings; no-op when nothing is resource-derived.

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	61715b2
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/6a1f3d56d566dd00080c85e1