testing-harness.md

Testing Harness

Cross-runtime acceptance is owned by the tri-runtime parity harness (space-data-module parity): one dist/isomorphic/module.wasm, identical behavior in browser / native WasmEdge / Docker WasmEdge, byte-diffed. See docs/tri-runtime-parity.md. "Works in X" is not module evidence.

This repo now includes two related testing/runtime surfaces in space-data-module-sdk/testing:

• the manifest-driven harness generator for contract smoke tests
• the canonical dynamic runtime-host harness for multi-module process and WasmEdge execution

The runtime-host harness is the new baseline for paid-module/server/browser-node execution. Single-module invoke is still supported, but it now sits on top of the same host-controlled process model as a compatibility profile.

Public API

Use the generator when you want contract-level smoke coverage for an arbitrary module or flow manifest:

import {
  generateManifestHarnessPlan,
  materializeHarnessScenario,
} from "space-data-module-sdk/testing";

const plan = generateManifestHarnessPlan({
  manifest,
  preferredWireFormat: "aligned-binary",
  payloadForPort({ methodId, portId, typeRef }) {
    if (methodId === "echo" && portId === "in") {
      console.log("selected wire format:", typeRef?.wireFormat ?? "flatbuffer");
      return "hello";
    }
    return null;
  },
});

const commandCase = materializeHarnessScenario(
  plan.scenarios.find((scenario) => scenario.id === "command:echo"),
);

For actual runtime-host execution, use createModuleHarness(...) or the lower level process clients:

import {
  createModuleHarness,
  createPluginInvokeProcessClient,
  resolveWasmEdgePluginLaunchPlan,
} from "space-data-module-sdk";

const harness = await createModuleHarness({
  runtime: {
    kind: "process",
    hostProfile: "runtime-host",
    command: process.execPath,
    args: ["--input-type=module", "--eval", runtimeHostServerScript],
    modules: [
      { moduleId: "alpha", metadata: { tier: "paid" } },
      { moduleId: "beta", metadata: { tier: "paid" } },
    ],
    defaultModuleId: "alpha",
  },
});

await harness.installModule({ moduleId: "gamma", metadata: { tier: "trial" } });
const modules = await harness.listModules();
const rowHandle = await harness.appendRow({
  schemaFileId: "OMM",
  payload: { noradCatId: 25544 },
});
const region = await harness.allocateRegion({
  layoutId: "propagator-state",
  recordByteLength: 64,
  alignment: 16,
  initialRecords: [new Uint8Array(64)],
});
const response = await harness.invokeModule("beta", {
  methodId: "echo",
  inputs: [{ portId: "request", payload: new TextEncoder().encode("hello") }],
});

The same host control surface is available from createPluginInvokeProcessClient and createWasmEdgeStreamProcessClient:

• installModule(definition)
• listModules()
• unloadModule(moduleId)
• invokeModule(moduleId, request)
• appendRow({ schemaFileId, payload })
• listRows(schemaFileId?)
• resolveRow({ schemaFileId, rowId })
• allocateRegion({ layoutId, recordByteLength, alignment, initialRecords })
• describeRegion(regionId)
• resolveRecord({ regionId, recordIndex })

In-process createRuntimeHost() also exposes registerExternalRegion(...), setRegionRecordCount(...), and resolveRecordView(...) for hosts that already own resident buffers and need transient region descriptors without turning raw addresses into durable harness identity. Those helpers are not part of the current process/WasmEdge control protocol, which remains copy-based.

For outer transport ingest, the runtime-host surface also exposes createFlatBufferStreamIngestor(...). That helper is intentionally outside the module invoke ABI. It accepts little-endian size-prefixed FlatBuffer stream chunks and appends durable rows through createFlatSqlRuntimeStore().

Identity rules remain:

• standards rows are addressed by ($SCHEMA_FILE_ID, rowId)
• aligned runtime records are addressed by (regionId, recordIndex)
• raw pointers remain internal execution details, not durable harness identity

The generator does two things:

1. It derives smoke cases from the manifest shape and invoke surfaces.
2. It classifies declared capabilities by runtime surface:
   • WASI-native
   • sync hostcall
   • async host API

When a port advertises multiple payload wire formats, pass preferredWireFormat to select which declared type ref the generated smoke cases should use. This is useful for contracts that accept regular FlatBuffer inputs but also expose aligned-binary ports or dual-format test fixtures. Aligned-binary entries are expected to ship with a regular FlatBuffer fallback for the same schema in the same accepted type set, and the harness will choose between those declared type refs rather than inventing one.

Flows

Flows are treated as degenerate modules for harness generation. The generator does not require a separate flow-only code path. A flow manifest still produces method-level invoke cases and capability classifications.

For runtime execution, flows should target the same dynamic runtime-host model. Host runtime packages may still compile or bundle flows for deployment, but the harness contract in this repo is now the canonical host surface for:

• installing multiple modules into one host
• wiring host-owned row and region services
• exercising the same control plane in process mode and WasmEdge mode

Runtime Matrix

The runtime matrix lives in test/runtime-matrix.test.js and is intentionally a separate integration script:

npm run test:runtime-matrix

Languages covered

• Node.js: native WASI runner
• Go: native WASI runner via wazero
• Python: native WASI runner via wasmtime
• C#: fallback wrapper around the canonical Node runner
• Java: fallback wrapper around the canonical Node runner
• Rust: fallback wrapper around the canonical Node runner
• Swift: fallback wrapper around the canonical Node runner

Why some languages are wrappers today

../flatbuffers/wasm documents a wider language/runtime ecosystem than this SDK currently maintains natively. The SDK matrix therefore uses:

• native WASI execution where the runtime path is clean and stable in this repo
• a shared Node-based fallback where the language/runtime bridge is still in flux or would otherwise make the test suite fragile

That keeps every supported language in the matrix immediately while preserving a single canonical execution baseline.

Dynamic Runtime-Host Harness

The SDK now treats dynamic host composition as the default server/browser/WasmEdge execution model. The harness supports two execution profiles:

1. Compatibility profile:

   • single module
   • plain invoke(...)
   • same shape as the earlier process harness

2. Runtime-host profile:

   • multiple installed modules in one host
   • explicit module install/list/unload/invoke controls
   • host-owned row services and aligned-binary region services

Process mode can emulate the runtime host with a normal Node child process. For WasmEdge, the runner can start in standalone host mode with:

const plan = resolveWasmEdgePluginLaunchPlan({
  hostProfile: "runtime-host",
  wasmEdgeRunnerBinary: process.env.WASMEDGE_RUNNER_BINARY,
});

That launches the runner with --serve-runtime-host and no preloaded guest module. Modules are then installed over the host control channel.

Covered Surfaces

The matrix and the existing Node host tests together cover:

• method calling through command-mode invoke envelopes
• aligned-binary metadata preservation on command-mode echoes
• stdin/stdout/stderr capture for WASI guests
• args and environment wiring
• filesystem path smoke through WASI/preopen setup
• WASI clock/time smoke
• sync space_data_module_host byte-envelope coverage for random.bytes
• host capability surface classification
• the Node host API suite for HTTP, TCP, UDP, TLS, WebSocket, MQTT, exec, context, crypto, timers, and filesystem semantics

Run the host-only suite directly with:

npm run test:host-surfaces

Run the streamed FlatBuffer ingest checks with:

npm run test:stream-ingest
npm run test:module-stream
npm run benchmark:stream-1gib
npm run benchmark:module-stream-1gib

The 1 GiB path is env-gated and intended for local stress/perf work. It measures either outer transport ingest or chunked resident-module ingest, not one giant SDS $PIV invoke request.

Important Edge Cases

1. Generated harnesses are contract smoke tests, not semantic proofs

The manifest can tell the harness how to invoke a method, but not what the method should do. Generated cases therefore verify ABI compatibility and surface shape. Any module-specific semantics still need scenario fixtures or custom validators.

2. Command mode owns stdout

For command-surface plugin tests, stdout is the invoke response channel. A guest method that writes arbitrary bytes to stdout can corrupt the response envelope. Use stderr for diagnostics in command-mode methods.

3. Direct invoke is not the cross-language baseline today

The matrix uses command-mode invoke for broad portability. Direct invoke still exists and is covered in the Node-based SDK tests, but it requires runtime- specific memory management and export calling that is not yet normalized across every language adapter.

For dynamic runtime-host tests, use the host-control APIs instead of assuming a single command-surface guest is the whole runtime.

Direct invoke is also not the right path for multi-hundred-megabyte or 1 GiB single-request benchmarks. The invoke ABI is still batch-oriented and fully buffered.

If a module needs resident internal state, such as a FlatSQL instance imported inside the SDN module itself, prefer createModuleFlatBufferStreamPump(...) with a persistent direct-surface harness. That keeps the stream chunked and binary while preserving guest state across many small invokes.

4. WASI filesystem behavior is still only smoke-tested cross-runtime

Preopen path handling is not perfectly consistent across all runtimes used by the matrix. The integration suite therefore asserts that filesystem wiring does not crash and that path-based probes run, but the authoritative filesystem semantics remain in the Node host tests.

5. Pipes are partially portable, not universally portable

The outer harness processes use normal OS pipes for subprocess communication. Inside the guest, many runtimes are more reliable with file-backed stdin/stdout/ stderr descriptors than with arbitrary pipe wiring. The matrix currently uses file-backed guest stdio for portability. That covers the portable stdio-style pipe contract, but not arbitrary named-pipe services.

6. Network and most Node-RED-style host services are not WASI-native

Pure WASI does not provide a portable HTTP/TCP/UDP/WebSocket/MQTT/process-exec surface here. Those capabilities exist today on the async host API side, and the repo tests them through the Node host, browser host, and browser module harness suites. Treat the runtime matrix as the portable guest suite and npm run test:host-surfaces as the authoritative host-adapter suite for the default Node-RED-style services.

7. Random entropy is not a stable libc-level cross-runtime assertion here

The capability matrix still marks random as WASI-native in principle, but the libc entry points used by standalone C guests are not consistently portable across every runtime in this matrix. The suite therefore classifies random correctly but does not hard-fail the matrix on a specific libc randomness call.

8. The sync guest hostcall ABI is intentionally narrow

The current space_data_module_host sync ABI is JSON-over-memory and only supports synchronous operations cleanly. That makes these capabilities practical for WASM guests today:

• clock
• schedule_cron
• filesystem.resolvePath

These remain async host API capabilities today from a guest execution perspective:

• network
• http
• tcp
• udp
• tls
• websocket
• mqtt
• timers
• process_exec
• context_read
• context_write
• crypto_hash
• crypto_sign
• crypto_verify
• crypto_encrypt
• crypto_decrypt
• crypto_key_agreement
• crypto_kdf
• ipfs
• protocol_handle
• protocol_dial

Portable-but-narrow today:

• logging via stdout/stderr
• pipe via stdio descriptors only

The harness generator exposes this distinction explicitly so module authors can see which capabilities are portable in the raw guest ABI today and which require the async host adapter path.

That async host adapter path is now a single generic capability boundary:

• NodeHost.invoke(...)
• BrowserHost.invoke(...)
• createRuntimeHost().invoke(...)
• loadModule(...).callHost(...)
• createBrowserModuleHarness(...).callHost(...)

Those entry points all dispatch the same awaited capability ids and operation names. If you override reference behavior, do it with explicit capabilityAdapters keyed by canonical capability id rather than inventing a repo-local host API variant.

This JSON-over-memory note applies only to the current sync space_data_module_host guest ABI. It is not the canonical FlatBuffer stream-ingest contract, which uses direct size-prefixed FlatBuffer frames and keeps payload bytes binary end-to-end.

Strict Standalone WASI Target

When a manifest declares runtimeTargets: ["wasi"], the SDK should interpret that as a stricter statement than "can run inside some wrapper-managed WASM host." It means the artifact is expected to run as a standalone WASI program.

The current SDK enforcement for that target is:

• invokeSurfaces must include command
• capabilities must stay within the current standalone-WASI subset:
  • logging
  • clock
  • random
  • filesystem
  • pipe
• hosted protocols may only use wasi-pipe

Anything else still requires a host adapter, delegated service, or a future raw guest async ABI expansion and therefore should not claim the canonical wasi runtime target yet.

WasmEdge Deployment Target

Use runtimeTargets: ["wasmedge"] for the standard server-side deployment target when the goal is maximum WASI compatibility rather than strict standalone-WASI purity.

That target assumes a WasmEdge environment with socket/TLS extensions available. In this repo’s capability matrix, these families become reasonable guest-side targets even though they are not plain standalone WASI:

• network
• http
• tcp
• udp
• tls

The full Node-RED parity map lives in docs/node-red-default-node-parity.md.