NSI — Neural Substrate Interface

A complete, working, real-time I/O system for wet electrophysiology, in Rust — the "narrow waist" between any neural rig and any tool. One typed, time-synchronized acquisition stream plus a symmetric stimulation envelope that any device can lower into and any tool can read from, so integrations are m + n adapters instead of m × n.

It runs end-to-end on real Neuropixels data (384 channels @ 30 kHz, IBL/DANDI), std-only, offline, with zero heap allocation on the hot path.

acquire ──▶ band-pass + common-median-reference ──▶ spike detection
   │                                                      │
   ├─ emulator (realistic spiking)                        ▼
   ├─ file replay  (SpikeGLX .bin / Open Ephys / Zarr)   online spike sorting
   ├─ real Intan RHX over TCP                             │   (SpikeInterface-readable .npz)
   └─ (HAL: one trait, any vendor)                        ▼
                                          closed-loop stim ──▶ storage (Zarr / Open Ephys)
                                                              └▶ live web dashboard (HTTP+SSE)

Why it's interesting (the engineering)

• Vendor-agnostic narrow waist. A single typed wire format + a stable driver contract (Rust trait and a #[repr(C)] C-ABI vtable). New hardware = one driver; nothing above changes.
• Hard real-time, lock-free. SPSC lanes, a loaned-message shared-memory pool, buffer-granular processing (process_chunk, never per-sample). A counting-allocator test proves 0 alloc / 0 realloc on the on_frame hot path.
• Honest time-synchronization. Per-frame clock-domain + a surfaced uncertainty bound — software sync never pretends to hide on-device delay.
• std-only, offline, zero external crates. The HTTP+SSE dashboard, the JSON, the Zarr v2 store, the Open Ephys + SpikeGLX readers, and all the DSP are hand-rolled on std.
• Conformance-as-contract. A driver is valid iff it passes the suite (seq monotonicity + gap reporting, capability honesty, zero-alloc hot path, closed-loop equivalence).

Quick start

cargo build --workspace            # offline, std-only, 0 warnings

# End-to-end demo: synthesize a recording -> replay -> filter+CMR+detect
# -> sort -> closed-loop stim -> Zarr/Open-Ephys store, then verify the round-trip.
cargo run --release -p nsi-cli -- demo

# Live web dashboard on the realistic emulator (open the printed URL).
cargo run --release -p nsi-cli -- monitor --source emu --channels 8 --rate 30000 --port 8080 --sort

# Real data, no hardware: replay a SpikeGLX/Neuropixels recording (geometry from the .meta).
cargo run --release -p nsi-cli -- sort --source bin:/path/to/recording.ap.bin --out sorting.npz

nsid subcommands: record · replay · process · sort · loop · monitor · info · demo · intan-selftest.

Verified results

• Runs on real IBL Neuropixels (sub-CSHL045, 384 ch @ 30 kHz, streamed from DANDI): acquire → CMR → detect → sort, live in the dashboard.
• Benchmark (emulator → runtime → writer, 1024 ch @ 20 kHz): sustained ~78 frames/s ≈ 147 GB/h, low p99 jitter, and 0 alloc / 0 realloc on the steady-state hot path.
• cargo test --workspace: 100+ unit & integration tests green, 0 warnings; the gated conformance suite (--features nsi-conformance/track-alloc) is green.
• Sorting output is a SpikeInterface-readable NpzSortingExtractor .npz; storage is a Zarr v2 store / Open Ephys binary, both re-readable.

Crate map (all implemented)

Crate	Role
nsi-core	Locked wire types (64-byte POD AcquisitionFrame, StimulationCommand), NeuralDevice/FrameSink traits, capability & clock models, errors.
nsi-abi	#[repr(C)] driver vtable + C-ABI for out-of-process vendor .so/.dll drivers.
nsi-runtime	Lock-free SPSC lanes, loaned-message SHM pool, clock-sync service, dispatch.
nsi-dsp	RBJ band-pass biquad, common-median reference, threshold spike detector.
nsi-sort	Online template-matching spike sorter + batch sort; SpikeInterface .npz output.
nsi-store	Zarr v2 + Open Ephys binary writers/readers; raw SpikeGLX .bin reader.
nsi-driver-emu	Realistic synthetic spiking generator (baseline + sparse biphasic spikes).
nsi-driver-file	File-replay device (Zarr / Open Ephys / SpikeGLX .bin), real-time paced.
nsi-driver-intan	Real Intan RHX TCP-client driver + an in-crate mock RHX server for tests.
nsi-loop	Closed-loop controller (spike → StimulationCommand) with latency tracking.
nsi-monitor	Live dashboard: a hand-rolled std HTTP+SSE server + a self-contained canvas UI.
nsi-cli	The nsid binary tying it all together.
nsi-conformance	The conformance suite — the suite is the contract.
nsi-bench	Std-only throughput/jitter/allocation benchmark.

Design

The normative design — the four waist primitives, the 64-byte frame layout, the C-ABI driver contract, the clock/sync model, versioning, and conformance — is in docs/DESIGN.md.

Status

v0 — a working prototype that proves the architecture end-to-end on real data. It is not production-grade: the spike sorter is a simple online template matcher, the real-time pipeline is single-threaded (compute-bound past ~100 channels at real time), and the live hardware driver is verified against a faithful mock of the documented Intan RHX protocol rather than on a physical rig. The bones — vendor-agnostic narrow waist, lock-free real-time core, honest sync, conformance-as-contract — are sound and consistently respected.

License

MIT — see LICENSE.