configuration.md

Reference

Configuration

Most users can rely on the default pipeline behavior, but SpectralBridge does expose a small set of runtime controls for execution engine, restart-safe extraction, merge tuning, and Ray diagnostics.

CLI first

Most configuration is surfaced through spectralbridge-pipeline flags rather than through large external config files.

Restart-safe defaults

The pipeline validates outputs and skips good artifacts by default, so reruns are part of the intended workflow.

Ray is optional

The default engine is Ray, but thread and process execution remain available for constrained or debugging environments.

Control surface

Where configuration comes from

1. CLI arguments and Python function parameters
2. A small set of environment variables used mainly for Ray behavior
3. Package defaults in the pipeline and merge helpers

There is not currently a project-wide YAML or TOML runtime configuration file for the main processing workflow.

Core runtime options

Common pipeline controls

--base-folder

Root location for downloaded HDF5 inputs, ENVI products, parquet outputs, merged tables, and QA artefacts.

--engine

Execution backend: ray, thread, or process. Ray remains the default in the NEON CLI path.

--max-workers

Upper bound for concurrent work. Higher values can improve throughput but also increase memory pressure.

--parquet-chunk-size

Controls chunk size for ENVI-to-parquet extraction and is one of the safest tuning knobs when memory is limited.

--merge-memory-limit

Tunes the DuckDB merge stage when parquet joins become memory-heavy.

--merge-temp-directory

Lets the merge stage spill to a specific scratch location instead of relying on the default temp directory.

Environment variables

Supported environment knobs

The current codebase uses a relatively small set of environment variables directly:

Variable	Purpose
CSC_RAY_NUM_CPUS	Override Ray helper CPU selection when you want a different cap than --max-workers.
CSC_RAY_DEBUG	Enable extra Ray diagnostics for troubleshooting initialization and dispatch behavior.
RAY_DISABLE_DASHBOARD	Can override the default suppression of the Ray dashboard if you intentionally want Ray’s dashboard behavior changed.

Older documentation sometimes referenced CSCAL_TMPDIR, CSCAL_LOGLEVEL, or CSCAL_RAY_ADDRESS. Those are not current first-class runtime controls in the present package code.

Sensor and workflow configuration

What is built into the package?

Sensor outputs

Canonical sensor products are defined through path helpers and data tables, including Landsat TM, ETM+, OLI, OLI-2, and the current MicaSense-related products.

Spectral data tables

Band centers, FWHM values, and brightness coefficient tables live under src/spectralbridge/data/ and are part of the package contract.

If you change sensor support or brightness tables, update the code, tests, and naming/output documentation together.

Operational guidance

How to tune safely

• Prefer lowering --max-workers before assuming a scientific stage is broken.
• Lower --parquet-chunk-size if parquet extraction or polygon workflows are memory-bound.
• Point --merge-temp-directory at a fast local scratch disk for large merges.
• Use thread for reproducible first-pass debugging and ray for larger production runs.

Where to go next

Related references

CLI reference See the concrete command patterns for the main entry points. Pipeline stages Understand which stages consume each setting. Validation metrics Review the QA and output checks that make reruns safe.