test-suite.md

layout	default
title	Developer Test Suite

This page is a developer-focused guide to OpenTestability testing strategy, execution, and debugging.

1. Test suite goals

• protect metric correctness across COP, SCOAP, reconvergence, and TPI paths
• prevent regressions in the Yosys JSON analysis and TPI flows
• keep quick confidence checks available for daily development
• provide deeper end-to-end validation before major merges

2. Test directory layout

tests/
|-- conftest.py
|-- fixtures/
|-- unit/
|   |-- core/
|   |-- parsers/
|   |-- testpoint/
|   |-- utils/
|   `-- yosys/
|-- integration/
|   `-- yosys_mode/
|-- system/
`-- benchmark/

Typical intent:

• unit/: fast isolated checks for functions and small modules
• integration/: behavior across module boundaries and mode workflows
• system/: full end-to-end flows against known benchmark designs
• benchmark/: pytest-benchmark performance harnesses (engines, TPI), run separately from the correctness suite

3. Marker map

Marker	Purpose	Typical command
smoke	fastest confidence checks	pytest tests/ -m smoke -v
slow	large-design tests (>30s); excluded by default via addopts in pytest.ini	pytest tests/ -m slow -v
integration	cross-module behavior	pytest tests/ -m integration -v
system	end-to-end flow validation	pytest tests/ -m system -v
yosys	yosys mode specific tests	pytest tests/ -m yosys -v

There is no unit marker — pytest tests/ -m unit -v silently selects zero tests. Unit tests are selected by path instead: pytest tests/unit/ -v.

4. Running tests locally

Linux/macOS workflow (on Windows, use WSL; or nix develop for a one-command shell):

cd <path-to>/OpenTestability
source venv/bin/activate

# All tests
pytest tests/ -v

# Fast loop during development
pytest tests/ -m smoke -v
pytest tests/unit/ -v

# Yosys-mode validation
pytest tests/ -m yosys -v

Useful targeted runs:

pytest tests/unit/utils/test_logging_config.py -v
pytest tests/unit/utils/test_session_manager.py -v
pytest tests/unit/testpoint/ -v
pytest tests/integration/test_verbose_logging.py -v
pytest tests/system/test_s27.py -v
pytest tests/system/test_tpi_quality.py -v

5. Recommended workflow for new changes

1. add or update unit tests close to changed logic
2. add at least one integration test for the affected flow
3. run the smoke marker and tests/unit/ first
4. run full affected marker class (integration or system) before merge
5. run the full integration suite when touching shared code

6. Fixtures and shared setup

• tests/conftest.py holds shared fixtures and common setup hooks
• tests/fixtures/ stores reusable test data and helpers
• keep fixtures deterministic and small so CI and local runs stay stable

7. Reports and artifacts during testing

Depending on test type and verbosity:

• pytest summary is printed to terminal
• .pytest_cache/ stores local test cache metadata
• verbose OpenTest runs can generate logs in results/log/
• verbose OpenTest runs can generate reports in results/reports/
• metrics or generated netlists may appear in data/results/, data/TPI/, or output/

When writing tests, prefer temporary paths/fixtures to avoid polluting shared output directories.

8. Common failure patterns

• invalid input: the netlist is not valid Yosys JSON
• missing environment setup: virtualenv not activated or dependency missing
• path assumptions: hardcoded paths instead of project utilities
• flaky output comparisons: assertions tied to non-deterministic ordering or timestamps

9. Developer PR checklist

• tests added/updated for changed behavior
• relevant marker subsets pass locally
• no regressions in both modes for shared-code changes
• docs updated for user-visible behavior changes
• generated artifacts are either ignored or intentionally tracked