Skip to content

Integrate BI tests with Runt & CI#282

Open
ngernest wants to merge 16 commits into
mainfrom
bi_test_suite
Open

Integrate BI tests with Runt & CI#282
ngernest wants to merge 16 commits into
mainfrom
bi_test_suite

Conversation

@ngernest

@ngernest ngernest commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

This PR addresses Kevin's comment here in #270, adding BI Runt tests to our testing framework / CI:

Can you make a separate PR for adding the bi to the tests and to CI? Then we can iterate on how to best integrate this with the new testing infrastrucutre.

Most files changed in this PR are new .expect files containing the BI output, the key code changes to review are:

  • scripts/generate_runt_config.py (I followed Nikil's instructions in Migrate to Runt #252 to add support for Runt tests here. Note that the BI uses the same set of test cases as the monitor, i.e. the BI test suite uses MONITOR_CASES in test_catalog.py. This is possible since BI and monitor share largely the same CLI args.)
  • bi/src/main.rs (described below)

There are some minor changes to the bi executable in this PR:

  • Added a new CLI arg --color never so that error messages are printed in plaintext (no color) in .expect files, like the monitor
  • Following the advice here (https://users.rust-lang.org/t/suppress-panic-message/6303/3), I added a panic hook so that when bi panics, instead of the default Rust error message:
thread 'main' (11301631) panicked at bi/src/bi.rs:628:21:
[add_busy_wait@00!] Cannot fork at step zero!
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

we instead print (and only save in .expect files) the following:

panicked at bi/src/bi.rs:628:21:
[add_busy_wait@00!] Cannot fork at step zero!

This avoids the OS thread ID (the thread 'main' (11301631) ... information) in the default error message from appearing in the .expect files, since the OS thread ID changes every time we run the executable.

Two test infra TODOs for future PRs

  1. For the monitor (not BI), when the monitor is expected to fail (e.g. running on a Brave New World buggy waveform), the corresponding .expect file only displays the error code:
---CODE---
124

Ideally, we change this behavior so that monitor error messages are also captured in .expect files (matching what is done for bi in this PR). This is a one-line change in generate_runt_configs.py, but also affects a bunch of .expect files, so I wanted to defer it to a subsequent PR.

  1. Kevin mentioned in (WIP) Use same protocol to both drive DUT + infer transactions for BNW axi-lite-s1 bug #270 that:

You should also try to only have a single .prot for each one of the brave new world benchmarks. After all, our claim is that a single correct protocol can distinguish between buggy and fixed version. I think the reason we used to maintain two copies was because of how turnt test discovery works. With the new testing infrastructure, we should not have to do that anymore.

To implement this, we have to adapt generate_runt_configs.py so that for the Brave New World test cases, one single .prot can correspond to multiple .expect files (different output for buggy / fixed waveforms). This is doable, but I'd like to defer this to a future PR since it will modify (remove) a bunch of .expect files.

@ngernest ngernest marked this pull request as ready for review July 11, 2026 20:10
@Nikil-Shyamsunder

Copy link
Copy Markdown
Collaborator

As mentioned in Slack, this is okay but maybe we can find a way to have passing tests share a .expect between monitor and bi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants