Integrate BI tests with Runt & CI#282
Open
ngernest wants to merge 16 commits into
Open
Conversation
…nic error messages
Nikil-Shyamsunder
approved these changes
Jul 11, 2026
Collaborator
|
As mentioned in Slack, this is okay but maybe we can find a way to have passing tests share a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses Kevin's comment here in #270, adding BI Runt tests to our testing framework / CI:
Most files changed in this PR are new
.expectfiles containing the BI output, the key code changes to review are:scripts/generate_runt_config.py(I followed Nikil's instructions in Migrate to Runt #252 to add support for Runt tests here. Note that the BI uses the same set of test cases as the monitor, i.e. the BI test suite usesMONITOR_CASESintest_catalog.py. This is possible since BI and monitor share largely the same CLI args.)bi/src/main.rs(described below)There are some minor changes to the
biexecutable in this PR:--color neverso that error messages are printed in plaintext (no color) in.expectfiles, like the monitorbipanics, instead of the default Rust error message:we instead print (and only save in
.expectfiles) the following:This avoids the OS thread ID (the
thread 'main' (11301631) ...information) in the default error message from appearing in the.expectfiles, since the OS thread ID changes every time we run the executable.Two test infra TODOs for future PRs
.expectfile only displays the error code:Ideally, we change this behavior so that monitor error messages are also captured in
.expectfiles (matching what is done forbiin this PR). This is a one-line change ingenerate_runt_configs.py, but also affects a bunch of.expectfiles, so I wanted to defer it to a subsequent PR.axi-lite-s1bug #270 that:To implement this, we have to adapt
generate_runt_configs.pyso that for the Brave New World test cases, one single.protcan correspond to multiple.expectfiles (different output for buggy / fixed waveforms). This is doable, but I'd like to defer this to a future PR since it will modify (remove) a bunch of.expectfiles.