Skip to content

ci: per-test GNU/bfs comparison and comment, like uutils sed/grep#726

Merged
sylvestre merged 1 commit into
uutils:mainfrom
sylvestre:ci-per-test-comparison
Jun 13, 2026
Merged

ci: per-test GNU/bfs comparison and comment, like uutils sed/grep#726
sylvestre merged 1 commit into
uutils:mainfrom
sylvestre:ci-per-test-comparison

Conversation

@sylvestre

Copy link
Copy Markdown
Contributor

The old comparison only commented when the PASS/FAIL/ERROR/SKIP counts changed vs main, so it was blind to net-zero changes (one test fixed, one broken). diff-gnu.sh also compared against an unextracted (empty) baseline, flagging every current failure as a regression, and the published baseline could be stale.

Adopt the sed/grep design:

  • build-gnu.sh / build-bfs.sh now emit a per-test JSON summary (findutils-gnu-full-result.json / bfs-full-result.json) listing every test's name + PASS/FAIL/SKIP, via the new gnu_json_result.py / bfs_json_result.py.
  • compare_test_results.py diffs the set of failing test names against a reference, writing a human-readable result file (empty when nothing changed) and exiting non-zero only on new, non-intermittent failures.
  • compat.yml fetches the reference *-full-result.json from the default branch's latest run (dawidd6/action-download-artifact) and uploads a comparison artifact per suite.
  • comment.yml posts a PR comment whenever the comparison is non-empty, i.e. on any per-test improvement or regression regardless of net count.
  • ignore-intermittent.txt lists known-flaky tests, reported but not failing the job.

Remove the superseded compare_gnu_result.py / compare_bfs_result.py / diff-gnu.sh / diff-bfs.sh.

The old comparison only commented when the PASS/FAIL/ERROR/SKIP *counts*
changed vs main, so it was blind to net-zero changes (one test fixed, one
broken). diff-gnu.sh also compared against an unextracted (empty) baseline,
flagging every current failure as a regression, and the published baseline
could be stale.

Adopt the sed/grep design:

- build-gnu.sh / build-bfs.sh now emit a per-test JSON summary
  (findutils-gnu-full-result.json / bfs-full-result.json) listing every
  test's name + PASS/FAIL/SKIP, via the new gnu_json_result.py /
  bfs_json_result.py.
- compare_test_results.py diffs the set of failing test names against a
  reference, writing a human-readable result file (empty when nothing
  changed) and exiting non-zero only on new, non-intermittent failures.
- compat.yml fetches the reference *-full-result.json from the default
  branch's latest run (dawidd6/action-download-artifact) and uploads a
  comparison artifact per suite.
- comment.yml posts a PR comment whenever the comparison is non-empty,
  i.e. on any per-test improvement or regression regardless of net count.
- ignore-intermittent.txt lists known-flaky tests, reported but not
  failing the job.

Remove the superseded compare_gnu_result.py / compare_bfs_result.py /
diff-gnu.sh / diff-bfs.sh.
@codecov

codecov Bot commented Jun 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.86%. Comparing base (5398003) to head (4ae0f1a).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #726   +/-   ##
=======================================
  Coverage   91.85%   91.86%           
=======================================
  Files          35       35           
  Lines        7149     7155    +6     
  Branches      375      375           
=======================================
+ Hits         6567     6573    +6     
  Misses        437      437           
  Partials      145      145           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq

codspeed-hq Bot commented Jun 9, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 20 untouched benchmarks


Comparing sylvestre:ci-per-test-comparison (4ae0f1a) with main (001966f)

Open in CodSpeed

@sylvestre sylvestre marked this pull request as ready for review June 13, 2026 07:10
@sylvestre sylvestre merged commit bb6a5ba into uutils:main Jun 13, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant