Skip to content

[Detail Bug] Test harness: compare_streams reports wrong line number when one output stream ends early #604

Description

@detail-app

Detail Bug Report

https://app.detail.dev/org_89d327b3-b883-4365-b6a3-46b6701342a9/bugs/bug_e789b0b3-2212-41e4-b4af-e42805c579f2

Introduced in #542 by @sgbalogh on Jun 19, 2026

Summary

  • Context: When compare_streams detects that stream A has fewer lines than stream B, it reports "differs at line N" where N is the line count from stream A.
  • Bug: When stream A ends before stream B, the reported line number points to the last matching line (stream A’s line count) rather than the comparison round where the mismatch is detected.
  • Actual vs. expected: Actual: message like stdout differs at line 3 even though line 3 matched in both streams; the real mismatch is the next comparison round where stream B still has content. Expected: the line number should correspond to the comparison round where the mismatch is detected (e.g., differs at line 4 when B has an extra 4th line).
  • Impact: Misleading error messages during determinism/stream comparisons; developers may check the reported line and find it matches, wasting time before realizing the issue is one stream having extra content.

Code with Bug

// sim/src/meta.rs
if read_a > 0 {
    lines += 1; // <-- BUG 🔴 counts only stream A lines; under-reports mismatch position when A ends first
}

Explanation

compare_streams increments lines only when read_a > 0 (i.e., only when stream A produces a line). If stream A ends early but stream B continues, the mismatch is detected on the next comparison round where read_a == 0 and read_b > 0, but lines is not incremented for that round. As a result, the error message uses a line number that refers to the previous (matching) line.

This also produces asymmetric reporting: comparing (A shorter vs B longer) reports a different line number than the reversed comparison, even though the mismatch occurs on the same comparison round.

Recommended Fix

// sim/src/meta.rs
if read_a > 0 || read_b > 0 {
    lines += 1;
}

History

This bug was introduced in commit b00f2c8. The initial implementation incremented lines only when stream A advanced (if read_a > 0 { lines += 1 }), which causes misleading line numbers when stream A ends before stream B.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions