Skip to content

Add minimal mutant setup#104

Open
mbj wants to merge 1 commit into
thoughtbot:mainfrom
mbj:add/mutant
Open

Add minimal mutant setup#104
mbj wants to merge 1 commit into
thoughtbot:mainfrom
mbj:add/mutant

Conversation

@mbj

@mbj mbj commented Apr 27, 2026

Copy link
Copy Markdown

This configures mutant for in top_secret, it intentionally does not:

  • Add mutant as a CI gate.
  • Attempt to cover any mutation.

This is for after I had a discussion with Luis who invited me to show mutant here ;)

Full mutant run produces this summary:

Mutant environment:
Usage:           opensource
Matcher:         #<Mutant::Matcher::Config subjects: [TopSecret*]>
Integration:     rspec
Jobs:            128
Includes:        ["lib"]
Requires:        ["top_secret"]
Operators:       light
MutationTimeout: 5
Subjects:        61
All-Tests:       116
Available-Tests: 116
Selected-Tests:  116
Tests/Subject:   1.90 avg
Mutations:       1482
Results:         1482
Kills:           1121
Alive:           361
Timeouts:        0
Runtime:         4.91s
Killtime:        101.32s
Efficiency:      2061.66%
Mutations/s:     301.57
Coverage:        75.64%

Note I work on a riddiculous machine (128 threads) will be slower on other computers. And a full pass is only useful for very small projects, regular CI / interactive / agent use would normally go for --since main / --fail-fast.

Claude was so nice to make me this summary on the alive mutations:

  Top alive-mutation hotspots (worth attacking first):

  ┌────────────────────────────────────────┬─────────────────┬──────────────────────────────────────────┐
  │                Subject                 │ Alive (visible) │                   File                   │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text::Result.with_global_labels        │ 86              │ lib/top_secret/text/result.rb:47         │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text::Result.from_messages             │ 50              │ lib/top_secret/text/result.rb:34         │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text::BatchResult.from_messages        │ 48              │ lib/top_secret/text/batch_result.rb:33   │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text.shared_model                      │ 24              │ lib/top_secret/text.rb:21                │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text::BatchResult#initialize           │ 5               │ lib/top_secret/text/batch_result.rb:21   │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text::Result#initialize                │ 3               │ lib/top_secret/text/result.rb:21         │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text::GlobalMapping#process_result     │ 1+              │ lib/top_secret/text/global_mapping.rb:40 │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text::GlobalMapping#build_from_results │ 2               │ lib/top_secret/text/global_mapping.rb:26 │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text#substitute_text                   │ 4               │ lib/top_secret/text.rb                   │
  ├────────────────────────────────────────┼─────────────────┼──────────────────────────────────────────┤
  │ Text.clear_model_cache!                │ 1               │ lib/top_secret/text.rb:38                │
  └────────────────────────────────────────┴─────────────────┴──────────────────────────────────────────┘

  Patterns of weak coverage that jump out:
  - Batch / global-mapping pipeline is the largest gap — most of the 361 alive mutations cluster there. Specs likely
  assert end-to-end shape but not the intermediate transformations (the global-label substitution loop, the
  value_to_label mapping, etc.).
  - Memoization & mutex semantics in Text.shared_model / clear_model_cache! aren't pinned down — the mutex
  synchronization can be removed without breaking any test.
  - Constructor field assignments (@input, @items) — initializers often have alive nil-substitution mutations because
  tests don't read every getter back after construction.
  - Splat usage (*default_filters.keys vs default_filters.keys) — assert_valid_keys happens to accept both shapes, so
  behavior diverges only on edge inputs no test exercises.

  Want me to start with Text::Result.with_global_labels (the biggest single subject), or open a session subject report
  (mutant session subject ...) for the full mutation list on a target?

Most of the alive mutations are "semantically dead code" that can likely just be removed, and many others are a sign of behaviore the code exhibits that probably should not be removed but is "not being pulled into existance by a test that demands that behavior".

I do not expect this to be merged as is will, as I said: Discuss with Luis on this PR ;)

@stevepolitodesign

Copy link
Copy Markdown
Contributor

@louis-antonopoulos tagging you just so I don't forget.

@github-actions

Copy link
Copy Markdown

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions Bot added the Stale label May 28, 2026
@mbj

mbj commented May 28, 2026

Copy link
Copy Markdown
Author

@louis-antonopoulos you want to get this merged? I'm happy to rebase.

@louis-antonopoulos

Copy link
Copy Markdown

I still need to evaluate and talk with @stevepolitodesign first!

@mbj

mbj commented May 28, 2026

Copy link
Copy Markdown
Author

kk, np at least we have defeated the stale bot for the next 30d ;)

@github-actions github-actions Bot removed the Stale label May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants