`agent-scan` false positive: #36085 graded `automated` despite disclosed AI assistance and full proof of work

[valentinpalkovic]

cc @huang-julien @Sidnioulz

The issue:
Our PR triage treats the agent-scan:* label as the single source of truth for AI-generated handling and deliberately never second-guesses it.
That contract only works if disagreements between the scan and the PR's actual content flow back to the scanner.
Here is the first clearly documented one.

#36085 (author: @ethanstoner, new contributor) was labeled agent-scan:automated by the workflow on 2026-08-30.
Against that grade, the PR body contains:

• both required template sections, filled in
• a 4-step "Manual testing" section with the exact sandbox commands to reproduce
• the real pre-fix failing-test output (snapshot mismatch, quoted verbatim)
• the exact commands that were run
• an explicit AI-disclosure statement, exactly as CONTRIBUTING asks for:

Per CONTRIBUTING's AI policy: this PR was written with AI assistance (Claude). A real person (me) is behind it, has reviewed the diff, and will respond to review comments.

A disclosed AI-assisted session is what agent-scan:mixed exists to describe, and triage explicitly does not treat mixed as AI-generated.
I would argue this one should have been graded mixed.

The cost:
Because of the automated grade, the author received the proof-of-work verification request, got the verification:pending label, and landed in the "Human verification" board column - for proof the PR already contained.
Not a great first-contribution experience.

Questions:

1. Is this tunable on our side (workflow config), or does it need to go upstream to MatteoGabriele/agentscan-action (we pin v1.8.0)?
2. Can we use this issue as the standing collection point for future scan-vs-content disagreements from triage? Triage already records them in its run reports; they just had nowhere to go until now.

Happy to pull more cases from the triage run reports if useful.

WDYT?

[valentinpalkovic]

Two more datapoints, from the 2026-09-02 sweep. These point in the opposite direction than the original report: the scan graded disclosed AI assistance as human instead of mixed.

PR	Author	Scan grade	Signal it contradicts
#36145	theRizwan	agent-scan:human	explicit AI-assistance disclosure in the body
#36146	theRizwan	agent-scan:human	identical disclosure sentence

Both bodies contain, word for word:

Claude Code assisted with the investigation and the patch, as CONTRIBUTING.md asks contributors to disclose. I reviewed and tested every change.

That is exactly the disclosed AI-assisted session that mixed exists to describe, in the wording CONTRIBUTING.md asks for. I would argue both should have been graded mixed.

The cost: none for triage this time. human and mixed are both treated as not AI-generated, so no handling changed and nothing was blocked, unlike the #36085 case above. It matters for the stats, though: a contributor who follows our own disclosure instruction gets classified as if they had not disclosed, so the mixed bucket is undercounting exactly the population it is meant to describe.

A guess at the mechanism: both PRs are pretty high quality. #36145 ships a type-level test that fails on next today, and #36146 ships two regression tests plus a fallback design that was refined during review. So human is a defensible read of the quality signal - it is the disclosure signal that was missed. If the grader weighs "does this look like unreviewed AI output" over "does the body disclose assistance", that would explain both misses. I might be wrong, though - take it with a grain of salt.

[huang-julien]

@valentinpalkovic this is not tunable at the moment. We use Agent Scan, @unveil/identity is the package that compute those data.

---

Regarding @ethanstoner, the account is still flagged as automation by agent scan AND there has been multiple reports from maintainers in the community:

The account leverages AI to produce nearly everything, from PR descriptions to follow-up responses for maintainers. Additionally, the PR merge numbers on their CV suggest they care more about stats farming than genuine contribution.

Giving full proof of work once doesn't mean the account doesn't automate. Agent scan consider multiple metric to classify an account as automated. From numbers of PRs, forks to time activity.
Automation does not mean no human behind the account, but human using AI to automate PRs.

Also, accounts that are community flagged will be forever flagged in agent scan until report and proof.

---

Regarding theRizwan, agent scan does not read data from the PR itself, it scans data from GH activity, meaning it watches over someone's behavior in the ecosystem rather than within a single PR.

You may be interested in https://github.com/peakoss/anti-slop in that case for PR specific analysis