[Experiment] Agentic Triage Workflow #55244
Open
nagilson wants to merge 11 commits into
Open
Conversation
Verbatim copy of microsoft/mxc .github/workflows/issue-triage.md as the starting point for a .NET SDK agentic triage workflow. No modifications yet.
Strip the mxc container/executor label allowlist, the mxc owner logins and area->owner map, and the mxc macOS-backend synonym example. Leave placeholders for the dotnet/sdk labels (commit 2.5) and CODEOWNERS + round-robin owner routing (commits 3-6).
Populate add-labels.allowed with the real dotnet/sdk Area-* component labels and core type labels (Bug/enhancement/Feature Request/question/documentation/Task). Switch the removed lifecycle label from mxc's 'Needs-Triage' to dotnet/sdk's 'untriaged' (the label the policy service auto-adds). Update body guidance to reference Area-* + type labels.
Replace the owner-assignment placeholder with Azure/azure-sdk-for-net's CODEOWNERS owner-lookup and routing sections (their Step 5 + Step 6), copied verbatim with headings demoted to nest under ## Task. Still Azure-specific (ServiceLabel/AzureSdkOwners/ServiceOwners/Service Attention/mention_owners); commit 4 adapts it to how dotnet/sdk CODEOWNERS actually works.
Rewrite the Azure owner-lookup section to match how dotnet/sdk CODEOWNERS works: root-level CODEOWNERS organized into '# Area-<Name>' sections that map directly to Area-* labels, with individual (@login) vs team (@dotnet/team) owners. Assign individuals; route team-only areas via the 'needs team triage' label. Remove all Azure-isms (ServiceLabel/%/AzureSdkOwners/ServiceOwners/Service Attention/needs-team-attention/customer-reported/mention_owners and the Event Hubs/ARM/Mgmt examples).
Insert microsoft/pylance-release's round-robin rotation body verbatim (rotation order, stateful dotnet#4462 pointer, skip-reassign, steps 1-6). Headings demoted to nest under ## Task. Commit 6 makes it stateless and merges it with expertise-based owner routing.
Replace pylance's stateful rotation with a stateless, expertise-first, load-balanced selection: prefer the area's CODEOWNERS individual owner(s), measure each candidate's current open-assigned load via 'repo:... is:open assignee:<login>', and assign the least-loaded; when the SME is overloaded and an alternate owner exists, assign the lighter teammate and name the SME. is:open avoids penalizing tenured members and avoids flooding newer ones. Populate assign-to-user.allowed with the 12 CODEOWNERS individuals and add the 'needs team triage' routing label. No persisted rotation state, no dotnet#4462 pointer, no unassign-from-user.
Teach the triager .NET SDK phrasing -> Area-* mapping (build/restore/publish/pack/test/run/watch/format, MSBuild, NuGet, workloads, templates, trimming/AOT/single-file/R2R, source-build/VMR, SWA/Blazor/Razor). Add special-purpose labels to the allowlist and body guidance: cookie (Copilot-suitable easy issues: small fixes, doc changes, backports, simple tests), Test Debt, performance, dotnetup, breaking-change, good first issue, help wanted, backport. Add a needs-info (needs-more-info) behavior: detect incomplete bug reports, apply needs-info, ask for the specific missing details, and hold owner routing.
Make the description dotnet/sdk-specific; add 'roles: all' so external-contributor issues are processed; switch to 'min-integrity: none' with a justification so external issue content is read (not filtered) while still treated as untrusted data; add github toolsets [issues, labels] and a 'noop' safe output; add a 'Before you start' anti-noise guard (skip already-triaged issues, re-triage after needs-info is answered) and a prompt-injection security note; drop the stale internal commit reference in the label comment.
Add a workflow_dispatch trigger with an issue_number input so the workflow can be run manually against a specific existing issue for testing (in addition to issues opened/edited). Resolve the target issue as github.event.issue.number || github.event.inputs.issue_number, and set target: '*' on the write safe-outputs so a dispatched run acts on the specified issue. Fix compile errors: nest 'roles: all' under 'on:', and drop the fixed add-labels allowlist (dotnet/sdk exceeds the 50-item cap) in favor of an instruction to apply only existing repo labels. Generate issue-triage.lock.yml via 'gh aw compile' and update the pinned actions-lock.json.
The gateway rejected the guard policy with "allow-only.repos string must be all or public" because allowed-repos was a scalar repo string. A specific repo must be a list, so repos compiles to an array the gateway accepts. This is why the agent step failed with nothing listening on 8080 -- the gateway never started.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an agentic workflow to triage newly opened/edited issues in dotnet/sdk, applying labels, requesting missing info, and routing/assigning based on CODEOWNERS, backed by a compiled gh-aw lock workflow.
Changes:
- Introduces a gh-aw Issue Triage workflow definition that guides labeling, needs-info handling, and CODEOWNERS-based routing with load balancing.
- Adds the compiled/pinned workflow (
issue-triage.lock.yml) that runs the Copilot-driven triage under the gh-aw harness. - Updates the gh-aw actions lock to the newer
github/gh-aw-actions/setupversion.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
.github/workflows/issue-triage.md |
New gh-aw workflow spec/prompt defining how to triage issues, apply labels, and route to CODEOWNERS owners. |
.github/workflows/issue-triage.lock.yml |
Generated GitHub Actions workflow executing the agentic triage flow with pinned actions/containers and safe-output handling. |
.github/aw/actions-lock.json |
Bumps pinned github/gh-aw-actions/setup action version/SHA used by gh-aw workflows. |
| mkdir -p /tmp/gh-aw/safeoutputs | ||
| mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs | ||
| cat > "${RUNNER_TEMP}/gh-aw/safeoutputs/config.json" << 'GH_AW_SAFE_OUTPUTS_CONFIG_0931caca8efa60f5_EOF' | ||
| {"add_comment":{"max":1,"target":"*"},"add_labels":{"max":6,"target":"*"},"assign_to_user":{"allowed":["akoeplinger","lbussell","lewing","maraf","MichaelSimons","MiYanni","mthalman","pavelsavara","phil-allen-msft","sujitnayak","tmat","vijayrkn"],"max":2,"target":"*"},"create_report_incomplete_issue":{},"missing_data":{},"missing_tool":{},"noop":{"max":1,"report-as-issue":"true"},"remove_labels":{"allowed":["untriaged"],"target":"*"},"report_incomplete":{}} |
| GH_AW_WORKFLOW_SOURCE_URL: "${{ github.server_url }}/${{ github.repository }}/blob/${{ github.ref_name }}/.github/workflows/issue-triage.md" | ||
| GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | ||
| GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} | ||
| GH_AW_NOOP_REPORT_AS_ISSUE: "true" |
Comment on lines
+1217
to
+1219
| GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} | ||
| GH_AW_MISSING_TOOL_CREATE_ISSUE: "true" | ||
| GH_AW_WORKFLOW_NAME: "Issue Triage" |
Comment on lines
+1231
to
+1235
| env: | ||
| GH_AW_AGENT_OUTPUT: ${{ steps.setup-agent-output-env.outputs.GH_AW_AGENT_OUTPUT }} | ||
| GH_AW_REPORT_INCOMPLETE_CREATE_ISSUE: "true" | ||
| GH_AW_WORKFLOW_NAME: "Issue Triage" | ||
| GH_AW_WORKFLOW_SOURCE_URL: "${{ github.server_url }}/${{ github.repository }}/blob/${{ github.ref_name }}/.github/workflows/issue-triage.md" |
| GH_AW_ALLOWED_DOMAINS: "api.business.githubcopilot.com,api.enterprise.githubcopilot.com,api.github.com,api.githubcopilot.com,api.individual.githubcopilot.com,api.snapcraft.io,archive.ubuntu.com,azure.archive.ubuntu.com,crl.geotrust.com,crl.globalsign.com,crl.identrust.com,crl.sectigo.com,crl.thawte.com,crl.usertrust.com,crl.verisign.com,crl3.digicert.com,crl4.digicert.com,crls.ssl.com,github.com,host.docker.internal,json-schema.org,json.schemastore.org,keyserver.ubuntu.com,ocsp.digicert.com,ocsp.geotrust.com,ocsp.globalsign.com,ocsp.identrust.com,ocsp.sectigo.com,ocsp.ssl.com,ocsp.thawte.com,ocsp.usertrust.com,ocsp.verisign.com,packagecloud.io,packages.cloud.google.com,packages.microsoft.com,ppa.launchpad.net,raw.githubusercontent.com,registry.npmjs.org,s.symcb.com,s.symcd.com,security.ubuntu.com,telemetry.enterprise.githubcopilot.com,ts-crl.ws.symantec.com,ts-ocsp.ws.symantec.com,www.googleapis.com" | ||
| GITHUB_SERVER_URL: ${{ github.server_url }} | ||
| GITHUB_API_URL: ${{ github.api_url }} | ||
| GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG: "{\"add_comment\":{\"max\":1,\"target\":\"*\"},\"add_labels\":{\"max\":6,\"target\":\"*\"},\"assign_to_user\":{\"allowed\":[\"akoeplinger\",\"lbussell\",\"lewing\",\"maraf\",\"MichaelSimons\",\"MiYanni\",\"mthalman\",\"pavelsavara\",\"phil-allen-msft\",\"sujitnayak\",\"tmat\",\"vijayrkn\"],\"max\":2,\"target\":\"*\"},\"create_report_incomplete_issue\":{},\"missing_data\":{},\"missing_tool\":{},\"noop\":{\"max\":1,\"report-as-issue\":\"true\"},\"remove_labels\":{\"allowed\":[\"untriaged\"],\"target\":\"*\"},\"report_incomplete\":{}}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces an agentic triager flow to the .NET SDK repository.
An example run is found in nagilson#14.
Related #55185
Expected Impact
needs-more-infostyle issues to request follow up automaticallyCriteria
needs-more-infolabels on issues that do not need that.codeowners.mdcookielabelFor Reviewers
I looked at a variety of existing triaging skills and imported them directly into the commit history but then made edits on top of them to better suit our scenarios.
I did not find any that meet all of the criteria above being used in production across GitHub. However, I found pieces with the best ones.
Criteria Coverage on Existing Triager Agentic Workflows (✓ = validated live where noted)
needs-triage/untriaged¹ mxc validated on a thin sample (3 issues, all maintainer-filed):
#612assignedindividualsand posted as the bot;#613/#620got no owner (area→owner map is sparse);#613received two triage comments (re-run noise).² Azure validated on
#60838: labeled +Service Attention+needs-team-attentionas the bot, but did not assign an individual³ agentics via typespec
#11227: labels applied, verbose "Agentic Triage" comment, and no owner assignment.Examples of other Triagers (with validated caveats)
issue-triage.md#612)github-actions[bot];min-integrity+ integrity-guard fix; Model Aissue-triage.md(imports agentics)#11227)min-integrity: nonejustified for public-repo external contributorsissue-triage-agent.md#2728was the policy-service (fabricbot), not the AI agent; recent spam (#3014,#2996) sat untriaged → AI triage not reliably running. Group @mention only, no individual assignmentmin-integrity: none+allowed-reposscoped; add-only labels; never closesissue-repro-triage.md(+ quality/report workflows)Needs: Additional Info; reproduces & drafts fix PRs#16234): agentic output posted under a maintainer's personal account (nohwnd), not a bot — write-back uses a personal PAT.min-integrity: none; org GitHub App fallback intended but not in effect hereauto-triage-issues.mdneeds-triagefallback;communitydetection; per-run caps +max-daily-ai-creditsmin-integrity: approved(labeler-only); shows cost/rate-limit knobsmin-integrityvalueallowed-repos