Skip to content

Commit 171290e

Browse files
authored
Add E2E coverage for WSL profile agents (#486)
* Add E2E coverage for WSL profile agents Add environment-gated integration coverage for profile-scoped WSL agent routing, settings hot reload, and authenticated chat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a64788a-aabb-458d-a6d4-878a5cc3f69b * Fix WSL agent discovery in E2E Probe literal agent commands individually so wsl.exe command-line expansion cannot erase shell loop variables and falsely skip native agents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a64788a-aabb-458d-a6d4-878a5cc3f69b * Track WSL profile agents in release checklist Add stable release items for the profile picker, strict WSL routing, and WSL chat, and map the two automated runtime checks into release reports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8a64788a-aabb-458d-a6d4-878a5cc3f69b --------- Copilot-Session: 8a64788a-aabb-458d-a6d4-878a5cc3f69b
1 parent 680f605 commit 171290e

4 files changed

Lines changed: 184 additions & 4 deletions

File tree

doc/release-check-list.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,11 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
9797
- [ ] `C047` `[E2E]` **Session hooks remove works:** Per-CLI remove buttons remove hook state without breaking the Settings page.
9898
- [ ] `C048` `[UT~]` `[E2E]` **Policy lock UI works:** Locked controls are disabled and show the policy message. _(UT: Effective*/IsLocked gates.)_
9999

100+
### Profile Agent pane agent
101+
102+
- [ ] `C237` `[new]` `[UT~]` `[E2E]` **Profile Agent pane agent picker works:** Each profile's Agent pane agent picker lists installed, policy-allowed Windows agents plus agents installed natively in that profile's WSL distro; saving persists `agentPaneBackend`, while an unconfigured profile inherits the global Windows agent. _(#481; UT: `ProfileTests::AgentPaneBackendDefaultsAndInherits`.)_
103+
- [ ] `C238` `[new]` `[E2E]` **Profile WSL agent routing is strict:** Changing a profile to a WSL backend rebuilds its helper, routes the exact selected agent through `wsl:<distro>`, and does not fall back to a Windows-hosted agent. _(#481; E2E: `Feature.WslAgentBackend`.)_
104+
100105
## 2. Agent pane chat
101106

102107
**Feature definition:** The agent pane is a per-tab AI chat pane backed by WTA helper/master and an ACP-capable agent. It should be reusable, able to be hidden, and stable across tab/window operations.
@@ -131,6 +136,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
131136
- [ ] `C057` `[E2E]` `[MANUAL]` **Copilot chat works:** User can send a prompt and Copilot responds successfully.
132137
- [x] `C058` `[UT✓]` `[E2E]` **Copilot missing CLI path works:** Missing Copilot shows actionable setup/auth guidance, not a silent failure. _(UT: `is_cli_available_*` availability check + the auth/setup screen renders `render_auth_sign_in_card` / `render_auth_screen_shows_agent_name`; a missing binary degrades to a guidance screen, not a silent failure. Exercising a truly uninstalled Copilot stays MANUAL.)_
133138
- [ ] `C059` `[E2E]` **Non-Copilot agents chat works:** Each installed+authenticated non-Copilot built-in agent (Claude/Codex/Gemini) connects through its ACP adapter and answers a prompt. _(One consolidated matrix case — all built-in agents share the same agent-pane/ACP path, so per-agent behavioural depth is covered by the Copilot suites.)_
139+
- [ ] `C239` `[new]` `[E2E]` **Profile WSL agent chat works:** An installed and authenticated agent selected by a WSL profile connects through the helper/master architecture from inside that distro and completes a real chat round trip. _(#481; E2E: `Feature.WslAgentBackend`.)_
134140
- [x] `C060` `[UT✓]` `[E2E]` **Agent auth failure works:** Unauthenticated agents show clear login guidance and can recover after sign-in. _(UT: `auth_error_routes_to_signin_not_connection_lost` (AuthRequired → sign-in, not a generic failure) + the in-pane auth screen renders `render_auth_screen_shows_agent_name` / `render_auth_sign_in_card` / `render_auth_checking_with_status_message` (login guidance + post-sign-in checking state). Driving a real sign-out stays MANUAL.)_
135141
- [ ] `C216` `[new]` `[E2E]` **GitHub Enterprise Copilot sign-in works:** On the auth screen, pressing **E** lets the user enter a GHE domain (e.g. `*.ghe.com`) and sign in; the last-used host is remembered and the device-verification URL targets that host. _(#362.)_
136142
- [ ] `C061` `[E2E]` **Agent restart after settings change works:** Changing the selected agent or model restarts/reconnects cleanly.

test/e2e/README.md

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,17 @@ environment. Current status (run on the Store package):
2626
| `Feature.AgentProposedCommand.Tests.ps1` | §2 agent-proposed command Insert/Run into the shell pane (non-autofix chat path) | 2 |
2727
| `Feature.AgentMatrix.Tests.ps1` | §2 non-Copilot built-in agents (Claude/Codex/Gemini) connect+chat through the ACP adapter — ONE consolidated case (Copilot is the in-depth suite); skips when none installed+authed | 1 |
2828
| `Feature.PerTabAgent.Tests.ps1` | C225-C228: `/agent` picker/direct selection, invalid-id safety, per-tab isolation/shared-master reuse, and global-default/override behavior | 6 |
29+
| `Feature.WslAgentBackend.Tests.ps1` | PR #481 profile-scoped WSL agent backend: settings hot reload, helper/master source routing, and authenticated chat | 2 (environment-gated) |
2930
| `Feature.AgentChat.Tests.ps1` / `Feature.AgentPopup.Tests.ps1` | agent chat + `/` popup/menu interaction | 1 + 3 |
3031

31-
**Coverage: all 101 automatable `[E2E]` checklist items are implemented.**
32-
**Test status: 98 feature cases pass + 2 documented skips** (`wta sessions list` is
33-
identity-gated — see `Feature.SessionList.Tests.ps1`); the 101 checklist items map to these
34-
cases plus the deterministic settings/persistence assertions. Remaining
32+
**Coverage: 103 of 104 automatable `[E2E]` checklist items are implemented.**
33+
**Test status: 98 baseline feature cases pass + 2 documented skips** (`wta sessions list` is
34+
identity-gated — see `Feature.SessionList.Tests.ps1`), plus 2 PR #481 WSL-backend cases that
35+
run only when a dev package, runnable distro, and native supported agent are available; the
36+
chat case also requires authentication. The 103 implemented checklist items map to the
37+
baseline cases plus the deterministic settings/persistence assertions. The remaining new
38+
item is the profile Agent pane agent picker UI; it stays explicit E2E work rather than being
39+
falsely credited by the JSON-level runtime tests. Other
3540
environment-dependent items are tracked and auto-skipped when their prerequisite is absent:
3641
**other agent CLIs** (`Feature.AgentMatrix.Tests.ps1` now covers Claude/Codex/Gemini chat,
3742
auth-gated per CLI — each Context runs only when that CLI is installed *and* authenticated,

test/e2e/release-coverage-map.psd1

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,11 @@
3636
# §1 settings
3737
'Model control appears' = 'Model control / model changes apply'
3838
'Model changes apply' = 'Model control / model changes apply'
39+
# PR #481 profile-scoped WSL backend runtime. The profile picker UI is deliberately
40+
# NOT mapped: Feature.WslAgentBackend writes the profile JSON directly and therefore
41+
# cannot prove that the Settings picker renders or saves correctly.
42+
'Profile WSL agent routing is strict' = 'Hot reload routes the profile agent through its WSL distro without host fallback'
43+
'Profile WSL agent chat works' = 'profile-selected WSL agent connects and answers a chat round trip'
3944

4045
# §0 FRE auto-error (on/off both covered by the single off/on test)
4146
'Automatic error detection on' = 'Automatic error detection off/on'
Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
#Requires -Modules @{ ModuleName='Pester'; ModuleVersion='5.0.0' }
2+
# PR #481: a profile can pin its agent pane to a supported ACP agent installed
3+
# inside that profile's WSL distro. The routing assertion is deterministic once
4+
# a native WSL agent is present; the chat assertion additionally requires auth.
5+
6+
BeforeDiscovery {
7+
$devPkg = Get-AppxPackage | Where-Object { $_.PackageFamilyName -like 'IntelligentTerminal_*' }
8+
$script:Ready = [bool](
9+
$devPkg -and
10+
(Get-Command winapp -ErrorAction SilentlyContinue) -and
11+
(Get-Command wsl.exe -ErrorAction SilentlyContinue)
12+
)
13+
}
14+
15+
Describe 'Feature profile-scoped WSL agent backend' -Tag 'Feature' -Skip:(-not $script:Ready) {
16+
BeforeAll {
17+
Import-Module (Join-Path $PSScriptRoot '..\ItE2E\ItE2E.psd1') -Force
18+
19+
$script:app = $null
20+
$script:skipReason = $null
21+
$script:oldAgentPaneId = $null
22+
$script:profileGuid = '{e2e48100-4810-4810-9810-000000000001}'
23+
$script:profileName = 'ITE2E PR481 WSL Agent'
24+
25+
$distroProbe = Invoke-Native -FilePath 'wsl.exe' -Arguments @(
26+
'-e', 'sh', '-lc', 'printf "%s" "${WSL_DISTRO_NAME:-}"'
27+
) -TimeoutSec 45
28+
$script:distro = $distroProbe.StdOut.Trim()
29+
if ($distroProbe.ExitCode -ne 0 -or -not $script:distro) {
30+
$script:skipReason = 'no runnable default WSL distro is available'
31+
return
32+
}
33+
if ($script:distro -match '["\r\n]') {
34+
$script:skipReason = 'the default WSL distro name cannot be represented safely in a test profile'
35+
return
36+
}
37+
38+
# Match the product's source probe: reject Windows executables leaked
39+
# through WSL interop, and require native npx for adapter-backed agents.
40+
# Probe one literal command at a time; wsl.exe's command-line expansion
41+
# can consume loop variables in a compound bash script before bash runs.
42+
$knownAgents = @('copilot', 'gemini', 'opencode', 'claude', 'codex')
43+
$script:agent = $null
44+
foreach ($candidate in $knownAgents) {
45+
$pathProbe = Invoke-Native -FilePath 'wsl.exe' -Arguments @(
46+
'-d', $script:distro, '--', 'sh', '-lc', "command -v $candidate 2>/dev/null"
47+
) -TimeoutSec 15
48+
$path = $pathProbe.StdOut.Trim()
49+
if ($pathProbe.ExitCode -ne 0 -or
50+
-not $path -or
51+
$path -match '(?i)^/mnt/[a-z]/|\.(exe|cmd|bat)$') {
52+
continue
53+
}
54+
55+
if ($candidate -in @('claude', 'codex')) {
56+
$npxProbe = Invoke-Native -FilePath 'wsl.exe' -Arguments @(
57+
'-d', $script:distro, '--', 'sh', '-lc', 'command -v npx 2>/dev/null'
58+
) -TimeoutSec 15
59+
$npxPath = $npxProbe.StdOut.Trim()
60+
if ($npxProbe.ExitCode -ne 0 -or
61+
-not $npxPath -or
62+
$npxPath -match '(?i)^/mnt/[a-z]/|\.(exe|cmd|bat)$') {
63+
continue
64+
}
65+
}
66+
67+
$script:agent = $candidate
68+
break
69+
}
70+
if (-not $script:agent) {
71+
$script:skipReason = "$($script:distro) has no supported native Linux ACP agent"
72+
return
73+
}
74+
75+
$unconfiguredProfile = [pscustomobject][ordered]@{
76+
guid = $script:profileGuid
77+
name = $script:profileName
78+
commandline = "wsl.exe -d `"$($script:distro)`""
79+
}
80+
$unconfiguredProfiles = [pscustomobject][ordered]@{
81+
defaults = [pscustomobject]@{}
82+
list = @($unconfiguredProfile)
83+
}
84+
$script:backend = "wsl:$($script:distro):$($script:agent)"
85+
$script:app = Start-Terminal -Package Dev -PassFre $true -Settings @{
86+
acpAgent = $script:agent
87+
defaultProfile = $script:profileGuid
88+
profiles = $unconfiguredProfiles
89+
}
90+
$oldAgentPane = Get-AgentPaneSession -App $script:app
91+
if ($oldAgentPane) {
92+
$script:oldAgentPaneId = $oldAgentPane.PaneSessionId
93+
}
94+
95+
# Apply the profile backend only after Start-Terminal has captured log
96+
# offsets. This exercises settings hot reload and keeps source-routing
97+
# assertions scoped to this run.
98+
$configuredProfile = [pscustomobject][ordered]@{
99+
guid = $script:profileGuid
100+
name = $script:profileName
101+
commandline = "wsl.exe -d `"$($script:distro)`""
102+
agentPaneBackend = $script:backend
103+
}
104+
$configuredProfiles = [pscustomobject][ordered]@{
105+
defaults = [pscustomobject]@{}
106+
list = @($configuredProfile)
107+
}
108+
Set-WtSetting -App $script:app -Key 'profiles' -Value $configuredProfiles | Out-Null
109+
Open-AgentPane -App $script:app | Out-Null
110+
}
111+
AfterAll {
112+
if ($script:app) {
113+
Stop-Terminal -App $script:app
114+
}
115+
}
116+
117+
It 'Hot reload routes the profile agent through its WSL distro without host fallback' {
118+
if ($script:skipReason) {
119+
Set-ItResult -Skipped -Because $script:skipReason
120+
return
121+
}
122+
123+
$settings = Get-WtSettingsObject -App $script:app
124+
$profile = @($settings.profiles.list) |
125+
Where-Object { $_.guid -eq $script:profileGuid } |
126+
Select-Object -First 1
127+
$profile.agentPaneBackend | Should -Be $script:backend
128+
129+
if ($script:oldAgentPaneId) {
130+
(Test-Until -TimeoutSec 30 -IntervalSec 0.5 -Condition {
131+
-not (Get-AgentPaneSession -App $script:app -PaneSessionId $script:oldAgentPaneId)
132+
}) | Should -BeTrue -Because 'changing this profile backend must replace its old host helper'
133+
}
134+
135+
$agentPattern = [regex]::Escape($script:agent)
136+
$sourcePattern = [regex]::Escape("wsl:$($script:distro)")
137+
Assert-Log -App $script:app -Name 'wta-main_master.log' -Pattern (
138+
'resolving agent CLI for helper.*requested_agent_id=Some\("' + $agentPattern +
139+
'"\).*resolved_agent_source=' + $sourcePattern
140+
) -TimeoutSec 60
141+
Assert-Log -App $script:app -Name 'wta-main_master.log' -Pattern (
142+
'agent CLI spawned.*agent_source=' + $sourcePattern
143+
) -TimeoutSec 60
144+
}
145+
146+
It 'The profile-selected WSL agent connects and answers a chat round trip' {
147+
if ($script:skipReason) {
148+
Set-ItResult -Skipped -Because $script:skipReason
149+
return
150+
}
151+
if (-not (Wait-AgentReady -App $script:app -TimeoutSec 150)) {
152+
Set-ItResult -Skipped -Because "$($script:agent) is installed in $($script:distro) but is not authenticated or its ACP server did not connect"
153+
return
154+
}
155+
156+
$oldAgentPaneIds = @($script:oldAgentPaneId) | Where-Object { $_ }
157+
$agentPane = Wait-NewAgentPaneSession -App $script:app `
158+
-ExcludePaneSessionId $oldAgentPaneIds -TimeoutSec 30
159+
Send-AgentPrompt -App $script:app -PaneSessionId $agentPane.PaneSessionId `
160+
-Text 'What is 480 plus 1? Reply with only the number.' | Out-Null
161+
Assert-AgentPaneText -App $script:app -PaneSessionId $agentPane.PaneSessionId `
162+
-Pattern '\b481\b' -TimeoutSec 150
163+
}
164+
}

0 commit comments

Comments
 (0)