Skip to content

Commit 1b2fce9

Browse files
vanzueCopilot
andauthored
test(e2e): expand agent-pane coverage, add release report, fix harness flakiness (#356)
* test(e2e): strengthen agent-pane coverage; add shell-integration & proposed-command suites - Remove redundant fixed Start-Sleep in AutofixPane (rely on poll-based Assert-Pane). - Strengthen /model picker, all-four pane positions, helper-cleanup, real Shift+Enter (new Send-AgentShiftEnter win32-input-mode helper). - Add Feature.ShellIntegration.Tests.ps1 (OSC 133 marks + cmd.exe missing-integration safety). - Add Feature.AgentProposedCommand.Tests.ps1 (non-autofix chat Insert/Run recommendation card). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): align new suites to Get-ItTestPackage package selector Match the post-merge convention so the new ShellIntegration and AgentProposedCommand suites honor ITE2E_PACKAGE instead of hardcoding Store. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): cover §2 view-switch draft-input preservation Add a deterministic SessionList test: a typed-but-unsubmitted draft survives a round-trip through the session view (open + Esc back to chat). No LLM involved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): add multi-CLI agent chat matrix (Claude/Codex/Gemini, auth-gated) Cover §2 Claude/Codex/Gemini chat through the IT agent pane's ACP adapters. Each per-CLI Context runs only when the CLI is installed AND authenticated (print-mode auth probe at discovery), else skips with the reason recorded. Verified live: Claude + Codex pass, Gemini skips (installed but unauthenticated here). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): add release-report generator + autofix matrix (Claude/Codex/Gemini) New-ReleaseReport.ps1 turns the release checklist into a clean human-facing report driven purely by test results: tags (UT/E2E/MANUAL) stripped; [x]=automation passed, 'AUTOMATION FAILED'=test failed, plain [ ]=not covered, verify manually. Mapping is title-substring + a curated override map (release-coverage-map.psd1), conservative by design (unmapped -> manual, never a false [x]). Also extend the agent matrix with a per-CLI autofix case (§3 Autofix with Claude/Codex/Gemini). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): report generator merges multiple results + map refinements - New-ReleaseReport.ps1 accepts multiple -ResultsXml (later overrides earlier per test name), so an isolated re-run of a flaky suite layers onto the full run. - release-coverage-map.psd1: map a few passing tests whose names differ from their checklist titles (Focus hotkey, Model control/changes). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): clear stale IT instances before every launch (fixes new-tab E_FAIL flake) A prior test whose AfterAll/Stop-Terminal didn't run (e.g. a BeforeAll that threw) leaves an IT window behind. The single-instance AUMID launch then hands off to that stale, often half-initialised window (Launched=false), so the harness drives a broken instance where new-tab returns CreateTab E_FAIL (0x80004005); and because the store and dev packages share one per-brand COM CLSID, a stale window of the OTHER package steals wtcli's CoCreateInstance and misroutes every call. Start-Terminal now calls Stop-StaleItInstances first, closing every leftover IT window (store + dev, matched by *IntelligentTerminal* install location only — never the user's stock WT) so each launch is deterministic and freshly-owned. Verified: a simulated leftover is cleaned and the fresh instance's new-tab wsl.exe succeeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): harden WSL-autofix gate + Shift+Enter injection - WSL autofix Describe: wrap the WSL-pane setup in try/catch so a build that can't create a wsl.exe tab via the protocol (stale dev pkg predating OSC 9001 -> CreateTab E_FAIL) SKIPS via the per-It guards instead of failing the Describe in BeforeAll. - Shift+Enter on a live session row: skip if no selectable row; retry the raw win32-input keystroke up to 3x while polling for the view to dismiss (the injection can drop under load). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): report credits UT-verified items + fixes/extends title->test map - Generator: an originally-ticked [x] item is verified by an automated unit test, so credit it as passed (unless a mapped E2E test failed). Unit tests are automation; the human needn't re-verify them. - Map: drop backticks from keys (the report strips them from titles, so backticked keys never matched -> false manual); add /model, Shift+Enter, Autofix-with-Copilot, FRE auto-error on-variants, session-mgmt-choice-persists, packaging/logging name mismatches. - Net on the last full run: 79 -> 104 verified, 156 -> 130 manual. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): make Wait-AgentReady deterministic (poll the real precondition) The agent-pane-readiness flake: Wait-AgentReady matched the helper-log 'acp_initialize' marker, which fires several seconds BEFORE the helper writes its session origin ('recording agent-pane session origin') -> the jsonl that Get-AgentPaneSession reads. So it returned ready too early and the next agent-pane call (Send-AgentKey/Open-SessionList) raced a not-yet-written record and timed out. The agent_status connected/failed event is NOT broadcast to wtcli listen (verified), so events can't be used. Wait-AgentReady now polls Get-AgentPaneSession (the exact precondition every primitive needs: a recorded, running pane session) and returns the instant it resolves — deterministic, not a fixed delay — for both the initial connect and a reconnect after /restart or a settings-driven rebuild (newest running record wins). A logged auth/fatal failure short-circuits. The AgentRestart test now waits for reconnect-readiness after the settings change before driving the menu. Verified: 3/3 consecutive green (the test previously flaked on a 20s timeout). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): judge agent readiness by the connected UI, not the session registry Per review: gating Wait-AgentReady on Get-AgentPaneSession (which reads the agent-pane-sessions.jsonl session registry) is verifying a feature with that same feature — if the registry breaks, the gate false-readies or hangs and masks the bug. Wait-AgentReady now matches the agent pane buffer for the connected input placeholder ('Ask anything, / for commands..'), which the TUI renders ONLY in ConnectionState::Connected (ui/input.rs:62; the connecting/disconnected placeholders are distinct strings). That is the user-visible ground truth of 'ready to chat', independent of the session-tracking feature, and still returns the instant it's observed (deterministic). Auth/fatal log markers short-circuit. Verified: AgentRestart 2/2 green (initial connect + reconnect). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): require full connected placeholder in Wait-AgentReady (Copilot review) Match 'Ask anything … for commands' in order on one line instead of either fragment anywhere in the captured scrollback, so stray transcript/help text can't false-positive the readiness gate. Verified: AgentRestart 2/2 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): isolate proposed-command Insert/Run in fresh terminals (Copilot review) Split the Insert and Run cases into separate Describes, each with its own fresh terminal (matching Feature.AutofixPane). With the shared terminal, a prior card's 'Run command'/'Insert in Terminal' text lingered in the scrollback and could co-occur with the next case's marker (echoed in the prompt) to false-positive the card-readiness check before a fresh card rendered. Verified: both cases 2/2 green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): precise matrix skip reasons + word-boundary assert + Harness comment (Copilot review) - AgentMatrix: report a PRECISE per-agent skip reason (not installed vs installed-but- unauthenticated vs package missing) via Set-ItResult instead of a boolean Context -Skip, so CI shows why; no terminal is launched when skipping. Re-checks package presence in BeforeAll because a script-scoped var from BeforeDiscovery does not persist into the run phase (only the -ForEach data does). - AgentMatrix: chat assertion uses a word-boundary match instead of a bare '7'. - Harness: correct the Stop-StaleItInstances comment - it makes -ColdStart redundant, but -ShowFre still controls whether the FRE overlay is shown. Verified: Claude/Codex chat+autofix pass; Gemini skips with "installed but not authenticated". Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): consolidate non-Copilot agents into one matrix case; rewrite checklist Per design decision: Copilot is the primary agent and its full behaviour (chat, autofix, insert/run, permission, render, slash, sessions) is covered in depth by the copilot-only suites. All built-in agents share the same agent-pane -> helper -> master -> agent-CLI (ACP) path; the only per-agent difference is the spawned command. So we stop re-testing every behaviour per agent. - Feature.AgentMatrix.Tests.ps1: collapsed from a per-agent (Claude/Codex/Gemini) x chat+autofix matrix into ONE consolidated test case that, for each installed+authenticated non-Copilot agent, does a single connect + chat round-trip in its own fresh terminal; skips when none is available. - doc/release-check-list.md: collapsed the per-agent items (Claude/Codex/Gemini chat, autofix, delegate, installed, hook-install; and the custom-agent behavioural items) into single consolidated items, keeping Copilot as the primary and the config/selection/tracking items. Total 235 -> 220 items. - release-coverage-map.psd1 + README updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): add deterministic §9/§10/§4 coverage (CLSID inject, log retention, /sessions) Adds three high-confidence, deterministic E2E cases (no mock/agent variance) plus one mapping fix, expanding genuine release-checklist coverage: - §9 "WT_COM_CLSID is injected": read $env:WT_COM_CLSID back from a shell pane and assert a braced CLSID, proving WT injects protocol discovery into panes. - §10 "Old log cleanup is safe": seed a sentinel in the running version's log dir + a stale other-version dir, restart the build, assert the running version's logs survive and the stale version dir is pruned wholesale. - §4 "Slash command works": /sessions opens the session view (the command-menu path, complementing the existing button path). - §10 "Early startup failures are logged": coverage-map override (the test "...would be logged" already exists and passes). All three new cases validated live against the Store package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): locale-robust agent-readiness gate + strict CLSID assert (Copilot review) - Wait-AgentReady: the connected input placeholder is localized (input.placeholder.connected), so a hard-coded English regex would time out on non-en-US machines. Derive the readiness regex from EVERY bundled wta locale's connected placeholder (Get-AgentConnectedPlaceholderRegex, cached; falls back to the en-US literal outside a repo checkout). Verified it matches the real zh-CN placeholder and rejects the connecting/disconnected strings. - WT_COM_CLSID test: tighten the assertion to a real GUID shape ({8-4-4-4-12} hex) so malformed values like {----} can no longer pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): stable /model assert + actionable agent-matrix skip reason (Copilot review) - /model picker: replace the loose name alternation (auto|claude|gpt|sonnet|opus — "auto" also matches autofix/automatic and drifts with releases) with an assertion on the locale-independent "> " highlight selector that the picker marks its selected row with (model_popup.rs highlight_symbol). Proves a selectable model row actually rendered. Validated live (full suite 13/13). - AgentMatrix: classify each CLI as not-installed / installed-unauthenticated / authed and record every CLI's state in the skip -Because, so CI results are actionable and an "installed but unauthenticated" regression isn't silently collapsed. Validated the classifier distinguishes gemini (unauthenticated) from a missing binary (not-installed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): fix Wait-AgentReady auth fail-fast log probe (Copilot review) The helper logs the typed failure via tracing as `target=failure … class=auth_required` (app.rs) and `non_compliant_auth=true` (failure.rs string-fallback shim), so the old literal substring `agent failure class="auth` never matched — auth-required connects burned the full timeout despite the comment claiming a short-circuit. Match the stable class labels instead (quote/field-order agnostic): auth_required / non_compliant_auth, plus the helper `exiting with error`. Deliberately NOT matching the bare message "agent failure", which also fires for a benign cancel (logged before the is_cancelled early-return). Verified real release helper logs from successful connects contain 0 of these tokens (no false-trip), and the placeholder check runs before the probe each iteration so a slow-but-OK connect still wins. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): locale-robust /model title + assert agent-readiness in restart (Copilot review) - Generalize the locale-bundle reader into Get-WtaLocalizedTextRegex (matches a localization key's value across EVERY bundled wta locale; strips a trailing " (…)" key-hint and ".." run). Get-AgentConnectedPlaceholderRegex now delegates to it. The /model picker assertion uses it for model_picker.title instead of the hardcoded en-US "Select model" (which would fail on non-en-US). Exported the helper so tests can use it. Verified it matches the real zh-CN title. - Feature.AgentRestart: assert the two Wait-AgentReady gates with Should -BeTrue (was | Out-Null) so a failed reconnect fails immediately and attributably, instead of surfacing later as an opaque prompt-assertion timeout. Validated live: Feature.AgentPaneInteraction 13/13, Feature.AgentRestart 2/2. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): remove dead -ColdStart param from Start-Terminal (Copilot review) Stop-StaleItInstances now always enforces cold-start semantics, so the -ColdStart switch was an unused no-op while the doc claimed it was "redundant" — misleading. Removed the parameter and its doc, updated the -ShowFre doc to state a fresh monarch is always started, and dropped the -ColdStart argument from the sole caller (Start-TerminalFre). Validated live: Start-Terminal path (Packaging CLSID 1/1) and the -ShowFre FRE path (Feature.FreFlow 5/5) both green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): throttle readiness log reads, distinct probe-timeout, localize card/anchor selector (Copilot review) Five review fixes: - Wait-AgentReady: throttle the helper-log fail-fast read to every ~2s (UI placeholder still polled at 500ms). Get-ItLogText re-reads the whole appended slice each call and the log grows while connecting, so reading every loop was O(n^2) IO on long waits. - AgentMatrix Get-CliStatus: return a distinct `probe-timeout` status instead of collapsing a hung/changed probe into `installed-unauthenticated`, so CI stays actionable. - AgentProposedCommand (both Insert + Run Describes): the recommendation card buttons ([ Run command ] / Insert in Terminal) are localized — match them across all bundled locales via Get-WtaLocalizedTextRegex (en-US fallback) instead of hardcoded English. - /model selector assertion: anchor the "> " highlight marker to line start (allowing a leading popup border │/║/|) so a stray "->" in chat output can't false-positive. Validated: Feature.AgentPaneInteraction 13/13 live (anchored selector against the real bordered picker); card regex + anchored selector + parse all checked offline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): robust YAML scalar parse + ensure report output dir (Copilot review) - Get-WtaLocalizedTextRegex previously only matched double-quoted YAML values, so any unquoted (or single-quoted) locale entry would yield $null and silently fall back to the en-US literal. Now handles double-quoted (ignoring trailing # comment / Locked hint), single-quoted (with '' escaping), and bare scalars (trailing # comment stripped). (Note: the current bundle happens to double-quote all four keys we read, so there is no behavior change today — this hardens against future unquoted entries.) Verified it still matches en-US/zh placeholder, en-US/de run-button, en-US title, and extracts an unquoted scalar correctly. - New-ReleaseReport.ps1: create the -OutFile parent directory before Set-Content, so running with a custom path (or without a prior Invoke-ItE2EReport) no longer throws. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): resolve per-package COM CLSID + co-location asserts + matrix timeout (local-build run) Running the suite against the locally-built Dev package surfaced four failures; all are test harness/assertion issues, not product regressions: - ROOT CAUSE FIX (Paths.ps1): Resolve-WtComClsid blindly probed every brand CLSID and took the first responder. With BOTH the Store (Release brand) and Dev sideload installed, it probed Release first, COM cold-launched the window-less Store server (whose list-windows returns an empty-but-non-null array), and latched onto the WRONG CLSID -> every later wtcli call hit a server with no windows (GetActivePane 0x80004005). Now we map each known package family to its brand (Store=Release, Dev=Dev) and probe ONLY that brand, so Dev binds to its own {D5B7C9E1...} server and works even with Store installed. Verified live: Dev resolves its own CLSID and active-pane succeeds with Store still present — no more uninstall dance. - Feature.Packaging (2 cases): the "co-located wta.exe / wtcli" assertions hardcoded a 'WindowsApps' path, which only holds for an installed Store package; a Dev F5/run-from-layout build resolves binaries under bin\x64\Debug\AppX. Assert co-location under the package InstallLocation (covers Store AND Dev) and explicitly reject the unpackaged tools\wta\target fallback — preserving the original intent on both. Verified for Store and Dev app objects. - Feature.AgentMatrix: bumped the non-Copilot answer assert 90s -> 150s. These agents answer via the npx ACP adapter (extra hop + remote model latency); 90s was demonstrably too tight and the consolidated external-CLI case flaked on a slow turn (passes at ~143s). Not masking a bug — the agent does answer; the window just needs to fit a realistic adapter+model latency. All four previously-failing cases verified green individually against the local Dev build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): harden autofix card (skip on LLM-explain), make Open-SessionList idempotent, drop wrong Shift+Enter case Hardening the two flaky cases the full Dev run flagged: - Autofix card-render + AI-oracle: when the LLM returns an explanation (not a runnable-fix card) for ALL retried typos, that's model variance, not a product failure — SKIP with a clear reason instead of failing, matching the existing WSL-autofix variance skip. The AI-oracle case also skips when no card was produced. - Open-SessionList: made idempotent. SessionToggleButton is a TOGGLE, so a single press could CLOSE an already-open view (or race a transition), leaving us in chat → render-wait timeout. Now it checks-then-toggles (returns immediately if already shown) and retries, fixing the "session view did not render" flake. - Removed the "Shift+Enter on a live session row" E2E case: its premise was WRONG. It asserted the session view dismisses back to chat, but Shift+Enter on a Live row dispatches FocusPane (wtcli focus-pane → move WT focus to that session's pane); it does NOT close the view (verified live + by docs). The contract is deterministically covered by the Rust unit test shift_enter_on_class_a_live_row_focuses (7/7 shift_enter unit tests green). Focus-pane semantics aren't stably observable in E2E — the MVP picker shows only Class B shell sessions whose panes are usually already closed (Focus → NotFound). Updated the checklist item to [UT✓]-only, removed the coverage-map E2E mapping, and adjusted README counts (90→89). Validated live on the local Dev build: autofix pair + agent-restart all green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): require BOTH card buttons + clarify Stop-StaleItInstances ownership (Copilot review) - Feature.AgentProposedCommand (both Insert + Run Describes): the recommendation-card readiness predicate matched an OR of the Run/Insert labels, contradicting the comment that it waits for BOTH actions and risking driving the wrong UI state. Split into CardRunRegex + CardInsertRegex (each localized across all bundled locales, en-US fallback) and require BOTH plus the marker. Verified: a full "[ Run command ] Insert in Terminal" card matches; a Run-only line does not. - Stop-StaleItInstances doc: state plainly that it unconditionally kills ALL running IT windows (store + dev) — not just crashed-test leftovers but also a manually-started window and even the IT window hosting the current shell if tests are run from IT — so the harness owns every IT window during a run. Stock Windows Terminal (Microsoft.WindowsTerminal_*) is never matched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): preserve fully-parenthesized locale values + locale-robust session-view detection (Copilot review) - Get-WtaLocalizedTextRegex: the trailing key-hint strip used `\s*\([^)]*\)\s*$`, which also matched a value that is ENTIRELY parenthesized (e.g. agents.footer_hint "(↑ ↓ … Esc to exit …)"), reducing it to an empty string -> $null regex. Require whitespace BEFORE the hint (`\s+\(`) so only true trailing hints (e.g. "Select model (↑ ↓ …)") are stripped while fully-parenthesized values survive. Verified footer_hint now yields a non-null regex; model-title strip and placeholder matching unchanged. - Open-SessionList / Test-SessionListShown: detected the rendered session view with hard-coded English substrings ("to launch session|to exit|navigate"), which the agents view localizes via agents.footer_hint -> would never match on non-en-US machines. Added Get-SessionViewRenderRegex (the footer hint is drawn in BOTH the loading and populated branches per ui/agents_view.rs render_footer_hint, so it's the reliable open-signal) matched across all bundled locales, en-US fallback. Live-validated: Session button / refresh / /sessions all green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): assert deterministic cmd errorlevel (ITERR=9009) for cmd.exe safety (Copilot review) The cmd.exe safety assertion matched "not recognized|...|nonexistentcmd" — the nonexistentcmd alternative matches the ECHOED input line even if nothing executed, and "not recognized" is localized. Switched to the deterministic, locale-robust cmd errorlevel signal as suggested. Implementation note: the suggested single-line `badcmd & echo ITERR=%errorlevel%` does NOT work — cmd expands %errorlevel% at PARSE time, before the bad command runs, so it captures the OLD value (verified live: prints ITERR=0). The echo must be a SEPARATE command to observe the failure's 9009, so the test now sends the bad command + Enter, then `echo ITERR=%errorlevel%` + Enter, and asserts ITERR=9009 (verified live: prints ITERR=9009; the echoed line shows "%errorlevel%" literally, so the match only hits the expanded output — also proving Enter executed the command). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): Wait-AgentReady fail-fast scans newest helper log in full (Copilot review) The fail-fast auth/fatal scan used Get-ItLogText -SinceStart, which reads only bytes appended after Initialize-LogOffsets (captured at the END of Start-Terminal). But the helper is pre-warmed during tab init, so an auth/fatal connect failure can be logged BEFORE that offset — -SinceStart would miss it and Wait-AgentReady would burn the full timeout instead of returning $false fast. Now it reads the NEWEST wta-main_helper-*.log file in full (via a FileShare.ReadWrite stream, matching Get-ItLogText, since the helper holds it open for writing). The newest file is this launch's helper (fresh PID → fresh file, because Stop-StaleItInstances killed any prior terminal), so reading from the top catches a pre-offset failure without false-matching a previous run's log. Verified live: the authed happy path still returns ready (no false positive). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): dispose helper-log StreamReader + require popup border in /model selector (Copilot review) - Wait-AgentReady: dispose the StreamReader used to read the newest helper log (it also disposes the underlying FileStream), instead of leaking it. - /model picker assertion: the selector regex allowed the "> " marker with no border prefix, so a Markdown blockquote ("> …") in the prior chat transcript could false-positive. The picker draws its highlight symbol INSIDE the popup's left border (verified live: "│> ● Claude Sonnet 4.6 …"), so require a border char (│/║/|) immediately before the marker: ^\s*[│║|]\s*>\s+\S. Verified the real picker row matches while "> some text" and "foo -> bar" are rejected; full Feature.AgentPaneInteraction suite 13/13 green live. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): drop misleading tools/wta/target check from wtcli co-location assert (Copilot review) The wtcli negative-path line `Should -Not -Match 'tools/wta/target'` was copy-pasted from the wta.exe case — but wtcli is never resolved from tools/wta/target. Its non-packaged fallbacks are a PATH-resolved standalone (Get-Command wtcli) or bin/x64/{Debug,Release}/wtcli/wtcli.exe (cli_channel.rs), none under the InstallLocation. The preceding -BeLike "InstallLocation*" already fully guarantees co-location, so the extra check was redundant and misleading — removed it and documented why (the tools/wta/target check stays in the wta.exe case where it's correct). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): drop unneeded copilot pin from ShellIntegration + rename wtcli co-location test (Copilot review) - Feature.ShellIntegration: removed the acpAgent='copilot' pin. The suite is deterministic (OSC 133 marks + cmd.exe safety) and never opens the agent pane, and BeforeDiscovery gates only on the IT package — not copilot — so pinning the agent added an unnecessary copilot install/auth dependency (the pre-warmed helper is irrelevant here). Keeps autoFixEnabled=$false. Verified the two cases still pass without the pin. - Feature.Packaging: renamed "Wrong unpackaged WTA is not used" → "Packaged wtcli is co-located in the package (no unpackaged fallback)" so the name matches what it asserts (wtcli co-location). Added a release-coverage-map entry so the checklist item "Wrong unpackaged WTA is not used" still maps (to the wta.exe not-tools\wta\target case + the wtcli co-location case together). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): match structured auth-failure fields in Wait-AgentReady fail-fast (Copilot review) The fail-fast scan matched bare `auth_required` / `non_compliant_auth` anywhere in the helper log, which could false-trigger if those tokens appear in an unrelated field/value (→ Wait-AgentReady returns $false even though the agent might still connect). Tightened to the STRUCTURED tracing fields: `class="?auth_required` (app.rs; the &str value may be quoted, so the quote is optional) and `non_compliant_auth=true` (failure.rs), plus `exiting with error`. Verified it still matches both quoted/unquoted class=auth_required, non_compliant_auth=true, and the helper exit, while rejecting a bare "auth_required" mention and non_compliant_auth=false. This only narrows the match, so the authed happy path (zero matches) is unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): localize autofix card detection + dispose FileStream if reader ctor throws (Copilot review) - Feature.AutofixPane: CardShown and the two other card-detection predicates (lines 30/102/271) hard-coded the English "Run command|Insert in Terminal" labels, so they'd mis-skip/fail on non-en-US machines. Added an exported Get-RecommendationCardRegex helper (EITHER button label, localized across all bundled locales via Get-WtaLocalizedTextRegex, en-US fallback) and routed all three through it. Verified it matches the en-US card line; the variance-skip path still works live. - Wait-AgentReady: if the StreamReader ctor throws, $fs was left undisposed (file-handle leak). Wrapped the reader in a nested try/finally so $fs is always disposed (double-dispose after the reader closes it is a safe no-op). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): unescape YAML double-quoted escapes in Get-WtaLocalizedTextRegex (Copilot review) Double-quoted YAML scalars were captured raw, so a value containing \" (e.g. setup.subtitle.* "Your agent \"%{agent}\" …") kept the literal backslashes — the generated regex then looked for backslashes absent from the rendered UI text, breaking locale-robust assertions for such keys. Now the double-quoted branch unescapes \" \\ \n \t \r (\x -> x) before the value is regex-escaped. Verified: setup.subtitle.copilot_missing no longer yields a regex containing \" (the escape is resolved to a literal "), while the keys the tests actually use (no backslashes) are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): assert Wait-AgentReady in autofix/proposed-command BeforeAll (Copilot review) Four BeforeAll blocks piped Wait-AgentReady to Out-Null, discarding its boolean — so an auth/fatal connect failure would proceed in a not-ready state and surface later as opaque card-polling failures. Assert | Should -BeTrue with a clear -Because in all four (AutofixPane card-render + AutofixPane WSL setup, AgentProposedCommand Insert + Run), so a failed/again-auth connect fails immediately and attributably. (The WSL one is inside the best-effort try/catch, so a readiness failure there is logged and degrades to a skip via the existing per-It $wslShell guards.) Verified live: the Insert BeforeAll assertion passes when copilot connects. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): truncation-robust session-view detection + pane-scoped failure mark (Copilot review) - Get-SessionViewRenderRegex: matched the FULL localized agents.footer_hint, but the TUI end-truncates that hint to the pane width (agents_view.rs render_footer_hint -> trunc), so the full line may never appear and Open/Test-SessionListShown could time out. Every bundled locale leads the hint with the invariant nav arrows "↑ ↓" (en "(↑ ↓ to navigate …)", zh "(↑ ↓ 导航 …)"), and being at the start they survive truncation — so match those (en-US footer words kept as an extra fallback). Verified live: the rendered footer matches; the slash-/sessions path is green. - Feature.ShellIntegration failure-mark test: Wait-WtCommandFailure listened to the global vt_sequence stream, so an unrelated OSC 133;D mark could satisfy it. The event's `pane_id` equals the pane session_id (Get-ActivePane.session_id), whereas its `tab_id` is a GUID and Get-ActivePane/Get-WtTabs expose tab_id only as a numeric INDEX — so added a -PaneId filter to Wait-WtCommandFailure and scoped the assertion to the active pane. Verified live: passes scoped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): localize last autofix Run-card check + drop redundant English reconnect probe (Copilot review) - Feature.AutofixPane Run-action: the card-detection predicate still hard-coded the English "Run command" label (missed in the earlier sweep). Routed it through Get-RecommendationCardRegex like the other card-detection sites so it's locale-robust. - Feature.AgentRestart: removed the post-/restart `Test-Until … -match 'Ask anything|Copilot|Agent'` reconnect probe — it matched hard-coded English (not locale-robust) and was redundant with the Wait-AgentReady | Should -BeTrue gate immediately after, which is the deterministic reconnect-and-ready signal. Verified live: the restart case still passes (37s). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(e2e): tighten Non-Copilot-chat coverage-map regex to not match Copilot restart (Copilot review) The override regex 'connects and answers' for "Non-Copilot agents chat works" also matched the Copilot restart test name "(/restart reconnects and answers)" — "reconnects and answers" contains "connects and answers" — which could credit the checklist item from the wrong test in the report. Anchored on "non-Copilot agent.*connects and answers" so it uniquely matches the AgentMatrix case. Verified: matches the AgentMatrix name, does NOT match the Copilot restart name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 18bc4d8 commit 1b2fce9

19 files changed

Lines changed: 958 additions & 136 deletions

doc/release-check-list.md

Lines changed: 7 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
4848

4949
- [ ] `[UT~]` `[E2E]` **Copilot without install:** Copilot appears as an available/default choice, is labeled as needing install, and the setup path installs or clearly explains how to install it. _(UT: registry/policy filter.)_
5050
- [ ] `[UT~]` `[E2E]` **Copilot preinstalled:** Copilot appears as installed; saving does not reinstall unnecessarily; opening the agent pane uses Copilot successfully.
51-
- [ ] `[UT~]` `[E2E]` **Claude installed:** Claude appears only when available; selecting it saves correctly; Node/npx requirement guidance appears when relevant.
52-
- [ ] `[UT~]` `[E2E]` **Codex installed:** Codex appears only when available; selecting it saves correctly; Node/npx requirement guidance appears when relevant.
53-
- [ ] `[UT~]` `[E2E]` **Gemini installed:** Gemini appears only when available; selecting it saves correctly and can connect in agent-pane mode.
51+
- [ ] `[UT~]` `[E2E]` **Non-Copilot agents appear when installed:** Claude/Codex/Gemini appear as selectable only when installed; selecting one saves correctly and can connect in agent-pane mode; Node/npx requirement guidance appears when relevant.
5452
- [ ] `[UT~]` `[E2E]` **Unavailable non-Copilot agents:** Claude/Codex/Gemini that are not installed do not appear as broken selectable options.
5553
- [ ] `[UT✓]` `[E2E]` **Agent selection persists:** The selected agent remains selected after FRE completion and app restart. _(UT: `BuiltInAcpAgentRoundtrips`.)_
5654

@@ -127,9 +125,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
127125

128126
- [ ] `[E2E]` `[MANUAL]` **Copilot chat works:** User can send a prompt and Copilot responds successfully.
129127
- [ ] `[UT~]` `[E2E]` **Copilot missing CLI path works:** Missing Copilot shows actionable setup/auth guidance, not a silent failure. _(UT: registry install hint.)_
130-
- [ ] `[E2E]` `[MANUAL]` **Claude chat works:** User can send a prompt and Claude responds successfully through the ACP adapter.
131-
- [ ] `[E2E]` `[MANUAL]` **Codex chat works:** User can send a prompt and Codex responds successfully through the ACP adapter.
132-
- [ ] `[E2E]` `[MANUAL]` **Gemini chat works:** User can send a prompt and Gemini responds successfully.
128+
- [ ] `[E2E]` **Non-Copilot agents chat works:** Each installed+authenticated non-Copilot built-in agent (Claude/Codex/Gemini) connects through its ACP adapter and answers a prompt. _(One consolidated matrix case — all built-in agents share the same agent-pane/ACP path, so per-agent behavioural depth is covered by the Copilot suites.)_
133129
- [ ] `[UT~]` `[E2E]` **Agent auth failure works:** Unauthenticated agents show clear login guidance and can recover after sign-in. _(UT: `AgentFailure::AuthRequired` classification; in-pane auth screen render via `render_auth_screen_shows_agent_name` / `render_auth_sign_in_card` / `render_auth_checking_with_status_message`.)_
134130
- [ ] `[E2E]` **Agent restart after settings change works:** Changing the selected agent or model restarts/reconnects cleanly.
135131

@@ -193,10 +189,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
193189
- [ ] `[UT✓]` `[E2E]` **Reject/dismiss works:** User can dismiss an autofix suggestion without side effects. _(UT: `trigger_echo_pane_clears_when_state_returns_to_idle`.)_
194190
- [ ] `[UT✓]` `[E2E]` **Autofix target pane is correct:** Failure in one pane does not offer/run a fix in the wrong pane. _(UT: target-tab routing — busy-pane tests + `autofix_still_triggers_for_non_agent_pane`.)_
195191
- [ ] `[E2E]` `[MANUAL]` **Autofix with Copilot works:** Copilot returns a useful suggestion.
196-
- [ ] `[E2E]` `[MANUAL]` **Autofix with Claude works:** Claude returns a useful suggestion.
197-
- [ ] `[E2E]` `[MANUAL]` **Autofix with Codex works:** Codex returns a useful suggestion.
198-
- [ ] `[E2E]` `[MANUAL]` **Autofix with Gemini works:** Gemini returns a useful suggestion.
199-
- [ ] `[E2E]` **Autofix with custom ACP agent works:** Custom agent-pane command can receive autofix prompts and respond.
192+
- [ ] `[E2E]` **Autofix with non-Copilot agents works:** Autofix produces a usable suggestion with a non-Copilot built-in agent (Claude/Codex/Gemini) and a custom ACP agent — same path as Copilot, covered once across the available agents.
200193

201194
### Autofix across layout changes
202195

@@ -237,7 +230,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
237230
- [ ] `[UT✓]` `[E2E]` **Restore old agent-pane session:** Supported agent-pane sessions resume through agent-pane/session-load path when enabled. _(UT: `ResumeInAgentPane` decision.)_
238231
- [ ] `[UT✓]` `[E2E]` **Unsupported restore is clear:** Unknown CLI, missing resume support, or missing on-disk session shows a clear not-resumable message. _(UT: `NotResumable` reasons.)_
239232
- [ ] `[UT✓]` `[E2E]` **Enter behavior works:** Enter performs the expected focus/resume action.
240-
- [ ] `[UT✓]` `[E2E]` **Shift+Enter behavior works:** Shift+Enter performs the alternate resume path for dead sessions and same focus path for live sessions. _(UT: `decide_enter_action` shift.)_
233+
- [ ] `[UT✓]` **Shift+Enter behavior works:** Shift+Enter performs the alternate resume path for dead sessions and same focus path for live sessions. _(UT: `decide_enter_action` shift + `shift_enter_on_class_a_live_row_focuses`. Not E2E: Live-row Shift+Enter dispatches FocusPane — moves WT focus to the session's pane, it does NOT dismiss the view — and the MVP picker shows only Class B shell sessions whose panes are usually already closed, so there is no stable E2E observable.)_
241234

242235
### Session-management scope and custom agents
243236

@@ -259,9 +252,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
259252
- [ ] `[E2E]` **Command palette prompt launches delegate:** Typing a request and pressing Enter creates a delegate task.
260253
- [ ] `[E2E]` **Command palette cancel is safe:** Esc/cancel closes the palette without launching a delegate.
261254
- [ ] `[E2E]` `[MANUAL]` **Delegate with Copilot works:** Copilot delegate task starts and responds.
262-
- [ ] `[E2E]` `[MANUAL]` **Delegate with Claude works:** Claude delegate task starts and responds if supported by delegate mode.
263-
- [ ] `[E2E]` `[MANUAL]` **Delegate with Codex works:** Codex delegate task starts and responds if supported by delegate mode.
264-
- [ ] `[E2E]` `[MANUAL]` **Delegate with Gemini works:** Gemini delegate task starts and responds if supported by delegate mode.
255+
- [ ] `[E2E]` `[MANUAL]` **Delegate with non-Copilot agents works:** Claude/Codex/Gemini delegate tasks start and respond where supported by delegate mode.
265256
- [ ] `[UT~]` `[E2E]` **Delegate errors are actionable:** Missing CLI/auth errors are clear.
266257

267258
## 6. Custom agents
@@ -276,10 +267,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
276267
- [ ] `[UT✓]` `[E2E]` **Edit custom ACP agent:** Editing updates the command used by new agent panes.
277268
- [ ] `[UT~]` `[E2E]` **Delete custom ACP agent:** Deleting returns to a valid built-in/default selection.
278269
- [ ] `[UT~]` `[E2E]` **Model selection visible:** Model picker/textbox remains visible when custom agent is selected.
279-
- [ ] `[E2E]` **Custom direct chat works:** Agent pane can talk to the custom ACP agent.
280-
- [ ] `[E2E]` **Custom command request works:** Custom agent can request a command/tool action and the UI handles it.
281-
- [ ] `[E2E]` **Custom insert/run works:** Insert into pane and run in pane work with custom agent requests.
282-
- [ ] `[E2E]` **Custom autofix works:** Autofix can use the custom ACP agent when configured as the agent-pane provider.
270+
- [ ] `[E2E]` **Custom agent runs the standard agent-pane behaviours:** A configured custom ACP agent can chat, request a command/tool action, insert/run into the pane, and drive autofix — the same agent-pane behaviours verified in depth with Copilot.
283271
- [ ] `[UT~]` `[E2E]` **Custom failure is safe:** Bad command, missing executable, or non-ACP behavior shows a clear error and does not crash Terminal. _(UT: failure classification.)_
284272

285273
### Custom delegate agent
@@ -312,10 +300,7 @@ Net effect: UT shrinks the manual matrix to "did the wiring and UI connect", not
312300

313301
- [ ] `[E2E]` **Install hooks from FRE works:** Session-management toggle can install supported hooks during first run.
314302
- [ ] `[E2E]` **Install hooks from Settings works:** Install hooks button works after FRE.
315-
- [ ] `[E2E]` **Copilot hook install works:** Copilot hook is installed or reports why it cannot be installed.
316-
- [ ] `[E2E]` **Claude hook install works:** Claude hook is installed or reports why it cannot be installed.
317-
- [ ] `[E2E]` **Gemini hook install works:** Gemini hook is installed or reports why it cannot be installed.
318-
- [ ] `[E2E]` **Codex hook behavior is understood:** Codex hook/session support is tested according to the current implementation.
303+
- [ ] `[E2E]` **Per-CLI hook install works:** Each supported CLI (Copilot/Claude/Gemini) installs its hook or reports why it can't; Codex hook/session support behaves per the current implementation.
319304
- [ ] `[E2E]` **Hook remove works:** Removing a hook disables future session tracking for that CLI.
320305
- [ ] `[UT✓]` `[E2E]` **Disabled plugin is respected:** Disabled agent plugin is skipped and not force-enabled. _(UT: `decide_skip_when_disabled`.)_
321306
- [ ] `[UT✓]` `[E2E]` **Hook auto-upgrade works:** After package upgrade, previously installed hooks are updated silently when bundle version changes. _(UT: `decide_upgrade` + `upgrade_state` round-trip.)_

test/e2e/ItE2E/ItE2E.psm1

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ foreach ($scope in @('Private', 'Public')) {
1515
$publicFns = @(
1616
# Harness
1717
'Resolve-ItApp', 'Resolve-WtComClsid', 'Get-ItTestPackage', 'Start-Terminal', 'Start-TerminalClean', 'Stop-Terminal',
18-
'Reset-TerminalState', 'Backup-WtConfig', 'Restore-WtConfig', 'Get-WtProcessesForApp', 'Stop-AppInstances', 'Start-TerminalFre', 'Get-DescendantWtaIds',
18+
'Reset-TerminalState', 'Backup-WtConfig', 'Restore-WtConfig', 'Get-WtProcessesForApp', 'Stop-AppInstances', 'Stop-StaleItInstances', 'Start-TerminalFre', 'Get-DescendantWtaIds',
1919
# Core (useful in tests)
2020
'Wait-Until', 'Test-Until', 'Invoke-Native', 'Write-ItLog', 'ConvertFrom-JsonSafe',
2121
# Wt
@@ -36,9 +36,9 @@ $publicFns = @(
3636
'Get-ItLogDir', 'Initialize-LogOffsets', 'Get-ItLogText', 'Start-WtEventListener', 'Get-WtEvents',
3737
'Wait-WtEvent', 'Stop-WtEventListener', 'Get-ContextBundle', 'ConvertTo-ContextText',
3838
# Agent / Autofix / Sessions
39-
'Open-AgentPane', 'Set-AgentPaneFocus', 'Wait-AgentReady', 'Send-AgentPrompt', 'Wait-AgentState',
39+
'Open-AgentPane', 'Set-AgentPaneFocus', 'Wait-AgentReady', 'Get-WtaLocalizedTextRegex', 'Get-RecommendationCardRegex', 'Send-AgentPrompt', 'Wait-AgentState',
4040
'Test-AgentPaneOpen', 'Stop-AgentPane', 'Restore-AgentPane', 'Get-AgentPaneSession', 'Get-AgentPaneText',
41-
'Send-AgentKey', 'Clear-AgentInput', 'Send-AgentWin32Key', 'Send-AgentAltV',
41+
'Send-AgentKey', 'Send-AgentShiftEnter', 'Clear-AgentInput', 'Send-AgentWin32Key', 'Send-AgentAltV',
4242
'Open-AgentCommandMenu', 'Get-AgentMenuSelection', 'Invoke-AgentMenuItem',
4343
'Test-AgentPopupShown', 'Wait-AgentPermission', 'Resolve-AgentPermission', 'Assert-AgentPaneText',
4444
'Set-ClipboardImage',

test/e2e/ItE2E/Private/Paths.ps1

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,16 @@ $script:ItKnownFamilies = [ordered]@{
1515
Dev = 'IntelligentTerminal_rd9vj3e6a2mbr'
1616
}
1717

18+
# Which brand CLSID a known package family registers (TerminalProtocolComServer.h brands).
19+
# The Store package ships the Release brand; the dev sideload ships the Dev brand. This lets
20+
# Resolve-WtComClsid probe the CORRECT brand for the package under test instead of blindly
21+
# taking the first responder — critical when BOTH Store and Dev are installed, since otherwise
22+
# probing Release first cold-launches the (window-less) Store server and misroutes wtcli.
23+
$script:ItFamilyBrand = @{
24+
'Microsoft.IntelligentTerminal_8wekyb3d8bbwe' = 'Release'
25+
'IntelligentTerminal_rd9vj3e6a2mbr' = 'Dev'
26+
}
27+
1828
function Get-ItRepoRoot {
1929
<# Walk up from the module until we find the git root (best-effort). #>
2030
$d = $PSScriptRoot
@@ -153,14 +163,23 @@ function Resolve-WtComClsid {
153163
$wtcli = $App.WtcliPath
154164
if (-not $wtcli) { throw "wtcli not resolved for $($App.Package)." }
155165

166+
# Probe the brand that THIS package registers (Store=Release, Dev=Dev). When both packages
167+
# are installed they share the COM activation namespace, so blindly taking the first
168+
# responding brand can cold-launch the OTHER package's window-less server and latch onto the
169+
# wrong CLSID (→ later GetActivePane 0x80004005). For a KNOWN package family we therefore
170+
# probe ONLY its brand (the Wait-Until retry absorbs startup); an unknown/custom family falls
171+
# back to scanning every brand.
172+
$expectedBrand = $script:ItFamilyBrand[$App.Package]
173+
$probeOrder = if ($expectedBrand) { @($expectedBrand) } else { @($script:ItBrandClsids.Keys) }
174+
156175
$found = Wait-Until -TimeoutSec $TimeoutSec -IntervalSec 1 -Because "a brand CLSID that wtcli can connect to" -Quiet -Condition {
157-
foreach ($brand in $script:ItBrandClsids.Keys) {
176+
foreach ($brand in $probeOrder) {
158177
$clsid = $script:ItBrandClsids[$brand]
159178
$r = Invoke-Native -FilePath $wtcli -Arguments @('--json', 'list-windows') -TimeoutSec 8 -Environment @{ WT_COM_CLSID = $clsid }
160179
if ($r.ExitCode -eq 0) {
161180
$j = $r.StdOut | ConvertFrom-JsonSafe
162181
if ($null -ne $j -and $null -ne $j.windows) {
163-
Write-ItLog -Level INFO -Message "Resolved COM CLSID brand=$brand clsid=$clsid"
182+
Write-ItLog -Level INFO -Message "Resolved COM CLSID brand=$brand clsid=$clsid (expected=$expectedBrand)"
164183
return $clsid
165184
}
166185
}

0 commit comments

Comments
 (0)