Commit 1b2fce9
test(e2e): expand agent-pane coverage, add release report, fix harness flakiness (#356)
* test(e2e): strengthen agent-pane coverage; add shell-integration & proposed-command suites
- Remove redundant fixed Start-Sleep in AutofixPane (rely on poll-based Assert-Pane).
- Strengthen /model picker, all-four pane positions, helper-cleanup, real Shift+Enter
(new Send-AgentShiftEnter win32-input-mode helper).
- Add Feature.ShellIntegration.Tests.ps1 (OSC 133 marks + cmd.exe missing-integration safety).
- Add Feature.AgentProposedCommand.Tests.ps1 (non-autofix chat Insert/Run recommendation card).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): align new suites to Get-ItTestPackage package selector
Match the post-merge convention so the new ShellIntegration and
AgentProposedCommand suites honor ITE2E_PACKAGE instead of hardcoding Store.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): cover §2 view-switch draft-input preservation
Add a deterministic SessionList test: a typed-but-unsubmitted draft survives a
round-trip through the session view (open + Esc back to chat). No LLM involved.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): add multi-CLI agent chat matrix (Claude/Codex/Gemini, auth-gated)
Cover §2 Claude/Codex/Gemini chat through the IT agent pane's ACP adapters. Each
per-CLI Context runs only when the CLI is installed AND authenticated (print-mode
auth probe at discovery), else skips with the reason recorded. Verified live:
Claude + Codex pass, Gemini skips (installed but unauthenticated here).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): add release-report generator + autofix matrix (Claude/Codex/Gemini)
New-ReleaseReport.ps1 turns the release checklist into a clean human-facing report
driven purely by test results: tags (UT/E2E/MANUAL) stripped; [x]=automation passed,
'AUTOMATION FAILED'=test failed, plain [ ]=not covered, verify manually. Mapping is
title-substring + a curated override map (release-coverage-map.psd1), conservative by
design (unmapped -> manual, never a false [x]).
Also extend the agent matrix with a per-CLI autofix case (§3 Autofix with Claude/Codex/Gemini).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): report generator merges multiple results + map refinements
- New-ReleaseReport.ps1 accepts multiple -ResultsXml (later overrides earlier per
test name), so an isolated re-run of a flaky suite layers onto the full run.
- release-coverage-map.psd1: map a few passing tests whose names differ from their
checklist titles (Focus hotkey, Model control/changes).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): clear stale IT instances before every launch (fixes new-tab E_FAIL flake)
A prior test whose AfterAll/Stop-Terminal didn't run (e.g. a BeforeAll that threw)
leaves an IT window behind. The single-instance AUMID launch then hands off to that
stale, often half-initialised window (Launched=false), so the harness drives a broken
instance where new-tab returns CreateTab E_FAIL (0x80004005); and because the store and
dev packages share one per-brand COM CLSID, a stale window of the OTHER package steals
wtcli's CoCreateInstance and misroutes every call.
Start-Terminal now calls Stop-StaleItInstances first, closing every leftover IT window
(store + dev, matched by *IntelligentTerminal* install location only — never the user's
stock WT) so each launch is deterministic and freshly-owned. Verified: a simulated
leftover is cleaned and the fresh instance's new-tab wsl.exe succeeds.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): harden WSL-autofix gate + Shift+Enter injection
- WSL autofix Describe: wrap the WSL-pane setup in try/catch so a build that can't create a
wsl.exe tab via the protocol (stale dev pkg predating OSC 9001 -> CreateTab E_FAIL) SKIPS
via the per-It guards instead of failing the Describe in BeforeAll.
- Shift+Enter on a live session row: skip if no selectable row; retry the raw win32-input
keystroke up to 3x while polling for the view to dismiss (the injection can drop under load).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): report credits UT-verified items + fixes/extends title->test map
- Generator: an originally-ticked [x] item is verified by an automated unit test, so credit
it as passed (unless a mapped E2E test failed). Unit tests are automation; the human needn't
re-verify them.
- Map: drop backticks from keys (the report strips them from titles, so backticked keys never
matched -> false manual); add /model, Shift+Enter, Autofix-with-Copilot, FRE auto-error
on-variants, session-mgmt-choice-persists, packaging/logging name mismatches.
- Net on the last full run: 79 -> 104 verified, 156 -> 130 manual.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): make Wait-AgentReady deterministic (poll the real precondition)
The agent-pane-readiness flake: Wait-AgentReady matched the helper-log 'acp_initialize'
marker, which fires several seconds BEFORE the helper writes its session origin
('recording agent-pane session origin') -> the jsonl that Get-AgentPaneSession reads. So it
returned ready too early and the next agent-pane call (Send-AgentKey/Open-SessionList) raced
a not-yet-written record and timed out. The agent_status connected/failed event is NOT
broadcast to wtcli listen (verified), so events can't be used.
Wait-AgentReady now polls Get-AgentPaneSession (the exact precondition every primitive needs:
a recorded, running pane session) and returns the instant it resolves — deterministic, not a
fixed delay — for both the initial connect and a reconnect after /restart or a settings-driven
rebuild (newest running record wins). A logged auth/fatal failure short-circuits. The
AgentRestart test now waits for reconnect-readiness after the settings change before driving
the menu. Verified: 3/3 consecutive green (the test previously flaked on a 20s timeout).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): judge agent readiness by the connected UI, not the session registry
Per review: gating Wait-AgentReady on Get-AgentPaneSession (which reads the
agent-pane-sessions.jsonl session registry) is verifying a feature with that same feature —
if the registry breaks, the gate false-readies or hangs and masks the bug.
Wait-AgentReady now matches the agent pane buffer for the connected input placeholder
('Ask anything, / for commands..'), which the TUI renders ONLY in ConnectionState::Connected
(ui/input.rs:62; the connecting/disconnected placeholders are distinct strings). That is the
user-visible ground truth of 'ready to chat', independent of the session-tracking feature, and
still returns the instant it's observed (deterministic). Auth/fatal log markers short-circuit.
Verified: AgentRestart 2/2 green (initial connect + reconnect).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): require full connected placeholder in Wait-AgentReady (Copilot review)
Match 'Ask anything … for commands' in order on one line instead of either fragment
anywhere in the captured scrollback, so stray transcript/help text can't false-positive the
readiness gate. Verified: AgentRestart 2/2 green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): isolate proposed-command Insert/Run in fresh terminals (Copilot review)
Split the Insert and Run cases into separate Describes, each with its own fresh terminal
(matching Feature.AutofixPane). With the shared terminal, a prior card's
'Run command'/'Insert in Terminal' text lingered in the scrollback and could co-occur with the
next case's marker (echoed in the prompt) to false-positive the card-readiness check before a
fresh card rendered. Verified: both cases 2/2 green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): precise matrix skip reasons + word-boundary assert + Harness comment (Copilot review)
- AgentMatrix: report a PRECISE per-agent skip reason (not installed vs installed-but-
unauthenticated vs package missing) via Set-ItResult instead of a boolean Context -Skip, so CI
shows why; no terminal is launched when skipping. Re-checks package presence in BeforeAll
because a script-scoped var from BeforeDiscovery does not persist into the run phase (only the
-ForEach data does).
- AgentMatrix: chat assertion uses a word-boundary match instead of a bare '7'.
- Harness: correct the Stop-StaleItInstances comment - it makes -ColdStart redundant, but
-ShowFre still controls whether the FRE overlay is shown.
Verified: Claude/Codex chat+autofix pass; Gemini skips with "installed but not authenticated".
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): consolidate non-Copilot agents into one matrix case; rewrite checklist
Per design decision: Copilot is the primary agent and its full behaviour (chat, autofix,
insert/run, permission, render, slash, sessions) is covered in depth by the copilot-only
suites. All built-in agents share the same agent-pane -> helper -> master -> agent-CLI (ACP)
path; the only per-agent difference is the spawned command. So we stop re-testing every
behaviour per agent.
- Feature.AgentMatrix.Tests.ps1: collapsed from a per-agent (Claude/Codex/Gemini) x
chat+autofix matrix into ONE consolidated test case that, for each installed+authenticated
non-Copilot agent, does a single connect + chat round-trip in its own fresh terminal; skips
when none is available.
- doc/release-check-list.md: collapsed the per-agent items (Claude/Codex/Gemini chat,
autofix, delegate, installed, hook-install; and the custom-agent behavioural items) into
single consolidated items, keeping Copilot as the primary and the config/selection/tracking
items. Total 235 -> 220 items.
- release-coverage-map.psd1 + README updated to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): add deterministic §9/§10/§4 coverage (CLSID inject, log retention, /sessions)
Adds three high-confidence, deterministic E2E cases (no mock/agent variance)
plus one mapping fix, expanding genuine release-checklist coverage:
- §9 "WT_COM_CLSID is injected": read $env:WT_COM_CLSID back from a shell pane
and assert a braced CLSID, proving WT injects protocol discovery into panes.
- §10 "Old log cleanup is safe": seed a sentinel in the running version's log
dir + a stale other-version dir, restart the build, assert the running
version's logs survive and the stale version dir is pruned wholesale.
- §4 "Slash command works": /sessions opens the session view (the command-menu
path, complementing the existing button path).
- §10 "Early startup failures are logged": coverage-map override (the test
"...would be logged" already exists and passes).
All three new cases validated live against the Store package.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): locale-robust agent-readiness gate + strict CLSID assert (Copilot review)
- Wait-AgentReady: the connected input placeholder is localized
(input.placeholder.connected), so a hard-coded English regex would time out
on non-en-US machines. Derive the readiness regex from EVERY bundled wta
locale's connected placeholder (Get-AgentConnectedPlaceholderRegex, cached;
falls back to the en-US literal outside a repo checkout). Verified it matches
the real zh-CN placeholder and rejects the connecting/disconnected strings.
- WT_COM_CLSID test: tighten the assertion to a real GUID shape
({8-4-4-4-12} hex) so malformed values like {----} can no longer pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): stable /model assert + actionable agent-matrix skip reason (Copilot review)
- /model picker: replace the loose name alternation (auto|claude|gpt|sonnet|opus —
"auto" also matches autofix/automatic and drifts with releases) with an
assertion on the locale-independent "> " highlight selector that the picker
marks its selected row with (model_popup.rs highlight_symbol). Proves a
selectable model row actually rendered. Validated live (full suite 13/13).
- AgentMatrix: classify each CLI as not-installed / installed-unauthenticated /
authed and record every CLI's state in the skip -Because, so CI results are
actionable and an "installed but unauthenticated" regression isn't silently
collapsed. Validated the classifier distinguishes gemini (unauthenticated)
from a missing binary (not-installed).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): fix Wait-AgentReady auth fail-fast log probe (Copilot review)
The helper logs the typed failure via tracing as `target=failure … class=auth_required`
(app.rs) and `non_compliant_auth=true` (failure.rs string-fallback shim), so the old
literal substring `agent failure class="auth` never matched — auth-required connects
burned the full timeout despite the comment claiming a short-circuit.
Match the stable class labels instead (quote/field-order agnostic): auth_required /
non_compliant_auth, plus the helper `exiting with error`. Deliberately NOT matching the
bare message "agent failure", which also fires for a benign cancel (logged before the
is_cancelled early-return). Verified real release helper logs from successful connects
contain 0 of these tokens (no false-trip), and the placeholder check runs before the
probe each iteration so a slow-but-OK connect still wins.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): locale-robust /model title + assert agent-readiness in restart (Copilot review)
- Generalize the locale-bundle reader into Get-WtaLocalizedTextRegex (matches a
localization key's value across EVERY bundled wta locale; strips a trailing
" (…)" key-hint and ".." run). Get-AgentConnectedPlaceholderRegex now delegates
to it. The /model picker assertion uses it for model_picker.title instead of
the hardcoded en-US "Select model" (which would fail on non-en-US). Exported
the helper so tests can use it. Verified it matches the real zh-CN title.
- Feature.AgentRestart: assert the two Wait-AgentReady gates with Should -BeTrue
(was | Out-Null) so a failed reconnect fails immediately and attributably,
instead of surfacing later as an opaque prompt-assertion timeout.
Validated live: Feature.AgentPaneInteraction 13/13, Feature.AgentRestart 2/2.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): remove dead -ColdStart param from Start-Terminal (Copilot review)
Stop-StaleItInstances now always enforces cold-start semantics, so the -ColdStart
switch was an unused no-op while the doc claimed it was "redundant" — misleading.
Removed the parameter and its doc, updated the -ShowFre doc to state a fresh
monarch is always started, and dropped the -ColdStart argument from the sole
caller (Start-TerminalFre). Validated live: Start-Terminal path (Packaging CLSID
1/1) and the -ShowFre FRE path (Feature.FreFlow 5/5) both green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): throttle readiness log reads, distinct probe-timeout, localize card/anchor selector (Copilot review)
Five review fixes:
- Wait-AgentReady: throttle the helper-log fail-fast read to every ~2s (UI placeholder
still polled at 500ms). Get-ItLogText re-reads the whole appended slice each call and
the log grows while connecting, so reading every loop was O(n^2) IO on long waits.
- AgentMatrix Get-CliStatus: return a distinct `probe-timeout` status instead of
collapsing a hung/changed probe into `installed-unauthenticated`, so CI stays actionable.
- AgentProposedCommand (both Insert + Run Describes): the recommendation card buttons
([ Run command ] / Insert in Terminal) are localized — match them across all bundled
locales via Get-WtaLocalizedTextRegex (en-US fallback) instead of hardcoded English.
- /model selector assertion: anchor the "> " highlight marker to line start (allowing a
leading popup border │/║/|) so a stray "->" in chat output can't false-positive.
Validated: Feature.AgentPaneInteraction 13/13 live (anchored selector against the real
bordered picker); card regex + anchored selector + parse all checked offline.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): robust YAML scalar parse + ensure report output dir (Copilot review)
- Get-WtaLocalizedTextRegex previously only matched double-quoted YAML values, so any
unquoted (or single-quoted) locale entry would yield $null and silently fall back to
the en-US literal. Now handles double-quoted (ignoring trailing # comment / Locked
hint), single-quoted (with '' escaping), and bare scalars (trailing # comment stripped).
(Note: the current bundle happens to double-quote all four keys we read, so there is no
behavior change today — this hardens against future unquoted entries.) Verified it still
matches en-US/zh placeholder, en-US/de run-button, en-US title, and extracts an unquoted
scalar correctly.
- New-ReleaseReport.ps1: create the -OutFile parent directory before Set-Content, so
running with a custom path (or without a prior Invoke-ItE2EReport) no longer throws.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): resolve per-package COM CLSID + co-location asserts + matrix timeout (local-build run)
Running the suite against the locally-built Dev package surfaced four failures; all are test
harness/assertion issues, not product regressions:
- ROOT CAUSE FIX (Paths.ps1): Resolve-WtComClsid blindly probed every brand CLSID and took the
first responder. With BOTH the Store (Release brand) and Dev sideload installed, it probed
Release first, COM cold-launched the window-less Store server (whose list-windows returns an
empty-but-non-null array), and latched onto the WRONG CLSID -> every later wtcli call hit a
server with no windows (GetActivePane 0x80004005). Now we map each known package family to its
brand (Store=Release, Dev=Dev) and probe ONLY that brand, so Dev binds to its own
{D5B7C9E1...} server and works even with Store installed. Verified live: Dev resolves its own
CLSID and active-pane succeeds with Store still present — no more uninstall dance.
- Feature.Packaging (2 cases): the "co-located wta.exe / wtcli" assertions hardcoded a
'WindowsApps' path, which only holds for an installed Store package; a Dev F5/run-from-layout
build resolves binaries under bin\x64\Debug\AppX. Assert co-location under the package
InstallLocation (covers Store AND Dev) and explicitly reject the unpackaged tools\wta\target
fallback — preserving the original intent on both. Verified for Store and Dev app objects.
- Feature.AgentMatrix: bumped the non-Copilot answer assert 90s -> 150s. These agents answer via
the npx ACP adapter (extra hop + remote model latency); 90s was demonstrably too tight and the
consolidated external-CLI case flaked on a slow turn (passes at ~143s). Not masking a bug — the
agent does answer; the window just needs to fit a realistic adapter+model latency.
All four previously-failing cases verified green individually against the local Dev build.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): harden autofix card (skip on LLM-explain), make Open-SessionList idempotent, drop wrong Shift+Enter case
Hardening the two flaky cases the full Dev run flagged:
- Autofix card-render + AI-oracle: when the LLM returns an explanation (not a runnable-fix
card) for ALL retried typos, that's model variance, not a product failure — SKIP with a
clear reason instead of failing, matching the existing WSL-autofix variance skip. The
AI-oracle case also skips when no card was produced.
- Open-SessionList: made idempotent. SessionToggleButton is a TOGGLE, so a single press could
CLOSE an already-open view (or race a transition), leaving us in chat → render-wait timeout.
Now it checks-then-toggles (returns immediately if already shown) and retries, fixing the
"session view did not render" flake.
- Removed the "Shift+Enter on a live session row" E2E case: its premise was WRONG. It asserted
the session view dismisses back to chat, but Shift+Enter on a Live row dispatches FocusPane
(wtcli focus-pane → move WT focus to that session's pane); it does NOT close the view (verified
live + by docs). The contract is deterministically covered by the Rust unit test
shift_enter_on_class_a_live_row_focuses (7/7 shift_enter unit tests green). Focus-pane semantics
aren't stably observable in E2E — the MVP picker shows only Class B shell sessions whose panes
are usually already closed (Focus → NotFound). Updated the checklist item to [UT✓]-only, removed
the coverage-map E2E mapping, and adjusted README counts (90→89).
Validated live on the local Dev build: autofix pair + agent-restart all green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): require BOTH card buttons + clarify Stop-StaleItInstances ownership (Copilot review)
- Feature.AgentProposedCommand (both Insert + Run Describes): the recommendation-card readiness
predicate matched an OR of the Run/Insert labels, contradicting the comment that it waits for
BOTH actions and risking driving the wrong UI state. Split into CardRunRegex + CardInsertRegex
(each localized across all bundled locales, en-US fallback) and require BOTH plus the marker.
Verified: a full "[ Run command ] Insert in Terminal" card matches; a Run-only line does not.
- Stop-StaleItInstances doc: state plainly that it unconditionally kills ALL running IT windows
(store + dev) — not just crashed-test leftovers but also a manually-started window and even the
IT window hosting the current shell if tests are run from IT — so the harness owns every IT
window during a run. Stock Windows Terminal (Microsoft.WindowsTerminal_*) is never matched.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): preserve fully-parenthesized locale values + locale-robust session-view detection (Copilot review)
- Get-WtaLocalizedTextRegex: the trailing key-hint strip used `\s*\([^)]*\)\s*$`, which also
matched a value that is ENTIRELY parenthesized (e.g. agents.footer_hint
"(↑ ↓ … Esc to exit …)"), reducing it to an empty string -> $null regex. Require whitespace
BEFORE the hint (`\s+\(`) so only true trailing hints (e.g. "Select model (↑ ↓ …)") are
stripped while fully-parenthesized values survive. Verified footer_hint now yields a non-null
regex; model-title strip and placeholder matching unchanged.
- Open-SessionList / Test-SessionListShown: detected the rendered session view with hard-coded
English substrings ("to launch session|to exit|navigate"), which the agents view localizes via
agents.footer_hint -> would never match on non-en-US machines. Added Get-SessionViewRenderRegex
(the footer hint is drawn in BOTH the loading and populated branches per
ui/agents_view.rs render_footer_hint, so it's the reliable open-signal) matched across all
bundled locales, en-US fallback. Live-validated: Session button / refresh / /sessions all green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): assert deterministic cmd errorlevel (ITERR=9009) for cmd.exe safety (Copilot review)
The cmd.exe safety assertion matched "not recognized|...|nonexistentcmd" — the nonexistentcmd
alternative matches the ECHOED input line even if nothing executed, and "not recognized" is
localized. Switched to the deterministic, locale-robust cmd errorlevel signal as suggested.
Implementation note: the suggested single-line `badcmd & echo ITERR=%errorlevel%` does NOT work —
cmd expands %errorlevel% at PARSE time, before the bad command runs, so it captures the OLD value
(verified live: prints ITERR=0). The echo must be a SEPARATE command to observe the failure's
9009, so the test now sends the bad command + Enter, then `echo ITERR=%errorlevel%` + Enter, and
asserts ITERR=9009 (verified live: prints ITERR=9009; the echoed line shows "%errorlevel%"
literally, so the match only hits the expanded output — also proving Enter executed the command).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): Wait-AgentReady fail-fast scans newest helper log in full (Copilot review)
The fail-fast auth/fatal scan used Get-ItLogText -SinceStart, which reads only bytes appended
after Initialize-LogOffsets (captured at the END of Start-Terminal). But the helper is pre-warmed
during tab init, so an auth/fatal connect failure can be logged BEFORE that offset — -SinceStart
would miss it and Wait-AgentReady would burn the full timeout instead of returning $false fast.
Now it reads the NEWEST wta-main_helper-*.log file in full (via a FileShare.ReadWrite stream,
matching Get-ItLogText, since the helper holds it open for writing). The newest file is this
launch's helper (fresh PID → fresh file, because Stop-StaleItInstances killed any prior terminal),
so reading from the top catches a pre-offset failure without false-matching a previous run's log.
Verified live: the authed happy path still returns ready (no false positive).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): dispose helper-log StreamReader + require popup border in /model selector (Copilot review)
- Wait-AgentReady: dispose the StreamReader used to read the newest helper log (it also disposes
the underlying FileStream), instead of leaking it.
- /model picker assertion: the selector regex allowed the "> " marker with no border prefix, so a
Markdown blockquote ("> …") in the prior chat transcript could false-positive. The picker draws
its highlight symbol INSIDE the popup's left border (verified live: "│> ● Claude Sonnet 4.6 …"),
so require a border char (│/║/|) immediately before the marker: ^\s*[│║|]\s*>\s+\S. Verified the
real picker row matches while "> some text" and "foo -> bar" are rejected; full
Feature.AgentPaneInteraction suite 13/13 green live.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): drop misleading tools/wta/target check from wtcli co-location assert (Copilot review)
The wtcli negative-path line `Should -Not -Match 'tools/wta/target'` was copy-pasted from the
wta.exe case — but wtcli is never resolved from tools/wta/target. Its non-packaged fallbacks are a
PATH-resolved standalone (Get-Command wtcli) or bin/x64/{Debug,Release}/wtcli/wtcli.exe
(cli_channel.rs), none under the InstallLocation. The preceding -BeLike "InstallLocation*" already
fully guarantees co-location, so the extra check was redundant and misleading — removed it and
documented why (the tools/wta/target check stays in the wta.exe case where it's correct).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): drop unneeded copilot pin from ShellIntegration + rename wtcli co-location test (Copilot review)
- Feature.ShellIntegration: removed the acpAgent='copilot' pin. The suite is deterministic (OSC
133 marks + cmd.exe safety) and never opens the agent pane, and BeforeDiscovery gates only on
the IT package — not copilot — so pinning the agent added an unnecessary copilot install/auth
dependency (the pre-warmed helper is irrelevant here). Keeps autoFixEnabled=$false. Verified the
two cases still pass without the pin.
- Feature.Packaging: renamed "Wrong unpackaged WTA is not used" → "Packaged wtcli is co-located in
the package (no unpackaged fallback)" so the name matches what it asserts (wtcli co-location).
Added a release-coverage-map entry so the checklist item "Wrong unpackaged WTA is not used" still
maps (to the wta.exe not-tools\wta\target case + the wtcli co-location case together).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): match structured auth-failure fields in Wait-AgentReady fail-fast (Copilot review)
The fail-fast scan matched bare `auth_required` / `non_compliant_auth` anywhere in the helper log,
which could false-trigger if those tokens appear in an unrelated field/value (→ Wait-AgentReady
returns $false even though the agent might still connect). Tightened to the STRUCTURED tracing
fields: `class="?auth_required` (app.rs; the &str value may be quoted, so the quote is optional)
and `non_compliant_auth=true` (failure.rs), plus `exiting with error`. Verified it still matches
both quoted/unquoted class=auth_required, non_compliant_auth=true, and the helper exit, while
rejecting a bare "auth_required" mention and non_compliant_auth=false. This only narrows the match,
so the authed happy path (zero matches) is unaffected.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): localize autofix card detection + dispose FileStream if reader ctor throws (Copilot review)
- Feature.AutofixPane: CardShown and the two other card-detection predicates (lines 30/102/271)
hard-coded the English "Run command|Insert in Terminal" labels, so they'd mis-skip/fail on
non-en-US machines. Added an exported Get-RecommendationCardRegex helper (EITHER button label,
localized across all bundled locales via Get-WtaLocalizedTextRegex, en-US fallback) and routed
all three through it. Verified it matches the en-US card line; the variance-skip path still works
live.
- Wait-AgentReady: if the StreamReader ctor throws, $fs was left undisposed (file-handle leak).
Wrapped the reader in a nested try/finally so $fs is always disposed (double-dispose after the
reader closes it is a safe no-op).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): unescape YAML double-quoted escapes in Get-WtaLocalizedTextRegex (Copilot review)
Double-quoted YAML scalars were captured raw, so a value containing \" (e.g. setup.subtitle.*
"Your agent \"%{agent}\" …") kept the literal backslashes — the generated regex then looked for
backslashes absent from the rendered UI text, breaking locale-robust assertions for such keys.
Now the double-quoted branch unescapes \" \\ \n \t \r (\x -> x) before the value is regex-escaped.
Verified: setup.subtitle.copilot_missing no longer yields a regex containing \" (the escape is
resolved to a literal "), while the keys the tests actually use (no backslashes) are unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): assert Wait-AgentReady in autofix/proposed-command BeforeAll (Copilot review)
Four BeforeAll blocks piped Wait-AgentReady to Out-Null, discarding its boolean — so an auth/fatal
connect failure would proceed in a not-ready state and surface later as opaque card-polling
failures. Assert | Should -BeTrue with a clear -Because in all four (AutofixPane card-render +
AutofixPane WSL setup, AgentProposedCommand Insert + Run), so a failed/again-auth connect fails
immediately and attributably. (The WSL one is inside the best-effort try/catch, so a readiness
failure there is logged and degrades to a skip via the existing per-It $wslShell guards.) Verified
live: the Insert BeforeAll assertion passes when copilot connects.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): truncation-robust session-view detection + pane-scoped failure mark (Copilot review)
- Get-SessionViewRenderRegex: matched the FULL localized agents.footer_hint, but the TUI
end-truncates that hint to the pane width (agents_view.rs render_footer_hint -> trunc), so the
full line may never appear and Open/Test-SessionListShown could time out. Every bundled locale
leads the hint with the invariant nav arrows "↑ ↓" (en "(↑ ↓ to navigate …)", zh "(↑ ↓ 导航 …)"),
and being at the start they survive truncation — so match those (en-US footer words kept as an
extra fallback). Verified live: the rendered footer matches; the slash-/sessions path is green.
- Feature.ShellIntegration failure-mark test: Wait-WtCommandFailure listened to the global
vt_sequence stream, so an unrelated OSC 133;D mark could satisfy it. The event's `pane_id`
equals the pane session_id (Get-ActivePane.session_id), whereas its `tab_id` is a GUID and
Get-ActivePane/Get-WtTabs expose tab_id only as a numeric INDEX — so added a -PaneId filter to
Wait-WtCommandFailure and scoped the assertion to the active pane. Verified live: passes scoped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): localize last autofix Run-card check + drop redundant English reconnect probe (Copilot review)
- Feature.AutofixPane Run-action: the card-detection predicate still hard-coded the English
"Run command" label (missed in the earlier sweep). Routed it through Get-RecommendationCardRegex
like the other card-detection sites so it's locale-robust.
- Feature.AgentRestart: removed the post-/restart `Test-Until … -match 'Ask anything|Copilot|Agent'`
reconnect probe — it matched hard-coded English (not locale-robust) and was redundant with the
Wait-AgentReady | Should -BeTrue gate immediately after, which is the deterministic
reconnect-and-ready signal. Verified live: the restart case still passes (37s).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test(e2e): tighten Non-Copilot-chat coverage-map regex to not match Copilot restart (Copilot review)
The override regex 'connects and answers' for "Non-Copilot agents chat works" also matched the
Copilot restart test name "(/restart reconnects and answers)" — "reconnects and answers" contains
"connects and answers" — which could credit the checklist item from the wrong test in the report.
Anchored on "non-Copilot agent.*connects and answers" so it uniquely matches the AgentMatrix case.
Verified: matches the AgentMatrix name, does NOT match the Copilot restart name.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent 18bc4d8 commit 1b2fce9
19 files changed
Lines changed: 958 additions & 136 deletions
File tree
- doc
- test/e2e
- ItE2E
- Private
- Public
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
52 | | - | |
53 | | - | |
| 51 | + | |
54 | 52 | | |
55 | 53 | | |
56 | 54 | | |
| |||
127 | 125 | | |
128 | 126 | | |
129 | 127 | | |
130 | | - | |
131 | | - | |
132 | | - | |
| 128 | + | |
133 | 129 | | |
134 | 130 | | |
135 | 131 | | |
| |||
193 | 189 | | |
194 | 190 | | |
195 | 191 | | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
| 192 | + | |
200 | 193 | | |
201 | 194 | | |
202 | 195 | | |
| |||
237 | 230 | | |
238 | 231 | | |
239 | 232 | | |
240 | | - | |
| 233 | + | |
241 | 234 | | |
242 | 235 | | |
243 | 236 | | |
| |||
259 | 252 | | |
260 | 253 | | |
261 | 254 | | |
262 | | - | |
263 | | - | |
264 | | - | |
| 255 | + | |
265 | 256 | | |
266 | 257 | | |
267 | 258 | | |
| |||
276 | 267 | | |
277 | 268 | | |
278 | 269 | | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
| 270 | + | |
283 | 271 | | |
284 | 272 | | |
285 | 273 | | |
| |||
312 | 300 | | |
313 | 301 | | |
314 | 302 | | |
315 | | - | |
316 | | - | |
317 | | - | |
318 | | - | |
| 303 | + | |
319 | 304 | | |
320 | 305 | | |
321 | 306 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
18 | 28 | | |
19 | 29 | | |
20 | 30 | | |
| |||
153 | 163 | | |
154 | 164 | | |
155 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
156 | 175 | | |
157 | | - | |
| 176 | + | |
158 | 177 | | |
159 | 178 | | |
160 | 179 | | |
161 | 180 | | |
162 | 181 | | |
163 | | - | |
| 182 | + | |
164 | 183 | | |
165 | 184 | | |
166 | 185 | | |
| |||
0 commit comments