|
| 1 | +# PR-to-Integration-Test Workflow |
| 2 | + |
| 3 | +Use this procedure for a target Intelligent Terminal PR. Adapt the exact test |
| 4 | +suite and build commands to the files changed by that PR. |
| 5 | + |
| 6 | +## 1. Resolve the Target and Branch Strategy |
| 7 | + |
| 8 | +Inspect the PR before reading implementation files: |
| 9 | + |
| 10 | +```powershell |
| 11 | +$prNumber = 482 |
| 12 | +gh pr view $prNumber --repo microsoft/intelligent-terminal ` |
| 13 | + --json number,title,body,state,mergedAt,mergeCommit,baseRefName,headRefName,commits,files,closingIssuesReferences,reviews,url |
| 14 | +gh pr diff $prNumber --repo microsoft/intelligent-terminal |
| 15 | +``` |
| 16 | + |
| 17 | +Read linked issues and relevant review threads. Record: |
| 18 | + |
| 19 | +- The pre-fix user symptom and exact reproduction. |
| 20 | +- The intended behavior and explicitly accepted limitations. |
| 21 | +- The changed components and every boundary crossed at runtime. |
| 22 | +- Whether the PR is open, merged, or superseded. |
| 23 | + |
| 24 | +Choose the branch deliberately: |
| 25 | + |
| 26 | +- **Merged PR / follow-up test PR:** update the PR's base branch with |
| 27 | + `git pull --ff-only`, delete the merged feature branch only after verifying |
| 28 | + merge state, then create a new test branch from the merged commit. |
| 29 | +- **Open PR and tests belong in it:** use the PR head only when the user expects |
| 30 | + commits on that branch and it is safe to push there. |
| 31 | +- **Open PR but independent validation is requested:** create a branch from the |
| 32 | + PR head and clearly state that the test branch depends on the unmerged PR. |
| 33 | + |
| 34 | +Verify merge state and a clean worktree before deleting a branch. Squash-merged |
| 35 | +branches may require `git branch -D` because Git cannot infer ancestry. |
| 36 | + |
| 37 | +## 2. Reconstruct the Behavioral Contract |
| 38 | + |
| 39 | +Describe the full path as components and observable handoffs: |
| 40 | + |
| 41 | +```text |
| 42 | +user trigger |
| 43 | + -> producer/component A |
| 44 | + -> protocol or process boundary |
| 45 | + -> consumer/component B |
| 46 | + -> user-visible effect |
| 47 | +``` |
| 48 | + |
| 49 | +For a regression, distinguish: |
| 50 | + |
| 51 | +- What the producer emitted before the fix. |
| 52 | +- What downstream code interpreted. |
| 53 | +- Why existing tests did not catch the gap. |
| 54 | +- Which observable separates the regression from a legitimate success case. |
| 55 | + |
| 56 | +## 3. Audit Existing Coverage and Harness Primitives |
| 57 | + |
| 58 | +Search before adding helpers: |
| 59 | + |
| 60 | +```powershell |
| 61 | +$pattern = 'feature|event|setting|command' |
| 62 | +git grep -n -E $pattern -- test/e2e tools/wta/src src/cascadia |
| 63 | +``` |
| 64 | + |
| 65 | +Read: |
| 66 | + |
| 67 | +- Related `test/e2e/tests/Feature.*.Tests.ps1` suites. |
| 68 | +- Relevant functions under `test/e2e/ItE2E/Public/`. |
| 69 | +- Unit tests around classification, state transitions, and parsing. |
| 70 | +- Matching items in `doc/release-check-list.md`. |
| 71 | +- `test/e2e/release-coverage-map.psd1`. |
| 72 | +- `test/e2e/release-exclude.psd1`, so a new title is not silently omitted from |
| 73 | + the generated report. |
| 74 | + |
| 75 | +Classify current coverage: |
| 76 | + |
| 77 | +| Layer | What it should prove | |
| 78 | +|-------|----------------------| |
| 79 | +| Unit | Local decisions, parsing, reducer/state-machine branches | |
| 80 | +| Component | Generated scripts, serialization, API adapters | |
| 81 | +| Integration/E2E | Real process/protocol/package/UI wiring | |
| 82 | +| Release checklist | Which user-facing behaviors count as signed off | |
| 83 | + |
| 84 | +Add a shared helper only when multiple suites need it or it provides a more |
| 85 | +precise oracle. |
| 86 | + |
| 87 | +## 4. Build the Behavior Matrix |
| 88 | + |
| 89 | +Start with the regression, then add only risk-driven controls: |
| 90 | + |
| 91 | +1. **Regression positive:** the exact old failure now reaches the intended final |
| 92 | + effect. |
| 93 | +2. **Ordinary baseline:** the preexisting successful or failure path still |
| 94 | + works. |
| 95 | +3. **False-positive control:** a similar but legitimate case remains ignored. |
| 96 | +4. **Replay/idempotency:** redraw, retry, duplicate event, or repeated command |
| 97 | + does not double-submit or reuse stale state. |
| 98 | +5. **Lifecycle/routing:** hidden panes, split panes, moved tabs, reconnects, or |
| 99 | + process restarts only when the PR touches those risks. |
| 100 | +6. **Compatibility:** alternate shell, agent, policy, or language mode only when |
| 101 | + the changed code is shared with it. |
| 102 | + |
| 103 | +Each case must correspond to a plausible failure mode introduced or exposed by |
| 104 | +the target PR. |
| 105 | + |
| 106 | +For every case, identify both the immediate trigger and the downstream effect. |
| 107 | +For example, proving a failure event exists is insufficient when the user |
| 108 | +contract is that Autofix receives it; assert the event and the Autofix request. |
| 109 | + |
| 110 | +## 5. Implement Deterministic ItE2E Tests |
| 111 | + |
| 112 | +Follow existing suite structure: |
| 113 | + |
| 114 | +```powershell |
| 115 | +#Requires -Modules @{ ModuleName='Pester'; ModuleVersion='5.0.0' } |
| 116 | +
|
| 117 | +BeforeDiscovery { |
| 118 | + $package = Get-AppxPackage | Where-Object Name -like '*IntelligentTerminal*' |
| 119 | + $script:Ready = $null -ne $package |
| 120 | +} |
| 121 | +
|
| 122 | +Describe 'Feature: <behavior>' -Tag 'Feature' -Skip:(-not $script:Ready) { |
| 123 | + BeforeAll { |
| 124 | + Import-Module (Join-Path $PSScriptRoot '..\ItE2E\ItE2E.psd1') -Force |
| 125 | + $script:app = Start-Terminal -Package (Get-ItTestPackage) -PassFre $true |
| 126 | + } |
| 127 | + AfterAll { if ($script:app) { Stop-Terminal -App $script:app } } |
| 128 | +
|
| 129 | + It '<exact release-checklist title>' { |
| 130 | + # Start observers before the action, scope the oracle, and assert the |
| 131 | + # real downstream contract. |
| 132 | + } |
| 133 | +} |
| 134 | +``` |
| 135 | + |
| 136 | +Implementation rules: |
| 137 | + |
| 138 | +- Start listeners before the action; scope predicates by stable IDs and poll for |
| 139 | + positive outcomes. |
| 140 | +- For a negative case, first prove the action completed, then use a bounded |
| 141 | + observation window. |
| 142 | +- Use unique inputs when the product deduplicates, isolate state between cases, |
| 143 | + and clean up in `finally`/`AfterAll`. |
| 144 | +- Keep model-semantic assertions separate from deterministic pipeline checks. |
| 145 | + |
| 146 | +## 6. Wire the Release Checklist |
| 147 | + |
| 148 | +Add unchecked E2E items near the related feature section: |
| 149 | + |
| 150 | +```markdown |
| 151 | +- [ ] `[new]` `[E2E]` **Exact behavior title:** User-visible contract. _(#issue; E2E: `Feature.Suite`.)_ |
| 152 | +``` |
| 153 | + |
| 154 | +Use the same `Exact behavior title` in the Pester `It` name. Then assign IDs: |
| 155 | + |
| 156 | +```powershell |
| 157 | +pwsh -NoProfile -File test/e2e/Set-ChecklistIds.ps1 |
| 158 | +``` |
| 159 | + |
| 160 | +Never reuse, insert, or renumber IDs manually. If exact-title matching is not |
| 161 | +appropriate, add a narrow regex to `test/e2e/release-coverage-map.psd1` and |
| 162 | +explain why it cannot over-credit another behavior. |
| 163 | + |
| 164 | +Update the suite table in `test/e2e/README.md` when adding a new feature file. |
| 165 | + |
| 166 | +## 7. Validate Tests and Report Mapping |
| 167 | + |
| 168 | +Verify prerequisites: |
| 169 | + |
| 170 | +```powershell |
| 171 | +pwsh -NoProfile -File test/e2e/bootstrap.ps1 -Check |
| 172 | +``` |
| 173 | + |
| 174 | +Run the new suite through the report driver: |
| 175 | + |
| 176 | +```powershell |
| 177 | +$suite = 'test/e2e/tests/Feature.AutofixParser.Tests.ps1' |
| 178 | +pwsh -NoProfile -File test/e2e/Invoke-ItE2EReport.ps1 ` |
| 179 | + -Path $suite ` |
| 180 | + -UpdateReport |
| 181 | +``` |
| 182 | + |
| 183 | +Confirm: |
| 184 | + |
| 185 | +- No new case failed or skipped unexpectedly. |
| 186 | +- Every new checklist ID appears as `- [x]` in |
| 187 | + `test/e2e/artifacts/release-report.md`. |
| 188 | +- A failure would produce `AUTOMATION FAILED`, not a false pass. |
| 189 | + |
| 190 | +`-UpdateReport` incrementally overlays matched results when a prior report |
| 191 | +exists and falls back to a fresh report otherwise. To validate the underlying |
| 192 | +incremental script explicitly or use a custom output directory, run: |
| 193 | + |
| 194 | +```powershell |
| 195 | +$report = 'test/e2e/artifacts/release-report.md' |
| 196 | +$results = 'test/e2e/artifacts/results.xml' |
| 197 | +$updated = 'test/e2e/artifacts/release-report-updated.md' |
| 198 | +pwsh -NoProfile -File test/e2e/Update-ReleaseReport.ps1 ` |
| 199 | + -Report $report ` |
| 200 | + -ResultsXml $results ` |
| 201 | + -OutFile $updated |
| 202 | +``` |
| 203 | + |
| 204 | +Verify only matched items changed and every new ID is `[x]`. A skipped-only |
| 205 | +result must leave an existing checkbox unchanged. |
| 206 | + |
| 207 | +Run related existing suites in the same Pester invocation when practical. At |
| 208 | +minimum include: |
| 209 | + |
| 210 | +- The suite that previously covered the ordinary path. |
| 211 | +- The lower-layer suite that produces the new trigger. |
| 212 | +- Routing/lifecycle suites affected by shared state. |
| 213 | + |
| 214 | +Escalate to the broader Feature suite only when targeted runs expose shared |
| 215 | +regressions or the PR changes common harness/product infrastructure. |
| 216 | + |
| 217 | +## 8. Prove the Correct Build Ran |
| 218 | + |
| 219 | +Build and deploy the changed area. For WTA changes, build the explicit target |
| 220 | +matching the package architecture before the C++ package, deploy it, select it |
| 221 | +with `ITE2E_PACKAGE` or `-Package Dev`, and verify a runtime version, path, log, |
| 222 | +or changed observable. Compilation alone does not prove the deployed package |
| 223 | +contains the new `wta.exe` or generated shell integration. |
| 224 | + |
| 225 | +## 9. Deliver the Test PR |
| 226 | + |
| 227 | +Before committing, run `git -c core.whitespace=cr-at-eol diff --check`. |
| 228 | + |
| 229 | +The PR description must include: |
| 230 | + |
| 231 | +- Target PR/issue and the integration boundary added. |
| 232 | +- Cases and checklist IDs. |
| 233 | +- Targeted/regression totals and skip reasons. |
| 234 | +- Package/build tested. |
0 commit comments