Skip to content

Commit bc4f1a9

Browse files
vanzueCopilot
andcommitted
Trim PR integration test skill
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b88f4443-0b18-4d44-ae20-dde46ff2eea8
1 parent dfc7ee8 commit bc4f1a9

2 files changed

Lines changed: 35 additions & 105 deletions

File tree

.github/skills/pr-integration-test/SKILL.md

Lines changed: 13 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -11,38 +11,23 @@ checklist.
1111

1212
## When to Use This Skill
1313

14-
- Add integration or E2E tests for a specific PR.
15-
- Add follow-up regression coverage after a fix has merged.
16-
- Prove a bug from an issue cannot recur across real component boundaries.
14+
- Convert an open or merged PR into cross-component regression coverage.
1715
- Extend `doc/release-check-list.md` and make report scripts check the new rows.
1816
- Audit whether a PR's existing tests cover the user-visible behavior rather
1917
than only its implementation details.
2018

2119
## Prerequisites
2220

23-
- Work on a feature branch, never directly on `main` or `master`.
24-
- Use `gh` to inspect the target PR, linked issues, review discussion, and state.
25-
- Read repository and area-specific instructions before editing.
2621
- Read `test/e2e/README.md` and reuse the ItE2E framework instead of creating a
2722
parallel harness.
2823
- Run `pwsh -File test/e2e/bootstrap.ps1 -Check` before live E2E validation.
2924

3025
## Workflow
3126

3227
Follow [workflow.md](./references/workflow.md). Track the phases with a TODO
33-
list because PR analysis, test design, checklist wiring, live validation, and
34-
delivery must all complete.
35-
36-
1. Resolve the target PR and select the correct base commit and branch strategy.
37-
2. Reconstruct the user-visible behavior and the complete component path.
38-
3. Audit existing unit, integration, E2E, and release-checklist coverage.
39-
4. Build a positive/negative/regression behavior matrix before writing tests.
40-
5. Implement the smallest deterministic integration suite using existing ItE2E
41-
primitives.
42-
6. Map each new release-signoff behavior to a stable checklist item.
43-
7. Run the new suite, related existing regressions, and release-report scripts.
44-
8. Commit, push, and prepare a PR whose evidence names passes, skips, checklist
45-
IDs, and the exact build tested.
28+
list.
29+
30+
`analyze PR -> reconstruct behavior -> audit coverage -> design matrix -> write ItE2E -> wire checklist -> validate -> deliver`
4631

4732
## Test Design Standard
4833

@@ -58,11 +43,6 @@ For every proposed case, record:
5843
| Existing protection | Which existing tests protect old behavior and must still run? |
5944
| Checklist title | Which exact bold release-checklist title contains the Pester test name? |
6045

61-
An integration test is justified only when it proves a boundary that lower-level
62-
tests do not. Keep focused unit tests for branch logic; use E2E for wiring,
63-
packaging, process boundaries, protocol events, persistence, and real UI
64-
interaction.
65-
6646
## Oracle Priority
6747

6848
Prefer the earliest deterministic product-owned signal:
@@ -73,9 +53,8 @@ Prefer the earliest deterministic product-owned signal:
7353
4. Rendered terminal or UI state
7454
5. LLM-generated text
7555

76-
Use an LLM output or AI judge only when usefulness or semantic correctness is
77-
the behavior under test. Do not make routing, trigger, suppression, or
78-
single-flight regressions depend on model wording.
56+
Use model output only when it is the behavior under test; never use it to prove
57+
routing, triggers, suppression, or idempotency.
7958

8059
## Release Checklist Contract
8160

@@ -85,26 +64,18 @@ single-flight regressions depend on model wording.
8564
- Prefer exact-title matching. Add `test/e2e/release-coverage-map.psd1` entries
8665
only when an exact test name would be misleading or one case intentionally
8766
covers multiple checklist items.
88-
- Run `pwsh -File test/e2e/Set-ChecklistIds.ps1`; never assign or renumber
89-
stable checklist IDs manually.
90-
- Run the suite through the E2E report driver, not only `Invoke-Pester`, and
91-
inspect `release-report.md` to prove every new ID is `[x]`.
92-
- Verify `Update-ReleaseReport.ps1` too when the suite is expected to support
93-
partial release-signoff runs.
67+
- Assign stable IDs and verify `[x]` output through the full and incremental
68+
report paths in [workflow.md](./references/workflow.md).
9469

9570
## Completion Gate
9671

9772
Do not call the work complete until all of these are true:
9873

99-
- The original regression fails on the old behavior or is otherwise tied to a
100-
verified pre-fix symptom.
101-
- The fixed path passes through the real integration boundary.
102-
- False-positive and replay/idempotency risks are covered where applicable.
103-
- Related preexisting suites pass, or every skip/failure is explicitly
104-
classified as environment, model variance, product regression, or test bug.
105-
- New checklist IDs become `[x]` through both applicable report paths.
106-
- The deployed package or executable under test is proven to contain the target
107-
change; no stale artifact is being exercised.
74+
- Pre-fix evidence identifies the regression, and the fixed path crosses the
75+
real integration boundary.
76+
- Relevant false positives, replay risks, and existing behavior are covered.
77+
- The correct build passes related suites and marks new checklist IDs `[x]`;
78+
every skip is explained.
10879

10980
## Gotchas
11081

@@ -118,16 +89,6 @@ Do not call the work complete until all of these are true:
11889
- **Do not turn product failures into skips.** Skip only when an external
11990
prerequisite is genuinely unavailable. A connected product that behaves
12091
incorrectly must fail.
121-
- **Do not rely on fixed sleeps for positive completion.** Start listeners
122-
before actions and poll for a scoped event/state. Use a short bounded
123-
observation window only to prove that something does not happen.
124-
- **Do not let unrelated panes or windows satisfy assertions.** Scope events by
125-
pane/session/tab/window identifiers whenever the protocol exposes them.
126-
- **Do not trust a build command alone.** Confirm package selection, deployed
127-
version, co-located binaries, and runtime logs when stale artifacts are
128-
possible.
129-
- **Do not add checklist text without validating report matching.** A passing
130-
Pester case that leaves its release row unchecked is incomplete coverage.
13192

13293
## References
13394

.github/skills/pr-integration-test/references/workflow.md

Lines changed: 22 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,8 @@ Choose the branch deliberately:
3131
- **Open PR but independent validation is requested:** create a branch from the
3232
PR head and clearly state that the test branch depends on the unmerged PR.
3333

34-
Never force-delete a branch based only on its name. Verify the PR state and a
35-
clean worktree first. A squash-merged branch may require `git branch -D`
36-
because Git cannot infer ancestry even though GitHub confirms the merge.
34+
Verify merge state and a clean worktree before deleting a branch. Squash-merged
35+
branches may require `git branch -D` because Git cannot infer ancestry.
3736

3837
## 2. Reconstruct the Behavioral Contract
3938

@@ -54,9 +53,6 @@ For a regression, distinguish:
5453
- Why existing tests did not catch the gap.
5554
- Which observable separates the regression from a legitimate success case.
5655

57-
Do not accept a PR description as the sole source of truth. Compare it with the
58-
code, issue reproduction, logs/events, and live behavior when available.
59-
6056
## 3. Audit Existing Coverage and Harness Primitives
6157

6258
Search before adding helpers:
@@ -85,8 +81,8 @@ Classify current coverage:
8581
| Integration/E2E | Real process/protocol/package/UI wiring |
8682
| Release checklist | Which user-facing behaviors count as signed off |
8783

88-
Reuse public ItE2E primitives. Add a shared helper only when multiple suites
89-
need the same operation or the helper itself provides a more precise oracle.
84+
Add a shared helper only when multiple suites need it or it provides a more
85+
precise oracle.
9086

9187
## 4. Build the Behavior Matrix
9288

@@ -104,8 +100,8 @@ Start with the regression, then add only risk-driven controls:
104100
6. **Compatibility:** alternate shell, agent, policy, or language mode only when
105101
the changed code is shared with it.
106102

107-
Avoid combinatorial matrices. Each case must correspond to a plausible failure
108-
mode introduced or exposed by the target PR.
103+
Each case must correspond to a plausible failure mode introduced or exposed by
104+
the target PR.
109105

110106
For every case, identify both the immediate trigger and the downstream effect.
111107
For example, proving a failure event exists is insufficient when the user
@@ -139,19 +135,13 @@ Describe 'Feature: <behavior>' -Tag 'Feature' -Skip:(-not $script:Ready) {
139135

140136
Implementation rules:
141137

142-
- Start event listeners before triggering the behavior.
143-
- Scope event predicates by stable IDs; do not accept unrelated startup events.
144-
- Use `Wait-WtEvent`, `Test-Until`, or assertion helpers for positive outcomes.
145-
- For negative outcomes, first prove the command/action completed, then observe
146-
a short bounded window and assert the forbidden event/state is absent.
147-
- Use unique command text when the product intentionally deduplicates repeated
148-
requests.
149-
- Use fresh applications or explicit state cleanup when one case can leave an
150-
agent turn, setting, pane, or listener active.
151-
- Put cleanup in `finally`/`AfterAll`.
152-
- Keep model-semantic tests separate from deterministic routing tests. If model
153-
variance is accepted, skip only the semantic assertion after proving the
154-
deterministic pipeline succeeded.
138+
- Start listeners before the action; scope predicates by stable IDs and poll for
139+
positive outcomes.
140+
- For a negative case, first prove the action completed, then use a bounded
141+
observation window.
142+
- Use unique inputs when the product deduplicates, isolate state between cases,
143+
and clean up in `finally`/`AfterAll`.
144+
- Keep model-semantic assertions separate from deterministic pipeline checks.
155145

156146
## 6. Wire the Release Checklist
157147

@@ -226,40 +216,19 @@ regressions or the PR changes common harness/product infrastructure.
226216

227217
## 8. Prove the Correct Build Ran
228218

229-
Build and deploy according to the changed area. For WTA + Terminal changes:
230-
231-
1. Build WTA with the explicit target matching the package architecture:
232-
`--target x86_64-pc-windows-msvc` for x64 or
233-
`--target aarch64-pc-windows-msvc` for ARM64. Package deployment prefers the
234-
explicit-target artifact.
235-
2. Build the C++ package after WTA.
236-
3. Deploy/redeploy the package and select it with `ITE2E_PACKAGE` or
237-
`-Package Dev`.
238-
4. Verify package version/path, runtime logs, or a changed observable.
239-
240-
Do not infer deployment success from compilation alone. A stale packaged
241-
`wta.exe`, generated shell-integration script, profile reference, or AppX
242-
staging directory can make a new test exercise old code.
219+
Build and deploy the changed area. For WTA changes, build the explicit target
220+
matching the package architecture before the C++ package, deploy it, select it
221+
with `ITE2E_PACKAGE` or `-Package Dev`, and verify a runtime version, path, log,
222+
or changed observable. Compilation alone does not prove the deployed package
223+
contains the new `wta.exe` or generated shell integration.
243224

244225
## 9. Deliver the Test PR
245226

246-
Before committing:
247-
248-
```powershell
249-
git -c core.whitespace=cr-at-eol diff --check
250-
git status --short --branch
251-
```
227+
Before committing, run `git -c core.whitespace=cr-at-eol diff --check`.
252228

253229
The PR description must include:
254230

255-
- Target PR and issue.
256-
- Missing integration boundary now covered.
257-
- Positive, negative, and compatibility cases added.
258-
- New checklist IDs and proof they become `[x]`.
259-
- Exact targeted and regression test totals.
260-
- Every skip with its reason.
231+
- Target PR/issue and the integration boundary added.
232+
- Cases and checklist IDs.
233+
- Targeted/regression totals and skip reasons.
261234
- Package/build tested.
262-
263-
Do not claim complete regression coverage if environment-dependent suites were
264-
not run. Distinguish deterministic passes from accepted model variance and
265-
unavailable prerequisites.

0 commit comments

Comments
 (0)