feat(testing-framework): config-driven framework suite runner (#2509 PR1)#2570
Closed
quanru wants to merge 6 commits into
Closed
feat(testing-framework): config-driven framework suite runner (#2509 PR1)#2570quanru wants to merge 6 commits into
quanru wants to merge 6 commits into
Conversation
Deploying midscene with
|
| Latest commit: |
69db689
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://459eacf5.midscene.pages.dev |
| Branch Preview URL: | https://codex-testing-framework-conf.midscene.pages.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b9258795a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+117
to
+119
| await options.agent.runYaml( | ||
| dumpSingleStepTask(`${options.caseName}:${stepName}`, step), | ||
| ); |
There was a problem hiding this comment.
When yamlSteps is configured, every built-in step is sent through a separate agent.runYaml() call here. runYaml() creates a fresh ScriptPlayer each time, and ScriptPlayer resolves $variables from its per-player result, so a built-in step like aiQuery: ...; name: product followed later by aiAssert: "$product ..." fails with the variable undefined as soon as any custom step handler is registered. This regresses normal YAML flows that work through runBuiltinYamlCase; the split execution needs to carry the accumulated result forward or avoid splitting built-in segments that depend on each other.
Useful? React with 👍 / 👎.
Add the `@midscene/testing-framework` package implementing the Framework Config Runner MVP from #2509: - `defineMidsceneConfig` and `runMidsceneSuite` - load `midscene.config.ts` (testDir/include/exclude, testRunner, output, agentOptions, target, setup, yamlSteps) with validation - YAML case discovery (glob include/exclude, .yaml/.yml/.test.ts, stable sort) - default web (PlaywrightAgent) and android (agentFromAdbDevice) targets, plus fully custom setup - custom yamlSteps interleaved with built-in steps in authored order - each YAML case runs as one Rstest test via runRstest + virtual modules, with suite-level setup/teardown and shared state
When a project's midscene.config.ts imports playwright (or
@midscene/web/playwright), playwright's bundled @vitest/expect copy
defines Symbol.for('$$jest-matchers-object') on globalThis as a
non-configurable property. Calling runRstest in the same process then
fails with TypeError: Cannot redefine property because Rstest's own
@vitest/expect copy tries to redefine the same symbol.
Spawn a child process via child_process.fork() that loads runRstest
fresh, never having imported playwright. The parent forwards stdout
lines while reserving a sentinel-prefixed line for the structured run
result.
Mirror @midscene/cli behaviour so a project's midscene.config.ts can rely on env-driven model configuration without the user having to source .env manually before tsx run-suite.ts. Defaults: look for .env in cwd and the config directory, skip missing files silently, do not override existing process.env. The new config.env field opts a project out (enabled: false) or points the loader at a custom list of files (path).
After rebasing onto main (which introduced new top-level packages like @rstest/core via #2537), pnpm install regenerated entries for the testing-framework's @rstest/core dependency that were lost during the lockfile auto-merge.
quanru
force-pushed
the
codex/testing-framework-config-runner
branch
from
5eadd1c to
0c666f0
Compare
…ted" The worker silenced all rstest reporters to keep stdout clean, which also hid per-test errors. When a case never produced its own result file (e.g. a `beforeAll` hook threw before any case ran), the framework only reported "Not executed" with no clue about the real cause. Walk `result.files[]` to extract both case-level and suite-level errors (tagged with `kind`), forward them through `WorkerOutput.testErrors`, and fall through case match -> suite errors -> unhandledErrors in `readCaseResult` so failures always carry a real stack.
Replace runMidsceneSuite + fork worker with an in-process runMidsceneTest: the runner only resolves the config path, while the user's midscene.config (which may import Playwright) is imported solely inside the Rstest worker via a generated bootstrap module, so Playwright's @vitest/expect global no longer collides with Rstest's copy and forking is unnecessary. - registerMidsceneSuite (mode A bootstrap) discovers cases with top-level await and registers one test per YAML case against a shared agent - defineMidsceneCaseTest + emitRstestProject export a native Rstest project - runCase returns its result instead of throwing; summary counts a retried case once - add midscene-testing-framework bin (test/emit)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the Framework Config Runner MVP from the UI Testing Framework topic (#2509). Adds a new self-contained package
@midscene/testing-frameworkthat turns amidscene.config.tsplus natural-language YAML cases into an Rstest run.Scope is strictly limited to what #2509 and the
midscene-examplecodex/add-framework-setup-demodemos describe. Nomidscene test/midscene emitcommand, no extra DSL/watch/reporters, and no changes to the existing YAML CLI.What's included
defineMidsceneConfig— type-only helper that returns the config unchanged.runMidsceneSuite— loads the config, discovers cases, runs them through Rstest, writes the summary. Designed so a project'srun-suite.tscollapses toawait runMidsceneSuite().target,testDir,include,exclude,testRunner,output,agentOptions,setup,yamlSteps. Errors on missingtestDir/include, ontarget+setuptogether, and on custom steps that override built-in step names.include/exclude,.yaml/.yml/.test.ts, stable sort.web(PlaywrightAgent: launch → context(viewport) → goto(url) → agent, teardown closes context/browser) andandroid(agentFromAdbDevicewithdeviceId/launch/device-option passthrough). Customsetup({ projectDir, agentOptions })takes precedence.yamlStepsinterleaved with built-in steps in authored order; multi-key built-in steps (e.g.aiInput+value) are handled; unknown steps throw a clear error;stateis shared across the suite.test, generated into a single virtual suite module so the agent is created once (beforeAll), torn down once (afterAll, even on failure), andstateis shared. Reuses the samerunRstest+VirtualModulesPluginmechanism as the existing CLI YAML runner instead of a bespoke concurrent runner.testRunner.{maxConcurrency,bail,testTimeout,retry}map to Rstest config.Design notes
@midscene/cli: the long-term direction iscli → testing-framework(the plannedemitPR), so a reverse dependency would create a cycle. The existingpackages/cli/src/framework/*YAML path is left untouched.@rstest/coreis a peer dependency (*); thepkg.pr.newpreview is only a devDependency for local validation and never a published hard dependency.@rsbuild/coreis resolved at runtime relative to the consumer's@rstest/core.Validation
npx nx build testing-framework— ✅ (incl. d.ts)packages/testing-frameworkunit tests — ✅ 33/33 (config load/validate, discovery, web/android setup, custom setup, yamlSteps ordering/unknown/state, Rstest param mapping, summary)runYaml, custom steps invoked, sharedstate, and teardown once — including teardown still running when every case fails, withexitCode=1and a failed summary.npx nx test cli— ✅ 148/148 (existingmidscene ./script.yaml/--configsemantics unchanged; no CLI source touched)pnpm run lint— ✅Out of scope / follow-ups
midscene emit ./project-folder— emit an explicit Rstest project and have the CLI consume this package.@rstest/corepkg.pr.newpreview with a published version before release (consumer/emitted projects).output.reportDiris typed for fidelity with the doc but not yet consumed (reports come fromagentOptions);case.test.tshas minimal discovery + type entry only, no extra DSL.midscene-exampledemos'run-suite.tstorunMidsceneSuite()lives in the example repo.