Skip to content

Commit 5c576b0

Browse files
ottomaoottomao
committed
docs(site): add UI testing framework topic
1 parent 62c0ba7 commit 5c576b0

7 files changed

Lines changed: 886 additions & 0 deletions

‎apps/site/docs/en/ui-testing-engineering.mdx‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Engineering UI Tests with Midscene
2+
3+
YAML gets a UI test running quickly. Engineering practice keeps it reliable after the test suite grows. The main rule is simple: keep user-facing UI intent in YAML, and keep deterministic environment and business logic in code.
4+
5+
## Recommended project shape
6+
7+
```text
8+
.
9+
.env
10+
midscene.config.yaml
11+
setup.ts
12+
e2e/
13+
login.yaml
14+
checkout.yaml
15+
mobile-smoke.yaml
16+
fixtures/
17+
account.ts
18+
device.ts
19+
reports/
20+
```
21+
22+
Use this shape as a guide, not a strict requirement. Small projects can start with one YAML file. Larger projects usually need separate setup, fixtures, and CI configuration.
23+
24+
## Keep the boundary clear
25+
26+
| Concern | Put it in | Why |
27+
| --- | --- | --- |
28+
| User path, visual state, popup handling, navigation | YAML | Natural language is concise and resilient to UI changes |
29+
| Login, cookies, SSO, accounts, environment preparation | Setup scripts or fixtures | The logic is project-specific and often needs secrets or internal tools |
30+
| API responses, database records, analytics events, amount calculation | JavaScript/TypeScript assertions | These checks need deterministic data and exact failure messages |
31+
| Batch execution, concurrency, summaries, report artifacts | CLI and CI configuration | They are execution concerns, not test intent |
32+
33+
## Environment configuration
34+
35+
Midscene CLI loads `.env` from the command working directory. A typical file contains model settings:
36+
37+
```ini filename=.env
38+
MIDSCENE_MODEL_BASE_URL="https://your-model-service.example.com/v1"
39+
MIDSCENE_MODEL_API_KEY="your API Key"
40+
MIDSCENE_MODEL_NAME="your model name"
41+
MIDSCENE_MODEL_FAMILY="your model family"
42+
```
43+
44+
In CI, store sensitive values in the CI secret manager and expose them as environment variables. Use `--dotenv-debug` when you need to inspect how local variables are loaded, and `--dotenv-override` only when the `.env` file should replace existing process variables.
45+
46+
## Login and setup
47+
48+
Most real tests should not spend every run manually completing login. Prepare state before the YAML flow starts:
49+
50+
- create or select a test account;
51+
- complete SSO and inject cookies;
52+
- install or launch the app under test;
53+
- select an Android or iOS device;
54+
- configure a lane, feature flag, or internal environment;
55+
- seed backend data needed by the test.
56+
57+
After setup, keep the YAML focused on the business behavior:
58+
59+
```yaml
60+
web:
61+
url: https://internal.example.com/dashboard
62+
63+
tasks:
64+
- name: Check dashboard
65+
flow:
66+
- aiAssert: The dashboard is loaded and user information is visible
67+
```
68+
69+
## Batch runs and CI
70+
71+
Use the CLI for suite-level execution:
72+
73+
```bash
74+
midscene --files './e2e/**/*.yaml' --concurrent 4 --continue-on-error --summary index.json
75+
```
76+
77+
Recommended CI artifacts:
78+
79+
- the summary JSON file;
80+
- each YAML run result;
81+
- visual report HTML files;
82+
- logs from the application or device when available.
83+
84+
Keep reports even for successful scheduled runs. They make UI drift and flaky behavior easier to investigate later.
85+
86+
## Browser sessions
87+
88+
For Web tests, choose the connection mode based on the state you need:
89+
90+
| Mode | Best for |
91+
| --- | --- |
92+
| Default browser launch | Clean and repeatable tests |
93+
| Headed mode | Local debugging |
94+
| CDP connection | Remote browsers or managed browser services |
95+
| Chrome bridge mode | Reusing an existing desktop Chrome session, cookies, extensions, or internal login |
96+
97+
See [YAML script runner](./yaml-script-runner) and [Bridge to the desktop Chrome](./bridge-mode) for the exact configuration.
98+
99+
## Mobile devices
100+
101+
Mobile tests usually need more setup than Web tests. Keep device management outside the YAML journey when possible:
102+
103+
- reserve a device from a device pool;
104+
- install the target build;
105+
- clear or seed app state;
106+
- configure network, region, or account data;
107+
- collect device logs after failure.
108+
109+
Use platform-specific YAML helpers for small local actions, such as `runAdbShell`, `runWdaRequest`, `launch`, and `terminate`. Use setup scripts for broader device orchestration.
110+
111+
## Deterministic checks
112+
113+
AI assertions are best for UI state: visible text, layout meaning, workflow completion, and visual conditions. Use code when correctness depends on exact values:
114+
115+
```ts
116+
expect(createOrderResponse.status).toBe(200);
117+
expect(order.total).toBe(expectedTotal);
118+
expect(analyticsEvents).toContainEqual({
119+
name: 'checkout_submit',
120+
source: 'recommendation',
121+
});
122+
```
123+
124+
This keeps failures actionable. A visual assertion explains what the user saw. A code assertion explains which business invariant failed.
125+
126+
## Reports and debugging
127+
128+
When a test fails, inspect the report before editing the prompt. Check:
129+
130+
- the screenshot before the failed step;
131+
- the AI action and assertion text;
132+
- whether the page or app was still loading;
133+
- whether login or setup state was missing;
134+
- whether the test should use code for an exact business rule.
135+
136+
Prompt changes should clarify intent. They should not encode brittle layout details unless the layout itself is what you are testing.
137+
138+
## Agent orchestration
139+
140+
Some internal workflows require more than a fixed runner: device pools, SSO, logs, network tools, backend queries, and report analysis. In those cases, treat YAML as the stable test asset and let a coding agent or internal runner orchestrate the surrounding tools.
141+
142+
Midscene remains responsible for UI understanding, actions, screenshots, and reports. The orchestrator prepares the environment, calls the right tests, gathers external evidence, and summarizes failures.
143+
144+
## Reference pages
145+
146+
- [Write UI tests with YAML](./ui-testing-yaml-quick-start)
147+
- [Workflow in YAML format](./automate-with-scripts-in-yaml)
148+
- [YAML script runner](./yaml-script-runner)
149+
- [Integrate Midscene with any interface](./integrate-with-any-interface)

‎apps/site/docs/en/ui-testing-framework.mdx‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# AI-native UI Testing Framework
2+
3+
Midscene helps teams build UI tests around user intent instead of fragile selectors. Natural-language steps describe what a user wants to do, while scripts, setup code, CLI execution, and reports keep the workflow maintainable enough for real projects.
4+
5+
This guide introduces the framework-level view. If you want to run your first test immediately, start with [Write UI tests with YAML](./ui-testing-yaml-quick-start).
6+
7+
## Why UI testing needs a new shape
8+
9+
Traditional UI testing frameworks such as Playwright and Puppeteer are stable and programmable, but they require teams to maintain selectors, waits, fixtures, browser contexts, device access, and report pipelines. AI-driven automation lowers the authoring cost by letting people write steps such as "open the first product" or "confirm the cart contains one item", but a prompt alone is not enough for production testing.
10+
11+
Real UI tests still need:
12+
13+
- login state, SSO, accounts, cookies, and environment setup;
14+
- deterministic checks for APIs, databases, analytics events, and business rules;
15+
- Web, Android, iOS, desktop, and other runtime environments;
16+
- command-line execution, CI integration, reports, screenshots, and debugging artifacts.
17+
18+
Midscene's testing framework is the layer that connects those pieces.
19+
20+
| Pain point | What it looks like | Midscene's approach |
21+
| --- | --- | --- |
22+
| Simple smoke tests require too much framework code | A business path needs fixtures, waits, selectors, and runner setup before it can run | Express the path in YAML with natural-language steps |
23+
| Prompt demos do not scale into test projects | Login, data setup, CI, and debugging remain outside the prompt | Keep setup and deterministic checks in scripts, while AI handles UI intent |
24+
25+
## Adoption path
26+
27+
Midscene should be adopted gradually: first make the business path runnable, then connect environment setup, and only move into a fully custom project when the team needs control over the runner, fixtures, assertions, and orchestration.
28+
29+
| Stage | Best for | Recommended shape |
30+
| --- | --- | --- |
31+
| Smoke test | Verify a key business path quickly | YAML + case. Write the entry point, steps, and assertions in YAML, then run it from the CLI and inspect the report |
32+
| Simple customization | Tests need login state, accounts, cookies, devices, or a small amount of environment preparation | YAML + `setup.js`. Keep the business path in YAML and put pre-run preparation in setup |
33+
| Fully custom | The team needs to integrate an existing test project, CI, internal tools, data checks, or a custom runner | Use `midscene emit` to export an independent Rstest project, then maintain it like a standard test project |
34+
35+
### Smoke test: YAML + case
36+
37+
This stage is about one thing: make the core path easy to express, run, and review. The test author writes YAML for the entry point, actions, and expected result. Midscene executes the flow, captures screenshots, and generates the report.
38+
39+
```yaml
40+
web:
41+
url: https://shop.example.com
42+
43+
tasks:
44+
- name: Guest checkout smoke test
45+
flow:
46+
- aiAct: Search for "running shoes"
47+
- aiAct: Open the first product
48+
- aiAssert: The cart page shows one product and the checkout button
49+
```
50+
51+
### Simple customization: YAML + setup.js
52+
53+
Once tests become part of daily work, they usually need login state, test accounts, cookies, device preparation, staging lanes, or other environment details. Those concerns should not be squeezed into natural-language steps. They belong in `setup.js`.
54+
55+
Here, `setup.js` means the pre-run setup script for the test project. It runs before YAML cases and prepares the browser, device, account, or backend data for the test. A simple customized project can look like this:
56+
57+
```text
58+
.
59+
setup.js
60+
midscene.config.yaml
61+
e2e/
62+
dashboard.yaml
63+
checkout.yaml
64+
```
65+
66+
`midscene.config.yaml` manages model settings, runtime options, report output, and selected cases. `setup.js` manages project-specific preparation. `e2e/*.yaml` only describes the business path.
67+
68+
YAML still describes the business path:
69+
70+
```yaml
71+
web:
72+
url: https://internal.example.com/dashboard
73+
74+
tasks:
75+
- name: Check dashboard
76+
flow:
77+
- aiAssert: The dashboard is loaded and user information is visible
78+
```
79+
80+
`setup.js` gets the test to the right starting point, such as logging in, injecting cookies, preparing data, or connecting a device. Its value is giving deterministic preparation logic a clear home while the YAML case stays lightweight:
81+
82+
```js
83+
export default async function setup({ browser, context, device }) {
84+
const cookies = await loginByTestAccount(process.env.TEST_ACCOUNT);
85+
await context.addCookies(cookies);
86+
87+
await prepareTestData({
88+
user: process.env.TEST_ACCOUNT,
89+
scenario: 'dashboard-smoke',
90+
});
91+
}
92+
```
93+
94+
The code above shows the boundary: login, accounts, data, and devices belong in setup. What the user should accomplish after the page opens stays in YAML.
95+
96+
### Fully custom: emit an independent Rstest project
97+
98+
When a team already has a test platform, CI rules, internal fixtures, data checks, and report systems, Midscene should not force everything through a fixed YAML runner. The core capability for fully custom projects is `emit`: export a lightweight Midscene project into an independent Rstest project.
99+
100+
```bash
101+
midscene emit ./project-folder
102+
```
103+
104+
The emitted project can look like this:
105+
106+
```text
107+
project-folder/
108+
package.json
109+
rstest.config.ts
110+
setup.ts
111+
e2e/
112+
dashboard.test.ts
113+
checkout.test.ts
114+
fixtures/
115+
account.ts
116+
device.ts
117+
reports/
118+
midscene-report/
119+
```
120+
121+
At this stage, YAML is no longer the boundary of what the framework can do. It is the migration entry point. Teams can keep the business expression from YAML cases, or move complex logic into Rstest test files, fixtures, and internal tools. Midscene handles UI actions and visual assertions, while your own code handles environment orchestration, API checks, database checks, and failure analysis.
122+
123+
The point of this stage is not to write everything in YAML. The point is to let Midscene generate a standard test project that the team can fully own.
124+
125+
## Next steps
126+
127+
- Run your first test: [Write UI tests with YAML](./ui-testing-yaml-quick-start)
128+
- Add setup, CI, reports, and deterministic checks: [Engineering UI tests with Midscene](./ui-testing-engineering)
129+
- Look up every YAML field: [Workflow in YAML format](./automate-with-scripts-in-yaml)
130+
- Look up every CLI flag: [YAML script runner](./yaml-script-runner)

0 commit comments

Comments
 (0)