Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/site/docs/en/integrate-with-playwright.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ This usually happens when the page uses a native `select` element for the dropdo

First, check the screenshot in the report. If the dropdown options do not appear in the report screenshot after the click, this is very likely the cause.

You can enable `forceChromeSelectRendering` to force Chrome to render the `select` dropdown instead. Once enabled, the dropdown will appear in screenshots and can be recognized by Playwright. The dropdown style will also look noticeably different from the operating system's default style.
Midscene enables `forceChromeSelectRendering` by default, which forces Chrome to render the `select` dropdown so it appears in screenshots and can be recognized by Playwright. The dropdown style will look noticeably different from the operating system's default style. If you need the native rendering back, set `forceChromeSelectRendering: false`.

### The webpage continues to flash when running in headed mode

Expand Down
2 changes: 1 addition & 1 deletion apps/site/docs/en/integrate-with-puppeteer.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ npx puppeteer browsers install chrome --base-url="https://registry.npmmirror.com

If the dropdown options do not appear in the report screenshot after you click the field, the page is usually using a native `select` element. In that case, the expanded dropdown is rendered by the operating system instead of inside the webpage.

You can enable `forceChromeSelectRendering` to force Chrome to render the dropdown. For the detailed explanation and how to confirm the issue, see [Playwright FAQ — Cannot click the dropdown](./integrate-with-playwright#cannot-click-the-dropdown).
Midscene enables `forceChromeSelectRendering` by default to force Chrome to render the dropdown; set it to `false` to opt out. For the detailed explanation and how to confirm the issue, see [Playwright FAQ — Cannot click the dropdown](./integrate-with-playwright#cannot-click-the-dropdown).

### The webpage continues to flash when running in headed mode

Expand Down
4 changes: 2 additions & 2 deletions apps/site/docs/en/web-api-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ In addition to the base agent options, Puppeteer exposes:
- `waitForNetworkIdleTimeout: number` — Wait for network idle between actions to reduce flakiness. Default `2000` (set `0` to skip waiting).
- `enableTouchEventsInActionSpace: boolean` — Add touch gestures (like swipe) to the action space so the agent can handle touch-only interactions. Default `false`.
- `keyboardTypeDelay: number` — Per-character delay (ms) forwarded to Puppeteer's `page.keyboard.type`. Default `undefined`, which leaves the option unset and uses Puppeteer's own default. You usually do not need to configure it; raise it (e.g. `80`) only when a controlled input drops characters under fast typing.
- `forceChromeSelectRendering: boolean` — Force `select` elements to render with Chrome's base-select styling so they're visible in screenshots/element extraction; requires Puppeteer > `24.6.0`.
- `forceChromeSelectRendering: boolean` — Force `select` elements to render with Chrome's base-select styling so they're visible in screenshots/element extraction; requires Puppeteer > `24.6.0`. Defaults to `true`; set to `false` to opt out (e.g. on older Chrome/Puppeteer versions).
- `customActions: DeviceAction[]` — Register bespoke actions defined via `defineAction` so planning can call domain-specific steps.

### Usage notes
Expand Down Expand Up @@ -136,7 +136,7 @@ const agent = new PlaywrightAgent(page, {
- `waitForNetworkIdleTimeout: number` — Wait between actions for network idle. Default `2000` (set `0` to disable).
- `enableTouchEventsInActionSpace: boolean` — Add touch gestures (like swipe) to the action space so the agent can handle touch-only interactions. Default `false`.
- `keyboardTypeDelay: number` — Per-character delay (ms) forwarded to Playwright's `page.keyboard.type`. Default `undefined`, which leaves the option unset and uses Playwright's own default. You usually do not need to configure it; raise it (e.g. `80`) only when a controlled input drops characters under fast typing.
- `forceChromeSelectRendering: boolean` — Force `select` elements to render with Chrome's base-select styling so they're visible in screenshots/element extraction; requires Playwright ≥ `1.52.0`.
- `forceChromeSelectRendering: boolean` — Force `select` elements to render with Chrome's base-select styling so they're visible in screenshots/element extraction; requires Playwright ≥ `1.52.0`. Defaults to `true`; set to `false` to opt out (e.g. on older Chrome/Playwright versions).
- `customActions: DeviceAction[]` — Extend planning with project-specific actions.

### Usage notes
Expand Down
2 changes: 1 addition & 1 deletion apps/site/docs/zh/integrate-with-playwright.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ npx playwright install --with-deps chromium

建议先检查报告中的截图:如果点击下拉框后,报告截图里确实没有出现下拉选项,基本就可以判断是这个问题。

可以开启 `forceChromeSelectRendering` 选项,强制由 Chrome 来渲染 `select` 下拉框这样下拉框就会出现在页面截图中,也能被 Playwright 正常识别。开启后,下拉框样式通常会和操作系统默认样式有明显区别。
Midscene 默认开启 `forceChromeSelectRendering` 选项,强制由 Chrome 来渲染 `select` 下拉框这样下拉框就会出现在页面截图中,也能被 Playwright 正常识别。开启后,下拉框样式通常会和操作系统默认样式有明显区别。如果需要恢复系统原生渲染,可将 `forceChromeSelectRendering` 设为 `false`

### 浏览器界面持续闪动

Expand Down
2 changes: 1 addition & 1 deletion apps/site/docs/zh/integrate-with-puppeteer.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ npx puppeteer browsers install chrome --base-url="https://registry.npmmirror.com

如果点击下拉框后,报告中的截图没有出现下拉选项,通常是页面使用了原生 `select` 标签,导致展开后的下拉面板由操作系统原生控件渲染,没有真正出现在浏览器页面里。

这时可以开启 `forceChromeSelectRendering` 选项,强制由 Chrome 来渲染下拉框。详细原因、判断方式和效果说明,请参考 [Playwright FAQ — 下拉框点击不到](./integrate-with-playwright#下拉框点击不到)。
Midscene 默认开启 `forceChromeSelectRendering` 选项,强制由 Chrome 来渲染下拉框;如需关闭可将其设为 `false`。详细原因、判断方式和效果说明,请参考 [Playwright FAQ — 下拉框点击不到](./integrate-with-playwright#下拉框点击不到)。

### 浏览器界面持续闪动

Expand Down
4 changes: 2 additions & 2 deletions apps/site/docs/zh/web-api-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ const agent = new PuppeteerAgent(page, {
- `waitForNetworkIdleTimeout: number` —— 每次操作后等待网络空闲的时间,默认 `2000`(设为 `0` 关闭)。
- `enableTouchEventsInActionSpace: boolean` —— 在动作空间里增加触摸手势(如滑动),用于需要触摸事件的页面,默认 `false`。
- `keyboardTypeDelay: number` —— 透传给 Puppeteer `page.keyboard.type` 的每字符延迟(毫秒)。默认值为 `undefined`,表示 Midscene 不传该选项,使用 Puppeteer 自身默认值。通常无需配置;只有当受控输入框在快速输入下出现丢字等特殊情况时,再调大该值(例如 `80`)。
- `forceChromeSelectRendering: boolean` —— 强制 `select` 元素使用 Chrome 的 base-select 样式,避免系统原生样式导致截图/元素提取不可见;需要 Puppeteer > `24.6.0`。
- `forceChromeSelectRendering: boolean` —— 强制 `select` 元素使用 Chrome 的 base-select 样式,避免系统原生样式导致截图/元素提取不可见;需要 Puppeteer > `24.6.0`。默认值为 `true`;如需关闭(例如旧版 Chrome/Puppeteer)可设为 `false`。
- `customActions: DeviceAction[]` —— 借助 `defineAction` 注册自定义动作,让规划器可以调用领域特定步骤。

### 使用说明
Expand Down Expand Up @@ -136,7 +136,7 @@ const agent = new PlaywrightAgent(page, {
- `waitForNetworkIdleTimeout: number` —— 每次操作后等待网络空闲的时间,默认 `2000`(设为 `0` 关闭)。
- `enableTouchEventsInActionSpace: boolean` —— 在动作空间里增加触摸手势(如滑动),用于需要触摸事件的页面,默认 `false`。
- `keyboardTypeDelay: number` —— 透传给 Playwright `page.keyboard.type` 的每字符延迟(毫秒)。默认值为 `undefined`,表示 Midscene 不传该选项,使用 Playwright 自身默认值。通常无需配置;只有当受控输入框在快速输入下出现丢字等特殊情况时,再调大该值(例如 `80`)。
- `forceChromeSelectRendering: boolean` —— 强制 `select` 元素使用 Chrome 的 base-select 样式,避免系统原生样式导致截图/元素提取不可见;需要 Playwright ≥ `1.52.0`。
- `forceChromeSelectRendering: boolean` —— 强制 `select` 元素使用 Chrome 的 base-select 样式,避免系统原生样式导致截图/元素提取不可见;需要 Playwright ≥ `1.52.0`。默认值为 `true`;如需关闭(例如旧版 Chrome/Playwright)可设为 `false`。
- `customActions: DeviceAction[]` —— 追加项目特有的动作,供规划器调用。

### 使用说明
Expand Down
10 changes: 4 additions & 6 deletions packages/core/src/report-generator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,6 @@ export class ReportGenerator implements IReportGenerator {
if (options.reuseExistingReport) {
this.hydrateStateFromExistingReport();
}
this.printReportPath('will be generated at');
}

static create(
Expand Down Expand Up @@ -205,25 +204,24 @@ export class ReportGenerator implements IReportGenerator {
return undefined;
}

this.printReportPath('finalized');
return this.reportPath;
}

getReportPath(): string | undefined {
return this.reportPath;
}

private printReportPath(verb: string): void {
private printReportPath(): void {
if (!this.autoPrint || !this.reportPath) return;
if (globalConfigManager.getEnvConfigInBoolean(MIDSCENE_REPORT_QUIET))
return;

if (this.screenshotMode === 'directory') {
logMsg(
`Midscene - report ${verb}: npx serve ${dirname(this.reportPath)}`,
`Midscene - report file updated: npx serve ${dirname(this.reportPath)}`,
);
} else {
logMsg(`Midscene - report ${verb}: ${this.reportPath}`);
logMsg(`Midscene - report file updated: ${this.reportPath}`);
}
}

Expand All @@ -245,7 +243,7 @@ export class ReportGenerator implements IReportGenerator {

if (!this.firstWriteDone) {
this.firstWriteDone = true;
this.printReportPath('generated');
this.printReportPath();
}
}

Expand Down
85 changes: 84 additions & 1 deletion packages/core/tests/unit-test/report-generator.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import {
type ReportMeta,
type UIContext,
} from '@/types';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
countGroupedDumpScripts,
extractGroupedDumpScripts,
Expand Down Expand Up @@ -602,6 +602,89 @@ describe('ReportGenerator — append-only model', () => {
});
});

describe('autoPrint — report path logging', () => {
let logSpy: ReturnType<typeof vi.spyOn>;

beforeEach(() => {
logSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
});

afterEach(() => {
logSpy.mockRestore();
});

const updatedLogs = () =>
logSpy.mock.calls
.map((args) => String(args[0]))
.filter((msg) => msg.includes('Midscene - report file updated:'));

it('does not log on construction, and logs "report file updated" once on first write', async () => {
const reportPath = join(tmpDir, 'autoprint-inline.html');
const generator = new ReportGenerator({
reportPath,
screenshotMode: 'inline',
});

// Constructing the generator must not print anything yet.
expect(updatedLogs()).toHaveLength(0);

const execution = createExecution([
ScreenshotItem.create(fakeBase64(100), Date.now()),
]);

generator.onExecutionUpdate(execution, defaultReportMeta);
await generator.flush();

expect(updatedLogs()).toEqual([
`Midscene - report file updated: ${reportPath}`,
]);

// Subsequent updates and finalize must not print the tip again.
generator.onExecutionUpdate(execution, defaultReportMeta);
await generator.flush();
await generator.finalize();

expect(updatedLogs()).toHaveLength(1);
});

it('does not log when autoPrint is disabled', async () => {
const reportPath = join(tmpDir, 'autoprint-disabled.html');
const generator = new ReportGenerator({
reportPath,
screenshotMode: 'inline',
autoPrint: false,
});

generator.onExecutionUpdate(
createExecution([ScreenshotItem.create(fakeBase64(100), Date.now())]),
defaultReportMeta,
);
await generator.flush();
await generator.finalize();

expect(updatedLogs()).toHaveLength(0);
});

it('points at "npx serve <dir>" in directory mode', async () => {
const reportDir = join(tmpDir, 'autoprint-dir');
const reportPath = join(reportDir, 'index.html');
const generator = new ReportGenerator({
reportPath,
screenshotMode: 'directory',
});

generator.onExecutionUpdate(
createExecution([ScreenshotItem.create(fakeBase64(100), Date.now())]),
defaultReportMeta,
);
await generator.flush();

expect(updatedLogs()).toEqual([
`Midscene - report file updated: npx serve ${reportDir}`,
]);
});
});

describe('directory mode — incremental PNG writes', () => {
it('should write each screenshot as a PNG file exactly once', async () => {
const reportDir = join(tmpDir, 'dir-test');
Expand Down
17 changes: 10 additions & 7 deletions packages/web-integration/src/playwright/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,20 +52,23 @@ export class PlaywrightAgent extends PageAgent<PlaywrightWebPage> {
const webPage = new PlaywrightWebPage(page, opts);
super(webPage, opts);

const { forceSameTabNavigation = true, forceChromeSelectRendering } =
const { forceSameTabNavigation = true, forceChromeSelectRendering = true } =
opts ?? {};

if (forceSameTabNavigation) {
forceClosePopup(page, debug);
}

if (forceChromeSelectRendering) {
// Check Playwright version requirement (>= 1.52)
const playwrightVersion = getPlaywrightVersion();
if (playwrightVersion && !semver.gte(playwrightVersion, '1.52.0')) {
console.warn(
`[midscene:error] forceChromeSelectRendering requires Playwright >= 1.52.0, but current version is ${playwrightVersion}. This feature may not work correctly.`,
);
// Only warn about version requirements when the user explicitly opted in;
// it is on by default, so we should not nag users on older Playwright.
if (opts?.forceChromeSelectRendering === true) {
const playwrightVersion = getPlaywrightVersion();
if (playwrightVersion && !semver.gte(playwrightVersion, '1.52.0')) {
console.warn(
`[midscene:error] forceChromeSelectRendering requires Playwright >= 1.52.0, but current version is ${playwrightVersion}. This feature may not work correctly.`,
);
}
}
applyChromeSelectRendering(page);
}
Expand Down
11 changes: 11 additions & 0 deletions packages/web-integration/src/puppeteer/base-page.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1198,9 +1198,20 @@ export function forceClosePopup(
*
* Adds a style tag with CSS rules to make all select elements use base-select appearance.
*/
// Track pages that already have the select-rendering style wired up so the
// immediate injection and the `load` listener are only registered once per page,
// even if multiple agents are created for the same page.
const forceSelectRenderingPages = new WeakSet<object>();

export function forceChromeSelectRendering(
page: PuppeteerPage | PlaywrightPage,
): void {
// Only inject once per page to avoid stacking duplicate `load` listeners.
if (forceSelectRenderingPages.has(page)) {
return;
}
forceSelectRenderingPages.add(page);

// Force Chrome to render select elements using base-select appearance
// Reference: https://developer.chrome.com/blog/a-customizable-select
const styleContent = `
Expand Down
16 changes: 10 additions & 6 deletions packages/web-integration/src/puppeteer/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,19 +46,23 @@ export class PuppeteerAgent extends PageAgent<PuppeteerWebPage> {
const webPage = new PuppeteerWebPage(page, opts);
super(webPage, opts);

const { forceSameTabNavigation = true, forceChromeSelectRendering } =
const { forceSameTabNavigation = true, forceChromeSelectRendering = true } =
opts ?? {};

if (forceSameTabNavigation) {
forceClosePopup(page, debug);
}

if (forceChromeSelectRendering) {
const puppeteerVersion = getPuppeteerVersion();
if (puppeteerVersion && !semver.gte(puppeteerVersion, '24.6.0')) {
console.warn(
`[midscene:error] forceChromeSelectRendering requires Puppeteer > 24.6.0, but current version is ${puppeteerVersion}. This feature may not work correctly.`,
);
// Only warn about version requirements when the user explicitly opted in;
// it is on by default, so we should not nag users on older Puppeteer.
if (opts?.forceChromeSelectRendering === true) {
const puppeteerVersion = getPuppeteerVersion();
if (puppeteerVersion && !semver.gte(puppeteerVersion, '24.6.0')) {
console.warn(
`[midscene:error] forceChromeSelectRendering requires Puppeteer > 24.6.0, but current version is ${puppeteerVersion}. This feature may not work correctly.`,
);
}
}
applyChromeSelectRendering(page);
}
Expand Down
3 changes: 3 additions & 0 deletions packages/web-integration/src/web-element.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ export type WebPageOpt = {
* Reference: https://developer.chrome.com/blog/a-customizable-select
*
* When enabled, adds a style tag with `select { appearance: base-select !important; }` to the page.
*
* Defaults to `true`. Set to `false` to opt out (e.g. on older Chrome/driver
* versions that do not support `appearance: base-select`).
*/
forceChromeSelectRendering?: boolean;
beforeInvokeAction?: () => Promise<void>;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
import { forceChromeSelectRendering } from '@/puppeteer/base-page';
import { describe, expect, it, vi } from 'vitest';

const createMockPage = () =>
({
evaluate: vi.fn().mockResolvedValue(undefined),
on: vi.fn(),
}) as any;

describe('forceChromeSelectRendering', () => {
it('injects the style and registers a single load listener per page', async () => {
const page = createMockPage();

forceChromeSelectRendering(page);
// allow the immediate (async) injection to settle
await Promise.resolve();

expect(page.evaluate).toHaveBeenCalledTimes(1);
expect(page.on).toHaveBeenCalledTimes(1);
expect(page.on).toHaveBeenCalledWith('load', expect.any(Function));
});

it('is a no-op when called again for the same page', async () => {
const page = createMockPage();

forceChromeSelectRendering(page);
forceChromeSelectRendering(page);
forceChromeSelectRendering(page);
await Promise.resolve();

// still only injected once and only one load listener attached
expect(page.evaluate).toHaveBeenCalledTimes(1);
expect(page.on).toHaveBeenCalledTimes(1);
});

it('wires up each distinct page independently', async () => {
const pageA = createMockPage();
const pageB = createMockPage();

forceChromeSelectRendering(pageA);
forceChromeSelectRendering(pageB);
await Promise.resolve();

expect(pageA.on).toHaveBeenCalledTimes(1);
expect(pageB.on).toHaveBeenCalledTimes(1);
});
});
Loading