Skip to content

refactor(core): update describeElementAtPoint integration#2692

Open
fishear wants to merge 2 commits into
mainfrom
feat/describe_eval
Open

refactor(core): update describeElementAtPoint integration#2692
fishear wants to merge 2 commits into
mainfrom
feat/describe_eval

Conversation

@fishear

@fishear fishear commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Move element description helpers out of Agent and expose describeElementAtPoint utilities from core
  • Add locatorVerifyFn so callers can customize describeElementAtPoint verification and retry/deepLocate behavior
  • Move recorder UI describer implementation and tests into playground, keeping recorder fallback out of core public APIs
  • Update Studio/playground integration and related tests

Validation

  • pnpm run lint
  • pnpm --dir packages/core test -- tests/unit-test/agent-describe-element.test.ts
  • pnpm --dir packages/core build
  • pnpm --dir packages/playground test -- tests/unit/recorder-ui-describer.test.ts
  • pnpm --dir apps/studio test -- tests/studio-recorder-main-codegen.test.ts
  • pnpm --dir apps/studio build
  • git diff --check

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 17, 2026

Copy link
Copy Markdown

Deploying midscene with  Cloudflare Pages  Cloudflare Pages

Latest commit: ace70ae
Status: ✅  Deploy successful!
Preview URL: https://954c8e37.midscene.pages.dev
Branch Preview URL: https://feat-describe-eval.midscene.pages.dev

View logs

@fishear fishear requested review from EAGzzyCSL and yuyutaotao and removed request for yuyutaotao June 17, 2026 11:23
@fishear fishear enabled auto-merge (squash) June 17, 2026 11:24
…er describer out of core

Move recorder UI describer implementation and tests into playground.

- Keep describeElementAtPoint and locatorVerifyFn exposed from core as
- general-purpose element description utilities.

- Update Studio recorder codegen to import recorder describer from playground.
@fishear fishear force-pushed the feat/describe_eval branch from a94debd to 4fe7b07 Compare June 17, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant