Browser Agent Guide 🧭

Guardrails and a guide that keep browser-operating AIs on track.

English · 한국어 · 日本語 · 中文

---

Browser Agent Guide is a Chrome (Manifest V3) extension that lets non-engineers add notes and drawings (circle/box/arrow/freehand around an element) to any web page just by clicking — so a browser-operating AI (this extension's chat, or an external AI chat UI) understands the page context and behaves deterministically. (Markers and cue-buttons still exist, but are added by the AI through its verb registry rather than the click-to-add form.)

Annotations are persisted per page (origin + path) and restored on every visit (reproducibility), and each target element is re-resolved through a robust multi-signal anchor (stable-ID path, data-testid, name, aria-label, text match) so the same element is found every time (determinism). Copy for AI exports deterministic guide context to paste into an external AI chat.

You can also instruct the AI directly from the side-panel chat — but only through a closed, deterministic verb registry (clickAffordance, fillAffordance, markElement, addNote, …). Combined with stable element IDs and Structured Outputs, the same prompt produces the same actions.

Demo

Here is a demo showing the browser-operating AI shopping on Amazon, searching for specific items, and adding them to the cart:

If the animation does not load, you can 🎥 watch the original MP4 video directly.

Why guardrails?

Goal	How Browser Agent Guide delivers it
AI integration + Structured Outputs	The service worker calls the AI with a strict JSON Schema (reply + actions)
Store the API key	chrome.storage.local only, set from the options page
Remember pages as needed	Rules are added automatically after saved annotations or chat-driven page changes; injection memory can target the current URL, current domain, or all sites, and rules can still be edited manually
Add cues from the sidebar	The AI can use safe verb-registry actions; explicit HTML / CSS / JS injections can be saved as re-apply recipes
Deterministic, consistent operation	AI can only use registered verbs; elements get stable aiIds; recipes re-apply every load

The AI never improvises raw DOM calls. It chooses from the verb registry, so behaviour stays predictable: same instruction → same verb → same result. Page text and HTML attributes are treated as untrusted data. High-risk DOM mutation verbs such as setStyle, removeElement, and defineMarker are hidden from chat AI and rejected from chat / auto-recipe execution paths. injectHtml, injectCss, and injectScript are available only for explicit user-requested page additions and are saved as re-apply recipes when they succeed. injectScript runs through Chrome's User Scripts API, so it does not rely on page inline-script CSP.

Install (unpacked)

1. Open chrome://extensions
2. Toggle Developer mode on
3. Load unpacked → select this folder
4. Click the toolbar icon to open the side panel

Requires Chrome 135+ (Side Panel API + User Scripts API). On Chrome 138+, enable Allow User Scripts in the extension details page before using injectScript.

Setup

1. Right-click the icon → Options (or the 設定 / Settings button in the side panel)
2. ① AI connection — choose a provider (OpenAI / Anthropic / Gemini / custom OpenAI-compatible), paste the API key and model, then save
   • OpenAI-compatible: uses response_format: json_schema with strict: true
   • Anthropic: forces structured output via tool_choice
   • Gemini: uses generationConfig.responseMimeType: "application/json" with responseJsonSchema
3. Choose ② AI injection auto-save to decide whether injected HTML / CSS / JS should be saved for the current URL, current domain, or all sites
4. Save an annotation or ask the chat to change the page; Browser Agent Guide automatically remembers that scope
5. Optionally edit ③ remembered URLs / activation rules and ④ memory rules for verbs that re-apply on every page load

Usage

Open any page and type natural-language instructions into the side-panel chat:

• "Find the login button and highlight it"
• "Plant a Summarize button bottom-right; its purpose is to ask the AI to summarize this page"
• "Type Chrome extension into the search box and submit"
• "Inject a fixed notice on this page and keep showing it next time"
• "Inject CSS for this domain so headings are easier to scan"

The AI returns a structured reply (explanation) plus actions (verbs), which run in order. Each verb's result is shown inline in the chat. When a saved annotation or chat-driven page change succeeds, the selected URL / domain / all-sites scope and any durable re-apply rules are remembered automatically. Saved visual changes such as outlineElement re-apply after page refresh. To revert them, delete the matching remembered URL or memory rule in Settings, then refresh the page.

Toolbar:

• Leave cues: Add note (click an element to attach one AI-facing note; the target is framed in red) and Draw (circle/box/arrow/freehand a target and attach a comment).
• Guide AI: Copy for AI packages the current URL, page title, saved cues, and operable elements as text to paste into another AI chat. When drawings exist, Send image to AI burns the visual cues into a screenshot.
• Inspect page: List elements shows current operable elements.
• History reuses sent prompts, also via ↑/↓ at the textarea edge. Settings opens AI connection, memory, recipe, and daemon settings.

The side panel also shows the addressed Chrome tab (tabId, windowId, and tab position). Page-feedback captures store the same tab metadata in annotation.json / memo.md and expose it through the daemon MCP context, so two tabs on the same URL can be disambiguated by passing tabId (or windowId) to the MCP tools.

Drawing (circle/box/arrow/freehand around an element)

Press 🖍 お描き/Draw to enter drawing mode and sketch on the page (◯circle / ▭box / ↗arrow / ✎pen + color), then Done to attach a single AI-facing instruction. Each drawing is anchored to the element it encircles; coordinates are stored as fractions of the element's box, so the sketch follows the element and is restored at the same place on revisit, scroll, and reflow. The AI receives a plain-language description (e.g. "circled in red. comment: …"), so "look here / fix this circled part" instructions land precisely. Saved drawings (kind: Drawing) are editable/removable from the Page cues list.

Recovering a note you forgot to send (advanced)

Add note only saves to the extension's own chrome.storage.local (key aiAdvisorAnnotations) — it is not sent to the daemon/MCP until you press Send image to AI (or enable the daemon in Options — off by default; once enabled, notes are auto-sent). If you left a note and closed the tab before sending it, scripts/read-annotations-cdp.mjs reads it back directly from the browser via the Chrome DevTools Protocol — no daemon, no re-capture, no lost work:

# Chrome must be running with a remote-debugging port (see the doc for a safe launch recipe):
npm run read-annotations                         # prints every note/drawing, newest first
node scripts/read-annotations-cdp.mjs --kind note --scope example.com

Full setup, flags, and a note on why this bypasses chrome-devtools-mcp (its --categoryExtensions flag doesn't work over an attach connection as of v1.4.0) are in docs/reading-annotations-via-cdp.md.

Verb registry (excerpt)

Implemented in content/content-script.js as AI_VERBS. Every function name is a verb.

Verb	Purpose
annotatePage / listAffordances	Assign stable IDs / list interactive elements
clickAffordance / clickElement	Click
fillAffordance / fillInput / selectOption	Fill / select
submitForm / focusElement / scrollToElement	Submit / focus / scroll
highlightElement / outlineElement	Temporary highlight / persistent outline
injectHtml / injectCss / injectScript	Inject explicit HTML / CSS / JS and re-apply it on revisit
injectButton / injectPanel	Plant cue buttons and sanitized panels
waitForElement	Wait for an element
navigateTo / goBack / notify	Navigate / toast
readText / extractData / readSignals	Read / extract / read user signals
startAnnotating / startDrawing	Start the click-to-annotate / draw (circle/box/arrow/pen) mode

Affordances & human-AI cooperation

annotatePage assigns deterministic aiIds in document order (button#1, input-text#2, …). The AI references those IDs via clickAffordance({aiId}), so there's no drift from guessed selectors.

Buttons planted with injectButton carry a neutral DOM attribute such as data-bag-intent and, when clicked, record a signal readable via readSignals. This closes the loop: a human clicks a planted button → the AI reads the intent and continues the task.

For elements an author wants the AI to target reliably across reflows, the repo also defines the @agent: marker convention — a stable anchor data-agent-id="@agent:<path>" found with rg -n 'data-agent-id="@agent:'. See docs/agent-markers.md. (Dynamically-injected UI is out of scope — it already has data-bag-id + signals.) The /bag-workflow skill uses these markers, and npm run check:markers lints them.

Recipes on SPAs & async pages

Saved recipes (the injectHtml / injectCss / injectScript / outlineElement / injectButton / injectPanel allow-list) re-apply automatically. They also keep working on single-page apps and lazily-rendered content:

• SPA internal navigation — the content script watches location.href (via a MutationObserver, plus popstate / hashchange) and notifies the service worker, which re-evaluates the URL and re-applies matching recipes. No full page reload is required.
• Wait for async elements — add waitFor: { selector, timeoutMs } to a recipe action to defer it until the target element appears (deferred loading / client-side rendering). The default timeout is 5000 ms; if the element never shows up the action fails (it does not throw). Since a missing element waits out the full timeoutMs, gate it with when: { selectorExists } first if you'd rather skip than wait.
• Render only when a condition holds — add when: { urlContains, selectorExists, selectorAbsent } to a recipe action and it is skipped unless the condition matches. Use urlContains for per-screen targeting in a SPA, and selectorAbsent to avoid duplicate injection on re-apply.

waitFor and when are optional per-action fields you set in the recipe JSON (Options → Recipes; the template button ships with examples).

Architecture

sidepanel (chat UI)
   │  CHAT {text, history, tabId}
   ▼
background (service worker)
   │  ① COLLECT_CONTEXT → content (collect verb catalog + affordances)
   │  ② callAI (Structured Outputs → reply + actions)
   │  ③ RUN_ACTIONS → content (run verbs in order)
   ▼
content-script (in the page / verb registry + executor)

• lib/ai-client.js — provider-agnostic Structured Outputs call
• lib/prompt.js — system prompt carrying the verb catalog + affordances
• lib/site-matcher.js — URL / domain / regex matching
• lib/storage.js — settings persistence

The tabId is the live Chrome-session target for page actions. Page-feedback exports additionally persist {tabId, windowId, index, active} as capture metadata; tabId is not a durable browser-history ID, but it is the right discriminator while the tabs are open.

Security / privacy

• The API key is stored only in chrome.storage.local and sent only to the AI API you choose.
• Prompt history and page-scoped chat history are also stored in chrome.storage.local.
• Destructive/irreversible actions (submit, purchase, delete) are stated in reply before running.
• Recorded-workflow auto-run holds final confirm/purchase/delete clicks by default; Options can explicitly trust recorded steps and allow them.
• The verb registry is a closed set, so the AI cannot perform unexpected DOM operations.

Extending

Add a verb by appending one { description, args, run } entry to AI_VERBS in content/content-script.js. Its description and args automatically flow into the system prompt and the Structured Outputs schema (the verb enum).

Tech

Vanilla JavaScript, no build step. Manifest V3, Side Panel API, chrome.scripting, chrome.storage.

Quality gate

Run npm run check before packaging. It runs JavaScript syntax checks, the deterministic anchor test, and Playwright + axe UI checks for the side panel and options page.

The adopted anti-slop workflow is documented in docs/ui-quality-workflow.md.

License

MIT