feat(cdp): report the page's real child frame tree by xrip · Pull Request #623 · h4ckf0r0day/obscura

xrip · 2026-08-10T20:35:52Z

What changed

Third of three stacked PRs for #600. Fixes #600 once all three land.

Depends on #621 and #622 and is stacked on them. GitHub will not let a cross-fork PR use a fork branch as its base, so this targets main and therefore also contains their commits. Merge #621, then #622, then this. Its own commit is feat(cdp): report the page's real child frame tree.

Page.getFrameTree returned "childFrames": [] however many frames a page had built, and no frame lifecycle event was ever emitted, so Playwright and Puppeteer saw a single-frame page and could not address a child at all. That is the protocol half of the issue report.

getFrameTree now carries the live hierarchy, nesting included. A child's protocol id is derived from its page's frame id and stays stable for the life of the frame, which is what lets a client match an attach event to the frame it later sees in the tree.
Each child is announced with Page.frameAttached, Page.frameNavigated and Page.frameStoppedLoading, and retracted with Page.frameDetached. Attach is emitted before navigate, because a client builds its frame from the attach event and treats navigation of a frame it has never seen as a protocol error.
The events come from a diff drawn after every dispatch, alongside the existing binding-call drain, rather than from the navigation handler. Script can add an iframe at any time, so a frame that first appears on a later command still gets reported, exactly once.

One behavior change worth review attention: loading a document now includes building the frames in it, so navigation does that work instead of leaving it to a caller that settles. Without this a CDP client that only navigates was told the page had no frames, because nothing had given them realms yet. Pages with no iframe skip it after a single native query_selector("iframe"), so the common case pays one selector query and no event-loop time. The pumping is bounded (8 rounds x 50ms) because a frame can add a frame and an unbounded loop would never finish on a page that adds one every turn.

Validation

cargo nextest run --release --features render --no-fail-fast                                          # 1416/1417, see note
cargo nextest run --release --no-default-features -p obscura-js -p obscura-browser                    # 342/342
cargo build --release -p obscura-cli --bins --no-default-features                                     # clean
CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=2 cargo build --release -p obscura-cli --bins --features render  # clean

New test crates/obscura-cdp/tests/child_frame_tree.rs drives real CDP against a local server whose page embeds a child that itself embeds a grandchild. It asserts the nested tree with correct parentId links, that frameAttached precedes frameNavigated for the same frame, and that a second command does not announce the same frame twice.

The one workspace failure is obscura-cdp::max_connections_cap max_connections_refuses_then_recovers, which fails identically on an unmodified main worktree in this environment.

Obstacle course: 29/33, identical to unmodified main run back to back on the same machine (observer-intersection, textdecoder, charset-shiftjis, fingerprint; the middle two are Windows console codepage mojibake and the last is local timezone). No regression from this change, but I cannot attest to 33/33 from this environment.

Rendering

Not applicable.

Performance

Navigation of a page without an iframe adds one native selector query and nothing else. A page with frames pays bounded pumping while its frames load, which is work the page needs done regardless.

Checklist

The change is focused and does not remove existing behavior without justification.
Tests cover the failure or feature.
Existing tests pass, including render and no-render configurations when affected.
I checked for CPU, latency, and memory regressions.
Public API or user-facing behavior changes are documented.

A child iframe's document was fetched and parsed, but the frame never got a scripting context: its <script> elements sat in the DOM as inert nodes. A parser-created <iframe src> was worse, because nothing started its load at all, so only a frame whose src was assigned from script was ever fetched. A frame now gets a realm of its own, a second v8::Context in the page's isolate. The startup snapshot already contains the bootstrap, so building one is a context restore rather than a re-parse. deno_core binds ops into the main context only, so bootstrap hands the host its op table and the host copies those functions into each realm; the handoff global is deleted in the same step, before page script runs. Ops resolve the realm that called them from the entered-or-microtask context, never the current one, which reports the realm an op was bound in rather than the caller's. Without that a frame's deferred work writes to the parent's document. Frame timers cannot use deno_core's queue. op_timer_queue reads per-context state that only a deno_core-created context carries, so queueing from a snapshot realm dereferences uninitialized memory and aborts the process. A frame schedules through op_sleep instead, whose continuation is an ordinary microtask, which V8 reports with the frame as the microtask context. Refs h4ckf0r0day#600.

A frame could run its scripts but not report anything back, which is what an embedded widget exists to do. `iframe.contentWindow.postMessage` dispatched the event on the parent's own window, so a page never reached the document inside its frame, and `window.postMessage` was a no-op stub, so a page posting to itself waited forever. A realm cannot reach another realm's context on its own, so postMessage is queued for the host and delivered into the target realm, the same shape as frame attachment. A delivered event is marked trusted, because the user agent delivers it rather than script dispatching it: embedders gate on the flag and drop an untrusted message silently, so getting this wrong looks like the widget hanging rather than like an error. The receiver's `event.source` is the sending window, so a reply reaches the sender. A framed realm gets real `parent` and `top` windows. `parent === window` is how a document decides it is top-level, so they are installed before any of the frame's own scripts run. The queue is capped by entries and bytes. Script can post in a synchronous loop while the host drains only between event loop turns, and this buffer is on the process heap rather than V8's, so an unbounded queue would let a page grow memory in the one place the heap-limit guard cannot see. Refs h4ckf0r0day#600.

Page.getFrameTree returned `childFrames: []` however many frames a page had built, and no frame lifecycle event was ever emitted, so Playwright and Puppeteer saw a single-frame page and could not address a child at all. getFrameTree now carries the live hierarchy, nesting included, and each child is announced with Page.frameAttached, Page.frameNavigated and Page.frameStoppedLoading, with a Page.frameDetached when it goes away. The events come from a diff drawn after every dispatch, next to the binding-call drain, rather than from the navigation handler: script can add an iframe at any time, so a frame that first appears on a later command still gets reported exactly once. Loading a document includes loading the frames in it, so navigation now builds them instead of leaving it to a caller that settles. A client that only navigates was otherwise told the page had no frames. Pages with no iframe skip that work after one native selector query. Fixes h4ckf0r0day#600.

wntic · 2026-08-11T06:30:08Z

I built and tested this stack against #600 (I filed that issue). The engine half is a
real fix — thank you. But the protocol half does not reach a real CDP client, and this
PR's own test does not catch that.

Setup. Merge-base of the stack is main @ 6fa87e1; PR #624's head 4fd0654
carries all four commits (6a4683d, 3db9c60, bbc7d80, 4fd0654), so I built that
head as "stack" and 6fa87e1 as "base", both with
cargo build --release -p obscura-cli --bins --features render,stealth.

What works: `obscura fetch`

My loopback repro from #600 flips completely, with and without --stealth:

loopback repro via CLI	base `6fa87e1`	stack `4fd0654`
child `<script>` runs (`window.__ran`)	`undefined`	`"YES"`
child `document.title`	`""`	`"RAN-IN-CHILD"`
child → parent `postMessage`	`[]`	`["FROM-CHILD"]`

It also covers more than I asked for. A cross-origin child (a second loopback port)
runs its script and reaches the parent, and so does the exact shape Turnstile uses — a
cross-origin iframe inside a closed shadow root — for both insertion orders
(iframe into shadow then host into document, and the reverse):

cross-origin child             : parentGot ["FROM-XO-CHILD"]
closed shadow root + x-origin  : got ["FROM-XO-CHILD", "FROM-XO-CHILD"]

What does not: `obscura serve` + CDP

Same fixtures, same binary, through the CDP server instead of the CLI. No change from
base at all — including the childFrames this PR is about:

child frame via `serve` + CDP	base	stack
child `<script>` runs	`undefined`	`undefined`
child → parent `postMessage`	`[]`	`[]`
`Page.getFrameTree` → `childFrames`	`[]`	`[]`
`Page.frameAttached` events	0	0

Controls I ran so this is not an artifact of my client:

with --stealth and without it — identical
sampled at 0 / 2 / 5 / 9 s after Page.navigate, plus a Runtime.evaluate settle
nudge mirroring what child_frame_tree.rs does between navigate and getFrameTree
attaching to the reused startup target (Target.createTarget returns the
existing page-1) and to a freshly created one (page-2) — identical
only Page.frameNavigated (1) and Page.frameStoppedLoading (1) are ever emitted;
no Page.frameAttached

Verbatim tree from the stack build:

{"frameTree": {"frame": {"id": "page-1", "loaderId": "initial-loader",
  "url": "http://127.0.0.1:49361/parent.html", "domainAndRegistry": "",
  "securityOrigin": "http://127.0.0.1:49361/parent.html", "mimeType": "text/html",
  "adFrameStatus": {"adFrameType": "none"}}, "childFrames": []}}

Why the test passes anyway

crates/obscura-cdp/tests/child_frame_tree.rs passes in-tree (1/1, and
child_frame_scripts.rs 5/5). It drives the protocol like this:

let mut ctx = CdpContext::new();
let page_id = ctx.create_page();
ctx.sessions.insert(session.to_string(), page_id);
// ... dispatch(&CdpRequest { .., session_id: Some(session) }, &mut ctx)

so it calls dispatch() against a page from ctx.create_page() with a hand-inserted
session entry. A real client instead gets its page and session through
Target.createTarget → Target.attachToTarget {flatten: true} against
ws://…/devtools/browser. The frame work appears not to be wired into that path — and
it is the only path Puppeteer/Playwright can take, which is the scenario the test's own
doc comment describes ("a Playwright or Puppeteer client saw a single-frame page").

A regression test that goes through Target.createTarget + Target.attachToTarget
would close the gap.

Reproduction

Fixtures (two files) and a static server

child.html and parent.html are exactly the two files from #600. Serve them on
loopback:

python3 -m http.server 8899 --bind 127.0.0.1 --directory ./fixtures &

CLI — fixed on the stack, broken on base:

obscura --allow-private-network fetch http://127.0.0.1:8899/parent.html --wait 3 -q \
  --eval '(function(){var f=document.querySelector("iframe");return JSON.stringify({
    childRan:(function(){try{return String(f.contentWindow.__ran)}catch(e){return "blocked"}})(),
    childTitle:(function(){try{return f.contentDocument.title}catch(e){return "blocked"}})(),
    parentGot: window.__res ? window.__res.parentGot : "no-res"});})()'

# base   6fa87e1 : {"childRan":"undefined","childTitle":"","parentGot":[]}
# stack  4fd0654 : {"childRan":"YES","childTitle":"RAN-IN-CHILD","parentGot":["FROM-CHILD"]}

CDP — unchanged on the stack. Start obscura serve --port 9222 --allow-private-network,
then take the browser socket from /json/version, Target.createTarget →
Target.attachToTarget {flatten:true}, pass the sessionId on every command,
Page.navigate to the same parent.html, wait ~6 s, Runtime.evaluate once to settle,
then read window.__res and Page.getFrameTree:

__res        = {"parentGot":[],"childRan":"undefined","childTitle":"","iframeLoaded":true}
childFrames  = []
frameAttached events = 0

Secondary notes

Still broken, exactly as fix(js): give child iframes a realm so their scripts run #621 documents in its known limitations: srcdoc
iframes never fire onload (base and stack alike), and data:text/html iframes fire
onload but their inline script stays inert. Flagging only so it is on the record —
not a request to widen this PR.
Turnstile is unchanged: 300030, no token, hang at ~43 s, on both the CLI and CDP
paths. A logging CONNECT proxy (--proxy is honored — a dead port fails with
ProxyConnect) shows base and stack are network-identical: 2 connections to
challenges.cloudflare.com each. That measures connections, not requests, so it does
not say whether the challenge frame itself fetched anything.
For calibration, headless Chrome 145 on the same page makes 5 requests to
challenges.cloudflare.com, including
/cdn-cgi/challenge-platform/h/g/orchestrate/jsch/v1 and a second versioned
api.js. Note that run also had no token and an empty status at 50 s, so Chrome is
not automatically a pass on that page either.
Correcting two things I put in Child iframes never get a scripting context: <script> in a child frame never executes, Page.getFrameTree reports no childFrames #600 that turned out to be bad instrumentation, in
case they mislead anyone here: document.querySelectorAll('iframe').length === 0 is
not evidence of a missing widget frame (Chrome reports 0 too — the frame is in a
closed shadow root), and request counts taken from CDP network events are unusable
against obscura, which does not report fetch/XHR traffic.

Environment

macOS 26.6.1 arm64 (Apple Silicon), Rust 1.97.1, cmake 4.4.2,
LIBCLANG_PATH=/Library/Developer/CommandLineTools/usr/lib. Baseline build 2 m 01 s,
stack 26 s incremental. Reference browser: Google Chrome 145 headless.

navigate_single returns as soon as wait_until is DomContentLoaded, and build_document_frames sat past that return. Page.navigate defaults to DomContentLoaded, so a Puppeteer or Playwright client, which sends no waitUntil at all, drove the one path that never built the frames and got a tree with no children. Only a caller asking for `load` reached the build, which is why the CLI half of h4ckf0r0day#600 worked and the CDP half did not. Build the frames before any wait_until can return: they belong to the document rather than to one readiness level. The events also went to a single arbitrary session per page. A client that reaches a page the ordinary way holds two of them, because Target.createTarget opens a session and the Target.attachToTarget that follows opens another, and a client discards any event whose sessionId is not the one it attached with. Announce to every session on the page. The test hid both: it drove Page.navigate with an explicit waitUntil "load" against a hand-inserted session. It now does the real Target.createTarget and Target.attachToTarget handshake, lets waitUntil default, and requires every session on the page to be told the frame attached.

xrip · 2026-08-11T08:33:27Z

Thank you — this reproduced exactly, and the test really was blind to it. Fixed in b5fa41b.

The cause turned out not to be the target path. It is the waitUntil default.

navigate_single returns early the moment wait_until is DomContentLoaded:

self.lifecycle = LifecycleState::DomContentLoaded;

if wait_until == crate::lifecycle::WaitUntil::DomContentLoaded {
    return Ok(());
}

and this PR put build_document_frames() at the end of that function, past that return. Page.navigate deliberately defaults to DomContentLoaded rather than Load (domains/page.rs, to keep JS-heavy sites from pushing navigation past client timeouts), so a client that sends no waitUntil — which is what Puppeteer and Playwright do, and what you did — drove the one path that never built the frames. The CLI defaults to Load, which is why your obscura fetch column flipped and your serve column did not.

Your controls were all sound; the variable that mattered was one neither of us was holding. On your own repro against the unfixed stack, adding waitUntil to the navigate makes it pass:

Page.navigate {url}                     -> childFrames [], frameAttached 0, parentGot []
Page.navigate {url, waitUntil: "load"}  -> childFrames [1], frameAttached 1, parentGot ["FROM-CHILD"]

The frames are now built before any waitUntil can return, since they belong to the document rather than to one readiness level.

A second defect your setup exposed

With that fixed, Page.frameAttached came out carrying sessionId: page-1-session while my client's session was page-1-session-1. drain_frame_events collapsed every session of a page into one arbitrary HashMap entry, and a client drops events addressed to a session it does not hold — so the events were emitted and still invisible. Going through Target.createTarget is what surfaces this: it opens a session, and the Target.attachToTarget after it opens a second one on the same page. The events now go to every session on the page.

So your instinct that the target path was implicated was right, just one layer over: it is not that the frame work is unwired there, it is that the extra session that path creates was swallowing the announcements.

Verified the way you asked

Real binary, real /devtools/browser socket, Target.createTarget → Target.attachToTarget {flatten:true} → Page.navigate with no waitUntil:

childFrames  : [{"frame":{"id":"page-1-frame-1","parentId":"page-1","url":".../child.html", ...},"childFrames":[]}]
__res        : {"parentGot":["FROM-CHILD"],"iframeLoaded":true}
frameAttached: [{"s":"page-1-session"},{"s":"page-1-session-1"}]    my sessionId: page-1-session-1

child_frame_tree.rs now does the real Target.createTarget + Target.attachToTarget handshake instead of inserting a session for a hand-made page, lets waitUntil default, and requires every session on the page to receive the attach. Both halves fail without their fix:

without the navigate fix: no child frame in the tree: ...childFrames:[] — your output, verbatim
without the session fan-out: session page-1-session-1 on the page was never told the child frame attached

The second one needed care: asserting only "my session got it" passes roughly half the time on HashMap ordering, which is the same coin flip the bug itself was.

Validation

cargo nextest run --release --no-default-features -p obscura-js -p obscura-browser -p obscura-cdp   # 457/458

The one failure is obscura-cdp::max_connections_cap max_connections_refuses_then_recovers, which I re-ran on the unmodified PR head in the same worktree and it fails identically — unchanged from what this PR already noted.

On your secondary notes

srcdoc and data:text/html frames are still as #621 documents them; not widened here. Turnstile is a separate matter and I am not claiming this moves it.

Also noted, and being handled as their own PRs rather than widened into this one:

drain_binding_calls has the identical one-arbitrary-session flaw, so Runtime.bindingCalled can be delivered to a session the client does not hold. Pre-existing, and it would bite exposeFunction through the same createTarget path you used.
suspend_js drops the runtime without clearing self.frames, while init_js only clears them under if self.js.is_some(). After a suspend/resume, FrameRealm V8 handles into a destroyed isolate survive.

Thanks again for building both binaries and running the controls — the serve-versus-fetch split in your table is what made this findable.

xrip · 2026-08-11T09:22:53Z

Follow-up: #624 has been rebased onto this PR's fix (b5fa41b) and force-pushed, so the stack is linear again and each PR contains the ones below it. Merge order is unchanged: #621, #622, #623, then #624.

While reviewing #624 on top of the corrected #623, one of its own changes turned out to break the case you had specifically confirmed working.

The shadow-root frame was being torn down

#624 discards a frame realm when its iframe element leaves the document, which it decided with:

[...document.querySelectorAll('iframe')].map(f => f._frameId >>> 0).filter(id => id)

That query does not reach into a shadow root. You made exactly this point in your review here — document.querySelectorAll('iframe').length === 0 is not evidence of a missing frame, because Chrome reports 0 as well. So an iframe inside a shadow root was read as detached and had its realm destroyed moments after it loaded, along with the published window and document that #624 exists to provide.

It is the one shape where the whole four-PR stack matters most, and it is the shape you verified:

closed shadow root + x-origin  : got ["FROM-XO-CHILD", "FROM-XO-CHILD"]

Measured on the same fixture, same binary, before and after:

closed shadow root, before:  childRan "undefined"  childTitle ""              frameObjects []
closed shadow root, after:   childRan "YES"        childTitle "RAN-IN-CHILD"  frameObjects ["1"]
plain iframe, unchanged:     childRan "YES"                                   frameObjects ["1"]

Liveness is now asked of the element through isConnected, which reports a shadow-contained iframe correctly where the document query returns nothing:

iframeIsConnected true,  querySelectorAll('iframe').length 0

A failed liveness query now also leaves the frame tree alone instead of reading as "every frame is gone" — holding a frame is bounded by the existing cap, whereas destroying a live one is not recoverable.

Two regression tests cover both directions: a shadow-root frame survives the sweep and stays reachable, and a genuinely removed iframe still has its realm released with every reference the page realm held for it cleared, so nothing is retained.

On the rest of the stack

Cross-origin frames still stay opaque — the frame loads and can postMessage out, while nothing is published and the page reads undefined for anything inside it. The three new bootstrap globals are captured by the snapshot hide list; Object.getOwnPropertyNames(globalThis) shows none of them on a running page.

Performance, against this PR's head on the same machine:

DOM-heavy page, no frames, in-page ms (median of 7):   #623 382     #624 378
navigation wall clock, 0 frames (avg of 5):            #623 1041ms  #624 1038ms
navigation wall clock, 8 frames (avg of 5):            #623 1069ms  #624 1072ms

Suite after the rebase: 462/463 on -p obscura-js -p obscura-browser -p obscura-cdp --no-default-features, the one failure being max_connections_cap, unchanged from an unmodified worktree here.

Two things on the record, not in this PR

srcdoc and data:text/html frames remain as fix(js): give child iframes a realm so their scripts run #621 documents them. Still not widened.
Four globals (__obscura_viewport_w, __obscura_viewport_h, __obscura_screen_emulated, __obscura_click_target) are missing from the snapshot hide list and are enumerable on window. That is pre-existing — identical on an unmodified build — and unrelated to the frame work, but it is a detection surface and I did not want to leave it unmentioned given how much of this stack is about not adding one.

Separately, and also found while working on this stack: Runtime.bindingCalled had the same one-arbitrary-session flaw the frame events had (#632), and suspend_js dropped the page's runtime while its frame realms still held V8 handles into it, which aborts the process (#633).

xrip added 3 commits August 10, 2026 22:59

This was referenced Aug 10, 2026

fix(js): let a page reach into its own same-origin frame #624

Closed

Child iframes never get a scripting context: <script> in a child frame never executes, Page.getFrameTree reports no childFrames #600

Closed

This was referenced Aug 11, 2026

fix(cdp): deliver Runtime.bindingCalled to the session that subscribed #632

Open

fix(browser): drop frame realms before the isolate they point into #633

Merged

xrip mentioned this pull request Aug 11, 2026

fix(frames): release iframe realms safely #637

Merged

SGavrl closed this Aug 14, 2026

SGavrl reopened this Aug 14, 2026

SGavrl requested review from SGavrl and h4ckf0r0day as code owners August 14, 2026 16:57

SGavrl merged commit b5fa41b into h4ckf0r0day:main Aug 14, 2026
3 checks passed

Navigation Menu

feat(cdp): report the page's real child frame tree - #623

feat(cdp): report the page's real child frame tree#623
SGavrl merged 4 commits into
h4ckf0r0day:mainfrom
xrip:feat/cdp-child-frame-tree-600

xrip commented Aug 10, 2026

Uh oh!

wntic commented Aug 11, 2026

Uh oh!

xrip commented Aug 11, 2026

Uh oh!

xrip commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

xrip commented Aug 10, 2026

What changed

Validation

Rendering

Performance

Checklist

Uh oh!

wntic commented Aug 11, 2026

What works: obscura fetch

What does not: obscura serve + CDP

Why the test passes anyway

Reproduction

Secondary notes

Environment

Uh oh!

xrip commented Aug 11, 2026

A second defect your setup exposed

Verified the way you asked

Validation

On your secondary notes

Uh oh!

xrip commented Aug 11, 2026

The shadow-root frame was being torn down

On the rest of the stack

Two things on the record, not in this PR

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

What works: `obscura fetch`

What does not: `obscura serve` + CDP