fix(desktop): calibrate browser element capture against guest viewport

[kassamkhoja]

Linked issue

No issue filed yet, happy to open one with the raw evidence if preferred. Repro details are in QA below.

Type of change: Bug fix

Reasoning

Element screenshots for browser annotations were inaccurate: agents received an image that did not match the annotated div. Pixel analysis of a real case showed the captured image contained entirely different content than the expected element.

The paseo:browser:capture-element handler passed the CSS-pixel rect from getBoundingClientRect() straight into contents.capturePage(rect). The returned frame is device-scaled, and page zoom shifts the mapping further, so whenever display scale or zoom was not exactly 1 the crop landed offset and undersized. A second failure mode: any scroll between clicking an element and the screenshot landing used the stale click-time rect.

This matters to users because annotated element screenshots are what agents see when driving the browser pane, a wrong crop means the agent reasons about pixels the user never pointed at. Also, I presume many people use Paseo

Reproduced with real Electron under Xvfb + MacOS (27.0 Beta 26A5416b) against a fixture page (fraction = share of captured pixels matching the annotated element):

Scenario	legacy capturePage(rect)	calibrated crop (this PR)
Display scale 1.25	100%	99.5%
Page zoom 1.25 (dpr 1.5625)	67.6%	99.1%
Scroll between measure and capture (+ zoom)	54.9%	99.1%

The fix captures the full guest frame and crops itself, deriving the CSS→pixel scale from the actual frame size versus the measured guest viewport (innerWidth/innerHeight). That makes the result independent of how Electron maps rects internally across platforms, display configurations, and versions. The selector flow stamps a unique data-paseo-capture-id on the clicked node, and the element is re-measured by that marker at capture time so scroll drift cannot skew the region and a stale/ambiguous selector cannot retarget the crop. The guest is measured again after the frame lands and the crop is only applied when both measurements agree (one retry). The crop is clamped to frame bounds. The CSS rect is never passed to capturePage; when no trustworthy crop can be produced the call returns null rather than an unrelated screenshot.

New module packages/desktop/src/features/browser-webviews/capture-element.ts, wired through the existing browserCapture service; preload and both call sites pass selection.selector + selection.captureId.

Goals

• Element annotation screenshots match the annotated element on any display scale factor and page zoom
• Scroll between selection and capture cannot corrupt the region
• Partially-offscreen elements clamp instead of producing out-of-bounds crops
• Never produce a wrong-region crop: crop the freshest known rect when metrics are trustworthy; return null rather than a misleading image otherwise

Non-goals

• No changes to the automation screenshot path
• Not addressing capture latency/polling behavior

QA

• Reproduced the bug first: real Electron + forced --force-device-scale-factor=1.25 + page zoom 1.25 under Xvfb; legacy capturePage(rect) captured only 67.6% of the target element (54.9% after a scroll), confirming both failure modes above.
• Confirmed fixed: same harness, calibrated crop → 99.1–99.5% across all scenarios (table above).
• New unit tests: npx vitest run src/features/browser-webviews/capture-element.test.ts src/features/browser-capture.test.ts → 18 passed.
• npm run typecheck -w @getpaseo/desktop -w @getpaseo/app: clean. npm run lint on changed files: 0 warnings/errors.

Platforms: tested on Linux (Xvfb,Electron). and MacOS (27.0 Beta 26A5416b). Not tested Windows, I don't have a windows machine. However, the calibration does not depend on platform capture internals, but confirmation on a HiDPI Windows display would be welcome.

Before and After Screenshots

Reproduced at 1.5x zoom on the Paseo desktop app on MacOS.

Before screenshots from website:

Screenshot 1:

Image sent to agent:

Screenshot 2:

Image sent to agent:

In each “before” pair, the image sent to the agent showed different content than the annotated element: e.g., highlighting the stats card delivered a crop of an unrelated region.

After screenshots from website:

Screenshot 1:

Image sent to agent:

Screenshot 2:

Image sent to agent:

As seen in the after screenshots, the images sent to the agents matched identically to the annotation area selected with the annotation tool.

Checklist

• One focused change
• npm run typecheck passes
• npm run lint passes
• npm run format passes
• QA evidence
• Tests added or updated where it made sense

[greptile-apps]

Greptile Summary

The PR calibrates browser-element screenshots against the captured guest viewport and remeasures selected elements around capture time.

• Captures full guest frames and converts CSS rectangles into frame-pixel crops.
• Passes a selection-specific marker through the renderer, preload, and main-process capture boundary.
• Adds stability checks, bounded retries, crop clamping, diagnostics, and interface-level tests.

Confidence Score: 4/5

The PR is not yet safe to merge because replacing the marked element can make the capture silently target another node through the selector fallback.

The capture-ID change preserves identity only while the stamped node remains in the DOM; after an SPA or virtualized render replaces it, the original selector can resolve a different node whose stable rectangle is accepted and returned.

Files Needing Attention: packages/desktop/src/features/browser-webviews/capture-element.ts

Important Files Changed

Filename	Overview
packages/desktop/src/features/browser-webviews/capture-element.ts	Adds calibrated full-frame cropping and stability checks, but the selector fallback can still retarget capture after the marked node is replaced.
packages/app/src/desktop/browser/pane/element-selector.electron.ts	Stamps the selected node with a session-specific capture marker and includes it in the selection result.
packages/desktop/src/features/browser-capture.ts	Routes validated capture requests through the calibrated element-capture module.
packages/desktop/src/preload.ts	Extends the context-isolated bridge with selector and capture-ID arguments.
packages/desktop/src/main.ts	Keeps IPC capture requests scoped to the sender-owned browser guest while forwarding the new identity fields.
packages/desktop/src/features/browser-webviews/capture-element.test.ts	Exercises calibration, marker lookup, movement retries, failure reporting, and frame validation through the capture interface.

Sequence Diagram

sequenceDiagram
  participant UI as Browser pane
  participant IPC as Preload/Main IPC
  participant Guest as Browser guest
  UI->>IPC: capture(rect, selector, captureId)
  IPC->>Guest: measure marked element
  Guest-->>IPC: viewport and element rect
  IPC->>Guest: capture full frame
  IPC->>Guest: remeasure marked element
  alt Measurements stable
    IPC-->>UI: calibrated element crop
  else Measurements diverge
    IPC->>Guest: retry once
  end

Loading

_{Reviews (4): Last reviewed commit: "fix(desktop): calibrate browser element ..." | Re-trigger Greptile}

[greptile-apps]

Selector fallback retargets capture

When an SPA render or virtualized-list update replaces the selected element before capture, the capture marker disappears and this fallback resolves the original positional or duplicate-ID selector. Two stable measurements of that replacement are then accepted, causing the annotation or clipboard screenshot to contain a different element.