Add BH_SCREENSHOT_COMPACT to size screenshots for an LLM

interaction-skills/screenshots.md

@@ -1,17 +1,42 @@
 # Screenshots
 
-`capture_screenshot()` writes a PNG of the current viewport. The file is in **device pixels** — on a 2× display a 2296×1143 CSS viewport produces a 4592×2286 PNG.
+`capture_screenshot()` writes a PNG of the current viewport and returns its path. The file is in **device pixels**: on a 2× display a 2296×1143 CSS viewport produces a 4592×2286 PNG.
 
-That matters for two reasons:
+```python
+capture_screenshot()                       # ~/.config/browser-harness/tmp/shot.png, native resolution
+capture_screenshot("/tmp/page.png")        # your path
+capture_screenshot("/tmp/page.png", max_dim=1800)   # downscale the long edge
+capture_screenshot("/tmp/page.jpg")        # JPEG instead of PNG
+```
 
-1. **Click coordinates are CSS pixels.** Don't read a target off the image and pass it to `click_at_xy()` directly without dividing by `devicePixelRatio`. The simplest workflow is to take the screenshot, look at it in a viewer that shows CSS coordinates, or measure relative positions and use `js("window.devicePixelRatio")` to convert.
+## Compact mode, for screenshots a model will read
 
-2. **Some LLMs reject images > 2000 px per side.** Long sessions on 2× displays will eventually hit this. Pass `max_dim=1800` to downscale the file before it gets into the conversation:
+Set `BH_SCREENSHOT_COMPACT=1` to size every capture for an LLM: the long edge is capped at 1568px and the default filename becomes `shot.jpg`.
 
-```python
-capture_screenshot("/tmp/shot.png", max_dim=1800)
-```
+1568 is not arbitrary. An image-aware LLM scales anything larger than that down to it before the model sees the image, and charges roughly `(width × height) / 750` tokens. Pixels above 1568 therefore cost nothing in tokens and add nothing to legibility. They only enlarge the transcript the screenshot is pasted into, which is the part that grows without bound over a long session.
+
+Measured on a real 3024×1432 capture:
+
+| Output | File | Tokens charged |
+|---|---|---|
+| 3024px PNG | 360 KB | 1,551 |
+| 1568px JPEG q75 | 77 KB | 1,551 |
+| 900px JPEG q75 | 40 KB | 511 |
+
+The first two rows are the point: identical cost to the model, no visible difference when read back, 4.7× less transcript.
+
+Going below 1568 does cut tokens, but it is a genuine trade. At 900px body text stays readable while dimmed sidebars, small labels and exact identifiers do not, and a misread that forces a recapture costs more than one clean 1568px shot would have. Downscale further only when you are checking layout rather than reading text.
+
+Capture always happens at native resolution and the resize comes afterwards. That is deliberate: downscaling a supersampled 2× capture is **sharper** than asking Chrome to render at `deviceScaleFactor: 1`.
+
+## Gotchas
+
+**Click coordinates are CSS pixels.** Don't read a target off the image and pass it to `click_at_xy()` without dividing by `devicePixelRatio`, and note that a resized image is neither CSS nor device pixels. Prefer selectors; if you must go by pixels, capture at native resolution with `max_dim=None`.
+
+**Pixel-diff baselines need `max_dim=None`.** Resampling and JPEG are both lossy, so a compact shot is not a valid comparison baseline. An explicit `max_dim` always overrides the environment variable.
+
+**Format follows the extension.** `.jpg`/`.jpeg` writes JPEG, anything else writes PNG, so passing an explicit `.png` path keeps PNG even in compact mode.
 
-The downscale only happens when the image actually exceeds `max_dim`, so it's safe to leave on for every shot.
+**Some LLMs reject images over 2000px per side.** Long sessions on 2× displays will hit this; `max_dim=1800` or compact mode both avoid it.
 
-Use full-page screenshots (`full=True`) only when you need to see content below the fold — they are much larger and slower than viewport-only.
+**`full=True` only when you need content below the fold.** Full-page captures are much larger and slower than viewport-only, and on a long page the long-edge cap squeezes the width badly. Scroll and take viewport shots instead when you need to read text.

src/browser_harness/helpers.py

@@ -155,7 +155,9 @@ def click_at_xy(x, y, button="left", clicks=1):
         try:
             from PIL import Image, ImageDraw
             dpr = js("window.devicePixelRatio") or 1
-            path = capture_screenshot(str(ipc._TMP / f"debug_click_{_debug_click_counter}.png"))
+            # Native resolution, not the default downscale: the marker below is
+            # positioned in device pixels, so any resampling would misplace it.
+            path = capture_screenshot(str(ipc._TMP / f"debug_click_{_debug_click_counter}.png"), max_dim=None)
             img = Image.open(path)
             draw = ImageDraw.Draw(img)
             px, py = int(x * dpr), int(y * dpr)
@@ -239,18 +241,66 @@ def scroll(x, y, dy=-300, dx=0):
 
 
 # --- visual ---
-def capture_screenshot(path=None, full=False, max_dim=None):
-    """Save a PNG of the current viewport. Set max_dim=1800 on a 2× display to
-    keep the file under the 2000px-per-side limit some image-aware LLMs enforce."""
-    path = path or str(ipc._TMP / "shot.png")
+
+# 1568px is the longest edge an image-aware LLM actually uses: larger images are
+# scaled down to it before the model sees them, and cost is roughly
+# (width * height) / 750 tokens. Pixels past that buy no legibility and no extra
+# tokens, they only enlarge the transcript the screenshot is pasted into.
+SCREENSHOT_MAX_DIM = 1568
+SCREENSHOT_JPEG_QUALITY = 75
+
+_UNSET = object()
+
+
+def _compact_screenshots():
+    """True when BH_SCREENSHOT_COMPACT asks for LLM-sized screenshots."""
+    return os.environ.get("BH_SCREENSHOT_COMPACT", "") not in ("", "0")
+
+
+def capture_screenshot(path=None, full=False, max_dim=_UNSET,
+                       quality=SCREENSHOT_JPEG_QUALITY):
+    """Save a screenshot of the current viewport and return its path.
+
+    By default this writes a full-resolution PNG, unchanged.
+
+    Set `BH_SCREENSHOT_COMPACT=1` to size captures for a model instead: the
+    long edge is capped at SCREENSHOT_MAX_DIM and the default filename becomes
+    `shot.jpg`. On a 2× display that turned a real 3024×1432 capture from
+    360 KB into 77 KB with no change in token cost and no visible difference
+    when read back.
+
+    `max_dim` overrides that per call, whatever the environment says. Capture
+    always happens at native resolution and the resize comes after, because
+    downscaling a supersampled 2× capture is sharper than asking Chrome to
+    render at deviceScaleFactor 1.
+
+    Output format follows the path extension: `.jpg`/`.jpeg` writes JPEG,
+    anything else writes PNG. So a caller passing an explicit `.png` keeps PNG
+    and only picks up the resize.
+    """
+    compact = _compact_screenshots()
+    if max_dim is _UNSET:
+        max_dim = SCREENSHOT_MAX_DIM if compact else None
+    path = path or str(ipc._TMP / ("shot.jpg" if compact else "shot.png"))
+
     r = cdp("Page.captureScreenshot", format="png", captureBeyondViewport=full)
-    open(path, "wb").write(base64.b64decode(r["data"]))
-    if max_dim:
-        from PIL import Image
-        img = Image.open(path)
-        if max(img.size) > max_dim:
-            img.thumbnail((max_dim, max_dim))
-            img.save(path)
+    raw = base64.b64decode(r["data"])
+
+    as_jpeg = str(path).lower().endswith((".jpg", ".jpeg"))
+    if max_dim is None and not as_jpeg:
+        open(path, "wb").write(raw)
+        return path
+
+    from io import BytesIO
+    from PIL import Image
+    img = Image.open(BytesIO(raw))
+    if max_dim and max(img.size) > max_dim:
+        img.thumbnail((max_dim, max_dim), Image.LANCZOS)
+    if as_jpeg:
+        # JPEG carries no alpha channel, and a screenshot never needs one.
+        img.convert("RGB").save(path, "JPEG", quality=quality, optimize=True)
+    else:
+        img.save(path, "PNG", optimize=True)
     return path
 
 

tests/unit/test_helpers.py

@@ -9,24 +9,73 @@
 from browser_harness import helpers
 
 
-def _run(fake_png, width, height, **kwargs):
+def _run(fake_png, width, height, name="shot.png", **kwargs):
     fake = lambda method, **_: {"data": fake_png(width, height)}
     with patch("browser_harness.helpers.cdp", side_effect=fake), tempfile.TemporaryDirectory() as d:
-        path = os.path.join(d, "shot.png")
+        path = os.path.join(d, name)
         helpers.capture_screenshot(path, **kwargs)
-        return Image.open(path).size
+        img = Image.open(path)
+        return img.size, img.format, os.path.getsize(path)
 
 
 def test_max_dim_downsizes_oversized_image(fake_png):
-    assert max(_run(fake_png, 4592, 2286, max_dim=1800)) == 1800
+    assert max(_run(fake_png, 4592, 2286, max_dim=1800)[0]) == 1800
 
 
 def test_max_dim_skips_when_image_already_small(fake_png):
-    assert _run(fake_png, 800, 400, max_dim=1800) == (800, 400)
+    assert _run(fake_png, 800, 400, max_dim=1800)[0] == (800, 400)
 
 
-def test_max_dim_default_is_no_resize(fake_png):
-    assert _run(fake_png, 4592, 2286) == (4592, 2286)
+def test_max_dim_default_is_no_resize(fake_png, monkeypatch):
+    monkeypatch.delenv("BH_SCREENSHOT_COMPACT", raising=False)
+    assert _run(fake_png, 4592, 2286)[0] == (4592, 2286)
+
+
+def test_compact_env_caps_the_long_edge(fake_png, monkeypatch):
+    # Anything past 1568 is resized away before a model sees it, so compact
+    # mode stops there rather than shipping pixels nobody reads.
+    monkeypatch.setenv("BH_SCREENSHOT_COMPACT", "1")
+    size = _run(fake_png, 4592, 2286)[0]
+    assert max(size) == helpers.SCREENSHOT_MAX_DIM == 1568
+    assert size[1] == round(2286 * 1568 / 4592)  # aspect ratio preserved
+
+
+def test_compact_env_off_by_default_and_disableable(fake_png, monkeypatch):
+    monkeypatch.setenv("BH_SCREENSHOT_COMPACT", "0")
+    assert _run(fake_png, 4592, 2286)[0] == (4592, 2286)
+
+
+def test_explicit_max_dim_overrides_the_env(fake_png, monkeypatch):
+    # Pixel-diff baselines need the native capture even in compact mode.
+    monkeypatch.setenv("BH_SCREENSHOT_COMPACT", "1")
+    assert _run(fake_png, 4592, 2286, max_dim=None)[0] == (4592, 2286)
+
+
+def test_format_follows_the_path_extension(fake_png):
+    assert _run(fake_png, 4592, 2286, name="shot.jpg")[1] == "JPEG"
+    assert _run(fake_png, 4592, 2286, name="shot.jpeg")[1] == "JPEG"
+    assert _run(fake_png, 4592, 2286, name="shot.png")[1] == "PNG"
+
+
+def test_jpeg_output_drops_the_alpha_channel(fake_png):
+    # JPEG carries no alpha and PIL raises rather than converting silently.
+    assert _run(fake_png, 2000, 1000, name="shot.jpg")[1] == "JPEG"
+
+
+def test_compact_default_filename_is_jpeg(fake_png, monkeypatch, tmp_path):
+    monkeypatch.setenv("BH_SCREENSHOT_COMPACT", "1")
+    monkeypatch.setattr(helpers.ipc, "_TMP", tmp_path)
+    fake = lambda method, **_: {"data": fake_png(4592, 2286)}
+    with patch("browser_harness.helpers.cdp", side_effect=fake):
+        assert helpers.capture_screenshot().endswith("shot.jpg")
+
+
+def test_default_filename_stays_png(fake_png, monkeypatch, tmp_path):
+    monkeypatch.delenv("BH_SCREENSHOT_COMPACT", raising=False)
+    monkeypatch.setattr(helpers.ipc, "_TMP", tmp_path)
+    fake = lambda method, **_: {"data": fake_png(4592, 2286)}
+    with patch("browser_harness.helpers.cdp", side_effect=fake):
+        assert helpers.capture_screenshot().endswith("shot.png")
 
 
 def _seed_skill(tmp_path):