Skip to content

Extension: show live fetch status in the popup (API paging, 429/retry, sleeping) #43

Description

@rtanglao

Motivation

The extension popup currently shows one static line while a fetch runs:

Fetching in the page… (this can take a while with answers)

For a multi-day, answers-included window that's a long, silent wait: the user
can't tell whether it's paging questions, fetching per-question answers, being
rate-limited, or hung. aaqFetch already console.logs 429 waits — but to the
page console, not the popup, so the attended operator never sees it.

Because the popup is dismissed the instant it loses focus (see
docs/manual-refresh-runbook.md gotcha), an operator who can't see progress is
more likely to click away mid-fetch and drop the run. Live status keeps them
engaged and makes rate-limit stalls legible.

Goal

Stream progress from the in-page fetch loop to the popup and render it, without
changing what data is fetched or the output bundle.

Statuses to surface:

  • Paging questions — e.g. Questions: page 3 (142 so far)
  • Fetching answers — e.g. Answers: question 57/142
  • Rate limited — e.g. HTTP 429 — waiting 90s (retry 1/3) with a countdown if cheap
  • Sleeping between calls (low priority; likely folded into the above rather than its own noisy line)
  • Done / error — unchanged final line

Design

Progress needs to flow while aaqFetch runs. scripting.executeScript
returns only once (can't stream), so use runtime messaging for progress
events, independent of which path started the fetch:

  1. fetch-core.js — emit progress from the loop via a guarded reporter:
    const rt = (globalThis.browser ?? globalThis.chrome)?.runtime;
    const report = (p) => { try { rt?.sendMessage({ type: "aaq-progress", ...p }); } catch (e) {} };
    • Keeps aaqFetch self-contained: rt is present in the content-script and
      executeScript ISOLATED worlds, and absent in the page's own context
      (console-snippet.js), where report() silently no-ops. No behavior change
      there.
    • Call report() at: each question page fetched, each answer question index,
      entering a 429 wait (with waitS, attempt, max429Retries).
  2. popup.js — add a runtime.onMessage listener for aaq-progress that
    calls setStatus(...), registered before runInPage(...) and removed in
    finally. Keep the existing aaq-fetch / aaq-ping reply protocol intact.
  3. content.js — no protocol change needed (progress is a separate
    runtime.sendMessage, not part of the aaq-fetch response).

Edge cases

  • Console-snippet path: no runtime → reporter no-ops; snippet keeps logging
    to the console as today. (Optionally have the snippet's CONFIG log the same
    strings — nice-to-have, not required.)
  • Popup closed mid-fetch: progress messages have no receiver; sendMessage
    may reject → the try/catch swallows it. No crash, matches today's drop
    behavior.
  • Message volume: throttle answer-progress (e.g. every question is fine at
    ~1 msg / 2s given delayMs, so no extra throttling likely needed).

Acceptance criteria

  • Popup status updates live during a Chrome fetch: question paging, answer
    progress, and 429 waits are all visible.
  • 429 wait shows the wait seconds and retry count.
  • Output bundle (aaq-<product>-<dates>.json) is byte-identical to before —
    progress is UI-only.
  • console-snippet.js still works unchanged (reporter no-ops with no runtime).
  • npx web-ext lint reports 0 errors; manifest.json version bumped.

Out of scope

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions