Move the scraper off Playwright to httpx + containerize for the k8s CronJob

.dockerignore

@@ -0,0 +1,11 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at https://mozilla.org/MPL/2.0/.
+.git
+.venv
+__pycache__
+*.pyc
+20*/
+docs/
+backfill-reports/
+tests/

.github/workflows/aaq-scraper-image.yml

@@ -0,0 +1,75 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# Build & push the scraper image to the shared ECR via GitHub OIDC.
+# One-time setup (see platform-infrastructure Task B1):
+#   - GitHub Environment "image-aaq-scraper" (protected to main)
+#   - repo variable IMAGE_PUSH_ROLE_ARN = the push-only role ARN
+name: Build scraper image
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "**.py"
+      - "pyproject.toml"
+      - "uv.lock"
+      - "Dockerfile"
+      - "deploy/entrypoint.sh"
+      - ".github/workflows/aaq-scraper-image.yml"
+  workflow_dispatch:
+    inputs:
+      tag:
+        description: "Explicit image tag (default git-<sha>)"
+        required: false
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    environment: image-aaq-scraper
+    env:
+      ECR: 826971876779.dkr.ecr.us-east-1.amazonaws.com
+      REPO: aaq-scraper
+    steps:
+      - uses: actions/checkout@v5
+      - name: Compute tag
+        id: tag
+        env:
+          # Pass the workflow_dispatch input through an env var rather than
+          # interpolating ${{ github.event.* }} directly into the run script
+          # (that is a shell-injection vector, since this job holds ECR-push
+          # OIDC creds). Then validate the tag charset before use.
+          TAG_INPUT: ${{ github.event.inputs.tag }}
+        run: |
+          T="$TAG_INPUT"
+          [ -n "$T" ] || T="git-$(git rev-parse --short HEAD)"
+          case "$T" in
+            ""|*[!A-Za-z0-9._-]*)
+              echo "invalid tag: $T" >&2; exit 1 ;;
+          esac
+          printf 'tag=%s\n' "$T" >> "$GITHUB_OUTPUT"
+      - name: Configure AWS credentials (OIDC)
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          role-to-assume: ${{ vars.IMAGE_PUSH_ROLE_ARN }}
+          aws-region: us-east-1
+      - name: Login to ECR
+        uses: aws-actions/amazon-ecr-login@v2
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: arm64
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@v3
+      - name: Build and push
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          platforms: linux/arm64
+          push: true
+          tags: ${{ env.ECR }}/${{ env.REPO }}:${{ steps.tag.outputs.tag }}

.gitignore

@@ -19,8 +19,9 @@ poc-sample.json
 /extension/web-ext-artifacts/
 /extension/.amo-upload-uuid
 
-# Incremental-refresh high-water mark (persisted via Actions cache, not git)
-/.refresh-hwm
+# NOTE: /.refresh-hwm is intentionally NOT ignored — the k8s CronJob commits the
+# incremental-refresh high-water mark to the repo (the pod is stateless and the
+# GitHub Actions cache that used to hold it is gone).
 
 # Month-backfill failure log (local record of dates to redo)
 /backfill-failures.txt

.python-version

@@ -0,0 +1 @@
+3.12

CLAUDE.md

@@ -8,19 +8,21 @@ Scrape the Mozilla SUMO (support.mozilla.org) "Ask a Question" API for Thunderbi
 Desktop and Thunderbird Android, producing CSVs compatible with the legacy Ruby
 reports in `thunderbird/github-action-thunderbird-aaq`. Since ~June 2026 the API
 sits behind a JavaScript challenge that blocks headless HTTP (issue
-thunderbird/github-action-thunderbird-aaq#34), so we drive a **real browser** to
-pass the challenge and call the JSON API from inside the browser's authenticated
-context.
+thunderbird/github-action-thunderbird-aaq#34); a real-browser (Playwright/
+Chromium) workaround passed the challenge for a while but is now itself
+fingerprinted and blocked (issue #28), so the scraper instead calls the API
+over a plain **`httpx`** HTTP client, and is moving to a Kubernetes CronJob
+deployment with a stable, allowlistable egress IP (issue #27).
 
 ## Core architecture & crucial decisions
 
-- **Browser-passes-challenge approach** (`sumo.py`): `SumoBrowser` launches
-  Chromium (Playwright), loads the site once to acquire challenge cookies, then
-  `fetch_json()` does an **in-page `fetch()`** (`page.evaluate`) so the request
-  reuses the page's cookies/origin. Headless works (locally **and in GitHub
-  Actions** — this is what resolves #34).
-- **Stack: Python + Playwright, managed with `uv`** — use `uv sync` / `uv run`,
-  never pip or raw venv. Deps in `pyproject.toml`.
+- **HTTP-client approach** (`sumo.py`): `SumoBrowser` (name kept for
+  compatibility) drives a plain `httpx` client; historically it launched
+  Chromium via Playwright to pass the JS challenge, but that is now
+  fingerprinted/blocked (#28).
+- **Stack: Python + `httpx`, managed with `uv`** — use `uv sync` / `uv run`,
+  never pip or raw venv. Deps in `pyproject.toml`. (`poc.py` still uses Playwright,
+  which is no longer a project dependency — install it separately to run the PoC.)
 - **`fetch_json` retries** transient failures with exponential backoff: HTTP 429
   (honours `Retry-After`), 5xx, and a 200-but-non-JSON challenge hiccup.
   Non-retryable 4xx fail fast.
@@ -111,8 +113,7 @@ context.
 ## Commands
 
 ```sh
-uv sync
-uv run playwright install chromium                 # one-time
+uv sync                                            # httpx client; no browser install needed
 
 # Questions (single day = same date twice). Add --product thunderbird-android for Android.
 uv run python scrape_questions.py 2026 6 10 2026 6 10 --headless
@@ -184,6 +185,49 @@ vary 2–10s (`--min-delay`/`--max-delay`). Use `--headless` for CI parity.
   `run_refresh` instead.
 - Actions are pinned to Node-24 versions: `actions/checkout@v5`,
   `astral-sh/setup-uv@v8.2.0`.
+- **k8s CronJob deployment (manifests merged-pending; job ships suspended):**
+  the scraper is moving off GitHub Actions onto an ArgoCD-managed **Kubernetes
+  CronJob** on the workloads EKS cluster, because that cluster's NAT egress IPs
+  are **already allowlisted** by Mozilla while Actions runners (shared, rotating
+  IPs) are not. Verified 2026-07-27 from a pod in each AZ with the scraper's own
+  httpx client: `3.67.52.124` (eu-central-1a) and `63.182.70.185` (eu-central-1b)
+  both return 200 + JSON; the same request from a non-allowlisted network returns
+  the challenge HTML. **So the API is NOT blocked from the cluster** — only from
+  Actions and from developer workstations, which is why a local run still raises
+  `ChallengeError` and is not evidence of an outage. The k8s manifests/Pulumi/
+  ArgoCD app live in the
+  separate `platform-infrastructure` repo; this repo only builds the image
+  (`Dockerfile`, `.github/workflows/aaq-scraper-image.yml` → shared ECR via
+  OIDC) and ships `deploy/entrypoint.sh` (clone → `run_refresh.py` → commit).
+  Two prerequisites already landed here: (1) `sumo.py` **dropped Playwright**
+  entirely — the Chromium challenge-bypass is itself now fingerprinted and
+  blocked (issue #28) — and drives the same `SumoBrowser`/`fetch_json` public
+  API over a plain **`httpx`** client instead — Playwright is kept only as an
+  **optional dependency group** (`uv sync --group playwright`) so `poc.py` still
+  runs; it is not installed by default and is absent from the image; (2)
+  **`.refresh-hwm` is now
+  tracked in git** (removed from `.gitignore`, no longer the Actions cache)
+  since the pod is stateless and the repo is the durable state. The pod pushes
+  its commits authenticated with a **fine-grained GitHub PAT** sourced from AWS
+  Secrets Manager (synced in by External Secrets Operator) via a **git
+  credential helper** — the token is read from the environment at call time and
+  never appears in argv or a URL. The CronJob ships **suspended** with a
+  placeholder image tag — the only remaining gate is mechanical (`pulumi up` for
+  the ECR repo + OIDC role, create the Secrets Manager PAT, merge so the image
+  builds, then pin the tag and unsuspend). Full design:
+  `docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md`
+  and `docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md`.
+- **What is actually producing data right now: the browser extension**, not any
+  GitHub workflow. As of 2026-07-27 **all three workflows are `disabled_manually`**
+  (`scrape.yml` last ran 2026-07-10) — do not describe `scrape.yml` as the live
+  refresh. The `extension/` add-on runs a scheduled auto-fetch from a real
+  browser and its JSON bundles are imported by `import_json.py`, which reuses the
+  scrapers' own writers (`build_fieldnames`, `flatten_answer`, `COLUMNS`), so
+  extension-imported and `run_refresh.py`-generated CSVs are format-compatible.
+  **Open decision:** whether the k8s CronJob *replaces* the extension or runs
+  alongside it — both would commit the same day-CSVs on `main`, so running both
+  duplicates work (the pod's rebase-retry copes with the race, but it is not a
+  plan). Decide before unsuspending.
 
 ## Notes
 

Dockerfile

@@ -0,0 +1,31 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at https://mozilla.org/MPL/2.0/.
+
+# Browser-free image: the old Playwright/Chromium approach is dead (issue #28);
+# this runs the plain-httpx scraper. git + a shell are needed at runtime because
+# the entrypoint clones aaq-scraper, runs the refresh, and commits CSVs back.
+FROM python:3.12-slim
+
+# uv for dependency management (matches local/CI: uv sync / uv run).
+COPY --from=ghcr.io/astral-sh/uv:0.5.11 /uv /usr/local/bin/uv
+
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends git ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+COPY pyproject.toml uv.lock ./
+RUN uv sync --frozen --no-dev
+
+COPY . .
+
+# Activate venv by adding it to PATH so `python` resolves to the venv's interpreter.
+ENV PATH="/app/.venv/bin:$PATH"
+
+# Non-root. The clone/commit workdir is a writable emptyDir mounted at runtime
+# (the root FS is read-only in the pod), so this user only needs to read /app.
+RUN useradd --create-home --uid 65532 appuser
+USER 65532
+
+ENTRYPOINT ["deploy/entrypoint.sh"]

README.md

@@ -1,16 +1,32 @@
 # aaq-scraper
 
 Until we get an official API that has a way to prevent DOS-ing, we scrape the SUMO
-(support.mozilla.org) API by driving a real browser. Since ~June 2026 the API sits
-behind a JavaScript challenge that blocks headless HTTP (see
-[thunderbird/github-action-thunderbird-aaq#34](https://github.com/thunderbird/github-action-thunderbird-aaq/issues/34));
-a real browser passes the challenge, then we call the JSON API from inside the
-browser's authenticated context.
+(support.mozilla.org) API. Since ~June 2026 the API sits behind a JavaScript
+challenge that blocks headless HTTP (see
+[thunderbird/github-action-thunderbird-aaq#34](https://github.com/thunderbird/github-action-thunderbird-aaq/issues/34)).
+A real-browser (Playwright/Chromium) workaround passed the challenge for a
+while but is now itself fingerprinted and blocked (issue #28), so the scraper
+is a plain **`httpx`** client instead, and is being redeployed as a
+**Kubernetes CronJob** on a cluster with a stable egress IP that Mozilla can
+allowlist (issue #27) — see the design and plan docs under
+[`docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md`](docs/superpowers/specs/2026-07-13-k8s-argocd-scraper-deployment-design.md)
+and
+[`docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md`](docs/superpowers/plans/2026-07-13-k8s-argocd-scraper-deployment.md).
+
+That cluster's egress IPs turn out to be **already allowlisted** (verified
+2026-07-27 from a pod in each AZ), so the API is reachable from the cluster —
+but *not* from GitHub Actions or a developer workstation, where a run still
+fails with `ChallengeError`. That is expected, not an outage. Data is currently
+produced by the browser extension under [`extension/`](extension); all GitHub
+workflows are disabled.
 
 ## Proof of concept (Bucket 0)
 
+Historical Bucket-0 script. `playwright` is no longer a runtime dependency, but
+it is kept as an **optional dependency group** so this still runs:
+
 ```sh
-uv sync
+uv sync --group playwright
 uv run playwright install chromium
 uv run python poc.py            # headed — most likely to pass the challenge
 uv run python poc.py --headless # try headless (closer to CI)

deploy/entrypoint.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# Pod entrypoint: clone aaq-scraper, run the incremental refresh, commit changed
+# CSVs (and the high-water mark) back to the branch. Mirrors the logic that used
+# to live in .github/workflows/scrape.yml (checkout + run_refresh + commit with
+# rebase-onto-latest-main retry). The pod is stateless; the repo IS the state.
+set -euo pipefail
+
+: "${GIT_REPO_URL:?set GIT_REPO_URL}"
+: "${GITHUB_TOKEN:?set GITHUB_TOKEN}"
+GIT_BRANCH="${GIT_BRANCH:-main}"
+GIT_AUTHOR_NAME="${GIT_AUTHOR_NAME:-aaq-scraper-bot}"
+GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-aaq-scraper-bot@thunderbird.net}"
+REFRESH_ARGS="${REFRESH_ARGS:---soft-deadline 40 --max-429-wait 120}"
+# Paths staged for the commit. Defaults to the real data tree; a shakedown
+# deployment writing under AAQ_DATA_ROOT sets this to that directory instead so
+# it never stages the committed CSVs. Word-split deliberately (multiple paths).
+GIT_ADD_PATHS="${GIT_ADD_PATHS:-20*/ .refresh-hwm}"
+
+# $HOME is the writable emptyDir mounted in the pod (Phase B sets HOME=/work);
+# /tmp is on the read-only root filesystem (readOnlyRootFilesystem: true), so
+# mktemp must be pointed at $HOME rather than its default /tmp base.
+WORKDIR="$(mktemp -d "${HOME:-/tmp}/aaq-scraper.XXXXXX")"
+trap 'rm -rf "$WORKDIR"' EXIT
+
+# Supply the PAT via a credential helper that reads $GITHUB_TOKEN from the
+# environment at call time, so only the variable NAME (never its value) ever
+# appears in argv (visible via ps / /proc/<pid>/cmdline). The clone/pull/push
+# all use the plain GIT_REPO_URL -- no token spliced into the URL.
+export GIT_TERMINAL_PROMPT=0
+CRED_HELPER='!f() { echo "username=x-access-token"; echo "password=${GITHUB_TOKEN}"; }; f'
+
+git -c credential.helper="$CRED_HELPER" clone --branch "$GIT_BRANCH" "$GIT_REPO_URL" "$WORKDIR/repo"
+cd "$WORKDIR/repo"
+git config credential.helper "$CRED_HELPER"
+git config user.name "$GIT_AUTHOR_NAME"
+git config user.email "$GIT_AUTHOR_EMAIL"
+
+# Run the refresh. REFRESH_CMD lets tests inject a fake; default is the real one.
+# Preserve the deferral exit code (75) as a non-error signal.
+set +e
+if [ -n "${REFRESH_CMD:-}" ]; then
+  eval "$REFRESH_CMD"
+else
+  uv run python run_refresh.py $REFRESH_ARGS
+fi
+rc=$?
+set -e
+if [ "$rc" -eq 75 ]; then
+  echo "refresh deferred (exit 75); committing whatever completed"
+elif [ "$rc" -ne 0 ]; then
+  echo "refresh failed (exit $rc)"; exit "$rc"
+fi
+
+# Stage each configured path independently and tolerate ones that do not exist
+# (a fresh checkout has no .refresh-hwm; a shakedown root has no 20*/ dirs).
+# $GIT_ADD_PATHS is intentionally unquoted so the shell word-splits and expands
+# the 20*/ glob before git sees it.
+# shellcheck disable=SC2086
+for _p in $GIT_ADD_PATHS; do
+  git add -- "$_p" 2>/dev/null || true
+done
+if git diff --cached --quiet; then
+  echo "No changes to commit."
+  exit 0
+fi
+git commit -m "Hourly refresh $(date -u +%Y-%m-%dT%H:%MZ)"
+
+# main also receives pushes from manual backfills, so rebase onto latest and
+# retry with backoff rather than failing on a rejected push.
+#
+# --autostash: the scrape can leave unstaged files in the clone (anything not
+# staged by GIT_ADD_PATHS), and a plain `git pull --rebase` REFUSES to run with
+# a dirty tree -- which would wedge this loop for all 5 attempts and fail the
+# run even though the commit itself is fine.
+for attempt in 1 2 3 4 5; do
+  if ! git pull --rebase --autostash origin "$GIT_BRANCH"; then
+    git rebase --abort || true
+    echo "rebase failed (attempt $attempt/5); retrying"; sleep $((attempt * 5)); continue
+  fi
+  if git push origin "$GIT_BRANCH"; then
+    echo "pushed on attempt $attempt"; exit 0
+  fi
+  echo "push rejected (attempt $attempt/5); retrying"; sleep $((attempt * 5))
+done
+echo "ERROR: could not push after 5 attempts"; exit 1