Ask
@aatchison — once #56 lands, what should happen to the two monitoring workflows that are still switched off?
Current state (gh workflow list --all):
| Workflow |
State |
Hits the SUMO API? |
scrape.yml (Refresh SUMO AAQ) |
disabled_manually |
yes — superseded by the CronJob |
schema-check.yml (API schema check) |
disabled_manually |
yes |
kitsune-api-watch.yml (Kitsune API watch) |
disabled_manually |
no — GitHub API only |
aaq-scraper-image.yml (Build scraper image) |
active |
no |
All three were disabled together on 2026-07-10 during the Fastly fingerprinting incident (#26). That incident is now largely resolved for the scraper — the cluster's NAT egress IPs are allowlisted and PR #44 moved it to httpx in a CronJob — but the resolution does not extend to GitHub Actions, whose runners still get the challenge.
The two cases differ
1. schema-check.yml — blocked where it currently runs. It calls /api/2/ from an Actions runner, so simply re-enabling it would fail with ChallengeError and open spurious api-blocked issues (it exits 2 for exactly this case). It needs one of:
- move the check into the pod / CronJob (run it on a schedule alongside the refresh, from the allowlisted IP), or
- run it as a separate k8s Job, or
- leave it off and accept losing the empirical safety net.
Worth weighing: this is our only detector for response-shape changes regardless of origin, including edge/WAF behaviour. It's also what would have caught #55-style drift. Leaving it off indefinitely seems like the weakest option, but it's your call how much pod plumbing it's worth.
2. kitsune-api-watch.yml — arguably safe to re-enable as-is. It only queries the GitHub API for new upstream mozilla/kitsune commits on the paths that serve our endpoints; it never touches support.mozilla.org. It was disabled as part of the blanket 2026-07-10 shutdown rather than because it was failing. If that's right, gh workflow enable "Kitsune API watch" should just work and restores the code-level early warning.
Please sanity-check that second claim before flipping it — I'm going off the workflow definition, not a test run.
Context
Ask
@aatchison — once #56 lands, what should happen to the two monitoring workflows that are still switched off?
Current state (
gh workflow list --all):scrape.yml(Refresh SUMO AAQ)disabled_manuallyschema-check.yml(API schema check)disabled_manuallykitsune-api-watch.yml(Kitsune API watch)disabled_manuallyaaq-scraper-image.yml(Build scraper image)All three were disabled together on 2026-07-10 during the Fastly fingerprinting incident (#26). That incident is now largely resolved for the scraper — the cluster's NAT egress IPs are allowlisted and PR #44 moved it to httpx in a CronJob — but the resolution does not extend to GitHub Actions, whose runners still get the challenge.
The two cases differ
1.
schema-check.yml— blocked where it currently runs. It calls/api/2/from an Actions runner, so simply re-enabling it would fail withChallengeErrorand open spuriousapi-blockedissues (it exits 2 for exactly this case). It needs one of:Worth weighing: this is our only detector for response-shape changes regardless of origin, including edge/WAF behaviour. It's also what would have caught #55-style drift. Leaving it off indefinitely seems like the weakest option, but it's your call how much pod plumbing it's worth.
2.
kitsune-api-watch.yml— arguably safe to re-enable as-is. It only queries the GitHub API for new upstreammozilla/kitsunecommits on the paths that serve our endpoints; it never touchessupport.mozilla.org. It was disabled as part of the blanket 2026-07-10 shutdown rather than because it was failing. If that's right,gh workflow enable "Kitsune API watch"should just work and restores the code-level early warning.Please sanity-check that second claim before flipping it — I'm going off the workflow definition, not a test run.
Context