docs: docs-tests harness for Console UI drift detection

docs-tests/.env.example

@@ -0,0 +1,8 @@
+# Staging Console base URL (required).
+CONSOLE_STAGING_URL=https://console.staging-example.apify.dev
+
+# Seeded staging user (required to run the tests). Used by tests/auth.fixture.ts
+# to log in fresh each run. Locally these live here; in CI they come from GitHub
+# Secrets, never from a committed file. The seeded account has no 2FA.
+CONSOLE_STAGING_USER_EMAIL=
+CONSOLE_STAGING_USER_PASSWORD=

docs-tests/.gitignore

@@ -0,0 +1,11 @@
+# Never commit a logged-in session or local secrets to this public repo.
+node_modules/
+.env
+auth.json
+playwright-report/
+test-results/
+.DS_Store
+
+# Runtime outputs. The stored assertion baseline (assertions/) IS committed;
+# only the per-run report lives here.
+output/

docs-tests/README.md

@@ -0,0 +1,125 @@
+# docs-tests
+
+Docs-as-tests for the Apify Console. Every UI claim in the platform docs — a
+route resolves, a tab is named X, a button exists on page Y — is a testable
+assertion. This package extracts those assertions from the docs with an LLM,
+stores them as a reviewed baseline, and verifies them against Console staging
+with Playwright, so documentation drift is caught automatically.
+
+```
+pages.json ──extract──▶ assertions/*.json ──Playwright──▶ output/issues.json
+(page list)  (claude -p)  (committed baseline)  (vs staging)   (drift report)
+```
+
+## Model
+
+1. **`pages.json`** is an adjustable list of documentation pages (real source
+   files under `sources/platform/…`) to cover.
+2. **`scripts/extract.sh`** feeds one page to `claude -p` with a strict JSON
+   schema and writes the result to `assertions/<slug>.json`.
+   `scripts/extract-all.sh` does the whole manifest.
+3. **`assertions/`** is the *stored, reviewed baseline* — committed to the repo.
+   Regenerate it with the LLM whenever docs change, review the diff, commit.
+   The assertion set is owned by humans even though a model drafts it.
+4. **`tests/from-doc.spec.ts`** reads every stored assertion and emits one
+   Playwright `test()` per assertion, run against `$CONSOLE_STAGING_URL`.
+5. Failures point back to `source_file:line` so the offending prose is one click
+   away, and land in `output/issues.json` for downstream triage.
+
+The Notion plan *"AI-based testing for docs"* (its Part 1 routes + Part 2
+elements) is the inspiration for which pages and claims to cover — not a fixed
+transcription. The authoritative set is whatever the manifest + extractor
+produce and a human commits.
+
+## Assertion kinds
+
+| Kind             | Checks                                                              |
+| ---------------- | ------------------------------------------------------------------ |
+| `route`          | Documented path is reachable (HTTP < 400)                          |
+| `element_tab`    | Documented tab label exists on the page named in `at`              |
+| `element_button` | Documented button label exists on the page named in `at`          |
+| `element_text`   | Documented heading/label/field is visible on the page named in `at` |
+
+## One-time setup
+
+```bash
+pnpm install
+pnpm exec playwright install chromium
+cp .env.example .env   # fill in CONSOLE_STAGING_URL + seeded-user email/password
+```
+
+## Generate / refresh the assertion baseline
+
+```bash
+# Every page in pages.json:
+pnpm extract:all
+
+# Or a single page:
+pnpm extract sources/platform/console/settings.md
+```
+
+Review the diff in `assertions/`, then commit. This is the step a human owns.
+
+## Run the tests
+
+```bash
+pnpm test            # evaluate all stored assertions against staging
+pnpm issues          # machine-readable, action-oriented failures
+pnpm report          # HTML report (failures include screenshots, video, trace)
+```
+
+Authentication is automatic: a worker-scoped fixture (`tests/auth.fixture.ts`)
+logs in once per run with `CONSOLE_STAGING_USER_EMAIL` / `_PASSWORD` and keeps
+the session in memory. **No `auth.json` is written or read** — nothing has to
+pre-exist, so it behaves identically locally and in CI (where the credentials
+come from GitHub Secrets). The seeded staging user has no 2FA.
+
+`pnpm test` always writes `output/issues.json` — a summary plus one entry per
+failing assertion, sorted by `source_line`, each carrying `source_file:line`,
+the offending `source_quote`, and a one-line error. For `element_*` failures it
+also captures the live page's same-kind labels (`observed_candidates`) and, when
+unambiguous, a `suggested_target`, so a downstream LLM can propose a doc fix
+without re-running the browser.
+
+## Adjusting coverage
+
+Edit `pages.json` and re-run `pnpm extract:all`. Add a page → it gets an
+assertion set; remove one → delete its `assertions/<slug>.json`.
+
+## Known gaps (deferred)
+
+- **Detail-page fixtures.** Assertions about Actor-detail, Schedule-detail, etc.
+  need a known fixture to navigate to. The runner currently *skips* element
+  assertions with no `at` route — surfacing the gap without false negatives.
+  Requires the seeded-user fixtures (1 Actor, 1 task, 1 schedule, named storages,
+  1 webhook, 1 completed run) from the Notion plan.
+- **Left-nav group check.** The documented global nav items (Dashboard/Store/
+  Actors/…) are a Console-wide check, not a per-page claim — not modeled yet.
+- **Session-gated pages.** Pages like `/settings/security` re-prompt for
+  credentials even within a valid session; needs a `requires_fresh_session`
+  field plus a re-auth flow.
+- **Multi-step flows.** The schema only supports atomic claims (one
+  navigate-then-check). "Click X, then Y, then Z" sequences are not modeled.
+- **No CI yet.** Local only; wiring into a scheduled GitHub Action (modeled on
+  `.github/workflows/lychee.yml`) is the follow-up.
+
+## Files
+
+```
+docs-tests/
+├── pages.json                   # adjustable list of docs pages to cover
+├── assertions/                  # committed baseline, one JSON per page (generated)
+├── prompts/
+│   ├── extract-system.md        # system prompt + known-routes table
+│   └── assertion-schema.json    # JSON Schema for the extractor output
+├── scripts/
+│   ├── extract.sh               # one page  → assertions/<slug>.json
+│   └── extract-all.sh           # whole manifest
+├── reporters/issues-reporter.ts # custom Playwright reporter → output/issues.json
+├── tests/
+│   ├── auth.fixture.ts          # worker-scoped login from env creds (in-memory session)
+│   ├── from-doc.spec.ts         # reads assertions/*.json, emits tests
+│   └── similarity.ts            # suggest-replacement helper for failures
+├── playwright.config.ts
+└── .env                         # CONSOLE_STAGING_URL (gitignored)
+```

docs-tests/assertions/console-billing.json

@@ -0,0 +1 @@
+{"source_file":"sources/platform/console/billing.md","assertions":[{"id":"billing-route","kind":"route","target":"/billing","page_context":"Billing > Current period page","source_quote":"The **Current period** tab is a comprehensive resource for understanding your platform usage during the ongoing billing cycle.","source_line":11,"needs_auth":true},{"id":"billing-historical-usage-route","kind":"route","target":"/billing/historical-usage","page_context":"Billing > Historical usage page","source_quote":"The **Historical usage** tab provides a detailed view of your monthly platform usage, excluding any free Actor compute units or discounts from your subscription plan.","source_line":21,"needs_auth":true},{"id":"billing-subscription-route","kind":"route","target":"/billing/subscription","page_context":"Billing > Subscription page","source_quote":"Navigate to [Subscription](https://console.apify.com/billing/subscription) section in Apify Console, and click the **Buy add-ons** button to explore the available options.","source_line":53,"needs_auth":true},{"id":"billing-limits-route","kind":"route","target":"/billing#/limits","page_context":"Billing > Limits page","source_quote":"The **Limits** tab displays the usage limits for the Apify platform based on your current subscription plan.","source_line":73,"needs_auth":true},{"id":"current-period-tab","kind":"element_tab","target":"Current period","at":"/billing","page_context":"Billing page tab navigation","source_quote":"The **Current period** tab is a comprehensive resource for understanding your platform usage during the ongoing billing cycle.","source_line":11,"needs_auth":true},{"id":"historical-usage-tab","kind":"element_tab","target":"Historical usage","at":"/billing","page_context":"Billing page tab navigation","source_quote":"The **Historical usage** tab provides a detailed view of your monthly platform usage","source_line":21,"needs_auth":true},{"id":"subscription-tab","kind":"element_tab","target":"Subscription","at":"/billing","page_context":"Billing page tab navigation","source_quote":"The **Subscription** tab offers a central location to manage various aspects of your subscription plan.","source_line":37,"needs_auth":true},{"id":"pricing-tab","kind":"element_tab","target":"Pricing","at":"/billing","page_context":"Billing page tab navigation","source_quote":"The **Pricing** tab offers a way to quickly check all unit pricing for various platform services related to Apify usage for your account.","source_line":61,"needs_auth":true},{"id":"invoices-tab","kind":"element_tab","target":"Invoices","at":"/billing","page_context":"Billing page tab navigation","source_quote":"The **Invoices** tab is where you can find your current and previous invoices for Apify platform usage.","source_line":67,"needs_auth":true},{"id":"limits-tab","kind":"element_tab","target":"Limits","at":"/billing","page_context":"Billing page tab navigation","source_quote":"The **Limits** tab displays the usage limits for the Apify platform based on your current subscription plan.","source_line":73,"needs_auth":true},{"id":"plan-consumption-graph-text","kind":"element_text","target":"Plan Consumption","at":"/billing","page_context":"Billing > Current period tab, Plan Consumption graph","source_quote":"the tab features a **Plan Consumption** Graph. It shows how much of your free or paid plan has been utilized up to this point.","source_line":13,"needs_auth":true},{"id":"actors-statistics-section","kind":"element_text","target":"Actors","at":"/billing","page_context":"Billing > Current period tab, statistics sections under graph","source_quote":"You can access detailed statistics related to **Actors**, **Data transfer**, **Proxy**, and **Storage**.","source_line":15,"needs_auth":true},{"id":"data-transfer-statistics-section","kind":"element_text","target":"Data transfer","at":"/billing","page_context":"Billing > Current period tab, statistics sections under graph","source_quote":"You can access detailed statistics related to **Actors**, **Data transfer**, **Proxy**, and **Storage**.","source_line":15,"needs_auth":true},{"id":"proxy-statistics-section","kind":"element_text","target":"Proxy","at":"/billing","page_context":"Billing > Current period tab, statistics sections under graph","source_quote":"You can access detailed statistics related to **Actors**, **Data transfer**, **Proxy**, and **Storage**.","source_line":15,"needs_auth":true},{"id":"storage-statistics-section","kind":"element_text","target":"Storage","at":"/billing","page_context":"Billing > Current period tab, statistics sections under graph","source_quote":"You can access detailed statistics related to **Actors**, **Data transfer**, **Proxy**, and **Storage**.","source_line":15,"needs_auth":true},{"id":"usage-by-actors-table","kind":"element_text","target":"Usage by Actors","at":"/billing/historical-usage","page_context":"Billing > Historical usage tab, table below the bar chart","source_quote":"Below the bar chart, there is a table titled **Usage by Actors**. This table presents a detailed breakdown of the Compute units used per Actor and the associated costs.","source_line":31,"needs_auth":true},{"id":"buy-add-ons-button","kind":"element_button","target":"Buy add-ons","at":"/billing/subscription","page_context":"Billing > Subscription tab, Plan add-ons section","source_quote":"click the **Buy add-ons** button to explore the available options.","source_line":53,"needs_auth":true}]}

docs-tests/assertions/console-index.json

@@ -0,0 +1 @@
+{"source_file":"sources/platform/console/index.md","assertions":[{"id":"sign-up-route","kind":"route","target":"/sign-up","page_context":"Apify Console sign-up page","source_quote":"create an account at the [sign-up page](https://console.apify.com/sign-up).","source_line":13,"needs_auth":false},{"id":"sign-in-route","kind":"route","target":"/sign-in","page_context":"Apify Console sign-in page","source_quote":"To sign in to your account, visit the [sign-in page](https://console.apify.com/sign-in).","source_line":42,"needs_auth":false},{"id":"settings-security-route","kind":"route","target":"/settings/security","page_context":"Settings > Login & Privacy section","source_quote":"go to the [Login & Privacy](https://console.apify.com/settings/security) section of your account settings.","source_line":38,"needs_auth":true},{"id":"sign-up-button","kind":"element_button","target":"Sign up","at":"/sign-up","page_context":"Sign-up form page","source_quote":"After you click the **Sign up** button, we will send you a verification email.","source_line":19,"needs_auth":false},{"id":"resend-verification-email-button","kind":"element_button","target":"Resend verification email","page_context":"Email verification page reached after sign-up flow","source_quote":"On the verification page, you can click on the **Resend verification email** button to send the email again.","source_line":25,"needs_auth":true},{"id":"continue-with-google-button","kind":"element_button","target":"Continue with Google","at":"/sign-up","page_context":"Sign-up form page","source_quote":"click the **Continue with Google** or **Continue with GitHub** buttons.","source_line":29,"needs_auth":false},{"id":"continue-with-github-button","kind":"element_button","target":"Continue with GitHub","at":"/sign-up","page_context":"Sign-up form page","source_quote":"click the **Continue with Google** or **Continue with GitHub** buttons.","source_line":29,"needs_auth":false},{"id":"recently-viewed-text","kind":"element_text","target":"Recently viewed","page_context":"Dashboard section shown after sign-in","source_quote":"**Recently viewed**: Displays Actors you have recently accessed.","source_line":48,"needs_auth":true},{"id":"suggested-actors-for-you-text","kind":"element_text","target":"Suggested Actors for you","page_context":"Dashboard section shown after sign-in","source_quote":"**Suggested Actors for you**: Recommends Actors that might interest you based on your and other users' recent activities.","source_line":50,"needs_auth":true},{"id":"actor-runs-text","kind":"element_text","target":"Actor runs","page_context":"Dashboard section shown after sign-in","source_quote":"**Actor runs**: Shows your recent Actor runs, as well as scheduled runs and tasks.","source_line":52,"needs_auth":true},{"id":"store-route","kind":"route","target":"/store","page_context":"Apify Store section accessible from left-side panel","source_quote":"| [Apify Store](/platform/console/store) | <kbd>G</kbd> + <kbd>O</kbd> | Search for Actors that suit your web-scraping needs. |","source_line":62,"needs_auth":true},{"id":"actors-route","kind":"route","target":"/actors","page_context":"Actors section accessible from left-side panel","source_quote":"| [Actors](/platform/actors) | <kbd>G</kbd> + <kbd>A</kbd> | View recent and bookmarked Actors. |","source_line":64,"needs_auth":true},{"id":"tasks-route","kind":"route","target":"/actors/tasks","page_context":"Saved tasks section accessible from left-side panel","source_quote":"| [Saved tasks](/platform/actors/running/tasks) | <kbd>G</kbd> + <kbd>T</kbd> | View your saved tasks. |","source_line":66,"needs_auth":true},{"id":"schedules-route","kind":"route","target":"/schedules","page_context":"Schedules section accessible from left-side panel","source_quote":"| [Schedules](/platform/schedules) | <kbd>G</kbd> + <kbd>U</kbd> | Schedule Actor runs and tasks to run at a specified time. |","source_line":68,"needs_auth":true},{"id":"proxy-route","kind":"route","target":"/proxy","page_context":"Proxy section accessible from left-side panel","source_quote":"| [Proxy](/platform/proxy) | <kbd>G</kbd> + <kbd>P</kbd> | View your proxy usage and credentials. |","source_line":70,"needs_auth":true},{"id":"storage-route","kind":"route","target":"/storage","page_context":"Storage section accessible from left-side panel","source_quote":"| [Storage](/platform/storage) | <kbd>G</kbd> + <kbd>E</kbd> | View stored results of your runs in various data formats. |","source_line":71,"needs_auth":true},{"id":"billing-route","kind":"route","target":"/billing","page_context":"Billing section accessible from left-side panel","source_quote":"| [Billing](/platform/console/billing) | <kbd>G</kbd> + <kbd>B</kbd> | View billing information, statistics, and invoices. |","source_line":72,"needs_auth":true},{"id":"settings-route","kind":"route","target":"/settings","page_context":"Settings section accessible from left-side panel","source_quote":"| [Settings](/platform/console/settings) | <kbd>G</kbd> + <kbd>S</kbd> | Change settings of your account. |","source_line":73,"needs_auth":true}]}