Fix the three silent failures behind the empty harvest

docs/superpowers/plans/2026-08-21-harvest-net-and-grading.md

@@ -0,0 +1,164 @@
+# Harvest Reach and Grading Reliability Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use
+> superpowers:subagent-driven-development (recommended) or
+> superpowers:executing-plans to implement this plan task-by-task. Steps use
+> checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make adjacent vocabulary, deleted-domain harvesting, and free DR
+grading broad, retry-safe, and operationally bounded.
+
+**Architecture:** Keep parsing, matching, retry classification, and grading
+orchestration in Worker-runtime-free modules with injected I/O. Repositories
+provide atomic claims; server functions adapt Worker bindings; the React panel
+invokes only explicit actions.
+
+**Tech Stack:** TypeScript, Vitest, TanStack Start/Query, Drizzle ORM,
+Cloudflare D1, Postgres, OpenRouter AI SDK.
+
+**Spec:** `docs/superpowers/specs/2026-08-21-harvest-net-and-grading-design.md`
+
+## Global Constraints
+
+- Keep medium reasoning, usage accounting, the ZDR provider order, and
+  `minimax/minimax-m3` for adjacent terms.
+- Only HTTP 401 plus exact message `You cannot download file.` is permanent.
+- TLDs are exactly `com`, `net`, `org`, and `co`; daily cap is 450.
+- Keep one 50-query Free-plan budget constant and derive harvest/grading sizes.
+- Each expired-domain cron tick harvests at most one fairly rotated project or
+  grades one batch, never both; rank checks use a separate trigger invocation.
+- Grade at most eight rows per invocation, newest first, concurrency three,
+  three attempts.
+- Keep 15-row inserts and at most 100 D1 bound parameters.
+- `null` means unknown; paid work requires an explicit click.
+- Vitest imports no module that statically imports `cloudflare:workers`.
+- Generate additive D1/Postgres migrations but do not apply them remotely.
+
+---
+
+### Task 1: Adjacent-term response and cache safety
+
+**Files:**
+
+- Modify: `src/server/features/expired-domains/adjacentTerms.test.ts`
+- Modify: `src/server/features/expired-domains/adjacentTerms.ts`
+- Modify: `src/server/features/expired-domains/harvestVocabulary.test.ts`
+- Modify: `src/server/features/expired-domains/harvestVocabulary.ts`
+
+**Interfaces:** Produces strict whole-response trace rejection, pinned M3
+request options, and a v2 vocabulary payload with a legacy reader.
+
+- [ ] Add failing tests for a realistic `<think>` blob, whitespace response,
+      pinned model/options, reasoning headroom, no-cache-on-empty, v2 reads/writes,
+      legacy arrays, and null categories.
+- [ ] Run the two focused Vitest files and confirm failures are caused by the
+      missing behavior.
+- [ ] Build the adjacent model with the explicit M3 slug, request about 1,500
+      output tokens, reject trace-shaped replies, and log finish reason/text length
+      for empty visible text.
+- [ ] Write `{ terms, categoryByTerm }`, fall back to v1 arrays, and normalize
+      absent/null categories to `uncategorised`.
+- [ ] Re-run the focused tests.
+
+### Task 2: Permanent feed skips and four-TLD harvesting
+
+**Files:**
+
+- Modify: `src/server/lib/whoisfreaks.test.ts`
+- Modify: `src/server/lib/whoisfreaks.ts`
+- Modify: `src/server/features/expired-domains/domainHarvest.test.ts`
+- Modify: `src/server/features/expired-domains/domainHarvest.ts`
+- Modify: `src/server/features/expired-domains/manualDomainHarvest.test.ts`
+- Modify: `src/server/features/expired-domains/manualDomainHarvest.ts`
+- Modify: `src/server/features/expired-domains/repositories/HarvestedDomainRepository.ts`
+- Modify: `src/server/features/expired-domains/services/scheduledDomainHarvest.ts`
+- Modify: `src/serverFunctions/domainHarvest.ts`
+
+**Interfaces:** Produces a typed permanent-date error, atomic `skipRun`,
+`skippedRuns`, separate response dates, one compiled matcher per project, and
+shared-stream cancellation only when all matchers are full.
+
+- [ ] Add failing tests proving only the exact 401 body is permanent; bad-key
+      and transient responses retry.
+- [ ] Add failing orchestration tests for skip persistence, separate response
+      dates, four TLDs, the 450 ceiling, and shared-stream cancellation.
+- [ ] Parse the response body once, carry the typed permanent signal through the
+      injected stream, complete the owned claim with `skip_reason`, and release
+      all retryable failures.
+- [ ] Change both feed adapters to the four approved TLDs and retain streaming,
+      the compiled alternation, and 15-row insert chunks.
+- [ ] Re-run WhoisFreaks, vocabulary matcher, harvest, and manual-harvest tests.
+
+### Task 3: Atomic bounded DR grading
+
+**Files:**
+
+- Create: `src/server/features/expired-domains/domainRatingGrading.test.ts`
+- Create: `src/server/features/expired-domains/domainRatingGrading.ts`
+- Modify: `src/server/features/expired-domains/repositories/HarvestedDomainRepository.ts`
+- Modify: `src/server/features/expired-domains/services/scheduledDomainHarvest.ts`
+
+**Interfaces:** Produces atomic attempt claims and an injected grading
+orchestrator capped at eight / concurrency three.
+
+- [ ] Add failing tests for newest-first selection, at-most-eight calls,
+      concurrency three, attempt ceiling three, overlapping claims, real zero,
+      unknown null, and one aggregated failure log.
+- [ ] Implement the pure concurrency worker and an update/returning repository
+      claim that increments attempts atomically.
+- [ ] Route scheduled grading through the shared service and re-run tests.
+
+### Task 4: Explicit free Grade now action
+
+**Files:**
+
+- Modify: `src/serverFunctions/domainHarvest.ts`
+- Modify: `src/client/features/expired-domains/HarvestedDomainsPanel.tsx`
+- Add focused server/client behavior tests where existing boundaries allow.
+
+- [ ] Add a failing test proving grading has no APIVerve/billing/availability
+      dependency and no grade work runs on mount.
+- [ ] Wire the action to the shared free grading service in sequential
+      eight-domain HTTP requests, refresh after each batch, report remaining
+      rows, and provide progress plus cancellation.
+- [ ] Re-run focused tests and type checking.
+
+### Task 5: Additive dual-database migrations
+
+**Files:**
+
+- Modify: `src/db/app.schema.ts`
+- Modify: `src/db/pg/app.schema.ts`
+- Create: next generated D1 and Postgres migrations and metadata snapshots.
+
+- [ ] Add `domain_rating_attempts` with non-null default zero and nullable
+      `skip_reason` in both schemas.
+- [ ] Generate migrations in sequence and inspect that their SQL is additive.
+- [ ] Do not run a migration-apply command.
+
+### Task 6: Integrated verification and review
+
+- [ ] Run all focused regression files.
+- [ ] Run `pnpm ci:check` and read its complete exit output.
+- [ ] Run `pnpm test` and read its complete exit output.
+- [ ] Inspect `git diff --check`, generated SQL, Worker import boundaries, D1
+      bind counts, and explicit-click spend paths.
+- [ ] Request whole-branch review and resolve critical/important findings.
+
+### Task 7: Free-plan invocation budgeting
+
+**Files:**
+
+- Create: `src/shared/workerQueryBudget.ts`
+- Create: `src/shared/workerQueryBudget.test.ts`
+- Create: `src/server/features/expired-domains/scheduledDomainHarvestPolicy.ts`
+- Create: `src/server/features/expired-domains/scheduledDomainHarvestPolicy.test.ts`
+- Modify: `src/server/features/expired-domains/services/scheduledDomainHarvest.ts`
+- Modify: `src/server.ts` and `wrangler.jsonc`
+
+- [ ] Prove with three capped projects that one invocation selects exactly one
+      harvest and remains within the 50-query budget.
+- [ ] Derive the 450-match and eight-domain limits from the one plan constant.
+- [ ] Select the oldest pending project/date with deterministic tie rotation.
+- [ ] Run grading only on ticks with no pending harvest.
+- [ ] Separate rank-check and expired-domain Cron Triggers.

docs/superpowers/specs/2026-08-21-harvest-net-and-grading-design.md

@@ -0,0 +1,115 @@
+# Harvest Reach and Grading Reliability
+
+Date: 2026-08-21
+Status: Approved for implementation
+
+## Goal
+
+Make the deleted-domain harvest broad enough to be useful, safe when its model
+or feed returns an ambiguous response, and bounded when automatic domain-rating
+work repeatedly fails.
+
+## Adjacent vocabulary
+
+The adjacent-term request uses `minimax/minimax-m3` explicitly and does not
+read `OPENROUTER_MODEL`. It keeps OpenRouter usage accounting, the existing
+ZDR provider order and fallbacks, and `reasoning: { effort: "medium" }`. Its
+output budget is about 1,500 tokens so a realistic reasoning trace cannot
+consume the entire visible response. Up to 50 usable terms are retained from a
+model response.
+
+An empty or whitespace-only visible answer is a failed enrichment: log the
+finish reason and visible text length, return no adjacent terms, and do not
+cache it. A reply that resembles prose or a `<think>` trace is rejected as a
+whole before individual hostname-shaped words are considered.
+
+Vocabulary cache writes use a v2 object:
+
+```ts
+{ terms: string[], categoryByTerm: Record<string, string | null> }
+```
+
+Legacy arrays remain readable. A missing or null category resolves to the
+literal `uncategorised`. The current flat adjacent-term protocol has no
+category source, so newly derived flat terms also use `uncategorised`.
+
+## Feed completion and retry semantics
+
+`harvest_runs.skip_reason` distinguishes a completed harvest from a permanent
+date skip. Only a WhoisFreaks HTTP 401 response whose parsed body has the exact
+message `You cannot download file.` is the measured subscription-window case
+and may permanently skip the date. A bad-key 401, malformed error body, rate
+limit, transport error, and every other failure remain retryable.
+
+Server responses expose harvested dates and skipped dates separately. A skip is
+not presented as a harvest.
+
+## Harvest breadth and limits
+
+Each feed pass accepts `.com`, `.net`, `.org`, and `.co`. Each
+project/date may store at most 450 matches. The feed remains streamed;
+matching uses one compiled alternation per project, inserts remain in 15-row
+chunks (therefore below D1's 100-bound-parameter ceiling), and the stream is
+cancelled only after every participating project's cap is full.
+
+## Cron scheduling and query budget
+
+`WORKER_QUERY_BUDGET` is the single plan-level switch. It is 50 for the
+current Workers Free deployment; Cloudflare D1 allows 1,000 queries per Worker
+invocation on Workers Paid. Harvest and grading batch sizes are derived from
+that constant with explicit headroom.
+
+Expired-domain work runs on a dedicated offset Cron Trigger, separate from rank
+checking. A tick performs exactly one unit: it either harvests one deterministic
+project/date candidate or, when no harvest is pending, grades one batch.
+Candidates are ordered by oldest pending date and rotated by tick among ties so
+three harvestable projects cannot starve.
+
+A capped harvest uses at most 30 insert statements for 450 matches. Together
+with the fleet read, preparation, cache, feed, claim, ownership, and completion
+work, the modeled worst-case ceiling is 41 subrequests (including both a lost
+completion write and its fenced release), leaving nine of headroom. A day is
+completed only after the stream and all inserts finish; hitting the cap is a
+complete bounded harvest, while an error remains retryable and is never
+recorded as a partial completed day.
+
+## Domain-rating grading
+
+`domain_rating_attempts` begins at zero and is capped at three. Work is
+claimed atomically so overlapping invocations cannot grade the same row.
+Candidates are newest first. One invocation performs at most eight lookups with
+concurrency three. Candidate selection plus the remaining-count query and eight
+worst-case five-subrequest lookups total 42, leaving eight subrequests of
+headroom. Failures stay unknown (`domain_rating = null`) and are
+summarized in one batch log containing each domain and its reason; they do not
+emit one log per row.
+
+The stored-list UI has a free `Grade now` action. It uses the same grading
+service, does not call APIVerve, has no spend gate, and runs only after the user
+clicks. Each HTTP invocation accepts no more than eight loaded domains and
+returns the ungraded remainder. The panel sends sequential requests, refreshes
+the rows and shows progress after every batch, and exposes cancellation. The
+loop stops retrying a batch when it stalls or reaches the three-attempt cap,
+then continues to later loaded rows. The overall operation ends after every
+batch has had its bounded turn or the user cancels, so one stubborn row cannot
+block the rest and the loop can never spin forever. Availability remains a
+separate paid explicit-click action.
+
+## Persistence and rollout
+
+Additive migrations are generated for both D1/SQLite and Postgres. They add
+`harvest_runs.skip_reason` and
+`harvested_domains.domain_rating_attempts` without rewriting or deleting
+existing data. Migrations are generated only and are not applied to any remote
+database.
+
+## Verification constraints
+
+- Write and observe a failing Vitest regression before each production fix.
+- No module imported by Vitest may statically import `cloudflare:workers`.
+- D1 statements use at most 100 bound parameters.
+- Modeled expired-domain work stays at or below 50 subrequests per invocation.
+- `null` always means unknown.
+- Paid work occurs only after an explicit click.
+- Final verification is `pnpm ci:check` followed by `pnpm test`, with
+  the equivalent npm scripts allowed when pnpm is unavailable.

drizzle-pg/0018_worried_omega_red.sql

@@ -0,0 +1 @@
+ALTER TABLE "harvested_domains" ADD COLUMN "domain_rating_attempts" integer DEFAULT 0 NOT NULL;

drizzle-pg/0019_eager_wraith.sql

@@ -0,0 +1 @@
+ALTER TABLE "harvest_runs" ADD COLUMN "skip_reason" text;

drizzle-pg/0020_solid_robbie_robertson.sql

@@ -0,0 +1,2 @@
+ALTER TABLE "harvested_domains" ADD COLUMN "domain_rating_claim_id" text;--> statement-breakpoint
+ALTER TABLE "harvested_domains" ADD COLUMN "domain_rating_lease_expires_at" text;