Catch up governed curve campaign and analysis workflow

[fallen-pc]

Summary

• publish the preserved July/August AutoSniper worktree as reviewable commits
• improve repair-aware AI Analysis, classification references, Carsales/Apify exact-URL ingestion, and canonical-tag regressions
• advance governed curves from 115 to 196 canonical tags with portable manifest snapshots and reproducible Carsales evidence inputs
• add five final Grays-backed lanes from Apify run 7WqoauuNvqzoVM5jO, raising covered sold vehicles from 4,592 to 4,696
• make the VPS the single production runtime with a main-only governed release for code, curves, repair decisions, and repair pricing
• make VPS navigation runtime/read-only while keeping curve, repair, and interactive scraper authoring local
• preserve generated CatBoost output, the newer live repair-review queue, the unfinished third curve batch, raw per-run JSON, backups, and local editor files outside the PR

Why

The previous code deployment deliberately preserved all CSV_data, which also left governed curves and repair pricing stale on production. The new release path separates VPS-owned runtime data from Git-governed business inputs, validates and backs up both release layers, records hashes and row counts, and rolls back if activation fails.

Validation

• venv\Scripts\python.exe -m pytest -q — 734 passed
• venv\Scripts\python.exe scripts\readiness_smoke.py — passed
• venv\Scripts\python.exe scripts\governance_checks.py check — 408/408 observed tags covered, 0 monotonicity errors
• venv\Scripts\python.exe scripts\project_memory.py check — passed
• focused VPS release/navigation suite — 6 passed
• focused release/curve/repair suite — 46 passed
• Ruff on the new release/navigation files — passed
• PowerShell and staged remote Bash syntax — passed

Runtime boundary

This PR still updates GitHub source and governed evidence only. It has not been deployed to the DigitalOcean VPS, and VPS-owned runtime data has not changed. Production release is intentionally blocked until this PR is reviewed, merged to main, and the working release paths are clean.

[devin-ai-integration]

Code review — PR #10 (Catch up governed curve campaign and analysis workflow)

Reviewed at git diff $(git merge-base main codex/local-dirty-catchup-20260809)..codex/local-dirty-catchup-20260809, source/logic files only (the 92k added lines are mostly governed curve snapshots and evidence CSVs).

Bugs / regressions

1. AI reasoning and expected-sale source are no longer rendered anywhere (pages/6_AI_ANALYSIS.py).
_render_overview_tab was deleted and the Overview tab now shows card_html. With it went the only consumers of confidence_notes and expected_sale_note:

• _split_notes (line ~3687) now has zero callers — dead code.
• confidence_notes is not referenced anywhere in the file after this change; expected_sale_note is only written into the analysis frame (~3549/3587) and never displayed.
• The old "Listing profile" bullets (canonical_tag, curve tag, expected sale source) are gone too, so a card whose curve tag differs from its canonical tag can no longer be diagnosed from the UI.

If dropping AI notes is intentional, please also delete _split_notes and stop persisting expected_sale_note for display; if not, re-add the notes block to the Overview tab. Same for _confidence_badges_html (~3960), dead since #6.

2. merge_output in scripts/import_carsales_apify_run.py can duplicate and accumulate rows.

identified = identified.drop_duplicates(subset=["ad_id", "url"], keep="last")
combined = pd.concat([identified, unidentified], ignore_index=True)

• Dedupe is on the pair. The same listing arriving from two actors (memo23 sets adId+canonicalUrl, abotapi may set listingId/url) yields differing pairs and survives as two market rows, inflating the comparable counts that feed curve anchors. Prefer dedupe on ad_id when non-empty, else on url.
• Rows with both keys empty are now kept unconditionally, so every re-import appends another copy of every unidentified row. Previously they collapsed. Worth capping (drop them, or dedupe on a content hash).
• sort_values(["_sort_scraped_at", "_merge_order"]) uses the default quicksort, which is not stable, so among rows sharing a timestamp and order the keep="last" winner is arbitrary. Pass kind="mergesort".
• _sort_scraped_at is a lexicographic string compare over scrapedAt values from two different actors; mixed formats/offsets (2026-08-05T00:00:00Z vs 2026-08-05 00:00:00+10:00) will not order by real time. Normalise with pd.to_datetime(..., utc=True, errors="coerce") before sorting.

3. Preflight can be silently skipped for URLs it cannot parse (scripts/run_carsales_apify.py).

url_targets = [target for url in exact_urls if (target := _carsales_url_target(url))]
targets = url_targets or [(args.make, args.model)]

_carsales_url_target returns None unless the path contains a private segment. With a --start-url-file mixing private and dealer/used URLs, the unparsable ones are dropped from targets and never preflighted — a paid actor run proceeds against them with no coverage guard, which is exactly what the preflight exists to prevent. Fail (or force --allow-covered-refresh) when any supplied URL cannot be resolved to a make/model target. Also nothing validates the URLs are on a Carsales host before they are sent to a billable actor.

4. --import-results returns exit code 0 when the run has not reached a terminal status.

if status not in TERMINAL_STATUSES:
    print("import_deferred=true ...")
    return 0

A scheduler/wrapper sees success while zero rows were imported. Prefer a distinct non-zero exit (or at minimum a deferred marker the caller checks).

Risky heuristics that need coverage

5. ("commercial", "van") in BODY_ALIASES (shared/canonical_tagging.py).
_body_from_text returns the first alias hit in tuple order, and commercial sits before people mover/coupe. Any Carsales/Grays title that says "Commercial" without also containing an earlier ute/cab keyword now resolves to van and lands in a different curve group. Suggest either dropping this alias or moving it last with a test proving ute/cab-chassis titles still win.

6. ("fastback", "coupe") will pull four-door fastbacks (Octavia-style, Sportback-adjacent naming) into the coupe body bucket. Lower risk than #5, but worth a named test.

7. _compile_alias boundary change:

-return re.compile(rf"\b{escaped}\b", re.IGNORECASE)
+return re.compile(rf"(?<!\w){escaped}(?!\w)", re.IGNORECASE)

This is not equivalent for aliases whose first/last character is non-word (\b requires an adjacent word char, the lookarounds do not), so aliases starting or ending in punctuation/digits become matchable in new contexts. The intent is presumably to match around ./- separators; please state which aliases motivated it and add a regression test, since this predicate gates every alias in the module.

8. re.fullmatch(r"ml\d{3}", code) in _normalize_series_code discards any ml### token. That is broader than the {"ix35", "cx5", "cx9"} model-token list next to it and will silently drop a genuine ML### series code if one ever appears. A comment naming the actual offending listings would keep this reviewable.

9. _normalise_fuel_type hardcodes Toyota hybrid series prefixes in the importer.

if series_code.startswith(("AXAH", "AXVH", "ZWE")):
    return "Hybrid"

Series→drivetrain knowledge lives in shared/canonical_tagging.py; putting a second copy in an ingestion script means the two can drift. Also this overrides a fuel type supplied by the source rather than only filling a blank — consider raw_fuel or hybrid_from_series(series).

10. Falsy-or chains like _numeric(specs.get("odometer") or item.get("odometer") or item.get("kms")) treat a legitimate 0 as missing. Minor for odometer, but the pattern is repeated across the new field mapping.

UI robustness (minor)

• [data-testid="stVerticalBlockBorderWrapper"]:has(.listing-shell-marker) and the [data-testid="stMetricValue"] { ... !important } override depend on private Streamlit test IDs and will break silently on a Streamlit upgrade. Given requirements.txt pins streamlit==1.55.0 that is survivable, but a comment recording the pinned-version dependency would help the next upgrade.
• expandPriceBands: int(max_listings) > 75 is an undocumented magic threshold on a billable actor input; a named constant with a one-line rationale would help.

Process

The commit set mixes source/logic changes with ~90k lines of governed curve snapshots and evidence CSVs. Per AGENTS.md's independent-governance rule, the four logic commits (import_carsales_apify_run, run_carsales_apify, canonical_tagging, curves) are the reviewable part and would be much easier to land and revert on their own, with the snapshot/evidence commits following separately.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a676675aee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Hide links to pages excluded from VPS routing

When vps_mode=True, this condition leaves the authoring pages unregistered, but the production-default pages/00_SCRAPER_OPERATIONS.py still unconditionally links to pages/7_AUTOTRADER_SCRAPER.py, and pages/03_CURVES.py still offers a st.switch_page button for Curve Builder. Clicking these controls on the VPS therefore targets pages absent from Streamlit navigation; the Curves fallback even tells the user to use a sidebar entry that no longer exists. Hide or replace these controls in VPS mode, or retain the pages as hidden routable entries.

Useful? React with 👍 / 👎.