Consolidate AutoPredict around an auditable evaluation-to-shadow path

[howdymary]

Objective

Make AutoPredict a trustworthy forecast-evaluation and shadow-deployment framework with one canonical path:

record/ingest -> validate -> evaluate -> report -> shadow -> promote

The product should prove that a user-provided forecast adds held-out value over the market baseline before real-money execution is exposed.

Prioritized issue groups

P0 — repository and evaluation integrity

• AP-001/AP-002: remove broken legacy backtest surfaces and the root package marker.
• AP-003: converge overlapping CLI and compatibility entrypoints.
• AP-004: add compile, source-test, installed-wheel import, format/lint, and type gates.
• AP-005: define one versioned dataset contract and route evaluation through one engine.

P1 — credible end-to-end product

• AP-006: replace the no-op paper loop with real shadow execution.
• AP-007: align live-product messaging and fail closed.
• AP-008/AP-009: make risk direction-aware and persist/reconcile live order state.
• AP-010: build a first-party point-in-time Polymarket recorder and deterministic replay.

P2/P3 — defensible learning and product clarity

• AP-011–AP-013: aggregate out-of-fold evidence, minimum event counts, uncertainty, and multiple-testing protection.
• AP-014: replace optimistic passive-fill assumptions with event replay and sensitivity bands.
• AP-015: describe the current capability as recalibration/parameter search until a forecast-provider interface exists.
• AP-016: rebase maturity and roadmap claims on measurable release gates.

Implementation sequence

1. Repository cleanup and release-integrity CI.
2. Canonical schema and evaluation CLI.
3. Forecast-provider boundary.
4. Polymarket recorder and deterministic replay.
5. Real shadow trading with durable state.
6. Statistical promotion hardening.
7. Live-safety readiness review.

Definition of done

• Source and installed-wheel integrity checks pass.
• One versioned data-to-report workflow is documented and implemented.
• Shadow and future live execution share normalization, decision, risk, and logging paths.
• Promotion uses aggregate held-out evidence against the market baseline.
• Live submission remains gated until reconciliation and operational safety requirements pass.

Detailed evidence, acceptance criteria, and work packets are included in the repository-consolidation specification package on the cleanup branch:

• Issue audit
• PRD
• Technical PRP
• Task plan

[howdymary]

Final implementation report — merged to main

Main commit: faab11f664c99bf339f766974eae18c2cb8c8d93
Main CI: https://github.com/howdymary/autopredict/actions/runs/29380326645 — PASS

Completed packets and pull requests

• Repository cleanup and product/spec package: Consolidate repository and document product plan #3
• Packet 0, supported live commands fail closed: Disable supported live execution commands #12 / issue Packet 0: hard-disable the reachable live submission entrypoint #4
• Packet 2, release-integrity CI: Add release integrity gates #13 / issue Packet 2: add release-integrity CI gates #5
• Packet 3, canonical dataset and evaluation CLI: Packet 3: canonical dataset evaluation #14 / issue Packet 3: define the canonical dataset schema and evaluation CLI #6
• Packet 4, typed forecast-provider boundary: Packet 4: typed forecast provider boundary #15 / issue Packet 4: introduce a typed forecast-provider boundary #8
• Packet 5, deterministic Polymarket recording and replay: Add deterministic Polymarket recording and replay #16 / issue Packet 5: build a point-in-time Polymarket recorder and deterministic replay #9
• Packet 7, aggregate statistical promotion hardening: Harden statistical frontier promotion #17 / issue Packet 7: harden statistical selection and frontier promotion #10
• Packet 6, durable deterministic shadow execution: Add durable deterministic shadow execution #18 / issue Packet 6: implement durable shadow execution and direction-aware risk #7
• Packet 8, hard-disabled live capability and NO-GO review: Hard-disable live execution capability #19 / issue Packet 8: conduct an independent live-safety readiness review #11

All packet issues are closed. No implementation PR remains open.

Final architecture and data flow

record/ingest -> validate -> evaluate -> report -> shadow -> promote

• Packet 5 records immutable public Gamma/CLOB/trade evidence with source timestamps, identity validation, hashes, pagination evidence, sticky partial completeness, and explicit gaps.
• Packet 3 validates the versioned dataset and produces a versioned evaluation report with provenance, dataset hash, code/config identity, market baseline comparison, and uncertainty.
• Packet 4 supplies typed market-baseline, recalibration, and user forecast providers; outcomes are unavailable at the provider boundary.
• Packet 6 consumes validated captures through the same provider/strategy path, applies signed risk, deterministic observed-depth/trade fills, and persists fenced/idempotent SQLite state with restart reconciliation and breakers.
• Packet 7 evaluates aggregate out-of-fold paired skill against the market baseline, clusters by independent event, applies finite-sample Student-t uncertainty and Bonferroni correction, and binds promotion to complete attempted/evaluated artifact manifests.
• Packet 8 blocks live execution at CLI, direct Python, concrete venue mutation, raw-client, installed-wheel, and audit boundaries.

Interface and schema decisions

• Canonical dataset: autopredict.dataset.v1 JSON manifest + immutable JSONL records.
• Evaluation report: provider-aware versioned report with explicit market baseline and abstentions.
• Recorder/replay: immutable capture contract; incomplete or gapped evidence never upgrades to complete.
• Forecast provider: typed probability, confidence, provenance/version, optional rationale; no untyped snapshot forecast metadata.
• Shadow state: SQLite schema v1, fixed-point price/quantity/cash, deterministic IDs and canonical state hash.
• Promotion method: paired-event-clustered-brier-student-t-v2, trusted floor >=30 independent events, familywise alpha <=0.05, Bonferroni correction.
• Live release capability: unconditional disabled gate; credentials/configuration cannot enable it.

Verification matrix

• Final local suite: 455 passed.
• Main GitHub matrix: Python 3.9, 3.10, 3.11, 3.12 all passed.
• Main package build/install/import audit: passed.
• Main Black quality and canonical mypy gate: passed.
• Compile audit: 164 Python files passed on Python 3.9 and 3.12 locally; all tracked Python compiled in every CI matrix job.
• Canonical dataset validation/evaluation: deterministic and adversarially reviewed.
• Capture replay: deterministic; gaps/partial evidence surfaced and rejected for performance claims.
• Shadow replay: repeat, restart, and fresh-database state hashes identical.
• Crash/reconciliation: atomic multi-order trade allocation, queue consumption, decision/fill tamper, fencing takeover, failed admin cleanup, and post-commit daily-loss recovery tested.
• Promotion: duplicate/reweighted/swapped rows, understated hypotheses, fake folds/artifacts, provider/genome mismatch, and finite-sample counterexample rejected.
• Installed-wheel live bypass matrix: constructor flags, __new__, injected adapter/client, venue place/submit/cancel/raw client, entrypoints, and credentialed audit all fail closed.

Documentation and migration

• Canonical workflow, dataset/evaluation, recording, shadow operations, deployment, security, and learning docs are updated.
• Versioned live-readiness artifacts are under docs/live-readiness/v1/:
  • NO_GO.md
  • THREAT_MODEL.md
  • RUNBOOK.md
  • evidence-manifest.template.json
• Legacy backtest/live surfaces were removed or made explicit fail-closed compatibility boundaries.
• Legacy promotion schema 1 is rejected with migration guidance rather than silently reinterpreted.

Known residual risks / open operational milestone

• No seven-day representative shadow evidence package exists yet. The evidence template is present, but no evidence was fabricated.
• Direct public polling is intentionally unsupported in shadow v1; operators persist and validate Packet 5 captures first.
• REST offset pagination cannot prove a transactionally frozen upstream trade set; polling completeness is reported accordingly.
• Queue-ahead is a conservative deterministic simulation assumption, not venue truth.
• Canonical hashes detect mutation but do not authenticate wholly fabricated archives; signed provenance is future hardening.
• GitHub currently warns that actions/checkout@v4 and actions/setup-python@v5 target deprecated Node 20 while runners force Node 24; jobs pass, but action-version maintenance should be scheduled.

Recommended next milestone

Run the documented representative shadow program for at least seven continuous days, produce the signed evidence manifest, exercise restart/cancel/breaker drills, and have an independent reviewer verify zero authenticated calls, zero unexplained gaps, reproducible terminal hashes, and zero reconciliation drift.

Live execution remains disabled and NO-GO. Any future live enablement requires a separate, explicitly human-authorized project after the operational evidence gate; it must not be enabled by credentials, configuration, or a routine follow-up patch.