[Bug] : sync-omitted stage runs measure stale scratch cache, poisoning the catalog with old evidence under new content hashes

[Sagar-024]

Version or commit

Current main

Environment

Ubuntu 24.04 (WSL2), Python 3.12, x86_64, file:// obstore root

Minimal reproduction

Using a file:// bucket-backed storage root with independent worker mirrors:

1. Worker 1 runs the full profile on clean footage (v1). Scratch is populated, measurements pass (black_frame_pct=0.00).
2. The canonical object in the remote store is overwritten with altered camera payload (v2, bumped pipeline_version, true black_frame_pct=50.00). Worker 2 processes this and records the true v2 measurements.
3. Worker 1 runs a sync-omitted profile (e.g., META + MEDIA only). Its etag-verified fetch downloads the new v2 canonical, but the scratch directory is not invalidated.

Expected behavior

Worker 1 fetches the new v2 canonical, invalidates or bypasses the stale scratch cache, measures the v2 pixels, and records the true v2 measurements (50.00) under the new episode_id.

Actual behavior

Worker 1 fetches the v2 canonical but keeps the v1 scratch (identical sha and mtime_ns). It measures the v1 pixels and records v1 measurements (0.00) under the v2 episode_id.

Because the stale run's measurement values differ from the truth, its _run_fingerprint is unique, so the catalog appends it as a new row. measurements_latest ranks by recorded_at, so the stale, newer measurement wins the wide view.

Worse, the poisoning is self-perpetuating: if a fresh worker (Worker 3) later tries to repair this by re-measuring, it produces the true values, which match the original v2 fingerprint. The catalog dedupes the repair (written=False), and it fails to overwrite the stale latest view. The system actively protects the lie.

Additional context

Root cause: The only scratch invalidation is shutil.rmtree(scratch_dir) at app.py:2391, guarded by if Stage.SYNC in enabled_stages (app.py:2347). The sync-omitted route (app.py:2395-2412) fetches the new canonical but never touches the scratch directory. Episode.video() relies on a bare output.exists() check (episode.py:551-554), serving the stale remux.

Why it matters: Curation cuts and gates accept or reject episodes using footage that isn't the footage. The downstream training corpus inherits mislabeled quality verdicts that survive the exact byte-change the content-hash was supposed to catch. The append-only deduplication logic (_run_fingerprint) weaponizes the repair path, making the poison durable.

Fix direction: Make the scratch content-bound: key scratch_dir on content_episode_id(canonical_path), or drop a digest marker inside the scratch dir and remux whenever it disagrees with the fetched canonical. Additionally, the measurement deduplication logic needs a retraction or force-overwrite path for later-arriving stale appends that lose to the truth.

Definition of done:

• A sync-omitted run on a worker with stale scratch correctly measures the newly fetched canonical bytes, not the old ones.
• _run_fingerprint deduplication does not allow a stale measurement to permanently shadow a true measurement in the latest views.
• Mutation proof: tamper the canonical payload without touching scratch, and the recorded measurements must change.
• Existing app/storage/catalog suites stay green.

[Sagar-024]

Leaving this one open for the community! I'm currently heads-down on the cache collision issue, so I won't be picking this up immediately.

This is a fantastic deep-dive for anyone wanting to understand how App.fetch, the scratch cache, and _run_fingerprint deduplication interact. The first half of the fix (content-keying the scratch directory) is pretty straightforward, but the second half (handling the deduplication/retraction in the catalog) is a really interesting architectural puzzle.

If anyone wants to take a shot at the PR, please go for it! I'd be more than happy to review your code, help you set up the local file:// bucket repro environment I used, and help you write the mutation proof. Let's get this fixed together.

And if no one claims this, I will pick it up myself after my cache collision PR is merged.

[kstonekuan]

Confirmed. app.py:2351 puts scratch under run_dir, which is keyed on the source identity, not the canonical content, so new bytes at the same source reuse the old mp4 through episode.py:551.

Key the scratch on the canonical's content, not the source. episode_id already is that hash.

The repair trap is the worse half and is worth its own issue if the fix does not cover it: a correct re-measure reproduces the original fingerprint, dedupes to written=False, and cannot displace the stale row that measurements_latest is ranking first. Say which of the two you are fixing.

[kstonekuan]

Reopening: #539 fixed the stale-scratch half only. The repair trap is still open, and the PR said so.

A correct re-measure reproduces the original fingerprint, dedupes to written=False, and cannot displace the stale row that measurements_latest ranks first. That needs a supersede or retraction path.

[kstonekuan]

Both halves are in now: #539 keyed scratch on canonical content, #541 makes a recurring outcome current again.

The repair path is covered by test_recurring_outcome_becomes_current_and_consecutive_retries_deduplicate, which asserts episodes.status flips back on the third attempt of an A/B/A sequence.