feat: reclaim old task records from the download status file

[phil-IO-p]

What this does

Bounds the growth of the per-queue download-status file, which until now only ever grew. _build_status_file_content merges each deadline queue sync-output run onto the previous file and nothing was ever removed, so size scaled with the total number of frames ever downloaded through a queue.

Once more than 100,000 task records are retained, the records of the least recently updated eligible jobs are emptied. Job rows themselves are never removed.

Counterpart change on the reader side, a 64 MiB read ceiling in Deadline Cloud Monitor.

Why the budget counts frames, not jobs

Task records dominate size. Measured on the pretty-printed file, two independent measurements agreeing within 3%:

Item	Bytes
One task record	230 to 236
One job row	329 to 406

Capping job count instead would leave size at the mercy of frames-per-job:

Frames per job	File size under "newest 50 jobs"
100	1 MB
1,000	11.5 MB
10,000	115 MB

That last row fails worst for the large-render studios the feature exists to serve.

What is protected, and why

Two sets of records are never reclaimed.

Jobs that did not finish cleanly. A job is eligible only if every task record is a plain success, with no failed, no farm_failed, and no error_code. The Monitor derives a job's partial-failure indicator from the records alone rather than from the job's counts, so dropping them from a partly-failed job would repaint it as a clean success. Job status alone is not sufficient here: a downloaded job can legitimately hold failed tasks.

This run's download candidates. The farm_failed guard in this module reads records from disk to suppress a failure the API replays after a requeue, so emptying them for a job being actively synced would let an already-downloaded frame be recorded as a farm failure.

That second protection deliberately covers the candidate categories only, not every job the checkpoint knows about. An inactive job is re-appended to the checkpoint on every run so a later requeue can be detected (_incremental_download.py:1201), and inactive is recomputed as "in the checkpoint but not a candidate", so a finished job is inactive permanently. Protecting that set would exempt exactly the accumulated history this change exists to reclaim, and the overshoot branch would become the steady state rather than the exception.

Accepted consequences

• The file is not bounded. Job rows accumulate at ~330 bytes each, roughly 33 MB per 100,000 jobs. Only records are reclaimed. See the next section for why rows are left alone.
• The record target is a reclaim budget, not a ceiling. Records held by an ineligible job, or by a candidate this run, are kept without spending it, so the retained total can exceed the target. The overshoot is logged at info with both figures broken out, so an operator can tell protected records from unreclaimed ones.
• A reclaimed job that is later requeued can misreport one frame. With no on-disk record for the farm_failed guard to match, a frame whose output was downloaded earlier can be recorded as a farm failure. The same applies to a job another machine is tracking, since one machine cannot see another's candidate set. This is the price of the change reclaiming anything at all; the alternatives were a no-op or zeroed file counts.
• A queue whose jobs mostly carry failures stays large. Little is eligible there. Lifting this needs a marker on reclaimed jobs plus new logic in the merge path, left as a follow-up.
• The first run after this ships still parses the old oversized file in full inside the lock, since the read precedes any reclaim. Measured at 367 ms for a 101 MB file. One-time, and the file it writes back is smaller.

Why job rows are never removed

Deleting a row makes the next run rebuild it from scratch with zeroed file counts, because the preserve-counts branch needs the existing entry to be there. Avoiding that requires knowing a job is finished with, and nothing in this file can show it:

• last_updated moves only on a status change, never on a sync. A job that finished weeks ago and is still enumerated every run carries an old timestamp while being actively tracked.
• Several machines write this file with independent checkpoints, so one machine's job set proves nothing about what another still tracks.

An age gate on last_updated was tried and removed for that reason: it reads "nothing changed recently" as "safe to delete", which is the normal steady state of a healthy finished job. Bounding rows needs a field that advances on every sync. That would also lift the eligibility restriction above, so it belongs in its own change rather than bolted on here.

Shared-filesystem hazards handled

This file is written by several machines to a shared location, so the retention order cannot trust its contents:

• A timestamp further ahead than a five-minute skew allowance is clamped to now, not demoted to oldest. A machine whose clock is fast is writing its newest entries, so treating them as oldest would make every other writer reclaim that machine's records first. The allowance is one-sided by nature: a clock reading in the past produces timestamps indistinguishable from genuinely old ones.
• fromisoformat only accepts a trailing Z from Python 3.11, and requires-python is >=3.9, so which records were reclaimed first would otherwise depend on the interpreter running the sync. Z is now normalised.
• Entries and records of an unexpected JSON shape are treated as ineligible instead of raising. write_download_status_file swallows every exception and only logs, so a shape error in the reclaim pass would have left that queue's file frozen at its old content indefinitely.

Why feat: and not feat!:

Deliberate, and worth a reviewer's disagreement. The schema is unchanged, no CLI subcommand, argument or default changes, and no job row ever disappears. What does change is that an empty tasks dict can now mean "records reclaimed" as well as "nothing downloaded", which is called out on the schema-version constant.

Testing

• hatch run test: 3842 passed, 22 skipped, 0 failures. Coverage 77.67% against the 69% gate; the changed module is at 95%, with the remaining uncovered lines pre-existing.
• hatch run fmt, hatch run lint (ruff + mypy), hatch build: all clean.
• 21 new tests plus 7 parametrized cases, including one that goes through the public write_download_status_file path with a real read/write cycle rather than only the internal builder.
• Every guard was verified by removing it and confirming its test fails. That check caught three tests of mine that passed for reasons unrelated to what they were written to check, including one whose subject was unreachable because a recently-settled row sorts to the front rather than into the tail.
• Not run: the integration suite. Nothing was measured against a real network share, so any timing figure here is CPU only.

[github-actions]

This test guards the invariant that matters most about trimming — that a retained "downloaded" record still suppresses a stale farm_failed — but as written it cannot distinguish the untrimmable path from the live-candidate path, and it does not cover the case where the invariant actually breaks.

MOCK_JOB_ID is passed in completed={MOCK_JOB_ID}, so it lands in all_job_ids, which is exactly what _build_status_file_content hands to _trim_retained_history as candidate_job_ids. In the trim loop that means the elif job_id in candidate_job_ids branch retains it regardless of _is_history_trimmable. Stub _is_history_trimmable to return True unconditionally and this test still passes: job-new (day 9, 10 records, not a candidate) spends the whole shrunken budget, and MOCK_JOB_ID is then exempted as live rather than as untrimmable. So the assertion pins the merge guard in _build_status_file_content, not the trimming decision the test name refers to.

It cannot be fixed by just dropping the job from the categorized sets, because task_download_results is only consumed inside the for job_id in all_job_ids loop — any job receiving a farm_failed record this run is necessarily a candidate, hence necessarily exempt. Which is the real point: within a single run the invariant is unbreakable, and the risk is entirely cross-run — trim on run N while the job is not a candidate, then have it become a candidate again on run N+1 and receive a farm_failed record for a task whose "downloaded" evidence is gone. That is the regression the _trim_retained_history docstring already admits ("A reclaimed job that is later requeued can report an already-downloaded frame as a farm failure"), and it is reachable in practice: _retrieve_session_actions_for_session collects FAILED taskRun ids from every retrieved session before the checkpoint_job_session_completed_indexes filter is applied, so a re-retrieved session can re-surface an old farm failure.

Worth restructuring as two builder calls — run 1 with the job in no categorized set so its records are actually reclaimed, run 2 feeding that output back in with the job in completed plus a farm_failed task result — and asserting whatever the intended behaviour is there. Even if the answer is "the frame is reported as farm_failed", pinning it makes the accepted cost visible in the suite instead of only in a docstring; today the suite reads as though the invariant holds unconditionally.

[larrygao001]

nit: can we make this more visible to the users?

[larrygao001]

Can we make this more visible to the users?

[andychoquette]

In instances where a task row has been reclaimed, what is the customer experience. Are those task rows stale until synced? I'm not super familiar with this functionality.

[phil-IO-p]

In instances where a task row has been reclaimed, what is the customer experience. Are those task rows stale until synced? I'm not super familiar with this functionality.

@andychoquette Good question, and it turned up something we had missed, so thanks for asking!

Short answer
They do not come back on their own. A sync only writes per-frame records for frames it downloads on that run, and a job that already finished and downloaded is not re-downloaded, so once its per-frame detail is reclaimed it stays gone. "Stale until synced" would actually be the better outcome; the reality is closer to "gone unless something forces a re-download of that job".

What a customer would see today, for a job whose per-frame detail has been reclaimed

• The job-level Download status stays correct. The row keeps its status and its file counts, so the job still reads as Downloaded with the right totals.
• The per-frame Download status column in the job panel is where the problem is. With no per-frame record, and the job still present in the file, the column concludes that the sync looked and found nothing, and shows No outputs for every frame. Those outputs were downloaded and are on disk, so that is simply wrong, and it is worse than showing a dash because it makes a positive claim.

The underlying reason is that the reader cannot currently tell "these records were reclaimed to keep the file small" apart from "this job never had records", and those two cases deserve different labels.

So I am holding this as a draft rather than pushing it forward. Landing it safely needs two things together:

1. A marker in the status file saying a job's per-frame detail was intentionally reclaimed, and
2. A change on the reader side to honor that marker instead of concluding No outputs. I had deliberately avoided adding a field to the file, but this is the third separate problem that points at needing one, so that looks like the right call now.

[phil-IO-p]

Closing this. Clearing a finished job's task records makes Deadline Cloud Monitor report downloaded frames as having no outputs, and deleting the whole row breaks the same way as soon as the job is requeued.

Replaced by #1361: write the file compactly (40% smaller) and report its size, so any ceiling comes from real numbers instead of a guess.