Fix test_flow_run_lateness flake by tying tolerance to API request latency #19405
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Fixes a flaky test (
test_flow_run_lateness) that occasionally fails in CI due to timing sensitivity. The test compares client-siderequest_timeagainst server-sidesa.func.now()(evaluated during query execution), and the original tolerance calculation didn't properly account for this drift.Root Cause
The test was calculating tolerance as
(now - start_of_test), which included unrelated test setup time. When API calls took longer due to CI runner contention, the tolerance became insufficient. Additionally, since the test sums lateness for 2 scheduled runs, the request→DB "now" drift applies per run and must be multiplied by 2.Changes
(now - start_of_test)to(now - request_time)to measure actual API latency2.0intofudge_secondsvariable used in both tolerance calculation and expected value adjustmenttest_elapsedby 2 since the test sums lateness for 2 scheduled runsTesting
Review Focus
2multiplier intolerance = fudge_seconds + 2 * test_elapsedassumes this test always sums exactly 2 scheduled runs. If the test changes in the future, this will need updating. Consider whether this should be derived frominterval["states"][2]["count_runs"]instead.Link to Devin run: https://app.devin.ai/sessions/bf389bc601d2448cbf9140fd612d43da
Requested by: Nate Nowack ([email protected]) / @zzstoatzz
Checklist
<link to issue>"mint.json.