Read the MLX cache after eviction has run, not before - #236
Merged
Conversation
`test_mlx_cache_limit_actually_bounds_the_cache` has failed on CI every run
since it landed -- `main` on 2026-08-25 and every branch since, always with the
byte-identical `assert 35340292 <= 33554432`. Locally it fails about one run in
six.
MLX admits a freed buffer to the cache and trims down to the limit on the
*next* allocation, not on the free. Sampling immediately after the churn loop
therefore catches a transient in which the last buffer -- up to 17 MB for the
sizes this test allocates -- is still sitting above the line. Whether it does
depends on the allocation history of the process, which is why it is
deterministic on CI, where the test order is fixed, and intermittent locally.
Measured across six fresh processes, cache size after the churn:
before one further allocation: 17.00 MB x5, 34.30 MB x1 (limit 32 MB)
after one further allocation: 17.30 MB x6
So the fix is to allocate once more before reading the total, which is what
any real workload does anyway. Eight consecutive local runs pass.
The limit itself was never in doubt and the test's premise is sound: the same
churn with no limit leaves 501.61 MB cached.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Independent of my three stacked PRs (#233, #234, #235) and the reason all three show red CI — this is what they're failing on.
The failure
test_mlx_cache_limit_actually_bounds_the_cachehas failed on every CI run since it landed on 2026-08-24, always with the byte-identical assertion:main, run 32881198675, 2026-08-25 — same bytesslicer-hash, 2026-09-02dev's last green run was 2026-08-09, two weeks before the test existed, which is why this hasn't been noticed. Locally it fails about one run in six.Cause
MLX admits a freed buffer to the cache and trims down to the limit on the next allocation, not on the free. Sampling immediately after the churn loop catches a transient in which the last buffer — up to 17 MB for the sizes this test allocates — is still above the line.
Whether it does depends on the process's allocation history, which is why it's deterministic on CI (fixed test order) and intermittent locally.
Measured across six fresh processes, cache size after the churn:
Fix
Allocate once more before reading the total — which is what any real workload does anyway. Eight consecutive local runs pass.
The limit itself was never in doubt and the test's premise is sound: the same churn with no limit at all leaves 501.61 MB cached. The test was asserting on a transient rather than on the settled state it meant to measure.
🤖 Generated with Claude Code