data: Tie the lifetime of C-allocated memory to its views - #3018
Merged
Conversation
`Data.__del__` released the memory as soon as the `Data` object died, but the views handed out of it outlive that. `Function._data_allocated` in particular returns `np.asarray(self._data)`, whose `base` NumPy collapses past the `Data` down to the array `alloc` built, so it keeps the `Data` alive not at all; holding onto one of those past the lifetime of the owning Function read freed memory. That array is, however, exactly what every view of the allocation -- the `Data` included -- does anchor on. So hang the deallocation off it with a `weakref.finalize`, and the memory outlives every view of it.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3018 +/- ##
==========================================
+ Coverage 83.72% 83.74% +0.01%
==========================================
Files 257 257
Lines 54822 54884 +62
Branches 4693 4694 +1
==========================================
+ Hits 45901 45963 +62
Misses 8110 8110
Partials 811 811
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
FabioLuporini
approved these changes
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Function._data_allocated(and anything else built onnp.asarray(self._data))hands out a plain
ndarrayview whosebasechain has collapsed past theDatasubclass: NumPy anchors it on the array returned byMemoryAllocator.alloc, not on theDatawrapping it. TheData, however, wasthe only object holding
_memfree_args, and its__del__released theallocation. So as soon as the owning
Functionwent away, the buffer wasfreed while those views were still alive and still pointing at it — ause-after-free that shows up as whatever the allocator later put on that block.
Devito.jl is one such consumer:
Devito.data(f)keepsf.o."_data_allocated"and wraps its pointer with
unsafe_wrap.The fix moves the release off the
Dataand onto the array every view isanchored on:
Data.__del__goes away entirely. The memory is now released when the lastview of it — the
Data, a_data_allocated, a slice handed to a third party —is collected, which is the invariant the code always assumed. Nothing in
allocators.pychanges except one docstring sentence spelling out thatmemfree_argsmust not refer back to the array, or nothing would ever becollected.
This works uniformly for allocators that override
alloctoo (ExternalAllocator,and out-of-tree ones), since the finalizer is attached to whatever array
allocreturned.
Tests
tests/test_data.py::TestMemoryLifetimecovers it, via aPosixAllocatorsubclass that counts
freecalls:test_view_not_clobbered_by_later_allocations— hold a view, drop theFunction,clear_cache(),gc.collect(), then allocate eight same-shapedFunctions onto the freed block and check the view still reads what it did.test_view_outlives_function— nofreewhile a view is alive.test_self_built_allocator— analloc-overriding allocator still freesexactly once, and not early.
test_memory_released_with_last_view— and the memory is released once thelast view goes, i.e. this is not a leak.
Three of the four fail on
main; all four pass here.Verification
1 failed, 3763 passed, 24 skipped, 4 xfailed— the one failure(
test_dse.py::TestAliases::test_min_storage_in_isolation) is pre-existing onmaintests/test_mpi.py: 229 passedallocations): freed exactly once, never early
🤖 Generated with Claude Code
https://claude.ai/code/session_01JemwmBMrw8cnacDmWuGKdq