Skip to content

data: Tie the lifetime of C-allocated memory to its views - #3018

Merged
FabioLuporini merged 2 commits into
mainfrom
fix-dangling-data-views
Sep 7, 2026
Merged

data: Tie the lifetime of C-allocated memory to its views#3018
FabioLuporini merged 2 commits into
mainfrom
fix-dangling-data-views

Conversation

@mloubout

@mloubout mloubout commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Function._data_allocated (and anything else built on np.asarray(self._data))
hands out a plain ndarray view whose base chain has collapsed past the
Data subclass: NumPy anchors it on the array returned by
MemoryAllocator.alloc, not on the Data wrapping it. The Data, however, was
the only object holding _memfree_args, and its __del__ released the
allocation. So as soon as the owning Function went away, the buffer was
freed while those views were still alive and still pointing at it — a
use-after-free that shows up as whatever the allocator later put on that block.

Devito.jl is one such consumer: Devito.data(f) keeps f.o."_data_allocated"
and wraps its pointer with unsafe_wrap.

The fix moves the release off the Data and onto the array every view is
anchored on:

if memfree_args is not None:
    weakref.finalize(ndarray, allocator.free, *memfree_args)

Data.__del__ goes away entirely. The memory is now released when the last
view of it — the Data, a _data_allocated, a slice handed to a third party —
is collected, which is the invariant the code always assumed. Nothing in
allocators.py changes except one docstring sentence spelling out that
memfree_args must not refer back to the array, or nothing would ever be
collected.

This works uniformly for allocators that override alloc too (ExternalAllocator,
and out-of-tree ones), since the finalizer is attached to whatever array alloc
returned.

Tests

tests/test_data.py::TestMemoryLifetime covers it, via a PosixAllocator
subclass that counts free calls:

  • test_view_not_clobbered_by_later_allocations — hold a view, drop the
    Function, clear_cache(), gc.collect(), then allocate eight same-shaped
    Functions onto the freed block and check the view still reads what it did.
  • test_view_outlives_function — no free while a view is alive.
  • test_self_built_allocator — an alloc-overriding allocator still frees
    exactly once, and not early.
  • test_memory_released_with_last_view — and the memory is released once the
    last view goes, i.e. this is not a leak.

Three of the four fail on main; all four pass here.

Verification

  • full suite: 1 failed, 3763 passed, 24 skipped, 4 xfailed — the one failure
    (test_dse.py::TestAliases::test_min_storage_in_isolation) is pre-existing on
    main
  • tests/test_mpi.py: 229 passed
  • allocator audit (aligned/guard/numa/external/device/shm, padded and zero-size
    allocations): freed exactly once, never early

🤖 Generated with Claude Code

https://claude.ai/code/session_01JemwmBMrw8cnacDmWuGKdq

`Data.__del__` released the memory as soon as the `Data` object died, but the
views handed out of it outlive that. `Function._data_allocated` in particular
returns `np.asarray(self._data)`, whose `base` NumPy collapses past the `Data`
down to the array `alloc` built, so it keeps the `Data` alive not at all;
holding onto one of those past the lifetime of the owning Function read freed
memory.

That array is, however, exactly what every view of the allocation -- the
`Data` included -- does anchor on. So hang the deallocation off it with a
`weakref.finalize`, and the memory outlives every view of it.
@codecov

codecov Bot commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.74%. Comparing base (b330c14) to head (954a1fd).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3018      +/-   ##
==========================================
+ Coverage   83.72%   83.74%   +0.01%     
==========================================
  Files         257      257              
  Lines       54822    54884      +62     
  Branches     4693     4694       +1     
==========================================
+ Hits        45901    45963      +62     
  Misses       8110     8110              
  Partials      811      811              
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX 68.63% <75.00%> (-0.02%) ⬇️
pytest-gpu-gcc- 78.37% <100.00%> (+0.03%) ⬆️
pytest-gpu-icx- 78.30% <100.00%> (+0.03%) ⬆️
pytest-gpu-nvc-nvidiaX 69.14% <75.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@FabioLuporini
FabioLuporini merged commit e875143 into main Sep 7, 2026
42 checks passed
@FabioLuporini
FabioLuporini deleted the fix-dangling-data-views branch September 7, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants