Skip to content

Fix NWBFile.objects going stale after the file is modified - #2243

Open
HugoFara wants to merge 1 commit into
NeurodataWithoutBorders:devfrom
HugoFara:fix-nwbfile-objects-stale-cache
Open

Fix NWBFile.objects going stale after the file is modified#2243
HugoFara wants to merge 1 commit into
NeurodataWithoutBorders:devfrom
HugoFara:fix-nwbfile-objects-stale-cache

Conversation

@HugoFara

Copy link
Copy Markdown
Contributor

Motivation

NWBFile.objects populates its dict once, lazily, and nothing invalidates it, so the first read decides its contents for the life of the file: containers added afterwards are missing from it, containers removed are still in it. all_children() repopulates the same cache as a side effect, so whether objects is correct depends on how reads and writes in unrelated code interleave.

Nothing raises when it goes wrong. In #2242 a read-only inspection froze the dict at 42 objects in a file that ended up with 85; NeuroConv's backend configuration then walked .objects and wrote the other 43 containers with backend defaults, i.e. uncompressed — visible only by reopening the file and checking its filters.

objects is now rebuilt on every access, and all_children() no longer writes to cached state. I considered keeping the cache and invalidating it on mutation, but HDMF exposes no hook for "a descendant was added or removed", so invalidation could only be best-effort. Rebuilding costs one walk per access, so reading .objects inside a loop over containers is now O(n²) where it was O(n); happy to trade the other way if you prefer.

How to test the behavior?

The snippet in #2242 prints False / True / True / False / True on dev and True / True / True / True / False with this change. End to end through NeuroConv 0.9.4, the three TimeSeries added after the .objects read are written compression=None, chunks=None on dev and gzip, chunks=(1000,) here.

Three regression tests in tests/unit/test_file.pytest_objects_includes_container_added_after_first_access, test_objects_includes_nested_container_added_after_first_access, test_objects_excludes_removed_container — fail on dev and pass with the change. pytest tests/unit: 575 passed, 3 skipped. pytest tests/integration/hdf5: 223 passed, 3 skipped. ruff check . and codespell clean. Python 3.13, hdmf 6.1.0, h5py 3.16.0.

Fix #2242

Checklist

  • Did you update CHANGELOG.md with your changes?
  • Have you checked our Contributing document?
  • Have you ensured the PR clearly describes the problem and the solution?
  • Is your contribution compliant with our coding style? This can be checked running ruff check . && codespell from the source directory.
  • Have you checked to ensure that there aren't other open Pull Requests for the same change?
  • Have you included the relevant issue number using "Fix #XXX" notation where XXX is the issue number? By including "Fix #XXX" you allow GitHub to close issue #XXX when the PR is merged.

Pull request created by Claude Opus 5 using the principles of knock-first.

NWBFile.objects populated its LabelledDict once, lazily, and nothing ever
invalidated it, so the first read of the property decided its contents for
the life of the NWBFile: containers added afterwards were absent from it and
containers removed afterwards were still in it. all_children() also
repopulated the cache as a side effect, so whether objects was correct
depended on how reads and writes in otherwise unrelated code interleaved.

Build the dict on every access instead. all_children() no longer touches any
cached state; objects walks it and keys the result by object ID, keeping the
warning for children without an object_id in the walk where it was before.

Fixes NeurodataWithoutBorders#2242
@codecov

codecov Bot commented Aug 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.22%. Comparing base (6285dd8) to head (33354b3).

Files with missing lines Patch % Lines
src/pynwb/file.py 66.66% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #2243      +/-   ##
==========================================
- Coverage   96.22%   96.22%   -0.01%     
==========================================
  Files          30       30              
  Lines        2993     2992       -1     
  Branches      433      434       +1     
==========================================
- Hits         2880     2879       -1     
  Misses         64       64              
  Partials       49       49              
Flag Coverage Δ
integration 75.00% <66.66%> (-0.01%) ⬇️
unit 86.59% <66.66%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@HugoFara

Copy link
Copy Markdown
Contributor Author

CI is not passing, I can create a second PR to fix it but I'd rather ask maintainers before stacking up work.

Two solutions from what I see:

  1. Change the nwbinspector test to test_sweep_table.parent = nwbfile (and location=None, since a real child has no in-memory path). I verified that passes both with and without this PR, so it can merge on its own schedule and this gate goes green with no further commit here. Happy to open it.
  2. Keep write-through in pynwb: hold one persistent dict, refresh real children in place, leave externally-inserted keys alone. Roughly ten lines, but it makes writing into a derived index a supported thing, and injected entries would then never expire.

1 is cleaner, but 2 is easy if you would rather not break that usage. The codecov patch percentage is the two branch partials in the new loop; I can add a case for a child without an object_id if you want it green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: NWBFile.objects is cached on first access and never invalidated

1 participant