Skip to content

[codex] Fix data parser cache leaks - #735

Closed
ObservedObserver wants to merge 1 commit into
mainfrom
codex/fix-data-parser-cache-leak
Closed

[codex] Fix data parser cache leaks#735
ObservedObserver wants to merge 1 commit into
mainfrom
codex/fix-data-parser-cache-leak

Conversation

@ObservedObserver

Copy link
Copy Markdown
Member

Summary

  • Replace method-level @lru_cache() on data parser cached properties with per-instance cached_property.
  • Cover pandas parser raw_fields and field_metas with a GC regression test.
  • Verify parser instances remain picklable after cached properties are loaded.

Root Cause

functools.lru_cache stores calls in the decorated function wrapper. For instance methods and properties, that cache key includes self, so parser instances are retained by the class-level cache after callers drop their references. Those parser instances keep their DataFrames alive, causing memory growth in long-running processes that parse many datasets.

Notes

This reimplements the core fix from #724 against current main, avoiding the merge conflicts and avoiding a per-instance threading.Lock so parser pickling keeps working.

Fixes #723

Validation

  • ./venv/bin/python -m pytest tests/test_data_parsers.py -q
  • ./venv/bin/python -m pytest -q
  • ./venv/bin/python -m ruff check .

@ObservedObserver

Copy link
Copy Markdown
Member Author

Closing in favor of #736, which preserves the original #724 author commits while keeping the same final fix and passing CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Memory Leak in Data Parsers: Instance Methods Decorated with @lru_cache Retain Large DataFrames

1 participant