Skip to content

Utility for detection of resource leaks - #4324

Merged
cody-littley merged 7 commits into
mainfrom
cjl/must-close
Sep 28, 2026
Merged

cody-littley merged 7 commits into
mainfrom
cjl/must-close

Conversation

@cody-littley

Copy link
Copy Markdown
Contributor

Creates a new utility for detecting and recovering from resource leaks.

Simply register an object that must be closed with this method. This should be done in constructors.

func newReverseIterator(table *DiskTable, segs []*segment.Segment) *reverseIterator {
	it := &reverseIterator{
		onClose: closeLiveIterator(table, segs),
		segs:    segs,
		segPos:  len(segs) - 1,
	}
	utils.MustCloseE(it, "littdb reverse iterator", (*reverseIterator).isClosed, (*reverseIterator).Close)
	return it
}

If the object becomes GC eligible without being properly closed, then the following happens:

  • If in a unit test
    • log a stack trace to where the leaked object was created
    • fail the test
  • If not in a unit test:
    • Log an angry message
    • No stack trace, too costly for production code
    • close the object

Using this utility, I detected and fixed several potential production leaks, as well as a dozen or so leaks in unit tests.

@cursor

cursor Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches core DB/view lifecycle across iterators, flatkv commit pipelines, and state stores; incorrect Abandon vs Release could pin versions or mask real leaks, but changes are mostly observability and test hygiene with explicit shutdown handling.

Overview
Adds MustClose / CloseMarker in sei-db/common/utils: constructors register closeable resources with runtime.AddCleanup; if the object is GC’d without CloseMarker.Close, tests panic with a creation stack and non-test builds log (no automatic close in the diff).

That pattern is wired through litt iterators/keymaps, pebbledb batches/DB/iterators/MVCC, view manager views and tracked iterators, flatkv (exporters, state views, atomic store views), hashvault, state WAL, giga StateDB, EVM SS, and staged snapshot types—replacing ad-hoc closed booleans and calling closed.Close(owner) from each Close/Stop.

View.Abandon() is added for shutdown/failure paths where reservations must stay pinned but the view should not count as leaked; flatkv pipelines (finalization, hash engine, snapshot writer) drain or abandon queued work when stopped or when enqueue races cancellation. DecrementReferenceCount now returns whether the last reservation was released so viewImpl.Release can mark the close marker correctly.

Smaller fixes: NewDiskTable stops the keymap on open failure; flatkv importer closes batches after commit; many tests explicitly Close/Abandon and use context.Background() for store lifetimes so teardown isn’t tied to cancelled t.Context().

Reviewed by Cursor Bugbot for commit 5945a33. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedSep 28, 2026, 7:48 PM

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a runtime.SetFinalizer-based leak detector and fixes a number of real leaks in production code and tests. I found no merge blockers, but there are three problems to address: finalizers on objects in reference cycles (they are never collected), a missing KeepAlive in the view-release path that can race, and finalizers added to hot-path iterators and batches.

Findings: 0 blocking | 5 non-blocking | 3 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • [suggestion] sei-db/common/utils/must_close.go has no unit tests. Add tests that cover the leak-detected path, the path where an object was closed properly and nothing fires, and the MustClose wrapper. Because the utility depends on GC timing, it also needs a documented pattern such as runtime.GC() plus a polling loop so tests can check it deterministically.
  • [suggestion] In a test binary the finalizer panics on the shared finalizer goroutine. That crashes the whole package's test process at a point GC decides, so the failure is reported under whatever test happens to be running, not the one that leaked the object. Consider recording leaks and failing from a TestMain/t.Cleanup hook, with a forced GC, so failures are deterministic and credited to the right test.
  • 3 suggestion(s)/nit(s) flagged inline on specific lines.

Comment thread sei-db/state_db/ss/cosmos/store.go Resolved
Comment thread sei-db/db_engine/pebbledb/iterator.go Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit fd61f1b. Configure here.

Comment thread sei-db/db_engine/litt/util/file_lock.go Resolved
Comment thread sei-db/state_db/sc/flatkv/state_view.go Resolved
@codecov

codecov Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.26966% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.99%. Comparing base (0a017df) to head (5945a33).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sei-db/state_db/sc/flatkv/store_write.go 52.94% 8 Missing ⚠️
sei-db/state_db/giga/state_db.go 0.00% 2 Missing ⚠️
sei-db/state_db/sc/flatkv/finalization_manager.go 81.81% 2 Missing ⚠️
sei-db/state_db/sc/flatkv/importer.go 75.00% 1 Missing ⚠️
sei-db/state_db/sc/flatkv/snapshot_writer.go 83.33% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4324      +/-   ##
==========================================
- Coverage   67.70%   66.99%   -0.72%     
==========================================
  Files        2168     2133      -35     
  Lines      168283   162448    -5835     
==========================================
- Hits       113938   108824    -5114     
+ Misses      54336    53614     -722     
- Partials        9       10       +1     
Flag Coverage Δ
sei-db 74.81% <100.00%> (+0.30%) ⬆️
sei-db-state-db ?
sei-db-state-db-pr 80.63% <82.92%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/db_engine/litt/disktable/disk_table.go 81.38% <ø> (ø)
...ei-db/db_engine/litt/disktable/forward_iterator.go 89.10% <ø> (-0.80%) ⬇️
...b_engine/litt/disktable/keymap/pebble_db_keymap.go 77.65% <ø> (ø)
...ei-db/db_engine/litt/disktable/reverse_iterator.go 91.30% <ø> (-1.29%) ⬇️
sei-db/db_engine/litt/util/file_lock.go 74.48% <ø> (-0.52%) ⬇️
sei-db/db_engine/pebbledb/batch.go 95.45% <ø> (-0.20%) ⬇️
sei-db/db_engine/pebbledb/db.go 82.00% <ø> (-0.36%) ⬇️
sei-db/db_engine/pebbledb/iterator.go 84.61% <ø> (ø)
sei-db/db_engine/pebbledb/mvcc/iterator.go 75.00% <ø> (-0.11%) ⬇️
...i-db/db_engine/pebbledb/mvcc/iterator_ascending.go 76.68% <ø> (+0.39%) ⬆️
... and 22 more

... and 237 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cody-littley
cody-littley added this pull request to the merge queue Sep 28, 2026
Merged via the queue into main with commit 3f14aa0 Sep 28, 2026
64 checks passed
@cody-littley
cody-littley deleted the cjl/must-close branch September 28, 2026 20:12
alexander-sei added a commit that referenced this pull request Oct 30, 2026
The RocksDB job now runs against Ubuntu's librocksdb-dev package, which
is built with assertions enabled. TestDatabasePrune left an iterator
open when its deferred Close ran, and the column family that iterator
still referenced tripped the `last_ref` assertion in ~ColumnFamilySet,
aborting the whole mvcc test binary. main closes the iterator on the
same line, added there by #4324.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants