Skip to content

WAL wrapper improvements - #2934

Merged
cody-littley merged 20 commits into
mainfrom
cody-littley/wal-improvements
Feb 23, 2026
Merged

cody-littley merged 20 commits into
mainfrom
cody-littley/wal-improvements

Conversation

@cody-littley

@cody-littley cody-littley commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

Describe your changes and provide context

Ticket: https://linear.app/seilabs/issue/STO-361/improve-wal-library-correctness

This PR fixes several issues identified in this thread: https://sei-network-io.slack.com/archives/C08KPKN3SLT/p1771432603483039?thread_ts=1771432091.719279&cid=C08KPKN3SLT

  • Async vs. non-async write mode is now an explicit toggle in settings
    • legacy code that expects the write buffer size to control async behavior has been modified to maintain legacy behavior
  • Added toggles for fsync mode and shallow copy mode.
  • Threading model altered. Instead of using mutexes, all work is performed on a background goroutine, and communication with this goroutine is mediated with channels.
    • Fixes identified thread safety issues.
    • This refactor is not the only way to fix the issues identified in the thread, and so if people don't like this change I'm willing to roll to the old mutex-based implementation.
    • My personal opinion: when it comes to reasoning about multi-threaded code, this pattern is easier to reason about than using mutexes. Since there is only one thread "doing things", it's trivial to convince yourself of its safety.
    • This threading change can sometimes lead to performance improvements. But for this specific case, I think it highly unlikely that this will have a performance impact. It's very unlikely that we will see mutex contention impact performance in this code, and so performance will likely be near identical to a mutex-based implementation.
  • Added behind-the-scenes batching. Top level API remains identical. But if there is a bunch of work that builds up in the queue, the worker thread aggregates multiple writes into larger batches.
    • Large impact for small writes in async mode. 2-3x throughput improvement with writes less than 10kb.
    • Impact tapers off the larger the writes get. This is expected.
    • Extreme impact (orders of magnitude) when writing with fsync mode enabled, since fsync is super expensive and this lets us do fewer fsync operations.
      • For the consensus team's use case, this is unlikely to help much though, since they will be doing one WAL per lane and blocking on each write before proceeding to the next write.
      • If we move to all lanes in a single WAL, then consensus team will likely realize a large speedup from batching.
      • It's possible to make the consensus team's use case much faster, but to do so I'd need to completely change wal.go's API and semantics. I didn't think that appropriate in this PR.

Testing performed to validate your change

I utilized @yzang2019's benchmark. My changes had no impact on performance in some cases, and had a strong positive impact in others.

Screenshot 2026-02-18 at 3 54 48 PM

@github-actions

github-actions Bot commented Feb 19, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedFeb 23, 2026, 2:46 PM

Comment thread sei-db/wal/wal.go Fixed
@codecov

codecov Bot commented Feb 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.11321% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.55%. Comparing base (80b9aae) to head (386db4a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sei-db/wal/wal.go 72.85% 42 Missing and 15 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##             main    #2934       +/-   ##
===========================================
+ Coverage   57.83%   69.55%   +11.72%     
===========================================
  Files        2110       11     -2099     
  Lines      174123     1429   -172694     
===========================================
- Hits       100710      994    -99716     
+ Misses      64451      300    -64151     
+ Partials     8962      135     -8827     
Flag Coverage Δ
sei-chain 70.06% <73.11%> (+12.25%) ⬆️
sei-db 68.49% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/ledger_db/parquet/wal.go 75.75% <100.00%> (+0.37%) ⬆️
sei-db/wal/changelog.go 100.00% <100.00%> (ø)
sei-db/wal/wal.go 71.75% <72.85%> (+4.48%) ⬆️

... and 2099 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread sei-db/wal/changelog.go Outdated

// Legacy compatibility: originally, BufferSize > 0 implied async writes. Preserve that for
// callers who set BufferSize without explicitly setting AsyncWrites.
if config.BufferSize > 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If already we have BufferSize to indicate async or not, why do we still need to add another config for AsyncWrites (true/false) here? Isn't that a redundant config?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, we would only use the buffer when running in async mode. After my refactor, we always use the buffer regardless of whether we are running in async mode. This means we need to configure buffer size independently of async mode configuration.

My reasoning behind always using the buffer is twofold. It reduces complexity, since there is one fewer code pathway. Additionally, it allows for batching (internal) even when running in sync mode (although you need multiple writer goroutines to take advantage of batching in sync mode).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me know if you feel strongly about this. I can revert back to the old pattern of configuration without a huge rewrite.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me take a look at the new code path first for how buffer is being used now, thanks for explanation!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already pushed a change to revert to the previous config pattern. We can still use the old pattern, you'd just lose the ability to configure internal buffer sizes when you are running in sync mode. Perhaps not too much of a problem? Since internal buffer size is unlikely to be important super important when running in sync mode.

Comment thread sei-db/wal/wal.go Outdated

bufferSize := config.BufferSize
if config.BufferSize == 0 {
bufferSize = 1024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe have a const for these default value? Same as writeBatchSize

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Const added for both.

Comment thread sei-db/wal/wal.go Outdated
PruneInterval time.Duration

// If true, the writes are asynchronous, and will return immediately without waiting for the write to be durable
AsyncWrites bool

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel it sometimes could be confusing to have two configs to control one feature, user don't know which one dominates without looking into the code. Would suggest still keep only one here which is the bufferSize > 0 to indicate whether we are doing async writes or not

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config reverted to a single setting, as suggested.

Comment thread sei-db/wal/wal.go Outdated
func (walLog *WAL[T]) mainLoop() {

pruneInterval := walLog.config.PruneInterval
if pruneInterval < time.Second {

@yzang2019 yzang2019 Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about the case where we want to disable auto pruning? I think previously we have assumption if pruneInterval <= 0 then we will disable pruning, and we still need to support that case right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still supported, but I can see how it is confusing in its current form. Currently, prune() becomes a noop if it is disabled:

func (walLog *WAL[T]) prune() {
	keepRecent := walLog.config.KeepRecent
	if keepRecent <= 0 || walLog.config.PruneInterval <= 0 {
		// pruning is disabled
		return
	}

I've refactored the code to make it a more obvious that pruning is disabled.

	var pruneChan <-chan time.Time
	if walLog.config.PruneInterval > 0 && walLog.config.KeepRecent > 0 {
		pruneTicker := time.NewTicker(walLog.config.PruneInterval)
		defer pruneTicker.Stop()
		pruneChan = pruneTicker.C
	}

// ...

		case <-pruneChan:
			walLog.prune()

The pruneChan is nil if pruning is not configured. <- nil is legal in go, it will just never be selected/executed.

@blindchaser

Copy link
Copy Markdown
Contributor

I like the single-writer event loop design for writes/truncates.

one question, all reads (ReadAt, Replay, FirstOffset, LastOffset) are now also serialized through the mainLoop. In the old code these were direct calls to tidwall/wal. Have you considered keeping reads as direct calls? It would eliminate ~5 request/response types and simplify the code. I can see the argument for keeping them serialized (avoids read/truncate interleave), but given that reads are mostly only called at startup, curious about your thinking here.

@cody-littley

Copy link
Copy Markdown
Contributor Author

one question, all reads (ReadAt, Replay, FirstOffset, LastOffset) are now also serialized through the mainLoop. In the old code these were direct calls to tidwall/wal. Have you considered keeping reads as direct calls? It would eliminate ~5 request/response types and simplify the code. I can see the argument for keeping them serialized (avoids read/truncate interleave), but given that reads are mostly only called at startup, curious about your thinking here.

Converted the read methods back into direct calls.

Comment thread sei-db/wal/wal.go
}

// drain pending work, then tear down
walLog.drain()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we do this instead to be safer?
walLog.cancel() // 1. stop accepting new requests
walLog.drain() // 2. finish in-flight work already in the buffer

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Between drain() returning and cancel() being called , the context is still active. A concurrent caller (e.g. Write()) can successfully send a request into the buffered channel because the context check in Write() passes. But mainLoop has already stopped reading — that request is silently dropped and its errChan will never receive a response.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that even if a caller manages to submit a message to the buffer after Close() is called but before the context is cancelled, they will not block forever waiting for a response. When waiting for a response, we abort early if the context is cancelled:

	select {
	case _, ok := <-walLog.ctx.Done():
		if !ok {
			return fmt.Errorf("WAL was closed after write was submitted but before write was finalized, " +
				"write may or may not be durable")
		}
	case err := <-req.errChan:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, I don't think there is any harm in cancelling the context before draining additional work. Change made.

Comment thread sei-db/wal/wal.go Outdated
}

select {
case _, ok := <-walLog.ctx.Done():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we usually just do this:
case <-walLog.ctx.Done():
return err

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I follow. In this specific case, there is no pre-existing err to return.

In order to make this code easier to read, I created some helper methods for inserting and removing things from channels. Eventually I'll probably extract these methods to a utility package, but for the time being I put them inside wal.go since there are no other callers.

// Push to a channel, returning an error if the context is cancelled before the value is pushed.
func interuptablePush[T any](ctx context.Context, ch chan T, value T) error {
	select {
	case <-ctx.Done():
		return fmt.Errorf("context cancelled: %w", ctx.Err())
	case ch <- value:
		return nil
	}
}

// Pull from a channel, returning an error if the context is cancelled before the value is pulled.
func interuptablePull[T any](ctx context.Context, ch <-chan T) (T, error) {
	var zero T
	select {
	case <-ctx.Done():
		return zero, fmt.Errorf("context cancelled: %w", ctx.Err())
	case value, ok := <-ch:
		if !ok {
			return zero, fmt.Errorf("channel closed")
		}
		return value, nil
	}
}

@yzang2019 yzang2019 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM!

Comment thread sei-db/wal/wal.go Outdated
asyncWriteErrCh: make(chan error, 1),

bufferSize := config.WriteBufferSize
if config.WriteBufferSize == 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should this be config.WriteBufferSize <= 0? I think memiavl sometimes passes -1 to mean sync mode. With == 0, negative values can still flow into make(chan, size) and panic.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah seems like it's breaking some unit test

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed

@bdchatham bdchatham left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm aligned with the channel based approach

Comment thread sei-db/wal/changelog.go Resolved
Comment thread sei-db/ledger_db/parquet/wal.go Resolved
Comment thread sei-db/wal/wal.go Resolved
Comment thread sei-db/wal/wal.go Resolved
Comment thread sei-db/wal/wal.go Resolved
Comment thread sei-db/wal/wal.go Resolved
Comment thread sei-db/wal/wal.go

// Shut down the WAL. Sends a close request to the main loop so in-flight writes (and other work)
// can complete before teardown. Idempotent.
func (walLog *WAL[T]) Close() error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this isn't really thread safe although the current call pattern is safe. Could be more defensive/explicit to use sync.once. Two racing threads could try to put multiple entries into the channel. One will hang.

@cody-littley cody-littley Feb 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is thread safe even if you have concurrent Close() calls. The first caller will send a message to the background thread. Other callers may become temporarily blocked. But once the background thread cancels the context, all other callers to Close() will stop waiting.

I added a unit test to verify this behavior.

	// Calling close lots of times shouldn't cause any problems.
	for i := 0; i < 10; i++ {
		require.NoError(t, changelog.Close())
	}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling it serially is definitely no problem.

The callout is that thread safety of Close() is implicitly dependent on properly handling closeErrChan. If you protect it differently you don't have to have this implicit dependency.

Not a critical problem but wanted to mention the alternative.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main challenge that sync.Once() doesn't resolve in this scenario is ensuring that Close() blocks all callers until everything is fully closed. We don't want to just block the first caller while things are being closed, we want to block all callers. So we'd still have to either block all callers on closeErrChan, or we'd have to introduce an additional synchronization primitive like a semaphore.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's a good point. The race between the removal and replacement was just the smell I wanted to dig into more. Appreciate the discussion on it

Comment thread sei-db/wal/wal.go Resolved
Comment thread sei-db/wal/wal.go
lastOffset++
}

if err := walLog.log.WriteBatch(batch); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems problematic to return the error to all error channels since some of them may have been successfully written. In the case the wal implementation spans multiple segments then you could write sub-batches but return the error only for the last segment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it is important to differentiate which writes became durable on disk and which ones didn't within a batch. In general, I'm not sure that this is a recoverable error if the WAL stops being able to accept new writes, since we're probably bumping into something like a full disk.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm yeah. I don't have as much context as I'd like. Idk entirely how the WAL is used, but my main concern was the possibility of writing something to it but reporting to the caller that there was actually an error and what that caller will do with that signal

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In sei-chain currently if we did fail write to the WAL, e.g. write() returns an error, it will panic and cause the whole process to exit

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it - thanks for the context and discussion on this

Comment thread sei-db/wal/wal.go Dismissed
@cody-littley
cody-littley enabled auto-merge (squash) February 23, 2026 14:44
@cody-littley
cody-littley merged commit f23d05a into main Feb 23, 2026
38 checks passed
@cody-littley
cody-littley deleted the cody-littley/wal-improvements branch February 23, 2026 14:59
yzang2019 pushed a commit that referenced this pull request Feb 25, 2026
## Describe your changes and provide context

Ticket:
https://linear.app/seilabs/issue/STO-361/improve-wal-library-correctness

This PR fixes several issues identified in this thread:
https://sei-network-io.slack.com/archives/C08KPKN3SLT/p1771432603483039?thread_ts=1771432091.719279&cid=C08KPKN3SLT

- Async vs. non-async write mode is now an explicit toggle in settings
- legacy code that expects the write buffer size to control async
behavior has been modified to maintain legacy behavior
- Added toggles for fsync mode and shallow copy mode. 
- Threading model altered. Instead of using mutexes, all work is
performed on a background goroutine, and communication with this
goroutine is mediated with channels.
    - Fixes identified thread safety issues.
- This refactor is not the only way to fix the issues identified in the
thread, and so if people don't like this change I'm willing to roll to
the old mutex-based implementation.
- My personal opinion: when it comes to reasoning about multi-threaded
code, this pattern is easier to reason about than using mutexes. Since
there is only one thread "doing things", it's trivial to convince
yourself of its safety.
- This threading change can sometimes lead to performance improvements.
But for this specific case, I think it highly unlikely that this will
have a performance impact. It's very unlikely that we will see mutex
contention impact performance in this code, and so performance will
likely be near identical to a mutex-based implementation.
- Added behind-the-scenes batching. Top level API remains identical. But
if there is a bunch of work that builds up in the queue, the worker
thread aggregates multiple writes into larger batches.
- Large impact for small writes in async mode. 2-3x throughput
improvement with writes less than 10kb.
    - Impact tapers off the larger the writes get. This is expected.
- Extreme impact (orders of magnitude) when writing with fsync mode
enabled, since fsync is super expensive and this lets us do fewer fsync
operations.
- For the consensus team's use case, this is unlikely to help much
though, since they will be doing one WAL per lane and blocking on each
write before proceeding to the next write.
- If we move to all lanes in a single WAL, then consensus team will
likely realize a large speedup from batching.
- It's possible to make the consensus team's use case much faster, but
to do so I'd need to completely change `wal.go`'s API and semantics. I
didn't think that appropriate in this PR.

## Testing performed to validate your change

I utilized @yzang2019's benchmark. My changes had no impact on
performance in some cases, and had a strong positive impact in others.

 
<img width="986" height="607" alt="Screenshot 2026-02-18 at 3 54 48 PM"
src="https://github.com/user-attachments/assets/023871c3-91ed-479e-81d4-ec0a6c6d3c5d"
/>

---------

Co-authored-by: Cody James Littley <cody@MacBook-Pro.local>
Co-authored-by: Cody Littley <cody.littley@seinetwork.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants