macOS: cross-process WAL writer lock (flock/fcntl) fails on FileStream-derived fd

[mrdevrobot]

Summary

3 tests in MultiProcessWalSharedMemoryTests fail on macOS (confirmed on Darwin 25.5.0 / arm64) with a locking failure that is not present on Linux CI. Root cause is deeper than a single wrong constant and isn't fully resolved yet.

What's already fixed (see commit ccda67b)

WalShmFcntl.cs used the Linux value of F_WRLCK (1) unchanged on macOS, where 1 is actually F_RDLCK (macOS F_WRLCK is 3, verified against the SDK's sys/fcntl.h). Every "write" lock request on macOS was silently taking a read lock instead — a real, independent bug, now fixed.

Separately, fcntl(F_SETLK) was replaced with flock() on macOS after finding that fcntl(F_SETLK) reliably returns EINVAL when called on a file descriptor obtained from a .NET FileStream on this platform — reproduced with a bare P/Invoke + FileStream probe, independent of BLite entirely. A raw libc open()-backed descriptor is unaffected; only FileStream-derived descriptors trigger it, and it happens regardless of the lock type/offset passed.

What's still broken

Switching to flock() fixed 10 of the 13 originally-failing tests, but exposed a different problem: calling flock() manually on a FileStream's underlying fd appears to conflict with something in .NET's own Unix FileStream implementation. Isolated with a minimal probe:

• Two plain FileStream opens on the same path (FileShare.ReadWrite, no manual locking at all) succeed concurrently — fine.
• If a manual flock(fdA, LOCK_EX | LOCK_NB) is taken on the first FileStream's fd, a second, independent FileStream.Open on the same path then throws IOException: ... because it is being used by another process — even though the second open only needs FileShare.ReadWrite and isn't attempting to lock anything itself.

This suggests .NET's Unix FileStream open path consults/participates in the file's flock() state on macOS, and a lock taken manually on one fd is visible to (and blocks) an unrelated FileStream.Open on the same path. That's a different, more invasive problem than the wrong constant — it looks like WalSharedMemory's current design (call fcntl/flock directly on the fd owned by its FileStream) may not be safe on macOS at all, regardless of which syscall is used.

Suspect: OS build

This machine runs Darwin 25.5.0 (arm64) — a very recent, likely pre-release build as of August 2026. Not yet confirmed whether this reproduces on a stable, publicly-released macOS version. If it doesn't, this may be specific to this beta build rather than a general macOS issue.

Failing tests (as of commit ccda67b)

• MultiProcessWalSharedMemoryTests.WriterLock_IsMutualExclusion_AcrossInstances
• MultiProcessWalSharedMemoryTests.BeginTransaction_Phase7_ReplaysCrossProcessCommits
• (one more in the same class — full list via dotnet test --filter FullyQualifiedName~MultiProcessWalSharedMemoryTests)

Practical impact

Only affects AllowMultiProcessAccess = true (opt-in, defaults to false). Not exercised by CI (ubuntu-latest only) — this is why the existing 5.0.6 release shipped despite the (pre-fix) failures never being caught.

Possible fix direction (not yet implemented)

Open a separate, raw native fd via open() (bypassing FileStream entirely) dedicated to holding the advisory lock, while keeping the existing FileStream for actual I/O on the .wal-shm file. Confirmed via probe that a raw open()-backed fd does not hit either the fcntl EINVAL or the flock/FileStream.Open conflict. This would require restructuring how WalSharedMemory opens its backing file on macOS, not just WalShmFcntl.cs.