`connect()` after `disconnectAndClear()` silently never starts the sync client (native/Android); all subsequent connect/disconnect calls hang too

[daimos123]

Summary

During an account-switch flow (sign out, then sign in with a different user), we call disconnectAndClear() followed by connect() with a fresh connector.
On one real-device occurrence, connect() never started the sync client: no Starting Rust sync iteration log, no error, no isolate exit — total silence. Every subsequent disconnect() / connect() call (triggered by further auth transitions) also produced no sync activity, consistent with them queueing forever on syncConnectMutex behind the stuck connect().
Only killing and restarting the app recovered. The same flow works fine most of the time — this looks like a race.

Environment

• powersync 2.2.0 (Flutter, Rust sync client, Sync Streams)
• Android, real device (Samsung, arm64), debug build
• Supabase auth/backend, PowerSyncBackendConnector recreated on each connect()
• Attachment queue (@experimental attachments API) active with 62 synced attachments, periodic sync every 30 s
• We diffed connection_manager.dart / native_powersync_database.dart between 2.2.0 and 2.3.2: the connect/disconnect/mutex mechanics are
  identical, so we believe 2.3.2 is affected as well.

Timeline (from logcat, single app process)

Context: for ~20 minutes before the incident the device had no DNS (Failed host lookup), with the Rust client retrying every 5 s — the sync recovered right before the account switch (checkpoint applied).

21:32:03  Starting Rust sync iteration
21:32:08  Validated and applied checkpoint          (old account, recovery after DNS outage)
21:32:10  supabase signedOut -> our code calls db.disconnect()
21:32:10  Ending Rust sync iteration. Immediate restart: false
21:32:11  Sync Isolate exit
21:32:30  supabase signedIn (DIFFERENT user id) -> our code calls:
            await db.disconnectAndClear();          // completes: local db cleared
            await db.connect(connector: fresh);     // <- never starts the sync client
21:32:30  Found 0 active attachments               (attachment queue reacting to the cleared table)
—— from here on: NOTHING from the sync client, forever ——
21:45:16  signedOut -> db.disconnect()              // no "Ending ...", no "Isolate exit": hangs silently
21:45:38  signedIn (same user) -> disconnectAndClear() + connect()  // silent too

The attachment queue's periodic timer kept running normally the whole time (so the main isolate event loop was healthy). db.currentStatus stayed
frozen at the state set by disconnectAndClear() (connected: false, connecting: false, hasSynced: false) — that is how our UI-level watchdog
now detects the condition.

What we could rule out

• disconnectAndClear() completed (the cleared attachments table triggered watch queries; our code after it ran — the new owner id was persisted).
• The previous sync isolate exited cleanly 19 s before the failing connect() (Sync Isolate exit logged at 21:32:11).
• No exception surfaced from connect() into our try/catch around the auth-transition handler (or it was swallowed — see next point): nothing
  was reported through our error pipeline either.
• The app was NOT hot-restarted between those events (single process).

Candidate spots we noticed while reading 2.2.0 sources

We could not pin down the exact mechanism, but while reading connection_manager.dart / native_powersync_database.dart / sync_isolate_protocol.dart we noticed a few places that could produce a silent never-completing connect():

1. ConnectionManager.connect() holds syncConnectMutex across connectInternal(), which awaits hasInitPort.future. If the freshly spawned isolate never sends init, connect() never returns and every later connect()/disconnect() queues on the mutex forever — matching the observed "everything silent from this point on".
2. _abortCurrentSync()'s else branch (disconnector.aborted == true) awaits onCompletion but does not reset _abortActiveSync to null; the following assert(_abortActiveSync == null) in connect() would throw in debug builds (an AssertionError escaping through the caller rather than a clean state).
3. _RemoteMutex.lock() ignores its abortTrigger parameter, so a sync isolate stuck acquiring the remote sync/crud mutex cannot be aborted: openedStreamingSync?.abort() would hang in shutdown(), the isolate never exits, and disconnect() then hangs in waitForShutdown() while holding syncConnectMutex.
4. Mutex grant/release messages (receiveMessages) and the exit notification (receiveExit) arrive on different ports with no ordering guarantee; handleChildIsolateExit() completing held grants relies on those orderings being benign.

Given the DNS-outage recovery, an abort 2 s after a checkpoint, and an attachment queue writing concurrently, a race between those pieces seems
plausible.

Repro

Not deterministic — the same account-switch flow passed twice under instrumentation the previous day. Happy to run an instrumented build or
provide the full logcat if that helps.

Workaround we ship meanwhile

App-level watchdog: after asking for a reconnection, if the status stream stays frozen with no sign of life (connected/connecting/downloading/
errors all absent) for 20 s while the first sync is pending, we tell the user to restart the app.

[daimos123]

Second occurrence, now with instrumentation — connect() confirmed to never return

We hit the same hang again on the same device, this time with debug traces around every step of our account-switch flow. Timeline (single process):

22:26:15  last sync iteration before the app went to background
          (app backgrounded / idle for ~28 minutes)
22:54:27  app resumed, sync iterations resumed normally
22:54:36  signedOut -> db.disconnect() -> "Sync Isolate exit" (clean)
22:54:50  signedIn (different user) -> disconnectAndClear() completed
          -> our trace right before db.connect()
          [nothing further from the sync client, ever]
22:55:00  our 10-second watchdog fires: connect() still has not returned

Key data points:

• db.connect() never returns and never throws — we log right after the await and have a catch-all around it; neither fired.
• No Starting Rust sync iteration, no isolate exit or error — the sync client never starts.
• Reproduction pattern (2 out of 2 occurrences now): app in background for ~30 minutes -> resume -> sign out -> sign in with a different user (our flow then runs disconnectAndClear() + connect()) -> hang. Rapid account switches without the long background period do NOT reproduce (6+ clean switches the same day).
• Device: Samsung (Android), debug build — aggressive OEM background management may be relevant given the "app slept first" precondition.

Restarting the app recovers every time. Happy to test a patched build.

[simolus3]

When the app is resumed, after half an hour of being backgrounded, I assume that's a new process? (As in, main() runs again?)

Given that connect() is stuck, I've added many more log lines to that method on the debug-connect-hang branch. Could you try depending on the PowerSync SDK from that branch and report logs when you run into this again?

dependency_overrides:
  powersync:
    git:
      url: https://github.com/powersync-ja/powersync.dart.git
      ref: debug-connect-hang
      path: packages/powersync

I hope that at least provides a better clue. It also adds a timeout when the sync isolate requests a mutex from the main isolate to debug deadlocks.

---

ConnectionManager.connect() holds syncConnectMutex across connectInternal(), which awaits hasInitPort.future. If the freshly spawned isolate never sends init, connect() never returns and every later connect()/disconnect() queues on the mutex forever — matching the observed "everything silent from this point on".

Assuming that the isolate doesn't enter a deadlock, the only way for that to happen would be for the isolate to crash. Do you see any Sync Isolate error logs we'd log in receiveUnhandledErrors.listen?

_abortCurrentSync()'s else branch (disconnector.aborted == true) awaits onCompletion but does not reset _abortActiveSync to null; the following assert(_abortActiveSync == null) in connect() would throw in debug builds (an AssertionError escaping through the caller rather than a clean state).

The else branch indeed looks fishy, considering that calling abort() on a controller that has already been aborted doesn't do anything. The comment about races also looks outdated given that the whole thing is wrapped in a critical section. I'll fix that, but I think that's harmless.

_RemoteMutex.lock() ignores its abortTrigger parameter, so a sync isolate stuck acquiring the remote sync/crud mutex cannot be aborted: openedStreamingSync?.abort() would hang in shutdown(), the isolate never exits, and disconnect() then hangs in waitForShutdown() while holding syncConnectMutex.

Right, but Sync Isolate exit was logged which means that mutexServer.handleChildIsolateExit() was called and mutexes potentially still held by the prior isolate would have been unlocked.

Mutex grant/release messages (receiveMessages) and the exit notification (receiveExit) arrive on different ports with no ordering guarantee; handleChildIsolateExit() completing held grants relies on those orderings being benign.

Do you mean acquireRequest being called after handleChildIsolateExit because we receive a request on a different port? There's an assert(!_didExit) in acquireRequest, so that should also throw an assertion error.

[daimos123]

Same process, not a cold start: on resume the sync iterations picked up normally without any of our boot-time traces firing (they log on every main()).

No Sync Isolate error at any point — the only isolate log ever seen is the clean Sync Isolate exit at sign-out, then total silence after disconnectAndClear().

We've switched our dev builds to debug-connect-hang via dependency_overrides and will report the enhanced logs at the next occurrence. Heads-up: the repro needs the ~30 min background precondition, so it may take a few days to trigger naturally.

[daimos123]

Reproduced today on the debug-connect-hang branch — here are the logs you asked for.

Key finding: connect() runs to completion and the isolate is spawned — it just never starts its first sync iteration. No error, no isolate exit, nothing. That narrows it down compared to my earlier reports, where I could only say "connect() is silent".

Logs (instrumented branch, resolved-ref 9bbbe1f)

Failing sequence — logout, login as a different user, disconnectAndClear(), then connect():

21:36:13.287  supabase.auth: Signing out user with scope: SignOutScope.local
21:36:13.297  App: auth transition signedOut -> disconnect (uid: null)
21:36:13.299  [PowerSync] INFO: aborting previous sync on connect call
21:36:13.301  [PowerSync] INFO: waitForShutdown called. tried spawning? true
21:36:13.312  [PowerSync] FINE: Sync Isolate exit
21:36:13.312  [PowerSync] INFO: cleanup complete
21:36:13.372  App: disconnect done

21:36:22.465  App: auth transition signedIn -> reconnect (uid: cae349fc-…)
21:36:22.465  App: local DB purge (owner: 130e790c-… -> cae349fc-…)   <- disconnectAndClear()
21:36:22.720  App: identity guard passed, reconnect requested
21:36:22.720  [PowerSync] INFO: connect() called
21:36:22.720  [PowerSync] INFO: trying to acquire sync connect mutex
21:36:22.721  [PowerSync] INFO: has mutex, reconnecting...
21:36:22.721  [PowerSync] INFO: connectWithSyncLock() called
21:36:22.721  [PowerSync] INFO: spawning sync isolate
21:36:22.728  [PowerSync] INFO: sync isolate spawned
                                    <- nothing else, ever
21:36:32.722  App: ALERT connect() silent after 10 s

Nothing further was logged for the remaining 99 seconds of the capture (buffer ends 21:38:01): no Starting Rust sync iteration, no credentials fetch, no Sync Isolate exit, no exception. Only an app restart recovers.

Healthy connect() from the same log file, for contrast — the first iteration follows the spawn by 91 ms:

20:06:07.234  [PowerSync] INFO: spawning sync isolate
20:06:07.239  [PowerSync] INFO: sync isolate spawned
20:06:07.330  [PowerSync] INFO: Starting Rust sync iteration

Preconditions that were present this time

The two earlier occurrences and this one all share "app backgrounded ~30-40 min, then account switch". What the instrumented log adds is what the sync client was doing at the moment of the disconnect — it was not idle, it was in an error retry loop:

20:52:52  [PowerSync] WARNING: Sync error: Sync service error
          ClientException with SocketException: Failed host lookup:
          '…powersync.journeyapps.com' (No address associated with hostname, errno = 7)
          (device lost network while backgrounded, retried every ~5 s)

21:36:03  [PowerSync] WARNING: Sync error: Authorization error   <- token expired
21:36:08  [PowerSync] WARNING: Sync error: Authorization error       (retry loop, ~5 s apart)
21:36:13  logout -> abort / waitForShutdown / Sync Isolate exit / cleanup complete

So the shutdown path did report a clean exit while an errored iteration was in flight. If it helps, my working hypothesis is a race between the aborted iteration's teardown and the next spawn, where the new isolate ends up waiting on something the previous one owned — but the isolate side is silent, so I can't see past the spawn from here.

Environment

• powersync from debug-connect-hang, resolved-ref 9bbbe1f389a3dd9a0043d35d80dd9d5d3c04d744 (reports version 2.3.2)
• Flutter 3.44.1 stable, Dart 3.x
• Samsung SM-S918B, Android 16 — physical device, debug build
• App: Supabase auth, disconnectAndClear() on account switch (local DB has a single owner by design)

Happy to add more instrumentation on the isolate side if you push anything to the branch — the repro takes ~40 min of wall time but it's now reliable enough that I can run it again on demand.

[simolus3]

Healthy connect() from the same log file, for contrast — the first iteration follows the spawn by 91 ms:

Did you omit the has init port message there or was that actually not shown in both cases?

We've recently fixed another issue that could cause this (#452), I have updated the debug-connect-hang branch (ref is now 022ffb9c195512a7df9f19b92e4c5c11d3a8e97f). Are you using dio in your connector? It's possible that sending the error itself might have caused the issue.

[daimos123]

Did you omit the has init port message there or was that actually not shown in both cases?

Sorry — that was over-trimming on my side. In the full log file, both healthy connect() occurrences do log it:

20:06:07.234938: spawning sync isolate
20:06:07.239026: sync isolate spawned
20:06:07.254481: has init port          <- +15 ms after spawn

20:44:36.808486: spawning sync isolate
20:44:36.812061: sync isolate spawned
20:44:36.882031: has init port          <- +70 ms after spawn

The stuck occurrence (immediately after disconnectAndClear() during an account switch) is the only one where it never arrives:

21:36:22.721251: spawning sync isolate
21:36:22.728810: sync isolate spawned
(nothing further from PowerSync — our app-side 10 s watchdog fires at 21:36:32)

So: has init port present in every healthy connect, never sent in the stuck one — the isolate spawns but never gets as far as sending init.

Also, to answer your earlier question: there is no Sync Isolate error line anywhere in the file, so receiveUnhandledErrors.listen never fired.

Are you using dio in your connector?

No — no dio anywhere in the dependency tree (checked pubspec.lock, direct and transitive). Our connector's fetchCredentials()/uploadData() go through supabase_flutter, which uses package:http internally. If #452's failure mode is an unsendable error object crossing the isolate boundary, another exception type could conceivably take the same path — the stuck occurrence happened right after disconnectAndClear() during a user switch, a moment where a transient failure in the connector wouldn't be surprising.

I've updated our override to the new pinned ref (022ffb9c) and will post the instrumented logs at the next occurrence. If it stops reproducing now that the #452 fix is on board, I'll report that too.