Postgres replication silently stops (no reconnect, no error) after a half-open source connection; health probe stays green

[embune99]

Summary

When the connection between the replication worker and the source Postgres becomes half-open (e.g. a DB restart / failover / network drop that leaves the TCP connection hung with no RST/FIN), the WAL streaming read blocks indefinitely without throwing. As a result:

• PowerSync never reconnects — the replication stream is dead but no retry is attempted.
• The process keeps running and the liveness probe stays green ("healthy"), so nothing (k8s, alerting) notices.
• replication_lag stays 0 (it only reflects received messages), so it is also blind.
• The only way to recover is to restart the replication service.

This matches a production incident where sync silently stopped after a same-day DB restart and only a manual restart restored it.

Environment

• journeyapps/powersync-service v1.21.0 (Open Edition). Behaviour also reviewed against main.
• Source: PostgreSQL (AWS RDS in production).
• Run mode: dedicated replication worker (start -r sync).
• Storage: Postgres.

Steps to reproduce

Minimal docker stack: postgres (source, wal_level=logical) + storage Postgres + powersync-service:1.21.0 replicating one table.

1. Start the stack; confirm replication is healthy (rows replicate, slot active=t, behind≈0).
2. Black-hole all traffic between the PowerSync container and the source Postgres without sending RST (simulates a DB restart / failover / network partition):

   # inside the powersync container's network namespace
   iptables -A OUTPUT -d <source_ip> -j DROP
   iptables -A INPUT  -s <source_ip> -j DROP

   (Using DROP, not REJECT, is important — REJECT would send an RST and PowerSync would error & reconnect. The silent failure needs the no-RST case.)
3. INSERT a row into the replicated table on the source.

Observed

• The new row is never replicated. rows_replicated does not increase.
• No fatal error, no reconnect attempt. The main WAL read just hangs.
• Liveness/health probe stays PASS (green) the whole time.
• replication_lag stays 0.
• The only non-fatal noise is a few warn lines from the periodic keepAlive() side-query hitting a TCP retransmit timeout:

  warn  Query error, retrying postgres query failed
  warn  [<slot>] KeepAlive failed, unable to post to WAL postgres query failed
  

  These are caught and retried forever; the replication connection is never recreated.
• Source-side pg_replication_slots shows the truth: confirmed_flush_lsn frozen, active flips t→f, lag grows.

Recovery

Restarting the replication service reconnects, replays the missed rows, and returns to healthy.

Expected behaviour

PowerSync should not be able to silently sit on a dead replication stream while reporting healthy. At minimum one of:

1. A read/socket timeout on the WAL streaming read (or tcp_user_timeout / TCP keepalive on the replication connection) so a half-open connection surfaces as an error → triggers the existing retry/reconnect path.
2. A liveness signal tied to replication progress (e.g. confirmed-flush / received-LSN advancing or a keepalive round-trip), instead of the run-loop touching the probe unconditionally — so the health check fails when the stream is actually dead.

Possible cause (pointers into the source)

Line numbers are against main at the time of writing; the behaviour was observed on v1.21.0.

1. The WAL read blocks here and never throws —
   WalStream.ts L709:

   for await (const chunk of replicationStream.pgoutputDecode()) {
     this.touch();
     ...

   When the source connection is half-open, this for await simply never yields another chunk and never rejects, so control stays inside the loop forever.

2. So the retry path is never reached —
   WalStreamReplicationJob.ts L45-L88:
   replicate() → await this.replicateOnce() (L46); the catch that calls this.rateLimiter.reportError(e) (L77) and the finally { this.abortController.abort() } (L88) only run if the read throws — which it doesn't here. No error → no reportError → no reconnect.

3. Meanwhile the liveness probe is touched unconditionally —
   AbstractReplicator.ts L99-L149:
   the runLoop calls await container.probes.touch(); (L108) on every iteration, independent of whether the WAL stream is progressing; the only failure it catches is refresh() throwing (L120-L121, logged as Failed to refresh replication jobs). So the probe reflects "the loop is spinning", not "data is flowing" → stays green while the stream is dead.

Note: there is also keepalive-handling logic in WalStream.ts (around L528) that may be relevant to how RDS keepalives interact with a hung stream — worth a maintainer's eye, but I couldn't pin an exact root-cause line there.

Impact

Silent, undetected sync outage. No alert fires because every built-in signal (probe, replication_lag) reports healthy; users hit stale data until someone manually restarts the service.

[rkistner]

@embune99 Thanks for the report and analysis.

TCP keepalives are configured on these connections by default, which should help to some extent, although much later than what we'd like. Unfortunately NodeJS 24.x does not allow configuring TCP_KEEPINTVL or TCP_KEEPCNT.

1. How long did you wait before restarting the process? I'd expect that replication should eventually fail with read ETIMEDOUT. On my test machine it happens after around 8m30s.
2. Could you please check what these values are on the machine where you're seeing the issue?

sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes net.ipv4.tcp_retries2

Example output:

net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_retries2 = 15

Note that we set TCP_KEEPALIVE_TIME to 40 in the app, but it's still useful to see the original value.

[embune99]

@rkistner Thanks for the detailed explanation — that clarifies a lot.

Q1 — how long we waited / does it recover:
Originally we restarted within ~2 minutes, so we never reached the keepalive timeout — which is why it looked like it "never" recovered. I re-ran the repro (PowerSync v1.21.0, Node v24.15.0) and let it sit this time:

• Source black-holed with iptables -j DROP (no RST) at T+0.
• The main replication read hung silently until ~T+16 min, when it finally threw read ETIMEDOUT (errno -110, at TCP.onStreamRead) → Replication error.
• After I then restored connectivity (no process restart), it reconnected and replayed the missed rows within ~15s.

So you're right — it does self-heal; it's "recovers after ~16 min", not "never". (Slightly longer than your ~12 min, presumably due to pending-data/RTO behaviour on top of the keepalive.)

Q2 — keepalive sysctls (read inside the powersync-replication pod on our AWS EKS node):

net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9

With the app's keepidle=40 plus the un-overridable intvl=75 × probes=9, the dead socket should surface at ~40 + 675 ≈ 715s (~12 min), consistent with your number; our observed ~16 min is likely the pending-data path on top.

On production impact: even though it does eventually self-recover, a ~12–16 minute window of silently-stalled replication — during which the process still reports healthy and replication_lag stays 0, so no probe or alert fires — is fairly significant for a production system. Is this timeout something you'd consider tightening — e.g. an application-level read/progress timeout on the replication stream (independent of the OS TCP keepalive that can't be fully configured on Node 24), or reflecting the stalled state in the health check — so it recovers in seconds rather than minutes without an operator having to restart manually?

[rkistner]

Thanks for those details, that helps.

I agree that this is way too long for silently stalled replication, and we'll need to look into options to handle this. The main options:

1. Set TCP_USER_TIMEOUT, TCP_KEEPINTVL and TCP_KEEPCNT to lower values. That handles everything on the socket layer, which I think is the best place to detect these issues. It does require using additional low-level calls using a package such as (net-keepalive)[https://www.npmjs.com/package/net-keepalive].
2. Add application-level checks. Does not require low-level system calls, but may still need driver changes to implement properly. For example, on the application level we may not be able to reliably detect the difference between receiving no data, versus receiving a large replication message over a slow connection.

We'll also have to check whether this affects database connections other than Postgres.

In the meantime, two places you can use to detect this:

1. The diagnostics API will give you last_keepalive_ts and last_checkpoint_ts. If these fall behind, it indicates an issue. However, it is only updated once a minute on idle connections, so it won't give you fast detection of the issue.
2. Like you mentioned, monitoring pg_replication_slots.active on the source database can highlight the issue. This would only reflect after the server detects the dropped connection - I'm not sure how long that would take.

Unfortunately, neither of those are good candidates as a health check for the specific process, but it can be used to alert on issues after say 5 minutes.

[rkistner]

Out of interest, do you know what caused this in your production environment? Personally I've seen similar issues in two cases:

1. IPv6 on old AWS m5 instances randomly dropping connections after a while, resolved by using newer instance families. This did produce connection timeouts after around 5 minutes.
2. Running too many pods (160+) on a single Kubernetes node occasionally had symptoms like this, triggered on all the pods on the Node at the same time.

[embune99]

@rkistner Thanks, that all makes sense — and agreed that the socket layer (option 1) is the cleaner place to catch this.

On detection in the meantime: that's exactly the direction we're taking. We're adding a small watchdog that:

• reads last_keepalive_ts — via the diagnostics API, thanks for pointing that out; much better than reading the internal storage table directly — and
• cross-checks pg_replication_slots.active / lag on the source,

then alerts and restarts the replication pod once those stay stale for a few minutes. We measured the keepalive cadence at ~60s on our instance (matching your "once a minute on idle"), so we'll set the staleness threshold to a few multiples above that. Agreed neither is a true process-level liveness probe — we're treating it strictly as an external watchdog/alert.

One related suggestion: that ~60s cadence — and the resulting "once a minute" detection floor you mentioned — comes from the hardcoded PING_INTERVAL (1_000_000_000n * 60n) in AbstractReplicator. Would you consider making it configurable? A shorter interval would let last_keepalive_ts-based detection react in seconds instead of being capped at ~1 minute, which would help as an interim measure until the socket-layer fix lands.

On the cause (your second comment): in our case it correlated with an RDS (source DB) restart/maintenance event the same day, which we believe left the connection half-open. Neither of your two cases seems to match our setup:

• We're on a newer instance family than m5, so probably not the IPv6/m5 issue.
• Pod density per node is moderate — well below the 160+ you mentioned.

So for us it points at the DB-side restart rather than node/instance networking. Happy to share more if it's useful.