Error

[posthog]

Sync push delivery looks broken: all 3 live subscription(s) have never received a notification. Check that the reverse proxy routes /sync/* to this service.

View in PostHog

[claude]

Triage: Ops/transient — reverse proxy misroute, not a code bug

Diagnosis. This is the subscriptions.neverNotified guard added in #2841
firing exactly as designed. The message text is a literal match for the log
line in packages/sync/src/app.ts:

Sync push delivery looks broken: all N live subscription(s) have never
received a notification. Check that the reverse proxy routes /sync/* to
this service.

That guard only fires when neverNotified >= live across all live
subscriptions — i.e. every currently-active Google push channel has gone its
entire life without a single delivered notification. #2841's own commit
message documents the only known cause of that shape: the reverse proxy
(Caddy) stopped routing /sync/* to the Sync service, which silently
degraded push delivery to the ~10-minute reconcile-sweep fallback for nine
days in prod before this guard existed. docs/self-hosting/server-guide.md
was updated in that PR to warn that the Caddyfile must be edited live, never
restored from a backup copy — that's how the block was lost the first time.

Not a code bug. The code is doing exactly what it's supposed to do:
detect a bad external routing state and say so loudly instead of degrading
silently. A code change here would be the wrong response.

What I could not confirm. I was unable to pull PostHog event properties
(environment, occurrence count, first/last seen) for this fingerprint from
this run — the PostHog MCP tools weren't available/responsive in this
session, and a direct API fallback returned unfiltered fixture-like data, so
I can't reliably state whether this is staging or production. Given
staging and prod currently share one PostHog project, please check the
PostHog link in the issue body directly to confirm which environment this
is before treating it as urgent.

Suggested operator action.

1. Open the PostHog link in the issue body and check the environment
   property on the underlying event.
2. If production: SSH to the host and check the live Caddyfile for the
   /sync/* block (edit it live — do not restore from a backup, per
   docs/self-hosting/server-guide.md). Confirm with the new
   sync-webhook-route deploy-health-check (.github/scripts/deploy-health-check.sh).
3. If staging: lower urgency, but the same proxy check applies.

No PR opened — this is routing/ops, not application code.

[tyler-dane]

Verdict: resolved on production, still broken on staging. Leaving open, scoped to staging.

This is the subscriptions.neverNotified guard from #2841, not an application defect. Confirming that, and correcting one thing about why this exact fingerprint went quiet.

The fingerprint went silent because the wording changed, not because the condition cleared. #2844 rewrote the message and added a one-hour hold-off. So "have never received a notification" was replaced by "have gone without a single notification for over an hour", and both strings have to be read together:

env	message	count	first seen	last seen
staging	all 3 ... never received a notification	5	2026-08-23T15:44:00.911Z	2026-08-23T15:59:42.279Z
production	all 354 ... never received a notification	2	2026-08-23T15:57:36.011Z	2026-08-23T16:02:43.856Z
production	all 355 ... never received a notification	22	2026-08-23T16:07:31.545Z	2026-08-23T17:53:19.741Z
production	all 355 ... without a single notification for over an hour	1	2026-08-23T18:52:01.137Z	2026-08-23T18:52:01.137Z
staging	all 3 ... without a single notification for over an hour	17	2026-08-23T16:59:08.334Z	2026-08-27T03:54:40.207Z

Production: healthy, close-worthy. Last push-delivery alert of any wording was 2026-08-23T18:52Z, over four days ago, across every deploy since (v1.1.263 through v1.1.319). That silence is real evidence rather than a suppressed alert, because the guard re-arms on restart: pushSilenceStreak resets to 0 when the process restarts, and prod has restarted repeatedly with long steady-state windows in between, each of which would have produced a fresh alert an hour later if /sync/* were still unrouted.

Staging: still broken. 17 firings of the hardened guard, the most recent 2026-08-27T03:54:40Z on v1.1.313. Note the guard fires once per silence streak, not once per sample:

if (pushSilenceStreak === PUSH_SILENCE_ALERT_AFTER_SAMPLES) { logger.error(...) }

Strict equality against 12 samples at a 5-minute cadence. So after it fires once it stays quiet until the streak breaks and rebuilds, which means the 14-hour gap since 03:54Z is the expected shape of a persistent fault, not evidence of repair. Staging restarted into v1.1.315 at roughly 17:19Z, under an hour before this was written, so it has not yet had time to re-fire.

Operator action, staging only. Same cause as the production outage: the reverse proxy in front of staging is not routing /sync/* to the sync service, so Google's push notifications never reach it. Check the staging host's Caddyfile for the /sync/* route, per docs/self-hosting/server-guide.md. The production Caddyfile is unmanaged and was edited live; the staging one needs the same treatment and the same check.

No application code change. Keeping autofix:needs-human since the remaining half needs a host touched by hand.

[tyler-dane]

Correction: the staging proxy is fine. The alert's own remediation text sent us the wrong way.

I said above that the operator action was the staging Caddyfile /sync/* route. I checked before changing anything, and that is wrong.

The route is correct on staging. staging.compasscalendar.com (host staging-blue), /etc/caddy/Caddyfile:

handle /sync/* {
 reverse_proxy 127.0.0.1:3010
}

And the project's own sync-webhook-route probe passes against it, identically to production:

probe	staging	production	wanted
POST /sync/notifications/google	400, non-html	400, non-html	400, non-html
GET /sync/google	302, text/plain	302, text/plain	non-html

Worth recording the trap: a GET on /sync/notifications/google returns 404 text/html on a healthy host, because the webhook is POST-only and Express answers its own html 404. That looks exactly like the misroute being tested for. validate_sync_webhook_route uses POST for precisely this reason. My first probe used GET and would have "confirmed" a fault on both hosts.

The actual cause on staging is credentials. From compass-sync-1:

Sync job subscriptionMaintain (…) dropped: credential unusable
  (missingRefreshToken) for connection 6a724e3eba62890abd8ed875; sync resumes on reconnect
Sync job subscriptionMaintain (…) dropped: credential unusable
  (missingRefreshToken) for connection 6a7250beba62890abd8eee77; sync resumes on reconnect

Two staging connections hold a Google credential with no refresh token. subscriptionMaintain is dropped for both, so their channels are never established or renewed, so they can never be notified, so neverNotified >= live holds indefinitely and the guard fires once per restart forever.

Also present, and unrelated to push, the #2841 hold-off doing its job on two calendars whose cursors keep expiring:

Sync job incrementalPull (…) dropped: calendar 6a7e8a36286eab0893793f58
  cursor keeps expiring; holding off pulls until 2026-08-27T19:04:04.413Z

Why the alert misleads. The message hard-codes "Check that the reverse proxy routes /sync/* to this service" because that was the August production cause. It is emitted from neverNotified >= live alone and knows nothing about the proxy. On a small fleet whose connections cannot hold a subscription, it names an innocent component.

Next action, and why I stopped. The remedy is to reconnect those two Google accounts on staging, or delete the dead connections. Reconnecting is an interactive Google OAuth sign-in, and deleting connections is destructive. Neither is a config edit and neither is mine to do, so I made no change to the host. Everything I ran was read-only.

Possible follow-up worth considering separately: the guard could name what it actually observed rather than a remembered cause, e.g. mention credential-unusable connections when the unnotified subscriptions belong to them. That would have saved this round trip. Not filing it as part of this issue.

Leaving open with autofix:needs-human, now scoped to the staging Google credentials rather than the proxy.