fix(aws): Endpoint.none to avoid deadlock

[sam-goodwin]

Closes #1231.

alchemy deploy on an AWS stack could hang forever with zero output and no network activity. It is a self-deadlock, not credentials, S3, or the auth lock.

getAccountId runs STS GetCallerIdentity from inside the construction of AWSEnvironment. It already guards Region against the resulting cycle, but passed Layer.empty for the endpoint:

             Region.of(region),
-            endpoint ? Endpoint.of(endpoint) : Layer.empty,
+            endpoint ? Endpoint.of(endpoint) : Endpoint.none,

Layer.empty lets the call fall through to the ambient Endpoint.fromEnvironment — provideMerge'd by both AWS.state() and AWS.providers() — whose service Effect reads the endpoint off AWSEnvironment:

// distilled protocol.ts — every AWS op resolves this
const customEndpoint = yield* yield* Effect.serviceOption(Endpoint.Endpoint)

That re-enters the Effect.cached AWSEnvironment resolution already in flight on the same fiber, so the fiber awaits its own cache deferred forever: no timer, no socket, event loop idle in kevent. The cross-process auth lock is held for the whole hang, which is why the next run reports Timed out waiting for the alchemy auth lock … held for over 120s.

Trigger

method: "env" credentials (what exported AWS_* gives you) with no AWS_ACCOUNT_ID (so STS is needed) and no AWS_ENDPOINT_URL (so Layer.empty is taken). The stored-without-accountId backfill path hits it too. Not state-store specific — Endpoint.fromEnvironment is in AWS.providers() as well.

Verification

Probe stack with AWS.state(), profile configured { method: "env" }, dummy AWS_* credentials:

run	before	after
no AWS_ENDPOINT_URL	hangs indefinitely, 0 bytes out, 0 inet fds	fails in ~2.7s: The security token included in the request is invalid
AWS_ENDPOINT_URL set	fails in ~4s (took the Endpoint.of branch)	unchanged

The silence (issue's second ask)

A stalled deploy printed nothing at any log level, and .alchemy/log/out was empty too — nothing on the deploy → state_store.init → auth lock path logged at all.

• Debug lines for the state-store init and each auth-lock acquire/acquired/release.
• A stall notice every 30s naming the operation and lock path:

alchemy: AWS.read (profile 'prod') has been running for 30s while holding the
auth lock '/Users/me/.alchemy/lock/prod-AWS.lock'. Re-run with --log-level debug for detail.

Via Console.error, not the logger — with telemetry on, TelemetryLive replaces the console logger with the OTLP one, so log records never reach the terminal of the user watching it hang. Suppressed for login/configure, which legitimately block on the user and have a prompt on screen.

• --log-level now reaches the console. Telemetry-on runs sent every record to .alchemy/log/out and nothing to stderr, so --log-level debug changed nothing visible. A console sink is added when the flag is explicitly passed; with telemetry off the default logger is still in the set, so nothing double-prints.

No hard timeout on the locked effect: a slow-but-real SSO refresh or OAuth round-trip would be killed by one. The notice makes a stall visible without declaring it fatal.

Before / after on the same stalling run, telemetry on, no flags:

-(nothing, indefinitely)
+alchemy: AWS.read (profile 'probe') has been running for 30s while holding the auth lock '…'.

[alchemy-version-bot]

Install the packages built from this commit:

Alchemy

alchemy

bun add https://pkg.ing/alchemy/c5fe43a

@alchemy.run/better-auth

bun add https://pkg.ing/@alchemy.run/better-auth/c5fe43a

@alchemy.run/cloudflare-runtime

bun add https://pkg.ing/@alchemy.run/cloudflare-runtime/c5fe43a

@alchemy.run/frontend-frameworks

bun add https://pkg.ing/@alchemy.run/frontend-frameworks/c5fe43a

@alchemy.run/node-utils

bun add https://pkg.ing/@alchemy.run/node-utils/c5fe43a

@alchemy.run/pr-package

bun add https://pkg.ing/@alchemy.run/pr-package/c5fe43a

@alchemy.run/floci

bun add https://pkg.ing/@alchemy.run/floci/c5fe43a

Distilled

@distilled.cloud/core

bun add https://pkg.ing/@distilled.cloud/core/dd2a322

@distilled.cloud/aws

bun add https://pkg.ing/@distilled.cloud/aws/dd2a322

@distilled.cloud/axiom

bun add https://pkg.ing/@distilled.cloud/axiom/dd2a322

@distilled.cloud/cloudflare

bun add https://pkg.ing/@distilled.cloud/cloudflare/dd2a322

@distilled.cloud/hetzner

bun add https://pkg.ing/@distilled.cloud/hetzner/dd2a322

@distilled.cloud/neon

bun add https://pkg.ing/@distilled.cloud/neon/dd2a322

@distilled.cloud/planetscale

bun add https://pkg.ing/@distilled.cloud/planetscale/dd2a322

[arthtyagi]

Confirming this is still needed against the current published release, and adding an independent verification of the fix.

2.0.0-beta.74 (published 2026-08-21, current latest) still ships the unfixed branch, and so does main at e9fe594:

packages/alchemy/src/AWS/AuthProvider.ts:186
    endpoint ? Endpoint.of(endpoint) : Layer.empty,

Endpoint.none is absent from the published tarball in both src/AWS/Endpoint.ts and lib/AWS/Endpoint.js.

One detail from the beta.72 to beta.74 window is worth naming, because it makes the line look already handled. beta.72's getAccountId had no endpoint provision at all. beta.74's gained the endpoint parameter and the Endpoint.of(endpoint) branch. But the Layer.empty fallback is the branch every ordinary run takes, since a user with plain exported AWS_* credentials has no AWS_ENDPOINT_URL. So the deadlock is unchanged in practice across all three releases, while the diff reads as if it were addressed.

Independent verification, arrived at separately before finding this PR:

• Applied the semantically identical one-liner, Layer.succeed(Endpoint.Endpoint, Effect.succeed(undefined)) in place of Layer.empty, as a patch over 2.0.0-beta.72.
• Profile { method: "env" }, valid credentials in the environment, AWS_ACCOUNT_ID deliberately unset, no other workaround.
• Before: deploy produced zero bytes and never terminated.
• After: deploy of a stack using AWS.state() plus eight resources reached Done: 8 succeeded in about 60 seconds, destroy reached Done: 4 succeeded in about 71 seconds, and a sweep afterwards found no orphaned instances, queues, functions or roles, with the state bucket empty.

Two notes for anyone working around this before the merge:

1. The patch has to reach src/, not only lib/. Under bun the package resolves to src/AWS/AuthProvider.ts, so a lib/-only patch changes nothing and reads as though the fix did not work.
2. The silence is worth keeping in scope rather than splitting off. The stall emits nothing at --log-level trace either, and the records that do accumulate under .alchemy/log/out are not pointed to from anywhere a user would look while watching a hang. The console sink and the 30s stall notice in the second commit are what turn this from a multi-hour bisect into one line on screen.

Also relevant: #1106 describes the same self-await and proposes the same one-line fix, so this PR should close that issue too. It is currently linked only to #1231.

[arthtyagi]

This is still reproducible on 2.0.0-beta.74, the current latest on npm. I checked the published
tarball rather than the branch, in both the source and the build that actually ships:

x-2.0.0-beta.74/package/src/AWS/AuthProvider.ts:186
    endpoint ? Endpoint.of(endpoint) : Layer.empty,

x-2.0.0-beta.74/package/lib/AWS/AuthProvider.js:79
    endpoint ? Endpoint.of(endpoint) : Layer.empty)), Effect.flatMap((self) => self.Account

When no custom endpoint is configured, which is the default for anyone deploying to real AWS,
Layer.empty leaves the ambient Endpoint.fromEnvironment to resolve. That provider reads
env.endpoint off the Effect.cached AWSEnvironment effect that this very STS call is in the
middle of constructing, so the fiber waits on itself.

What makes this worth merging rather than working around: the comment two lines above the same call
site already documents this exact hazard, for the sibling service.

// Provide Region directly from the resolved inputs. Relying on the
// ambient Region provider (Region.fromEnvironment) here would
// deadlock: it derives the region from AWSEnvironment, which is the
// very service still being constructed by this STS call.
Region.of(region),

Region is pinned for precisely the reason Endpoint needs to be. Endpoint.none in this PR is
that same pin, which is why the change is small and why the failure is total rather than partial.

The symptom is bad out of proportion to the fix. There is no error, no timeout and no network
activity: alchemy deploy hangs with zero output and the event loop parked in kevent, which reads
as a credentials or connectivity problem rather than a resolution cycle. That is what #1231 was
opened about.

We have been carrying this exact change as a patchedDependencies patch against beta.72 in
production since it was diagnosed, applied to both src/AWS/AuthProvider.ts and
lib/AWS/AuthProvider.js, because bun resolves alchemy/* to src/*.ts while node resolves to
lib/*.js and the launcher re-execs whichever runtime invoked it. Patching only one entrypoint
leaves the other still hanging. No regressions since.

We also confirmed the deadlock is not a beta.72 artifact that later releases fixed by accident:
beta.73 and beta.74 both ship Layer.empty at this site, so upgrading is not a workaround.

Measured in our repository against the real AWSEnvironment, the real Endpoint.fromEnvironment
and the real Endpoint tag, with no network and no deploy:

ambient Endpoint only (unpatched shape):    DEADLOCK, timed out after 3008ms
Endpoint pinned at the call site (this PR):  completed in 1ms

Could this be merged? It is a two-line fix for a failure mode that is silent, total, and hits the
default path for every AWS user.

One small note on shape, take it or leave it: endpoint ? Endpoint.of(endpoint) : Endpoint.none is
strictly better than what we patched in, which pins unconditionally. Ours is fine on beta.72
because that call site has no endpoint parameter yet, but on beta.74 the conditional is what
keeps a LocalStack or floci account lookup from reaching real STS. We will align to yours when this
lands.