fix(middleman): resolve broadcast failures through the verifier

[miguel502]

A broadcast with no clean answer was written Failure with no hash: invisible to the verifier forever, though the tx may have landed.

sendTransaction now derives the hash locally (sha256 of the bytes it broadcasts, as matchTxInBlock does), so every outcome carries one.

It also stops flattening errors: dedup is success, a deterministic CheckTx rejection is rejected, a refused or unresolvable connection is neverSent (nothing reached the node), and everything else is unknown.

ExecuteTransaction anchors hash + heights BEFORE broadcasting; an unknown outcome stays pending for the chain to settle.

Unknown outcomes are retried, not parked. executeTransaction returns only a definitive answer: success, a CheckTx rejection, or a sdk code 32 (sequence already consumed, which a re-send cannot change and the verifier settles by hash or by the sequence rule). Anything else — node unreachable, connection dropped mid-request, mempool full — throws a retryable failure so Temporal re-broadcasts the same bytes, on a retry policy scoped to that one activity (5s → 30s backoff, 5 attempts, ~3.5 min worst case). This closes the gap for Soothe-signed transactions, which carry no timeoutHeight and would otherwise sit pending, keys held, until the signer's sequence was consumed by some other tx. Once the policy is spent, both flows catch the failure: the anchored flow records the last error on the row, and the legacy flow (pre-upgrade histories) anchors the row in that run — never failing the run, since a relaunched run with no hash would re-send as attempt 1 and could trust a rejection while the stake is on chain. An outage longer than the retry window still leaves a Soothe-signed row waiting on the sequence rule; an alert for pending rows with no timeout is a follow-up.

parseSignerAndSequence decoded hex payloads as base64 and returned nulls for every transaction, so no middleman transaction could reach a failure verdict; fixed and tested both ways.

The workflow bundle is now guarded by a test that builds it: a runtime import from the activities module would pull drizzle/pg/node:crypto into Temporal's sandbox and crash the worker at start-up, which nothing else in CI can see.

Closes #339

Not done, flagged for follow-up:

A Soothe-signed tx carries no timeoutHeight, so a broadcast that never reached a mempool can
only be failed once the signer's sequence is consumed by another tx — for an idle delegator,
never. The retries here shrink the window to ~3.5 min but do not close it. Alerting on those
rows (never auto-failing them) is #351.

[miguel502]

PR description — fixes #339

Summary

A broadcast that returned no clean answer was recorded as a permanent Failure with no hash. The
verifier sweep selects on status = pending AND hash IS NOT NULL AND executionHeight IS NOT NULL,
so such a row was never revisited: if the node had accepted the transaction before the connection
dropped, it still landed on chain while our database said it failed — forever, with no supplier
rows ever created for it.

The broadcaster no longer decides the outcome. It anchors the transaction (hash + heights), sends
it, retries while the node gives no answer, and lets the verifier settle it against the chain. Only
a rejection the node states deterministically, on the first attempt, short-circuits to Failure.

Design

1. The tx hash is derived locally. deriveTxHash (exported from @igniter/pocket) is
   sha256(txBytes) uppercased, the same derivation matchTxInBlock uses. The anchor and every
   non-success outcome carry it; non-hex input is rejected rather than hashed as truncated bytes.
2. The hash is persisted before broadcasting. persistBroadcastAnchor writes hash +
   executionHeight + timeoutHeight, then the bytes go out. Once anchored, a crash anywhere
   re-enters at the transaction.hash guard and hands the tx to the verifier instead of
   broadcasting twice. The unknown-outcome path never writes a status.
3. Only deterministic rejections are terminal, and only on attempt 1. sdk codes 20 (mempool
   full) and 32 (sequence mismatch) are not definitive: 32 also means the tx already landed.
   Codes are codespace-scoped, so a module error reusing those numbers is still a hard rejection.
   And a rejection is only trusted on attempt 1: a retry re-sends identical bytes, and if the earlier
   attempt landed the tx, the node answers about the world that tx created — insufficient funds now
   that the stake deducted the balance, or a consumed sequence. On a retry the row goes to the
   verifier.
4. Re-broadcast on re-entry is deliberately not adopted. It would be safe under the
   classification above (an in-mempool repeat answers code 19 → dedup success, a landed one answers
   code 32 → indeterminate), and the provider does it — but the provider can re-sign its own
   transactions and middleman cannot, since the wallet holds the key. Cost: a run that dies in the
   seconds between the anchor write and the broadcast does not re-send, so the verifier settles it
   Failure once coverage passes timeoutHeight (Keplr-signed txs; a Soothe-signed tx has no
   timeoutHeight and waits on the sequence rule). An unreachable node no longer opens this window,
   see "Broadcast retries". Adopting re-broadcast on re-entry is the natural follow-up, left out so
   the behavioural switch is made deliberately rather than as a side effect of this fix.
5. The dead if (!result) branch is gone from the new flow. It is retained verbatim inside
   legacyFlow, which must not change shape for Temporal replay.

Broadcast retries

A tx signed through Soothe carries no timeoutHeight (the Keplr path builds the body itself and
embeds one; the Soothe path hands the wallet only messages and gas). Without a timeout the verifier
can only fail an absent tx once the signer's sequence is consumed by some other tx — for an idle
delegator, never. So a broadcast that never reached a mempool must be re-sent, not parked:

• executeTransaction returns only an answer a re-send cannot improve on: success, a CheckTx
  rejection, or sdk code 32. Anything else — node unreachable, connection dropped mid-request,
  mempool full — throws a retryable ApplicationFailure (BroadcastOutcomeUnknown), so Temporal
  re-runs the activity with the same bytes. That activity has its own proxy with a 5s/2x/30s-cap
  backoff over 5 attempts (~3.5 min worst case); every other activity keeps the 30s/3 policy, and
  the activity's deterministic guards (not found, not signed) are non-retryable.
• sdk code 32 is returned rather than retried: a re-send cannot change a consumed sequence, and the
  verifier settles it by hash (it landed) or by the sequence rule (stale signature).
• When the policy is spent, the anchored flow catches the ActivityFailure — the activity's typed
  failure, or a TimeoutFailure when the last attempt hit startToClose, in which case the previous
  attempt's answer is kept — records the last error through the pending-guarded diagnostics write,
  and leaves the row to the verifier. The anchor is never rolled back: clearing a hash for a tx that
  might be in a mempool would hide it from the verifier, which is Broadcast failures mark transactions as permanently failed, with no hash and no way back #339 itself.
• The legacy flow (pre-upgrade histories) catches it too and anchors the row via
  persistBroadcastAnchor in that run. An earlier attempt may have landed the bytes (the upgrade
  itself kills in-flight activities), and a failed run would be relaunched hashless, re-sending as
  attempt 1 and able to trust a code-5 rejection while the stake is on chain — the key-release
  hazard described under "Also included". Replay-safe: no pre-upgrade history holds a command after
  a failed broadcast.
• sendTransaction classifies a refused or unresolvable connection (ECONNREFUSED / ENOTFOUND /
  EAI_AGAIN / EHOSTUNREACH / ENETUNREACH, walking undici's TypeError('fetch failed') cause chain
  and AggregateError members) as neverSent; ECONNRESET stays unknown since the request may have
  been written. The cosmjs TimeoutError branch is gone: broadcastTxSync never throws it.
• The attempt number defaults to null outside an activity context, so an unknown attempt can never
  turn a rejection into a Failure.
• Worst case per struggling tx is ~3.5 min, which also holds the dispatcher run (it awaits its
  children under a SKIP overlap policy); this only occurs while the node is unreachable, when no
  other transaction could be broadcast either. An outage longer than the retry window still leaves
  a Soothe-signed row waiting on the sequence rule; an alert for pending rows with no timeout is a
  follow-up.

The blocker that had to be fixed first

The anchor could not have worked as intended. parseSignerAndSequence decoded the signed payload
as base64; middleman stores it as hex. Because hex digits are valid base64 characters the
mistake never threw — it produced garbage bytes, TxRaw.decode failed, and the catch returned
{sequence: null, timeoutHeight: null} for every transaction since v0.13.0.

Both consumers were affected, so decideVerification computed
orderedRequiredCoverage = POSITIVE_INFINITY and its hash: absent → failure branch was
unreachable. No middleman transaction could reach a failure verdict. Success verdicts were
unaffected (a tx found on chain settles on the hash alone), which is why the stranded population is
only transactions that were broadcast and never landed — and why this surfaced as one user report
rather than a flood.

Measured against the real function with a real TxRaw:

hex payload parsed as base64 (production): {"sequence":null,"timeoutHeight":null}
same payload parsed as hex:                {"sequence":7,"timeoutHeight":1030}

Without this, the change would have moved the defect rather than removed it: from "wrongly marked
failed, quickly" to "stuck pending, silently, forever". It now has the test file it never had,
including a two-sided assertion that the encodings are not interchangeable.

Also included

• The rejection path claims the row through claimTerminalTransition's CAS before running any
  effect, and stands down if it loses. Anchoring pre-broadcast makes the row visible to the
  verifier while the broadcaster is still retrying, and the verifier can reach the opposite
  verdict (success on goal-state alone, when a sibling tx staked the same operator). Releasing the
  addresses afterwards is unrecoverable: the provider's markStaked requires state = Delivered,
  so a release landing first turns the verifier's success effect into a silent no-op and the key
  can be re-delivered to another delegator while the first is staked on-chain.

  This deliberately inverts the effects-before-CAS invariant documented on
  claimTerminalTransition, which assumes both racers reach the same decision. The
  expired-before-broadcast branch keeps the original order — its row has no hash, so no competing
  writer exists.

• The broadcast-rejection exit releases the provider's addresses. It was the only terminal exit
  that did not, unlike the expired path and the verifier's apply-failure.

• The "transaction is not pending" guard returns instead of throwing. A plain Error from
  workflow code does not fail the run — the SDK turns it into an unhandled rejection, failing the
  workflow task, which the server retries forever while the run sits RUNNING and
  ALLOW_DUPLICATE_FAILED_ONLY blocks a replacement. Pre-existing line; anchoring before the
  broadcast widened the window from microseconds to the whole broadcast phase.

• Deterministic handling for an unbroadcastable payload. The anchor returns null for an empty
  or non-hex payload and the workflow terminalizes it, instead of failing the run and letting the
  10s dispatcher relaunch it forever.

• patched('execute-transaction-command-sequence-v2'). The workflow's command sequence changed
  (activity inserted, one dropped, terminal branches reordered). Temporal replays history rather
  than resuming, and Igniter is upgraded by restarting the process, so a run open at that moment
  would mismatch, fail the workflow task, and retry forever — a silent permanent wedge needing a
  manual terminate. The gate keeps pre-upgrade runs on the old path. Those runs still get the
  substance of the fix, because activities are not version-pinned: the fixed executeTransaction
  returns a hash on every definitive outcome and the legacy flow anchors on an unknown one.

• The workflow bundle is guarded by a test that builds it. The failure type the workflow matches
  on lives in lib/broadcastOutcome.ts, a leaf module with no imports: importing a runtime value
  from the activities module pulls drizzle, pg and node:crypto into Temporal's sandboxed bundle
  and crashes the worker at start-up, which tsc and unit tests cannot see.

Correction to the issue text

The issue says the middleman never got the #308 broadcast/verify split. It got half — the
if (transaction.hash) handoff guard and the "verification moved to the verifier" docstring were
already there. What was missing was persisting the hash before broadcasting.

Tests

• packages/pocket/src/sendTransaction.test.ts — the error ladder, codespace scoping, never-sent
  classification with the real undici and axios shapes, the bounded cause walk, and the invariant
  the anchor rests on: the bytes hashed are the bytes broadcast, cross-checked against node:crypto.
• apps/middleman-workflows/src/workflows/ExecuteTransaction.test.ts — every outcome branch in both
  flows, ordering assertions via invocationCallOrder, the CAS-lost path, the retry-policy routing,
  the exhausted-retry and timed-out-attempt paths, and the legacy anchor-in-catch.
• apps/middleman-workflows/src/activities/executeTransaction.test.ts — the real activity functions
  against a mocked DAL/RPC: throw vs return per outcome, attempt handling, the anchor with a real
  TxRaw, the non-retryable guards.
• apps/middleman-workflows/src/lib/dal/transaction.test.ts — recordPendingDiagnostics rendered
  through the Postgres dialect, pinning the status = pending guard.
• apps/middleman-workflows/src/workflows/bundle.test.ts — builds the real Temporal workflow bundle.
• apps/middleman-workflows/src/activities/parseSignerAndSequence.test.ts — the hex/base64
  regression, both directions.