Skip to content

Add the human verification step with an embedded live view - #40

Merged
akxue merged 10 commits into
mainfrom
hypeship/ma-captcha-hitl
Oct 30, 2026
Merged

akxue merged 10 commits into
mainfrom
hypeship/ma-captcha-hitl

Conversation

@akxue

@akxue akxue commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

context

managed auth logins can now pause on a captcha the browser can't solve (AWAITING_HUMAN_INTERVENTION) when the connection has hitl on. the component needs a screen where the person completes the check in the login browser and tells us they're done.

changes

  • new awaiting_human_intervention UI state and HumanInterventionStep:
    • centered "Human verification required" heading, then the login browser's live view embedded in a 16:9 frame (the stream is the full 1920x1080 display, so other ratios letterbox). the frame border is an outline so it doesn't shrink the 16:9 box.
    • "Complete the check in {site} above, then continue. We'll continue signing you in after." next to a Continue button, which submits the human_intervention choice.
    • the live view sits in a LiveViewFrame that follows the live view's postMessage protocol: if no video plays within 60s it reloads the frame up to twice, then says to reload the page. a network failure gets one "Try again"; a browser that can't play the stream gets a message and no button.
    • the step stays mounted while continue is in flight, so the live view isn't torn down; a failed submit returns to whatever screen the flow state says, not a hard-coded form.
    • from the second pause on, the choice's description shows as a note ("{site} is asking to verify again.").
    • the card widens to 64rem on this screen only.
  • the human_intervention choice is kept out of sign_in_options and submitted through the existing sign-in choice path.
  • the captcha_blocked error screen shows the API's error_message when there is one, so a timeout reads "a person was asked to complete it but didn't finish in time", and falls back to "A CAPTCHA blocked the login." when there isn't.
  • types, localization keys, appearance slots (humanInterventionNote, humanInterventionLiveView, humanInterventionMessage), demo states and a changeset. the changeset notes that sites with a CSP need the live view origin in frame-src.

alternatives considered

  • opening the live view in a new tab: dropped in favor of embedding, so the person stays in the flow.
  • a left-aligned header with an icon, or a site tile like the consent screen: the site tile pushed the continue button below the fold on a 1440x900 laptop. the centered title-only heading matches the other screens and keeps the browser large.

testing

head is 221ea9f.

  • bun test passes (100), including hook tests for the state, the submitted body and the failed-submit path, a component test that Continue stays disabled after an accepted submit until the flow moves on, LiveViewFrame tests (one retry for network failures, none for unsupported browsers, other origins ignored, silent stalls reloaded then reported), the error screen description (the server's message when there is one, a captcha fallback when there isn't), and a fresh live view when its url changes. the package builds, the demo typechecks, and prettier is clean.
  • @onkernel/sdk (dev dependency, used only by the contract test) is bumped from 0.93.0 to 0.138.0, the release with the new step and choice type from kernel#5015. typecheck (package and demo), build and prettier are clean, and CI's test job passes, including sdk-contract.test.ts.
  • manually checked in the local hosted UI against real pauses on walmart.com and ebay.com, with an earlier kernel#5015 commit (the step, choice and submit shape this screen uses haven't changed since), and in the demo states. checked the frame edges on a real live view: the old border left a black pixel column, the outline doesn't.
  • not tested: the screen against kernel#5015's current head, staging, and mobile, where the embedded page renders at roughly 20% scale.

related

merge order: kernel#5015 and the SDK release are done; after this, bump the hosted UI before announcing hitl.


Note

Medium Risk
Touches the managed-auth login state machine and submit error recovery, and embeds a third-party origin in an iframe—integrators must allow the live view in CSP.

Overview
Adds AWAITING_HUMAN_INTERVENTION support for HITL connections when a CAPTCHA blocks automated login. The all-in-one flow renders HumanInterventionStep, which embeds the session’s live_view_url via LiveViewFrame (postMessage watchdog, auto-reloads, one manual retry on transport failures) and submits the human_intervention choice on Continue. The step stays mounted during submit so the iframe is not torn down; failed submits return to the session’s current step instead of defaulting to the input form.

The human_intervention choice is excluded from sign-in options. captcha_blocked errors prefer the API error_message when present. Protocol/types align with @onkernel/sdk 0.138.0, plus localization, appearance slots, styles, demo states, README CSP frame-src note, and broad test coverage.

Reviewed by Cursor Bugbot for commit 221ea9f. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel

vercel Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
managed-auth-react-demo Ready Ready Preview Oct 30, 2026 1:17am UTC

@akxue
akxue marked this pull request as ready for review October 28, 2026 10:08
@akxue
akxue requested a review from masnwilliams October 28, 2026 10:08

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f7fd933. Configure here.

Comment thread packages/managed-auth-react/src/KernelManagedAuth.tsx

@masnwilliams masnwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. the double submit on Continue is fixed in 09a2052 and the new KernelManagedAuth test covers it.

three optional cleanups inline, none blocking.

merge order reminder: the test check fails on sdk-contract.test.ts until kernel/kernel#5015 merges and the SDK is regenerated with the new step and choice type, so this should land after that.

Comment on lines +305 to +310
setUIState((current) => {
if (isTerminal(current)) return current;
return stateRef.current
? deriveUIState(stateRef.current)
: "awaiting_input";
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: this now does almost the same thing as the stale_interaction updater above. re-deriving a screen that isn't submitting gives back the screen it already has, and stateRef.current can't be null once a submit can start, so the "awaiting_input" fallback never runs. one shared updater would cover both:

const leaveSubmitting = () =>
  setUIState((current) =>
    current === "submitting" && stateRef.current
      ? deriveUIState(stateRef.current)
      : current,
  );

Comment on lines 68 to 70
captcha_blocked: {
title: "Verification required",
description: "A CAPTCHA challenge blocked the login.",
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: with the description removed, a captcha_blocked failure that has no error_message now shows the generic error copy instead of the captcha text. keeping description: "A CAPTCHA challenge blocked the login." and preferring the server message for this code when one is present would avoid that, and description could stay required for the other codes.

{errorMessage && <ErrorBanner message={errorMessage} />}

<LiveViewFrame url={liveViewUrl} />

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: the reload/retry counters in LiveViewFrame live in refs and don't reset if live_view_url changes, and a failure that's already set keeps the effect inactive for a new url. <LiveViewFrame key={liveViewUrl ?? ""} url={liveViewUrl} /> would start the frame fresh on a new url.

@akxue
akxue merged commit 5e32de9 into main Oct 30, 2026
5 checks passed
@akxue
akxue deleted the hypeship/ma-captcha-hitl branch October 30, 2026 01:19

This branch was successfully deployed

1 active deployment
Preview — 221ea9f2 Deployed Oct 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants