Skip to content

Commit ee643c4

Browse files
rgarciargarcia
committed
Add the pi extension that will replace the CLI
pi owns the agent loop, session, and UI; the extension contributes the tools, the browser they run against, and the provider wiring native surfaces need. Adapted from an earlier spike onto current APIs. Notably it uses neither attach() nor the harness: pi owns the model collection, so the extension takes the two pieces that are not pi-shaped -- the catalog compiler and CuaExecutionResources -- and applies headers and payload transforms through pi's own hooks. That is evidence for the framework-neutral split the rename is built on. Selectors now cover every provider-native surface rather than only Anthropic's computer tool, and /cua-tools with no argument lists what the current model can take, with the catalog compiler's own reason for anything it cannot.
1 parent 8ffce0c commit ee643c4

22 files changed

Lines changed: 1701 additions & 11 deletions

‎.github/workflows/ci.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,21 @@ jobs:
4747
- name: Agent unit tests
4848
run: npm test --workspace @onkernel/cua-agent -- --exclude "**/*.live.test.ts"
4949

50+
pi-extension-unit:
51+
runs-on: ubuntu-latest
52+
timeout-minutes: 15
53+
steps:
54+
- uses: actions/checkout@v5
55+
- uses: actions/setup-node@v5
56+
with:
57+
node-version: 22
58+
cache: npm
59+
- run: npm ci
60+
- run: npm run build --workspace @onkernel/cua-ai
61+
- run: npm run build --workspace @onkernel/cua-agent
62+
- name: Pi extension unit tests
63+
run: npm test --workspace @onkernel/cua-pi-extension
64+
5065
cli-unit:
5166
runs-on: ubuntu-latest
5267
timeout-minutes: 15

‎README.md‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,17 @@ All of them expect you to:
3434

3535
```
3636
packages/
37-
├── ai/ # @onkernel/cua-ai - model catalog, tool schemas, provider adapters
38-
├── agent/ # @onkernel/cua-agent - Kernel-browser tool execution
39-
├── cli/ # @onkernel/cua-cli - the `cua` binary
40-
└── ptywright/ # @onkernel/ptywright - development-only PTY/TUI test infrastructure
37+
├── ai/ # @onkernel/cua-ai - model catalog, tool schemas, provider adapters
38+
├── agent/ # @onkernel/cua-agent - Kernel-browser tool execution
39+
├── pi-extension/ # @onkernel/cua-pi-extension - Kernel browser tools inside pi's own session
40+
├── cli/ # @onkernel/cua-cli - the `cua` binary
41+
└── ptywright/ # @onkernel/ptywright - development-only PTY/TUI test infrastructure
4142
```
4243

44+
**Using pi already?** [`packages/pi-extension`](packages/pi-extension) adds these
45+
tools to a pi session without a second agent loop: `pi install` it, select tools
46+
with `--cua-tools`, and pi keeps owning the session, UI, and model.
47+
4348
**Building your own agent? Start here:** [`packages/agent`](packages/agent)
4449
(`@onkernel/cua-agent`) — `attach()` binds a Kernel browser and compiles a
4550
(model, tools) pair into plain pi objects you drive yourself. It sits on
@@ -53,21 +58,27 @@ flowchart LR
5358
ai[("@onkernel/cua-ai")]
5459
agent[("@onkernel/cua-agent")]
5560
cli[("@onkernel/cua-cli")]
61+
ext[("@onkernel/cua-pi-extension")]
5662
pi[("pi-agent-core / pi-ai / pi-tui / pi-coding-agent")]
5763
sdk[("@onkernel/sdk")]
5864
ai --> agent
5965
agent --> cli
6066
ai --> cli
67+
agent --> ext
68+
ai --> ext
6169
pi --> agent
6270
pi --> cli
71+
pi --> ext
6372
sdk --> agent
6473
sdk --> cli
74+
sdk --> ext
6575
```
6676

6777
| Package | What it ships |
6878
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
6979
| [`@onkernel/cua-ai`](packages/ai) | Computer-use model catalog, tool factories/toolsets, compatibility checks, and provider adapters. |
70-
| [`@onkernel/cua-agent`](packages/agent) | Agent and harness APIs that run selected computer-use tools against a Kernel browser. |
80+
| [`@onkernel/cua-agent`](packages/agent) | `attach()`: binds a Kernel browser and compiles a (model, tools) pair into plain pi objects. |
81+
| [`@onkernel/cua-pi-extension`](packages/pi-extension) | A pi extension contributing these tools to pi's own agent session. |
7182
| [`@onkernel/cua-cli`](packages/cli) | The `cua` binary: argv parsing, sessions, skills, JSONL output, pi-tui front-end. |
7283
| [`@onkernel/ptywright`](packages/ptywright) | Development-only PTY/TUI test infrastructure. |
7384

‎docs/architecture.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,14 @@ both explicitly and may use pi's orchestration primitives directly.
1919
`AgentTool` or materialization types and no `pi-agent-core` dependency.
2020
- `@onkernel/cua-agent` is provider-neutral runtime glue around
2121
`pi-agent-core`. It defines `CuaAgentTool`, materializes catalog specs
22-
exactly once per shared resource pool against a Kernel browser, owns
23-
implementation identity for replacement detection, owns shared execution
24-
resources, and applies catalog plans supplied as data.
22+
exactly once per shared resource pool against a Kernel browser, owns shared
23+
execution resources, and applies catalog plans supplied as data.
24+
- `@onkernel/cua-pi-extension` contributes these tools to a pi session that pi
25+
itself owns. It is the one consumer that uses neither `attach()` nor the
26+
harness: pi owns the model collection and the agent loop, so the extension
27+
takes the two pieces that are not pi-shaped — the catalog compiler and
28+
`CuaExecutionResources` — and applies headers and payload transforms through
29+
pi's own `before_provider_headers` and `before_provider_request` hooks.
2530
- `@onkernel/cua-cli` owns application policy: it chooses an explicit tool list
2631
for each selected model, adds pi coding tools, supplies the system prompt,
2732
resolves credentials/sessions/skills, and renders text, JSONL, or TUI output.

‎package-lock.json‎

Lines changed: 28 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@
88
"packages/ai",
99
"packages/agent",
1010
"packages/ptywright",
11-
"packages/cli"
11+
"packages/cli",
12+
"packages/pi-extension"
1213
],
1314
"scripts": {
1415
"build": "npm run build --workspace @onkernel/cua-ai && npm run build --workspace @onkernel/cua-agent && tsc -b && npm run build --workspace @onkernel/cua-cli && npm run build:native --workspace @onkernel/ptywright --if-present",

‎packages/pi-extension/CHANGELOG.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Changelog
2+
3+
## Unreleased
4+
5+
- Add `@onkernel/cua-pi-extension`, an installable pi extension that contributes
6+
Kernel browser tools to pi's own agent session. Selectors cover the CDP browser
7+
toolset, the canonical computer toolset, the batch and Playwright tools, and
8+
every provider-native surface CUA carries (Anthropic computer and browser,
9+
OpenAI computer, Google's predefined browser set).
10+
- A selection is validated by compiling it for the active model, so an
11+
incompatible tool deactivates with the catalog compiler's own reason instead of
12+
failing at request time. `/cua-tools` with no argument lists every selector for
13+
the current model with those reasons.
14+
- One browser is provisioned lazily per session on first tool execution and
15+
deleted on shutdown if this session created it. Declaration compilation, header
16+
generation, and payload transforms never provision a browser.

‎packages/pi-extension/README.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# `@onkernel/cua-pi-extension`
2+
3+
An installable [pi](https://pi.dev) extension that adds Kernel browser tools to
4+
pi's existing agent session. pi owns the agent loop, session, and UI; this
5+
extension contributes the tools, the browser they run against, and the provider
6+
wiring that provider-native surfaces need.
7+
8+
It does not start a second model loop, and it adds no implicit screenshots or
9+
prompt instructions.
10+
11+
## Install
12+
13+
```sh
14+
pi install ./packages/pi-extension
15+
# or, once published
16+
pi install npm:@onkernel/cua-pi-extension
17+
```
18+
19+
`KERNEL_API_KEY` is required when a tool first executes, not at startup.
20+
`KERNEL_BASE_URL` is honored. Neither is written to session entries or output.
21+
22+
## Use
23+
24+
No selector means no Kernel tool is active and no browser is provisioned.
25+
26+
```sh
27+
pi -p --provider openai --model gpt-5.6-sol \
28+
--cua-tools browser,browser-act "Open example.com and report its heading"
29+
30+
pi --mode rpc --no-session --provider openai --model gpt-5.6-sol --cua-tools browser
31+
32+
pi -p --provider anthropic --model claude-opus-5 --cua-tools anthropic-computer \
33+
"Open example.com and report its heading"
34+
```
35+
36+
### Selectors
37+
38+
| selector | tools |
39+
| --- | --- |
40+
| `browser` | the CDP browser toolset |
41+
| `computer` | the canonical computer toolset |
42+
| `mixed` | both, deduplicated |
43+
| `browser-act` | `browser_act` alone, the verified-plan tool |
44+
| `browser-batch`, `computer-batch` | one mechanical batch tool |
45+
| `playwright` | `playwright_execute` |
46+
| `anthropic-computer`, `anthropic-browser` | Anthropic's native surfaces |
47+
| `openai-computer` | OpenAI's native computer tool |
48+
| `google-browser` | Google's predefined browser action set |
49+
| any individual tool name | that tool alone |
50+
51+
`--cua-coordinates` selects `pixels` (default) or `normalized-1000` for the
52+
computer toolset's coordinate contract.
53+
54+
### Commands
55+
56+
- `/cua` — current selectors, active tools, and browser status.
57+
- `/cua-tools` — with no argument, list every selector for the current model,
58+
marking the selected ones and showing the compiler's own reason for any that
59+
this model cannot take. With an argument, replace the selection. `none` clears
60+
it.
61+
62+
A selection is checked by compiling it, so a model that cannot take a tool
63+
deactivates it with a reason rather than failing at request time. Switching
64+
models re-checks, and restores a previously forced-off selection when the new
65+
model can take it.
66+
67+
### Browser
68+
69+
| flag | effect |
70+
| --- | --- |
71+
| `--cua-browser-session` | attach an existing session; never deleted on exit |
72+
| `--cua-profile-id`, `--cua-profile-save-changes` | load and optionally persist a profile |
73+
| `--cua-proxy-id` | route through a Kernel proxy |
74+
| `--cua-browser-timeout` | owned-browser timeout in seconds (default 300) |
75+
76+
One browser is provisioned lazily per session, on first tool execution.
77+
Compiling declarations, generating headers, and transforming a payload never
78+
provision one. An owned browser is deleted on session shutdown.
79+
80+
## Development
81+
82+
```bash
83+
npm run typecheck --workspace @onkernel/cua-pi-extension
84+
npm test --workspace @onkernel/cua-pi-extension
85+
```
86+
87+
The test suite includes an end-to-end run that spawns real `pi` in print and RPC
88+
modes against a fake provider and Kernel server.
89+
90+
## License
91+
92+
MIT

‎packages/pi-extension/package.json‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
{
2+
"name": "@onkernel/cua-pi-extension",
3+
"version": "0.10.0",
4+
"description": "Kernel browser tools for pi",
5+
"license": "MIT",
6+
"type": "module",
7+
"repository": {
8+
"type": "git",
9+
"url": "git+https://github.com/kernel/cua.git",
10+
"directory": "packages/pi-extension"
11+
},
12+
"homepage": "https://github.com/kernel/cua/tree/main/packages/pi-extension#readme",
13+
"keywords": [
14+
"pi-package",
15+
"pi-extension",
16+
"computer-use",
17+
"kernel"
18+
],
19+
"pi": {
20+
"extensions": [
21+
"./src/index.ts"
22+
]
23+
},
24+
"files": [
25+
"src",
26+
"README.md",
27+
"CHANGELOG.md",
28+
"package.json"
29+
],
30+
"publishConfig": {
31+
"access": "public"
32+
},
33+
"engines": {
34+
"node": ">=22.19.0"
35+
},
36+
"scripts": {
37+
"build": "tsc -b",
38+
"typecheck": "tsc -b",
39+
"clean": "tsc -b --clean && rm -rf dist-tsc",
40+
"test": "vitest --run"
41+
},
42+
"dependencies": {
43+
"@onkernel/cua-agent": "0.10.0",
44+
"@onkernel/cua-ai": "0.10.0",
45+
"@onkernel/sdk": "0.49.0"
46+
},
47+
"peerDependencies": {
48+
"@earendil-works/pi-agent-core": "*",
49+
"@earendil-works/pi-ai": "*",
50+
"@earendil-works/pi-coding-agent": "*",
51+
"@earendil-works/pi-tui": "*"
52+
},
53+
"devDependencies": {
54+
"vitest": "^3.2.4"
55+
}
56+
}

0 commit comments

Comments
 (0)