diff --git a/.agents/skills/cross-platform-gui-e2e-test/SKILL.md b/.agents/skills/cross-platform-gui-e2e-test/SKILL.md new file mode 100644 index 0000000..0465449 --- /dev/null +++ b/.agents/skills/cross-platform-gui-e2e-test/SKILL.md @@ -0,0 +1,105 @@ +--- +name: cross-platform-gui-e2e-test +description: Run packaged Ropy desktop GUI end-to-end, smoke, and compatibility tests across macOS, Windows, and Linux. Use when validating a Ropy release or CI artifact, reproducing platform-specific clipboard, global-hotkey, tray, focus, X11, or window behavior, or collecting cross-platform GUI evidence. Run macOS locally; run Windows and Linux in isolated AgentBay virtual desktops. +--- + +# Ropy Cross-Platform GUI E2E Test + +Validate the packaged desktop application on every supported operating system. Complement GPUI in-process tests with real process, clipboard, global-hotkey, tray, focus, and window interactions. + +## Enforce the Platform Contract + +| Target | Execution environment | Release artifact | +| --- | --- | --- | +| macOS Apple Silicon | Local matching Mac | `ropy-aarch64-apple-darwin.dmg` | +| macOS Intel | Local matching Mac | `ropy-x86_64-apple-darwin.dmg` | +| Windows x86-64 | AgentBay Windows virtual desktop | `ropy-x86_64-pc-windows-msvc.zip` | +| Linux x86-64 | AgentBay Linux X11 virtual desktop | `ropy-x86_64-unknown-linux-gnu.tar.xz` | +| Linux ARM64 | Matching AgentBay custom Linux X11 image | `ropy-aarch64-unknown-linux-gnu.tar.xz` | + +- Inspect the host or guest architecture before selecting an artifact. Never use emulation silently. +- Run macOS on the local Mac. Do not spend AgentBay resources for macOS unless the user explicitly changes this policy. +- Run every Windows and Linux GUI test in a fresh virtual desktop. Do not substitute a local container, headless shell, WSL, or compile-only CI job. +- Require X11 connectivity for Linux. Ropy's Linux GUI integration is X11-specific; a `DISPLAY` variable without a reachable X server is insufficient. +- Report a cross-platform pass only when every requested platform row passes. Preserve separate results for each OS and architecture. + +Read only the reference for the platform being exercised: + +- [macOS local run](references/macos.md) +- [Windows AgentBay run](references/windows.md) +- [Linux AgentBay run](references/linux.md) + +## Apply Shared Guardrails + +- Resolve the release tag or commit requested by the user. Otherwise use the latest published release. +- Download artifacts to a temporary controller directory, inspect their contents, calculate SHA-256, and compare any publisher-provided digest before execution. +- Keep AgentBay credentials in an environment variable, secret store, or user-level MCP configuration. Never commit, print, or include a credential-bearing URL in output. +- Confirm paid-session authorization unless the current request already authorizes the virtual test. Do not enable postpaid billing, buy credits, or raise limits without separate authorization. +- Before creating a virtual session, inspect active sessions and billing controls. Prefer concurrency `1`, a maximum runtime of no more than `30` minutes, and an idle timeout of at least `10` minutes. +- Create only one virtual session at a time. Record its identifier immediately and release it in a `finally` path. Explicitly terminate it; do not leave it running or hibernating. +- Use a unique marker such as `ropy-e2e--` so stale clipboard contents cannot create a false pass. +- Never record unrelated local clipboard contents, credentials, signed URLs, or user data in screenshots and logs. + +## Prefer Stable Virtual Control + +For Windows and Linux, prefer these AgentBay paths in order: + +1. Use a configured MCP endpoint with command, filesystem, screenshot, keyboard, and window capabilities. +2. Use the official AgentBay SDK with `AGENTBAY_API_KEY` supplied through the environment. +3. Use an AgentBay console debug session through the in-app browser only as a manual fallback. + +Discover current tool names rather than assuming one SDK version. Prefer filesystem `write_large_file` or an AgentBay Context for artifact transfer. Avoid downloading release assets inside the guest: slow downloads may trigger idle-session reclamation. + +If no supported control path is available, stop before creating a paid session and report the missing capability. Never place the secret endpoint in the repository as a workaround. + +## Run the Shared GUI Scenario + +Use platform-native UI automation, screenshots, and window inspection at every assertion boundary: + +1. Launch the verified packaged executable and assert that its process remains alive after three seconds. +2. Expect no initial visible window: Ropy intentionally starts hidden in the system tray. +3. Send `Control+Shift+D`, the default activation hotkey on all supported platforms, and assert that the Ropy window becomes visible. +4. Open a platform-native plain-text editor, enter the unique marker, select it, and copy it. +5. Wait briefly for clipboard ingestion, reactivate Ropy, and assert that the exact marker appears in history. +6. Press `/`, search for a unique substring, and assert that the matching record remains visible. +7. Select the record and press `Enter`. The default confirm mode copies the record back to the clipboard. +8. Return to the editor, paste, and assert that the pasted text exactly equals the marker. + +Prefer semantic window, text, and accessibility queries. Use coordinates only after taking a fresh screenshot. Retry a flaky clipboard ingestion once with a new marker; collect state before retrying. + +## Add Focused Checks Deliberately + +- Test favorite, delete, clear, plain-text paste, grid navigation, pinning, settings, theme, language, opacity, autostart, or update behavior only when relevant to the requested change. +- Avoid destructive history operations on local macOS by default because they can touch the user's real Ropy data. +- Keep the common smoke scenario identical across platforms; isolate platform-specific assertions in the relevant run. +- Verify Linux tray behavior only when the image has a working desktop panel and session bus. Do not classify a missing tray host as a Ropy regression. + +## Record a Platform Matrix + +For every requested target, record: + +- OS version, architecture, execution environment, and image identifier when virtual; +- release tag or commit SHA, artifact name, size, and SHA-256; +- start/end timestamps and elapsed time; +- checksum, extraction, launch, process, window, and assertion results; +- screenshots of the visible Ropy window, captured marker, filtered result, and final editor paste; +- cleanup confirmation and remaining active virtual-session count. + +Classify each row independently: + +- **pass**: the packaged executable launches and all requested GUI assertions pass; +- **application failure**: the verified artifact runs in a healthy environment but exits, cannot activate, or violates an assertion; +- **infrastructure blocked**: architecture, upload, MCP/SDK access, X11, desktop image, session expiry, or provisioning prevents reaching the assertion; +- **partial**: artifact transfer or launch is proven but the GUI scenario does not finish. + +State the last verified boundary. Never describe an infrastructure interruption as a Ropy failure. + +## Clean Up Reliably + +- Collect platform logs before teardown on failure; use the location in the platform reference. +- Terminate only the Ropy process started by this run and close helper applications when practical. +- Remove temporary local mounts and files. +- Always release AgentBay sessions after Windows or Linux runs, including after exceptions and timeouts. +- Recheck the AgentBay session list and report any cleanup failure immediately. + +Before changing the virtual workflow, consult current official AgentBay Computer Use, desktop UI automation, filesystem, command, application, and window-management documentation. diff --git a/.agents/skills/cross-platform-gui-e2e-test/agents/openai.yaml b/.agents/skills/cross-platform-gui-e2e-test/agents/openai.yaml new file mode 100644 index 0000000..6bbbea0 --- /dev/null +++ b/.agents/skills/cross-platform-gui-e2e-test/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Ropy Cross-Platform GUI E2E" + short_description: "Run Ropy GUI tests locally and in virtual desktops" + default_prompt: "Use $cross-platform-gui-e2e-test to validate the latest Ropy build on macOS, Windows, and Linux." diff --git a/.agents/skills/cross-platform-gui-e2e-test/references/linux.md b/.agents/skills/cross-platform-gui-e2e-test/references/linux.md new file mode 100644 index 0000000..a4d3961 --- /dev/null +++ b/.agents/skills/cross-platform-gui-e2e-test/references/linux.md @@ -0,0 +1,51 @@ +# Linux AgentBay Run + +Use a fresh AgentBay Linux Computer Use virtual desktop. Ropy supports Linux through X11, so require a real desktop session; a successful shell command or headless build is not GUI evidence. + +## Match the image + +- Inspect `uname -m` and select the matching GNU/Linux artifact. Use `x86_64-unknown-linux-gnu` for an x86-64 `linux_latest` image. +- Use an ARM64 custom image for `aarch64-unknown-linux-gnu`; never execute it through implicit emulation. +- Require a non-empty `DISPLAY` and verify the connection with `xdpyinfo` or an equivalent X11 query. +- Accept XWayland only when the X11 connection is reachable and Ropy's window, hotkey, and clipboard behavior work. Do not claim native Wayland support. +- Check that a desktop window manager is running. Require a desktop panel and session bus only for tray-specific assertions. + +## Transfer and launch + +Upload the verified archive to `/tmp/ropy-e2e/ropy.tar.xz` with the AgentBay filesystem API. Run an equivalent shell sequence: + +```bash +set -eu +root=/tmp/ropy-e2e +archive="$root/ropy.tar.xz" +expected='' +actual="$(sha256sum "$archive" | awk '{print $1}')" +test "$actual" = "$expected" +mkdir -p "$root/app" +tar -xJf "$archive" -C "$root/app" +exe="$(find "$root/app" -type f -name ropy -perm -u+x -print -quit)" +test -n "$exe" +test -n "${DISPLAY:-}" +xdpyinfo >/dev/null +ldd "$exe" | tee "$root/ldd.txt" +if grep -q 'not found' "$root/ldd.txt"; then exit 64; fi +nohup "$exe" >"$root/ropy.stdout.log" 2>"$root/ropy.stderr.log" & +pid=$! +sleep 3 +kill -0 "$pid" +printf '%s\n' "$pid" +``` + +If `ldd` reports missing libraries, classify the stock image as infrastructure-blocked or use an approved custom image containing the runtime equivalents of the repository's GTK3, X11, `libxdo`, and `libxkbcommon-x11` build dependencies. Do not install arbitrary packages silently into a supposedly reproducible compatibility image. + +Discover an installed graphical text editor and use it for the shared marker scenario. Send `Control+Shift+D` to reveal Ropy. Use AgentBay screenshot, keyboard, application, and window tools rather than `xdotool` for the assertions unless a lower-level diagnostic is specifically required. + +## Troubleshoot and clean up + +- If `DISPLAY` is unset or `xdpyinfo` fails, the image is not ready for Ropy GUI testing. +- If the process lives but activation fails, collect X11 session details and Ropy logs before classifying the result. +- If the tray is absent while hotkey activation works, check the desktop panel and `DBUS_SESSION_BUS_ADDRESS`; report missing host infrastructure separately. +- Collect JSONL logs from `~/.config/ropy/logs` plus `$root/ropy.stdout.log` and `$root/ropy.stderr.log` on failure. +- Stop the test process by its recorded PID, then release the AgentBay session and verify that it is gone. + +Current official references: [Computer Use](https://help.aliyun.com/en/agentbay/computeruse), [desktop UI automation](https://help.aliyun.com/en/agentbay/developer-reference/ui-automation), and [application management](https://help.aliyun.com/en/agentbay/developer-reference/application-management). diff --git a/.agents/skills/cross-platform-gui-e2e-test/references/macos.md b/.agents/skills/cross-platform-gui-e2e-test/references/macos.md new file mode 100644 index 0000000..0ef7a07 --- /dev/null +++ b/.agents/skills/cross-platform-gui-e2e-test/references/macos.md @@ -0,0 +1,34 @@ +# macOS Local Run + +Use the local Mac for macOS GUI validation. Inspect `uname -m` first and select the matching DMG; do not run an Intel artifact on Apple Silicon through Rosetta without explicitly labeling that separate compatibility run. + +## Protect Local State + +- Check for an existing Ropy process. Do not terminate or replace a user-owned running instance without explicit permission. +- Warn that the smoke test temporarily changes the system clipboard and can add the unique marker to the user's Ropy history. +- Never run clear-history or bulk-delete checks locally by default. Remove only the unique test marker when that can be done unambiguously. +- Do not inspect or report pre-existing clipboard history. + +## Verify and launch + +Download the matching `.dmg` to a temporary directory and verify it against the release digest: + +```bash +shasum -a 256 /path/to/ropy-apple-darwin.dmg +hdiutil attach -nobrowse -readonly /path/to/ropy-apple-darwin.dmg +``` + +Locate `Ropy.app` on the mounted volume and launch that exact bundle with `open`. Record the mount point, bundle path, bundle version, and process identifier. Do not validate an already-installed copy by accident. + +Use local Computer Use or another approved macOS UI-control capability for screenshots, keyboard input, focus, and the shared GUI scenario. Ropy starts hidden; send `Control+Shift+D` to reveal it. + +If macOS blocks the downloaded application through quarantine or Gatekeeper, capture the dialog and report an application packaging/signing result. Do not remove quarantine attributes or bypass platform security unless the user explicitly asks for that diagnostic. + +Use a plain-text editor such as TextEdit in plain-text mode for the marker copy/paste assertion. + +## Collect and clean up + +- Collect JSONL logs from `~/Library/Application Support/ropy/logs` on failure. +- Quit only the Ropy instance launched for this test. +- Eject the mounted DMG with `hdiutil detach` and remove the temporary download. +- Restore the prior plain-text clipboard value when it was captured without exposing it; otherwise state that clipboard restoration was not lossless. diff --git a/.agents/skills/cross-platform-gui-e2e-test/references/windows.md b/.agents/skills/cross-platform-gui-e2e-test/references/windows.md new file mode 100644 index 0000000..2bef25f --- /dev/null +++ b/.agents/skills/cross-platform-gui-e2e-test/references/windows.md @@ -0,0 +1,38 @@ +# Windows AgentBay Run + +Use a fresh AgentBay Windows Computer Use virtual desktop. Record its session and image identifiers before transferring the artifact. + +## Transfer and launch + +Upload the verified archive to `C:\ropy-e2e\ropy.zip` with the AgentBay filesystem API. Use shell or command APIs for setup; do not type long PowerShell commands through remote-desktop automation because keyboard layouts can corrupt punctuation. + +Run an equivalent non-interactive PowerShell sequence: + +```powershell +$ErrorActionPreference = 'Stop' +$root = 'C:\ropy-e2e' +$zip = Join-Path $root 'ropy.zip' +$expected = '' +$actual = (Get-FileHash -Algorithm SHA256 $zip).Hash.ToLowerInvariant() +if ($actual -ne $expected) { throw "SHA-256 mismatch: $actual" } +$app = Join-Path $root 'app' +Expand-Archive -Path $zip -DestinationPath $app -Force +$exe = Join-Path $app 'ropy.exe' +if (-not (Test-Path $exe)) { throw "Missing $exe" } +$process = Start-Process -FilePath $exe -PassThru +Start-Sleep -Seconds 3 +if ($process.HasExited) { throw "ropy.exe exited with $($process.ExitCode)" } +$process.Id +``` + +Use Notepad for the shared marker scenario. Treat an alive process with no initial window as expected, then send `Control+Shift+D`. + +## Troubleshoot and clean up + +- If PowerShell 5 cannot download from GitHub, keep using controller-side upload; do not weaken TLS settings. +- If typed commands replace `:` or other punctuation, use the command API. +- If the archive arrives but the session expires before extraction, use direct file upload and an idle timeout of at least `10` minutes. +- Collect JSONL logs from `%APPDATA%\ropy\logs` on failure. +- Stop the test process by its recorded PID, then release the AgentBay session and verify that it is gone. + +Current official references: [Windows Computer Use](https://help.aliyun.com/en/agentbay/developer-reference/computer-use-windows-server-2022), [desktop UI automation](https://help.aliyun.com/en/agentbay/developer-reference/ui-automation), and [window management](https://help.aliyun.com/en/agentbay/support/window-management).