Skip to content

[codex] Improve CI test coverage#212

Merged
pbakaus merged 11 commits into
mainfrom
codex/ci-test-suites
Jun 8, 2026
Merged

[codex] Improve CI test coverage#212
pbakaus merged 11 commits into
mainfrom
codex/ci-test-suites

Conversation

@pbakaus

@pbakaus pbakaus commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add a central test suite registry and runner so bun run test is composed from named suites instead of one long inline command.
  • Add a CI test planner that runs core coverage always, conditionally runs detector/live/framework suites on pull requests, and keeps expensive or remote suites opt-in/manual.
  • Split CLI E2E coverage into deterministic local-bundle tests for regular runs and a separate remote production smoke suite.
  • Add guardrail tests so every new tests/**/*.test.{js,mjs} file must be registered in a default or opt-in suite.

Why

CI was only exercising a subset of the repository's available tests, and the previous CLI E2E path mixed useful install/update coverage with production network dependency. This makes the default signal broader and more maintainable while keeping non-deterministic or costly checks isolated.

Validation

  • bun run test
  • bun run build
  • git diff --check
  • git diff --exit-code -- .agents .claude .cursor .gemini .github/skills plugin cli/engine/detect-antipatterns-browser.js
  • node scripts/run-tests.mjs --list
  • package JSON parse check

Note: a sandboxed bun run test:core attempt failed only because local listen(127.0.0.1) is blocked in the desktop sandbox; the same command passed outside the sandbox, and the full default suite passed outside the sandbox.


Note

Medium Risk
CI workflow behavior changes broadly affect what runs on every PR, and live-browser accept/DOM cleanup touches user-visible preview state after variant accept.

Overview
Replaces the monolithic bun run test command with a named suite registry (scripts/test-suites.mjs, scripts/run-tests.mjs) and wires CI to a changes job that runs scripts/ci-test-plan.mjs so PRs get core always plus path-triggered detector/live/framework jobs, while Playwright live E2E, remote CLI smoke, skill-behavior, and provider regressions stay opt-in (manual workflow_dispatch or matching diffs). The workflow also splits live E2E into matrix smoke/full runs, adds caching, verifies generated skill copies after build, and introduces test:cli-remote-e2e separate from local-bundle CLI tests; impeccable skills can use IMPECCABLE_BUNDLE_PATH for offline bundles.

Live browser (canonical skill/scripts/live-browser.js, mirrored to provider skill trees) gains JSX source-fallback normalization before DOMParser, safer class-based element matching, a 750ms delayed injectVariantsFromSource when HMR is slow, quieter optional sendEvent failures, and post-accept cleanup that finds carbonize wrappers and removes stale runtime chrome when the DOM is already clean.

Live E2E adds fixture/scenario filters, configurable timeouts, failure artifact capture, stricter accept/carbonize source checks, and steer-agent support for template className; tests and docs were updated accordingly (tests/ci-test-plan.test.mjs, tests/test-suites.test.mjs, framework fixture README).

Reviewed by Cursor Bugbot for commit fa09f3e. Bugbot is set up for automated code reviews on this repo. Configure here.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 6, 2026

Copy link
Copy Markdown

Deploying impeccable with  Cloudflare Pages  Cloudflare Pages

Latest commit: fa09f3e
Status: ✅  Deploy successful!
Preview URL: https://924b8765.impeccable-2rv.pages.dev
Branch Preview URL: https://codex-ci-test-suites.impeccable-2rv.pages.dev

View logs

@pbakaus pbakaus marked this pull request as ready for review June 6, 2026 01:57
Comment thread scripts/ci-test-plan.mjs

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit df491d5. Configure here.

Comment thread skill/scripts/live-browser.js
@pbakaus pbakaus merged commit 82801a4 into main Jun 8, 2026
11 checks passed
@pbakaus pbakaus deleted the codex/ci-test-suites branch June 8, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant