diff --git a/openspec/changes/runtime-rule-execution/.openspec.yaml b/openspec/changes/archive/2026-07-03-runtime-rule-execution/.openspec.yaml similarity index 100% rename from openspec/changes/runtime-rule-execution/.openspec.yaml rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/.openspec.yaml diff --git a/openspec/changes/runtime-rule-execution/design.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md similarity index 94% rename from openspec/changes/runtime-rule-execution/design.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md index e296f60d..02a22ff3 100644 --- a/openspec/changes/runtime-rule-execution/design.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md @@ -17,8 +17,8 @@ hashed, globally-unique `id` (`${ruleSlug}-${sha1(ruleBody).slice(0,8)}`) for sc attribution, and a stable model-assigned `name` the check branches on. Capture rules may carry full ast-grep config (`constraints`/`utils`/`transform` as **siblings** of `rule`). -The local harness is resolved in TSKL-245: assemble a rule's capture rules → **one** -`ast-grep scan` (anchor `--inline-rules --json=stream`; broad `--files-with-matches`) → gate +The local harness is resolved in TSKL-245: assemble a rule's capture rules → an +`ast-grep scan` (one per mode: anchor `--json=stream`; broad `--files-with-matches`) → gate on matches → invoke `check.ts`'s **default export** with `(root, matches)` via a bundled, pinned `tsx`; use the **returned** `Finding[]`. `Finding.severity ∈ error|warning|info` maps onto static-rule gating with no translation. Measured cost is `tsx` per-worker startup @@ -126,9 +126,13 @@ would run unverified code the user never opted into. ### Decision: The narrow → gate → `check.ts` harness, per TSKL-245 -For each executable runtime rule: collect its capture rules and run **one** `ast-grep` scan — -`--inline-rules --json=stream` in anchor mode, `--files-with-matches` in broad mode -(`kind: program` enumerators). Normalize each match to +For each executable runtime rule: collect its capture rules into a generated ast-grep config +(a temp `ruleDirs` config — `--inline-rules` carries only a single rule, and a runtime rule has +multiple capture rules plus full `constraints`/`utils`/`transform`) and run **one scan per +mode** — `--json=stream` for anchor capture rules, `--files-with-matches` for broad +(`kind: program`) enumerators. An all-anchor rule is a single scan; mixing modes is one scan +per mode (broad is kept separate so a `kind: program` rule isn't streamed as whole-file text). +Normalize each match to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed `ruleId` back to the model `name` as `match.rule`. If there are **zero** matches, `check.ts` is not invoked. Otherwise invoke its **default export** as a function with diff --git a/openspec/changes/runtime-rule-execution/proposal.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/proposal.md similarity index 97% rename from openspec/changes/runtime-rule-execution/proposal.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/proposal.md index 84918445..76c1aa2b 100644 --- a/openspec/changes/runtime-rule-execution/proposal.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/proposal.md @@ -20,8 +20,8 @@ the default safety mechanism is a server-validated signature, not a local sandbo ## What Changes - Add a **`cli-runtime-rule-execution`** capability: recognize a runtime-rule directory - (`metadata.taskless.kind: runtime`), run its capture rules as **one** `ast-grep` narrow - (anchor `--inline-rules --json=stream`; broad `--files-with-matches` for `kind: program` + (`metadata.taskless.kind: runtime`), run its capture rules as an `ast-grep` narrow (one scan + per mode: anchor `--json=stream`; broad `--files-with-matches` for `kind: program` enumerators), **gate on matches**, and only then invoke `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`. Zero matches ⇒ `check.ts` is never invoked. Normalize matches to `{ rule, ruleId, file, line, column, text, captures }` diff --git a/openspec/changes/runtime-rule-execution/specs/cli-check/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md similarity index 92% rename from openspec/changes/runtime-rule-execution/specs/cli-check/spec.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md index 23fc716f..385e0dbf 100644 --- a/openspec/changes/runtime-rule-execution/specs/cli-check/spec.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md @@ -165,3 +165,18 @@ non-zero code solely because reconciliation failed, and the warning SHALL be sup - **WHEN** the CLI degrades and `--json` is set - **THEN** stdout SHALL contain the machine JSON shape (`{ success, results }` plus the additive optional `skipped` array for the skipped runtime rules) - **AND** SHALL NOT contain the human-readable degrade warning + +## REMOVED Requirements + +### Requirement: Check warns on reconciliation mismatches + +**Reason**: The cutover scopes reconciliation to runtime `check.ts` only; static rules are no +longer reconciled. Runtime-rule mismatches (a `check.ts` that is `unsafe`/`unknown`/`missing`) +are now surfaced as withheld/skipped notices and the `--json` `skipped` array (see the ADDED +requirements), so the static-rule drift/unknown/missing warnings this described no longer exist. + +### Requirement: Check exits cleanly when the run set is empty + +**Reason**: Static rules are no longer gated by a server `run` set — they always scan. An empty +runtime allow-list simply runs no runtime rules (reported via `skipped`/withheld); there is no +static empty-run-set skip behavior left to specify. diff --git a/openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md similarity index 100% rename from openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md diff --git a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md similarity index 91% rename from openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md index bf81cf81..1e170346 100644 --- a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md @@ -25,10 +25,12 @@ SHALL NOT be executed by `check`. ### Requirement: The harness narrows with one ast-grep scan and gates on matches -For a runtime rule the CLI SHALL assemble the rule's capture rules and run **one** `ast-grep` -scan as the narrow: `--inline-rules --json=stream` in `anchor` mode, and -`--files-with-matches` in `broad` mode (whole-language `kind: program` enumerators). When the -narrow produces **zero** matches the CLI SHALL NOT invoke `check.ts`. +For a runtime rule the CLI SHALL assemble the rule's capture rules into an ast-grep +configuration and run **one scan per mode** as the narrow: `--json=stream` for `anchor` capture +rules, and `--files-with-matches` for `broad` capture rules (whole-language `kind: program` +enumerators). A rule with only `anchor` capture rules therefore runs in a single scan; a rule +mixing modes runs one scan per mode. When the narrow produces **zero** matches the CLI SHALL +NOT invoke `check.ts`. #### Scenario: Zero matches skips the check @@ -36,10 +38,10 @@ narrow produces **zero** matches the CLI SHALL NOT invoke `check.ts`. - **THEN** the CLI SHALL NOT invoke that rule's `check.ts` - **AND** the rule SHALL contribute no findings -#### Scenario: One scan per rule +#### Scenario: Capture rules of a mode run together -- **WHEN** a runtime rule has multiple capture rules -- **THEN** the CLI SHALL run them as a single `ast-grep` scan, not one scan per capture rule +- **WHEN** a runtime rule has multiple `anchor` capture rules +- **THEN** the CLI SHALL run them in a single `ast-grep` scan, not one scan per capture rule ### Requirement: Matches are normalized and attributed to the model name diff --git a/openspec/changes/archive/2026-07-03-runtime-rule-execution/tasks.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/tasks.md new file mode 100644 index 00000000..8a072450 --- /dev/null +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/tasks.md @@ -0,0 +1,45 @@ +## 1. Runtime-rule recognition + +- [x] 1.1 Add `packages/cli/src/rules/runtime/discover.ts`: enumerate `.taskless/runtime-rules/` for rule directories, parse each capture `*.yml`'s `metadata.taskless` (`kind`, `name`, `check`, `match`), and confirm the class via `kind: runtime`. Return a typed `RuntimeRule` (`{ dir, captureFiles, checkFile, match }`). `.taskless/runtime-rule-tests/` is not enumerated for execution. (Also mirrored the harness↔check contract types into `src/types/runtime-rule.ts`.) +- [x] 1.2 Static rules stay sourced from `.taskless/rules/`; runtime rules from `.taskless/runtime-rules/` — location is the class split. + +## 2. Narrow → gate → check harness + +- [x] 2.1 Add `packages/cli/src/rules/runtime/narrow.ts`: assemble a rule's capture rules and run ONE `ast-grep` scan per mode — anchor `--json=stream`, broad `--files-with-matches` (`kind: program`). (Used a temp `--config` rules dir rather than `--inline-rules` — a runtime rule has multiple capture rules + full ast-grep config, which `--inline-rules` can't carry; **copies the original capture `*.yml` bytes** to avoid a YAML round-trip. Grouped by mode: all-anchor = 1 scan, mixed = 1 per mode to avoid broad's whole-file streaming. Reuses `findSgBinary`/`buildPath`.) +- [x] 2.2 Add match normalization to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed capture `id` back to the model `name` surfaced as `match.rule`. (ast-grep 0-indexed → 1-indexed; captures from `metaVariables.single`.) +- [x] 2.3 Gate on matches: when the narrow yields zero matches, do NOT invoke `check.ts`. +- [x] 2.4 Add `packages/cli/src/rules/runtime/invoke.ts`: call `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`; use the returned `Finding[]`. Isolate a throwing check to a single error-severity finding for that rule. (Runs an embedded ESM runner under `tsx`; findings via an out-file so the check's stdout can't pollute the channel.) +- [x] 2.5 Decide and implement scheduling: process-per-check (each invoke spawns a `tsx` process), rules run sequentially; bound each check with a default wall-clock timeout (`DEFAULT_CHECK_TIMEOUT_MS`, overridable via `--timeout`) that SIGKILLs the check and records a single error-severity finding. +- [x] 2.6 Map each `Finding` (`severity ∈ error|warning|info`, omitted → warning) onto `CheckResult` with `source: taskless-runtime` (`src/rules/runtime/harness.ts`); feeds the existing aggregation and exit-code logic. + +## 3. tsx bundling + +- [x] 3.1 Add a pinned `tsx` to the CLI `dependencies` and resolve its bin at runtime via `createRequire`/`tsx/package.json` (no repo-local toolchain assumed); externalized from the Vite bundle (spawned as a subprocess). +- [x] 3.2 Verify `check.ts` executes with no `node_modules` and no precompile — smoke-tested against a temp-dir fixture (discovery → narrow → `tsx` invoke → finding); automated coverage lands in Group 7. + +## 4. Reconcile scoping & materialization + +- [x] 4.1 Add `src/rules/runtime/run-set.ts` (runtime-cohesive rather than bloating the static `run-set.ts`): enumerate via `discoverRuntimeRules`, `signRuntimeChecks` signs each rule's `check.ts` only, `reportRuntimeChecks` maps to `{ file, signature }` (capture `*.yml` and static rules are inert — not reported). +- [x] 4.2 `selectBlessedRuntimeRules(signed, run)` computes per-rule eligibility by content-join: a rule is blessed iff its `check.ts` signature is in `run`; the rest are `withheld` (advisory). +- [x] 4.3 `materializeRuntimeRules` copies blessed rule dirs into `.taskless/.run/runtime-rules//` and re-discovers them (via `discoverRuntimeRulesIn`) so the narrow + `check.ts` execute the blessed bytes; `addToGitignore` keeps `.run/` ignored. + +## 5. check dispatch & modes + +- [x] 5.1 Rewrote `src/commands/check.ts`: static rules under `.taskless/rules/` always scan; runtime rules route through the harness only on a validated path. **Cutover** — removed the stacked-under static-reconcile gating (deleted `src/rules/run-set.ts`). +- [x] 5.2 `planRuntime` implements the mode table: authed → reconcile & run blessed rules, withhold the rest (advisory); logged-out/`--anonymous`/no-remote/reconcile-unavailable → skip runtime + report skipped; `--dangerously-run-scripts` → run all runtime rules (no network). +- [x] 5.3 Added `--dangerously-run-scripts` (skips reconciliation, runs all runtime rules, prominent stderr warning suppressed under `--json`) and `--timeout ` (→ `parseTimeoutMs`). +- [x] 5.4 The former degrade path now scans static rules but **never executes runtime `check.ts`** unverified — skip-with-notice instead. +- [x] 5.5 Skipped-runtime notices are human-output; under `--json` an additive optional `skipped: [{ rule, reason }]` field is added (schema updated) leaving `success`/`results` unchanged. Fixed a `Finding`→`CheckResult` off-by-one (findings are 1-indexed; `CheckResult.range` is 0-indexed). Exit code stays governed solely by error-severity findings. Removed the now-obsolete `test/reconcile-check.test.ts` + `test/run-set.test.ts` (Group 7 adds runtime-dispatch tests). + +## 6. Help & docs + +- [x] 6.1 Updated `check.txt` (topic v2): the two rule kinds, what runs per mode, the `--dangerously-run-scripts` warning, `--timeout`, and the `--json` `skipped` array. +- [x] 6.2 Updated `ci.txt` step 7: static rules always run unauthenticated; the `TASKLESS_TOKEN` backstop is the authoritative enforcement point for runtime `check.ts`. + +## 7. Tests & verification + +- [x] 7.1 `test/runtime-harness.test.ts` (imports src, real ast-grep + tsx): discovery, gate-on-zero-matches (check never invoked), match normalization + `Finding`→`CheckResult` indexing, throwing-check isolation, timeout → error finding. +- [x] 7.2 `test/runtime-check.test.ts` (subprocess against `dist/`, temp fixtures + mock reconcile server + git origin): authed runs blessed rules; empty-run withholds; logged-out and `--anonymous` skip + report; reconcile-unavailable (503) skips; `--dangerously-run-scripts` runs offline with a warning. +- [x] 7.3 Static-always-run asserted in every mode; reconcile receives ONLY the runtime `check.ts` (never the static YAML). +- [x] 7.4 `--json` shape: warnings/notices suppressed; runtime findings share the `results` shape; the optional `skipped` array present when runtime rules don't run. +- [x] 7.5 `pnpm --filter @taskless/cli typecheck`, `pnpm lint`, and full `pnpm test` (338 tests) — all green. Also fixed a real timeout bug (SIGKILL the tsx process _group_, not just the wrapper, so a runaway check is actually terminated). diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/runtime-rule-execution/tasks.md deleted file mode 100644 index c48ac578..00000000 --- a/openspec/changes/runtime-rule-execution/tasks.md +++ /dev/null @@ -1,45 +0,0 @@ -## 1. Runtime-rule recognition - -- [ ] 1.1 Add `packages/cli/src/rules/runtime/discover.ts`: enumerate `.taskless/runtime-rules/` for rule directories, parse each capture `*.yml`'s `metadata.taskless` (`kind`, `name`, `check`, `match`), and confirm the class via `kind: runtime`. Return a typed `RuntimeRule` (`{ dir, captureFiles, checkFile, match }`). `.taskless/runtime-rule-tests/` is not enumerated for execution. -- [ ] 1.2 Static rules stay sourced from `.taskless/rules/`; runtime rules from `.taskless/runtime-rules/` — location is the class split. - -## 2. Narrow → gate → check harness - -- [ ] 2.1 Add `packages/cli/src/rules/runtime/narrow.ts`: assemble a rule's capture rules and run ONE `ast-grep` scan — anchor mode `--inline-rules --json=stream`, broad mode `--files-with-matches` (`kind: program`). Reuse the existing scan plumbing where possible. -- [ ] 2.2 Add match normalization to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed capture `id` back to the model `name` surfaced as `match.rule`. -- [ ] 2.3 Gate on matches: when the narrow yields zero matches, do NOT invoke `check.ts`. -- [ ] 2.4 Add `packages/cli/src/rules/runtime/invoke.ts`: call `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`; use the returned `Finding[]`. Isolate a throwing check to a single error-severity finding for that rule. -- [ ] 2.5 Decide and implement scheduling (process-per-check vs. `worker_threads` pool vs. `import()`); bound each check with a default wall-clock timeout (overridable via `--timeout `) that terminates the check and records a single error-severity finding. -- [ ] 2.6 Map each `Finding` (`severity ∈ error|warning|info`) onto `CheckResult` with a runtime `source`; feed into the existing aggregation and exit-code logic. - -## 3. tsx bundling - -- [ ] 3.1 Add a pinned `tsx` (or equivalent loader) to the CLI bundle and resolve its path at runtime without assuming any repo-local toolchain. -- [ ] 3.2 Verify `check.ts` executes with no `node_modules` and no precompile in a temp-dir fixture. - -## 4. Reconcile scoping & materialization - -- [ ] 4.1 Extend `src/rules/run-set.ts` to enumerate `.taskless/runtime-rules/` and sign each rule's `check.ts` only (capture `*.yml` and static rules are inert — not reported to reconcile). -- [ ] 4.2 Compute per-rule eligibility: a runtime rule executes only if its `check.ts` is in `run`; otherwise withhold and surface as advisory. -- [ ] 4.3 Materialize blessed runtime rules into `.taskless/.run/` and execute `check.ts` from there (read-hash-execute); ensure `.taskless/.gitignore` still covers `.run/`. - -## 5. check dispatch & modes - -- [ ] 5.1 In `src/commands/check.ts`, split the corpus via `classifyRules`; always run static rules; route runtime rules through the harness only on a validated path. -- [ ] 5.2 Implement the mode table: authed/API-key → reconcile & run fully-blessed runtime rules; logged-out/`--anonymous` → skip runtime + report skipped; reconcile-cannot-complete → skip runtime with notice; `--dangerously-run-scripts` → run all runtime rules trusting local signatures. -- [ ] 5.3 Add the `--dangerously-run-scripts` flag: skip reconciliation entirely (no network) and run all present runtime rules, behind a prominent unverified-execution warning (stderr, suppressed under `--json`). Add the `--timeout ` flag. -- [ ] 5.4 Narrow the stacked-under degrade path so it scans static rules but never executes runtime `check.ts`. -- [ ] 5.5 Emit skipped-runtime notices (human output); under `--json` add an additive optional `skipped: [{ rule, reason }]` field leaving `success`/`results` unchanged. Confirm the exit code stays governed solely by error-severity findings. - -## 6. Help & docs - -- [ ] 6.1 Update `packages/cli/src/help/check.txt` with a runtime-rule section: what runs per mode, and the `--dangerously-run-scripts` warning. -- [ ] 6.2 Update `packages/cli/src/help/ci.txt` to note the enforced (`--enforce`) sandbox as the authoritative runtime-rule enforcement point. - -## 7. Tests & verification - -- [ ] 7.1 Harness unit tests: narrow assembly (anchor/broad), gate-on-zero-matches (check never invoked), match normalization (hashed id → model name), `Finding` → `CheckResult` mapping, throwing-check isolation, timeout → error finding. -- [ ] 7.2 Mode integration tests (subprocess against `dist/`, temp-dir fixtures with a mock reconcile server + git origin): authed runs blessed runtime rules; partially-blessed rule withheld; logged-out and `--anonymous` skip runtime + report skipped; reconcile-unavailable skips runtime; `--dangerously-run-scripts` runs runtime offline with a warning. -- [ ] 7.3 Static-always-run tests: static rules run in every mode; static rules are not reported to reconcile. -- [ ] 7.4 `--json` shape tests: warnings/notices suppressed; runtime findings appear under the same `results` shape as static findings. -- [ ] 7.5 Run `pnpm typecheck`, `pnpm lint`, and the full `pnpm test` — all green. diff --git a/openspec/specs/cli-check/spec.md b/openspec/specs/cli-check/spec.md index b3bf88ea..f6cde842 100644 --- a/openspec/specs/cli-check/spec.md +++ b/openspec/specs/cli-check/spec.md @@ -247,83 +247,164 @@ When `taskless check --json` exits with an error, the output SHALL conform to th ### Requirement: Check selects what it runs from auth state -`taskless check` SHALL NOT require authentication, and it SHALL choose what it runs from the current auth state. When no token is available, or when `--anonymous` is set, the CLI SHALL run all local rule files under `.taskless/rules/` without contacting the server (the offline linter posture). When a token is available and `--anonymous` is not set, the CLI SHALL reconcile against the server (per "Check reconciles rule files before scanning") and run only the returned `run` set. The unauthenticated path SHALL succeed with no network access and SHALL NOT emit a warning; any informational notice on that path is optional. - -#### Scenario: Unauthenticated check runs local rules +`taskless check` SHALL NOT require authentication, and it SHALL choose what it runs from the +current auth state. **Static ast-grep rules** (single `*.yml` files under `.taskless/rules/`) +SHALL always run without contacting the server, on every path (the offline linter posture). +**Runtime rules** (directories with `metadata.taskless.kind: runtime`) SHALL run only on a +signature-validated path: when a token is available and `--anonymous` is not set the CLI SHALL +reconcile and execute the runtime rules fully returned in `run`; when no token is available, +when `--anonymous` is set, or when reconciliation cannot complete, the CLI SHALL skip runtime +execution unless `--dangerously-run-scripts` is set. The unauthenticated path SHALL succeed +with no network access for static rules and SHALL NOT emit a warning about missing +authentication. + +#### Scenario: Unauthenticated check runs static rules and skips runtime rules - **WHEN** a user runs `taskless check` with no available token -- **THEN** the CLI SHALL scan all local rule files -- **AND** SHALL NOT call `POST /cli/api/reconcile` +- **THEN** the CLI SHALL scan all static rule files +- **AND** SHALL NOT call `POST /cli/api/reconcile` for the purpose of running static rules +- **AND** SHALL skip runtime rules - **AND** SHALL NOT emit a warning about missing authentication -#### Scenario: Authenticated check reconciles +#### Scenario: Authenticated check reconciles runtime rules - **WHEN** a user runs `taskless check` with an available token and without `--anonymous` -- **THEN** the CLI SHALL call `POST /cli/api/reconcile` and run only the `run` set +- **THEN** the CLI SHALL reconcile and execute the runtime rules fully returned in `run` +- **AND** SHALL run static rules unconditionally #### Scenario: Anonymous forces the logged-out path - **WHEN** a user runs `taskless check --anonymous` while a token is available -- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (run all local rules, no reconcile call) +- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (static rules run, runtime rules skipped, no reconcile call) ### Requirement: Check reconciles rule files before scanning -`taskless check` SHALL reconcile before scanning whenever a bearer token and a `repositoryUrl` are resolvable and `--anonymous` is not set. It SHALL compute the signature envelope for every `.yml` file under `.taskless/rules/`, call `POST /cli/api/reconcile` with `{ repositoryUrl, files }`, and then scan **only** the files returned in the `run` set, matched back to local files by signature (per the `cli-rule-reconciliation` capability). +`taskless check` SHALL reconcile before running runtime rules whenever a bearer token and a +`repositoryUrl` are resolvable and `--anonymous` is not set. It SHALL compute the signature +envelope for the `check.ts` of every runtime rule under `.taskless/runtime-rules/`, call +`POST /cli/api/reconcile` with `{ repositoryUrl, files }`, and then execute **only** the +runtime rules whose `check.ts` is returned in the `run` set, matched back to local files by +signature (per the `cli-rule-reconciliation` capability). Capture `*.yml` and static rules +SHALL NOT be gated by reconciliation. -#### Scenario: Only blessed rules are scanned +#### Scenario: Only rules with a blessed check.ts execute -- **WHEN** a user runs `taskless check` while authenticated and reconciliation returns a - `run` set that is a subset of the local rule files -- **THEN** the CLI SHALL invoke `sg scan` against only the `run`-set rule files -- **AND** SHALL NOT scan any local rule file absent from `run` +- **WHEN** a user runs `taskless check` while authenticated and reconciliation returns a `run` + set covering the `check.ts` of some runtime rules and not others +- **THEN** the CLI SHALL execute only the runtime rules whose `check.ts` is in `run` +- **AND** SHALL run all static rules regardless of the `run` set -### Requirement: Check warns on reconciliation mismatches +### Requirement: Check degrades to a local scan when reconciliation cannot complete -`taskless check` SHALL warn on every mismatch reported by a successful reconciliation and SHALL NOT let those warnings change the exit code. `unsafe` entries SHALL be warned as tamper/drift, `unknown` as not-issued-by-the-server, and `missing` as audit-only. The exit code SHALL continue to be governed solely by error-severity scan results from the executed `run` set. Warnings SHALL be human-readable output only; when `--json` is set the output SHALL remain the existing `{ success, results }` shape and SHALL NOT include the warnings. +`taskless check` SHALL NOT fail solely because an attempted reconciliation cannot complete. +When a token is available and `--anonymous` is not set but reconciliation cannot complete (no +resolvable git remote, or the reconcile endpoint is unreachable or not yet deployed, or a +transport error), the CLI SHALL warn that rule verification could not be performed, SHALL fall +back to scanning all **static** rule files, and SHALL **skip runtime rules** (their `check.ts` +SHALL NOT run) unless `--dangerously-run-scripts` is set. The CLI SHALL NOT exit with a +non-zero code solely because reconciliation failed, and the warning SHALL be suppressed under +`--json`. -#### Scenario: Unsafe drift is warned without failing the exit code +#### Scenario: Endpoint unreachable degrades static and skips runtime -- **WHEN** reconciliation returns an `unsafe` entry and the `run`-set scan produces no - error-severity results -- **THEN** the CLI SHALL warn about the drift -- **AND** SHALL exit with code 0 +- **WHEN** an authenticated `check` attempts reconciliation and the endpoint is unreachable or returns a not-deployed error +- **THEN** the CLI SHALL warn that verification could not be performed +- **AND** SHALL scan all static rule files +- **AND** SHALL NOT execute any runtime rule's `check.ts` +- **AND** SHALL NOT exit with a non-zero code solely due to the reconcile failure -#### Scenario: Missing rules are warned as audit-only +#### Scenario: Degrade warning is suppressed under --json -- **WHEN** reconciliation returns `missing` entries -- **THEN** the CLI SHALL warn about them as audit information -- **AND** SHALL NOT change the exit code on their account +- **WHEN** the CLI degrades and `--json` is set +- **THEN** stdout SHALL contain the machine JSON shape (`{ success, results }` plus the additive optional `skipped` array for the skipped runtime rules) +- **AND** SHALL NOT contain the human-readable degrade warning -#### Scenario: Warnings are suppressed under --json +### Requirement: Check dispatches static and runtime rules to distinct executors -- **WHEN** reconciliation returns mismatches and `--json` is set -- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape -- **AND** SHALL NOT contain the human-readable mismatch warnings +`taskless check` SHALL execute **static** ast-grep rules under `.taskless/rules/` with the +ast-grep scanner as before, and **runtime** rules under `.taskless/runtime-rules/` (directories +with `metadata.taskless.kind: runtime`, per the `cli-runtime-rule-execution` capability) with +the runtime harness. Findings from both executors SHALL be aggregated into the same result set +and SHALL count toward the exit code identically. -### Requirement: Check degrades to a local scan when reconciliation cannot complete +#### Scenario: Mixed corpus runs both executors -`taskless check` SHALL NOT fail solely because an attempted reconciliation cannot complete. When a token is available and `--anonymous` is not set but reconciliation cannot complete (no resolvable git remote, or the reconcile endpoint is unreachable or not yet deployed, or a transport error), the CLI SHALL warn that rule verification could not be performed and SHALL fall back to scanning all local rule files. The CLI SHALL NOT exit with a non-zero code solely because reconciliation failed, and the warning SHALL be suppressed under `--json`. +- **WHEN** `.taskless/rules/` contains static rules and `.taskless/runtime-rules/` contains runtime rules +- **THEN** the CLI SHALL run static rules through `sg scan` and runtime rules through the runtime harness +- **AND** SHALL merge their findings into one result set -#### Scenario: Endpoint unreachable degrades to local scan +### Requirement: Check runs runtime rules only on a signature-validated path -- **WHEN** an authenticated `check` attempts reconciliation and the endpoint is unreachable or returns a not-deployed error -- **THEN** the CLI SHALL warn that verification could not be performed -- **AND** SHALL scan all local rule files -- **AND** SHALL NOT exit with a non-zero code solely due to the reconcile failure +`taskless check` SHALL execute a runtime rule's `check.ts` only when that `check.ts` has been +validated by the server — returned in the reconciliation `run` set — or when +`--dangerously-run-scripts` is set. When a token is available and `--anonymous` is not set, the +CLI SHALL reconcile and execute every runtime rule whose `check.ts` is in `run`. An API key +SHALL be treated identically to an interactive token. On any path where the runtime rule's +`check.ts` signature is not validated — logged out, `--anonymous`, or a reconciliation that +cannot complete — the CLI SHALL NOT execute the rule's `check.ts`. -#### Scenario: Degrade warning is suppressed under --json +#### Scenario: Authenticated check runs blessed runtime rules -- **WHEN** the CLI degrades to a local scan and `--json` is set -- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape -- **AND** SHALL NOT contain the human-readable degrade warning +- **WHEN** an authenticated `check` reconciles and a runtime rule's `check.ts` is returned in `run` +- **THEN** the CLI SHALL execute that runtime rule through the harness -### Requirement: Check exits cleanly when the run set is empty +#### Scenario: A rule whose check.ts is not blessed is withheld -`taskless check` SHALL NOT invoke the scanner when a successful reconciliation returns an empty `run` set (for example an empty corpus, or every reported file classified `unsafe`/`unknown`); it SHALL report no scan results and SHALL exit with code 0. Any `unsafe` / `unknown` mismatch warnings SHALL still be emitted. +- **WHEN** reconciliation does not return a runtime rule's `check.ts` in `run` (it lands in `unsafe`/`unknown`/`missing`) +- **THEN** the CLI SHALL NOT execute that runtime rule +- **AND** SHALL surface it as an advisory mismatch -#### Scenario: Empty run set skips the scan +#### Scenario: API key behaves like a token -- **WHEN** reconciliation returns an empty `run` set -- **THEN** the CLI SHALL NOT invoke `sg scan` -- **AND** SHALL exit with code 0 -- **AND** SHALL still warn about any `unsafe`/`unknown` mismatches +- **WHEN** `check` runs with an API key +- **THEN** the CLI SHALL reconcile and run validated runtime rules exactly as with an interactive token + +### Requirement: Check skips runtime rules on unverified paths and reports the skip + +`taskless check` SHALL skip a runtime rule's execution when it cannot validate the rule's +signature — logged out, `--anonymous`, or reconciliation cannot complete — and +`--dangerously-run-scripts` is not set. It SHALL report that runtime rules exist and were not +run, SHALL still run static rules on these paths, and SHALL NOT change the exit code because +rules were skipped. In human output the report SHALL be a notice; under `--json` it SHALL be an +additive, optional `skipped` array of `{ rule, reason }`, leaving the existing `success` and +`results` fields unchanged, so machine callers (for example CI) can detect that runtime rules +did not run. + +#### Scenario: Logged-out check skips runtime rules + +- **WHEN** a user runs `taskless check` with no available token and runtime rules are present +- **THEN** the CLI SHALL run static rules +- **AND** SHALL NOT execute any runtime rule's `check.ts` +- **AND** SHALL report that the runtime rules were skipped +- **AND** SHALL NOT change the exit code because rules were skipped + +#### Scenario: Skipped runtime rules appear under --json + +- **WHEN** runtime rules are skipped and `--json` is set +- **THEN** stdout SHALL include a `skipped` array of `{ rule, reason }` alongside the unchanged `success` and `results` fields + +#### Scenario: Anonymous skips runtime rules while authenticated + +- **WHEN** a user runs `taskless check --anonymous` while a token is available +- **THEN** the CLI SHALL skip runtime rules exactly as an unauthenticated `check` + +### Requirement: Check accepts --dangerously-run-scripts to run runtime rules without server validation + +`taskless check` SHALL accept a `--dangerously-run-scripts` flag that runs **all** runtime +rules without server validation, regardless of auth state. +When the flag is set the CLI SHALL NOT reconcile — it SHALL skip the network entirely (matching +how `--anonymous` forces the no-network path) and execute every present runtime rule. The CLI +SHALL emit a prominent warning that runtime rule code is being executed unverified. The flag +SHALL be the only way to execute runtime rules on an unverified path. + +#### Scenario: Dangerously-run-scripts executes runtime rules offline + +- **WHEN** a user runs `taskless check --dangerously-run-scripts` with no available token +- **THEN** the CLI SHALL execute the present runtime rules' `check.ts` +- **AND** SHALL emit a warning that runtime rule code ran unverified + +#### Scenario: Warning is suppressed under --json + +- **WHEN** `--dangerously-run-scripts` and `--json` are both set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** the unverified-execution warning SHALL NOT appear in stdout diff --git a/openspec/specs/cli-rule-reconciliation/spec.md b/openspec/specs/cli-rule-reconciliation/spec.md index 3f37b76a..5d4c1170 100644 --- a/openspec/specs/cli-rule-reconciliation/spec.md +++ b/openspec/specs/cli-rule-reconciliation/spec.md @@ -104,20 +104,28 @@ be a release blocker (the test SHALL fail the build). - **THEN** the test SHALL fail - **AND** the build SHALL NOT pass -### Requirement: Reconcile reports every held rule file +### Requirement: Reconcile reports every runtime rule's check.ts -The CLI SHALL reconcile by sending `POST /cli/api/reconcile` with an -`Authorization: Bearer ` header and a JSON body `{ repositoryUrl, files }`, where -`repositoryUrl` is the full repository URL and `files` is an array of -`{ file, signature }` covering **every** rule file the CLI would otherwise run under -`.taskless/rules/`. `file` SHALL be the rule file's delivered name as it exists on disk and -`signature` SHALL be the full envelope computed for that file's bytes. The CLI SHALL send the -whole signature envelope, not a bare digest. +Reconciliation SHALL be scoped to the **`check.ts` of runtime rules** — the only artifact that +carries arbitrary code execution. Static ast-grep rules and runtime-rule capture `*.yml` are +inert data, always available, and SHALL NOT be reported to or gated by reconciliation. The CLI +SHALL reconcile by sending `POST /cli/api/reconcile` with an `Authorization: Bearer ` +header and a JSON body `{ repositoryUrl, files }`, where `repositoryUrl` is the full repository +URL and `files` is an array of `{ file, signature }` covering the `check.ts` of **every** +runtime rule the CLI holds under `.taskless/runtime-rules/`. `file` SHALL be the `check.ts`'s +delivered path as it exists on disk and `signature` SHALL be the full envelope computed for its +bytes. The CLI SHALL send the whole signature envelope, not a bare digest. -#### Scenario: All rule files are reported +#### Scenario: Every runtime rule's check.ts is reported -- **WHEN** the CLI reconciles with rule files present under `.taskless/rules/` -- **THEN** the request body SHALL include one `{ file, signature }` entry per rule file +- **WHEN** the CLI reconciles with runtime rules present under `.taskless/runtime-rules/` +- **THEN** the request body SHALL include one `{ file, signature }` entry for the `check.ts` of each runtime rule + +#### Scenario: Inert files are not reported + +- **WHEN** the CLI reconciles and static `*.yml` rules and capture `*.yml` are also present +- **THEN** the request body SHALL NOT include entries for those inert files +- **AND** the static rules SHALL run regardless of the reconcile response #### Scenario: Full envelope is sent @@ -165,20 +173,22 @@ join), so a file that was moved but not changed still resolves. ### Requirement: The CLI executes only the server run set -When reconciliation succeeds, the CLI SHALL execute exactly the files in the server's `run` -set and no other file. The CLI SHALL NOT run a file on the basis of its own local comparison -of signatures. This replaces any local classification of rules. +When reconciliation succeeds, the CLI SHALL execute a runtime rule only if its **`check.ts`** +is present in the server's `run` set, and SHALL execute no runtime rule whose `check.ts` is +absent from `run`. The CLI SHALL NOT run a runtime rule on the basis of its own local +comparison of signatures. This replaces any local classification of runtime rules. Static +ast-grep rules and capture `*.yml` are outside this gate. -#### Scenario: Only run-set files execute +#### Scenario: Only rules with a blessed check.ts execute -- **WHEN** reconciliation returns a `run` set that is a strict subset of the reported files -- **THEN** the CLI SHALL execute only the files in `run` -- **AND** SHALL NOT execute any reported file absent from `run` +- **WHEN** reconciliation returns a `run` set covering the `check.ts` of some runtime rules but not others +- **THEN** the CLI SHALL execute only the runtime rules whose `check.ts` is in `run` +- **AND** SHALL withhold any runtime rule whose `check.ts` is absent from `run` #### Scenario: No local self-classification -- **WHEN** the CLI holds a local signature or sidecar for a file -- **THEN** it SHALL NOT treat that local value as authorization to run the file +- **WHEN** the CLI holds a local signature or sidecar for a runtime rule's `check.ts` +- **THEN** it SHALL NOT treat that local value as authorization to execute the rule ### Requirement: Reconcile is scoped to the token's organization diff --git a/openspec/specs/cli-runtime-rule-execution/spec.md b/openspec/specs/cli-runtime-rule-execution/spec.md new file mode 100644 index 00000000..263516f5 --- /dev/null +++ b/openspec/specs/cli-runtime-rule-execution/spec.md @@ -0,0 +1,143 @@ +# CLI Runtime Rule Execution + +## Purpose + +Defines how the CLI executes a **runtime rule** in `taskless check`: the on-disk shape it recognizes, the local harness that runs the rule's ast-grep capture narrow and gates on matches, how it invokes the rule's `check.ts` as a function under a bundled TypeScript loader, and how the returned findings map onto the scanner-agnostic result type. A runtime rule's `check.ts` is arbitrary code; the reconciliation gate that authorizes running it is defined in `cli-rule-reconciliation`, and the auth-state dispatch is defined in `cli-check`. + +## Requirements + +### Requirement: Runtime rules are directories recognized by metadata + +The CLI SHALL recognize a **runtime rule** as a directory under `.taskless/runtime-rules//` +containing one or more ast-grep capture `*.yml` (one per capture rule) and a single `check.ts`, +its capture rules declaring `metadata.taskless.kind: runtime`. The rule's check file SHALL be +the `check.ts` in the rule directory. The CLI SHALL read **each capture rule's** +`metadata.taskless.match` (`anchor` or `broad`) to select that capture rule's ast-grep +invocation mode; capture rules within one runtime rule MAY mix modes (each is independent). +Rule files under `.taskless/rules/` SHALL +continue to be treated as static ast-grep rules, not runtime rules. +`.taskless/runtime-rule-tests//` holds `valid/` and `invalid/` verification fixtures and +SHALL NOT be executed by `check`. + +#### Scenario: A runtime-rules directory entry is a runtime rule + +- **WHEN** `.taskless/runtime-rules//` contains capture `*.yml` with `metadata.taskless.kind: runtime` and a `check.ts` +- **THEN** the CLI SHALL treat it as a runtime rule with `check.ts` as its check file and route it to the runtime harness + +#### Scenario: Rules under .taskless/rules remain static + +- **WHEN** a rule file lives under `.taskless/rules/` +- **THEN** the CLI SHALL treat it as a static rule and SHALL NOT route it to the runtime harness + +### Requirement: The harness narrows with one ast-grep scan and gates on matches + +For a runtime rule the CLI SHALL assemble the rule's capture rules into an ast-grep +configuration and run **one scan per mode** as the narrow: `--json=stream` for `anchor` capture +rules, and `--files-with-matches` for `broad` capture rules (whole-language `kind: program` +enumerators). A rule with only `anchor` capture rules therefore runs in a single scan; a rule +mixing modes runs one scan per mode. When the narrow produces **zero** matches the CLI SHALL +NOT invoke `check.ts`. + +#### Scenario: Zero matches skips the check + +- **WHEN** a runtime rule's narrow scan produces no matches +- **THEN** the CLI SHALL NOT invoke that rule's `check.ts` +- **AND** the rule SHALL contribute no findings + +#### Scenario: Capture rules of a mode run together + +- **WHEN** a runtime rule has multiple `anchor` capture rules +- **THEN** the CLI SHALL run them in a single `ast-grep` scan, not one scan per capture rule + +### Requirement: Matches are normalized and attributed to the model name + +The CLI SHALL normalize every narrow match to +`{ rule, ruleId, file, line, column, text, captures }`, where `file` is root-relative, `line` +is 1-indexed, and `rule` is the capture rule's stable model-assigned `name`. The CLI SHALL map +the hashed capture-rule `id` used by the scan back to that `name` so `match.rule` is the value +the check branches on, never the hash. A `broad` (path-only, `--files-with-matches`) match +carries no location or captures: its `line` and `column` SHALL be `1`, and its `text` and +`captures` SHALL be empty. + +#### Scenario: Hashed id maps to model name + +- **WHEN** the scan emits a match whose rule id is the hashed `${ruleSlug}-${sha1}` identifier +- **THEN** the normalized match's `rule` SHALL be the capture rule's model-assigned `name` + +### Requirement: The check is invoked as a function and its return value is used + +The CLI SHALL invoke a runtime rule's `check.ts` by calling its **default export** as a +function with `(root, matches)`, where `root` is the repository root and `matches` are the +normalized matches. The CLI SHALL use the `Finding[]` value the function **returns** as the +rule's result; it SHALL NOT infer results from process exit codes or stdout. A `check.ts` that +throws SHALL be isolated to a single error-severity finding for that rule and SHALL NOT abort +the overall `check` run. + +#### Scenario: Returned findings are the result + +- **WHEN** a runtime rule's `check.ts` default export returns a `Finding[]` +- **THEN** the CLI SHALL treat exactly those findings as the rule's output + +#### Scenario: A throwing check is isolated + +- **WHEN** a runtime rule's `check.ts` throws during execution +- **THEN** the CLI SHALL record a single error-severity finding for that rule +- **AND** SHALL continue executing the remaining rules and produce output + +### Requirement: check.ts execution is bounded by a timeout + +The CLI SHALL bound each `check.ts` invocation with a wall-clock timeout and SHALL accept a +`--timeout ` flag on `check` to override the default. When a `check.ts` exceeds the +timeout the CLI SHALL terminate it, record a single error-severity finding for that rule, and +continue executing the remaining rules — a runaway check SHALL NOT wedge the overall `check` +run. + +#### Scenario: A hanging check is terminated at the timeout + +- **WHEN** a runtime rule's `check.ts` runs longer than the effective timeout +- **THEN** the CLI SHALL terminate it and record a single error-severity finding for that rule +- **AND** SHALL continue executing the remaining rules + +#### Scenario: --timeout overrides the default + +- **WHEN** a user runs `taskless check --timeout ` +- **THEN** the CLI SHALL use that value as the per-check wall-clock bound + +### Requirement: check.ts runs via a bundled TypeScript loader + +The CLI SHALL execute `check.ts` using a pinned TypeScript loader bundled with the CLI (e.g. +`tsx`) and SHALL NOT require the user's repository to provide a TypeScript toolchain, +`node_modules`, or a precompile step. The CLI MAY schedule invocations by any mechanism +(process-per-check, worker pool, or in-process import); the function contract does not +constrain the choice. + +#### Scenario: No user toolchain required + +- **WHEN** a repository with a runtime rule has no local TypeScript toolchain installed +- **THEN** the CLI SHALL still execute the rule's `check.ts` using its bundled loader + +### Requirement: Findings map onto the scanner-agnostic result type + +The CLI SHALL map each `Finding` returned by a `check.ts` onto the existing `CheckResult` +shape with a runtime `source`, so runtime findings are aggregated, formatted, and counted +toward the exit code identically to static findings. `Finding.severity` (`error` / `warning` / +`info`) SHALL map directly onto the corresponding `CheckResult` severity with no translation. + +#### Scenario: Runtime findings gate the exit code like static findings + +- **WHEN** a runtime rule returns a finding with `severity: "error"` +- **THEN** the CLI SHALL count it toward the error total that sets a non-zero exit code +- **AND** the finding SHALL appear in `--json` output under the same `results` shape as a static finding + +### Requirement: Blessed runtime rules execute from the materialized run directory + +When a runtime rule is executed on a validated path, the CLI SHALL execute it from the +ephemeral, gitignored `.taskless/.run/` materialization of the blessed bytes, not from the +live `.taskless/runtime-rules/` tree, so the bytes executed are the exact bytes reconciliation +blessed (read-hash-execute ordering). + +#### Scenario: Execution uses the blessed bytes + +- **WHEN** a runtime rule is blessed and executed +- **THEN** the CLI SHALL invoke the `check.ts` materialized under `.taskless/.run/` +- **AND** SHALL NOT execute a copy modified in `.taskless/runtime-rules/` after reconciliation diff --git a/packages/cli/package.json b/packages/cli/package.json index 9add16a3..e54dd4ac 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -42,6 +42,7 @@ "posthog-node": "^5.28.11", "smol-toml": "^1.6.1", "sprintf-js": "^1.1.3", + "tsx": "^4.21.0", "yaml": "^2.8.2", "zod": "^4.3.6" }, diff --git a/packages/cli/src/commands/check.ts b/packages/cli/src/commands/check.ts index 86ba9ea3..b2021c84 100644 --- a/packages/cli/src/commands/check.ts +++ b/packages/cli/src/commands/check.ts @@ -12,14 +12,18 @@ import { makeErrorEnvelope } from "../types/errors"; import { getToken } from "../auth/token"; import { resolveRepositoryUrl } from "../util/git-remote"; import { getCliPrefix } from "../util/package-manager"; -import { reconcile, type ReconcileResponse } from "../api/reconcile"; +import { reconcile } from "../api/reconcile"; import { - RUN_RULES_DIR, - materializeRunDirectory, - selectRunFiles, - signRuleFiles, - type SignedRuleFile, -} from "../rules/run-set"; + discoverRuntimeRules, + type RuntimeRule, +} from "../rules/runtime/discover"; +import { + materializeRuntimeRules, + reportRuntimeChecks, + selectBlessedRuntimeRules, + signRuntimeChecks, +} from "../rules/runtime/run-set"; +import { executeRuntimeRules } from "../rules/runtime/harness"; async function pathExists(absolutePath: string): Promise { try { @@ -83,7 +87,9 @@ function extractPositionalPaths(rawArguments: string[]): string[] { if (argument.startsWith("-")) { // Skip value for short/long flags that take a value if ( - (argument === "-d" || argument === "--dir") && + (argument === "-d" || + argument === "--dir" || + argument === "--timeout") && index + 1 < rawArguments.length ) { index += 1; @@ -95,79 +101,141 @@ function extractPositionalPaths(rawArguments: string[]): string[] { return paths; } +/** A runtime rule that will not run, with why (advisory). */ +interface SkippedRuntimeRule { + rule: string; + reason: string; +} + +/** The runtime-execution plan resolved from auth state and flags. */ +interface RuntimePlan { + /** Rules to execute — materialized when gated, live under `--dangerously-run-scripts`. */ + execute: RuntimeRule[]; + /** Rules that will not run, with a reason. */ + skipped: SkippedRuntimeRule[]; + /** Human-only notices about the runtime disposition. */ + notices: string[]; +} + +/** Skip every runtime rule with a shared reason (an unverified path). */ +function skipAllRuntime(rules: RuntimeRule[], reason: string): RuntimePlan { + return { + execute: [], + skipped: rules.map((rule) => ({ rule: rule.name, reason })), + notices: [], + }; +} + /** - * How `check` will scan, decided from auth state: - * - `local`: unauthenticated or `--anonymous` — scan all local rules silently. - * - `degrade`: authenticated but reconciliation could not complete — scan all - * local rules and warn that verification could not be performed. - * - `gated`: reconciliation succeeded — scan only the blessed `run` set. + * Decide which runtime rules run. A runtime rule's `check.ts` is arbitrary code + * execution, so it runs only when its signature is server-validated (an + * authenticated reconcile that returns it in `run`) or `--dangerously-run-scripts` + * is set. Every unverified path — anonymous, logged out, no remote, or a + * reconcile that cannot complete — skips runtime rules without failing. */ -type ScanMode = - | { kind: "local" } - | { kind: "degrade"; reason: string } - | { kind: "gated"; result: ReconcileResponse; signed: SignedRuleFile[] }; - -async function resolveScanMode( +async function planRuntime( cwd: string, - anonymous: boolean, - ruleFiles: string[] -): Promise { - if (anonymous) return { kind: "local" }; + discovered: RuntimeRule[], + options: { anonymous: boolean; dangerouslyRunScripts: boolean } +): Promise { + if (discovered.length === 0) return { execute: [], skipped: [], notices: [] }; + + if (options.dangerouslyRunScripts) { + return { + execute: discovered, + skipped: [], + notices: [ + "Warning: --dangerously-run-scripts is executing runtime rule code without server verification.", + ], + }; + } + + if (options.anonymous) { + return skipAllRuntime( + discovered, + "anonymous mode — runtime rules were not verified and did not run" + ); + } const token = await getToken(cwd, { silent: true }); - if (!token) return { kind: "local" }; + if (!token) { + return skipAllRuntime( + discovered, + "not authenticated — runtime rules were not verified and did not run" + ); + } let repositoryUrl: string; try { repositoryUrl = await resolveRepositoryUrl(cwd); } catch { - return { - kind: "degrade", - reason: "no GitHub remote was found, so rules could not be verified", - }; + return skipAllRuntime( + discovered, + "no GitHub remote — runtime rules could not be verified and did not run" + ); } - const signed = await signRuleFiles(cwd, ruleFiles); + // A rule whose check.ts is missing/unreadable is reported, not fatal: signing + // never throws, and such rules are surfaced as skipped so static checks and + // the other runtime rules are unaffected. + const { signed, unreadable } = await signRuntimeChecks(discovered); + const unreadableSkips: SkippedRuntimeRule[] = unreadable.map((rule) => ({ + rule: rule.name, + reason: "its check.ts is missing or unreadable", + })); + const outcome = await reconcile(token, { repositoryUrl, - files: signed.map(({ file, signature }) => ({ file, signature })), + files: reportRuntimeChecks(cwd, signed), }); - if (outcome.status === "ok") { - return { kind: "gated", result: outcome.result, signed }; - } if (outcome.status === "unauthorized") { - return { - kind: "degrade", - reason: `authentication was rejected — run \`${getCliPrefix()} auth login\` to re-authenticate`, - }; - } - return { - kind: "degrade", - reason: `the rule service was unavailable (${outcome.reason})`, - }; -} - -/** Emit advisory reconciliation warnings (human output only). */ -function surfaceReconcileWarnings( - result: ReconcileResponse, - warn: (message: string) => void -): void { - for (const entry of result.unsafe) { - warn( - `Warning: ${entry.file} differs from the blessed rule (tamper/drift) and will not run.` + return skipAllRuntime( + discovered, + `authentication was rejected — run \`${getCliPrefix()} auth login\` to re-authenticate` ); } - for (const entry of result.unknown) { - warn( - `Notice: ${entry.file} was not issued by the server and will not run.` + if (outcome.status === "unavailable") { + return skipAllRuntime( + discovered, + `the rule service was unavailable (${outcome.reason})` ); } - for (const entry of result.missing) { - warn( - `Notice: expected rule ${entry.ruleId} (${entry.file}) is not present locally.` + + const { blessed, withheld } = selectBlessedRuntimeRules( + signed, + outcome.result.run + ); + let execute: RuntimeRule[] = []; + try { + execute = + blessed.length > 0 ? await materializeRuntimeRules(cwd, blessed) : []; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + return skipAllRuntime( + discovered, + `runtime rules could not be materialized (${message})` ); } + return { + execute, + skipped: [ + ...unreadableSkips, + ...withheld.map((rule) => ({ + rule: rule.name, + reason: "not blessed by the server (unsafe / unknown / drift)", + })), + ], + notices: [], + }; +} + +/** Parse `--timeout ` into milliseconds; invalid/absent → undefined (default). */ +function parseTimeoutMs(raw: string | undefined): number | undefined { + if (raw === undefined) return undefined; + const seconds = Number(raw); + if (!Number.isFinite(seconds) || seconds <= 0) return undefined; + return Math.round(seconds * 1000); } export const checkCommand = defineCommand({ @@ -188,16 +256,27 @@ export const checkCommand = defineCommand({ }, anonymous: { type: "boolean", - description: "Skip server reconciliation and scan local rules unverified", + description: + "Run only trusted static rules; skip runtime rules (no reconciliation)", + default: false, + }, + "dangerously-run-scripts": { + type: "boolean", + description: + "Run runtime-rule check.ts without server verification (executes untrusted code)", default: false, }, + timeout: { + type: "string", + description: "Per-runtime-check timeout in seconds (default 10)", + }, }, async run({ args, rawArgs }) { const cwd = resolve(args.dir ?? process.cwd()); const telemetry = await getTelemetry(cwd); - // Warnings are advisory human output; suppress them under --json so the - // machine output stays the existing { success, results } shape. + // Warnings/notices are advisory human output; suppress them under --json so + // the machine output stays the { success, results, skipped? } shape. const warn = (message: string) => { if (!args.json) console.error(message); }; @@ -227,17 +306,19 @@ export const checkCommand = defineCommand({ return; } - // Check for rule files + // Static rules (trusted ast-grep YAML) always run; runtime rules + // (untrusted check.ts) are gated separately. const rulesDirectory = join(cwd, ".taskless", "rules"); - let ruleFiles: string[] = []; + let staticRuleFiles: string[] = []; try { const entries = await readdir(rulesDirectory); - ruleFiles = entries.filter((f) => f.endsWith(".yml")); + staticRuleFiles = entries.filter((f) => f.endsWith(".yml")); } catch { // .taskless/ or rules/ directory doesn't exist } + const runtimeRules = await discoverRuntimeRules(cwd); - if (ruleFiles.length === 0) { + if (staticRuleFiles.length === 0 && runtimeRules.length === 0) { if (args.json) { console.log( JSON.stringify( @@ -252,30 +333,33 @@ export const checkCommand = defineCommand({ return; } - // Decide what to run from auth state, then scan. try { - const mode = await resolveScanMode(cwd, args.anonymous, ruleFiles); + const results: CheckResult[] = []; - let results: CheckResult[] = []; - if (mode.kind === "gated") { - surfaceReconcileWarnings(mode.result, warn); - const runFiles = selectRunFiles(mode.signed, mode.result.run); - // Empty run set (empty corpus, or all unsafe/unknown): run nothing. - if (runFiles.length > 0) { - await materializeRunDirectory(cwd, runFiles); - await generateSgConfig(cwd, { rulesDirectory: RUN_RULES_DIR }); - const scan = await runAstGrepScan(cwd, existingPaths); - results = scan.results; - } - } else { - if (mode.kind === "degrade") { - warn( - `Notice: ${mode.reason}. Scanning all local rules unverified; the CI backstop enforces the server-owned rule set.` - ); - } + // Static rules: always scan, no verification (inert data). + if (staticRuleFiles.length > 0) { await generateSgConfig(cwd); const scan = await runAstGrepScan(cwd, existingPaths); - results = scan.results; + results.push(...scan.results); + } + + // Runtime rules: run only what the server validated (or forced). + const plan = await planRuntime(cwd, runtimeRules, { + anonymous: args.anonymous, + dangerouslyRunScripts: Boolean(args["dangerously-run-scripts"]), + }); + for (const notice of plan.notices) warn(notice); + for (const skipped of plan.skipped) { + warn( + `Notice: runtime rule ${skipped.rule} was not run — ${skipped.reason}.` + ); + } + if (plan.execute.length > 0) { + const runtimeResults = await executeRuntimeRules(cwd, plan.execute, { + paths: existingPaths, + timeoutMs: parseTimeoutMs(args.timeout), + }); + results.push(...runtimeResults); } let errorCount = 0; @@ -287,11 +371,11 @@ export const checkCommand = defineCommand({ const hasErrors = errorCount > 0; scanCounts = { errorCount, warningCount, findings: results.length }; - // Format output if (args.json) { const output = checkOutputSchema.parse({ success: !hasErrors, results, + ...(plan.skipped.length > 0 ? { skipped: plan.skipped } : {}), }); console.log(JSON.stringify(output)); } else { diff --git a/packages/cli/src/help/check.txt b/packages/cli/src/help/check.txt index 937dadf7..10f38fec 100644 --- a/packages/cli/src/help/check.txt +++ b/packages/cli/src/help/check.txt @@ -1,37 +1,52 @@ -# Topic: check (CLI v%(CLI_VERSION)s / topic v1) +# Topic: check (CLI v%(CLI_VERSION)s / topic v2) ## Goal -Run the applicable rules in `.taskless/rules/` against the codebase and -report matches. Used standalone (full project scan), in CI (diff-only -scan), or after rule create/improve to validate. +Run the applicable rules against the codebase and report matches. Two +rule kinds run: **static** ast-grep rules in `.taskless/rules/`, and +**runtime** rules in `.taskless/runtime-rules/` (a directory of ast-grep +capture rules plus a `check.ts`). Used standalone (full project scan), +in CI (diff-only scan), or after rule create/improve to validate. ## Preconditions - `.taskless/` directory exists. -- At least one rule exists in `.taskless/rules/`. (If none exist, - the CLI exits 0 with a friendly message suggesting - `taskless rule create`.) -- No auth required. What actually runs depends on auth state — see - "What runs (auth state)". - -## What runs (auth state) - -`check` never requires auth, but auth state decides which rules execute: - -- **Logged out** (or `--anonymous`) — scans every rule in - `.taskless/rules/` locally, with no network call. The offline linter - posture; no warnings. -- **Logged in** — reconciles the local rule files against the Taskless - service and runs ONLY the server-blessed `run` set. Files that drift - from the blessed copy (`unsafe`), were never issued (`unknown`), or - are expected but absent locally (`missing`) are reported as warnings - and do NOT run. If reconciliation can't complete (no GitHub remote, - service unavailable/not deployed, or a rejected token), `check` warns - that verification was skipped and falls back to a full local scan. - -Warnings are human-readable stderr only: they never change the exit -code and never appear in `--json` output (which stays the -`{ success, results }` shape). The authoritative allow-list is the -server's; the CI backstop (`taskless help ci`) is the enforcement point. +- At least one rule exists in `.taskless/rules/` or + `.taskless/runtime-rules/`. (If none exist, the CLI exits 0 with a + friendly message suggesting `taskless rule create`.) +- No auth required. Static rules always run; whether runtime rules run + depends on auth state — see "What runs". + +## What runs + +`check` never requires auth. The two rule kinds run differently: + +- **Static rules** (`.taskless/rules/*.yml`) are inert ast-grep patterns + and **always run**, in every mode, with no network call. The offline + linter posture. +- **Runtime rules** (`.taskless/runtime-rules//`) execute a + `check.ts` — arbitrary code — so they run ONLY when that code is + verified: + - **Logged in** (token or API key) — each rule's `check.ts` is + reconciled against the Taskless service; rules the server blessed + (`run`) execute, and the rest are withheld and reported (advisory). + - **Logged out, `--anonymous`, no GitHub remote, or service + unavailable** — runtime rules are **skipped** (reported, never run). + Static rules still run. + - **`--dangerously-run-scripts`** — runs every runtime rule trusting + local signatures, with no network call, behind a prominent warning. + This is the only way to run runtime rules unverified. + +Notices about skipped/withheld runtime rules are human-readable stderr +only: they never change the exit code. Under `--json` they do NOT appear +as warnings; instead an additive optional `skipped: [{ rule, reason }]` +array is included alongside the unchanged `{ success, results }`. The +authoritative allow-list is the server's; the CI backstop +(`taskless help ci`) is the enforcement point for runtime rules. + +## Flags +- `--json` — machine output (`{ success, results, skipped? }`). +- `--anonymous` — run only static rules; skip runtime rules. +- `--dangerously-run-scripts` — run runtime `check.ts` unverified. +- `--timeout ` — per-runtime-check wall-clock bound (default 10). ## Steps @@ -54,6 +69,11 @@ server's; the CI backstop (`taskless help ci`) is the enforcement point. Paths that don't exist on disk are silently filtered, so you can pipe raw `git diff` output directly without pre-filtering. + When runtime rules were present but not run (e.g. logged out), the + JSON also carries `"skipped": [{ "rule": "", "reason": "…" }]` + alongside `success`/`results`. Surface it so CI can tell that runtime + rules did not execute; it never affects the exit code. + 3. **Parse the JSON output.** Shape: ```json { diff --git a/packages/cli/src/help/ci.txt b/packages/cli/src/help/ci.txt index 087baa05..ded38138 100644 --- a/packages/cli/src/help/ci.txt +++ b/packages/cli/src/help/ci.txt @@ -161,13 +161,18 @@ different structure for CircleCI. The six steps stay the same. `taskless check` does NOT require authentication. The generated CI config works out of the box with no secrets and scans all local rules. -Optionally, exposing a `TASKLESS_TOKEN` secret turns CI into the -**backstop**: an authenticated `check` reconciles the repo's rule files -against the Taskless service and runs exactly the server-blessed `run` -set, warning on drift (`unsafe`), unissued files (`unknown`), or -missing rules. This is the enforcement point for the server-owned rule -set — local developer runs are advisory. To wire it, set the token as -an env var on the check step (GitHub Actions): +Static ast-grep rules always run in CI with no secrets. **Runtime +rules** (`.taskless/runtime-rules/`, which execute a `check.ts`) only +run when their code is server-verified — so an unauthenticated CI job +runs the static rules and skips the runtime ones. + +Exposing a `TASKLESS_TOKEN` secret turns CI into the **backstop** for +runtime rules: an authenticated `check` reconciles each runtime rule's +`check.ts` against the Taskless service and runs exactly the +server-blessed set, withholding any that drift or were never issued. +This is the enforcement point for runtime rules — local developer runs +skip them unless `--dangerously-run-scripts` is passed. To wire it, set +the token as an env var on the check step (GitHub Actions): ```yaml - name: Taskless check diff --git a/packages/cli/src/rules/run-set.ts b/packages/cli/src/rules/run-set.ts deleted file mode 100644 index 44861a09..00000000 --- a/packages/cli/src/rules/run-set.ts +++ /dev/null @@ -1,72 +0,0 @@ -import { copyFile, mkdir, rm } from "node:fs/promises"; -import { join } from "node:path"; - -import { addToGitignore } from "../filesystem/gitignore"; -import type { RunEntry } from "../api/reconcile"; -import { signRuleFile } from "./rule-hash"; - -/** A local rule file paired with its canonical signature. */ -export interface SignedRuleFile { - /** Delivered file name as it exists under `.taskless/rules/`. */ - file: string; - /** Absolute path to the file on disk. */ - path: string; - /** Canonical signature envelope for the file's bytes. */ - signature: string; -} - -/** Directory (relative to `.taskless/`) holding the materialized run set. */ -export const RUN_RULES_DIR = ".run/rules"; - -/** Sign each rule file, returning its delivered name, path, and signature. */ -export async function signRuleFiles( - cwd: string, - fileNames: string[] -): Promise { - const rulesDirectory = join(cwd, ".taskless", "rules"); - return Promise.all( - fileNames.map(async (file) => { - const path = join(rulesDirectory, file); - return { file, path, signature: await signRuleFile(path) }; - }) - ); -} - -/** - * Resolve the server's `run` entries back to local files by signature - * (content-based join, so a moved-but-unchanged file still resolves). Entries - * whose signature is not held locally are dropped. - */ -export function selectRunFiles( - signed: SignedRuleFile[], - run: RunEntry[] -): SignedRuleFile[] { - const bySignature = new Map(signed.map((s) => [s.signature, s])); - const selected: SignedRuleFile[] = []; - const seen = new Set(); - for (const entry of run) { - const match = bySignature.get(entry.signature); - if (match && !seen.has(match.path)) { - seen.add(match.path); - selected.push(match); - } - } - return selected; -} - -/** - * Materialize the blessed run set into a fresh, gitignored - * `.taskless/.run/rules/` so ast-grep evaluates only those files. The directory - * is rebuilt on every call to avoid stale rules leaking into a scan. - */ -export async function materializeRunDirectory( - cwd: string, - files: SignedRuleFile[] -): Promise { - const runRoot = join(cwd, ".taskless", ".run"); - const runRules = join(runRoot, "rules"); - await rm(runRoot, { recursive: true, force: true }); - await mkdir(runRules, { recursive: true }); - await Promise.all(files.map((f) => copyFile(f.path, join(runRules, f.file)))); - await addToGitignore(cwd, [".run/"]); -} diff --git a/packages/cli/src/rules/runtime/discover.ts b/packages/cli/src/rules/runtime/discover.ts new file mode 100644 index 00000000..65c516e8 --- /dev/null +++ b/packages/cli/src/rules/runtime/discover.ts @@ -0,0 +1,138 @@ +import { readdir, readFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { parse } from "yaml"; + +import type { CaptureRule, MatchMode } from "../../types/runtime-rule"; + +/** Directory (relative to `.taskless/`) that holds runtime rules. */ +export const RUNTIME_RULES_DIR = "runtime-rules"; + +/** A parsed capture `*.yml` of a runtime rule, with the fields the harness needs. */ +export interface LoadedCaptureRule { + /** Absolute path to the capture `*.yml`. */ + file: string; + /** Basename of the capture file. */ + fileName: string; + /** Baked-in ast-grep rule id (used to attribute scan matches back to `name`). */ + id: string; + /** Stable, model-assigned name (`metadata.taskless.name`), surfaced as `match.rule`. */ + name: string; + /** ast-grep `language`. */ + language: string; + /** Scan mode; `anchor` when omitted. */ + match: MatchMode; + /** The full parsed capture rule (for running the narrow). */ + rule: CaptureRule; +} + +/** A discovered runtime rule directory under `.taskless/runtime-rules/`. */ +export interface RuntimeRule { + /** Rule directory basename (e.g. `no-default-export-abc12345`). */ + name: string; + /** Absolute path to the rule directory. */ + dir: string; + /** The rule's parsed capture rules, in filename order. */ + captureRules: LoadedCaptureRule[]; + /** Absolute path to the rule's `check.ts`. */ + checkFile: string; +} + +/** Narrow an unknown parsed YAML value to a runtime `CaptureRule`, or return null. */ +function asRuntimeCaptureRule(value: unknown): CaptureRule | null { + if (typeof value !== "object" || value === null) return null; + const candidate = value as Partial; + const taskless = candidate.metadata?.taskless; + if (!taskless || taskless.kind !== "runtime") return null; + if (typeof candidate.language !== "string") return null; + if (typeof taskless.name !== "string") return null; + return candidate as CaptureRule; +} + +/** Load and parse the capture rules of a single runtime-rule directory. */ +async function loadCaptureRules( + directory: string +): Promise { + let entries: string[]; + try { + entries = await readdir(directory); + } catch { + return []; + } + const ymlFiles = entries.filter( + (f) => f.endsWith(".yml") || f.endsWith(".yaml") + ); + const loaded: LoadedCaptureRule[] = []; + for (const fileName of ymlFiles.toSorted()) { + const file = join(directory, fileName); + let parsed: unknown; + try { + parsed = parse(await readFile(file, "utf8")); + } catch { + continue; // not valid YAML — skip + } + const rule = asRuntimeCaptureRule(parsed); + if (!rule || typeof rule.id !== "string") continue; + loaded.push({ + file, + fileName, + id: rule.id, + name: rule.metadata.taskless.name, + language: rule.language, + match: rule.metadata.taskless.match ?? "anchor", + rule, + }); + } + return loaded; +} + +/** + * Enumerate `.taskless/runtime-rules/` under `cwd` and return each rule + * directory that holds at least one `kind: runtime` capture rule. + * `.taskless/runtime-rule-tests/` is never enumerated — it holds verification + * fixtures, not executable rules. + */ +export async function discoverRuntimeRules( + cwd: string +): Promise { + return discoverRuntimeRulesIn(join(cwd, ".taskless", RUNTIME_RULES_DIR)); +} + +/** + * Enumerate runtime rules under an explicit `runtime-rules` root — used to + * re-discover rules from the materialized `.taskless/.run/` tree so the executed + * bytes are the blessed ones. + */ +export async function discoverRuntimeRulesIn( + root: string +): Promise { + let directoryEntries; + try { + directoryEntries = await readdir(root, { withFileTypes: true }); + } catch { + return []; // no runtime-rules directory + } + + const rules: RuntimeRule[] = []; + const sorted = directoryEntries.toSorted((a, b) => + a.name.localeCompare(b.name) + ); + for (const entry of sorted) { + if (!entry.isDirectory()) continue; + const directory = join(root, entry.name); + const captureRules = await loadCaptureRules(directory); + if (captureRules.length === 0) continue; // not a runtime rule + + // The check file is always `check.ts` inside the rule directory (per spec). + // We deliberately do NOT resolve `metadata.taskless.check` as a path — an + // arbitrary value (e.g. `../../evil.ts`) must not be able to point execution + // or signing at a file outside the rule directory. + rules.push({ + name: entry.name, + dir: directory, + captureRules, + checkFile: join(directory, "check.ts"), + }); + } + return rules; +} diff --git a/packages/cli/src/rules/runtime/harness.ts b/packages/cli/src/rules/runtime/harness.ts new file mode 100644 index 00000000..f6ffd056 --- /dev/null +++ b/packages/cli/src/rules/runtime/harness.ts @@ -0,0 +1,113 @@ +import { relative } from "node:path"; + +import type { CheckResult } from "../../types/check"; +import type { Finding } from "../../types/runtime-rule"; +import type { RuntimeRule } from "./discover"; +import { runNarrow } from "./narrow"; +import { DEFAULT_CHECK_TIMEOUT_MS, invokeCheck } from "./invoke"; + +/** Scanner-agnostic `source` label for runtime-rule findings. */ +export const RUNTIME_SOURCE = "taskless-runtime"; + +/** Options controlling a runtime-rule run. */ +export interface RuntimeRunOptions { + /** Restrict the narrow to these paths (diff scope); empty scans the repo. */ + paths?: string[]; + /** Per-check wall-clock bound in ms. */ + timeoutMs?: number; +} + +/** + * Map a check `Finding` onto the scanner-agnostic `CheckResult`. `Finding` + * line/column are 1-indexed (harness contract); `CheckResult.range` is 0-indexed + * (ast-grep native — display and `--json` consumers add 1), so convert down. + */ +function findingToCheckResult( + rule: RuntimeRule, + finding: Finding +): CheckResult { + const line = finding.line === undefined ? 0 : Math.max(0, finding.line - 1); + const column = + finding.column === undefined ? 0 : Math.max(0, finding.column - 1); + return { + source: RUNTIME_SOURCE, + ruleId: rule.name, + severity: finding.severity ?? "warning", + message: finding.message, + file: finding.file, + range: { start: { line, column }, end: { line, column } }, + matchedText: "", + }; +} + +/** A harness failure (throw / timeout / bad output) becomes one error finding. */ +function harnessErrorResult( + root: string, + rule: RuntimeRule, + message: string +): CheckResult { + return { + source: RUNTIME_SOURCE, + ruleId: rule.name, + severity: "error", + message: `runtime rule ${rule.name} failed: ${message}`, + file: relative(root, rule.checkFile), + range: { start: { line: 0, column: 0 }, end: { line: 0, column: 0 } }, + matchedText: "", + }; +} + +/** + * Execute one runtime rule: run the ast-grep narrow, gate on matches (zero + * matches ⇒ `check.ts` is never invoked), invoke `check.ts`, and map its + * findings onto `CheckResult`. A harness failure is isolated to a single + * error-severity finding and never throws. + */ +export async function executeRuntimeRule( + root: string, + rule: RuntimeRule, + options: RuntimeRunOptions = {} +): Promise { + let matches; + try { + matches = await runNarrow(root, rule, options.paths ?? []); + } catch (error) { + return [ + harnessErrorResult( + root, + rule, + `narrow failed: ${error instanceof Error ? error.message : String(error)}` + ), + ]; + } + + if (matches.length === 0) return []; // gate: no matches, no check + + const result = await invokeCheck( + rule.checkFile, + root, + matches, + options.timeoutMs ?? DEFAULT_CHECK_TIMEOUT_MS + ); + if (result.status === "error") { + return [harnessErrorResult(root, rule, result.message)]; + } + return result.findings.map((finding) => findingToCheckResult(rule, finding)); +} + +/** + * Execute runtime rules in sequence (process-per-check scheduling) and return + * the aggregated findings. Sequential keeps `tsx` worker startup predictable; + * the function contract leaves room for a pool later. + */ +export async function executeRuntimeRules( + root: string, + rules: RuntimeRule[], + options: RuntimeRunOptions = {} +): Promise { + const results: CheckResult[] = []; + for (const rule of rules) { + results.push(...(await executeRuntimeRule(root, rule, options))); + } + return results; +} diff --git a/packages/cli/src/rules/runtime/invoke.ts b/packages/cli/src/rules/runtime/invoke.ts new file mode 100644 index 00000000..3a00ae91 --- /dev/null +++ b/packages/cli/src/rules/runtime/invoke.ts @@ -0,0 +1,174 @@ +import { spawn } from "node:child_process"; +import { createRequire } from "node:module"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +import type { Finding, Match } from "../../types/runtime-rule"; + +/** Default per-check wall-clock bound (ms); overridable via `--timeout`. */ +export const DEFAULT_CHECK_TIMEOUT_MS = 10_000; + +/** Outcome of invoking a single `check.ts`. */ +export type InvokeResult = + | { status: "ok"; findings: Finding[] } + | { status: "error"; message: string }; + +/** + * The self-contained ESM runner executed under `tsx`. It imports the delivered + * `check.ts` (which `tsx` transpiles), calls its default export with + * `(root, matches)`, and writes the returned findings to `outPath` — so the + * check's own stdout/stderr never pollutes the result channel. + */ +const RUNNER_SOURCE = `import { readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const [checkFile, root, matchesPath, outPath] = process.argv.slice(2); +try { + const matches = JSON.parse(readFileSync(matchesPath, "utf8")); + const mod = await import(pathToFileURL(checkFile).href); + const check = mod.default; + if (typeof check !== "function") { + throw new TypeError("check.ts must export a default function"); + } + const findings = await check(root, matches); + writeFileSync(outPath, JSON.stringify(findings ?? [])); +} catch (error) { + process.stderr.write(error instanceof Error ? (error.stack ?? error.message) : String(error)); + process.exit(1); +} +`; + +/** Resolve the bundled `tsx` CLI entry so `check.ts` runs without a repo toolchain. */ +function resolveTsxCli(): string { + const require = createRequire(import.meta.url); + const packageJsonPath = require.resolve("tsx/package.json"); + const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8")) as { + bin?: string | Record; + }; + const binField = packageJson.bin; + const relative = typeof binField === "string" ? binField : binField?.tsx; + if (!relative) { + throw new Error("Bundled tsx has no resolvable bin entry"); + } + return resolve(dirname(packageJsonPath), relative); +} + +/** + * Invoke a runtime rule's `check.ts` as a function and return its findings. + * Bounds execution with `timeoutMs`; a throw, a non-zero exit, or a timeout is + * isolated into an `error` result (the caller records one error-severity finding) + * and never aborts the overall run. + */ +export async function invokeCheck( + checkFile: string, + root: string, + matches: Match[], + timeoutMs: number = DEFAULT_CHECK_TIMEOUT_MS +): Promise { + let tsxCli: string; + try { + tsxCli = resolveTsxCli(); + } catch (error) { + return { + status: "error", + message: `runtime harness unavailable: ${ + error instanceof Error ? error.message : String(error) + }`, + }; + } + + const workDirectory = await mkdtemp(join(tmpdir(), "tskl-check-")); + const matchesPath = join(workDirectory, "matches.json"); + const runnerPath = join(workDirectory, "runner.mjs"); + const outPath = join(workDirectory, "findings.json"); + + try { + await writeFile(matchesPath, JSON.stringify(matches)); + await writeFile(runnerPath, RUNNER_SOURCE); + + const result = await new Promise((resolvePromise) => { + // `detached` makes the child its own process-group leader so a timeout + // can SIGKILL the whole tree — `tsx` re-execs node as a grandchild, and + // killing only the wrapper would leave a runaway check running. + const child = spawn( + process.execPath, + [tsxCli, runnerPath, checkFile, root, matchesPath, outPath], + { stdio: ["ignore", "ignore", "pipe"], detached: true } + ); + const stderrChunks: string[] = []; + let timedOut = false; + const killTree = () => { + if (child.pid === undefined) return; + if (process.platform === "win32") { + // Negative PIDs aren't supported on Windows; `taskkill /T` terminates + // the whole tree (the tsx wrapper and its re-exec'd node grandchild). + try { + spawn("taskkill", ["/pid", String(child.pid), "/T", "/F"], { + stdio: "ignore", + }); + } catch { + child.kill("SIGKILL"); + } + return; + } + try { + process.kill(-child.pid, "SIGKILL"); + } catch { + child.kill("SIGKILL"); + } + }; + const timer = setTimeout(() => { + timedOut = true; + killTree(); + }, timeoutMs); + + child.stderr.on("data", (chunk: Buffer) => + stderrChunks.push(chunk.toString()) + ); + child.on("error", (error) => { + clearTimeout(timer); + resolvePromise({ status: "error", message: error.message }); + }); + child.on("close", (code) => { + clearTimeout(timer); + if (timedOut) { + resolvePromise({ + status: "error", + message: `check timed out after ${String(timeoutMs)}ms`, + }); + return; + } + if (code !== 0) { + const detail = stderrChunks.join("").trim(); + resolvePromise({ + status: "error", + message: `check exited with code ${String(code)}${ + detail ? `: ${detail}` : "" + }`, + }); + return; + } + void readFile(outPath, "utf8") + .then((raw) => + resolvePromise({ + status: "ok", + findings: JSON.parse(raw) as Finding[], + }) + ) + .catch((error: unknown) => + resolvePromise({ + status: "error", + message: `check produced no readable findings: ${ + error instanceof Error ? error.message : String(error) + }`, + }) + ); + }); + }); + return result; + } finally { + await rm(workDirectory, { recursive: true, force: true }); + } +} diff --git a/packages/cli/src/rules/runtime/narrow.ts b/packages/cli/src/rules/runtime/narrow.ts new file mode 100644 index 00000000..7f340777 --- /dev/null +++ b/packages/cli/src/rules/runtime/narrow.ts @@ -0,0 +1,173 @@ +import { spawn } from "node:child_process"; +import { copyFile, mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { createInterface } from "node:readline"; +import { join } from "node:path"; + +import { stringify } from "yaml"; + +import { buildPath, findSgBinary } from "../scan"; +import type { Match } from "../../types/runtime-rule"; +import type { LoadedCaptureRule, RuntimeRule } from "./discover"; + +/** + * The subset of an ast-grep `--json=stream` match this harness reads. ast-grep + * reports `range.start` 0-indexed; the harness normalizes to 1-indexed `Match`. + */ +interface AstGrepStreamMatch { + text: string; + file: string; + ruleId: string; + range: { start: { line: number; column: number } }; + metaVariables?: { + single?: Record; + }; +} + +/** Spawn `sg scan` with a temp config and stream stdout lines to `onLine`. */ +function runSg( + root: string, + configPath: string, + extraArguments: string[], + paths: string[], + onLine: (line: string) => void +): Promise { + return new Promise((resolve, reject) => { + const argv = [ + "scan", + "--config", + configPath, + ...extraArguments, + ...(paths.length > 0 ? ["--", ...paths] : []), + ]; + const child = spawn(findSgBinary(), argv, { + cwd: root, + stdio: ["ignore", "pipe", "pipe"], + env: { ...process.env, PATH: buildPath() }, + }); + const stderrChunks: string[] = []; + const rl = createInterface({ input: child.stdout }); + rl.on("line", onLine); + child.stderr.on("data", (chunk: Buffer) => + stderrChunks.push(chunk.toString()) + ); + child.on("error", reject); + child.on("close", (code, signal) => { + // ast-grep exits 1 when matches are found — expected. A `null` code means + // the process was killed by a signal (e.g. OOM); treat that and any exit + // >1 as a real failure rather than silently dropping matches. + if (code === null || code > 1) { + const cause = + code === null ? `signal ${String(signal)}` : `exit ${String(code)}`; + reject( + new Error( + `ast-grep narrow failed (${cause})${ + stderrChunks.length > 0 ? `: ${stderrChunks.join("").trim()}` : "" + }` + ) + ); + return; + } + resolve(); + }); + }); +} + +/** Copy the given capture files into `/rules/` and a config pointing at them. */ +async function writeRuleConfig( + directory: string, + captureRules: LoadedCaptureRule[] +): Promise { + const rulesDirectory = join(directory, "rules"); + await mkdir(rulesDirectory, { recursive: true }); + // Copy the original `*.yml` bytes rather than re-serializing the parsed rule — + // a YAML round-trip can subtly alter an exotic ast-grep config, and the file is + // a valid ast-grep rule as delivered. + await Promise.all( + captureRules.map((c) => copyFile(c.file, join(rulesDirectory, c.fileName))) + ); + const configPath = join(directory, "sgconfig.yml"); + await writeFile(configPath, stringify({ ruleDirs: ["rules"] })); + return configPath; +} + +/** + * Run a runtime rule's capture rules as the ast-grep narrow and return the + * normalized matches. Anchor captures produce full matches with `captures`; + * broad (`kind: program`) captures are path-only enumerators. `ruleId` is mapped + * back to the stable model `name` surfaced as `match.rule`. + */ +export async function runNarrow( + root: string, + rule: RuntimeRule, + paths: string[] = [] +): Promise { + const nameById = new Map(rule.captureRules.map((c) => [c.id, c.name])); + const anchor = rule.captureRules.filter((c) => c.match !== "broad"); + const broad = rule.captureRules.filter((c) => c.match === "broad"); + + const workDirectory = await mkdtemp(join(tmpdir(), "tskl-narrow-")); + const matches: Match[] = []; + try { + if (anchor.length > 0) { + const config = await writeRuleConfig( + join(workDirectory, "anchor"), + anchor + ); + await runSg(root, config, ["--json=stream"], paths, (line) => { + const trimmed = line.trim(); + if (trimmed === "") return; + let parsed: AstGrepStreamMatch; + try { + parsed = JSON.parse(trimmed) as AstGrepStreamMatch; + } catch { + return; // non-JSON status line + } + const captures: Record = {}; + for (const [key, value] of Object.entries( + parsed.metaVariables?.single ?? {} + )) { + if (typeof value.text === "string") captures[key] = value.text; + } + matches.push({ + rule: nameById.get(parsed.ruleId) ?? parsed.ruleId, + ruleId: parsed.ruleId, + file: parsed.file, + line: parsed.range.start.line + 1, + column: parsed.range.start.column + 1, + text: parsed.text, + captures, + }); + }); + } + + // `--files-with-matches` reports paths only — it can't say which rule + // matched — so run one broad scan per broad capture rule and attribute each + // file to that rule. (Broad rules are whole-language enumerators, so this is + // cheap and the common case is a single broad rule.) + for (const broadRule of broad) { + const config = await writeRuleConfig( + join(workDirectory, `broad-${broadRule.id}`), + [broadRule] + ); + const seen = new Set(); + await runSg(root, config, ["--files-with-matches"], paths, (line) => { + const file = line.trim(); + if (file === "" || seen.has(file)) return; + seen.add(file); + matches.push({ + rule: broadRule.name, + ruleId: broadRule.id, + file, + line: 1, + column: 1, + text: "", + captures: {}, + }); + }); + } + } finally { + await rm(workDirectory, { recursive: true, force: true }); + } + return matches; +} diff --git a/packages/cli/src/rules/runtime/run-set.ts b/packages/cli/src/rules/runtime/run-set.ts new file mode 100644 index 00000000..fff8c188 --- /dev/null +++ b/packages/cli/src/rules/runtime/run-set.ts @@ -0,0 +1,109 @@ +import { cp, mkdir, rm } from "node:fs/promises"; +import { join, relative, sep } from "node:path"; + +import { addToGitignore } from "../../filesystem/gitignore"; +import { signRuleFile } from "../rule-hash"; +import type { ReportedFile, RunEntry } from "../../api/reconcile"; +import { discoverRuntimeRulesIn, type RuntimeRule } from "./discover"; + +/** Directory (relative to `.taskless/`) holding materialized runtime rules. */ +export const RUNTIME_RUN_DIR = ".run/runtime-rules"; + +/** A runtime rule paired with its `check.ts` signature — the reconcile gate. */ +export interface SignedRuntimeRule { + rule: RuntimeRule; + /** Canonical signature envelope for the rule's `check.ts` bytes. */ + signature: string; +} + +/** Result of signing a set of runtime rules' `check.ts` files. */ +export interface RuntimeSigningResult { + /** Rules whose `check.ts` was read and signed. */ + signed: SignedRuntimeRule[]; + /** Rules whose `check.ts` was missing or unreadable (cannot be reconciled). */ + unreadable: RuntimeRule[]; +} + +/** + * Sign each runtime rule's `check.ts` (only) — the sole artifact carrying + * arbitrary code execution. Capture `*.yml` are inert and are neither signed nor + * reported. A rule whose `check.ts` cannot be read is returned in `unreadable` + * (never thrown) so one malformed rule never aborts the whole `check`. + */ +export async function signRuntimeChecks( + rules: RuntimeRule[] +): Promise { + const signed: SignedRuntimeRule[] = []; + const unreadable: RuntimeRule[] = []; + await Promise.all( + rules.map(async (rule) => { + try { + signed.push({ rule, signature: await signRuleFile(rule.checkFile) }); + } catch { + unreadable.push(rule); + } + }) + ); + return { signed, unreadable }; +} + +/** Map signed runtime rules to the reconcile report (`check.ts` path + signature). */ +export function reportRuntimeChecks( + cwd: string, + signed: SignedRuntimeRule[] +): ReportedFile[] { + return signed.map(({ rule, signature }) => ({ + // Reconcile paths are repo-relative POSIX; normalize Windows separators. + file: relative(cwd, rule.checkFile).split(sep).join("/"), + signature, + })); +} + +/** The blessed/withheld split from a successful reconciliation. */ +export interface RuntimeSelection { + /** Rules whose `check.ts` signature is in the server `run` set. */ + blessed: RuntimeRule[]; + /** Rules whose `check.ts` was not blessed (advisory). */ + withheld: RuntimeRule[]; +} + +/** + * Split signed runtime rules into those whose `check.ts` is present in the + * server's `run` set (blessed, execute) and the rest (withheld, advisory). The + * join is by signature — content-based, so a moved-but-unchanged rule resolves. + */ +export function selectBlessedRuntimeRules( + signed: SignedRuntimeRule[], + run: RunEntry[] +): RuntimeSelection { + const runSignatures = new Set(run.map((entry) => entry.signature)); + const blessed: RuntimeRule[] = []; + const withheld: RuntimeRule[] = []; + for (const { rule, signature } of signed) { + if (runSignatures.has(signature)) blessed.push(rule); + else withheld.push(rule); + } + return { blessed, withheld }; +} + +/** + * Materialize blessed runtime rules into the gitignored + * `.taskless/.run/runtime-rules/` and return them re-discovered from there, so + * the narrow and `check.ts` execute the blessed bytes rather than whatever is + * live in `.taskless/runtime-rules/`. + */ +export async function materializeRuntimeRules( + cwd: string, + blessed: RuntimeRule[] +): Promise { + const runtimeRunRoot = join(cwd, ".taskless", RUNTIME_RUN_DIR); + await rm(runtimeRunRoot, { recursive: true, force: true }); + await mkdir(runtimeRunRoot, { recursive: true }); + await Promise.all( + blessed.map((rule) => + cp(rule.dir, join(runtimeRunRoot, rule.name), { recursive: true }) + ) + ); + await addToGitignore(cwd, [".run/"]); + return discoverRuntimeRulesIn(runtimeRunRoot); +} diff --git a/packages/cli/src/schemas/check.ts b/packages/cli/src/schemas/check.ts index c66b8909..ebe596e5 100644 --- a/packages/cli/src/schemas/check.ts +++ b/packages/cli/src/schemas/check.ts @@ -24,10 +24,20 @@ const checkResultSchema = z.object({ fix: z.string().nullable().optional().describe("Suggested fix replacement"), }); +/** A runtime rule that was present but not executed (advisory), for `--json`. */ +const skippedRuntimeRuleSchema = z.object({ + rule: z.string().describe("Runtime rule name that did not run"), + reason: z.string().describe("Why the runtime rule was not run"), +}); + /** Output schema for `taskless check --json` on success */ export const outputSchema = z.object({ success: z.boolean(), results: z.array(checkResultSchema).describe("Check results"), + skipped: z + .array(skippedRuntimeRuleSchema) + .optional() + .describe("Runtime rules present but not executed"), }); /** Error schema for `taskless check --json` on failure */ diff --git a/packages/cli/src/types/runtime-rule.ts b/packages/cli/src/types/runtime-rule.ts new file mode 100644 index 00000000..fc61bf09 --- /dev/null +++ b/packages/cli/src/types/runtime-rule.ts @@ -0,0 +1,112 @@ +/** + * Runtime rule file format + harness↔check contract — mirrored structurally from + * `@taskless/types` (workers/generator). A runtime rule is a DIRECTORY under + * `.taskless/runtime-rules//`: one or more ast-grep capture `*.yml` plus + * exactly one `check.ts`. The capture rules are the cheap syntactic narrow; the + * `check.ts` refines only where the narrow matched. + * + * These types are kept structurally identical to what a generated `check.ts` + * declares inline: a delivered check imports NOTHING from `@taskless/*`, so the + * contract is structural, not an import. They exist here for the CLI's own + * harness code (discovery, narrow, invocation). + */ + +/** Harness↔check protocol version; bumped only on a breaking `(root, matches)` change. */ +export const RUNTIME_CHECK_PROTOCOL_VERSION = 1; + +/** + * Finding severity — a subset of ast-grep's enum. `error` blocks (non-zero + * result); `warning` and `info` are advisory. An omitted value is treated as + * `warning`. + */ +export type FindingSeverity = "error" | "warning" | "info"; + +/** A single result a check returns. `file` is repo-root-relative. */ +export interface Finding { + /** Repo-root-relative path the finding is about. */ + file: string; + /** 1-indexed line, when the finding is line-scoped. */ + line?: number; + /** 1-indexed column, when the finding is column-scoped. */ + column?: number; + /** Human-readable description of the finding. */ + message: string; + /** Per-finding severity; an omitted value is treated as `warning`. */ + severity?: FindingSeverity; +} + +/** + * One normalized ast-grep match handed to a check. The check branches on `rule` + * (the stable, model-assigned name), NEVER on `ruleId` (the baked-in hash). + */ +export interface Match { + /** Stable, model-assigned capture-rule name to branch on (e.g. `exports`). */ + rule: string; + /** Baked-in, globally-unique hashed id; opaque to the check. */ + ruleId: string; + /** Path RELATIVE to `root`; `path.join(root, file)` to read the file. */ + file: string; + /** 1-indexed line (ast-grep reports 0-indexed; the harness normalizes). */ + line: number; + /** 1-indexed column (ast-grep reports 0-indexed; the harness normalizes). */ + column: number; + /** The matched source text. */ + text: string; + /** Captured metavariables by name; empty for a `broad` enumerator match. */ + captures: Record; +} + +/** + * The harness↔check contract: a check module's default export is an async + * function taking the repo `root` and the narrow's `matches`, reading any repo + * file it needs from disk under `root`, and RETURNING its findings. + */ +export type CheckFunction = ( + root: string, + matches: Match[] +) => Promise; + +/** + * Scan mode for a capture rule. `anchor` (the default) is the syntactic narrow + * (`ast-grep scan --json=stream`, matches carry `captures`); `broad` is a + * whole-language enumerator (`rule: { kind: program }` + `--files-with-matches`, + * paths only, empty `captures`). + */ +export type MatchMode = "anchor" | "broad"; + +/** The `metadata.taskless` block folded into every capture rule's YAML. */ +export interface RuntimeRuleMetadata { + /** Metadata schema version for the runtime rule format. */ + version: number; + /** Discriminates a runtime capture rule from a static rule. */ + kind: "runtime"; + /** Stable, model-assigned capture-rule name (surfaced on matches as `rule`). */ + name: string; + /** Filename of the `check.ts` this capture rule pairs with. */ + check: string; + /** Scan mode; treated as `anchor` when omitted. */ + match?: MatchMode; +} + +/** One on-disk ast-grep capture `*.yml` of a runtime rule. */ +export interface CaptureRule { + /** Baked-in, globally-unique ast-grep rule id (`${ruleSlug}-${sha1}`). */ + id?: string; + /** ast-grep `language` — scopes which files this rule parses. */ + language: string; + /** The ast-grep matcher. */ + rule: Record; + /** Optional ast-grep `constraints` — a SIBLING of `rule`, not nested. */ + constraints?: Record; + /** Optional ast-grep `utils` — a sibling of `rule`. */ + utils?: Record; + /** Optional ast-grep `transform` — a sibling of `rule`. */ + transform?: Record; + /** Taskless metadata folded into the YAML. */ + metadata: { + /** The runtime rule's self-describing Taskless block. */ + taskless: RuntimeRuleMetadata; + /** Other metadata keys are permitted alongside. */ + [key: string]: unknown; + }; +} diff --git a/packages/cli/test/reconcile-check.test.ts b/packages/cli/test/reconcile-check.test.ts deleted file mode 100644 index 194d5d1d..00000000 --- a/packages/cli/test/reconcile-check.test.ts +++ /dev/null @@ -1,310 +0,0 @@ -import { execFile } from "node:child_process"; -import { createServer, type Server } from "node:http"; -import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; -import { resolve, join } from "node:path"; -import { tmpdir } from "node:os"; -import { promisify } from "node:util"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; - -const execFileAsync = promisify(execFile); -const binPath = resolve(import.meta.dirname, "../dist/index.js"); - -interface ReportedFile { - file: string; - signature: string; -} -interface ReconcileRequestBody { - repositoryUrl: string; - files: ReportedFile[]; -} -interface MockResponse { - statusCode: number; - body?: unknown; -} -type Responder = (request: ReconcileRequestBody) => MockResponse; - -interface MockServer { - apiUrl: string; - requests: ReconcileRequestBody[]; - close: () => Promise; -} - -/** Start a mock reconcile endpoint on a random port. */ -function startMockServer(responder: Responder): Promise { - const requests: ReconcileRequestBody[] = []; - const server: Server = createServer((request, response) => { - if (request.method !== "POST" || request.url !== "/cli/api/reconcile") { - response.writeHead(404).end("{}"); - return; - } - let raw = ""; - request.on("data", (chunk: Buffer) => (raw += chunk.toString())); - request.on("end", () => { - const parsed = JSON.parse(raw) as ReconcileRequestBody; - requests.push(parsed); - const { statusCode, body } = responder(parsed); - response.writeHead(statusCode, { "content-type": "application/json" }); - response.end(JSON.stringify(body ?? {})); - }); - }); - return new Promise((resolvePromise) => { - server.listen(0, "127.0.0.1", () => { - const address = server.address(); - const port = typeof address === "object" && address ? address.port : 0; - resolvePromise({ - apiUrl: `http://127.0.0.1:${String(port)}/cli`, - requests, - close: () => - new Promise((done) => { - server.close(() => done()); - }), - }); - }); - }); -} - -/** Look up a reported file's signature so the mock can echo it back. */ -function sig(request: ReconcileRequestBody, file: string): string { - return request.files.find((f) => f.file === file)?.signature ?? ""; -} - -async function runCli( - args: string[], - env: Record = {} -): Promise<{ stdout: string; stderr: string; exitCode: number }> { - try { - const { stdout, stderr } = await execFileAsync("node", [binPath, ...args], { - env: { ...process.env, ...env }, - }); - return { stdout, stderr, exitCode: 0 }; - } catch (error) { - const execError = error as { stdout: string; stderr: string; code: number }; - return { - stdout: execError.stdout ?? "", - stderr: execError.stderr ?? "", - exitCode: execError.code, - }; - } -} - -const RULE_NO_EVAL = [ - "id: no-eval", - "language: javascript", - "severity: error", - "rule:", - " pattern: eval($$$)", - "message: Avoid eval", -].join("\n"); - -const RULE_NO_CONSOLE = [ - "id: no-console", - "language: javascript", - "severity: error", - "rule:", - " pattern: console.log($$$)", - "message: Avoid console.log", -].join("\n"); - -describe("check reconciliation", () => { - let directory: string; - - beforeEach(async () => { - directory = await mkdtemp(join(tmpdir(), "taskless-reconcile-")); - const rules = join(directory, ".taskless", "rules"); - await mkdir(rules, { recursive: true }); - await writeFile(join(rules, "no-eval.yml"), RULE_NO_EVAL, "utf8"); - await writeFile(join(rules, "no-console.yml"), RULE_NO_CONSOLE, "utf8"); - await writeFile( - join(directory, "app.js"), - 'eval("x");\nconsole.log("y");\n', - "utf8" - ); - // A GitHub origin is required for reconciliation to run. - await execFileAsync("git", ["init"], { cwd: directory }); - await execFileAsync( - "git", - ["remote", "add", "origin", "https://github.com/acme/widgets.git"], - { cwd: directory } - ); - }); - - afterEach(async () => { - await rm(directory, { recursive: true, force: true }); - }); - - it("runs ONLY the blessed run set, excluding non-run rules", async () => { - const server = await startMockServer((request) => ({ - statusCode: 200, - body: { - run: [ - { - ruleId: "no-eval", - file: "no-eval.yml", - signature: sig(request, "no-eval.yml"), - }, - ], - unsafe: [], - unknown: [], - missing: [], - }, - })); - try { - const { stdout, exitCode } = await runCli( - ["check", "-d", directory, "--json"], - { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } - ); - expect(server.requests).toHaveLength(1); - expect(exitCode).toBe(1); // no-eval is an error and matches - const parsed = JSON.parse(stdout.trim()) as { - results: { ruleId: string }[]; - }; - const ruleIds = new Set(parsed.results.map((r) => r.ruleId)); - expect(ruleIds.has("no-eval")).toBe(true); - expect(ruleIds.has("no-console")).toBe(false); // not in run set - } finally { - await server.close(); - } - }); - - it("warns on unsafe/unknown/missing and exits 0 on an empty run set", async () => { - const server = await startMockServer((request) => ({ - statusCode: 200, - body: { - run: [], - unsafe: [ - { - file: "no-console.yml", - expected: "1;h=sha-256;d=" + "0".repeat(64), - got: sig(request, "no-console.yml"), - }, - ], - unknown: [{ file: "no-eval.yml" }], - missing: [{ ruleId: "no-var", file: "no-var-abc.yml" }], - }, - })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory], - { - TASKLESS_TOKEN: "fake.token.value", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(exitCode).toBe(0); // empty run set → nothing scanned - expect(stdout).toContain("No issues found"); - expect(stderr).toContain("no-console.yml"); // unsafe drift - expect(stderr).toContain("no-eval.yml"); // unknown - expect(stderr).toContain("no-var"); // missing - } finally { - await server.close(); - } - }); - - it("suppresses mismatch warnings under --json", async () => { - const server = await startMockServer((request) => ({ - statusCode: 200, - body: { - run: [], - unsafe: [ - { - file: "no-console.yml", - expected: "1;h=sha-256;d=" + "0".repeat(64), - got: sig(request, "no-console.yml"), - }, - ], - unknown: [], - missing: [], - }, - })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory, "--json"], - { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } - ); - expect(exitCode).toBe(0); - expect(stderr).not.toContain("tamper"); - expect(stderr).not.toContain("no-console.yml"); - const parsed = JSON.parse(stdout.trim()) as { - success: boolean; - results: unknown[]; - }; - expect(parsed).toEqual({ success: true, results: [] }); - } finally { - await server.close(); - } - }); - - it("degrades to a local scan when reconciliation is unavailable", async () => { - const server = await startMockServer(() => ({ statusCode: 503 })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory], - { - TASKLESS_TOKEN: "fake.token.value", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(server.requests).toHaveLength(1); - expect(exitCode).toBe(1); // fell back to a full local scan; no-eval matches - expect(stdout).toContain("no-eval"); - expect(stdout).toContain("no-console"); // ALL local rules ran - expect(stderr).toContain("Scanning all local rules unverified"); - } finally { - await server.close(); - } - }); - - it("does not warn under --json when degrading", async () => { - const server = await startMockServer(() => ({ statusCode: 503 })); - try { - const { stdout, stderr } = await runCli( - ["check", "-d", directory, "--json"], - { - TASKLESS_TOKEN: "fake.token.value", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(stderr).not.toContain("Scanning all local rules unverified"); - const parsed = JSON.parse(stdout.trim()) as { results: unknown[] }; - expect(parsed.results.length).toBeGreaterThan(0); - } finally { - await server.close(); - } - }); - - it("scans all local rules silently when logged out (no reconcile)", async () => { - const server = await startMockServer(() => ({ statusCode: 200, body: {} })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory], - { - TASKLESS_TOKEN: "", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(server.requests).toHaveLength(0); // never reconciled - expect(exitCode).toBe(1); - expect(stdout).toContain("no-eval"); - expect(stdout).toContain("no-console"); - expect(stderr).not.toContain("unverified"); - } finally { - await server.close(); - } - }); - - it("forces the logged-out path under --anonymous even with a token", async () => { - const server = await startMockServer(() => ({ statusCode: 200, body: {} })); - try { - const { stdout, exitCode } = await runCli( - ["check", "-d", directory, "--anonymous"], - { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } - ); - expect(server.requests).toHaveLength(0); // --anonymous skips reconcile - expect(exitCode).toBe(1); - expect(stdout).toContain("no-eval"); - expect(stdout).toContain("no-console"); - } finally { - await server.close(); - } - }); -}); diff --git a/packages/cli/test/run-set.test.ts b/packages/cli/test/run-set.test.ts deleted file mode 100644 index eeddc9c6..00000000 --- a/packages/cli/test/run-set.test.ts +++ /dev/null @@ -1,131 +0,0 @@ -import { - mkdtemp, - rm, - mkdir, - writeFile, - readdir, - readFile, -} from "node:fs/promises"; -import { join } from "node:path"; -import { tmpdir } from "node:os"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; - -import { canonicalHash } from "../src/rules/rule-hash"; -import { - materializeRunDirectory, - selectRunFiles, - signRuleFiles, - type SignedRuleFile, -} from "../src/rules/run-set"; - -async function writeRule( - cwd: string, - name: string, - body: string -): Promise { - const rulesDirectory = join(cwd, ".taskless", "rules"); - await mkdir(rulesDirectory, { recursive: true }); - await writeFile(join(rulesDirectory, name), body, "utf8"); -} - -describe("signRuleFiles", () => { - let directory: string; - beforeEach(async () => { - directory = await mkdtemp(join(tmpdir(), "taskless-runset-")); - }); - afterEach(async () => { - await rm(directory, { recursive: true, force: true }); - }); - - it("signs each rule file with its canonical signature", async () => { - await writeRule(directory, "a.yml", "id: a\n"); - await writeRule(directory, "b.yml", "id: b\n"); - - const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); - - expect(signed.map((s) => s.file).toSorted()).toEqual(["a.yml", "b.yml"]); - for (const entry of signed) { - const expected = await canonicalHash(await readFile(entry.path, "utf8")); - expect(entry.signature).toBe(expected); - } - }); -}); - -describe("selectRunFiles", () => { - const signed: SignedRuleFile[] = [ - { file: "a.yml", path: "/x/a.yml", signature: "1;h=sha-256;d=aaaa" }, - { file: "b.yml", path: "/x/b.yml", signature: "1;h=sha-256;d=bbbb" }, - ]; - - it("selects only files whose signature the server blessed", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "a", file: "a.yml", signature: "1;h=sha-256;d=aaaa" }, - ]); - expect(runFiles.map((s) => s.file)).toEqual(["a.yml"]); - }); - - it("matches by signature, not path (a moved-but-unchanged file resolves)", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "b", file: "moved/b.yml", signature: "1;h=sha-256;d=bbbb" }, - ]); - expect(runFiles.map((s) => s.file)).toEqual(["b.yml"]); - }); - - it("drops run entries with no local signature match", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "c", file: "c.yml", signature: "1;h=sha-256;d=cccc" }, - ]); - expect(runFiles).toEqual([]); - }); - - it("does not select the same local file twice", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "a", file: "a.yml", signature: "1;h=sha-256;d=aaaa" }, - { ruleId: "a", file: "a-copy.yml", signature: "1;h=sha-256;d=aaaa" }, - ]); - expect(runFiles.map((s) => s.file)).toEqual(["a.yml"]); - }); -}); - -describe("materializeRunDirectory", () => { - let directory: string; - beforeEach(async () => { - directory = await mkdtemp(join(tmpdir(), "taskless-runset-")); - }); - afterEach(async () => { - await rm(directory, { recursive: true, force: true }); - }); - - it("copies only the selected files and gitignores .run/", async () => { - await writeRule(directory, "a.yml", "id: a\n"); - await writeRule(directory, "b.yml", "id: b\n"); - const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); - const onlyA = signed.filter((s) => s.file === "a.yml"); - - await materializeRunDirectory(directory, onlyA); - - const runRules = join(directory, ".taskless", ".run", "rules"); - expect(await readdir(runRules)).toEqual(["a.yml"]); - - const gitignore = await readFile( - join(directory, ".taskless", ".gitignore"), - "utf8" - ); - expect(gitignore).toContain(".run/"); - }); - - it("rebuilds the run dir, dropping stale files from a prior run", async () => { - await writeRule(directory, "a.yml", "id: a\n"); - await writeRule(directory, "b.yml", "id: b\n"); - const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); - - await materializeRunDirectory(directory, signed); // both - await materializeRunDirectory( - directory, - signed.filter((s) => s.file === "b.yml") - ); // only b - - const runRules = join(directory, ".taskless", ".run", "rules"); - expect(await readdir(runRules)).toEqual(["b.yml"]); - }); -}); diff --git a/packages/cli/test/runtime-check.test.ts b/packages/cli/test/runtime-check.test.ts new file mode 100644 index 00000000..81ef4b02 --- /dev/null +++ b/packages/cli/test/runtime-check.test.ts @@ -0,0 +1,329 @@ +import { execFile } from "node:child_process"; +import { createServer, type Server } from "node:http"; +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { resolve, join } from "node:path"; +import { tmpdir } from "node:os"; +import { promisify } from "node:util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const execFileAsync = promisify(execFile); +const binPath = resolve(import.meta.dirname, "../dist/index.js"); + +interface ReportedFile { + file: string; + signature: string; +} +interface ReconcileRequestBody { + repositoryUrl: string; + files: ReportedFile[]; +} +type Responder = (request: ReconcileRequestBody) => { + statusCode: number; + body?: unknown; +}; +interface MockServer { + apiUrl: string; + requests: ReconcileRequestBody[]; + close: () => Promise; +} + +/** Start a mock reconcile endpoint on a random port. */ +function startMockServer(responder: Responder): Promise { + const requests: ReconcileRequestBody[] = []; + const server: Server = createServer((request, response) => { + if (request.method !== "POST" || request.url !== "/cli/api/reconcile") { + response.writeHead(404).end("{}"); + return; + } + let raw = ""; + request.on("data", (chunk: Buffer) => (raw += chunk.toString())); + request.on("end", () => { + const parsed = JSON.parse(raw) as ReconcileRequestBody; + requests.push(parsed); + const { statusCode, body } = responder(parsed); + response.writeHead(statusCode, { "content-type": "application/json" }); + response.end(JSON.stringify(body ?? {})); + }); + }); + return new Promise((resolvePromise) => { + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + const port = typeof address === "object" && address ? address.port : 0; + resolvePromise({ + apiUrl: `http://127.0.0.1:${String(port)}/cli`, + requests, + close: () => new Promise((done) => server.close(() => done())), + }); + }); + }); +} + +/** Echo a reported file's signature back so the mock can bless it. */ +function sig(request: ReconcileRequestBody, endsWith: string): string { + return request.files.find((f) => f.file.endsWith(endsWith))?.signature ?? ""; +} + +async function runCli( + args: string[], + env: Record = {} +): Promise<{ stdout: string; stderr: string; exitCode: number }> { + try { + const { stdout, stderr } = await execFileAsync("node", [binPath, ...args], { + env: { ...process.env, ...env }, + }); + return { stdout, stderr, exitCode: 0 }; + } catch (error) { + const execError = error as { stdout: string; stderr: string; code: number }; + return { + stdout: execError.stdout ?? "", + stderr: execError.stderr ?? "", + exitCode: execError.code, + }; + } +} + +/** The check `--json` line, ignoring any preceding migration output. */ +function parseJson(stdout: string): { + success: boolean; + results: { source: string; ruleId: string }[]; + skipped?: { rule: string; reason: string }[]; +} { + const line = stdout + .trim() + .split("\n") + .findLast((l) => l.trim().startsWith("{")); + return JSON.parse(line ?? "{}") as { + success: boolean; + results: { source: string; ruleId: string }[]; + skipped?: { rule: string; reason: string }[]; + }; +} + +const STATIC_RULE = [ + "id: no-console", + "language: typescript", + "severity: warning", + "rule:", + " pattern: console.log($$$A)", + "message: avoid console.log", + "", +].join("\n"); + +const RUNTIME_CAPTURE = [ + "id: logs-abc12345", + "language: typescript", + "rule:", + " pattern: console.log($A)", + "metadata:", + " taskless:", + " version: 1", + " kind: runtime", + " name: logs", + " check: check.ts", + " match: anchor", + "", +].join("\n"); + +const RUNTIME_CHECK = `export default async function (root, matches) { + return matches.map((m) => ({ file: m.file, line: m.line, message: "runtime " + m.rule, severity: "warning" })); +} +`; + +const CHECK_REPORT_PATH = ".taskless/runtime-rules/demo/check.ts"; + +describe("check: static vs runtime dispatch", () => { + let directory: string; + + beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "tskl-rt-check-")); + const rules = join(directory, ".taskless", "rules"); + const runtime = join(directory, ".taskless", "runtime-rules", "demo"); + await mkdir(rules, { recursive: true }); + await mkdir(runtime, { recursive: true }); + await writeFile(join(rules, "no-console.yml"), STATIC_RULE, "utf8"); + await writeFile(join(runtime, "logs.yml"), RUNTIME_CAPTURE, "utf8"); + await writeFile(join(runtime, "check.ts"), RUNTIME_CHECK, "utf8"); + await writeFile(join(directory, "src.ts"), 'console.log("hi");\n', "utf8"); + await execFileAsync("git", ["init"], { cwd: directory }); + await execFileAsync( + "git", + ["remote", "add", "origin", "https://github.com/acme/widgets.git"], + { cwd: directory } + ); + }); + + afterEach(async () => { + await rm(directory, { recursive: true, force: true }); + }); + + it("logged out: static runs, runtime is skipped and reported in --json", async () => { + const { stdout, exitCode } = await runCli([ + "check", + "-d", + directory, + "--json", + ]); + const output = parseJson(stdout); + expect(exitCode).toBe(0); // only warnings + const ids = new Set(output.results.map((r) => r.ruleId)); + expect(ids.has("no-console")).toBe(true); // static always runs + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + false + ); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + }); + + it("logged out: skip notice on stderr, static findings on stdout", async () => { + const { stdout, stderr } = await runCli(["check", "-d", directory]); + expect(stderr).toContain("was not run"); + expect(stdout).toContain("no-console"); + }); + + it("authed + blessed check.ts: runtime runs; only check.ts is reported", async () => { + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + run: [ + { + ruleId: "demo", + file: CHECK_REPORT_PATH, + signature: sig(request, "check.ts"), + }, + ], + unsafe: [], + unknown: [], + missing: [], + }, + })); + try { + const { stdout } = await runCli(["check", "-d", directory, "--json"], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + }); + // Only the runtime check.ts is reported — never the static rule. + expect(server.requests).toHaveLength(1); + expect(server.requests[0]!.files).toHaveLength(1); + expect(server.requests[0]!.files[0]!.file.endsWith("check.ts")).toBe( + true + ); + const output = parseJson(stdout); + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + true + ); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + } finally { + await server.close(); + } + }); + + it("authed + empty run set: runtime withheld, static still runs", async () => { + const server = await startMockServer(() => ({ + statusCode: 200, + body: { run: [], unsafe: [], unknown: [], missing: [] }, + })); + try { + const { stdout } = await runCli(["check", "-d", directory, "--json"], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + }); + const output = parseJson(stdout); + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + false + ); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + } finally { + await server.close(); + } + }); + + it("reconcile unavailable: runtime skipped, static runs, exit 0", async () => { + const server = await startMockServer(() => ({ statusCode: 503 })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + const output = parseJson(stdout); + expect(exitCode).toBe(0); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + } finally { + await server.close(); + } + }); + + it("--anonymous with a token: skips runtime and never calls reconcile", async () => { + const server = await startMockServer(() => ({ + statusCode: 200, + body: { run: [], unsafe: [], unknown: [], missing: [] }, + })); + try { + const { stdout } = await runCli( + ["check", "-d", directory, "--json", "--anonymous"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + expect(server.requests).toHaveLength(0); + const output = parseJson(stdout); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + } finally { + await server.close(); + } + }); + + it("--dangerously-run-scripts: runs runtime offline behind a warning", async () => { + const { stdout, stderr } = await runCli([ + "check", + "-d", + directory, + "--dangerously-run-scripts", + ]); + expect(stderr).toContain("dangerously-run-scripts"); + expect(stdout).toContain("demo"); // runtime finding surfaced + }); + + it("a runtime rule missing check.ts is skipped, not fatal; static still runs", async () => { + // A malformed rule (capture yml, no check.ts) must not abort the whole check. + const broken = join(directory, ".taskless", "runtime-rules", "broken"); + await mkdir(broken, { recursive: true }); + await writeFile(join(broken, "logs.yml"), RUNTIME_CAPTURE, "utf8"); + + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + run: [ + { + ruleId: "demo", + file: CHECK_REPORT_PATH, + signature: sig(request, "check.ts"), + }, + ], + unsafe: [], + unknown: [], + missing: [], + }, + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + } + ); + const output = parseJson(stdout); + expect(exitCode).toBe(0); // not SCAN_FAILED + // The good runtime rule still ran and static still ran. + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + true + ); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + // The broken rule is reported as skipped, not crashed. + expect(output.skipped?.some((s) => s.rule === "broken")).toBe(true); + // Only the readable check.ts was reported to the server. + expect(server.requests[0]!.files).toHaveLength(1); + } finally { + await server.close(); + } + }); +}); diff --git a/packages/cli/test/runtime-harness.test.ts b/packages/cli/test/runtime-harness.test.ts new file mode 100644 index 00000000..5568fef2 --- /dev/null +++ b/packages/cli/test/runtime-harness.test.ts @@ -0,0 +1,190 @@ +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { discoverRuntimeRules } from "../src/rules/runtime/discover"; +import { executeRuntimeRule } from "../src/rules/runtime/harness"; + +/** Build a capture `*.yml` with a `metadata.taskless` runtime block. */ +function capture(options: { + id: string; + name: string; + pattern: string; + match?: "anchor" | "broad"; + rule?: string; +}): string { + const ruleBlock = options.rule ?? ` pattern: ${options.pattern}`; + return [ + `id: ${options.id}`, + "language: typescript", + "rule:", + ruleBlock, + "metadata:", + " taskless:", + " version: 1", + " kind: runtime", + ` name: ${options.name}`, + " check: check.ts", + ` match: ${options.match ?? "anchor"}`, + "", + ].join("\n"); +} + +/** Write a runtime rule directory (capture files + check.ts) under the repo. */ +async function writeRuntimeRule( + root: string, + name: string, + captures: Record, + check: string +): Promise { + const directory = join(root, ".taskless", "runtime-rules", name); + await mkdir(directory, { recursive: true }); + for (const [file, body] of Object.entries(captures)) { + await writeFile(join(directory, file), body, "utf8"); + } + await writeFile(join(directory, "check.ts"), check, "utf8"); +} + +/** A check that echoes each match back as a finding (proves it was invoked). */ +const ECHO_CHECK = `export default async function (root, matches) { + return matches.map((m) => ({ + file: m.file, + line: m.line, + column: m.column, + message: "rule=" + m.rule + " cap=" + JSON.stringify(m.captures), + severity: "warning", + })); +} +`; + +describe("runtime harness", () => { + let root: string; + + beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "tskl-harness-")); + }); + afterEach(async () => { + await rm(root, { recursive: true, force: true }); + }); + + it("discovers a runtime rule directory", async () => { + await writeRuntimeRule( + root, + "logs", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + ECHO_CHECK + ); + const rules = await discoverRuntimeRules(root); + expect(rules).toHaveLength(1); + expect(rules[0]!.captureRules[0]!.name).toBe("logs"); + expect(rules[0]!.captureRules[0]!.match).toBe("anchor"); + expect(rules[0]!.checkFile.endsWith("check.ts")).toBe(true); + }); + + it("gates on matches: zero matches never invokes check.ts", async () => { + // The source has no console.log, so the narrow yields nothing and the echo + // check (which would return a finding per match) must never run. + await writeFile(join(root, "src.ts"), "const x = 1;\n", "utf8"); + await writeRuntimeRule( + root, + "logs", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + ECHO_CHECK + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!); + expect(results).toHaveLength(0); + }); + + it("normalizes matches (1-indexed, rule name, captures) and maps findings to 0-indexed range", async () => { + await writeFile( + join(root, "src.ts"), + 'const x = 1;\nconsole.log("hi");\n', + "utf8" + ); + await writeRuntimeRule( + root, + "logs", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + ECHO_CHECK + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!); + expect(results).toHaveLength(1); + const finding = results[0]!; + expect(finding.source).toBe("taskless-runtime"); + expect(finding.ruleId).toBe("logs"); + expect(finding.file).toBe("src.ts"); + // The match is on line 2 (1-indexed); the echo check returns m.line, which + // the harness converts down to the 0-indexed CheckResult range. + expect(finding.range.start.line).toBe(1); + expect(finding.message).toContain("rule=logs"); + // captures echoed through: the single metavariable A holds the string "hi". + expect(finding.message).toContain("cap="); + expect(finding.message).toContain("hi"); + }); + + it("isolates a throwing check into a single error finding", async () => { + await writeFile(join(root, "src.ts"), 'console.log("hi");\n', "utf8"); + await writeRuntimeRule( + root, + "boom", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + `export default async function () { throw new Error("kaboom"); }\n` + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!); + expect(results).toHaveLength(1); + expect(results[0]!.severity).toBe("error"); + expect(results[0]!.message).toContain("boom"); + }); + + it("times out a slow check into an error finding", async () => { + await writeFile(join(root, "src.ts"), 'console.log("hi");\n', "utf8"); + await writeRuntimeRule( + root, + "slow", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + `export default async function () { + await new Promise((r) => setTimeout(r, 5000)); + return []; + }\n` + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!, { timeoutMs: 200 }); + expect(results).toHaveLength(1); + expect(results[0]!.severity).toBe("error"); + expect(results[0]!.message).toContain("timed out"); + }, 15_000); +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index bbbc1243..5261fdde 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -98,6 +98,9 @@ importers: sprintf-js: specifier: ^1.1.3 version: 1.1.3 + tsx: + specifier: ^4.21.0 + version: 4.21.0 yaml: specifier: ^2.8.2 version: 2.8.2