From 673724ac1e79c247fcf4b8279dca138f0b7805c9 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 2 Jul 2026 18:22:13 -0700 Subject: [PATCH 01/21] feat(cli): add canonical rule-hash signature module normalize() + canonicalHash() + parseSignature() implementing the algoVersion-1 signature envelope for server-owned rule reconciliation (TSKL-270), on web-standard crypto only to match the server reference. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../.openspec.yaml | 2 + .../design.md | 156 +++++++++++++ .../proposal.md | 66 ++++++ .../specs/cli-check/spec.md | 135 ++++++++++++ .../specs/cli-rule-reconciliation/spec.md | 208 ++++++++++++++++++ .../server-owned-rule-reconciliation/tasks.md | 48 ++++ packages/cli/src/rules/rule-hash.ts | 117 ++++++++++ 7 files changed, 732 insertions(+) create mode 100644 openspec/changes/server-owned-rule-reconciliation/.openspec.yaml create mode 100644 openspec/changes/server-owned-rule-reconciliation/design.md create mode 100644 openspec/changes/server-owned-rule-reconciliation/proposal.md create mode 100644 openspec/changes/server-owned-rule-reconciliation/specs/cli-check/spec.md create mode 100644 openspec/changes/server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md create mode 100644 openspec/changes/server-owned-rule-reconciliation/tasks.md create mode 100644 packages/cli/src/rules/rule-hash.ts diff --git a/openspec/changes/server-owned-rule-reconciliation/.openspec.yaml b/openspec/changes/server-owned-rule-reconciliation/.openspec.yaml new file mode 100644 index 00000000..43e65ca6 --- /dev/null +++ b/openspec/changes/server-owned-rule-reconciliation/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-03 diff --git a/openspec/changes/server-owned-rule-reconciliation/design.md b/openspec/changes/server-owned-rule-reconciliation/design.md new file mode 100644 index 00000000..3ebb51d5 --- /dev/null +++ b/openspec/changes/server-owned-rule-reconciliation/design.md @@ -0,0 +1,156 @@ +## Context + +`taskless check` today (`packages/cli/src/commands/check.ts`) enumerates every `*.yml` under +`.taskless/rules/`, writes a fixed `.taskless/sgconfig.yml` with `ruleDirs: [rules]` +(`src/filesystem/sgconfig.ts`), and shells out to `sg scan` (`src/rules/scan.ts`). It has no +notion of authenticity, no signing, no network call, and no auth dependency — `--anonymous` +is a pure no-op. There is no rule-hash code, no conformance harness, and the generated API +schema (`src/generated/api.d.ts`) exposes only `whoami` / `rule` / `rule/{id}` / +`rule/{id}/iterate`. Reconciliation is entirely greenfield. + +The backend (TSKL-270) is introducing `POST /cli/api/reconcile`: the CLI reports the rule +files it holds as `{ file, signature }`, and the server returns the exact subset that may run. +The signature format, the `normalize()` procedure, and the conformance vectors are **frozen** +and specified in `tmp/rule-signatures.md`; the endpoint may not be live in every environment +on day one. The existing plumbing we build on: `resolveIdentity(cwd)` → +`{ token, orgId, repositoryUrl }` (`src/auth/identity.ts`), `getToken(cwd)` +(`src/auth/token.ts`), the `openapi-fetch` client + base-URL resolution +(`src/api/client.ts`, `src/api/config.ts`), and the stable error envelope +(`src/types/errors.ts`). + +## Goals / Non-Goals + +**Goals:** + +- A signature module that reproduces the server's `rule-hash.ts` byte-for-byte, verified by + the shared conformance vectors. +- A reconcile client for `POST /cli/api/reconcile` returning typed `run`/`unsafe`/`unknown`/ + `missing` buckets. +- `check` executes only the `run` set (matched by signature) when it can reconcile, and + surfaces the other buckets as advisory. +- A fallback that keeps `check` working offline / logged-out / `--anonymous` / before the + endpoint ships, preserving today's linter posture. + +**Non-Goals:** + +- Defining or implementing how a **server-owned/runtime rule executes**. The reconciliation + contract is rule-type-agnostic (it gates which files run). Rule execution is a deliberate, + team-owned follow-up shipped as a **separate proposal stacked on top of this change**; this + change only makes such rules gate-able. +- A `sync` / `pull` / `list` command to bulk-download rules (none exists today; reconciliation + does not require one — the CLI reports what is already on disk). +- Server-side approval of `unknown` files (explicitly server-side future work). +- Changing the on-disk rule naming scheme. The reconcile join key is the signature, not the + path, so `.yml` naming is fine as-is. + +## Decisions + +### Decision: New `rule-hash.ts` mirroring the server, web-standard crypto only + +Add `packages/cli/src/rules/rule-hash.ts` exporting `ALGO_VERSION`, `normalize(text)`, +`canonicalHash(bytes|text)` (returns the envelope string), and `parseSignature(sig)`. Use +`crypto.subtle.digest('SHA-256', …)` + `TextEncoder` exclusively — no `node:crypto`. This +matches the server reference (`packages/shared/src/rule-hash.ts`) and runs identically in +workerd and Node 20+. `normalize()` operates on raw decoded text (BOM strip, CRLF/CR→LF, +collapse trailing newlines to one LF) and never parses YAML, so it is future-proof for any +rule type. + +_Alternative rejected:_ `node:crypto` `createHash`. Simpler locally but diverges from the +web-standard reference the server pins, and risks subtle cross-repo drift the vectors exist to +prevent. + +### Decision: Conformance test fetches vectors, commits a fixture, asserts exact reproduction + +Add a script/step to fetch `GET /cli/api/rule-hash-vectors` and commit the result as +`packages/cli/test/fixtures/rule-hash-vectors.json`, plus a vitest test +(`test/rule-hash.test.ts`) that parses each `input` as JSON (decoding `\uXXXX`) and asserts +`canonicalHash(input) === signature`. A mismatch fails the build. Committing the fixture (vs. +fetching at test time) keeps tests hermetic/offline; refresh is a manual step when the algo +version bumps, mirroring `generate:ast-grep-schema`. + +_Alternative rejected:_ fetch vectors live during the test run — introduces network flakiness +into CI and couples the unit suite to endpoint availability. + +### Decision: Reconcile client via a hand-typed request over the existing fetch layer + +The reconcile endpoint is not in the generated `paths`, and the handoff says it may not be +deployed everywhere yet. Add `packages/cli/src/api/reconcile.ts` exporting +`reconcile(token, { repositoryUrl, files })`. Reuse `createApiClient(token)` where possible; +because the path is absent from the generated schema, type the request/response with local +interfaces (`ReconcileRequest`, `ReconcileResponse` with `run`/`unsafe`/`unknown`/`missing`) +and issue the call with an explicit `Authorization: Bearer` header against `getApiBaseUrl()`'s +origin. Map a `401` to an unauthorized signal and any transport/`404`/not-deployed outcome to +a distinct "reconcile unavailable" signal that drives the fallback (never a hard failure). +When the endpoint lands in the published schema, this can be migrated onto the typed client +without changing `check`. + +_Alternative rejected:_ add the path to `api.d.ts` by hand — the file is generated +(`pnpm generate:api`) and hand-edits would be clobbered; wait for the server schema to expose +it, then regenerate. + +### Decision: Gate the scan by materializing a run-set rule directory + +`sg scan` selects rules via `ruleDirs` in `sgconfig.yml`, so limiting execution to the `run` +set means pointing ast-grep at only those files. Materialize an ephemeral, gitignored +`.taskless/.run/rules/` containing just the blessed files (copied from their local matches by +signature), generate an sgconfig whose `ruleDirs` points there, and scan that. Keeps +`src/rules/scan.ts` unchanged (it already accepts a config path and positional paths) and +avoids mutating the user's `.taskless/rules/`. On fallback, generate the current sgconfig +(`ruleDirs: [rules]`) and scan everything, exactly as today. + +_Alternatives considered:_ (a) delete non-run files in place — destructive, unacceptable; +(b) pass each blessed rule as an inline `--rule` arg — brittle across ast-grep versions and +loses `testConfigs`. The ephemeral-dir approach is the least invasive and reuses existing +config generation. + +### Decision: Auth state is the behavior axis; degrade (not fail) when authed reconcile can't complete + +`check` requires no auth and picks its path from auth state: + +- **No token** (or `--anonymous`) → run all local rules with no network. This is the normal + offline linter posture, so it emits no warning (an informational line is optional). +- **Authenticated** (token + resolvable `repositoryUrl`, `--anonymous` unset) → reconcile, + run only the `run` set, and **warn** on `unsafe`/`unknown`/`missing` mismatches. +- **Authenticated but reconcile can't complete** (no git remote, endpoint unreachable / + not-deployed, transport error) → **degrade**: warn that verification couldn't be performed + and scan all local rules, without a non-zero exit. + +All warnings are human-output only and suppressed under `--json` to keep the machine shape +stable. This honors the handoff's trust model — local `check` stays advisory like a linter, +the server-owned allow-list applies when authenticated, and the paid-plan CI backstop is the +real enforcement point — and prevents the not-yet-live endpoint from bricking `check`. +Separating "no auth" (silent, expected) from "authed-but-couldn't-verify" (warned) keeps +routine offline use quiet while still flagging a genuine verification gap. + +### Decision: Add `RECONCILE_FAILED` to the error enum, used sparingly + +Extend `CLIErrorCode` in `src/types/errors.ts` with `RECONCILE_FAILED`. Because reconcile +failure normally triggers the fallback (not an error), this code is reserved for the case +where reconciliation itself is the requested operation and hard-fails in a way the user asked +to be surfaced. Adding a code is permitted by the `cli` capability without a major bump. + +## Risks / Trade-offs + +- **[Endpoint not deployed on day one]** → Fallback treats unreachable/404 as "reconcile + unavailable" and scans locally; no behavior regression versus today until the endpoint + ships. +- **[Signature drift from the server reference]** → Committed conformance vectors + a + build-blocking test catch any divergence in `normalize()`/hashing before release. +- **[`check` gains a network + auth dependency]** → Reconciliation is strictly additive and + gated; unauthenticated and `--anonymous` runs keep working with no auth via the fallback. +- **[Ephemeral run-dir leaks into git]** → Write under `.taskless/.run/` and ensure + `.taskless/.gitignore` covers it (same mechanism that hides `sgconfig.yml`); regenerate on + every run. +- **[Users read the fallback as "verified"]** → The one-line notice explicitly states rules + are unverified and names the CI backstop as the enforcement point. +- **[Reconcile latency on large corpora]** → One request per `check`; signatures are cheap + SHA-256 over small files. Acceptable; can batch/cache later if needed. + +## Open Questions + +- **Warning verbosity**: how loudly to warn on `unknown`/`missing` in human output (always, + or behind a `--verbose`/`--strict` flag) — resolve during implementation against the help + copy for `check`/`ci`. `unsafe` (tamper/drift) should warn by default. +- **CI `--strict` mode**: whether the CI backstop invocation should turn `unsafe` into a + non-zero exit (the enforcement posture) versus advisory locally. Defer unless the backstop + spec requires it here. diff --git a/openspec/changes/server-owned-rule-reconciliation/proposal.md b/openspec/changes/server-owned-rule-reconciliation/proposal.md new file mode 100644 index 00000000..55af9bcc --- /dev/null +++ b/openspec/changes/server-owned-rule-reconciliation/proposal.md @@ -0,0 +1,66 @@ +## Why + +The backend is moving to **server-owned rule reconciliation** (TSKL-270): the server, not +the CLI, decides which rule files may run. This is the CLI-side enabler for a new class of +server-blessed rules (including runtime rules) — instead of executing whatever YAML happens +to sit in `.taskless/rules/`, the CLI must report the rule files it holds and execute only +the subset the server returns as `run`. Today `taskless check` runs every `.yml` in the +rules directory with no notion of authenticity or drift; the frozen `POST /cli/api/reconcile` +contract lets us close that gap now, ahead of the endpoint going live in every environment. + +## What Changes + +- Add a canonical **rule-signature** module (`normalize()` + `canonicalHash()` + + `parseSignature()`, algoVersion `1`, `1;h=sha-256;d=`) built on web-standard + `crypto.subtle` + `TextEncoder`, byte-for-byte matching the server reference. +- Add a **conformance test** that fetches `GET /cli/api/rule-hash-vectors`, commits the + fixture, and asserts the local hasher reproduces every vector's signature exactly (a + mismatch is a release blocker). +- Add a **reconcile API client** for `POST /cli/api/reconcile` that reports every rule file + under `.taskless/rules/` as `{ file, signature }` and parses the `run` / `unsafe` / + `unknown` / `missing` response. +- **Make `taskless check` behavior depend on auth state** (it still requires no auth): + - **No token** → run all local rules, no network (today's offline linter posture). + - **Authenticated** → reconcile, execute **exactly** the server's `run` set (matched back + to local files by signature), and **warn on mismatches** (`unsafe` / `unknown` / + `missing`) without changing the exit code. + - **`--anonymous`** → force the logged-out path (run local, no reconcile call). +- Add a **degrade path**: when an authenticated reconciliation cannot complete (no git + remote, or the endpoint is unreachable / not yet deployed), `check` warns that verification + could not be performed and falls back to a local scan without failing — so the not-yet-live + endpoint never bricks `check`. +- Add a stable `RECONCILE_FAILED` error code to the CLI error enum for `--json` callers. + +Out of scope (deliberately, by team decision): how a server-owned rule (including a runtime +rule) _executes_. The reconciliation contract is rule-type-agnostic — it decides which files +run, not how a runtime rule is evaluated. Rule execution is a **separate proposal, landed as a +stacked PR on top of this work**; this change only makes such rules gate-able. + +## Capabilities + +### New Capabilities + +- `cli-rule-reconciliation`: the rule-signature envelope and `normalize()` procedure, the + conformance-vector contract, the `POST /cli/api/reconcile` client, the four response + buckets and their required CLI actions, and the run-set-only execution rule. + +### Modified Capabilities + +- `cli-check`: `check` chooses its behavior from auth state — unauthenticated/`--anonymous` + runs all local rules; authenticated reconciles and executes only the `run` set, warning on + `unsafe` / `unknown` / `missing`, and degrading to a local scan if reconciliation can't complete. + +## Impact + +- **Code:** new `packages/cli/src/rules/rule-hash.ts` and reconcile client under + `packages/cli/src/api/`; changes to `src/commands/check.ts` (reconcile-then-gate flow) and + `src/filesystem/sgconfig.ts` / rule enumeration (scan only the blessed set); new error code + in `src/types/errors.ts`. +- **APIs consumed:** `POST /cli/api/reconcile` (Bearer ``, `repositoryUrl` + + reported files) and `GET /cli/api/rule-hash-vectors` (unauthenticated, for conformance). +- **Behavioral shift:** `check` gains an auth-state-dependent path — authenticated runs get a + server-owned allow-list and mismatch warnings; unauthenticated/`--anonymous` runs are + unchanged (all local rules, no network). +- **Tests:** new conformance-vector test and reconcile/gating tests (vitest, subprocess + against `dist/`, temp-dir fixtures per existing conventions). +- **Docs:** `check.txt` / `ci.txt` help updated to explain the run-set gate and the CI backstop. diff --git a/openspec/changes/server-owned-rule-reconciliation/specs/cli-check/spec.md b/openspec/changes/server-owned-rule-reconciliation/specs/cli-check/spec.md new file mode 100644 index 00000000..ed1a0691 --- /dev/null +++ b/openspec/changes/server-owned-rule-reconciliation/specs/cli-check/spec.md @@ -0,0 +1,135 @@ +## ADDED Requirements + +### Requirement: Check selects what it runs from auth state + +`taskless check` SHALL NOT require authentication, and it SHALL choose what it runs from the current auth state. When no token is available, or when `--anonymous` is set, the CLI SHALL run all local rule files under `.taskless/rules/` without contacting the server (the offline linter posture). When a token is available and `--anonymous` is not set, the CLI SHALL reconcile against the server (per "Check reconciles rule files before scanning") and run only the returned `run` set. The unauthenticated path SHALL succeed with no network access and SHALL NOT emit a warning; any informational notice on that path is optional. + +#### Scenario: Unauthenticated check runs local rules + +- **WHEN** a user runs `taskless check` with no available token +- **THEN** the CLI SHALL scan all local rule files +- **AND** SHALL NOT call `POST /cli/api/reconcile` +- **AND** SHALL NOT emit a warning about missing authentication + +#### Scenario: Authenticated check reconciles + +- **WHEN** a user runs `taskless check` with an available token and without `--anonymous` +- **THEN** the CLI SHALL call `POST /cli/api/reconcile` and run only the `run` set + +#### Scenario: Anonymous forces the logged-out path + +- **WHEN** a user runs `taskless check --anonymous` while a token is available +- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (run all local rules, no reconcile call) + +### Requirement: Check reconciles rule files before scanning + +`taskless check` SHALL reconcile before scanning whenever a bearer token and a `repositoryUrl` are resolvable and `--anonymous` is not set. It SHALL compute the signature envelope for every `.yml` file under `.taskless/rules/`, call `POST /cli/api/reconcile` with `{ repositoryUrl, files }`, and then scan **only** the files returned in the `run` set, matched back to local files by signature (per the `cli-rule-reconciliation` capability). + +#### Scenario: Only blessed rules are scanned + +- **WHEN** a user runs `taskless check` while authenticated and reconciliation returns a + `run` set that is a subset of the local rule files +- **THEN** the CLI SHALL invoke `sg scan` against only the `run`-set rule files +- **AND** SHALL NOT scan any local rule file absent from `run` + +### Requirement: Check warns on reconciliation mismatches + +`taskless check` SHALL warn on every mismatch reported by a successful reconciliation and SHALL NOT let those warnings change the exit code. `unsafe` entries SHALL be warned as tamper/drift, `unknown` as not-issued-by-the-server, and `missing` as audit-only. The exit code SHALL continue to be governed solely by error-severity scan results from the executed `run` set. Warnings SHALL be human-readable output only; when `--json` is set the output SHALL remain the existing `{ success, results }` shape and SHALL NOT include the warnings. + +#### Scenario: Unsafe drift is warned without failing the exit code + +- **WHEN** reconciliation returns an `unsafe` entry and the `run`-set scan produces no + error-severity results +- **THEN** the CLI SHALL warn about the drift +- **AND** SHALL exit with code 0 + +#### Scenario: Missing rules are warned as audit-only + +- **WHEN** reconciliation returns `missing` entries +- **THEN** the CLI SHALL warn about them as audit information +- **AND** SHALL NOT change the exit code on their account + +#### Scenario: Warnings are suppressed under --json + +- **WHEN** reconciliation returns mismatches and `--json` is set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** SHALL NOT contain the human-readable mismatch warnings + +### Requirement: Check degrades to a local scan when reconciliation cannot complete + +`taskless check` SHALL NOT fail solely because an attempted reconciliation cannot complete. When a token is available and `--anonymous` is not set but reconciliation cannot complete (no resolvable git remote, or the reconcile endpoint is unreachable or not yet deployed, or a transport error), the CLI SHALL warn that rule verification could not be performed and SHALL fall back to scanning all local rule files. The CLI SHALL NOT exit with a non-zero code solely because reconciliation failed, and the warning SHALL be suppressed under `--json`. + +#### Scenario: Endpoint unreachable degrades to local scan + +- **WHEN** an authenticated `check` attempts reconciliation and the endpoint is unreachable or returns a not-deployed error +- **THEN** the CLI SHALL warn that verification could not be performed +- **AND** SHALL scan all local rule files +- **AND** SHALL NOT exit with a non-zero code solely due to the reconcile failure + +#### Scenario: Degrade warning is suppressed under --json + +- **WHEN** the CLI degrades to a local scan and `--json` is set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** SHALL NOT contain the human-readable degrade warning + +### Requirement: Check exits cleanly when the run set is empty + +`taskless check` SHALL NOT invoke the scanner when a successful reconciliation returns an empty `run` set (for example an empty corpus, or every reported file classified `unsafe`/`unknown`); it SHALL report no scan results and SHALL exit with code 0. Any `unsafe` / `unknown` mismatch warnings SHALL still be emitted. + +#### Scenario: Empty run set skips the scan + +- **WHEN** reconciliation returns an empty `run` set +- **THEN** the CLI SHALL NOT invoke `sg scan` +- **AND** SHALL exit with code 0 +- **AND** SHALL still warn about any `unsafe`/`unknown` mismatches + +## MODIFIED Requirements + +### Requirement: Check subcommand executes ast-grep scan + +The CLI SHALL generate an ephemeral `sgconfig.yml` in `.taskless/` and execute +`sg scan --config .taskless/sgconfig.yml --json=stream` using `child_process.spawn` with +`shell: true` for cross-platform binary resolution. The `sg` binary SHALL be resolved from +the `@ast-grep/cli` dependency via PATH. When reconciliation succeeds, the scan SHALL cover +only the blessed `run`-set rule files; on the unauthenticated/`--anonymous` path, or when an +authenticated reconciliation degrades to a local scan, the scan SHALL cover all local rule +files as before. + +#### Scenario: ast-grep scan runs with generated config + +- **WHEN** the CLI executes the scanner +- **THEN** it SHALL first write `.taskless/sgconfig.yml` +- **AND** it SHALL invoke `sg scan` with `--config .taskless/sgconfig.yml` and `--json=stream` +- **AND** the working directory for the spawned process SHALL be the resolved project directory + +#### Scenario: Scan is limited to the run set when reconciled + +- **WHEN** reconciliation succeeded and returned a `run` set +- **THEN** the generated scan configuration SHALL cause `sg scan` to evaluate only the + `run`-set rule files + +#### Scenario: ast-grep binary is not found + +- **WHEN** the `sg` binary cannot be resolved from PATH +- **THEN** the CLI SHALL print an error message indicating ast-grep is not available +- **AND** the CLI SHALL exit with code 1 + +### Requirement: Check accepts --anonymous as a no-op + +The `taskless check` command SHALL accept the global `--anonymous` flag (per the `cli` +capability). Because `check` reconciles against the Taskless API when authenticated, +`--anonymous` SHALL force the logged-out path: it SHALL suppress the reconcile network call +and run all local rule files. Aside from forcing the logged-out path, `--anonymous` SHALL NOT +change scan behavior, output shape, or exit codes relative to an unauthenticated `check`. + +#### Scenario: check --anonymous skips reconciliation + +- **WHEN** a user runs `taskless check --anonymous` +- **THEN** the CLI SHALL NOT call `POST /cli/api/reconcile` +- **AND** SHALL scan all local rule files + +#### Scenario: check --anonymous matches an unauthenticated check + +- **WHEN** a user runs `taskless check --anonymous` +- **THEN** its scan behavior, output, and exit code SHALL match `taskless check` run with no + available token diff --git a/openspec/changes/server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md b/openspec/changes/server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md new file mode 100644 index 00000000..de609e1b --- /dev/null +++ b/openspec/changes/server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md @@ -0,0 +1,208 @@ +## ADDED Requirements + +### Requirement: Canonical rule signature envelope + +The CLI SHALL represent a rule file's canonical signature as a single self-describing +string of the form `;h=;d=`. For algoVersion `1` this is +`1;h=sha-256;d=`, where `` is the digest as lowercase hexadecimal. The token +before the **first** `;` is the algoVersion and SHALL be read up to that one delimiter to +detect the version (and therefore the normalization procedure and hash algorithm) before +any `key=value` parameters are parsed. Signatures SHALL be compared as whole strings. + +#### Scenario: Envelope is emitted for algoVersion 1 + +- **WHEN** the CLI computes a signature for a rule file's bytes using algoVersion 1 +- **THEN** the result SHALL be a string `1;h=sha-256;d=` with `` lowercase + +#### Scenario: Version is read before parameters + +- **WHEN** the CLI parses a signature string +- **THEN** it SHALL read the algoVersion as the substring before the first `;` +- **AND** SHALL NOT rely on the `key=value` parameter syntax to determine the version + +#### Scenario: Signatures compare as whole strings + +- **WHEN** the CLI compares two signatures for equality +- **THEN** it SHALL compare the full envelope strings, not the bare digests + +### Requirement: Signature normalization procedure (algoVersion 1) + +The CLI SHALL compute an algoVersion-1 digest as `SHA-256( normalize(fileText) )`, +hex-encoded lowercase, wrapped in the envelope. `normalize()` SHALL operate on raw decoded +text only and SHALL NOT parse or re-serialize YAML. In order, `normalize()` SHALL: + +1. Decode the file as UTF-8. +2. Strip a single leading UTF-8 byte-order mark if present (decoded as `U+FEFF`). +3. Convert every CRLF and lone CR to LF. +4. Strip all trailing newlines, then append exactly one LF. +5. Re-encode as UTF-8 and SHA-256, hex lowercase. + +The CLI SHALL NOT apply Unicode normalization (NFC/NFD): canonically-equivalent strings in +different composition forms SHALL hash differently. A change to `normalize()` SHALL ship as +a new algoVersion, never as a redefinition of an existing one. + +#### Scenario: CRLF and LF hash identically + +- **WHEN** two files differ only in CRLF versus LF line endings +- **THEN** their algoVersion-1 signatures SHALL be equal + +#### Scenario: Trailing newlines are collapsed to one + +- **WHEN** two files differ only in the number of trailing newlines (including none) +- **THEN** their algoVersion-1 signatures SHALL be equal + +#### Scenario: Leading BOM is stripped + +- **WHEN** a file has a leading UTF-8 BOM and an otherwise identical file does not +- **THEN** their algoVersion-1 signatures SHALL be equal + +#### Scenario: Meaningful content change differs + +- **WHEN** two files differ in any non-newline, non-BOM byte +- **THEN** their algoVersion-1 signatures SHALL differ + +#### Scenario: Combining marks are not NFC-folded + +- **WHEN** one file contains a precomposed character and another the decomposed form +- **THEN** their algoVersion-1 signatures SHALL differ + +### Requirement: Signature hashing uses web-standard APIs only + +The signature implementation SHALL use only web-standard APIs (`crypto.subtle.digest('SHA-256', …)` +and `TextEncoder`) and SHALL NOT depend on a Node-specific crypto module, so the CLI +reproduces the server's reference implementation byte-for-byte. + +#### Scenario: No node-specific crypto dependency + +- **WHEN** the signature module hashes a file's normalized bytes +- **THEN** it SHALL use `crypto.subtle` and `TextEncoder` +- **AND** SHALL NOT import `node:crypto` + +### Requirement: Conformance vectors are fetched and asserted + +The CLI SHALL consume the cross-repo conformance vectors served at +`GET /cli/api/rule-hash-vectors` (unauthenticated) as `{ vectors: [{ name, input, signature }] }`, +commit a copy as a fixture, and assert in its test suite that its independent +`normalize()`-plus-hash reproduces every vector's `signature` exactly. Non-ASCII `input` +SHALL be parsed as JSON (decoding `\uXXXX` escapes) before hashing. A vector mismatch SHALL +be a release blocker (the test SHALL fail the build). + +#### Scenario: Local hasher reproduces every vector + +- **WHEN** the conformance test runs against the committed vectors +- **THEN** the CLI SHALL compute the exact `signature` for every vector entry + +#### Scenario: A mismatch blocks release + +- **WHEN** any vector's computed signature does not match its expected `signature` +- **THEN** the test SHALL fail +- **AND** the build SHALL NOT pass + +### Requirement: Reconcile reports every held rule file + +The CLI SHALL reconcile by sending `POST /cli/api/reconcile` with an +`Authorization: Bearer ` header and a JSON body `{ repositoryUrl, files }`, where +`repositoryUrl` is the full repository URL and `files` is an array of +`{ file, signature }` covering **every** rule file the CLI would otherwise run under +`.taskless/rules/`. `file` SHALL be the rule file's delivered name as it exists on disk and +`signature` SHALL be the full envelope computed for that file's bytes. The CLI SHALL send the +whole signature envelope, not a bare digest. + +#### Scenario: All rule files are reported + +- **WHEN** the CLI reconciles with rule files present under `.taskless/rules/` +- **THEN** the request body SHALL include one `{ file, signature }` entry per rule file + +#### Scenario: Full envelope is sent + +- **WHEN** the CLI reports a file's signature +- **THEN** it SHALL send the complete `1;h=sha-256;d=` envelope, not only the digest + +### Requirement: Reconcile response buckets drive execution + +The reconcile response SHALL be interpreted as four buckets — `run`, `unsafe`, `unknown`, +and `missing` — and each SHALL drive a specific CLI action: + +- `run`: the file's content matches a rule the server blessed. The CLI SHALL execute it. This + is the complete allow-list. +- `unsafe`: a rule held by delivered name whose content differs from what the server blessed + (`expected` vs `got`). The CLI SHALL NOT run it and SHALL surface it as tamper/drift. +- `unknown`: a reported file the server never issued. The CLI SHALL NOT run it and SHALL + surface it as advisory. +- `missing`: a rule the server expected that the CLI did not report. It is not actionable for + execution and SHALL be treated as advisory/audit only. + +The CLI SHALL match each `run` entry back to a local file by its `signature` (content-based +join), so a file that was moved but not changed still resolves. + +#### Scenario: Run entries are matched by signature + +- **WHEN** the CLI processes a `run` entry +- **THEN** it SHALL locate the corresponding local file by matching the `signature`, not the path + +#### Scenario: Unsafe is surfaced and not run + +- **WHEN** a reported file lands in `unsafe` +- **THEN** the CLI SHALL NOT execute it +- **AND** SHALL surface it as tamper/drift + +#### Scenario: Unknown is surfaced and not run + +- **WHEN** a reported file lands in `unknown` +- **THEN** the CLI SHALL NOT execute it +- **AND** SHALL surface it as an advisory notice + +#### Scenario: Missing is advisory only + +- **WHEN** the response includes `missing` entries +- **THEN** the CLI SHALL treat them as advisory/audit only and SHALL NOT fail execution on them + +### Requirement: The CLI executes only the server run set + +When reconciliation succeeds, the CLI SHALL execute exactly the files in the server's `run` +set and no other file. The CLI SHALL NOT run a file on the basis of its own local comparison +of signatures. This replaces any local classification of rules. + +#### Scenario: Only run-set files execute + +- **WHEN** reconciliation returns a `run` set that is a strict subset of the reported files +- **THEN** the CLI SHALL execute only the files in `run` +- **AND** SHALL NOT execute any reported file absent from `run` + +#### Scenario: No local self-classification + +- **WHEN** the CLI holds a local signature or sidecar for a file +- **THEN** it SHALL NOT treat that local value as authorization to run the file + +### Requirement: Reconcile is scoped to the token's organization + +The reconcile endpoint SHALL be authorized by the same bearer-token / `orgId`-claim scheme as +all `/cli/api/*` endpoints and SHALL be scoped to the organization the token owns. The CLI +SHALL handle the documented edges: a `401` with `{ error: "unauthorized" }` for a missing or +invalid token; an empty corpus (empty `run`/`missing`, every reported file in `unknown`, +nothing runs); and an empty report (empty `run`/`unsafe`/`unknown`, the full corpus in +`missing`). + +#### Scenario: Unauthorized token + +- **WHEN** the CLI calls reconcile without a valid bearer token +- **THEN** the server SHALL return `401` with `{ error: "unauthorized" }` +- **AND** the CLI SHALL NOT execute any rule from a `run` set + +#### Scenario: Empty corpus runs nothing + +- **WHEN** the repository has no blessed rules and the CLI reports files +- **THEN** every reported file SHALL be returned in `unknown` +- **AND** the CLI SHALL execute nothing + +### Requirement: Local signatures are advisory only + +Any signature the CLI persists into the repository (for example as sidecar metadata) SHALL be +treated as an offline-fallback convenience only and SHALL NOT be treated as an authorization +signal. The server-side record is authoritative and the server decides what runs. + +#### Scenario: Sidecar signature is not authorization + +- **WHEN** a rule file has a locally stored signature that matches its content +- **THEN** the CLI SHALL NOT run the file on that basis alone +- **AND** SHALL rely on the server's `run` set for authorization diff --git a/openspec/changes/server-owned-rule-reconciliation/tasks.md b/openspec/changes/server-owned-rule-reconciliation/tasks.md new file mode 100644 index 00000000..84c64d79 --- /dev/null +++ b/openspec/changes/server-owned-rule-reconciliation/tasks.md @@ -0,0 +1,48 @@ +## 1. Signature module + +- [x] 1.1 Create `packages/cli/src/rules/rule-hash.ts` exporting `ALGO_VERSION`, `normalize(text)`, `canonicalHash(input)` (returns the `1;h=sha-256;d=` envelope), and `parseSignature(sig)` (validates the envelope, reads algoVersion up to the first `;`). +- [x] 1.2 Implement `normalize()` exactly: UTF-8 decode, strip one leading `U+FEFF` BOM, convert CRLF and lone CR to LF, strip all trailing newlines then append one LF. No YAML parsing, no Unicode NFC/NFD. +- [x] 1.3 Hash with web-standard APIs only: `crypto.subtle.digest('SHA-256', …)` + `TextEncoder`; do NOT import `node:crypto`. Hex-encode lowercase. +- [x] 1.4 Add a helper to read a rule file as UTF-8 and produce its signature (used by reconcile and any sidecar write). + +## 2. Conformance vectors + +- [ ] 2.1 Add a fetch script (mirroring `scripts/fetch-ast-grep-schema.ts`) that pulls `GET /cli/api/rule-hash-vectors` and writes `packages/cli/test/fixtures/rule-hash-vectors.json`; wire an npm script (e.g. `generate:rule-hash-vectors`). +- [ ] 2.2 Commit the fetched vectors fixture. +- [ ] 2.3 Add `packages/cli/test/rule-hash.test.ts` that parses each vector's `input` as JSON (decoding `\uXXXX`) and asserts `canonicalHash(input) === signature` for every entry; failure blocks the build. +- [ ] 2.4 Add focused unit tests for each invariant: CRLF-equals-LF, trailing-newlines-collapse, BOM-stripped, content-change-differs, multibyte UTF-8, combining-mark-not-NFC-folded. + +## 3. Reconcile API client + +- [ ] 3.1 Create `packages/cli/src/api/reconcile.ts` with local `ReconcileRequest` (`{ repositoryUrl, files: { file, signature }[] }`) and `ReconcileResponse` (`run`/`unsafe`/`unknown`/`missing`) types. +- [ ] 3.2 Implement `reconcile(token, request)` issuing `POST /cli/api/reconcile` with `Authorization: Bearer ` against `getApiBaseUrl()`'s origin (reuse `createApiClient`/config where practical). +- [ ] 3.3 Map outcomes to typed results: success → parsed buckets; `401` `{ error: "unauthorized" }` → unauthorized signal; transport error / `404` / not-deployed → "reconcile unavailable" signal (never a thrown hard failure that aborts `check`). +- [ ] 3.4 Add `RECONCILE_FAILED` to the `CLIErrorCode` union in `packages/cli/src/types/errors.ts`. + +## 4. Run-set gating in check + +- [ ] 4.1 Add a rule-enumeration + signing step: read every `.yml` under `.taskless/rules/`, compute `{ file, signature }` for each. +- [ ] 4.2 In `packages/cli/src/commands/check.ts`, branch on auth state: no token (or `--anonymous`) → run all local rules with no network; token + resolvable `repositoryUrl` + not `--anonymous` → reconcile-then-gate to the `run` set. +- [ ] 4.3 Materialize an ephemeral, gitignored `.taskless/.run/rules/` containing only the `run`-set files (matched to local files by signature); ensure `.taskless/.gitignore` covers `.run/`. +- [ ] 4.4 Generate an sgconfig pointing `ruleDirs` at the ephemeral run dir (extend `src/filesystem/sgconfig.ts` to accept a rules dir / target) and scan that set via the existing `runAstGrepScan`. +- [ ] 4.5 When the `run` set is empty, skip `sg scan`, produce zero results, and exit 0 (still surface advisories). + +## 5. Mismatch warnings, degrade & exit codes + +- [ ] 5.1 On a successful authenticated reconcile, warn on `unsafe` (tamper/drift), `unknown` (not server-issued), and `missing` (audit-only) without changing the exit code. +- [ ] 5.2 Keep the unauthenticated/`--anonymous` path silent (run all local rules, no warning; optional informational line only). +- [ ] 5.3 Implement the authed degrade path: token present but reconcile can't complete (no git remote / endpoint unreachable / not-deployed / transport error) → warn "verification could not be performed" and scan all local rules, no non-zero exit. +- [ ] 5.4 Suppress all warnings/notices under `--json`; keep the existing `{ success, results }` shape (and the error envelope on scan failure). +- [ ] 5.5 Ensure the exit code stays governed solely by error-severity results from the executed rule set. + +## 6. Help & docs + +- [ ] 6.1 Update `packages/cli/src/help/check.txt` to explain the auth-state behavior (offline runs local; authed reconciles and warns on mismatches), the run-set gate, and `--anonymous`. +- [ ] 6.2 Update `packages/cli/src/help/ci.txt` to describe the CI backstop as the enforcement point over the reconciled run set. + +## 7. Tests & verification + +- [ ] 7.1 Add reconcile-gating integration tests (vitest, subprocess against `dist/`, temp-dir fixtures): only `run`-set rules scanned; non-run files excluded. +- [ ] 7.2 Add auth-state tests: logged-out and `--anonymous` scan all rules silently; authed endpoint-unavailable warns and degrades without failing the exit code; `--json` omits warnings. +- [ ] 7.3 Add mismatch-warning tests: `unsafe`/`unknown`/`missing` warn without changing the exit code; empty run set exits 0 without invoking the scanner. +- [ ] 7.4 Run `pnpm build` then `pnpm --filter @taskless/cli test`, plus `pnpm typecheck` and `pnpm lint`; fix any failures. diff --git a/packages/cli/src/rules/rule-hash.ts b/packages/cli/src/rules/rule-hash.ts new file mode 100644 index 00000000..d292bc35 --- /dev/null +++ b/packages/cli/src/rules/rule-hash.ts @@ -0,0 +1,117 @@ +import { readFile } from "node:fs/promises"; + +/** + * Canonical rule hashing — the CLI half of the server-owned reconciliation + * contract (TSKL-270). This MUST reproduce the server reference + * (`packages/shared/src/rule-hash.ts`) byte-for-byte; the conformance vectors + * at `GET /cli/api/rule-hash-vectors` exist to catch any divergence. + * + * The hash is built on web-standard APIs only (`crypto.subtle` + `TextEncoder`) + * so it matches in workerd and Node 20+ without a Node-specific crypto module. + */ + +/** Algorithm version carried as the leading token of every signature. */ +export const ALGO_VERSION = 1; + +/** Hash algorithm for algoVersion 1. */ +const ALGO = "sha-256"; + +/** A parsed signature envelope. */ +export interface ParsedSignature { + algoVersion: number; + algo: string; + digest: string; +} + +/** + * Normalize raw decoded rule text prior to hashing. Operates on text only — + * it never parses or re-serializes YAML, so it works for any rule type. + * + * In order: strip a single leading BOM, convert CRLF and lone CR to LF, strip + * all trailing newlines, then append exactly one LF. Unicode is intentionally + * NOT NFC/NFD-folded — a change here ships as a new algoVersion. + */ +export function normalize(fileText: string): string { + let text = fileText; + // Strip a single leading UTF-8 BOM (decoded as U+FEFF). + if (text.codePointAt(0) === 0xfeff) { + text = text.slice(1); + } + // Convert every CRLF and lone CR to LF. + text = text.replaceAll(/\r\n?/g, "\n"); + // Strip all trailing newlines, then append exactly one LF. + text = text.replaceAll(/\n+$/g, "") + "\n"; + return text; +} + +/** Lowercase-hex encode a digest buffer. */ +function toHex(buffer: ArrayBuffer): string { + return [...new Uint8Array(buffer)] + .map((byte) => byte.toString(16).padStart(2, "0")) + .join(""); +} + +/** + * Compute the canonical signature envelope `1;h=sha-256;d=` for a rule + * file's text. The digest is `SHA-256(normalize(text))`, hex lowercase. + */ +export async function canonicalHash(fileText: string): Promise { + const bytes = new TextEncoder().encode(normalize(fileText)); + const digest = await crypto.subtle.digest("SHA-256", bytes); + return `${String(ALGO_VERSION)};h=${ALGO};d=${toHex(digest)}`; +} + +/** + * Parse and validate a signature envelope. The algoVersion is read up to the + * first `;` — before any `key=value` parsing — so versioning never depends on + * the parameter syntax. Throws on a malformed or unsupported envelope. + */ +export function parseSignature(signature: string): ParsedSignature { + const firstDelimiter = signature.indexOf(";"); + if (firstDelimiter <= 0) { + throw new Error(`Malformed signature (no algoVersion): "${signature}"`); + } + + const versionToken = signature.slice(0, firstDelimiter); + const algoVersion = Number(versionToken); + if ( + !Number.isInteger(algoVersion) || + algoVersion <= 0 || + String(algoVersion) !== versionToken + ) { + throw new Error(`Malformed signature (bad algoVersion): "${signature}"`); + } + + const parameters = new Map(); + for (const part of signature.slice(firstDelimiter + 1).split(";")) { + const eq = part.indexOf("="); + if (eq === -1) { + throw new Error(`Malformed signature (bad parameter): "${signature}"`); + } + parameters.set(part.slice(0, eq), part.slice(eq + 1)); + } + + const algo = parameters.get("h"); + const digest = parameters.get("d"); + if (algo === undefined || digest === undefined) { + throw new Error(`Malformed signature (missing h/d): "${signature}"`); + } + + if ( + algoVersion === ALGO_VERSION && + (algo !== ALGO || !/^[0-9a-f]{64}$/.test(digest)) + ) { + throw new Error(`Invalid algoVersion-1 signature: "${signature}"`); + } + + return { algoVersion, algo, digest }; +} + +/** + * Read a rule file as UTF-8 and produce its canonical signature envelope. + * Used when reporting files for reconciliation (and for any sidecar write). + */ +export async function signRuleFile(filePath: string): Promise { + const text = await readFile(filePath, "utf8"); + return canonicalHash(text); +} From 840c729b011e1b51504a76f9db1c70e0b0313ba6 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 2 Jul 2026 20:22:07 -0700 Subject: [PATCH 02/21] feat(cli): add rule-hash conformance vectors with resilient fetch Commit the cross-repo vectors fixture and a prebuild-wired fetch that refreshes from GET /cli/api/rule-hash-vectors, falling back to the committed cache offline. Conformance test asserts our hasher reproduces every vector exactly (release-blocking on mismatch). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../design.md | 26 ++-- .../server-owned-rule-reconciliation/tasks.md | 8 +- packages/cli/package.json | 2 + .../cli/scripts/fetch-rule-hash-vectors.ts | 84 ++++++++++++ .../cli/test/fixtures/rule-hash.vectors.json | 67 ++++++++++ packages/cli/test/rule-hash.test.ts | 125 ++++++++++++++++++ 6 files changed, 297 insertions(+), 15 deletions(-) create mode 100644 packages/cli/scripts/fetch-rule-hash-vectors.ts create mode 100644 packages/cli/test/fixtures/rule-hash.vectors.json create mode 100644 packages/cli/test/rule-hash.test.ts diff --git a/openspec/changes/server-owned-rule-reconciliation/design.md b/openspec/changes/server-owned-rule-reconciliation/design.md index 3ebb51d5..8bd64a11 100644 --- a/openspec/changes/server-owned-rule-reconciliation/design.md +++ b/openspec/changes/server-owned-rule-reconciliation/design.md @@ -59,17 +59,21 @@ _Alternative rejected:_ `node:crypto` `createHash`. Simpler locally but diverges web-standard reference the server pins, and risks subtle cross-repo drift the vectors exist to prevent. -### Decision: Conformance test fetches vectors, commits a fixture, asserts exact reproduction - -Add a script/step to fetch `GET /cli/api/rule-hash-vectors` and commit the result as -`packages/cli/test/fixtures/rule-hash-vectors.json`, plus a vitest test -(`test/rule-hash.test.ts`) that parses each `input` as JSON (decoding `\uXXXX`) and asserts -`canonicalHash(input) === signature`. A mismatch fails the build. Committing the fixture (vs. -fetching at test time) keeps tests hermetic/offline; refresh is a manual step when the algo -version bumps, mirroring `generate:ast-grep-schema`. - -_Alternative rejected:_ fetch vectors live during the test run — introduces network flakiness -into CI and couples the unit suite to endpoint availability. +### Decision: Build-wired resilient fetch with a committed cache; test asserts exact reproduction + +Commit the vectors as `packages/cli/test/fixtures/rule-hash.vectors.json` in the cross-repo +source-of-truth format (a bare `[{ name, input, signature }]` array kept pure-ASCII with +`\uXXXX` escapes, so git stays byte-stable). `scripts/fetch-rule-hash-vectors.ts` (npm +`generate:rule-hash-vectors`) refreshes it from `GET /cli/api/rule-hash-vectors`, unwrapping +the endpoint's `{ vectors: [...] }` and re-escaping to ASCII. It is wired as `prebuild`, so +every build/CI run tries to refresh but falls back to the committed cache on any +network/HTTP/shape failure (only a missing cache is fatal). A vitest test +(`test/rule-hash.test.ts`) parses each `input` as JSON (decoding `\uXXXX`) and asserts +`canonicalHash(input) === signature` for every entry; a mismatch fails the build. CI gets the +freshest vectors when reachable while offline builds and the unit suite stay hermetic. + +_Alternatives rejected:_ (a) fetch live during the test run — couples the unit suite to +endpoint availability; (b) a purely manual refresh — drifts silently from the server. ### Decision: Reconcile client via a hand-typed request over the existing fetch layer diff --git a/openspec/changes/server-owned-rule-reconciliation/tasks.md b/openspec/changes/server-owned-rule-reconciliation/tasks.md index 84c64d79..a4bb68ba 100644 --- a/openspec/changes/server-owned-rule-reconciliation/tasks.md +++ b/openspec/changes/server-owned-rule-reconciliation/tasks.md @@ -7,10 +7,10 @@ ## 2. Conformance vectors -- [ ] 2.1 Add a fetch script (mirroring `scripts/fetch-ast-grep-schema.ts`) that pulls `GET /cli/api/rule-hash-vectors` and writes `packages/cli/test/fixtures/rule-hash-vectors.json`; wire an npm script (e.g. `generate:rule-hash-vectors`). -- [ ] 2.2 Commit the fetched vectors fixture. -- [ ] 2.3 Add `packages/cli/test/rule-hash.test.ts` that parses each vector's `input` as JSON (decoding `\uXXXX`) and asserts `canonicalHash(input) === signature` for every entry; failure blocks the build. -- [ ] 2.4 Add focused unit tests for each invariant: CRLF-equals-LF, trailing-newlines-collapse, BOM-stripped, content-change-differs, multibyte UTF-8, combining-mark-not-NFC-folded. +- [x] 2.1 Add a fetch script (mirroring `scripts/fetch-ast-grep-schema.ts`) that pulls `GET /cli/api/rule-hash-vectors`, unwraps `{ vectors }`, re-escapes to pure ASCII, and writes `packages/cli/test/fixtures/rule-hash.vectors.json`; wire the `generate:rule-hash-vectors` npm script AND a `prebuild` hook that always tries the network but falls back to the committed cache (only a missing cache is fatal). +- [x] 2.2 Commit the vectors fixture (cross-repo source-of-truth bare-array format). +- [x] 2.3 Add `packages/cli/test/rule-hash.test.ts` that parses each vector's `input` as JSON (decoding `\uXXXX`) and asserts `canonicalHash(input) === signature` for every entry; failure blocks the build. +- [x] 2.4 Add focused unit tests for each invariant: CRLF-equals-LF, lone-CR, trailing-newlines-collapse, empty→single-LF, BOM-stripped (leading/double/interior), content-change-differs, multibyte UTF-8, combining-mark-not-NFC-folded. ## 3. Reconcile API client diff --git a/packages/cli/package.json b/packages/cli/package.json index 478b18fc..9add16a3 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -8,11 +8,13 @@ "directory": "packages/cli" }, "scripts": { + "prebuild": "tsx scripts/fetch-rule-hash-vectors.ts", "build": "vite build", "build:dev": "TASKLESS_BUILD_TARGET=dev vite build", "build:self": "TASKLESS_BUILD_TARGET=self vite build", "generate:api": "openapi-typescript https://app.taskless.io/cli/api/__schema -o src/generated/api.d.ts", "generate:ast-grep-schema": "tsx scripts/fetch-ast-grep-schema.ts", + "generate:rule-hash-vectors": "tsx scripts/fetch-rule-hash-vectors.ts", "test": "vitest run", "typecheck": "tsc --noEmit" }, diff --git a/packages/cli/scripts/fetch-rule-hash-vectors.ts b/packages/cli/scripts/fetch-rule-hash-vectors.ts new file mode 100644 index 00000000..6994e58d --- /dev/null +++ b/packages/cli/scripts/fetch-rule-hash-vectors.ts @@ -0,0 +1,84 @@ +import { writeFileSync, existsSync } from "node:fs"; +import { resolve, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const OUTPUT_PATH = resolve( + __dirname, + "..", + "test", + "fixtures", + "rule-hash.vectors.json" +); + +/** + * Serialize as pure ASCII: every non-ASCII code unit becomes a `\uXXXX` escape + * so the committed fixture matches the cross-repo source-of-truth format and + * stays byte-stable in git regardless of the platform's console encoding. + * Iterating code units (not code points) makes a surrogate pair escape as two + * `\uXXXX`, exactly like the reference file. + */ +function toAsciiJson(value: unknown): string { + const json = JSON.stringify(value, null, 2); + let out = ""; + for (let index = 0; index < json.length; index++) { + // eslint-disable-next-line unicorn/prefer-code-point -- need per-UTF-16-unit escaping so surrogate pairs become two \uXXXX + const code = json.charCodeAt(index); + out += + code > 0x7f + ? String.raw`\u` + code.toString(16).padStart(4, "0") + : json[index]; + } + return out + "\n"; +} + +// Resolve the API origin the same way the runtime client does: honor +// TASKLESS_API_URL, else the production default. The vectors endpoint is +// unauthenticated and lives under /cli/api/. +const baseUrl = ( + process.env.TASKLESS_API_URL ?? "https://app.taskless.io/cli" +).replace(/\/cli\/?$/, ""); +const sourceUrl = `${baseUrl}/cli/api/rule-hash-vectors`; + +console.log(`Fetching rule-hash conformance vectors...`); +console.log(` URL: ${sourceUrl}`); + +/** + * Refresh is best-effort: always try the network, but fall back to the + * committed cache when the endpoint is unreachable or unhealthy so offline and + * CI-without-network builds still succeed. Only a missing cache is fatal. + */ +function fallBackToCache(reason: string): never | void { + if (existsSync(OUTPUT_PATH)) { + console.warn(` ${reason}`); + console.warn(` Falling back to committed vectors cache.`); + return; + } + throw new Error(`${reason} and no committed vectors cache exists.`); +} + +let response: Response | undefined; +try { + response = await fetch(sourceUrl); +} catch (error) { + const message = error instanceof Error ? error.message : String(error); + fallBackToCache(`Could not reach the vectors endpoint (${message})`); +} + +if (response?.ok) { + // The endpoint wraps the source-of-truth array as { vectors: [...] }; the + // committed cache stores the bare array to match the server reference file. + const body = (await response.json()) as { + vectors?: { name: string; input: string; signature: string }[]; + }; + if (Array.isArray(body.vectors) && body.vectors.length > 0) { + writeFileSync(OUTPUT_PATH, toAsciiJson(body.vectors), "utf8"); + console.log( + ` Wrote ${String(body.vectors.length)} vectors to: ${OUTPUT_PATH}` + ); + } else { + fallBackToCache(`Response did not contain a non-empty "vectors" array`); + } +} else if (response) { + fallBackToCache(`HTTP ${String(response.status)} fetching ${sourceUrl}`); +} diff --git a/packages/cli/test/fixtures/rule-hash.vectors.json b/packages/cli/test/fixtures/rule-hash.vectors.json new file mode 100644 index 00000000..16d69dd5 --- /dev/null +++ b/packages/cli/test/fixtures/rule-hash.vectors.json @@ -0,0 +1,67 @@ +[ + { + "name": "lf-baseline", + "input": "rule:\n id: foo\n message: no foo\n", + "signature": "1;h=sha-256;d=7d0f9d1822594679d83800361278d6e3ecb3182669cf1aa2d54469ef6df24dbe" + }, + { + "name": "crlf-equals-lf", + "input": "rule:\r\n id: foo\r\n message: no foo\r\n", + "signature": "1;h=sha-256;d=7d0f9d1822594679d83800361278d6e3ecb3182669cf1aa2d54469ef6df24dbe" + }, + { + "name": "no-trailing-newline-equals-single", + "input": "rule:\n id: foo\n message: no foo", + "signature": "1;h=sha-256;d=7d0f9d1822594679d83800361278d6e3ecb3182669cf1aa2d54469ef6df24dbe" + }, + { + "name": "multiple-trailing-newlines-equal-single", + "input": "rule:\n id: foo\n message: no foo\n\n\n", + "signature": "1;h=sha-256;d=7d0f9d1822594679d83800361278d6e3ecb3182669cf1aa2d54469ef6df24dbe" + }, + { + "name": "leading-bom-stripped", + "input": "\ufeffrule:\n id: foo\n message: no foo\n", + "signature": "1;h=sha-256;d=7d0f9d1822594679d83800361278d6e3ecb3182669cf1aa2d54469ef6df24dbe" + }, + { + "name": "meaningful-content-change-differs", + "input": "rule:\n id: bar\n message: no foo\n", + "signature": "1;h=sha-256;d=417c6b37ea6f4f7082836ec875d161cf874bfc19587a6b0ac029a28e56f55bc1" + }, + { + "name": "multibyte-utf8", + "input": "rule:\n message: caf\u00e9 \u2615 \u65e5\u672c\u8a9e\n", + "signature": "1;h=sha-256;d=91cccc7d330ab09b77bd4daa31acebb080256e34638516be5b570ba600bf6284" + }, + { + "name": "combining-mark-not-nfc-folded", + "input": "rule:\n id: cafe\u0301\n", + "signature": "1;h=sha-256;d=75d78b40aed8ddc5d3e6338644e043d7925907ede842d192ed777507838c29d8" + }, + { + "name": "lone-cr-equals-lf", + "input": "rule:\r id: foo\r message: no foo\r", + "signature": "1;h=sha-256;d=7d0f9d1822594679d83800361278d6e3ecb3182669cf1aa2d54469ef6df24dbe" + }, + { + "name": "empty-input-becomes-single-lf", + "input": "", + "signature": "1;h=sha-256;d=01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b" + }, + { + "name": "interior-bom-preserved", + "input": "rule:\n id: a\ufeffb\n", + "signature": "1;h=sha-256;d=85c354beeecb2124ea714d37fad08fc25fa0b3c1a3c225de236f38cc264fb9fb" + }, + { + "name": "double-leading-bom-strips-one", + "input": "\ufeff\ufeffrule:\n id: foo\n message: no foo\n", + "signature": "1;h=sha-256;d=644a709d438d1c267bcc615d1ca83977fcaf61fe774443ba9f6f44e371c8af62" + }, + { + "name": "astral-utf8-surrogate-pair", + "input": "rule:\n message: party \ud83c\udf89\n", + "signature": "1;h=sha-256;d=13ab09346df90129fc2d562a6bcbf1fad13cdb27bda09091d5355f9876881518" + } +] diff --git a/packages/cli/test/rule-hash.test.ts b/packages/cli/test/rule-hash.test.ts new file mode 100644 index 00000000..5232de55 --- /dev/null +++ b/packages/cli/test/rule-hash.test.ts @@ -0,0 +1,125 @@ +import { describe, expect, it } from "vitest"; + +import { + canonicalHash, + normalize, + parseSignature, +} from "../src/rules/rule-hash"; +import vectorsFixture from "./fixtures/rule-hash.vectors.json"; + +interface Vector { + name: string; + input: string; + signature: string; +} + +// The committed fixture is the cross-repo source-of-truth format: a bare array +// of { name, input, signature }, refreshed via `pnpm generate:rule-hash-vectors`. +const vectors = vectorsFixture as Vector[]; + +// Non-ASCII is built from code points so this source file stays pure ASCII. +const BOM = String.fromCodePoint(0xfeff); + +describe("rule-hash conformance vectors", () => { + it("has a non-empty committed vectors fixture", () => { + expect(vectors.length).toBeGreaterThan(0); + }); + + // A mismatch here is a cross-repo release blocker: our normalize()+hash must + // reproduce the server reference for every vector, exactly. + it.each(vectors.map((v) => [v.name, v] as const))( + "reproduces vector %s", + async (_name, vector) => { + // JSON.parse already decoded any \uXXXX escapes into real code points. + expect(await canonicalHash(vector.input)).toBe(vector.signature); + } + ); +}); + +describe("normalize invariants", () => { + it("treats CRLF the same as LF", async () => { + expect(await canonicalHash("a\r\nb")).toBe(await canonicalHash("a\nb")); + }); + + it("treats a lone CR the same as LF", async () => { + expect(await canonicalHash("a\rb")).toBe(await canonicalHash("a\nb")); + }); + + it("collapses any number of trailing newlines to one", async () => { + const base = await canonicalHash("a\nb"); + expect(await canonicalHash("a\nb\n")).toBe(base); + expect(await canonicalHash("a\nb\n\n\n")).toBe(base); + expect(await canonicalHash("a\nb")).toBe(base); + }); + + it("normalizes empty input to a single LF", () => { + expect(normalize("")).toBe("\n"); + }); + + it("strips a single leading BOM", async () => { + expect(await canonicalHash(BOM + "abc")).toBe(await canonicalHash("abc")); + }); + + it("strips only one leading BOM and preserves the rest", async () => { + // Two leading BOMs collapse to one remaining BOM, distinct from none. + expect(await canonicalHash(BOM + BOM + "abc")).not.toBe( + await canonicalHash("abc") + ); + }); + + it("preserves an interior BOM", async () => { + expect(await canonicalHash("a" + BOM + "b")).not.toBe( + await canonicalHash("ab") + ); + }); + + it("differs on a meaningful content change", async () => { + expect(await canonicalHash("abc")).not.toBe(await canonicalHash("abd")); + }); + + it("hashes multibyte UTF-8 stably", async () => { + // e+acute, CJK, and an astral emoji (party popper, U+1F389). + const text = + "caf" + + String.fromCodePoint(0xe9) + + " " + + String.fromCodePoint(0x65e5, 0x672c, 0x8a9e) + + " " + + String.fromCodePoint(0x1f389) + + "\n"; + const sig = await canonicalHash(text); + expect(sig).toMatch(/^1;h=sha-256;d=[0-9a-f]{64}$/); + }); + + it("does not NFC-fold combining marks", async () => { + const precomposed = "caf" + String.fromCodePoint(0xe9); // e-acute U+00E9 + const decomposed = "cafe" + String.fromCodePoint(0x301); // e + U+0301 + expect(precomposed).not.toBe(decomposed); + expect(await canonicalHash(precomposed)).not.toBe( + await canonicalHash(decomposed) + ); + }); +}); + +describe("parseSignature", () => { + it("round-trips a computed signature", async () => { + const sig = await canonicalHash("abc"); + const parsed = parseSignature(sig); + expect(parsed).toEqual({ + algoVersion: 1, + algo: "sha-256", + digest: sig.slice(sig.indexOf("d=") + 2), + }); + }); + + it("reads the algoVersion before parameters", () => { + // Unknown future version: parsed, not rejected, and params still read. + expect(parseSignature("2;h=sha-512;d=deadbeef").algoVersion).toBe(2); + }); + + it("rejects a malformed envelope", () => { + expect(() => parseSignature("nope")).toThrow(); + expect(() => parseSignature("1;hsha256")).toThrow(); + expect(() => parseSignature("1;h=sha-256;d=xyz")).toThrow(); + }); +}); From e9b49111128ef6189e0c60681ace3861a515c134 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 2 Jul 2026 20:28:17 -0700 Subject: [PATCH 03/21] feat(cli): add reconcile API client POST /cli/api/reconcile client returning a discriminated ok/unauthorized/unavailable outcome that never throws on expected network/auth/not-deployed conditions, so check can degrade to a local scan. Adds the RECONCILE_FAILED error code. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../server-owned-rule-reconciliation/tasks.md | 8 +- packages/cli/src/api/reconcile.ts | 128 ++++++++++++++++++ packages/cli/src/types/errors.ts | 1 + 3 files changed, 133 insertions(+), 4 deletions(-) create mode 100644 packages/cli/src/api/reconcile.ts diff --git a/openspec/changes/server-owned-rule-reconciliation/tasks.md b/openspec/changes/server-owned-rule-reconciliation/tasks.md index a4bb68ba..91506de0 100644 --- a/openspec/changes/server-owned-rule-reconciliation/tasks.md +++ b/openspec/changes/server-owned-rule-reconciliation/tasks.md @@ -14,10 +14,10 @@ ## 3. Reconcile API client -- [ ] 3.1 Create `packages/cli/src/api/reconcile.ts` with local `ReconcileRequest` (`{ repositoryUrl, files: { file, signature }[] }`) and `ReconcileResponse` (`run`/`unsafe`/`unknown`/`missing`) types. -- [ ] 3.2 Implement `reconcile(token, request)` issuing `POST /cli/api/reconcile` with `Authorization: Bearer ` against `getApiBaseUrl()`'s origin (reuse `createApiClient`/config where practical). -- [ ] 3.3 Map outcomes to typed results: success → parsed buckets; `401` `{ error: "unauthorized" }` → unauthorized signal; transport error / `404` / not-deployed → "reconcile unavailable" signal (never a thrown hard failure that aborts `check`). -- [ ] 3.4 Add `RECONCILE_FAILED` to the `CLIErrorCode` union in `packages/cli/src/types/errors.ts`. +- [x] 3.1 Create `packages/cli/src/api/reconcile.ts` with local `ReconcileRequest` (`{ repositoryUrl, files: { file, signature }[] }`) and `ReconcileResponse` (`run`/`unsafe`/`unknown`/`missing`) types. +- [x] 3.2 Implement `reconcile(token, request)` issuing `POST /cli/api/reconcile` with `Authorization: Bearer ` against `getApiBaseUrl()`'s origin (plain fetch, since the path is not in the generated schema). +- [x] 3.3 Map outcomes to a discriminated `ReconcileOutcome`: success → parsed buckets (`ok`); `401` → `unauthorized`; transport error / `404` / not-deployed / any other non-2xx → `unavailable` (never a thrown hard failure that aborts `check`). Verified against the live origin: reconcile returns 405 (not yet deployed) → `unavailable`. +- [x] 3.4 Add `RECONCILE_FAILED` to the `CLIErrorCode` union in `packages/cli/src/types/errors.ts`. ## 4. Run-set gating in check diff --git a/packages/cli/src/api/reconcile.ts b/packages/cli/src/api/reconcile.ts new file mode 100644 index 00000000..b3ff6124 --- /dev/null +++ b/packages/cli/src/api/reconcile.ts @@ -0,0 +1,128 @@ +import { getApiBaseUrl } from "./config"; + +/** + * Server-owned rule reconciliation (TSKL-270). The CLI reports the rule files + * it holds and the server returns the exact subset that may run. This endpoint + * is not in the generated schema (it may not be deployed everywhere yet), so it + * is called with a hand-typed request over plain `fetch`; migrate it onto the + * typed client once it lands in `GET /cli/api/__schema`. + */ + +/** A rule file reported for reconciliation. */ +export interface ReportedFile { + file: string; + signature: string; +} + +export interface ReconcileRequest { + repositoryUrl: string; + files: ReportedFile[]; +} + +/** A file the server blessed — execute exactly these. */ +export interface RunEntry { + ruleId: string; + file: string; + signature: string; +} + +/** A held rule whose content differs from what the server blessed. */ +export interface UnsafeEntry { + file: string; + expected: string; + got: string; +} + +/** A reported file the server never issued. */ +export interface UnknownEntry { + file: string; +} + +/** A rule the server expected that the client did not report. */ +export interface MissingEntry { + ruleId: string; + file: string; +} + +export interface ReconcileResponse { + run: RunEntry[]; + unsafe: UnsafeEntry[]; + unknown: UnknownEntry[]; + missing: MissingEntry[]; +} + +/** + * The result of an attempted reconciliation. `check` branches on this: + * - `ok`: use `result.run` as the complete allow-list; warn on the rest. + * - `unauthorized`: the token was missing/invalid (server returned 401). + * - `unavailable`: the endpoint could not be reached or is not deployed — + * degrade to a local scan. Never a thrown error for these expected cases. + */ +export type ReconcileOutcome = + | { status: "ok"; result: ReconcileResponse } + | { status: "unauthorized" } + | { status: "unavailable"; reason: string }; + +/** Coerce an untyped bucket into a typed array, tolerating a missing field. */ +function asArray(value: unknown): T[] { + return Array.isArray(value) ? (value as T[]) : []; +} + +/** + * Reconcile the reported files against the server. Returns a `ReconcileOutcome` + * and never throws for expected network/auth/deployment conditions. + */ +export async function reconcile( + token: string, + request: ReconcileRequest +): Promise { + // Schema paths include the /cli/ prefix, so the base URL is the origin. + const baseUrl = getApiBaseUrl().replace(/\/cli\/?$/, ""); + const url = `${baseUrl}/cli/api/reconcile`; + + let response: Response; + try { + response = await fetch(url, { + method: "POST", + headers: { + Authorization: `Bearer ${token}`, + "Content-Type": "application/json", + }, + body: JSON.stringify(request), + }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + return { status: "unavailable", reason: `network error: ${message}` }; + } + + if (response.status === 401) { + return { status: "unauthorized" }; + } + + // 404 (endpoint not deployed), 5xx, and any other non-2xx are treated as + // "unavailable" so `check` degrades to a local scan rather than failing. + if (!response.ok) { + return { + status: "unavailable", + reason: `HTTP ${String(response.status)}`, + }; + } + + let body: unknown; + try { + body = await response.json(); + } catch { + return { status: "unavailable", reason: "invalid response body" }; + } + + const data = body as Partial; + return { + status: "ok", + result: { + run: asArray(data.run), + unsafe: asArray(data.unsafe), + unknown: asArray(data.unknown), + missing: asArray(data.missing), + }, + }; +} diff --git a/packages/cli/src/types/errors.ts b/packages/cli/src/types/errors.ts index ab5504f9..a459b78f 100644 --- a/packages/cli/src/types/errors.ts +++ b/packages/cli/src/types/errors.ts @@ -14,6 +14,7 @@ export type CLIErrorCode = | "INVALID_INPUT" | "NETWORK_ERROR" | "SCAN_FAILED" + | "RECONCILE_FAILED" | "INTERNAL_ERROR"; /** From d1571789a2fb86c1ec04293ad2d0b6312e127a69 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 2 Jul 2026 20:59:59 -0700 Subject: [PATCH 04/21] feat(cli): gate check on server reconciliation by auth state check now picks its behavior from auth state: unauthenticated/--anonymous scans all local rules silently; authenticated reconciles and runs only the blessed run set, warning on unsafe/unknown/missing; a failed reconcile degrades to a local scan with a notice. Warnings are suppressed under --json and never affect the exit code. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../server-owned-rule-reconciliation/tasks.md | 20 +-- packages/cli/src/commands/check.ts | 123 +++++++++++++++++- packages/cli/src/filesystem/sgconfig.ts | 25 +++- packages/cli/src/rules/run-set.ts | 72 ++++++++++ 4 files changed, 219 insertions(+), 21 deletions(-) create mode 100644 packages/cli/src/rules/run-set.ts diff --git a/openspec/changes/server-owned-rule-reconciliation/tasks.md b/openspec/changes/server-owned-rule-reconciliation/tasks.md index 91506de0..ca08bbfd 100644 --- a/openspec/changes/server-owned-rule-reconciliation/tasks.md +++ b/openspec/changes/server-owned-rule-reconciliation/tasks.md @@ -21,19 +21,19 @@ ## 4. Run-set gating in check -- [ ] 4.1 Add a rule-enumeration + signing step: read every `.yml` under `.taskless/rules/`, compute `{ file, signature }` for each. -- [ ] 4.2 In `packages/cli/src/commands/check.ts`, branch on auth state: no token (or `--anonymous`) → run all local rules with no network; token + resolvable `repositoryUrl` + not `--anonymous` → reconcile-then-gate to the `run` set. -- [ ] 4.3 Materialize an ephemeral, gitignored `.taskless/.run/rules/` containing only the `run`-set files (matched to local files by signature); ensure `.taskless/.gitignore` covers `.run/`. -- [ ] 4.4 Generate an sgconfig pointing `ruleDirs` at the ephemeral run dir (extend `src/filesystem/sgconfig.ts` to accept a rules dir / target) and scan that set via the existing `runAstGrepScan`. -- [ ] 4.5 When the `run` set is empty, skip `sg scan`, produce zero results, and exit 0 (still surface advisories). +- [x] 4.1 Add a rule-enumeration + signing step (`src/rules/run-set.ts` → `signRuleFiles`): read every `.yml` under `.taskless/rules/`, compute `{ file, path, signature }` for each. +- [x] 4.2 In `packages/cli/src/commands/check.ts` (`resolveScanMode`), branch on auth state: no token (or `--anonymous`) → local; token + resolvable `repositoryUrl` + not `--anonymous` → reconcile, gating to the `run` set on `ok`. +- [x] 4.3 Materialize an ephemeral, gitignored `.taskless/.run/rules/` (`materializeRunDirectory` + `selectRunFiles`) containing only the `run`-set files matched by signature; `.taskless/.gitignore` gets `.run/`. +- [x] 4.4 Extend `src/filesystem/sgconfig.ts` to accept a target rules directory and point `ruleDirs` at the ephemeral run dir; scan via the existing `runAstGrepScan`. +- [x] 4.5 When the `run` set is empty, skip `sg scan`, produce zero results, and exit 0 (still surface advisories). ## 5. Mismatch warnings, degrade & exit codes -- [ ] 5.1 On a successful authenticated reconcile, warn on `unsafe` (tamper/drift), `unknown` (not server-issued), and `missing` (audit-only) without changing the exit code. -- [ ] 5.2 Keep the unauthenticated/`--anonymous` path silent (run all local rules, no warning; optional informational line only). -- [ ] 5.3 Implement the authed degrade path: token present but reconcile can't complete (no git remote / endpoint unreachable / not-deployed / transport error) → warn "verification could not be performed" and scan all local rules, no non-zero exit. -- [ ] 5.4 Suppress all warnings/notices under `--json`; keep the existing `{ success, results }` shape (and the error envelope on scan failure). -- [ ] 5.5 Ensure the exit code stays governed solely by error-severity results from the executed rule set. +- [x] 5.1 On a successful authenticated reconcile, `surfaceReconcileWarnings` warns on `unsafe` (tamper/drift), `unknown` (not server-issued), and `missing` (audit-only) without changing the exit code. +- [x] 5.2 The unauthenticated/`--anonymous` path is silent (runs all local rules, no warning). +- [x] 5.3 The authed degrade path warns and scans all local rules with no non-zero exit — covers no git remote, `unauthorized` (401, with a re-auth hint), and `unavailable` (unreachable / not-deployed / transport). +- [x] 5.4 All warnings/notices are suppressed under `--json` (warnings go to stderr and are gated on `!args.json`); the existing `{ success, results }` / error-envelope shapes are unchanged. +- [x] 5.5 The exit code stays governed solely by error-severity results from the executed rule set. ## 6. Help & docs diff --git a/packages/cli/src/commands/check.ts b/packages/cli/src/commands/check.ts index f1e82b49..86ba9ea3 100644 --- a/packages/cli/src/commands/check.ts +++ b/packages/cli/src/commands/check.ts @@ -3,11 +3,23 @@ import { readdir, stat } from "node:fs/promises"; import { defineCommand } from "citty"; import { runAstGrepScan } from "../rules/scan"; +import type { CheckResult } from "../types/check"; import { formatText } from "../util/format"; import { generateSgConfig } from "../filesystem/sgconfig"; import { getTelemetry } from "../telemetry"; import { outputSchema as checkOutputSchema } from "../schemas/check"; import { makeErrorEnvelope } from "../types/errors"; +import { getToken } from "../auth/token"; +import { resolveRepositoryUrl } from "../util/git-remote"; +import { getCliPrefix } from "../util/package-manager"; +import { reconcile, type ReconcileResponse } from "../api/reconcile"; +import { + RUN_RULES_DIR, + materializeRunDirectory, + selectRunFiles, + signRuleFiles, + type SignedRuleFile, +} from "../rules/run-set"; async function pathExists(absolutePath: string): Promise { try { @@ -83,6 +95,81 @@ function extractPositionalPaths(rawArguments: string[]): string[] { return paths; } +/** + * How `check` will scan, decided from auth state: + * - `local`: unauthenticated or `--anonymous` — scan all local rules silently. + * - `degrade`: authenticated but reconciliation could not complete — scan all + * local rules and warn that verification could not be performed. + * - `gated`: reconciliation succeeded — scan only the blessed `run` set. + */ +type ScanMode = + | { kind: "local" } + | { kind: "degrade"; reason: string } + | { kind: "gated"; result: ReconcileResponse; signed: SignedRuleFile[] }; + +async function resolveScanMode( + cwd: string, + anonymous: boolean, + ruleFiles: string[] +): Promise { + if (anonymous) return { kind: "local" }; + + const token = await getToken(cwd, { silent: true }); + if (!token) return { kind: "local" }; + + let repositoryUrl: string; + try { + repositoryUrl = await resolveRepositoryUrl(cwd); + } catch { + return { + kind: "degrade", + reason: "no GitHub remote was found, so rules could not be verified", + }; + } + + const signed = await signRuleFiles(cwd, ruleFiles); + const outcome = await reconcile(token, { + repositoryUrl, + files: signed.map(({ file, signature }) => ({ file, signature })), + }); + + if (outcome.status === "ok") { + return { kind: "gated", result: outcome.result, signed }; + } + if (outcome.status === "unauthorized") { + return { + kind: "degrade", + reason: `authentication was rejected — run \`${getCliPrefix()} auth login\` to re-authenticate`, + }; + } + return { + kind: "degrade", + reason: `the rule service was unavailable (${outcome.reason})`, + }; +} + +/** Emit advisory reconciliation warnings (human output only). */ +function surfaceReconcileWarnings( + result: ReconcileResponse, + warn: (message: string) => void +): void { + for (const entry of result.unsafe) { + warn( + `Warning: ${entry.file} differs from the blessed rule (tamper/drift) and will not run.` + ); + } + for (const entry of result.unknown) { + warn( + `Notice: ${entry.file} was not issued by the server and will not run.` + ); + } + for (const entry of result.missing) { + warn( + `Notice: expected rule ${entry.ruleId} (${entry.file}) is not present locally.` + ); + } +} + export const checkCommand = defineCommand({ meta: { name: "check", @@ -101,7 +188,7 @@ export const checkCommand = defineCommand({ }, anonymous: { type: "boolean", - description: "Accepted for compatibility; check has no auth dependency", + description: "Skip server reconciliation and scan local rules unverified", default: false, }, }, @@ -109,6 +196,12 @@ export const checkCommand = defineCommand({ const cwd = resolve(args.dir ?? process.cwd()); const telemetry = await getTelemetry(cwd); + // Warnings are advisory human output; suppress them under --json so the + // machine output stays the existing { success, results } shape. + const warn = (message: string) => { + if (!args.json) console.error(message); + }; + // Set when a scan actually runs; drives cli_check_completed with counts // only (never matched code). let scanCounts: @@ -159,10 +252,32 @@ export const checkCommand = defineCommand({ return; } - // Generate ephemeral sgconfig.yml and run scanner + // Decide what to run from auth state, then scan. try { - await generateSgConfig(cwd); - const { results } = await runAstGrepScan(cwd, existingPaths); + const mode = await resolveScanMode(cwd, args.anonymous, ruleFiles); + + let results: CheckResult[] = []; + if (mode.kind === "gated") { + surfaceReconcileWarnings(mode.result, warn); + const runFiles = selectRunFiles(mode.signed, mode.result.run); + // Empty run set (empty corpus, or all unsafe/unknown): run nothing. + if (runFiles.length > 0) { + await materializeRunDirectory(cwd, runFiles); + await generateSgConfig(cwd, { rulesDirectory: RUN_RULES_DIR }); + const scan = await runAstGrepScan(cwd, existingPaths); + results = scan.results; + } + } else { + if (mode.kind === "degrade") { + warn( + `Notice: ${mode.reason}. Scanning all local rules unverified; the CI backstop enforces the server-owned rule set.` + ); + } + await generateSgConfig(cwd); + const scan = await runAstGrepScan(cwd, existingPaths); + results = scan.results; + } + let errorCount = 0; let warningCount = 0; for (const result of results) { diff --git a/packages/cli/src/filesystem/sgconfig.ts b/packages/cli/src/filesystem/sgconfig.ts index 423fa263..9849a319 100644 --- a/packages/cli/src/filesystem/sgconfig.ts +++ b/packages/cli/src/filesystem/sgconfig.ts @@ -3,21 +3,32 @@ import { join } from "node:path"; import { ensureTasklessDirectory } from "./directory"; -const SGCONFIG_CONTENT = `ruleDirs: - - rules -testConfigs: - - testDir: rule-tests -`; +/** Build sgconfig contents pointing `ruleDirs` at the given directory. */ +function sgConfigContent(rulesDirectory: string): string { + return `ruleDirs:\n - ${rulesDirectory}\ntestConfigs:\n - testDir: rule-tests\n`; +} + +export interface SgConfigOptions { + /** + * Directory (relative to `.taskless/`) that ast-grep should load rules from. + * Defaults to `rules`. Reconciliation points this at the ephemeral run + * directory so only the server-blessed run set is evaluated. + */ + rulesDirectory?: string; +} /** * Generate an ephemeral `sgconfig.yml` in `.taskless/` for ast-grep. * Runs migrations and ensures the directory structure is up-to-date. */ -export async function generateSgConfig(cwd: string): Promise { +export async function generateSgConfig( + cwd: string, + options: SgConfigOptions = {} +): Promise { await ensureTasklessDirectory(cwd); await writeFile( join(cwd, ".taskless", "sgconfig.yml"), - SGCONFIG_CONTENT, + sgConfigContent(options.rulesDirectory ?? "rules"), "utf8" ); } diff --git a/packages/cli/src/rules/run-set.ts b/packages/cli/src/rules/run-set.ts new file mode 100644 index 00000000..44861a09 --- /dev/null +++ b/packages/cli/src/rules/run-set.ts @@ -0,0 +1,72 @@ +import { copyFile, mkdir, rm } from "node:fs/promises"; +import { join } from "node:path"; + +import { addToGitignore } from "../filesystem/gitignore"; +import type { RunEntry } from "../api/reconcile"; +import { signRuleFile } from "./rule-hash"; + +/** A local rule file paired with its canonical signature. */ +export interface SignedRuleFile { + /** Delivered file name as it exists under `.taskless/rules/`. */ + file: string; + /** Absolute path to the file on disk. */ + path: string; + /** Canonical signature envelope for the file's bytes. */ + signature: string; +} + +/** Directory (relative to `.taskless/`) holding the materialized run set. */ +export const RUN_RULES_DIR = ".run/rules"; + +/** Sign each rule file, returning its delivered name, path, and signature. */ +export async function signRuleFiles( + cwd: string, + fileNames: string[] +): Promise { + const rulesDirectory = join(cwd, ".taskless", "rules"); + return Promise.all( + fileNames.map(async (file) => { + const path = join(rulesDirectory, file); + return { file, path, signature: await signRuleFile(path) }; + }) + ); +} + +/** + * Resolve the server's `run` entries back to local files by signature + * (content-based join, so a moved-but-unchanged file still resolves). Entries + * whose signature is not held locally are dropped. + */ +export function selectRunFiles( + signed: SignedRuleFile[], + run: RunEntry[] +): SignedRuleFile[] { + const bySignature = new Map(signed.map((s) => [s.signature, s])); + const selected: SignedRuleFile[] = []; + const seen = new Set(); + for (const entry of run) { + const match = bySignature.get(entry.signature); + if (match && !seen.has(match.path)) { + seen.add(match.path); + selected.push(match); + } + } + return selected; +} + +/** + * Materialize the blessed run set into a fresh, gitignored + * `.taskless/.run/rules/` so ast-grep evaluates only those files. The directory + * is rebuilt on every call to avoid stale rules leaking into a scan. + */ +export async function materializeRunDirectory( + cwd: string, + files: SignedRuleFile[] +): Promise { + const runRoot = join(cwd, ".taskless", ".run"); + const runRules = join(runRoot, "rules"); + await rm(runRoot, { recursive: true, force: true }); + await mkdir(runRules, { recursive: true }); + await Promise.all(files.map((f) => copyFile(f.path, join(runRules, f.file)))); + await addToGitignore(cwd, [".run/"]); +} From ebf410ee15e414f06acb4e2cc423ecf1d372c609 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 2 Jul 2026 21:03:10 -0700 Subject: [PATCH 05/21] docs(cli): document reconciliation auth-state behavior in help check.txt gains a "What runs (auth state)" section; ci.txt step 7 documents the optional TASKLESS_TOKEN backstop as the enforcement point over the server-blessed run set. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../server-owned-rule-reconciliation/tasks.md | 4 +-- packages/cli/src/help/check.txt | 29 ++++++++++++++++--- packages/cli/src/help/ci.txt | 26 +++++++++++++++-- 3 files changed, 50 insertions(+), 9 deletions(-) diff --git a/openspec/changes/server-owned-rule-reconciliation/tasks.md b/openspec/changes/server-owned-rule-reconciliation/tasks.md index ca08bbfd..101e26c1 100644 --- a/openspec/changes/server-owned-rule-reconciliation/tasks.md +++ b/openspec/changes/server-owned-rule-reconciliation/tasks.md @@ -37,8 +37,8 @@ ## 6. Help & docs -- [ ] 6.1 Update `packages/cli/src/help/check.txt` to explain the auth-state behavior (offline runs local; authed reconciles and warns on mismatches), the run-set gate, and `--anonymous`. -- [ ] 6.2 Update `packages/cli/src/help/ci.txt` to describe the CI backstop as the enforcement point over the reconciled run set. +- [x] 6.1 Update `packages/cli/src/help/check.txt` with a "What runs (auth state)" section (offline/`--anonymous` runs local; authed reconciles and warns on `unsafe`/`unknown`/`missing`; warnings are stderr-only and never affect exit code or `--json`). +- [x] 6.2 Update `packages/cli/src/help/ci.txt` step 7 to describe the optional `TASKLESS_TOKEN` backstop as the enforcement point over the reconciled run set. ## 7. Tests & verification diff --git a/packages/cli/src/help/check.txt b/packages/cli/src/help/check.txt index 76111b62..937dadf7 100644 --- a/packages/cli/src/help/check.txt +++ b/packages/cli/src/help/check.txt @@ -1,16 +1,37 @@ # Topic: check (CLI v%(CLI_VERSION)s / topic v1) ## Goal -Run all rules in `.taskless/rules/` against the codebase and report -matches. Used standalone (full project scan), in CI (diff-only scan), -or after rule create/improve to validate. +Run the applicable rules in `.taskless/rules/` against the codebase and +report matches. Used standalone (full project scan), in CI (diff-only +scan), or after rule create/improve to validate. ## Preconditions - `.taskless/` directory exists. - At least one rule exists in `.taskless/rules/`. (If none exist, the CLI exits 0 with a friendly message suggesting `taskless rule create`.) -- No auth required. +- No auth required. What actually runs depends on auth state — see + "What runs (auth state)". + +## What runs (auth state) + +`check` never requires auth, but auth state decides which rules execute: + +- **Logged out** (or `--anonymous`) — scans every rule in + `.taskless/rules/` locally, with no network call. The offline linter + posture; no warnings. +- **Logged in** — reconciles the local rule files against the Taskless + service and runs ONLY the server-blessed `run` set. Files that drift + from the blessed copy (`unsafe`), were never issued (`unknown`), or + are expected but absent locally (`missing`) are reported as warnings + and do NOT run. If reconciliation can't complete (no GitHub remote, + service unavailable/not deployed, or a rejected token), `check` warns + that verification was skipped and falls back to a full local scan. + +Warnings are human-readable stderr only: they never change the exit +code and never appear in `--json` output (which stays the +`{ success, results }` shape). The authoritative allow-list is the +server's; the CI backstop (`taskless help ci`) is the enforcement point. ## Steps diff --git a/packages/cli/src/help/ci.txt b/packages/cli/src/help/ci.txt index 320216e7..087baa05 100644 --- a/packages/cli/src/help/ci.txt +++ b/packages/cli/src/help/ci.txt @@ -16,7 +16,8 @@ you recognize one not on the list, apply the same patterns. produces an always-green check that gives false confidence.) - A local `taskless check` succeeds (or fails with real findings the user is OK with seeing in CI's first run). -- No auth required for CI — `check` is unauthenticated. +- No auth required for CI — `check` is unauthenticated by default. + (Optionally tokenized as a server-enforced backstop; see step 7.) ## Steps @@ -155,10 +156,29 @@ The six universal steps: YAML for GitHub/GitLab/Azure/Bitbucket; Groovy for Jenkins; different structure for CircleCI. The six steps stay the same. -### 7. Authentication in CI +### 7. Authentication in CI (optional backstop) `taskless check` does NOT require authentication. The generated CI -config works out of the box with no secrets. Only mention auth if +config works out of the box with no secrets and scans all local rules. + +Optionally, exposing a `TASKLESS_TOKEN` secret turns CI into the +**backstop**: an authenticated `check` reconciles the repo's rule files +against the Taskless service and runs exactly the server-blessed `run` +set, warning on drift (`unsafe`), unissued files (`unknown`), or +missing rules. This is the enforcement point for the server-owned rule +set — local developer runs are advisory. To wire it, set the token as +an env var on the check step (GitHub Actions): + +```yaml + - name: Taskless check + env: + TASKLESS_TOKEN: ${{ secrets.TASKLESS_TOKEN }} + run: | + ... +``` + +Add this only when the user wants server-enforced rules in CI; the +unauthenticated default remains fully supported. Also mention auth if the user explicitly asks to run authenticated commands (e.g. `rule create`/`rule improve`) in CI — uncommon. From 3618204ca424b48ee4e4173b77e32b45632a0f42 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 2 Jul 2026 21:10:35 -0700 Subject: [PATCH 06/21] test(cli): cover reconciliation gating, auth state, and warnings Unit tests for the run-set helpers and mock-server integration tests proving check runs only the blessed run set, degrades on an unavailable endpoint, stays silent when logged out or --anonymous, and warns on unsafe/unknown/missing without affecting the exit code or --json output. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../server-owned-rule-reconciliation/tasks.md | 8 +- packages/cli/test/reconcile-check.test.ts | 310 ++++++++++++++++++ packages/cli/test/run-set.test.ts | 131 ++++++++ 3 files changed, 445 insertions(+), 4 deletions(-) create mode 100644 packages/cli/test/reconcile-check.test.ts create mode 100644 packages/cli/test/run-set.test.ts diff --git a/openspec/changes/server-owned-rule-reconciliation/tasks.md b/openspec/changes/server-owned-rule-reconciliation/tasks.md index 101e26c1..961f2c37 100644 --- a/openspec/changes/server-owned-rule-reconciliation/tasks.md +++ b/openspec/changes/server-owned-rule-reconciliation/tasks.md @@ -42,7 +42,7 @@ ## 7. Tests & verification -- [ ] 7.1 Add reconcile-gating integration tests (vitest, subprocess against `dist/`, temp-dir fixtures): only `run`-set rules scanned; non-run files excluded. -- [ ] 7.2 Add auth-state tests: logged-out and `--anonymous` scan all rules silently; authed endpoint-unavailable warns and degrades without failing the exit code; `--json` omits warnings. -- [ ] 7.3 Add mismatch-warning tests: `unsafe`/`unknown`/`missing` warn without changing the exit code; empty run set exits 0 without invoking the scanner. -- [ ] 7.4 Run `pnpm build` then `pnpm --filter @taskless/cli test`, plus `pnpm typecheck` and `pnpm lint`; fix any failures. +- [x] 7.1 Add reconcile-gating integration tests (vitest, subprocess against `dist/`, temp-dir fixtures with a mock reconcile server + git origin): only `run`-set rules scanned; non-run files excluded. Also unit tests for `signRuleFiles`/`selectRunFiles`/`materializeRunDirectory` in `test/run-set.test.ts`. +- [x] 7.2 Add auth-state tests: logged-out and `--anonymous` scan all rules silently (no reconcile call); authed endpoint-unavailable (503) warns and degrades without failing the exit code; `--json` omits warnings. +- [x] 7.3 Add mismatch-warning tests: `unsafe`/`unknown`/`missing` warn without changing the exit code; empty run set exits 0 without invoking the scanner. +- [x] 7.4 Ran `pnpm typecheck`, `pnpm lint`, and the full `pnpm test` (turbo build + 340 tests) — all green. diff --git a/packages/cli/test/reconcile-check.test.ts b/packages/cli/test/reconcile-check.test.ts new file mode 100644 index 00000000..194d5d1d --- /dev/null +++ b/packages/cli/test/reconcile-check.test.ts @@ -0,0 +1,310 @@ +import { execFile } from "node:child_process"; +import { createServer, type Server } from "node:http"; +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { resolve, join } from "node:path"; +import { tmpdir } from "node:os"; +import { promisify } from "node:util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const execFileAsync = promisify(execFile); +const binPath = resolve(import.meta.dirname, "../dist/index.js"); + +interface ReportedFile { + file: string; + signature: string; +} +interface ReconcileRequestBody { + repositoryUrl: string; + files: ReportedFile[]; +} +interface MockResponse { + statusCode: number; + body?: unknown; +} +type Responder = (request: ReconcileRequestBody) => MockResponse; + +interface MockServer { + apiUrl: string; + requests: ReconcileRequestBody[]; + close: () => Promise; +} + +/** Start a mock reconcile endpoint on a random port. */ +function startMockServer(responder: Responder): Promise { + const requests: ReconcileRequestBody[] = []; + const server: Server = createServer((request, response) => { + if (request.method !== "POST" || request.url !== "/cli/api/reconcile") { + response.writeHead(404).end("{}"); + return; + } + let raw = ""; + request.on("data", (chunk: Buffer) => (raw += chunk.toString())); + request.on("end", () => { + const parsed = JSON.parse(raw) as ReconcileRequestBody; + requests.push(parsed); + const { statusCode, body } = responder(parsed); + response.writeHead(statusCode, { "content-type": "application/json" }); + response.end(JSON.stringify(body ?? {})); + }); + }); + return new Promise((resolvePromise) => { + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + const port = typeof address === "object" && address ? address.port : 0; + resolvePromise({ + apiUrl: `http://127.0.0.1:${String(port)}/cli`, + requests, + close: () => + new Promise((done) => { + server.close(() => done()); + }), + }); + }); + }); +} + +/** Look up a reported file's signature so the mock can echo it back. */ +function sig(request: ReconcileRequestBody, file: string): string { + return request.files.find((f) => f.file === file)?.signature ?? ""; +} + +async function runCli( + args: string[], + env: Record = {} +): Promise<{ stdout: string; stderr: string; exitCode: number }> { + try { + const { stdout, stderr } = await execFileAsync("node", [binPath, ...args], { + env: { ...process.env, ...env }, + }); + return { stdout, stderr, exitCode: 0 }; + } catch (error) { + const execError = error as { stdout: string; stderr: string; code: number }; + return { + stdout: execError.stdout ?? "", + stderr: execError.stderr ?? "", + exitCode: execError.code, + }; + } +} + +const RULE_NO_EVAL = [ + "id: no-eval", + "language: javascript", + "severity: error", + "rule:", + " pattern: eval($$$)", + "message: Avoid eval", +].join("\n"); + +const RULE_NO_CONSOLE = [ + "id: no-console", + "language: javascript", + "severity: error", + "rule:", + " pattern: console.log($$$)", + "message: Avoid console.log", +].join("\n"); + +describe("check reconciliation", () => { + let directory: string; + + beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "taskless-reconcile-")); + const rules = join(directory, ".taskless", "rules"); + await mkdir(rules, { recursive: true }); + await writeFile(join(rules, "no-eval.yml"), RULE_NO_EVAL, "utf8"); + await writeFile(join(rules, "no-console.yml"), RULE_NO_CONSOLE, "utf8"); + await writeFile( + join(directory, "app.js"), + 'eval("x");\nconsole.log("y");\n', + "utf8" + ); + // A GitHub origin is required for reconciliation to run. + await execFileAsync("git", ["init"], { cwd: directory }); + await execFileAsync( + "git", + ["remote", "add", "origin", "https://github.com/acme/widgets.git"], + { cwd: directory } + ); + }); + + afterEach(async () => { + await rm(directory, { recursive: true, force: true }); + }); + + it("runs ONLY the blessed run set, excluding non-run rules", async () => { + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + run: [ + { + ruleId: "no-eval", + file: "no-eval.yml", + signature: sig(request, "no-eval.yml"), + }, + ], + unsafe: [], + unknown: [], + missing: [], + }, + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } + ); + expect(server.requests).toHaveLength(1); + expect(exitCode).toBe(1); // no-eval is an error and matches + const parsed = JSON.parse(stdout.trim()) as { + results: { ruleId: string }[]; + }; + const ruleIds = new Set(parsed.results.map((r) => r.ruleId)); + expect(ruleIds.has("no-eval")).toBe(true); + expect(ruleIds.has("no-console")).toBe(false); // not in run set + } finally { + await server.close(); + } + }); + + it("warns on unsafe/unknown/missing and exits 0 on an empty run set", async () => { + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + run: [], + unsafe: [ + { + file: "no-console.yml", + expected: "1;h=sha-256;d=" + "0".repeat(64), + got: sig(request, "no-console.yml"), + }, + ], + unknown: [{ file: "no-eval.yml" }], + missing: [{ ruleId: "no-var", file: "no-var-abc.yml" }], + }, + })); + try { + const { stdout, stderr, exitCode } = await runCli( + ["check", "-d", directory], + { + TASKLESS_TOKEN: "fake.token.value", + TASKLESS_API_URL: server.apiUrl, + } + ); + expect(exitCode).toBe(0); // empty run set → nothing scanned + expect(stdout).toContain("No issues found"); + expect(stderr).toContain("no-console.yml"); // unsafe drift + expect(stderr).toContain("no-eval.yml"); // unknown + expect(stderr).toContain("no-var"); // missing + } finally { + await server.close(); + } + }); + + it("suppresses mismatch warnings under --json", async () => { + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + run: [], + unsafe: [ + { + file: "no-console.yml", + expected: "1;h=sha-256;d=" + "0".repeat(64), + got: sig(request, "no-console.yml"), + }, + ], + unknown: [], + missing: [], + }, + })); + try { + const { stdout, stderr, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } + ); + expect(exitCode).toBe(0); + expect(stderr).not.toContain("tamper"); + expect(stderr).not.toContain("no-console.yml"); + const parsed = JSON.parse(stdout.trim()) as { + success: boolean; + results: unknown[]; + }; + expect(parsed).toEqual({ success: true, results: [] }); + } finally { + await server.close(); + } + }); + + it("degrades to a local scan when reconciliation is unavailable", async () => { + const server = await startMockServer(() => ({ statusCode: 503 })); + try { + const { stdout, stderr, exitCode } = await runCli( + ["check", "-d", directory], + { + TASKLESS_TOKEN: "fake.token.value", + TASKLESS_API_URL: server.apiUrl, + } + ); + expect(server.requests).toHaveLength(1); + expect(exitCode).toBe(1); // fell back to a full local scan; no-eval matches + expect(stdout).toContain("no-eval"); + expect(stdout).toContain("no-console"); // ALL local rules ran + expect(stderr).toContain("Scanning all local rules unverified"); + } finally { + await server.close(); + } + }); + + it("does not warn under --json when degrading", async () => { + const server = await startMockServer(() => ({ statusCode: 503 })); + try { + const { stdout, stderr } = await runCli( + ["check", "-d", directory, "--json"], + { + TASKLESS_TOKEN: "fake.token.value", + TASKLESS_API_URL: server.apiUrl, + } + ); + expect(stderr).not.toContain("Scanning all local rules unverified"); + const parsed = JSON.parse(stdout.trim()) as { results: unknown[] }; + expect(parsed.results.length).toBeGreaterThan(0); + } finally { + await server.close(); + } + }); + + it("scans all local rules silently when logged out (no reconcile)", async () => { + const server = await startMockServer(() => ({ statusCode: 200, body: {} })); + try { + const { stdout, stderr, exitCode } = await runCli( + ["check", "-d", directory], + { + TASKLESS_TOKEN: "", + TASKLESS_API_URL: server.apiUrl, + } + ); + expect(server.requests).toHaveLength(0); // never reconciled + expect(exitCode).toBe(1); + expect(stdout).toContain("no-eval"); + expect(stdout).toContain("no-console"); + expect(stderr).not.toContain("unverified"); + } finally { + await server.close(); + } + }); + + it("forces the logged-out path under --anonymous even with a token", async () => { + const server = await startMockServer(() => ({ statusCode: 200, body: {} })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--anonymous"], + { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } + ); + expect(server.requests).toHaveLength(0); // --anonymous skips reconcile + expect(exitCode).toBe(1); + expect(stdout).toContain("no-eval"); + expect(stdout).toContain("no-console"); + } finally { + await server.close(); + } + }); +}); diff --git a/packages/cli/test/run-set.test.ts b/packages/cli/test/run-set.test.ts new file mode 100644 index 00000000..eeddc9c6 --- /dev/null +++ b/packages/cli/test/run-set.test.ts @@ -0,0 +1,131 @@ +import { + mkdtemp, + rm, + mkdir, + writeFile, + readdir, + readFile, +} from "node:fs/promises"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { canonicalHash } from "../src/rules/rule-hash"; +import { + materializeRunDirectory, + selectRunFiles, + signRuleFiles, + type SignedRuleFile, +} from "../src/rules/run-set"; + +async function writeRule( + cwd: string, + name: string, + body: string +): Promise { + const rulesDirectory = join(cwd, ".taskless", "rules"); + await mkdir(rulesDirectory, { recursive: true }); + await writeFile(join(rulesDirectory, name), body, "utf8"); +} + +describe("signRuleFiles", () => { + let directory: string; + beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "taskless-runset-")); + }); + afterEach(async () => { + await rm(directory, { recursive: true, force: true }); + }); + + it("signs each rule file with its canonical signature", async () => { + await writeRule(directory, "a.yml", "id: a\n"); + await writeRule(directory, "b.yml", "id: b\n"); + + const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); + + expect(signed.map((s) => s.file).toSorted()).toEqual(["a.yml", "b.yml"]); + for (const entry of signed) { + const expected = await canonicalHash(await readFile(entry.path, "utf8")); + expect(entry.signature).toBe(expected); + } + }); +}); + +describe("selectRunFiles", () => { + const signed: SignedRuleFile[] = [ + { file: "a.yml", path: "/x/a.yml", signature: "1;h=sha-256;d=aaaa" }, + { file: "b.yml", path: "/x/b.yml", signature: "1;h=sha-256;d=bbbb" }, + ]; + + it("selects only files whose signature the server blessed", () => { + const runFiles = selectRunFiles(signed, [ + { ruleId: "a", file: "a.yml", signature: "1;h=sha-256;d=aaaa" }, + ]); + expect(runFiles.map((s) => s.file)).toEqual(["a.yml"]); + }); + + it("matches by signature, not path (a moved-but-unchanged file resolves)", () => { + const runFiles = selectRunFiles(signed, [ + { ruleId: "b", file: "moved/b.yml", signature: "1;h=sha-256;d=bbbb" }, + ]); + expect(runFiles.map((s) => s.file)).toEqual(["b.yml"]); + }); + + it("drops run entries with no local signature match", () => { + const runFiles = selectRunFiles(signed, [ + { ruleId: "c", file: "c.yml", signature: "1;h=sha-256;d=cccc" }, + ]); + expect(runFiles).toEqual([]); + }); + + it("does not select the same local file twice", () => { + const runFiles = selectRunFiles(signed, [ + { ruleId: "a", file: "a.yml", signature: "1;h=sha-256;d=aaaa" }, + { ruleId: "a", file: "a-copy.yml", signature: "1;h=sha-256;d=aaaa" }, + ]); + expect(runFiles.map((s) => s.file)).toEqual(["a.yml"]); + }); +}); + +describe("materializeRunDirectory", () => { + let directory: string; + beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "taskless-runset-")); + }); + afterEach(async () => { + await rm(directory, { recursive: true, force: true }); + }); + + it("copies only the selected files and gitignores .run/", async () => { + await writeRule(directory, "a.yml", "id: a\n"); + await writeRule(directory, "b.yml", "id: b\n"); + const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); + const onlyA = signed.filter((s) => s.file === "a.yml"); + + await materializeRunDirectory(directory, onlyA); + + const runRules = join(directory, ".taskless", ".run", "rules"); + expect(await readdir(runRules)).toEqual(["a.yml"]); + + const gitignore = await readFile( + join(directory, ".taskless", ".gitignore"), + "utf8" + ); + expect(gitignore).toContain(".run/"); + }); + + it("rebuilds the run dir, dropping stale files from a prior run", async () => { + await writeRule(directory, "a.yml", "id: a\n"); + await writeRule(directory, "b.yml", "id: b\n"); + const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); + + await materializeRunDirectory(directory, signed); // both + await materializeRunDirectory( + directory, + signed.filter((s) => s.file === "b.yml") + ); // only b + + const runRules = join(directory, ".taskless", ".run", "rules"); + expect(await readdir(runRules)).toEqual(["b.yml"]); + }); +}); From efb5eb5574a8470538b4cf1aae21497e6e1e51d1 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Thu, 2 Jul 2026 21:16:40 -0700 Subject: [PATCH 07/21] chore(openspec): sync + archive server-owned-rule-reconciliation Applies the delta specs to the living specs (new cli-rule-reconciliation capability; cli-check gains auth-state gating, warnings, degrade, and run-set-only execution) and archives the completed change. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/cli-check/spec.md | 0 .../specs/cli-rule-reconciliation/spec.md | 0 .../tasks.md | 0 openspec/specs/cli-check/spec.md | 118 +++++++++- .../specs/cli-rule-reconciliation/spec.md | 214 ++++++++++++++++++ 8 files changed, 325 insertions(+), 7 deletions(-) rename openspec/changes/{server-owned-rule-reconciliation => archive/2026-07-02-server-owned-rule-reconciliation}/.openspec.yaml (100%) rename openspec/changes/{server-owned-rule-reconciliation => archive/2026-07-02-server-owned-rule-reconciliation}/design.md (100%) rename openspec/changes/{server-owned-rule-reconciliation => archive/2026-07-02-server-owned-rule-reconciliation}/proposal.md (100%) rename openspec/changes/{server-owned-rule-reconciliation => archive/2026-07-02-server-owned-rule-reconciliation}/specs/cli-check/spec.md (100%) rename openspec/changes/{server-owned-rule-reconciliation => archive/2026-07-02-server-owned-rule-reconciliation}/specs/cli-rule-reconciliation/spec.md (100%) rename openspec/changes/{server-owned-rule-reconciliation => archive/2026-07-02-server-owned-rule-reconciliation}/tasks.md (100%) create mode 100644 openspec/specs/cli-rule-reconciliation/spec.md diff --git a/openspec/changes/server-owned-rule-reconciliation/.openspec.yaml b/openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/.openspec.yaml similarity index 100% rename from openspec/changes/server-owned-rule-reconciliation/.openspec.yaml rename to openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/.openspec.yaml diff --git a/openspec/changes/server-owned-rule-reconciliation/design.md b/openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/design.md similarity index 100% rename from openspec/changes/server-owned-rule-reconciliation/design.md rename to openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/design.md diff --git a/openspec/changes/server-owned-rule-reconciliation/proposal.md b/openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/proposal.md similarity index 100% rename from openspec/changes/server-owned-rule-reconciliation/proposal.md rename to openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/proposal.md diff --git a/openspec/changes/server-owned-rule-reconciliation/specs/cli-check/spec.md b/openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/specs/cli-check/spec.md similarity index 100% rename from openspec/changes/server-owned-rule-reconciliation/specs/cli-check/spec.md rename to openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/specs/cli-check/spec.md diff --git a/openspec/changes/server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md b/openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md similarity index 100% rename from openspec/changes/server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md rename to openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/specs/cli-rule-reconciliation/spec.md diff --git a/openspec/changes/server-owned-rule-reconciliation/tasks.md b/openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/tasks.md similarity index 100% rename from openspec/changes/server-owned-rule-reconciliation/tasks.md rename to openspec/changes/archive/2026-07-02-server-owned-rule-reconciliation/tasks.md diff --git a/openspec/specs/cli-check/spec.md b/openspec/specs/cli-check/spec.md index 70003d7d..b3bf88ea 100644 --- a/openspec/specs/cli-check/spec.md +++ b/openspec/specs/cli-check/spec.md @@ -60,15 +60,27 @@ The CLI SHALL check for the presence of YAML rule files in the `.taskless/rules/ ### Requirement: Check subcommand executes ast-grep scan -The CLI SHALL generate an ephemeral `sgconfig.yml` in `.taskless/` and execute `sg scan --config .taskless/sgconfig.yml --json=stream` using `child_process.spawn` with `shell: true` for cross-platform binary resolution. The `sg` binary SHALL be resolved from the `@ast-grep/cli` dependency via PATH. +The CLI SHALL generate an ephemeral `sgconfig.yml` in `.taskless/` and execute +`sg scan --config .taskless/sgconfig.yml --json=stream` using `child_process.spawn` with +`shell: true` for cross-platform binary resolution. The `sg` binary SHALL be resolved from +the `@ast-grep/cli` dependency via PATH. When reconciliation succeeds, the scan SHALL cover +only the blessed `run`-set rule files; on the unauthenticated/`--anonymous` path, or when an +authenticated reconciliation degrades to a local scan, the scan SHALL cover all local rule +files as before. #### Scenario: ast-grep scan runs with generated config - **WHEN** the CLI executes the scanner -- **THEN** it SHALL first write `.taskless/sgconfig.yml` with `ruleDirs: ['rules']` +- **THEN** it SHALL first write `.taskless/sgconfig.yml` - **AND** it SHALL invoke `sg scan` with `--config .taskless/sgconfig.yml` and `--json=stream` - **AND** the working directory for the spawned process SHALL be the resolved project directory +#### Scenario: Scan is limited to the run set when reconciled + +- **WHEN** reconciliation succeeded and returned a `run` set +- **THEN** the generated scan configuration SHALL cause `sg scan` to evaluate only the + `run`-set rule files + #### Scenario: ast-grep binary is not found - **WHEN** the `sg` binary cannot be resolved from PATH @@ -205,14 +217,23 @@ Before forwarding, the CLI SHALL silently drop any path that does not exist on d ### Requirement: Check accepts --anonymous as a no-op -The `taskless check` command SHALL accept the global `--anonymous` flag (per the `cli` capability) without changing its behavior. `check` does not call the Taskless API, so the flag is effectively a no-op for this command. +The `taskless check` command SHALL accept the global `--anonymous` flag (per the `cli` +capability). Because `check` reconciles against the Taskless API when authenticated, +`--anonymous` SHALL force the logged-out path: it SHALL suppress the reconcile network call +and run all local rule files. Aside from forcing the logged-out path, `--anonymous` SHALL NOT +change scan behavior, output shape, or exit codes relative to an unauthenticated `check`. + +#### Scenario: check --anonymous skips reconciliation + +- **WHEN** a user runs `taskless check --anonymous` +- **THEN** the CLI SHALL NOT call `POST /cli/api/reconcile` +- **AND** SHALL scan all local rule files -#### Scenario: check --anonymous behaves identically to check +#### Scenario: check --anonymous matches an unauthenticated check - **WHEN** a user runs `taskless check --anonymous` -- **THEN** the CLI SHALL execute the same logic as `taskless check` -- **AND** SHALL produce identical output (no warning, no error) -- **AND** SHALL exit with the same code as `taskless check` would +- **THEN** its scan behavior, output, and exit code SHALL match `taskless check` run with no + available token ### Requirement: Check error output uses standardized error envelope @@ -223,3 +244,86 @@ When `taskless check --json` exits with an error, the output SHALL conform to th - **WHEN** `taskless check --json` fails (e.g. ast-grep invocation error) - **THEN** stdout SHALL contain a JSON object matching the standardized error envelope - **AND** SHALL include a stable `code` field (e.g. `SCAN_FAILED` if added to the enum) + +### Requirement: Check selects what it runs from auth state + +`taskless check` SHALL NOT require authentication, and it SHALL choose what it runs from the current auth state. When no token is available, or when `--anonymous` is set, the CLI SHALL run all local rule files under `.taskless/rules/` without contacting the server (the offline linter posture). When a token is available and `--anonymous` is not set, the CLI SHALL reconcile against the server (per "Check reconciles rule files before scanning") and run only the returned `run` set. The unauthenticated path SHALL succeed with no network access and SHALL NOT emit a warning; any informational notice on that path is optional. + +#### Scenario: Unauthenticated check runs local rules + +- **WHEN** a user runs `taskless check` with no available token +- **THEN** the CLI SHALL scan all local rule files +- **AND** SHALL NOT call `POST /cli/api/reconcile` +- **AND** SHALL NOT emit a warning about missing authentication + +#### Scenario: Authenticated check reconciles + +- **WHEN** a user runs `taskless check` with an available token and without `--anonymous` +- **THEN** the CLI SHALL call `POST /cli/api/reconcile` and run only the `run` set + +#### Scenario: Anonymous forces the logged-out path + +- **WHEN** a user runs `taskless check --anonymous` while a token is available +- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (run all local rules, no reconcile call) + +### Requirement: Check reconciles rule files before scanning + +`taskless check` SHALL reconcile before scanning whenever a bearer token and a `repositoryUrl` are resolvable and `--anonymous` is not set. It SHALL compute the signature envelope for every `.yml` file under `.taskless/rules/`, call `POST /cli/api/reconcile` with `{ repositoryUrl, files }`, and then scan **only** the files returned in the `run` set, matched back to local files by signature (per the `cli-rule-reconciliation` capability). + +#### Scenario: Only blessed rules are scanned + +- **WHEN** a user runs `taskless check` while authenticated and reconciliation returns a + `run` set that is a subset of the local rule files +- **THEN** the CLI SHALL invoke `sg scan` against only the `run`-set rule files +- **AND** SHALL NOT scan any local rule file absent from `run` + +### Requirement: Check warns on reconciliation mismatches + +`taskless check` SHALL warn on every mismatch reported by a successful reconciliation and SHALL NOT let those warnings change the exit code. `unsafe` entries SHALL be warned as tamper/drift, `unknown` as not-issued-by-the-server, and `missing` as audit-only. The exit code SHALL continue to be governed solely by error-severity scan results from the executed `run` set. Warnings SHALL be human-readable output only; when `--json` is set the output SHALL remain the existing `{ success, results }` shape and SHALL NOT include the warnings. + +#### Scenario: Unsafe drift is warned without failing the exit code + +- **WHEN** reconciliation returns an `unsafe` entry and the `run`-set scan produces no + error-severity results +- **THEN** the CLI SHALL warn about the drift +- **AND** SHALL exit with code 0 + +#### Scenario: Missing rules are warned as audit-only + +- **WHEN** reconciliation returns `missing` entries +- **THEN** the CLI SHALL warn about them as audit information +- **AND** SHALL NOT change the exit code on their account + +#### Scenario: Warnings are suppressed under --json + +- **WHEN** reconciliation returns mismatches and `--json` is set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** SHALL NOT contain the human-readable mismatch warnings + +### Requirement: Check degrades to a local scan when reconciliation cannot complete + +`taskless check` SHALL NOT fail solely because an attempted reconciliation cannot complete. When a token is available and `--anonymous` is not set but reconciliation cannot complete (no resolvable git remote, or the reconcile endpoint is unreachable or not yet deployed, or a transport error), the CLI SHALL warn that rule verification could not be performed and SHALL fall back to scanning all local rule files. The CLI SHALL NOT exit with a non-zero code solely because reconciliation failed, and the warning SHALL be suppressed under `--json`. + +#### Scenario: Endpoint unreachable degrades to local scan + +- **WHEN** an authenticated `check` attempts reconciliation and the endpoint is unreachable or returns a not-deployed error +- **THEN** the CLI SHALL warn that verification could not be performed +- **AND** SHALL scan all local rule files +- **AND** SHALL NOT exit with a non-zero code solely due to the reconcile failure + +#### Scenario: Degrade warning is suppressed under --json + +- **WHEN** the CLI degrades to a local scan and `--json` is set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** SHALL NOT contain the human-readable degrade warning + +### Requirement: Check exits cleanly when the run set is empty + +`taskless check` SHALL NOT invoke the scanner when a successful reconciliation returns an empty `run` set (for example an empty corpus, or every reported file classified `unsafe`/`unknown`); it SHALL report no scan results and SHALL exit with code 0. Any `unsafe` / `unknown` mismatch warnings SHALL still be emitted. + +#### Scenario: Empty run set skips the scan + +- **WHEN** reconciliation returns an empty `run` set +- **THEN** the CLI SHALL NOT invoke `sg scan` +- **AND** SHALL exit with code 0 +- **AND** SHALL still warn about any `unsafe`/`unknown` mismatches diff --git a/openspec/specs/cli-rule-reconciliation/spec.md b/openspec/specs/cli-rule-reconciliation/spec.md new file mode 100644 index 00000000..3f37b76a --- /dev/null +++ b/openspec/specs/cli-rule-reconciliation/spec.md @@ -0,0 +1,214 @@ +# CLI Rule Reconciliation + +## Purpose + +Defines the CLI-side contract for server-owned rule reconciliation: how the CLI computes a canonical signature envelope for each rule file, how it calls the `POST /cli/api/reconcile` endpoint, and how it executes only the server's `run` set. The server-side record is authoritative; local signatures are advisory only. + +## Requirements + +### Requirement: Canonical rule signature envelope + +The CLI SHALL represent a rule file's canonical signature as a single self-describing +string of the form `;h=;d=`. For algoVersion `1` this is +`1;h=sha-256;d=`, where `` is the digest as lowercase hexadecimal. The token +before the **first** `;` is the algoVersion and SHALL be read up to that one delimiter to +detect the version (and therefore the normalization procedure and hash algorithm) before +any `key=value` parameters are parsed. Signatures SHALL be compared as whole strings. + +#### Scenario: Envelope is emitted for algoVersion 1 + +- **WHEN** the CLI computes a signature for a rule file's bytes using algoVersion 1 +- **THEN** the result SHALL be a string `1;h=sha-256;d=` with `` lowercase + +#### Scenario: Version is read before parameters + +- **WHEN** the CLI parses a signature string +- **THEN** it SHALL read the algoVersion as the substring before the first `;` +- **AND** SHALL NOT rely on the `key=value` parameter syntax to determine the version + +#### Scenario: Signatures compare as whole strings + +- **WHEN** the CLI compares two signatures for equality +- **THEN** it SHALL compare the full envelope strings, not the bare digests + +### Requirement: Signature normalization procedure (algoVersion 1) + +The CLI SHALL compute an algoVersion-1 digest as `SHA-256( normalize(fileText) )`, +hex-encoded lowercase, wrapped in the envelope. `normalize()` SHALL operate on raw decoded +text only and SHALL NOT parse or re-serialize YAML. In order, `normalize()` SHALL: + +1. Decode the file as UTF-8. +2. Strip a single leading UTF-8 byte-order mark if present (decoded as `U+FEFF`). +3. Convert every CRLF and lone CR to LF. +4. Strip all trailing newlines, then append exactly one LF. +5. Re-encode as UTF-8 and SHA-256, hex lowercase. + +The CLI SHALL NOT apply Unicode normalization (NFC/NFD): canonically-equivalent strings in +different composition forms SHALL hash differently. A change to `normalize()` SHALL ship as +a new algoVersion, never as a redefinition of an existing one. + +#### Scenario: CRLF and LF hash identically + +- **WHEN** two files differ only in CRLF versus LF line endings +- **THEN** their algoVersion-1 signatures SHALL be equal + +#### Scenario: Trailing newlines are collapsed to one + +- **WHEN** two files differ only in the number of trailing newlines (including none) +- **THEN** their algoVersion-1 signatures SHALL be equal + +#### Scenario: Leading BOM is stripped + +- **WHEN** a file has a leading UTF-8 BOM and an otherwise identical file does not +- **THEN** their algoVersion-1 signatures SHALL be equal + +#### Scenario: Meaningful content change differs + +- **WHEN** two files differ in any non-newline, non-BOM byte +- **THEN** their algoVersion-1 signatures SHALL differ + +#### Scenario: Combining marks are not NFC-folded + +- **WHEN** one file contains a precomposed character and another the decomposed form +- **THEN** their algoVersion-1 signatures SHALL differ + +### Requirement: Signature hashing uses web-standard APIs only + +The signature implementation SHALL use only web-standard APIs (`crypto.subtle.digest('SHA-256', …)` +and `TextEncoder`) and SHALL NOT depend on a Node-specific crypto module, so the CLI +reproduces the server's reference implementation byte-for-byte. + +#### Scenario: No node-specific crypto dependency + +- **WHEN** the signature module hashes a file's normalized bytes +- **THEN** it SHALL use `crypto.subtle` and `TextEncoder` +- **AND** SHALL NOT import `node:crypto` + +### Requirement: Conformance vectors are fetched and asserted + +The CLI SHALL consume the cross-repo conformance vectors served at +`GET /cli/api/rule-hash-vectors` (unauthenticated) as `{ vectors: [{ name, input, signature }] }`, +commit a copy as a fixture, and assert in its test suite that its independent +`normalize()`-plus-hash reproduces every vector's `signature` exactly. Non-ASCII `input` +SHALL be parsed as JSON (decoding `\uXXXX` escapes) before hashing. A vector mismatch SHALL +be a release blocker (the test SHALL fail the build). + +#### Scenario: Local hasher reproduces every vector + +- **WHEN** the conformance test runs against the committed vectors +- **THEN** the CLI SHALL compute the exact `signature` for every vector entry + +#### Scenario: A mismatch blocks release + +- **WHEN** any vector's computed signature does not match its expected `signature` +- **THEN** the test SHALL fail +- **AND** the build SHALL NOT pass + +### Requirement: Reconcile reports every held rule file + +The CLI SHALL reconcile by sending `POST /cli/api/reconcile` with an +`Authorization: Bearer ` header and a JSON body `{ repositoryUrl, files }`, where +`repositoryUrl` is the full repository URL and `files` is an array of +`{ file, signature }` covering **every** rule file the CLI would otherwise run under +`.taskless/rules/`. `file` SHALL be the rule file's delivered name as it exists on disk and +`signature` SHALL be the full envelope computed for that file's bytes. The CLI SHALL send the +whole signature envelope, not a bare digest. + +#### Scenario: All rule files are reported + +- **WHEN** the CLI reconciles with rule files present under `.taskless/rules/` +- **THEN** the request body SHALL include one `{ file, signature }` entry per rule file + +#### Scenario: Full envelope is sent + +- **WHEN** the CLI reports a file's signature +- **THEN** it SHALL send the complete `1;h=sha-256;d=` envelope, not only the digest + +### Requirement: Reconcile response buckets drive execution + +The reconcile response SHALL be interpreted as four buckets — `run`, `unsafe`, `unknown`, +and `missing` — and each SHALL drive a specific CLI action: + +- `run`: the file's content matches a rule the server blessed. The CLI SHALL execute it. This + is the complete allow-list. +- `unsafe`: a rule held by delivered name whose content differs from what the server blessed + (`expected` vs `got`). The CLI SHALL NOT run it and SHALL surface it as tamper/drift. +- `unknown`: a reported file the server never issued. The CLI SHALL NOT run it and SHALL + surface it as advisory. +- `missing`: a rule the server expected that the CLI did not report. It is not actionable for + execution and SHALL be treated as advisory/audit only. + +The CLI SHALL match each `run` entry back to a local file by its `signature` (content-based +join), so a file that was moved but not changed still resolves. + +#### Scenario: Run entries are matched by signature + +- **WHEN** the CLI processes a `run` entry +- **THEN** it SHALL locate the corresponding local file by matching the `signature`, not the path + +#### Scenario: Unsafe is surfaced and not run + +- **WHEN** a reported file lands in `unsafe` +- **THEN** the CLI SHALL NOT execute it +- **AND** SHALL surface it as tamper/drift + +#### Scenario: Unknown is surfaced and not run + +- **WHEN** a reported file lands in `unknown` +- **THEN** the CLI SHALL NOT execute it +- **AND** SHALL surface it as an advisory notice + +#### Scenario: Missing is advisory only + +- **WHEN** the response includes `missing` entries +- **THEN** the CLI SHALL treat them as advisory/audit only and SHALL NOT fail execution on them + +### Requirement: The CLI executes only the server run set + +When reconciliation succeeds, the CLI SHALL execute exactly the files in the server's `run` +set and no other file. The CLI SHALL NOT run a file on the basis of its own local comparison +of signatures. This replaces any local classification of rules. + +#### Scenario: Only run-set files execute + +- **WHEN** reconciliation returns a `run` set that is a strict subset of the reported files +- **THEN** the CLI SHALL execute only the files in `run` +- **AND** SHALL NOT execute any reported file absent from `run` + +#### Scenario: No local self-classification + +- **WHEN** the CLI holds a local signature or sidecar for a file +- **THEN** it SHALL NOT treat that local value as authorization to run the file + +### Requirement: Reconcile is scoped to the token's organization + +The reconcile endpoint SHALL be authorized by the same bearer-token / `orgId`-claim scheme as +all `/cli/api/*` endpoints and SHALL be scoped to the organization the token owns. The CLI +SHALL handle the documented edges: a `401` with `{ error: "unauthorized" }` for a missing or +invalid token; an empty corpus (empty `run`/`missing`, every reported file in `unknown`, +nothing runs); and an empty report (empty `run`/`unsafe`/`unknown`, the full corpus in +`missing`). + +#### Scenario: Unauthorized token + +- **WHEN** the CLI calls reconcile without a valid bearer token +- **THEN** the server SHALL return `401` with `{ error: "unauthorized" }` +- **AND** the CLI SHALL NOT execute any rule from a `run` set + +#### Scenario: Empty corpus runs nothing + +- **WHEN** the repository has no blessed rules and the CLI reports files +- **THEN** every reported file SHALL be returned in `unknown` +- **AND** the CLI SHALL execute nothing + +### Requirement: Local signatures are advisory only + +Any signature the CLI persists into the repository (for example as sidecar metadata) SHALL be +treated as an offline-fallback convenience only and SHALL NOT be treated as an authorization +signal. The server-side record is authoritative and the server decides what runs. + +#### Scenario: Sidecar signature is not authorization + +- **WHEN** a rule file has a locally stored signature that matches its content +- **THEN** the CLI SHALL NOT run the file on that basis alone +- **AND** SHALL rely on the server's `run` set for authorization From 959c43ef07e1b78f086b1396f4b20dac7f3484fa Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 07:56:13 -0700 Subject: [PATCH 08/21] docs(openspec): propose runtime rule execution Add the OpenSpec change defining how a runtime rule executes, stacked on server-owned rule reconciliation. Reconciliation gates which files may run; this proposal defines the local harness that evaluates a runtime rule. A runtime rule is a directory under .taskless/runtime-rules/ (capture *.yml + a check.ts). The harness assembles the capture rules into one ast-grep narrow, gates on matches, then invokes check.ts's default export via a bundled tsx. check.ts is arbitrary code execution, so it runs only when its signature is validated by reconciliation, or under --dangerously-run-scripts. Static ast-grep rules and inert capture *.yml are never gated. Adds the five auth/flag modes, a --timeout bound, and an additive --json skipped field. Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../runtime-rule-execution/.openspec.yaml | 2 + .../changes/runtime-rule-execution/design.md | 187 ++++++++++++++++++ .../runtime-rule-execution/proposal.md | 83 ++++++++ .../specs/cli-check/spec.md | 167 ++++++++++++++++ .../specs/cli-rule-reconciliation/spec.md | 48 +++++ .../specs/cli-runtime-rule-execution/spec.md | 130 ++++++++++++ .../changes/runtime-rule-execution/tasks.md | 45 +++++ 7 files changed, 662 insertions(+) create mode 100644 openspec/changes/runtime-rule-execution/.openspec.yaml create mode 100644 openspec/changes/runtime-rule-execution/design.md create mode 100644 openspec/changes/runtime-rule-execution/proposal.md create mode 100644 openspec/changes/runtime-rule-execution/specs/cli-check/spec.md create mode 100644 openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md create mode 100644 openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md create mode 100644 openspec/changes/runtime-rule-execution/tasks.md diff --git a/openspec/changes/runtime-rule-execution/.openspec.yaml b/openspec/changes/runtime-rule-execution/.openspec.yaml new file mode 100644 index 00000000..43e65ca6 --- /dev/null +++ b/openspec/changes/runtime-rule-execution/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-03 diff --git a/openspec/changes/runtime-rule-execution/design.md b/openspec/changes/runtime-rule-execution/design.md new file mode 100644 index 00000000..18d9aa31 --- /dev/null +++ b/openspec/changes/runtime-rule-execution/design.md @@ -0,0 +1,187 @@ +## Context + +`taskless check` (`packages/cli/src/commands/check.ts`) has a single executor. The +stacked-under change added reconciliation: enumerate `.taskless/rules/*.yml`, sign each file +(`src/rules/rule-hash.ts`, envelope `1;h=sha-256;d=` over normalized bytes), call +`POST /cli/api/reconcile`, materialize the blessed `run` set into a gitignored +`.taskless/.run/rules/`, and `sg scan` it (`src/rules/run-set.ts`, `src/filesystem/sgconfig.ts`, +`src/rules/scan.ts`). Findings surface through the scanner-agnostic `CheckResult` +(`src/types/check.ts`, `source: "ast-grep"` today). + +A **runtime rule** is a different on-disk shape (TSKL-243, resolved): a **directory** under +`.taskless/runtime-rules/` (with fixtures under `.taskless/runtime-rule-tests/`) holding one or +more ast-grep capture `*.yml` and exactly one `check.ts`. Each capture rule carries +`metadata.taskless`: `version`, `kind: runtime`, `name`, `check`, and `match: anchor|broad` +(broad = whole-language `kind: program` enumerator). A capture rule has two identifiers — a +hashed, globally-unique `id` (`${ruleSlug}-${sha1(ruleBody).slice(0,8)}`) for scan→rule +attribution, and a stable model-assigned `name` the check branches on. Capture rules may carry +full ast-grep config (`constraints`/`utils`/`transform` as **siblings** of `rule`). + +The local harness is resolved in TSKL-245: assemble a rule's capture rules → **one** +`ast-grep scan` (anchor `--inline-rules --json=stream`; broad `--files-with-matches`) → gate +on matches → invoke `check.ts`'s **default export** with `(root, matches)` via a bundled, +pinned `tsx`; use the **returned** `Finding[]`. `Finding.severity ∈ error|warning|info` maps +onto static-rule gating with no translation. Measured cost is `tsx` per-worker startup +(~590ms), not per-import (~9–14ms warm), so scheduling (process / `worker_threads` pool / +`import()`) is left to the harness. + +The one thing that shape does not carry is trust: `check.ts` is arbitrary code execution. This +change makes the server signature the gate for running it, and reuses reconciliation (already +built, already authenticated) as that gate rather than inventing a local sandbox. + +## Goals / Non-Goals + +**Goals:** + +- Recognize a runtime-rule directory and execute it with the TSKL-245 harness + (narrow → gate → `check.ts`), producing `CheckResult`s indistinguishable downstream from + static findings. +- Run `check.ts` **only** on a signature-validated path; skip (never run) on every unverified + path; provide `--dangerously-run-scripts` as the sole local override. +- Keep static ast-grep rules exactly as they are (always run, no gating, offline linter). +- Bundle a pinned TypeScript loader so `check.ts` runs without the user's toolchain. + +**Non-Goals:** + +- The **hardened, sandboxed** enforced runner (network/credential-isolated substrate) and its + commit-bound integrity report. That is the Taskless-hosted `--enforce` runtime, owned + server-side (TSKL-237/262). This change is the **local** harness (eslint-equivalent trust, + gated by signature), not the sandbox. +- **Generating** runtime rules or classifying static-vs-runtime — owned by the service + (`classifyStep`, TSKL-241/244). +- Redefining the signature envelope or `normalize()` — reused unchanged from the stacked-under + change. + +## Decisions + +### Decision: Runtime rules live in their own tree; location is the primary classifier + +Runtime rules live under `.taskless/runtime-rules/` — each rule a directory holding its capture +`*.yml` and `check.ts` — with fixtures under `.taskless/runtime-rule-tests/`. Static ast-grep +rules stay under `.taskless/rules/`. **Location is the primary rule-class split**: `check` +scans `.taskless/rules/` for static rules and `.taskless/runtime-rules/` for runtime rules, +and `metadata.taskless.kind: runtime` confirms the class. The CLI reads +`metadata.taskless.check` to locate the directory's `check.ts` and `metadata.taskless.match` +to pick the ast-grep invocation mode, and never parses rule intent beyond this metadata +envelope and the ast-grep config it already understands. `.taskless/runtime-rule-tests/` holds +verification fixtures and is not executed by `check`. + +_Alternative rejected:_ co-locating runtime rules under `.taskless/rules/` and splitting on +directory-vs-file. The separate tree is what the generator writes, and a distinct path removes +any ambiguity about which executor owns a given entry. + +### Decision: The gate is the rule's `check.ts`; capture `*.yml` are inert and ungated + +The signature gate is the one artifact that carries arbitrary code execution: `check.ts`. +`src/rules/run-set.ts` grows to enumerate runtime-rule directories and sign each rule's +`check.ts` (only) with the existing envelope, reporting it as `{ file, signature }`. A runtime +rule is **eligible to execute only if its `check.ts` is returned in `run`**; a `check.ts` in +`unsafe`/`unknown`/`missing` withholds the rule and surfaces it as advisory. Capture `*.yml` +are inert ast-grep patterns — they cannot execute code, so they are not signed or gated; the +worst a tampered capture can do is change which matches feed an already-authentic, already- +blessed `check.ts`, which the enforced runner remains the authority over. + +_Alternative rejected:_ sign every file of the rule (each capture `*.yml` plus `check.ts`) and +require all in `run`. It adds withholding churn over inert data for no ACE benefit; the YAML +is harmless. + +### Decision: Static rules are not gated; reconciliation is scoped to runtime rules + +Static ast-grep `*.yml` are inert data — they always run, with no network, exactly as before +the stacked-under change. Only runtime rules are reported to and gated by reconciliation. This +refines the stacked-under `cli-rule-reconciliation` requirement ("report every rule file"): +the reported corpus is the runtime rules. Concretely, the stacked-under degrade path — which +scans "all local rules unverified" — is narrowed so it scans static rules but **never executes +runtime `check.ts`** without a validated signature. + +_Alternative rejected:_ keep gating everything and rely on the server to always bless static +files. That leaves a not-deployed / offline `check` unable to run harmless static rules, a +regression against today's linter posture for zero security benefit. + +### Decision: Execution is driven by a single question — "is this runtime rule's signature validated?" + +`check` resolves a runtime-execution disposition from auth state and flags: + +| State | Runtime rules | +| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| Authenticated (token) | reconcile; execute every rule fully in `run`; report the rest as advisory | +| API key | same as authenticated | +| Logged out / `--anonymous` | skip; report that runtime rules exist and were not run | +| Reconcile cannot complete (no remote, endpoint down) | skip; notice that runtime rules could not be verified | +| `--dangerously-run-scripts` (any auth state) | execute **all** runtime rules, trusting local signatures, no server validation, behind a loud warning | + +Skipping is never an error and never changes the exit code. This mirrors the stacked-under +degrade philosophy (a not-live endpoint never bricks `check`) while inverting it for ACE: +where static rules degrade to _run-unverified_, runtime rules degrade to _skip_. + +_Alternative rejected:_ run runtime rules on locally-cached signatures when the endpoint is +down. That is exactly what `--dangerously-run-scripts` makes explicit; doing it implicitly +would run unverified code the user never opted into. + +### Decision: The narrow → gate → `check.ts` harness, per TSKL-245 + +For each executable runtime rule: collect its capture rules and run **one** `ast-grep` scan — +`--inline-rules --json=stream` in anchor mode, `--files-with-matches` in broad mode +(`kind: program` enumerators). Normalize each match to +`{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the +hashed `ruleId` back to the model `name` as `match.rule`. If there are **zero** matches, +`check.ts` is not invoked. Otherwise invoke its **default export** as a function with +`(root, matches)` and use the **returned** `Finding[]`; map each `Finding` onto `CheckResult` +with a runtime `source` and feed it into the existing aggregation and error-severity exit-code +logic. A `check.ts` that throws is isolated to a single error-severity finding for that rule — +it never aborts the whole `check` run. Scheduling (process-per-check vs. worker pool vs. +`import()`) is an implementation choice; the function contract leaves it open. + +### Decision: Bundle a pinned `tsx`; execute from the materialized run directory + +Ship a pinned `tsx` (or equivalent loader) with the CLI so `check.ts` runs without any +`node_modules`/toolchain in the user's repo. Execute blessed runtime rules from the ephemeral, +gitignored `.taskless/.run/` (extending the stacked-under materialize step), so the bytes +executed are exactly the reconciled-and-blessed bytes (read-hash-execute ordering), not +whatever is live in `.taskless/rules/` at exec time. + +_Alternative rejected:_ require the user to have `tsx`/`ts-node`. Non-hermetic, version-drift +prone, and breaks the "no toolchain assumptions" posture the rest of the CLI keeps. + +### Decision: Bound `check.ts` with a timeout; `--timeout` overrides + +Each `check.ts` invocation runs under a default wall-clock timeout; a `--timeout ` +flag on `check` overrides it. A check that exceeds the bound is terminated and recorded as a +single error-severity finding for that rule, and the run continues — a runaway or hanging +check never wedges the overall `check`. This is a robustness requirement, not a tuning knob: +runtime rules are third-party code and must be time-bounded by default. + +_Alternative rejected:_ no timeout / rely on the OS. A hung check would block CI indefinitely +with no attributable finding. + +## Risks / Trade-offs + +- **[Running unverified ACE]** → `check.ts` never runs without either a server-validated + signature or an explicit `--dangerously-run-scripts`; the degrade path skips runtime rules + rather than running them. +- **[Capture-file tampering around an authentic `check.ts`]** → accepted: capture `*.yml` are + inert ast-grep patterns and cannot execute code, so they are ungated; a tampered capture can + only change which matches feed an already-blessed `check.ts`, and the enforced runner remains + the authority over rule behavior. +- **[`tsx` startup cost on large corpora]** → the narrow gates first (zero matches ⇒ no + invoke), and startup amortizes across a worker pool; measured ~590ms per worker, ~9–14ms + warm per import. +- **[A slow or hanging `check.ts`]** → the harness owns scheduling and SHOULD bound execution + (timeout → error finding); a runaway check must not wedge `check`. +- **[Silent skips read as "passed"]** → the skipped-runtime notice names the rules skipped and + points at `--dangerously-run-scripts` / authenticated `check` as the way to run them. +- **[Divergence from the enforced runner]** → local findings are advisory-equivalent; the + Taskless `--enforce` sandbox remains the authoritative enforcement point and is out of scope + here. + +## Resolved Questions + +- **`--json` shape for skipped runtime rules:** resolved — add an additive, optional `skipped` + array of `{ rule, reason }` to `--json` output, leaving `success`/`results` unchanged, so CI + can detect that runtime rules did not run. +- **Timeout policy for `check.ts`:** resolved — a default wall-clock bound with a `--timeout` + override; a timeout terminates the check and records an error-severity finding (see the + timeout Decision). +- **`--dangerously-run-scripts` with a resolvable token:** resolved — skip the network + entirely (no reconcile) and execute every present runtime rule, matching how `--anonymous` + forces the no-network path. diff --git a/openspec/changes/runtime-rule-execution/proposal.md b/openspec/changes/runtime-rule-execution/proposal.md new file mode 100644 index 00000000..84918445 --- /dev/null +++ b/openspec/changes/runtime-rule-execution/proposal.md @@ -0,0 +1,83 @@ +## Why + +Server-owned reconciliation (the stacked-under change) decides **which** rule files may run. +This change defines **how a runtime rule executes** — the deliberate follow-up that +reconciliation was built to enable. + +Today every rule is a single ast-grep `*.yml` and `taskless check` has exactly one executor: +`sg scan`. A **runtime rule** is a new class (TSKL-243/245): a directory of one or more +ast-grep capture `*.yml` **plus a `check.ts`** that expresses constraints a single syntactic +pattern cannot — cross-file invariants, import/call graphs, config-vs-code consistency. Its +`check.ts` is **arbitrary code execution**, so the CLI cannot run it the way it runs a +declarative YAML rule. + +The trust model follows from that. Static ast-grep rules are inert data and stay an +offline-linter posture: they **always run**. A runtime rule's `check.ts` executes **only when +a valid server signature says so** — reconciliation is the safe-harness gate. Runtime rules +therefore supersede the earlier "advisory vs. enforced / `--dangerously-run-scripts`" framing: +the default safety mechanism is a server-validated signature, not a local sandbox. + +## What Changes + +- Add a **`cli-runtime-rule-execution`** capability: recognize a runtime-rule directory + (`metadata.taskless.kind: runtime`), run its capture rules as **one** `ast-grep` narrow + (anchor `--inline-rules --json=stream`; broad `--files-with-matches` for `kind: program` + enumerators), **gate on matches**, and only then invoke `check.ts`'s default export with + `(root, matches)` via a CLI-bundled, pinned `tsx`. Zero matches ⇒ `check.ts` is never + invoked. Normalize matches to `{ rule, ruleId, file, line, column, text, captures }` + (hashed `ruleId` mapped back to the model `name`, surfaced as `match.rule`) and map the + returned `Finding[]` (`severity ∈ error|warning|info`) onto the existing scanner-agnostic + `CheckResult`. +- **Make `check.ts` execution conditional on a validated signature.** The harness SHALL invoke + a runtime rule's `check.ts` only when reconciliation returned that rule's `check.ts` in + `run`, or when `--dangerously-run-scripts` is set. On any unverified path (logged out, + `--anonymous`, or a reconcile that cannot complete) runtime rules are **skipped with a + notice** and never executed. +- **Scope reconciliation to each runtime rule's `check.ts`** (refines the stacked-under + change): static ast-grep `*.yml` and runtime-rule capture `*.yml` are inert and are not + gated — they always run/apply. Reconciliation reports and gates only the ACE-bearing + `check.ts`. The stacked-under degrade path ("scan all local rules unverified") is narrowed + so it **never executes runtime `check.ts`**. +- Add a **`--dangerously-run-scripts`** flag to `check`: assume every runtime rule's signature + is valid and execute without server validation, behind a loud warning. +- **Materialize blessed runtime rules** into the ephemeral, gitignored `.taskless/.run/` and + execute from there (read-hash-execute ordering), so the bytes executed are exactly the + reconciled bytes. + +## Capabilities + +### New Capabilities + +- `cli-runtime-rule-execution`: the runtime-rule on-disk shape the CLI recognizes, the + narrow→gate→`check.ts` local harness, the pinned-`tsx` invocation contract, match + normalization, and the `Finding` → `CheckResult` mapping. + +### Modified Capabilities + +- `cli-check`: static rules always run; runtime rules execute only on a signature-validated + path; the five auth/flag modes (authed / logged-out / API-key / `--dangerously-run-scripts` / + `--anonymous`); skipped-runtime reporting; the new `--dangerously-run-scripts` flag; and the + narrowed degrade path. +- `cli-rule-reconciliation`: the reconciled corpus is scoped to each runtime rule's `check.ts`; + static ast-grep rules and capture `*.yml` are inert and are not reported or gated. A runtime + rule is eligible to execute only if its `check.ts` is returned in `run`. + +## Impact + +- **Code:** new `packages/cli/src/rules/runtime/` (directory recognition, narrow assembly, + match normalization, `check.ts` invocation via bundled `tsx`, `Finding`→`CheckResult`); + changes to `src/commands/check.ts` (static-vs-runtime dispatch, the mode table, the + `--dangerously-run-scripts` and `--timeout` flags, skipped-runtime notices); extension of + `src/rules/run-set.ts` (enumerate `.taskless/runtime-rules/`, sign each rule's `check.ts`, + materialize blessed rules) and the reconcile report to carry those `check.ts`. +- **Dependencies:** a pinned `tsx` (or equivalent TypeScript loader) bundled with the CLI so + `check.ts` runs without the user's toolchain. +- **Behavioral shift:** authenticated `check` gains a second executor for runtime rules; + unauthenticated / `--anonymous` `check` skips runtime rules (static behavior unchanged); + `--dangerously-run-scripts` is the explicit local escape hatch. +- **Tests:** runtime-harness unit tests (narrow, gate-on-zero-matches, normalization, + `Finding` mapping, a throwing `check.ts` isolated to an error finding) and integration tests + for each of the five modes against a mock reconcile server. +- **Docs:** `check.txt` gains a runtime-rule section (what runs per mode, the + `--dangerously-run-scripts` warning); `ci.txt` notes the enforced backstop over runtime + rules. diff --git a/openspec/changes/runtime-rule-execution/specs/cli-check/spec.md b/openspec/changes/runtime-rule-execution/specs/cli-check/spec.md new file mode 100644 index 00000000..98f04e54 --- /dev/null +++ b/openspec/changes/runtime-rule-execution/specs/cli-check/spec.md @@ -0,0 +1,167 @@ +## ADDED Requirements + +### Requirement: Check dispatches static and runtime rules to distinct executors + +`taskless check` SHALL execute **static** ast-grep rules under `.taskless/rules/` with the +ast-grep scanner as before, and **runtime** rules under `.taskless/runtime-rules/` (directories +with `metadata.taskless.kind: runtime`, per the `cli-runtime-rule-execution` capability) with +the runtime harness. Findings from both executors SHALL be aggregated into the same result set +and SHALL count toward the exit code identically. + +#### Scenario: Mixed corpus runs both executors + +- **WHEN** `.taskless/rules/` contains static rules and `.taskless/runtime-rules/` contains runtime rules +- **THEN** the CLI SHALL run static rules through `sg scan` and runtime rules through the runtime harness +- **AND** SHALL merge their findings into one result set + +### Requirement: Check runs runtime rules only on a signature-validated path + +`taskless check` SHALL execute a runtime rule's `check.ts` only when that `check.ts` has been +validated by the server — returned in the reconciliation `run` set — or when +`--dangerously-run-scripts` is set. When a token is available and `--anonymous` is not set, the +CLI SHALL reconcile and execute every runtime rule whose `check.ts` is in `run`. An API key +SHALL be treated identically to an interactive token. On any path where the runtime rule's +`check.ts` signature is not validated — logged out, `--anonymous`, or a reconciliation that +cannot complete — the CLI SHALL NOT execute the rule's `check.ts`. + +#### Scenario: Authenticated check runs blessed runtime rules + +- **WHEN** an authenticated `check` reconciles and a runtime rule's `check.ts` is returned in `run` +- **THEN** the CLI SHALL execute that runtime rule through the harness + +#### Scenario: A rule whose check.ts is not blessed is withheld + +- **WHEN** reconciliation does not return a runtime rule's `check.ts` in `run` (it lands in `unsafe`/`unknown`/`missing`) +- **THEN** the CLI SHALL NOT execute that runtime rule +- **AND** SHALL surface it as an advisory mismatch + +#### Scenario: API key behaves like a token + +- **WHEN** `check` runs with an API key +- **THEN** the CLI SHALL reconcile and run validated runtime rules exactly as with an interactive token + +### Requirement: Check skips runtime rules on unverified paths and reports the skip + +`taskless check` SHALL skip a runtime rule's execution when it cannot validate the rule's +signature — logged out, `--anonymous`, or reconciliation cannot complete — and +`--dangerously-run-scripts` is not set. It SHALL report that runtime rules exist and were not +run, SHALL still run static rules on these paths, and SHALL NOT change the exit code because +rules were skipped. In human output the report SHALL be a notice; under `--json` it SHALL be an +additive, optional `skipped` array of `{ rule, reason }`, leaving the existing `success` and +`results` fields unchanged, so machine callers (for example CI) can detect that runtime rules +did not run. + +#### Scenario: Logged-out check skips runtime rules + +- **WHEN** a user runs `taskless check` with no available token and runtime rules are present +- **THEN** the CLI SHALL run static rules +- **AND** SHALL NOT execute any runtime rule's `check.ts` +- **AND** SHALL report that the runtime rules were skipped +- **AND** SHALL NOT change the exit code because rules were skipped + +#### Scenario: Skipped runtime rules appear under --json + +- **WHEN** runtime rules are skipped and `--json` is set +- **THEN** stdout SHALL include a `skipped` array of `{ rule, reason }` alongside the unchanged `success` and `results` fields + +#### Scenario: Anonymous skips runtime rules while authenticated + +- **WHEN** a user runs `taskless check --anonymous` while a token is available +- **THEN** the CLI SHALL skip runtime rules exactly as an unauthenticated `check` + +### Requirement: Check accepts --dangerously-run-scripts to run runtime rules without server validation + +`taskless check` SHALL accept a `--dangerously-run-scripts` flag that runs **all** runtime +rules by trusting their local signatures without server validation, regardless of auth state. +When the flag is set the CLI SHALL NOT reconcile — it SHALL skip the network entirely (matching +how `--anonymous` forces the no-network path) and execute every present runtime rule. The CLI +SHALL emit a prominent warning that runtime rule code is being executed unverified. The flag +SHALL be the only way to execute runtime rules on an unverified path. + +#### Scenario: Dangerously-run-scripts executes runtime rules offline + +- **WHEN** a user runs `taskless check --dangerously-run-scripts` with no available token +- **THEN** the CLI SHALL execute the present runtime rules' `check.ts` +- **AND** SHALL emit a warning that runtime rule code ran unverified + +#### Scenario: Warning is suppressed under --json + +- **WHEN** `--dangerously-run-scripts` and `--json` are both set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** the unverified-execution warning SHALL NOT appear in stdout + +## MODIFIED Requirements + +### Requirement: Check selects what it runs from auth state + +`taskless check` SHALL NOT require authentication, and it SHALL choose what it runs from the +current auth state. **Static ast-grep rules** (single `*.yml` files under `.taskless/rules/`) +SHALL always run without contacting the server, on every path (the offline linter posture). +**Runtime rules** (directories with `metadata.taskless.kind: runtime`) SHALL run only on a +signature-validated path: when a token is available and `--anonymous` is not set the CLI SHALL +reconcile and execute the runtime rules fully returned in `run`; when no token is available, +when `--anonymous` is set, or when reconciliation cannot complete, the CLI SHALL skip runtime +execution unless `--dangerously-run-scripts` is set. The unauthenticated path SHALL succeed +with no network access for static rules and SHALL NOT emit a warning about missing +authentication. + +#### Scenario: Unauthenticated check runs static rules and skips runtime rules + +- **WHEN** a user runs `taskless check` with no available token +- **THEN** the CLI SHALL scan all static rule files +- **AND** SHALL NOT call `POST /cli/api/reconcile` for the purpose of running static rules +- **AND** SHALL skip runtime rules +- **AND** SHALL NOT emit a warning about missing authentication + +#### Scenario: Authenticated check reconciles runtime rules + +- **WHEN** a user runs `taskless check` with an available token and without `--anonymous` +- **THEN** the CLI SHALL reconcile and execute the runtime rules fully returned in `run` +- **AND** SHALL run static rules unconditionally + +#### Scenario: Anonymous forces the logged-out path + +- **WHEN** a user runs `taskless check --anonymous` while a token is available +- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (static rules run, runtime rules skipped, no reconcile call) + +### Requirement: Check reconciles rule files before scanning + +`taskless check` SHALL reconcile before running runtime rules whenever a bearer token and a +`repositoryUrl` are resolvable and `--anonymous` is not set. It SHALL compute the signature +envelope for the `check.ts` of every runtime rule under `.taskless/runtime-rules/`, call +`POST /cli/api/reconcile` with `{ repositoryUrl, files }`, and then execute **only** the +runtime rules whose `check.ts` is returned in the `run` set, matched back to local files by +signature (per the `cli-rule-reconciliation` capability). Capture `*.yml` and static rules +SHALL NOT be gated by reconciliation. + +#### Scenario: Only rules with a blessed check.ts execute + +- **WHEN** a user runs `taskless check` while authenticated and reconciliation returns a `run` + set covering the `check.ts` of some runtime rules and not others +- **THEN** the CLI SHALL execute only the runtime rules whose `check.ts` is in `run` +- **AND** SHALL run all static rules regardless of the `run` set + +### Requirement: Check degrades to a local scan when reconciliation cannot complete + +`taskless check` SHALL NOT fail solely because an attempted reconciliation cannot complete. +When a token is available and `--anonymous` is not set but reconciliation cannot complete (no +resolvable git remote, or the reconcile endpoint is unreachable or not yet deployed, or a +transport error), the CLI SHALL warn that rule verification could not be performed, SHALL fall +back to scanning all **static** rule files, and SHALL **skip runtime rules** (their `check.ts` +SHALL NOT run) unless `--dangerously-run-scripts` is set. The CLI SHALL NOT exit with a +non-zero code solely because reconciliation failed, and the warning SHALL be suppressed under +`--json`. + +#### Scenario: Endpoint unreachable degrades static and skips runtime + +- **WHEN** an authenticated `check` attempts reconciliation and the endpoint is unreachable or returns a not-deployed error +- **THEN** the CLI SHALL warn that verification could not be performed +- **AND** SHALL scan all static rule files +- **AND** SHALL NOT execute any runtime rule's `check.ts` +- **AND** SHALL NOT exit with a non-zero code solely due to the reconcile failure + +#### Scenario: Degrade warning is suppressed under --json + +- **WHEN** the CLI degrades and `--json` is set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** SHALL NOT contain the human-readable degrade warning diff --git a/openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md b/openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md new file mode 100644 index 00000000..7195f909 --- /dev/null +++ b/openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md @@ -0,0 +1,48 @@ +## MODIFIED Requirements + +### Requirement: Reconcile reports every held rule file + +Reconciliation SHALL be scoped to the **`check.ts` of runtime rules** — the only artifact that +carries arbitrary code execution. Static ast-grep rules and runtime-rule capture `*.yml` are +inert data, always available, and SHALL NOT be reported to or gated by reconciliation. The CLI +SHALL reconcile by sending `POST /cli/api/reconcile` with an `Authorization: Bearer ` +header and a JSON body `{ repositoryUrl, files }`, where `repositoryUrl` is the full repository +URL and `files` is an array of `{ file, signature }` covering the `check.ts` of **every** +runtime rule the CLI holds under `.taskless/runtime-rules/`. `file` SHALL be the `check.ts`'s +delivered path as it exists on disk and `signature` SHALL be the full envelope computed for its +bytes. The CLI SHALL send the whole signature envelope, not a bare digest. + +#### Scenario: Every runtime rule's check.ts is reported + +- **WHEN** the CLI reconciles with runtime rules present under `.taskless/runtime-rules/` +- **THEN** the request body SHALL include one `{ file, signature }` entry for the `check.ts` of each runtime rule + +#### Scenario: Inert files are not reported + +- **WHEN** the CLI reconciles and static `*.yml` rules and capture `*.yml` are also present +- **THEN** the request body SHALL NOT include entries for those inert files +- **AND** the static rules SHALL run regardless of the reconcile response + +#### Scenario: Full envelope is sent + +- **WHEN** the CLI reports a file's signature +- **THEN** it SHALL send the complete `1;h=sha-256;d=` envelope, not only the digest + +### Requirement: The CLI executes only the server run set + +When reconciliation succeeds, the CLI SHALL execute a runtime rule only if its **`check.ts`** +is present in the server's `run` set, and SHALL execute no runtime rule whose `check.ts` is +absent from `run`. The CLI SHALL NOT run a runtime rule on the basis of its own local +comparison of signatures. This replaces any local classification of runtime rules. Static +ast-grep rules and capture `*.yml` are outside this gate. + +#### Scenario: Only rules with a blessed check.ts execute + +- **WHEN** reconciliation returns a `run` set covering the `check.ts` of some runtime rules but not others +- **THEN** the CLI SHALL execute only the runtime rules whose `check.ts` is in `run` +- **AND** SHALL withhold any runtime rule whose `check.ts` is absent from `run` + +#### Scenario: No local self-classification + +- **WHEN** the CLI holds a local signature or sidecar for a runtime rule's `check.ts` +- **THEN** it SHALL NOT treat that local value as authorization to execute the rule diff --git a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md b/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md new file mode 100644 index 00000000..112b921f --- /dev/null +++ b/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md @@ -0,0 +1,130 @@ +## ADDED Requirements + +### Requirement: Runtime rules are directories recognized by metadata + +The CLI SHALL recognize a **runtime rule** as a directory under `.taskless/runtime-rules/` +containing one or more ast-grep capture `*.yml` and exactly one `check.ts`, its capture rules +declaring `metadata.taskless.kind: runtime`. The CLI SHALL read `metadata.taskless.check` to +locate the `check.ts` within the directory and `metadata.taskless.match` (`anchor` or `broad`) +to select the ast-grep invocation mode. Rule files under `.taskless/rules/` SHALL continue to +be treated as static ast-grep rules, not runtime rules. `.taskless/runtime-rule-tests/` holds +verification fixtures and SHALL NOT be executed by `check`. + +#### Scenario: A runtime-rules directory entry is a runtime rule + +- **WHEN** `.taskless/runtime-rules//` contains capture `*.yml` with `metadata.taskless.kind: runtime` and a `check.ts` +- **THEN** the CLI SHALL treat it as a runtime rule and route it to the runtime harness + +#### Scenario: Rules under .taskless/rules remain static + +- **WHEN** a rule file lives under `.taskless/rules/` +- **THEN** the CLI SHALL treat it as a static rule and SHALL NOT route it to the runtime harness + +### Requirement: The harness narrows with one ast-grep scan and gates on matches + +For a runtime rule the CLI SHALL assemble the rule's capture rules and run **one** `ast-grep` +scan as the narrow: `--inline-rules --json=stream` in `anchor` mode, and +`--files-with-matches` in `broad` mode (whole-language `kind: program` enumerators). When the +narrow produces **zero** matches the CLI SHALL NOT invoke `check.ts`. + +#### Scenario: Zero matches skips the check + +- **WHEN** a runtime rule's narrow scan produces no matches +- **THEN** the CLI SHALL NOT invoke that rule's `check.ts` +- **AND** the rule SHALL contribute no findings + +#### Scenario: One scan per rule + +- **WHEN** a runtime rule has multiple capture rules +- **THEN** the CLI SHALL run them as a single `ast-grep` scan, not one scan per capture rule + +### Requirement: Matches are normalized and attributed to the model name + +The CLI SHALL normalize every narrow match to +`{ rule, ruleId, file, line, column, text, captures }`, where `file` is root-relative, `line` +is 1-indexed, and `rule` is the capture rule's stable model-assigned `name`. The CLI SHALL map +the hashed capture-rule `id` used by the scan back to that `name` so `match.rule` is the value +the check branches on, never the hash. + +#### Scenario: Hashed id maps to model name + +- **WHEN** the scan emits a match whose rule id is the hashed `${ruleSlug}-${sha1}` identifier +- **THEN** the normalized match's `rule` SHALL be the capture rule's model-assigned `name` + +### Requirement: The check is invoked as a function and its return value is used + +The CLI SHALL invoke a runtime rule's `check.ts` by calling its **default export** as a +function with `(root, matches)`, where `root` is the repository root and `matches` are the +normalized matches. The CLI SHALL use the `Finding[]` value the function **returns** as the +rule's result; it SHALL NOT infer results from process exit codes or stdout. A `check.ts` that +throws SHALL be isolated to a single error-severity finding for that rule and SHALL NOT abort +the overall `check` run. + +#### Scenario: Returned findings are the result + +- **WHEN** a runtime rule's `check.ts` default export returns a `Finding[]` +- **THEN** the CLI SHALL treat exactly those findings as the rule's output + +#### Scenario: A throwing check is isolated + +- **WHEN** a runtime rule's `check.ts` throws during execution +- **THEN** the CLI SHALL record a single error-severity finding for that rule +- **AND** SHALL continue executing the remaining rules and produce output + +### Requirement: check.ts execution is bounded by a timeout + +The CLI SHALL bound each `check.ts` invocation with a wall-clock timeout and SHALL accept a +`--timeout ` flag on `check` to override the default. When a `check.ts` exceeds the +timeout the CLI SHALL terminate it, record a single error-severity finding for that rule, and +continue executing the remaining rules — a runaway check SHALL NOT wedge the overall `check` +run. + +#### Scenario: A hanging check is terminated at the timeout + +- **WHEN** a runtime rule's `check.ts` runs longer than the effective timeout +- **THEN** the CLI SHALL terminate it and record a single error-severity finding for that rule +- **AND** SHALL continue executing the remaining rules + +#### Scenario: --timeout overrides the default + +- **WHEN** a user runs `taskless check --timeout ` +- **THEN** the CLI SHALL use that value as the per-check wall-clock bound + +### Requirement: check.ts runs via a bundled TypeScript loader + +The CLI SHALL execute `check.ts` using a pinned TypeScript loader bundled with the CLI (e.g. +`tsx`) and SHALL NOT require the user's repository to provide a TypeScript toolchain, +`node_modules`, or a precompile step. The CLI MAY schedule invocations by any mechanism +(process-per-check, worker pool, or in-process import); the function contract does not +constrain the choice. + +#### Scenario: No user toolchain required + +- **WHEN** a repository with a runtime rule has no local TypeScript toolchain installed +- **THEN** the CLI SHALL still execute the rule's `check.ts` using its bundled loader + +### Requirement: Findings map onto the scanner-agnostic result type + +The CLI SHALL map each `Finding` returned by a `check.ts` onto the existing `CheckResult` +shape with a runtime `source`, so runtime findings are aggregated, formatted, and counted +toward the exit code identically to static findings. `Finding.severity` (`error` / `warning` / +`info`) SHALL map directly onto the corresponding `CheckResult` severity with no translation. + +#### Scenario: Runtime findings gate the exit code like static findings + +- **WHEN** a runtime rule returns a finding with `severity: "error"` +- **THEN** the CLI SHALL count it toward the error total that sets a non-zero exit code +- **AND** the finding SHALL appear in `--json` output under the same `results` shape as a static finding + +### Requirement: Blessed runtime rules execute from the materialized run directory + +When a runtime rule is executed on a validated path, the CLI SHALL execute it from the +ephemeral, gitignored `.taskless/.run/` materialization of the blessed bytes, not from the +live `.taskless/rules/` tree, so the bytes executed are the exact bytes reconciliation blessed +(read-hash-execute ordering). + +#### Scenario: Execution uses the blessed bytes + +- **WHEN** a runtime rule is blessed and executed +- **THEN** the CLI SHALL invoke the `check.ts` materialized under `.taskless/.run/` +- **AND** SHALL NOT execute a copy modified in `.taskless/rules/` after reconciliation diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/runtime-rule-execution/tasks.md new file mode 100644 index 00000000..c48ac578 --- /dev/null +++ b/openspec/changes/runtime-rule-execution/tasks.md @@ -0,0 +1,45 @@ +## 1. Runtime-rule recognition + +- [ ] 1.1 Add `packages/cli/src/rules/runtime/discover.ts`: enumerate `.taskless/runtime-rules/` for rule directories, parse each capture `*.yml`'s `metadata.taskless` (`kind`, `name`, `check`, `match`), and confirm the class via `kind: runtime`. Return a typed `RuntimeRule` (`{ dir, captureFiles, checkFile, match }`). `.taskless/runtime-rule-tests/` is not enumerated for execution. +- [ ] 1.2 Static rules stay sourced from `.taskless/rules/`; runtime rules from `.taskless/runtime-rules/` — location is the class split. + +## 2. Narrow → gate → check harness + +- [ ] 2.1 Add `packages/cli/src/rules/runtime/narrow.ts`: assemble a rule's capture rules and run ONE `ast-grep` scan — anchor mode `--inline-rules --json=stream`, broad mode `--files-with-matches` (`kind: program`). Reuse the existing scan plumbing where possible. +- [ ] 2.2 Add match normalization to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed capture `id` back to the model `name` surfaced as `match.rule`. +- [ ] 2.3 Gate on matches: when the narrow yields zero matches, do NOT invoke `check.ts`. +- [ ] 2.4 Add `packages/cli/src/rules/runtime/invoke.ts`: call `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`; use the returned `Finding[]`. Isolate a throwing check to a single error-severity finding for that rule. +- [ ] 2.5 Decide and implement scheduling (process-per-check vs. `worker_threads` pool vs. `import()`); bound each check with a default wall-clock timeout (overridable via `--timeout `) that terminates the check and records a single error-severity finding. +- [ ] 2.6 Map each `Finding` (`severity ∈ error|warning|info`) onto `CheckResult` with a runtime `source`; feed into the existing aggregation and exit-code logic. + +## 3. tsx bundling + +- [ ] 3.1 Add a pinned `tsx` (or equivalent loader) to the CLI bundle and resolve its path at runtime without assuming any repo-local toolchain. +- [ ] 3.2 Verify `check.ts` executes with no `node_modules` and no precompile in a temp-dir fixture. + +## 4. Reconcile scoping & materialization + +- [ ] 4.1 Extend `src/rules/run-set.ts` to enumerate `.taskless/runtime-rules/` and sign each rule's `check.ts` only (capture `*.yml` and static rules are inert — not reported to reconcile). +- [ ] 4.2 Compute per-rule eligibility: a runtime rule executes only if its `check.ts` is in `run`; otherwise withhold and surface as advisory. +- [ ] 4.3 Materialize blessed runtime rules into `.taskless/.run/` and execute `check.ts` from there (read-hash-execute); ensure `.taskless/.gitignore` still covers `.run/`. + +## 5. check dispatch & modes + +- [ ] 5.1 In `src/commands/check.ts`, split the corpus via `classifyRules`; always run static rules; route runtime rules through the harness only on a validated path. +- [ ] 5.2 Implement the mode table: authed/API-key → reconcile & run fully-blessed runtime rules; logged-out/`--anonymous` → skip runtime + report skipped; reconcile-cannot-complete → skip runtime with notice; `--dangerously-run-scripts` → run all runtime rules trusting local signatures. +- [ ] 5.3 Add the `--dangerously-run-scripts` flag: skip reconciliation entirely (no network) and run all present runtime rules, behind a prominent unverified-execution warning (stderr, suppressed under `--json`). Add the `--timeout ` flag. +- [ ] 5.4 Narrow the stacked-under degrade path so it scans static rules but never executes runtime `check.ts`. +- [ ] 5.5 Emit skipped-runtime notices (human output); under `--json` add an additive optional `skipped: [{ rule, reason }]` field leaving `success`/`results` unchanged. Confirm the exit code stays governed solely by error-severity findings. + +## 6. Help & docs + +- [ ] 6.1 Update `packages/cli/src/help/check.txt` with a runtime-rule section: what runs per mode, and the `--dangerously-run-scripts` warning. +- [ ] 6.2 Update `packages/cli/src/help/ci.txt` to note the enforced (`--enforce`) sandbox as the authoritative runtime-rule enforcement point. + +## 7. Tests & verification + +- [ ] 7.1 Harness unit tests: narrow assembly (anchor/broad), gate-on-zero-matches (check never invoked), match normalization (hashed id → model name), `Finding` → `CheckResult` mapping, throwing-check isolation, timeout → error finding. +- [ ] 7.2 Mode integration tests (subprocess against `dist/`, temp-dir fixtures with a mock reconcile server + git origin): authed runs blessed runtime rules; partially-blessed rule withheld; logged-out and `--anonymous` skip runtime + report skipped; reconcile-unavailable skips runtime; `--dangerously-run-scripts` runs runtime offline with a warning. +- [ ] 7.3 Static-always-run tests: static rules run in every mode; static rules are not reported to reconcile. +- [ ] 7.4 `--json` shape tests: warnings/notices suppressed; runtime findings appear under the same `results` shape as static findings. +- [ ] 7.5 Run `pnpm typecheck`, `pnpm lint`, and the full `pnpm test` — all green. From 537345ac7e1acbe07a7a64ffc7e07825e509386c Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 08:02:02 -0700 Subject: [PATCH 09/21] docs(openspec): confirm runtime rule layout against generator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified against workers/generator/src/actions/add-runtime-rule.ts: runtime rules are written to .taskless/runtime-rules/-/ (one .yml per capture rule + a check.ts), with fixtures under .taskless/runtime-rule-tests/. The check file is always check.ts, and its bytes are hashed with the same canonicalHash envelope reconcile uses — so gating on check.ts matches what the server captures. Tighten the recognition spec (drop the metadata.taskless.check locator claim; check.ts is fixed) and record the confirmation in design.md. Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../changes/runtime-rule-execution/design.md | 15 ++++++++++----- .../specs/cli-runtime-rule-execution/spec.md | 17 +++++++++-------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/openspec/changes/runtime-rule-execution/design.md b/openspec/changes/runtime-rule-execution/design.md index 18d9aa31..4c31d10c 100644 --- a/openspec/changes/runtime-rule-execution/design.md +++ b/openspec/changes/runtime-rule-execution/design.md @@ -60,11 +60,16 @@ Runtime rules live under `.taskless/runtime-rules/` — each rule a directory ho `*.yml` and `check.ts` — with fixtures under `.taskless/runtime-rule-tests/`. Static ast-grep rules stay under `.taskless/rules/`. **Location is the primary rule-class split**: `check` scans `.taskless/rules/` for static rules and `.taskless/runtime-rules/` for runtime rules, -and `metadata.taskless.kind: runtime` confirms the class. The CLI reads -`metadata.taskless.check` to locate the directory's `check.ts` and `metadata.taskless.match` -to pick the ast-grep invocation mode, and never parses rule intent beyond this metadata -envelope and the ast-grep config it already understands. `.taskless/runtime-rule-tests/` holds -verification fixtures and is not executed by `check`. +and `metadata.taskless.kind: runtime` confirms the class. The check file is always `check.ts` +in the rule directory; the CLI reads `metadata.taskless.match` to pick the ast-grep invocation +mode, and never parses rule intent beyond this metadata envelope and the ast-grep config it +already understands. `.taskless/runtime-rule-tests//` holds `valid/`/`invalid/` fixtures +and is not executed by `check`. + +This layout is confirmed against the generator +(`workers/generator/src/actions/add-runtime-rule.ts`), which writes +`.taskless/runtime-rules/-/` with `.yml` per capture rule and a +`check.ts`, and hashes `check.ts` with the same `canonicalHash` envelope reconcile uses. _Alternative rejected:_ co-locating runtime rules under `.taskless/rules/` and splitting on directory-vs-file. The separate tree is what the generator writes, and a distinct path removes diff --git a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md b/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md index 112b921f..000566cc 100644 --- a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md +++ b/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md @@ -2,18 +2,19 @@ ### Requirement: Runtime rules are directories recognized by metadata -The CLI SHALL recognize a **runtime rule** as a directory under `.taskless/runtime-rules/` -containing one or more ast-grep capture `*.yml` and exactly one `check.ts`, its capture rules -declaring `metadata.taskless.kind: runtime`. The CLI SHALL read `metadata.taskless.check` to -locate the `check.ts` within the directory and `metadata.taskless.match` (`anchor` or `broad`) -to select the ast-grep invocation mode. Rule files under `.taskless/rules/` SHALL continue to -be treated as static ast-grep rules, not runtime rules. `.taskless/runtime-rule-tests/` holds -verification fixtures and SHALL NOT be executed by `check`. +The CLI SHALL recognize a **runtime rule** as a directory under `.taskless/runtime-rules//` +containing one or more ast-grep capture `*.yml` (one per capture rule) and a single `check.ts`, +its capture rules declaring `metadata.taskless.kind: runtime`. The rule's check file SHALL be +the `check.ts` in the rule directory. The CLI SHALL read `metadata.taskless.match` (`anchor` or +`broad`) to select the ast-grep invocation mode. Rule files under `.taskless/rules/` SHALL +continue to be treated as static ast-grep rules, not runtime rules. +`.taskless/runtime-rule-tests//` holds `valid/` and `invalid/` verification fixtures and +SHALL NOT be executed by `check`. #### Scenario: A runtime-rules directory entry is a runtime rule - **WHEN** `.taskless/runtime-rules//` contains capture `*.yml` with `metadata.taskless.kind: runtime` and a `check.ts` -- **THEN** the CLI SHALL treat it as a runtime rule and route it to the runtime harness +- **THEN** the CLI SHALL treat it as a runtime rule with `check.ts` as its check file and route it to the runtime harness #### Scenario: Rules under .taskless/rules remain static From fa9e836f3195d5a4130dc728087a40a3f7010848 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 08:35:00 -0700 Subject: [PATCH 10/21] feat(cli): discover runtime rules under .taskless/runtime-rules Group 1 of runtime-rule-execution. Add the harness-side recognition of runtime rules: a directory under .taskless/runtime-rules/ holding kind: runtime capture *.yml plus a check.ts. - src/types/runtime-rule.ts mirrors the structural harness<->check contract (Finding, Match, CheckFunction, CaptureRule, metadata block) from the generator's @taskless/types; a delivered check imports nothing, so the contract is structural. - src/rules/runtime/discover.ts enumerates .taskless/runtime-rules/, parses each capture *.yml, confirms kind: runtime, and returns a typed RuntimeRule (capture rules with id/name/language/match + the check.ts path). .taskless/runtime-rule-tests/ is never enumerated. Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../changes/runtime-rule-execution/tasks.md | 4 +- packages/cli/src/rules/runtime/discover.ts | 127 ++++++++++++++++++ packages/cli/src/types/runtime-rule.ts | 112 +++++++++++++++ 3 files changed, 241 insertions(+), 2 deletions(-) create mode 100644 packages/cli/src/rules/runtime/discover.ts create mode 100644 packages/cli/src/types/runtime-rule.ts diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/runtime-rule-execution/tasks.md index c48ac578..fc850be0 100644 --- a/openspec/changes/runtime-rule-execution/tasks.md +++ b/openspec/changes/runtime-rule-execution/tasks.md @@ -1,7 +1,7 @@ ## 1. Runtime-rule recognition -- [ ] 1.1 Add `packages/cli/src/rules/runtime/discover.ts`: enumerate `.taskless/runtime-rules/` for rule directories, parse each capture `*.yml`'s `metadata.taskless` (`kind`, `name`, `check`, `match`), and confirm the class via `kind: runtime`. Return a typed `RuntimeRule` (`{ dir, captureFiles, checkFile, match }`). `.taskless/runtime-rule-tests/` is not enumerated for execution. -- [ ] 1.2 Static rules stay sourced from `.taskless/rules/`; runtime rules from `.taskless/runtime-rules/` — location is the class split. +- [x] 1.1 Add `packages/cli/src/rules/runtime/discover.ts`: enumerate `.taskless/runtime-rules/` for rule directories, parse each capture `*.yml`'s `metadata.taskless` (`kind`, `name`, `check`, `match`), and confirm the class via `kind: runtime`. Return a typed `RuntimeRule` (`{ dir, captureFiles, checkFile, match }`). `.taskless/runtime-rule-tests/` is not enumerated for execution. (Also mirrored the harness↔check contract types into `src/types/runtime-rule.ts`.) +- [x] 1.2 Static rules stay sourced from `.taskless/rules/`; runtime rules from `.taskless/runtime-rules/` — location is the class split. ## 2. Narrow → gate → check harness diff --git a/packages/cli/src/rules/runtime/discover.ts b/packages/cli/src/rules/runtime/discover.ts new file mode 100644 index 00000000..729d8833 --- /dev/null +++ b/packages/cli/src/rules/runtime/discover.ts @@ -0,0 +1,127 @@ +import { readdir, readFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { parse } from "yaml"; + +import type { CaptureRule, MatchMode } from "../../types/runtime-rule"; + +/** Directory (relative to `.taskless/`) that holds runtime rules. */ +export const RUNTIME_RULES_DIR = "runtime-rules"; + +/** A parsed capture `*.yml` of a runtime rule, with the fields the harness needs. */ +export interface LoadedCaptureRule { + /** Absolute path to the capture `*.yml`. */ + file: string; + /** Basename of the capture file. */ + fileName: string; + /** Baked-in ast-grep rule id (used to attribute scan matches back to `name`). */ + id: string; + /** Stable, model-assigned name (`metadata.taskless.name`), surfaced as `match.rule`. */ + name: string; + /** ast-grep `language`. */ + language: string; + /** Scan mode; `anchor` when omitted. */ + match: MatchMode; + /** The full parsed capture rule (for running the narrow). */ + rule: CaptureRule; +} + +/** A discovered runtime rule directory under `.taskless/runtime-rules/`. */ +export interface RuntimeRule { + /** Rule directory basename (e.g. `no-default-export-abc12345`). */ + name: string; + /** Absolute path to the rule directory. */ + dir: string; + /** The rule's parsed capture rules, in filename order. */ + captureRules: LoadedCaptureRule[]; + /** Absolute path to the rule's `check.ts`. */ + checkFile: string; +} + +/** Narrow an unknown parsed YAML value to a runtime `CaptureRule`, or return null. */ +function asRuntimeCaptureRule(value: unknown): CaptureRule | null { + if (typeof value !== "object" || value === null) return null; + const candidate = value as Partial; + const taskless = candidate.metadata?.taskless; + if (!taskless || taskless.kind !== "runtime") return null; + if (typeof candidate.language !== "string") return null; + if (typeof taskless.name !== "string") return null; + return candidate as CaptureRule; +} + +/** Load and parse the capture rules of a single runtime-rule directory. */ +async function loadCaptureRules( + directory: string +): Promise { + let entries: string[]; + try { + entries = await readdir(directory); + } catch { + return []; + } + const ymlFiles = entries.filter( + (f) => f.endsWith(".yml") || f.endsWith(".yaml") + ); + const loaded: LoadedCaptureRule[] = []; + for (const fileName of ymlFiles.toSorted()) { + const file = join(directory, fileName); + let parsed: unknown; + try { + parsed = parse(await readFile(file, "utf8")); + } catch { + continue; // not valid YAML — skip + } + const rule = asRuntimeCaptureRule(parsed); + if (!rule || typeof rule.id !== "string") continue; + loaded.push({ + file, + fileName, + id: rule.id, + name: rule.metadata.taskless.name, + language: rule.language, + match: rule.metadata.taskless.match ?? "anchor", + rule, + }); + } + return loaded; +} + +/** + * Enumerate `.taskless/runtime-rules/` and return each rule directory that holds + * at least one `kind: runtime` capture rule. `.taskless/runtime-rule-tests/` is + * never enumerated — it holds verification fixtures, not executable rules. + */ +export async function discoverRuntimeRules( + cwd: string +): Promise { + const root = join(cwd, ".taskless", RUNTIME_RULES_DIR); + let directoryEntries; + try { + directoryEntries = await readdir(root, { withFileTypes: true }); + } catch { + return []; // no runtime-rules directory + } + + const rules: RuntimeRule[] = []; + const sorted = directoryEntries.toSorted((a, b) => + a.name.localeCompare(b.name) + ); + for (const entry of sorted) { + if (!entry.isDirectory()) continue; + const directory = join(root, entry.name); + const captureRules = await loadCaptureRules(directory); + if (captureRules.length === 0) continue; // not a runtime rule + + // Every capture rule of a runtime rule names the same `check.ts`; the + // generator always writes `check.ts`, so fall back to that. + const checkName = + captureRules[0]!.rule.metadata.taskless.check || "check.ts"; + rules.push({ + name: entry.name, + dir: directory, + captureRules, + checkFile: join(directory, checkName), + }); + } + return rules; +} diff --git a/packages/cli/src/types/runtime-rule.ts b/packages/cli/src/types/runtime-rule.ts new file mode 100644 index 00000000..fc61bf09 --- /dev/null +++ b/packages/cli/src/types/runtime-rule.ts @@ -0,0 +1,112 @@ +/** + * Runtime rule file format + harness↔check contract — mirrored structurally from + * `@taskless/types` (workers/generator). A runtime rule is a DIRECTORY under + * `.taskless/runtime-rules//`: one or more ast-grep capture `*.yml` plus + * exactly one `check.ts`. The capture rules are the cheap syntactic narrow; the + * `check.ts` refines only where the narrow matched. + * + * These types are kept structurally identical to what a generated `check.ts` + * declares inline: a delivered check imports NOTHING from `@taskless/*`, so the + * contract is structural, not an import. They exist here for the CLI's own + * harness code (discovery, narrow, invocation). + */ + +/** Harness↔check protocol version; bumped only on a breaking `(root, matches)` change. */ +export const RUNTIME_CHECK_PROTOCOL_VERSION = 1; + +/** + * Finding severity — a subset of ast-grep's enum. `error` blocks (non-zero + * result); `warning` and `info` are advisory. An omitted value is treated as + * `warning`. + */ +export type FindingSeverity = "error" | "warning" | "info"; + +/** A single result a check returns. `file` is repo-root-relative. */ +export interface Finding { + /** Repo-root-relative path the finding is about. */ + file: string; + /** 1-indexed line, when the finding is line-scoped. */ + line?: number; + /** 1-indexed column, when the finding is column-scoped. */ + column?: number; + /** Human-readable description of the finding. */ + message: string; + /** Per-finding severity; an omitted value is treated as `warning`. */ + severity?: FindingSeverity; +} + +/** + * One normalized ast-grep match handed to a check. The check branches on `rule` + * (the stable, model-assigned name), NEVER on `ruleId` (the baked-in hash). + */ +export interface Match { + /** Stable, model-assigned capture-rule name to branch on (e.g. `exports`). */ + rule: string; + /** Baked-in, globally-unique hashed id; opaque to the check. */ + ruleId: string; + /** Path RELATIVE to `root`; `path.join(root, file)` to read the file. */ + file: string; + /** 1-indexed line (ast-grep reports 0-indexed; the harness normalizes). */ + line: number; + /** 1-indexed column (ast-grep reports 0-indexed; the harness normalizes). */ + column: number; + /** The matched source text. */ + text: string; + /** Captured metavariables by name; empty for a `broad` enumerator match. */ + captures: Record; +} + +/** + * The harness↔check contract: a check module's default export is an async + * function taking the repo `root` and the narrow's `matches`, reading any repo + * file it needs from disk under `root`, and RETURNING its findings. + */ +export type CheckFunction = ( + root: string, + matches: Match[] +) => Promise; + +/** + * Scan mode for a capture rule. `anchor` (the default) is the syntactic narrow + * (`ast-grep scan --json=stream`, matches carry `captures`); `broad` is a + * whole-language enumerator (`rule: { kind: program }` + `--files-with-matches`, + * paths only, empty `captures`). + */ +export type MatchMode = "anchor" | "broad"; + +/** The `metadata.taskless` block folded into every capture rule's YAML. */ +export interface RuntimeRuleMetadata { + /** Metadata schema version for the runtime rule format. */ + version: number; + /** Discriminates a runtime capture rule from a static rule. */ + kind: "runtime"; + /** Stable, model-assigned capture-rule name (surfaced on matches as `rule`). */ + name: string; + /** Filename of the `check.ts` this capture rule pairs with. */ + check: string; + /** Scan mode; treated as `anchor` when omitted. */ + match?: MatchMode; +} + +/** One on-disk ast-grep capture `*.yml` of a runtime rule. */ +export interface CaptureRule { + /** Baked-in, globally-unique ast-grep rule id (`${ruleSlug}-${sha1}`). */ + id?: string; + /** ast-grep `language` — scopes which files this rule parses. */ + language: string; + /** The ast-grep matcher. */ + rule: Record; + /** Optional ast-grep `constraints` — a SIBLING of `rule`, not nested. */ + constraints?: Record; + /** Optional ast-grep `utils` — a sibling of `rule`. */ + utils?: Record; + /** Optional ast-grep `transform` — a sibling of `rule`. */ + transform?: Record; + /** Taskless metadata folded into the YAML. */ + metadata: { + /** The runtime rule's self-describing Taskless block. */ + taskless: RuntimeRuleMetadata; + /** Other metadata keys are permitted alongside. */ + [key: string]: unknown; + }; +} From 01bb83198cfc6228242f60b2808ae5dd022be767 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 09:06:01 -0700 Subject: [PATCH 11/21] feat(cli): add the runtime-rule narrow -> gate -> check harness Groups 2-3 of runtime-rule-execution. Implement the local harness that evaluates a runtime rule, plus the bundled tsx loader it runs check.ts under. - narrow.ts: run a rule's capture rules as ONE ast-grep scan (anchor --json=stream, broad --files-with-matches) and normalize matches to the contract shape (0-indexed -> 1-indexed, ruleId -> model name, captures from metaVariables). Uses a temp --config rules dir so multiple captures + full ast-grep config run in a single scan. - invoke.ts: run check.ts's default export (root, matches) via a pinned tsx resolved at runtime (no repo toolchain). An embedded ESM runner writes the returned Finding[] to an out-file; a throw, non-zero exit, or timeout is isolated to an error result. Default 10s bound, overridable. - harness.ts: narrow -> gate-on-matches -> invoke -> map Finding to CheckResult (source: taskless-runtime); process-per-check, sequential. - Add tsx to the CLI dependencies (externalized from the Vite bundle). Verified end-to-end against a temp-dir fixture: discovery, narrow (correct line normalization + rule attribution), tsx invocation, and finding mapping. Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../changes/runtime-rule-execution/tasks.md | 16 +- packages/cli/package.json | 1 + packages/cli/src/rules/runtime/harness.ts | 108 ++++++++++++ packages/cli/src/rules/runtime/invoke.ts | 151 ++++++++++++++++ packages/cli/src/rules/runtime/narrow.ts | 164 ++++++++++++++++++ pnpm-lock.yaml | 3 + 6 files changed, 435 insertions(+), 8 deletions(-) create mode 100644 packages/cli/src/rules/runtime/harness.ts create mode 100644 packages/cli/src/rules/runtime/invoke.ts create mode 100644 packages/cli/src/rules/runtime/narrow.ts diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/runtime-rule-execution/tasks.md index fc850be0..5df454dd 100644 --- a/openspec/changes/runtime-rule-execution/tasks.md +++ b/openspec/changes/runtime-rule-execution/tasks.md @@ -5,17 +5,17 @@ ## 2. Narrow → gate → check harness -- [ ] 2.1 Add `packages/cli/src/rules/runtime/narrow.ts`: assemble a rule's capture rules and run ONE `ast-grep` scan — anchor mode `--inline-rules --json=stream`, broad mode `--files-with-matches` (`kind: program`). Reuse the existing scan plumbing where possible. -- [ ] 2.2 Add match normalization to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed capture `id` back to the model `name` surfaced as `match.rule`. -- [ ] 2.3 Gate on matches: when the narrow yields zero matches, do NOT invoke `check.ts`. -- [ ] 2.4 Add `packages/cli/src/rules/runtime/invoke.ts`: call `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`; use the returned `Finding[]`. Isolate a throwing check to a single error-severity finding for that rule. -- [ ] 2.5 Decide and implement scheduling (process-per-check vs. `worker_threads` pool vs. `import()`); bound each check with a default wall-clock timeout (overridable via `--timeout `) that terminates the check and records a single error-severity finding. -- [ ] 2.6 Map each `Finding` (`severity ∈ error|warning|info`) onto `CheckResult` with a runtime `source`; feed into the existing aggregation and exit-code logic. +- [x] 2.1 Add `packages/cli/src/rules/runtime/narrow.ts`: assemble a rule's capture rules and run ONE `ast-grep` scan — anchor mode `--json=stream`, broad mode `--files-with-matches` (`kind: program`). (Used a temp `--config` rules dir rather than `--inline-rules` so multiple capture rules + full ast-grep config run in a single scan; reuses `findSgBinary`/`buildPath` from `scan.ts`.) +- [x] 2.2 Add match normalization to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed capture `id` back to the model `name` surfaced as `match.rule`. (ast-grep 0-indexed → 1-indexed; captures from `metaVariables.single`.) +- [x] 2.3 Gate on matches: when the narrow yields zero matches, do NOT invoke `check.ts`. +- [x] 2.4 Add `packages/cli/src/rules/runtime/invoke.ts`: call `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`; use the returned `Finding[]`. Isolate a throwing check to a single error-severity finding for that rule. (Runs an embedded ESM runner under `tsx`; findings via an out-file so the check's stdout can't pollute the channel.) +- [x] 2.5 Decide and implement scheduling: process-per-check (each invoke spawns a `tsx` process), rules run sequentially; bound each check with a default wall-clock timeout (`DEFAULT_CHECK_TIMEOUT_MS`, overridable via `--timeout`) that SIGKILLs the check and records a single error-severity finding. +- [x] 2.6 Map each `Finding` (`severity ∈ error|warning|info`, omitted → warning) onto `CheckResult` with `source: taskless-runtime` (`src/rules/runtime/harness.ts`); feeds the existing aggregation and exit-code logic. ## 3. tsx bundling -- [ ] 3.1 Add a pinned `tsx` (or equivalent loader) to the CLI bundle and resolve its path at runtime without assuming any repo-local toolchain. -- [ ] 3.2 Verify `check.ts` executes with no `node_modules` and no precompile in a temp-dir fixture. +- [x] 3.1 Add a pinned `tsx` to the CLI `dependencies` and resolve its bin at runtime via `createRequire`/`tsx/package.json` (no repo-local toolchain assumed); externalized from the Vite bundle (spawned as a subprocess). +- [x] 3.2 Verify `check.ts` executes with no `node_modules` and no precompile — smoke-tested against a temp-dir fixture (discovery → narrow → `tsx` invoke → finding); automated coverage lands in Group 7. ## 4. Reconcile scoping & materialization diff --git a/packages/cli/package.json b/packages/cli/package.json index 9add16a3..e54dd4ac 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -42,6 +42,7 @@ "posthog-node": "^5.28.11", "smol-toml": "^1.6.1", "sprintf-js": "^1.1.3", + "tsx": "^4.21.0", "yaml": "^2.8.2", "zod": "^4.3.6" }, diff --git a/packages/cli/src/rules/runtime/harness.ts b/packages/cli/src/rules/runtime/harness.ts new file mode 100644 index 00000000..14105793 --- /dev/null +++ b/packages/cli/src/rules/runtime/harness.ts @@ -0,0 +1,108 @@ +import { relative } from "node:path"; + +import type { CheckResult } from "../../types/check"; +import type { Finding } from "../../types/runtime-rule"; +import type { RuntimeRule } from "./discover"; +import { runNarrow } from "./narrow"; +import { DEFAULT_CHECK_TIMEOUT_MS, invokeCheck } from "./invoke"; + +/** Scanner-agnostic `source` label for runtime-rule findings. */ +export const RUNTIME_SOURCE = "taskless-runtime"; + +/** Options controlling a runtime-rule run. */ +export interface RuntimeRunOptions { + /** Restrict the narrow to these paths (diff scope); empty scans the repo. */ + paths?: string[]; + /** Per-check wall-clock bound in ms. */ + timeoutMs?: number; +} + +/** Map a check `Finding` onto the scanner-agnostic `CheckResult`. */ +function findingToCheckResult( + rule: RuntimeRule, + finding: Finding +): CheckResult { + const line = finding.line ?? 0; + const column = finding.column ?? 0; + return { + source: RUNTIME_SOURCE, + ruleId: rule.name, + severity: finding.severity ?? "warning", + message: finding.message, + file: finding.file, + range: { start: { line, column }, end: { line, column } }, + matchedText: "", + }; +} + +/** A harness failure (throw / timeout / bad output) becomes one error finding. */ +function harnessErrorResult( + root: string, + rule: RuntimeRule, + message: string +): CheckResult { + return { + source: RUNTIME_SOURCE, + ruleId: rule.name, + severity: "error", + message: `runtime rule ${rule.name} failed: ${message}`, + file: relative(root, rule.checkFile), + range: { start: { line: 0, column: 0 }, end: { line: 0, column: 0 } }, + matchedText: "", + }; +} + +/** + * Execute one runtime rule: run the ast-grep narrow, gate on matches (zero + * matches ⇒ `check.ts` is never invoked), invoke `check.ts`, and map its + * findings onto `CheckResult`. A harness failure is isolated to a single + * error-severity finding and never throws. + */ +export async function executeRuntimeRule( + root: string, + rule: RuntimeRule, + options: RuntimeRunOptions = {} +): Promise { + let matches; + try { + matches = await runNarrow(root, rule, options.paths ?? []); + } catch (error) { + return [ + harnessErrorResult( + root, + rule, + `narrow failed: ${error instanceof Error ? error.message : String(error)}` + ), + ]; + } + + if (matches.length === 0) return []; // gate: no matches, no check + + const result = await invokeCheck( + rule.checkFile, + root, + matches, + options.timeoutMs ?? DEFAULT_CHECK_TIMEOUT_MS + ); + if (result.status === "error") { + return [harnessErrorResult(root, rule, result.message)]; + } + return result.findings.map((finding) => findingToCheckResult(rule, finding)); +} + +/** + * Execute runtime rules in sequence (process-per-check scheduling) and return + * the aggregated findings. Sequential keeps `tsx` worker startup predictable; + * the function contract leaves room for a pool later. + */ +export async function executeRuntimeRules( + root: string, + rules: RuntimeRule[], + options: RuntimeRunOptions = {} +): Promise { + const results: CheckResult[] = []; + for (const rule of rules) { + results.push(...(await executeRuntimeRule(root, rule, options))); + } + return results; +} diff --git a/packages/cli/src/rules/runtime/invoke.ts b/packages/cli/src/rules/runtime/invoke.ts new file mode 100644 index 00000000..f44b6d09 --- /dev/null +++ b/packages/cli/src/rules/runtime/invoke.ts @@ -0,0 +1,151 @@ +import { spawn } from "node:child_process"; +import { createRequire } from "node:module"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; + +import type { Finding, Match } from "../../types/runtime-rule"; + +/** Default per-check wall-clock bound (ms); overridable via `--timeout`. */ +export const DEFAULT_CHECK_TIMEOUT_MS = 10_000; + +/** Outcome of invoking a single `check.ts`. */ +export type InvokeResult = + | { status: "ok"; findings: Finding[] } + | { status: "error"; message: string }; + +/** + * The self-contained ESM runner executed under `tsx`. It imports the delivered + * `check.ts` (which `tsx` transpiles), calls its default export with + * `(root, matches)`, and writes the returned findings to `outPath` — so the + * check's own stdout/stderr never pollutes the result channel. + */ +const RUNNER_SOURCE = `import { readFileSync, writeFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +const [checkFile, root, matchesPath, outPath] = process.argv.slice(2); +try { + const matches = JSON.parse(readFileSync(matchesPath, "utf8")); + const mod = await import(pathToFileURL(checkFile).href); + const check = mod.default; + if (typeof check !== "function") { + throw new TypeError("check.ts must export a default function"); + } + const findings = await check(root, matches); + writeFileSync(outPath, JSON.stringify(findings ?? [])); +} catch (error) { + process.stderr.write(error instanceof Error ? (error.stack ?? error.message) : String(error)); + process.exit(1); +} +`; + +/** Resolve the bundled `tsx` CLI entry so `check.ts` runs without a repo toolchain. */ +function resolveTsxCli(): string { + const require = createRequire(import.meta.url); + const packageJsonPath = require.resolve("tsx/package.json"); + const packageJson = JSON.parse(readFileSync(packageJsonPath, "utf8")) as { + bin?: string | Record; + }; + const binField = packageJson.bin; + const relative = typeof binField === "string" ? binField : binField?.tsx; + if (!relative) { + throw new Error("Bundled tsx has no resolvable bin entry"); + } + return resolve(dirname(packageJsonPath), relative); +} + +/** + * Invoke a runtime rule's `check.ts` as a function and return its findings. + * Bounds execution with `timeoutMs`; a throw, a non-zero exit, or a timeout is + * isolated into an `error` result (the caller records one error-severity finding) + * and never aborts the overall run. + */ +export async function invokeCheck( + checkFile: string, + root: string, + matches: Match[], + timeoutMs: number = DEFAULT_CHECK_TIMEOUT_MS +): Promise { + let tsxCli: string; + try { + tsxCli = resolveTsxCli(); + } catch (error) { + return { + status: "error", + message: `runtime harness unavailable: ${ + error instanceof Error ? error.message : String(error) + }`, + }; + } + + const workDirectory = await mkdtemp(join(tmpdir(), "tskl-check-")); + const matchesPath = join(workDirectory, "matches.json"); + const runnerPath = join(workDirectory, "runner.mjs"); + const outPath = join(workDirectory, "findings.json"); + + try { + await writeFile(matchesPath, JSON.stringify(matches)); + await writeFile(runnerPath, RUNNER_SOURCE); + + const result = await new Promise((resolvePromise) => { + const child = spawn( + process.execPath, + [tsxCli, runnerPath, checkFile, root, matchesPath, outPath], + { stdio: ["ignore", "ignore", "pipe"] } + ); + const stderrChunks: string[] = []; + let timedOut = false; + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGKILL"); + }, timeoutMs); + + child.stderr.on("data", (chunk: Buffer) => + stderrChunks.push(chunk.toString()) + ); + child.on("error", (error) => { + clearTimeout(timer); + resolvePromise({ status: "error", message: error.message }); + }); + child.on("close", (code) => { + clearTimeout(timer); + if (timedOut) { + resolvePromise({ + status: "error", + message: `check timed out after ${String(timeoutMs)}ms`, + }); + return; + } + if (code !== 0) { + const detail = stderrChunks.join("").trim(); + resolvePromise({ + status: "error", + message: `check exited with code ${String(code)}${ + detail ? `: ${detail}` : "" + }`, + }); + return; + } + void readFile(outPath, "utf8") + .then((raw) => + resolvePromise({ + status: "ok", + findings: JSON.parse(raw) as Finding[], + }) + ) + .catch((error: unknown) => + resolvePromise({ + status: "error", + message: `check produced no readable findings: ${ + error instanceof Error ? error.message : String(error) + }`, + }) + ); + }); + }); + return result; + } finally { + await rm(workDirectory, { recursive: true, force: true }); + } +} diff --git a/packages/cli/src/rules/runtime/narrow.ts b/packages/cli/src/rules/runtime/narrow.ts new file mode 100644 index 00000000..abd727f2 --- /dev/null +++ b/packages/cli/src/rules/runtime/narrow.ts @@ -0,0 +1,164 @@ +import { spawn } from "node:child_process"; +import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { createInterface } from "node:readline"; +import { join } from "node:path"; + +import { stringify } from "yaml"; + +import { buildPath, findSgBinary } from "../scan"; +import type { Match } from "../../types/runtime-rule"; +import type { LoadedCaptureRule, RuntimeRule } from "./discover"; + +/** + * The subset of an ast-grep `--json=stream` match this harness reads. ast-grep + * reports `range.start` 0-indexed; the harness normalizes to 1-indexed `Match`. + */ +interface AstGrepStreamMatch { + text: string; + file: string; + ruleId: string; + range: { start: { line: number; column: number } }; + metaVariables?: { + single?: Record; + }; +} + +/** Spawn `sg scan` with a temp config and stream stdout lines to `onLine`. */ +function runSg( + root: string, + configPath: string, + extraArguments: string[], + paths: string[], + onLine: (line: string) => void +): Promise { + return new Promise((resolve, reject) => { + const argv = [ + "scan", + "--config", + configPath, + ...extraArguments, + ...(paths.length > 0 ? ["--", ...paths] : []), + ]; + const child = spawn(findSgBinary(), argv, { + cwd: root, + stdio: ["ignore", "pipe", "pipe"], + env: { ...process.env, PATH: buildPath() }, + }); + const stderrChunks: string[] = []; + const rl = createInterface({ input: child.stdout }); + rl.on("line", onLine); + child.stderr.on("data", (chunk: Buffer) => + stderrChunks.push(chunk.toString()) + ); + child.on("error", reject); + child.on("close", (code) => { + // ast-grep exits 1 when matches are found — expected. >1 is a real failure. + if (code !== null && code > 1) { + reject( + new Error( + `ast-grep narrow failed (exit ${String(code)})${ + stderrChunks.length > 0 ? `: ${stderrChunks.join("").trim()}` : "" + }` + ) + ); + return; + } + resolve(); + }); + }); +} + +/** Write the given capture rules into `/rules/` and a config pointing at them. */ +async function writeRuleConfig( + directory: string, + captureRules: LoadedCaptureRule[] +): Promise { + const rulesDirectory = join(directory, "rules"); + await mkdir(rulesDirectory, { recursive: true }); + await Promise.all( + captureRules.map((c) => + writeFile(join(rulesDirectory, `${c.id}.yml`), stringify(c.rule)) + ) + ); + const configPath = join(directory, "sgconfig.yml"); + await writeFile(configPath, stringify({ ruleDirs: ["rules"] })); + return configPath; +} + +/** + * Run a runtime rule's capture rules as the ast-grep narrow and return the + * normalized matches. Anchor captures produce full matches with `captures`; + * broad (`kind: program`) captures are path-only enumerators. `ruleId` is mapped + * back to the stable model `name` surfaced as `match.rule`. + */ +export async function runNarrow( + root: string, + rule: RuntimeRule, + paths: string[] = [] +): Promise { + const nameById = new Map(rule.captureRules.map((c) => [c.id, c.name])); + const anchor = rule.captureRules.filter((c) => c.match !== "broad"); + const broad = rule.captureRules.filter((c) => c.match === "broad"); + + const workDirectory = await mkdtemp(join(tmpdir(), "tskl-narrow-")); + const matches: Match[] = []; + try { + if (anchor.length > 0) { + const config = await writeRuleConfig( + join(workDirectory, "anchor"), + anchor + ); + await runSg(root, config, ["--json=stream"], paths, (line) => { + const trimmed = line.trim(); + if (trimmed === "") return; + let parsed: AstGrepStreamMatch; + try { + parsed = JSON.parse(trimmed) as AstGrepStreamMatch; + } catch { + return; // non-JSON status line + } + const captures: Record = {}; + for (const [key, value] of Object.entries( + parsed.metaVariables?.single ?? {} + )) { + if (typeof value.text === "string") captures[key] = value.text; + } + matches.push({ + rule: nameById.get(parsed.ruleId) ?? parsed.ruleId, + ruleId: parsed.ruleId, + file: parsed.file, + line: parsed.range.start.line + 1, + column: parsed.range.start.column + 1, + text: parsed.text, + captures, + }); + }); + } + + if (broad.length > 0) { + const config = await writeRuleConfig(join(workDirectory, "broad"), broad); + // A single ruleId can't be recovered from --files-with-matches; attribute + // broad matches to the (usually sole) broad capture rule. + const broadRule = broad[0]!; + const seen = new Set(); + await runSg(root, config, ["--files-with-matches"], paths, (line) => { + const file = line.trim(); + if (file === "" || seen.has(file)) return; + seen.add(file); + matches.push({ + rule: broadRule.name, + ruleId: broadRule.id, + file, + line: 1, + column: 1, + text: "", + captures: {}, + }); + }); + } + } finally { + await rm(workDirectory, { recursive: true, force: true }); + } + return matches; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index bbbc1243..5261fdde 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -98,6 +98,9 @@ importers: sprintf-js: specifier: ^1.1.3 version: 1.1.3 + tsx: + specifier: ^4.21.0 + version: 4.21.0 yaml: specifier: ^2.8.2 version: 2.8.2 From 7ff6ca091a4f4b434dacad8f96e0ca591d51846c Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 09:39:03 -0700 Subject: [PATCH 12/21] feat(cli): scope reconcile to runtime check.ts and materialize blessed rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Group 4 of runtime-rule-execution. Add the runtime reconcile-set: sign each rule's check.ts (only) for reconciliation, select the rules the server blessed, and materialize them for read-hash-execute. - src/rules/runtime/run-set.ts: signRuntimeChecks (check.ts only — capture *.yml are inert), reportRuntimeChecks -> { file, signature }, selectBlessedRuntimeRules (content-join: a rule runs iff its check.ts signature is in the run set; the rest are withheld/advisory), and materializeRuntimeRules (copy blessed rule dirs into .taskless/.run/runtime-rules/ and re-discover so execution uses the blessed bytes; .run/ stays gitignored). - discover.ts: extract discoverRuntimeRulesIn(root) so materialized rules can be re-discovered from .run/. - narrow.ts: copy the original capture *.yml bytes into the temp config instead of re-serializing the parsed object — a YAML round-trip can alter an exotic ast-grep config. Verified end-to-end: report only check.ts, bless on signature match, materialize to .run/, execute the materialized copy; empty run set withholds the rule. Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../changes/runtime-rule-execution/tasks.md | 8 +- packages/cli/src/rules/runtime/discover.ts | 19 +++- packages/cli/src/rules/runtime/narrow.ts | 11 ++- packages/cli/src/rules/runtime/run-set.ts | 93 +++++++++++++++++++ 4 files changed, 118 insertions(+), 13 deletions(-) create mode 100644 packages/cli/src/rules/runtime/run-set.ts diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/runtime-rule-execution/tasks.md index 5df454dd..65d68caa 100644 --- a/openspec/changes/runtime-rule-execution/tasks.md +++ b/openspec/changes/runtime-rule-execution/tasks.md @@ -5,7 +5,7 @@ ## 2. Narrow → gate → check harness -- [x] 2.1 Add `packages/cli/src/rules/runtime/narrow.ts`: assemble a rule's capture rules and run ONE `ast-grep` scan — anchor mode `--json=stream`, broad mode `--files-with-matches` (`kind: program`). (Used a temp `--config` rules dir rather than `--inline-rules` so multiple capture rules + full ast-grep config run in a single scan; reuses `findSgBinary`/`buildPath` from `scan.ts`.) +- [x] 2.1 Add `packages/cli/src/rules/runtime/narrow.ts`: assemble a rule's capture rules and run ONE `ast-grep` scan per mode — anchor `--json=stream`, broad `--files-with-matches` (`kind: program`). (Used a temp `--config` rules dir rather than `--inline-rules` — a runtime rule has multiple capture rules + full ast-grep config, which `--inline-rules` can't carry; **copies the original capture `*.yml` bytes** to avoid a YAML round-trip. Grouped by mode: all-anchor = 1 scan, mixed = 1 per mode to avoid broad's whole-file streaming. Reuses `findSgBinary`/`buildPath`.) - [x] 2.2 Add match normalization to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed capture `id` back to the model `name` surfaced as `match.rule`. (ast-grep 0-indexed → 1-indexed; captures from `metaVariables.single`.) - [x] 2.3 Gate on matches: when the narrow yields zero matches, do NOT invoke `check.ts`. - [x] 2.4 Add `packages/cli/src/rules/runtime/invoke.ts`: call `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`; use the returned `Finding[]`. Isolate a throwing check to a single error-severity finding for that rule. (Runs an embedded ESM runner under `tsx`; findings via an out-file so the check's stdout can't pollute the channel.) @@ -19,9 +19,9 @@ ## 4. Reconcile scoping & materialization -- [ ] 4.1 Extend `src/rules/run-set.ts` to enumerate `.taskless/runtime-rules/` and sign each rule's `check.ts` only (capture `*.yml` and static rules are inert — not reported to reconcile). -- [ ] 4.2 Compute per-rule eligibility: a runtime rule executes only if its `check.ts` is in `run`; otherwise withhold and surface as advisory. -- [ ] 4.3 Materialize blessed runtime rules into `.taskless/.run/` and execute `check.ts` from there (read-hash-execute); ensure `.taskless/.gitignore` still covers `.run/`. +- [x] 4.1 Add `src/rules/runtime/run-set.ts` (runtime-cohesive rather than bloating the static `run-set.ts`): enumerate via `discoverRuntimeRules`, `signRuntimeChecks` signs each rule's `check.ts` only, `reportRuntimeChecks` maps to `{ file, signature }` (capture `*.yml` and static rules are inert — not reported). +- [x] 4.2 `selectBlessedRuntimeRules(signed, run)` computes per-rule eligibility by content-join: a rule is blessed iff its `check.ts` signature is in `run`; the rest are `withheld` (advisory). +- [x] 4.3 `materializeRuntimeRules` copies blessed rule dirs into `.taskless/.run/runtime-rules//` and re-discovers them (via `discoverRuntimeRulesIn`) so the narrow + `check.ts` execute the blessed bytes; `addToGitignore` keeps `.run/` ignored. ## 5. check dispatch & modes diff --git a/packages/cli/src/rules/runtime/discover.ts b/packages/cli/src/rules/runtime/discover.ts index 729d8833..949b5200 100644 --- a/packages/cli/src/rules/runtime/discover.ts +++ b/packages/cli/src/rules/runtime/discover.ts @@ -87,14 +87,25 @@ async function loadCaptureRules( } /** - * Enumerate `.taskless/runtime-rules/` and return each rule directory that holds - * at least one `kind: runtime` capture rule. `.taskless/runtime-rule-tests/` is - * never enumerated — it holds verification fixtures, not executable rules. + * Enumerate `.taskless/runtime-rules/` under `cwd` and return each rule + * directory that holds at least one `kind: runtime` capture rule. + * `.taskless/runtime-rule-tests/` is never enumerated — it holds verification + * fixtures, not executable rules. */ export async function discoverRuntimeRules( cwd: string ): Promise { - const root = join(cwd, ".taskless", RUNTIME_RULES_DIR); + return discoverRuntimeRulesIn(join(cwd, ".taskless", RUNTIME_RULES_DIR)); +} + +/** + * Enumerate runtime rules under an explicit `runtime-rules` root — used to + * re-discover rules from the materialized `.taskless/.run/` tree so the executed + * bytes are the blessed ones. + */ +export async function discoverRuntimeRulesIn( + root: string +): Promise { let directoryEntries; try { directoryEntries = await readdir(root, { withFileTypes: true }); diff --git a/packages/cli/src/rules/runtime/narrow.ts b/packages/cli/src/rules/runtime/narrow.ts index abd727f2..c95d58dc 100644 --- a/packages/cli/src/rules/runtime/narrow.ts +++ b/packages/cli/src/rules/runtime/narrow.ts @@ -1,5 +1,5 @@ import { spawn } from "node:child_process"; -import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { copyFile, mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { createInterface } from "node:readline"; import { join } from "node:path"; @@ -69,17 +69,18 @@ function runSg( }); } -/** Write the given capture rules into `/rules/` and a config pointing at them. */ +/** Copy the given capture files into `/rules/` and a config pointing at them. */ async function writeRuleConfig( directory: string, captureRules: LoadedCaptureRule[] ): Promise { const rulesDirectory = join(directory, "rules"); await mkdir(rulesDirectory, { recursive: true }); + // Copy the original `*.yml` bytes rather than re-serializing the parsed rule — + // a YAML round-trip can subtly alter an exotic ast-grep config, and the file is + // a valid ast-grep rule as delivered. await Promise.all( - captureRules.map((c) => - writeFile(join(rulesDirectory, `${c.id}.yml`), stringify(c.rule)) - ) + captureRules.map((c) => copyFile(c.file, join(rulesDirectory, c.fileName))) ); const configPath = join(directory, "sgconfig.yml"); await writeFile(configPath, stringify({ ruleDirs: ["rules"] })); diff --git a/packages/cli/src/rules/runtime/run-set.ts b/packages/cli/src/rules/runtime/run-set.ts new file mode 100644 index 00000000..8ce5e0ea --- /dev/null +++ b/packages/cli/src/rules/runtime/run-set.ts @@ -0,0 +1,93 @@ +import { cp, mkdir, rm } from "node:fs/promises"; +import { join, relative } from "node:path"; + +import { addToGitignore } from "../../filesystem/gitignore"; +import { signRuleFile } from "../rule-hash"; +import type { ReportedFile, RunEntry } from "../../api/reconcile"; +import { discoverRuntimeRulesIn, type RuntimeRule } from "./discover"; + +/** Directory (relative to `.taskless/`) holding materialized runtime rules. */ +export const RUNTIME_RUN_DIR = ".run/runtime-rules"; + +/** A runtime rule paired with its `check.ts` signature — the reconcile gate. */ +export interface SignedRuntimeRule { + rule: RuntimeRule; + /** Canonical signature envelope for the rule's `check.ts` bytes. */ + signature: string; +} + +/** + * Sign each runtime rule's `check.ts` (only) — the sole artifact carrying + * arbitrary code execution. Capture `*.yml` are inert and are neither signed nor + * reported. + */ +export async function signRuntimeChecks( + rules: RuntimeRule[] +): Promise { + return Promise.all( + rules.map(async (rule) => ({ + rule, + signature: await signRuleFile(rule.checkFile), + })) + ); +} + +/** Map signed runtime rules to the reconcile report (`check.ts` path + signature). */ +export function reportRuntimeChecks( + cwd: string, + signed: SignedRuntimeRule[] +): ReportedFile[] { + return signed.map(({ rule, signature }) => ({ + file: relative(cwd, rule.checkFile), + signature, + })); +} + +/** The blessed/withheld split from a successful reconciliation. */ +export interface RuntimeSelection { + /** Rules whose `check.ts` signature is in the server `run` set. */ + blessed: RuntimeRule[]; + /** Rules whose `check.ts` was not blessed (advisory). */ + withheld: RuntimeRule[]; +} + +/** + * Split signed runtime rules into those whose `check.ts` is present in the + * server's `run` set (blessed, execute) and the rest (withheld, advisory). The + * join is by signature — content-based, so a moved-but-unchanged rule resolves. + */ +export function selectBlessedRuntimeRules( + signed: SignedRuntimeRule[], + run: RunEntry[] +): RuntimeSelection { + const runSignatures = new Set(run.map((entry) => entry.signature)); + const blessed: RuntimeRule[] = []; + const withheld: RuntimeRule[] = []; + for (const { rule, signature } of signed) { + if (runSignatures.has(signature)) blessed.push(rule); + else withheld.push(rule); + } + return { blessed, withheld }; +} + +/** + * Materialize blessed runtime rules into the gitignored + * `.taskless/.run/runtime-rules/` and return them re-discovered from there, so + * the narrow and `check.ts` execute the blessed bytes rather than whatever is + * live in `.taskless/runtime-rules/`. + */ +export async function materializeRuntimeRules( + cwd: string, + blessed: RuntimeRule[] +): Promise { + const runtimeRunRoot = join(cwd, ".taskless", RUNTIME_RUN_DIR); + await rm(runtimeRunRoot, { recursive: true, force: true }); + await mkdir(runtimeRunRoot, { recursive: true }); + await Promise.all( + blessed.map((rule) => + cp(rule.dir, join(runtimeRunRoot, rule.name), { recursive: true }) + ) + ); + await addToGitignore(cwd, [".run/"]); + return discoverRuntimeRulesIn(runtimeRunRoot); +} From 0bad732a310fa726b94167d9d8fc55a0ac885d86 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 09:53:38 -0700 Subject: [PATCH 13/21] feat(cli): dispatch static vs runtime rules in check; cut over reconcile gating MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Group 5 of runtime-rule-execution. Rewire `taskless check` around the two rule classes and complete the cutover to signing only untrusted code. - Static ast-grep rules under .taskless/rules/ always run (trusted, no network). - Runtime rules run only on a validated path: planRuntime resolves the mode from auth state + flags — authed reconcile runs blessed rules and withholds the rest (advisory); logged-out / --anonymous / no-remote / reconcile-unavailable skip runtime with a notice; --dangerously-run-scripts runs all runtime rules with no network behind a loud warning. - Add --dangerously-run-scripts and --timeout ; runtime findings merge into the same results and exit-code logic; --json gains an additive optional `skipped` array (schema updated), warnings/notices stay stderr-only. - Fix a Finding->CheckResult off-by-one: findings are 1-indexed, CheckResult.range is 0-indexed (display/json add 1). Cutover: remove the stacked-under static-reconcile gating — delete src/rules/run-set.ts and the now-obsolete test/reconcile-check.test.ts + test/run-set.test.ts (runtime-dispatch tests land in Group 7). Static rules are no longer signed or gated; only runtime check.ts is. Verified end-to-end via the built CLI: static-only-runs (runtime skipped + notice), --dangerously-run-scripts (both run), and --json (skipped array, warnings suppressed). Full CLI suite green (326 tests). Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../changes/runtime-rule-execution/tasks.md | 10 +- packages/cli/src/commands/check.ts | 238 +++++++++----- packages/cli/src/rules/run-set.ts | 72 ---- packages/cli/src/rules/runtime/harness.ts | 11 +- packages/cli/src/schemas/check.ts | 10 + packages/cli/test/reconcile-check.test.ts | 310 ------------------ packages/cli/test/run-set.test.ts | 131 -------- 7 files changed, 174 insertions(+), 608 deletions(-) delete mode 100644 packages/cli/src/rules/run-set.ts delete mode 100644 packages/cli/test/reconcile-check.test.ts delete mode 100644 packages/cli/test/run-set.test.ts diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/runtime-rule-execution/tasks.md index 65d68caa..5af550d0 100644 --- a/openspec/changes/runtime-rule-execution/tasks.md +++ b/openspec/changes/runtime-rule-execution/tasks.md @@ -25,11 +25,11 @@ ## 5. check dispatch & modes -- [ ] 5.1 In `src/commands/check.ts`, split the corpus via `classifyRules`; always run static rules; route runtime rules through the harness only on a validated path. -- [ ] 5.2 Implement the mode table: authed/API-key → reconcile & run fully-blessed runtime rules; logged-out/`--anonymous` → skip runtime + report skipped; reconcile-cannot-complete → skip runtime with notice; `--dangerously-run-scripts` → run all runtime rules trusting local signatures. -- [ ] 5.3 Add the `--dangerously-run-scripts` flag: skip reconciliation entirely (no network) and run all present runtime rules, behind a prominent unverified-execution warning (stderr, suppressed under `--json`). Add the `--timeout ` flag. -- [ ] 5.4 Narrow the stacked-under degrade path so it scans static rules but never executes runtime `check.ts`. -- [ ] 5.5 Emit skipped-runtime notices (human output); under `--json` add an additive optional `skipped: [{ rule, reason }]` field leaving `success`/`results` unchanged. Confirm the exit code stays governed solely by error-severity findings. +- [x] 5.1 Rewrote `src/commands/check.ts`: static rules under `.taskless/rules/` always scan; runtime rules route through the harness only on a validated path. **Cutover** — removed the stacked-under static-reconcile gating (deleted `src/rules/run-set.ts`). +- [x] 5.2 `planRuntime` implements the mode table: authed → reconcile & run blessed rules, withhold the rest (advisory); logged-out/`--anonymous`/no-remote/reconcile-unavailable → skip runtime + report skipped; `--dangerously-run-scripts` → run all runtime rules (no network). +- [x] 5.3 Added `--dangerously-run-scripts` (skips reconciliation, runs all runtime rules, prominent stderr warning suppressed under `--json`) and `--timeout ` (→ `parseTimeoutMs`). +- [x] 5.4 The former degrade path now scans static rules but **never executes runtime `check.ts`** unverified — skip-with-notice instead. +- [x] 5.5 Skipped-runtime notices are human-output; under `--json` an additive optional `skipped: [{ rule, reason }]` field is added (schema updated) leaving `success`/`results` unchanged. Fixed a `Finding`→`CheckResult` off-by-one (findings are 1-indexed; `CheckResult.range` is 0-indexed). Exit code stays governed solely by error-severity findings. Removed the now-obsolete `test/reconcile-check.test.ts` + `test/run-set.test.ts` (Group 7 adds runtime-dispatch tests). ## 6. Help & docs diff --git a/packages/cli/src/commands/check.ts b/packages/cli/src/commands/check.ts index 86ba9ea3..300ed5b5 100644 --- a/packages/cli/src/commands/check.ts +++ b/packages/cli/src/commands/check.ts @@ -12,14 +12,18 @@ import { makeErrorEnvelope } from "../types/errors"; import { getToken } from "../auth/token"; import { resolveRepositoryUrl } from "../util/git-remote"; import { getCliPrefix } from "../util/package-manager"; -import { reconcile, type ReconcileResponse } from "../api/reconcile"; +import { reconcile } from "../api/reconcile"; import { - RUN_RULES_DIR, - materializeRunDirectory, - selectRunFiles, - signRuleFiles, - type SignedRuleFile, -} from "../rules/run-set"; + discoverRuntimeRules, + type RuntimeRule, +} from "../rules/runtime/discover"; +import { + materializeRuntimeRules, + reportRuntimeChecks, + selectBlessedRuntimeRules, + signRuntimeChecks, +} from "../rules/runtime/run-set"; +import { executeRuntimeRules } from "../rules/runtime/harness"; async function pathExists(absolutePath: string): Promise { try { @@ -83,7 +87,9 @@ function extractPositionalPaths(rawArguments: string[]): string[] { if (argument.startsWith("-")) { // Skip value for short/long flags that take a value if ( - (argument === "-d" || argument === "--dir") && + (argument === "-d" || + argument === "--dir" || + argument === "--timeout") && index + 1 < rawArguments.length ) { index += 1; @@ -95,79 +101,121 @@ function extractPositionalPaths(rawArguments: string[]): string[] { return paths; } +/** A runtime rule that will not run, with why (advisory). */ +interface SkippedRuntimeRule { + rule: string; + reason: string; +} + +/** The runtime-execution plan resolved from auth state and flags. */ +interface RuntimePlan { + /** Rules to execute — materialized when gated, live under `--dangerously-run-scripts`. */ + execute: RuntimeRule[]; + /** Rules that will not run, with a reason. */ + skipped: SkippedRuntimeRule[]; + /** Human-only notices about the runtime disposition. */ + notices: string[]; +} + +/** Skip every runtime rule with a shared reason (an unverified path). */ +function skipAllRuntime(rules: RuntimeRule[], reason: string): RuntimePlan { + return { + execute: [], + skipped: rules.map((rule) => ({ rule: rule.name, reason })), + notices: [], + }; +} + /** - * How `check` will scan, decided from auth state: - * - `local`: unauthenticated or `--anonymous` — scan all local rules silently. - * - `degrade`: authenticated but reconciliation could not complete — scan all - * local rules and warn that verification could not be performed. - * - `gated`: reconciliation succeeded — scan only the blessed `run` set. + * Decide which runtime rules run. A runtime rule's `check.ts` is arbitrary code + * execution, so it runs only when its signature is server-validated (an + * authenticated reconcile that returns it in `run`) or `--dangerously-run-scripts` + * is set. Every unverified path — anonymous, logged out, no remote, or a + * reconcile that cannot complete — skips runtime rules without failing. */ -type ScanMode = - | { kind: "local" } - | { kind: "degrade"; reason: string } - | { kind: "gated"; result: ReconcileResponse; signed: SignedRuleFile[] }; - -async function resolveScanMode( +async function planRuntime( cwd: string, - anonymous: boolean, - ruleFiles: string[] -): Promise { - if (anonymous) return { kind: "local" }; + discovered: RuntimeRule[], + options: { anonymous: boolean; dangerouslyRunScripts: boolean } +): Promise { + if (discovered.length === 0) return { execute: [], skipped: [], notices: [] }; + + if (options.dangerouslyRunScripts) { + return { + execute: discovered, + skipped: [], + notices: [ + "Warning: --dangerously-run-scripts is executing runtime rule code without server verification.", + ], + }; + } + + if (options.anonymous) { + return skipAllRuntime( + discovered, + "anonymous mode — runtime rules were not verified and did not run" + ); + } const token = await getToken(cwd, { silent: true }); - if (!token) return { kind: "local" }; + if (!token) { + return skipAllRuntime( + discovered, + "not authenticated — runtime rules were not verified and did not run" + ); + } let repositoryUrl: string; try { repositoryUrl = await resolveRepositoryUrl(cwd); } catch { - return { - kind: "degrade", - reason: "no GitHub remote was found, so rules could not be verified", - }; + return skipAllRuntime( + discovered, + "no GitHub remote — runtime rules could not be verified and did not run" + ); } - const signed = await signRuleFiles(cwd, ruleFiles); + const signed = await signRuntimeChecks(discovered); const outcome = await reconcile(token, { repositoryUrl, - files: signed.map(({ file, signature }) => ({ file, signature })), + files: reportRuntimeChecks(cwd, signed), }); - if (outcome.status === "ok") { - return { kind: "gated", result: outcome.result, signed }; - } if (outcome.status === "unauthorized") { - return { - kind: "degrade", - reason: `authentication was rejected — run \`${getCliPrefix()} auth login\` to re-authenticate`, - }; + return skipAllRuntime( + discovered, + `authentication was rejected — run \`${getCliPrefix()} auth login\` to re-authenticate` + ); } + if (outcome.status === "unavailable") { + return skipAllRuntime( + discovered, + `the rule service was unavailable (${outcome.reason})` + ); + } + + const { blessed, withheld } = selectBlessedRuntimeRules( + signed, + outcome.result.run + ); + const execute = + blessed.length > 0 ? await materializeRuntimeRules(cwd, blessed) : []; return { - kind: "degrade", - reason: `the rule service was unavailable (${outcome.reason})`, + execute, + skipped: withheld.map((rule) => ({ + rule: rule.name, + reason: "not blessed by the server (unsafe / unknown / drift)", + })), + notices: [], }; } -/** Emit advisory reconciliation warnings (human output only). */ -function surfaceReconcileWarnings( - result: ReconcileResponse, - warn: (message: string) => void -): void { - for (const entry of result.unsafe) { - warn( - `Warning: ${entry.file} differs from the blessed rule (tamper/drift) and will not run.` - ); - } - for (const entry of result.unknown) { - warn( - `Notice: ${entry.file} was not issued by the server and will not run.` - ); - } - for (const entry of result.missing) { - warn( - `Notice: expected rule ${entry.ruleId} (${entry.file}) is not present locally.` - ); - } +/** Parse `--timeout ` into milliseconds; invalid/absent → undefined (default). */ +function parseTimeoutMs(raw: string | undefined): number | undefined { + if (raw === undefined) return undefined; + const seconds = Number(raw); + if (!Number.isFinite(seconds) || seconds <= 0) return undefined; + return Math.round(seconds * 1000); } export const checkCommand = defineCommand({ @@ -188,16 +236,27 @@ export const checkCommand = defineCommand({ }, anonymous: { type: "boolean", - description: "Skip server reconciliation and scan local rules unverified", + description: + "Run only trusted static rules; skip runtime rules (no reconciliation)", default: false, }, + "dangerously-run-scripts": { + type: "boolean", + description: + "Run runtime-rule check.ts without server verification (executes untrusted code)", + default: false, + }, + timeout: { + type: "string", + description: "Per-runtime-check timeout in seconds (default 10)", + }, }, async run({ args, rawArgs }) { const cwd = resolve(args.dir ?? process.cwd()); const telemetry = await getTelemetry(cwd); - // Warnings are advisory human output; suppress them under --json so the - // machine output stays the existing { success, results } shape. + // Warnings/notices are advisory human output; suppress them under --json so + // the machine output stays the { success, results, skipped? } shape. const warn = (message: string) => { if (!args.json) console.error(message); }; @@ -227,17 +286,19 @@ export const checkCommand = defineCommand({ return; } - // Check for rule files + // Static rules (trusted ast-grep YAML) always run; runtime rules + // (untrusted check.ts) are gated separately. const rulesDirectory = join(cwd, ".taskless", "rules"); - let ruleFiles: string[] = []; + let staticRuleFiles: string[] = []; try { const entries = await readdir(rulesDirectory); - ruleFiles = entries.filter((f) => f.endsWith(".yml")); + staticRuleFiles = entries.filter((f) => f.endsWith(".yml")); } catch { // .taskless/ or rules/ directory doesn't exist } + const runtimeRules = await discoverRuntimeRules(cwd); - if (ruleFiles.length === 0) { + if (staticRuleFiles.length === 0 && runtimeRules.length === 0) { if (args.json) { console.log( JSON.stringify( @@ -252,30 +313,33 @@ export const checkCommand = defineCommand({ return; } - // Decide what to run from auth state, then scan. try { - const mode = await resolveScanMode(cwd, args.anonymous, ruleFiles); + const results: CheckResult[] = []; - let results: CheckResult[] = []; - if (mode.kind === "gated") { - surfaceReconcileWarnings(mode.result, warn); - const runFiles = selectRunFiles(mode.signed, mode.result.run); - // Empty run set (empty corpus, or all unsafe/unknown): run nothing. - if (runFiles.length > 0) { - await materializeRunDirectory(cwd, runFiles); - await generateSgConfig(cwd, { rulesDirectory: RUN_RULES_DIR }); - const scan = await runAstGrepScan(cwd, existingPaths); - results = scan.results; - } - } else { - if (mode.kind === "degrade") { - warn( - `Notice: ${mode.reason}. Scanning all local rules unverified; the CI backstop enforces the server-owned rule set.` - ); - } + // Static rules: always scan, no verification (inert data). + if (staticRuleFiles.length > 0) { await generateSgConfig(cwd); const scan = await runAstGrepScan(cwd, existingPaths); - results = scan.results; + results.push(...scan.results); + } + + // Runtime rules: run only what the server validated (or forced). + const plan = await planRuntime(cwd, runtimeRules, { + anonymous: args.anonymous, + dangerouslyRunScripts: Boolean(args["dangerously-run-scripts"]), + }); + for (const notice of plan.notices) warn(notice); + for (const skipped of plan.skipped) { + warn( + `Notice: runtime rule ${skipped.rule} was not run — ${skipped.reason}.` + ); + } + if (plan.execute.length > 0) { + const runtimeResults = await executeRuntimeRules(cwd, plan.execute, { + paths: existingPaths, + timeoutMs: parseTimeoutMs(args.timeout), + }); + results.push(...runtimeResults); } let errorCount = 0; @@ -287,11 +351,11 @@ export const checkCommand = defineCommand({ const hasErrors = errorCount > 0; scanCounts = { errorCount, warningCount, findings: results.length }; - // Format output if (args.json) { const output = checkOutputSchema.parse({ success: !hasErrors, results, + ...(plan.skipped.length > 0 ? { skipped: plan.skipped } : {}), }); console.log(JSON.stringify(output)); } else { diff --git a/packages/cli/src/rules/run-set.ts b/packages/cli/src/rules/run-set.ts deleted file mode 100644 index 44861a09..00000000 --- a/packages/cli/src/rules/run-set.ts +++ /dev/null @@ -1,72 +0,0 @@ -import { copyFile, mkdir, rm } from "node:fs/promises"; -import { join } from "node:path"; - -import { addToGitignore } from "../filesystem/gitignore"; -import type { RunEntry } from "../api/reconcile"; -import { signRuleFile } from "./rule-hash"; - -/** A local rule file paired with its canonical signature. */ -export interface SignedRuleFile { - /** Delivered file name as it exists under `.taskless/rules/`. */ - file: string; - /** Absolute path to the file on disk. */ - path: string; - /** Canonical signature envelope for the file's bytes. */ - signature: string; -} - -/** Directory (relative to `.taskless/`) holding the materialized run set. */ -export const RUN_RULES_DIR = ".run/rules"; - -/** Sign each rule file, returning its delivered name, path, and signature. */ -export async function signRuleFiles( - cwd: string, - fileNames: string[] -): Promise { - const rulesDirectory = join(cwd, ".taskless", "rules"); - return Promise.all( - fileNames.map(async (file) => { - const path = join(rulesDirectory, file); - return { file, path, signature: await signRuleFile(path) }; - }) - ); -} - -/** - * Resolve the server's `run` entries back to local files by signature - * (content-based join, so a moved-but-unchanged file still resolves). Entries - * whose signature is not held locally are dropped. - */ -export function selectRunFiles( - signed: SignedRuleFile[], - run: RunEntry[] -): SignedRuleFile[] { - const bySignature = new Map(signed.map((s) => [s.signature, s])); - const selected: SignedRuleFile[] = []; - const seen = new Set(); - for (const entry of run) { - const match = bySignature.get(entry.signature); - if (match && !seen.has(match.path)) { - seen.add(match.path); - selected.push(match); - } - } - return selected; -} - -/** - * Materialize the blessed run set into a fresh, gitignored - * `.taskless/.run/rules/` so ast-grep evaluates only those files. The directory - * is rebuilt on every call to avoid stale rules leaking into a scan. - */ -export async function materializeRunDirectory( - cwd: string, - files: SignedRuleFile[] -): Promise { - const runRoot = join(cwd, ".taskless", ".run"); - const runRules = join(runRoot, "rules"); - await rm(runRoot, { recursive: true, force: true }); - await mkdir(runRules, { recursive: true }); - await Promise.all(files.map((f) => copyFile(f.path, join(runRules, f.file)))); - await addToGitignore(cwd, [".run/"]); -} diff --git a/packages/cli/src/rules/runtime/harness.ts b/packages/cli/src/rules/runtime/harness.ts index 14105793..f6ffd056 100644 --- a/packages/cli/src/rules/runtime/harness.ts +++ b/packages/cli/src/rules/runtime/harness.ts @@ -17,13 +17,18 @@ export interface RuntimeRunOptions { timeoutMs?: number; } -/** Map a check `Finding` onto the scanner-agnostic `CheckResult`. */ +/** + * Map a check `Finding` onto the scanner-agnostic `CheckResult`. `Finding` + * line/column are 1-indexed (harness contract); `CheckResult.range` is 0-indexed + * (ast-grep native — display and `--json` consumers add 1), so convert down. + */ function findingToCheckResult( rule: RuntimeRule, finding: Finding ): CheckResult { - const line = finding.line ?? 0; - const column = finding.column ?? 0; + const line = finding.line === undefined ? 0 : Math.max(0, finding.line - 1); + const column = + finding.column === undefined ? 0 : Math.max(0, finding.column - 1); return { source: RUNTIME_SOURCE, ruleId: rule.name, diff --git a/packages/cli/src/schemas/check.ts b/packages/cli/src/schemas/check.ts index c66b8909..ebe596e5 100644 --- a/packages/cli/src/schemas/check.ts +++ b/packages/cli/src/schemas/check.ts @@ -24,10 +24,20 @@ const checkResultSchema = z.object({ fix: z.string().nullable().optional().describe("Suggested fix replacement"), }); +/** A runtime rule that was present but not executed (advisory), for `--json`. */ +const skippedRuntimeRuleSchema = z.object({ + rule: z.string().describe("Runtime rule name that did not run"), + reason: z.string().describe("Why the runtime rule was not run"), +}); + /** Output schema for `taskless check --json` on success */ export const outputSchema = z.object({ success: z.boolean(), results: z.array(checkResultSchema).describe("Check results"), + skipped: z + .array(skippedRuntimeRuleSchema) + .optional() + .describe("Runtime rules present but not executed"), }); /** Error schema for `taskless check --json` on failure */ diff --git a/packages/cli/test/reconcile-check.test.ts b/packages/cli/test/reconcile-check.test.ts deleted file mode 100644 index 194d5d1d..00000000 --- a/packages/cli/test/reconcile-check.test.ts +++ /dev/null @@ -1,310 +0,0 @@ -import { execFile } from "node:child_process"; -import { createServer, type Server } from "node:http"; -import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; -import { resolve, join } from "node:path"; -import { tmpdir } from "node:os"; -import { promisify } from "node:util"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; - -const execFileAsync = promisify(execFile); -const binPath = resolve(import.meta.dirname, "../dist/index.js"); - -interface ReportedFile { - file: string; - signature: string; -} -interface ReconcileRequestBody { - repositoryUrl: string; - files: ReportedFile[]; -} -interface MockResponse { - statusCode: number; - body?: unknown; -} -type Responder = (request: ReconcileRequestBody) => MockResponse; - -interface MockServer { - apiUrl: string; - requests: ReconcileRequestBody[]; - close: () => Promise; -} - -/** Start a mock reconcile endpoint on a random port. */ -function startMockServer(responder: Responder): Promise { - const requests: ReconcileRequestBody[] = []; - const server: Server = createServer((request, response) => { - if (request.method !== "POST" || request.url !== "/cli/api/reconcile") { - response.writeHead(404).end("{}"); - return; - } - let raw = ""; - request.on("data", (chunk: Buffer) => (raw += chunk.toString())); - request.on("end", () => { - const parsed = JSON.parse(raw) as ReconcileRequestBody; - requests.push(parsed); - const { statusCode, body } = responder(parsed); - response.writeHead(statusCode, { "content-type": "application/json" }); - response.end(JSON.stringify(body ?? {})); - }); - }); - return new Promise((resolvePromise) => { - server.listen(0, "127.0.0.1", () => { - const address = server.address(); - const port = typeof address === "object" && address ? address.port : 0; - resolvePromise({ - apiUrl: `http://127.0.0.1:${String(port)}/cli`, - requests, - close: () => - new Promise((done) => { - server.close(() => done()); - }), - }); - }); - }); -} - -/** Look up a reported file's signature so the mock can echo it back. */ -function sig(request: ReconcileRequestBody, file: string): string { - return request.files.find((f) => f.file === file)?.signature ?? ""; -} - -async function runCli( - args: string[], - env: Record = {} -): Promise<{ stdout: string; stderr: string; exitCode: number }> { - try { - const { stdout, stderr } = await execFileAsync("node", [binPath, ...args], { - env: { ...process.env, ...env }, - }); - return { stdout, stderr, exitCode: 0 }; - } catch (error) { - const execError = error as { stdout: string; stderr: string; code: number }; - return { - stdout: execError.stdout ?? "", - stderr: execError.stderr ?? "", - exitCode: execError.code, - }; - } -} - -const RULE_NO_EVAL = [ - "id: no-eval", - "language: javascript", - "severity: error", - "rule:", - " pattern: eval($$$)", - "message: Avoid eval", -].join("\n"); - -const RULE_NO_CONSOLE = [ - "id: no-console", - "language: javascript", - "severity: error", - "rule:", - " pattern: console.log($$$)", - "message: Avoid console.log", -].join("\n"); - -describe("check reconciliation", () => { - let directory: string; - - beforeEach(async () => { - directory = await mkdtemp(join(tmpdir(), "taskless-reconcile-")); - const rules = join(directory, ".taskless", "rules"); - await mkdir(rules, { recursive: true }); - await writeFile(join(rules, "no-eval.yml"), RULE_NO_EVAL, "utf8"); - await writeFile(join(rules, "no-console.yml"), RULE_NO_CONSOLE, "utf8"); - await writeFile( - join(directory, "app.js"), - 'eval("x");\nconsole.log("y");\n', - "utf8" - ); - // A GitHub origin is required for reconciliation to run. - await execFileAsync("git", ["init"], { cwd: directory }); - await execFileAsync( - "git", - ["remote", "add", "origin", "https://github.com/acme/widgets.git"], - { cwd: directory } - ); - }); - - afterEach(async () => { - await rm(directory, { recursive: true, force: true }); - }); - - it("runs ONLY the blessed run set, excluding non-run rules", async () => { - const server = await startMockServer((request) => ({ - statusCode: 200, - body: { - run: [ - { - ruleId: "no-eval", - file: "no-eval.yml", - signature: sig(request, "no-eval.yml"), - }, - ], - unsafe: [], - unknown: [], - missing: [], - }, - })); - try { - const { stdout, exitCode } = await runCli( - ["check", "-d", directory, "--json"], - { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } - ); - expect(server.requests).toHaveLength(1); - expect(exitCode).toBe(1); // no-eval is an error and matches - const parsed = JSON.parse(stdout.trim()) as { - results: { ruleId: string }[]; - }; - const ruleIds = new Set(parsed.results.map((r) => r.ruleId)); - expect(ruleIds.has("no-eval")).toBe(true); - expect(ruleIds.has("no-console")).toBe(false); // not in run set - } finally { - await server.close(); - } - }); - - it("warns on unsafe/unknown/missing and exits 0 on an empty run set", async () => { - const server = await startMockServer((request) => ({ - statusCode: 200, - body: { - run: [], - unsafe: [ - { - file: "no-console.yml", - expected: "1;h=sha-256;d=" + "0".repeat(64), - got: sig(request, "no-console.yml"), - }, - ], - unknown: [{ file: "no-eval.yml" }], - missing: [{ ruleId: "no-var", file: "no-var-abc.yml" }], - }, - })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory], - { - TASKLESS_TOKEN: "fake.token.value", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(exitCode).toBe(0); // empty run set → nothing scanned - expect(stdout).toContain("No issues found"); - expect(stderr).toContain("no-console.yml"); // unsafe drift - expect(stderr).toContain("no-eval.yml"); // unknown - expect(stderr).toContain("no-var"); // missing - } finally { - await server.close(); - } - }); - - it("suppresses mismatch warnings under --json", async () => { - const server = await startMockServer((request) => ({ - statusCode: 200, - body: { - run: [], - unsafe: [ - { - file: "no-console.yml", - expected: "1;h=sha-256;d=" + "0".repeat(64), - got: sig(request, "no-console.yml"), - }, - ], - unknown: [], - missing: [], - }, - })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory, "--json"], - { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } - ); - expect(exitCode).toBe(0); - expect(stderr).not.toContain("tamper"); - expect(stderr).not.toContain("no-console.yml"); - const parsed = JSON.parse(stdout.trim()) as { - success: boolean; - results: unknown[]; - }; - expect(parsed).toEqual({ success: true, results: [] }); - } finally { - await server.close(); - } - }); - - it("degrades to a local scan when reconciliation is unavailable", async () => { - const server = await startMockServer(() => ({ statusCode: 503 })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory], - { - TASKLESS_TOKEN: "fake.token.value", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(server.requests).toHaveLength(1); - expect(exitCode).toBe(1); // fell back to a full local scan; no-eval matches - expect(stdout).toContain("no-eval"); - expect(stdout).toContain("no-console"); // ALL local rules ran - expect(stderr).toContain("Scanning all local rules unverified"); - } finally { - await server.close(); - } - }); - - it("does not warn under --json when degrading", async () => { - const server = await startMockServer(() => ({ statusCode: 503 })); - try { - const { stdout, stderr } = await runCli( - ["check", "-d", directory, "--json"], - { - TASKLESS_TOKEN: "fake.token.value", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(stderr).not.toContain("Scanning all local rules unverified"); - const parsed = JSON.parse(stdout.trim()) as { results: unknown[] }; - expect(parsed.results.length).toBeGreaterThan(0); - } finally { - await server.close(); - } - }); - - it("scans all local rules silently when logged out (no reconcile)", async () => { - const server = await startMockServer(() => ({ statusCode: 200, body: {} })); - try { - const { stdout, stderr, exitCode } = await runCli( - ["check", "-d", directory], - { - TASKLESS_TOKEN: "", - TASKLESS_API_URL: server.apiUrl, - } - ); - expect(server.requests).toHaveLength(0); // never reconciled - expect(exitCode).toBe(1); - expect(stdout).toContain("no-eval"); - expect(stdout).toContain("no-console"); - expect(stderr).not.toContain("unverified"); - } finally { - await server.close(); - } - }); - - it("forces the logged-out path under --anonymous even with a token", async () => { - const server = await startMockServer(() => ({ statusCode: 200, body: {} })); - try { - const { stdout, exitCode } = await runCli( - ["check", "-d", directory, "--anonymous"], - { TASKLESS_TOKEN: "fake.token.value", TASKLESS_API_URL: server.apiUrl } - ); - expect(server.requests).toHaveLength(0); // --anonymous skips reconcile - expect(exitCode).toBe(1); - expect(stdout).toContain("no-eval"); - expect(stdout).toContain("no-console"); - } finally { - await server.close(); - } - }); -}); diff --git a/packages/cli/test/run-set.test.ts b/packages/cli/test/run-set.test.ts deleted file mode 100644 index eeddc9c6..00000000 --- a/packages/cli/test/run-set.test.ts +++ /dev/null @@ -1,131 +0,0 @@ -import { - mkdtemp, - rm, - mkdir, - writeFile, - readdir, - readFile, -} from "node:fs/promises"; -import { join } from "node:path"; -import { tmpdir } from "node:os"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; - -import { canonicalHash } from "../src/rules/rule-hash"; -import { - materializeRunDirectory, - selectRunFiles, - signRuleFiles, - type SignedRuleFile, -} from "../src/rules/run-set"; - -async function writeRule( - cwd: string, - name: string, - body: string -): Promise { - const rulesDirectory = join(cwd, ".taskless", "rules"); - await mkdir(rulesDirectory, { recursive: true }); - await writeFile(join(rulesDirectory, name), body, "utf8"); -} - -describe("signRuleFiles", () => { - let directory: string; - beforeEach(async () => { - directory = await mkdtemp(join(tmpdir(), "taskless-runset-")); - }); - afterEach(async () => { - await rm(directory, { recursive: true, force: true }); - }); - - it("signs each rule file with its canonical signature", async () => { - await writeRule(directory, "a.yml", "id: a\n"); - await writeRule(directory, "b.yml", "id: b\n"); - - const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); - - expect(signed.map((s) => s.file).toSorted()).toEqual(["a.yml", "b.yml"]); - for (const entry of signed) { - const expected = await canonicalHash(await readFile(entry.path, "utf8")); - expect(entry.signature).toBe(expected); - } - }); -}); - -describe("selectRunFiles", () => { - const signed: SignedRuleFile[] = [ - { file: "a.yml", path: "/x/a.yml", signature: "1;h=sha-256;d=aaaa" }, - { file: "b.yml", path: "/x/b.yml", signature: "1;h=sha-256;d=bbbb" }, - ]; - - it("selects only files whose signature the server blessed", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "a", file: "a.yml", signature: "1;h=sha-256;d=aaaa" }, - ]); - expect(runFiles.map((s) => s.file)).toEqual(["a.yml"]); - }); - - it("matches by signature, not path (a moved-but-unchanged file resolves)", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "b", file: "moved/b.yml", signature: "1;h=sha-256;d=bbbb" }, - ]); - expect(runFiles.map((s) => s.file)).toEqual(["b.yml"]); - }); - - it("drops run entries with no local signature match", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "c", file: "c.yml", signature: "1;h=sha-256;d=cccc" }, - ]); - expect(runFiles).toEqual([]); - }); - - it("does not select the same local file twice", () => { - const runFiles = selectRunFiles(signed, [ - { ruleId: "a", file: "a.yml", signature: "1;h=sha-256;d=aaaa" }, - { ruleId: "a", file: "a-copy.yml", signature: "1;h=sha-256;d=aaaa" }, - ]); - expect(runFiles.map((s) => s.file)).toEqual(["a.yml"]); - }); -}); - -describe("materializeRunDirectory", () => { - let directory: string; - beforeEach(async () => { - directory = await mkdtemp(join(tmpdir(), "taskless-runset-")); - }); - afterEach(async () => { - await rm(directory, { recursive: true, force: true }); - }); - - it("copies only the selected files and gitignores .run/", async () => { - await writeRule(directory, "a.yml", "id: a\n"); - await writeRule(directory, "b.yml", "id: b\n"); - const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); - const onlyA = signed.filter((s) => s.file === "a.yml"); - - await materializeRunDirectory(directory, onlyA); - - const runRules = join(directory, ".taskless", ".run", "rules"); - expect(await readdir(runRules)).toEqual(["a.yml"]); - - const gitignore = await readFile( - join(directory, ".taskless", ".gitignore"), - "utf8" - ); - expect(gitignore).toContain(".run/"); - }); - - it("rebuilds the run dir, dropping stale files from a prior run", async () => { - await writeRule(directory, "a.yml", "id: a\n"); - await writeRule(directory, "b.yml", "id: b\n"); - const signed = await signRuleFiles(directory, ["a.yml", "b.yml"]); - - await materializeRunDirectory(directory, signed); // both - await materializeRunDirectory( - directory, - signed.filter((s) => s.file === "b.yml") - ); // only b - - const runRules = join(directory, ".taskless", ".run", "rules"); - expect(await readdir(runRules)).toEqual(["b.yml"]); - }); -}); From cf9901d98a3f153a3c0df09d3eaa0e694062562a Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 10:44:04 -0700 Subject: [PATCH 14/21] test(cli): cover runtime rule execution; document the two rule kinds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Groups 6-7 of runtime-rule-execution. Docs (Group 6): - check.txt (topic v2): static rules always run; runtime check.ts runs only when server-verified; the mode table, --dangerously-run-scripts, --timeout, and the --json skipped array. - ci.txt: unauthenticated CI runs static rules and skips runtime; the TASKLESS_TOKEN backstop is the enforcement point for runtime check.ts. Tests (Group 7): - runtime-harness.test.ts: discovery, gate-on-zero-matches (check never invoked), match normalization + Finding->CheckResult indexing, throwing-check isolation, timeout -> error finding. - runtime-check.test.ts: end-to-end dispatch via the built CLI with a mock reconcile server + git origin — authed-blessed, empty-run withheld, logged-out and --anonymous skip + report, reconcile-unavailable skips, dangerously-run- scripts runs offline; asserts static always runs and only check.ts is reported. Also fix a real timeout bug found by the harness test: tsx re-execs node as a grandchild, so spawn detached and SIGKILL the whole process group — otherwise a runaway check keeps running past the timeout. Full suite green (338 tests). Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../changes/runtime-rule-execution/tasks.md | 14 +- packages/cli/src/help/check.txt | 78 +++-- packages/cli/src/help/ci.txt | 19 +- packages/cli/src/rules/runtime/invoke.ts | 14 +- packages/cli/test/runtime-check.test.ts | 284 ++++++++++++++++++ packages/cli/test/runtime-harness.test.ts | 190 ++++++++++++ 6 files changed, 554 insertions(+), 45 deletions(-) create mode 100644 packages/cli/test/runtime-check.test.ts create mode 100644 packages/cli/test/runtime-harness.test.ts diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/runtime-rule-execution/tasks.md index 5af550d0..8a072450 100644 --- a/openspec/changes/runtime-rule-execution/tasks.md +++ b/openspec/changes/runtime-rule-execution/tasks.md @@ -33,13 +33,13 @@ ## 6. Help & docs -- [ ] 6.1 Update `packages/cli/src/help/check.txt` with a runtime-rule section: what runs per mode, and the `--dangerously-run-scripts` warning. -- [ ] 6.2 Update `packages/cli/src/help/ci.txt` to note the enforced (`--enforce`) sandbox as the authoritative runtime-rule enforcement point. +- [x] 6.1 Updated `check.txt` (topic v2): the two rule kinds, what runs per mode, the `--dangerously-run-scripts` warning, `--timeout`, and the `--json` `skipped` array. +- [x] 6.2 Updated `ci.txt` step 7: static rules always run unauthenticated; the `TASKLESS_TOKEN` backstop is the authoritative enforcement point for runtime `check.ts`. ## 7. Tests & verification -- [ ] 7.1 Harness unit tests: narrow assembly (anchor/broad), gate-on-zero-matches (check never invoked), match normalization (hashed id → model name), `Finding` → `CheckResult` mapping, throwing-check isolation, timeout → error finding. -- [ ] 7.2 Mode integration tests (subprocess against `dist/`, temp-dir fixtures with a mock reconcile server + git origin): authed runs blessed runtime rules; partially-blessed rule withheld; logged-out and `--anonymous` skip runtime + report skipped; reconcile-unavailable skips runtime; `--dangerously-run-scripts` runs runtime offline with a warning. -- [ ] 7.3 Static-always-run tests: static rules run in every mode; static rules are not reported to reconcile. -- [ ] 7.4 `--json` shape tests: warnings/notices suppressed; runtime findings appear under the same `results` shape as static findings. -- [ ] 7.5 Run `pnpm typecheck`, `pnpm lint`, and the full `pnpm test` — all green. +- [x] 7.1 `test/runtime-harness.test.ts` (imports src, real ast-grep + tsx): discovery, gate-on-zero-matches (check never invoked), match normalization + `Finding`→`CheckResult` indexing, throwing-check isolation, timeout → error finding. +- [x] 7.2 `test/runtime-check.test.ts` (subprocess against `dist/`, temp fixtures + mock reconcile server + git origin): authed runs blessed rules; empty-run withholds; logged-out and `--anonymous` skip + report; reconcile-unavailable (503) skips; `--dangerously-run-scripts` runs offline with a warning. +- [x] 7.3 Static-always-run asserted in every mode; reconcile receives ONLY the runtime `check.ts` (never the static YAML). +- [x] 7.4 `--json` shape: warnings/notices suppressed; runtime findings share the `results` shape; the optional `skipped` array present when runtime rules don't run. +- [x] 7.5 `pnpm --filter @taskless/cli typecheck`, `pnpm lint`, and full `pnpm test` (338 tests) — all green. Also fixed a real timeout bug (SIGKILL the tsx process _group_, not just the wrapper, so a runaway check is actually terminated). diff --git a/packages/cli/src/help/check.txt b/packages/cli/src/help/check.txt index 937dadf7..10f38fec 100644 --- a/packages/cli/src/help/check.txt +++ b/packages/cli/src/help/check.txt @@ -1,37 +1,52 @@ -# Topic: check (CLI v%(CLI_VERSION)s / topic v1) +# Topic: check (CLI v%(CLI_VERSION)s / topic v2) ## Goal -Run the applicable rules in `.taskless/rules/` against the codebase and -report matches. Used standalone (full project scan), in CI (diff-only -scan), or after rule create/improve to validate. +Run the applicable rules against the codebase and report matches. Two +rule kinds run: **static** ast-grep rules in `.taskless/rules/`, and +**runtime** rules in `.taskless/runtime-rules/` (a directory of ast-grep +capture rules plus a `check.ts`). Used standalone (full project scan), +in CI (diff-only scan), or after rule create/improve to validate. ## Preconditions - `.taskless/` directory exists. -- At least one rule exists in `.taskless/rules/`. (If none exist, - the CLI exits 0 with a friendly message suggesting - `taskless rule create`.) -- No auth required. What actually runs depends on auth state — see - "What runs (auth state)". - -## What runs (auth state) - -`check` never requires auth, but auth state decides which rules execute: - -- **Logged out** (or `--anonymous`) — scans every rule in - `.taskless/rules/` locally, with no network call. The offline linter - posture; no warnings. -- **Logged in** — reconciles the local rule files against the Taskless - service and runs ONLY the server-blessed `run` set. Files that drift - from the blessed copy (`unsafe`), were never issued (`unknown`), or - are expected but absent locally (`missing`) are reported as warnings - and do NOT run. If reconciliation can't complete (no GitHub remote, - service unavailable/not deployed, or a rejected token), `check` warns - that verification was skipped and falls back to a full local scan. - -Warnings are human-readable stderr only: they never change the exit -code and never appear in `--json` output (which stays the -`{ success, results }` shape). The authoritative allow-list is the -server's; the CI backstop (`taskless help ci`) is the enforcement point. +- At least one rule exists in `.taskless/rules/` or + `.taskless/runtime-rules/`. (If none exist, the CLI exits 0 with a + friendly message suggesting `taskless rule create`.) +- No auth required. Static rules always run; whether runtime rules run + depends on auth state — see "What runs". + +## What runs + +`check` never requires auth. The two rule kinds run differently: + +- **Static rules** (`.taskless/rules/*.yml`) are inert ast-grep patterns + and **always run**, in every mode, with no network call. The offline + linter posture. +- **Runtime rules** (`.taskless/runtime-rules//`) execute a + `check.ts` — arbitrary code — so they run ONLY when that code is + verified: + - **Logged in** (token or API key) — each rule's `check.ts` is + reconciled against the Taskless service; rules the server blessed + (`run`) execute, and the rest are withheld and reported (advisory). + - **Logged out, `--anonymous`, no GitHub remote, or service + unavailable** — runtime rules are **skipped** (reported, never run). + Static rules still run. + - **`--dangerously-run-scripts`** — runs every runtime rule trusting + local signatures, with no network call, behind a prominent warning. + This is the only way to run runtime rules unverified. + +Notices about skipped/withheld runtime rules are human-readable stderr +only: they never change the exit code. Under `--json` they do NOT appear +as warnings; instead an additive optional `skipped: [{ rule, reason }]` +array is included alongside the unchanged `{ success, results }`. The +authoritative allow-list is the server's; the CI backstop +(`taskless help ci`) is the enforcement point for runtime rules. + +## Flags +- `--json` — machine output (`{ success, results, skipped? }`). +- `--anonymous` — run only static rules; skip runtime rules. +- `--dangerously-run-scripts` — run runtime `check.ts` unverified. +- `--timeout ` — per-runtime-check wall-clock bound (default 10). ## Steps @@ -54,6 +69,11 @@ server's; the CI backstop (`taskless help ci`) is the enforcement point. Paths that don't exist on disk are silently filtered, so you can pipe raw `git diff` output directly without pre-filtering. + When runtime rules were present but not run (e.g. logged out), the + JSON also carries `"skipped": [{ "rule": "", "reason": "…" }]` + alongside `success`/`results`. Surface it so CI can tell that runtime + rules did not execute; it never affects the exit code. + 3. **Parse the JSON output.** Shape: ```json { diff --git a/packages/cli/src/help/ci.txt b/packages/cli/src/help/ci.txt index 087baa05..ded38138 100644 --- a/packages/cli/src/help/ci.txt +++ b/packages/cli/src/help/ci.txt @@ -161,13 +161,18 @@ different structure for CircleCI. The six steps stay the same. `taskless check` does NOT require authentication. The generated CI config works out of the box with no secrets and scans all local rules. -Optionally, exposing a `TASKLESS_TOKEN` secret turns CI into the -**backstop**: an authenticated `check` reconciles the repo's rule files -against the Taskless service and runs exactly the server-blessed `run` -set, warning on drift (`unsafe`), unissued files (`unknown`), or -missing rules. This is the enforcement point for the server-owned rule -set — local developer runs are advisory. To wire it, set the token as -an env var on the check step (GitHub Actions): +Static ast-grep rules always run in CI with no secrets. **Runtime +rules** (`.taskless/runtime-rules/`, which execute a `check.ts`) only +run when their code is server-verified — so an unauthenticated CI job +runs the static rules and skips the runtime ones. + +Exposing a `TASKLESS_TOKEN` secret turns CI into the **backstop** for +runtime rules: an authenticated `check` reconciles each runtime rule's +`check.ts` against the Taskless service and runs exactly the +server-blessed set, withholding any that drift or were never issued. +This is the enforcement point for runtime rules — local developer runs +skip them unless `--dangerously-run-scripts` is passed. To wire it, set +the token as an env var on the check step (GitHub Actions): ```yaml - name: Taskless check diff --git a/packages/cli/src/rules/runtime/invoke.ts b/packages/cli/src/rules/runtime/invoke.ts index f44b6d09..60d73b56 100644 --- a/packages/cli/src/rules/runtime/invoke.ts +++ b/packages/cli/src/rules/runtime/invoke.ts @@ -89,16 +89,26 @@ export async function invokeCheck( await writeFile(runnerPath, RUNNER_SOURCE); const result = await new Promise((resolvePromise) => { + // `detached` makes the child its own process-group leader so a timeout + // can SIGKILL the whole tree — `tsx` re-execs node as a grandchild, and + // killing only the wrapper would leave a runaway check running. const child = spawn( process.execPath, [tsxCli, runnerPath, checkFile, root, matchesPath, outPath], - { stdio: ["ignore", "ignore", "pipe"] } + { stdio: ["ignore", "ignore", "pipe"], detached: true } ); const stderrChunks: string[] = []; let timedOut = false; + const killTree = () => { + try { + if (child.pid !== undefined) process.kill(-child.pid, "SIGKILL"); + } catch { + child.kill("SIGKILL"); + } + }; const timer = setTimeout(() => { timedOut = true; - child.kill("SIGKILL"); + killTree(); }, timeoutMs); child.stderr.on("data", (chunk: Buffer) => diff --git a/packages/cli/test/runtime-check.test.ts b/packages/cli/test/runtime-check.test.ts new file mode 100644 index 00000000..7f803104 --- /dev/null +++ b/packages/cli/test/runtime-check.test.ts @@ -0,0 +1,284 @@ +import { execFile } from "node:child_process"; +import { createServer, type Server } from "node:http"; +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { resolve, join } from "node:path"; +import { tmpdir } from "node:os"; +import { promisify } from "node:util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const execFileAsync = promisify(execFile); +const binPath = resolve(import.meta.dirname, "../dist/index.js"); + +interface ReportedFile { + file: string; + signature: string; +} +interface ReconcileRequestBody { + repositoryUrl: string; + files: ReportedFile[]; +} +type Responder = (request: ReconcileRequestBody) => { + statusCode: number; + body?: unknown; +}; +interface MockServer { + apiUrl: string; + requests: ReconcileRequestBody[]; + close: () => Promise; +} + +/** Start a mock reconcile endpoint on a random port. */ +function startMockServer(responder: Responder): Promise { + const requests: ReconcileRequestBody[] = []; + const server: Server = createServer((request, response) => { + if (request.method !== "POST" || request.url !== "/cli/api/reconcile") { + response.writeHead(404).end("{}"); + return; + } + let raw = ""; + request.on("data", (chunk: Buffer) => (raw += chunk.toString())); + request.on("end", () => { + const parsed = JSON.parse(raw) as ReconcileRequestBody; + requests.push(parsed); + const { statusCode, body } = responder(parsed); + response.writeHead(statusCode, { "content-type": "application/json" }); + response.end(JSON.stringify(body ?? {})); + }); + }); + return new Promise((resolvePromise) => { + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + const port = typeof address === "object" && address ? address.port : 0; + resolvePromise({ + apiUrl: `http://127.0.0.1:${String(port)}/cli`, + requests, + close: () => new Promise((done) => server.close(() => done())), + }); + }); + }); +} + +/** Echo a reported file's signature back so the mock can bless it. */ +function sig(request: ReconcileRequestBody, endsWith: string): string { + return request.files.find((f) => f.file.endsWith(endsWith))?.signature ?? ""; +} + +async function runCli( + args: string[], + env: Record = {} +): Promise<{ stdout: string; stderr: string; exitCode: number }> { + try { + const { stdout, stderr } = await execFileAsync("node", [binPath, ...args], { + env: { ...process.env, ...env }, + }); + return { stdout, stderr, exitCode: 0 }; + } catch (error) { + const execError = error as { stdout: string; stderr: string; code: number }; + return { + stdout: execError.stdout ?? "", + stderr: execError.stderr ?? "", + exitCode: execError.code, + }; + } +} + +/** The check `--json` line, ignoring any preceding migration output. */ +function parseJson(stdout: string): { + success: boolean; + results: { source: string; ruleId: string }[]; + skipped?: { rule: string; reason: string }[]; +} { + const line = stdout + .trim() + .split("\n") + .findLast((l) => l.trim().startsWith("{")); + return JSON.parse(line ?? "{}") as { + success: boolean; + results: { source: string; ruleId: string }[]; + skipped?: { rule: string; reason: string }[]; + }; +} + +const STATIC_RULE = [ + "id: no-console", + "language: typescript", + "severity: warning", + "rule:", + " pattern: console.log($$$A)", + "message: avoid console.log", + "", +].join("\n"); + +const RUNTIME_CAPTURE = [ + "id: logs-abc12345", + "language: typescript", + "rule:", + " pattern: console.log($A)", + "metadata:", + " taskless:", + " version: 1", + " kind: runtime", + " name: logs", + " check: check.ts", + " match: anchor", + "", +].join("\n"); + +const RUNTIME_CHECK = `export default async function (root, matches) { + return matches.map((m) => ({ file: m.file, line: m.line, message: "runtime " + m.rule, severity: "warning" })); +} +`; + +const CHECK_REPORT_PATH = ".taskless/runtime-rules/demo/check.ts"; + +describe("check: static vs runtime dispatch", () => { + let directory: string; + + beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "tskl-rt-check-")); + const rules = join(directory, ".taskless", "rules"); + const runtime = join(directory, ".taskless", "runtime-rules", "demo"); + await mkdir(rules, { recursive: true }); + await mkdir(runtime, { recursive: true }); + await writeFile(join(rules, "no-console.yml"), STATIC_RULE, "utf8"); + await writeFile(join(runtime, "logs.yml"), RUNTIME_CAPTURE, "utf8"); + await writeFile(join(runtime, "check.ts"), RUNTIME_CHECK, "utf8"); + await writeFile(join(directory, "src.ts"), 'console.log("hi");\n', "utf8"); + await execFileAsync("git", ["init"], { cwd: directory }); + await execFileAsync( + "git", + ["remote", "add", "origin", "https://github.com/acme/widgets.git"], + { cwd: directory } + ); + }); + + afterEach(async () => { + await rm(directory, { recursive: true, force: true }); + }); + + it("logged out: static runs, runtime is skipped and reported in --json", async () => { + const { stdout, exitCode } = await runCli([ + "check", + "-d", + directory, + "--json", + ]); + const output = parseJson(stdout); + expect(exitCode).toBe(0); // only warnings + const ids = new Set(output.results.map((r) => r.ruleId)); + expect(ids.has("no-console")).toBe(true); // static always runs + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + false + ); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + }); + + it("logged out: skip notice on stderr, static findings on stdout", async () => { + const { stdout, stderr } = await runCli(["check", "-d", directory]); + expect(stderr).toContain("was not run"); + expect(stdout).toContain("no-console"); + }); + + it("authed + blessed check.ts: runtime runs; only check.ts is reported", async () => { + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + run: [ + { + ruleId: "demo", + file: CHECK_REPORT_PATH, + signature: sig(request, "check.ts"), + }, + ], + unsafe: [], + unknown: [], + missing: [], + }, + })); + try { + const { stdout } = await runCli(["check", "-d", directory, "--json"], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + }); + // Only the runtime check.ts is reported — never the static rule. + expect(server.requests).toHaveLength(1); + expect(server.requests[0]!.files).toHaveLength(1); + expect(server.requests[0]!.files[0]!.file.endsWith("check.ts")).toBe( + true + ); + const output = parseJson(stdout); + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + true + ); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + } finally { + await server.close(); + } + }); + + it("authed + empty run set: runtime withheld, static still runs", async () => { + const server = await startMockServer(() => ({ + statusCode: 200, + body: { run: [], unsafe: [], unknown: [], missing: [] }, + })); + try { + const { stdout } = await runCli(["check", "-d", directory, "--json"], { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + }); + const output = parseJson(stdout); + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + false + ); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + } finally { + await server.close(); + } + }); + + it("reconcile unavailable: runtime skipped, static runs, exit 0", async () => { + const server = await startMockServer(() => ({ statusCode: 503 })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + const output = parseJson(stdout); + expect(exitCode).toBe(0); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + } finally { + await server.close(); + } + }); + + it("--anonymous with a token: skips runtime and never calls reconcile", async () => { + const server = await startMockServer(() => ({ + statusCode: 200, + body: { run: [], unsafe: [], unknown: [], missing: [] }, + })); + try { + const { stdout } = await runCli( + ["check", "-d", directory, "--json", "--anonymous"], + { TASKLESS_TOKEN: "fake.token", TASKLESS_API_URL: server.apiUrl } + ); + expect(server.requests).toHaveLength(0); + const output = parseJson(stdout); + expect(output.skipped?.some((s) => s.rule === "demo")).toBe(true); + } finally { + await server.close(); + } + }); + + it("--dangerously-run-scripts: runs runtime offline behind a warning", async () => { + const { stdout, stderr } = await runCli([ + "check", + "-d", + directory, + "--dangerously-run-scripts", + ]); + expect(stderr).toContain("dangerously-run-scripts"); + expect(stdout).toContain("demo"); // runtime finding surfaced + }); +}); diff --git a/packages/cli/test/runtime-harness.test.ts b/packages/cli/test/runtime-harness.test.ts new file mode 100644 index 00000000..5568fef2 --- /dev/null +++ b/packages/cli/test/runtime-harness.test.ts @@ -0,0 +1,190 @@ +import { mkdtemp, rm, mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { discoverRuntimeRules } from "../src/rules/runtime/discover"; +import { executeRuntimeRule } from "../src/rules/runtime/harness"; + +/** Build a capture `*.yml` with a `metadata.taskless` runtime block. */ +function capture(options: { + id: string; + name: string; + pattern: string; + match?: "anchor" | "broad"; + rule?: string; +}): string { + const ruleBlock = options.rule ?? ` pattern: ${options.pattern}`; + return [ + `id: ${options.id}`, + "language: typescript", + "rule:", + ruleBlock, + "metadata:", + " taskless:", + " version: 1", + " kind: runtime", + ` name: ${options.name}`, + " check: check.ts", + ` match: ${options.match ?? "anchor"}`, + "", + ].join("\n"); +} + +/** Write a runtime rule directory (capture files + check.ts) under the repo. */ +async function writeRuntimeRule( + root: string, + name: string, + captures: Record, + check: string +): Promise { + const directory = join(root, ".taskless", "runtime-rules", name); + await mkdir(directory, { recursive: true }); + for (const [file, body] of Object.entries(captures)) { + await writeFile(join(directory, file), body, "utf8"); + } + await writeFile(join(directory, "check.ts"), check, "utf8"); +} + +/** A check that echoes each match back as a finding (proves it was invoked). */ +const ECHO_CHECK = `export default async function (root, matches) { + return matches.map((m) => ({ + file: m.file, + line: m.line, + column: m.column, + message: "rule=" + m.rule + " cap=" + JSON.stringify(m.captures), + severity: "warning", + })); +} +`; + +describe("runtime harness", () => { + let root: string; + + beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "tskl-harness-")); + }); + afterEach(async () => { + await rm(root, { recursive: true, force: true }); + }); + + it("discovers a runtime rule directory", async () => { + await writeRuntimeRule( + root, + "logs", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + ECHO_CHECK + ); + const rules = await discoverRuntimeRules(root); + expect(rules).toHaveLength(1); + expect(rules[0]!.captureRules[0]!.name).toBe("logs"); + expect(rules[0]!.captureRules[0]!.match).toBe("anchor"); + expect(rules[0]!.checkFile.endsWith("check.ts")).toBe(true); + }); + + it("gates on matches: zero matches never invokes check.ts", async () => { + // The source has no console.log, so the narrow yields nothing and the echo + // check (which would return a finding per match) must never run. + await writeFile(join(root, "src.ts"), "const x = 1;\n", "utf8"); + await writeRuntimeRule( + root, + "logs", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + ECHO_CHECK + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!); + expect(results).toHaveLength(0); + }); + + it("normalizes matches (1-indexed, rule name, captures) and maps findings to 0-indexed range", async () => { + await writeFile( + join(root, "src.ts"), + 'const x = 1;\nconsole.log("hi");\n', + "utf8" + ); + await writeRuntimeRule( + root, + "logs", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + ECHO_CHECK + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!); + expect(results).toHaveLength(1); + const finding = results[0]!; + expect(finding.source).toBe("taskless-runtime"); + expect(finding.ruleId).toBe("logs"); + expect(finding.file).toBe("src.ts"); + // The match is on line 2 (1-indexed); the echo check returns m.line, which + // the harness converts down to the 0-indexed CheckResult range. + expect(finding.range.start.line).toBe(1); + expect(finding.message).toContain("rule=logs"); + // captures echoed through: the single metavariable A holds the string "hi". + expect(finding.message).toContain("cap="); + expect(finding.message).toContain("hi"); + }); + + it("isolates a throwing check into a single error finding", async () => { + await writeFile(join(root, "src.ts"), 'console.log("hi");\n', "utf8"); + await writeRuntimeRule( + root, + "boom", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + `export default async function () { throw new Error("kaboom"); }\n` + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!); + expect(results).toHaveLength(1); + expect(results[0]!.severity).toBe("error"); + expect(results[0]!.message).toContain("boom"); + }); + + it("times out a slow check into an error finding", async () => { + await writeFile(join(root, "src.ts"), 'console.log("hi");\n', "utf8"); + await writeRuntimeRule( + root, + "slow", + { + "logs.yml": capture({ + id: "logs-a1", + name: "logs", + pattern: "console.log($A)", + }), + }, + `export default async function () { + await new Promise((r) => setTimeout(r, 5000)); + return []; + }\n` + ); + const [rule] = await discoverRuntimeRules(root); + const results = await executeRuntimeRule(root, rule!, { timeoutMs: 200 }); + expect(results).toHaveLength(1); + expect(results[0]!.severity).toBe("error"); + expect(results[0]!.message).toContain("timed out"); + }, 15_000); +}); From 829d2a05f7ed938847c81c4ea5eb217661858024 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 12:23:51 -0700 Subject: [PATCH 15/21] docs(openspec): correct narrow spec to match the config-based implementation Verification (opsx:verify) caught a spec-vs-impl divergence: the narrow requirement, proposal, and design named `--inline-rules --json=stream`, but the harness assembles the capture rules into a temp `--config` (--inline-rules carries only one rule; a runtime rule has multiple capture rules + full ast-grep config). Reworded to "one scan per mode" via a generated config so the spec that gets synced to canonical on archive matches reality. Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../changes/runtime-rule-execution/design.md | 14 +++++++++----- .../changes/runtime-rule-execution/proposal.md | 4 ++-- .../specs/cli-runtime-rule-execution/spec.md | 16 +++++++++------- 3 files changed, 20 insertions(+), 14 deletions(-) diff --git a/openspec/changes/runtime-rule-execution/design.md b/openspec/changes/runtime-rule-execution/design.md index 4c31d10c..c25d1781 100644 --- a/openspec/changes/runtime-rule-execution/design.md +++ b/openspec/changes/runtime-rule-execution/design.md @@ -17,8 +17,8 @@ hashed, globally-unique `id` (`${ruleSlug}-${sha1(ruleBody).slice(0,8)}`) for sc attribution, and a stable model-assigned `name` the check branches on. Capture rules may carry full ast-grep config (`constraints`/`utils`/`transform` as **siblings** of `rule`). -The local harness is resolved in TSKL-245: assemble a rule's capture rules → **one** -`ast-grep scan` (anchor `--inline-rules --json=stream`; broad `--files-with-matches`) → gate +The local harness is resolved in TSKL-245: assemble a rule's capture rules → an +`ast-grep scan` (one per mode: anchor `--json=stream`; broad `--files-with-matches`) → gate on matches → invoke `check.ts`'s **default export** with `(root, matches)` via a bundled, pinned `tsx`; use the **returned** `Finding[]`. `Finding.severity ∈ error|warning|info` maps onto static-rule gating with no translation. Measured cost is `tsx` per-worker startup @@ -125,9 +125,13 @@ would run unverified code the user never opted into. ### Decision: The narrow → gate → `check.ts` harness, per TSKL-245 -For each executable runtime rule: collect its capture rules and run **one** `ast-grep` scan — -`--inline-rules --json=stream` in anchor mode, `--files-with-matches` in broad mode -(`kind: program` enumerators). Normalize each match to +For each executable runtime rule: collect its capture rules into a generated ast-grep config +(a temp `ruleDirs` config — `--inline-rules` carries only a single rule, and a runtime rule has +multiple capture rules plus full `constraints`/`utils`/`transform`) and run **one scan per +mode** — `--json=stream` for anchor capture rules, `--files-with-matches` for broad +(`kind: program`) enumerators. An all-anchor rule is a single scan; mixing modes is one scan +per mode (broad is kept separate so a `kind: program` rule isn't streamed as whole-file text). +Normalize each match to `{ rule, ruleId, file (root-relative), line (1-indexed), column, text, captures }`, mapping the hashed `ruleId` back to the model `name` as `match.rule`. If there are **zero** matches, `check.ts` is not invoked. Otherwise invoke its **default export** as a function with diff --git a/openspec/changes/runtime-rule-execution/proposal.md b/openspec/changes/runtime-rule-execution/proposal.md index 84918445..76c1aa2b 100644 --- a/openspec/changes/runtime-rule-execution/proposal.md +++ b/openspec/changes/runtime-rule-execution/proposal.md @@ -20,8 +20,8 @@ the default safety mechanism is a server-validated signature, not a local sandbo ## What Changes - Add a **`cli-runtime-rule-execution`** capability: recognize a runtime-rule directory - (`metadata.taskless.kind: runtime`), run its capture rules as **one** `ast-grep` narrow - (anchor `--inline-rules --json=stream`; broad `--files-with-matches` for `kind: program` + (`metadata.taskless.kind: runtime`), run its capture rules as an `ast-grep` narrow (one scan + per mode: anchor `--json=stream`; broad `--files-with-matches` for `kind: program` enumerators), **gate on matches**, and only then invoke `check.ts`'s default export with `(root, matches)` via a CLI-bundled, pinned `tsx`. Zero matches ⇒ `check.ts` is never invoked. Normalize matches to `{ rule, ruleId, file, line, column, text, captures }` diff --git a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md b/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md index 000566cc..0c23383a 100644 --- a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md +++ b/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md @@ -23,10 +23,12 @@ SHALL NOT be executed by `check`. ### Requirement: The harness narrows with one ast-grep scan and gates on matches -For a runtime rule the CLI SHALL assemble the rule's capture rules and run **one** `ast-grep` -scan as the narrow: `--inline-rules --json=stream` in `anchor` mode, and -`--files-with-matches` in `broad` mode (whole-language `kind: program` enumerators). When the -narrow produces **zero** matches the CLI SHALL NOT invoke `check.ts`. +For a runtime rule the CLI SHALL assemble the rule's capture rules into an ast-grep +configuration and run **one scan per mode** as the narrow: `--json=stream` for `anchor` capture +rules, and `--files-with-matches` for `broad` capture rules (whole-language `kind: program` +enumerators). A rule with only `anchor` capture rules therefore runs in a single scan; a rule +mixing modes runs one scan per mode. When the narrow produces **zero** matches the CLI SHALL +NOT invoke `check.ts`. #### Scenario: Zero matches skips the check @@ -34,10 +36,10 @@ narrow produces **zero** matches the CLI SHALL NOT invoke `check.ts`. - **THEN** the CLI SHALL NOT invoke that rule's `check.ts` - **AND** the rule SHALL contribute no findings -#### Scenario: One scan per rule +#### Scenario: Capture rules of a mode run together -- **WHEN** a runtime rule has multiple capture rules -- **THEN** the CLI SHALL run them as a single `ast-grep` scan, not one scan per capture rule +- **WHEN** a runtime rule has multiple `anchor` capture rules +- **THEN** the CLI SHALL run them in a single `ast-grep` scan, not one scan per capture rule ### Requirement: Matches are normalized and attributed to the model name From 9bc23ea1497f9ce95114efb3a5514e9667bc26fb Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Fri, 3 Jul 2026 13:29:45 -0700 Subject: [PATCH 16/21] chore(openspec): sync + archive runtime-rule-execution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sync the runtime-rule-execution delta specs into the canonical specs and archive the change (verify → sync → archive). - cli-runtime-rule-execution: new canonical spec (the runtime harness contract). - cli-check: static-vs-runtime dispatch, the validated-path rule, skip+report, and --dangerously-run-scripts added; auth-state/reconcile/degrade requirements updated for the cutover; the now-obsolete 'warns on reconciliation mismatches' and 'exits cleanly when the run set is empty' requirements removed (their static-reconcile behavior was deleted in the cutover — the delta records the removal with reasons). - cli-rule-reconciliation: reporting + run-set requirements rescoped to each runtime rule's check.ts. Change archived to openspec/changes/archive/2026-07-03-runtime-rule-execution/; no unarchived changes remain, so the tip's check-openspec-archived job passes. Refs TSKL-245 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../.openspec.yaml | 0 .../design.md | 0 .../proposal.md | 0 .../specs/cli-check/spec.md | 15 ++ .../specs/cli-rule-reconciliation/spec.md | 0 .../specs/cli-runtime-rule-execution/spec.md | 0 .../tasks.md | 0 openspec/specs/cli-check/spec.md | 175 +++++++++++++----- .../specs/cli-rule-reconciliation/spec.md | 48 +++-- .../specs/cli-runtime-rule-execution/spec.md | 139 ++++++++++++++ 10 files changed, 311 insertions(+), 66 deletions(-) rename openspec/changes/{runtime-rule-execution => archive/2026-07-03-runtime-rule-execution}/.openspec.yaml (100%) rename openspec/changes/{runtime-rule-execution => archive/2026-07-03-runtime-rule-execution}/design.md (100%) rename openspec/changes/{runtime-rule-execution => archive/2026-07-03-runtime-rule-execution}/proposal.md (100%) rename openspec/changes/{runtime-rule-execution => archive/2026-07-03-runtime-rule-execution}/specs/cli-check/spec.md (92%) rename openspec/changes/{runtime-rule-execution => archive/2026-07-03-runtime-rule-execution}/specs/cli-rule-reconciliation/spec.md (100%) rename openspec/changes/{runtime-rule-execution => archive/2026-07-03-runtime-rule-execution}/specs/cli-runtime-rule-execution/spec.md (100%) rename openspec/changes/{runtime-rule-execution => archive/2026-07-03-runtime-rule-execution}/tasks.md (100%) create mode 100644 openspec/specs/cli-runtime-rule-execution/spec.md diff --git a/openspec/changes/runtime-rule-execution/.openspec.yaml b/openspec/changes/archive/2026-07-03-runtime-rule-execution/.openspec.yaml similarity index 100% rename from openspec/changes/runtime-rule-execution/.openspec.yaml rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/.openspec.yaml diff --git a/openspec/changes/runtime-rule-execution/design.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md similarity index 100% rename from openspec/changes/runtime-rule-execution/design.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md diff --git a/openspec/changes/runtime-rule-execution/proposal.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/proposal.md similarity index 100% rename from openspec/changes/runtime-rule-execution/proposal.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/proposal.md diff --git a/openspec/changes/runtime-rule-execution/specs/cli-check/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md similarity index 92% rename from openspec/changes/runtime-rule-execution/specs/cli-check/spec.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md index 98f04e54..026d6c52 100644 --- a/openspec/changes/runtime-rule-execution/specs/cli-check/spec.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md @@ -165,3 +165,18 @@ non-zero code solely because reconciliation failed, and the warning SHALL be sup - **WHEN** the CLI degrades and `--json` is set - **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape - **AND** SHALL NOT contain the human-readable degrade warning + +## REMOVED Requirements + +### Requirement: Check warns on reconciliation mismatches + +**Reason**: The cutover scopes reconciliation to runtime `check.ts` only; static rules are no +longer reconciled. Runtime-rule mismatches (a `check.ts` that is `unsafe`/`unknown`/`missing`) +are now surfaced as withheld/skipped notices and the `--json` `skipped` array (see the ADDED +requirements), so the static-rule drift/unknown/missing warnings this described no longer exist. + +### Requirement: Check exits cleanly when the run set is empty + +**Reason**: Static rules are no longer gated by a server `run` set — they always scan. An empty +runtime allow-list simply runs no runtime rules (reported via `skipped`/withheld); there is no +static empty-run-set skip behavior left to specify. diff --git a/openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md similarity index 100% rename from openspec/changes/runtime-rule-execution/specs/cli-rule-reconciliation/spec.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md diff --git a/openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md similarity index 100% rename from openspec/changes/runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md diff --git a/openspec/changes/runtime-rule-execution/tasks.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/tasks.md similarity index 100% rename from openspec/changes/runtime-rule-execution/tasks.md rename to openspec/changes/archive/2026-07-03-runtime-rule-execution/tasks.md diff --git a/openspec/specs/cli-check/spec.md b/openspec/specs/cli-check/spec.md index b3bf88ea..a519018f 100644 --- a/openspec/specs/cli-check/spec.md +++ b/openspec/specs/cli-check/spec.md @@ -247,83 +247,164 @@ When `taskless check --json` exits with an error, the output SHALL conform to th ### Requirement: Check selects what it runs from auth state -`taskless check` SHALL NOT require authentication, and it SHALL choose what it runs from the current auth state. When no token is available, or when `--anonymous` is set, the CLI SHALL run all local rule files under `.taskless/rules/` without contacting the server (the offline linter posture). When a token is available and `--anonymous` is not set, the CLI SHALL reconcile against the server (per "Check reconciles rule files before scanning") and run only the returned `run` set. The unauthenticated path SHALL succeed with no network access and SHALL NOT emit a warning; any informational notice on that path is optional. - -#### Scenario: Unauthenticated check runs local rules +`taskless check` SHALL NOT require authentication, and it SHALL choose what it runs from the +current auth state. **Static ast-grep rules** (single `*.yml` files under `.taskless/rules/`) +SHALL always run without contacting the server, on every path (the offline linter posture). +**Runtime rules** (directories with `metadata.taskless.kind: runtime`) SHALL run only on a +signature-validated path: when a token is available and `--anonymous` is not set the CLI SHALL +reconcile and execute the runtime rules fully returned in `run`; when no token is available, +when `--anonymous` is set, or when reconciliation cannot complete, the CLI SHALL skip runtime +execution unless `--dangerously-run-scripts` is set. The unauthenticated path SHALL succeed +with no network access for static rules and SHALL NOT emit a warning about missing +authentication. + +#### Scenario: Unauthenticated check runs static rules and skips runtime rules - **WHEN** a user runs `taskless check` with no available token -- **THEN** the CLI SHALL scan all local rule files -- **AND** SHALL NOT call `POST /cli/api/reconcile` +- **THEN** the CLI SHALL scan all static rule files +- **AND** SHALL NOT call `POST /cli/api/reconcile` for the purpose of running static rules +- **AND** SHALL skip runtime rules - **AND** SHALL NOT emit a warning about missing authentication -#### Scenario: Authenticated check reconciles +#### Scenario: Authenticated check reconciles runtime rules - **WHEN** a user runs `taskless check` with an available token and without `--anonymous` -- **THEN** the CLI SHALL call `POST /cli/api/reconcile` and run only the `run` set +- **THEN** the CLI SHALL reconcile and execute the runtime rules fully returned in `run` +- **AND** SHALL run static rules unconditionally #### Scenario: Anonymous forces the logged-out path - **WHEN** a user runs `taskless check --anonymous` while a token is available -- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (run all local rules, no reconcile call) +- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (static rules run, runtime rules skipped, no reconcile call) ### Requirement: Check reconciles rule files before scanning -`taskless check` SHALL reconcile before scanning whenever a bearer token and a `repositoryUrl` are resolvable and `--anonymous` is not set. It SHALL compute the signature envelope for every `.yml` file under `.taskless/rules/`, call `POST /cli/api/reconcile` with `{ repositoryUrl, files }`, and then scan **only** the files returned in the `run` set, matched back to local files by signature (per the `cli-rule-reconciliation` capability). +`taskless check` SHALL reconcile before running runtime rules whenever a bearer token and a +`repositoryUrl` are resolvable and `--anonymous` is not set. It SHALL compute the signature +envelope for the `check.ts` of every runtime rule under `.taskless/runtime-rules/`, call +`POST /cli/api/reconcile` with `{ repositoryUrl, files }`, and then execute **only** the +runtime rules whose `check.ts` is returned in the `run` set, matched back to local files by +signature (per the `cli-rule-reconciliation` capability). Capture `*.yml` and static rules +SHALL NOT be gated by reconciliation. -#### Scenario: Only blessed rules are scanned +#### Scenario: Only rules with a blessed check.ts execute -- **WHEN** a user runs `taskless check` while authenticated and reconciliation returns a - `run` set that is a subset of the local rule files -- **THEN** the CLI SHALL invoke `sg scan` against only the `run`-set rule files -- **AND** SHALL NOT scan any local rule file absent from `run` +- **WHEN** a user runs `taskless check` while authenticated and reconciliation returns a `run` + set covering the `check.ts` of some runtime rules and not others +- **THEN** the CLI SHALL execute only the runtime rules whose `check.ts` is in `run` +- **AND** SHALL run all static rules regardless of the `run` set -### Requirement: Check warns on reconciliation mismatches +### Requirement: Check degrades to a local scan when reconciliation cannot complete -`taskless check` SHALL warn on every mismatch reported by a successful reconciliation and SHALL NOT let those warnings change the exit code. `unsafe` entries SHALL be warned as tamper/drift, `unknown` as not-issued-by-the-server, and `missing` as audit-only. The exit code SHALL continue to be governed solely by error-severity scan results from the executed `run` set. Warnings SHALL be human-readable output only; when `--json` is set the output SHALL remain the existing `{ success, results }` shape and SHALL NOT include the warnings. +`taskless check` SHALL NOT fail solely because an attempted reconciliation cannot complete. +When a token is available and `--anonymous` is not set but reconciliation cannot complete (no +resolvable git remote, or the reconcile endpoint is unreachable or not yet deployed, or a +transport error), the CLI SHALL warn that rule verification could not be performed, SHALL fall +back to scanning all **static** rule files, and SHALL **skip runtime rules** (their `check.ts` +SHALL NOT run) unless `--dangerously-run-scripts` is set. The CLI SHALL NOT exit with a +non-zero code solely because reconciliation failed, and the warning SHALL be suppressed under +`--json`. -#### Scenario: Unsafe drift is warned without failing the exit code +#### Scenario: Endpoint unreachable degrades static and skips runtime -- **WHEN** reconciliation returns an `unsafe` entry and the `run`-set scan produces no - error-severity results -- **THEN** the CLI SHALL warn about the drift -- **AND** SHALL exit with code 0 +- **WHEN** an authenticated `check` attempts reconciliation and the endpoint is unreachable or returns a not-deployed error +- **THEN** the CLI SHALL warn that verification could not be performed +- **AND** SHALL scan all static rule files +- **AND** SHALL NOT execute any runtime rule's `check.ts` +- **AND** SHALL NOT exit with a non-zero code solely due to the reconcile failure -#### Scenario: Missing rules are warned as audit-only +#### Scenario: Degrade warning is suppressed under --json -- **WHEN** reconciliation returns `missing` entries -- **THEN** the CLI SHALL warn about them as audit information -- **AND** SHALL NOT change the exit code on their account +- **WHEN** the CLI degrades and `--json` is set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** SHALL NOT contain the human-readable degrade warning -#### Scenario: Warnings are suppressed under --json +### Requirement: Check dispatches static and runtime rules to distinct executors -- **WHEN** reconciliation returns mismatches and `--json` is set -- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape -- **AND** SHALL NOT contain the human-readable mismatch warnings +`taskless check` SHALL execute **static** ast-grep rules under `.taskless/rules/` with the +ast-grep scanner as before, and **runtime** rules under `.taskless/runtime-rules/` (directories +with `metadata.taskless.kind: runtime`, per the `cli-runtime-rule-execution` capability) with +the runtime harness. Findings from both executors SHALL be aggregated into the same result set +and SHALL count toward the exit code identically. -### Requirement: Check degrades to a local scan when reconciliation cannot complete +#### Scenario: Mixed corpus runs both executors -`taskless check` SHALL NOT fail solely because an attempted reconciliation cannot complete. When a token is available and `--anonymous` is not set but reconciliation cannot complete (no resolvable git remote, or the reconcile endpoint is unreachable or not yet deployed, or a transport error), the CLI SHALL warn that rule verification could not be performed and SHALL fall back to scanning all local rule files. The CLI SHALL NOT exit with a non-zero code solely because reconciliation failed, and the warning SHALL be suppressed under `--json`. +- **WHEN** `.taskless/rules/` contains static rules and `.taskless/runtime-rules/` contains runtime rules +- **THEN** the CLI SHALL run static rules through `sg scan` and runtime rules through the runtime harness +- **AND** SHALL merge their findings into one result set -#### Scenario: Endpoint unreachable degrades to local scan +### Requirement: Check runs runtime rules only on a signature-validated path -- **WHEN** an authenticated `check` attempts reconciliation and the endpoint is unreachable or returns a not-deployed error -- **THEN** the CLI SHALL warn that verification could not be performed -- **AND** SHALL scan all local rule files -- **AND** SHALL NOT exit with a non-zero code solely due to the reconcile failure +`taskless check` SHALL execute a runtime rule's `check.ts` only when that `check.ts` has been +validated by the server — returned in the reconciliation `run` set — or when +`--dangerously-run-scripts` is set. When a token is available and `--anonymous` is not set, the +CLI SHALL reconcile and execute every runtime rule whose `check.ts` is in `run`. An API key +SHALL be treated identically to an interactive token. On any path where the runtime rule's +`check.ts` signature is not validated — logged out, `--anonymous`, or a reconciliation that +cannot complete — the CLI SHALL NOT execute the rule's `check.ts`. -#### Scenario: Degrade warning is suppressed under --json +#### Scenario: Authenticated check runs blessed runtime rules -- **WHEN** the CLI degrades to a local scan and `--json` is set -- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape -- **AND** SHALL NOT contain the human-readable degrade warning +- **WHEN** an authenticated `check` reconciles and a runtime rule's `check.ts` is returned in `run` +- **THEN** the CLI SHALL execute that runtime rule through the harness -### Requirement: Check exits cleanly when the run set is empty +#### Scenario: A rule whose check.ts is not blessed is withheld -`taskless check` SHALL NOT invoke the scanner when a successful reconciliation returns an empty `run` set (for example an empty corpus, or every reported file classified `unsafe`/`unknown`); it SHALL report no scan results and SHALL exit with code 0. Any `unsafe` / `unknown` mismatch warnings SHALL still be emitted. +- **WHEN** reconciliation does not return a runtime rule's `check.ts` in `run` (it lands in `unsafe`/`unknown`/`missing`) +- **THEN** the CLI SHALL NOT execute that runtime rule +- **AND** SHALL surface it as an advisory mismatch -#### Scenario: Empty run set skips the scan +#### Scenario: API key behaves like a token -- **WHEN** reconciliation returns an empty `run` set -- **THEN** the CLI SHALL NOT invoke `sg scan` -- **AND** SHALL exit with code 0 -- **AND** SHALL still warn about any `unsafe`/`unknown` mismatches +- **WHEN** `check` runs with an API key +- **THEN** the CLI SHALL reconcile and run validated runtime rules exactly as with an interactive token + +### Requirement: Check skips runtime rules on unverified paths and reports the skip + +`taskless check` SHALL skip a runtime rule's execution when it cannot validate the rule's +signature — logged out, `--anonymous`, or reconciliation cannot complete — and +`--dangerously-run-scripts` is not set. It SHALL report that runtime rules exist and were not +run, SHALL still run static rules on these paths, and SHALL NOT change the exit code because +rules were skipped. In human output the report SHALL be a notice; under `--json` it SHALL be an +additive, optional `skipped` array of `{ rule, reason }`, leaving the existing `success` and +`results` fields unchanged, so machine callers (for example CI) can detect that runtime rules +did not run. + +#### Scenario: Logged-out check skips runtime rules + +- **WHEN** a user runs `taskless check` with no available token and runtime rules are present +- **THEN** the CLI SHALL run static rules +- **AND** SHALL NOT execute any runtime rule's `check.ts` +- **AND** SHALL report that the runtime rules were skipped +- **AND** SHALL NOT change the exit code because rules were skipped + +#### Scenario: Skipped runtime rules appear under --json + +- **WHEN** runtime rules are skipped and `--json` is set +- **THEN** stdout SHALL include a `skipped` array of `{ rule, reason }` alongside the unchanged `success` and `results` fields + +#### Scenario: Anonymous skips runtime rules while authenticated + +- **WHEN** a user runs `taskless check --anonymous` while a token is available +- **THEN** the CLI SHALL skip runtime rules exactly as an unauthenticated `check` + +### Requirement: Check accepts --dangerously-run-scripts to run runtime rules without server validation + +`taskless check` SHALL accept a `--dangerously-run-scripts` flag that runs **all** runtime +rules by trusting their local signatures without server validation, regardless of auth state. +When the flag is set the CLI SHALL NOT reconcile — it SHALL skip the network entirely (matching +how `--anonymous` forces the no-network path) and execute every present runtime rule. The CLI +SHALL emit a prominent warning that runtime rule code is being executed unverified. The flag +SHALL be the only way to execute runtime rules on an unverified path. + +#### Scenario: Dangerously-run-scripts executes runtime rules offline + +- **WHEN** a user runs `taskless check --dangerously-run-scripts` with no available token +- **THEN** the CLI SHALL execute the present runtime rules' `check.ts` +- **AND** SHALL emit a warning that runtime rule code ran unverified + +#### Scenario: Warning is suppressed under --json + +- **WHEN** `--dangerously-run-scripts` and `--json` are both set +- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **AND** the unverified-execution warning SHALL NOT appear in stdout diff --git a/openspec/specs/cli-rule-reconciliation/spec.md b/openspec/specs/cli-rule-reconciliation/spec.md index 3f37b76a..28fea4ee 100644 --- a/openspec/specs/cli-rule-reconciliation/spec.md +++ b/openspec/specs/cli-rule-reconciliation/spec.md @@ -106,18 +106,26 @@ be a release blocker (the test SHALL fail the build). ### Requirement: Reconcile reports every held rule file -The CLI SHALL reconcile by sending `POST /cli/api/reconcile` with an -`Authorization: Bearer ` header and a JSON body `{ repositoryUrl, files }`, where -`repositoryUrl` is the full repository URL and `files` is an array of -`{ file, signature }` covering **every** rule file the CLI would otherwise run under -`.taskless/rules/`. `file` SHALL be the rule file's delivered name as it exists on disk and -`signature` SHALL be the full envelope computed for that file's bytes. The CLI SHALL send the -whole signature envelope, not a bare digest. +Reconciliation SHALL be scoped to the **`check.ts` of runtime rules** — the only artifact that +carries arbitrary code execution. Static ast-grep rules and runtime-rule capture `*.yml` are +inert data, always available, and SHALL NOT be reported to or gated by reconciliation. The CLI +SHALL reconcile by sending `POST /cli/api/reconcile` with an `Authorization: Bearer ` +header and a JSON body `{ repositoryUrl, files }`, where `repositoryUrl` is the full repository +URL and `files` is an array of `{ file, signature }` covering the `check.ts` of **every** +runtime rule the CLI holds under `.taskless/runtime-rules/`. `file` SHALL be the `check.ts`'s +delivered path as it exists on disk and `signature` SHALL be the full envelope computed for its +bytes. The CLI SHALL send the whole signature envelope, not a bare digest. -#### Scenario: All rule files are reported +#### Scenario: Every runtime rule's check.ts is reported -- **WHEN** the CLI reconciles with rule files present under `.taskless/rules/` -- **THEN** the request body SHALL include one `{ file, signature }` entry per rule file +- **WHEN** the CLI reconciles with runtime rules present under `.taskless/runtime-rules/` +- **THEN** the request body SHALL include one `{ file, signature }` entry for the `check.ts` of each runtime rule + +#### Scenario: Inert files are not reported + +- **WHEN** the CLI reconciles and static `*.yml` rules and capture `*.yml` are also present +- **THEN** the request body SHALL NOT include entries for those inert files +- **AND** the static rules SHALL run regardless of the reconcile response #### Scenario: Full envelope is sent @@ -165,20 +173,22 @@ join), so a file that was moved but not changed still resolves. ### Requirement: The CLI executes only the server run set -When reconciliation succeeds, the CLI SHALL execute exactly the files in the server's `run` -set and no other file. The CLI SHALL NOT run a file on the basis of its own local comparison -of signatures. This replaces any local classification of rules. +When reconciliation succeeds, the CLI SHALL execute a runtime rule only if its **`check.ts`** +is present in the server's `run` set, and SHALL execute no runtime rule whose `check.ts` is +absent from `run`. The CLI SHALL NOT run a runtime rule on the basis of its own local +comparison of signatures. This replaces any local classification of runtime rules. Static +ast-grep rules and capture `*.yml` are outside this gate. -#### Scenario: Only run-set files execute +#### Scenario: Only rules with a blessed check.ts execute -- **WHEN** reconciliation returns a `run` set that is a strict subset of the reported files -- **THEN** the CLI SHALL execute only the files in `run` -- **AND** SHALL NOT execute any reported file absent from `run` +- **WHEN** reconciliation returns a `run` set covering the `check.ts` of some runtime rules but not others +- **THEN** the CLI SHALL execute only the runtime rules whose `check.ts` is in `run` +- **AND** SHALL withhold any runtime rule whose `check.ts` is absent from `run` #### Scenario: No local self-classification -- **WHEN** the CLI holds a local signature or sidecar for a file -- **THEN** it SHALL NOT treat that local value as authorization to run the file +- **WHEN** the CLI holds a local signature or sidecar for a runtime rule's `check.ts` +- **THEN** it SHALL NOT treat that local value as authorization to execute the rule ### Requirement: Reconcile is scoped to the token's organization diff --git a/openspec/specs/cli-runtime-rule-execution/spec.md b/openspec/specs/cli-runtime-rule-execution/spec.md new file mode 100644 index 00000000..17c52670 --- /dev/null +++ b/openspec/specs/cli-runtime-rule-execution/spec.md @@ -0,0 +1,139 @@ +# CLI Runtime Rule Execution + +## Purpose + +Defines how the CLI executes a **runtime rule** in `taskless check`: the on-disk shape it recognizes, the local harness that runs the rule's ast-grep capture narrow and gates on matches, how it invokes the rule's `check.ts` as a function under a bundled TypeScript loader, and how the returned findings map onto the scanner-agnostic result type. A runtime rule's `check.ts` is arbitrary code; the reconciliation gate that authorizes running it is defined in `cli-rule-reconciliation`, and the auth-state dispatch is defined in `cli-check`. + +## Requirements + +### Requirement: Runtime rules are directories recognized by metadata + +The CLI SHALL recognize a **runtime rule** as a directory under `.taskless/runtime-rules//` +containing one or more ast-grep capture `*.yml` (one per capture rule) and a single `check.ts`, +its capture rules declaring `metadata.taskless.kind: runtime`. The rule's check file SHALL be +the `check.ts` in the rule directory. The CLI SHALL read `metadata.taskless.match` (`anchor` or +`broad`) to select the ast-grep invocation mode. Rule files under `.taskless/rules/` SHALL +continue to be treated as static ast-grep rules, not runtime rules. +`.taskless/runtime-rule-tests//` holds `valid/` and `invalid/` verification fixtures and +SHALL NOT be executed by `check`. + +#### Scenario: A runtime-rules directory entry is a runtime rule + +- **WHEN** `.taskless/runtime-rules//` contains capture `*.yml` with `metadata.taskless.kind: runtime` and a `check.ts` +- **THEN** the CLI SHALL treat it as a runtime rule with `check.ts` as its check file and route it to the runtime harness + +#### Scenario: Rules under .taskless/rules remain static + +- **WHEN** a rule file lives under `.taskless/rules/` +- **THEN** the CLI SHALL treat it as a static rule and SHALL NOT route it to the runtime harness + +### Requirement: The harness narrows with one ast-grep scan and gates on matches + +For a runtime rule the CLI SHALL assemble the rule's capture rules into an ast-grep +configuration and run **one scan per mode** as the narrow: `--json=stream` for `anchor` capture +rules, and `--files-with-matches` for `broad` capture rules (whole-language `kind: program` +enumerators). A rule with only `anchor` capture rules therefore runs in a single scan; a rule +mixing modes runs one scan per mode. When the narrow produces **zero** matches the CLI SHALL +NOT invoke `check.ts`. + +#### Scenario: Zero matches skips the check + +- **WHEN** a runtime rule's narrow scan produces no matches +- **THEN** the CLI SHALL NOT invoke that rule's `check.ts` +- **AND** the rule SHALL contribute no findings + +#### Scenario: Capture rules of a mode run together + +- **WHEN** a runtime rule has multiple `anchor` capture rules +- **THEN** the CLI SHALL run them in a single `ast-grep` scan, not one scan per capture rule + +### Requirement: Matches are normalized and attributed to the model name + +The CLI SHALL normalize every narrow match to +`{ rule, ruleId, file, line, column, text, captures }`, where `file` is root-relative, `line` +is 1-indexed, and `rule` is the capture rule's stable model-assigned `name`. The CLI SHALL map +the hashed capture-rule `id` used by the scan back to that `name` so `match.rule` is the value +the check branches on, never the hash. + +#### Scenario: Hashed id maps to model name + +- **WHEN** the scan emits a match whose rule id is the hashed `${ruleSlug}-${sha1}` identifier +- **THEN** the normalized match's `rule` SHALL be the capture rule's model-assigned `name` + +### Requirement: The check is invoked as a function and its return value is used + +The CLI SHALL invoke a runtime rule's `check.ts` by calling its **default export** as a +function with `(root, matches)`, where `root` is the repository root and `matches` are the +normalized matches. The CLI SHALL use the `Finding[]` value the function **returns** as the +rule's result; it SHALL NOT infer results from process exit codes or stdout. A `check.ts` that +throws SHALL be isolated to a single error-severity finding for that rule and SHALL NOT abort +the overall `check` run. + +#### Scenario: Returned findings are the result + +- **WHEN** a runtime rule's `check.ts` default export returns a `Finding[]` +- **THEN** the CLI SHALL treat exactly those findings as the rule's output + +#### Scenario: A throwing check is isolated + +- **WHEN** a runtime rule's `check.ts` throws during execution +- **THEN** the CLI SHALL record a single error-severity finding for that rule +- **AND** SHALL continue executing the remaining rules and produce output + +### Requirement: check.ts execution is bounded by a timeout + +The CLI SHALL bound each `check.ts` invocation with a wall-clock timeout and SHALL accept a +`--timeout ` flag on `check` to override the default. When a `check.ts` exceeds the +timeout the CLI SHALL terminate it, record a single error-severity finding for that rule, and +continue executing the remaining rules — a runaway check SHALL NOT wedge the overall `check` +run. + +#### Scenario: A hanging check is terminated at the timeout + +- **WHEN** a runtime rule's `check.ts` runs longer than the effective timeout +- **THEN** the CLI SHALL terminate it and record a single error-severity finding for that rule +- **AND** SHALL continue executing the remaining rules + +#### Scenario: --timeout overrides the default + +- **WHEN** a user runs `taskless check --timeout ` +- **THEN** the CLI SHALL use that value as the per-check wall-clock bound + +### Requirement: check.ts runs via a bundled TypeScript loader + +The CLI SHALL execute `check.ts` using a pinned TypeScript loader bundled with the CLI (e.g. +`tsx`) and SHALL NOT require the user's repository to provide a TypeScript toolchain, +`node_modules`, or a precompile step. The CLI MAY schedule invocations by any mechanism +(process-per-check, worker pool, or in-process import); the function contract does not +constrain the choice. + +#### Scenario: No user toolchain required + +- **WHEN** a repository with a runtime rule has no local TypeScript toolchain installed +- **THEN** the CLI SHALL still execute the rule's `check.ts` using its bundled loader + +### Requirement: Findings map onto the scanner-agnostic result type + +The CLI SHALL map each `Finding` returned by a `check.ts` onto the existing `CheckResult` +shape with a runtime `source`, so runtime findings are aggregated, formatted, and counted +toward the exit code identically to static findings. `Finding.severity` (`error` / `warning` / +`info`) SHALL map directly onto the corresponding `CheckResult` severity with no translation. + +#### Scenario: Runtime findings gate the exit code like static findings + +- **WHEN** a runtime rule returns a finding with `severity: "error"` +- **THEN** the CLI SHALL count it toward the error total that sets a non-zero exit code +- **AND** the finding SHALL appear in `--json` output under the same `results` shape as a static finding + +### Requirement: Blessed runtime rules execute from the materialized run directory + +When a runtime rule is executed on a validated path, the CLI SHALL execute it from the +ephemeral, gitignored `.taskless/.run/` materialization of the blessed bytes, not from the +live `.taskless/rules/` tree, so the bytes executed are the exact bytes reconciliation blessed +(read-hash-execute ordering). + +#### Scenario: Execution uses the blessed bytes + +- **WHEN** a runtime rule is blessed and executed +- **THEN** the CLI SHALL invoke the `check.ts` materialized under `.taskless/.run/` +- **AND** SHALL NOT execute a copy modified in `.taskless/rules/` after reconciliation From 5a7e18cbf99a254149f5361d3abc25106b7529da Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 4 Jul 2026 11:15:07 -0700 Subject: [PATCH 17/21] fix(cli): harden vectors prebuild and correct parseSignature doc Address Copilot review on #47: - fetch-rule-hash-vectors.ts: a 200 response with a non-JSON body threw an uncaught error and broke the prebuild even when a committed cache existed. Wrap response.json() and fall back to the cache like every other failure. - rule-hash.ts: the parseSignature JSDoc claimed it throws on an 'unsupported' envelope, but it deliberately tolerates unknown future algoVersions (forward-compat). Correct the comment to match the implementation and tests. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../cli/scripts/fetch-rule-hash-vectors.ts | 20 ++++++++++++++----- packages/cli/src/rules/rule-hash.ts | 4 +++- 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/packages/cli/scripts/fetch-rule-hash-vectors.ts b/packages/cli/scripts/fetch-rule-hash-vectors.ts index 6994e58d..fa66ac55 100644 --- a/packages/cli/scripts/fetch-rule-hash-vectors.ts +++ b/packages/cli/scripts/fetch-rule-hash-vectors.ts @@ -68,15 +68,25 @@ try { if (response?.ok) { // The endpoint wraps the source-of-truth array as { vectors: [...] }; the // committed cache stores the bare array to match the server reference file. - const body = (await response.json()) as { - vectors?: { name: string; input: string; signature: string }[]; - }; - if (Array.isArray(body.vectors) && body.vectors.length > 0) { + let body: + | { vectors?: { name: string; input: string; signature: string }[] } + | undefined; + try { + body = (await response.json()) as { + vectors?: { name: string; input: string; signature: string }[]; + }; + } catch (error) { + // A 200 with a non-JSON body (e.g. a proxy error page) must fall back to the + // cache like any other failure, not throw and break the build. + const message = error instanceof Error ? error.message : String(error); + fallBackToCache(`Response was not valid JSON (${message})`); + } + if (body && Array.isArray(body.vectors) && body.vectors.length > 0) { writeFileSync(OUTPUT_PATH, toAsciiJson(body.vectors), "utf8"); console.log( ` Wrote ${String(body.vectors.length)} vectors to: ${OUTPUT_PATH}` ); - } else { + } else if (body) { fallBackToCache(`Response did not contain a non-empty "vectors" array`); } } else if (response) { diff --git a/packages/cli/src/rules/rule-hash.ts b/packages/cli/src/rules/rule-hash.ts index d292bc35..069e8387 100644 --- a/packages/cli/src/rules/rule-hash.ts +++ b/packages/cli/src/rules/rule-hash.ts @@ -64,7 +64,9 @@ export async function canonicalHash(fileText: string): Promise { /** * Parse and validate a signature envelope. The algoVersion is read up to the * first `;` — before any `key=value` parsing — so versioning never depends on - * the parameter syntax. Throws on a malformed or unsupported envelope. + * the parameter syntax. Throws on a malformed envelope; an unknown (future) + * algoVersion is parsed leniently (its `algo`/`digest` are not validated) so + * newer signatures stay forward-compatible rather than being rejected. */ export function parseSignature(signature: string): ParsedSignature { const firstDelimiter = signature.indexOf(";"); From 0343af7db16af50ca83bfd66897f4eaaddc00dd5 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 4 Jul 2026 11:17:15 -0700 Subject: [PATCH 18/21] docs(openspec): fix live-tree path in the materialize requirement Address Copilot review on #49: the 'Blessed runtime rules execute from the materialized run directory' requirement, its scenario, and the design decision referred to the live tree as `.taskless/rules/`, but runtime rules live under `.taskless/runtime-rules/`. Correct all three so the read-hash-execute guarantee names the right tree. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../archive/2026-07-03-runtime-rule-execution/design.md | 2 +- .../specs/cli-runtime-rule-execution/spec.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md index c25d1781..f73ff3a7 100644 --- a/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md @@ -147,7 +147,7 @@ Ship a pinned `tsx` (or equivalent loader) with the CLI so `check.ts` runs witho `node_modules`/toolchain in the user's repo. Execute blessed runtime rules from the ephemeral, gitignored `.taskless/.run/` (extending the stacked-under materialize step), so the bytes executed are exactly the reconciled-and-blessed bytes (read-hash-execute ordering), not -whatever is live in `.taskless/rules/` at exec time. +whatever is live in `.taskless/runtime-rules/` at exec time. _Alternative rejected:_ require the user to have `tsx`/`ts-node`. Non-hermetic, version-drift prone, and breaks the "no toolchain assumptions" posture the rest of the CLI keeps. diff --git a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md index 0c23383a..a63ac7cb 100644 --- a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md @@ -123,11 +123,11 @@ toward the exit code identically to static findings. `Finding.severity` (`error` When a runtime rule is executed on a validated path, the CLI SHALL execute it from the ephemeral, gitignored `.taskless/.run/` materialization of the blessed bytes, not from the -live `.taskless/rules/` tree, so the bytes executed are the exact bytes reconciliation blessed -(read-hash-execute ordering). +live `.taskless/runtime-rules/` tree, so the bytes executed are the exact bytes reconciliation +blessed (read-hash-execute ordering). #### Scenario: Execution uses the blessed bytes - **WHEN** a runtime rule is blessed and executed - **THEN** the CLI SHALL invoke the `check.ts` materialized under `.taskless/.run/` -- **AND** SHALL NOT execute a copy modified in `.taskless/rules/` after reconciliation +- **AND** SHALL NOT execute a copy modified in `.taskless/runtime-rules/` after reconciliation From f3c5dc1c81773ad6b90b16234aabba90000d209b Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 4 Jul 2026 11:23:03 -0700 Subject: [PATCH 19/21] fix(cli): address Copilot review on runtime rule execution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six fixes from Copilot's review of #50, plus the canonical-spec twin of the #49 doc fix: - invoke.ts: on Windows, kill the check via taskkill /T (negative PIDs aren't supported), so a timeout actually terminates the tsx+node tree. - discover.ts: pin the check file to check.ts inside the rule dir; do not resolve metadata.taskless.check as a path (prevents escaping the dir via ../). - run-set.ts: signRuntimeChecks is now per-rule resilient (returns unreadable rules instead of throwing); reported reconcile paths are POSIX-normalized so Windows backslashes don't defeat the server-side path match. - check.ts: a missing/unreadable check.ts is reported as skipped and materialization errors degrade to a runtime-skip — a malformed runtime rule no longer aborts the whole check (static keeps running). - narrow.ts: run one broad scan per broad capture rule so matches are attributed to the right rule (was mislabeling all broad matches as the first rule); treat a signal-killed narrow (null exit code) as a failure instead of success. - specs/cli-runtime-rule-execution: fix the materialize requirement's live-tree path (.taskless/rules/ -> .taskless/runtime-rules/) to match #49. Add an integration test: a runtime rule missing check.ts is skipped (not fatal) and static rules still run. Full suite green (339). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../specs/cli-runtime-rule-execution/spec.md | 6 +-- packages/cli/src/commands/check.ts | 34 +++++++++++--- packages/cli/src/rules/runtime/discover.ts | 10 ++--- packages/cli/src/rules/runtime/invoke.ts | 15 ++++++- packages/cli/src/rules/runtime/narrow.ts | 26 +++++++---- packages/cli/src/rules/runtime/run-set.ts | 34 ++++++++++---- packages/cli/test/runtime-check.test.ts | 45 +++++++++++++++++++ 7 files changed, 136 insertions(+), 34 deletions(-) diff --git a/openspec/specs/cli-runtime-rule-execution/spec.md b/openspec/specs/cli-runtime-rule-execution/spec.md index 17c52670..76ff68d0 100644 --- a/openspec/specs/cli-runtime-rule-execution/spec.md +++ b/openspec/specs/cli-runtime-rule-execution/spec.md @@ -129,11 +129,11 @@ toward the exit code identically to static findings. `Finding.severity` (`error` When a runtime rule is executed on a validated path, the CLI SHALL execute it from the ephemeral, gitignored `.taskless/.run/` materialization of the blessed bytes, not from the -live `.taskless/rules/` tree, so the bytes executed are the exact bytes reconciliation blessed -(read-hash-execute ordering). +live `.taskless/runtime-rules/` tree, so the bytes executed are the exact bytes reconciliation +blessed (read-hash-execute ordering). #### Scenario: Execution uses the blessed bytes - **WHEN** a runtime rule is blessed and executed - **THEN** the CLI SHALL invoke the `check.ts` materialized under `.taskless/.run/` -- **AND** SHALL NOT execute a copy modified in `.taskless/rules/` after reconciliation +- **AND** SHALL NOT execute a copy modified in `.taskless/runtime-rules/` after reconciliation diff --git a/packages/cli/src/commands/check.ts b/packages/cli/src/commands/check.ts index 300ed5b5..b2021c84 100644 --- a/packages/cli/src/commands/check.ts +++ b/packages/cli/src/commands/check.ts @@ -175,7 +175,15 @@ async function planRuntime( ); } - const signed = await signRuntimeChecks(discovered); + // A rule whose check.ts is missing/unreadable is reported, not fatal: signing + // never throws, and such rules are surfaced as skipped so static checks and + // the other runtime rules are unaffected. + const { signed, unreadable } = await signRuntimeChecks(discovered); + const unreadableSkips: SkippedRuntimeRule[] = unreadable.map((rule) => ({ + rule: rule.name, + reason: "its check.ts is missing or unreadable", + })); + const outcome = await reconcile(token, { repositoryUrl, files: reportRuntimeChecks(cwd, signed), @@ -198,14 +206,26 @@ async function planRuntime( signed, outcome.result.run ); - const execute = - blessed.length > 0 ? await materializeRuntimeRules(cwd, blessed) : []; + let execute: RuntimeRule[] = []; + try { + execute = + blessed.length > 0 ? await materializeRuntimeRules(cwd, blessed) : []; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + return skipAllRuntime( + discovered, + `runtime rules could not be materialized (${message})` + ); + } return { execute, - skipped: withheld.map((rule) => ({ - rule: rule.name, - reason: "not blessed by the server (unsafe / unknown / drift)", - })), + skipped: [ + ...unreadableSkips, + ...withheld.map((rule) => ({ + rule: rule.name, + reason: "not blessed by the server (unsafe / unknown / drift)", + })), + ], notices: [], }; } diff --git a/packages/cli/src/rules/runtime/discover.ts b/packages/cli/src/rules/runtime/discover.ts index 949b5200..65c516e8 100644 --- a/packages/cli/src/rules/runtime/discover.ts +++ b/packages/cli/src/rules/runtime/discover.ts @@ -123,15 +123,15 @@ export async function discoverRuntimeRulesIn( const captureRules = await loadCaptureRules(directory); if (captureRules.length === 0) continue; // not a runtime rule - // Every capture rule of a runtime rule names the same `check.ts`; the - // generator always writes `check.ts`, so fall back to that. - const checkName = - captureRules[0]!.rule.metadata.taskless.check || "check.ts"; + // The check file is always `check.ts` inside the rule directory (per spec). + // We deliberately do NOT resolve `metadata.taskless.check` as a path — an + // arbitrary value (e.g. `../../evil.ts`) must not be able to point execution + // or signing at a file outside the rule directory. rules.push({ name: entry.name, dir: directory, captureRules, - checkFile: join(directory, checkName), + checkFile: join(directory, "check.ts"), }); } return rules; diff --git a/packages/cli/src/rules/runtime/invoke.ts b/packages/cli/src/rules/runtime/invoke.ts index 60d73b56..3a00ae91 100644 --- a/packages/cli/src/rules/runtime/invoke.ts +++ b/packages/cli/src/rules/runtime/invoke.ts @@ -100,8 +100,21 @@ export async function invokeCheck( const stderrChunks: string[] = []; let timedOut = false; const killTree = () => { + if (child.pid === undefined) return; + if (process.platform === "win32") { + // Negative PIDs aren't supported on Windows; `taskkill /T` terminates + // the whole tree (the tsx wrapper and its re-exec'd node grandchild). + try { + spawn("taskkill", ["/pid", String(child.pid), "/T", "/F"], { + stdio: "ignore", + }); + } catch { + child.kill("SIGKILL"); + } + return; + } try { - if (child.pid !== undefined) process.kill(-child.pid, "SIGKILL"); + process.kill(-child.pid, "SIGKILL"); } catch { child.kill("SIGKILL"); } diff --git a/packages/cli/src/rules/runtime/narrow.ts b/packages/cli/src/rules/runtime/narrow.ts index c95d58dc..7f340777 100644 --- a/packages/cli/src/rules/runtime/narrow.ts +++ b/packages/cli/src/rules/runtime/narrow.ts @@ -52,12 +52,16 @@ function runSg( stderrChunks.push(chunk.toString()) ); child.on("error", reject); - child.on("close", (code) => { - // ast-grep exits 1 when matches are found — expected. >1 is a real failure. - if (code !== null && code > 1) { + child.on("close", (code, signal) => { + // ast-grep exits 1 when matches are found — expected. A `null` code means + // the process was killed by a signal (e.g. OOM); treat that and any exit + // >1 as a real failure rather than silently dropping matches. + if (code === null || code > 1) { + const cause = + code === null ? `signal ${String(signal)}` : `exit ${String(code)}`; reject( new Error( - `ast-grep narrow failed (exit ${String(code)})${ + `ast-grep narrow failed (${cause})${ stderrChunks.length > 0 ? `: ${stderrChunks.join("").trim()}` : "" }` ) @@ -137,11 +141,15 @@ export async function runNarrow( }); } - if (broad.length > 0) { - const config = await writeRuleConfig(join(workDirectory, "broad"), broad); - // A single ruleId can't be recovered from --files-with-matches; attribute - // broad matches to the (usually sole) broad capture rule. - const broadRule = broad[0]!; + // `--files-with-matches` reports paths only — it can't say which rule + // matched — so run one broad scan per broad capture rule and attribute each + // file to that rule. (Broad rules are whole-language enumerators, so this is + // cheap and the common case is a single broad rule.) + for (const broadRule of broad) { + const config = await writeRuleConfig( + join(workDirectory, `broad-${broadRule.id}`), + [broadRule] + ); const seen = new Set(); await runSg(root, config, ["--files-with-matches"], paths, (line) => { const file = line.trim(); diff --git a/packages/cli/src/rules/runtime/run-set.ts b/packages/cli/src/rules/runtime/run-set.ts index 8ce5e0ea..fff8c188 100644 --- a/packages/cli/src/rules/runtime/run-set.ts +++ b/packages/cli/src/rules/runtime/run-set.ts @@ -1,5 +1,5 @@ import { cp, mkdir, rm } from "node:fs/promises"; -import { join, relative } from "node:path"; +import { join, relative, sep } from "node:path"; import { addToGitignore } from "../../filesystem/gitignore"; import { signRuleFile } from "../rule-hash"; @@ -16,20 +16,35 @@ export interface SignedRuntimeRule { signature: string; } +/** Result of signing a set of runtime rules' `check.ts` files. */ +export interface RuntimeSigningResult { + /** Rules whose `check.ts` was read and signed. */ + signed: SignedRuntimeRule[]; + /** Rules whose `check.ts` was missing or unreadable (cannot be reconciled). */ + unreadable: RuntimeRule[]; +} + /** * Sign each runtime rule's `check.ts` (only) — the sole artifact carrying * arbitrary code execution. Capture `*.yml` are inert and are neither signed nor - * reported. + * reported. A rule whose `check.ts` cannot be read is returned in `unreadable` + * (never thrown) so one malformed rule never aborts the whole `check`. */ export async function signRuntimeChecks( rules: RuntimeRule[] -): Promise { - return Promise.all( - rules.map(async (rule) => ({ - rule, - signature: await signRuleFile(rule.checkFile), - })) +): Promise { + const signed: SignedRuntimeRule[] = []; + const unreadable: RuntimeRule[] = []; + await Promise.all( + rules.map(async (rule) => { + try { + signed.push({ rule, signature: await signRuleFile(rule.checkFile) }); + } catch { + unreadable.push(rule); + } + }) ); + return { signed, unreadable }; } /** Map signed runtime rules to the reconcile report (`check.ts` path + signature). */ @@ -38,7 +53,8 @@ export function reportRuntimeChecks( signed: SignedRuntimeRule[] ): ReportedFile[] { return signed.map(({ rule, signature }) => ({ - file: relative(cwd, rule.checkFile), + // Reconcile paths are repo-relative POSIX; normalize Windows separators. + file: relative(cwd, rule.checkFile).split(sep).join("/"), signature, })); } diff --git a/packages/cli/test/runtime-check.test.ts b/packages/cli/test/runtime-check.test.ts index 7f803104..81ef4b02 100644 --- a/packages/cli/test/runtime-check.test.ts +++ b/packages/cli/test/runtime-check.test.ts @@ -281,4 +281,49 @@ describe("check: static vs runtime dispatch", () => { expect(stderr).toContain("dangerously-run-scripts"); expect(stdout).toContain("demo"); // runtime finding surfaced }); + + it("a runtime rule missing check.ts is skipped, not fatal; static still runs", async () => { + // A malformed rule (capture yml, no check.ts) must not abort the whole check. + const broken = join(directory, ".taskless", "runtime-rules", "broken"); + await mkdir(broken, { recursive: true }); + await writeFile(join(broken, "logs.yml"), RUNTIME_CAPTURE, "utf8"); + + const server = await startMockServer((request) => ({ + statusCode: 200, + body: { + run: [ + { + ruleId: "demo", + file: CHECK_REPORT_PATH, + signature: sig(request, "check.ts"), + }, + ], + unsafe: [], + unknown: [], + missing: [], + }, + })); + try { + const { stdout, exitCode } = await runCli( + ["check", "-d", directory, "--json"], + { + TASKLESS_TOKEN: "fake.token", + TASKLESS_API_URL: server.apiUrl, + } + ); + const output = parseJson(stdout); + expect(exitCode).toBe(0); // not SCAN_FAILED + // The good runtime rule still ran and static still ran. + expect(output.results.some((r) => r.source === "taskless-runtime")).toBe( + true + ); + expect(output.results.some((r) => r.ruleId === "no-console")).toBe(true); + // The broken rule is reported as skipped, not crashed. + expect(output.skipped?.some((s) => s.rule === "broken")).toBe(true); + // Only the readable check.ts was reported to the server. + expect(server.requests[0]!.files).toHaveLength(1); + } finally { + await server.close(); + } + }); }); From e61fb12aa42dd4ea4e4e8242e0a4ea964520902f Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 4 Jul 2026 11:48:52 -0700 Subject: [PATCH 20/21] docs(openspec): tighten runtime-rule specs per Copilot review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address the remaining #49 review comments (items 1-4, 6-7): - cli-check: the degrade --json scenario now allows the additive `skipped` array (not 'only { success, results }'); --dangerously-run-scripts is described as 'without server validation' rather than 'trusting local signatures' (which the reconciliation spec forbids as an auth mechanism). - cli-rule-reconciliation: rename the requirement heading to 'Reconcile reports every runtime rule's check.ts' (RENAMED op) so the title matches the scoped body instead of the old 'every held rule file'. - cli-runtime-rule-execution: clarify that match mode is read per capture rule (rules may mix modes); state that a broad path-only match carries line/column = 1 and empty text/captures. - design: note the generator path lives in the internal taskless/taskless repo, not this one. (Item 5 — the 'exactly as before' wording — is left as-is by decision.) Co-Authored-By: Claude Opus 4.8 (1M context) --- .../2026-07-03-runtime-rule-execution/design.md | 5 +++-- .../specs/cli-check/spec.md | 4 ++-- .../specs/cli-rule-reconciliation/spec.md | 7 ++++++- .../specs/cli-runtime-rule-execution/spec.md | 10 +++++++--- 4 files changed, 18 insertions(+), 8 deletions(-) diff --git a/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md index f73ff3a7..02a22ff3 100644 --- a/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/design.md @@ -66,8 +66,9 @@ mode, and never parses rule intent beyond this metadata envelope and the ast-gre already understands. `.taskless/runtime-rule-tests//` holds `valid/`/`invalid/` fixtures and is not executed by `check`. -This layout is confirmed against the generator -(`workers/generator/src/actions/add-runtime-rule.ts`), which writes +This layout is confirmed against the Taskless internal generator — in the sibling +`taskless/taskless` repo, not this one — at `workers/generator/src/actions/add-runtime-rule.ts`, +which writes `.taskless/runtime-rules/-/` with `.yml` per capture rule and a `check.ts`, and hashes `check.ts` with the same `canonicalHash` envelope reconcile uses. diff --git a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md index 026d6c52..385e0dbf 100644 --- a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-check/spec.md @@ -72,7 +72,7 @@ did not run. ### Requirement: Check accepts --dangerously-run-scripts to run runtime rules without server validation `taskless check` SHALL accept a `--dangerously-run-scripts` flag that runs **all** runtime -rules by trusting their local signatures without server validation, regardless of auth state. +rules without server validation, regardless of auth state. When the flag is set the CLI SHALL NOT reconcile — it SHALL skip the network entirely (matching how `--anonymous` forces the no-network path) and execute every present runtime rule. The CLI SHALL emit a prominent warning that runtime rule code is being executed unverified. The flag @@ -163,7 +163,7 @@ non-zero code solely because reconciliation failed, and the warning SHALL be sup #### Scenario: Degrade warning is suppressed under --json - **WHEN** the CLI degrades and `--json` is set -- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **THEN** stdout SHALL contain the machine JSON shape (`{ success, results }` plus the additive optional `skipped` array for the skipped runtime rules) - **AND** SHALL NOT contain the human-readable degrade warning ## REMOVED Requirements diff --git a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md index 7195f909..13710714 100644 --- a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-rule-reconciliation/spec.md @@ -1,6 +1,11 @@ +## RENAMED Requirements + +- FROM: `### Requirement: Reconcile reports every held rule file` +- TO: `### Requirement: Reconcile reports every runtime rule's check.ts` + ## MODIFIED Requirements -### Requirement: Reconcile reports every held rule file +### Requirement: Reconcile reports every runtime rule's check.ts Reconciliation SHALL be scoped to the **`check.ts` of runtime rules** — the only artifact that carries arbitrary code execution. Static ast-grep rules and runtime-rule capture `*.yml` are diff --git a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md index a63ac7cb..1e170346 100644 --- a/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md +++ b/openspec/changes/archive/2026-07-03-runtime-rule-execution/specs/cli-runtime-rule-execution/spec.md @@ -5,8 +5,10 @@ The CLI SHALL recognize a **runtime rule** as a directory under `.taskless/runtime-rules//` containing one or more ast-grep capture `*.yml` (one per capture rule) and a single `check.ts`, its capture rules declaring `metadata.taskless.kind: runtime`. The rule's check file SHALL be -the `check.ts` in the rule directory. The CLI SHALL read `metadata.taskless.match` (`anchor` or -`broad`) to select the ast-grep invocation mode. Rule files under `.taskless/rules/` SHALL +the `check.ts` in the rule directory. The CLI SHALL read **each capture rule's** +`metadata.taskless.match` (`anchor` or `broad`) to select that capture rule's ast-grep +invocation mode; capture rules within one runtime rule MAY mix modes (each is independent). +Rule files under `.taskless/rules/` SHALL continue to be treated as static ast-grep rules, not runtime rules. `.taskless/runtime-rule-tests//` holds `valid/` and `invalid/` verification fixtures and SHALL NOT be executed by `check`. @@ -47,7 +49,9 @@ The CLI SHALL normalize every narrow match to `{ rule, ruleId, file, line, column, text, captures }`, where `file` is root-relative, `line` is 1-indexed, and `rule` is the capture rule's stable model-assigned `name`. The CLI SHALL map the hashed capture-rule `id` used by the scan back to that `name` so `match.rule` is the value -the check branches on, never the hash. +the check branches on, never the hash. A `broad` (path-only, `--files-with-matches`) match +carries no location or captures: its `line` and `column` SHALL be `1`, and its `text` and +`captures` SHALL be empty. #### Scenario: Hashed id maps to model name From 949565d2283183657310309476d4e42d7e721c61 Mon Sep 17 00:00:00 2001 From: Jakob Heuser Date: Sat, 4 Jul 2026 11:57:05 -0700 Subject: [PATCH 21/21] docs(openspec): sync runtime-rule spec tightening into canonical specs Propagate the #49 review-comment fixes into the canonical specs (the archived change copy came via merge): - cli-check: degrade --json scenario allows the additive skipped array; --dangerously-run-scripts described as 'without server validation'. - cli-rule-reconciliation: heading renamed to 'Reconcile reports every runtime rule's check.ts'. - cli-runtime-rule-execution: per-capture match mode; broad matches are path-only (line/column 1, empty text/captures). Co-Authored-By: Claude Opus 4.8 (1M context) --- openspec/specs/cli-check/spec.md | 4 ++-- openspec/specs/cli-rule-reconciliation/spec.md | 2 +- openspec/specs/cli-runtime-rule-execution/spec.md | 10 +++++++--- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/openspec/specs/cli-check/spec.md b/openspec/specs/cli-check/spec.md index a519018f..f6cde842 100644 --- a/openspec/specs/cli-check/spec.md +++ b/openspec/specs/cli-check/spec.md @@ -316,7 +316,7 @@ non-zero code solely because reconciliation failed, and the warning SHALL be sup #### Scenario: Degrade warning is suppressed under --json - **WHEN** the CLI degrades and `--json` is set -- **THEN** stdout SHALL contain only the existing `{ success, results }` JSON shape +- **THEN** stdout SHALL contain the machine JSON shape (`{ success, results }` plus the additive optional `skipped` array for the skipped runtime rules) - **AND** SHALL NOT contain the human-readable degrade warning ### Requirement: Check dispatches static and runtime rules to distinct executors @@ -391,7 +391,7 @@ did not run. ### Requirement: Check accepts --dangerously-run-scripts to run runtime rules without server validation `taskless check` SHALL accept a `--dangerously-run-scripts` flag that runs **all** runtime -rules by trusting their local signatures without server validation, regardless of auth state. +rules without server validation, regardless of auth state. When the flag is set the CLI SHALL NOT reconcile — it SHALL skip the network entirely (matching how `--anonymous` forces the no-network path) and execute every present runtime rule. The CLI SHALL emit a prominent warning that runtime rule code is being executed unverified. The flag diff --git a/openspec/specs/cli-rule-reconciliation/spec.md b/openspec/specs/cli-rule-reconciliation/spec.md index 28fea4ee..5d4c1170 100644 --- a/openspec/specs/cli-rule-reconciliation/spec.md +++ b/openspec/specs/cli-rule-reconciliation/spec.md @@ -104,7 +104,7 @@ be a release blocker (the test SHALL fail the build). - **THEN** the test SHALL fail - **AND** the build SHALL NOT pass -### Requirement: Reconcile reports every held rule file +### Requirement: Reconcile reports every runtime rule's check.ts Reconciliation SHALL be scoped to the **`check.ts` of runtime rules** — the only artifact that carries arbitrary code execution. Static ast-grep rules and runtime-rule capture `*.yml` are diff --git a/openspec/specs/cli-runtime-rule-execution/spec.md b/openspec/specs/cli-runtime-rule-execution/spec.md index 76ff68d0..263516f5 100644 --- a/openspec/specs/cli-runtime-rule-execution/spec.md +++ b/openspec/specs/cli-runtime-rule-execution/spec.md @@ -11,8 +11,10 @@ Defines how the CLI executes a **runtime rule** in `taskless check`: the on-disk The CLI SHALL recognize a **runtime rule** as a directory under `.taskless/runtime-rules//` containing one or more ast-grep capture `*.yml` (one per capture rule) and a single `check.ts`, its capture rules declaring `metadata.taskless.kind: runtime`. The rule's check file SHALL be -the `check.ts` in the rule directory. The CLI SHALL read `metadata.taskless.match` (`anchor` or -`broad`) to select the ast-grep invocation mode. Rule files under `.taskless/rules/` SHALL +the `check.ts` in the rule directory. The CLI SHALL read **each capture rule's** +`metadata.taskless.match` (`anchor` or `broad`) to select that capture rule's ast-grep +invocation mode; capture rules within one runtime rule MAY mix modes (each is independent). +Rule files under `.taskless/rules/` SHALL continue to be treated as static ast-grep rules, not runtime rules. `.taskless/runtime-rule-tests//` holds `valid/` and `invalid/` verification fixtures and SHALL NOT be executed by `check`. @@ -53,7 +55,9 @@ The CLI SHALL normalize every narrow match to `{ rule, ruleId, file, line, column, text, captures }`, where `file` is root-relative, `line` is 1-indexed, and `rule` is the capture rule's stable model-assigned `name`. The CLI SHALL map the hashed capture-rule `id` used by the scan back to that `name` so `match.rule` is the value -the check branches on, never the hash. +the check branches on, never the hash. A `broad` (path-only, `--files-with-matches`) match +carries no location or captures: its `line` and `column` SHALL be `1`, and its `text` and +`captures` SHALL be empty. #### Scenario: Hashed id maps to model name