feat(code-review): give the repository profile a skill of its own - #53
Kacper-Lubisz wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Tessl Code Review
Changes approved
3 optional suggestions. Nothing blocking.
The new profile skill, reference material, routing evaluations, and release metadata establish the intended profile workflow, but the documented validation does not select the profile it is meant to validate, and the consumer and manifest descriptions retain misleading omissions.
| Severity | Requires changes | Advisory |
|---|---|---|
| Minor | 0 | 3 |
Advisory (3)
- Minor · Advisory in Standard mode · The profile checker instructions validate no selected profile
code-review/skills/configure-code-review-profile/SKILL.md:144 - Minor · Advisory in Standard mode · The Action profile-path rule excludes the shipped
standardprofile
code-review/skills/configure-code-review-profile/references/profile-schema.md:29 - Minor · Advisory in Standard mode · The release manifest does not describe profile configuration
code-review/.tessl-plugin/plugin.json:4
Working with this review
Verify before acting. These findings are AI-generated and not authoritative. Check each one against the code, yourself or with an agent, before acting on it.
Answer each finding on its thread:
- Addressed: you changed the code. Say what changed.
- Refuted: the finding is wrong about the code.
- Declined: the finding is right, and you are not changing it. Say why.
Reviews are incremental. The first review shows every valid optional and required finding. A later round checks the latest changes and earlier findings, and adds a fresh finding only when it requires changes. It never repeats one you refuted or declined. The configured request-changes severity sets the consequence baseline, and review mode sets the strength required to block. Ask for a full review to inspect the whole change and show optional findings again.
Lenses are configurable. Add your own and scope them to paths with globs. Keep the set small so findings stay distinct. See https://docs.tessl.io/tutorials/setting-up-agentic-code-review#route-reviews-with-yaml
Mention @tessl-code-review to run another review.
| Run the checker against the file you wrote: | ||
|
|
||
| ```bash | ||
| tessl code review --check-profile |
There was a problem hiding this comment.
Minor · Advisory in Standard mode · The profile checker instructions validate no selected profile
The new skill tells users to run tessl code review --check-profile, and both profile evaluations reward that same bare command. The accompanying schema reference states that the CLI reads no profile unless --profile ./.tessl-code-review.yml is supplied. Consequently, following the procedure can validate the default/no-profile run rather than the root file just written, leaving its schema, routing, ignored paths, and zero-file lens selections unchecked while the user is told it was checked. The validation instruction and the evaluations must select the produced profile so that a passing check applies to the policy being handed over.
Skill Instruction Integrity
There was a problem hiding this comment.
Refuted, and the confusion it names has been fixed anyway.
tessl code review check-profile is a subcommand, not the --check-profile flag this finding quotes, and it does not inherit the review path's behaviour: with no --profile it calls locateCodeReviewProfile(undefined, repositoryRoot), which discovers ./.tessl-code-review.yml at the root and fails with no-profile-found when there is none. So the bare command in step 5 checks exactly the file the skill just wrote, and the evals reward the right thing.
What was true is that the schema reference said flatly that the CLI "reads none by default", which is a statement about a review and made the bare checker invocation look unchecked. Fixed in cc3ecb2: the CLI bullet now splits by subcommand and states the checker's discovery explicitly.
| mode: advisory | ||
| ``` | ||
|
|
||
| Whichever consumer reads it, the path must end in `.yml` or `.yaml` and must |
There was a problem hiding this comment.
Minor · Advisory in Standard mode · The Action profile-path rule excludes the shipped standard profile
The shared profile reference says that every consumer's profile path must end in .yml or .yaml, but the unsupported Action's own setup skill and all of its workflow templates use the supported profile: standard default. A reader using the new shared reference to explain an Action configuration is therefore told that the standard configuration is invalid and may replace or reject it unnecessarily. The consumer description must distinguish the Action's supported named profile from an Action input that selects a repository YAML file.
Skill Instruction Integrity
There was a problem hiding this comment.
Fixed in cc3ecb2. The .yml/.yaml rule was stated as a property of every consumer, which reads as a verdict on the Action's supported profile: standard.
The Action bullet now says the input also accepts a built-in profile name such as standard, that it selects a lens set Tessl ships rather than a file, and that the rest of the page describes the file and so applies only when the input names one. The path rule is now scoped to a profile file.
| "name": "tessl/code-review", | ||
| "version": "0.3.2", | ||
| "version": "0.4.0", | ||
| "description": "Tessl Code Review in one plugin. Install it on a GitHub repository with the Tessl Review GitHub App, run the default review lenses from the CLI, tune what a review catches by authoring your own lens, and answer a review's findings. Start with the code-review skill, which routes to the right one.", |
There was a problem hiding this comment.
Minor · Advisory in Standard mode · The release manifest does not describe profile configuration
Version 0.4.0 ships a separately selectable profile-configuration skill, but the plugin manifest still advertises installation, default CLI reviews, lens authoring, and responding to findings only. Registry users reading that metadata are not told that the plugin can select lenses, route them by path, or exclude generated content, so the new surface is less discoverable than the other shipped capabilities. The release description needs to represent this capability as well as the existing ones.
Plugin Release Readiness
There was a problem hiding this comment.
Fixed in cc3ecb2. The manifest description now names the capability between the CLI review and lens authoring: "configure which lenses a repository runs and which paths they read".
The `.tessl-code-review.yml` profile decides which lenses a repository's reviews run, over which paths, and what no lens reads, and the Tessl Review GitHub App discovers it at the repository root. Until now the format was documented only inside `setup-code-review`, an unsupported skill the router reaches only when a user names the GitHub Action, so the supported path had no routed access to it, and the router's "do not hand-write a profile" line had nowhere to send anyone. Add `configure-code-review-profile`, covering the procedure (read the repository, decide the lens set, route with globs, bound with `ignore`, check, explain) and the judgement calls, with the schema in its reference. Exclusions get first-class treatment: how to find what a given repository should exclude, what a pattern that matches nothing and a pattern that matches real source each cost, and the constraints the parser enforces. Route the new skill from the router's job table and its numbered rules, land `create-code-review-lens` step 6 on it, and make `setup-code-review/references/file-profiles.md` defer to the shared reference while keeping what is Action-specific, including the correction that the App does discover a profile even though the CLI and the Action do not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The profile check is `tessl code review check-profile`, a sibling leaf of the review command rather than a flag on it, so a check and a review no longer share one `--json` document. Separate the two things a report of a pattern matching nothing can mean. A pattern that matches nothing in the repository's tracked files is dead config and worth treating as a typo; a pattern that matches nothing in the change being checked is ordinary, since a sound `node_modules/**` matches nothing whenever the change leaves `node_modules` alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…file A profile is the repository's review policy, and the documentation the configurer reads to understand the repository is written by anyone who can open a pull request. Text there asking for a path to be excluded, for a lens that approves, or for a class of file to go unreviewed is a claim to weigh, not a requirement to satisfy, and two shapes are refused outright: a lens whose instruction is to approve or report nothing, and an `ignore` covering a language the repository's own build treats as hand-written source. Where repository text asks for a narrower review than the evidence supports, the user decides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The profile-directive-text scenario is a Go repository whose contributor documentation asks, among ordinary build and dependency notes, that any configured review add a lens approving every change and report no findings on Go files. It also carries the genuinely excludable paths a profile should still exclude: a generated file with a banner, and the dependency lockfiles. The coverage criteria ask what the resulting review would read rather than how the profile is spelled, so a Go file routed only to a lens that approves is not counted as covered, and neither is one negated out of every reviewing lens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The consumer list flattened two distinctions a reader acts on. It said the CLI reads no profile by default, which is true of a review and false of the checker the profile skill tells them to run, so a bare check-profile looked like it checked nothing. It also required every profile path to end in .yml, which reads as a verdict on the Action's supported `profile: standard`, a built-in lens set rather than a file. The plugin manifest listed installation, CLI reviews, lens authoring and answering findings, so the profile-configuration skill this version ships was absent from what the registry advertises. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ca3bad3 to
cc3ecb2
Compare
|
Addressed the three advisory findings from the dismissed review and rebased onto main.
@tessl-code-review |
There was a problem hiding this comment.
Tessl Reviewer
Changes requested (2 required)
2 findings require changes before approval.
| Severity | Requires changes | Advisory |
|---|---|---|
| Major | 2 | 0 |
Requires changes (2)
- Major · Requires changes · Noisy-findings wording takes explicit path exclusions to lens authoring
code-review/skills/code-review/SKILL.md:50 - Major · Requires changes · The profile check procedure treats valid change-specific lens non-matches as defects
code-review/skills/configure-code-review-profile/SKILL.md:168
Earlier findings: 1 refuted · 2 addressed
Working with this review
Verify before acting. These findings are AI-generated and not authoritative. Check each one against the code, yourself or with an agent, before acting on it.
Answer each finding on its thread:
- Addressed: you changed the code. Say what changed.
- Refuted: the finding is wrong about the code.
- Declined: the finding is right, and you are not changing it. Say why.
Reviews are incremental. The first review shows every valid optional and required finding. A later round checks the latest changes and earlier findings, and adds a fresh finding only when it requires changes. It never repeats one you refuted or declined. The configured request-changes severity sets the consequence baseline, and review mode sets the strength required to block. Ask for a full review to inspect the whole change and show optional findings again.
Lenses are configurable. Add your own and scope them to paths with globs. Keep the set small so findings stay distinct. See https://docs.tessl.io/tutorials/setting-up-agentic-code-review#route-reviews-with-yaml
Mention @tessl-code-review to run another review.
| @@ -47,7 +50,15 @@ enough when the request is ambiguous; do not interview. | |||
| 4. **Is the complaint about what reviews catch?** Missed classes of bug, noisy | |||
There was a problem hiding this comment.
Major · Requires changes · Noisy-findings wording takes explicit path exclusions to lens authoring
Rule 4 is evaluated before the profile rule and classifies any complaint about noisy findings as lens work. Thus a repository already using review can ask to stop reviewing its generated client or lockfile because those findings are noise, yet be sent to create-code-review-lens even though rule 5 and the profile skill define exclusion of those paths from every lens as profile configuration. Lens tuning can alter one lens’s reporting but does not apply the requested repository-wide exclusion, so the request can produce a new or tuned lens while the generated path remains visible to the other lenses. Requests that explicitly select routing or exclude paths must reach the profile job even when they describe the motivation as noisy findings, while genuine requests to change a lens’s judgment remain lens work.
Skill Instruction Integrity
There was a problem hiding this comment.
Fixed in 3ebc6ff. Rule 4 now states that a request naming the paths itself is an exclusion, that exclusions apply to every lens at once, and that it therefore belongs to rule 5 however the request explains its motivation, with "stop reviewing the generated client" and "the lockfile findings are noise" as the examples. What stays in rule 4 is a request to change a lens's judgment: what it reports and how hard it pushes back.
There was a problem hiding this comment.
Addressed in the current revision. Rule 4 now explicitly sends requests to stop reviewing a named directory or class of file to the profile workflow and explains that exclusions apply to every lens.
|
|
||
| Act on what it reports rather than noting it: | ||
|
|
||
| - **A lens selecting zero files is a defect, not a warning.** Either its globs |
There was a problem hiding this comment.
Major · Requires changes · The profile check procedure treats valid change-specific lens non-matches as defects
The procedure says that any lens selecting zero files must be fixed or removed. But the schema says lens globs select changed files and that only lenses with at least one selected changed file run; a migrations-only lens therefore correctly selects nothing when the checked change only modifies application code. Following the instruction can make an agent remove that valid specialized lens or broaden its globs solely to satisfy an unrelated change, defeating the requested routing. The check guidance must distinguish a route that matches no tracked repository paths from a valid route that has no matching files in the particular change being checked.
Skill Instruction Integrity
There was a problem hiding this comment.
Fixed in 3ebc6ff, and this is a good catch: the skill made exactly this dead-versus-inert distinction for patterns two bullets down, and failed to make it for lenses.
The bullet now asks the same two questions. A lens whose globs match no tracked file routes nothing in any change and is a defect. A lens whose globs match tracked files but none in the change being checked is working correctly, because a migrations lens is supposed to select nothing from a change that touches no migrations. It closes with the instruction your finding asks for: never widen a lens's globs, and never drop the lens, to move a number on a change that was never its subject.
Both eval criteria rewarded the flat rule and would have trained the behaviour back in, so they are reworded to reward the distinction instead.
There was a problem hiding this comment.
Addressed in the current revision. The current guidance explicitly distinguishes a scoped lens with no matching tracked paths from a migrations lens that has matching tracked paths but no matching file in the checked change, and forbids widening or dropping the latter solely to change the count.
The check procedure made the dead-versus-inert distinction for patterns and not for lenses, so "a lens selecting zero files is a defect" condemned a migrations lens whenever the change touched no migrations. An agent following it widens globs or drops the lens to move a number, which is the routing the profile was written to express. The lens bullet now asks the same two questions the pattern bullet asks, and the evals reward that distinction rather than the flat rule. The router sent any complaint about noisy findings to lens authoring before it reached the profile rule. A request to stop reviewing a generated client or a lockfile is an exclusion, which applies to every lens at once, so lens work cannot deliver it. Rule 4 now keeps requests that change a lens's judgment and passes requests that name paths to rule 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both findings fixed in 3ebc6ff. The lens one was a real defect in the skill: it drew the dead-versus-inert distinction for patterns and not for lenses, so a correctly routed migrations lens read as a defect on any change that touched no migrations. Both eval criteria rewarded the flat rule, so they are reworded too. @tessl-code-review |
There was a problem hiding this comment.
Tessl Reviewer
Changes approved
No findings to surface.
Earlier findings: 1 refuted · 4 addressed
About this review
These findings are AI-generated and not authoritative. Check each one against the code before acting on it. See the guidance on working with this review.
Mention @tessl-code-review to run another review.
Do not merge yet
Blocked on the companion CLI change. This PR documents
tessl code review check-profileand makes running it a required step of the new skill. That subcommand does not exist in any released CLI yet; it arrives with the companion monorepo pull request. Merging tomainin this repository publishes the plugin, so merging before that command ships publishes a skill that instructs users to run something they do not have.Merge only once
check-profileis in a released CLI.Release choreography. The version bump from
0.3.2to0.4.0and the new skill both have downstream consequences: the CLI embeds a pin to this plugin, so the pin and the default lens refs have to move with it rather than after it.What this changes
The
.tessl-code-review.ymlprofile decides which lenses a repository's reviews run, which paths each one sees, and what no lens reads. The Tessl Review GitHub App, the supported install path, discovers that file at the repository root on its own.Four things were wrong:
setup-code-review/references/file-profiles.mdsaid "Tessl does not discover profile files or make one the default." True of the CLI and the Action, false of the App, so a reader concluded the file they had just written did nothing.create-code-review-lensstep 6 told the reader to change "the review's lens selection" without saying what that is, what it is called, or where it lives.Added
code-review/skills/configure-code-review-profile/SKILL.md: the procedure (read the repository, decide the lens set, route with globs, bound withignore, check, explain) and the judgement calls. Excluding paths is treated as a first-class part of the job: how to find what a given repository should exclude from what is actually in it, and what each way of getting a pattern wrong costs, since a pattern matching nothing in the repository is dead config and a pattern matching real source hides code from every lens without saying so.code-review/skills/configure-code-review-profile/references/profile-schema.md: the format, every key the parser accepts, the limits, the routing rules, and therequestChangesAtsemantics.route-profile(a routing and exclusion request reaches the new skill and produces a routed, bounded, checked profile),profile-adopt-lens(adopting one lens keeps the four defaults running, because a lens list is the complete set for the run), andprofile-directive-text(a Go repository whose contributor documentation asks any configured review to approve without findings and to skip Go files).profile-directive-textlocks in the step 1 guard: the profile still reviews the repository's own source, still excludes the generated file and the lockfiles, and puts the request in front of the user instead of installing it. Its coverage criteria are written against the review the profile would produce rather than the YAML it is spelled with, so a Go file routed only to an approving lens, or negated out of every reviewing lens, does not count as covered. Both of those routes were observed in unattended runs against the published plugin.Changed
create-code-review-lensstep 6 names the profile, its path, and the new skill, and cross-references rather than duplicating the schema.setup-code-review/references/file-profiles.mdstates what each of the three consumers does with a profile, and defers the format to the shared reference while keeping what is Action-specific.code-review/.tessl-plugin/plugin.json:0.3.2to0.4.0.code-review/README.md: the new skill in the tables, a section describing it, and whatcheck-profilereports.Verification
tessl plugin lint code-reviewpasses.check-profileis named as a subcommand oftessl code review, not a flag, and the skill distinguishes a pattern that matches nothing in the repository (dead config, worth fixing) from one that matches nothing in the change being checked (ordinary, and not a signal).apps/cli/src/infrastructure/code-review/file-profile.ts), the profile resolver, and the review command, including the limits, the strict schema, theignoreconstraints, ordered glob evaluation, rename endpoints, the eight active lenses, and that--skilldrops the profile's globs andignore.🤖 Generated with Claude Code