Skip to content

docs(skills): add pine-visual-test-pr visual regression skill - #788

Merged
QuintonJason merged 6 commits into
mainfrom
chore/pine-visual-test-pr-skill
Aug 14, 2026
Merged

docs(skills): add pine-visual-test-pr visual regression skill#788
QuintonJason merged 6 commits into
mainfrom
chore/pine-visual-test-pr-skill

Conversation

@QuintonJason

@QuintonJason QuintonJason commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What

Adds a pine-visual-test-pr Claude Code skill at .claude/skills/pine-visual-test-pr/SKILL.md — a self-iterating, Playwright-MCP visual-test harness for Pine PRs.

It's modeled on kajabi-products' kp-visual-test-pr, re-pointed at Pine's actual visual surface (Storybook) instead of Rails worktree servers.

How it works

  1. Preflightgh auth, root deps, libs/core scripts, Playwright MCP reachable.
  2. PR scope — reduces the diff to changed libs/core/src/components/pds-* dirs.
  3. Storybook upbuild.stencilstart.storybook (:6006) → enumerate every story from /index.json (the Pine analog of worktree-server list; never guesses story IDs).
  4. Capture plan — maps changed components → story IDs via each story file's title:, then builds a shot matrix.
  5. Capture + evaluate loop — screenshots the chrome-free iframe.html?id=…&globals=theme:… across light/dark themes, viewports, and interactive states (hover/focus/open/disabled/error); self-evaluates each against embedded rules; retries harness failures (max 3), records genuine theme/focus/console/token problems as findings.
  6. Report — posts a markdown report as a PR comment.

Optional before/after baseline mode (checkout base → rebuild Stencil → re-shoot the same matrix) labels each delta intended vs regression, gated behind a clean-tree confirmation.

Design notes

  • Dark theme is mandatory — dark-mode/token regressions are Pine's most common and easiest-to-miss visual bug, so light+dark is always captured.
  • Findings ≠ blockers — a broken theme, missing focus ring, or console error is the point of the run and is recorded as a finding, not retried away.
  • Complements Chromatic — positions itself as the fast, local, pre-CI pass; Chromatic (@chromatic-com/storybook) remains the authoritative pixel-diff gate in CI.

Scope

Docs/tooling only — one new markdown skill file. No component, build, or runtime code touched.


Note

Low Risk
Single new markdown skill file with no changes to components, builds, or production runtime.

Overview
Adds a new Claude Code skill at .claude/skills/pine-visual-test-pr/SKILL.md that documents an end-to-end Playwright MCP visual regression workflow for Pine PRs (adapted from kajabi-products’ kp-visual-test-pr, but aimed at Storybook instead of app servers).

The skill defines a seven-phase runbook: environment preflight, PR/component scoping from gh pr diff, Stencil build + Storybook on :6006 with story IDs from /index.json, a capture matrix (light/dark, viewports, hover/focus/open states), optional base-branch baseline screenshots (with explicit git/Storybook cleanup around createComponent.tsx and port 6006), rule-based evaluation with retries vs findings, and a deduped gh pr comment report under .claude/visual-test-reports/.

It positions the harness as Pine’s current primary local pixel check until Chromatic CI merges, and links it to pine-design-review and pine-run-gauntlet. Docs/tooling only — no component, build, or runtime code changes.

Reviewed by Cursor Bugbot for commit ddf7b5d. Bugbot is set up for automated code reviews on this repo. Configure here.

Self-iterating Playwright-MCP visual test harness for Pine PRs, modeled on
kajabi-products' kp-visual-test-pr but re-pointed at Pine's Storybook.

Builds Stencil, serves Storybook, maps changed pds-* components to their
stories via /index.json, and captures screenshots across light/dark themes,
viewports, and interactive states. Self-evaluates each shot against embedded
rules (dark-mode/theme, focus-ring, console errors, token discipline),
retries harness failures, records genuine visual/a11y regressions as
findings, and posts a markdown report as a PR comment. Optional before/after
baseline diff for modified components. Complements Chromatic (local, fast,
pre-CI); does not replace it.
@QuintonJason
QuintonJason requested a review from a team as a code owner August 10, 2026 13:38
@netlify

netlify Bot commented Aug 10, 2026

Copy link
Copy Markdown

Deploy Preview for pine-design-system ready!

Name Link
🔨 Latest commit ddf7b5d
🔍 Latest deploy log https://app.netlify.com/projects/pine-design-system/deploys/6a7a3fddf985470008963978
😎 Deploy Preview https://deploy-preview-788--pine-design-system.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Dry run against the pds-tabs PR surfaced three fixes:
- Rule 4: whitelist harness noise (favicon/static 404s, Vite HMR); only
  component-origin errors (pine-core/*.js stack) count as findings.
- Screenshots: require an absolute output path into a per-run dir — the
  Playwright MCP browser cwd differs from the shell cwd, so relative
  filenames land outside the repo.
- Story mapping: match story titles by exact/nested-path equality, not
  substring (includes('tab') wrongly matched Table and Sortable).
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Running baseline mode against #785 surfaced two more fixes:
- Console read must be per-navigation, not all:true — session history
  floods with stale ERR_CONNECTION_REFUSED / NoStoryMatchError after any
  Storybook restart, none of which are findings.
- Phase 5b: build.stencil dirties tracked generated files
  (components.d.ts, react createComponent.tsx), aborting the branch switch;
  reset them before each checkout, restart Storybook on the base (stale
  index → NoStoryMatchError), and compare only stories shared with base.
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated

@pixelflips pixelflips left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review — 9 inline findings, see comments on the relevant lines.

Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Resolves Cursor + pixelflips findings on pine-visual-test-pr:
- Chromatic hedged everywhere — the CI job isn't merged (only the local
  Storybook panel is); this skill is currently Pine's only pixel-level check.
- Phase 1 threads an explicit <number> through gh pr diff/view (bare calls
  resolve against the current branch, mis-scoping an explicitly-passed PR).
- Dropped the half-wired RTL axis (planned but never looped/graded).
- All Phase 5/6 capture commands use the absolute <REPORT_DIR> path (the
  L201-213 rule was only in prose, not the example commands).
- Console check is per-navigation AND per-theme (a single end-of-story read
  missed the earlier theme); never all:true.
- Phase 5b: createComponent.tsx carries an authored double-registration
  guard that build.stencil strips — restore via git checkout -- after the
  final rebuild; added an explicit Storybook stop (lsof kill) before restart;
  Phase 2 'reuse the server' now carves out the post-branch-switch case.
- Phase 6: write the report file before cat-ing it; post via --body-file
  with --repo (not double-quoted $(cat)); added a once-per-PR dedup guard
  and the 'Posted by pine-visual-test-pr' marker.
@QuintonJason

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review — all findings addressed in e83a33d.

pixelflips (9):

  1. Chromatic present-tense CI gate — hedged everywhere (intro, when-to-use, Phase 4, Completion, anti-patterns, related-skills). Now states plainly that only the local @chromatic-com/storybook panel is merged and this skill is currently Pine's only pixel-level check until the CI job lands.
  2. PR number never reaches gh pr diff — Phase 1 now resolves an explicit <number> and threads it through every gh pr diff/view call, with a note on the wrong-branch trap.
  3. RTL axis half-wired — dropped it (removed from the axes and the URL template; report template no longer lists +rtl), with a note on how to add it as a full axis later.
  4. Capture commands violate the absolute-path rule — all Phase 5 capture calls (Steps 3 & 4) and the interactive-state calls now use the absolute <REPORT_DIR> path; report template updated too.
  5. createComponent.tsx is not "safe to discard" — corrected: it carries the hand-authored double-registration guard; git checkout -- is kept because it restores the committed (guarded) copy, and Phase 5b step 5 now re-runs it after the final rebuild (which re-strips the guard), with a git status clean-check. Added an anti-pattern against git add-ing the stripped version.
  6. "restart" had no implementation — Phase 5b now has an explicit lsof -ti:6006 | xargs kill stop step before re-starting; Phase 2's "reuse the existing server" carves out the post-branch-switch case in both the prose and the error-handling table.
  7. Report file never written but cat-read — Phase 6 step 1 now writes the filled report to $REPORT via the Write tool before posting.
  8. No dedup guard on posting — added the once-per-PR guard (grep for the Posted by pine-visual-test-pr marker, skip unless asked), matching the pine-run-gauntlet pattern; the marker is now in the report template footer.
  9. Double-quoted --body + missing --repo — posting now uses gh pr comment <number> --repo Kajabi/pine --body-file "$REPORT".

Cursor:

  • Console check spans whole session (all: true) — already removed in a prior commit; the read is per-navigation.
  • Per-navigation read misses the earlier theme — fixed: the console read now runs after each theme's navigation (inside the light/dark loop in Step 3, and again after interactions in Step 4), so both themes are judged rather than only the last one.

(This reply posted via --body-file — dogfooding fix #9.)

Comment thread .claude/skills/pine-visual-test-pr/SKILL.md
Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Cursor Bugbot re-review flagged two issues introduced by the prior fix:
- Cross-branch capture (High): threading the PR number scoped the right
  files but Phase 2 still builds/screenshots the checked-out tree. Added a
  Phase 1 branch-match guard that stops on mismatch (no auto-switch).
- Stash ordering (Medium): Phase 5b stashed WIP before resetting the
  generated files, so the stash could snapshot the vendor-stripped
  createComponent.tsx and reintroduce it on pop. Reordered: reset generated
  files first, then stash only genuine WIP.
@QuintonJason

Copy link
Copy Markdown
Contributor Author

Both re-review findings fixed in the latest commit:

  • Cross-branch PR screenshots (High) — good catch; threading the number scoped the right files but capture still renders the checked-out tree. Phase 1 now has an explicit branch-match guard (git branch --show-current vs the PR's headRefName) that stops on mismatch and tells the user to check out the head branch first — it does not auto-switch.
  • Stash can restore stripped guard (Medium) — reordered Phase 5b: reset the two build-generated files (git checkout --) before stashing any genuine WIP, so the stash can never snapshot the vendor-stripped createComponent.tsx and reintroduce it on pop.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e857714. Configure here.

Comment thread .claude/skills/pine-visual-test-pr/SKILL.md Outdated
Re-review flagged that restoring the step-3 WIP stash and then requiring a
clean git status contradict each other. The clean check now applies to the
skill's own artifacts (the two generated files match committed) and runs
before restoring the stash; the stash restore comes last and legitimately
re-dirties the tree with the user's own WIP.
@QuintonJason

Copy link
Copy Markdown
Contributor Author

Fixed in the latest commit. The clean check now applies to skill-induced changes only (the two generated files matching committed), and runs before restoring the step-3 stash. Restoring the user's WIP is done last and legitimately re-dirties the tree — that's expected and no longer treated as a failed clean check.

@pixelflips pixelflips left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing all the feedback. LGTM! 👍🏼

@QuintonJason
QuintonJason merged commit a010fe8 into main Aug 14, 2026
21 checks passed
@QuintonJason
QuintonJason deleted the chore/pine-visual-test-pr-skill branch August 14, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants