Skip to content

post_bot_commits counts local-tier agent commits as human rework #1440

Description

@The01Geek

Problem Statement

The retrospective's post_bot_commits field is meant to measure human rework after the bot's last push. It miscounts the engine's own local-tier agent commits as human rework, so the clean gate forces LLM analysis on PRs that had none.

Current Behavior

lib/fetch-pr-context.sh (the POST_BOT_COMMITS jq block) finds the last commit whose author_login or committer_login ends in [bot] or equals the PR author, then counts every non-merge commit after it. Local-tier Claude Code sessions commit under a git identity GitHub cannot resolve to an account, so the API returns a blank login. A blank login fails the [bot] test, so every local-tier agent commit after the last cloud-bot commit is counted as human rework.

Measured over 232 merged PRs / 1,912 commits: only three distinct author_login values exist — blank (1,080), github-actions[bot] (673), and the repo owner (159, of which 158 are merge commits already excluded). Essentially one identifiable human non-merge commit exists in the corpus, yet post_bot_commits > 0 on 96 of 232 PRs (41%). A run-boundary cross-check (commits after the ready_for_review timeline event) disagrees on 42 of 219 PRs, systematically the local-tier-agent case; PR 772 reports post_bot_commits: 8 where all eight counted commits carry fix-loop messages and predate ready_for_review.

Desired Behavior

A non-merge commit counts toward post_bot_commits only when it is positively attributable to a human: its author_login or committer_login is non-blank, does not end in [bot], and is not the PR author. A commit whose login the API returns blank is treated as agent-side, never as human rework — unknown is not a human.

User Impact

The weekly retrospective's clean gate stops forcing LLM analysis on ~41% of PRs whose only "rework" was the engine's own fix-loop commits, and consumers of the field stop drawing a wrong human-effort conclusion.

Technical Context

Scope note: The files and details below are the known starting points, not the full list. Before implementing, trace the change through the codebase to find every affected call site, consumer, and layer — this issue maps the work, it does not bound it.

  • Relevant Classes/Fileslib/fetch-pr-context.sh (the POST_BOT_COMMITS jq filter and its explanatory comment), lib/cheap-gate.jq (sole consumer: the post_bot_commits > 0 non-clean arm and the field's header comment).
  • Architecture Alignment — this repo's unknown is not zero rule: an unestablished measurement never collapses onto a real value. A blank login is unestablished identity and must not collapse onto "human".
  • Dependencies — none beyond the existing gh/jq path.
  • Data/Schema Considerations — the field's type and name are unchanged; only its counting rule changes.
  • Cross-layer Impact — retrospective scan path only; no skill, workflow, or config surface.

Acceptance Criteria

  • A PR whose commits after the last bot/author commit all carry blank author_login and blank committer_login reports post_bot_commits: 0.
  • A non-merge commit after the last bot/author commit whose author_login or committer_login is a non-blank login that neither ends in [bot] nor equals the PR author is still counted.
  • Merge commits (parents_count > 1) remain excluded from the count.
  • The comment above the jq block and the field's description in lib/cheap-gate.jq state what the field now measures, including that blank-login commits are classified agent-side.
  • lib/cheap-gate.jq's post_bot_commits > 0 non-clean arm is unchanged.

Implementation Notes

  • Approach — narrow the counted tail in the POST_BOT_COMMITS jq filter so only positively human-attributable non-merge commits count; the bot-side "last bot commit" anchor logic keeps its current meaning.
  • Relevant files — this likely touches only lib/fetch-pr-context.sh, plus the field comment in lib/cheap-gate.jq.
  • Code Patterns — the filter runs on API-shaped commit JSON; guard possibly-non-string fields with (.x | strings) before string ops, per the repo's embedded-jq rules.
  • Testing Strategylib/fetch-pr-context.sh is unmodularized (run.sh-resident), so drive the jq filter directly against fixture commit arrays: a blank-login tail (asserts 0), a mixed tail with one named human commit (asserts 1), a merge-commit-only tail (asserts 0), and a null-login record (asserts no filter abort). Each fixture assertion fails first against today's filter where the behavior changes.
  • Documentation Needed — none. The behavior is internal to the retrospective scan and its contract lives in the two file comments the ACs cover.
  • Potential Gotchas — a genuinely human commit made with an unresolvable email becomes invisible to the count; the measured corpus shows this population is ~1 commit in 1,912, and the field's purpose (a cheap non-clean trigger) tolerates that false negative far better than the current 41% false-positive rate. Historical retrospective records are not rewritten.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions