Skip to content

docs: add flaky test quarantine and remediation policy#35

Merged
GoogilyBoogily merged 2 commits intomainfrom
add-flaky-test-policy
May 1, 2026
Merged

docs: add flaky test quarantine and remediation policy#35
GoogilyBoogily merged 2 commits intomainfrom
add-flaky-test-policy

Conversation

@GoogilyBoogily
Copy link
Copy Markdown
Contributor

Summary

  • Add CONTRIBUTING.md with Test Stability, Quarantine & Remediation Policy
  • Defines flaky test detection, quarantine via .skip, and 30-day remediation SLA

Test plan

  • Verify CONTRIBUTING.md renders correctly on GitHub

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 30, 2026 19:50
@GoogilyBoogily GoogilyBoogily requested a review from a team as a code owner April 30, 2026 19:50
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds contributor documentation describing local dev/test commands and a policy for detecting, quarantining, and remediating flaky tests in this repository.

Changes:

  • Introduces a new CONTRIBUTING.md with development + test instructions.
  • Documents a flaky test quarantine workflow using .skip plus a tracking issue.
  • Defines remediation expectations (30-day SLA, escalation after 14 days).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread CONTRIBUTING.md
});
```

3. **Skipped tests are excluded from coverage reporting.** Since quarantined tests do not execute, their associated source code is not counted toward coverage metrics. This is expected and acceptable while the test is in quarantine.
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The statement that skipped tests are "excluded from coverage reporting" is inaccurate for the current Jest setup (collectCoverageFrom includes all src files). Skipping a test usually means less code gets executed and coverage can drop (and potentially fail the configured coverageThreshold). Please reword to reflect how coverage works here and what maintainers should expect when quarantining tests.

Suggested change
3. **Skipped tests are excluded from coverage reporting.** Since quarantined tests do not execute, their associated source code is not counted toward coverage metrics. This is expected and acceptable while the test is in quarantine.
3. **Skipped tests do not run, so coverage may decrease.** In this Jest setup, coverage reporting still includes the configured source files even when a quarantined test is skipped. Because the skipped test no longer exercises its code paths, overall coverage can drop and may cause the configured `coverageThreshold` to fail. Maintainers should expect this when quarantining a test and either remediate the test promptly or make any necessary coverage-related follow-up as part of the quarantine plan.

Copilot uses AI. Check for mistakes.
Comment thread CONTRIBUTING.md

### Test Stability, Quarantine & Remediation Policy

Flaky tests are any test that passes and fails intermittently without code changes (this could be due to things like timeouts, inconsistent return values, poor assertions, etc). Flaky tests erode confidence in the test suite and slow down development. This section defines how we detect, quarantine, and remediate them.
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor grammar: "Flaky tests are any test" should be rephrased (e.g., "A flaky test is a test..." or "Flaky tests are tests...") to read correctly.

Suggested change
Flaky tests are any test that passes and fails intermittently without code changes (this could be due to things like timeouts, inconsistent return values, poor assertions, etc). Flaky tests erode confidence in the test suite and slow down development. This section defines how we detect, quarantine, and remediate them.
A flaky test is a test that passes and fails intermittently without code changes (this could be due to things like timeouts, inconsistent return values, poor assertions, etc). Flaky tests erode confidence in the test suite and slow down development. This section defines how we detect, quarantine, and remediate them.

Copilot uses AI. Check for mistakes.
Comment thread CONTRIBUTING.md

Thanks for considering contributing to this project. Ways you can help:

- [Create a pull request](https://help.github.com/articles/creating-a-pull-request)
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GitHub Help URL here is deprecated (help.github.com). Please update the link to the current docs.github.com page for creating a pull request so it doesn't break or redirect unexpectedly.

Suggested change
- [Create a pull request](https://help.github.com/articles/creating-a-pull-request)
- [Create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request)

Copilot uses AI. Check for mistakes.
Comment thread CONTRIBUTING.md

#### Detection

- **Automatic retries**: Jest unit tests can be configured to automatically retry on failure via `jest.retryTimes()`. Playwright integration tests (where applicable) have retries configured per-project. A test that fails once but passes on retry can be considered as potentially flaky.
Copy link

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This repo doesn't appear to use Playwright (no dependency/config found). Mentioning Playwright retries here is likely confusing—either remove the Playwright reference or replace it with the actual integration/e2e test tooling used in this project.

Suggested change
- **Automatic retries**: Jest unit tests can be configured to automatically retry on failure via `jest.retryTimes()`. Playwright integration tests (where applicable) have retries configured per-project. A test that fails once but passes on retry can be considered as potentially flaky.
- **Automatic retries**: Jest unit tests can be configured to automatically retry on failure via `jest.retryTimes()`. A test that fails once but passes on retry can be considered as potentially flaky.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member

@codentacos codentacos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 🧙

@GoogilyBoogily GoogilyBoogily merged commit 20b946b into main May 1, 2026
5 checks passed
@GoogilyBoogily GoogilyBoogily deleted the add-flaky-test-policy branch May 1, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants