-
Notifications
You must be signed in to change notification settings - Fork 1
docs: add flaky test quarantine and remediation policy #35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,66 @@ | ||||||
| # Contributing | ||||||
|
|
||||||
| Thanks for considering contributing to this project. Ways you can help: | ||||||
|
|
||||||
| - [Create a pull request](https://help.github.com/articles/creating-a-pull-request) | ||||||
| - [Add an issue](https://github.com/braintree/class-list/issues) | ||||||
|
|
||||||
| ## Development | ||||||
|
|
||||||
| Clone this repo, then install the project's dependencies: | ||||||
|
|
||||||
| ``` | ||||||
| npm install | ||||||
| ``` | ||||||
|
|
||||||
| ## Tests | ||||||
|
|
||||||
| ``` | ||||||
| npm test | ||||||
| ``` | ||||||
|
|
||||||
| ### Test Stability, Quarantine & Remediation Policy | ||||||
|
|
||||||
| Flaky tests are any test that passes and fails intermittently without code changes (this could be due to things like timeouts, inconsistent return values, poor assertions, etc). Flaky tests erode confidence in the test suite and slow down development. This section defines how we detect, quarantine, and remediate them. | ||||||
|
||||||
| Flaky tests are any test that passes and fails intermittently without code changes (this could be due to things like timeouts, inconsistent return values, poor assertions, etc). Flaky tests erode confidence in the test suite and slow down development. This section defines how we detect, quarantine, and remediate them. | |
| A flaky test is a test that passes and fails intermittently without code changes (this could be due to things like timeouts, inconsistent return values, poor assertions, etc). Flaky tests erode confidence in the test suite and slow down development. This section defines how we detect, quarantine, and remediate them. |
Copilot
AI
Apr 30, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This repo doesn't appear to use Playwright (no dependency/config found). Mentioning Playwright retries here is likely confusing—either remove the Playwright reference or replace it with the actual integration/e2e test tooling used in this project.
| - **Automatic retries**: Jest unit tests can be configured to automatically retry on failure via `jest.retryTimes()`. Playwright integration tests (where applicable) have retries configured per-project. A test that fails once but passes on retry can be considered as potentially flaky. | |
| - **Automatic retries**: Jest unit tests can be configured to automatically retry on failure via `jest.retryTimes()`. A test that fails once but passes on retry can be considered as potentially flaky. |
Copilot
AI
Apr 30, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The statement that skipped tests are "excluded from coverage reporting" is inaccurate for the current Jest setup (collectCoverageFrom includes all src files). Skipping a test usually means less code gets executed and coverage can drop (and potentially fail the configured coverageThreshold). Please reword to reflect how coverage works here and what maintainers should expect when quarantining tests.
| 3. **Skipped tests are excluded from coverage reporting.** Since quarantined tests do not execute, their associated source code is not counted toward coverage metrics. This is expected and acceptable while the test is in quarantine. | |
| 3. **Skipped tests do not run, so coverage may decrease.** In this Jest setup, coverage reporting still includes the configured source files even when a quarantined test is skipped. Because the skipped test no longer exercises its code paths, overall coverage can drop and may cause the configured `coverageThreshold` to fail. Maintainers should expect this when quarantining a test and either remediate the test promptly or make any necessary coverage-related follow-up as part of the quarantine plan. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The GitHub Help URL here is deprecated (help.github.com). Please update the link to the current docs.github.com page for creating a pull request so it doesn't break or redirect unexpectedly.