fix(config): stop alwaysRun on @Before* from bypassing a configuration failure

[juherr]

Fix #1622

The bug

alwaysRun is documented to mean two different things, and every @Before*/@After* annotation repeats it verbatim:

For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.
For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.

Since 6.9.5 a @Before* method got both. The fix for #420 added the four getBefore*() terms to MethodHelper.isAlwaysRun, and ConfigInvoker.invokeConfigurations read that one predicate for two independent decisions:

• may a configuration of a @Test(enabled = false) class still run — this is what Before/AfterSuite methods may not run, when classes from suite use inheritance, and one of them has enabled=false set. #420 needs;
• may it run past a configuration failure — it does not.

So a failing @BeforeSuite, which marks the whole suite failed regardless of configFailurePolicy, was followed by every alwaysRun @BeforeTest, @BeforeClass and @BeforeMethod of the run — each setting up something no test would use, since the @Test methods were skipped either way. On the reporter's class that reads as 4 configuration failures and 0 skips where 6.9.4 reported 1 and 3.

The fix

The failure bypass now comes from a separate predicate, MethodHelper.canBypassConfigurationFailure, true only for @AfterSuite, @AfterTest, @AfterClass, @AfterMethod and @AfterGroups. isAlwaysRun keeps its meaning and its single remaining caller, the enabled decision #420 covers, so both halves of the documented contract now have a name. The two share their phase lists rather than repeating them.

This is deliberately not tied to configFailurePolicy (the approach #1633 took): a @BeforeSuite failure marks the suite failed whatever the policy says, and the same run reports the same invocations under SKIP and under CONTINUE. #420 is untouched, and so is the group meaning of alwaysRun on a @Before* method.

The second commit

Removing the bypass exposed a defect it had been hiding, so this PR carries it as its own commit. TestInvoker.invokeMethod already exempts a retried test method from the configuration failures of the attempt it retries, but nothing exempted that attempt's setup — so a retry whose @AfterMethod had failed ran the test method with no setup at all. Until now that was only reachable for a @BeforeMethod without alwaysRun, because alwaysRun bypassed the failure and hid it.

The retry bit already lives in FailureContext.representsRetriedMethod; ConfigMethodArguments now carries it, so the setup configurations of a retried invocation are exempt on the same terms as the method they set up. Teardown configurations are unchanged. It lands first so that no commit in this branch is red.

Behaviour changes

• An alwaysRun @Before* whose parent configuration failed is reported as skipped instead of run. A suite that relied on the regression — an alwaysRun @BeforeMethod doing work after its @BeforeClass failed — will see that method skipped.
• A retried invocation runs its @BeforeMethod again whether or not it says alwaysRun.

No API break.

Tests

test.configuration.issue1622.IssueTest drives the reporter's own class — a failing @BeforeSuite, alwaysRun @BeforeTest/@BeforeClass/@BeforeMethod, a @Test, and the four alwaysRun @After* — and asserts the exact invocation list with containsExactly, plus the reported configuration failure/skip/skipped-test counts. It runs the same suite twice, once with the default SKIP policy and once with CONTINUE, asserting the same list both times; that is the statement that these semantics do not depend on configFailurePolicy.

The oracle is a static list in the sample rather than an invoked-method listener, because ConfigInvoker fires BEFORE_INVOCATION on the skip path too and a listener could not tell "ran" from "skipped".

Measured, before → after (identical under both policies):

- [failingBeforeSuite, beforeTest, beforeClass, beforeMethod, afterMethod, afterClass, afterTest, afterSuite]
+ [failingBeforeSuite,                                        afterMethod, afterClass, afterTest, afterSuite]
- confFailures=1  confSkips=0  skippedTests=1
+ confFailures=1  confSkips=3  skippedTests=1

which is what 6.9.4 reported.

test.configuration.issue1753 relied on the regression: its parent @BeforeMethod failed and the alwaysRun child one ran anyway. The child is now the one that fails, which keeps the failing-configuration path #1753 is about and leaves its assertions unchanged.

Since that flips which method fails, a second hierarchy — FailingParentClassSample / ChildOfFailingParentSample — covers the case the first one no longer reaches: a parent @BeforeMethod that fails still contributes its attributes to the skipped result, while the child one, now skipped, contributes none. Measured keys on the skipped result:

ChildOfFailingParentSample-parentClassBeforeMethod   (failed, still contributed)
ChildOfFailingParentSample-parentClassAfterMethod    (alwaysRun @After)
ChildOfFailingParentSample-childClassAfterMethod     (alwaysRun @After)

containsOnlyKeys makes that exact, so it is also a guard on this PR: with the alwaysRun bypass restored, ChildOfFailingParentSample-childClassBeforeMethod reappears and the assertion fails.

Validation

./gradlew build is green on both commits — 0 failures, 0 errors across every result file, on the retry commit alone and on the branch head.

Red → green → red was checked on the final code: with the canBypassConfigurationFailure term reverted to !alwaysRun and the test kept, test.configuration.issue1622.IssueTest fails with but some elements were not expected: ["beforeTest", "beforeClass", "beforeMethod"] under both policies; restoring it turns it green.

Summary by CodeRabbit

• Bug Fixes

  • Corrected configuration execution after failures and during test retries.
  • Improved ordering for parallel configuration barriers and dependent groups.
  • Fixed repeated class/include execution and parameter handling in reports.
  • Reports now safely display null, failing, and array parameter values.
  • Improved CLI value parsing and suite validation.
  • Fixed XML reporting diagnostics and chronological panel rendering.

• New Features

  • Added configuration-method priorities.
  • Added test context start and end timestamps.
  • Added expanded schema support for suite validation.

• Documentation

  • Added TestNG 7.13.0 release notes and clarified group configuration lifecycle behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: c28f6112-0dca-45f3-a428-975f2f3bb3af

📥 Commits

Reviewing files that changed from the base of the PR and between b38122f and b03fc12.

📒 Files selected for processing (1)

• CHANGES.txt

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

📝 Walkthrough

Walkthrough

Changes

Configuration failure handling

Layer / File(s)	Summary
Configuration execution semantics testng-core/src/main/java/org/testng/internal/MethodHelper.java, testng-core/src/main/java/org/testng/internal/invokers/*	Configuration calls track retried test methods. Only eligible @After* methods can bypass prior configuration failures.
Configuration regression tests testng-core/src/test/java/test/configuration/issue1622/*, testng-core/src/test/java/test/configuration/issue1753/*, testng-core/src/test/resources/testng.xml	Tests verify setup skipping after failed suite or parent setup configurations and preserve eligible teardown execution.
Release notes and compatibility record CHANGES.txt	The 7.13.0 release notes document configuration, scheduling, reporting, API, parsing, CLI, concurrency, build, and compatibility changes.

Estimated code review effort: 2 (Simple) | ~15 minutes

Merge Risk: ⚪ Minimal · up to b03fc

This change documents the corrected configuration failure behavior, including skipping dependent always-run setup methods after prior setup failures. No current merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 11.90% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 42 functions across 10 files. (1 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main fix: alwaysRun on @Before* methods no longer bypasses configuration failures.
Linked Issues check	✅ Passed	The changes satisfy issue [#1622]. @Before* methods marked alwaysRun no longer bypass prior configuration failures, while @After* methods can still bypass them. Tests cover failed @BeforeSuite…
Out of Scope Changes check	✅ Passed	The changes remain related to configuration-failure handling. Added retry handling, regression fixtures, test registration, and release-note updates support or document the configuration behavior.

Full details: Linked Issues check

Explanation

The changes satisfy issue [#1622]. @Before* methods marked alwaysRun no longer bypass prior configuration failures, while @After* methods can still bypass them. Tests cover failed @BeforeSuite behavior under both SKIP and CONTINUE policies.

Full details: Docstring Coverage

Explanation

Docstring coverage is 11.90% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 42 functions across 10 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}