Classify Python tests by the SDK device-requirement marker by j-ororke · Pull Request #399 · project-chip/certification-tool-backend

j-ororke · 2026-10-14T16:10:29Z

What

test_harness_client.py reports each test class's device-requirement marker (device_requirement() from matter.testing.matter_testing), and parse_python_script uses it to pick the suite:

Marker	Suite
MatterTestCommissionedDevice	COMMISSIONING
MatterTestUncommissionedDevice, MatterTestCommissioner, CertificationUnitTestNoDevice	NO_COMMISSIONING
None, an unknown marker (logged as a warning), or an SDK without markers	Step 1's is_commissioning flag (unchanged)

MANDATORY and LEGACY classification is unchanged.

Why

The SDK declares each test's required DUT state with a marker base class (project-chip/connectedhomeip#73214). project-chip/connectedhomeip#73565 drops the now redundant is_commissioning=True from step 1. Without this change, those tests would land in the "No commissioning" suite and run against an uncommissioned DUT.

Validation on a TH

Raspberry Pi (arm64) running TH v2.16-beta3+summer2026, with the backend image built from this branch:

Baseline: brought up stock beta3 and saved its suite listing (975 test cases).
With the pinned SDK (99a81bd): the suite contents are identical to stock beta3, every test-info entry has device_requirement, the 27 marked tests are in the suite their marker implies, and there are no contradiction warnings.
Loaded [Framework] Enforce commissioning-state preconditions in device markers connectedhomeip#73565 with fetch_sdk_tests_and_runner.sh --sdk-path and th-cli rescan-tests. No chip-cert-bins image newer than 99a81bd is published, so the SDK container used chip-cert-bins:99a81bd with that PR's matter_testing.py.
With #73565, all 605 compared tests are in the expected suite. The 483 commissioned-device tests whose step 1 is no longer flagged are in "Auto commissioning"; the old rule would put them in "No auto commissioning". TC_WebRTCR_2_1 to 2_7 (MatterTestCommissioner) are in "No auto commissioning".
Ran with th-cli against all-devices-app, commissioned on-network:
- TC_IDM_9_1 (no marker, step-1 flag): passed.
- TC_CADMIN_1_9 (MatterTestCommissionedDevice, no flag): passed.
- TC_CADMIN_1_9 against a DUT that isn't on the TH fabric: fails in setup_class on #73565's precondition, as intended.
- TC_DD_1_16 and TC_DD_1_17 (MatterTestUncommissionedDevice, no flag), on a factory-fresh DUT: passed.

Tests

New test_python_parser_device_requirement. With the pinned SDK (99a81bd), all Python tests keep their current suite.

Validated on a Raspberry Pi TH (v2.16-beta3 top level, this backend) with connectedhomeip#73565 loaded through fetch_sdk_tests_and_runner.sh --sdk-path. TC_IDM_9_1 (unmarked, step flag) and TC_CADMIN_1_9 (MatterTestCommissionedDevice, no flag) pass through th-cli against all-devices-app.

Notes

Merge connectedhomeip#73565 before, or together with, any SDK_SHA bump past 99a81bd. On current SDK master, TC_WEBRTCR_2_1–2_7 are marked MatterTestCommissionedDevice, so this change would move them into the commissioning suite. #73565 marks them MatterTestCommissioner.

The SDK now declares each test's required DUT state with a marker base class and is dropping is_commissioning=True from step 1. Report the marker from test_harness_client.py and use it to pick COMMISSIONING or NO_COMMISSIONING, falling back to the step-1 flag for unmarked tests and SDKs without markers. This is a companion change to work with changes made in SDK PR project-chip/connectedhomeip#73565

coderabbitai · 2026-10-14T16:10:41Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 80236441-c1dd-410e-9fd7-2e3cc8573c76

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 0bbedc6b-eace-4971-a277-92f5e109c745

📥 Commits

Reviewing files that changed from the base of the PR and between c7635b6 and af4adcf.

📒 Files selected for processing: 4

test_collections/matter/sdk_tests/support/python_testing/models/python_test_parser.py
test_collections/matter/sdk_tests/support/python_testing/models/rpc_client/test_harness_client.py
test_collections/matter/sdk_tests/support/tests/python_tests/test_python_parser.py
test_collections/matter/sdk_tests/support/tests/python_tests/test_python_script/python_tests_info_markers.json

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Test-info generation now adds a device_requirement marker name to each test entry when available. The parser uses recognized markers to determine whether a commissioned device is required. If the marker is missing or unrecognized, it uses the first step’s commissioning flag, or False when there are no steps. Contradictions produce a warning. Tests cover marker, fallback, and contradiction cases.

Sequence Diagram(s)

sequenceDiagram
  participant TestHarnessClient
  participant SDK
  participant TestInfo
  participant python_test_parser
  TestHarnessClient->>SDK: Resolve device requirement for test class
  SDK-->>TestHarnessClient: Return marker when available
  TestHarnessClient->>TestInfo: Add device_requirement marker name
  TestInfo->>python_test_parser: Provide marker and step flags
  python_test_parser->>python_test_parser: Determine test-level commissioning requirement

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to af4ad

No concrete failure is established, so the change is mergeable with normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 3 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely identifies the main change: classifying Python tests using the SDK device-requirement marker.
Description check		The description directly explains the marker-based classification, fallback behavior, motivation, validation, and SDK compatibility considerations.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 3 files. (1 skipped: 1 unsupported.)

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

mergify · 2026-10-14T16:11:11Z

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

…r warnings

We discovered this dead code during review of project-chip#399 . This removes it and some more code that can be removed because of _that_ removal. Also adjusts the description of to be accurate.

Split out of #73565 at review: the step tag removal and marker fixes land here, and #73565 keeps the setup_class precondition. The device-requirement marker declares each test's DUT state, and the TH classifies tests by it (project-chip/certification-tool-backend#399), so drop the redundant step-1 is_commissioning=True from MatterTestCommissionedDevice tests. BasicCompositionTests-based tests have no marker and keep the flag. TH builds without #399 still classify by the flag, so this needs a TH that includes it. Also fix markers the TH now relies on: TC_WEBRTCR_2_1-2_7 are MatterTestCommissioner (the DUT is the commissioner and never joins the TH fabric), and TestCreateNewController is CertificationUnitTestNoDevice.

j-ororke self-assigned this Oct 14, 2026

j-ororke marked this pull request as draft October 14, 2026 16:11

j-ororke marked this pull request as ready for review October 14, 2026 18:45

j-ororke requested review from greens and rquidute October 14, 2026 19:00

j-ororke mentioned this pull request Oct 14, 2026

[Framework] Enforce commissioning-state preconditions in device markers project-chip/connectedhomeip#73565

Open

greens reviewed Oct 14, 2026

View reviewed changes

Comment thread test_collections/matter/sdk_tests/support/python_testing/models/python_test_parser.py Outdated

greens reviewed Oct 14, 2026

View reviewed changes

Comment thread test_collections/matter/sdk_tests/support/python_testing/models/python_test_parser.py

greens reviewed Oct 14, 2026

View reviewed changes

Comment thread test_collections/matter/sdk_tests/support/python_testing/models/python_test_parser.py

greens reviewed Oct 14, 2026

View reviewed changes

Comment thread test_collections/matter/sdk_tests/support/python_testing/models/python_test_parser.py

rquidute requested a review from antonio-amjr October 14, 2026 19:48

rquidute approved these changes Oct 14, 2026

View reviewed changes

Comment thread test_collections/matter/sdk_tests/support/python_testing/models/python_test_parser.py

Comment thread ...collections/matter/sdk_tests/support/python_testing/models/rpc_client/test_harness_client.py Outdated

j-ororke added 3 commits October 14, 2026 13:18

Fix is_commissioning typo in parse_python_script

d04f499

Warn on unknown device-requirement markers

263bcc9

Treat a missing step is_commissioning as unflagged and test the marke…

47470db

…r warnings

greens approved these changes Oct 14, 2026

View reviewed changes

antonio-amjr approved these changes Oct 15, 2026

View reviewed changes

Warn when a test class declares conflicting device-requirement markers

2905b5a

rquidute merged commit 5ab3b1d into project-chip:v2.16-develop Oct 15, 2026
9 checks passed

j-ororke mentioned this pull request Oct 15, 2026

Remove is_commissioning=True from commissioned-device tests project-chip/connectedhomeip#74796

Merged

greens mentioned this pull request Oct 15, 2026

[Cleanup] Removes dead code #400

Merged

Navigation Menu

Repository navigation

Classify Python tests by the SDK device-requirement marker - #399

Classify Python tests by the SDK device-requirement marker#399
rquidute merged 5 commits into
project-chip:v2.16-developfrom
j-ororke:feature/runner-usage-device-markers

j-ororke commented Oct 14, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading

Review skipped

Sequence Diagram(s)

Uh oh!

mergify Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

j-ororke commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Validation on a TH

Tests

Notes

Uh oh!

coderabbitai Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Sequence Diagram(s)

Uh oh!

mergify Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

j-ororke commented Oct 14, 2026 •

edited

Loading

coderabbitai Bot commented Oct 14, 2026 •

edited

Loading

mergify Bot commented Oct 14, 2026 •

edited

Loading