api compat tests - #448
Draft
nashif wants to merge 9 commits into
Draft
api compat tests#448nashif wants to merge 9 commits into
nashif wants to merge 9 commits into
Conversation
nashif
commented
Jul 27, 2026
nashif
commented
Member
- scripts: ci: add API lifecycle compatibility checks
- scripts: ci: api_compat: match removal advice to the lifecycle
- scripts: ci: api_compat: add an HTML report
- scripts: ci: api_compat: propose lifecycle states from tree evidence
- scripts: ci: api_compat: inherit @Version from enclosing groups
- .github: add an API compatibility workflow
- dummy API change
Add tooling that reports changes to public API headers which break the guarantees implied by the API's documented lifecycle state. A textual diff cannot answer either question that matters. It misses changes that never touch a prototype -- reordered struct fields, enumerators inserted mid-enum, changed macro values -- all of which keep compiling and change behavior at run time. Nor can it say whether a change was permitted: that follows from the semantic version on the owning Doxygen group, per doc/develop/api/overview.rst, which makes the same edit routine churn in an experimental API and a contract violation in a stable one. Three checks are provided. group-metadata requires a newly declared @defgroup to carry a valid @SInCE and @Version. deprecation-version enforces the minor version bump that overview.rst requires when an API gains a deprecation. Both need only the headers and the diff. signature compares every public symbol between two Doxygen XML snapshots and splits findings by whether they break the build or silently change behavior, grading severity by the lifecycle of the enclosing group. Doxygen output is used rather than a C parser because it already resolves group membership through @addtogroup / @{ @} nesting and carries the @SInCE and @Version tags, with no need for include paths or Kconfig-dependent preprocessor state. Doxygen's macro expansion is not stable across runs, so types are normalized before comparison; without that, 11 of 13 findings on a measured run were artefacts in files that had not changed. Groups with no @Version, currently 78% of the tree, are treated as stable so that the check fails closed. group-metadata looks only at newly added groups so the existing backlog does not drown the signal. The package is standalone and does not depend on check_compliance.py; a ComplianceTest wrapper is sketched in the README for wiring it in later. Tested with pytest scripts/tests/ci/test_api_compat.py (60 tests), an audit over include/zephyr (1244 groups), the header checks against real merged commits, and compare-revs over HEAD~5..HEAD, which correctly identified the two functions removed by the reverts at HEAD. Assisted-by: Claude:claude-opus-4-8
The detail text reported for a removed symbol was hardcoded to the stable API rule, so an experimental or unstable API was told it needed a two release deprecation period even though the same finding was correctly graded as a note. The advice contradicted the severity sitting next to it. Derive the text from the same lifecycle the severity is derived from. api_lifecycle.rst imposes a deprecation period only on stable APIs: experimental ones may be removed at any time, and unstable ones may be removed without announcement. A group carrying no @Version now also states that it is being treated as stable and points at --unversioned-is, which until now was an invisible assumption. Found while comparing v4.4.0 with 3df5fa4, where the removal of min3 from the experimental sys-util group was reported with advice that only applies to stable APIs. Tested with pytest scripts/tests/ci/test_api_compat.py (62 tests), including a case asserting that the experimental wording never claims a deprecation period is required, so the text cannot drift from the severity again. Assisted-by: Claude:claude-opus-4-8
A release-to-release comparison produces thousands of findings, which is more than terminal output is useful for. Add --format html, which writes a standalone page grouping findings by file and filtering them client side by severity, lifecycle state and check, with a search box over symbol, group, file and message. Summary tiles count errors, warnings, notes and silent behavior changes. The last of those is the number worth reading first: those findings still compile for downstream users and change what the code does. The page has no external references, so it opens from disk and can be attached to a CI job as an artifact. It follows the viewer's light or dark theme, and every severity carries a text label rather than relying on color alone. Add -o/--output so a report can be written to a file from any subcommand. The lifecycle filter is built from the values actually present rather than from the four known states. A finding that resolves to no group carries no lifecycle, which the deprecation check emits for a deprecation outside any @defgroup; without a matching checkbox such a finding was rendered but hidden by the filter permanently. Tested with pytest scripts/tests/ci/test_api_compat.py (70 tests), which now covers filter reachability for every facet value, balanced markup, escaping of hostile content, and that the page stays self contained. The report was generated from a real Doxygen comparison, and at 6000 synthetic findings renders to 5 MB in 0.05 s. Assisted-by: Claude:claude-opus-4-8
78% of API groups declare no @Version, so every other check in this package has to guess at their guarantees. Add a propose mode that walks headers under a given path, gathers the evidence api_lifecycle.rst asks for, and suggests a state per group along with the @SInCE and @Version tags to paste onto the @defgroup. Peripheral APIs are judged on releases and implementations: fewer than two of either keeps them experimental, two of each makes them unstable, and three releases plus substantial use makes them a stable candidate. Hardware-agnostic APIs have no implementations to count, so they are judged on releases and in-tree users. Three measurements back this, each with a trap that had to be handled. Release age comes from git following renames. Without --follow every header dates to 3.0, when headers moved under include/zephyr. Release tags are the vX.Y.0 tags whose timestamps increase in version order: sorting alone admits a stray v5.0.0 pointing at an unrelated commit, while filtering by reachability from HEAD discards the LTS releases tagged on branches. Validated against the groups that do declare @SInCE: gpio resolves to 1.0, w1 to 3.2 and nvmem to 4.3. Implementations come from DEVICE_API(<class>, ...) definitions, with the class read from the header's struct <class>_driver_api rather than guessed from the filename, since watchdog.h declares class wdt and uart.h keeps its vtable in a nested _internal.h. Where a devicetree binding directory exists the count is capped by the distinct non-Zephyr vendor prefixes among its compatibles, which counts hardware rather than files. Around eighteen classes have no vtable at all, among them pinctrl, hwinfo and cache; for those the count is unknown rather than zero and the API falls back to the agnostic rule, because treating it as zero marks long-stable APIs experimental. Users exclude the API's own implementation directories. w1.h has 18 includes but 15 are its own drivers, which made a niche API look adopted enough to propose as stable. Per-vendor and emulator headers nested under a driver class are skipped by default, and the number skipped is reported. They extend an API rather than declaring one, and left in they collect the SoC files that include them as users: they accounted for 16 of 46 stable candidates under include/zephyr/drivers before being excluded. A stable proposal stays a candidate: the remaining criteria, 100% test coverage and complete documentation, are not measured here. Tested with pytest scripts/tests/ci/test_api_compat.py (96 tests) and against include/zephyr/drivers, which yields 47 proposals over 80 API headers whose stable candidates are gpio, uart, flash, sensor, adc, pwm, can, i2c, i3c, display, pinctrl, regulator and counter. Assisted-by: Claude:claude-opus-4-8
A group nested inside a versioned group is part of that API and carries the same promise, but a group carrying no @Version of its own was read as unversioned regardless of where it sat. thread_apis, timer_apis and clock_apis all sit inside kernel_apis and inherit its 1.0.0, yet each was treated as untagged and graded by the --unversioned-is policy instead of by what the tree already states. Resolve a group's version by walking up to the nearest versioned ancestor. @InGroup names the parent and wins over lexical nesting inside another group's @{ ... @} block: it is the explicit statement of where the group belongs, and in the two places in the tree where the two disagree it names the more specific parent. Resolution needs the whole tree rather than one file, since @InGroup routinely points at a parent declared in another header, so a GroupIndex collects groups across headers and walks the chain with a visited set, because nothing stops an author writing a cycle of @InGroup tags. The signature comparison takes the same relation from Doxygen's own <innergroup> elements rather than re-deriving nesting from headers. This resolves 438 of the 967 untagged groups, leaving 529 genuinely unversioned: 352 groups are now known to be stable rather than 66. A finding still reports one of the four state names, so the HTML report's filter facets stay bounded, and names the group it inherited from in the detail instead. Three consumers follow. group-metadata no longer demands @SInCE and @Version on a group that inherits them. propose skips groups already covered by an enclosing group, which drops the drivers backlog from 47 proposals to 22. audit reports declared and effective states side by side. Tested with pytest scripts/tests/ci/test_api_compat.py (109 tests) covering @InGroup and lexical inheritance, cross-header parents, multi-hop chains, own-version precedence and cycle safety, and verified against the Doxygen XML of a real tree, where clock_apis now grades stable through kernel_apis and says so. Assisted-by: Claude:claude-opus-4-8
Add a workflow that reports changes to public API headers which break the guarantees implied by the API's documented lifecycle state, as annotations on the pull request. It runs the two halves of scripts/ci/check_api_compat.py. The lifecycle metadata checks read only the headers and the diff and take seconds, so they gate directly. The signature comparison needs a Doxygen tree per revision, which dominates the runtime; one pair of trees is built and then reused for the annotations, the job summary and an HTML artifact rather than being built three times. No west workspace is set up, because every INPUT in doc/zephyr.doxyfile.in is relative to the Zephyr repository, so a bare checkout is enough to build both snapshots. The job passes --unversioned-is unstable, which is weaker than the tool's fail-closed default. Much of the tree is still untagged and grading all of it as stable would fail pull requests over APIs that never promised anything; with the flag, only an API that explicitly declares itself stable can fail the job. Two fixes were needed to make the annotations land. Doxygen strips the include roots listed in STRIP_FROM_PATH from the locations it records, so include/zephyr/drivers/gpio.h was reported as zephyr/drivers/gpio.h. GitHub anchors an annotation to a line only when the path it is given exists in the repository, so that prefix is now put back by probing the known stripped roots, with a fallback for headers the change itself deleted, which exist in no checkout to probe. Declaration paths are also resolved against the repository rather than the snapshot worktrees, which compare-revs removes before comparing. A changed symbol is now located in the head revision instead of the base, so the annotation lands on the line the change actually touched. A removed symbol has no such line and keeps its base location. Tested with pytest scripts/tests/ci/test_api_compat.py (115 tests) and by running the workflow's commands against a branch carrying two deliberate breaks in the stable gpio_interface API: a removed function and a parameter retyped from gpio_pin_t to uint8_t. Both were reported at include/zephyr/drivers/gpio.h with resolvable paths, the retyped parameter anchored on its declaration and flagged as a silent behavior change, and the gating step exited non-zero. Assisted-by: Claude:claude-opus-4-8
Three problems, the first two of which failed the first run of the workflow on zephyr-testing. "git fetch --depth=0" is not valid: git rejects it with "depth 0 is not a positive number". The step was unnecessary anyway, because the cached clone plus a checkout with fetch-depth 0 already leaves the base ref in place, so it is now only fetched when it is genuinely missing. doxmlparser, which the signature comparison reads the Doxygen XML with, comes from doc/requirements.txt rather than scripts/requirements-actions.txt. Only the latter was installed, so every step that touched the comparison failed on the import. The docs workflow installs both; so does this one now. The third problem did not fail the run but would have produced wrong findings. Both checks compared against the tip of the target branch rather than the merge base. The two differ whenever the target has moved on since the pull request branched, and every API change made by anyone else in the meantime would have been reported as though this pull request had made it. A branch that changes nothing at all compares clean against its merge base and dirty against a tip that has advanced. The merge base is now resolved once and used by both checks. Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
A test run on zephyr-testing reported 159 breaking changes for a pull request that modified one header. 156 of them were members of the led_fake group, which the change never went near. include/zephyr/drivers/led/led_fake.h declares its API through seven Fake Function Framework macros, and those generate the members that were reported: call_count, arg0_val, arg0_history and the rest. Doxygen does not expand macro-generated declarations reproducibly, so the members expanded in one snapshot and not the other, and every one of them looked removed. The same comparison run locally produced nothing, which is what non-determinism looks like. Limit findings to the files the change actually touches. A signature change originates in a changed file; anything else is snapshot noise. This costs a blind spot, since a change to a macro in one header can legitimately alter what another header expands to, but that is rarer than the noise it removes. Pass --all-files to opt out. Also fix the annotation format. GitHub reads "::" and "," as annotation syntax, and struct members are displayed as parent::member, so titles like "member 'led_fake::arg0_history' was removed" truncated the annotation partway through and left the rest as message text. That is visible in the run: the titles are cut off mid-word. Properties are now escaped as GitHub specifies, and the message escapes percent signs as well as newlines. Tested with pytest scripts/tests/ci/test_api_compat.py (122 tests) and against the pull request that produced the noise, which now reports nothing rather than 159 phantom removals. Assisted-by: Claude:claude-opus-4-8
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| unversioned_is=Lifecycle(unversioned), | ||
| base_root=base_root, | ||
| head_root=head_root, | ||
| ) |
| # comparison reads the Doxygen XML with. | ||
| - name: Install Python packages | ||
| run: | | ||
| pip install -r scripts/requirements-actions.txt --require-hashes |
| - name: Install Python packages | ||
| run: | | ||
| pip install -r scripts/requirements-actions.txt --require-hashes | ||
| pip install -r doc/requirements.txt --require-hashes |
nashif
force-pushed
the
main
branch
3 times, most recently
from
August 9, 2026 01:23
0f2df86 to
4b8671a
Compare
nashif
force-pushed
the
main
branch
2 times, most recently
from
August 15, 2026 14:20
af6d02f to
cddc603
Compare
nashif
force-pushed
the
main
branch
10 times, most recently
from
August 31, 2026 15:41
4e4fac7 to
d9b0d41
Compare
nashif
force-pushed
the
main
branch
2 times, most recently
from
September 14, 2026 13:36
4a3006c to
8bedb3c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.