Interop Investigation 2026 Scoring Criteria

[danae404]

We should define "success" for the 2026 investigation.

[danae404]

My initial thoughts: We should go through each rendering test (250 tests, probably dividing work by subdirectory), convert each test to updated format with shared css file, and reach agreement on the reference file during meetings. A bulk of our score can be a percentage of the 250 tests that we complete. Perhaps bonus points for adding new test coverage?

[danae404]

For reference here is last year's scoring criteria: #2

[danae404]

Our first priority is rendering tests, then 2nd is api and parsing tests. 75% of score based on the percentage of renderings tests that we go through, and 25% of score based on percentage of parsing and api tests that we go through.

[nigelmegitt]

Is the assumption that a parsing fault will be exposed as a rendering test fail, where the input is valid and the parsing error relates to rendering semantics, i.e. the priority is tests that stress the whole path from parsing to rendering, and more specific parsing tests that reveal where errors are occurring in the first part of that path are less important initially?

[gkatsev]

The rendering tests are the biggest group of tests with failures and the biggest interop difference. Additionally, I think that they're the most subjective compared to the parsing and API tests. While we didn't think of your point, I think it's an additional benefit to focusing on these.