Skip to content

fix: report generalisation gap instead of inverted accuracy gap#472

Draft
shamykyzer wants to merge 1 commit into
mainfrom
468-generalisation-gap
Draft

fix: report generalisation gap instead of inverted accuracy gap#472
shamykyzer wants to merge 1 commit into
mainfrom
468-generalisation-gap

Conversation

@shamykyzer

@shamykyzer shamykyzer commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

As reported in #468, StructuralAttack's generalisation metric was inverted and mislabelled: it returned the train/test accuracy gap with the sign flipped, so an overfit model showed a negative value.

This changes it to report the generalisation gap = test error - train error (for classification, (1 - test_acc) - (1 - train_acc)), which is positive when the model overfits. Classification only; regression is left to #414 as discussed on the issue.

  • Flip the sign in the SklearnModel and PytorchModel generalisation methods.
  • Clarify the docstrings, and fix the swapped train_acc/test_acc descriptions in StructuralAttackResults.
  • Update the affected test assertions; add a CHANGELOG entry.

Closes #468

@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.66%. Comparing base (b7885f1) to head (9c05bfe).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #472   +/-   ##
=======================================
  Coverage   99.66%   99.66%           
=======================================
  Files          28       28           
  Lines        3633     3633           
=======================================
  Hits         3621     3621           
  Misses         12       12           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shamykyzer shamykyzer self-assigned this Jul 10, 2026
@shamykyzer
shamykyzer requested a review from rpreen July 10, 2026 14:24
@rpreen

rpreen commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Thanks for this. Please can you remove any references to issue numbers to keep the source clean? Also the variable/function names (pretty much anywhere that deals with "generalisation error") like generalisation_error need to be renamed to generalisation gap for clarity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Behaviour Change] Generalisation error metric is mislabelled

2 participants