Repository navigation
Make IQ Scanner check #6 (data source count) configurable with safe-composition warning band - #225
Open
wscalioni wants to merge 1 commit into
Conversation
…igurable with safe-composition warning band Resolves databricks-solutions#212. Sub-task of Epic databricks-solutions#199. Replaces the hard binary "1-12" gate with three env-var-driven thresholds: - GSO_MAX_SAFE_DATA_SOURCES (default 12) — the soft cap - GSO_MAX_WIDE_TABLE_COLUMNS (default 60) — what counts as a wide raw table - GSO_MIN_METRIC_VIEW_RATIO (default 0.5) — qualifier for the warning band Defaults reproduce the original behaviour exactly. Above the cap, a space qualifies for severity="warning" (passed=True) when its composition is safe (metric-view ratio at-or-above min, zero wide raw tables); otherwise it stays at severity="fail" (passed=False) as before. get_maturity_label() gates on `passed`, not `severity`, so the warning band correctly unblocks Ready-to-Optimize / Trusted tiers without weakening the ranking. The advisory text lands in `warnings` / `warning_next_steps` (not `findings`), so Quick Fix stays quiet for safe over-cap spaces. Same warning-band pattern is already used by check databricks-solutions#9 (entity matching). The check label is renamed from "Data source count 1-12" to "Data source count" since the range is now configurable; historical scans persist the original label string so the history view is unaffected. Tested: - 7 new unit-test cases in TestDataSourceCountSoftCap (warning-band pass, wide-table fail, low-ratio fail, env-var raise/lower of all three knobs, maturity-unblock proof via get_maturity_label directly) - All 87 backend/tests/test_scanner.py tests pass - All 6 backend/tests/test_scanner_parity.py tests pass - 437 passes / 0 failures across all scan/scor/check_6/data_source tests Filed by Will Scalioni (Databricks SA) using personal github wscalioni because EMU accounts cannot write to external public repos. Co-authored-by: Isaac
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #212. Sub-task of Epic #199.
Replaces the hard binary "1-12" data-source gate in IQ Scanner check #6 with three env-var-driven thresholds:
GSO_MAX_SAFE_DATA_SOURCES(default 12) — the soft capGSO_MAX_WIDE_TABLE_COLUMNS(default 60) — what counts as a wide raw tableGSO_MIN_METRIC_VIEW_RATIO(default 0.5) — qualifier for the warning bandDefaults reproduce the original behaviour exactly. Above the cap, a space qualifies for
severity="warning"(passed=True) when its composition is safe (metric-view ratio at-or-above min, zero wide raw tables); otherwise it stays atseverity="fail"(passed=False) as before.get_maturity_label()gates onpassed, notseverity, so the warning band correctly unblocks Ready-to-Optimize / Trusted tiers without weakening the ranking. The advisory text lands inwarnings/warning_next_steps(notfindings), so Quick Fix stays quiet for safe over-cap spaces.Same warning-band pattern is already used by check #9 (entity matching).
The check label is renamed from "Data source count 1-12" to "Data source count" since the range is now configurable; historical scans persist the original label string so the history view is unaffected.
Testing
TestDataSourceCountSoftCap(warning-band pass, wide-table fail, low-ratio fail, env-var raise/lower of all three knobs, maturity-unblock proof viaget_maturity_labeldirectly)backend/tests/test_scanner.pytests passbackend/tests/test_scanner_parity.pytests pass