Add has_no_sequence_gaps dataset-level check - #1456
Open
ghanse wants to merge 4 commits into
Open
Conversation
Adds a dataset-level check that flags gaps in a numeric sequence, i.e. expected values that are missing between values that are present (for example no invoice numbered 1002 while 1001 and 1003 are present). This is the numeric counterpart of has_no_gaps_per_time_window: distinct values are bucketed onto a fixed grid of `step` aligned to zero, a window lead() finds the next present bucket, and a gap is flagged wherever the next present bucket starts more than one step after the current one. Since a missing value has no row to attach a violation to, the gap is reported on every row in the last present bucket before the gap. With the default `step` of 1 the grid is the integers, so detection is exact sequence-gap detection - the common case for invoice numbers, ledger entries and sequence IDs. Only interior gaps are detected. The bounds of a sequence are its own lowest and highest present values, so missing values beyond either end are not reported because there is no row to anchor them to. When `group_by` is provided, gaps are detected independently within each group and each group is bounded by its own lowest and highest present value, so a group whose sequence simply stops earlier than another group's is not flagged. Null values are ignored and pass with no violation.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1456 +/- ##
===========================================
- Coverage 92.51% 78.51% -14.01%
===========================================
Files 141 141
Lines 13577 13601 +24
Branches 151 151
===========================================
- Hits 12561 10679 -1882
- Misses 946 2855 +1909
+ Partials 70 67 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
❌ 966/977 passed, 1 flaky, 11 failed, 49 skipped, 8h14m8s total ❌ TestDataContractIntegration.test_generate_rules_with_text_processing: litellm.exceptions.RateLimitError: litellm.RateLimitError: DatabricksException - {"error_code":"REQUEST_LIMIT_EXCEEDED","message":"REQUEST_LIMIT_EXCEEDED: Exceeded workspace input tokens per minute rate limit for databricks-claude-sonnet-4-5. Work with your Databricks account team to request a higher FMAPI rate limit tier."} (28.881s)❌ test_has_no_sequence_gaps: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 40.00000 % ) (5.775s)❌ test_has_no_sequence_gaps_multiple_gaps: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 66.66667 % ) (7.247s)❌ test_has_no_sequence_gaps_custom_step: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 50.00000 % ) (11.93s)❌ test_has_no_sequence_gaps_negative_values: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 33.33333 % ) (6.402s)❌ test_has_no_sequence_gaps_group_by: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 20.00000 % ) (6.44s)❌ test_has_no_sequence_gaps_group_by_bounded_independently: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 20.00000 % ) (7.275s)❌ test_has_no_sequence_gaps_group_by_null_key: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 50.00000 % ) (8.82s)❌ test_has_no_sequence_gaps_group_by_mixed_null_and_named_keys: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 33.33333 % ) (8.932s)❌ test_has_no_sequence_gaps_group_by_column_expression: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 25.00000 % ) (9.304s)❌ test_apply_checks_and_split_has_no_sequence_gaps: pyspark.errors.exceptions.base.PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 33.33333 % ) (18.625s)Flaky tests:
Running from acceptance #5656 |
Contributor
|
✅ 1/1 passed, 30m28s total Running from mcp #405 |
Contributor
|
✅ 138/138 passed, 1 skipped, 8h58m42s total Running from anomaly #1770 |
…as-no-sequence-gaps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a new dataset-level check
has_no_sequence_gaps(column, step=1, group_by=None)that detects gaps in a numeric sequence.This follows the approach established in #1370 for
has_no_gaps_per_time_window. Distinct values are bucketed onto a fixed grid ofstep, a windowedlead()finds the next bucket, and a gap is flagged whenever the next bucket starts more than one step after the current bucket. Gaps are reported for every row in the last present bucket before the gap.The bounds of a sequence are its own lowest and highest present values. When
group_byis provided, gaps are detected independently within each group (e.g. to detect missing primary keys by source system) and each group is bounded by its own lowest and highest value.Linked issues
Tests
Documentation and Demos
This pull request and its description were co-written by Isaac.