Describe the bug
cudf::mixed_left_semi_join can return an incorrect gather map when the equality tables are non-nullable but the conditional tables contain nulls. The result is sensitive to the insertion order of otherwise identical logical rows.
This was isolated while investigating NVIDIA/cudf-spark#14076, where NDS query 16 intermittently produced incorrect aggregate results.
The mixed-semi hash set builds two right-side comparators. The comparator over right_conditional currently receives the null-state derived from the equality table (right). When equality keys have no nulls, nulls in the conditional values can therefore be ignored while selecting hash-set representatives.
Steps/Code to reproduce bug
Use identical 30-row left and right inputs composed of 10 equality-key groups:
equality key per group: [k, k, k]
conditional, null-first: [null, 0, 1]
conditional, null-last: [0, 1, null]
predicate: left.conditional != right.conditional
null equality: UNEQUAL
Call cudf::mixed_left_semi_join, using the equality-key column for the equality tables and both columns for the conditional tables. Repeat each ordering 1,000 times and compare the sorted gather map with the non-null row indices.
Observed with the original implementation:
case=nonnull_control iterations=1000 mismatches=0
case=nullable_null_last iterations=1000 mismatches=0
case=nullable_null_first actual_size=10 expected_size=20
case=nullable_null_first iterations=1000 mismatches=1000
The same signature was reproduced on both x86_64/Ada and aarch64/GH200.
Expected behavior
Both nullable row orders should return the same 20 non-null left-row indices on every iteration. Conditional-table null handling must not depend on whether the equality table contains nulls.
Environment overview
- Environment location: bare metal
- Method of cuDF install: built from source locally; exact packaged library from the cudf-spark build on GH200
Environment details
Local reproduction:
cuDF revision (unpatched binary used for the output above): f37d0dfd860ff464ccfe747b4ba5998b9f223558
architecture: x86_64
GPU: NVIDIA RTX 5880 Ada Generation
driver: 580.95.05
gcc/g++: 13.3.0
CUDA compiler: 13.0
CMake: 4.2.3
GH200 reproduction:
cuDF revision: a6bd95e24bebc35c5ea92283410df4f48bf3be89
architecture: aarch64
GPU: NVIDIA GH200 480GB
Additional context
The same buggy comparator construction was confirmed by source inspection at PR base f588e0c838b437271e1a1016236822f4a2ef3826; the retained unpatched local binary used for the output above was f37d0dfd860ff464ccfe747b4ba5998b9f223558.
Using a null-state derived from right_conditional for its comparator eliminates the failure in 1,000/1,000 null-first iterations on both devices. A regression test should cover non-null equality keys with nullable conditional values in both null-first and null-last order.
Describe the bug
cudf::mixed_left_semi_joincan return an incorrect gather map when the equality tables are non-nullable but the conditional tables contain nulls. The result is sensitive to the insertion order of otherwise identical logical rows.This was isolated while investigating NVIDIA/cudf-spark#14076, where NDS query 16 intermittently produced incorrect aggregate results.
The mixed-semi hash set builds two right-side comparators. The comparator over
right_conditionalcurrently receives the null-state derived from the equality table (right). When equality keys have no nulls, nulls in the conditional values can therefore be ignored while selecting hash-set representatives.Steps/Code to reproduce bug
Use identical 30-row left and right inputs composed of 10 equality-key groups:
Call
cudf::mixed_left_semi_join, using the equality-key column for the equality tables and both columns for the conditional tables. Repeat each ordering 1,000 times and compare the sorted gather map with the non-null row indices.Observed with the original implementation:
The same signature was reproduced on both x86_64/Ada and aarch64/GH200.
Expected behavior
Both nullable row orders should return the same 20 non-null left-row indices on every iteration. Conditional-table null handling must not depend on whether the equality table contains nulls.
Environment overview
Environment details
Local reproduction:
GH200 reproduction:
Additional context
The same buggy comparator construction was confirmed by source inspection at PR base
f588e0c838b437271e1a1016236822f4a2ef3826; the retained unpatched local binary used for the output above wasf37d0dfd860ff464ccfe747b4ba5998b9f223558.Using a null-state derived from
right_conditionalfor its comparator eliminates the failure in 1,000/1,000 null-first iterations on both devices. A regression test should cover non-null equality keys with nullable conditional values in both null-first and null-last order.