Conversation
Add HyperLogLog-based approximate unique count for high-cardinality UV queries without O(cardinality) memory. Uses axiomhq/hyperloglog p=14 with a versioned wire format for cluster merge. Relates to VictoriaMetrics#1414.
Cover typed column hash consistency, remote/local/proxy merge paths, merge algebra, wire golden round-trips, fuzzed unmarshal, accuracy distribution, state budget/heap checks, and basic benchmarks.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 29 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Merge on importState so repeated remote states for the same group are unioned instead of replaced. Normalize -0 to unsigned zero, strip leading zeros before parsing, and apply the same numeric canonical encoding on multi-field tuples.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Validate Axiom payload precision against p=14 and return an error instead of Panicf when merging untrusted imported wire state.
Resolve conflicts in go.mod/go.sum, vendor/modules.txt, and CHANGELOG while keeping count_uniq_hll and its HyperLogLog dependencies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
count_uniq_hll(field1, ..., fieldN)for fixed-memory HyperLogLog UV estimation (p=14, axiomhq/hyperloglog v0.3.0), addressing high-cardinality OOM withcount_uniq/count_uniq_hash.vlselectandvlstoragemust both be upgraded to use this function.1/01share the same hash domain.Closes #1414
Test plan
go test -mod=vendor ./lib/logstorage/ -run 'CountUniqHLL|TestHLL'FuzzHLLUnmarshalState(short)BenchmarkHLL*vlstoragenodes; result must not equal sum of per-node estimates