lib/logstorage: fix collapse_nums treating hex-letter word prefixes as numbers

[niladrix719]

Summary

Don't treat the leading hex chars of a word as a number in collapse_nums.
Previously words like default collapsed into <N>ult, beefsteak into
<N>steak and feed_url into <N>_url.

* | generate_sequence 1
      | format "default beefsteak feed_url is_default deadbeef1234 10us" as _msg
      | collapse_nums | fields _msg

Before: <N>ult <N>steak <N>_url is_<N>ult <N> <N>us
After: default beefsteak feed_url is_default <N> <N>us

Notes

But doing this one thing to focus on is a hex num immediately followed by _ T Z s m h u n is no longer collapsed i.e deadbeef1234s stays as is instead of becoming <N>s, and dead_beef gives dead_<N> instead of <N>_<N>.

However Hex nums followed by anything else are unaffected: deadbeef1234 and deadbeef1234 done still collapse.

fixes: #1760

[cubic-dev-ai]

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/victorialogs/CHANGELOG.md">

<violation number="1" location="docs/victorialogs/CHANGELOG.md:43">
P2: Custom agent: **Changelog Review Agent**

This entry describes a narrow edge case limited to users of `collapse_nums` or pattern-match filters whose values begin with hexadecimal-looking prefixes, so it does not meet the changelog rule's majority-of-users, high-signal threshold. Remove this specialized bugfix from the changelog or combine it with a broader user-facing LogsQL fix.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Custom agent: Changelog Review Agent

This entry describes a narrow edge case limited to users of collapse_nums or pattern-match filters whose values begin with hexadecimal-looking prefixes, so it does not meet the changelog rule's majority-of-users, high-signal threshold. Remove this specialized bugfix from the changelog or combine it with a broader user-facing LogsQL fix.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At docs/victorialogs/CHANGELOG.md, line 43:

<comment>This entry describes a narrow edge case limited to users of `collapse_nums` or pattern-match filters whose values begin with hexadecimal-looking prefixes, so it does not meet the changelog rule's majority-of-users, high-signal threshold. Remove this specialized bugfix from the changelog or combine it with a broader user-facing LogsQL fix.</comment>

<file context>
@@ -40,6 +40,7 @@ according to the following docs:
 * BUGFIX: [cluster version](https://docs.victoriametrics.com/victorialogs/cluster/): properly cancel queries already running on `vlstorage` when the corresponding query is canceled or times out on `vlselect`. Previously, `vlstorage` could fail to detect the disconnected `vlselect`, causing these queries to continue running and waste CPU and disk IO. See [#1672](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1672).
 * BUGFIX: [data ingestion](https://docs.victoriametrics.com/victorialogs/data-ingestion/) and [querying](https://docs.victoriametrics.com/victorialogs/querying/): properly handle logs containing duplicate [stream field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) names. Previously, [v1.52.0](https://github.com/VictoriaMetrics/VictoriaLogs/releases/tag/v1.52.0) could panic when ingesting such logs in single-node VictoriaLogs, drop them during ingestion in VictoriaLogs cluster, or panic when querying such data written by earlier releases. See [#1603](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1603) and [#1604](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1604).
 * BUGFIX: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): fix [`week_range[Sun,Sun]` filter](https://docs.victoriametrics.com/victorialogs/logsql/#week-range-filter) when it is used inside the [`filter` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#filter-pipe). Previously, it could fail to match rows on Sunday. See [#1335](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1335).
+* BUGFIX: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): fix [`collapse_nums` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#collapse_nums-pipe) and [pattern match filters](https://docs.victoriametrics.com/victorialogs/logsql/#pattern-match-filter) incorrectly detecting hexadecimal numbers at the beginning of words. Previously, words such as `default` could be collapsed into `<N>ult`. See [#1760](https://github.com/VictoriaMetrics/VictoriaLogs/issues/1760).
 * BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): prevent long group-by values from overflowing group headers. See [#1663](https://github.com/VictoriaMetrics/VictoriaLogs/pull/1663).
 * BUGFIX: [web UI](https://docs.victoriametrics.com/victorialogs/querying/#web-ui): render only inline Markdown links with explicit `http` or `https` destinations, such as `[text](https://example.com)`, as clickable links in log messages. Bare URLs, autolinks, reference-style links, links using other schemes, and images are rendered as plain text. See [#1470](https://github.com/VictoriaMetrics/VictoriaLogs/pull/1470).
</file context>

[Vandit1604]

@niladrix719 I had written the same change locally before you pushed, so this is the fix I would have sent too. I ran your version here and go test ./lib/logstorage/ passes, 890 tests.

One gap, on the line comment.

[Vandit1604]

isValidNum has a second caller: indexPlaceholderNumEnd in pattern_matcher.go, which backs the pattern_match filter. Your changelog mentions the filter, but no test covers it. pattern_match("<N>ult") matches default on master and stops matching with this patch. That is the behaviour you want, and nothing in TestPatternMatcherMatch pins it.

I also tried letting _ through here, to keep dead_beef collapsing to <N>_<N>. Every pre-existing case still passes, but dead_letter then becomes <N>_letter, which is the reported bug again. The strict rule is right.

[niladrix719]

Thanks, Added the PatternMatcher tests

[niladrix719]

I also tried letting _ through here, to keep dead_beef collapsing to _. Every pre-existing case still passes, but dead_letter then becomes _letter, which is the reported bug again. The strict rule is right.

I did run into the exact same problem, which seems to be the trade-off, I mentioned it in the notes