fix: strip email domains from FTS search input - #140
Open
c1-dev-bot[bot] wants to merge 1 commit into
Open
Conversation
![c1-dev-bot[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
When users search by full email address (e.g. user@domain.com), the jargon tokenizer splits at '@' producing domain tokens that never exist in the FTS index (which only indexes display names). This causes 0 results for email searches. Add stripEmailDomains() preprocessing to extract only the local-part before '@', so "user@domain.com" becomes "user" before tokenization. Fixes: IGA-1508
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dluu@zscaler.com),FullTextSearchQueryproduces tokens likedluuandzscalercom(domain with dot stripped). Since the FTS index only contains display name tokens, the domain token never matches, causing 0 results.stripEmailDomains()preprocessing that detects email-format tokens in the search input and extracts only the local-part before@, sodluu@zscaler.combecomesdluubefore tokenization.stripEmailDomainsand an integration test (TestSearchEmailAddress) verifying email search matches against display name vectors.Fixes: IGA-1508
Test plan
TestStripEmailDomains— verifies email domain stripping for various inputs (single email, multiple emails, non-email text, edge cases like@orphan, empty strings)TestSearchEmailAddress— integration test against Postgres confirming thatdluu@zscaler.commatches a vector containingdluuwhilenotauser@example.comdoes notAutomated PR Notice
This PR was automatically created by c1-dev-bot as a potential implementation.
This code requires: