Fix query cache heap leak from wildcard field queries

[Adam-Horse]

Description

Fixes a heap exhaustion caused by WildcardMatchingQuery (the two-phase query behind the wildcard field type) being admitted to the LRU query cache while holding shard-scoped state.

When built from a QueryShardContext, the query captures the shard's SearchLookup and a ValueFetcher supplier for second-phase verification, but its weight unconditionally reported isCacheable() == true. Cache entries outlive the search, so each cached wildcard query pinned its captured object graph on the heap. Because equals/hashCode ignore the captured lookup, every unique pattern pinned a distinct graph, and the cache's RAM accounting (via Query#visit, which only sees the first-phase terms) missed the pinned state entirely, so memory-based eviction never triggered. Under steady wildcard query traffic this accumulates until the node runs out of heap.

Changes:

• The weight now reports isCacheable() == false whenever a SearchLookup was captured, keeping the stateful two-phase query out of the cache.
• The first-phase trigram query is created through searcher.createWeight(firstPhaseQuery, ScoreMode.COMPLETE_NO_SCORES, 1f) instead of firstPhaseQuery.createWeight(...), so the cheap, stateless part of the query remains independently cacheable (its scores were never used; the outer weight is constant-score).

Tests:

• testSearchLookupNotPinnedByQueryCache — regression test for the reported leak: runs a context-bound query through an LRUQueryCache, retains only a WeakReference to the captured SearchLookup, releases the query, and asserts the reference clears under GC. On main it fails with SearchLookup is still strongly reachable: the query cache is pinning shard state; with this fix it passes.
• testCacheabilityDependsOnSearchLookup — the weight is cacheable without a QueryShardContext and not cacheable with one.
• testFirstPhaseQueryCachedIndependently — the first-phase query still enters the cache and serves hits, while the outer WildcardMatchingQuery never does.

Verified on a live node built from source (1 GB heap, 20k-doc single-segment wildcard index, 2000 unique patterns x 6 repetitions in filter context with request_cache=false, then forced GC + jcmd GC.class_histogram):

After GC	main	with fix
Retained SearchLookup / QueryShardContext	4000 / 2000	0 / 0
Heap used	134 MB	47 MB (baseline)
Query cache entries / hits	2000 / 0	2000 / 4000

On main the cache self-reported ~2.4 MB while actually pinning ~88 MB, even with a near-empty mapping — consistent with the tens-of-GB retention reported in the issue at production scale.

A follow-up restructuring WildcardMatchingQuery so the second phase does not capture shard-scoped state at all (making the full query safely cacheable again) is discussed in the linked issue.

Related Issues

Resolves #22419

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 3f4507f)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 3f4507f
Explore these optional code suggestions:

Category	Suggestion	Impact
General	Verify score mode passed to first-phase weight Passing ScoreMode.COMPLETE_NO_SCORES unconditionally may prevent cache use when the caller requested a scoring mode that requires it, but more importantly, when the outer scoreMode does not need scores, using a fixed mode is fine; however, if the outer scoreMode.needsScores() is false the first phase should also skip scores. Consider deriving from the passed-in scoreMode (e.g., scoreMode.needsScores() ? ScoreMode.COMPLETE_NO_SCORES : scoreMode) or explicitly documenting why the hard-coded value is safe, since first-phase scores are discarded anyway — the current choice is correct but should ensure the first-phase weight's caching key is stable across invocations. server/src/main/java/org/opensearch/index/mapper/WildcardFieldMapper.java [844] +Weight firstPhaseWeight = searcher.createWeight(firstPhaseQuery, ScoreMode.COMPLETE_NO_SCORES, 1f); - Suggestion importance[1-10]: 2 __ Why: The suggestion is vague and its improved_code is identical to existing_code. Since the outer query is a ConstantScoreWeight, first-phase scores are always discarded, making COMPLETE_NO_SCORES a correct and intentional choice.	Low

---

Previous suggestions

Suggestions up to commit f845311

Category	Suggestion	Impact
General	Ensure IndexWriter is closed on exceptions The IndexWriter is not enclosed in a try-with-resources or try/finally, so if addDocument or DirectoryReader.open throws, the writer will leak and the directory will not be closable. Use try-with-resources for the writer or close it before opening the reader outside a try block. This issue is repeated in all three new tests. server/src/test/java/org/opensearch/index/mapper/WildcardFieldTypeTests.java [275-278] -IndexWriter iw = new IndexWriter(dir, new IndexWriterConfig(Lucene.KEYWORD_ANALYZER)); -iw.addDocument(appleTrigramDocument()); -try (IndexReader reader = DirectoryReader.open(iw)) { - iw.close(); +try (IndexWriter iw = new IndexWriter(dir, new IndexWriterConfig(Lucene.KEYWORD_ANALYZER))) { + iw.addDocument(appleTrigramDocument()); +} +try (IndexReader reader = DirectoryReader.open(dir)) { Suggestion importance[1-10]: 4 __ Why: Valid point about resource leak on exceptions in test code, but the impact is minor since these are tests and the leak only occurs on failure paths. The improved code is a reasonable refactor.	Low

[github-actions]

Persistent review updated to latest commit 3f4507f

[github-actions]

✅ Gradle check result for 3f4507f: SUCCESS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.47%. Comparing base (27e5d58) to head (3f4507f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #22432      +/-   ##
============================================
+ Coverage     73.39%   73.47%   +0.07%     
- Complexity    76397    76448      +51     
============================================
  Files          6105     6105              
  Lines        346613   346613              
  Branches      49888    49889       +1     
============================================
+ Hits         254403   254680     +277     
+ Misses        71958    71608     -350     
- Partials      20252    20325      +73     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.