Repository navigation
perf: faster minimatch() and filter() pattern cache - #339
Conversation
| p: string, | ||
| pattern: string, | ||
| options: MinimatchOptions = {}, | ||
| options?: MinimatchOptions, |
There was a problem hiding this comment.
How much of the perf improvement comes from making this optional? This is a breaking change, Glob uses the .options in a few places.
There was a problem hiding this comment.
Free to double check, but I sanity checked the output bundle to verify my assumption and - as far as I remember - the emitted .d.ts files are byte-identical between this proposal and the latest released version, as options = {} already compiled to options?: MinimatchOptions. So no breaking change in my book 😄.
There was a problem hiding this comment.
Oh, ok, but the object always has an options field then?
Sorry, I'm traveling this week, only looking at code on my phone, and so you're not yet getting the frontier model /effort xhigh isaacs ;)
There was a problem hiding this comment.
Yep, it does. Only the minimatch() function's parameter became optional. The Minimatch constructor still has options = {}, and the cache builds instances with { ...options }, which gives {} for undefined. So mm.options is always an object, and filter()/defaults() didn't change. Checked on the built output: minimatch(p, pat), minimatch(p, pat, undefined) and filter(pat)(p) all give mm.options as {}.
No worries, safe travels! 😄 As a matter of fact, thank you 2x for checking this while traveling! 🙇
| patterns. | ||
| - `minimatch()` keeps the 256 `Minimatch` objects that it used most | ||
| recently, so it does not compile the same pattern again. | ||
| - `minimatch()` keeps at least the 256 `Minimatch` objects that it |
There was a problem hiding this comment.
So the cache size is no longer bounded? That could be a problem.
| if (cacheSize >= cacheMax) { | ||
| oldCache = cache | ||
| cache = new Map() | ||
| cacheSize = 0 |
There was a problem hiding this comment.
Oh, i see, it's not that the cache is no longer bounded, it's just not an LRU anymore. When it hits the max, you're throwing the whole thing away and starting over with no cache.
That might be a good approach, but I'm not sure. It's definitely a significant behavior change. I'll need to benchmark this against some representative samples, to ensure it's not just optimized for the benchmark.
There was a problem hiding this comment.
Oh, no, not starting over with no cache. Items do get pulled forward into the new cache. Still, I need to comb through this carefully to ensure we understand the behavior change.
There was a problem hiding this comment.
Ok, yeah, I was mistaken. This is still just a normal LRU, it's just doing the two-stage trick that works for small N / small memory, where frequent disposal is more expensive than the cost of duplicating the data. Good change! (I'd explored such a trick in the lru-cache module, but it is not good if you want to have strict memory-usage contracts, and not ideal for larger data sets, which that module supports. But here it's fine.)
There was a problem hiding this comment.
Yes, basically for a fixed N of small objects the extra generation costs less than a delete + set for every hit.
Also, remove basically all cases where `JSON.stringify` would get used.
optionsKey() now loops over the keys of the options object, in place
of reading each known option by name. Its speed no longer depends on
how many object shapes it has seen. Before, after 6 other shapes,
{ dot: true } took about 31 ms on the realistic-paths benchmark. Now
it takes about 19 ms (HEAD before the key work: about 27 ms).
The loop reads inherited keys, and the cached object now gets a copy
that keeps them too. So minimatch() and filter() use inherited
options, the same as new Minimatch().
This drops the benchmark timing down considerably by improving cache key
calculation and also making more things share a cache entry that have
the same behavior.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1c0330b to
4633525
Compare
Summary
Makes
minimatch(path, pattern)2.3x faster andminimatch.filter()2.8x faster on everyday globs. Matching itself is unchanged. The time saved was all spent finding the cachedMinimatchobject.JSON.stringifyand no new string to hash. The cache now has two generations, as quick-lru does: a hit is twoMap.gets, with nodelete+set.filter(): makes itsMinimatchonce, the first time it tests a path. An invalid pattern still throws at that first call.Closes #338
Results
15,505 paths (
benchmark/realistic-paths.json) × 10 globs. Each figure is the median of 5 alternating process runs, each of those the median of 15 rounds. Range in brackets.ea0a4a6)minimatch(f, p)minimatch(f, p, { dot: true })minimatch.filter(p)minimatch.defaults({ dot: true })(f, p)Minimatch#match(not changed)minimatch(f, p)is now within 12% ofMinimatch#match, andfilter()within 21%.The match counts are identical on both builds: 35,020, and 35,213 with
dot.benchmark/realistic.mjsgives the same totals (33.52 vs 33.53 ms).Behavior changes
filter()uses the options as they are at its first test. A later change to the options object no longer applies to that filter.minimatch()still applies such changes (test/cache.ts).Tests
npm test: 7303/7303 pass, with full coverage.oxlintandprettier --checkare clean.test/cache.ts: a pattern used again from the old generation, and thefilter()options.Repro
fnbench.mjs