Improve package scan performance #4606

AyanSinhaMahapatra · 2025-11-17T09:49:46Z

This PR improve package scan performance by....

References:

fast-scan: Apply performance improvements to code: batch 2 (file system walking or from benchmark results) #4064
fast-scan: Create scan benchmark for package detection in ScanCode Toolkit #4061

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁
Updated documentation pages (if applicable)
Updated CHANGELOG.rst (if applicable)

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Reference: https://github.com/Quantco/multiregex Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Use multiregex to use a cached regex path patterns and datafile handlers mapping to detect package datafiles faster. Reference: #4064 Reference: #4061 Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

src/packagedcode/cache.py

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

src/packagedcode/cache.py

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Introduce a new option --binary-packages which looks for package/dependency data in binaries. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

We do not need the license index in a --package-only scan as this is designed to do a fast package detection only scan which skips the license detection. As license index loading takes a couple seconds in each case, this makes the package only scan much faster. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

pombredanne · 2025-12-15T12:57:37Z

docs/source/rst_snippets/basic_options.rst

 --system-package             Scan ``<input>`` for installed system package
                             databases.

+-b, --binary-package         Scan <input> for package and dependency related


What about this:

--package-in-exec Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust binaries.

Or

--package-in-compiled Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust compiled binaries.

pombredanne · 2025-12-15T13:04:44Z

etc/release/scancode-create-pypi-wheel.sh


 ./configure --dev
 venv/bin/scancode-reindex-licenses
+venv/bin/scancode-cache-package-patterns


What about naming this venv/bin/scancode-reindex-package-patterns to be consistent?

pombredanne · 2025-12-15T13:07:11Z

src/packagedcode/__init__.py


+
+# These handlers are special as they use filetype to
+# detect these binaries instead of datafile path patterns


Suggested change

# detect these binaries instead of datafile path patterns

# detect these compiled executable binaries instead of datafile path patterns

pombredanne · 2025-12-15T13:14:24Z

src/packagedcode/cache.py

+# This is the Pickle protocol we use, which was added in Python 3.4.
+PICKLE_PROTOCOL = 4
+
+PACKAGE_INDEX_LOCK_TIMEOUT = 60 * 6


360 seconds is a lot. 60 secs should be enough.

pombredanne · 2025-12-15T13:15:24Z

src/packagedcode/cache.py

+PACKAGE_INDEX_DIR = 'package_patterns_index'
+PACKAGE_INDEX_FILENAME = 'index_cache'
+PACKAGE_LOCKFILE_NAME = 'scancode_package_index_lockfile'
+PACKAGE_CHECKSUM_FILE = 'scancode_package_index_tree_checksums'


This is not used anymore (also should be dropped from licensing)

Suggested change

PACKAGE_CHECKSUM_FILE = 'scancode_package_index_tree_checksums'

pombredanne

Here are some nits for your consideration!

pombredanne · 2025-12-15T13:20:23Z

src/packagedcode/cache.py

+            pickle.dump(self, fn, protocol=PICKLE_PROTOCOL)
+
+
+def get_prematchers_from_glob_pattern(pattern):


Can you add a docstring and a few doctests ?

pombredanne · 2025-12-15T13:22:11Z

src/packagedcode/cache.py

+    Return a mapping of regex patterns to datafile handler IDs and
+    multiregex patterns consisting of regex patterns and prematchers.
+    """
+    handler_by_regex = {}


This should be a defaultdict(list) maybe?

pombredanne · 2025-12-15T13:25:56Z

src/packagedcode/cache.py

+    multiregex patterns consisting of regex patterns and prematchers.
+    """
+    handler_by_regex = {}
+    multiregex_patterns = []


Suggested change

multiregex_patterns = []

# stores tuples of ....????

multiregex_patterns = []

What about instead having a small attrs or dataclass object to use until a last minute conversion?

class AcceleratedPattern: regex :str # regular expression string prematchers :list[str] # list of prematcher strinsg for this regex handler_datasource_id" :str #handler

something more or less like that would help avoid creating parallel list until the end.

pombredanne · 2025-12-15T13:33:37Z

src/packagedcode/cache.py

+                handler_by_regex[regex_pattern]= [handler.datasource_id]
+
+    for regex in handler_by_regex.keys():
+        regex_and_prematcher = (regex, prematchers_by_regex.get(regex, []))


Also in keeping with the idea of using an object above, this could be a namedtuple

from typing import NamedTuple class RegexPrematchers(NamedTuple): regex: str prematchers: list[str]

pombredanne · 2025-12-15T13:35:40Z

src/packagedcode/recognize.py

        location=location,
+        application=application,
+        system=system,
+        binary=binary,


may be compiled?

pombredanne · 2025-12-15T13:36:48Z

docs/source/rst_snippets/basic_options.rst

 --system-package             Scan ``<input>`` for installed system package
                             databases.

+-b, --binary-package         Scan <input> for package and dependency related


Or

--package-in-compiled Scan compiled executable binaries such as ELF, WinpE and Mach-O files, looking for structured package and dependency metadata as found for example in Go and Rust compiled binaries.

AyanSinhaMahapatra added 3 commits November 17, 2025 15:12

Simplify GemfileHandler path patterns

489af1b

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add multiregex as a dependency

8d6fa73

Reference: https://github.com/Quantco/multiregex Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Add initial multiregex implementation

4fc3af2

Use multiregex to use a cached regex path patterns and datafile handlers mapping to detect package datafiles faster. Reference: #4064 Reference: #4061 Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra marked this pull request as draft November 17, 2025 09:49

AyanSinhaMahapatra changed the title ~~Fast package scan~~ Improve package scan performance Nov 17, 2025

github-advanced-security bot found potential problems Nov 17, 2025

View reviewed changes

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Fixed Show fixed Hide fixed

src/packagedcode/cache.py Fixed Show fixed Hide fixed

AyanSinhaMahapatra added 2 commits November 17, 2025 19:27

Add minimal tests for package cache

4438526

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

Cache multiregex matchers instead of patterns

3bbf35e

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

github-advanced-security bot found potential problems Nov 18, 2025

View reviewed changes

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

src/packagedcode/cache.py Dismissed Show dismissed Hide dismissed

Restore binary package manifest scanning

e0460ef

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra requested review from JonoYang and pombredanne November 19, 2025 09:47

AyanSinhaMahapatra marked this pull request as ready for review November 19, 2025 09:47

Only scan for bianry packages optionally

dde6bc9

Introduce a new option --binary-packages which looks for package/dependency data in binaries. Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

AyanSinhaMahapatra force-pushed the fast-package-scan branch from a99c4f8 to dde6bc9 Compare November 19, 2025 10:26

AyanSinhaMahapatra added 2 commits November 19, 2025 16:15

Add a new console script to build the package patterns cache

6b6a79b

Signed-off-by: Ayan Sinha Mahapatra <[email protected]>

pombredanne reviewed Dec 15, 2025

View reviewed changes



		# These handlers are special as they use filetype to
		# detect these binaries instead of datafile path patterns

	# detect these binaries instead of datafile path patterns
	# detect these compiled executable binaries instead of datafile path patterns

		pickle.dump(self, fn, protocol=PICKLE_PROTOCOL)


		def get_prematchers_from_glob_pattern(pattern):

	multiregex_patterns = []
	# stores tuples of ....????
	multiregex_patterns = []

Uh oh!

Improve package scan performance #4606

Are you sure you want to change the base?

Improve package scan performance #4606

Conversation

AyanSinhaMahapatra commented Nov 17, 2025 • edited by pombredanne Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AyanSinhaMahapatra commented Nov 17, 2025 •

edited by pombredanne

Loading