Add num_subjects: advanced-search operator - #2827
Open
bendichter wants to merge 8 commits into
Open
Conversation
`num_subjects:N` matches dandisets whose `assetsSummary.numberOfSubjects` is at least N. The threshold reads as "studies with ≥N subjects", which is the search intent users actually have (an upper bound is rarely useful and isn't worth the syntax cost). A `num_sessions:` operator was considered but is intentionally NOT included: dandischema's `AssetsSummary` does not aggregate sessions, and assets carry no `sessionId` field from which to derive a count. Adding it would require upstream schema work. Implementation: - `COUNT_OPS` dispatch table in `operators.py` maps op → jsonpath into `Version.metadata`; adding any of `num_files`, `num_bytes`, `num_samples`, `num_cells` is a one-line entry. - `_apply_count_filter` builds a `jsonb_path_exists(metadata, ...)` predicate. The integer is bound via the jsonpath `vars` parameter (not inlined into SQL or jsonpath text), so the value is injection-safe. Versions whose metadata lacks the count field never match — the jsonpath `?` filter drops missing/null/non-numeric values naturally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
|
Some considerations here: GitHub uses comparison operators, e.g. "comments:>10" example Also in GitHub, you can express a range with "comments:10..100" I opted not to use that here, and just have all queries be ≥. So for example: subjects:2 means that a dandiset as 2 or more subjects. The justification is I think a query that uses < would be pretty uncommon (why would they want a dandiset that has fewer than 50 subjects?). It could be useful for analytics of the available data, but I don't think it would be used often for finding a dataset. I could be wrong and I could be convinced to change this. It's just a more complex parser. |
Member
Author
|
@CodyCBakerPhD requested the comparators be included here |
…advanced-search-counts
Count values now accept an optional comparator prefix: `num_subjects:>10`, `num_subjects:>=10`, `num_subjects:<10`, `num_subjects:<=10`, `num_subjects:=10` (exactly 10). A bare value (`num_subjects:10`) keeps its "at least N" meaning (equivalent to `>=`), which is the intent behind count search and preserves existing behavior. The comparator is parsed in `_apply_count_filter` and mapped through a fixed allowlist to a Postgres jsonpath operator (`=` → `==`); it is never interpolated verbatim from user input, and the integer is still bound via the jsonpath `vars` parameter, so the predicate stays injection-safe. A comparator without a number (e.g. `num_subjects:>=`) is rejected with a 400. Updates the OpenAPI help text, the search-field popover entry, and the integration tests (dedicated comparator test + extended invalid-value cases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This branch grows the operator list to 20 (owner, contributor, author, num_subjects, etc.), but the autocomplete E2E spec merged from master hardcoded toHaveCount(10), so "shows the full operator list on focus" failed with 20 elements. Compare the dropdown option count against the help-table row count (both render from the same source list in the component) so the test reflects "every documented operator is offered" and no longer needs updating when operators are added or removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Member
Author
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Draft. Stacked on #2822 — review will be cleaner once the contributor PR merges. The diff against master will collapse to this PR's commits only at that point.
Adds
num_subjects:N— filter dandisets by subject count, perVersion.metadata.assetsSummary.numberOfSubjects(populated by dandischema'saggregate_assets_summary()).Comparator syntax
The value is a non-negative integer, optionally prefixed with a comparator:
>,>=,<,<=, or=. A bare value defaults to>=("at least N"), which is the intent users most often have — "studies with at least N subjects" — while the explicit comparators cover ranges and exact matches without needing separate operators or..range syntax.Why no
num_sessions:Considered, deliberately omitted:
dandischema.AssetsSummarydoes not aggregate sessions. Its numeric fields arenumberOfBytes,numberOfFiles,numberOfSubjects,numberOfSamples,numberOfCells— nonumberOfSessions. See feat: add numberOfSessions to AssetsSummary dandi-schema#409BareAssethas nosessionIdfield either, so we can't derive a count from per-asset metadata at query time.num_sessions:would need upstream work in dandischema (addnumberOfSessionstoAssetsSummary) plus an aggregator that infers sessions from BIDSses-paths and/or NWB session metadata. That's its own project.Implementation
COUNT_OPSdict indandiapi/api/services/search/operators.pymapsnum_subjects→$.assetsSummary.numberOfSubjects. Adding any ofnum_files,num_bytes,num_samples,num_cellsis a one-line entry — same dispatch._apply_count_filter(filters.py) parses the value with_COUNT_VALUE_RE(^(>=|<=|>|<|=)?\s*(\d+)$), maps the comparator through the_COUNT_COMPARATORSallowlist (never taken verbatim from user input), and buildsjsonb_path_exists(metadata, '$path ? (@ <op> $val)', '{"val": N}'::jsonb). The integer is bound via the jsonpathvarsparameter (not inlined into SQL or jsonpath text) and the comparator is a mapped constant — injection-safe.?predicate drops missing/null/non-numeric values naturally. A freshly-created dandiset with no assets summarized yet won't satisfynum_subjects:>=1.Files
dandiapi/api/services/search/operators.py—COUNT_OPStable; folded intoOPERATOR_KEYSdandiapi/api/services/search/filters.py—_apply_count_filter()(comparator parsing) + dispatch entrydandiapi/api/views/serializers.py— OpenAPI help textweb/src/components/DandisetSearchField.vue— advanced-search popover entry (documents the comparator forms)dandiapi/api/tests/test_dandiset.py— consolidated integration test (comparators, missing-field exclusion, case-insensitive key, composition withcreated_after:) + invalid-value 400 testTest plan
pytest dandiapi/api/tests/test_dandiset.py dandiapi/api/tests/test_search_parser.py -k 'advanced_search or search_parser'ruff check dandiapi/api/services/search/ dandiapi/api/tests/test_dandiset.py— cleanvue-tsc --noEmit+eslint— cleanManual, full local stack (Docker backend on this branch +
npm run devfrontend pointed at it), seeded four non-empty dandisets withnumberOfSubjects= 5 / 10 / 15 / 40:num_subjects:>=10num_subjects:10(bare ⇒ ≥)num_subjects:>10num_subjects:<10num_subjects:<=10num_subjects:=40num_subjects:0num_subjects:abc