Skip to content

Add num_subjects: advanced-search operator - #2827

Open
bendichter wants to merge 8 commits into
dandi:advanced-search-contributorfrom
bendichter:advanced-search-counts
Open

Add num_subjects: advanced-search operator#2827
bendichter wants to merge 8 commits into
dandi:advanced-search-contributorfrom
bendichter:advanced-search-counts

Conversation

@bendichter

@bendichter bendichter commented May 13, 2026

Copy link
Copy Markdown
Member

Draft. Stacked on #2822 — review will be cleaner once the contributor PR merges. The diff against master will collapse to this PR's commits only at that point.

Adds num_subjects:N — filter dandisets by subject count, per Version.metadata.assetsSummary.numberOfSubjects (populated by dandischema's aggregate_assets_summary()).

num_subjects:10                       # ≥10 subjects (bare value means "at least")
num_subjects:>=10                     # ≥10 subjects (explicit)
num_subjects:>10                      # strictly more than 10
num_subjects:<5                       # fewer than 5
num_subjects:<=5                      # at most 5
num_subjects:=0                       # exactly 0
num_subjects:>=1 species:mouse        # AND'd: ≥1 subject AND has a mouse asset
NUM_SUBJECTS:50                       # case-insensitive operator key
num_subjects:abc                      # 400 — non-negative integer required

Comparator syntax

The value is a non-negative integer, optionally prefixed with a comparator: >, >=, <, <=, or =. A bare value defaults to >= ("at least N"), which is the intent users most often have — "studies with at least N subjects" — while the explicit comparators cover ranges and exact matches without needing separate operators or .. range syntax.

Why no num_sessions:

Considered, deliberately omitted:

  • dandischema.AssetsSummary does not aggregate sessions. Its numeric fields are numberOfBytes, numberOfFiles, numberOfSubjects, numberOfSamples, numberOfCells — no numberOfSessions. See feat: add numberOfSessions to AssetsSummary dandi-schema#409
  • BareAsset has no sessionId field either, so we can't derive a count from per-asset metadata at query time.
  • A useful num_sessions: would need upstream work in dandischema (add numberOfSessions to AssetsSummary) plus an aggregator that infers sessions from BIDS ses- paths and/or NWB session metadata. That's its own project.

Implementation

  • COUNT_OPS dict in dandiapi/api/services/search/operators.py maps num_subjects$.assetsSummary.numberOfSubjects. Adding any of num_files, num_bytes, num_samples, num_cells is a one-line entry — same dispatch.
  • _apply_count_filter (filters.py) parses the value with _COUNT_VALUE_RE (^(>=|<=|>|<|=)?\s*(\d+)$), maps the comparator through the _COUNT_COMPARATORS allowlist (never taken verbatim from user input), and builds jsonb_path_exists(metadata, '$path ? (@ <op> $val)', '{"val": N}'::jsonb). The integer is bound via the jsonpath vars parameter (not inlined into SQL or jsonpath text) and the comparator is a mapped constant — injection-safe.
  • Versions missing the field don't match: the jsonpath ? predicate drops missing/null/non-numeric values naturally. A freshly-created dandiset with no assets summarized yet won't satisfy num_subjects:>=1.
  • Match semantics mirror the contributor pattern: a dandiset matches if at least one of its versions satisfies the predicate.

Files

  • dandiapi/api/services/search/operators.pyCOUNT_OPS table; folded into OPERATOR_KEYS
  • dandiapi/api/services/search/filters.py_apply_count_filter() (comparator parsing) + dispatch entry
  • dandiapi/api/views/serializers.py — OpenAPI help text
  • web/src/components/DandisetSearchField.vue — advanced-search popover entry (documents the comparator forms)
  • dandiapi/api/tests/test_dandiset.py — consolidated integration test (comparators, missing-field exclusion, case-insensitive key, composition with created_after:) + invalid-value 400 test

Test plan

  • pytest dandiapi/api/tests/test_dandiset.py dandiapi/api/tests/test_search_parser.py -k 'advanced_search or search_parser'

  • ruff check dandiapi/api/services/search/ dandiapi/api/tests/test_dandiset.py — clean

  • vue-tsc --noEmit + eslint — clean

  • Manual, full local stack (Docker backend on this branch + npm run dev frontend pointed at it), seeded four non-empty dandisets with numberOfSubjects = 5 / 10 / 15 / 40:

    query matches
    num_subjects:>=10 10, 15, 40
    num_subjects:10 (bare ⇒ ≥) 10, 15, 40
    num_subjects:>10 15, 40
    num_subjects:<10 5
    num_subjects:<=10 5, 10
    num_subjects:=40 40
    num_subjects:0 5, 10, 15, 40
    num_subjects:abc 400 with a helpful message

bendichter and others added 2 commits May 13, 2026 19:56
`num_subjects:N` matches dandisets whose
`assetsSummary.numberOfSubjects` is at least N. The threshold reads as
"studies with ≥N subjects", which is the search intent users actually
have (an upper bound is rarely useful and isn't worth the syntax cost).

A `num_sessions:` operator was considered but is intentionally NOT
included: dandischema's `AssetsSummary` does not aggregate sessions,
and assets carry no `sessionId` field from which to derive a count.
Adding it would require upstream schema work.

Implementation:
- `COUNT_OPS` dispatch table in `operators.py` maps op → jsonpath into
  `Version.metadata`; adding any of `num_files`, `num_bytes`,
  `num_samples`, `num_cells` is a one-line entry.
- `_apply_count_filter` builds a `jsonb_path_exists(metadata, ...)`
  predicate. The integer is bound via the jsonpath `vars` parameter
  (not inlined into SQL or jsonpath text), so the value is
  injection-safe. Versions whose metadata lacks the count field never
  match — the jsonpath `?` filter drops missing/null/non-numeric
  values naturally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bendichter

bendichter commented Jun 1, 2026

Copy link
Copy Markdown
Member Author

Some considerations here:

GitHub uses comparison operators, e.g. "comments:>10" example

Also in GitHub, you can express a range with "comments:10..100"

I opted not to use that here, and just have all queries be ≥. So for example: subjects:2 means that a dandiset as 2 or more subjects. The justification is I think a query that uses < would be pretty uncommon (why would they want a dandiset that has fewer than 50 subjects?). It could be useful for analytics of the available data, but I don't think it would be used often for finding a dataset. I could be wrong and I could be convinced to change this. It's just a more complex parser.

@bendichter

Copy link
Copy Markdown
Member Author

@CodyCBakerPhD requested the comparators be included here

bendichter and others added 2 commits June 8, 2026 15:59
Count values now accept an optional comparator prefix: `num_subjects:>10`,
`num_subjects:>=10`, `num_subjects:<10`, `num_subjects:<=10`, `num_subjects:=10`
(exactly 10). A bare value (`num_subjects:10`) keeps its "at least N" meaning
(equivalent to `>=`), which is the intent behind count search and preserves
existing behavior.

The comparator is parsed in `_apply_count_filter` and mapped through a fixed
allowlist to a Postgres jsonpath operator (`=` → `==`); it is never interpolated
verbatim from user input, and the integer is still bound via the jsonpath `vars`
parameter, so the predicate stays injection-safe. A comparator without a number
(e.g. `num_subjects:>=`) is rejected with a 400.

Updates the OpenAPI help text, the search-field popover entry, and the
integration tests (dedicated comparator test + extended invalid-value cases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bendichter and others added 3 commits July 6, 2026 14:51
This branch grows the operator list to 20 (owner, contributor, author,
num_subjects, etc.), but the autocomplete E2E spec merged from master
hardcoded toHaveCount(10), so "shows the full operator list on focus"
failed with 20 elements.

Compare the dropdown option count against the help-table row count (both
render from the same source list in the component) so the test reflects
"every documented operator is offered" and no longer needs updating when
operators are added or removed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@bendichter

Copy link
Copy Markdown
Member Author

Requires backend changes, so it doesn't work in preview, but here's a demo I ran locally:

Screen.Recording.2026-07-07.at.9.06.33.AM.mov
image

@bendichter
bendichter requested review from yarikoptic and removed request for yarikoptic July 7, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant