[ENH]: BEP032 formal review

[CodyCBakerPhD]

BEP 032: Under review

This BEP is currently under review: #2406

Please post significant discussion to that thread. Minor issues, such as typos or clarifications, may be posted as suggestions to this PR. These suggestions may be accepted or rejected without discussion, so do not make substantial proposals in this way.

---

Original post

To all @bids-standard/maintainers

In line with the penultimate steps outlined in the Proposed BEP Review process, the BEP032 team would like to submit our pull request for formal review by the maintainers and community!

This PR adds a specification for microelectrode electrophysiology datasets based on the BEP032 proposal.

We happily welcome all enhancements, fixes, suggestions, requests, and critiques by the broader community.

This work is the result of some enormous efforts over the last year by @ree-gupta @bendichter @yarikoptic @twachtler @robertoostenveld @lzehl @apdavison and many others whose full work history has been squashed as an extended description on this commit and can be seen in the original draft PR, a copy of which will be kept on a branch in perpetuity on my personal fork (https://github.com/CodyCBakerPhD/bids-specification/tree/backup-bep032-work).

Quick links

• HTML preview of this BEP
• Working overleaf manuscript project with source and git history here
• Example dataset
• Another example dataset

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (4235ae7) to head (2aa6e30).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2307   +/-   ##
=======================================
  Coverage   83.07%   83.07%           
=======================================
  Files          22       22           
  Lines        1696     1696           
=======================================
  Hits         1409     1409           
  Misses        287      287           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bendichter]

@effigies it looks like there are minor merge conflicts. Should we resolve these?

[effigies]

Yes. They look easy to resolve.

[ree-gupta]

Hey! @CodyCBakerPhD Thanks again for the PR. I found a potential issue/improvement in the spec:

I missed adding ecephys and icephys datatypes to the allowed datatypes for physio suffix.

Reference:

bids-specification/src/schema/rules/files/raw/task.yaml

Lines 10 to 14 in 4b40578

	datatypes:
	- beh
	- eeg
	- ieeg
	- nirs

This would allow physiological recordings (cardiac, eye tracking) and stim to be stored alongside microelectrode electrophysiology data.

I am still not sure if we can push to this PR branch or collect all the reviews for the end, hence not making changes yet.

[robertoostenveld]

@bids-maintenance can we get this announced to the community through the mailing list for formal review, and get a few experts explicitly assigned/invited as potential reviewers?

[lzehl]

@bids-maintenance could you help us with the question: "can we get this announced to the community through the mailing list for formal review, and get a few experts explicitly assigned/invited as potential reviewers?"

[julia-pfarr]

Hi, we have two scheduled BEP community review periods per year, the next one is in April. We put you on the list for this review period! Before that, some maintainers would like to meet with all of you and go through the BEP, to make sure everything is ready for the community review. @effigies and @rwblair, can you post a when2meet with your availabilities here? Thank you!

[ree-gupta]

While curating a set of resting-state ecephys datasets (NHP, no pharmacology, no stimuli), I noticed that the pharmaceutical fields introduced as recommended don't accept "n/a" as a value. Other fields types like HardwareFilters (objects/metadata.yaml:1750-1756) and SoftwareFilters (objects/metadata.yaml:3964-3970) handle this by including {type: string, enum: [n/a]} in their anyOf, but these don't have that escape.

Since they're recommended, omitting them still triggers SIDECAR_KEY_RECOMMENDED warnings.

Two possible fixes:

1. Add "n/a" to the type definitions in objects/metadata.yaml (like HardwareFilters and SoftwareFilters do). But since these definitions are shared with other modalities, this would also allow "n/a" there, which may be undesirable.
2. Downgrade from recommended to optional in microephys.yaml:62-71. This suppresses the warning while keeping the fields available for experiments that do use pharmacology, but loses the signal that they should be provided when applicable.

[yarikoptic]

I think there is no specific rule, but @effigies struggled to move many RECOMMENDs into OPTIONAL for MRI since there was too much noise from validator.

May be rule of thumb could be simple: What % of studies use some pharmacology worth adding? if majority (over 50%) -- keep in RECOMMENDED, otherwise - go for OPTIONAL. Eventually, with all the LLMing, and after underlying papers are associated with the dataset for mining and informed metadata editing like with WiP by @bendichter https://medit.dandiarchive.org for DANDI, those OPTIONAL could be auto-filled out or requested based on the study background mined from the extra materials.

[effigies]

Another approach could be to add a required boolean field like "PharmacologicalIntervention". If False, then fields are OPTIONAL; if True, they are REQUIRED/RECOMMENDED (as appropriate).

[effigies]

Forgot to send a when2meet. @CodyCBakerPhD and anybody who wants to meet with us: https://www.when2meet.com/?35364181-QNHek

[ree-gupta]

While curating a set of resting-state ecephys datasets (NHP, no pharmacology, no stimuli), I noticed that the pharmaceutical fields introduced as recommended don't accept "n/a" as a value. Other fields types like HardwareFilters (objects/metadata.yaml:1750-1756) and SoftwareFilters (objects/metadata.yaml:3964-3970) handle this by including {type: string, enum: [n/a]} in their anyOf, but these don't have that escape.

Since they're recommended, omitting them still triggers SIDECAR_KEY_RECOMMENDED warnings.

Two possible fixes:

1. Add `"n/a"` to the type definitions in `objects/metadata.yaml` (like `HardwareFilters` and `SoftwareFilters` do). But since these definitions are shared with other modalities, this would also allow `"n/a"` there, which may be undesirable.

2. Downgrade from `recommended` to `optional` in [microephys.yaml:62-71](https://github.com/bids-standard/bids-specification/blob/bep032-review/src/schema/rules/sidecars/microephys.yaml#L62-L71). This suppresses the warning while keeping the fields available for experiments that do use pharmacology, but loses the signal that they _should be_ provided when applicable.

Decision on 2026.03.04: Make it optional for this BEP.

[effigies]

@bendichter @CodyCBakerPhD All of the top availability slots are missing one of you. Who would be best to have on this call?

[CodyCBakerPhD]

@effigies Probably default to Ben as he's been of a driving force on the BEP whereas I sit in the background and make sure nwb2bids tooling keeps up to date and works out any bugs, ensures validity, etc

But TBH it is really just next week, week-and-a-half I think that both of us are very busy; BBQS has a workshop going on and then COSYNE. So once those are done it should be easier to schedule

[effigies]

We can push off if you prefer, but if we're aiming for a review period in April, there's only so much time to push. I'll leave it to you. Otherwise, it looks like Monday at 2pm EDT is going to be the best time. LMK.

[CodyCBakerPhD]

Unless the other folks from the group have confirmed 2pm time already, probably best to try next week

[ree-gupta]

Both this week or the next works well for me.

[effigies]

I scheduled review for next week at 2pm EDT. If you did not receive an invitation, please add this event to your calendar.

[CodyCBakerPhD]

For dealing with comments from the first round of review by BIDS maintainers (on 3/16, thank you @rwblair and @effigies!) in preparation for community review in April, here are some TODO items for the group

@ree-gupta From above comment

I missed adding ecephys and icephys datatypes to the allowed datatypes for physio suffix.

Can you do that so we can ensure this BEP is up to date for the planned April timeline?

@CodyCBakerPhD will review what nwb2bids is capable of auto-extracting and reduce any impractical metadata fields from recommended to optional per the maintainers' suggestion

[ree-gupta]

Can you do that so we can ensure this BEP is up to date for the planned April timeline?

Yes I will change this and draft a rule to also perform the channels -> electrodes -> probes consistency checks.

[CodyCBakerPhD]

The other follow-up from the meeting with the BIDS maintainers related to very basic file content validation: bids-standard/bids-validator#369

[ree-gupta]

Hi, we have two scheduled BEP community review periods per year, the next one is in April. We put you on the list for this review period

Hi BIDS-maintainers :) do we have an update in which review period would this proposal fall in? Are there any more open questions or changes pending on our end?
Thanks!

Tagging to notify: @julia-pfarr @effigies

[effigies]

Was just going to send out emails. We're planning to open review on Monday.

[effigies]

This PR is under review: #2406

Please post significant discussion to that thread. Minor issues, such as typos or clarifications, may be posted as suggestions to this PR. These suggestions may be accepted or rejected without discussion, so do not make substantial proposals in this way.

[effigies]

Still working on a final review. These are my comments from reading the text. I haven't yet gone through the schema closely.

[effigies]

I don't understand this TODO. Apologies if I missed a discussion.

[effigies]

Missing table here. Looks like the following, rendered:

[effigies]

This TODO appears undone. If it is not ready, I would remove this section and open an issue to add it later.

[yarikoptic]

I think it came from good old times when also @JuliaSprenger was involved in this BEP, so that https://gin.g-node.org/sprenger/multielectrode_grasp/src/bep_animalephys was created. But overall it was left as TODO since it is a chicken-egg issue. We can't have this until we finalize BEP since otherwise those datasets might not be valid ones. I wonder also if we could/should just point to https://github.com/bids-dandisets/ which would have many of such examples. I would not bother with https://gin.g-node.org/NeuralEnsemble/BEP032-examples though since those are largely replaced with bids-examples. But I would leave in smth like

Suggested change

	Several real-world datasets have been formatted using this specification and can be used for practical guidance when curating a new dataset.
	<!-- TODO: Update with current real datasets. A current version of these datasets [can be found on GIN](https://gin.g-node.org/NeuralEnsemble/BEP032-examples) .
	For a complete dataset including all data samples the extracellular microelectrode dataset published in [Brochier (2018)](https://doi.org/10.1038/sdata.2018.55) has been reorganized according to the current version of this BEP, using the NIX data format.
	The up-to-date version of the dataset [can be found on GIN](https://gin.g-node.org/sprenger/multielectrode_grasp/src/bep_animalephys) .
	We will also publish another dataset using the NWB data format in the near future, and a dataset acquired -->
	Several real-world datasets have been formatted using this specification and can be used for practical guidance when curating a new dataset.
	<!-- TODO: Upon finalization of the BEP032, update with current real datasets if they are updated to the spec and pass bids-validator.
	For a complete dataset including all data samples the extracellular microelectrode dataset published in [Brochier (2018)](https://doi.org/10.1038/sdata.2018.55) has been reorganized according to the current version of this BEP, using the NIX data format and [hosted on GIN](https://gin.g-node.org/sprenger/multielectrode_grasp/src/bep_animalephys) .
	Also numerous datasets from the [DANDI Archive](https://dandiarchive.org) have been converted to BIDS using [nwb2bids]() and shared [on GitHub](https://github.com/bids-dandisets/).
	>

[effigies]

The sample-<label> entity is first introduced here and only described in the photos section. As a reader it came out of nowhere.

This appears to have been taken from microscopy. I would reuse text from there to the extent that it's applicable.

Is samples.tsv required if the sample-<label> entity is used? Is it permitted?

[bendichter]

Hey BEP032 team, let's discuss @effigies 's point about sample-<label> coming out of nowhere. He's right that it's under-explained, and digging in I found the requirement level is inconsistent between the prose and the examples, plus a schema gap. Here's what I'd suggest, but curious what you all think:

Introduce the entity up front. Right now its only real definition is buried in the Photos section, framed as just for histological photos, but the intracellular examples use it on every file (sample-cell001, etc.). I'd propose adding a definition near the top in "Terminology: Modality and Datatypes," adapted from the microscopy text.

Split the requirement level by datatype rather than making it uniformly optional or copying microscopy's blanket REQUIRED:

• ecephys (in-vivo extracellular): OPTIONAL, and typically absent, since there's no tissue sample. Requiring it here would be wrong.
• icephys (patch): REQUIRED. A patched cell effectively is a sample, and the datatype inherently deals in discrete cells/slices. This matches what our intracellular example already does (every file is sample-cellNNN) and mirrors how micr treats it.

In the schema this is just leaving sample: optional for ecephys and changing it to sample: required in the icephys block of microephys.yaml.

One sub-question: micr does not exempt the single-sample case (one sample per subject still gets sample-01). I'd suggest we follow that, since a single-sample exemption causes filename churn if a second cell is added later and forces tools to special-case it. So: required for icephys even when there's only one sample.

samples.tsv. Chris also asked whether it's required when the entity is used. It currently isn't enforced at all for microephys, since SamplesTSVMissing only triggers on "micr" in dataset.modalities. I'd suggest requiring samples.tsv whenever the sample- entity is present (which, with the split above, means always for icephys and only-if-used for ecephys), and adding a check keyed on entity-presence rather than modality.

samples.tsv columns we inherit. These are modality-agnostic, so microephys gets them for free: sample_id (required), participant_id (required), sample_type (required), pathology (recommended), derived_from (recommended). They all fit microephys fine, so I don't think we need to define anything new. Two things worth a quick look:

• sample_type is a required enum, and its full value list is broader than just cell types: tissue, whole organisms, organoid, technical sample, plus cell line / in vitro differentiated cells / primary cell / cell-free sample / cloning host. So an acute slice maps cleanly to tissue and organoid recordings to organoid — no missing values. The only thing to settle is a granularity convention: for patch clamp, is the "sample" the slice or the individual cell? If it's the cell, tissue is still the closest sample_type. We could also register both and use derived_from to link sample-cell001 back to sample-slice01. I'd suggest we pick one convention and document it.
• sample_id labels are restricted to ^sample-[0-9a-zA-Z+]+$ (no underscores/hyphens), so sample-cell001 is fine but keep that in mind for examples.

Cleanup. Trim the Photos-section mention down to a back-reference to the new definition.

If the per-datatype split (optional ecephys / required icephys) and the entity-presence trigger for samples.tsv sound right, I'm happy to put up the edits. The main thing I'd want to settle first is the sample granularity convention (cell vs. slice) for patch data.

[yarikoptic]

nice dig! I do not know much about icephys, but is it really that there would be multiple samples per each subject? ATM we do mandate sub- entity already, so if it is actually typical to have multiple samples - people would just use it if/where needed to disambiguate within subject, so we do not even need to make it RECOMMENDED.

And yes, I would recommend having samples.tsv if _sample- entity is used.

NB/backref -- this will add one more "consistent" case for samples.tsv to be inconsistent' in having dual-index for a single plural entity filename as mentioned in #2283

[bendichter]

Yes, multiple samples per subject is normal for icephys. A single animal commonly yields several slices, and several cells are patched per slice, so >1 sample per subject is the rule, not the exception.

On the requirement level: sub- alone can't disambiguate samples within a subject, and as you note people would reach for sample- to do that anyway. So the real choice is whether sample- is present inconsistently (only when >1) or always. The problem with "only when needed" is churn: collect one cell → sub-X_..., collect a second later → you now have to rename the first file to add sample-, and tools have to special-case the single-sample form. micr avoids this by requiring sample- even for a single sample, and I'd lean toward icephys following it for the same reason.

Agreed on requiring samples.tsv whenever sample- is used.

That said, the required-vs-optional call (and the related cell-vs-slice granularity convention) has real trade-offs in both directions, so I think it warrants input from the larger BEP032 team before we settle it.

[effigies]

This appears to be vestigial, as there is no Procedure metadata defined in the schema, and PharmaceuticalName is shown directly in the top-level metadata. (I'm assuming "node" means a field in a JSON structure.)

[effigies]

These are implied by the $ref: rules.files.raw.channels.electrodes.

Suggested change

	suffixes:
	- electrodes
	extensions:
	- .json
	- .tsv

[effigies]

Suggested change

	suffixes:
	- channels
	extensions:
	- .json
	- .tsv

[effigies]

Suggested change

	suffixes:
	- channels
	extensions:
	- .json
	- .tsv