[BEP036] Tabular phenotype example datasets

[Arshitha]

Added pheno001 and pheno002 example dataset inspired by ds004215 on OpenNeuro but significantly modified to keep it simple and easy to convey the various use cases proposed in BEP036.

Use cases covered (and to be added to this PR):

• pheno001 - Single session with both phenotype and imaging data
• pheno002 - Two sessions with one imaging data only session
• pheno003 - Two sessions with one phenotype data only session
• pheno004 - Two sets of sessions. One set of sessions (e.g. screening, baseline, followup, etc) for phenotype data and another set of sessions (e.g. 01, 02, etc) for imaging data.

Still in draft state but would appreciate any and all feedback.

Pinging co-contributors: @ericearl @SamGuay @surchs

• BIDS Specification PR: [ENH] BEP036 - Phenotypic Data Guidelines bids-specification#2123
• BIDS Validator PR: Enh/bep036 extensions bids-validator#366

[ericearl]

This is great, thanks! I'm guessing you left it in draft state becausew of pheno001 and pheno002, right?

I think we should remove the age_at_visit column/field from all phenotype/ measurement tools and instead provide a root-level sessions file with that field. Should we maybe take that a step farther and RECOMMEND or say it's OPTIONAL to add age to the sessions file?

[Arshitha]

I like that idea. It's redundant information that can be aggregated to sessions level, and can be a recommendation in the BEP.

[Arshitha]

It's in Draft state because I haven't prepared pheno003 and pheno004 but yes, all four example datasets will violate the contribution guidelines.

[christinerogers]

Got a question from @dominikwelke --

Could this PR include an example showing how to represent multiple runs from one participant-session?

@ericearl mentioned today this is easily done by adding a run column in the .tsv, would be nice to see illustrated and mentioned here.

[ericearl]

I hijacked the not yet created pheno004 for use with some current bids-validator testing needs at bids-standard/bids-specification#2044. Can we make this a non-draft PR and get it merged hopefully? @effigies @Arshitha @SamGuay @surchs

Addressed in updates.

[effigies]

Please set the BIDS_SCHEMA environment variable to https://bids-specification--2044.org.readthedocs.build/en/2044/schema.json here:

bids-examples/.github/workflows/validate_datasets.yml

Line 114 in e52f77f

run: echo BIDS_SCHEMA=https://bids-specification.readthedocs.io/en/latest/schema.json >> $GITHUB_ENV

Please also add pheno004 to be skipped on legacy and stable:

bids-examples/.github/workflows/validate_datasets.yml

Lines 98 to 101 in e52f77f

	- name: Skip legacy validation for post-legacy datasets
	run: for DS in mrs_* dwi_deriv; do touch $DS/.SKIP_VALIDATION; done
	if: matrix.bids-validator == 'legacy'
	shell: bash

bids-examples/.github/workflows/validate_datasets.yml

Lines 103 to 106 in e52f77f

	- name: Skip stable validation for datasets with unreleased features
	run: for DS in dwi_deriv; do touch $DS/.SKIP_VALIDATION; done
	if: matrix.bids-validator != 'dev'
	shell: bash

[ericearl]

@effigies Is that comment just above here a note for me? I'm confused by most of it and don't feel safe editing those files as-is. If you need me to take care of that, can I sit with you, Ross, or Nell to figure it out or have it explained to me enough to be able to do the work?

[effigies]

Okay, I did what I asked. It looks like there are issues in the schema that need to be addressed, but also there are unrelated issues in pheno001-003: https://github.com/bids-standard/bids-examples/actions/runs/13188395001/job/36815880378?pr=465

[ericearl]

This is super-helpful @effigies, thank you! I'm bringing the errors out of the logs here for us (@Arshitha @SamGuay @surchs):

# pheno001

	[ERROR] MISSING_DATASET_DESCRIPTION A dataset_description.json file is required in the root of the dataset
		

	Please visit https://neurostars.org/search?q=MISSING_DATASET_DESCRIPTION for existing conversations about this issue.

	[ERROR] JSON_INVALID Not a valid JSON file.
		/sub-01/anat/sub-01_T1w.json
		/sub-01/anat/sub-01_T1w.json

		2 more files with the same issue

	Please visit https://neurostars.org/search?q=JSON_INVALID for existing conversations about this issue.

# pheno002

	[ERROR] MISSING_DATASET_DESCRIPTION A dataset_description.json file is required in the root of the dataset
		

	Please visit https://neurostars.org/search?q=MISSING_DATASET_DESCRIPTION for existing conversations about this issue.

	[ERROR] TSV_COLUMN_ORDER_INCORRECT Some TSV columns are in the incorrect order
		session_id
		/sessions.tsv - Column 0 (starting from 0) found at index 1.

	Please visit https://neurostars.org/search?q=TSV_COLUMN_ORDER_INCORRECT for existing conversations about this issue.

	[ERROR] TSV_INDEX_VALUE_NOT_UNIQUE An index column(s) was specified for the tsv file and not all of the values for it are unique.
		/sessions.tsv - Row: 4, Value: 01
		/sessions.tsv - Row: 5, Value: 02

	Please visit https://neurostars.org/search?q=TSV_INDEX_VALUE_NOT_UNIQUE for existing conversations about this issue.

	[ERROR] JSON_INVALID Not a valid JSON file.
		/sub-01/ses-01/anat/sub-01_ses-01_T1w.json
		/sub-01/ses-01/anat/sub-01_ses-01_T1w.json

		6 more files with the same issue

	Please visit https://neurostars.org/search?q=JSON_INVALID for existing conversations about this issue.

# pheno003

	[ERROR] MISSING_DATASET_DESCRIPTION A dataset_description.json file is required in the root of the dataset
		

	Please visit https://neurostars.org/search?q=MISSING_DATASET_DESCRIPTION for existing conversations about this issue.

	[ERROR] TSV_COLUMN_ORDER_INCORRECT Some TSV columns are in the incorrect order
		session_id
		/sessions.tsv - Column 0 (starting from 0) found at index 1.

	Please visit https://neurostars.org/search?q=TSV_COLUMN_ORDER_INCORRECT for existing conversations about this issue.

	[ERROR] TSV_INDEX_VALUE_NOT_UNIQUE An index column(s) was specified for the tsv file and not all of the values for it are unique.
		/sessions.tsv - Row: 4, Value: baseline

	Please visit https://neurostars.org/search?q=TSV_INDEX_VALUE_NOT_UNIQUE for existing conversations about this issue.

	[ERROR] JSON_INVALID Not a valid JSON file.
		/sub-01/ses-baseline/anat/sub-01_ses-baseline_T1w.json
		/sub-01/ses-baseline/anat/sub-01_ses-baseline_T1w.json

		2 more files with the same issue

	Please visit https://neurostars.org/search?q=JSON_INVALID for existing conversations about this issue.

[Arshitha]

@ericearl pheno006 example dataset is ready for your review! It's an example where a sessions file won't add any value.

[Arshitha]

@ericearl I've committed the suggested change! do we need anything else for approval by a maintainer?

[ericearl]

We need a couple more things:

1. The bids-validator needs to be able to manage optional columns in the initial_columns list
2. We need to try to trigger all of our bids-validator warnings and errors present in the schema upon AdditionalValidation

Perhaps we should draft a test plan doc and execute it?

[christinerogers]

can we add "phenotypic" or "pheno" in this PR title so that at a glance it's clear 🙏 @ericearl

[ericearl]

can we add "phenotypic" or "pheno" in this PR title so that at a glance it's clear 🙏 @ericearl

Title changed. Thanks for the suggestion!

[yarikoptic]

FTR -- I edited PR description to point to the BIDS PR to facilitate validation of the examples against that BEP. Note that there are conflicts now in that workflow (as one of the aspects -- no need to hardcode/patch for the BIDS PR now).

[yarikoptic]

validation against legacy was dropped, so I guess all changes to the workflow could be just dropped

[yarikoptic]

are we doing that fun dance again of mixing in description of columns and additional metadata?

All of that could be avoided if we followed somewhat established approach for any {entity} to have an {entity_plural}.tsv which would have description per each value of that entity. So here if it was e.g. pheno-ace.{tsv,json} and then phenotype.tsv (at top level of within phenotype/ folder -- doesn't matter due to Inheritance principle) with phenotype_id and description columns -- everything would have been nice and consistent.

I really do not get why there is ongoing brewing of some non-BIDS-like cocktail in BEP036.

Related issues backrefs:

• Formalize the concept of [{leading entities}_]{entity_plural}.{tsv,json} file(s) bids-specification#2283
• Formalization of "entities" flow or new-function-in-life for _mod- bids-2-devel#97 (relating to "seggregated" representations)