Conformity to DANDI standards?

[gwappa]

I would like to know how important the DANDI standards would be in organizing dandiset files.

Currently, we have all the subject folders as follows (and the subject folder is right under the dataset root, as the DANDI standard specifies):

sub-001/
│
├─ sub-001.csv  # metadata about all the sessions
│
├─ sub-001_1900-01-01_task_day1/  # the first day of task training
│  │
│  ├─ sub-001_1900-01-01_task_day1.nwb  # the NWB file
│  ├─ imaging/  # imaging data folder
│  │  ├─ sub-001_1900-01-01_task_day1_B.tif  # B channel data
│  │  └─ sub-001_1900-01-01_task_day1_V.tif  # V channel data
│  └─ videos/  # video data folder (if any)
│     ├─ sub-001_1900-01-01_task_day1_body.mp4  # body video
│     ├─ sub-001_1900-01-01_task_day1_eye.mp4   # eye video
│     └─ sub-001_1900-01-01_task_day1_face.mp4  # face video
│
├─ sub-001_1900-01-02_task_day2  # and so on
┆

And obviously the structure does not conform to the DANDI standards, so dataset validation fails.

Now I am tempted to skip the validation procedures in uploading the dataset because I believe that this is another way of keeping data organized.

So I am wondering what would be the consequences of skipping the validation step. Would it be just fine, provided the structure is the way I use? Or, would it harm so much the way DANDI is organized? Please let me know a bit more about the background of the DANDI standards.

Thank you very much!

[kabilar]

Hi @gwappa, thanks for reaching out. The DANDI standards largely conform to the BIDS specification and we are working on BIDS extensions so that DANDI and BIDS align. It would be best to adhere to a standard so that it is easier for users of your dataset to understand and reuse your files. Effectively, you could use the ses entity so that sub-001_1900-01-01_task_day1/ is replaced with sub-001_ses-1900-01-02/ or sub-001_ses-02/.

[satra]

DANDI would like organization itself to be standardized across datasets. The neurophysiology community has largely done ad hoc standards within labs and this has led to numerous differences across metadata and data organization. The other aspect of standardization is that analyses tool development gets more structured. When data are distributed through DANDI it is for a wider community of scientists and developers, and lately other technologies.

Therefore we do not allow a dandiset that does not pass validation to be published. Hence, we require conforming to DANDI's standards to publish and receive a DOI. These are meant to make data more usable by other researchers. We are here to help and to improve tools that help with this process.

[gwappa]

Thank you @kabilar and @satra for your inputs!

OK, then I start trying our data to conform to DANDI standards.
In doing so, could you tell me where you keep all the information related to naming / structuring conventions? (I just failed to find it)

I particularly have the concerns dealing with:

• multiple sessions (of different modalities) within a single day
• naming of externally referenced files
• external storage formatting of imaging data (we currently use TIFF)

I love to know how the resulting structure would look like. I am not very much keen to convert the dataset into a BIDS-like structure, but I would think about it if it results in a simpler file structure.

Thanks again!

[yarikoptic]

Have you tried dandi organize command (we describe it briefly as part of a generic upload process: https://docs.dandiarchive.org/example-notebooks/dandi/DANDI%20User%20Guide%2C%20Part%20I/#uploading-to-dandi ) .

[gwappa]

Thanks @yarikoptic ,

I see, dandi organize has some descriptions on the topic. I will try running this and see how it goes.

[gwappa]

After dealing with dandi organize, I found that the external imaging data file needs to be in AVI, not in TIFF.

But we have our imaging data in 16-bit unsigned int, and neither Fiji/ImageJ nor MATLAB efficiently handles 16-bit grayscale AVI.
In terms of DANDI standards, are we still allowed to use TIFF to represent these data, as long as the file structure is in accordance?

[satra]

@gwappa - could you please tell us what's in the TIFF file? is it a timeseries? static images, etc.,.? we hope to help you get the information into the most relevant option and format.

[gwappa]

thanks @satra , we have widefield calcium imaging data (time series of 40k-60k frames), and we want to have them as external data files. uint16 is one of the default output mode from our sCMOS camera.

[satra]

thanks @gwappa - that data should fit right into NWB, provide significant benefits for computing, and you can still access it from many common tools. if you have not seen the NWB Guide you may want to check it out. it will, in many cases, help individuals convert their data to NWB using known best practices for compression, and help organize and upload to DANDI.

@bendichter - what do you think?

[gwappa]

@satra thanks for your suggestions.

But including them into the NWB adds to its size by ~10 GB after rigorous compression. I was presuming it would become much harder to handle the NWB file, but would it still be acceptable?

[satra]

@gwappa - we may be able to help you choose appropriate lossless compression and chunking values. the files would be fine for us and tools to handle.

[bendichter]

@gwappa let me explain a bit about the situation you are seeing with the video.

For videos, we recommend storing behavioral videos externally (as you have done) and microscopy videos internally inside the NWB file. See our best practices here. We should be able to find a compression strategy that provides performance close to that of the TIFF files. @h-mayorquin, can you please help with this?

[gwappa]

Thanks @h-mayorquin , I think I managed to compress the data (though the compression ratio was no more than 1.5).

[satra]

@gwappa and @h-mayorquin - did either of you try blosc + zstd5 for compression instead of gzip?

[h-mayorquin]

I got a compression ratio of 2.32 blosc+zstd5 with clevel 5 for the same data as above. Is there any other configuration that you would like us to check or double check, @satra?

[satra]

@h-mayorquin - i think that's ok for now.

more generally it may be nice if we created a little function in nwb perhaps that did a quick evaluation of compression options for the user. @rly had done a comparative evaluation project in the past @bendichter is writing a dandi blog post about this, i think. it would be nice to have a utility for doing so with strategic options for different data types and modalities.

[h-mayorquin]

That would be great to have!

[gwappa]

This is to summarize what I have understood.

Thanks to @satra @kabilar and @h-mayorquin , I found out that we need to follow the standards (relatively strictly):

1. The wide-field calcium imaging data be as entries in the NWB file (according to the best practice): H5DataIO or neuroconv.tools.configure_and_write_nwbfile may be used. gzip compression at a level somewhere around 4–6 may be used.
2. File names and directory structure must follow DANDI standards: refer to the documentation of dandi organize (thanks @yarikoptic for pointing this out)
   1. the subject directory sub-<subject-id> be directly under the dandiset root.
   2. individual NWB files be directly under the corresponding subject directory.
      1. NWB file names in the format sub-<subject_id>[_key-<value>][_mod1+mod2+...].nwb
      2. external files be under the directory sub-<subject_id>[_key-<value>][_mod1+mod2+...] (i.e. the file base of the corresponding NWB file)
      3. external video data files in the format sub-<subject_id>[_key-<value>]_<modality>.<suffix>

• Keys: pick the most reasonable one from ses (session), tis (tissue), slice (slice), and cell (cell)
• Modalities: dandi organize can add them automatically (e.g. image, ophys, behavior), though they may be directly named as long as they are similar to the way they are categorized in the NWB.
• Suffix: only "normal" video files are assumed: accepts .mp4, .avi, .wmv, .mov, .flv, and .mkv.

Please let me know if there are any misunderstanding.

Thanks again for your cooperation!