Stream data from EMBER dandiset

[alessandratrapani]

Hi!
Is there a way to stream data from EMBER similarly to what is done from the DANDI Archive?
I would like to adapt the following code to stream from an emberset:

# Import necessary modules for accessing DANDI archive
from dandi.dandiapi import DandiAPIClient

# Create a search pattern to find the specific NWB file
# Replace underscores with hyphens to match DANDI naming convention
pattern = (f"sub-{subject_id.replace("_", "-")}/sub-{subject_id.replace("_", "-")}_ses-{session_id.replace("_", "-")}*.nwb")

# Initialize DANDI API client to access the archive
with DandiAPIClient() as client:
    # Authenticate with DANDI (required for embargoed datasets)
    # This line can be removed once the dataset is publicly available
    client.dandi_authenticate()
    
    # Search for assets matching our pattern in the draft version of the dandiset
    assets = client.get_dandiset(dandiset_id, "draft").get_assets_by_glob(pattern=pattern, order="path")
    
    # Collect S3 URLs for all matching assets
    s3_urls = []
    for asset in assets:
        # Get the direct S3 URL for streaming the file
        s3_url = asset.get_content_url(follow_redirects=1, strip_query=False)
        s3_urls.append(s3_url)
    
    # Validate that we found exactly one matching file
    if len(s3_urls) > 1:
        raise ValueError(f"More than one asset found for pattern {pattern}")
    elif len(s3_urls) == 0:
        raise ValueError(f"No asset found for pattern {pattern}")
    else:
        # Use the single matching asset
        s3_url = s3_urls[0]
        # Get the final S3 URL for the asset
        s3_url = asset.get_content_url(follow_redirects=1, strip_query=False)

# Import modules for remote file access and NWB reading
import h5py
import remfile
from pynwb import NWBHDF5IO

# Create a remote file object that can stream data from S3
file = remfile.File(s3_url)

# Wrap the remote file in an HDF5 file object for reading
h5_file = h5py.File(file, "r")

# Create NWB IO object for reading the streamed file
io = NWBHDF5IO(file=h5_file, load_namespaces=True)

# Read the NWB file into memory for analysis
nwbfile = io.read()

[yarikoptic]

there should be! Just when you initialize your DandiAPIClient , specify to use EMBER instance, e.g. DandiAPIClient(dandi_instance="ember") or DandiAPIClient(dandi_instance="ember-sandbox") .

Disclaimer: I said "should" since didn't try myself ;) Please report back if works.

[alessandratrapani]

with:

from dandi.dandiapi import DandiAPIClient
with DandiAPIClient(dandi_instance="ember") as client:
...

I get this error:

      1 from dandi.dandiapi import DandiAPIClient, DandiInstance
----> 5 with DandiAPIClient(dandi_instance="ember") as client:
      6     # client.dandi_authenticate()
      7
      8     # Search for assets matching our pattern in the draft version of the dandisets
      9     assets = client.get_dandiset(dandiset_id, "draft").get_assets()

File ~\anaconda3\envs\kind_lab_to_nwb_env\Lib\site-packages\dandi\dandiapi.py:434, in DandiAPIClient.__init__(self, api_url, token, dandi_instance)
    432         instance_name = os.environ.get("DANDI_INSTANCE", "dandi")
    433         dandi_instance = get_instance(instance_name)
--> 434     api_url = dandi_instance.api
    435 elif dandi_instance is not None:
    436     raise ValueError("api_url and dandi_instance are mutually exclusive")

AttributeError: 'str' object has no attribute 'api'

But if I define the DandiInstance, it is working:

from dandi.dandiapi import DandiAPIClient, DandiInstance
dandi_instance = DandiInstance(
    "ember",
    "https://dandi.emberarchive.org",
    "https://api-dandi.emberarchive.org/api",
)
with DandiAPIClient(dandi_instance=dandi_instance) as client:
...

However, I could not find documentation for this specific operation.

[yarikoptic]

well, done! sorry - I have missed that it must be dandi_instance: DandiInstance | None = None, there. I should have told to use with DandiAPIClient.for_dandi_instance(instance="ember") as client -- this way it should have worked (I think again ;) didn't check). Indeed, adding to docs would be handy, attn @dandi/ember (seems can't alert other org team as the @aplbrain/BBQS_EMBER )