Fixed the Zhang species so NWB uploads pass DANDI validation

.agents/skills/lab-conversion-plan/references/standards.md

@@ -53,7 +53,7 @@ Load-bearing details, verified for Kemere. Re-verify them against the PR's curre
 
 ## DANDI requirements that apply either way
 
-- Subject metadata must include species (Latin binomial), sex as one of `M`/`F`/`O`/`U`, and age as an ISO 8601 duration (`P30Y`, `P5W3D`) or a date of birth.
+- Subject metadata must include species (Latin binomial, or an NCBI taxonomy link such as `http://purl.obolibrary.org/obo/NCBITaxon_9669` for a subspecies, since DANDI's validation rejects trinomials like `Mustela putorius furo`), sex as one of `M`/`F`/`O`/`U`, and age as an ISO 8601 duration (`P30Y`, `P5W3D`) or a date of birth.
 - Subject and session labels are alphanumeric. Sanitize anything derived from folder or file names (`sanitize_label` in `labs/inman/code/batch_convert.py`).
 - Associated papers become `related_publications` DOI links in NWB session metadata.
 - Dandiset ids are six digits. Incoming and standardized may be one shared dandiset or two separate ones. See the field reference in `dispatch/README.md`.

.agents/skills/lab-scaffold/references/layout-contract.md

@@ -44,7 +44,7 @@ A lab spreadsheet that carries per-session facts (a session log, per-animal flag
 For NWB labs, the single metadata source the converter consumes. `labs/inman/code/config.yaml` is the pattern:
 
 - A header comment pointing at the NWB GUIDE for field meanings.
-- `session` and `subject` blocks covering the DANDI-required fields (species, sex `M`/`F`/`O`/`U`, age as ISO 8601 duration or date of birth), plus start time with explicit format, experimenter style notes, and `related_publications` for the intake's DOIs.
+- `session` and `subject` blocks covering the DANDI-required fields (species as a Latin binomial or, for a subspecies, an NCBI taxonomy link, sex `M`/`F`/`O`/`U`, age as ISO 8601 duration or date of birth), plus start time with explicit format, experimenter style notes, and `related_publications` for the intake's DOIs.
 - Per-stream blocks (timeseries descriptions and units, devices, electrode groups, channel maps) matching the plan's metadata section.
 - Every unconfirmed value marked `PROVISIONAL` inline, with what would confirm it. A file-top comment says provisional values keep the conversion runnable and must be replaced with real lab metadata.
 

dispatch/README.md

@@ -7,7 +7,7 @@ It is repo-level infra, not a lab — it doesn't do any conversion itself, it ju
 
 1. `dandi download` the project's incoming dandiset (from the `ember-dandi` instance, the only archive this pipeline uses) into `<ember-incoming>/<incoming_dandiset_id>/`. Runs inside `--dandi-image` (through Apptainer), not directly on the runner host.
 2. Discover its sessions (per `sessions.json`'s spec for the project) and diff them against the project's manifest (`<ember-standardized>/<standardized_dandiset_id>/.ingest_state.json`) to find sessions with no conversion recorded yet.
-3. If there are new sessions, **or** the conversion script's contents have changed since the manifest was last written (sha256, so any edit forces a full reprocess via `overwrite_flag`), run the lab's conversion command.
+3. If there are new sessions, **or** the conversion's code or config has changed since the manifest was last written, run the lab's conversion command. That covers the conversion script and every repository file `convert_command` names as `{repo_root}/...`, such as a batch driver or a config (sha256, so any edit forces a full reprocess via `overwrite_flag`). A project whose command names no other file is hashed by its script alone.
    If the project names a `container_image`, this step runs inside it (through Apptainer) instead of directly on the runner host — the image holds only the lab's runtime environment (e.g. FFmpeg for Kemere), not the code or data, which are bind-mounted in at run time from the same host paths. Otherwise it runs directly on the runner host, which must then already have whatever the conversion script needs installed.
    A command that converts many sessions can report the ones it couldn't through the `{results}` placeholder. That is a JSON file of `{"pending": {session_id: reason}, "failed": {session_id: reason}}`, keyed by the same session ids `sessions.json` discovers. When the command exits nonzero but lists its unconverted sessions there, the rest are recorded as converted and uploaded as usual, and only the listed ones are retried next run. `pending` sessions are waiting on inputs not uploaded yet: they are logged as a warning and the run still passes. `failed` sessions fail the project once the upload is done. Without a results file, a nonzero exit fails the whole project, as before.
 4. `dandi upload` the standardized directory, also inside `--dandi-image` — first fetching just that dandiset's `dandiset.yaml` (not a full download), since `dandi upload` needs one already on disk to know which dandiset it's uploading to, and one only lands there on its own when `standardized_dandiset_id` happens to equal `incoming_dandiset_id`. This step is skipped entirely when step 3 had nothing to do — upload's no-op check still re-checksums the whole local dandiset every pass (`DANDI_CACHE=ignore` disables the digest cache), a cost that grows with the dandiset. The tradeoff: if a run converts sessions but dies before its upload finishes, the next pass will not retry that upload on its own (the manifest already records the sessions). Recover by re-uploading manually, or by touching the conversion script so the hash change forces a reprocess + upload.

dispatch/dispatch.py

@@ -56,6 +56,7 @@
 from __future__ import annotations
 
 import argparse
+import hashlib
 import json
 import logging
 import os
@@ -327,6 +328,36 @@ class ConversionIncomplete(RuntimeError):
     """Some sessions failed to convert. The rest were still recorded and uploaded."""
 
 
+def conversion_inputs(project: Project, /, *, repo_root: Path) -> list[Path]:
+    """The conversion script plus every repository file its convert_command
+    names, such as a batch driver or a config. A change to any of them
+    changes what the conversion writes."""
+    named = []
+    for token in project.convert_command:
+        if not token.startswith("{repo_root}/"):
+            continue
+        path = Path(token.replace("{repo_root}", str(repo_root), 1))
+        if path.is_file():
+            named.append(path)
+    script = project.script_abspath(repo_root)
+    inputs = [script] + sorted({path for path in named if path != script})
+    return inputs
+
+
+def conversion_hash(project: Project, /, *, repo_root: Path) -> str:
+    """sha256 identifying the conversion's code and config. With only the
+    script to hash it is the script's own hash, so projects whose command
+    names no other file keep the hash their manifests already record."""
+    inputs = conversion_inputs(project, repo_root=repo_root)
+    if len(inputs) == 1:
+        return hash_file(inputs[0])
+    digest = hashlib.sha256()
+    for path in inputs:
+        digest.update(f"{path.relative_to(repo_root)}\0{hash_file(path)}\n".encode())
+    combined = digest.hexdigest()
+    return combined
+
+
 def process_project(
     project: Project,
     *,
@@ -355,7 +386,7 @@ def process_project(
     state = IngestState.load(standardized_dir, manifest_name=manifest_name)
 
     script_path = project.script_abspath(repo_root)
-    current_script_hash = hash_file(script_path) if script_path.is_file() else None
+    current_script_hash = conversion_hash(project, repo_root=repo_root) if script_path.is_file() else None
     if current_script_hash is None:
         log.warning("[%s] conversion script not found at %s, cannot hash it", project.key, script_path)
     script_changed = current_script_hash is not None and current_script_hash != state.script_sha256
@@ -376,7 +407,7 @@ def process_project(
 
     if script_changed:
         log.info(
-            "[%s] conversion script changed (%s -> %s); reprocessing all %d discovered session(s)",
+            "[%s] conversion script or config changed (%s -> %s); reprocessing all %d discovered session(s)",
             project.key,
             state.script_sha256,
             current_script_hash,

dispatch/tests/test_dispatch.py

@@ -732,3 +732,35 @@ def test_upload_is_skipped_when_every_session_is_still_pending(tmp_path, monkeyp
 
     assert (["upload"], {}) not in calls
     assert IngestState.load(standardized_dir).converted_sessions == {}
+
+
+def test_conversion_hash_is_the_script_hash_when_the_command_names_nothing_else(tmp_path):
+    repo_root = make_repo(tmp_path)
+    project = make_project()
+
+    combined = dispatch.conversion_hash(project, repo_root=repo_root)
+
+    assert combined == dispatch.hash_file(project.script_abspath(repo_root))
+
+
+def test_conversion_hash_covers_files_the_command_names(tmp_path):
+    repo_root = make_repo(tmp_path)
+    config = repo_root / "labs" / "test-lab" / "code" / "config.yaml"
+    config.write_text("species: a\n")
+    project = make_project(
+        convert_command=[
+            "python3",
+            "{repo_root}/labs/test-lab/code/convert.py",
+            "--config",
+            "{repo_root}/labs/test-lab/code/config.yaml",
+            "{repo_root}/labs/test-lab/code/not-a-file.yaml",
+        ]
+    )
+
+    before = dispatch.conversion_hash(project, repo_root=repo_root)
+    config.write_text("species: b\n")
+    after = dispatch.conversion_hash(project, repo_root=repo_root)
+
+    assert dispatch.conversion_inputs(project, repo_root=repo_root) == [project.script_abspath(repo_root), config]
+    assert before != after
+    assert before != dispatch.hash_file(project.script_abspath(repo_root))

labs/zhang/ferret-hyperscanning/code/config.yaml

@@ -33,7 +33,7 @@ session:
 
 subjects:
   defaults:
-    species: Mustela putorius furo
+    species: http://purl.obolibrary.org/obo/NCBITaxon_9669  # Mustela putorius furo (domestic ferret)
     sex: U        # PROVISIONAL (Q9) ("unknown"). Required by DANDI, should be one of "M", "F", "O", "U"
     age: P1Y      # PROVISIONAL (Q9). Required by DANDI, should be an ISO 8601 duration, or use date_of_birth instead
   "0235":

labs/zhang/ferret-hyperscanning/prompts/initial.md

@@ -524,3 +524,15 @@ now writes `--results`, splitting unconverted recordings into `pending`
 (a missing merged file, raised as `MissingInputError`) and `failed`
 (anything else). Dispatch records and uploads the rest, retries only those
 next run, and reports `pending` ones as a warning instead of a failure.
+
+## Request 12 — Uploads failing validation
+
+> cool ty keep watching the runs for issues
+
+The first run that kept its converted recordings uploaded every video but
+none of the 120 NWB files. DANDI rejected each one because the species,
+`Mustela putorius furo`, is a trinomial, and NWB Inspector accepts only a
+Latin binomial or an NCBI taxonomy link. `code/config.yaml` now gives the
+taxonomy link, `http://purl.obolibrary.org/obo/NCBITaxon_9669`. Dispatch
+now also reconverts when a file `convert_command` names changes, such as
+this config, so the fix reaches the recordings already converted.

labs/zhang/ferret-hyperscanning/tests/expected_output/summary.json

@@ -229,7 +229,7 @@
         "age": "P1Y",
         "description": "Ferret 0236, no shave",
         "sex": "U",
-        "species": "Mustela putorius furo",
+        "species": "http://purl.obolibrary.org/obo/NCBITaxon_9669",
         "subject_id": "0236"
       },
       "videos": {
@@ -540,7 +540,7 @@
         "age": "P1Y",
         "description": "Ferret 0237, identified by a shaved upper back",
         "sex": "U",
-        "species": "Mustela putorius furo",
+        "species": "http://purl.obolibrary.org/obo/NCBITaxon_9669",
         "subject_id": "0237"
       },
       "videos": {