Add script to serialize WorkflowHub metadata and create an RDF/Turtle file of all entries

[rioualen]

What this PR does

• Add script to serialize WorkflowHub entries and create a dump
• Remove previous, obsolete script (merged with the new script)
• Change import script to get EDAM URIs instead of labels

Follows PR #52, addresses issue #1547, and partly addresses issue #851.

New script

utils/bioschemas-gen/workflowhub_to_bioschemas.py

Deleted script

utils/bioschemas-gen/workflowhub_bioschemas_dump.py

Updated script

utils/workflowhub-import/workflowhub-import.py

What the new script does

• Fetch metadata from content/imports/workflowhub/{id}.workflowhub.json files
• Serialize metadata and produce an RDF/Turtle dump file content/datasets/workflowhub-dump.ttl

Using the same example as in PR #52, here is what content/imports/workflowhub/1104.workflowhub.json looks like:

{
    "content_blobs": [],
    "create_time": "2024-11-04",
    "creators": [
        "Diego De Panis",
        "ERGA"
    ],
    "description": "**Assembly Evaluation for ERGA-BGE Reports**\r\n\r\n_One Assembly, HiFi WGS reads + HiC reads_\r\n\r\nThe workflow requires the following:\r\n* Species Taxonomy ID number\r\n* NCBI Genome assembly accession code\r\n* BUSCO Lineage\r\n* WGS accurate reads accession code\r\n* NCBI HiC reads accession code\r\n\r\nThe workflow will get the data and process it to generate genome profiling (genomescope, smudgeplot -optional-), assembly stats (gfastats), merqury stats (QV, completeness), BUSCO, snailplot, contamination blobplot, and HiC heatmap.\r\n\r\n**Use this workflow for HiFi- or ONT-based assemblies**",
    "discussion_links": [
        {
            "id": "1133",
            "label": "",
            "url": "https://github.com/ERGA-consortium/pipelines/discussions"
        }
    ],
    "doi": "",
    "edam_operation": [
        "http://edamontology.org/operation_0525"
    ],
    "edam_topic": [
        "http://edamontology.org/topic_0622"
    ],
    "id": "1104",
    "latest_version": "2",
    "license": "MIT",
    "link": "https://workflowhub.eu/workflows/1104?version=2",
    "mapped_tools": [
        "bedtools",
        "blobtoolkit",
        "busco",
        "diamond",
        "genomescope_2.0",
        "gfastats",
        "hicexplorer",
        "merqury",
        "meryl",
        "minimap2",
        "ncbi_datasets",
        "rseqc",
        "smudgeplots",
        "sra-tools"
    ],
    "name": "Galaxy Workflow ERGA-BGE Genome Report ASM analyses (one-asm Long Reads + HiC)",
    "number_of_steps": 48,
    "projects": [],
    "source": "WorkflowHub",
    "tags": [
        "genome assembly",
        "genomics",
        "qc"
    ],
    "tools": [
        "__extract_dataset__",
        "__flatten__",
        "__unzip_collection__",
        "__zip_collection__",
        "bedtools",
        "bedtools_makewindowsbed",
        "bg_diamond",
        "blobtoolkit",
        "busco",
        "chromap",
        "collapse_dataset",
        "converter_fasta_to_fai",
        "cooler_cload_tabix",
        "cooler_csort_tabix",
        "cut1",
        "datasets_download_genome",
        "diamond",
        "fasterq_dump",
        "genomescope",
        "genomescope 2.0",
        "genomescope_2.0",
        "gfastats",
        "hicexplorer",
        "hicexplorer_hicmergematrixbins",
        "hicexplorer_hicplotmatrix",
        "lftp",
        "merqury",
        "meryl",
        "minimap2",
        "ncbi datasets",
        "ncbi_datasets",
        "pick_value",
        "rseqc",
        "sambamba_flagstat",
        "sambamba_merge",
        "smudgeplot",
        "smudgeplots",
        "sra software toolkit",
        "sra-tools",
        "tp_text_file_with_recurring_lines"
    ],
    "update_time": "2026-06-15",
    "versions": "2",
    "workflow_class": "Galaxy"
}

Which results in this dump entry:

@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix edam: <http://edamontology.org/> .
@prefix schema1: <http://schema.org/> .

<https://workflowhub.eu/workflows/1104?version=2> a schema1:ComputationalWorkflow ;
    dcterms:conformsTo "https://bioschemas.org/profiles/ComputationalWorkflow/1.0-RELEASE" ;
    schema1:applicationSubCategory edam:topic_0622 ;
    schema1:citation "" ;
    schema1:creator "Diego De Panis",
        "ERGA" ;
    schema1:dateCreated "2024-11-04" ;
    schema1:dateModified "2026-06-15" ;
    schema1:description """**Assembly Evaluation for ERGA-BGE Reports**\r
\r
_One Assembly, HiFi WGS reads + HiC reads_\r
\r
The workflow requires the following:\r
* Species Taxonomy ID number\r
* NCBI Genome assembly accession code\r
* BUSCO Lineage\r
* WGS accurate reads accession code\r
* NCBI HiC reads accession code\r
\r
The workflow will get the data and process it to generate genome profiling (genomescope, smudgeplot -optional-), assembly stats (gfastats), merqury stats (QV, completeness), BUSCO, snailplot, contamination blobplot, and HiC heatmap.\r
\r
**Use this workflow for HiFi- or ONT-based assemblies**""" ;
    schema1:featureList edam:operation_0525,
        edam:operation_3440 ;
    schema1:hasPart "bedtools",
        "blobtoolkit",
        "busco",
        "diamond",
        "genomescope_2.0",
        "gfastats",
        "hicexplorer",
        "merqury",
        "meryl",
        "minimap2",
        "ncbi_datasets",
        "rseqc",
        "smudgeplots",
        "sra-tools" ;
    schema1:identifier "1104" ;
    schema1:keywords "genome assembly",
        "genomics",
        "qc" ;
    schema1:license "MIT" ;
    schema1:name "Galaxy Workflow ERGA-BGE Genome Report ASM analyses (one-asm Long Reads + HiC)" ;
    schema1:url <https://workflowhub.eu/workflows/1104?version=2> ;
    schema1:version "2" .

A few notes:

• Missing Bioschemas “minimum” attributes from the ComputationalWorkflow profile:
  • schema:programmingLanguage
  • schema:sdPublisher, in Bioschemas “The host site for the ComputationalWorkflow”
    • Should this be “source”: “WorkflowHub” or “workflow_class”: “Galaxy”?
• This doesn't retrieve all of the metadata that we can extract with FAIR-Checker from individual workflow pages (e.g. programmingLanguage, sdPublisher, input, output, among others)
• Attributes from the json file that are not serialised: content_blobs, discussion_links, number_of_steps, projects, tools, versions.
• creator format is sometimes inconsistent, for example #10 and #16 (the whole list of creators in a unique string), and some entries are not actually creators for example 1741, 1740, 1739, and 1738 (a source of funding which could be a schema:funding attribute).
• There are a few exceptions due to special characters the name field (workflows 1741, 1740, 1739, 1738)
• EDAM terms are labels instead of URIs

One last question

• Right now we generate a jsonld and a ttl file for each workflow and a single ttl file for the whole dump. Should we keep the jsonld and/or a ttl files for workflows or just the final dump?

[hmenager]

As discussed with @rioualen , it would be better to only dump bioschemas/schema.org files as full dumps in the datasets folders.

[rioualen]

As discussed with @rioualen , it would be better to only dump bioschemas/schema.org files as full dumps in the datasets folders.

Made the changes as discussed and updated my original message accordingly, waiting for reviews 🙏🏻