Le WS DoiEnrich (InspireHep) renvoie des n/a et ne va pas au bout du traitement

[CarineBach]

Le ws DoiEnrich qui interroge la base InspireHep (in2p3) à partir de doi renvoie des n/a et ne va pas au bout du traitement (Lodex v16.10.11 et v16.11.2)

Url du ws : https://biblio-tools.services.istex.fr/v2/inspirehep/works/expand

198 n/a sur 216 lignes traitées dans Lodex (sur un corpus de 1137 lignes)

Problématique 1 : renvoie n/a dans la colonne d'enrichissement de Lodex même quand on retrouve un résultat dans le swagger
Exemple avec le doi : 10.1088/1742-6596/2687/2/022004

ok dans le swagger :

n/a dans la colonne d'enrichissement de Lodex:

Même chose pour les doi suivants qui affichent une réf bib dans le swagger, alors que l'enrichissement dans Lodex affiche un n/a :
10.1016/j.nimb.2024.165553
10.1103/g393-xx1w
10.1103/physrevc.111.054614

Problématique 2 : le ws n'est pas allé au bout du traitement du corpus entier. Même problème Lodex/Mongo que déjà signalé pour d'autres ws ?
Message d'erreur dans les log de l'enrichissement :
ganil-openalex-v2] Enrichement errored : item #1 [delegate] <Error: [LodexRunQuery] <MongoServerError: cursor id 806230549652139790 not found

[touv]

à re-tester avec la version 16.12.5 de Lodex, celle-ci supprime le message MongoServerError
pour éviter les réponses de type n/a, à tester en modifiant les paramètres de connexion au web service, notamment en passant noerror de true à false, Exemple :

[expand/URLConnect]
url = https://openalex-doctype.services.istex.fr/v1/isresearch-id?sid=lodex
timeout = 30000
noerror = false
retries = 0 

[parmentf]

La documentation de l'API Inspire/HEP indique qu'il y a un rate limiting que le web service ne respecte pas:

In order to avoid overwhelming the server, we enforce rate limits per IP address: every IP address is allowed 15 requests in a 5s window. If you exceed those limits, you will receive a response with HTTP status code 429. Please note that requests that are blocked due to exceeding the rate limit count towards the quota, so you'll need to wait at least 5s when receiving a 429 response before trying again.

Je vais essayer d'ajouter un throttle. En moyenne, 15 requêtes pour 5 secondes font 3 requêtes maximum par seconde. Pour être sûr, je vais la mettre à 2 requêtes par seconde.

[parmentf]

Le comportement est différent, mais il y a encore des erreurs, je rouvre l'issue.

[parmentf]

Ce genre d'erreur se multiplie:

07/07/2026 16:15:11] [Instance: ganil-test-inspire] The dollar ($) prefixed field '$schema' in 'WsInspire.metadata.$schema' is not valid for storage.
[07/07/2026 16:15:11] [Instance: ganil-test-inspire] The DBRef $ref field must be followed by a $id field

Un exemple de sortie (il y a des champs $ref et $schema):

[
    {
        "id": 1,
        "value": {
            "revision_id": 96,
            "links": {
                "bibtex": "https://inspirehep.net/api/doi/10.1103/physrevlett.19.1264?format=bibtex",
                "latex-eu": "https://inspirehep.net/api/doi/10.1103/physrevlett.19.1264?format=latex-eu",
                "latex-us": "https://inspirehep.net/api/doi/10.1103/physrevlett.19.1264?format=latex-us",
                "json": "https://inspirehep.net/api/doi/10.1103/physrevlett.19.1264?format=json",
                "json-expanded": "https://inspirehep.net/api/doi/10.1103/physrevlett.19.1264?format=json-expanded",
                "cv": "https://inspirehep.net/api/doi/10.1103/physrevlett.19.1264?format=cv"
            },
            "uuid": "f0318082-80cc-46bc-bb9a-ed8108097a46",
            "id": "10.1103/physrevlett.19.1264",
            "metadata": {
                "citation_count": 15450,
                "publication_info": [
                    {
                        "year": 1967,
                        "page_end": "1266",
                        "page_start": "1264",
                        "journal_title": "Phys.Rev.Lett.",
                        "journal_record": {
                            "$ref": "https://inspirehep.net/api/journals/1214495"
                        },
                        "journal_volume": "19"
                    },
                    {
                        "pubinfo_freetext": "Also in *Lichtenberg, D. B. (ed.), Rosen, S. P. (ed.): Developments In The Quark Theory Of Hadrons, Vol. 1*, 157-159"
                    },
                    {
                        "pubinfo_freetext": "In *Lai, C. H. (ed.): Gauge Theory Of Weak and Electromagnetic Interactions*, 185-187"
                    },
                    {
                        "pubinfo_freetext": "In *Rosner, J. L. (ed.): New Particles*, 26-28"
                    }
                ],
                "citation_count_without_self_citations": 15422,
                "core": true,
                "dois": [
                    {
                        "value": "10.1103/PhysRevLett.19.1264"
                    }
                ],
                "urls": [
                    {
                        "value": "http://cerncourier.com/cws/article/cern/70138",
                        "description": "CERN Courier article"
                    }
                ],
                "titles": [
                    {
                        "title": "A Model of Leptons"
                    }
                ],
                "$schema": "https://inspirehep.net/schemas/records/hep.json",
                "authors": [
                    {
                        "ids": [
                            {
                                "value": "INSPIRE-00135339",
                                "schema": "INSPIRE ID"
                            }
                        ],
                        "uuid": "5863d7d5-7db1-4d1c-af0c-6408b3584b3e",
                        "record": {
                            "$ref": "https://inspirehep.net/api/authors/983868"
                        },
                        "full_name": "Weinberg, Steven",
                        "affiliations": [
                            {
                                "value": "MIT, LNS",
                                "record": {
                                    "$ref": "https://inspirehep.net/api/institutions/903016"
                                }
                            }
                        ],
                        "signature_block": "WANBARGs",
                        "curated_relation": true
                    }
                ],
                "curated": true,
                "texkeys": [
                    "Weinberg:1967tq"
                ],
                "citeable": true,
                "keywords": [
                    {
                        "value": "LEPTON: MODEL",
                        "schema": "INSPIRE"
                    },
                    {
                        "value": "GAUGE FIELD THEORY",
                        "schema": "INSPIRE"
                    },
                    {
                        "value": "SPONTANEOUS SYMMETRY BREAKING",
                        "schema": "INSPIRE"
                    },
                    {
                        "value": "FIELD THEORY: GOLDSTONE THEOREM",
                        "schema": "INSPIRE"
                    },
                    {
                        "value": "ELECTROWEAK INTERACTION",
                        "schema": "INSPIRE"
                    },
                    {
                        "value": "RENORMALIZATION",
                        "schema": "INSPIRE"
                    },
                    {
                        "value": "gauge boson: mass",
                        "schema": "INSPIRE"
                    }
                ],
                "refereed": true,
                "references": [
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/3203"
                        },
                        "reference": {
                            "misc": [
                                "The history of attempts to unify weak and electro-magnetic interactions is very long, and will not be reviewed here. Possibly the earliest reference is"
                            ],
                            "label": "1",
                            "authors": [
                                {
                                    "full_name": "Fermi, E."
                                }
                            ],
                            "publication_info": {
                                "year": 1934,
                                "artid": "161",
                                "page_start": "161",
                                "journal_title": "Z.Phys.",
                                "journal_volume": "88"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/4328"
                        },
                        "reference": {
                            "misc": [
                                "A model similar to ours was discussed by"
                            ],
                            "label": "1",
                            "authors": [
                                {
                                    "full_name": "Glashow, S."
                                }
                            ],
                            "publication_info": {
                                "year": 1961,
                                "artid": "579",
                                "page_start": "579",
                                "journal_title": "Nucl.Phys.",
                                "journal_volume": "22"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "reference": {
                            "misc": [
                                "the chief difference is that Glashow introduces symmetry-breaking terms into the Lagrangian, and therefore gets less definite predictions"
                            ],
                            "label": "1"
                        }
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/12289"
                        },
                        "reference": {
                            "label": "2",
                            "authors": [
                                {
                                    "full_name": "Goldstone, J."
                                }
                            ],
                            "publication_info": {
                                "year": 1961,
                                "artid": "154",
                                "page_start": "154",
                                "journal_title": "Nuovo Cim.",
                                "journal_volume": "19"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/12290"
                        },
                        "reference": {
                            "label": "2",
                            "authors": [
                                {
                                    "full_name": "Goldstone, J."
                                },
                                {
                                    "full_name": "Salam, A."
                                },
                                {
                                    "full_name": "Weinberg, S."
                                }
                            ],
                            "publication_info": {
                                "year": 1962,
                                "artid": "965",
                                "page_start": "965",
                                "journal_title": "Phys.Rev.",
                                "journal_volume": "127"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/40440"
                        },
                        "reference": {
                            "label": "3",
                            "authors": [
                                {
                                    "full_name": "Higgs, P.W."
                                }
                            ],
                            "publication_info": {
                                "year": 1964,
                                "artid": "132",
                                "page_start": "132",
                                "journal_title": "Phys.Lett.",
                                "journal_volume": "12"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/11883"
                        },
                        "reference": {
                            "label": "3",
                            "publication_info": {
                                "year": 1964,
                                "artid": "508",
                                "page_start": "508",
                                "journal_title": "Phys.Rev.Lett.",
                                "journal_volume": "13"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/50073"
                        },
                        "reference": {
                            "misc": [
                                "and"
                            ],
                            "label": "3",
                            "publication_info": {
                                "year": 1966,
                                "artid": "1156",
                                "page_start": "1156",
                                "journal_title": "Phys.Rev.",
                                "journal_volume": "145"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/12291"
                        },
                        "reference": {
                            "label": "3",
                            "authors": [
                                {
                                    "full_name": "F.Englert"
                                },
                                {
                                    "full_name": "Brout, R."
                                }
                            ],
                            "publication_info": {
                                "year": 1964,
                                "artid": "321",
                                "page_start": "321",
                                "journal_title": "Phys.Rev.Lett.",
                                "journal_volume": "13"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/12292"
                        },
                        "reference": {
                            "label": "3",
                            "authors": [
                                {
                                    "full_name": "Guralnik, G.S."
                                },
                                {
                                    "full_name": "Hagen, C.R."
                                },
                                {
                                    "full_name": "Kibble, T.W.B."
                                }
                            ],
                            "publication_info": {
                                "year": 1964,
                                "artid": "585",
                                "page_start": "585",
                                "journal_title": "Phys.Rev.Lett.",
                                "journal_volume": "13"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/51165"
                        },
                        "reference": {
                            "misc": [
                                "See particularly A similar phenomenon occurs in the strong interactions"
                            ],
                            "label": "4",
                            "authors": [
                                {
                                    "full_name": "Kibble, T.W.B."
                                }
                            ],
                            "publication_info": {
                                "year": 1967,
                                "artid": "1554",
                                "page_start": "1554",
                                "journal_title": "Phys.Rev.",
                                "journal_volume": "155"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/51368"
                        },
                        "reference": {
                            "misc": [
                                "the p-meson mass in zeroth-order perturbation theory is just the bare mass, while the A1 meson picks up an extra contribution from the spontaneous breaking of chiral symmetry especially footnote 7"
                            ],
                            "label": "4",
                            "authors": [
                                {
                                    "full_name": "Weinberg, See S."
                                }
                            ],
                            "publication_info": {
                                "year": 1967,
                                "artid": "507",
                                "page_start": "507",
                                "journal_title": "Phys.Rev.Lett.",
                                "journal_volume": "18"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/51162"
                        },
                        "reference": {
                            "label": "4",
                            "authors": [
                                {
                                    "full_name": "Schwinger, J."
                                }
                            ],
                            "publication_info": {
                                "year": 1967,
                                "artid": "473",
                                "page_start": "473",
                                "journal_title": "Phys.Lett.B",
                                "journal_volume": "24"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/51507"
                        },
                        "reference": {
                            "misc": [
                                "Eq. (13) et seq"
                            ],
                            "label": "4",
                            "authors": [
                                {
                                    "full_name": "Glashow, S."
                                },
                                {
                                    "full_name": "Schnitzer, H."
                                },
                                {
                                    "full_name": "Weinberg, S."
                                }
                            ],
                            "publication_info": {
                                "year": 1967,
                                "artid": "139",
                                "page_start": "139",
                                "journal_title": "Phys.Rev.Lett.",
                                "journal_volume": "19"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/43797"
                        },
                        "reference": {
                            "label": "5",
                            "authors": [
                                {
                                    "full_name": "Lee, T.D."
                                },
                                {
                                    "full_name": "Yang, C.N."
                                }
                            ],
                            "publication_info": {
                                "year": 1955,
                                "artid": "101",
                                "page_start": "101",
                                "journal_title": "Phys.Rev.",
                                "journal_volume": "98"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "reference": {
                            "misc": [
                                "This is the same sort of transformation as that which eliminates the nonderivative 7t couplings in the a model"
                            ],
                            "label": "6"
                        }
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/50057"
                        },
                        "reference": {
                            "misc": [
                                "see The"
                            ],
                            "label": "6",
                            "authors": [
                                {
                                    "full_name": "Weinberg, S."
                                }
                            ],
                            "publication_info": {
                                "year": 1967,
                                "artid": "188",
                                "page_start": "188",
                                "journal_title": "Phys.Rev.Lett.",
                                "journal_volume": "18"
                            }
                        },
                        "curated_relation": false
                    },
                    {
                        "record": {
                            "$ref": "https://inspirehep.net/api/literature/1115221"
                        },
                        "reference": {
                            "misc": [
                                "7t reappears with derivative coupling because the strong-interaction Lagrangian is not invariant under chiral gauge transformation. 7 For a similar argument applied to the sigma meson, see Weinberg, Ref.6"
                            ],
                            "authors": [
                                {
                                    "full_name": "Feynman, R.P."
                                },
                                {
                                    "full_name": "Gell-Mann, M."
                                }
                            ],
                            "publication_info": {
                                "year": 1957,
                                "artid": "193",
                                "page_start": "193",
                                "journal_title": "Phys.Rev.",
                                "journal_volume": "109"
                            }
                        },
                        "curated_relation": false
                    }
                ],
                "document_type": [
                    "article"
                ],
                "preprint_date": "1967-11",
                "control_number": 51188,
                "legacy_version": "20180204042908.0",
                "number_of_pages": 3,
                "inspire_categories": [
                    {
                        "term": "Theory-HEP",
                        "source": "curator"
                    }
                ],
                "legacy_creation_date": "1981-01-01",
                "external_system_identifiers": [
                    {
                        "value": "4497676",
                        "schema": "OSTI"
                    },
                    {
                        "value": "1967PhRvL..19.1264W",
                        "schema": "ADS"
                    },
                    {
                        "value": "SPIRES-153990",
                        "schema": "SPIRES"
                    }
                ]
            },
            "created": "1981-01-01T00:00:00+00:00",
            "updated": "2023-03-09T15:45:47.191006+00:00"
        }
    }
]

Évidemment, mongodb (dans LODEX) est sensible à la présence du dollar dans les noms des champs.
C'est plus un problème Lodex que du web service, mais on va les supprimer pour parer au plus vite au problème.