Problem using `data-intake-esgf.yml` configuration file

[ehogan]

Describe the bug
If I have an unmodified data-intake-esgf.yml in my ~/.config/esmvaltool/ directory (created using the command esmvaltool config copy data-intake-esgf.yml), ESMValTool v2.14.0 fails with:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='search.api.globus.org', port=443): Max retries exceeded with url: /v1/index/a8ef4320-9e5a-4793-837b-c45161ca1845/scroll (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1032)')))

If I navigate to the page https://search.api.globus.org/v1/index/a8ef4320-9e5a-4793-837b-c45161ca1845/scroll I get:

(So sad the bunny doesn't render in this view! 🐰)

If I remove the data-intake-esgf.yml from my ~/.config/esmvaltool/ directory, ESMValTool works fine.

Does this happen to anyone else? This issue has been reported at the MO multiple times now, so it's not a user-specific issue, but it may be a MO-specific issue 😕

[valeriupredoi]

The scroll functionality happens when search is not happening - it's a more limited functionality than search, are you trying to look for a LOT of data per chance?

ChatGPT mentions a few cases when search is allowed, but scroll isn't, by the host institution:

Yes — potentially, but /scroll itself isn't inherently unusual or forbidden.

A server/institution could block it for several reasons:

• HTTP method restrictions: /scroll requires POST. Some institutional proxies/firewalls are configured to permit GET/HEAD but restrict or inspect POST requests.
• Domain allowlisting: they may allow search.api.globus.org generally, or perhaps only specific Globus endpoints, and reject anything not explicitly allowed.
• Path-based filtering: /v1/index/.../scroll could be blocked even if ordinary Globus Search requests work.
• Request-body inspection: the scroll query is in the POST body, so a restrictive proxy/WAF may treat it differently from a normal search request.
• Authentication/token handling: Globus Search APIs require appropriate OAuth credentials. A proxy can also interfere with the Authorization header.
• Rate/volume controls: scroll is specifically intended for retrieving large numbers of records, so a network/security service could throttle or terminate long-running/high-volume sequences.

In your particular case, there's an important distinction:

If intake-esgf works against the Globus Search /search endpoint but fails when it reaches /scroll, that would be quite plausible as an institutional-network restriction.

You can test this very cleanly from the affected machine with:

curl -v -X POST \
  https://search.api.globus.org/v1/index/a8ef4320-9e5a-4793-837b-c45161ca1845/scroll \
  -H 'Content-Type: application/json' \
  -d '{"q":"*"}'

You don't necessarily need valid Globus authentication to distinguish network blocking from API authentication failure. For example:

• 403/401 from Globus → you reached Globus; likely an auth/permission issue.
• 405 Method Not Allowed → endpoint/method issue.
• Connection timeout/reset → much more suggestive of proxy/firewall interference.
• A proxy-generated block page → almost certainly institutional filtering.
• Normal Globus JSON error → the network is allowing it through.

Given your earlier comments about institutional blocking of conda.anaconda.org, I would absolutely consider this a possibility if this is happening from the University of Reading network.