Keep track of all STAC Items/S3 objects used in notebooks

[clausmichele]

Due to the deprecation of the data in the rolling archive, we need to map the data used in the notebooks so that it doesn't get erased.

[clausmichele]

@jzvolensky @Yuvraj198920 @suriyahgit

I'd like to hear if you have ideas on how to tackle this issue. We would need have a list/table/database with all the products (STAC Items) used in the notebooks. It's not straightforward, especially for the notebooks using xcube-eopf, since it's internally "consuming" the STAC Items

[suriyahgit]

Maintain a small markdown table in the repo, let's assume for example (docs/data_availability.md) where each row is a notebook, listing its STAC Items (or a proxy/query for xcube-eopf) and a status column with ✅/❌. A scheduled GitHub Action checks reachability of these items/queries and automatically updates the status (green if alive, red if stale).

[jzvolensky]

Maintain a small markdown table in the repo, let's assume for example (docs/data_availability.md) where each row is a notebook, listing its STAC Items (or a proxy/query for xcube-eopf) and a status column with ✅/❌. A scheduled GitHub Action checks reachability of these items/queries and automatically updates the status (green if alive, red if stale).

I imagine this "small" markdown will grow rapidly and unless we have an automated mechanism to updated it, it will be a nightmare to maintain haha. Especially for notebooks that use 100s of items

[jzvolensky]

Perhaps we could keep a list or a "database" of AOIs and satellite used (S1, S2 etc.) in the notebooks and then have a single script which queries for all and extracts the list of items into a database. Then whenever we make a new notebook you just add an AOI entry and mission type into a sort of configuration instead of having to extract and update items manually

This way we also dont need to care about whether you used xcube or not.

[clausmichele]

NDSI notebook to be fixed via this data request: EOPF-Sample-Service/eopf-sample-data#35

[clausmichele]

Wildfire notebook data request: EOPF-Sample-Service/eopf-sample-data#36