Identify solutions for updating collections with a rolling window

[kyle-lesinger]

Summary

The Disasters near real-time (NRT) data is downloaded and processed on a rolling schedule which will require changes in the collection on either a 3-hour or 1-day basis (depending on the collection). New strategies will need to be identified to account for this type of data.

Potential solutions

1. Create a new DAG which can delete a collection
2. Create a new DAG which will delete items

@anayeaye

[anayeaye]

We have often discussed the need to formalize a dataset deprecation process. This could also be an opportunity to implement pipelines for item and collection level deprecation (at least metadata but ultimately we probably need a way to change data retention policies in S3).
cc: @smohiudd