AspireSync 3.0 design #63
Description
(Yes, our versions are kind of wacky, AS is currently at 2.8 while AC has yet to reach 1.0)
AspireSync originally started as some Symfony commands to batch-download metadata and zip downloads from .org and store them in S3. The download-to-s3 function has moved to AspireCloud, getting the updated slugs from subversion now happens in bash scripts, and metadata is now pushed to AC via HTTP, but batch-scraping metadata from the .org API is still AS's essential function. However, when AspireBuild hits the scene, it will take over asset downloads and metadata creation, essentially recreating .org's submission pipeline, making API scraping obsolete, and leaving AspireSync as currently constituted without a job.
Rather than put AS out to pasture, I'd like to reposition it as the glue between AB, AC, and likely other portions of the FAIR ecosystem and things we haven't thought of yet. To this end, it should act as a sort of message broker, such that when AB is done building a package, it pushes a notification to AS, which takes care of relaying any necessary info to AC. When a new package or update is discovered, AS would signal AB to build it. This discovery process would still be part of AS's job, leaving AB as essentially a pool of worker backends that AS invokes, and AC remaining unaware of where its packages ultimately come from.
The overall idea here is to say AspireSync 3.0 should be the orchestrator of AspirePress's processes: not executing any of them itself, but relaying messages to each component to relieve them of the burden of knowing which external systems they need to interface with. AS can collect the potentially dozens of repos, metadata services, labelers, and so on, and present a unified source and sink for the data that the components of the AspirePress and FAIR ecosystems depend on.
In short, AspireSync is getting sent to the glue factory. But glue is important!