GH sync won't work with v2 project files

[josephjclark]

• add projects: state to serialize to an old state file
• will this work with gh sync? I guess I need config too?
• alternatively, modify the old deploy to load from a new project file, if it exists

Acceptance criteria

• I have a v2 project locally set up (ie, I ran openfn project pull <uuid> from some lightning instance)
• I can run the GH sync pull command: openfn pull $OPENFN_PROJECT_ID --config-path $OPENFN_CONFIG_PATH --no-confirm and it will work
• I can run the GH sync deploy command: openfn deploy -c $OPENFN_CONFIG_PATH --no-confirm and it will work

We don't need to run a full github sync in production - we just need these two commands to work

Expected Issues

We have discovered a major constraint in this solution

If you have a v2 folder system using github sync, it'll work, but you can only have a single sync between a repo and lightning.

You cannot have multiple config files.

The reason for this is that we can only maintain one workflows/ folder. The whole concept of syncing your repo to an app breaks down because of how we use the file system.

So if you want to use github sync with the v2 folder structure, you can only sync one project.

There is a separate story for a v2 github sync action which will resolve this comprehnsively.

We could also explore a version of github sync which only pulls, deploys and commits the project.yaml file and ignores the workflows folder. Its just that I don't know what the point of this is - your working tree is always out of sync with the app, and you don't know the relationship between the project source and the connected app.

[josephjclark]

I see three ways of doing this:

1. Allow project v2 to eject to v1. But it's not just state - I also need to eject a config and project.yaml file
2. Update deploy to import the artefacts from v2 project files, if openfn.yaml is detected. This may not be too bad, but do I need to generate config and project files in memory?
3. Create a new v2 github sync action which simply uses deploy --beta instead of deploy. You could manually edit the github action file for this

[josephjclark]

Lining this up for Farhan to maybe take a look at tomorrow

[josephjclark]

A "fake" config.json file which has whatever is needed for v2 could be the solution

[midigofrank]

fyi, underneath, github sync are just these 2 github actions:

• https://github.com/openfn/cli-deploy-action
• https://github.com/openfn/cli-pull-action

We should consider updating them when cli v2 is stable and ready

[josephjclark]

Absolutely @midigofrank - well have a dedicated V2 action. But in the meantime we need a transition which doesn't break legacy sync setups

[josephjclark]

Taking a quick look at the existing actions with Farhan

All we need in a v2 deployment is the project uuid. We can work out the rest from there

So we should intercept openfn deploy and openfn pull. If we detect an openfn.yaml file in the working dir, we should REDIRECT to the v2 commands.

The pull command should actually just run a fetch to update the yaml file. It should not update the working directory.

The deploy command should take the working directly and deploy its contents, then update the local yaml file.

The Workspace class has the ability to load a project from a UUID

The downside of this strategy is that any openfn deploy will fail to use v1 logic if the user also has an openfn.yaml file in the same folder. I think this acceptable but might be a good idea to add a --legacy flag to disable this auto-lookup-v2 file. Also an env var actually, maybe OPENFN_FORCE_V1_DEPLOY

An integration test would be super nice