New task control features: workdir cleanup, input cleanup, output cleanup

[prioux]

When launching a task, we have at the top a 'control' panel where the user can select which data provider to send the results to, what group the task is in, a short description etc.

I'm suggesting we could add three more selection boxes for the user, for actions to be done at the end of a task:

1. an option to tell CBRAIN to erase the inputs from the cache from the local cache (dangerous, explain why)
2. an option to tell CBRAIN to erase the outputs from the cache (also dangerous, explain why)
3. an option to tell CBRAIN to erase (or archive) the work directory of the task (also also dangerous)

Users must be told about the consequences of turning on any of these features.

Each of these could be tied to particular states, e.g. only when the task is Completed or Failed etc. All the possibilities would be too much to enumerate (and probably not all that useful) so we should think of identifying only the most common cases.

Implementation of these would be performed by adding new attributes to the CBRAIN task model and having the TaskWorker (which is the process taking care of a task's life cycle) create appropriate BackgroundActivity objects to implement the operations.

I think a default behavior could be to erase the task workdir when a task is completed.

See also #111 for a description of the workdir cleanup ideas, which was already in the planning stage many years ago.

[prioux]

Just a few notes.

1. We have two Boutiques modules that provide some of that functionality already, but they work only because the admins configure the behavior, it's not up to the user to decide. The modules are BoutiquesInputCacheCleaner and BoutiquesOutputCacheCleaner

2. If users elect to erase some their input files, there should be some code to check the types of the inputs and never erase small text files (e.g. licenses, json option files, etc) because these are often shared with other tasks. In fact, maybe at the time of making the decision, we can see if the inputs are also in use by other tasks. The BoutiquesInputCacheCleaner module mentioned above has some logic using timestamps. Reproducing this as part of the framework could make the framework itself a bit messy.

[AviJxn]

Hi @prioux, I am interested in working on this issue since no one is currently assigned. I completely understand the safety concerns you mentioned, especially regarding accidentally 'double deleting' shared assets or removing important config files. I will be really careful to avoid anything that could lead to unintended data loss.

My plan is to start small with the 'Workdir Cleanup' logic from #111 first to get the flow right then move into the cache cleanup once that's solid. I will start by digging into the 'CbrainTask' model and the 'TaskWorker' to see where we can safely trigger these 'BackgroundActivity' objects during the Completed or Failed states.

Does that sound like a reasonable starting point?

[prioux]

Sure, but before you write any code, consider making a short bullet point list of which files you think need to be changed and in what way (just a summary) so I can tell you if you're heading the right way.

For the Background activity side, it's just creating an object and saving it in the DB, the rest of the BAC framework will take care performing all the work. E.g. to clean up a work directory for a task, the new code would do:

bac = BackgroundActivity::RemoveTaskWorkdir.new(
  :user_id            => task.userid,
  :remote_resource_id => task.bourreau_id,
  :status             => 'InProgress',
  :items              => [ task.id ],
)
bac.save

[prioux]

I'd like to add something. The life cycle of tasks is handled by the code in the BourreauWorker class (in Bourreau/app/models/bourreau_worker.rb, one of the only files in that folder that is not a symbolic link to the models in the portal side). The main loop in there is really, really horribly monolithic, and so since this project requires modifying it, it would be a good idea to NOT insert a bunch of statement in the already really long methods in there, but to package the functionality in small distinct methods that would be called at the appropriate places. Eventually, we should break the big methods into a set of smaller ones, but I don't have time for such refactoring.

[AviJxn]

Thank you for the detailed feedback. I have spent some time exploring the repository in VS code to map out exactly where these changes should live and here is the breakdown of files I plan to modify and logic flow.

Database & Schema

• file BrainPortal/db/migrate/xx_add_cleanup_options_to_cbrain_tasks.rb I will add three boolean columns to the cbrain_tasks table cleanup_workdir (defaulting 'true' as suggested), cleanup_inputs and cleanup_outputs.

Task Model Logic

• file BrainPortal/app/models/cbrain_task.rb I will add these attributes and implement a helper method to verify if the task has reached a terminal state (utilizing existing FINAL_STATUS list) before triggering background activities.

User Interface & Controller

• I will update the task control panel specifically BrainPortal/app/views/tasks/_control.html.erb or _params.html.erb to include the selection boxes. I will include clear warnings next to each to explain the risks of erasing cache vs work directories.
• As of controller i will update the task_params method in BrainPortal/app/controllers/tasks_controller.rb to permit these new boolean attributes.

Task Lifecycle (Bourreau Side)

• In file Bourreau/app/models/bourreau_worker.rb
• To keep the monolithic main loop clean I will implement a private method enqueue_post_task_cleanup(task). And I will call this only when a task hits a terminal state to instantiate and save the BackgroundActivity::RemoveTaskWorkdir (and eventually the cache cleaners) as you demonstrated.

Background Activities

• Creating remove_task_inputs.rb and remove_task_outputs.rb in BrainPortal/app/models/background_activity/
• For safety I will model these after the existing remove_task_workdir.rb logic and incorporate 'safe delete' checks leveraging logic from the 'Boutiques' cache cleaner modules to ensure we skip shared metadata like JSON and license files.

Does this summary align with what you had in mind? I am happy to refine any part of this before I start actual implementation.

[prioux]

Actually this seems like a fine breakdown of what do to. Except I don't think you're handling the more complicated cases, because I want users to have multiple options for different situations. Situations can be e.g. task completed, task failed on cluster, task failed to setup etc etc. Actions can be erase workdir, erase inputs, erase outputs, etc. There are multiple combinations and they don't all make sense. I'll come back with a list of maybe the only ones that make sense, a bit later.

[AviJxn]

That makes total sense. I see how the combinations of states (eg. 'failed on cluster' vs 'completed') require more nuanced options than simple booleans.
I will wait for your list of sensible combinations before finalizing the database schema. In the meantime I will start researching boutiques cache cleaner modules to understand safety logic for shared assets.

[AviJxn]

Hi @prioux, hope you are well! Just checking in to see if you have had a chance to think about the state action combinations for this feature.
I have already scaffolded the background activities and added the Boutiques inspired safety checks locally, so I am completely ready to wire things up whenever you have a moment. Thanks!

[prioux]

Hello @AviJxn , sorry for the delay, I'm not working full time these days, and I'm also in vacation half of the time that I do work. Anyway, here's a list of the state and action combinations I think we'd like to have:

Task State	Workdir Action	Input Cache Action	Output Cache Action	Default?
Completed	Keep	Keep	Keep	No, current behavior
Completed	Erase	Keep	Keep	No
Completed	Erase	Keep	Erase	New default
Completed	Erase	Erase	Erase	No
Completed	Keep	Erase	Erase	No
Failed	Keep	Keep	N/A	Yes
Failed	Erase	Keep	N/A	No
Failed	Erase	Keep	N/A	No

I really am not sure how to present all these choices to the user; they'll have to select one option for Completed and one option for Failed.