feat: add Dagster orchestration project for StrainRelief - #32
Open
erwallace wants to merge 8 commits into
Open
Conversation
Add the `dag/` Dagster project that wraps the StrainRelief workflow as assets (preprocess -> conformers -> local/global optimisation -> aggregate -> plot). Resources are config-driven, mirroring the Hydra `_target_` pattern: - CalculatorResource selects MACE/MMFF94/FAIRChem via a `kind`-discriminated union, loaded exactly once per run (module cache + in-process executor). - OptimiserResource base + Local/Global subclasses build BFGS and attach the shared calculator. Config classes in configs.py map 1:1 to the strain_relief function kwargs, with a full example run.yaml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n job Split preprocess_data into input_df + docked_mols assets (Option B) so the original DataFrame flows to aggregate_results for the id-merge in process_output. Route input_df through the DuckDB pandas io manager (dropping the unserialisable live RDKit mol column; molecules persist as mol_bytes and are rebuilt downstream, normalising DuckDB's bytearray back to bytes). Add an all_assets job (AssetSelection.all()) so the pipeline launches via `dg launch --job all_assets` — avoiding the antlr asset-selection grammar that conflicts with the omegaconf/hydra antlr pin. Make the optimiser config a discriminated union (BFGSConfig + placeholder) so calculator and optimiser configs share a uniform selector representation. Fix PyTorchIOManager to create parent dirs before torch.save, and correct the asset input type annotations (dict, not ConformerBatch). Document how to run the pipeline and where each asset's output is stored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Write all run artifacts under an absolute, run-scoped directory using a flat per-asset layout: data/<run_id>/<asset_name>.<ext> (DATA_DIR resolved from the repo layout, independent of the launch directory), so runs never overwrite each other. - Replace DuckDBPandasIOManager with a custom ParquetIOManager (data/<run_id>/<asset>.parquet); drop the dagster_duckdb_pandas dep. - PyTorchIOManager writes data/<run_id>/<asset>.pt. - aggregate_results takes AssetExecutionContext and writes the final parquet into the run directory; drop the DuckDB-specific bytearray->bytes workarounds (parquet preserves bytes). - Rename the job to `strain_relief`. - Update README run instructions and output-location table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ct the name of the assets output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the
dag/Dagster project that wraps the StrainRelief workflow as assets (preprocess -> conformers -> local/global optimisation -> aggregate -> plot).Resources are config-driven, mirroring the Hydra
_target_pattern:kind-discriminated union, loaded exactly once per run (module cache + in-process executor).Config classes in configs.py map 1:1 to the strain_relief function kwargs, with a full example run.yaml.