Merge branch 'develop_lenz' into migrate_to_p13

Author: RandomDefaultUser

.dockerignore

@@ -1,2 +1,2 @@
 *
-!install/*
+!pipeline/*

.github/workflows/cpu-tests.yml

@@ -183,7 +183,7 @@ jobs:
           # be there before it has been installed.
           sed -i '/materials-learning-algorithms/d' ./env_after.yml
 
-          # if comparison fails, `install/mala_cpu_[base]_environment.yml` needs to be aligned with
+          # if comparison fails, `pipeline/mala_cpu_[base]_environment.yml` needs to be aligned with
           # `requirements.txt` and/or extra dependencies are missing in the Docker Conda environment
 
           if diff --brief env_before.yml env_after.yml

.gitignore

@@ -153,6 +153,7 @@ cython_debug/
 *.out
 *.npy
 *.pkl
+*.pk
 *.pth
 *.json
 

Dockerfile

@@ -14,7 +14,7 @@ RUN apt-get --allow-releaseinfo-change update && apt-get upgrade -y && \
 
 # Choose 'cpu' or 'gpu'
 ARG DEVICE=cpu
-COPY install/mala_${DEVICE}_environment.yml .
+COPY pipeline/mala_${DEVICE}_environment.yml .
 RUN conda env create -f mala_${DEVICE}_environment.yml && rm -rf /opt/conda/pkgs/*
 
 # Install optional MALA dependencies into Conda environment with pip

docs/source/CONTRIBUTE.md

@@ -116,7 +116,7 @@ If you add additional dependencies, make sure to add them to `requirements.txt`
 if they are required or to `setup.py` under the appropriate `extras` tag if
 they are not.
 Further, in order for them to be available during the CI tests, make sure to
-add _required_ dependencies to the appropriate environment files in folder `install/` and _extra_ requirements directly in the `Dockerfile` for the `conda` environment build.
+add _required_ dependencies to the appropriate environment files in folder `pipeline/` and _extra_ requirements directly in the `Dockerfile` for the `conda` environment build.
 
 ## Pull Requests
 We actively welcome pull requests.

docs/source/advanced_usage/descriptors.rst

@@ -76,23 +76,20 @@ An example would be this:
 
       .. code-block:: python
 
-            hyperoptimizer.add_snapshot("espresso-out", os.path.join(data_path, "Be_snapshot1.out"),
-                                        "numpy", os.path.join(data_path, "Be_snapshot1.out.npy"),
+            hyperoptimizer.add_snapshot("espresso-out", os.path.join(data_path_be, "Be_snapshot1.out"),
+                                        "numpy", os.path.join(data_path_be, "Be_snapshot1.out.npy"),
                                         target_units="1/(Ry*Bohr^3)")
-            hyperoptimizer.add_snapshot("espresso-out", os.path.join(data_path, "Be_snapshot2.out"),
-                                        "numpy", os.path.join(data_path, "Be_snapshot2.out.npy"),
+            hyperoptimizer.add_snapshot("espresso-out", os.path.join(data_path_be, "Be_snapshot2.out"),
+                                        "numpy", os.path.join(data_path_be, "Be_snapshot2.out.npy"),
                                         target_units="1/(Ry*Bohr^3)")
 
 Once this is done, you can start the optimization via
 
       .. code-block:: python
 
-            hyperoptimizer.perform_study(return_plotting=False)
+            hyperoptimizer.perform_study()
             hyperoptimizer.set_optimal_parameters()
 
-If ``return_plotting`` is set to ``True``, relevant plotting data for the
-analysis are returned. This is useful for exploratory searches.
-
 Since the ACSD re-calculates the bispectrum descriptors for each combination
 of hyperparameters, it is useful to use parallel descriptor calculation.
 To do so, you can enable the `MPI <https://www.mpi-forum.org/>`_ capabilites

docs/source/advanced_usage/hyperparameters.rst

@@ -96,6 +96,34 @@ are started with ``wait_time`` time interval in between (to avoid race
 conditions when accessing the same data base) and further only use the data
 base, not MPI, for communication.
 
+The batch job on your HPC cluster will get killed after the designated runtime.
+Then unfinished trials will remain in the Optuna database in state RUNNING.
+
+The current workflow for resuming the study which makes use of MALA's own
+resume tooling
+(see ``examples/advanced/ex05_checkpoint_hyperparameter_optimization.py``) is
+this: before submitting the batch job again and let the script do the resume
+work, a user needs to modify the database like so:
+
+    .. code-block:: bash
+
+        python3 -c "import mala; mala.HyperOptOptuna.requeue_zombie_trials('hyperopt01', 'sqlite:///hyperopt.db')"
+
+which will set the RUNNING trials to state WAITING.
+When Optuna resumes, it will pick up and re-run those, before carrying on
+running the resumed study.
+
+Common questions related to this feature:
+
+- "Does "injecting" jobs like this disturb Optuna's operation in any way?":
+  No, the study object takes all of its information directly from the
+  data base, which in this case has "WAITING" trials now.
+- "Do those trials have to be run?": Technically not. One could simply ignore
+  them and re-run without them. The problem is that in this case, the study
+  will have missing data points from trials that have been suggested for a
+  reason, so even if Optuna would resume fine, we still want to re-run them
+  from an optimization point of view.
+
 If you do distributed hyperparameter optimization, another useful option
 is
 
@@ -114,7 +142,7 @@ a physical validation metric such as
 
       .. code-block:: python
 
-            parameters.running.after_training_metric = "band_energy"
+            parameters.running.final_validation_metric = "band_energy"
 
 Advanced optimization algorithms
 ********************************

docs/source/advanced_usage/openpmd.rst

@@ -33,16 +33,16 @@ be left untouched. Specifically, set
             ...
             # Changes for DataHandler
             data_handler = mala.DataHandler(parameters)
-            data_handler.add_snapshot("Be_snapshot0.in.h5", data_path,
-                                       "Be_snapshot0.out.h5", data_path, "tr",
+            data_handler.add_snapshot("Be_snapshot0.in.h5", data_path_be,
+                                       "Be_snapshot0.out.h5", data_path_be, "tr",
                                        snapshot_type="openpmd")
             ...
             # Changes for DataShuffler
             data_shuffler = mala.DataShuffler(parameters)
             # Data can be shuffle FROM and TO openPMD - but also from
             # numpy to openPMD.
-            data_shuffler.add_snapshot("Be_snapshot0.in.h5", data_path,
-                                        "Be_snapshot0.out.h5", data_path,
+            data_shuffler.add_snapshot("Be_snapshot0.in.h5", data_path_be,
+                                        "Be_snapshot0.out.h5", data_path_be,
                                         snapshot_type="openpmd")
             data_shuffler.shuffle_snapshots(...,
                                             save_name="Be_shuffled*.h5")

docs/source/advanced_usage/trainingmodel.rst

@@ -71,13 +71,13 @@ is directly outputted by MALA. By default, this validation loss gives the
 mean squared error between LDOS prediction and actual value. From a purely
 ML point of view, this is fine; however, the correctness of the LDOS itself
 does not hold much physical virtue. Thus, MALA implements physical validation
-metrics to be accessed before and after the training routine.
+metrics which can be evaluated for example after the training.
 
 Specifically, when setting
 
       .. code-block:: python
 
-            parameters.running.after_training_metric = "band_energy"
+            parameters.running.final_validation_metric = "band_energy"
 
 the error in the band energy between actual and predicted LDOS will be
 calculated and printed before and after network training (in meV/atom).
@@ -181,10 +181,10 @@ descriptors is supported.
             parameters.data.shuffling_seed = 1234
 
             data_shuffler = mala.DataShuffler(parameters)
-            data_shuffler.add_snapshot("Be_snapshot0.in.npy", data_path,
-                                       "Be_snapshot0.out.npy", data_path)
-            data_shuffler.add_snapshot("Be_snapshot1.in.npy", data_path,
-                                       "Be_snapshot1.out.npy", data_path)
+            data_shuffler.add_snapshot("Be_snapshot0.in.npy", data_path_be,
+                                       "Be_snapshot0.out.npy", data_path_be)
+            data_shuffler.add_snapshot("Be_snapshot1.in.npy", data_path_be,
+                                       "Be_snapshot1.out.npy", data_path_be)
             data_shuffler.shuffle_snapshots(complete_save_path="../",
                                             save_name="Be_shuffled*")
 
@@ -212,7 +212,7 @@ in the file ``advanced/ex03_tensor_board``. Simply select a logger prior to trai
       .. code-block:: python
 
             parameters.running.logger = "tensorboard"
-            parameters.running.logging_dir = "mala_vis"
+            parameters.running.logging_dir = "mala_logs"
 
 or
 
@@ -224,14 +224,14 @@ or
                   entity="your_wandb_entity"
             )
             parameters.running.logger = "wandb"
-            parameters.running.logging_dir = "mala_vis"
+            parameters.running.logging_dir = "mala_logs"
 
 where ``logging_dir`` specifies some directory in which to save the
 MALA logging data. You can also select which metrics to record via
 
       .. code-block:: python
 
-            parameters.validation_metrics = ["ldos", "dos", "density", "total_energy"]
+            parameters.logging_metrics = ["ldos", "dos", "density", "total_energy"]
 
 Full list of available metrics:
       - "ldos": MSE of the LDOS.
@@ -249,14 +249,14 @@ To save time and resources you can specify the logging interval via
 
       .. code-block:: python
 
-            parameters.running.validate_every_n_epochs = 10
+            parameters.running.logging_metrics_interval = 10
 
 If you want to monitor the degree to which the model overfits to the training data,
 you can use the option
 
       .. code-block:: python
             
-            parameters.running.validate_on_training_data = True
+            parameters.running.log_metrics_on_train_set = True
 
 MALA will evaluate the validation metrics on the training set as well as the validation set.
 

docs/source/basic_usage/more_data.rst

@@ -100,8 +100,8 @@ and fill it with data, e.g., by
       .. code-block:: python
 
             data_converter = mala.DataConverter(parameters)
-            outfile = os.path.join(data_path, "Be_snapshot0.out")
-            ldosfile = os.path.join(data_path, "cubes/tmp.pp*Be_ldos.cube")
+            outfile = os.path.join(data_path_be, "Be_snapshot0.out")
+            ldosfile = os.path.join(data_path_be, "cubes/tmp.pp*Be_ldos.cube")
 
             data_converter.add_snapshot(descriptor_input_type="espresso-out",
                                         descriptor_input_path=outfile,
@@ -133,12 +133,12 @@ Once data is provided, the conversion itself is simple.
                                              simulation_output_save_path="./",
                                              naming_scheme="Be_snapshot*.npy",
                                              descriptor_calculation_kwargs=
-                                             {"working_directory": data_path})
+                                             {"working_directory": data_path_be})
             # You can also provide only one path
             # data_converter.convert_snapshots(complete_save_path="./",
             #                                  naming_scheme="Be_snapshot*.npy",
             #                                  descriptor_calculation_kwargs=
-            #                                  {"working_directory": data_path})
+            #                                  {"working_directory": data_path_be})
 
 The ``convert_snapshots`` function will convert ALL snapshots added via
 ``add_snapshot`` and save the resulting volumetric numpy files to the