From 773730989c43c7096ac081ac28852373b05c9f46 Mon Sep 17 00:00:00 2001 From: Anthony Costarelli Date: Mon, 29 Jun 2026 17:50:18 -0400 Subject: [PATCH] Move operation lifecycle to IOM; collapse power problem types The domain-neutral build/solve/run lifecycle now lives in IOM. POM keeps only the power-specific extension-point implementations: - Delete the moved methods (build!/solve!/run!/build_model!/build_pre_step!/ reset!/handle_initial_conditions!/build_if_not_already_built!/execute_emulation!/ solve_and_write_initial_conditions!). Re-export build!/solve!/run! from IOM and import build_problem!/build_initial_conditions! to extend the IOM generics. - Collapse the problem-type hierarchy to a single linear chain AbstractPowerOperationProblem -> GenericPowerOperationProblem -> DefaultPowerOperationProblem. The decision<->emulation distinction is the wrapper's job (DecisionModel/EmulationModel), so the *Decision*/*Emulation* problem types are removed; constructors, validate_*, exports, and test mocks are rebound accordingly. - Drop the duplicated timer constants; import BUILD_PROBLEMS_TIMER from IOM so build_problem! timings aggregate into IOM's printout. Point [sources] at the IOM ac/move-op-models branch for co-development (revert to main before merge). Refresh the stale IOM/POM split notes in .claude/CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/CLAUDE.md | 37 +- Project.toml | 2 +- src/PowerOperationsModels.jl | 22 +- src/core/definitions.jl | 4 - src/core/problem_types.jl | 51 +-- src/initial_conditions/initialization.jl | 25 -- src/operation/decision_model.jl | 315 +---------------- src/operation/emulation_model.jl | 328 +----------------- test/Project.toml | 2 +- ..._device_thermal_generation_constructors.jl | 4 +- test/test_utils.jl | 2 +- test/test_utils/mock_operation_models.jl | 8 +- 12 files changed, 80 insertions(+), 720 deletions(-) diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 3a61a85..38e9adc 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -4,7 +4,7 @@ Platform-wide Sienna conventions (performance, type stability, formatter, enviro ## Purpose & place in the stack -POM is the **collection of operational optimization models for power systems**: the concrete device, service, and network formulations plus the `PowerOperationModel` problem-type chain. It builds JuMP models from PowerSystems components. +POM is the **collection of operational optimization models for power systems**: the concrete device, service, and network formulations plus the `AbstractPowerOperationProblem` problem-type chain. It builds JuMP models from PowerSystems components. Abstraction hierarchy (low → high level of generality): @@ -16,13 +16,30 @@ PSI (the old PowerSimulations.jl) ≈ POM + IOM. Many ports into POM originate f IS and IOM are **external package dependencies** (resolved via `Project.toml`/`[sources]`), not subdirectories. -### The IOM/POM split (PR #104 "Redistribute operation models") - -IOM was made domain-neutral. The power-flavoured taxonomy was **moved out of IOM into POM** (`src/core/problem_types.jl`): - -- POM owns `PowerOperationModel <: IOM.AbstractOptimizationProblem`, then `DecisionProblem`/`EmulationProblem`, `DefaultDecisionProblem`/`DefaultEmulationProblem`, `GenericOpProblem`/`GenericEmulationProblem`. POM dispatches on `DecisionModel{<:PowerOperationModel}` etc. The no-type-param `DecisionModel(template, sys)` default + the EmulationModel update chain live in POM (`src/operation/`). -- IOM removed `OperationModel`, `DecisionProblem`, `EmulationProblem`, the `Generic*`/`Default*` problem types, and the `Simulation*` stubs. The old `OperationModel` abstract → `IOM.AbstractOptimizationModel`. `validate_time_series!` / `validate_template` are stubs on the neutral abstract. -- **Gotcha:** when a symbol seems "not defined in IOM", check whether #104 moved it to POM — define/dispatch on the POM chain, do **not** re-add it to IOM. +### The IOM/POM split + +IOM is domain-neutral and owns the **operation-model lifecycle**. The power-flavoured +problem taxonomy lives in POM (`src/core/problem_types.jl`): + +- POM owns a single linear chain `AbstractPowerOperationProblem <: IOM.AbstractOptimizationProblem` + → `GenericPowerOperationProblem` (template-driven) → `DefaultPowerOperationProblem` (the + default `M`). Decision↔emulation is **not** in the problem chain — it is the wrapper's job + (`DecisionModel` vs `EmulationModel`, both in IOM). The no-type-param + `DecisionModel(template, sys)` / `EmulationModel(template, sys)` defaulting constructors live + in POM (`src/operation/`), defaulting `M = DefaultPowerOperationProblem`. +- **IOM owns the generic lifecycle**: `build!`/`solve!`/`run!`/`build_model!`/`build_pre_step!`/ + `reset!`/`handle_initial_conditions!`/`execute_emulation!`/`build_if_not_already_built!` are + single methods on `AbstractOptimizationModel`/`DecisionModel`/`EmulationModel` in IOM. POM + supplies the power-specific extension points: `build_problem!` (`src/operation/build_problem.jl`), + `build_initial_conditions!` (`src/initial_conditions/initialization.jl`), + `instantiate_network_model!`, `validate_template`, `validate_time_series!`. The two DM/EM + lifecycle differences are IOM compile-time hooks (`_mark_recurrent_solves!`, + `_apply_build_executions!`, `reset_time_series_type`). +- The operation timers (`BUILD_PROBLEMS_TIMER`, …) live in IOM; POM imports them so its + `build_problem!` timings aggregate into IOM's `build!` printout. +- **Gotcha:** when a symbol seems "not defined in IOM", check whether it is a POM extension-point + implementation (`build_problem!`/`build_initial_conditions!`/`validate_*`) — extend the IOM + generic (import it), do **not** create a shadowing POM function. ## Optimization Model Construction Conventions @@ -51,7 +68,7 @@ src/ area_interchange.jl # Area interchange balance core/ # Type definitions (no heavy logic) definitions.jl, physical_constant_definitions.jl - problem_types.jl # PowerOperationModel chain (moved from IOM, PR #104) + problem_types.jl # AbstractPowerOperationProblem chain (linear; d/e is the wrapper's job) interfaces.jl # incl. PowerFlowEvaluator wrapping PF models variables.jl, auxiliary_variables.jl, constraints.jl, expressions.jl, parameters.jl formulations.jl, network_formulations.jl # native DCP/ACP/PTDF/CopperPlate/Area structs @@ -122,7 +139,7 @@ Solvers: `HiGHS` (LP/MILP), `Ipopt` (NLP), `SCS` (SDP) — helpers `HiGHS_optimi ## POM-specific conventions, invariants, gotchas -- **Layer boundaries:** device/network formulations + `PowerOperationModel` chain live in POM; `OptimizationContainer`/`DecisionModel`/settings/generic builders in IOM; base key/TS types in IS. Don't push power-specific logic down into IOM. +- **Layer boundaries:** device/network formulations + `AbstractPowerOperationProblem` chain live in POM; `OptimizationContainer`/`DecisionModel`/`EmulationModel`/the build/solve/run lifecycle/settings/generic builders in IOM; base key/TS types in IS. Don't push power-specific logic down into IOM. - **`network_reduction` must always be populated** by POM's `instantiate_network_model!` for every network model (CopperPlate/AreaBalance set an identity reduction). It is `Union{Nothing,...}` on the IOM side. Concrete `BranchReductionOptimizationTracker` + `get_constraint_map_by_type` live in POM `network_models/network_reductions.jl`; IOM keeps only the abstract `AbstractBranchReductionTracker`. IOM evaluators must be `<:IOM.AbstractEvaluator`; POM wraps PF models in `PowerFlowEvaluator` (`core/interfaces.jl`). - **Parallel branches** between the same `(from_bus, to_bus)` pair are reduced to one equivalent branch by PowerNetworkMatrices **before** reaching POM. Index branch-pair-keyed variables (LPAC cosine vars, voltage approximations) by branch name directly — do not add dedupe bookkeeping; it's a non-problem at this layer. - **Security-constrained N-1 uses MODF** (Modified Outage Distribution Factors: PNM `VirtualMODF`/`ContingencySpec`), in `ac_transmission_models/security_constrained_branch.jl`. The old LODF-based `network_models/security_constrained_models.jl` and all generator-side (G-1) SC have been **removed** — do not reintroduce LODF or gen-side MODF. `NetworkModel.MODF_matrix`, `DeviceModel.outages`, and `supports_outages` (default false; POM specializes `true` for `AbstractSecurityConstrainedStaticBranch`) live in IOM. diff --git a/Project.toml b/Project.toml index fa73c03..e6328f7 100644 --- a/Project.toml +++ b/Project.toml @@ -27,7 +27,7 @@ PowerFlows = "94fada2c-fd9a-4e89-8d82-81405f5cb4f6" [sources] InfrastructureSystems = {rev = "IS4", url = "https://github.com/Sienna-Platform/InfrastructureSystems.jl"} PowerSystems = {rev = "psy6", url = "https://github.com/Sienna-Platform/PowerSystems.jl"} -InfrastructureOptimizationModels = {rev = "main", url = "https://github.com/Sienna-Platform/InfrastructureOptimizationModels.jl"} +InfrastructureOptimizationModels = {rev = "ac/move-op-models", url = "https://github.com/Sienna-Platform/InfrastructureOptimizationModels.jl"} PowerNetworkMatrices = {rev = "psy6", url = "https://github.com/Sienna-Platform/PowerNetworkMatrices.jl"} [extensions] diff --git a/src/PowerOperationsModels.jl b/src/PowerOperationsModels.jl index 41bdde2..5a87033 100644 --- a/src/PowerOperationsModels.jl +++ b/src/PowerOperationsModels.jl @@ -197,7 +197,14 @@ import InfrastructureOptimizationModels: DecisionModel, EmulationModel, make_empty_jump_model_with_settings, - set_model! + set_model!, + # Operation-model lifecycle: build!/solve!/run! live in IOM (re-exported below); + # build_problem!/build_initial_conditions! are IOM extension points POM implements. + build!, + solve!, + run!, + build_problem!, + build_initial_conditions! using InfrastructureOptimizationModels # TODO: use explicit imports. @@ -349,9 +356,10 @@ import InfrastructureOptimizationModels: CONTAINER_KEY_EMPTY_META # Import high-frequency IOM internals used throughout operation lifecycle code. -# Note: BUILD_PROBLEMS_TIMER and RUN_OPERATION_MODEL_TIMER are defined in POM's -# definitions.jl, so they are NOT imported from IOM. +# `BUILD_PROBLEMS_TIMER` is shared with IOM so POM's `build_problem!` timings aggregate +# into the same timer printout produced by IOM's `build!`/`solve!`. import InfrastructureOptimizationModels: + BUILD_PROBLEMS_TIMER, LOG_GROUP_OPTIMIZATION_CONTAINER, get_store, set_status!, @@ -378,10 +386,7 @@ export EmulationModel export PowerOperationsProblemTemplate export InitialCondition export AbstractPowerOperationProblem -export AbstractPowerDecisionProblem -export AbstractPowerEmulationProblem -export DefaultPowerDecisionProblem -export DefaultPowerEmulationProblem +export DefaultPowerOperationProblem # Network export NetworkModel @@ -476,8 +481,7 @@ export RunStatus export SimulationBuildStatus # Problem Types -export GenericPowerDecisionProblem -export GenericPowerEmulationProblem +export GenericPowerOperationProblem # Settings and Data Types export Settings diff --git a/src/core/definitions.jl b/src/core/definitions.jl index 54d8f1e..651ff99 100644 --- a/src/core/definitions.jl +++ b/src/core/definitions.jl @@ -5,10 +5,6 @@ const NamedMinMax = Tuple{String, MinMax} const UpDown = NamedTuple{(:up, :down), NTuple{2, Float64}} const InOut = NamedTuple{(:in, :out), NTuple{2, Float64}} -const BUILD_PROBLEMS_TIMER = TimerOutputs.TimerOutput() -const RUN_OPERATION_MODEL_TIMER = TimerOutputs.TimerOutput() -const RUN_SIMULATION_TIMER = TimerOutputs.TimerOutput() - # Type Alias for JuMP containers const JuMPOrFloat = Union{JuMP.AbstractJuMPScalar, Float64} const GAE = JuMP.GenericAffExpr{Float64, JuMP.VariableRef} diff --git a/src/core/problem_types.jl b/src/core/problem_types.jl index 1325046..788ee70 100644 --- a/src/core/problem_types.jl +++ b/src/core/problem_types.jl @@ -1,53 +1,30 @@ ################################################################################# # Operation-problem type hierarchy # -# IOM (#104 "Redistribute operation models") is a domain-neutral optimization -# library: it owns `DecisionModel{M}`/`EmulationModel{M}` and parameterizes them -# over the single abstract tag `IOM.AbstractOptimizationProblem`. The concrete, -# power-flavoured problem-type chain lives here in POM. POM-side methods dispatch -# on `DecisionModel{<:AbstractPowerOperationProblem}` (and its sub-tags) while IOM keeps only -# error-stub / extension-point methods on the neutral abstract. +# IOM is a domain-neutral optimization library: it owns `DecisionModel{M}`/ +# `EmulationModel{M}` and parameterizes them over `IOM.AbstractOptimizationProblem`. +# The decision↔emulation distinction is the wrapper's job (DecisionModel vs +# EmulationModel), so POM's power problem chain is a single linear hierarchy: the +# concrete problem type only describes domain content, not the solve strategy. ################################################################################# """ Umbrella supertype for every PowerOperationsModels optimization problem. A `DecisionModel`/`EmulationModel` parameterized over a subtype of this is a POM -operation model; dispatch on `DecisionModel{<:AbstractPowerOperationProblem}` selects the -POM-side implementations of IOM's extension points. +operation model. """ abstract type AbstractPowerOperationProblem <: IOM.AbstractOptimizationProblem end """ -Supertype for single-period (decision) operation problems solved by `DecisionModel`. +Operation problems whose build/validate behavior is fully driven by the +`ProblemTemplate` (the common case). The seam for a future custom-build problem is +to subtype `AbstractPowerOperationProblem` directly instead of this. """ -abstract type AbstractPowerDecisionProblem <: AbstractPowerOperationProblem end +abstract type GenericPowerOperationProblem <: AbstractPowerOperationProblem end """ -Supertype for rolling-horizon (emulation) operation problems solved by `EmulationModel`. +Default concrete template-driven operation problem. The `M` used when a +`DecisionModel`/`EmulationModel` is built from a template without an explicit problem +type. """ -abstract type AbstractPowerEmulationProblem <: AbstractPowerOperationProblem end - -""" -Decision problems whose build/validate behavior is fully driven by the -`ProblemTemplate` (the common case). `DecisionModel{<:GenericPowerDecisionProblem}` -gets the generic template-driven `build_model!`/`validate_template` methods. -""" -abstract type GenericPowerDecisionProblem <: AbstractPowerDecisionProblem end - -""" -Emulation problems whose build/validate behavior is fully driven by the -`ProblemTemplate`. -""" -abstract type GenericPowerEmulationProblem <: AbstractPowerEmulationProblem end - -""" -Default concrete template-driven decision problem. The `M` used when a `DecisionModel` -is built from a template without an explicit problem type. -""" -struct DefaultPowerDecisionProblem <: GenericPowerDecisionProblem end - -""" -Default concrete template-driven emulation problem. The `M` used when an `EmulationModel` -is built from a template without an explicit problem type. -""" -struct DefaultPowerEmulationProblem <: GenericPowerEmulationProblem end +struct DefaultPowerOperationProblem <: GenericPowerOperationProblem end diff --git a/src/initial_conditions/initialization.jl b/src/initial_conditions/initialization.jl index 2f95166..c68924b 100644 --- a/src/initial_conditions/initialization.jl +++ b/src/initial_conditions/initialization.jl @@ -126,28 +126,3 @@ function build_initial_conditions!(model::IOM.AbstractOptimizationModel) end return end - -# Called `initialize!` in PSI (lived in operation_model_interface.jl). -function solve_and_write_initial_conditions!(model::IOM.AbstractOptimizationModel) - container = get_optimization_container(model) - if get_initial_conditions_model_container(get_internal(model)) === nothing - return - end - @info "Solving Initialization Model for $(get_name(model))" - status = IOM.execute_optimizer!( - get_initial_conditions_model_container(get_internal(model)), - get_system(model), - ) - if status == RunStatus.FAILED - error("Model failed to initialize") - end - - IOM.write_initial_conditions_data!( - container, - get_initial_conditions_model_container(get_internal(model)), - ) - init_file = IOM.get_initial_conditions_file(model) - Serialization.serialize(init_file, IOM.get_initial_conditions_data(container)) - @info "Serialized initial conditions to $init_file" - return -end diff --git a/src/operation/decision_model.jl b/src/operation/decision_model.jl index 8873e87..8478350 100644 --- a/src/operation/decision_model.jl +++ b/src/operation/decision_model.jl @@ -1,4 +1,4 @@ -# Template-first constructors default the problem type to `DefaultPowerDecisionProblem`. +# Template-first constructors default the problem type to `DefaultPowerOperationProblem`. # IOM only ships the `DecisionModel{M}` / `DecisionModel(::Type{M}, ...)` variants # (`M` is the domain-neutral `AbstractOptimizationProblem`); the default problem # type is a POM concept, so the defaulting constructors live here. @@ -8,318 +8,31 @@ function DecisionModel( jump_model::Union{Nothing, JuMP.Model} = nothing; kwargs..., ) - return DecisionModel{DefaultPowerDecisionProblem}(template, sys, jump_model; kwargs...) + return DecisionModel{DefaultPowerOperationProblem}(template, sys, jump_model; kwargs...) end -# Generic (template-driven) decision problems require an AbstractProblemTemplate -# subtype, so the bare-system constructor (no template) is an error. +# Generic (template-driven) problems require an AbstractProblemTemplate subtype, so the +# bare-system constructor (no template) is an error. function DecisionModel{M}( sys::IS.InfrastructureSystemsContainer, jump_model::Union{Nothing, JuMP.Model} = nothing; kwargs..., -) where {M <: GenericPowerDecisionProblem} +) where {M <: GenericPowerOperationProblem} throw( IS.ArgumentError( - "GenericPowerDecisionProblem subtypes require a template. Use DecisionModel subtyping instead.", + "GenericPowerOperationProblem subtypes require a template. Use DecisionModel subtyping instead.", ), ) end -function build_pre_step!(model::DecisionModel{<:AbstractPowerDecisionProblem}) - TimerOutputs.@timeit BUILD_PROBLEMS_TIMER "Build pre-step" begin - validate_template(model) - if !isempty(model) - @info "OptimizationProblem status not ModelBuildStatus.EMPTY. Resetting" - reset!(model) - end - @info "Initializing Optimization Container For a DecisionModel" - init_optimization_container!( - get_optimization_container(model), - get_network_model(get_template(model)), - get_system(model), - ) - @info "Initializing ModelStoreParams" - IOM.init_model_store_params!(model) - set_status!(model, ModelBuildStatus.IN_PROGRESS) - end - return -end - -# Called `build_impl!(model)` in PSI (lived in decision_model.jl). -function build_model!(model::DecisionModel{<:AbstractPowerDecisionProblem}) - build_pre_step!(model) - @info "Instantiating Network Model" - IOM.instantiate_network_model!(model) - handle_initial_conditions!(model) - build_problem!( - get_optimization_container(model), - get_template(model), - get_system(model), - ) - IOM.serialize_metadata!(get_optimization_container(model), IOM.get_output_dir(model)) - log_values(get_settings(model)) - return -end - -""" -Build the Decision Model based on the specified AbstractPowerDecisionProblem. - -# Arguments - - - `model::DecisionModel{<:AbstractPowerDecisionProblem}`: DecisionModel object - - `output_dir::String`: Output directory for outputs - - `recorders::Vector{Symbol} = []`: recorder names to register - - `console_level = Logging.Error`: - - `file_level = Logging.Info`: - - `disable_timer_outputs = false` : Enable/Disable timing outputs - - `store_system_in_results::Bool = true`: If true, stores the system as JSON in the results HDF5 file. -""" -function build!( - model::DecisionModel{<:AbstractPowerDecisionProblem}; - output_dir::String, - recorders = [], - console_level = Logging.Error, - file_level = Logging.Info, - disable_timer_outputs = false, - store_system_in_results = true, -) - mkpath(output_dir) - IOM.set_output_dir!(model, output_dir) - IOM.set_console_level!(model, console_level) - IOM.set_file_level!(model, file_level) - TimerOutputs.reset_timer!(BUILD_PROBLEMS_TIMER) - disable_timer_outputs && TimerOutputs.disable_timer!(BUILD_PROBLEMS_TIMER) - file_mode = "w" - IOM.add_recorders!(model, recorders) - IOM.register_recorders!(model, file_mode) - logger = IS.configure_logging(get_internal(model), IOM.PROBLEM_LOG_FILENAME, file_mode) - if store_system_in_results - @warn "store_system_in_results is set to true. This will do nothing unless a Simulation is being built." - end - try - Logging.with_logger(logger) do - try - TimerOutputs.@timeit BUILD_PROBLEMS_TIMER "Problem $(get_name(model))" begin - build_model!(model) - end - set_status!(model, ModelBuildStatus.BUILT) - @info "\n$(BUILD_PROBLEMS_TIMER)\n" - catch e - set_status!(model, ModelBuildStatus.FAILED) - bt = catch_backtrace() - @error "DecisionModel Build Failed" exception = e, bt - end - end - finally - IOM.unregister_recorders!(model) - close(logger) - end - return IOM.get_status(model) -end - -function reset!(model::DecisionModel{<:GenericPowerDecisionProblem}) - was_built_for_recurrent_solves = built_for_recurrent_solves(model) - if was_built_for_recurrent_solves - IOM.set_execution_count!(model, 0) - end - sys = get_system(model) - ts_type = IOM.get_deterministic_time_series_type(sys) - IOM.set_container!( - get_internal(model), - OptimizationContainer( - get_system(model), - get_settings(model), - nothing, - ts_type, - ), - ) - get_optimization_container(model).built_for_recurrent_solves = - was_built_for_recurrent_solves - internal = get_internal(model) - IOM.set_initial_conditions_model_container!(internal, nothing) - IOM.empty_time_series_cache!(model) - empty!(get_store(model)) - set_status!(model, ModelBuildStatus.EMPTY) - return -end - -""" -Default solve method for models that conform to the requirements of -DecisionModel{<: AbstractPowerDecisionProblem}. - -This will call `build!` on the model if it is not already built. It will forward all -keyword arguments to that function. - -# Arguments - - - `model::IOM.AbstractOptimizationModel = model`: operation model - - `export_problem_outputs::Bool = false`: If true, export OptimizationProblemOutputs DataFrames to CSV files. - - `console_level = Logging.Error`: - - `file_level = Logging.Info`: - - `disable_timer_outputs = false` : Enable/Disable timing outputs - - `export_optimization_problem::Bool = true`: If true, serialize the model to a file to allow re-execution later. - - `store_system_in_results::Bool = true`: If true, stores the system as JSON in the results HDF5 file. - -# Examples - -```julia -outputs = solve!(OpModel) -outputs = solve!(OpModel, export_problem_outputs = true) -``` -""" -function solve!( - model::DecisionModel{<:AbstractPowerDecisionProblem}; - export_problem_outputs = false, - console_level = Logging.Error, - file_level = Logging.Info, - disable_timer_outputs = false, - export_optimization_problem = true, - store_system_in_results = true, - kwargs..., +# POM implementation of IOM's `validate_template` extension point for template-driven +# problems. Shared by both wrapper types (the body is wrapper-agnostic). +function validate_template( + model::Union{ + DecisionModel{<:GenericPowerOperationProblem}, + EmulationModel{<:GenericPowerOperationProblem}, + }, ) - if store_system_in_results - @warn "store_system_in_results is set to true. This will do nothing unless a Simulation is being built." - end - build_if_not_already_built!( - model; - console_level = console_level, - file_level = file_level, - disable_timer_outputs = disable_timer_outputs, - kwargs..., - ) - IOM.set_console_level!(model, console_level) - IOM.set_file_level!(model, file_level) - TimerOutputs.reset_timer!(RUN_OPERATION_MODEL_TIMER) - disable_timer_outputs && TimerOutputs.disable_timer!(RUN_OPERATION_MODEL_TIMER) - file_mode = "a" - IOM.register_recorders!(model, file_mode) - logger = IS.configure_logging( - get_internal(model), - IOM.PROBLEM_LOG_FILENAME, - file_mode, - ) - optimizer = get(kwargs, :optimizer, nothing) - try - Logging.with_logger(logger) do - try - IOM.initialize_storage!( - get_store(model), - get_optimization_container(model), - IOM.get_store_params(model), - ) - TimerOutputs.@timeit RUN_OPERATION_MODEL_TIMER "Solve" begin - IOM._pre_solve_model_checks(model, optimizer) - IOM.solve_model!(model) - current_time = get_initial_time(model) - write_outputs!(get_store(model), model, current_time, current_time) - IOM.write_optimizer_stats!( - get_store(model), - get_optimizer_stats(model), - current_time, - ) - end - if export_optimization_problem - TimerOutputs.@timeit RUN_OPERATION_MODEL_TIMER "Serialize" begin - serialize_optimization_model(model) - end - end - TimerOutputs.@timeit RUN_OPERATION_MODEL_TIMER "Outputs processing" begin - outputs = OptimizationProblemOutputs(model) - serialize_outputs(outputs, IOM.get_output_dir(model)) - export_problem_outputs && export_outputs(outputs) - end - @info "\n$(RUN_OPERATION_MODEL_TIMER)\n" - catch e - @error "Decision Problem solve failed" exception = (e, catch_backtrace()) - IOM.set_run_status!(model, RunStatus.FAILED) - end - end - finally - IOM.wait_for_serialization!(model) - IOM.unregister_recorders!(model) - close(logger) - end - - return IOM.get_run_status(model) -end - -function handle_initial_conditions!(model::DecisionModel{<:AbstractPowerDecisionProblem}) - TimerOutputs.@timeit BUILD_PROBLEMS_TIMER "Model Initialization" begin - if isempty(get_template(model)) - return - end - settings = get_settings(model) - initialize_model = get_initialize_model(settings) - deserialize_initial_conditions = get_deserialize_initial_conditions(settings) - serialized_initial_conditions_file = IOM.get_initial_conditions_file(model) - custom_init_file = get_initialization_file(settings) - - if !initialize_model && deserialize_initial_conditions - throw( - IS.ConflictingInputsError( - "!initialize_model && deserialize_initial_conditions", - ), - ) - elseif !initialize_model && !isempty(custom_init_file) - throw(IS.ConflictingInputsError("!initialize_model && initialization_file")) - end - - if !initialize_model - @info "Skip build of initial conditions" - return - end - - if !isempty(custom_init_file) - if !isfile(custom_init_file) - error("initialization_file = $custom_init_file does not exist") - end - if abspath(custom_init_file) != abspath(serialized_initial_conditions_file) - cp(custom_init_file, serialized_initial_conditions_file; force = true) - end - end - - if deserialize_initial_conditions && isfile(serialized_initial_conditions_file) - IOM.set_initial_conditions_data!( - get_optimization_container(model), - Serialization.deserialize(serialized_initial_conditions_file), - ) - @info "Deserialized initial_conditions_data" - else - @info "Make Initial Conditions Model" - build_initial_conditions!(model) - solve_and_write_initial_conditions!(model) - end - IOM.set_initial_conditions_model_container!( - get_internal(model), - nothing, - ) - end - return -end - -function build_if_not_already_built!(model::IOM.AbstractOptimizationModel; kwargs...) - status = IOM.get_status(model) - if status == ModelBuildStatus.EMPTY - if !haskey(kwargs, :output_dir) - error( - "'output_dir' must be provided as a kwarg if the model build status is $status", - ) - else - new_kwargs = Dict(k => v for (k, v) in kwargs if k != :optimizer) - status = build!(model; new_kwargs...) - end - end - if status != ModelBuildStatus.BUILT - error("build! of the $(typeof(model)) $(get_name(model)) failed: $status") - end - return -end - -function validate_template(::DecisionModel{M}) where {M <: AbstractPowerDecisionProblem} - error("validate_template is not implemented for DecisionModel{$M}") -end - -function validate_template(model::DecisionModel{<:GenericPowerDecisionProblem}) validate_template_impl!(model) return end @@ -328,7 +41,7 @@ end # POM provides the concrete check for its template-driven problem types. It reconciles # the model's resolution/interval/horizon settings against the forecast data in the # system and errors when the system has no forecast data. -function validate_time_series!(model::DecisionModel{<:GenericPowerDecisionProblem}) +function validate_time_series!(model::DecisionModel{<:GenericPowerOperationProblem}) sys = get_system(model) settings = get_settings(model) _reconcile_resolution!(settings, sys) diff --git a/src/operation/emulation_model.jl b/src/operation/emulation_model.jl index f638698..c1929b9 100644 --- a/src/operation/emulation_model.jl +++ b/src/operation/emulation_model.jl @@ -1,4 +1,4 @@ -# Template-first constructor defaults the problem type to `DefaultPowerEmulationProblem` +# Template-first constructor defaults the problem type to `DefaultPowerOperationProblem` # (mirrors the DecisionModel defaulting constructor; see decision_model.jl). function EmulationModel( template::IOM.AbstractProblemTemplate, @@ -6,7 +6,7 @@ function EmulationModel( jump_model::Union{Nothing, JuMP.Model} = nothing; kwargs..., ) - return EmulationModel{DefaultPowerEmulationProblem}( + return EmulationModel{DefaultPowerOperationProblem}( template, sys, jump_model; @@ -16,7 +16,7 @@ end # POM-side implementation of IOM's `validate_time_series!` extension point for # emulation problems. Emulation models solve a single step, so horizon == resolution. -function validate_time_series!(model::EmulationModel{<:GenericPowerEmulationProblem}) +function validate_time_series!(model::EmulationModel{<:GenericPowerOperationProblem}) sys = get_system(model) settings = get_settings(model) _reconcile_resolution!(settings, sys) @@ -34,325 +34,3 @@ function validate_time_series!(model::EmulationModel{<:GenericPowerEmulationProb end return end - -# FIXME untested. Moved to accommodate a few methods dispatching on EmulationModelStore, -# but not run in the tests and not yet refactored for IOM-POM split. -function build_pre_step!(model::EmulationModel) - TimerOutputs.@timeit BUILD_PROBLEMS_TIMER "Build pre-step" begin - validate_template(model) - if !isempty(model) - @info "AbstractPowerEmulationProblem status not ModelBuildStatus.EMPTY. Resetting" - reset!(model) - end - container = get_optimization_container(model) - container.built_for_recurrent_solves = true - - @info "Initializing Optimization Container For an EmulationModel" - init_optimization_container!( - get_optimization_container(model), - get_network_model(get_template(model)), - get_system(model), - ) - - @info "Initializing ModelStoreParams" - IOM.init_model_store_params!(model) - set_status!(model, ModelBuildStatus.IN_PROGRESS) - end - return -end - -# Called `build_impl!(model)` in PSI (lived in emulation_model.jl). -function build_model!(model::EmulationModel{<:AbstractPowerEmulationProblem}) - build_pre_step!(model) - @info "Instantiating Network Model" - IOM.instantiate_network_model!(model) - handle_initial_conditions!(model) - build_problem!( - get_optimization_container(model), - get_template(model), - get_system(model), - ) - IOM.serialize_metadata!(get_optimization_container(model), IOM.get_output_dir(model)) - log_values(get_settings(model)) - return -end - -""" -Implementation of build for any AbstractPowerEmulationProblem - - `store_system_in_results::Bool = true`: If true, stores the system as JSON in the results HDF5 file. -""" -function build!( - model::EmulationModel{<:AbstractPowerEmulationProblem}; - executions = 1, - output_dir::String, - recorders = [], - console_level = Logging.Error, - file_level = Logging.Info, - disable_timer_outputs = false, - store_system_in_results = true, -) - mkpath(output_dir) - IOM.set_output_dir!(model, output_dir) - IOM.set_console_level!(model, console_level) - IOM.set_file_level!(model, file_level) - TimerOutputs.reset_timer!(BUILD_PROBLEMS_TIMER) - disable_timer_outputs && TimerOutputs.disable_timer!(BUILD_PROBLEMS_TIMER) - file_mode = "w" - IOM.add_recorders!(model, recorders) - IOM.register_recorders!(model, file_mode) - logger = IS.configure_logging( - get_internal(model), - IOM.PROBLEM_LOG_FILENAME, - file_mode, - ) - if store_system_in_results - @warn "store_system_in_results is set to true. This will do nothing unless a Simulation is being built." - end - try - Logging.with_logger(logger) do - try - IOM.set_executions!(model, executions) - TimerOutputs.@timeit BUILD_PROBLEMS_TIMER "Problem $(get_name(model))" begin - build_model!(model) - end - set_status!(model, ModelBuildStatus.BUILT) - @info "\n$(BUILD_PROBLEMS_TIMER)\n" - catch e - set_status!(model, ModelBuildStatus.FAILED) - bt = catch_backtrace() - @error "EmulationModel Build Failed" exception = e, bt - end - end - finally - IOM.unregister_recorders!(model) - close(logger) - end - return IOM.get_status(model) -end - -function reset!(model::EmulationModel{<:AbstractPowerEmulationProblem}) - if built_for_recurrent_solves(model) - IOM.set_execution_count!(model, 0) - end - IOM.set_container!( - get_internal(model), - OptimizationContainer( - get_system(model), - get_settings(model), - nothing, - PSY.SingleTimeSeries, - ), - ) - IOM.set_initial_conditions_model_container!(get_internal(model), nothing) - IOM.empty_time_series_cache!(model) - empty!(get_store(model)) - set_status!(model, ModelBuildStatus.EMPTY) - return -end - -function _progress_meter_enabled() - return isa(stderr, Base.TTY) && - (get(ENV, "CI", nothing) != "true") && - (get(ENV, "RUNNING_SIENNA_TESTS", nothing) != "true") -end - -# Called `run_impl!` in PSI (lived in emulation_model.jl). -function execute_emulation!( - model::EmulationModel; - optimizer = nothing, - enable_progress_bar = _progress_meter_enabled(), - kwargs..., -) - IOM._pre_solve_model_checks(model, optimizer) - internal = get_internal(model) - executions = IOM.get_executions(internal) - # Temporary check. Needs better way to manage re-runs of the same model - if internal.execution_count > 0 - error("Call build! again") - end - prog_bar = ProgressMeter.Progress(executions; enabled = enable_progress_bar) - initial_time = get_initial_time(model) - for execution in 1:executions - TimerOutputs.@timeit RUN_OPERATION_MODEL_TIMER "Run execution" begin - # NOTE: PSI's run_impl! calls update_model!(model) here to update parameters - # and initial conditions between executions. That logic lives in IOM's - # emulation_model.jl and is invoked by PSI during simulation. Standalone - # emulation in POM does not update between steps. - IOM.solve_model!(model) - current_time = initial_time + (execution - 1) * get_resolution(model) - write_outputs!(get_store(model), model, execution, current_time) - IOM.write_optimizer_stats!( - get_store(model), - get_optimizer_stats(model), - execution, - ) - IOM.advance_execution_count!(model) - ProgressMeter.update!( - prog_bar, - IOM.get_execution_count(model); - showvalues = [(:Execution, execution)], - ) - end - end - return -end - -""" -Default run method for problems that conform to the requirements of -EmulationModel{<: AbstractPowerEmulationProblem} - -This will call `build!` on the model if it is not already built. It will forward all -keyword arguments to that function. - -# Arguments - - - `model::EmulationModel = model`: Emulation model - - `optimizer::MOI.OptimizerWithAttributes`: The optimizer that is used to solve the model - - `executions::Int`: Number of executions for the emulator run - - `export_problem_outputs::Bool`: If true, export OptimizationProblemOutputs DataFrames to CSV files. - - `output_dir::String`: Required if the model is not already built, otherwise ignored - - `enable_progress_bar::Bool`: Enables/Disable progress bar printing - - `export_optimization_problem::Bool`: If true, serialize the model to a file to allow re-execution later. - - `store_system_in_results::Bool = true`: If true, stores the system as JSON in the results HDF5 file. - -# Examples - -```julia -status = run!(model; optimizer = HiGHS.Optimizer, executions = 10) -status = run!(model; output_dir = ./model_output, optimizer = HiGHS.Optimizer, executions = 10) -``` -""" -function run!( - model::EmulationModel{<:AbstractPowerEmulationProblem}; - export_problem_outputs = false, - console_level = Logging.Error, - file_level = Logging.Info, - disable_timer_outputs = false, - export_optimization_problem = true, - enable_progress_bar = _progress_meter_enabled(), - store_system_in_results = true, - kwargs..., -) - if store_system_in_results - @warn "store_system_in_results is set to true. This will do nothing unless a Simulation is being built." - end - build_if_not_already_built!( - model; - console_level = console_level, - file_level = file_level, - disable_timer_outputs = disable_timer_outputs, - kwargs..., - ) - IOM.set_console_level!(model, console_level) - IOM.set_file_level!(model, file_level) - TimerOutputs.reset_timer!(RUN_OPERATION_MODEL_TIMER) - disable_timer_outputs && TimerOutputs.disable_timer!(RUN_OPERATION_MODEL_TIMER) - file_mode = "a" - IOM.register_recorders!(model, file_mode) - logger = IS.configure_logging( - get_internal(model), - IOM.PROBLEM_LOG_FILENAME, - file_mode, - ) - try - Logging.with_logger(logger) do - try - IOM.initialize_storage!( - get_store(model), - get_optimization_container(model), - IOM.get_store_params(model), - ) - TimerOutputs.@timeit RUN_OPERATION_MODEL_TIMER "Run" begin - execute_emulation!( - model; - enable_progress_bar = enable_progress_bar, - kwargs..., - ) - IOM.set_run_status!(model, RunStatus.SUCCESSFULLY_FINALIZED) - end - if export_optimization_problem - TimerOutputs.@timeit RUN_OPERATION_MODEL_TIMER "Serialize" begin - serialize_optimization_model(model) - end - end - TimerOutputs.@timeit RUN_OPERATION_MODEL_TIMER "Outputs processing" begin - outputs = OptimizationProblemOutputs(model) - serialize_outputs(outputs, IOM.get_output_dir(model)) - export_problem_outputs && export_outputs(outputs) - end - @info "\n$(RUN_OPERATION_MODEL_TIMER)\n" - catch e - @error "Emulation Problem Run failed" exception = (e, catch_backtrace()) - IOM.set_run_status!(model, RunStatus.FAILED) - end - end - finally - IOM.wait_for_serialization!(model) - IOM.unregister_recorders!(model) - close(logger) - end - return IOM.get_run_status(model) -end - -function handle_initial_conditions!(model::EmulationModel{<:AbstractPowerEmulationProblem}) - TimerOutputs.@timeit BUILD_PROBLEMS_TIMER "Model Initialization" begin - if isempty(get_template(model)) - return - end - settings = get_settings(model) - initialize_model = get_initialize_model(settings) - deserialize_initial_conditions = get_deserialize_initial_conditions(settings) - serialized_initial_conditions_file = IOM.get_initial_conditions_file(model) - custom_init_file = get_initialization_file(settings) - - if !initialize_model && deserialize_initial_conditions - throw( - IS.ConflictingInputsError( - "!initialize_model && deserialize_initial_conditions", - ), - ) - elseif !initialize_model && !isempty(custom_init_file) - throw(IS.ConflictingInputsError("!initialize_model && initialization_file")) - end - - if !initialize_model - @info "Skip build of initial conditions" - return - end - - if !isempty(custom_init_file) - if !isfile(custom_init_file) - error("initialization_file = $custom_init_file does not exist") - end - if abspath(custom_init_file) != abspath(serialized_initial_conditions_file) - cp(custom_init_file, serialized_initial_conditions_file; force = true) - end - end - - if deserialize_initial_conditions && isfile(serialized_initial_conditions_file) - IOM.set_initial_conditions_data!( - get_optimization_container(model), - Serialization.deserialize(serialized_initial_conditions_file), - ) - @info "Deserialized initial_conditions_data" - else - @info "Make Initial Conditions Model" - build_initial_conditions!(model) - solve_and_write_initial_conditions!(model) - end - IOM.set_initial_conditions_model_container!( - get_internal(model), - nothing, - ) - end - return -end - -function validate_template(::EmulationModel{M}) where {M <: AbstractPowerEmulationProblem} - error("validate_template is not implemented for EmulationModel{$M}") -end - -function validate_template(model::EmulationModel{<:GenericPowerEmulationProblem}) - validate_template_impl!(model) - return -end diff --git a/test/Project.toml b/test/Project.toml index 7a61d08..334d706 100644 --- a/test/Project.toml +++ b/test/Project.toml @@ -34,7 +34,7 @@ TimerOutputs = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f" UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" [sources] -InfrastructureOptimizationModels = {rev = "main", url = "https://github.com/Sienna-Platform/InfrastructureOptimizationModels.jl"} +InfrastructureOptimizationModels = {rev = "ac/move-op-models", url = "https://github.com/Sienna-Platform/InfrastructureOptimizationModels.jl"} InfrastructureSystems = {rev = "IS4", url = "https://github.com/Sienna-Platform/InfrastructureSystems.jl"} PowerSystems = {rev = "psy6", url = "https://github.com/Sienna-Platform/PowerSystems.jl"} PowerFlows = {rev = "psy6", url = "https://github.com/Sienna-Platform/PowerFlows.jl"} diff --git a/test/test_device_thermal_generation_constructors.jl b/test/test_device_thermal_generation_constructors.jl index 49b25d0..169dca5 100644 --- a/test/test_device_thermal_generation_constructors.jl +++ b/test/test_device_thermal_generation_constructors.jl @@ -933,7 +933,7 @@ end set_device_model!(template, ThermalStandard, ThermalStandardDispatch) set_device_model!(template, PowerLoad, StaticPowerLoad) ED = DecisionModel( - GenericPowerDecisionProblem, + GenericPowerOperationProblem, template, ramp_test_sys; optimizer = HiGHS_optimizer, @@ -948,7 +948,7 @@ end @testset "Solving UC with CopperPlate for testing Duration Constraints" begin template = get_thermal_standard_uc_template() UC = DecisionModel( - GenericPowerDecisionProblem, + GenericPowerOperationProblem, template, PSB.build_system(PSITestSystems, "c_duration_test"); optimizer = HiGHS_optimizer, diff --git a/test/test_utils.jl b/test/test_utils.jl index 8cc84fb..5ddc37a 100644 --- a/test/test_utils.jl +++ b/test/test_utils.jl @@ -4,7 +4,7 @@ end @testset "Test ProgressMeter" begin - @test !POM._progress_meter_enabled() + @test !IOM._progress_meter_enabled() end @testset "Axis Array to DataFrame" begin diff --git a/test/test_utils/mock_operation_models.jl b/test/test_utils/mock_operation_models.jl index 97ccdc3..57104cc 100644 --- a/test/test_utils/mock_operation_models.jl +++ b/test/test_utils/mock_operation_models.jl @@ -1,10 +1,10 @@ # NOTE: None of the models and function in this file are functional. All of these are used for testing purposes and do not represent valid examples either to develop custom # models. Please refer to the documentation. -struct MockOperationProblem <: GenericPowerDecisionProblem end -struct MockEmulationProblem <: GenericPowerEmulationProblem end -struct EconomicDispatchProblem <: GenericPowerDecisionProblem end -struct UnitCommitmentProblem <: GenericPowerDecisionProblem end +struct MockOperationProblem <: GenericPowerOperationProblem end +struct MockEmulationProblem <: GenericPowerOperationProblem end +struct EconomicDispatchProblem <: GenericPowerOperationProblem end +struct UnitCommitmentProblem <: GenericPowerOperationProblem end function IOM.DecisionModel( ::Type{MockOperationProblem},