Fix #11615 - Virtualenv leaks into PythonEngine + share common code between PythonEngine and PluginManager by jmarrec · Pull Request #11649 · NatLabRockies/EnergyPlus

jmarrec · 2026-06-19T10:49:26Z

Pull request overview

Fix Virtualenv leaks into PythonEngine #11615
Share common functions initPython, addToPythonPath and reportPythonError between PluginManager and PythonEngine
- Maintaining one proper code path is much easier, and improve readability in both files

Description of the purpose of this PR

Pull Request Author

Reviewer

…tter<PyStatus>

… DEBUG_PYTHON_CONFIG

…appening

…rted as my venv python! sys.executable=/home/julien/Virtualenvs/py312/bin/python

…char_t) ``` PyPreConfig initialized: PyPreConfig( parse_argv=1 isolated=0 use_environment=1 configure_locale=1 coerce_c_locale=-1 coerce_c_locale_warn=-1 utf8_mode=-1 dev_mode=-1 allocator=0 ) Final PyPreConfig: PyPreConfig( parse_argv=1 isolated=0 use_environment=0 configure_locale=1 coerce_c_locale=-1 coerce_c_locale_warn=-1 utf8_mode=1 dev_mode=-1 allocator=0 ) Isolated config initialized: PyConfig( isolated=1 use_environment=0 site_import=1 program_name=(null) home=(null) base_prefix=(null) prefix=(null) exec_prefix=(null) base_exec_prefix=(null) pythonpath_env=(null) executable=(null) module_search_paths_set=0, module_search_paths.length=0 ) Final PyConfig: PyConfig( isolated=1 use_environment=0 site_import=0 program_name=energyplus home=/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib base_prefix=/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib prefix=(null) exec_prefix=(null) base_exec_prefix=(null) pythonpath_env=(null) executable=/Users/julien/Software/Others/EnergyPlus-build-release/Products/energyplus-26.2.0 module_search_paths_set=1, module_search_paths.length=3 module_search_paths[0]=/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib module_search_paths[1]=/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/lib-dynload module_search_paths[2]=/Users/julien/Software/Others/EnergyPlus-build-release/Products ) sys.executable=/Users/julien/Software/Others/EnergyPlus-build-release/Products/energyplus-26.2.0 sys.version=3.12.2 (main, Dec 2 2024, 16:58:41) [Clang 16.0.0 (clang-1600.0.26.4)] {'sys.base_exec_prefix': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib', 'sys.base_prefix': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib', 'sys.exec_prefix': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib', 'sys.path': ['/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib', '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib', '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/lib-dynload', '/Users/julien/Software/Others/EnergyPlus-build-release/Products'], 'sys.prefix': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib'} {'data': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib', 'include': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/include/python3.12', 'platinclude': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/include/python3.12', 'platlib': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/lib/python3.12/site-packages', 'platstdlib': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/lib/python3.12', 'purelib': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/lib/python3.12/site-packages', 'scripts': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/bin', 'stdlib': '/Users/julien/Software/Others/EnergyPlus-build-release/Products/python_lib/lib/python3.12'} ```

…ror between PluginManager and PythonEngine

jmarrec · 2026-06-19T10:50:35Z

+# Debug Python Config
+option(DEBUG_PYTHON_CONFIG "Debug Python PyPreConfig and PyConfig initialization" OFF)
+mark_as_advanced(DEBUG_PYTHON_CONFIG)
+if(DEBUG_PYTHON_CONFIG)
+  # Linking PUBLIC so it reaches energyplusapi too
+  target_compile_definitions(energypluslib PUBLIC DEBUG_PYTHON_CONFIG=1)
+endif()


New advanced option, OFF by default. I could just remove this if you'd prefer, but I'm guessing this might not be the last time we have to debug this

jmarrec · 2026-06-19T10:51:16Z

+#        if DEBUG_PYTHON_CONFIG
+            cmd += R"python(
+import sys, sysconfig, pprint
+print(f"sys.executable={sys.executable}")
+print(f"sys.version={sys.version}")
+pprint.pprint({"sys.path": sys.path,
+             "sys.prefix": sys.prefix,
+             "sys.base_prefix": sys.base_prefix,
+             "sys.exec_prefix": sys.exec_prefix,
+             "sys.base_exec_prefix": sys.base_exec_prefix})
+pprint.pprint(sysconfig.get_paths())
+)python";
+#        endif


Extra debug if DEBUG_PYTHON_CONFIG=ON (non default) when you run
./Products/energyplus auxiliary updater

jmarrec · 2026-06-19T10:53:01Z

+#ifndef EPLUS_PYTHON_HELPERS_HH
+#define EPLUS_PYTHON_HELPERS_HH
+
+#if !LINK_WITH_PYTHON
+#    error "This file should only be included when LINK_WITH_PYTHON is defined"
+#endif


New helper, will throw if included when LINK_WITH_PYTHON is off.

jmarrec · 2026-06-19T10:53:31Z

+// Third Party Headers
+#ifdef _DEBUG
+// We don't want to try to import a debug build of Python here
+// so if we are building a Debug build of the C++ code, we need
+// to undefine _DEBUG during the #include command for Python.h.
+// Otherwise it will fail
+#    undef _DEBUG
+#    include <Python.h>
+#    define _DEBUG
+#else
+#    include <Python.h>
+#endif
+
+// C++ Headers
+#if DEBUG_PYTHON_CONFIG
+#    include <cwchar>
+#endif
+#include <format>
+
+#include <EnergyPlus/Formatters.hh>
+
+template <> struct std::formatter<PyStatus>
+{
+    // parse is inherited from formatter<string_view>.
+    constexpr auto parse(std::format_parse_context &ctx) -> std::format_parse_context::iterator
+    {
+        return ctx.begin();
+    }
+
+    template <typename FormatContext> auto format(const PyStatus &status, FormatContext &ctx) const
+    {
+        if (PyStatus_Exception(status) == 0) {
+            return ctx.out();
+        }
+        if (PyStatus_IsExit(status) != 0) {
+            return std::format_to(ctx.out(), "Exited with code {}", status.exitcode);
+        }
+        if (PyStatus_IsError(status) != 0) {
+            auto it = ctx.out();
+            it = std::format_to(it, "Fatal Python error: ");
+            if (status.func != nullptr) {
+                it = std::format_to(it, "{}: ", status.func);
+            }
+            it = std::format_to(it, "{}", status.err_msg);
+            return it;
+        }
+        return ctx.out();
+    }
+};


PyStatus formatter was in both PluginManager and PythonEngine cc files, now it's here.

jmarrec · 2026-06-19T10:53:53Z

+#if DEBUG_PYTHON_CONFIG
+static std::string narrowWide(const wchar_t *ws)
+{
+    if (ws == nullptr) {
+        return "(null)";
+    }
+    std::mbstate_t state{};
+    const wchar_t *tmp = ws;
+#    ifndef NDEBUG
+    // LC_TYPE is set to C.UTF-8 in PyPreConfig_InitPythonConfig already
+    const char *ctype = std::setlocale(LC_CTYPE, nullptr);
+    assert(ctype != nullptr && (std::strstr(ctype, "UTF-8") != nullptr || std::strstr(ctype, "utf8") != nullptr));
+#    endif
+
+    std::size_t n = std::wcsrtombs(nullptr, &tmp, 0, &state);
+    if (n == static_cast<std::size_t>(-1)) {
+        return "(conversion error)";
+    }
+    std::string result(n, '\0');
+    n = std::wcsrtombs(result.data(), &ws, n, &state);
+    if (n == static_cast<std::size_t>(-1)) {
+        return "(conversion error)";
+    }
+    return result;
+}
+
+template <> struct std::formatter<PyPreConfig>
+{
+    constexpr auto parse(std::format_parse_context &ctx) -> std::format_parse_context::iterator
+    {
+        return ctx.begin();
+    }
+
+    auto format(const PyPreConfig &c, std::format_context &ctx) const -> std::format_context::iterator
+    {
+        auto it = ctx.out();
+        it = std::format_to(it, "PyPreConfig(\n");
+        it = std::format_to(it, "  parse_argv={}\n", c.parse_argv);
+        it = std::format_to(it, "  isolated={}\n", c.isolated);
+        it = std::format_to(it, "  use_environment={}\n", c.use_environment);
+        it = std::format_to(it, "  configure_locale={}\n", c.configure_locale);
+        it = std::format_to(it, "  coerce_c_locale={}\n", c.coerce_c_locale);
+        it = std::format_to(it, "  coerce_c_locale_warn={}\n", c.coerce_c_locale_warn);
+        it = std::format_to(it, "  utf8_mode={}\n", c.utf8_mode);
+        it = std::format_to(it, "  dev_mode={}\n", c.dev_mode);
+        it = std::format_to(it, "  allocator={}\n", c.allocator);
+#    ifdef MS_WINDOWS
+        it = std::format_to(it, "  legacy_windows_fs_encoding={}\n", c.legacy_windows_fs_encoding);
+#    endif
+        it = std::format_to(it, ")");
+        return it;
+    }
+};
+
+template <> struct std::formatter<PyConfig>
+{
+    constexpr auto parse(std::format_parse_context &ctx)
+    {
+        return ctx.begin();
+    }
+
+    std::format_context::iterator format(const PyConfig &c, std::format_context &ctx) const
+    {
+        auto it = ctx.out();
+
+        it = std::format_to(it, "PyConfig(\n");
+        it = std::format_to(it, "  isolated={}\n", c.isolated);
+        it = std::format_to(it, "  use_environment={}\n", c.use_environment);
+        it = std::format_to(it, "  site_import={}\n", c.site_import);
+        it = std::format_to(it, "  program_name={}\n", narrowWide(c.program_name));
+        it = std::format_to(it, "  home={}\n", narrowWide(c.home));
+        it = std::format_to(it, "  base_prefix={}\n", narrowWide(c.base_prefix));
+        it = std::format_to(it, "  prefix={}\n", narrowWide(c.prefix));
+        it = std::format_to(it, "  exec_prefix={}\n", narrowWide(c.exec_prefix));
+        it = std::format_to(it, "  base_exec_prefix={}\n", narrowWide(c.base_exec_prefix));
+        it = std::format_to(it, "  pythonpath_env={}\n", narrowWide(c.pythonpath_env));
+        it = std::format_to(it, "  executable={}\n", narrowWide(c.executable));
+
+        it = std::format_to(
+            it, "  module_search_paths_set={}, module_search_paths.length={}\n", c.module_search_paths_set, c.module_search_paths.length);
+        if (c.module_search_paths.items != nullptr) {
+            for (Py_ssize_t i = 0; i < c.module_search_paths.length; ++i) {
+                it = std::format_to(it, "    module_search_paths[{}]={}\n", i, narrowWide(c.module_search_paths.items[i]));
+            }
+        }
+        it = std::format_to(it, ")");
+        return it;
+    }
+};
+#endif // DEBUG_PYTHON_CONFIG


If DEBUG_PYTHON_CONFIG, we define PyPreConfig and PyConfig formatters

jmarrec · 2026-06-19T10:57:45Z

+        PyWcharPath wcharProgramPath(FileSystem::getAbsolutePath(FileSystem::getProgramPath()));
+        status = PyConfig_SetString(&config, &config.executable, wcharProgramPath);
+        if (PyStatus_Exception(status) != 0) {
+            ShowFatalError(state, std::format("Could not initialize executable on PyConfig... {}", status));
+        }
+
+        status = PyConfig_SetString(&config, &config.program_name, L"energyplus");
+        if (PyStatus_Exception(status) != 0) {
+            ShowFatalError(state, std::format("Could not initialize program_name on PyConfig... {}", status));
+        }


This used to be in PythonEngine

status = PyConfig_SetBytesString(&config, &config.program_name, PluginManagement::programName);

programName was "python", and that's really the main problem that was happening, during site_import since the executable was not set it was resolving to my Venv python.

jmarrec · 2026-06-19T10:58:12Z

+        }
+
+        // Ensure site.py doesn't run, this picks up your virtualenv, even with isolated config
+        // config.site_import = 0;


Disable site_import works. But actually sys.executable is reported as my venv python, which is wrong, so I prefered the fix to set the executable. I'm not 100% sure of the side effects of not running site_import, so I commented it out.

jmarrec · 2026-06-19T10:58:39Z

+        // This is the other related line that caused Decent CI to start having trouble.  I'm putting it back to
+        // PyPreConfig_InitPythonConfig, even though I think it should be isolated.  Will deal with this after IO freeze.


this is an existing comment from Myoldmopar.

jmarrec · 2026-06-19T10:58:51Z

+        // PyPreConfig_InitIsolatedConfig sets configure_locale=0 which likely caused Decent CI failures
+        // https://github.com/python/cpython/blob/v3.12.2/Python/preconfig.c#L310-L345


That's a new comment.

jmarrec · 2026-06-19T11:00:21Z

+        // with config and get IO freeze going, then get back to solving it.
+        // Py_Initialize();
+        Py_InitializeFromConfig(&config);
+        PyConfig_Clear(&config);


New, this is recommended

https://docs.python.org/3/c-api/init_config.html#c.PyConfig

type PyConfig

Structure containing most parameters to configure Python.

When done, the PyConfig_Clear() function must be used to release the configuration memory.

jmarrec added 8 commits June 19, 2026 11:25

Start sharing some code between PythonEngine and PluginManager: forma…

0dff632

…tter<PyStatus>

Add a formatter for PyPreCOnfig/PyConfig based on new advanced option…

32a55bf

… DEBUG_PYTHON_CONFIG

Debug #11615 - ./Products/energyplus auxiliary updater shows what's h…

e3d8fa0

…appening

Try use_environment=0 in PreConfig - doesn't work

b6f088d

Disable site_import, which works. But actually sys.executable is repo…

8cfd359

…rted as my venv python! sys.executable=/home/julien/Virtualenvs/py312/bin/python

Remove Preconfig use_environmen and Config site_import

cea24e9

Share common functions initPython, addToPythonPath and reportPythonEr…

c8f5328

…ror between PluginManager and PythonEngine

jmarrec self-assigned this Jun 19, 2026

jmarrec added Defect Includes code to repair a defect in EnergyPlus NotIDDChange Code does not impact IDD (can be merged after IO freeze) python Pull requests that update Python code labels Jun 19, 2026

jmarrec commented Jun 19, 2026

View reviewed changes

Cleanup outdated comments

3bcfed0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #11615 - Virtualenv leaks into PythonEngine + share common code between PythonEngine and PluginManager#11649

Fix #11615 - Virtualenv leaks into PythonEngine + share common code between PythonEngine and PluginManager#11649
jmarrec wants to merge 9 commits into
developfrom
11615_Virtualenv_PythonEngine

jmarrec commented Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

jmarrec Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// This is the other related line that caused Decent CI to start having trouble. I'm putting it back to
		// PyPreConfig_InitPythonConfig, even though I think it should be isolated. Will deal with this after IO freeze.

		// PyPreConfig_InitIsolatedConfig sets configure_locale=0 which likely caused Decent CI failures
		// https://github.com/python/cpython/blob/v3.12.2/Python/preconfig.c#L310-L345

Conversation

jmarrec commented Jun 19, 2026

Pull request overview

Description of the purpose of this PR

Pull Request Author

Reviewer

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants