Closes #22603: Add experimental Python packaging support for NetBox

[pheus]

Closes: #22603

This PR adds the code and release infrastructure needed to build and test NetBox as an official Python package.

The packaged install path is experimental during the TestPyPI validation phase. Existing archive and git-clone workflows remain unchanged and continue to be the supported production installation methods until the packaged workflow is promoted.

This PR targets TestPyPI only. Production PyPI publishing is intentionally deferred until the packaged installation workflow is documented and promoted.

This PR covers:

• package build support;
• package metadata generation;
• package artifact layout;
• packaged-install runtime support;
• configuration discovery for packaged installs;
• one installed netbox command;
• pre-Django command handling for version output, setup, and secret key generation;
• upgrade task support for packaged deployments through netbox upgrade;
• package verification and smoke testing;
• TestPyPI publishing workflow.

This PR does not:

• replace the current archive or git-clone installation workflows;
• change NetBox’s plugin model;
• install or configure external services;
• automate service or web server configuration;
• add a separate netbox install command;
• add a second bootstrap binary;
• add user-facing installation documentation;
• enable production PyPI publishing.

The goal is to make the package artifacts buildable, testable, and safe to iterate on before the packaged install workflow is documented and promoted.

[pheus]

@claude /review Only report on issues of potential concern; we don't need commentary on anything that doesn't require further consideration.

[github-actions]

Claude finished @pheus's task in 2m 27s —— View job

---

Code Review

• Read changed files and gather context
• Analyze security concerns (CodeQL flags)
• Review CLI, scaffold, and settings utilities
• Review packaging/release workflow
• Review tests
• Post findings

---

Issues of Concern

1. Bug: Wrong command for docs build (upgrade_tasks.py:69)

subprocess.run(['zensical', 'build'], cwd=docs_root, check=True)

zensical is not a known tool. The project uses MkDocs (mkdocs build), as documented in AGENTS.md. This will raise FileNotFoundError at runtime whenever --build-docs is passed. Should be:

subprocess.run(['mkdocs', 'build'], cwd=docs_root, check=True)

Fix this →

---

2. CodeQL warnings — true false positives, but worth suppressing explicitly

Both flagged lines are intentional: netbox secret-key and generate_secret_key.py exist specifically to print a freshly-generated key to stdout for the administrator to copy. The key is newly generated and hasn't been used anywhere yet. There's no real information disclosure here.

However, since these warnings will block or clutter future CodeQL runs, suppressing them explicitly with a comment would be cleaner:

• netbox/generate_secret_key.py:5
• netbox/netbox/cli.py:63

A # noqa or # nosec comment (depending on which scanner matters most) would silence these permanently. Without a suppression, every future CodeQL scan will flag these.

---

3. PyPI classifier overstates stability

pyproject.toml:22:

"Development Status :: 5 - Production/Stable",

The PR explicitly describes this as experimental during a TestPyPI validation phase. 5 - Production/Stable will be the status pip shows to anyone who installs via pip install netbox. 4 - Beta or 3 - Alpha would be more appropriate while the packaged install path is still experimental. Once the workflow is promoted, it can be bumped to 5.

---

4. Smoke test skipped on pull requests

release.yml:138:

if: github.event_name != 'pull_request'

The smoke test (wheel install + database migration + netbox setup roundtrip) is skipped on every PR. This means a PR that breaks the wheel installation passes CI but only fails when the tag is pushed for release. Since publish-testpypi gates on smoke-test, the breakage would be caught before publishing — but the feedback loop is slow (requires a full tag push). Given that the smoke-test job is the most meaningful validation of the wheel install, it's worth documenting why PRs are excluded (cost/time) or adding a lighter alternative that runs on PRs.

---

5. Minor: sys.path.remove() removes first occurrence, not the inserted one (settings_utils.py:55-56)

sys.path.insert(0, module_dir)
...
finally:
    if module_dir in sys.path:
        sys.path.remove(module_dir)

list.remove() removes the first occurrence. If module_dir already existed elsewhere in sys.path before the insert(0, ...), the remove() would discard that pre-existing entry instead of the one we added, leaving our inserted entry in place. This is an unlikely edge case (config dirs are usually not in sys.path), but a safer pattern is to pop by index (sys.path.pop(0) if sys.path[0] == module_dir).

---

[pheus]

1. zensical build is intentional. NetBox switched from MkDocs to Zensical in v4.6.
2. Agreed, these are false positives. Both commands intentionally print a newly generated, unused secret key for the admin to copy.
3. I’d keep Production/Stable. NetBox itself is stable; only this new pip installation path is experimental.
4. Fair point. The publish job is still gated by the smoke test, but we can address PR smoke-test coverage or document why it’s skipped.
5. Good catch. I’ll update this to remove only the sys.path entry inserted by the helper.

[jeremystretch]

Nice work @pheus! I think we should talk through the wheel vs. checkout layout considerations; I'm sure there's a good reason for the disparity but it seems very fragile IMO.

[jeremystretch]

I'd like to dig into why this is. IMO it feels very fragile to accommodate two different file hierarchies.

[pheus]

The split comes from the wheel layout: immutable application/package data lives in the installed Python package, while mutable instance-local files need to live outside the wheel under the instance root. For checkout installs, the existing source tree layout remains unchanged.

I can try to reduce the fragility here by centralizing the install-mode/path decisions more clearly instead of spreading wheel-vs-checkout checks through settings.

[jeremystretch]

This seems largely redundant to upgrade.sh, which is retained in its original form. Should we update upgrade.sh to wrap the new manage.py upgrade command? I'd like to avoid having to maintain both avenues in parallel.

[pheus]

Good point. My intention is not to alter upgrade.sh during the NetBox v4.6.x release series, and instead switch the script to manage.py upgrade in NetBox v4.7.0.

That keeps the existing source/archive upgrade path unchanged for now, while still allowing us to test the management command and adopt it later.

[pheus]

Thank you for the thoughtful review! I appreciate you taking the time to go through the packaging approach in detail.

Wheel vs. checkout layout

The main reason for the split is that a wheel installs NetBox’s application code and bundled package data into the Python environment, while instance-local files like configuration.py, media, reports, scripts, and collected static files still need to live outside the package. This separates immutable application data in site-packages/ from mutable instance data.

There is also an import-path issue with the current configuration loading model. Today, NetBox loads its default configuration as the Python module netbox.configuration, which maps to the historical source/archive file path netbox/netbox/configuration.py. For a wheel install, however, netbox is the installed package from the virtual environment, so that import resolves against site-packages, not against an instance-local /opt/netbox/netbox/configuration.py file.

That is why the wheel install uses an instance-local conf/configuration.py file instead. Because this file is intentionally outside the installed netbox package, it has to be loaded by path rather than through the normal package import mechanism.

I did keep a fallback for the legacy source-style path for a grace period, but making that the default for wheel installs would require users to create a mostly empty netbox/netbox/ hierarchy under the instance root just to hold configuration. I wanted to avoid introducing that artificial structure for new pip-installed instances.

My goal was to keep the current checkout/archive layout unchanged and only introduce the separate instance root for wheel installs. Changing the configuration path for all installation methods could be discussed separately, but I think that would be better handled as a breaking-change release rather than as part of this packaging PR.

[pheus]

@claude /review Only report on issues of potential concern; we don't need commentary on anything that doesn't require further consideration.

[github-actions]

Claude finished @pheus's task in 4m 7s —— View job

---

Code Review

• Read key changed files
• Review settings_utils.py (sys.path fix, configuration loading)
• Review cli.py (CodeQL warnings)
• Review scaffold.py (scope concerns)
• Review settings.py (dual-path layout)
• Review release.yml (workflow)
• Review upgrade management command
• Post findings

---

Issues of Concern

1. Built docs write into site-packages — will break in restricted venvs and is clobbered by upgrade (unresolved maintainer concern)

upgrade.py:67-72 + settings_utils.py:47-48

In wheel mode, netbox upgrade --build-docs runs zensical build with cwd=<site-packages>/netbox/netbox/_data/. The site_dir in mkdocs.yml is netbox/project-static/docs (relative), so built docs land at:

<site-packages>/netbox/netbox/_data/netbox/project-static/docs/

Two concrete problems:

• Permissions: Venvs installed system-wide (e.g. as root via sudo pip install) will reject writes to site-packages from an unprivileged user running netbox upgrade.
• Clobbered on upgrade: pip install --upgrade netbox replaces the entire package directory. Any docs built there are deleted; the user must re-run --build-docs after every pip upgrade to restore UI-embedded docs.

The collectstatic step then copies them to NETBOX_ROOT/static/docs/ which persists, so the web server is fine. But the build step writing into site-packages is the problem. jeremystretch's comment — "We need to figure out a way to include the docs to ensure support for the UI-embedded documentation" — remains open. A plain pip install netbox will have no working UI docs until the user manually runs netbox upgrade --build-docs. Pre-building the docs as part of the release and including the output in the wheel would fix this.

---

2. CodeQL false positive still firing — no suppression added after author agreed to add one

Prior review identified cli.py (printing newly-generated key) as a CodeQL false positive. The author agreed to add a suppression. The latest commit didn't add it, and CodeQL has now re-flagged line 89 of cli.py in the most recent scan:

print(generate_secret_key())  # line 89 — no suppression

generate_secret_key.py:5 (print(generate_secret_key())) also still lacks suppression. Both were flagged in the prior review and the author agreed they needed it. Without suppression, every future scan will open a new alert.

Fix this →

---

3. netbox setup scope — maintainer objection not yet resolved

scaffold.py — jeremystretch said: "I feel this is out of scope for the goal of simply shipping a Python package. We should not take on administration of deployment-specific configurations." The author replied clarifying that it only copies templates verbatim (doesn't configure services), but the maintainer has not acknowledged this as acceptable. This appears to be an open design question that needs an explicit resolution before merge.

---

4. upgrade.py:_docs_source_root() silently skips docs build when mkdocs.yml is absent

def _docs_source_root():
    for candidate in (os.path.dirname(settings.BASE_DIR), settings.BASE_DIR):
        if os.path.isfile(os.path.join(candidate, 'mkdocs.yml')):
            return candidate
    return None

If mkdocs.yml is not found (returns None), the command prints a WARNING and silently continues (upgrade.py:68-69). In a fresh pip install where docs haven't been built yet, this will print the warning and skip docs regardless of --build-docs. The smoke test checks for this warning (line 255 of release.yml) but in practice a user who passes --build-docs expecting docs to be built would get no error — just a quiet skip — if they somehow run without the bundled data. More importantly, if this goes wrong during the smoke test (e.g., bundled _data/mkdocs.yml isn't found), the test passes (no crash) but the assertion on line 255 catches the skip message and exits 1. The warning-as-silent-skip pattern could mask misconfiguration in production.

[pheus]

@claude /review Only report on issues of potential concern; we don't need commentary on anything that doesn't require further consideration.

1. The wheel no longer builds documentation inside site-packages. The documentation is rendered before the package artifacts are built, and the rendered site is included in the wheel and sdist. A wheel install now uses the bundled rendered docs, which are picked up by collectstatic, so there is no runtime write into the installed package directory.

2. Agreed this is a false positive. I have dismissed the CodeQL alert. Both commands intentionally print a freshly generated, unused secret key for the administrator to copy.

3. I have reduced the scope of netbox setup. It now only creates the minimal local instance files and copies the bundled contrib examples verbatim. It does not rewrite service files, install systemd units, configure a web server, or otherwise take ownership of deployment-specific administration. Existing files are also left untouched.

4. This behavior is now intentional. Wheel installs do not need to build documentation anymore because the rendered docs are bundled in the package. --build-docs remains useful for checkout/source installs, but for wheel installs there is no documentation source tree to build from.

[github-actions]

Claude finished @pheus's task in 4m 14s —— View job

---

Code Review

• Read key changed files
• Verify issue 1: docs bundled in wheel (no site-packages write)
• Verify issue 2: CodeQL dismissal for secret key commands
• Verify issue 3: netbox setup scope reduction
• Verify issue 4: --build-docs behavior for wheel installs
• Check for any remaining concerns
• Post findings

---

All four stated resolutions are verified in the code.

1 — Docs bundled in wheel (resolved): pyproject.toml force-includes netbox/project-static/docs into the wheel at netbox/netbox/_data/docs before the package is built. The release workflow runs zensical build -c -s as a pre-build step (release.yml:55). settings_utils.py:45-48 sets both docs_root and static_docs_root to this bundled path for wheel installs, so collectstatic picks them up without any runtime write to site-packages. The smoke test (release.yml:262-263) directly asserts static/docs/index.html and a model doc page are present after netbox upgrade, providing hard verification.

2 — CodeQL (informational): There are no inline suppression comments on cli.py:89 or generate_secret_key.py:5. GitHub's UI dismissal is a manual per-alert action; it does not prevent the scanner from re-opening an alert when the flagged line changes in a future commit. The PR history already demonstrated this: the alert was first raised on 2026-07-03, then re-raised on 2026-07-06 after new commits landed. Adding # lgtm[py/clear-text-logging-sensitive-data] to those two lines would silence the scanner unconditionally regardless of future changes. This is not a blocker, but worth noting if future commits touch those files.

3 — netbox setup scope (resolved): scaffold.py now only creates conf/__init__.py, conf/configuration.py (copied verbatim from the bundled template), local_requirements.txt, and copies the contrib deployment examples unmodified into <target>/contrib/. Existing files are left untouched (scaffold.py:31-33). Service configuration and installation remain the administrator's responsibility, as stated in the docstring.

4 — --build-docs on wheel installs (resolved): upgrade.py:_docs_source_root() returns None for wheel installs (no mkdocs.yml at dirname(BASE_DIR)), and a WARNING is printed explaining the source tree is unavailable. The smoke test confirms bundled docs are collected to STATIC_ROOT without --build-docs. This is intentional and correctly documented in the code comment (upgrade.py:19-21).

---

One process note: jeremystretch's CHANGES_REQUESTED review is still open. His individual comments on the dual-hierarchy layout fragility and the settings.py docs root question have received author responses but no explicit maintainer sign-off. The PR will need a re-review from him before it can merge.

[jeremystretch]

Just one more minor note.

[jeremystretch]

Is there a need to expose STATIC_ROOT as a configuration parameter? If so, it needs to be documented in docs/configuration/system.md. Would prefer to keep this fixed if we can, as allowing administrators to change the path feels like it's inviting breakage.

[pheus]

You're right, there is no need to expose this as a configuration parameter. I've removed it and kept STATIC_ROOT fixed. Thanks for the pushback.