Repository navigation
Add additive corporate-CA support: APM_EXTRA_CA_BUNDLE (npm NODE_EXTRA_CA_CERTS parity) #2034
Description
Activity
- added a commit that references this issue
on Jul 5, 2026 Triage decision
acceptProposed labels
theme/security area/enterprise area/docs-site type/feature status/accepted priority/highMilestone
null-- No open milestone matching the v0.24.x release series was found. This is a priority/high enterprise feature; maintainer should assign to the appropriate next-minor milestone.Suggested next action
Draft the implementation plan for the SSLContext builder and child-runtime propagation strategy (especially how Python child processes via
requestswill honor the extra CA without the REQUESTS_CA_BUNDLE replacement footgun) before opening a PR.Suggested issue comment
This is accepted -- the REQUESTS_CA_BUNDLE footgun (replaces the trust set, breaks public HTTPS for corporate CA users) is a real enterprise blocker, and NODE_EXTRA_CA_CERTS parity is the right model. The additive APM_EXTRA_CA_BUNDLE approach is the correct design. A few implementation notes worth settling before the PR opens: 1. **Parent process**: SSLContext with load_verify_locations(APM_EXTRA_CA_BUNDLE) on top of OS/certifi defaults is the right Python primitive. 2. **Node children via apm run**: NODE_EXTRA_CA_CERTS export via build_child_tls_env() is straightforward. 3. **Python children via apm run**: `requests` does not use SSLContext directly -- the cleanest propagation is writing a merged PEM bundle to a temp file and setting REQUESTS_CA_BUNDLE to it, which avoids the replacement footgun for children too. This detail is worth capturing in the PR description. The acceptance criteria (e2e test: private-CA server trusted via APM_EXTRA_CA_BUNDLE while public HTTPS on OS/certifi roots still verifies) are concrete and testable. Documenting the precedence (APM_EXTRA_CA_BUNDLE additive / REQUESTS_CA_BUNDLE replace / APM_DISABLE_TRUSTSTORE opt-out) in enterprise/security.md is a required part of the implementing PR.
Per-lens notes (collapsed)
DevX UX Expert -- User-Need Reviewer
The current state -- REQUESTS_CA_BUNDLE replaces the trust set -- is a genuine enterprise UX failure:
developers must choose between trusting their corporate CA and trusting the public internet. APM_EXTRA_CA_BUNDLE
resolves this by layering on top of OS/certifi defaults, matching the NODE_EXTRA_CA_CERTS mental model
that Node.js ecosystem developers already know. The naming is clear, the behavior is predictable, and
the acceptance criteria (e2e test) are concrete. Documenting the precedence table (additive vs replace
vs opt-out) is essential -- enterprise developers need to reason about the behavior without trial and
error.Supply Chain Security Expert -- Risk-Surface Reviewer
This touches the TLS trust surface. Key security notes:
(1) The additive approach is correct and safe -- load_verify_locations() extends, not replaces, the
trust set. No weakening of existing verification.
(2) APM_EXTRA_CA_BUNDLE path must be validated at startup: file must exist and be a readable PEM
bundle; no path traversal should be possible.
(3) Child runtime propagation requires care: exporting NODE_EXTRA_CA_CERTS for Node children is safe
(additive by design). For Python children using requests, propagating a merged PEM bundle via
REQUESTS_CA_BUNDLE avoids the replacement footgun but introduces a temp-file lifecycle question.
(4) Precedence documentation is a security-auditor requirement: enterprises need to understand which
knob wins when multiple CA env vars are set.
theme/securityapplies (TLS trust configuration).area/enterpriseis the primary area.OSS Growth Hacker -- Contributor-Tone Reviewer
Not activated -- danielmeppiel is the site admin and project maintainer; no first-time contributor tone
adjustment is needed for this issue.Python Architect -- Architecture Reviewer
Implementation requires: (1) A new TLS utility (likely in
utils/tls.pyor extending existing TLS
helpers) that builds an SSLContext loading OS/certifi defaults + APM_EXTRA_CA_BUNDLE. (2) All HTTP
requests.Sessioncreation points must use the custom SSLContext (viarequests.adapters.HTTPAdapter
with a custom ssl_context). (3)build_child_tls_env()(introduced in #2005) extended to export
NODE_EXTRA_CA_CERTS for Node children. (4) For Python children that userequests, the cleanest
propagation is writing a merged PEM bundle to a temp file and setting REQUESTS_CA_BUNDLE -- this
avoids the replacement footgun but adds a temp-file lifecycle concern. The issue is correctly scoped
and the architectural seam (#2005) exists. Nostatus/needs-designneeded for the happy path;
the Python-child propagation detail is worth capturing in the PR description before coding.Doc Writer -- Documentation Reviewer
The implementing PR needs to update
docs/src/content/docs/enterprise/security.mdto include:
(1) APM_EXTRA_CA_BUNDLE in the environment variable reference alongside REQUESTS_CA_BUNDLE and
APM_DISABLE_TRUSTSTORE, with a clear precedence table (additive vs replace vs opt-out).
(2) A note on how APM_EXTRA_CA_BUNDLE propagates to child runtimes viaapm run(Node children:
NODE_EXTRA_CA_CERTS; Python children: merged bundle).area/docs-siteis warranted as a label since
the security model documentation requires a substantive addition. The suggested comment wording
correctly flags the Python-child propagation detail as something to capture before the PR opens.APM CEO -- Triage Arbiter
Well-scoped enterprise feature from the maintainer, filling a real gap: REQUESTS_CA_BUNDLE-as-replace
is a known footgun that blocks enterprise adoption for corporate CA customers. NODE_EXTRA_CA_CERTS
parity is a competitive-positioning win (APM matches Node.js behavior on a table-stakes enterprise
requirement). Thetheme/securitylabel is correct (TLS trust configuration).area/enterpriseis
the primary area;area/docs-siterides because the security model documentation requires a
substantive update.priority/highreflects the enterprise-blocker status. The Python-child
propagation detail (how to avoid the REQUESTS_CA_BUNDLE replacement footgun for children) is worth
settling in an implementation note before the PR opens -- but it is not a design blocker for accepting
the issue. Decision: accept. Milestone: null -- no open milestone for v0.24.x; maintainer assigns.{ "decision": "accept", "decision_detail": "", "theme": "theme/security", "areas": ["area/enterprise", "area/docs-site"], "type": "type/feature", "status": "status/accepted", "priority": "priority/high", "preserved_labels": [], "milestone": null, "next_action": "Draft the implementation plan for the SSLContext builder and Python-child propagation strategy (merged PEM temp bundle vs REQUESTS_CA_BUNDLE) before opening a PR.", "comment_markdown": "This is accepted -- the REQUESTS_CA_BUNDLE footgun (replaces the trust set, breaks public HTTPS for corporate CA users) is a real enterprise blocker, and NODE_EXTRA_CA_CERTS parity is the right model. The additive APM_EXTRA_CA_BUNDLE approach is the correct design.\n\nA few implementation notes worth settling before the PR opens:\n\n1. Parent process: SSLContext with load_verify_locations(APM_EXTRA_CA_BUNDLE) on top of OS/certifi defaults is the right Python primitive.\n2. Node children via apm run: NODE_EXTRA_CA_CERTS export via build_child_tls_env() is straightforward.\n3. Python children via apm run: requests does not use SSLContext directly -- the cleanest propagation is writing a merged PEM bundle to a temp file and setting REQUESTS_CA_BUNDLE to it, which avoids the replacement footgun for children too.\n\nThe acceptance criteria (e2e test: private-CA server trusted via APM_EXTRA_CA_BUNDLE while public HTTPS on OS/certifi roots still verifies) are concrete and testable. Documenting the precedence (APM_EXTRA_CA_BUNDLE additive / REQUESTS_CA_BUNDLE replace / APM_DISABLE_TRUSTSTORE opt-out) in enterprise/security.md is a required part of the implementing PR." }
Triage status: agentic proposal pending human ratification.
Silence is approval. Maintainers can:- Override any label or milestone above by editing it directly --
human edits are authoritative and will not be reverted on
subsequent runs. - Re-trigger triage by applying the
status/needs-triagelabel, or
by removingstatus/triagedto enroll the issue in the next
daily sweep.
Posted by the Triage Panel workflow. See .apm/skills/apm-triage-panel for the panel skill.
Generated by Triage Panel · 129.2 AIC · ⌖ 22.4 AIC · ⊞ 9.2K · ◷
- Override any label or milestone above by editing it directly --
- added
on Jul 5, 2026 - added a commit that references this issue
on Jul 12, 2026 Implementation plan before opening the PR:
- Keep
src/apm_cli/core/tls_trust.pyas the single trust-policy owner. Validate a non-emptyAPM_EXTRA_CA_BUNDLEas a readable, regular, parseable PEM before any HTTP module is imported. Invalid input will fail closed with an actionable error; verification is never disabled. - Preserve explicit precedence:
APM_DISABLE_TRUSTSTORE>REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE(replacement) >APM_EXTRA_CA_BUNDLE(additive) > OS trust store > certifi fallback. - For the parent process, inject
truststoreas today, then install a scopedtruststore.SSLContextsubclass whose constructor loads the validated extra PEM. Wire that context at the existing process-wide SSL/urllib3 seam and feature-detect Requests' optional preloaded context. This covers direct Requests calls, existing Sessions, custom adapters, and proxy pools without creating a second HTTP-session authority. - For Python child runtimes, extend the shipped, dependency-free
.pthbootstrap with the same additive context behavior. This deliberately avoids the proposed merged-certifi temp file: a merged PEM cannot portably enumerate and retain Windows/macOS native roots, while the bootstrap can preserve native trust plus the extra CA without a temp-file lifecycle. - Extend
build_child_tls_env()soapm runand managed runtime subprocesses receive the normalized extra path asNODE_EXTRA_CA_CERTS, while preserving an explicit non-emptyNODE_EXTRA_CA_CERTS. Apply that seam to bothapm runsubprocess branches. Rust/Codex keeps its runtime-owned trust configuration; the PR will not claim otherwise. - Add hermetic coverage with independent loopback CAs: default trust remains valid, the extra private CA becomes valid only in additive mode, replacement variables remain replacement, disable remains highest precedence, malformed/missing/directory paths fail closed, Node env propagation is non-destructive, and a foreign Python venv verifies through the shipped bootstrap.
- Update the enterprise security model, SSL troubleshooting page, environment-variable reference, docs scope assertions, and Unreleased changelog entry.
I also checked adjacent PR #2602. It touches the same TLS owner for diagnostics and already advertises this precedence, so I will rebase before opening and fold its diagnostic/isolation behavior in if it lands first.
- Keep
Implementation-plan amendment after adversarial review:
- I changed the child strategy to APM-owned per-process snapshots under
~/.apm/tls: an extra-only PEM for Node and acertifi+extra PEM for ordinary Requests children. This makes genericapm runPython children additive without depending on the managed bootstrap and makes source mutation after validation irrelevant. Managed Python runtimes still install the bootstrap so native trust can be retained whentruststoreis available. - The corrected precedence is
REQUESTS_CA_BUNDLE>CURL_CA_BUNDLE>APM_DISABLE_TRUSTSTORE>APM_EXTRA_CA_BUNDLE> OS trust >certififallback. The disable switch suppresses APM-managed OS/additive propagation but intentionally does not unset an explicit replacement bundle. - If
truststoreinjection is unavailable, the parent merged fallback is deliberately Requests-scoped. Parent stdliburllibpaths keep their normal fallback because they do not consume Requests bundle variables.
The PR now exercises private and public roots together, child and grandchild propagation, native Node override preservation, source-bundle mutation after validation, and exact rollback after failed TLS publication.
- I changed the child strategy to APM-owned per-process snapshots under
Implementation update: the proposed implementation is in #2741, now at ec035c10.
The latest follow-up removes redundant inner rollback, consolidates the parent fallback, and shares one HTTPS test server while preserving the additive trust and child-runtime behavior described above. It also corrects Windows CA-override and Bash test assumptions. The follow-up removes a net 93 lines across eight files.
Local validation on Windows/Python 3.12.13:
- 364 affected tests passed, with no skips or deselections, including all three Windows symlink tests. Two warnings remain in the unchanged lifecycle timeout test's subprocess-reader threads.
- Ruff lint/format, Pylint duplication, architecture/auth checks, and the CI source guards passed.
- Documentation built successfully: 124 pages and 1,007 relative links checked, with no broken links.
The tests exercise real private-CA HTTPS, retention of an independent existing root, Python/Node child propagation, and recovery after an additive TLS context has actually been published. The independent-root proof uses synthetic loopback CAs; this pass did not exercise live public-Internet HTTPS, a complete private-registry
apm install, a fresh packaged executable, or Linux/macOS execution.The PR validation section contains the current evidence and scope. The PR remains open; all six upstream workflows currently report
action_required, pending repository approval to run.The install acceptance gap identified in the review of #2741 is now covered in e9e0301a.
The new regression executes the real source CLI's
apm installagainst two independently signed loopback HTTPS registries. It proves thatAPM_EXTRA_CA_BUNDLEenables installation from the private-CA source while preserving an existing independent default root. Negative controls reject the private source without the extra CA and reject the default source when replacement-only trust excludes it. Successful installs verify package content, the lockfile, and deployed instructions.The default root is synthetic and seeded into the test process's certifi bundle; the test does not change the machine trust store, contact public services, or mock transport/resolution. This complements the existing real
apm run, Python/Node child, and independent-context trust tests.Local validation: 373 affected tests passed, with no skips, including the Windows symlink tests. Two existing lifecycle timeout reader-thread warnings remain. Required lint and documentation checks passed. The revision also clarifies shell-inline controls and makes managed-bootstrap refresh failure visible.
September 8 update: all six upstream workflows passed on
e9e0301a, including CI and the separate CodeQL findings check, which reported no new alerts and zero annotations. The latest commit, 4b923664, only corrects the changelog format and PR reference; its source and tests are unchanged. The PR remains open and conflict-free. The changelog commit's newly triggered workflows require maintainer approval, and human review is still pending.At the lead maintainer's direction, the current acceptance of this item is withdrawn as part of APM's roadmap and contribution reset. Previous acceptance labels or comments must not be treated as approval to start or continue substantive implementation for inclusion in APM.
Thank you for the work and ideas already contributed. Our small maintainer team needs to agree supported scope and review capacity before inviting more implementation. This is not a rejection of the proposal or a judgment of your work, and it does not undo completed work or erase its history.
Please follow the contribution rules: a responsible human maintainer needs to record the bounded scope, acceptance criteria and review contact on the issue before substantive work resumes. Existing issue discussion, assignments and contributions remain available for that decision. Trivial typo and broken-link corrections retain their standing preapproval.
- added and removed
on Sep 12, 2026 Understood. I’ll hold further work on #2741, including conflict repair, pending fresh scope approval and a review contact.
Daniel Meppiel (@danielmeppiel) Sergio Sisternes (@sergio-sisternes-epam) — could one of you confirm whether the existing proposal is work the project can currently support?
- Scope: opt-in
APM_EXTRA_CA_BUNDLEsupport for APM’s Python HTTPS paths and Python/Node children, preserving explicit replacement and opt-out controls. - Acceptance: real
apm installandapm runprivate-CA coverage, with independent existing-root retention and documented precedence/runtime limits. - Exclusions: Git/Rust trust configuration, machine trust-store changes, shell-command parsing, and the optional CA-directory setting.
The implementation and requested review follow-ups are already in the PR. The functional revision
e9e0301apassed upstream CI and the separate CodeQL findings check; the latest commit only changes the changelog.If this scope is still welcome, please record the fresh approval and review contact here under the current contribution process. If review capacity is unavailable, an explicit deferral would help us leave the work parked until it can be supported.
- Scope: opt-in
Thanks Josh Bazar (@TameTheGame) for the work in #2741 and for keeping it parked while the scope is reconsidered. We are revisiting this with npm/Node's additive-CA behavior as a concrete package-management precedent, rather than treating an undocumented deployment scenario as a reason to assume nobody is blocked.
Node's official NODE_EXTRA_CA_CERTS documentation states that the normal root CAs "will be extended with the extra certificates in
file." This is the additive route available to npm users; it is distinct from npm'sca/cafilesettings, which select replacement trust. Node also documents that an explicit TLScaoption bypasses both normal and extra roots.Are you or your team currently blocked by this in an actual APM workflow? A redacted example of the command/error, operating system, corporate HTTPS proxy versus private registry, and why the CA cannot be installed into normal system trust would help us define the right acceptance case. Please distinguish APM's own HTTPS requests, native Git access, and
apm runexecution if more than one is involved. Please do not share credentials, private keys, or sensitive internal endpoints.Your reported real-CLI private-CA installation coverage in this update already helps establish the technical behavior; the question is about the deployment requirement, not asking you to repeat that work. A general compatibility contribution is welcome too, even if it is not a personal blocker.
The product direction under discussion now favors equivalent additive trust for supported package-management HTTPS, retaining normal roots, certificate verification and explicit replacement/opt-out intent. Experimental execution-runtime propagation is a separate scope question; the intended
apm rundeprecation should not be used to dismiss a core installation need. Git's own TLS configuration also remains a distinct boundary.This is scenario clarification and product-direction feedback, not yet the fresh bounded scope approval required by CONTRIBUTING. No deferral is being applied, and no further implementation or conflict repair is requested before that record is agreed.
Recorded by Copilot on behalf of Daniel Meppiel (@danielmeppiel) following his npm-parity and scenario clarification on 2026-10-02.
Decision: approve
Area: project
Scope: Add opt-in APM_EXTRA_CA_BUNDLE support for APM's own package-management HTTPS, including installation, registry access and downloads, retaining the normal default trust roots alongside the additional corporate PEM certificates. Preserve existing explicit replacement-bundle and truststore opt-out intent; keep certificate verification enabled and centralize trust decisions in the existing TLS authority.
Done when: Real apm install against a private-CA HTTPS source succeeds with the additional bundle while an independently trusted default-root source remains usable; negative controls still reject untrusted certificates. Unset configuration preserves current behavior; invalid extra-bundle input produces a clear failure rather than silently discarding requested trust. Document and cover precedence with REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE and APM_DISABLE_TRUSTSTORE, update the TLS troubleshooting documentation and corresponding apm-usage guidance, and preserve supported existing TLS behavior.
Out of scope: APM_EXTRA_CA_DIR; new propagation into experimental apm run Python/Node execution children; native Git/Rust TLS reconfiguration; changing machine trust stores; disabling certificate verification; unrelated TLS-framework redesign; blanket approval of the existing broader PR #2741.
Review contact: Daniel Meppiel (@danielmeppiel)
Reason: Recorded by Copilot on behalf of Daniel Meppiel (@danielmeppiel) following his explicit acceptance on 2026-10-02: "yes accept issue 2034 on the extra corproate certificates". The confirmed npm/Node additive-trust precedent supports this core package-management capability. The scenario follow-up at #2034 (comment) remains useful for acceptance coverage but is not a condition for product acceptance. Josh Bazar (@TameTheGame) is invited to align the existing contribution in #2741 with this bounded scope through the contribution process; do not start competing work or assume approval of its excluded runtime portion. This is product-scope approval, not a PR review, merge permission or authorization to add a ninth item to the separately delegated eight-issue autopilot run.- added a commit that references this issue
on Oct 3, 2026 Updated #2741 in
73ae7920to match the approved package-management scope.APM_EXTRA_CA_BUNDLEadds corporate PEM certificates to APM's own HTTPS while retaining default roots, certificate verification, and explicit Requests/curl replacement and opt-out intent.- Removed the new Python/Node execution-child propagation and associated snapshot/refresh machinery. Existing runtime trust behavior is preserved; native Git/Rust configuration remains separate.
- Real source-CLI install coverage now runs ten fresh-project cases across normal OS trust and forced truststore fallback, proving private-CA installation, independent default-root retention, and untrusted/replacement-only rejection. Additional checks cover stdlib metadata HTTPS fallback and unchanged child trust.
- 363 affected tests passed with no skips, including elevated Windows symlink cases; 28 final focused checks and 64 repository quality checks passed. Required lint and documentation checks passed, and the PR is conflict-free.
This remains a general compatibility contribution with hermetic acceptance evidence; I am not claiming validation in a particular enterprise deployment. The PR description links the current scope approval and records the full evidence. Hosted workflows await maintainer approval, followed by review from Daniel Meppiel (@danielmeppiel).
Context
Fast-follow from the #2005 / #2004 OS-trust-store work. A red-team review (enterprise-proxy + package-manager-precedent panels) flagged that APM has no additive CA path:
REQUESTS_CA_BUNDLE/CURL_CA_BUNDLEreplace the trust set (and skip truststore injection entirely). Many enterprises hand developers a single corporate-root PEM but do not permit OS trust-store edits, so they need "OS roots plus this one extra CA", not "only this CA".This is the standard npm model (
NODE_EXTRA_CA_CERTS, additive) and Node's--use-system-ca(system + bundled + extra). APM lacks the equivalent.Proposal
Add an additive
APM_EXTRA_CA_BUNDLE(and optionallyAPM_EXTRA_CA_DIR) that is layered on top of the OS trust store rather than replacing it:SSLContextthat first loads the OS/certifi defaults, thenload_verify_locations(APM_EXTRA_CA_BUNDLE)— OS/certifi defaults remain intact.apm runchild runtimes through the samebuild_child_tls_env()seam introduced in feat(tls): verify against the OS trust store by default (closes #2004) #2005 (export the extra bundle so Python and Node children both honor it, e.g.NODE_EXTRA_CA_CERTSfor Node runtimes).REQUESTS_CA_BUNDLE(replace) andAPM_DISABLE_TRUSTSTORE(opt-out) knobs.Acceptance
APM_EXTRA_CA_BUNDLEis trusted in addition to the OS store for bothapm installandapm run.APM_EXTRA_CA_BUNDLEwhile public HTTPS (OS/certifi roots) still verifies.Deferred deliberately from #2005 to avoid half-shipping (Node-only) the additive path.