Support multiplatform pexes with uv lockfiles.

[benjyw]

Also supports building FaaS (awslambda/gcp cloud functions).

We leverage the fact that uv can create foreign-platform venvs
with the --python-platform flag. It still needs a local python
interpreter of the appropriate version, but it downloads and installs
wheels for the foreign platform. Of course there must be foreign
platform wheels available for all distributions. If there isn't one for
some dist, uv will build from the sdist for the local platform, which
may lead to a later runtime error.

Will close #23339

[benjyw]

This fixture was unused even before my change.

[benjyw]

Other tests were not using lockfiles but rather the legacy "resolve on demand without a lockfile" functionality. So this test is novel even in the pex case.

[benjyw]

Now we always compute the runtime python version, even if a complete_platforms is specified, because we don't yet know if we're using pex or uv.

[benjyw]

This was in a previous PR that was not in the release notes.

[benjyw]

Needed because we added a PythonSetup arg to the callee

[benjyw]

This is no longer ignored - it's used to compute uv_platform, even if that doesn't end up being used (because we're using pex resolution).

[benjyw]

AI disclosure: Claude was used to help with some plumbing, but it was mostly unhelpful with my attempts to get it to be smarter.

[tobni]

Every FaaS build materializes the entire resolve for that platform, so any partial-platform package anywhere in the resolve blocks unrelated targets.

Concretely for us: we build desktop apps for macOS & intel linux and lambdas for AL2023, and a lambda whose closure is 23 pure-python requirements fails to package because pyqt6-qt6 has no wheel for the lambda platform.

Pex just did "the right thing" by subsetting the lock per target. Here we'd need to enumerate platform markers in our requirements.txt so that install-time skipping kicks in. Workable, but users may need to be informed requirements.txt markers is how to get a build working.

Future:
Could the sync be restricted to the consumer's closure? uv.lock knows the package.dependencies graph and pants already knows the target's top-level requirements, so the complement could be excluded via --no-install-package at the cost of keying venvs by closure rather than by resolve+platform? Might be complicated but it would give us feature parity.

[tobni]

Another observation: (the empty uv.toml is on main, and #23454 only fixes uv lock) FaaS packaging uses sync and sync needs the indices.

uv.lock stores registry urls without credentials, so uv sync needs the index config again. The venv rule writes an empty uv.toml, so generation works but every install 401s against our CodeArtifact index.

Fix that works for us:

resolve_config = await determine_resolve_config(
    ResolveConfigRequest(metadata.resolve), **implicitly()
)
# snip
FileContent("uv.toml", resolve_config.uv_config().encode()),

[tobni]

Will add that with my comments addressed using hacks I got this to run in our repo, which is exciting!

[benjyw]

I'll fix this up ASAP per your comments

[cburroughs]

Thanks for the contribution. We've just branched for 2.33.x, so merging this pull request now will come out in 2.34.x, please move the release notes updates to docs/notes/2.34.x.md if that's appropriate.

[benjyw]

Future: Could the sync be restricted to the consumer's closure? uv.lock knows the package.dependencies graph and pants already knows the target's top-level requirements, so the complement could be excluded via --no-install-package at the cost of keying venvs by closure rather than by resolve+platform? Might be complicated but it would give us feature parity.

#23365 is an attempt in this direction. You can pass --only-group to restrict to the group and its transitive deps, and we already know the non-transitive direct deps to put in that group.

[benjyw]

Another observation: (the empty uv.toml is on main, and #23454 only fixes uv lock) FaaS packaging uses sync and sync needs the indices.

uv.lock stores registry urls without credentials, so uv sync needs the index config again. The venv rule writes an empty uv.toml, so generation works but every install 401s against our CodeArtifact index.

Fix that works for us:

resolve_config = await determine_resolve_config(
    ResolveConfigRequest(metadata.resolve), **implicitly()
)
# snip
FileContent("uv.toml", resolve_config.uv_config().encode()),

Does #23443 address this?

[tobni]

Another observation: (the empty uv.toml is on main, and #23454 only fixes uv lock) FaaS packaging uses sync and sync needs the indices.
uv.lock stores registry urls without credentials, so uv sync needs the index config again. The venv rule writes an empty uv.toml, so generation works but every install 401s against our CodeArtifact index.
Fix that works for us:

resolve_config = await determine_resolve_config(
    ResolveConfigRequest(metadata.resolve), **implicitly()
)
# snip
FileContent("uv.toml", resolve_config.uv_config().encode()),

Does #23443 address this?

No, the venv rule generate_pyproject_toml needs to be supplied indices and sources, I think.

[benjyw]

@tobni this is ready for another look, and if you can try it out in your repo that would be great. The main thing not yet addressed is syncing just the closure of the needed requirements. That can be done separately.

[tobni]

Yep, this works!

I do think the closure feature is important to get in though. The uv_platforms override unblocks for our build specifically, but it is a lie about the runtime: it happens to be safe for our case only because nothing we use hard requires glibc > 2.34 (not getting into the details here).

Some findings while running:

1. Explicit complete_platforms no longer short-circuits inference. On main, complete_platforms returned early from infer_runtime_platforms. On this branch inference continues so it can also compute uv_platforms, so a version with no runtime name now fails, including with the pex resolver, which arguably makes that feature worthless? Try e.g python 3.15 IC and a correct complete_platforms for some lambda runtime:

InvalidTargetException: In the 'python_aws_lambda_function' target: Could not find a known runtime for the
inferred Python version and machine architecture!
* Python version: (3, 15) [inferred from interpreter constraints]
* Machine architecture: x86_64
...
To fix, please: - specify `uv_platforms` for the given machine architecture,
  or specify a runtime that is known to Pants.

This is not a feature we use, mind you. Our IC and lambda runtimes are lockstep.

2. complete_platforms is ignored by uv. A warning when you mix a uv-resolve with complete_platforms pointing to uv_platforms is likely much better UX. Otherwise users might "flip the resolve switch" and start netting MacOs pexes that used to be cross-platform.

[benjyw]

1. Explicit complete_platforms no longer short-circuits inference. On main, complete_platforms returned early from infer_runtime_platforms. On this branch inference continues so it can also compute uv_platforms, so a version with no runtime name now fails, including with the pex resolver, which arguably makes that feature worthless? Try e.g python 3.15 IC and a correct complete_platforms for some lambda runtime:

InvalidTargetException: In the 'python_aws_lambda_function' target: Could not find a known runtime for the
inferred Python version and machine architecture!
* Python version: (3, 15) [inferred from interpreter constraints]
* Machine architecture: x86_64
...
To fix, please: - specify `uv_platforms` for the given machine architecture,
  or specify a runtime that is known to Pants.

This is not a feature we use, mind you. Our IC and lambda runtimes are lockstep.

Yeah, this is a little tricky - we also now must infer an interpreter version instead of allowing the ambient interpreter.

I wonder if we should do the following heuristic: only change the behavior if PythonSetup.resolver is uv. This would fail in the case where you have a uv-generated lockfile, but then you changed the resolver back to pex and didn't regen the lockfile. But that seems like a pretty unlikely corner case. WDYT?

[tobni]

Yeah that heuristic would catch that error.

My thoughts

• Why the global option instead of the resolve's lockfile format attr? Maybe option value is the better source of truth, I dont actually know.
• The side effect where users are using complete_platforms for faas and switch to uv resolve are a bit scary. Maybe it is an unlikely use case? Like I mentioned, we dont/wont.

[benjyw]

Yeah that heuristic would catch that error.

My thoughts

• Why the global option instead of the resolve's lockfile format attr? Maybe option value is the better source of truth, I dont actually know.

The lockfile format is the best source of truth, but we don't know it here, and changing that would mean pulling a lot of code inside out, which I really don't want to do.

• The side effect where users are using complete_platforms for faas and switch to uv resolve are a bit scary. Maybe it is an unlikely use case? Like I mentioned, we dont/wont.

We can detect this - if an explicit complete_platforms is provided then we refuse to infer a uv_platforms and fail later.

[benjyw]

Alternatively maybe the way to kill two birds with one stone and say that you cannot use uv if there is an explicit complete_platforms. You must first get rid of it to signal "yes I really mean to use uv". It's slightly clunky but it is at least easy to articulate and implement, and it is only a gotcha when transitioning from pex to uv for the first time, and then only if you kept the complete_platforms around.