[question] Why am I getting a different python_requires package version when resolving a test_package dependency graph?

[Alex-Riley]

What is your question?

In our CI setup we recently saw a test_package build failing due to missing dependencies. This was unexpected as our CI setup is running an in-house tool that effectively does the following: conan install -> manually run cmake && cmake build -> conan export-pkg. The tool ensures that the conan export-pkg command runs with the same profile, config and setting arguments that were used for the conan install command but errors if any packages are missing from the local cache (rather than downloading or building these), under the assumption that all dependencies needed for the test_package build will have been provided by running conan install already.

When investigating the issue I found that the failing build (for package foo) wanted a dependency (bar) to be built with version 1.7.x of an internal python_requires package that we use (conan-utils) but the conan install command for foo had built bar with conan-utils 1.8.x (we had recently released v1.8.0 of conan-utils).

The test_package graph looking for a build of bar with conan-utils 1.7.x was initially surprising to me as the bar recipe specifies that is uses >=1.0.0 <2.0.0 in its requirement for conan-utils. Additionally, we did not notice this behaviour in other packages, despite conan-utils being used throughout our setup.

On closer examination of foo's recipe, I noticed that its version requirement for conan-utils was explicitly 1.7.1 and that changing it to >=1.0.0 <2.0.0 caused the test_package to now accept the bar build with conan-utils v1.8.0. This all fits with the information in the docs about how ranges are resolved if there's already a requirement specified for a given package.

So my question is why didn't we see conan-utils 1.7.1 get picked when we ran the conan install command for the foo package but it did then get picked when we ran the test_package build?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @Alex-Riley

Thanks for the question.

I am a bit confused about some descriptions of the issue, like The test_package build looking wanting a build with 1.7.x, what that does mean? The test_package/conanfile.py has a python-requires itself?

As this is an issue with versions, it would probably be much better if it could be reproduced with very simple conanfiles, such as:

# bar/conanfile.py
class Bar(ConanFile):
     python_requires = "..."

# bar/test_package/conanfile.py
class TestBar(ConanFile):
     python_requires = "..."

Making everything fully explicit and non ambiguous. That also helps in creating unit tests to reproduce the potential issue.

[Alex-Riley]

I am a bit confused about some descriptions of the issue, like The test_package build looking wanting a build with 1.7.x, what that does mean? The test_package/conanfile.py has a python-requires itself?

Apologies for the typo. I meant to say:

The test_package graph looking for a build of Bar with ConanUtils 1.7.x was initially surprising...

(I have edited the original question to match this)

The test_package/conanfile.py has a python-requires itself?

It does. It's requirement for ConanUtils was set to [~1] (which I would expect to allow the v1.8.0 package).

As this is an issue with versions, it would probably be much better if it could be reproduced with very simple conanfiles, such as:

I'll see if I can put a reproducing example together but as we've fixed the issue it's lower priority so it won't be straight away.

[Alex-Riley]

# bar/test_package/conanfile.py

The Bar package was operating normally. It was the test_package for Foo that was failing.

To make this more concrete the layout was:

bar/conanfile.py (working):

class Bar(ConanFile):
     name = "bar"
     version = "8.0.0"
     python_requires = "conan-utils/[^1]"

bar/test_package/conanfile.py (working):

class TestBar(ConanFile):
     python_requires = "conan-utils/[^1]"
     requires = self.tested_reference_str

foo/conanfile.py (performed a build of bar/8.0.0 with conan-utils/1.8.0):

class Foo(ConanFile):
     name = "foo"
     python_requires = "conan-utils/1.7.1"
     requires = "bar/[^8]"

foo/test_package/conanfile.py (wanted a build of bar/8.0.0 with conan-utils/1.7.x):

class TestFoo(ConanFile):
     python_requires = "conan-utils/[~1]"
     requires = self.tested_reference_str

[memsharded]

But your conanfiles are not valid, they are using UpperCase packages which doesn't work for Conan 2, all packages need to be lowercase.
Can you please confirm you are using Conan 2 and what version?

[Alex-Riley]

Can you please confirm you are using Conan 2 and what version?

Our CI was using Conan 2.21.0.

they are using UpperCase packages which doesn't work for Conan 2

I just came up with some names to avoid using the actual names of our package. I've edited the package names to use lower case now. Apologies for any confusion 🙂

[memsharded]

Thanks for the feedback.

I managed to understand what was happening, and I have been since yesterday investigating it, and looking for possible improvements. I have opened a PR to try to improve it in #19344, but it might not be possible to move forward, the thing is that there are some reasons for that behavior, mostly related to reproducibility and lockfiles.

To try to summarize the current behavior:

• When a package is created, the first step is to "export" its recipe to the Conan cache, to compute its "recipe-revision", and start computing its dependencies, etc.
• At that export time, both the current package including its revision (in the example is foo/1.0#recipe-revision) and most importantly, its python-requires, such as utils/1.7.1#revision are added to a in-memory lockfile that will be used to expand the graph later.
• That utils/1.7.1 is the thing that is constraining the bar python-requires to resolve to it too, because it is in the valid range.

The alternative behavior is what is currently in the open PR #19344:

• When the recipe is exported to the cache, it only adds to the memory lockfile the foo/1.0#recipe-revision reference, but not the python-requires one.
• That allows for the utils/1.7.1 to work apparently smoothly. But this is a special situation, it works because the bar has a version range and it built against a newer utils/1.8.0, and the foo package happens to be using an older version.
• The possible problem is if the foo package used a utils/1.9.0 dependency instead. This case might apparently resolve, but it leads to the confusing situation where doing a conan create foo --lockfile-out=foo.lock will work, but then doing a new conan create foo --lockfile=foo.lock, that uses the just created lockfile as input fails!! So we are capturing a lockfile at conan create foo time for reproducibility, and if we use it a second later, it fails, so we can't reproduce the built that was done!
• The PR proposes in the test that the correct way to manage this would be first to do a conan lock create to create a lockfile, and such a lockfile will force the utils/1.9.0 dependency, so using the lockfile for the first time will give a Missing binary: bar/8.0 because it needs to be built against the newer utils/1.9.0.

As you can see, neither approach is perfect, it is a tradeoff:

• With the current behavior, it forces the utils/1.7.1 into the graph "early", so it gives the missing binary for bar/8.0 early
• With the new behavior, we can end up with a non reproducible "foo" package build, even if we capture the lockfile at conan create time, that lockfile will require bar to upgrade to utils/1.9.0 and build a new binary.

The need for a tradeoff is derived from the reduction in the degrees of freedom that happens in the mapping from a full dependency graph (a DAG) to the list of versions that are in the lockfile. Specially when new versions are brought into previously existing packages and binaries, the list of versions cannot map all the scenarios in which every node in the dependency graph can have its own and independent "python-requires" version. So lockfiles work on some assumptions, and the new behavior violates slightly one of those assumptions when applied at conan create time, because the dependencies version ranges previously resolved against utils/1.8.0 are affected by the utils/1.9.0 that the new foo package brings to the lockfile.

[Alex-Riley]

Thanks for the feedback.

I managed to understand what was happening, and I have been since yesterday investigating it, and looking for possible improvements.

Really appreciate your investigation and clear explanation. I'm continually surprised (pleasantly so!) of the Conan team's responsiveness and helpfulness 🙂

As you can see, neither approach is perfect, it is a tradeoff:

* With the current behavior, it forces the `utils/1.7.1` into the graph "early", so it gives the missing binary for `bar/8.0` early

* With the new behavior, we can end up with a non reproducible "foo" package build, even if we capture the lockfile at `conan create` time, that lockfile will require `bar` to upgrade to `utils/1.9.0` and build a new binary.

Understood. I think that of these two behaviours the current one makes more sense.

I don't suppose it would be possible for a package's python_requires to be added to the start of the graph at install time (in the same way it is at export-pkg time)? e.g. in this case when I conan install foo/, the requirement for conan-utils/1.7.1 is recorded. This would then lead to the conan-utils/[^1] requirement for bar resolving to conan-utils/1.7.1 and giving consistent behaviour between the conan install and conan export-pkg dependency graphs.

I'm aware that this is a behavioural change with an impact that's probably very hard to determine so please feel free to reject on that basis even it's technically feasible.