Click future direction

[Rowlando13]

In the last few years of working on Click, despite @kdeldycke adding many hundreds of tests, there has been a lot of breakage, specifically with 8.2 and 8.3. And 8.2 and 8.3 were mostly focused around fixing long standing bugs. Growing the test suite, has helped, but at this point it feels like the API is just too open, meaning there are many methods which are public. It feels like it is time for 9.0, where we actually fix the problem. In my mind 9.0 would try to:

1. Rewrite the parser to remove the optparse baggage
2. Make a lot of currently public methods private
3. Allow for fully customizable help page (which I would do)

However, I am sensitive to how much breakage this is likely to cause. 1. and 2. are mostly for the maintainers which I does not sell the breakage at all for people upgrading. 3. Is a strong reason to upgrade, but I am not convinced it will be enough right away. I think the solution is to land 9.0 with 1,2, and 3. Plan for lots of bug fix patches(9.0.1, 9.0.2...) And then try to land some much desire features in 9.1, maybe option groups, option ordering, ? However, those features should be planned in concert with 1.

I would also like to provide a clear upgrade path to users by:

• Provide a stable release for people to use since this will take a while. Maybe that is 8.3.1?
• If we make a method private, we would document what its replacement should be.
• ?

@untitaker @davidism @kdeldycke Does this sound right, or am I off base. I am very open to feedback since I have no experience with this sort of thing.

[kdeldycke]

• Make a lot of currently public methods private

If it would be on me I would be even more bolder about that by having them explicitly prefixed by an underscore _. 😅

And then try to land some much desire features in 9.1, maybe option groups, option ordering, ?

We can even plan both a 9.x and a 10.x in a short time frame (like less than a year), to have a major release dedicated to parser rewrite, and one to data/interface contracts.

Integers are cheap and it is a strong signal for maintainers to look twice at the changelog before blindly upgrading Click. See for instance how Graphviz was stagnant for a decade then went from 2.42.4 to 14.0.4 in the space of 5 years.

Other random ideas to review/re-evaluate for a major release:

• Unhide UNSET in internals
• Make UNSET falsy (cc @edgarrmondragon)
• Split-up core.py into context.py / parameter.py / command.py
• Split-up class Option into Option, FlagOption and BoolFlagOption
• Mirrors Option.is_flag/Option.flag_value concept into Parameter.is_default/Parameter.default_value
• Remove the implicit legacy cases of default and flag_value: remove some auto-magic around defaults and make behavior more strict
• Rename all private methods/functions/variables with _underscore style
• Remove utils.py / _utils.py: code needs to have a clear purpose and domain, else everything ends up being a vague utility or helper
• Remove isolated_filesystem()
• Change the data contract: move all type conversion embedded into the parser and core to Click's types (i.e. types.py::ParamType); preserve native types (Python/strings) from the edge of the framework to the console rendering. See @sirosen comment on data separation.
• consolidate signatures and improving typing

[sirosen]

We can even plan both a 9.x and a 10.x in a short time frame (like less than a year), to have a major release dedicated to parser rewrite, and one to data/interface contracts.

As a heavy user of, occasional contributor to, click I strongly endorse this path. There's nothing "wrong" with the version numbers going 9.0.0, 9.0.1, 10.0.0, if that's the best way to communicate with users. And for larger projects, porting their usage for each of these changes could be an endeavor in its own right.

I have seen projects get "stuck" on old dependency versions because a single-step upgrade changed too much at once. If I have to migrate a project across three or four or even more significant click releases in a row (e.g., 9.0, 10.0, 10.1, 11.0), that's generally going to be a lot easier on me as a downstream maintainer than trying to handle many changes all at once.

If click 9.0 simply made a lot of names private by renaming them with _, with few other intentional¹ user-facing changes, that would be extremely valuable. 10.0 could be the version for deeper structural changes.

I have written lots of custom param types, command and group types, and even custom option and argument subclasses, and undoubtedly have used methods meant to be private unknowingly. I remember that @davidism was surprised to hear I was overriding ParamType.get_metavar (see #2929 for context), but click had no prior way to indicate that this would be surprising.
In handling the removal of methods and attributes, projects will migrate to the APIs which are intended to be public (e.g., to_info_dict), and will therefore have a better experience with subsequent upgrades because the docs will direct them towards what changed.

Provide a stable release for people to use since this will take a while. Maybe that is 8.3.1?

I wouldn't worry about this at all. Users can rely on the 8.x versions for as long as those continue to work. As soon as 9.0.0 is released, everyone's expectation should be that 99%+ of click contributions are going into the 9.x line, and that 8.x will only get updates in unusual circumstances (e.g., security bugfix).

But the message should be "stop using 8.x as soon as you can."

---

One thing I've found valuable in recent projects is to write an "upgrading" guide as part of the change documentation for major releases. We maintain one for webargs, and we imitated that in a work project, globus-sdk.

To use my previous example, this means that if ParamType.get_metavar is becoming ParamType._get_metavar, the PR author for that change also has a place to write down what users who were using that method should do instead.

Footnotes

1. "intentional" being key. Doing internal cleanup, like moving around modules, which isn't supposed to impact users is, IMO, fine for that first cut. ↩

[davidism]

A difficult part about the "private" designation is that there's more nuance sometimes. We might be fine with people calling it, but once they override it becomes much hard to evolve.

[Rowlando13]

@sirosen @kdeldycke If anything from this list appeals to you, then make an issue and attach it to #3154, and we can discuss.

[untitaker]

I would caution against pushing out a major release that contains a lot of pent up breaking changes. Frankly the merits and risks of each change should be discussed and released separately

[Rowlando13]

Would you mind clarifying? Do you mean that releasing 1,2, and 3 is too much or "Make a lot of currently public methods private" is too vague and we should just consider each as method separately or something else?

[untitaker]

@Rowlando13 sorry for not responding. basically i'm suggesting to take a small breaking change, make a release with that and anticipate a wave of feedback from that. that feedback will probably influence how you roll out the rest. IMO you can't really incentivize folks to give you "proper" feedback until the damage is done, hence i suggest a slow rollout.

[Rowlando13]

Thanks for responding. That seems to be the consensus, so we are going with that.

[Rowlando13]

Based on input so far I think the tentative plan is split 1, 2, and 3 up. Anyone from the community is welcome to weigh in if they disagree.