Add blog post on why type-coverage matters - #4518
Conversation
|
This pull request has been imported. If you are a Meta employee, you can view this in D115732206. (Because this pull request was imported automatically, there will not be any future comments.) |
grievejia
left a comment
grievejia
left a comment
There was a problem hiding this comment.
Review automatically exported from Phabricator review in Meta.
jorenham
left a comment
jorenham
left a comment
There was a problem hiding this comment.
Awesome, thanks. I left some suggestions; feel free to ignore any of them.
|
|
||
| ## What's type coverage? | ||
|
|
||
| Similarly to how test coverage measures how much of a library's source code is hit by its tests, type coverage measures what percentage of a library's symbols have type annotations. Examples of symbols which get counted include function arguments and return types, class variables, and constants. Local variables in function bodies are excluded, as users would never interact with them anyway. |
There was a problem hiding this comment.
I've been calling them "typables", because there are symbols for which it doens't make much sense to type them, and because return types aren't really symbols.
I guess you could describe it as "a typable is something that can meaningfully annotated". I use "meaningfully" because there are typables where annotating it wouldn't be meaningfull, such as the return type of __init__ because that's always None, so annotating it wouldn't help much.
So although quite a bit will have to be rewritten, it'd probably be clearer if this would talk about these "typables" instead of "symbols". It'll at least be more accurate that way. Another benefit of using "typables" and actually defining what it means, is that it'll show that we've carefully thought out these coverage metrics (unlike pyright's --verifytypes, for example).
But even so, I'm probably the only one that cares about this detail haha; I'm sure the casual reader wouldn't even think twice about what exactly "symbol" means here.
| Note how there are two issues with the stub file: | ||
|
|
||
| - The `c` parameter in `two` is misnamed (it should be `b`). It's also unannotated. | ||
| - The `three` function is missing entirely. |
There was a problem hiding this comment.
I can imagine some users could be confused by this, because they might think that type-checkers fall back to the inline annotations if not present in the stub. So it might be worth explaining here that if a .pyi exists, type-checkers only look at that .pyi, completely ignoring the corresponding .py if it exists.
|
|
||
| Pyrefly coverage is a new-ish feature, yet it's already being used by a few projects: | ||
|
|
||
| - [SciPy](https://github.com/scipy/scipy). |
There was a problem hiding this comment.
SciPy itself doesn't use pyrefly coverage (it just uses pyrefly check), so I think you mean scipy-stubs.
| - [SciPy](https://github.com/scipy/scipy). | |
| - [SciPy (stubs)](https://github.com/scipy/scipy-stubs). |
or just
| - [SciPy](https://github.com/scipy/scipy). | |
| - [scipy-stubs](https://github.com/scipy/scipy-stubs). |
| - [NumPy](https://github.com/numpy/numpy). | ||
| - [Narwhals](https://github.com/narwhals-dev/narwhals). | ||
| - [sh](https://github.com/amoffat/sh). |
There was a problem hiding this comment.
Should we also include polars (pola-rs/polars#28795)?
| - [NumPy](https://github.com/numpy/numpy). | |
| - [Narwhals](https://github.com/narwhals-dev/narwhals). | |
| - [sh](https://github.com/amoffat/sh). | |
| - [NumPy](https://github.com/numpy/numpy). | |
| - [Polars](https://github.com/pola-rs/polars). | |
| - [Narwhals](https://github.com/narwhals-dev/narwhals). | |
| - [sh](https://github.com/amoffat/sh). |
|
|
||
| ## Who's using it? | ||
|
|
||
| Pyrefly coverage is a new-ish feature, yet it's already being used by a few projects: |
There was a problem hiding this comment.
| Pyrefly coverage is a new-ish feature, yet it's already being used by a few projects: | |
| Pyrefly coverage is a new-ish feature, yet it's already being used by a few major projects: |
maybe?
| ```console | ||
| WARN `foo.three` is untyped [coverage-missing] | ||
| --> src/foo/__init__.py:7:1 | ||
| | | ||
| 7 | / def three() -> None: | ||
| 8 | | return None | ||
| | |_______________- | ||
| | | ||
| WARN `foo.two` is not fully typed [coverage-partial] | ||
| --> src/foo/__init__.pyi:2:1 | ||
| | | ||
| 2 | def two(c) -> None: ... | ||
| | ----------------------- | ||
| | | ||
| ERROR type coverage 60.00% (3 of 5 typable) is below the 100.00% threshold | ||
| ``` |
There was a problem hiding this comment.
maybe include the full pyrefly coverage check command in here?
| - `def one(a: int) -> int` | ||
| - `def two(b: int) -> None` | ||
| - `def three() -> None` |
4 participants
Summary
I've put together a little blog post to explain what type coverage is, and why people might want to make use of it
Fixes #XXXX
Test Plan