Skip to content

Run docstring examples in CI (--doctest-modules) #866

Description

@mmcky

Summary

#864 fixed 31 broken docstring examples that a pytest --doctest-modules sweep turned up — stale pre-NumPy-1.14 array spacing, NumPy 2 scalar reprs, missing imports, and two outright NameErrors. Those examples had drifted precisely because nothing ever ran them. After that PR, 83 of the 85 modules are doctest-clean when each module is run in isolation.

The obvious follow-up is to run doctests in CI so they cannot drift again. Two things stand in the way, and neither has an obviously right answer, so raising it rather than deciding it in #864.

Blocker 1: global print state leaks between modules

Six game_theory docstrings call np.set_printoptions(precision=4) and never restore it:

  • game_theory/lemke_howson.py (two examples)
  • game_theory/support_enumeration.py (two examples)
  • game_theory/vertex_enumeration.py (two examples)

Print options are process-global. Under a package-wide --doctest-modules run, game_theory is collected before markov, optimize, random and util, so every array printed in those later modules comes out with 4 significant digits instead of 8. That produces 9 spurious failures in modules whose examples are individually correct — for instance markov/random.py, random/utilities.py::probvec, markov/ddp.py and optimize/nelder_mead.py all pass on their own and fail in the full run.

This is pre-existing and entirely harmless to the rendered documentation: a reader copying any one of those blocks gets exactly what is shown. It only bites the test harness.

The options are to drop set_printoptions from those six examples and paste full-precision expected output (more faithful, but noisier examples), to add a restoring call after each block (visible clutter in the published docs), or to reset print options between modules from a conftest fixture (invisible to readers, but the fixture has to be discovered and maintained). My weak preference is the conftest route, since it keeps the examples readable, but it is the least obvious to a future contributor.

Blocker 2: two modules cannot pass honestly

Module Why it cannot pass
util/notebooks.py::fetch_nb_dependencies Fetches over the network
util/timing.py::Timer and ::timeit Reports wall-clock durations, which vary per run

#864 deliberately left both untouched rather than annotating them with # doctest: +SKIP, because doctest directives render visibly in the published HTML and would be a cost paid by every reader to satisfy CI. If we do wire doctests into CI, we need to decide whether that visible cost is acceptable, or whether to exclude these modules at the pytest level instead — for example via --ignore, or collect_ignore in a conftest, which keeps the rendered docs clean.

util/array.py::searchsorted is a third, milder case: it is deliberately deprecated and emits a DeprecationWarning, which a strict warning filter would turn into a failure.

Scope

Once those are settled the change itself is small — a step in ci.yml alongside the existing test run. The value is that docstring examples become executable specifications rather than prose that silently rots, which is what let a call to a nonexistent lorenz() sit in _inequality.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions