2.0: modernize stack, replace vendored lmoments with lmoments3, add baseline/annual/SPEI features - #27
Merged
Conversation
Collapse 23 copy-paste test functions into a parametrized golden-value table. Add conftest.py with pathlib-based data fixtures so tests pass from any cwd (previously they required running from the repo root). Use pytest.approx with explicit tolerance instead of np.round equality. pe3-mle is marked xfail: scipy's pearson3_gen.fit uses the legacy super(type(self), self) idiom which infinitely recurses when subclassed by the vendored lmoments code — broken since scipy ~1.7, fixed in a later commit. Verified against the original suite under a 2021-era stack (numpy 1.23 / pandas 1.5 / scipy 1.9): all golden values unchanged.
…t free - freq="W": .dt.week was removed in pandas 2.0; use .dt.isocalendar().week - utils: np.histogram(normed=) was removed in numpy 1.24; use density=True. Add the missing lmoments import (best_fit_distribution raised NameError) - cdf_to_ppf: guard against params=None for all distributions, not only gam/pe3 (previously crashed with '** must be a mapping' for small groups) - lmoments_base: np.asarray(...).sort() mutated the caller's array in place; use np.sort, which copies. The old descending pre-sort in calculate() was masking this by accident; it is now removed as redundant - rolling_window_sum and calculate no longer mutate the caller's DataFrame - min-sample guard derived from the distribution (numargs + 3) instead of a hardcoded 4, so Wakeby on tiny groups yields NaN + warning, not ValueError - fit_distribution/cdf_to_ppf no longer communicate through instance state; instances are now reusable and thread-safe. fit_distribution returns (distribution, params, p_zero) - freq_col: the custom-frequency column is now carried into the working frame (previously always raised KeyError) and validated - grow-by-concat loop replaced with list accumulation (pandas 2.x deprecation and O(n^2) behavior); duplicate-date print() is now a UserWarning - wei MLE golden test now constrains floc=0: unconstrained 3-param Weibull MLE is ill-posed and scipy >= ~1.12 converges to a degenerate fit Golden values verified unchanged on both a 2021-era stack (numpy 1.23/ pandas 1.5/scipy 1.9) and a current stack (numpy 2.x/pandas 2.2/scipy 1.15).
Pin the full calculated-index series for all 19 supported (dist, fit) combinations on monthly_data.csv, generated against the vendored l-moments code. The next commit swaps in the external lmoments3 package; this gate proves the swap changes nothing numerically (rel tol 1e-9).
The standard_precip/lmoments/ directory was a GPL-3 copy of lmoments3 inside an Apache-2.0 project. It is replaced by a dependency on the maintained lmoments3 package (Ouranosinc, >=1.0.6) plus a small registry adapter (_distributions.py) that maps the public dist_type strings to fitting backends (closes #23). - L-moment fits use lmoments3 (the same Hosking code the vendored copy was extracted from). The regression gate passes at rel=1e-9: numerically identical. - MLE fits use plain scipy.stats distributions. Calling .fit() on the lmoments3 subclasses recurses infinitely for distributions where scipy's fit override uses the legacy super(type(self), self) idiom; this is also why pe3/mle had been broken for years. It now works again and reproduces its historical golden value. - New capability: glo and gno support L-moments fitting (Hosking's generalized logistic / generalized normal; previously NotImplementedError). kap L-moments is exposed too, but its L-moment ratios are unsolvable for some samples; such groups now warn and yield NaN instead of raising. - wak with MLE raises a clear ValueError (it never worked: the tuple result was passed to cdf(**params) and crashed).
- pyproject.toml with the hatchling backend replaces setup.py (which listed packages with a path separator and shipped no long_description, so the PyPI page showed UNKNOWN). requirements.txt and the dead Travis config are gone; dev tooling is a uv dependency group (uv sync --group dev), and uv.lock is committed. - Version 2.0.0, Python >= 3.10 (3.9 is EOL), dependencies bounded to the versions the code is actually tested against, lmoments3 declared. - standard_precip/__init__.py: the package previously relied on implicit namespace packages with no __init__.py anywhere. Adds __version__ and re-exports SPI and BaseStandardIndex, so 'from standard_precip import SPI' works as the README always implied. Verified: uv build; wheel installs into an isolated env and imports; test suite passes from any working directory.
calculate() gains three optional keyword arguments (closes #18, supersedes PR #20 - thanks @heroldn for the p_zero-from-baseline insight): - baseline_start / baseline_end: fit the distributions (and the probability of zero precipitation) on a reference period only - integers are years, strings/Timestamps are dates, inclusive - then transform the full record. This is the standard setup for climate-projection work: fit on a historical baseline, apply to the projected record. - return_params=True: additionally return a tidy dataframe of fitted parameters, one row per (column, frequency group), with the observation count and p_zero. The default path is unchanged (regression gate still green).
- freq=None fits a single distribution to the entire column: the correct semantics for annual precipitation totals, where there is no seasonal cycle to condition on (closes #25). The docstring warns against the per-year freq_col mistake, which would leave each year alone in its own fitting group. - freq_col is now actually usable: the column is carried into the working frame (it previously always raised KeyError), must exist, and must be an integer grouping column. Verified equivalent to freq="M" when the column is the calendar month. - Unknown freq values raise ValueError instead of AttributeError.
spei.py was deleted during the pandas-API rewrite, leaving the package
description ('SPI and SPEI') unfulfilled. SPEI shares the calculation with
BaseStandardIndex; the class docstring documents what actually matters: the
input is the user-computed climatic water balance D = P - PET (PET is not
computed by this package), and because D goes negative the zero-stripping
distributions (gam/pe3) are inappropriate - use dist_type='glo' with
fit_type='lmom' per Vicente-Serrano et al. (2010), which the lmoments3
migration made possible.
'from standard_precip import SPEI' now works.
The extreme SPI values reported for gamma MLE were caused by unconstrained-loc maximum-likelihood fits on zero-stripped (strictly positive) data: the location parameter drifts toward or past the data minimum, degenerating the fit and pushing the CDF toward 0/1. After zero removal the support is (0, inf) and every published SPI formulation uses a two-parameter gamma, so loc is now fixed at 0 unless the caller passes floc/loc explicitly. L-moments remains the recommended fit method (it is what NCAR's and R's implementations use). Adds the first tests for plot_index and best_fit_distribution (which never worked before this branch).
Replaces the long-dead Travis setup. CI lints with ruff and runs the test suite with coverage across Python 3.10-3.13 using uv. The publish workflow runs on GitHub releases via PyPI trusted publishing (OIDC); registering the repo as a trusted publisher on pypi.org is a one-time manual step.
uv run ruff check --fix . && uv run ruff format . Plus the two non-auto fixes: strict= on zip() in the distribution adapter and two overlong docstring lines. No behavior change; suite green.
README: fix the broken usage example (undefined variable, wrong column name), replace the dead Travis badge with CI + PyPI badges, fix typos (closes #26), document the new features (baseline period, annual mode, return_params, SPEI), update the distribution table (glo/gno/kap gained L-moments), note the lmoments3 GPL-3 dependency, and add uv dev setup. The example notebook is regenerated and fully executed against the new API, adding baseline, fitted-parameters, annual and SPEI examples. CHANGELOG.md documents 2.0.0 in Keep-a-Changelog format, including all breaking/behavior changes.
… floors
Methodology fixes (verified against synthetic reproductions):
- Sort by date before the rolling window: with scale > 1, unsorted input
produced silently wrong sums. Warn when dates are irregularly spaced,
since the window would span gaps. Output is now date-sorted.
- Per-column dropna: a NaN in one precipitation column no longer removes
that row from other columns' fits and outputs.
- Clip the CDF to [1e-16, 1 - 1e-16] (float64 resolution): observations
outside the fitted distribution's support map to large finite index
values (~ +/-8.2) instead of NaN - important when transforming records
against a baseline period. All previously finite values are unchanged;
the regression golden was regenerated for the 300 previously-NaN cells
and the change is documented in gen_golden.py and the CHANGELOG.
Code-review fixes:
- Empty gamma/pe3 fitting groups (partial-year baselines with freq D/W)
yield NaN instead of ZeroDivisionError.
- Fit failures no longer crash calculate(): lmoments3 raises a bare
Exception('Failed to converge') for some samples, which escaped the
ValueError-only handler. TypeErrors still propagate.
- Working-copy index is reset, so inputs with duplicate index labels
(pd.concat without ignore_index) work with baseline periods.
- baseline_start/baseline_end accept numpy integers as years; they were
previously interpreted as nanosecond timestamps.
- best_fit_distribution fits each candidate exactly as calculate() does
(zero-stripping, mixed-CDF weighting, gamma MLE floc=0), so its ranking
matches the model the index actually uses. Zero-handling and MLE
defaults now live on the immutable distribution registry.
- return_params: n_fit excludes zeros stripped before fitting.
- Unfittable groups warn once per column instead of once per group
(sparse daily data previously emitted 365 warnings).
- A user freq_col literally named 'freq' is no longer dropped.
- Removed the shared mutable class-level non_zero_distr list.
- Regression-gate docstring documents what the golden pins and when
regeneration is legitimate.
Compatibility:
- DistSpec uses default_factory for its mapping default; a bare
MappingProxyType default is rejected by dataclasses on Python <= 3.11.
Suite verified on 3.10, 3.11, 3.12 and 3.13.
- lmoments3 floor raised to 1.0.7: 1.0.6 produces incorrect Wakeby and
Generalized Normal L-moment fits (17 test failures). pandas floor
raised to 2.2 (tests and examples use the 'YE' frequency alias). All
floors are now verified by running the suite against the exact minimum
versions.
Also removes all inline comments from the Python sources in favor of
docstrings, and adds tests for every fix (87 tests total).
e-baumer
force-pushed
the
modernize-2.0
branch
from
August 12, 2026 22:18
38b934a to
cbaa425
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the package up to date after several years: it was broken on the modern scientific-Python stack, carried a GPL-3/Apache-2.0 license conflict, and had a backlog of feature requests. All 14 commits are self-contained and reviewable in order; the test suite grew from 23 golden-value tests to 87 tests including a numerical regression gate.
Fixes
freq="W"(.dt.weekremoved),best_fit_distribution(normed=removed plus a missing import that made it raiseNameErrorsince introduction).pearson3.fitbreaking under subclassing); it works again and reproduces its historical golden value.scale > 1, unsorted input produced wrong rolling sums (now sorted by date, with a warning for gapped records); a NaN in one precipitation column corrupted other columns' fits; the vendored l-moments code sorted the caller's array in place;calculate()mutated the input DataFrame.ZeroDivisionError,**None), duplicate index labels with a baseline period, unconvergeable kappa fits (bareExceptionfrom lmoments3), Wakeby on tiny groups. All now degrade to NaN with one aggregated warning per column.loc=0unless a constraint is passed. L-moments remains the recommended method.License / dependencies
>=1.0.7; 1.0.6 produces incorrect wak/gno fits). A committed regression gate proves the swap is numerically identical to 1e-9 across all 19 supported (distribution, fit) combinations. MLE uses plain scipy distributions.Features
calculate(..., baseline_start=1961, baseline_end=1990)fits distributions andp_zeroon a reference period, then transforms the whole record. Supersedes Specify base period and return distribution parameters #20 — thanks @heroldn for the p_zero-from-baseline insight.freq=Nonefits one distribution to the whole series — the correct semantics for annual totals.from standard_precip import SPEI, documented for user-supplied P − PET with the Vicente-Serrano 2010glo/lmomrecommendation.return_params=True: tidy per-(column, group) dataframe of fitted parameters, sample sizes, andp_zero.freq_colactually works now (it previously always raisedKeyError) and is validated.Tooling
pyproject.toml(hatchling) + committeduv.lock;setup.py,requirements.txt, and the dead Travis config removed. Python 3.10–3.13.e-baumer/standard_precip, workflowpublish.yml, environmentpypi).Incorporates the fixes from #24 and #19 — thanks @oshanmodi and @loicduffar. After merging, #19/#20/#24/#26 can be closed.
Breaking changes (2.0.0)
Python ≥ 3.10; vendored
standard_precip.lmomentsremoved; gamma MLE defaultsfloc=0;fit_distribution/cdf_to_ppfsignatures changed (stateless instances); output sorted by date; duplicate-date message is aUserWarning. Full details in CHANGELOG.md.