Skip to content

fix(distributions): keep two-sided TruncatedNormal samples within bounds - #2441

Open
rpinho wants to merge 2 commits into
pyro-ppl:masterfrom
rpinho:fix-truncated-normal-tail
Open

rpinho wants to merge 2 commits into
pyro-ppl:masterfrom
rpinho:fix-truncated-normal-tail

Conversation

@rpinho

@rpinho rpinho commented Nov 18, 2026 •

Copy link
Copy Markdown

TruncatedNormal(low=-16, high=-15) can draw -12.95 with float32. Call sample(key, (256,)) and check support for each draw. The patch computes and inverts the log CDF when the tail probability underflows. The test compares quantiles to SciPy and checks that draws lie between the bounds.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LKxc9GKLwnyeV2jaZBxH9N

@github-actions github-actions Bot added the awaiting review Awaiting review from maintainers label Nov 18, 2026
Comment thread numpyro/distributions/truncated.py Outdated


@jax.jit
def _truncated_normal_icdf(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we add this as a staticmethod in LeftTruncatedDistribution so that this is not a random aux function?

@juanitorduz juanitorduz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. The two-sided Normal case is correct now. Requesting changes for the items below.

Tests

The icdf check against osp.truncnorm.ppf and the (15, 16) case are the parts that matter. The sample + support assertion can go:

  • sample is icdf(uniform(tiny, 1)), and the new branch clips its output to [low, high], so the support check can only fail if the branch isn't taken. The icdf check catches that first.
  • 256 draws never get near q = 0 or 1, so it adds no coverage of the extremes.
  • A clip-only fix (jnp.clip(ppf, low, high)) would pass it, with every draw at -15. Only the icdf check rejects that.

Keep (15, 16): it's the only case that goes through the reflected branch (sign = -1), which covers the sign and the -inf/0 limit in log_tail_at, and loc + sign * scale * z.

The jnp.float32 casts only matter under JAX_ENABLE_X64=1, which CI doesn't use for this test. Keep or drop, either is fine.

Suggested version:

@pytest.mark.parametrize("low,high", [(-16, -15), (15, 16)])
def test_truncated_normal_icdf_in_tail(low, high):
    d = dist.TruncatedNormal(low=jnp.float32(low), high=jnp.float32(high))
    q = np.array([0.1, 0.5, 0.9], dtype=np.float32)
    assert_allclose(d.icdf(q), osp.truncnorm.ppf(q, low, high), rtol=0, atol=2e-5)

If you want a second assertion, check gradients rather than support. The fix changes reparameterized gradients in the tail (on master, d sample / d loc is 1), and a NaN leaking through the jnp.where(..., -100.0) placeholder is the regression nothing else would catch. Gradients are correct today (they match central finite differences), so this one is optional.

Code

  1. One-sided truncation has the same bug, with ±inf draws. On this branch, TruncatedNormal(low=15) returns +inf for every draw and TruncatedNormal(high=-15) returns -inf. support accepts these, since greater_than_eq(15) allows inf. The new helper already handles infinite bounds: calling _truncated_normal_icdf with (low=15, high=inf, sign=-1) and (low=-inf, high=-15, sign=1) matches SciPy to ~1e-6. Could you wire it into LeftTruncatedDistribution.icdf and RightTruncatedDistribution.icdf here and add the one-sided cases to the test? If you'd rather keep this PR to two-sided, please open an issue for it.
  2. Drop @jax.jit on _truncated_normal_icdf. No other helper in numpyro/distributions is jitted. Under an outer jit it does nothing, and in eager mode it adds a compile cache per input shape.
  3. Not blocking: Logistic and Laplace bases fail the same way far in the tail. TruncatedDistribution(Logistic(), low=-110, high=-105) puts every draw at -87.34, and Laplace gives -inf. Fine to leave out of this PR.

What I checked locally

  • The new test fails on master (max abs diff 2.2).
  • f32: icdf vs SciPy for q from tiny to 1 - 2**-24, max error ≤ 3.2e-6 for [-16, -15], [15, 16], [-13.5, -12.5], [12.5, 13.5], [-40, -39], loc=3, scale=2, and an explicit -inf lower bound. Master is off by 2 to 27.
  • x64: [-40, -39] and [39, 40] within 1.2e-13.
  • jit + vmap over a batch mixing tail and non-tail intervals stays in bounds.
  • Reparameterized gradients match central finite differences for loc, scale and high.
  • Speed (jitted f32, 1e6 draws, single run): non-tail 2.0 → 2.6 ms, tail 2.5 → 7.1 ms. The lax.cond keeps the common case cheap.
  • ruff check, ruff format --check and ty check pass; pytest test/test_distributions.py -k "runcated or Truncated": 1359 passed.

@rpinho

rpinho commented Nov 19, 2026

Copy link
Copy Markdown
Author

That's a good point. I changed those.

@github-actions

Copy link
Copy Markdown

Benchmark report

this PR fix-truncated-normal-tail at bb4e2c95 vs baseline master at fc7691c7

- run time:     1 slower, 1 faster
- compile time: 2 slower, 0 faster

Significant changes (4)

                                ──────── run time ───────     ────── compile time ──────
  benchmark                     baseline   this PR      Δ     baseline   this PR       Δ
────────────────────────────────────────────────────────────────────────────────────────
+ student_t_log_prob              1.2 ms    1.2 ms  -5.0%      54.4 ms   45.0 ms  -17.3%
- predictive_forward_sampling   524.6 ms  499.2 ms  -4.8%     134.3 ms  195.4 ms  +45.5%
- hmc_logistic_regression       753.9 ms  814.6 ms  +8.0%       1.79 s    2.08 s  +16.1%
- nuts_dense_mass_funnel        938.5 ms  967.8 ms  +3.1%       1.49 s    1.87 s  +25.7%

Red is slower, green is faster; a row is coloured by the worse of its two columns. A delta in parentheses cleared the threshold on a measurement below the resolution floor, so it is shown without being called a change. † marks a benchmark that could not be compared — see below.

Full results

distributions

                                 ───────── run time ────────     ────── compile time ──────
  benchmark                      baseline  this PR         Δ     baseline   this PR       Δ
───────────────────────────────────────────────────────────────────────────────────────────
  biject_to_constraints            1.9 ms   1.8 ms     -2.3%     125.8 ms  123.9 ms   -1.5%
  categorical_log_prob             871 µs   936 µs   (+7.6%)      45.0 ms   39.1 ms  -13.0%
  dirichlet_log_prob               277 µs   261 µs   (-6.1%)     448.2 ms  432.3 ms   -3.5%
  dirichlet_sample                27.5 ms  28.1 ms     +2.2%       1.44 s    1.41 s   -1.8%
  gamma_log_prob                   857 µs   844 µs     -1.6%       2.18 s    2.14 s   -1.7%
  gamma_sample                    10.8 ms  10.7 ms     -1.1%       1.32 s    1.34 s   +1.5%
  lkj_cholesky_sample              1.4 ms   1.5 ms     +2.0%       1.63 s    1.56 s   -4.8%
  mixture_same_family_log_prob     375 µs   365 µs     -2.6%      44.2 ms   42.5 ms   -3.8%
  multivariate_normal_log_prob     278 µs   371 µs  (+33.6%)     317.1 ms  265.3 ms  -16.3%
  normal_log_prob                  346 µs   340 µs     -1.9%      38.2 ms   35.4 ms   -7.4%
  normal_sample                   29.0 ms  29.1 ms     +0.5%     421.1 ms  415.1 ms   -1.4%
  stick_breaking_transform         2.9 ms   2.8 ms     -1.8%     112.7 ms  110.2 ms   -2.3%
+ student_t_log_prob               1.2 ms   1.2 ms     -5.0%      54.4 ms   45.0 ms  -17.3%
  truncated_normal_log_prob        356 µs   359 µs     +0.8%      43.1 ms   41.3 ms   -4.3%

handlers

                                  ───────── run time ────────     ────── compile time ──────
  benchmark                       baseline   this PR        Δ     baseline   this PR       Δ
────────────────────────────────────────────────────────────────────────────────────────────
  initialize_model_hierarchical    22.0 ms   21.8 ms    -0.7%       2.28 s    2.07 s   -9.3%
  log_density_hierarchical          1.6 ms    1.6 ms    -0.8%     686.9 ms  588.3 ms  -14.3%
  nested_handler_stack              628 µs    664 µs  (+5.7%)       865 µs    826 µs   -4.5%
  potential_energy_and_grad          16 µs     16 µs    +1.1%      57.6 ms   51.0 ms  -11.5%
- predictive_forward_sampling     524.6 ms  499.2 ms    -4.8%     134.3 ms  195.4 ms  +45.5%
  trace_seeded_model                338 µs    368 µs  (+8.8%)     422.1 ms  395.9 ms   -6.2%

mcmc

                             ──────── run time ───────     ────── compile time ─────
  benchmark                  baseline   this PR      Δ     baseline  this PR       Δ
────────────────────────────────────────────────────────────────────────────────────
- hmc_logistic_regression    753.9 ms  814.6 ms  +8.0%       1.79 s   2.08 s  +16.1%
- nuts_dense_mass_funnel     938.5 ms  967.8 ms  +3.1%       1.49 s   1.87 s  +25.7%
  nuts_eight_schools           1.29 s    1.32 s  +3.0%       1.57 s   1.82 s  +15.9%
  nuts_hierarchical_glm        1.61 s    1.60 s  -0.8%       2.60 s   3.05 s  +17.4%
  nuts_logistic_regression     1.23 s    1.22 s  -1.2%       1.91 s   2.01 s   +5.0%
  nuts_vectorized_chains       2.52 s    2.54 s  +0.7%       1.99 s   2.02 s   +1.5%

svi

                                             ──────── run time ───────     ────── compile time ─────
  benchmark                                  baseline   this PR      Δ     baseline  this PR       Δ
────────────────────────────────────────────────────────────────────────────────────────────────────
  svi_autodelta_map_logistic                 248.6 ms  252.4 ms  +1.5%       1.82 s   1.83 s   +0.4%
  svi_autodiagonalnormal_hierarchical        544.9 ms  525.0 ms  -3.6%       2.73 s   2.94 s   +7.9%
  svi_automultivariatenormal_eight_schools   462.1 ms  466.5 ms  +1.0%       2.69 s   2.67 s   -0.5%
  svi_autonormal_logistic                    797.9 ms  794.4 ms  -0.4%       2.18 s   2.01 s   -7.6%
  svi_multi_particle_elbo                    950.3 ms  989.0 ms  +4.1%       2.45 s   1.93 s  -21.2%
  svi_trace_mean_field_elbo                  754.6 ms  762.4 ms  +1.0%       3.30 s   3.24 s   -1.9%
Methodology and environment

Each benchmark is set up untimed, then called once with the JAX caches cleared and several more times warm. Run is the fastest warm call; compile is the first call minus that, i.e. the tracing, lowering and XLA compilation the warm calls did not have to pay for.

Both refs were measured on the same runner over 2 interleaved round(s), taking the best observation per benchmark. A result is called neutral when it moves less than ±5% (run) or ±25% (compile), or when the measurement itself is under 1 ms (run) / 50 ms (compile) — a shared CI runner cannot resolve changes below that. Compile time gets the looser band because it is measured once per round rather than best-of-N, and swings by roughly 20% even between two runs of identical code. A delta shown in parentheses did clear its threshold, but on a measurement below the resolution floor, so it is reported without being called a change.

baseline this PR
ref master fix-truncated-normal-tail
commit fc7691c7 bb4e2c95
numpyro 0.22.0 0.22.0
jax 0.12.1 0.12.1
backend cpu cpu
python 3.14.4 3.14.4

Runner: Linux-7.0.0-1019-azure-x86_64-with-glibc2.43, 4 CPUs.

Produced by this benchmark run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review Awaiting review from maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants