Treat integer hyperparameters as static in inject_hyperparams (fixes #412) - #1730
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
bba7c36 to
d6eb26b
Compare
`inject_hyperparams` converts every numeric argument into a traced array so it can be scheduled. Boolean arguments were already special-cased as static, because a traced boolean breaks Python control flow. Integer arguments have the exact same problem: several optimizers use them for structural decisions, e.g. `min_dim_size_to_factor` in `adafactor` (factorized.py) and `memory_size` in `lbfgs`. Injecting them as traced arrays raises a `TracerBoolConversionError` when the resulting transform is jitted, forcing users to manually pass `static_args=(...)`. Since `bool` is a subclass of `int`, treating `int` as static generalizes the existing behavior and lets `inject_hyperparams(optax.adafactor)` and `inject_hyperparams(optax.lbfgs)` be jitted out of the box. Integer hyperparameters cannot be meaningfully scheduled under jit anyway. Fixes google-deepmind#412.
d6eb26b to
8f973c3
Compare
|
Rebased on main; CLA is signed and CI is green. This fixes #412: Would someone be able to review? |
Summary
optax.inject_hyperparamsconverts every numeric argument into a traced array so it can be scheduled/overridden at runtime. Boolean arguments are already special-cased as static, because a traced boolean can't be used in Python control flow.Integer arguments have the identical problem, but currently fall through to the "numeric" branch and get traced. Several optimizers use integer arguments for structural decisions:
min_dim_size_to_factorinadafactor(optax/_src/factorized.py:55→if shape[...] < min_dim_size_to_factor)memory_sizeinlbfgs(optax/_src/transform.py:1717→if memory_size < 1)As a result, jitting these fails with
TracerBoolConversionErrorunless the user manually passesstatic_args=(...):This is exactly what #412 asks to fix ("all optimizers wrapped in
inject_hyperparamscan be jit compiled without any additionalstatic_args").Fix
Since
boolis a subclass ofint, extending the existing static-value check frombooltointgeneralizes the current behavior and resolves both optimizers with no new special cases. Integer hyperparameters cannot be meaningfully scheduled under jit anyway (a schedule returns floats, and structural ints break control flow when traced).Verification
inject_hyperparams(optax.adafactor)andinject_hyperparams(optax.lbfgs)nowjax.jitout of the box; the previously-requiredstatic_argsworkaround is no longer needed.optax/schedules/_inject_test.py.pytest optax/schedules/ optax/_src/alias_test.py→ 634 passed, 70 skipped, no regressions.Fixes #412.