Skip to content

Improve accuracy and performance of ChainRules derivatives - #138

Merged
devmotion merged 5 commits into
masterfrom
dmw/chainrules-accuracy
Nov 11, 2026
Merged

devmotion merged 5 commits into
masterfrom
dmw/chainrules-accuracy

Conversation

@devmotion

@devmotion devmotion commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

This PR fixes several numerical problems in the ChainRules definitions, makes some of them faster, and adds rules for the four functions that had none.

Each new formula was compared against 256-bit BigFloat references in Float64 (and in Float32 for logistic). Errors are given in ulps and timings in ns per call (Apple Silicon, Julia 1.13). The benchmarks used standalone copies of the formulas, not the extension code itself.

Rule Old → new Max error (ulp) Time (ns)
logistic Ω * (1 - Ω) → e / (1 + e)^2, e = exp(-abs(x)) ∞ (returns 0 at x = 40) → 3.7; Float32: 1.7e7 → 3.3 4.4 → 3.8
logabstanh inv(cosh(x) * sinh(x)) → 2 / sinh(2x) 3.2 → 2.2 14.0 → 7.2
log1psq 2x / (1 + x^2) → same for abs(x) < 1, else 2 / (x + inv(x)) overflows for |x| ≳ 1e154 → 1.4 –
log1mexp -exp(x - Ω) → -inv(expm1(-x)) 31 → 1.5 24.5 → 23.9
logexpm1 exp(x - Ω) → -inv(expm1(-x)) 25 → 1.8; NaN at Inf → 1 21.1 → 19.5
logaddexp exp(x - Ω), exp(y - Ω) → a single logistic call 511 → 2.0; NaN at (Inf, 1) → (1, 0) 27.6 → 24.9
logsubexp branched ±exp(· - Ω) → a single expm1 call 386 → 1.5; NaN at (Inf, 1) → (1, -0) 29.0 → 23.2

logaddexp and logsubexp. Both satisfy f(x + c, y + c) = f(x, y) + c, so ∂x + ∂y = 1. The rules compute the partial t for the smaller argument directly (logistic(-Δ) or -inv(expm1(Δ)), where Δ = |x - y|) and set the other partial to 1 - t. Since t ≤ 1/2, 1 - t does not suffer from cancellation. As in the primal implementations, Δ is set to zero if x - y is NaN (x == y == ±Inf for logaddexp, x == y == -Inf for logsubexp). Unlike the primal implementations, this replacement is not applied for finite x == y: there x - y is already zero, and replacing it would drop higher-order derivatives when the rules are evaluated on dual or tracked numbers whose == only compares primal values.

Behaviour changes at x == y, where the derivative isn't defined:

New rules. These functions had no rules: loglogistic (logistic(-x)), log1mlogistic (-logistic(x)), logitexp (-inv(expm1(x))) and logit1mexp (inv(expm1(x))).

Unchanged. The log2mexp rule stays as it is: the function is ill-conditioned near log(2), and the expm1-based alternative is no more accurate. The cloglog and cexpexp rules are now restricted to Real arguments; their formulas are unchanged.

Tests. I added the following tests:

  • test_scalar for the four new rules.
  • Comparisons against BigFloat references with rtol = 4eps. The old rules fail each of these:
    • logistic at large |x|, in Float64 and Float32
    • log1mexp(-1e-12)
    • logexpm1(6.04e-8)
    • logaddexp(300, 301)
    • logsubexp with arguments 1e-6 apart, where finite differences are unreliable
    • log1psq(1e200), which overflowed before
  • Cases at Inf, checked by hand: logexpm1, and logaddexp/logsubexp with one infinite argument.
  • Equal and NaN arguments of logaddexp and logsubexp, in Float64 and Float32: logaddexp(x, x) for x = -Inf, 1, Inf; logsubexp(x, x) for x = -Inf, 1 (and NaN for x = Inf); NaN propagation.

The full test suite passes locally.

The same rules are ported to DiffRules (used by ForwardDiff and ReverseDiff) in JuliaDiff/DiffRules.jl#121, which fixes #61.

🤖 Generated with Claude Code

devmotion and others added 3 commits November 10, 2026 13:34
- logistic: use e / (1 + e)^2 with e = exp(-|x|) instead of Ω * (1 - Ω),
  which suffers from cancellation for large x (returns 0 at x = 40)
- logabstanh: use 2 / sinh(2x) (one transcendental call instead of two)
- log1psq: avoid overflow of x^2 for large |x|
- log1mexp, logexpm1: use -inv(expm1(-x)) instead of exp(x - Ω), which
  amplifies the error of Ω and returns NaN for logexpm1 at Inf
- logaddexp, logsubexp: exploit ∂x + ∂y = 1 to compute both partials with
  a single logistic/expm1 call; fixes NaN for infinite arguments and loss
  of accuracy for arguments of large magnitude or close to each other
- add rules for loglogistic, log1mlogistic, logitexp, logit1mexp
- restrict cloglog and cexpexp rules to Real arguments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@devmotion
devmotion requested a review from tpapp November 10, 2026 14:47
devmotion added a commit to JuliaDiff/DiffRules.jl that referenced this pull request Nov 10, 2026
Port the rules from JuliaStats/LogExpFunctions.jl#138: more accurate logistic, log1psq, log1mexp, logexpm1, logaddexp and logsubexp rules, and new rules for loglogistic, log1mlogistic, logitexp and logit1mexp. Fixes NaN derivatives of logaddexp(±Inf, ±Inf) (JuliaStats/LogExpFunctions.jl#61).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t equal arguments

Replace `x - y` with zero only if it is `NaN` (`x == y == ±Inf` for `logaddexp`, `x == y == -Inf` for `logsubexp`): for finite `x == y` it is already zero, and the replacement would drop higher-order derivatives in nested AD. Add tests for equal and `NaN` arguments (#61), compare the `logistic` derivative with an exact reference, and shorten comments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@tpapp tpapp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep the original comments in the code, more or less, and if the concern is NaNs, just code it directly, instead of figuring out what will be NaN from the inputs.

Comment thread ext/LogExpFunctionsChainRulesCoreExt.jl Resolved
Comment thread ext/LogExpFunctionsChainRulesCoreExt.jl Resolved
# `x - y` is `NaN` if `x == y == ±Inf`.
function _∂_logaddexp(x::Real, y::Real)
Δ = x == y ? zero(x - y) : abs(x - y)
Δ = x == y && isinf(x) ? zero(x - y) : abs(x - y)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not simply

Suggested change
Δ = x == y && isinf(x) ? zero(x - y) : abs(x - y)
Δ = abs(x - y)
if isnan(Δ)
Δ = zero(Δ)
end

which conveys the intention?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If x or y are NaN, I think one could argue that it should return NaN, not 0. Hence I made the zero branch a bit more restrictive.

end
function _∂_logsubexp(x::Real, y::Real)
Δ = x == y && (isfinite(x) || x < 0) ? zero(x - y) : abs(x - y)
Δ = x == y == -Inf ? zero(x - y) : abs(x - y)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similarly as above

Comment thread test/chainrules.jl Resolved
Co-authored-by: David Müller-Widmann <devmotion@users.noreply.github.com>

@tpapp tpapp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@devmotion
devmotion merged commit d404e1f into master Nov 11, 2026
6 checks passed
@devmotion
devmotion deleted the dmw/chainrules-accuracy branch November 11, 2026 09:58
devmotion added a commit to JuliaDiff/DiffRules.jl that referenced this pull request Nov 11, 2026
* Add rules for logcosh, logabssinh and logabstanh

Requires LogExpFunctions 1 (logabstanh was added in 1.0.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Use 2/sinh(2x) for logabstanh and drop obsolete hasmethod guard

`2 / sinh(2x)` needs one transcendental call instead of two and is more
accurate than `inv(cosh(x) * sinh(x))`. With LogExpFunctions >= 1, all
unary rules have Float32 and Float64 methods, so the `hasmethod` guard
in the tests is no longer needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Improve accuracy of LogExpFunctions rules and add missing ones

Port the rules from JuliaStats/LogExpFunctions.jl#138: more accurate logistic, log1psq, log1mexp, logexpm1, logaddexp and logsubexp rules, and new rules for loglogistic, log1mlogistic, logitexp and logit1mexp. Fixes NaN derivatives of logaddexp(±Inf, ±Inf) (JuliaStats/LogExpFunctions.jl#61).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Update version from 1.17.0 to 1.17.1

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

logaddexp introduces NaNs for non-finite values

2 participants