Repository navigation
Improve accuracy and performance of ChainRules derivatives - #138
Merged
Merged
Conversation
- logistic: use e / (1 + e)^2 with e = exp(-|x|) instead of Ω * (1 - Ω), which suffers from cancellation for large x (returns 0 at x = 40) - logabstanh: use 2 / sinh(2x) (one transcendental call instead of two) - log1psq: avoid overflow of x^2 for large |x| - log1mexp, logexpm1: use -inv(expm1(-x)) instead of exp(x - Ω), which amplifies the error of Ω and returns NaN for logexpm1 at Inf - logaddexp, logsubexp: exploit ∂x + ∂y = 1 to compute both partials with a single logistic/expm1 call; fixes NaN for infinite arguments and loss of accuracy for arguments of large magnitude or close to each other - add rules for loglogistic, log1mlogistic, logitexp, logit1mexp - restrict cloglog and cexpexp rules to Real arguments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devmotion
added a commit
to JuliaDiff/DiffRules.jl
that referenced
this pull request
Nov 10, 2026
Port the rules from JuliaStats/LogExpFunctions.jl#138: more accurate logistic, log1psq, log1mexp, logexpm1, logaddexp and logsubexp rules, and new rules for loglogistic, log1mlogistic, logitexp and logit1mexp. Fixes NaN derivatives of logaddexp(±Inf, ±Inf) (JuliaStats/LogExpFunctions.jl#61). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t equal arguments Replace `x - y` with zero only if it is `NaN` (`x == y == ±Inf` for `logaddexp`, `x == y == -Inf` for `logsubexp`): for finite `x == y` it is already zero, and the replacement would drop higher-order derivatives in nested AD. Add tests for equal and `NaN` arguments (#61), compare the `logistic` derivative with an exact reference, and shorten comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tpapp
reviewed
Nov 11, 2026
tpapp
left a comment
Collaborator
There was a problem hiding this comment.
I would keep the original comments in the code, more or less, and if the concern is NaNs, just code it directly, instead of figuring out what will be NaN from the inputs.
| # `x - y` is `NaN` if `x == y == ±Inf`. | ||
| function _∂_logaddexp(x::Real, y::Real) | ||
| Δ = x == y ? zero(x - y) : abs(x - y) | ||
| Δ = x == y && isinf(x) ? zero(x - y) : abs(x - y) |
Collaborator
There was a problem hiding this comment.
why not simply
Suggested change
| Δ = x == y && isinf(x) ? zero(x - y) : abs(x - y) | |
| Δ = abs(x - y) | |
| if isnan(Δ) | |
| Δ = zero(Δ) | |
| end |
which conveys the intention?
Member
Author
There was a problem hiding this comment.
If x or y are NaN, I think one could argue that it should return NaN, not 0. Hence I made the zero branch a bit more restrictive.
| end | ||
| function _∂_logsubexp(x::Real, y::Real) | ||
| Δ = x == y && (isfinite(x) || x < 0) ? zero(x - y) : abs(x - y) | ||
| Δ = x == y == -Inf ? zero(x - y) : abs(x - y) |
Co-authored-by: David Müller-Widmann <devmotion@users.noreply.github.com>
devmotion
added a commit
to JuliaDiff/DiffRules.jl
that referenced
this pull request
Nov 11, 2026
* Add rules for logcosh, logabssinh and logabstanh Requires LogExpFunctions 1 (logabstanh was added in 1.0.0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Use 2/sinh(2x) for logabstanh and drop obsolete hasmethod guard `2 / sinh(2x)` needs one transcendental call instead of two and is more accurate than `inv(cosh(x) * sinh(x))`. With LogExpFunctions >= 1, all unary rules have Float32 and Float64 methods, so the `hasmethod` guard in the tests is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Improve accuracy of LogExpFunctions rules and add missing ones Port the rules from JuliaStats/LogExpFunctions.jl#138: more accurate logistic, log1psq, log1mexp, logexpm1, logaddexp and logsubexp rules, and new rules for loglogistic, log1mlogistic, logitexp and logit1mexp. Fixes NaN derivatives of logaddexp(±Inf, ±Inf) (JuliaStats/LogExpFunctions.jl#61). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Update version from 1.17.0 to 1.17.1 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes several numerical problems in the ChainRules definitions, makes some of them faster, and adds rules for the four functions that had none.
Each new formula was compared against 256-bit BigFloat references in Float64 (and in Float32 for
logistic). Errors are given in ulps and timings in ns per call (Apple Silicon, Julia 1.13). The benchmarks used standalone copies of the formulas, not the extension code itself.logisticΩ * (1 - Ω)→e / (1 + e)^2,e = exp(-abs(x))0atx = 40) → 3.7; Float32: 1.7e7 → 3.3logabstanhinv(cosh(x) * sinh(x))→2 / sinh(2x)log1psq2x / (1 + x^2)→ same forabs(x) < 1, else2 / (x + inv(x))log1mexp-exp(x - Ω)→-inv(expm1(-x))logexpm1exp(x - Ω)→-inv(expm1(-x))Inf→ 1logaddexpexp(x - Ω), exp(y - Ω)→ a singlelogisticcall(Inf, 1)→(1, 0)logsubexp±exp(· - Ω)→ a singleexpm1call(Inf, 1)→(1, -0)logaddexpandlogsubexp. Both satisfyf(x + c, y + c) = f(x, y) + c, so∂x + ∂y = 1. The rules compute the partialtfor the smaller argument directly (logistic(-Δ)or-inv(expm1(Δ)), whereΔ = |x - y|) and set the other partial to1 - t. Sincet ≤ 1/2,1 - tdoes not suffer from cancellation. As in the primal implementations,Δis set to zero ifx - yis NaN (x == y == ±Infforlogaddexp,x == y == -Infforlogsubexp). Unlike the primal implementations, this replacement is not applied for finitex == y: therex - yis already zero, and replacing it would drop higher-order derivatives when the rules are evaluated on dual or tracked numbers whose==only compares primal values.Behaviour changes at
x == y, where the derivative isn't defined:logaddexp(±Inf, ±Inf)now returns(1/2, 1/2)instead of NaN (the ChainRules counterpart of logaddexp introduces NaNs for non-finite values #61).logsubexp(x, x)now returns(Inf, -Inf), the limit asxapproachesyfrom above, instead of(-Inf, Inf).New rules. These functions had no rules:
loglogistic(logistic(-x)),log1mlogistic(-logistic(x)),logitexp(-inv(expm1(x))) andlogit1mexp(inv(expm1(x))).Unchanged. The
log2mexprule stays as it is: the function is ill-conditioned nearlog(2), and theexpm1-based alternative is no more accurate. Thecloglogandcexpexprules are now restricted toRealarguments; their formulas are unchanged.Tests. I added the following tests:
test_scalarfor the four new rules.rtol = 4eps. The old rules fail each of these:logisticat large|x|, in Float64 and Float32log1mexp(-1e-12)logexpm1(6.04e-8)logaddexp(300, 301)logsubexpwith arguments 1e-6 apart, where finite differences are unreliablelog1psq(1e200), which overflowed beforeInf, checked by hand:logexpm1, andlogaddexp/logsubexpwith one infinite argument.logaddexpandlogsubexp, in Float64 and Float32:logaddexp(x, x)forx = -Inf, 1, Inf;logsubexp(x, x)forx = -Inf, 1(and NaN forx = Inf); NaN propagation.The full test suite passes locally.
The same rules are ported to DiffRules (used by ForwardDiff and ReverseDiff) in JuliaDiff/DiffRules.jl#121, which fixes #61.
🤖 Generated with Claude Code