Repository navigation
Avoid catastrophic cancellation in the derivatives of tanh and inverse trigonometric/hyperbolic functions - #122
Merged
Conversation
…erivatives `1 - tanh(x)^2` returns 0 once `tanh(x)` rounds to ±1 (#102); use the branch-free `sech(x)^2`. Rules involving `1 - x^2` or `x^2 - 1` lose accuracy near `|x| = 1`; use the factorized forms `(1 - x) * (1 + x)` and `(x - 1) * (x + 1)` instead, as in JuliaDiff/ChainRules.jl#864. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use `sqrt(x - 1) * sqrt(x + 1)` as for `Base.acosh`: `(x - 1) * (x + 1)` overflows for large `x`, and the derivative is now NaN for `x < 1`, like the primal. Test both rules at a point where `x^2` overflows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #102. This ports the changes of JuliaDiff/ChainRules.jl#864 to DiffRules, as far as possible without branches.
Changes
tanh:1 - tanh(x)^2→sech(x)^2. Oncetanh(x)rounds to ±1, the old form returns 0. In Float64 that happens for |x| ≥ 20, where the relative error is 100%, and it is already 1e-8 at x = 10.sech(x)^2is accurate everywhere, with a relative error ≤ 4e-16 in the tested range up to |x| = 300.1 - x^2→(1 - x) * (1 + x)inasin,acos,asind,acosd,atanh,acoth,asech,NaNMath.asin,NaNMath.acosandNaNMath.atanh.x^2 - 1→(x - 1) * (x + 1)inasec,acsc,asecdandacscd.acoshandNaNMath.acoshusesqrt(x - 1) * sqrt(x + 1), as in ChainRules. This also avoids the overflow ofx^2for largex(e.g., the derivative ofacoshat1e200was 0), and makes the derivative ofNaNMath.acoshNaN forx < 1, like the function itself.log10/log2:inv(x) / logten→inv(logten * x), the same form as theNaNMathrules.The factored forms are exact for
xnear ±1 (Sterbenz) and cost the same number of operations.Why no branches
A rule like ChainRules'
abs(x) < 1 ? 1 - Ω^2 : sech(x)^2would be faster in some consumers. But DiffRules' expressions are also evaluated on symbolic arguments: Symbolics builds its derivative rules from them. A value branch would makeSymbolics.derivative(tanh(x), x)throw for every released Symbolics version.sech(x)^2has no branch and equals the derivative as a function. That also makes it valid for Taylor-mode AD, which evaluates the expression on truncated series (JuliaDiff/TaylorDiff.jl#113).Performance
The new forms give the same derivatives up to rounding. Measured with ForwardDiff (ns per
ForwardDiff.derivativecall, broadcast over 10⁴ points):asin/acos/atanh(|x| < 0.9)acosh/asec/acoth(1.1 < x < 6.1)sech(x)^2costs onecoshmore than1 - tanh(x)^2in consumers that reuse the primal value, as ForwardDiff does through CSE. ForwardDiff keeps its fast path for |x| < 1 by definingtanhby hand (JuliaDiff/ForwardDiff.jl#874).Downstream note
Symbolics' derivative expressions change shape, e.g.
Symbolics.derivative(asin(x), x)becomes1 / sqrt((1 - x) * (1 + x)). Downstream tests that compare printed expressions may need to be updated.Tests
New accuracy tests compare every changed rule, in Float32 and Float64, against the rule evaluated in 256-bit
BigFloat(rtol = 8 eps), near |x| = 1 and for large |x| (tanh, andacoshat a point wherex^2overflows). Every one of these test sets fails with the previous rules. All tests pass (1214 in total). ForwardDiff's test suite (v1.4.7) also passes with this branch.🤖 Generated with Claude Code