Skip to content

Accurate derivative of tanh for large |x| - #874

Open
devmotion wants to merge 1 commit into
masterfrom
dmw/accurate-tanh
Open

devmotion wants to merge 1 commit into
masterfrom
dmw/accurate-tanh

Conversation

@devmotion

Copy link
Copy Markdown
Member

The DiffRules rule for tanh is 1 - tanh(x)^2 (JuliaDiff/DiffRules.jl#102). Once tanh(x) rounds to ±1, it returns 0. In Float64 the relative error is already 1e-8 at x = 10 and reaches 100% for |x| ≥ 20.

This PR defines tanh(::Dual) by hand, next to sin/cos, and skips the DiffRules rule:

deriv = abs(x) < 1 ? 1 - val^2 : sech(x)^2

This is the same split as JuliaDiff/ChainRules.jl#864:

  • For |x| < 1 the derivative reuses the primal value, as before, and is accurate there.
  • Otherwise it uses sech(x)^2, which is accurate everywhere.

DiffRules can't use this branch itself, because Symbolics evaluates DiffRules' expressions on symbolic arguments. JuliaDiff/DiffRules.jl#122 switches the rule to the branch-free sech(x)^2 instead. Without this PR, ForwardDiff would pay for an extra cosh for all x. This PR works with both the current and the updated DiffRules.

Performance

ns per ForwardDiff.derivative(tanh, x), broadcast over 10⁴ points:

x before after
uniform in (-1, 1) 7.74 7.97
20 · randn 5.70 10.96
uniform in (20, 30) 3.76 8.84

The extra cost for |x| ≥ 1 is the cosh that's needed for an accurate result. At those inputs the old result was inaccurate, or exactly 0.

Tests

  • The DiffRules consistency test in DualTest.jl compares with ≈ for tanh, since the formula differs for |x| < 1.
  • New testset: accuracy against BigFloat (rtol = 4 eps) for Float32 and Float64 up to |x| = 300, for plain and nested duals.

🤖 Generated with Claude Code

DiffRules' `1 - tanh(x)^2` returns 0 once `tanh(x)` rounds to ±1
(JuliaDiff/DiffRules.jl#102). Define `tanh(::Dual)` by hand: reuse the
primal value for `|x| < 1` and use `sech(x)^2` otherwise, as in
JuliaDiff/ChainRules.jl#864.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.28%. Comparing base (70ef06c) to head (cbd429b).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #874      +/-   ##
==========================================
+ Coverage   91.24%   91.28%   +0.04%     
==========================================
  Files          11       11              
  Lines        1085     1090       +5     
==========================================
+ Hits          990      995       +5     
  Misses         95       95              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant