Fix floating point drift in generated linear scale ticks - #12282
Open
UlikGames wants to merge 1 commit into
Open
Fix floating point drift in generated linear scale ticks#12282UlikGames wants to merge 1 commit into
UlikGames wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12281
Reproduction from the issue: https://codepen.io/lamasse/pen/MYJzVpz
generateTicksrounds every tick withMath.round(tickValue * factor) / factorso the values reachingticks.callbackcome out clean. The factor comes from_decimalPlaces(niceMin), but withbounds: 'ticks'niceMin is itselfMath.floor(rmin / spacing) * spacing. For min 49.894 / max 51.5264 that is 49.800000000000004 instead of 49.8,_decimalPlacesreports 16, the factor becomes 1e16, and rounding to the 16th decimal does nothing.Before:
49.894, 50.00000000000001, 50.2, 50.400000000000006, 50.6, 50.800000000000004, 51.00000000000001, 51.2, 51.400000000000006, 51.5264
After:
49.894, 50, 50.2, 50.4, 50.6, 50.8, 51, 51.2, 51.4, 51.5264
The built-in formatter hides the drift, but a callback like
value => Number.isInteger(value) ? value : ''labels nothing at all.niceMin and niceMax are whole multiples of spacing, so they never need more decimals than spacing itself. Rounding them to that precision where they are computed is exact rather than a fudge, and it also covers the
countbranch below, which reuses the same value.Note this does change tick values for charts with non-round bounds. The values move to what they were always meant to be, and every image-based test still matches pixel for pixel, but it is a visible change rather than a pure internal cleanup.
Added a regression spec; it fails on master with the "before" list above.
Verified locally: karma 1703/1703 in Chrome including the image tests, eslint on js and md, tsc declarations and test/types, and the node, node-commonjs and typescript-node integration tests.