Skip to content

Handle large numbers in filter values - #253

Open
peterbroadhurst wants to merge 2 commits into
mainfrom
bignum-filter
Open

peterbroadhurst wants to merge 2 commits into
mainfrom
bignum-filter

Conversation

@peterbroadhurst

Copy link
Copy Markdown
Contributor

If you tried to pass a large JSON number into the JSONQuery structure, it would end up being incorrectly parsed into float64 in the utility during JSON unmarshal, and there's no way for a user of this utility package to avoid that rounding issue.

This PR uses json.Number+big.Rat parsing instead to handle large numbers, while still producing the same plain formatting in the string returned (removing 1.234e10 style syntax).

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
@peterbroadhurst
peterbroadhurst requested a review from a team as a code owner October 1, 2026 17:30
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
@peterbroadhurst
peterbroadhurst added this pull request to stack #255 October 1, 2026 18:58

@EnriqueL8 EnriqueL8 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @peterbroadhurst , looks good - we should definitely fix also query builder path

return toSimpleValue((fftypes.JSONObject{"v": v}).GetString("v"))

return true
}
exp, err := strconv.Atoi(s[i+1:])
limit := maxFilterNumberLength + maxFilterNumberDigits

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could move this limit to a const above

`0.0009765625`: "0.0009765625", // 1/1024 - more places than denominator digits
`9007199254740993e0`: "9007199254740993",
`123456789012345678901.5`: "123456789012345678901.5",
`1e30`: "1000000000000000000000000000000",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: worth adding an uppercase E test

// boundedLiteral reports whether s, a valid JSON number, is short enough and has a small enough
// exponent that the value might fit in maxFilterNumberBits.
// Check before parsing to big.Rat (per gosec G113)
func boundedLiteral(s string) bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that is safe after json.Number conversion, I believe a hex number would still pass this such as 0x1p999999999 and result in gosec 113. json.Number explicitly doesn't allow that hex format

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants