Skip to content

Speed up parsing of contiguous input: numbers, strings, and UTF-8 (with optional simdutf)#5283

Draft
nlohmann wants to merge 14 commits into
developfrom
claude/parser-performance-research-4hkdf8
Draft

Speed up parsing of contiguous input: numbers, strings, and UTF-8 (with optional simdutf)#5283
nlohmann wants to merge 14 commits into
developfrom
claude/parser-performance-research-4hkdf8

Conversation

@nlohmann

@nlohmann nlohmann commented Jul 20, 2026

Copy link
Copy Markdown
Owner

What & why

The parser reads input one character at a time through the input adapter and converts numbers with strtod/strtoull. For the common case of contiguous byte input (std::string, std::vector<char>/<std::uint8_t>, string literals, const char* ranges), the per-character get()/add() overhead and the locale/errno-heavy number conversion dominate. This PR adds fast paths for that case, borrowing ideas from the simdjson / fast_float ecosystem, while keeping the library header-only with a C++11 fallback on every path. Streaming inputs (files, std::istream, wide strings, user adapters) are untouched and keep the byte-at-a-time scanner. Error handling is unchanged throughout: every fast path defers to the byte-path scanner on anything malformed, so parse errors are reported at the same position with the same message.

The changes (each independent and reversible)

  1. Number conversion — replace strtoull/strtoll/strtod over the validated digit buffer with dedicated parsers: hand-rolled integer parsing with overflow→float fallback (matching the old semantics exactly), and Clinger's exact fast path for double, bit-identical to strtod on its subset and declining otherwise. float/long double keep strtof/strtold.
  2. SWAR string scanning — scan 8 bytes at a time with a uint64_t word test to bulk-append ordinary characters. Pure C++11, no intrinsics.
  3. UTF-8 bulk validation — fold well-formed multibyte sequences into the bulk run (validated with the same ranges as the byte-path switch), so non-ASCII text no longer drops to the slow per-byte path.
  4. Contiguous-container pointer adapter — route std::string & friends through a pointer-based adapter so 1–3 apply to json::parse(std::string) in C++11–17 (previously only const char*/literals, and C++20 contiguous iterators, qualified). The pointer keeps the container's element type, so char_type — and all behavior, including binary formats from std::vector<std::uint8_t> — is unchanged.
  5. Contiguous number fast path — parse the whole number token straight from the buffer (no per-byte get()/add()), materializing token_buffer in one copy with the locale decimal point substituted, then reusing the shared convert_number() tail.
  6. Optional simdutf backend (JSON_USE_SIMDUTF, off by default) — since simdutf ships a .cpp and uses runtime dispatch, it is not vendored; when defined, bulk UTF-8 validation routes through simdutf::validate_utf8 (the user supplies/links simdutf), otherwise the C++11 scalar validator is used. The library stays header-only either way.
  7. Refactor — the byte-level helpers are moved out of lexer.hpp (2357 → 1934 lines) into two focused headers, detail/input/string_scan.hpp and detail/input/number_parse.hpp, as free functions.

Measured gains

Cumulative, json::parse / accept(std::string), C++11, g++ 13 -O3, vs the v3.12.0 baseline (representative synthetic data):

dataset parse accept
floats 1.6× 1.8×
ints 1.2× 1.3×
strings (twitter-like) 1.3× 1.7×
long ASCII strings 8.5× 18×
dense CJK 2.4× 2.3×
mixed international 2.1× 2.2×

accept()/SAX/validation see the largest gains (not allocation-bound); full-DOM parse() gains on object-heavy input are capped by DOM allocation, which is a separate concern. With JSON_USE_SIMDUTF, dense-UTF-8 validation has further headroom (~4.7 GB/s ceiling measured with an SSE validator; ~7× the scalar path).

Verification

  • Millions of differential-fuzz documents (arbitrary bytes; overlong/surrogate/truncated UTF-8; overflow-range integers, long digit strings, %.17g doubles; escapes) parse and accept/reject identically via the contiguous path and the streaming byte path; the JSON_USE_SIMDUTF build matches too.
  • Number overflow boundaries verified; a 3M random-double fuzz matches strtod bit-for-bit.
  • Existing lexer/parser/diagnostic-position/deserialization/conversions/locale suites pass (error messages/positions unchanged); CBOR/MsgPack round-trips from std::vector<std::uint8_t> unchanged; a new "lexer number fast path" test checks contiguous-vs-streaming parity, token classification, and identical rejection of malformed numbers.
  • Warning-clean on g++ and clang, C++11/17/20, in both build configurations; clang-tidy clean on the new headers; make check-amalgamation clean.

Notes for reviewers / remaining work

  • No dedicated tests were added yet for the string/UTF-8/adapter paths — they are currently covered by differential fuzzing and the existing suites, not by tests/ cases, so coverage is not yet back at 100%. The fuzz harnesses can be adapted into tests/.
  • A JSON_USE_SIMDUTF CMake option (find_package(simdutf) + link + define) is not included yet.
  • The version noted in the JSON_USE_SIMDUTF doc page (3.12.1) is a placeholder for you to set.

  • The changes are described in detail, both the what and why.
  • If applicable, an existing issue is referenced. (none)
  • The code coverage remained at 100%. A test case for every new line of code. (partial: a number-fast-path test was added; the string/UTF-8/adapter paths are covered by differential fuzzing and existing suites, dedicated tests still to add)
  • If applicable, the documentation is updated. (added the JSON_USE_SIMDUTF macro page + nav entry)
  • The source code is amalgamated by running make amalgamate.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA

nlohmann and others added 6 commits July 20, 2026 09:51
…simdjson world)

The number scanner converted its already-validated digit buffer with
std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno
machinery and dominate number-heavy parsing (strtod runs at ~6 M/s).

Replace them with dedicated parsers over the validated buffer:

- parse_integer_unsigned / parse_integer_signed: accumulate digits with
  overflow detection, falling back to the float path on overflow exactly
  as the strtoull/strtoll round-trip check did. Overflow behavior is
  unchanged for narrower or wider custom number types.

- parse_float_fast: Clinger's exact fast path for `double` (<=19 significant
  digits, |exp10| <= 22, significand < 2^53), where significand * 10^exp is
  exact under IEEE round-to-nearest. This is the same fast path used by
  fast_float/simdjson. It is bit-identical to strtod on this subset and
  declines (falling back to strtod) otherwise. Only `double` uses it; float
  and long double keep std::strtof/std::strtold via a templated overload.

Measured on representative data (g++ 13, -O3):
  - integers:  DOM parse +11%, SAX +25-34%
  - floats:    DOM parse +37%, SAX +70%  (clang: float DOM ~1.9x)

No dependencies added; header-only and C++11-clean. Existing parser,
lexer, conversion and deserialization unit tests pass unchanged; a
3M-value random-double fuzz matches strtod bit-for-bit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_string() read the input one character at a time through the input
adapter and classified every byte with a large switch. For contiguous
byte buffers we can instead scan 8 bytes at a time with a SWAR word test
that finds the first byte needing individual handling (the closing quote,
an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append
the ordinary run in one go.

- input adapters expose supports_bulk_scan / bulk_data / bulk_remaining /
  bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges
  (raw pointers in every standard; std::string/std::vector/std::array and
  friends additionally in C++20 via std::contiguous_iterator).
- the lexer gains a bulk_scan capability (gated on lazy_token_string so
  bypassing the per-character capture cannot lose error diagnostics) and a
  scan_string_bulk() fast path; streaming/wide/user adapters are unchanged
  and keep the byte-at-a-time scanner.

The run contains no newline (all bytes < 0x20 are treated as special), so
position bookkeeping stays exact, and error tokens are still reconstructed
lazily from the consumed byte range. The SWAR special-byte test is pure
uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean.

Measured on representative data, pointer input, g++ 13 -O3
(string values discarded by accept() see the largest gains):

  long ASCII strings:  DOM +4.5x,  SAX +14x,   accept +17x  (to ~2 GB/s)
  short strings:       DOM +15%,   SAX +62%,   accept +85%
  escape-heavy:        DOM +31%,   SAX +26%,   accept +28%

Same-input parity verified: 200k randomized documents (escapes, multibyte
UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR
path and the streaming byte path; unit lexer/parser/diagnostic-position/
deserialization/conversions suites pass unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…I text)

The SWAR bulk string path stopped at the first non-ASCII byte and handed
every multibyte character to the byte-at-a-time scanner, whose per-byte
get()/next_byte_in_range()/add() machinery runs at roughly half the speed
of validating straight from the buffer. As a result, dense non-ASCII text
(CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII.

Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a
non-ASCII lead byte, validates one sequence with validate_one_utf8() -
which mirrors scan_string()'s per-byte switch ranges exactly (rejecting
overlong forms, surrogates, and out-of-range code points) - and appends it
in place, continuing until the closing quote, an escape, a control byte,
or an ill-formed sequence. All error handling still defers to the byte
path, so error messages and positions are byte-for-byte unchanged.

Because only well-formed content is fast-pathed and every rejection falls
through to the existing scanner, behavior is identical; the win is purely
throughput. Measured on pointer input (accept, string values discarded):

  content        g++ 13         clang 18
  dense CJK      277 -> 648     ~605  MB/s   (~2.3x)
  dense emoji    299 -> 857     ~702  MB/s   (~2.6-2.9x)
  mixed 90% ASCII 246 -> 331    ~334  MB/s   (~1.35x)
  pure ASCII     unchanged (~3.2 / 4.1 GB/s)

Verified: 2,000,000 randomized documents built from arbitrary bytes
(overlong, surrogate, truncated, out-of-range sequences) accept/reject and
parse identically via the contiguous path and the streaming byte path;
lexer/parser/diagnostic-position/deserialization/conversions suites pass
unchanged. Pure C++11, no intrinsics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…ths in C++11)

json::parse(std::string) - the most common entry point - did not benefit
from the contiguous fast paths (bulk string scanning, UTF-8 bulk
validation, memcpy for binary formats) in C++11..17: std::string::iterator
is a library wrapper, not a raw pointer, and pre-C++20 there is no portable
way to prove it contiguous, so supports_bulk_scan was false. Only raw
pointers, string literals, and C-arrays (and, in C++20, anything modelling
std::contiguous_iterator) took the fast path.

Detect contiguous single-byte containers (std::string, std::vector<char>,
std::vector<std::uint8_t>, std::string_view, ...) via is_contiguous_byte_
container and route them through an iterator_input_adapter built from
data()/data()+size(). The generic iterator-based container overload is
constrained to exclude these, so the two overloads are disjoint and there
is no ambiguity (a plain competing overload loses to the greedy
forwarding-reference container overload on reference binding, and a factory
partial-specialization is ambiguous - both were tried and rejected).

The pointer keeps the container's own element type, so char_type - and
therefore all parsing behavior - is byte-for-byte identical to the iterator
path (const char* for std::string, const std::uint8_t* for
std::vector<std::uint8_t>); only the raw pointer additionally turns on the
fast paths. Lifetimes are unchanged: the container outlives the adapter for
the full parse expression, exactly as the iterators it replaces did.

Measured, C++11, json::parse/accept(std::string), g++ 13:

  long ASCII strings:  accept 201 -> 3200 MB/s  (~16x),  parse 174 -> 1444
  dense CJK:           accept 263 ->  697 MB/s  (~2.6x)
  short strings:       accept 163 ->  243 MB/s  (~1.5x)

Verified: char_type preserved for std::string (char) and
std::vector<std::uint8_t> (uint8_t); CBOR/MsgPack round-trips from
std::vector<std::uint8_t> unchanged; 1,000,000 randomized documents accept
and parse identically via std::string and via std::istream;
deserialization/user-defined-input/parser/lexer/conversions/diagnostic-
position suites pass (20,480 assertions); warning-clean on g++ and clang in
C++11/17/20.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…UTF)

The bulk string scanner validates UTF-8 straight from a contiguous buffer.
The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD
validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8
validation (easy to get subtly wrong - a from-scratch SSE attempt rejected
valid CJK), wire in the vetted simdutf library behind an opt-in switch.

simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU
dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes
<simdutf.h> and routes the bulk validator through simdutf::validate_utf8;
the project supplies and links simdutf. Undefined (the default), nothing
external is included and the portable C++11 scalar path is used, so the
library stays header-only and its baseline behavior is unchanged.

Design keeps behavior identical either way:
- scan_string_bulk() now finds the run up to the next quote/escape/control
  byte (non-ASCII allowed) and validates it in one shot; on the rare
  validation failure it recomputes the exact valid prefix with the scalar
  helper, so ill-formed input still falls through to the byte path and is
  reported at the same position with the same message.
- the per-sequence scalar path is factored into scalar_string_bulk_run()
  and is the default backend; the refactor is behavior-preserving and does
  not change scalar throughput.

Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse
identically across 2,000,000 arbitrary-byte documents and 1,000,000
mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte
path); lexer/parser/diagnostic-position/deserialization suites pass under
both configurations (20,188 assertions with the backend enabled);
warning-clean on g++ and clang, C++11 and C++20, both configurations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_number() reads a number one character at a time through the input
adapter (get()) and appends each byte to token_buffer (add()) before
converting. For contiguous input, the per-character get()/add() overhead
dominates: it is roughly two thirds of the time spent on number-heavy
parsing, far more than the value conversion itself.

Add scan_number_bulk_contiguous(), which parses the whole number token
straight from the input buffer: it validates and classifies the extent
with the same grammar as scan_number()'s state machine, materializes
token_buffer in one copy (substituting the locale decimal point exactly as
scan_number() does), advances the adapter, and reuses the shared
convert_number() tail. On anything it does not recognize as a well-formed
number it makes no state change and returns token_type::uninitialized, so
the caller falls back to scan_number(), which then produces the exact
diagnostic. Errors and their positions are therefore unchanged.

The conversion tail is factored out of scan_number() into convert_number()
so both scanners share it; the fast path is selected by tag dispatch on the
existing bulk_scan capability, so streaming/wide/user adapters are
unaffected.

Measured on pointer input, g++ 13 -O3:
  - integers: parse +65%, accept +98%
  - floats:   parse +39%, accept +70%

Verified: 2,000,000 randomized number documents (including overflow-range
integers, long digit strings and %.17g doubles) parse identically via the
contiguous path and the streaming byte path, matching value, type and
round-trip text; the locale suite and existing parser/lexer/conversions/
deserialization tests pass; a new "lexer number fast path" test checks
contiguous-vs-streaming parity, token classification, and that malformed
numbers are rejected identically. Pure C++11, no intrinsics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the claude/parser-performance-research-4hkdf8 branch from c724f41 to 84939c1 Compare July 20, 2026 09:51
@github-actions github-actions Bot added the tests label Jul 20, 2026
nlohmann and others added 2 commits July 20, 2026 10:02
lexer.hpp had grown by ~600 lines of byte-level helpers that have no
dependency on the lexer's template parameters and clutter the state
machine. Move them, unchanged, into two focused headers as free functions
in namespace detail:

- number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now
  templated on the number type) and parse_float_fast (Clinger's exact
  double fast path, with the decimal point passed as an argument instead of
  read from a lexer member).
- string_scan.hpp: the SWAR string helpers (is_string_special,
  swar_string_special, find_string_special, validate_one_utf8,
  scalar_string_bulk_run) and the backend-dispatched string_bulk_run,
  including the optional simdutf include and find_string_delimiter.

lexer.hpp now includes these and calls the free functions; the methods that
touch lexer state (scan_string, scan_number, scan_string_bulk,
scan_number_bulk_contiguous, convert_number) stay put. This is a pure code
move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the
now-unused <cstdint>/<cstring>/<limits> includes are removed, and the
free-function form makes the SWAR helpers reusable elsewhere (e.g. the
serializer's string escaping).

Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and
2,000,000 arbitrary-byte-string differential-fuzz documents parse
identically to before; lexer/parser/conversions/deserialization/locale/
diagnostic-position suites pass (20,576 assertions); warning-clean on g++
and clang in C++11/17/20; the amalgamation regenerates and passes
check-amalgamation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- number_parse.hpp: use std::array for the powers-of-ten table
  (avoid-c-arrays) and `auto` for the cast-initialized result
  (modernize-use-auto), matching the codebase style (cf. the serializer's
  utf8d table). Indexing casts keep the -Wsign-conversion build clean.
- add JSON_USE_SIMDUTF to the mkdocs navigation so the macro page is
  reachable.

No behavior change; clang-tidy is clean on the new headers and the
amalgamation is regenerated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann nlohmann changed the title Speed up parsing: fast number/string/UTF-8 paths for contiguous input (with optional simdutf) Speed up parsing of contiguous input: numbers, strings, and UTF-8 (with optional simdutf) Jul 20, 2026
nlohmann and others added 5 commits July 20, 2026 10:15
The contiguous number fast path materialized token_buffer with
token_buffer.assign(data, len), but string_t is only required to provide
the minimal interface the rest of the lexer uses (push_back, append,
clear, operator[], ...). Custom string types such as the test's alt_string
do not implement assign(), so scan_number_bulk_contiguous() failed to
compile for them (unit-alt-string), breaking the gcc/clang standards and
old-compiler CI jobs.

reset() already clears token_buffer, so fill it with append() - which
alt_string and std::string both provide and which the string fast path
already relies on - instead of assign().

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
…rence

The CI clang-tidy (newer than the locally available version) reported two
additional checks on the new code:

- readability-math-missing-parentheses: parenthesize the (a * b) + c digit
  accumulations in number_parse.hpp.
- cppcoreguidelines-missing-std-forward: the contiguous-byte-container
  input_adapter overload took a forwarding reference but only reads
  data()/size() and never forwards it. It is already disjoint from the
  generic container overload via SFINAE, so a plain const& is correct and
  clearer (and keeps the container alive for the whole parse just as before).

No behavior change; char_type and routing are unchanged (std::string and
std::vector<std::uint8_t> still take the pointer adapter with char/uint8_t
char_type), CBOR/MsgPack round-trips and the 2M number fuzz still pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The Clinger fast path is exact only for the "easy" subset (<=19 significant
digits, |exp10| <= 22); high-precision and scientific floats fall through to
strtod, where the failed Clinger attempt actually makes parsing a net loss.
std::from_chars implements the Eisel-Lemire algorithm in modern standard
libraries: locale-independent, correctly rounded, and fast over the whole
value range.

convert_number() now tries parse_float_from_chars() first (guarded by
__cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the
Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is
used only when it consumes the entire token; a partial parse means a non-'.'
locale decimal point, and an under-/overflow (result_out_of_range) also
declines - in both cases the existing strtod fallback supplies the exact
value and the well-defined +/-inf/0 the parser expects, side-stepping the
P4168 divergence between implementations. float and long double now get the
fast path too (Clinger was double-only).

Measured, C++17, g++ 13 -O3, json::parse/accept:
  - canada-style floats: ~unchanged (Clinger already covered them)
  - high-precision (17 digits):  parse 2.1x, accept 2.5x
  - scientific (17 digits + exp): parse 3.6x, accept 4.1x

Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value -
including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow -
to bit-identical results; 2M number-fuzz clean; conversions/deserialization/
locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy
clean; warning-clean on g++ and clang in C++11/17/20.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
libstdc++ 15 defines __cpp_lib_to_chars even in C++14 mode (via
bits/version.h pulled in by other headers), but <charconv> is only included
under JSON_HAS_CPP_17. That made parse_float_from_chars() reference
std::from_chars without the header in C++14 builds, breaking gcc-latest,
icpx, and the offline-testdata jobs.

Gate the use on JSON_HAS_CPP_17 && __cpp_lib_to_chars so it matches the
include condition exactly; C++11/14 always take the scalar fallback.
Verified by forcing __cpp_lib_to_chars in a C++14 build: the guard
suppresses std::from_chars and it compiles. C++17 behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The contiguous number fast path uses a Clinger-style exact algorithm
(significand * 10^scale in double arithmetic), which is only correctly
rounded when double operations are evaluated in true 53-bit precision.
On the x87 FPU used by 32-bit x86 (FLT_EVAL_METHOD == 2) the single
multiply/divide is computed in 80-bit and then double-rounded to double,
so a small fraction of values land 1 ULP off.

This surfaced as test-cbor_cpp11 and test-msgpack_cpp11 failing on the
mingw (x86) job for regression/floats.json: the C++17 builds pass because
they take the correctly-rounded std::from_chars path, while C++11 falls
back to parse_float_fast(). A 5M-sample check over shortest round-trip
decimals reproduces it: 0 divergences with 53-bit doubles, ~1 in 25 000
with 80-bit intermediates; declining to std::strtod fixes all of them.

Guard parse_float_fast() on FLT_EVAL_METHOD so it declines whenever the
platform evaluates doubles in extended precision, letting the caller use
the correctly-rounded std::from_chars / std::strtod path instead. On
mainstream x86-64/ARM64 (FLT_EVAL_METHOD == 0) the fast path is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC 7 sets __cplusplus to the C++17 value under -std=gnu++1z, so
JSON_HAS_CPP_17 is defined, but its libstdc++ ships no <charconv> header
(added in GCC 8; floating-point from_chars in GCC 11). The unconditional
"#if defined(JSON_HAS_CPP_17) #include <charconv>" therefore failed to
compile there: "fatal error: charconv: No such file or directory" in the
ci_test_compilers_gcc (7) job.

Wrap the include in __has_include(<charconv>), mirroring the library's
existing handling of <version> and <filesystem> in macro_scope.hpp. When
the header is absent, __cpp_lib_to_chars stays undefined and
parse_float_from_chars() takes its scalar fallback, so the from_chars use
site (already gated on __cpp_lib_to_chars) is never reached. GCC 8-10,
which have <charconv> but no floating-point from_chars, are unaffected:
they include the header but still take the fallback. GCC 11+ is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant