fix: Improve jreader throughput with a single-pass tokenizer

[kinyoklion]

SDK-2884

The parse-side counterpart of #51. jreader's default tokenizer scanned input character by character: readString walked every character through bytes.Reader.ReadRune (a method call plus UTF-8 decode per character, even for plain ASCII — the single hottest function when parsing a real flag payload), escaped strings were built by appending one rune at a time, and scalar reads shuttled 48-byte token structs through two call layers. Parse throughput was ~110 MB/s against the rewritten writer's ~400 MB/s.

This PR rewrites the tokenizer as a single-pass scanner over the input byte slice:

• Strings scan in place with a 256-entry byte-class table. Unescaped strings keep the existing zero-copy behavior (the returned bytes are a subslice of the input, sliced identically to before). Strings containing escapes or invalid UTF-8 decode in one forward pass into a single buffer sized up front, bulk-copying clean runs; \uXXXX escapes and surrogate pairs decode by direct indexing. bytes.Reader and per-rune appends are gone.
• Scalar reads (Bool, Number, String, PropertyName, Any) dispatch on the first non-whitespace byte and parse directly, entering the token path only for pushed-back tokens or type mismatches, so all error construction still goes through the same primitives. next() parses into a reused field on the tokenReader instead of returning token structs; putBack is a flag flip.
• A bytes.IndexByte-based variant (find the closing quote, then validate the span) was benchmarked and rejected: 12-25% slower than the table loop across all string shapes, since validation must visit every byte anyway.

Observable behavior is unchanged: exported API, parsed values, error types, error message strings, error offsets, Offset() positions, and RawValue() byte ranges are all identical to the previous implementation.

Correctness validation

Beyond the existing suite (green with -race; the cross-permutation commontest matrices cover both dispatch paths), a differential harness ran the old and new tokenizers in lock-step over 1,013,258 paired cases with zero mismatches: exhaustive one- and two-byte string contents, all 65,536 \uXXXX values, surrogate-boundary grids plus random and handpicked pairs (truncated pairs, \uD800𐀀), invalid-UTF-8 sequences at eight positions, int64 extremes, -0.0, denormals, overflow exponents, 400-digit literals, malformed numbers, every prefix-truncation of a mixed document, and 8,000 random documents with trailing junk and byte mutations. Comparisons covered decoded values (bit-exact floats), string/raw bytes and nilness, error type/fields/message, Offset() after every operation, and failed-state stickiness. The harness's sensitivity was proven by seeded mutations (an off-by-one offset and a disabled surrogate-pairing branch were both detected).

One behavior worth naming because it is preserved, not changed: since the RFC 8259 work, invalid UTF-8 in strings is replaced with U+FFFD even when no escape sequence is present (each invalid byte forces the string off the zero-copy path). The new scanner matches that exactly, including the case where an unpairable surrogate emits U+FFFD and the following characters are re-read normally.

Benchmarks

go1.24, linux/amd64, interleaved old/new binaries (3 rounds x count 2, benchstat n=6, all listed deltas p=0.002). Micro geomean -52%:

ReadString                    72.5n -> 33.7n   (-53.5%)
ReadArrayOfStrings            25.6u -> 11.7u   (-54.3%, allocs 208 -> 158)
ReadStringKinds/longASCII     throughput 219 -> 1381 MB/s  (-84.2%)
ReadStringKinds/escaped       -65.6%  (allocs 101 -> 51)
ReadBooleanNoAlloc            -36.2%
ReadNumberIntNoAlloc          -35.0%
ReadObjectNoAlloc             -43.5%

On a real 3,228-flag / 3.2 MB LaunchDarkly payload parsed through ldmodel (go-server-sdk-evaluation): direct jreader parse 34.2 ms -> 18.5 ms (-45.9%); through the encoding/json-dispatch path, 62.1 ms -> 48.0 ms (-22.7%). For reference, the v3 easyjson lexer's advantage over the old default reader on the same payload was ~19% — relevant because v4 removed the easyjson adapters, and this closes that gap with margin.

All NoAlloc benchmarks remain at 0 allocs/op (CI gate).

Notes for reviewers

• The first-byte dispatch in any() and the pushed-back-token path (tokenToAnyValue) are parallel switches that must stay in sync; both are covered by the commontest permutation matrices and the differential corpus.
• decodedStringCapacity pre-scans an escaped string's remainder once to size the decode buffer — a deliberate two-scans/one-allocation trade.
• Number parsing still allocates one string per float (strconv.ParseFloat); avoiding it would require unsafe, which this repo does not use.

---

Note

Overview
Rewrites jreader’s default tokenReader to scan the input []byte in one pass instead of walking strings through bytes.Reader and shuttling token structs through next().

String parsing now uses a plainStringChars byte-class table to bulk-scan unescaped ASCII (still zero-copy subslices), with a separate decodeString path for escapes, \u / surrogate pairs, and invalid UTF-8 (replacing bytes.Reader and per-rune appends). Scalar reads (Bool, Number, String, Any, etc.) peek the first non-whitespace byte and parse inline when possible; next() writes into a reused r.tok and putBack() is a flag. Whitespace and keyword scanning use tighter index loops.

Adds BenchmarkReadStringKinds to exercise short/medium/long ASCII, multi-byte, and escaped string shapes via StringAsBytes.

^{Reviewed by Cursor Bugbot for commit 856c9de. Bugbot is set up for automated code reviews on this repo. Configure here.}