perf: single-pass tablet serialization — +52% write throughput, −61% latency

[CritasWang]

Problem

CPU profiling of an insertTablet-heavy workload (100 devices × 20 sensors × 1000-row batches, 20 clients) showed the event loop 79.5% busy, with serializeTabletValues at 26% self time and GC at 24.5% — the write path was dominated by allocation churn, not I/O:

1. serializeTabletValues transposed rows→columns (values.map(row => row[col]) — a fresh array per column), built a boolean[] null bitmap per column even when the column had no nulls, serialized each column into its own intermediate buffer, and finally Buffer.concat-ed everything (a full payload re-copy).
2. INT64/TIMESTAMP writes allocated a BigInt per value for writeBigInt64BE.
3. TEXT/STRING columns re-encoded UTF-8 for every row, even constant TAG columns.
4. BufferPool was acquire-only (zero release() calls in src/) — 0% hit rate, pure overhead over Buffer.allocUnsafe.
5. serializeBitMaps allocated a 1-byte Buffer.from([flag]) per column.

Changes

• Single-pass, single-buffer tablet serialization (serializeTabletValuesFast): pre-compute the exact payload size (fixed widths from schema; one byteLength scan for variable-width columns), allocate one buffer, write values column-by-column reading values[row][col] directly, and pack null bitmaps inline — flag byte stays 0 and no bitmap bytes are emitted when a column has no nulls. No transpose, no intermediate buffers, no Buffer.concat.
• BigInt-free int64 writes: sign-correct hi/lo 32-bit pair (hi = Math.floor(v / 2^32), lo = v >>> 0); BigInt inputs and non-safe integers still use writeBigInt64BE.
• String encode cache: single-entry memo reuses the encoded buffer for consecutive identical strings (constant TAG columns).
• BufferPool deprecated: removed from the write path (kept exported for API compatibility, marked @deprecated).

The legacy path (enableFastSerialization: false) is unchanged.

Results (IoTDB 2.0.10 standalone, 16-core box, DOUBLE workload, 10^9 points/run, interleaved A/B, 2 rounds)

Run	Throughput	Avg latency	P99
baseline #1	8.70M pts/s	26.83 ms	78.11 ms
optimized #1	13.11M pts/s	10.60 ms	23.61 ms
baseline #2	8.53M pts/s	27.07 ms	78.90 ms
optimized #2	13.16M pts/s	10.53 ms	23.18 ms

+52% throughput, −61% average latency, −70% P99. Server-side count(*) verified 100,000,000 rows for every run. Serializer self time dropped from 26% to 14.2% in --cpu-prof; the profiled process now shows 16% idle where it was previously saturated.

Wire compatibility

• All pre-existing serialization unit tests (which assert exact wire bytes) pass unchanged.
• New golden tests assert the fast path equals the legacy path byte-for-byte for mixed-type tablets with nulls, all-null columns, bitmap-width edge cases, BLOB/DATE, and negative/MIN_SAFE_INTEGER/BigInt int64 values.
• New unit tests cover hi/lo int64 correctness (two's complement negatives, i64 min/max BigInt) and string-cache correctness.
• e2e AllDataTypes + TableModelDataTypes suites pass against a live IoTDB 2.0.10.

Caveats

• BufferPool is deprecated but still exported; removal would be semver-major.
• The remaining GC in profiles is dominated by the benchmark harness's own tablet construction, not the client.

[CritasWang]

Rebased onto develop after #15 (DATE yyyyMMdd fix) merged. Conflict resolution notes: kept #15's parseDateToInt semantics everywhere and removed the BufferPool usage from serializeDateColumn; also updated the fast tablet path's DATE branch to use parseDateToInt (it predated #15), so the single-buffer path encodes yyyyMMdd identically — covered by the existing golden fast-vs-legacy byte-for-byte tests plus the exact-wire-bytes DATE tests from #15. 152/152 unit tests pass.

[HTHou]

The core serializer change looks sound: I verified that all 152 unit tests and the build pass, both GitHub E2E checks are green, and 2,000 randomized valid tablets matched the legacy path byte-for-byte. I left two inline comments, including one blocking scope issue.

One additional documentation issue is outside the changed diff: Config.enableFastSerialization still says that the option enables serialization “with buffer pooling” and claims a 2–3x improvement. Since this PR removes the pool from the write path and deprecates it, please update that public JSDoc to describe the new implementation and use performance wording consistent with the evidence in this PR.

[CritasWang]

Also addressed the Config.enableFastSerialization JSDoc (outside the original diff, included in c5f5522): it no longer mentions buffer pooling or the 2-3x claim — it now describes the actual implementation (single-pass single-buffer writes, inline null bitmaps, BigInt-free int64) and notes the output is byte-identical to the legacy path. All 153 unit tests pass; lint at baseline (61 errors, none new).

[HTHou]

Re-reviewed at c5f5522. The previously raised issues have been addressed: the unrelated SpecKit/Claude scaffolding was removed, non-Buffer BLOB serialization no longer performs duplicate materialization and now has golden coverage, and the public fast-serialization documentation was updated.

I also re-ran the 153 unit tests and the build successfully; both current E2E checks are green. The remaining raw ArrayBuffer compatibility edge is outside the documented/recommended BLOB input and is non-blocking for this change.

LGTM.

[CritasWang]

Addressed the ArrayBuffer P2 in 5f4c1e0: blobByteLength() now handles ArrayBuffer/SharedArrayBuffer via .byteLength (they have no .length, which previously made dataSize NaN → RangeError), and the write pass bulk-copies through a zero-copy new Uint8Array(v) view via buffer.set(). Added a golden fast-vs-legacy test covering ArrayBuffer, SharedArrayBuffer, empty ArrayBuffer, and null. Reproduced the failure first (RangeError, exactly as you described), then verified the fix end-to-end. 154/154 unit tests pass; lint at baseline (61, none new).