Skip to content

Receiver: ts-proto generated codec replaces the hand-rolled reader/writer - #20

Merged
lannbot merged 1 commit into
mainfrom
receiver-tsproto-codec
Sep 7, 2026
Merged

Receiver: ts-proto generated codec replaces the hand-rolled reader/writer#20
lannbot merged 1 commit into
mainfrom
receiver-tsproto-codec

Conversation

@lannbot

@lannbot lannbot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

What

Replaces receiver/src/proto.ts (hand-rolled protobuf Reader/Writer) and the per-op tag switch in frames.ts with ts-proto generated readers and writers over @bufbuild/protobuf/wire. Event payload encoding uses the generated writer too. No hand-written wire code remains in the receiver.

Why

docs/design.md said the JS codec was pbf; it never was — the hand-rolled codec was chosen silently in a dispatch and the record was not updated. Evaluating the options against the receiver's actual needs:

  • pbf (the documented choice): readSVarint is silently wrong above 2^53 (2^53+1 → 2^53), no bounds checks (a truncated string clamps silently), and its generated readers cannot represent proto3 optional presence — the schema uses that for ns/parent/anchor, and real streams contain InsertBefore.parent = 0.
  • protobuf.js 8: exact int64, bounds-checked, but generated readers are 4x slower than the hand-rolled decoder and the runtime is +20 KB gzipped.
  • ts-proto 2 + @bufbuild/protobuf/wire: exact sint64 (forceLong=number throws on unsafe values), bounds-checked, optionalundefined, +3–4 KB gzipped.

Measured over the bench producers' real captured streams (19, every op) into the same FrameSink (harness kept untracked under bench/decoders/; three runs, geomean stable to ±0.04):

arm time vs hand-rolled
ts-proto generated 0.67–0.71x
ts-proto hand-driven hybrid 0.73–0.78x
protobuf.js hybrid 0.64–0.68x
protobuf.js generated 4.1–4.3x
pbf (either) ~0.58x, disqualified above

The hand-rolled decoder loses on allocation (two Readers, a tuple per tag, a closure per frame), not on the wire format. Generated readers everywhere is both fastest-correct and the least hand-written code; the live receiver after this PR measures ~0.8x the old decoder's time on the same harness (with unknownFields=true for strict mode).

Codegen

buf from npm runs under Deno and spawns the ts-proto plugin through deno run — no Node, no protoc (buf.yaml, buf.gen.yaml, just proto-ts). receiver/src/gen/*.ts is committed; just check regenerates into a temp dir and diffs (just proto-drift). The unanchored gen/ .gitignore rule (for host/desktop/gen) is anchored so the generated code can be tracked.

Semantics preserved / changed

  • FrameSink and its exported types are unchanged.
  • Framing: partial-frame buffering across push, MAX_FRAME_BYTES, truncated length prefix = wait vs malformed = error (5-byte u32 bound, now tested at frame level).
  • Each frame is bounded by a post-decode position check on the shared reader: a nested length that overruns the frame, or a decode that stops early on a zero/end-group tag, is a malformed frame in both modes (regression tests added; the previous decoder threw on the first, tolerated a zero tag with wire type 0 by accident).
  • Strict mode walks _unknownFields with the same unknown field N in <Message> errors.
  • SetProperty.int outside Number.isSafeInteger now throws instead of silently losing bits.
  • Event payload bytes omit proto3 zero defaults the hand writer used to emit; decoded values identical.

Gates

just check (incl. drift gate), deno task test (128), deno fmt --check, cargo test -p stream-dom-guest, just site, just e2e, just bench-wire (byte-identical wire shapes — the encoder side is Rust). Equivalence gate: the live decoder's sink-call trace is identical to a frozen copy of the old decoder over all 19 corpus streams. Independent review: one blocking finding (per-frame bound lost with the shared reader), fixed with tests.

deno lint receiver has one pre-existing failure in policy-desktop.ts:476 (no-control-regex), untouched here.

Automerge armed (merge commit).

…iter

The JS receiver decoded frames with a hand-written protobuf Reader and a
per-op tag switch (receiver/src/proto.ts + frames.ts), and encoded event
payloads with a hand-written Writer. docs/design.md said the codec was
pbf; it never was — the hand-rolled codec was chosen silently in a
dispatch and the design record was not updated.

Measured over the bench producers' real captured streams (19 of them,
every op; harness kept untracked under bench/decoders): ts-proto
generated readers over @bufbuild/protobuf/wire decode in ~0.7x the time
of the hand-rolled decoder, and a hand-driven hybrid on the same runtime
is no faster. protobuf.js generated is 4x slower. pbf is fastest but
silently wrong on sint64 above 2^53, has no bounds checks, and cannot
represent proto3 `optional` presence (ns/parent/anchor).

So: generated readers and writers everywhere, no hand-written wire code.
Codegen is `buf` from npm under Deno spawning the ts-proto plugin via
`deno run` — no Node, no protoc (`just proto-ts`); generated code is
committed and drift-gated in `just check`. FrameSink and its types are
unchanged. Framing keeps partial-frame buffering, MAX_FRAME_BYTES, and
the wait-vs-malformed distinction for a truncated length prefix; each
frame is bounded by a post-decode position check, so a nested length
that overruns the frame (or a decode that stops early on a zero tag) is
a malformed frame in both modes. Strict mode walks the generated
messages' `_unknownFields` with the same message-name errors. sint64
outside Number.isSafeInteger now throws instead of losing bits.

The unanchored `gen/` .gitignore rule (for host/desktop/gen) is anchored
so receiver/src/gen can be tracked.
@lannbot
lannbot enabled auto-merge September 7, 2026 23:07
@lannbot
lannbot merged commit c2b0838 into main Sep 7, 2026
1 check passed
@lannbot
lannbot deleted the receiver-tsproto-codec branch September 7, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants