Skip to content

repr: pin the PackedNumeric layout to its durable byte offsets - #38402

Open
antiguru wants to merge 1 commit into
MaterializeInc:mainfrom
antiguru:moritz/pin-packed-numeric-layout
Open

repr: pin the PackedNumeric layout to its durable byte offsets#38402
antiguru wants to merge 1 commit into
MaterializeInc:mainfrom
antiguru:moritz/pin-packed-numeric-layout

Conversation

@antiguru

Copy link
Copy Markdown
Member

PackedNumeric is a FixedSizeCodec over forty bytes, so its layout is durable. It stores the coefficient units in the twenty-six bytes at offset eight, the flags in the byte at offset thirty-four, and leaves the last five unused. The encoder wrote thirteen units because that is what fits, but nothing in the code said so, and nothing tied it to NUMERIC_DATUM_WIDTH, which is where the thirteen comes from.

The coupling is a hazard in one direction. A wider Numeric would write its coefficient over the flags byte, which still fits inside SIZE, so the encoder would neither fail to compile nor panic. It would corrupt persisted values, and because the flags carry the sign, a negative value would come back positive.

This names the offsets, derives the unit count from them, and asserts at compile time that the datum width matches. Changing the datum width then fails the build and says what it needs, a new packed encoding and a migration, rather than silently changing a durable format.

The encoded bytes are unchanged, which the existing packed_numeric_stability test covers.

Tests

Adds packed_numeric_max_precision_keeps_flags, which round-trips the largest positive and negative values through the codec and checks the sign survives.

proptest_packed_numeric_roundtrip and packed_numeric_stability already fail on an overrun, so the new test is not the only guard; it names the specific failure mode and does not depend on a random input happening to use the last unit. All three were confirmed to fail when the encoder is modified to overrun the flags byte, and to pass once it is not.

Note for reviewers

bin/lint fails on this branch in check-pipeline-topics.sh and check-python-files.sh. Both concern files this change does not touch, reporting a "LiteralString" is unknown import symbol pyright error in two mzcompose.py files and a traceback inside lint_pipeline_topics.py. They look like local Python tooling problems rather than anything from this change; CI will confirm.

🤖 Generated with Claude Code

`PackedNumeric` is a `FixedSizeCodec` over forty bytes, so its layout is
durable. It stores the coefficient units in the twenty-six bytes at offset
eight, the flags in the byte at offset thirty-four, and leaves the last five
unused. The encoder wrote thirteen units because that is what fits, but
nothing said so, and nothing tied it to `NUMERIC_DATUM_WIDTH`.

That coupling is a hazard in one direction. A wider `Numeric` would write its
coefficient over the flags byte, which still fits inside `SIZE`, so the
encoder would neither fail to compile nor panic; it would corrupt persisted
values, and since the flags carry the sign, a negative value would come back
positive. Name the offsets, derive the unit count from them, and assert at
compile time that the datum width matches, so changing the width is a build
failure that says what it needs instead of a silent format change.

The encoded bytes are unchanged, which `packed_numeric_stability` covers.

Add `packed_numeric_max_precision_keeps_flags`, which round-trips the largest
positive and negative values through the codec. `proptest_packed_numeric_
roundtrip` and `packed_numeric_stability` already fail on an overrun, but only
for inputs that happen to use the last unit; this names the case and checks
the sign explicitly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@antiguru
antiguru requested a review from a team as a code owner August 21, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant