Skip to content

fix: cap the BYTEA read-path preview at 10 KB, sniff MIME from magic bytes (#87) - #107

Merged
aesslinger merged 1 commit into
mainfrom
fix/87-bytea-preview-truncation
Sep 16, 2026
Merged

aesslinger merged 1 commit into
mainfrom
fix/87-bytea-preview-truncation

Conversation

@aesslinger

Copy link
Copy Markdown
Collaborator

Summary

  • The BYTEA read path base64-encoded the entire byte array into the BLOB:<size>:<mime>:<b64> wire format, with no cap and a hardcoded application/octet-stream MIME type. The builtin driver truncates the read/preview path to the first 10,240 bytes (MAX_BLOB_PREVIEW_SIZE) while still reporting the true total_size in the header, and sniffs MIME from the (possibly truncated) content's magic bytes.
  • Extracted a shared src/utils/blob.rs (encode_blob for the truncated read/preview path, encode_blob_full for the untruncated file-export path) matching the builtin's drivers/common/blob.rs exactly — including sniffing MIME from the truncated preview slice specifically, not the full data, to match the builtin's encode_blob byte-for-byte.
  • Both extract.rs's Type::BYTEA scalar-column arm and extract_simple_kind_from_bytes (the array-element / composite-field decode path) now use the shared encode_blob; handlers/blob.rs's fetch_blob_as_data_url uses encode_blob_full, replacing its own duplicate copy of that function.

Second finding, fixed in this PR: while auditing every BYTEA decode path in extract.rs, found extract_simple_kind_from_bytes had its own independently-drifted, untruncated copy of the BYTEA arm — reached via array elements (bytea[]) and composite fields rather than top-level scalar columns. The builtin has exactly one BYTEA decode path shared by all three (Kind::Simple routes array/composite elements through the same simple::extract_or_null the scalar path uses); this repo had drifted into two independent copies. Both are now fixed and share the same encode_blob.

Third, out-of-scope finding — filed as #106: extract.rs's binary_blob_wrapper! macro (used for internal planner-statistics types like pg_mcv_list) has its own separate untruncated encoding whose doc comment incorrectly claims it matches the builtin. Low severity (those types are essentially never queried directly), so left as a follow-up rather than bundled here.

Fixes #87.

Test plan

  • TDD: added unit tests to utils/blob_tests.rs (encode_blob/encode_blob_full, including the 20 KB-input/10 KB-preview case the issue specifically requested) and a new bytea_array_element_... test in extract_tests.rs for the array-element path. Confirmed both the scalar-path and array-element-path tests fail against the pre-fix code (20480 vs expected 10240) before the fix, pass after.
  • cargo test --lib — 320 passed
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo fmt --all -- --check — clean
  • Live end-to-end verification against a real PostgreSQL instance (12/12 checks):
    • A 20 KB BYTEA value's read-path response reports the true size (20480) but truncates the base64 payload to exactly the first 10,240 bytes, matching the real data's first 10,240 bytes
    • The underlying stored data in PostgreSQL is confirmed untouched (octet_length still 20480) — only the read-path response is capped, never the actual data
    • fetch_blob_as_data_url (the file-export path) still returns the complete, untruncated 20,480-byte payload
    • PNG magic bytes are correctly sniffed to image/png on the read path (previously always application/octet-stream)
    • A small (under-cap) BYTEA value still round-trips in full, unaffected

…bytes (#87)

The BYTEA read path base64-encoded the entire byte array into the
BLOB:<size>:<mime>:<b64> wire format, with no cap and a hardcoded
application/octet-stream MIME. The builtin driver truncates the
read/preview path to the first 10,240 bytes (MAX_BLOB_PREVIEW_SIZE) while
still reporting the true total_size in the header, and sniffs MIME from
the (possibly truncated) content's magic bytes. Large BYTEA columns
produced much larger responses in this plugin than in the builtin,
slowing the data grid and inflating transfers -- and the hardcoded MIME
meant the UI couldn't preview image/PDF blobs the builtin's sniffed MIME
allows.

Extracted a shared src/utils/blob.rs (encode_blob for the truncated
read/preview path, encode_blob_full for the untruncated file-export
path) matching the builtin's drivers/common/blob.rs exactly, including
sniffing MIME from the truncated preview slice specifically (not the
full data) to match the builtin's encode_blob byte-for-byte. Both
extract.rs's Type::BYTEA scalar-column arm and its
extract_simple_kind_from_bytes (the array-element / composite-field
decode path) now use the shared encode_blob; handlers/blob.rs's
fetch_blob_as_data_url uses encode_blob_full, replacing its own
duplicate copy of that function.

Found and fixed a second, previously undetected instance of the same
bug while auditing extract.rs for every BYTEA decode path:
extract_simple_kind_from_bytes had its own independently-drifted,
untruncated copy of the BYTEA arm, reached via array elements
(bytea[]) and composite fields rather than top-level scalar columns.
The builtin has exactly one BYTEA decode path shared by all three
(Kind::Simple routes array/composite elements through the same
simple::extract_or_null the scalar path uses); the plugin had drifted
into two independent copies. Both are now fixed.

TDD: added unit tests to utils/blob_tests.rs (encode_blob/encode_blob_full,
including the 20 KB-input/10 KB-preview case the issue specifically
requested) and a new bytea_array_element_... test in extract_tests.rs
for the array-element path. Confirmed both the scalar and array-element
tests fail against the pre-fix code (20480 vs expected 10240) before the
fix, pass after.

Verified end-to-end against a live PostgreSQL instance: a 20 KB BYTEA
value's read-path response reports the true size (20480) but truncates
the base64 payload to exactly the first 10,240 bytes; the underlying
stored data in PostgreSQL is confirmed untouched (still 20480 bytes) --
only the read-path response is capped; fetch_blob_as_data_url (export
path) still returns the complete, untruncated payload; and PNG magic
bytes are correctly sniffed to image/png on the read path (previously
always application/octet-stream).

Filed #106 for a related, out-of-scope finding: extract.rs's
binary_blob_wrapper! macro (used for internal planner-statistics types
like pg_mcv_list) has its own separate untruncated encoding that should
also move to the new shared encode_blob, but is low-severity (those
types are essentially never queried directly) and left for a follow-up.
@aesslinger aesslinger added the prerelease:rc Version suggestion targets a release candidate label Sep 16, 2026
@github-actions

Copy link
Copy Markdown

Version suggestion

Based on this PR's title (fix) and the prerelease:rc label:

Current 1.0.0-rc.3
Suggested next tag v1.0.0-rc.4

This is informational only — no tag or release is created automatically yet.

@aesslinger aesslinger self-assigned this Sep 16, 2026
@aesslinger
aesslinger merged commit 341473a into main Sep 16, 2026
14 checks passed
@aesslinger
aesslinger deleted the fix/87-bytea-preview-truncation branch September 16, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

prerelease:rc Version suggestion targets a release candidate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BYTEA read path encodes the full base64 instead of a truncated preview (builtin caps at 10 KB)

1 participant