Skip to content

fix: implement RowIdMeta.asdict() and from_dict() for stable row ID serialization#6405

Open
pengw0048 wants to merge 1 commit intolance-format:mainfrom
pengw0048:fix/fragment-metadata-json-stable-rowids
Open

fix: implement RowIdMeta.asdict() and from_dict() for stable row ID serialization#6405
pengw0048 wants to merge 1 commit intolance-format:mainfrom
pengw0048:fix/fragment-metadata-json-stable-rowids

Conversation

@pengw0048
Copy link
Copy Markdown
Contributor

@pengw0048 pengw0048 commented Apr 3, 2026

Fixes #6403

FragmentMetadata.to_json() raised NotImplementedError when row_id_meta was present (enable_stable_row_ids=True), because PyRowIdMeta.asdict() was stubbed out. This also broke FragmentMetadata.from_json() for the round-trip, since RowIdMeta(**dict) doesn't work on a PyO3 class.

Repro

import lance, pyarrow as pa

uri = "/tmp/repro.lance"
ds = lance.write_dataset(pa.table({"x": [1, 2, 3]}), uri, enable_stable_row_ids=True)
ds.get_fragments()[0].metadata.to_json()  # NotImplementedError: PyRowIdMeta.asdict is not yet supported.

Fix

  • Implement PyRowIdMeta.asdict() via pythonize (Rust struct → Python dict)
  • Add PyRowIdMeta.from_dict() via depythonize (Python dict → Rust struct)
  • Update FragmentMetadata.from_json() to use RowIdMeta.from_dict() instead of RowIdMeta(**dict)
  • Add JSON round-trip test to the existing test_fragment_metadata_pickle parametrized test

@github-actions github-actions bot added bug Something isn't working python labels Apr 3, 2026
@pengw0048 pengw0048 force-pushed the fix/fragment-metadata-json-stable-rowids branch 2 times, most recently from c3dd12b to a3ac7f8 Compare April 3, 2026 20:29
@pengw0048 pengw0048 changed the title fix: support JSON serialization of FragmentMetadata with stable row IDs fix: implement RowIdMeta.asdict() and from_dict() for stable row ID serialization Apr 3, 2026
@pengw0048 pengw0048 force-pushed the fix/fragment-metadata-json-stable-rowids branch from a3ac7f8 to 3aa2e45 Compare April 3, 2026 20:39
`FragmentMetadata.to_json()` raised `NotImplementedError` when
`row_id_meta` was present (i.e. `enable_stable_row_ids=True`), because
`PyRowIdMeta.asdict()` was not implemented.

Implement `asdict()` via `pythonize` and add `from_dict()` via
`depythonize`, so both serialization and deserialization go directly
between Rust and Python dicts without JSON string intermediaries.
@pengw0048 pengw0048 force-pushed the fix/fragment-metadata-json-stable-rowids branch from 3aa2e45 to a997d63 Compare April 3, 2026 20:40
@pengw0048 pengw0048 marked this pull request as ready for review April 3, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FragmentMetadata.to_json() fails with stable row IDs enabled

1 participant