[rok]

The Parquet form of a validated measurement result (#2, §5.2 ingest object): the columns a result is stored under, which the local store's row begins with. It defines nothing about the message itself; where it and the schema differ, the schema wins.

• docs/design/measurement-result-parquet.md: the mapping from the message JSON Schema to Arrow and Parquet types, the resulting schema for message schema 0.1.0, the file metadata, the columns a store adds beside them, and how to read the file, with a DuckDB query over the six example results. One row is one result; closed objects are structs, open objects are canonical JSON columns, and observations are one typed list column.
• src/benchx/parquet.py: the mapping already lives here on main. This PR adds message_table() and MESSAGE_SCHEMA (the message columns alone), and records benchx.message_schema_sha256 in the file metadata, so files written under different drafts of 0.1.0 can be told apart. This includes store.parquet.
• tools/json_to_parquet.py: a small script built on benchx. It reads and checks each document the way bx validate does, then writes message_table() to one Parquet file. Nothing is written if any document is invalid.
• tests/test_parquet.py: the mapping over the examples (row order, nullability, canonical JSON, observations, metadata), the schema nodes the mapping must reject, and the tool.
• Indexes: the document is listed in README.md, docs/design/README.md, and the site navigation. The "Add Parquet storage schema and a PyArrow example #30" references in the code and docs now name the document instead.

pip install benchx
python tools/json_to_parquet.py results.parquet \
    src/benchx/schemas/measurement-result/0.1.0/examples/*.json

[rok]

I'm not sure this is ready to merge, but would invite reviews!

[MarcoGorelli]

I think this has drifted a bit since main, do you want to update and we can review?

[MarcoGorelli]

here it says

oneOf of closed objects

However, is that what the code does?

benchx/src/benchx/parquet.py

Lines 37 to 49 in 4778ea5

	if "oneOf" in node: # union of closed objects, or number-or-object observation
	branches = [defs[b["$ref"].rsplit("/", 1)[-1]] if "$ref" in b else b for b in node["oneOf"]]
	objects = [b for b in branches if b.get("type") == "object"]
	if not objects:
	raise ValueError(f"unsupported oneOf without an object alternative: {node}")
	required = set.intersection(*(set(b.get("required", [])) for b in objects))
	types = {} # in first-appearance order
	for b in objects:
	for k, v in b["properties"].items():
	t = nested(v)
	if types.setdefault(k, t) != t:
	raise ValueError(f"oneOf alternatives disagree on the type of {k!r}: {types[k]} and {t}")
	return pa.struct([pa.field(k, t, k not in required) for k, t in types.items()])

does it check that they're actually closed (i.e. additionalProperties: false)?

[MarcoGorelli]

this looks different to what's enforced?

it looks like we reject the case where no objects are present

benchx/src/benchx/parquet.py

Lines 40 to 41 in 4778ea5

	if not objects:
	raise ValueError(f"unsupported oneOf without an object alternative: {node}")

but then wouldn't oneOf: [{"type": "string"}, <closed object>] still get through?