Repository navigation
Conversation
rwb27
force-pushed
the
base64-ndarray
branch
from
November 12, 2026 10:01
12e3b49 to
3cf7d96
Compare
Barecheck - Code coverage reportTotal: 97.5%Your code coverage diff: 0.08% ▴ ✅ All code changes are covered |
rwb27
marked this pull request as ready for review
November 17, 2026 20:42
This commit introduces a Pydantic model for numpy arrays, and a corresponding class that may be used as an annotated type to serialise/validate arrays via that model, while keeping the type annotation as `np.ndarray`. There's a reasonable level of testing of both classes, and all of the supported datatypes are round-tripped via both simple Python objects and via JSON strings.
I now test against more arrays, some of which have foreign byte order. I've also removed the suggestion that we cope with fortran-ordered arrays, because the reconstruction currently doesn't.
This tests a sanity check that should never fail in practice.
Previously, some errors would only be caught when we converted from a model to an array. These errors ought now to get caught earlier. I've added a few test cases for invalid values. I'm pretty confident this tests all the cases. We could come up with more, but I'm not sure how much it gains.
This results in a simpler JSONSchema definition. The allowed values are identical.
rwb27
force-pushed
the
base64-ndarray
branch
from
November 18, 2026 14:41
181d7c2 to
e2fa623
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces a Pydantic model for numpy arrays, and a corresponding class that may be used as an annotated type to serialise/validate arrays via that model, while keeping the type annotation as
np.ndarray.There's a reasonable level of testing of both classes, and all of the supported datatypes are round-tripped via both simple Python objects and via JSON strings.
Todo before merging
to_arraydoesn't fail. E.g. we don't supportfloatwith length3.byte_order=Nonefor single-byte values.Limitations
This data structure is intended to support numeric arrays, because they make sense in any language. It's not intended to support arrays of arbitrary Python objects, or record arrays (essentially arrays of C
structs) because these are typically less portable. It's possible we will want to extend what's supported, for example adding support fortimedeltaordatetimetypes, but I am not convinced we will want to supportobjectorstringarrays in the medium term.To express this limitation in
numpyterminology, we support arrays wherearray.dtype.kindisb,i,u,f, orc. SupportingmandM(datetimeandtimedelta) ought to be possible but isn't done in this PR for simplicity. Supporting object, struct, or string types is probably a step too far.Data structure
The JSON object we create is intended to store the same information as a
.npyfile, which enables a round-trip through a file without losing any data. It doesn't use exactly the same structure, because.npyis very terse and prettynumpy-specific, but it is intended to save the same information in an easier-to-interpret way.The data saved in the model introduced here is equivalent to what is saved in a
.npyor.npzfile for an array where each element is one of the five supported types. Instead of a three-part string, consisting of byte order (<, '>' or '|'), datatype (one character) and size (1-2 digits), I've used three separate properties for ease of parsing/reading the JSON. The only limitation is that this structure can't handle record arrays, but I don't think that's a problem.Numpy saves:
array.dtype.descr. For numeric arrays (i.e. not record arrays) the relevant part of this is a short string usingnumpy-specific character codes to describe the data type.array.shapedescribing its size in each dimension (in elements, not bytes).Our datatype currently saves the following:
element_type: The element data type (float,int,unsigned int,bool,complex).element_length: The element length in bytes.byte_order: The byte order (littleorbigendian) orNonefor one-byte types.shape: The array shape.array_order: The element order (C or Fortran - though currently it's hard-coded as C).data: The data, as abase64-encoded string.I think the current structure is probably good. My one point of doubt is whether to have three properties for the element type (
byte_order,element_typeandelement_size) or whether to combine these into a nested object. A nested object could more neatly describe constraints like, for example,byte_orderbeing irrelevant forbooltypes, but I'm not sure that's worth the extra complexity.Full JSONSchema
The JSONSchema doesn't render beautifully here. You can paste it in to https://json.ophir.dev/ but honestly I think the class definition of
Base64NDArrayModelis probably as easy to read as anything else. You can skip the functions - just the attributes at the top are the ones that matter for the schema. The summary in the second bulleted list above is accurate as of 17/11 in the evening.Click here for the full schema
{ "description": "Numeric arrays that can be serialised to JSON.", "properties": { "element_type": { "description": "The datatype of each array element.", "enum": [ "bool", "uint", "int", "float", "complex" ], "title": "Element Type", "type": "string" }, "element_length": { "description": "The length of each array element in bytes.\n\nValid lengths are platform-dependent, but usually include\n1, 2, 4, 8 for integers and 2, 4, 8 and either 12 or 16 for\nfloating point numbers. Boolean arrays always use one byte\nper element.", "title": "Element Length", "type": "integer" }, "byte_order": { "default": null, "description": "Whether the data is in little-endian or big-endian format.\n\nNote that order is irrelevant for data types that are only\none byte long, i.e. ``element_length==1``. In that case, `None`\nis the only allowed value, though implementations may decide\nto ignore this field rather than raise an error.\n\nFor data types longer than one byte, ``byte_order`` must always\nbe specified and must be either `\"little\"` or `\"big\"`.", "enum": [ "little", "big", null ], "title": "Byte Order" }, "shape": { "description": "The shape of the array, i.e. the length in each dimension.", "items": { "type": "integer" }, "title": "Shape", "type": "array" }, "array_order": { "const": "c-contiguous", "description": "Whether the array elements are in C or fortran order.\n\nCurrently only c order is supported.", "title": "Array Order", "type": "string" }, "data": { "description": "The data for the array, Base64 encoded.\n\nWhen accessing a valid `Base64NDArrayModel` in Python, or when creating\none using `model_construct`, this field holds the array's data as a `bytes`\nobject. When constructing the model with `model_validate` or the constructor,\nthe value of ``data`` should be a Base64 encoded `bytes` object.\n\nThe value in a JSON string will always be Base64 encoded.", "format": "base64", "title": "Data", "type": "string" } }, "required": [ "element_type", "element_length", "shape", "array_order", "data" ], "title": "Numeric Array", "type": "object" }Why not just use
numpy.dtype.descr?numpyalready exposes its internal description of datatype asdtype.descr. This is a very flexible description: it works recursively for record arrays and can describe all sorts of things we don't yet support (and likely never will).The structure I've described above can cope with all numeric datatypes, and can be used to generate the expected value of
numpy.dtype.descras a double-check that we've understood the datatype. As an example:[('', '|u1')]becomes{"element_type": "uint", "element_length": 1, "byte_order": null}[('', '>f8')]becomes{"element_type": "float", "element_length": 8, "byte_order": "big"}[('key', '|u1'), ('val', '<f8')]is a record array so it's unsupportedArrays where the elements are numbers (or booleans) only ever have one field (i.e. the outer list is of length 1) and the title of the field is an empty string - so the
descrshould always look like[('', dtypestring)]. Said string consists of an endianness character (<,|or>), a "kind" field, and a length in bytes. The structure inBase64NDArrayModeldescribes that nicely, I think.Performance comparison
Serialising a large array of floating-point numbers is both much more efficient in terms of data, and much faster - by a factor of 10 - compared to the previous list-of-lists method. The table below compares the two methods for two arrays with shape
(1000, 1000, 3)and elements that are eitherfloatorint. For floating-point in particular, the new method is way faster. Note that the "Python" time ismodel_dump()and the JSON time ismodel_dump_json(). Clearlynumpyis doing something clever when we callarray.list()and deferring the work until later - I think that column doesn't mean much. We will more or less always be serialising usingmodel_dump_json()Note that the JSON size is actually only a factor of two bigger for the text representation - I'm a bit surprised by this, it's less awful than I anticipated. I guess it's because it's variable-length, so it doesn't always need to include information that's not required. That underlines the fact that the
base64representation is the only one guaranteed to preserve the numbers bit-for-bit.Click for the script
compare_array_serialisation.py
Are we tying our hands?
element_typeto take a more complex type, andelement_lengththen probably describes the total length of the struct. I'm pretty sure it could fit into the proposed structure without too much pain. I'm also pretty sure we don't want to!