Skip to content

Pydantic serialisation of numpy arrays to base64 - #425

Open
rwb27 wants to merge 8 commits into
mainfrom
base64-ndarray
Open

rwb27 wants to merge 8 commits into
mainfrom
base64-ndarray

Conversation

@rwb27

@rwb27 rwb27 commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

This PR introduces a Pydantic model for numpy arrays, and a corresponding class that may be used as an annotated type to serialise/validate arrays via that model, while keeping the type annotation as np.ndarray.

There's a reasonable level of testing of both classes, and all of the supported datatypes are round-tripped via both simple Python objects and via JSON strings.

Todo before merging

  • Add tests for validating from JSON with invalid values (currently, we only attempt to validate JSON we've generated, so it should always be valid).
  • Add validation logic so that to_array doesn't fail. E.g. we don't support float with length 3.
  • Implement byte_order=None for single-byte values.
  • Make sure docstrings are picked up by Pydantic and/or Sphinx.

Limitations

This data structure is intended to support numeric arrays, because they make sense in any language. It's not intended to support arrays of arbitrary Python objects, or record arrays (essentially arrays of C structs) because these are typically less portable. It's possible we will want to extend what's supported, for example adding support for timedelta or datetime types, but I am not convinced we will want to support object or string arrays in the medium term.

To express this limitation in numpy terminology, we support arrays where array.dtype.kind is b, i, u, f, or c. Supporting m and M (datetime and timedelta) ought to be possible but isn't done in this PR for simplicity. Supporting object, struct, or string types is probably a step too far.

Data structure

The JSON object we create is intended to store the same information as a .npy file, which enables a round-trip through a file without losing any data. It doesn't use exactly the same structure, because .npy is very terse and prettynumpy-specific, but it is intended to save the same information in an easier-to-interpret way.

The data saved in the model introduced here is equivalent to what is saved in a .npy or .npz file for an array where each element is one of the five supported types. Instead of a three-part string, consisting of byte order (<, '>' or '|'), datatype (one character) and size (1-2 digits), I've used three separate properties for ease of parsing/reading the JSON. The only limitation is that this structure can't handle record arrays, but I don't think that's a problem.

Numpy saves:

  • The value of array.dtype.descr. For numeric arrays (i.e. not record arrays) the relevant part of this is a short string using numpy-specific character codes to describe the data type.
  • The array.shape describing its size in each dimension (in elements, not bytes).
  • Whether the array is C or Fortran-contiguous.
  • The array data (in binary form).

Our datatype currently saves the following:

  • element_type: The element data type (float, int, unsigned int, bool, complex).
  • element_length: The element length in bytes.
  • byte_order: The byte order (little or big endian) or None for one-byte types.
  • shape: The array shape.
  • array_order: The element order (C or Fortran - though currently it's hard-coded as C).
  • data: The data, as a base64-encoded string.

I think the current structure is probably good. My one point of doubt is whether to have three properties for the element type (byte_order, element_type and element_size) or whether to combine these into a nested object. A nested object could more neatly describe constraints like, for example, byte_order being irrelevant for bool types, but I'm not sure that's worth the extra complexity.

Full JSONSchema

The JSONSchema doesn't render beautifully here. You can paste it in to https://json.ophir.dev/ but honestly I think the class definition of Base64NDArrayModel is probably as easy to read as anything else. You can skip the functions - just the attributes at the top are the ones that matter for the schema. The summary in the second bulleted list above is accurate as of 17/11 in the evening.

Click here for the full schema
{
  "description": "Numeric arrays that can be serialised to JSON.",
  "properties": {
    "element_type": {
      "description": "The datatype of each array element.",
      "enum": [
        "bool",
        "uint",
        "int",
        "float",
        "complex"
      ],
      "title": "Element Type",
      "type": "string"
    },
    "element_length": {
      "description": "The length of each array element in bytes.\n\nValid lengths are platform-dependent, but usually include\n1, 2, 4, 8 for integers and 2, 4, 8 and either 12 or 16 for\nfloating point numbers. Boolean arrays always use one byte\nper element.",
      "title": "Element Length",
      "type": "integer"
    },
    "byte_order": {
      "default": null,
      "description": "Whether the data is in little-endian or big-endian format.\n\nNote that order is irrelevant for data types that are only\none byte long, i.e. ``element_length==1``. In that case, `None`\nis the only allowed value, though implementations may decide\nto ignore this field rather than raise an error.\n\nFor data types longer than one byte, ``byte_order`` must always\nbe specified and must be either `\"little\"` or `\"big\"`.",
      "enum": [
        "little",
        "big",
        null
      ],
      "title": "Byte Order"
    },
    "shape": {
      "description": "The shape of the array, i.e. the length in each dimension.",
      "items": {
        "type": "integer"
      },
      "title": "Shape",
      "type": "array"
    },
    "array_order": {
      "const": "c-contiguous",
      "description": "Whether the array elements are in C or fortran order.\n\nCurrently only c order is supported.",
      "title": "Array Order",
      "type": "string"
    },
    "data": {
      "description": "The data for the array, Base64 encoded.\n\nWhen accessing a valid `Base64NDArrayModel` in Python, or when creating\none using `model_construct`, this field holds the array's data as a `bytes`\nobject. When constructing the model with `model_validate` or the constructor,\nthe value of ``data`` should be a Base64 encoded `bytes` object.\n\nThe value in a JSON string will always be Base64 encoded.",
      "format": "base64",
      "title": "Data",
      "type": "string"
    }
  },
  "required": [
    "element_type",
    "element_length",
    "shape",
    "array_order",
    "data"
  ],
  "title": "Numeric Array",
  "type": "object"
}

Why not just use numpy.dtype.descr?

numpy already exposes its internal description of datatype as dtype.descr. This is a very flexible description: it works recursively for record arrays and can describe all sorts of things we don't yet support (and likely never will).

The structure I've described above can cope with all numeric datatypes, and can be used to generate the expected value of numpy.dtype.descr as a double-check that we've understood the datatype. As an example:

  • [('', '|u1')] becomes {"element_type": "uint", "element_length": 1, "byte_order": null}
  • [('', '>f8')] becomes {"element_type": "float", "element_length": 8, "byte_order": "big"}
  • [('key', '|u1'), ('val', '<f8')] is a record array so it's unsupported

Arrays where the elements are numbers (or booleans) only ever have one field (i.e. the outer list is of length 1) and the title of the field is an empty string - so the descr should always look like [('', dtypestring)]. Said string consists of an endianness character (<, | or >), a "kind" field, and a length in bytes. The structure in Base64NDArrayModel describes that nicely, I think.

Performance comparison

Serialising a large array of floating-point numbers is both much more efficient in terms of data, and much faster - by a factor of 10 - compared to the previous list-of-lists method. The table below compares the two methods for two arrays with shape (1000, 1000, 3) and elements that are either float or int. For floating-point in particular, the new method is way faster. Note that the "Python" time is model_dump() and the JSON time is model_dump_json(). Clearly numpy is doing something clever when we call array.list() and deferring the work until later - I think that column doesn't mean much. We will more or less always be serialising using model_dump_json()

Method Array JSON Size/bytes Time (JSON)/ms Time (Python)/ms
list-of-lists randint 63138506 1172.44 0.0305698
new randint 32000124 272.241 86.91
list-of-lists float64 59813583 1938.03 0.0148129
new float64 32000126 167.634 91.7623

Note that the JSON size is actually only a factor of two bigger for the text representation - I'm a bit surprised by this, it's less awful than I anticipated. I guess it's because it's variable-length, so it doesn't always need to include information that's not required. That underlines the fact that the base64 representation is the only one guaranteed to preserve the numbers bit-for-bit.

Click for the script
from timeit import timeit
from typing import Annotated

import numpy as np
from pydantic import RootModel
from tabulate import tabulate

from labthings_fastapi.types.numpy import NDArrayAnnotation, NDArray


class LolRootModel(RootModel):
    root: NDArray


class NewRootModel(RootModel):
    root: Annotated[np.ndarray, NDArrayAnnotation]


models = {
    "list-of-lists": LolRootModel,
    "new": NewRootModel,
}


if __name__ == "__main__":
    arrays = {
        "randint": np.random.randint(
            np.iinfo(int).min,
            np.iinfo(int).max,
            size=(1000, 1000, 3),
        ),
        "float64": np.random.uniform(size=(1000, 1000, 3)),
    }

    N = 10

    cols = ["Method", "Array", "Size", "Time (JSON)/ms", "Time (Python)/ms"]
    table = []

    for aname, array in arrays.items():
        for mname, model in models.items():
            m = model(array)
            json_t = timeit("m.model_dump_json()", number=N, globals={"m": m}) / N
            json_l = len(m.model_dump_json())
            python_t = timeit("m.model_dump()", number=N, globals={"m": m}) / N
            #print(f"{mname} dumped {aname} to {json_l} bytes of JSON in {json_t}s")
            table.append([mname, aname, json_l, json_t*1000, python_t*1000])
            #print(f"{mname} dumped {aname} to Python in {python_t}s")

    print(tabulate(table, headers=cols, tablefmt="github"))

compare_array_serialisation.py

Are we tying our hands?

  • What if we decide we want to support string arrays?
    • We can add another datatype, and the current structure is fine.
  • What if we decide we want to support record arrays?
    • We would probably need to allow element_type to take a more complex type, and element_length then probably describes the total length of the struct. I'm pretty sure it could fit into the proposed structure without too much pain. I'm also pretty sure we don't want to!

@barecheck

barecheck Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Barecheck - Code coverage report

Total: 97.5%

Your code coverage diff: 0.08% ▴

✅ All code changes are covered

@rwb27
rwb27 marked this pull request as ready for review November 17, 2026 20:42
This commit introduces a Pydantic model for numpy arrays, and a
corresponding class that may be used as an annotated type to
serialise/validate arrays via that model, while keeping the
type annotation as `np.ndarray`.

There's a reasonable level of testing of both classes, and
all of the supported datatypes are round-tripped via both
simple Python objects and via JSON strings.
I now test against more arrays, some of which have foreign byte order.

I've also removed the suggestion that we cope with fortran-ordered arrays, because the reconstruction currently doesn't.
This tests a sanity check that should never fail in practice.
Previously, some errors would only be caught when we converted from a
model to an array.
These errors ought now to get caught earlier.

I've added a few test cases for invalid values. I'm pretty confident this tests all the cases. We could come up with more, but I'm not sure how much it gains.
This results in a simpler JSONSchema definition. The allowed values are identical.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant