Support for Code Generation for Pydantic v2

[almogtavor]

I would like to suggest the addition of support for code generation for Pydantic v2 types in the pkl-python library. In case others think that there's a reason to keep generating classes without Pydantic, I suggest letting the user to choose its target classes generation type (with Pydantic/ without). Pydantic v2 brings features that can enhance the usability and functionality of the pkl-python library.

Why Pydantic is Needed Even when using pkl:

1. Standardized Data Models:

   • Pydantic v2 models are widely used in modern Python applications, particularly in APIs and data processing pipelines. By supporting Pydantic v2, pkl-python can facilitate the integration and interoperability with other systems that rely on these standardized models.

2. Compatibility with Existing Ecosystem (especially with FastAPI):

   • Many tools and libraries in the Python ecosystem are updating to support Pydantic v2. FastAPI, a popular web framework, heavily utilizes Pydantic models. Adding support for Pydantic v2 in pkl-python would streamline the process of using pkl-python with FastAPI applications, making it easier to generate and manipulate models within the same ecosystem.
   • For example integration with FastAPI ORMs

3. Backward Compatibility:

   • Supporting Pydantic v2 types does not preclude the continued use of earlier versions or plain Python types. It simply expands the toolkit available to developers using pkl-python.

Anyway, thank you for considering this. I look forward to seeing pkl-python evolve with the support for Pydantic v2 types.

[jaanli]

@almogtavor - not sure what is missing from the standard dataclass implementation in Python, for which Pydantic is a better fit.

do you have guidance on learning about the business model of the company and/or the features that Python lacks that Pydantic solves for? I generally shy away from software if it makes money through another method than it’s open source (like observability), and is below a certain size

[jw-y]

Thanks for the suggestion.

I try my best to avoid using external libraries in Python, as they are often the biggest source of various problems. I believe that extra type validation by pydantic is unnecessary, since pkl and Python provide most of the validation needed.

However, I acknowledge the vast user base of FastAPI. Thus, if there is enough demand, supporting pydantic is a possibility. For now, the biggest concern is the correctness of code generation using dataclasses.

Please note that code generation has limitations, such as no default value support, which might not provide enough features for pydantic. Currently, it might be possible to use regex to handle converting from dataclasses to pydantic.

[almogtavor]

@jw-y yeah I think that in python there's a need for Pydantic especially due to the vast user base. People would want it bot for the validations, but for the integration with popular libraries such as FastAPI that heavily rely on Pydantic models. It's not a use case in other languages because typed languages kind of have "build in" Pydantic. Lots of people don't use data classes, and instead use Pydantic.

I also think that because Pydantic is so popular for Python properties loading, people won't migrate from this to pkl-python that's based on data classes because it will force them to change a lot of code, and "downgrade" from the feature-rich Pydantic models to data classes.

I agree that this may be less important than making the data classes generation more robust, but it's still important to tackle this someday. Even if it won't include all of Pydantic features.

A quick addition at this direction can be to give people to choose between dataclasses.dataclass and pydantic.dataclasses.dataclass.

[almogtavor]

@almogtavor - not sure what is missing from the standard dataclass implementation in Python, for which Pydantic is a better fit.

do you have guidance on learning about the business model of the company and/or the features that Python lacks that Pydantic solves for? I generally shy away from software if it makes money through another method than it’s open source (like observability), and is below a certain size

@jaanli I'd also try to prevent using third party libraries. But Pydantic isn't a classic third party library. People are using it massively, and sometimes rely on it like it's a built in tool.