Refactor/upgrade pydantic to v2

[georgeRobertson2]

Pydantic Changes:

• BaseModel:

  • fields to model_fields
  • dict() to model_dump()
  • pydantic.main.ModelMetaclass removed and replaced with BaseModel

• Field:

  • regex in pydantic.Field changed to pattern

• Pydantic Config:

  • Removed Config subclass in Pydantic models and replaced with model_config
    e.g. model_config = {...} OR ConfigDict()
  • allow_population_by_field_name in Confg renamed to populate_by_name
  • fields in Config (now model_config) depreciated
  • anystr_strip_whitespace in Config renamed to str_strip_whitespace

• Pydantic Validators:

  • pre arg in @validator now mode. E.g. pre=True -> mode="before"
  • allow_reuse arg in @validator no longer required, recommended to delete and no issue
  • always=True arg in @validator and @root_validator depreciated and you now define within the Field with validate_default arg
  • @root_validator changed to @model_validator
  • kwarg skip_on_failure depreciated and the logic of skip_on_failure=True is now default behaviour
  • added info kwarg to each field validator as that is how you now access the metadata of the model
  • removed values as @field_validator now just has cls, value and ValidationInfo. Reference to other fields will be obtained through ValidationInfo.data
  • @validate_arguments replaced with @validate_call and kwargs replaced with ConfigDict

• Pydantic Types:

  • get_validators replaced with get_pydantic_core_schema
    • The way you now call the validation functions has changed. You return a core_schema object which either returns a CoreSchema object or a ChainSchema consisting of CoreSchema objects. You can additionally pass in a type handler which makes checks like if value is None: return value irrelevant as the handler performs the check on the datatype based on whether the model allows optional or not.
  • types.ConstrainedStr replaced with types.StringConstraints
  • regex attr in ConstrainedStr replaced with pattern
  • ConstrainedDecimal retired and replaced with condecimal type
  • Optional for types when defined with no default will not allow nulls. So, in cases where it we seemed to be allowing nulls, I have placed a = None default
  • curtail_length kwarg for ConstrainedStr has been depreciated

Specifc DVE changes:

• Auditing:

  • changed the schema generation the BaseAuditor so it can work with FieldInfo object
  • changed the polars schema generation in DDBAuditor to use get_polars_type_from_annotation rather than using PYTHON_TYPE_TO_POLARS_TYPE map as the FieldInfo object needs to be evaluated into a Polars type
  • changed ddb_create_table_sql in DDBAuditor to use get_duckdb_typ_from_annotation as FieldInfo object needs to be evaluated into a Polars type
  • Removed is_complex() as no longer available due to model_fields returning FieldInfo objects. Replaced with a new custom is_field_complex() function

• Domain Types (custom types):

  • _SimpleRegexValidator base type switched to types.StringConstraints per ConstrainedStr being depreciated
  • NHSNumber:
    • removed parent type as no longer required
    • _warn_for_possible_invalid_number method has had loc argument removed as it can no longer be supplied with the way pydantic v2 validations are written
    • validate method has value to allow str or int type in the arguments as that was the expectation in the tests
    • validate method has had field argument removed as it can no longer be supplied with the way pydantic v2 validations are written
    • added a new method called get_pydantic_core_schema to facilitate new method of creating custom domain type validations. This now configures the validations required to validate the inbound value

• function wrapper:

  • _validator_kwargs has a number of arguments that have been depreciated in Pydantic v2 (pre, each_item, always, whole, allow_reuse)

• FieldSpecification:

  • Added a check for Annotated types that now frequently pass due to Pyd types now mapping to Annotated types rather than Pydantic type objects

Fixes post 23/05/26:

• Fixed file transformation error on Scenario: Validate complex nested XML data (duckdb)
  • This was due to an error with checking whether a type was complicated
    • wrote a new is_type_complex
    • wrote a strip optional
    • now check the type rather than the FieldInfo object when generating the template_row(s) in xml reading
• Change the "missing" check logic in the FeedbackMessage.from_pydantic_error to explicitly check for missing values as missing validation errors in Pydantic only trigger when a kwarg isn't provided

[sonarqubecloud]

Quality Gate failed

Failed conditions
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE