Support ODPS v1.1.0

AGENTS.md

@@ -67,9 +67,13 @@ before changing a command.
 
 ## Refreshing the bundled ODPS schema
 
-The ODPS JSON Schema is vendored at
-`dataproduct/schemas/odps-1.0.0.schema.json` from
-`https://raw.githubusercontent.com/bitol-io/open-data-product-standard/main/schema/odps-json-schema-v1.0.0.json`.
+The ODPS JSON Schemas are vendored at `dataproduct/schemas/odps-<version>.schema.json`;
+`dataproduct/schemas/download` refreshes them (parallel to datacontract-cli's
+`datacontract/schemas/download`). `lint` picks the bundled schema by the
+document's `apiVersion` (`ODPS_SCHEMA_VERSIONS` in `dataproduct/lint/schema.py`).
+A new ODPS release means: add it to `download` and run it, register it in
+`ODPS_SCHEMA_VERSIONS`, bump `DEFAULT_ODPS_SCHEMA_VERSION`, and move the init
+template to the new version.
 
 ## Release
 

CHANGELOG.md

@@ -5,6 +5,10 @@ what changed (user-facing).
 
 ## [Unreleased]
 
+- Support ODPS v1.1.0: `lint` validates against the bundled v1.1.0 JSON Schema (`type`, `context`, `synonyms`, `deprecated`, `customProperties[].vendor`, element `id`s, optional port `version`/`contractId`)
+- `lint` validates against the bundled JSON Schema for the `apiVersion` the document declares (`v1.1.0` → v1.1.0 schema; `v1.0.0`/`v0.9.0` → v1.0.0 schema; unknown → newest), and check names state which schema ran
+- `init` template now uses `apiVersion: v1.1.0`
+
 ## [0.1.0]
 
 - `init` command: create a valid `dataproduct.odps.yaml` from a bundled ODPS v1.0.0 template

README.md

@@ -2,7 +2,7 @@
 
 The `dataproduct` CLI is an open-source command-line tool for working with
 **data products** defined with the
-[Open Data Product Standard (ODPS)](https://bitol-io.github.io/open-data-product-standard/v1.0.0/).
+[Open Data Product Standard (ODPS)](https://bitol-io.github.io/open-data-product-standard/v1.1.0/).
 
 It is the data-product sibling of
 [`datacontract-cli`](https://github.com/datacontract/datacontract-cli) (which
@@ -41,8 +41,10 @@ dataproduct lint --output-format junit --output TEST-dataproduct.xml
 dataproduct lint --json-schema ./odps.schema.json  # validate against a custom schema
 ```
 
-Validation is schema-only in 0.1: the data product is checked against the
-bundled ODPS v1.0.0 JSON Schema. Exit code is `0` when valid, `1` otherwise.
+Validation is schema-only: the data product is checked against the bundled
+ODPS JSON Schema matching its `apiVersion` (`v1.1.0`, `v1.0.0`, or `v0.9.0`;
+unknown versions are validated against the latest). Exit code is `0` when
+valid, `1` otherwise.
 
 ### `publish` — publish to Entropy Data
 

dataproduct/data_product.py

@@ -11,7 +11,7 @@
 from dataproduct.config import Config
 from dataproduct.integration.entropy_data import publish_data_product_to_entropy_data
 from dataproduct.lint.files import read_resource
-from dataproduct.lint.schema import fetch_schema
+from dataproduct.lint.schema import fetch_schema, schema_version_for
 from dataproduct.lint.validate import parse_yaml, validate_against_schema
 from dataproduct.model.exceptions import DataProductException
 from dataproduct.model.run import Check, ResultEnum, Run
@@ -46,15 +46,16 @@ def _load_dict(self) -> dict:
         return parse_yaml(content)
 
     def lint(self) -> Run:
-        """Validate the data product against the ODPS JSON Schema (schema-only)."""
+        """Validate the data product against the ODPS JSON Schema matching its ``apiVersion`` (schema-only)."""
         run = Run.create_run()
         run.log_info("Linting data product")
         try:
             data = self._load_dict()
             run.dataProductId = data.get("id")
             run.dataProductVersion = data.get("version")
-            schema = fetch_schema(self._schema_location)
-            checks = validate_against_schema(data, schema, self._all_errors)
+            schema_version = None if self._schema_location else schema_version_for(data.get("apiVersion"))
+            schema = fetch_schema(self._schema_location, schema_version)
+            checks = validate_against_schema(data, schema, self._all_errors, schema_version)
             if checks:
                 run.checks.extend(checks)
                 for check in checks:
@@ -64,7 +65,9 @@ def lint(self) -> Run:
                     Check(
                         type="lint",
                         result=ResultEnum.passed,
-                        name="Data product is syntactically valid",
+                        name="Data product is syntactically valid"
+                        if schema_version is None
+                        else f"Data product is valid against ODPS v{schema_version}",
                     )
                 )
         except DataProductException as e:

dataproduct/init/init_template.py

@@ -3,7 +3,7 @@
 
 import requests
 
-DEFAULT_DATA_PRODUCT_INIT_TEMPLATE = "odps-1.0.0.init.yaml"
+DEFAULT_DATA_PRODUCT_INIT_TEMPLATE = "odps-1.1.0.init.yaml"
 
 
 def get_init_template(location: str = None) -> str:

dataproduct/lint/schema.py

@@ -3,26 +3,46 @@
 import logging
 import os
 from pathlib import Path
-from typing import Any, Dict, Union
+from typing import Any, Dict, Optional, Union
 
 import requests
 
 from dataproduct.model.exceptions import DataProductException
 from dataproduct.model.run import ResultEnum
 
-DEFAULT_DATA_PRODUCT_SCHEMA = "odps-1.0.0.schema.json"
+# ODPS v1.1.0 relaxed several required fields (e.g. `status`, port `version`/`contractId`)
+# and added new ones, so older documents must be validated against their own schema.
+# v0.9.0 has no dedicated bundled schema; the v1.0.0 schema accepts it.
+ODPS_SCHEMA_VERSIONS = {
+    "v1.1.0": "1.1.0",
+    "v1.0.0": "1.0.0",
+    "v0.9.0": "1.0.0",
+}
+DEFAULT_ODPS_SCHEMA_VERSION = "1.1.0"
 
 
-def fetch_schema(location: Union[str, Path] = None) -> Dict[str, Any]:
+def schema_version_for(api_version: Any = None) -> str:
+    """Return the bundled ODPS schema version for a document's ``apiVersion``.
+
+    Unknown or missing versions fall back to the newest bundled schema, which
+    then reports the invalid ``apiVersion`` as a schema violation.
+    """
+    if isinstance(api_version, str):
+        return ODPS_SCHEMA_VERSIONS.get(api_version, DEFAULT_ODPS_SCHEMA_VERSION)
+    return DEFAULT_ODPS_SCHEMA_VERSION
+
+
+def fetch_schema(location: Union[str, Path] = None, schema_version: Optional[str] = None) -> Dict[str, Any]:
     """Fetch the ODPS JSON Schema to validate against.
 
-    ``None`` uses the bundled ODPS v1.0.0 schema; otherwise ``location`` is a URL
-    or local path.
+    ``None`` uses the bundled schema for ``schema_version`` (newest when
+    omitted); otherwise ``location`` is a URL or local path.
     """
     if location is None:
-        logging.info("Use default bundled schema " + DEFAULT_DATA_PRODUCT_SCHEMA)
+        schema_name = f"odps-{schema_version or DEFAULT_ODPS_SCHEMA_VERSION}.schema.json"
+        logging.info("Use default bundled schema " + schema_name)
         schemas = resources.files("dataproduct")
-        schema_file = schemas.joinpath("schemas", DEFAULT_DATA_PRODUCT_SCHEMA)
+        schema_file = schemas.joinpath("schemas", schema_name)
         with schema_file.open("r") as file:
             return json.load(file)
 

dataproduct/lint/validate.py

@@ -1,4 +1,4 @@
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Optional
 
 import yaml
 from jsonschema.validators import validator_for
@@ -32,11 +32,15 @@ def parse_yaml(content: str) -> Dict[str, Any]:
     return data
 
 
-def validate_against_schema(data: Dict[str, Any], schema: Dict[str, Any], all_errors: bool = False) -> List[Check]:
+def validate_against_schema(
+    data: Dict[str, Any], schema: Dict[str, Any], all_errors: bool = False, schema_version: Optional[str] = None
+) -> List[Check]:
     """Validate ``data`` against the ODPS JSON Schema.
 
     Returns a list of ``error`` checks — empty when the document is valid. With
-    ``all_errors=False`` (default) only the first violation is reported.
+    ``all_errors=False`` (default) only the first violation is reported. Check
+    names state the bundled ``schema_version`` that ran; ``None`` means a custom
+    schema was supplied and no version is named.
     """
     validator_cls = validator_for(schema)
     validator_cls.check_schema(schema)
@@ -46,16 +50,21 @@ def validate_against_schema(data: Dict[str, Any], schema: Dict[str, Any], all_er
     if not all_errors:
         errors = errors[:1]
 
+    name = (
+        "Check that data product YAML is valid"
+        if schema_version is None
+        else f"Check that data product is valid against ODPS v{schema_version}"
+    )
     checks: List[Check] = []
     for error in errors:
-        path = "/".join(str(p) for p in error.absolute_path) or "(root)"
+        path = "/".join(str(p) for p in error.absolute_path)
         checks.append(
             Check(
                 type="lint",
                 result=ResultEnum.error,
-                name=f"Schema validation failed at '{path}'",
-                reason=error.message,
-                field=path,
+                name=name,
+                reason=f"{path}: {error.message}" if path else error.message,
+                field=path or "(root)",
             )
         )
     return checks

dataproduct/schemas/download

@@ -0,0 +1,9 @@
+#!/bin/bash
+# Refresh the vendored ODPS JSON Schemas (newest from its release tag, older ones from main,
+# as datacontract-cli does). After adding a version here, register it in ODPS_SCHEMA_VERSIONS
+# (dataproduct/lint/schema.py).
+set -e
+cd "$(dirname "$0")"
+
+curl -o odps-1.0.0.schema.json https://raw.githubusercontent.com/bitol-io/open-data-product-standard/refs/heads/main/schema/odps-json-schema-v1.0.0.json
+curl -o odps-1.1.0.schema.json https://raw.githubusercontent.com/bitol-io/open-data-product-standard/refs/tags/v1.1.0/schema/odps-json-schema-v1.1.0.json

dataproduct/schemas/odps-1.0.0.init.yaml → dataproduct/schemas/odps-1.1.0.init.yaml

@@ -1,9 +1,10 @@
-apiVersion: v1.0.0
+apiVersion: v1.1.0
 kind: DataProduct
 id: my-data-product-id
 name: My Data Product
 version: v1.0.0
 status: draft
+# type: consumerAligned  # sourceAligned | aggregate | consumerAligned
 
 description:
   purpose: Purpose of the data product.