data contracts and sharing agreements (#10)

agile-lab · web-flow · commit 7e171f531b55 · 2022-06-14T10:33:14.000+02:00
diff --git a/README.md b/README.md
@@ -37,6 +37,8 @@ The fixed structure must be technology agnostic. The first fields of teh fixed s
 * `DataProductOwner: [String]` Data Product owner, the unique identifier of the actual user that owns, manages, and receives notifications about the Data Product. To make it technology independent it is usually the email address of the owner.
 * `DataProductOwnerDisplayName [String]`: the human readable version of `DataProductOwner`.
 * `Email: [Option[String]]` point of contact between consumers and maintainers of the Data Product. It could be the owner or a distribution list, but must be reliable and responsive.
+* `OwnerGroup [String]`: LDAP user/group that is owning the data product
+* `DevGroup [String]`: LDAP user/group that is in charge to develop and maintain the data product
 * `InformationSLA: [Option[String]]` describes what SLA the Data Product team is providing to answer additional information requests about the Data Product itself.
 * `Status: [Option[String]]` this is an enum representing the status of this version of the Data Product. Allowed values are: `[Draft|Published|Retired]`. This is a metadata that communicates the overall status of the Data Product but is not reflected to the actual deployment status.
 * `Maturity: [Option[String]]` this is an enum to let the consumer understand if it is a tactical solution or not. It is really useful during migration from Data Warehouse or Data Lake. Allowed values are: `[Tactical|Strategic]`.
@@ -62,8 +64,6 @@ Constraints:
   * Major version of the Data Product is always the same as the major version of all of its components and it is the same version that is shown in both Data Product ID and component ID.
 * `InfrastructureTemplateId: [String]` the id of the microservice responsible for provisioning the component. A microservice may be capable of provisioning several components generated from different use case templates. 
 * `UseCaseTemplateId: [Option[String]]` the id of the template used in the builder to create the component. Could be empty in case the component was not created from a builder template.
-* `Allows: [Array[String]]` It is an array of user/role/group related to LDAP/AD user. This field is defining who has access in read-only to this specific output port.
-* `Owners: [Array[String]]` It is an array of user/role/group related to LDAP/AD user. This field defines who has all permissions on this specific output port.
 * `DependsOn: [Array[String]]` An output port could depend on other output ports or storage areas, for example a SQL Output port could be dependent on a Raw Output Port because it is just an external table.
 Constraints:
   * This array will only contain IDs of other components.
@@ -73,16 +73,24 @@ Constraints:
 * `CreationDate: [Optional[String]]` when this output port has been created.
 * `StartDate: [Optional[String]]` the first business date present in the dataset, leave it empty for events or we can use some standard semantic like: "-7D, -1Y".
 * `ProcessDescription: [Option[String]]` what is the underlying process that contributes to generate the data exposed by this output port.
-* `BillingPolicy: [Option[String]]` how a consumer will be charged back when it consumes this output port.
-* `SecurityPolicy: [Option[String]]` additional information related to security aspects, like restrictions, maskings, sensibile information.
-* `ConsumptionPolicy: [Option[String]]` any other information needed by the consumer in order to effectively consume the data, it could be related to technical stuff, regulation, security, etc.
-* `SLO:[Yaml]`
-  * `IntervalOfChange: [Option[String]]` how often changes in the data are reflected.
-  * `Timeliness: [Option[String]]` the skew between the time that a business fact occuts and when it becomes visibile in the data.
-* `Endpoint: [Option[URL]]` this is the API endpoint that self-describe the output port and provide insightful information at runtime about the physical location of the data, the protocol must be used, etc.
-* `Tags: [Array[Yaml]]` Tag labels at OutputPort level ( please refer to OpenMetadata https://docs.open-metadata.org/metadata-standard/schemas/types/taglabel).
+* `DataContract: [Yaml]`: In case something is going to change in this section, it represents a breaking change because the producer is breaking the contract, this will require to create a new version of the data product to keep backward compatibility
+  * `Schema: [Array[Yaml]]` when it comes to describe a schema we propose to leverage OpenMetadata specification: Ref https://docs.open-metadata.org/metadata-standard/schemas/entities/table#column. Each column can have a tag array and you can choose between simples LabelTags, ClassificationTags or DescriptiveTags. Here an example of classification Tag https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/data/tags/piiTags.json.
+  * `SLA: [Yaml]` Service Level Agreement, describe the quality of data delivery and the output port in general. It represents the producer's overall promise to the consumers.
+    * `IntervalOfChange: [Option[String]]` how often changes in the data are reflected.
+    * `Timeliness: [Option[String]]` the skew between the time that a business fact occuts and when it becomes visibile in the data.
+    * `UpTime: [Option[String]]` the percentage of port availability.
+  * `TermsAndConditions: [Option[String]]` If the data is usable only in specific environments.
+  * `Endpoint: [Option[URL]]` this is the API endpoint that self-describe the output port and provide insightful information at runtime about the physical location of the data, the protocol must be used, etc.
+* `DataSharingAgreement: [Yaml]` This part is covering usage, privacy, purpose, limitations and is indipendent by the data contract.
+  * `Purpose: [Option[String]]` what is the goal of this data set.
+  * `Billing: [Option[String]]` how a consumer will be charged back when it consumes this output port.
+  * `Security: [Option[String]]` additional information related to security aspects, like restrictions, maskings, sensibile information and privacy.
+  * `IntendedUsage: [Option[String]]` any other information needed by the consumer in order to effectively consume the data, it could be related to technical stuff (e.g. extract no more than one year of data for good performances ) or to business domains (e.g. this data is only useful in the marketing domains).
+  * `Limitations: [Option[String]]` If any limitation is present it must be made super clear to the consumers.
+  * `LifeCycle: [Option[String]]` Describe how the data will be historicized and how and when it will be deleted.
+  * `Confidentiality: [Option[String]]` Describe what a consumer should do to keep the information confidential, how to process and store it. Permission to share or report it.
+* `Tags: [Array[Yaml]]` Tag labels at OutputPort level, here we can have security classification for example (please refer to OpenMetadata https://docs.open-metadata.org/metadata-standard/schemas/types/taglabel).
 * `SampleData: [Option[Yaml]]` provides a sample data of your Output Port. See OpenMetadata specification: https://docs.open-metadata.org/openmetadata/schemas/entities/table#tabledata
-* `Schema: [Array[Yaml]]` when it comes to describe a schema we propose to leverage OpenMetadata specification: Ref https://docs.open-metadata.org/metadata-standard/schemas/entities/table#column. Each column can have a tag array and you can choose between simples LabelTags, ClassificationTags or DescriptiveTags. Here an example of classification Tag https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/data/tags/piiTags.json.
 * `SemanticLinking: [Option[Yaml]]` here we can express semantic relationships between this output port and other outputports (also coming from other domains and data products). For example we could say that column "customerId" of our SQL Output Port references the column "id" of the SQL Output Port of the "Customer" Data Product.
 * `Specific: [Yaml]` this is a custom section where we must put all the information strictly related to a specific technology or dependent from a standard/policy defined in the federated governance.
 
diff --git a/example.yaml b/example.yaml
@@ -9,6 +9,8 @@ environment: development
 dataProductOwner: tom_smith_corp.com
 dataProductOwnerDisplayName: Tom Smith
 email: mailto:distribution_list@corp.com
+ownerGroup: dataproduct1_corp.com
+devGroup: dataproduct1_dev_corp.com
 informationSLA: 2WD
 status: DRAFT
 maturity: Strategic
@@ -24,25 +26,32 @@ components:
     version: 1.0.1
     infrastructureTemplateId: microservice-id-1
     useCaseTemplateId: template-id-1
-    allows: [user-1]
-    owners: [user-2]
     dependsOn: []
     platform: CDP on AWS
     technology: s3_cdp
     outputPortType: Files
     creationDate: 05-04-2022 16:53:00
     startDate:
     processDescription: this output port is generated by a Spark Job scheduled every day at 2AM and it lasts for approx 2 hours
-    billingPolicy: 5$ for each full scan
-    securityPolicy: In order to consume this output port an additional security check with compliance must be done
-    consumerPolicy: This is only for HR department and not suitable for institutional reporting.
-    SLO:
-      intervalOfChange: 1 hours
-      timeliness: 1 minutes 
-    endpoint: https://myurl/development/my_domain/my_data_product/1.0.0/my_raw_s3_port
+    dataContract:
+      schema: []
+      SLA:
+        intervalOfChange: 1 hours
+        timeliness: 1 minutes 
+        upTime: 99.9%
+      termsAndConditions: only usable in development environment
+      endpoint: https://myurl/development/my_domain/my_data_product/1.0.0/my_raw_s3_port
+    dataSharingAgreements:
+      purpose: this output port want to provide a rich set of profitability KPIs related to the customer
+      billing: 5$ for each full scan
+      security: In order to consume this output port an additional security check with compliance must be done
+      intendedUsage: the dataset is huge so it is reccomended to extract maximum 1 year of data and to use these KPIs in the marketing or sales domain, but not for customer care
+      limitations: is not possible to use this data without a compliance check
+      lifeCycle: the maximum retention is 10 years, and eviction is happening on the first of january
+      confidentiality: if you want to store this data somewhere else, PII columns must be masked    
     tags: []
     sampleData: {}
-    schema: []
+    
     semanticLinking: {}
     specific:
       directory: history
@@ -55,22 +64,29 @@ components:
     version: 1.1.0
     infrastructureTemplateId: microservice-id-2
     useCaseTemplateId: template-id-2
-    allows: [user-1]
-    owners: [user-2]
     dependsOn: [urn:dmb:cmp:my_domain.my_data_product.1.my_raw_s3_port]
     platform: CDP on AWS
     technology: impala_cdp
     outputPortType: SQL
     creationDate: 05-04-2022 17:00:00
     startDate:
     processDescription:
-    billingPolicy:
-    securityPolicy:
-    consumerPolicy:
-    SLO:
-      intervalOfChange: 1 hours
-      timeliness: 1 minutes
-    endpoint:
+    dataContract:
+      schema: []
+      SLA:
+        intervalOfChange: 1 hours
+        timeliness: 1 minutes 
+        upTime: 99.9%
+      termsAndConditions: only usable in development environment
+      endpoint: https://myurl/development/my_domain/my_data_product/1.0.0/my_raw_s3_port
+    dataSharingAgreements:
+      purpose: this output port want to provide a rich set of profitability KPIs related to the customer
+      billing: 5$ for each full scan
+      security: In order to consume this output port an additional security check with compliance must be done
+      intendedUsage: the dataset is huge so it is reccomended to extract maximum 1 year of data and to use these KPIs in the marketing or sales domain, but not for customer care
+      limitations: is not possible to use this data without a compliance check
+      lifeCycle: the maximum retention is 10 years, and eviction is happening on the first of january
+      confidentiality: if you want to store this data somewhere else, PII columns must be masked    
     tags: []
     sampleData: {}
     schema: []
