You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+30-13Lines changed: 30 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,6 @@ This repository wants to define an open specification to define data products wi
8
8
With an open specification will be possible to create services for automatic deployment and interoperable components to build a Data Mesh platform.
9
9
10
10
11
-
12
11
# Data Product structure
13
12
14
13
The DP is composed by a general section with DP level information and four sub-structures:
@@ -33,7 +32,9 @@ The fixed structure must be technology agnostic.
33
32
*`Version: [String]` this is representing the version of the DP, because we consider the DP as an indipendent unit of deployment, so if a breaking change is needed, we create a brand new versionof the DP. If we introduce a new feature or patch it is not necessary create a new version, but we can change Y (new feature) or Z patch. Displayed as X.Y.Z where X is major version, Y is minor and Z is patch. Major version(X) is also shown in the ID and those 2 fields(version and ID) are always aligned with one another.
34
33
** Constraints:
35
34
*** Major version of the data product is always the same as the major version of the components and it is the same version that is shown in both data product ID and component ID
36
-
*`DataProductOwner: [String]` Data Product Owner, the actual user that receives the notifications about data product
35
+
*`Kind: [String]` type of component. Allowed values: [dataproduct | outputport | workload | storage | resource]
36
+
*`DataProductOwner: [String]` Data Product Owner, the id of actual user that receives the notifications about data product
37
+
*`DataProductOwnerDisplayName [String]`: The human readable version of `DataProductOwner`
37
38
*`Email: [String]` Point of contact, it could be the owner or a distribution list, but must be reliable and responsive.
38
39
*`InformationSLA: [String]` Describe what SLA the DP team is providing to answer additional information requests about the DP
39
40
*`Status: [String]` This is an enum representing the status of this version of the DP `[Draft|Published|Retired]`
@@ -45,22 +46,22 @@ The fixed structure must be technology agnostic.
45
46
The **unique identifier** of a DataProduct is the concatenation of Domain, Name and Version. So we will refer to the `DP_UK` as a string composed in the following way `$DPDomain.$DPID.$DPVersion`
46
47
47
48
48
-
49
-
50
49
### Output Ports
51
50
52
51
*`ID: [String]` the unique identifier of the output port --> not modifiable
53
52
** Constraints:
54
53
*** Allowed characters are `[a-zA-Z]` and `[_-]`
55
54
*** Output port ID is made of `$DPDomain.$DPIdentifier.$DPMajorVersion.$OutputPortIdentifier`
56
-
*`Name: [String]` the name of the DP
55
+
*`Name: [String]` the name of the Output Port
57
56
*`FullyQualifiedName: [String]` Human-readable that uniquely identifies an entity
58
-
*`ResourceType: [String]` the kind of output port: Files - SQL - Events. This should be extendible with GraphQL or others.
57
+
*`OutputPortType: [String]` the kind of output port: Files - SQL - Events. This should be extendible with GraphQL or others.
59
58
*`Technology: [String]` the underlying technology is useful for the consumer to understand better how to consume the output port and also needed for self serve provisioning specific stuff.
59
+
*`Platform`: This represents the vendor: Azure, GCP, AWS, CDP on AWS, etc. It is a free field but it is useful to understand better the platform where the component will be running
60
60
*`Description: [String]` detailed explanation about the function and the meaning of the output port
61
61
*`Version: [String]` Specific version of the output port. Displayed as X.Y.Z where X is the major version of the data product, Y is minor feature and Z is patch. Major version(X) is also shown in the component ID and those 2 fields(version and ID) are always aligned with one another.
62
62
** Constraints:
63
63
*** Major version of the data product is always the same as the major version of the components and it is the same version that is shown in both data product ID and component ID
64
+
*`Kind: [String]` type of component. Allowed values: [dataproduct | outputport | workload | storage | resource]
64
65
*`CreationDate: [String]` when this output port has been created
65
66
*`StartDate: [String]` the first business date present in the dataset, leave it null for events or we can use some standard semantic like: "-7D, -1Y"
66
67
*`ProcessDescription: [String]` what is the underlying process that contributes to generate the data exposed by this output port
@@ -71,14 +72,17 @@ The **unique identifier** of a DataProduct is the concatenation of Domain, Name
71
72
*`IntervalOfChange: [String]` How often changes in the data are reflected
72
73
*`Timeliness: [String]` The skew between the time that a business fact occuts and when it becomes visibile in the data
73
74
*`Endpoint: [URL]` this is the API endpoint that self-describe the output port and provide insightful information at runtime about the physical location of the data, the protocol must be used, etc
74
-
*`Allow: [Array[String]]` It is an array of user/role/group related to the specific technology ( each technology will have an associated authentication system ( Azure AD, AWS IAM, etc ). This field is defining who has access in read-only to this specific output port
75
+
*`Allows: [Array[String]]` It is an array of user/role/group related to LDAP/AD user. This field is defining who has access in read-only to this specific output port
76
+
*`Owners: [Array[String]]` It is an array of user/role/group related to LDAP/AD user. This field defines who has all permissions on this specific output port
77
+
*`InfrastructureTemplateId` the id of the microservice responsible for provisioning the component. A microservice may be capable of provisioning several UseCaseTemplateId
78
+
*`UseCaseTemplateId` the id of the template used in the builder to create the component
75
79
*`DependsOn: [Array[String]]` An output port could depend on other output ports or storage areas, for example a SQL Output port could be dependent on a Raw Output Port because it is just an external table.
76
80
** Constraints:
77
81
*** This array will only contain ID-s
78
82
*`Tags: [Array[Yaml]]` Free tags at OutputPort level ( please refer to OpenMetadata https://docs.open-metadata.org/openmetadata/schemas/entities/tagcategory )
79
83
*`SampleData: [Yaml]` - Provide a sample data of your outputport. See OpenMetadata specification: https://docs.open-metadata.org/openmetadata/schemas/entities/table#tabledata
80
84
*`Schema: [Array[Yaml]]` When it comes to describe a schema we propose to leverage OpenMetadata specification: Ref https://docs.open-metadata.org/openmetadata/schemas/entities/table#column. Each column can have a tag array and you can choose between simples LabelTags, ClassificationTags or DescriptiveTags. Here an example of classification Tag https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/data/tags/piiTags.json
81
-
*`SemanticLinking: [Yaml]` Here we can express semantic relationships between this output port and other outputports ( also coming from other domains and data products )
85
+
*`SemanticLinKind: [Yaml]` Here we can express semantic relationships between this output port and other outputports ( also coming from other domains and data products )
82
86
*`Specific: [Yaml]` this is a custom section where we can put all the information strictly related to a specific technology or dependent from a standard/policy defined in the federated governance.
83
87
84
88
@@ -90,14 +94,17 @@ The **unique identifier** of a DataProduct is the concatenation of Domain, Name
90
94
*** Workload ID is made of `$DPDomain.$DPIdentifier.$DPMajorVersion.$WorkloadIdentifier`
91
95
*`Name: [String]` the name of the workload
92
96
*`FullyQualifiedName: [String]` Human-readable that uniquely identifies an entity
93
-
*`Description: [String]` detailed description about the process, its purpose and characteristics
94
-
*`ResourceType: [String]` explain what type of workload is: Ingestion ETL, Streaming, Internal Process, etc.
95
-
*`Type: [String]` This is an enum `[HouseKeeping|DataPipeline]`, `Housekeeping` is for all the workloads that are acting on internal data without any external dependency. `DataPipeline` instead is for workloads that are reading from outputport of other DP or external systems.
96
-
*`Technology: [String]` this is a list of technologies: Airflow, Spark, Scala. It is a free field but it is useful to understand better how it is behaving
97
97
*`Description: [String]` detailed explaination about the purpose of the workload, what sources is reading, what business logic is apllying, etc
98
+
*`Kind: [String]` type of component. Allowed values: [dataproduct | outputport | workload | storage | resource]
99
+
*`WorkloadType: [String]` explain what type of workload is: Ingestion ETL, Streaming, Internal Process, etc.
100
+
*`ConnectionType: [String]` This is an enum `[HouseKeeping|DataPipeline]`, `Housekeeping` is for all the workloads that are acting on internal data without any external dependency. `DataPipeline` instead is for workloads that are reading from outputport of other DP or external systems.
101
+
*`Technology: [String]` this is a list of technologies: Airflow, Spark, Scala. It is a free field but it is useful to understand better how it is behaving
102
+
*`Platform`: This represents the vendor: Azure, GCP, AWS, CDP on AWS, etc. It is a free field but it is useful to understand better the platform where the component will be running
98
103
*`Version: [String]` Specific version of the workload. Displayed as X.Y.Z where X is the major version of the data product, Y is minor feature and Z is patch. Major version(X) is also shown in the component ID and those 2 fields(version and ID) are always aligned with one another.
99
104
** Constraints:
100
105
*** Major version of the data product is always the same as the major version of the components and it is the same version that is shown in both data product ID and component ID
106
+
*`InfrastructureTemplateId` the id of the microservice responsible for provisioning the component. A microservice may be capable of provisioning several UseCaseTemplateId
107
+
*`UseCaseTemplateId` the id of the template used in the builder to create the component
101
108
*`Tags: [Array[Yaml]]` Free tags at Workload level ( please refer to OpenMetadata https://docs.open-metadata.org/openmetadata/schemas/entities/tagcategory )
102
109
*`ReadsFrom: [Array[String]]` This is filled only for `DataPipeline` workloads and it represents the list of output ports or external systems that is reading. Output Ports are identified with `DP_UK.OutputPort_ID`, while external systems will be defined by a string `EX_$systemdescription`. Here we can elaborate a bit more and create a more semantic struct.
103
110
** Constraints:
@@ -107,11 +114,21 @@ The **unique identifier** of a DataProduct is the concatenation of Domain, Name
107
114
108
115
### Storage Area
109
116
110
-
*`ID: [String]` the unique identifier of the Storage Area
117
+
*`ID: [String]` the unique identifier of the Storage Area.
118
+
** Constraints:
119
+
*** Allowed characters are `[a-zA-Z]` and `[_-]`
120
+
*** Output port ID is made of `$DPDomain.$DPIdentifier.$DPMajorVersion.$OutputPortIdentifier`
111
121
*`Name: [String]` the name of the Storage Area
112
122
*`FullyQualifiedName: [String]` Human-readable that uniquely identifies an entity
123
+
*`StorageType: [String]` the kind of storage: Files - SQL - Events.
113
124
*`Technology: [String]` this is a list of technologies: S3, ADLS, SQLServer, Kafka.
125
+
*`Platform`: This represents the vendor: Azure, GCP, AWS, CDP on AWS, etc. It is a free field but it is useful to understand better the platform where the component will be running
114
126
*`Description: [String]` detailed explanation about the function and the meaning of this storage area
127
+
*`Kind: [String]` type of component. Allowed values: [dataproduct | outputport | workload | storage | resource]
128
+
*`Allows: [Array[String]]` It is an array of user/role/group related to LDAP/AD user. This field is defining who has access in read-only to this specific storage area
129
+
*`Owners: [Array[String]]` It is an array of user/role/group related to LDAP/AD user. This field defines who has all permissions on this specific storage area
130
+
*`InfrastructureTemplateId` the id of the microservice responsible for provisioning the component. A microservice may be capable of provisioning several UseCaseTemplateId
131
+
*`UseCaseTemplateId` the id of the template used in the builder to create the component
115
132
*`Tags: [Array[Yaml]]` Free tags at Storage area level ( please refer to OpenMetadata https://docs.open-metadata.org/openmetadata/schemas/entities/tagcategory )
116
133
*`Specific: [Yaml]` this is a custom section where we can put all the information strictly related to a specific technology or dependent from a standard/policy defined in the federated governance.
0 commit comments