diff --git a/docs/automate/processing-data-with-variable-input-workflows/index.md b/docs/automate/processing-data-with-variable-input-workflows/index.md index c3721a523..012b210b8 100644 --- a/docs/automate/processing-data-with-variable-input-workflows/index.md +++ b/docs/automate/processing-data-with-variable-input-workflows/index.md @@ -11,54 +11,122 @@ tags: ## Introduction This tutorial shows how you can create and use data integration workflows to process data coming from outside Corporate Memory (i.e., without registering datasets). -The concept to achieve this is named a **Variable Dataset**. -A variable dataset is created and used inside of a workflow as an input for other tasks (e.g. a transformation) at the place where a "regular" dataset (such as register CSV file) would be placed. +This is achieved with the **Allow replacement** flag on input and/or output datasets. +This flag is supported by most file dataset types. +A replaceable dataset is created and used inside a workflow as an input for other tasks (e.g., a transformation). -The workflow is then called via an HTTP REST call (or via [cmemc](../cmemc-command-line-interface/index.md)), thus uploading the payload data "at the place" of this variable input dataset and executing all following parts of the workflow. +The workflow is then called with the actual payload via an HTTP REST call or via [`cmemc`](../cmemc-command-line-interface/index.md). -This allows for solving all kinds of [☆ Automation](../index.md) tasks when you need to process lots of small data snippets or similar. +This allows you to solve all kinds of [☆ Automation](../index.md) tasks where you need to process lots of small data snippets or similar. -!!! Abstract +!!! Tutorial Package - The complete tutorial is available as a [project file](tutorial-varinput.project.zip). You can import this project + The complete tutorial is available as a Marketplace Package. You can install this package - - by using the [web interface](../../build/introduction-to-the-user-interface/index.md) (Create → Project → Import project file) or + - by using the web interface (:eccenca-module-marketplace: **Packages** → Search → "Variable Input") or - by using the [command line interface](../cmemc-command-line-interface/index.md) ``` shell-session - cmemc -c my-cmem project import tutorial-varinput.project.zip varinput + cmemc -c my-cmem package install ecc-variable-input-tutorial ``` -## 1 Install the required vocabularies +## 1 Install the required ontologies / vocabularies -First, install all required ontologies/vocabularies which are needed for mappings later in the tab **VOCABULARIES.** +This tutorial makes use of the `rdfs:` and `schema.org` ontologies. +Both can be installed from the **Marketplace**. -In this tutorial, we need the _Schema.org_ and the _RDFS_ vocabulary. Press the (toggle switch) button on the right to install them. +Click the :eccenca-module-marketplace: **Packages** icon in the main menu under the **Marketplace** section. -![Vocabulary catalog](pdwviw-vocab-catalog.png){ class="bordered" } +![Marketplace main menu](pdwviw-marketplace-menu.png){ class="bordered" } + +Search for the required ontologies / vocabularies and click the **Install** button. +Wait for a package installation to complete (the **Install** button will change to **Uninstall**) before installing the next package. + +![Search and install ontologies](pdwviw-marketplace-schema-search.png){ class="bordered" } ## 2 Create a new project -Second, create in the tab **DATA INTEGRATION** a new project. Provide it with a _Title_ and _Description_. +Click the :eccenca-artefact-project: **Projects** icon in the main menu under the **Build** section. +Then click on **Create new** in the top right corner to create a new project. + +![Create new project](pdwviw-build-project.png){ class="bordered" } + +Click on **Project** in the **Create new item** dialog, then click **Add**. + +![Add new project](pdwviw-create-new-project.png){ class="bordered" width="70%"} + +Provide it with a _Title_ and _Description_. +In this example we will use: + +- Title: `Variant Configuration Demo Project` +- Description: `This project contains a workflow that transforms excel files with variant configuration data into a knowledge graph.` + +Then click on **Create**. + +![Create new item of type project](pdwviw-build-project-title-description.png){ class="bordered" width="70%"} The project will include everything you need to build a workflow for extracting Feed XML data, transforming it into RDF, and loading it into a Knowledge Graph. -![Create new project dialog](pdwviw-create-new-project.png){ class="bordered" } -## 3 Create an (example) feed dataset and target graph dataset +## 3 Create and populate the workflow + +Click on **Create new** in the top right corner to create a new workflow. +Click on **Workflow** under the section **ITEM TYPE**, then click on **Workflow** and **Add**. + +![Create new workflow](pdwviw-create-new-workflow.png){ class="bordered" width="70%"} + +![Add new workflow](pdwviw-new-workflow.png){ class="bordered" width="70%" } -Upload a sample XML dataset (feed data) into your project: Create → XML → Upload new file. +Provide it with a _Label_ and _Description_. +In this example we will use: -For this tutorial, you may take this file: [feed.xml](feed.xml)(1) +- Label: `process feed documents (workflow io)` +- Description: `This workflow transforms an input with the feed transformation and outputs the data into the Feed Data graph.` + +![Provide new workflow](pdwviw-label-new-workflow.png){ class="bordered" width="70%"} + +Add the XML dataset (feed data) into your project via drag-and-drop. +In this tutorial we use this file: [feed.xml](feed.xml)(1) { .annotate } 1. Original feed source was: `https://www.ecdc.europa.eu/en/taxonomy/term/2942/feed` -![Create XML dataset dialog](pdwviw-create-xml-dataset.png){ class="bordered" } +![Add new file](pdwviw-dnd-feed-xml.png){ class="bordered" } + + + +![Create XML dataset dialog](pdwviw-create-xml-dataset.png){ class="bordered" width="70%" } + +Click the output port menu of the feed.xml dataset task. +And click **Connect to newly created Transformation**. + +![Create transformation](pdwviw-create-transformation.png){ class="bordered" width="70%"} + +If necessary change the details in the create dialog, complete by clicking the **Create** button. + +Click the output port menu of the Transform feed.xml transformation task. +And click **Connect to newly created Knowledge graph**. + +![Create knowledge graph](pdwviw-create-knowledge-graph.png){ class="bordered" width="90%"} + +Customize the _Label_ and provide _Graph_ IRI. +In this example we will use: + +- Label: `Feed Data` +- Graph: `http://example.org/feeds/` + +![Create knowledge graph dialog](pdwviw-create-knowledge-graph-dialog.png){ class="bordered" width="70%"} + +**Save** the workflow. ## 4 Create the feed transformation Based on the added sample feed XML Dataset, create a mapping to generate RDF triples. + +Click **Open details page** in the Transform feed.xml transformation task context menu to open the transformation editor in a new browser tab (use **Mapping editor** to open the transformation editor in a modal dialog). + +![Edit transformation](pdwviw-edit-transformation.png){ class="bordered" width="90%"} + The screenshot provides an example mapping to generate WebPages, which includes a label, a URL, a text, and the date they were published in the feed. The mappings are based on classes and properties defined by the _Schema.org_ and _RDFS_ vocabulary. @@ -66,12 +134,11 @@ In case you need help with mapping data from XML to RDF, feel free to visit your ![Feed transformation](pdwviw-feed-transformation.png){ class="bordered" } -## 5 Create the variable input and workflow +## 5 Allow input dataset replacement -Create a new workflow in your project. -Move the **input** XML feed dataset and the **Feed Data** Graph into the workflow editor and connect them with your created **Transform feed**. +Activate the **Allow replacement** flag in a datasets` context menu by activating the **Allow replacement** option. -![Workflow with variable input](pdwviw-variable-input-workflow.png){ class="bordered" } +![Activate allow replacement](pdwviw-allow-replacement.png){ class="bordered" width="90%" } ## 6 Use `cmemc` to feed data into the workflow @@ -105,6 +172,6 @@ http://feeds.bbci.co.uk/news/rss.xml ## 7 Explore the fetched Knowledge Graph -In **EXPLORATION**, you can study the ingested feed data in your Knowledge Graph. +In **EXPLORE** > **Knowledge graphs**, you can study the ingested feed data. ![Explore the result](pdwviw-review-knowledge-graph.png){ class="bordered" } diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-allow-replacement.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-allow-replacement.png new file mode 100644 index 000000000..a14f48759 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-allow-replacement.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-build-project-title-description.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-build-project-title-description.png new file mode 100644 index 000000000..29196a1ec Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-build-project-title-description.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-build-project.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-build-project.png new file mode 100644 index 000000000..b6ac56673 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-build-project.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-knowledge-graph-dialog.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-knowledge-graph-dialog.png new file mode 100644 index 000000000..06af8d3a7 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-knowledge-graph-dialog.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-knowledge-graph.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-knowledge-graph.png new file mode 100644 index 000000000..4bbc68455 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-knowledge-graph.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-new-project.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-new-project.png index 8cd2b8907..350f0bb90 100644 Binary files a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-new-project.png and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-new-project.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-new-workflow.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-new-workflow.png new file mode 100644 index 000000000..2fa058164 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-new-workflow.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-transformation.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-transformation.png new file mode 100644 index 000000000..87ebce054 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-transformation.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-xml-dataset.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-xml-dataset.png index eb02408d8..7d8276830 100644 Binary files a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-xml-dataset.png and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-create-xml-dataset.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-dnd-feed-xml.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-dnd-feed-xml.png new file mode 100644 index 000000000..dee1f8b86 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-dnd-feed-xml.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-edit-transformation.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-edit-transformation.png new file mode 100644 index 000000000..70ffb35dc Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-edit-transformation.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-label-new-workflow.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-label-new-workflow.png new file mode 100644 index 000000000..85e5c6675 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-label-new-workflow.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-marketplace-menu.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-marketplace-menu.png new file mode 100644 index 000000000..971b82d2f Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-marketplace-menu.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-marketplace-schema-search.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-marketplace-schema-search.png new file mode 100644 index 000000000..abf70428b Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-marketplace-schema-search.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-new-workflow.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-new-workflow.png new file mode 100644 index 000000000..4aa68c0e4 Binary files /dev/null and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-new-workflow.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-review-knowledge-graph.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-review-knowledge-graph.png index 7f4347b43..829a0a0be 100644 Binary files a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-review-knowledge-graph.png and b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-review-knowledge-graph.png differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-variable-input-workflow.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-variable-input-workflow.png deleted file mode 100644 index 49206cd96..000000000 Binary files a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-variable-input-workflow.png and /dev/null differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-vocab-catalog.png b/docs/automate/processing-data-with-variable-input-workflows/pdwviw-vocab-catalog.png deleted file mode 100644 index 0b7d23a9f..000000000 Binary files a/docs/automate/processing-data-with-variable-input-workflows/pdwviw-vocab-catalog.png and /dev/null differ diff --git a/docs/automate/processing-data-with-variable-input-workflows/tutorial-varinput.project.zip b/docs/automate/processing-data-with-variable-input-workflows/tutorial-varinput.project.zip deleted file mode 100644 index f0c7e257c..000000000 Binary files a/docs/automate/processing-data-with-variable-input-workflows/tutorial-varinput.project.zip and /dev/null differ diff --git a/overrides/.icons/eccenca/module-marketplace.svg b/overrides/.icons/eccenca/module-marketplace.svg new file mode 100644 index 000000000..e8b63c545 --- /dev/null +++ b/overrides/.icons/eccenca/module-marketplace.svg @@ -0,0 +1 @@ +