Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 90 additions & 23 deletions docs/automate/processing-data-with-variable-input-workflows/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,67 +11,134 @@ tags:
## Introduction

This tutorial shows how you can create and use data integration workflows to process data coming from outside Corporate Memory (i.e., without registering datasets).
The concept to achieve this is named a **Variable Dataset**.
A variable dataset is created and used inside of a workflow as an input for other tasks (e.g. a transformation) at the place where a "regular" dataset (such as register CSV file) would be placed.
This is achieved with the **Allow replacement** flag on input and/or output datasets.
This flag is supported by most file dataset types.
A replaceable dataset is created and used inside a workflow as an input for other tasks (e.g., a transformation).

The workflow is then called via an HTTP REST call (or via [cmemc](../cmemc-command-line-interface/index.md)), thus uploading the payload data "at the place" of this variable input dataset and executing all following parts of the workflow.
The workflow is then called with the actual payload via an HTTP REST call or via [`cmemc`](../cmemc-command-line-interface/index.md).

This allows for solving all kinds of [☆ Automation](../index.md) tasks when you need to process lots of small data snippets or similar.
This allows you to solve all kinds of [☆ Automation](../index.md) tasks where you need to process lots of small data snippets or similar.

!!! Abstract
!!! Tutorial Package

The complete tutorial is available as a [project file](tutorial-varinput.project.zip). You can import this project
The complete tutorial is available as a Marketplace Package. You can install this package

- by using the [web interface](../../build/introduction-to-the-user-interface/index.md) (Create → Project → Import project file) or
- by using the web interface (:eccenca-module-marketplace: **Packages** → Search → "Variable Input") or
- by using the [command line interface](../cmemc-command-line-interface/index.md)

``` shell-session
cmemc -c my-cmem project import tutorial-varinput.project.zip varinput
cmemc -c my-cmem package install ecc-variable-input-tutorial
```

## 1 Install the required vocabularies
## 1 Install the required ontologies / vocabularies

First, install all required ontologies/vocabularies which are needed for mappings later in the tab **VOCABULARIES.**
This tutorial makes use of the `rdfs:` and `schema.org` ontologies.
Both can be installed from the **Marketplace**.

In this tutorial, we need the _Schema.org_ and the _RDFS_ vocabulary. Press the (toggle switch) button on the right to install them.
Click the :eccenca-module-marketplace: **Packages** icon in the main menu under the **Marketplace** section.

![Vocabulary catalog](pdwviw-vocab-catalog.png){ class="bordered" }
![Marketplace main menu](pdwviw-marketplace-menu.png){ class="bordered" }

Search for the required ontologies / vocabularies and click the **Install** button.
Wait for a package installation to complete (the **Install** button will change to **Uninstall**) before installing the next package.

![Search and install ontologies](pdwviw-marketplace-schema-search.png){ class="bordered" }

## 2 Create a new project

Second, create in the tab **DATA INTEGRATION** a new project. Provide it with a _Title_ and _Description_.
Click the :eccenca-artefact-project: **Projects** icon in the main menu under the **Build** section.
Then click on **Create new** in the top right corner to create a new project.

![Create new project](pdwviw-build-project.png){ class="bordered" }

Click on **Project** in the **Create new item** dialog, then click **Add**.

![Add new project](pdwviw-create-new-project.png){ class="bordered" width="70%"}

Provide it with a _Title_ and _Description_.
In this example we will use:

- Title: `Variant Configuration Demo Project`
- Description: `This project contains a workflow that transforms excel files with variant configuration data into a knowledge graph.`

Then click on **Create**.

![Create new item of type project](pdwviw-build-project-title-description.png){ class="bordered" width="70%"}

The project will include everything you need to build a workflow for extracting Feed XML data, transforming it into RDF, and loading it into a Knowledge Graph.

![Create new project dialog](pdwviw-create-new-project.png){ class="bordered" }

## 3 Create an (example) feed dataset and target graph dataset
## 3 Create and populate the workflow

Click on **Create new** in the top right corner to create a new workflow.
Click on **Workflow** under the section **ITEM TYPE**, then click on **Workflow** and **Add**.

![Create new workflow](pdwviw-create-new-workflow.png){ class="bordered" width="70%"}

![Add new workflow](pdwviw-new-workflow.png){ class="bordered" width="70%" }

Upload a sample XML dataset (feed data) into your project: Create → XML → Upload new file.
Provide it with a _Label_ and _Description_.
In this example we will use:

For this tutorial, you may take this file: [feed.xml](feed.xml)(1)
- Label: `process feed documents (workflow io)`
- Description: `This workflow transforms an input with the feed transformation and outputs the data into the Feed Data graph.`

![Provide new workflow](pdwviw-label-new-workflow.png){ class="bordered" width="70%"}

Add the XML dataset (feed data) into your project via drag-and-drop.
In this tutorial we use this file: [feed.xml](feed.xml)(1)
{ .annotate }

1. Original feed source was: `https://www.ecdc.europa.eu/en/taxonomy/term/2942/feed`

![Create XML dataset dialog](pdwviw-create-xml-dataset.png){ class="bordered" }
![Add new file](pdwviw-dnd-feed-xml.png){ class="bordered" }



![Create XML dataset dialog](pdwviw-create-xml-dataset.png){ class="bordered" width="70%" }

Click the output port menu of the feed.xml dataset task.
And click **Connect to newly created Transformation**.

![Create transformation](pdwviw-create-transformation.png){ class="bordered" width="70%"}

If necessary change the details in the create dialog, complete by clicking the **Create** button.

Click the output port menu of the Transform feed.xml transformation task.
And click **Connect to newly created Knowledge graph**.

![Create knowledge graph](pdwviw-create-knowledge-graph.png){ class="bordered" width="90%"}

Customize the _Label_ and provide _Graph_ IRI.
In this example we will use:

- Label: `Feed Data`
- Graph: `http://example.org/feeds/`

![Create knowledge graph dialog](pdwviw-create-knowledge-graph-dialog.png){ class="bordered" width="70%"}

**Save** the workflow.

## 4 Create the feed transformation

Based on the added sample feed XML Dataset, create a mapping to generate RDF triples.

Click **Open details page** in the Transform feed.xml transformation task context menu to open the transformation editor in a new browser tab (use **Mapping editor** to open the transformation editor in a modal dialog).

![Edit transformation](pdwviw-edit-transformation.png){ class="bordered" width="90%"}

The screenshot provides an example mapping to generate WebPages, which includes a label, a URL, a text, and the date they were published in the feed.
The mappings are based on classes and properties defined by the _Schema.org_ and _RDFS_ vocabulary.

In case you need help with mapping data from XML to RDF, feel free to visit your respective tutorial: [Lift data from JSON and XML sources](../../build/lift-data-from-json-and-xml-sources/index.md).

![Feed transformation](pdwviw-feed-transformation.png){ class="bordered" }

## 5 Create the variable input and workflow
## 5 Allow input dataset replacement

Create a new workflow in your project.
Move the **input** XML feed dataset and the **Feed Data** Graph into the workflow editor and connect them with your created **Transform feed**.
Activate the **Allow replacement** flag in a datasets` context menu by activating the **Allow replacement** option.

![Workflow with variable input](pdwviw-variable-input-workflow.png){ class="bordered" }
![Activate allow replacement](pdwviw-allow-replacement.png){ class="bordered" width="90%" }

## 6 Use `cmemc` to feed data into the workflow

Expand Down Expand Up @@ -105,6 +172,6 @@ http://feeds.bbci.co.uk/news/rss.xml

## 7 Explore the fetched Knowledge Graph

In **EXPLORATION**, you can study the ingested feed data in your Knowledge Graph.
In **EXPLORE** > **Knowledge graphs**, you can study the ingested feed data.

![Explore the result](pdwviw-review-knowledge-graph.png){ class="bordered" }
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions overrides/.icons/eccenca/module-marketplace.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading