diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 60ef72ae6..80e30550e 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -61,6 +61,9 @@ jobs:
- name: reconfigure private repository access
run: ACCESS_TOKEN=${{ secrets.ACCESS_TOKEN }} task use:insider-https-token
+ - name: test
+ run: task test:unit
+
- name: check
run: task check
diff --git a/.markdownlint.jsonc b/.markdownlint.jsonc
index 54fc1d920..75b7c745a 100644
--- a/.markdownlint.jsonc
+++ b/.markdownlint.jsonc
@@ -21,7 +21,10 @@
"figcaption",
"figure",
"img",
+ "key",
+ "name",
"p",
+ "param",
"span",
"summary"
]
diff --git a/Taskfile.yml b/Taskfile.yml
index 0c34d034f..a9007f198 100644
--- a/Taskfile.yml
+++ b/Taskfile.yml
@@ -48,7 +48,7 @@ tasks:
- poetry install
check:
- desc: Run complete test suite
+ desc: Check documentation links and Markdown style
deps:
- check:links
- check:rumdl
@@ -60,6 +60,25 @@ tasks:
cmds:
- poetry run linkcheckMarkdown -r docs 2>&1 | grep -v "ResourceWarning"
+ test:
+ desc: Run the Python unit test suite
+ deps:
+ - test:unit
+
+ test:unit:
+ desc: Run tests that need no external service
+ deps:
+ - install
+ cmds:
+ - poetry run pytest -m "not integration"
+
+ test:integration:
+ desc: Run tests that need a live CO instance
+ deps:
+ - install
+ cmds:
+ - poetry run pytest -m integration
+
build:
desc: Build the page
deps:
diff --git a/data/plugins.json b/data/plugins.json
index 5cb265d48..abc48d780 100644
--- a/data/plugins.json
+++ b/data/plugins.json
@@ -132,7 +132,7 @@
],
"main_category": "Uncategorized",
"description": "Clears the dataset that is connected to the output of this operator.",
- "markdownDocumentation": "Clears the dataset that is connected to the output of this operator.\n",
+ "markdownDocumentation": "Clears the dataset that is connected to the output of this operator, e.g. deletes all triples of a Knowledge Graph or removes the contents of a CSV file.\n\nThe operator itself only emits a clear instruction. The **dataset node connected to its output** performs the physical clear when that node executes. Clearing a read-only dataset fails the workflow.\n\n## Execution order\n\nA dataset node fed by this operator (a \"clear node\") clears the dataset when the node executes. Its execution order relative to other nodes writing to the same dataset is undefined unless made explicit. Without an explicit order the clear may run before or after those writes. A clear that runs after them silently removes the just-written data. The workflow reports a warning when it detects this situation.\n\nThe order is explicit if one of the following holds:\n\n- A (transitive) data-flow or dependency path connects the clear node and the writing node; the direction of the path determines which of the two runs first.\n- Clear and write happen on the same dataset node; the input port order decides: a clear input on an earlier port runs before data inputs on later ports.\n\n## Recipes\n\n### Clear, then write single dataset\n\nExample: a `Customers` dataset that is rebuilt from scratch on every run.\n\nConnect the output of this operator to the first input port of the `Customers` node and the output of the data-producing task to a later input port of the same node. The port order guarantees that the dataset is emptied before the new data is written. No dependency connections are needed.\n\n### Clear before write on separate nodes\n\nIf the clear cannot share a node with the writes (for example because different branches of the workflow write to their own `Customers` nodes), place `Customers` on the canvas one more time and connect the output of this operator to it (the clear node). Then draw a *dependency* connection from the clear node to every node that `Customers` is written to, so all writes run after the clear.\n\nA dataset can also be cleared several times in one workflow (e.g. to reuse it freshly in a later phase), but then each clear node must be ordered against all writes this way. A clear that is left unordered may run after the writes and silently remove them.\n\n### Write before clear\n\nExample: a temporary `Staging` dataset that is filled and consumed during the workflow and should be left empty at the end.\n\nDraw a dependency connection from the node that writes to `Staging` to this operator (or to its clear node), so the clear runs after the write as the final step.\n",
"pluginIcon": null,
"properties": {},
"properties_advanced": {},
@@ -615,16 +615,16 @@
"Uncategorized"
],
"main_category": "Uncategorized",
- "description": "Removes duplicated entities based on a user-defined path. Note that this operator does not retain the order of the entities. Since this operator accepts a flexible input schema, it can only be connected to operators that provide a non-flexible output schema. A typical way to achieve this is to place a transform operator before it, which produces a fixed output schema.",
- "markdownDocumentation": "Removes duplicated entities based on a user-defined path. Note that this operator does not retain the order of the entities.\n\nSince this operator accepts a flexible input schema, it can only be connected to operators that provide a non-flexible output schema.\nA typical way to achieve this is to place a transform operator before it, which produces a fixed output schema.\n",
+ "description": "Removes duplicated entities based on user-defined paths. Duplicates can be resolved by keeping the first or last entity, or by keeping the entity with the minimum or maximum value of a compare path.",
+ "markdownDocumentation": "## 1. Introduction\n\nThe **Distinct by** operator removes duplicated entities based on one or more user-defined paths.\n\nAll entities that share the same values at all **distinct paths** (one path per line) are considered duplicates of each other,\nand exactly one of them is kept according to the chosen **duplicate resolution strategy**.\n\nNote that this operator does not retain the order of the entities.\n\n## 2. Duplicate Resolution Strategies\n\n- **Keep first duplicate** \u2013 keeps the first entity encountered in the input.\n- **Keep last duplicate** \u2013 keeps the last entity encountered in the input.\n- **Keep duplicate with minimum value** \u2013 keeps the entity with the lowest value at the **compare path**.\n- **Keep duplicate with maximum value** \u2013 keeps the entity with the highest value at the **compare path**.\n\nThe first and last strategies depend on the order of the input entities.\nThe minimum and maximum strategies are independent of the input order and follow these rules:\n\n- Entities that have a value at the compare path win over entities that do not have one.\n- If an entity has multiple values at the compare path, its lowest (minimum strategy) or highest (maximum strategy) value is used for comparison.\n- On ties, the first encountered entity is kept.\n\n## 3. Compare Order\n\nThe order used to compare values for the *Keep duplicate with minimum/maximum value* strategies can be configured:\n\n- `Autodetect` (default) \u2013 if both values are numbers, numerical order is used; otherwise, alphabetical order is used.\n- `Alphabetical` \u2013 values are always compared as strings.\n- `Numerical` \u2013 values are compared as decimal numbers; values that cannot be parsed never win a comparison.\n- `Integer` \u2013 values are compared as integers; values that cannot be parsed never win a comparison.\n\n## 4. Example\n\nInput:\n\n| key | value |\n|-----|-------|\n| A | 2 |\n| A | 1 |\n| B | 2 |\n| A | 3 |\n| B | 1 |\n\nConfiguration:\n\n| Parameter | Value |\n|--------------------|-----------------------------------|\n| Distinct paths | `key` |\n| Resolve duplicates | Keep duplicate with minimum value |\n| Compare path | `value` |\n| Compare order | Autodetect (default) |\n\nOutput:\n\n| key | value |\n|-----|-------|\n| A | 1 |\n| B | 1 |\n\nFor each distinct `key`, only the entity with the lowest `value` is kept.\n\n## 5. Connecting the Operator\n\nSince this operator accepts a flexible input schema, it can only be connected to operators that provide a\nnon-flexible output schema or an explicit schema, such as CSV datasets, which can be connected directly.\nFor other inputs, a typical way to achieve this is to place a transform operator before it,\nwhich produces a fixed output schema.\n\n## 6. Technical Notes\n\n- Entities are buffered in a temporary disk-based store, so the operator also works on datasets that do not fit into memory.\n",
"pluginIcon": null,
"properties": {
"distinctPath": {
"name": "distinctPath",
- "title": "Distinct path",
- "description": "Entities that share this path will be deduplicated.",
+ "title": "Distinct paths",
+ "description": "Entities that share the values of all these paths will be deduplicated. One path per line.",
"type": "string",
- "parameterType": "string",
+ "parameterType": "multiline string",
"value": null,
"advanced": false,
"visibleInDialog": true,
@@ -640,6 +640,28 @@
"advanced": false,
"visibleInDialog": true,
"properties": {}
+ },
+ "comparePath": {
+ "name": "comparePath",
+ "title": "Compare path",
+ "description": "Path whose value decides which duplicate is kept for the 'Keep duplicate with minimum/maximum value' strategies. Ignored otherwise.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "order": {
+ "name": "order",
+ "title": "Compare order",
+ "description": "Order used to compare values for the 'Keep duplicate with minimum/maximum value' strategies. Per default, if both values are numbers, numerical order is used for comparison. Otherwise, alphabetical order is used. Ignored for other strategies.",
+ "type": "string",
+ "parameterType": "enumeration",
+ "value": "Autodetect",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
}
},
"properties_advanced": {},
@@ -1642,7 +1664,18 @@
"description": "A list of messages comprising the conversation compatible with OpenAI chat completion API message object. Have look at [Message roles and instruction following](https://platform.openai.com/docs/guides/text#message-roles-and-instruction-following) to learn about different levels of priority to messages with different roles.",
"type": "string",
"parameterType": "code-json",
- "value": "[\n {\n \"role\": \"developer\",\n \"content\": \"You are a helpful assistant.\"\n },\n {\n \"role\": \"user\",\n \"content\": \"{{ instruction_prompt }}\"\n }\n]",
+ "value": "[\n {\n \"role\": \"developer\",\n \"content\": \"{{ developer_prompt }}\"\n },\n {\n \"role\": \"user\",\n \"content\": \"{{ instruction_prompt }}\"\n }\n]",
+ "advanced": true,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "developer_prompt_template": {
+ "name": "developer_prompt_template",
+ "title": "Developer Prompt Template",
+ "description": "The developer (system) prompt inserted at `{{ developer_prompt }}` in the Messages Template. Defines the LLM's role and behaviour. Leave the Messages Template's developer content hardcoded if this parameter is not needed.",
+ "type": "string",
+ "parameterType": "code-jinja2",
+ "value": "You are a helpful assistant.",
"advanced": true,
"visibleInDialog": true,
"properties": {}
@@ -2527,6 +2560,98 @@
"pluginType": "customtask",
"relatedPlugins": []
},
+ "cmem_plugin_random-GenerateEntities": {
+ "pluginId": "cmem_plugin_random-GenerateEntities",
+ "title": "Generate random values",
+ "categories": [
+ "Uncategorized"
+ ],
+ "main_category": "Uncategorized",
+ "description": "Generates entities with random values.",
+ "markdownDocumentation": "This workflow task generates entities with random values.\n\nThe plugin generates X entities with Y values, each value has a length of Z.\n\nAll parameters can be configured with the parameters.\n\nWarning: Please note that high numbers in any of the parameters will result in more\ncomputational time as well as disk usage to save the entities.\nFor example, while a configuration of 100 entities with 100 values / 100 characters\nresults in a 1.4 MB CSV file (which is generated in milliseconds),\na configuration of 1000 entities with 1000 values / 1000 characters will result\nalready in a 1.3 GB CSV file.\n",
+ "pluginIcon": "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjwhLS0gQ3JlYXRlZCB3aXRoIElua3NjYXBlIChodHRwOi8vd3d3Lmlua3NjYXBlLm9yZy8pIC0tPgoKPHN2ZwogICB3aWR0aD0iOTAuODU4NjA0bW0iCiAgIGhlaWdodD0iOTcuMjEwOG1tIgogICB2aWV3Qm94PSIwIDAgOTAuODU4NjA1IDk3LjIxMDgwMSIKICAgdmVyc2lvbj0iMS4xIgogICBpZD0ic3ZnNSIKICAgc29kaXBvZGk6ZG9jbmFtZT0iY3VzdG9tLnN2ZyIKICAgaW5rc2NhcGU6dmVyc2lvbj0iMS4xLjIgKGI4ZTI1YmU4LCAyMDIyLTAyLTA1KSIKICAgeG1sbnM6aW5rc2NhcGU9Imh0dHA6Ly93d3cuaW5rc2NhcGUub3JnL25hbWVzcGFjZXMvaW5rc2NhcGUiCiAgIHhtbG5zOnNvZGlwb2RpPSJodHRwOi8vc29kaXBvZGkuc291cmNlZm9yZ2UubmV0L0RURC9zb2RpcG9kaS0wLmR0ZCIKICAgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIgogICB4bWxuczpzdmc9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICA8c29kaXBvZGk6bmFtZWR2aWV3CiAgICAgaWQ9Im5hbWVkdmlldzgzNiIKICAgICBwYWdlY29sb3I9IiNmZmZmZmYiCiAgICAgYm9yZGVyY29sb3I9IiM2NjY2NjYiCiAgICAgYm9yZGVyb3BhY2l0eT0iMS4wIgogICAgIGlua3NjYXBlOnBhZ2VzaGFkb3c9IjIiCiAgICAgaW5rc2NhcGU6cGFnZW9wYWNpdHk9IjAuMCIKICAgICBpbmtzY2FwZTpwYWdlY2hlY2tlcmJvYXJkPSIwIgogICAgIGlua3NjYXBlOmRvY3VtZW50LXVuaXRzPSJtbSIKICAgICBzaG93Z3JpZD0iZmFsc2UiCiAgICAgaW5rc2NhcGU6em9vbT0iMi42OTMxODUxIgogICAgIGlua3NjYXBlOmN4PSIxNTkuMTA1MjkiCiAgICAgaW5rc2NhcGU6Y3k9IjE2OC43NTkyOSIKICAgICBpbmtzY2FwZTp3aW5kb3ctd2lkdGg9IjE5MjAiCiAgICAgaW5rc2NhcGU6d2luZG93LWhlaWdodD0iMTAyNyIKICAgICBpbmtzY2FwZTp3aW5kb3cteD0iMTcyOCIKICAgICBpbmtzY2FwZTp3aW5kb3cteT0iMjUiCiAgICAgaW5rc2NhcGU6d2luZG93LW1heGltaXplZD0iMCIKICAgICBpbmtzY2FwZTpjdXJyZW50LWxheWVyPSJzdmc1IgogICAgIGZpdC1tYXJnaW4tdG9wPSIxMCIKICAgICBmaXQtbWFyZ2luLWxlZnQ9IjEwIgogICAgIGZpdC1tYXJnaW4tcmlnaHQ9IjEwIgogICAgIGZpdC1tYXJnaW4tYm90dG9tPSIxMCIgLz4KICA8ZGVmcwogICAgIGlkPSJkZWZzMiIgLz4KICA8ZwogICAgIGlkPSJsYXllcjEiCiAgICAgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTU2Ljk1NDk3MSwtMTA2LjU0MjcxKSIKICAgICBzdHlsZT0iZmlsbC1ydWxlOmV2ZW5vZGQiPgogICAgPGcKICAgICAgIGlkPSJnODM2IgogICAgICAgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTIuMTAzODk2MSw0MS4wMjU5NzQpIgogICAgICAgc3R5bGU9ImZpbGwtcnVsZTpldmVub2RkIj4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDIuNzIwNzEsMTI1LjUxMDI1IGMgMCwtMC44MDAzNiAwLjA0OTYsLTEuMTI3NzggMC4xMTAyMiwtMC43Mjc2IDAuMDYwNiwwLjQwMDE4IDAuMDYwNiwxLjA1NTAzIDAsMS40NTUyMSAtMC4wNjA2LDAuNDAwMTggLTAuMTEwMjIsMC4wNzI4IC0wLjExMDIyLC0wLjcyNzYxIHogbSAzLjQwMjA3LC0zLjQzOTU4IGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ2LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ2LDAuMDMzMSAtMC4xMzM0NiwtMC4zMzA3MyB6IG0gLTI4LjA2NjcxOCwtNi41MDQzNCBjIDAuMDEyNjksLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MiwtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTY0LC0wLjM0NzI3IHogbSAyNi4xMDY3MTgsLTIuMTM3NTUgYyAwLjE4MTksLTAuMDczNCAwLjQ3OTU2LC0wLjA3MzQgMC42NjE0NiwwIDAuMTgxOSwwLjA3MzQgMC4wMzMxLDAuMTMzNDUgLTAuMzMwNzMsMC4xMzM0NSAtMC4zNjM4MSwwIC0wLjUxMjYzLC0wLjA2MDEgLTAuMzMwNzMsLTAuMTMzNDUgeiIKICAgICAgICAgaWQ9InBhdGg4NTIiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTM5LjcwMzk4LDEzMC4xMTg0MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NiAwLjA3NjQsMC4xOTEgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjgzIC0wLjEzODk2LC0wLjM0NzI2IHogbSAtMzYuOTc3ODcsLTEuNjk3NzUgYyAwLC0wLjk0NTg4IDAuMDQ4LC0xLjMzMjg0IDAuMTA2NzEsLTAuODU5ODkgMC4wNTg3LDAuNDcyOTQgMC4wNTg3LDEuMjQ2ODQgMCwxLjcxOTc5IC0wLjA1ODcsMC40NzI5NCAtMC4xMDY3MSwwLjA4NiAtMC4xMDY3MSwtMC44NTk5IHogbSAtMzMuNjY1ODgyLDAuNjM5NDEgYyAwLjAxMjcsLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MywtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI3IHogTSAxMDYuMTIyNzgsMTIzLjEyOSBjIDAsLTAuMzYzOCAwLjA2MDEsLTAuNTEyNjMgMC4xMzM0NiwtMC4zMzA3MiAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU1IDAsMC42NjE0NSAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NiwwLjAzMzEgLTAuMTMzNDYsLTAuMzMwNzMgeiBtIC0yOC4wNjY3MTgsLTguMzU2NDIgYyAwLjAxMjY5LC0wLjMwODIyIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODUgMC4wNzY0MywwLjE5MDk5IDAuMDY3MDMsMC40MTkyIC0wLjAyMDg5LDAuNTA3MTIgLTAuMDg3OTIsMC4wODc5IC0wLjE1MDQ1MywtMC4wNjgzIC0wLjEzODk2NCwtMC4zNDcyNyB6IG0gLTguOTk1ODM0LC0xMi40MzU0MSBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTM5LC0wLjM3MDkyIDAuMTU5ODUzLC0wLjE1OTg2IDAuMDc2NDMsMC4xOTEgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI2IHogbSAyNS45MDgyNzksLTkuMjM4Mzc0IGMgMC4zNDE3NzYsLTAuMzYzODAyIDAuNjgwOTQsLTAuNjYxNDU4IDAuNzUzNywtMC42NjE0NTggMC4wNzI3NiwwIC0wLjE0NzM0MSwwLjI5NzY1NiAtMC40ODkxMTcsMC42NjE0NTggLTAuMzQxNzc1LDAuMzYzODAyIC0wLjY4MDkzOSwwLjY2MTQ1OSAtMC43NTM3LDAuNjYxNDU5IC0wLjA3Mjc2LDAgMC4xNDczNDEsLTAuMjk3NjU3IDAuNDg5MTE3LC0wLjY2MTQ1OSB6IgogICAgICAgICBpZD0icGF0aDg1MCIgLz4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDIuNjYyMzEsMTQ4LjkwMzgzIGMgMC4wMTI3LC0wLjMwODIyIDAuMDc1NCwtMC4zNzA5MiAwLjE1OTg1LC0wLjE1OTg1IDAuMDc2NCwwLjE5MDk5IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTIgLTAuMDg3OSwwLjA4NzkgLTAuMTUwNDYsLTAuMDY4NCAtMC4xMzg5NywtMC4zNDcyNyB6IG0gMy40Mzk1OSwwIGMgMC4wMTI3LC0wLjMwODIyIDAuMDc1NCwtMC4zNzA5MiAwLjE1OTg1LC0wLjE1OTg1IDAuMDc2NCwwLjE5MDk5IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTIgLTAuMDg3OSwwLjA4NzkgLTAuMTUwNDUsLTAuMDY4NCAtMC4xMzg5NiwtMC4zNDcyNyB6IG0gLTMuMzcxNTIsLTE3LjA0MzU4IGMgMCwtMS4wOTE0IDAuMDQ2NywtMS41Mzc4OSAwLjEwMzc4LC0wLjk5MjE4IDAuMDU3MSwwLjU0NTcgMC4wNTcxLDEuNDM4NjcgMCwxLjk4NDM3IC0wLjA1NzEsMC41NDU3MSAtMC4xMDM3OCwwLjA5OTIgLTAuMTAzNzgsLTAuOTkyMTkgeiBtIC0zMy42NDkyNjMsLTMuNzA0MTYgYyAwLC0wLjM2MzggMC4wNjAwNSwtMC41MTI2MyAwLjEzMzQ1MiwtMC4zMzA3MyAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU2IDAsMC42NjE0NiAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NTIsMC4wMzMxIC0wLjEzMzQ1MiwtMC4zMzA3MyB6IG0gMzcuMDQxNjYzLC0zLjk2ODc1IGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ2LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ2LDAuMDMzMSAtMC4xMzM0NiwtMC4zMzA3MyB6IE0gNzguMDU2MDYyLDExMy45Nzg4MyBjIDAuMDEyNjksLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MiwtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODMgLTAuMTM4OTY0LC0wLjM0NzI3IHogbSAtOC45OTU4MzQsLTEwLjg0NzkxIGMgMC4wMTI3LC0wLjMwODIzIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTMsLTAuMTU5ODYgMC4wNzY0MywwLjE5MSAwLjA2NzAzLDAuNDE5MiAtMC4wMjA4OSwwLjUwNzEyIC0wLjA4NzkyLDAuMDg3OSAtMC4xNTA0NTMsLTAuMDY4MyAtMC4xMzg5NjMsLTAuMzQ3MjYgeiBtIDcwLjY0Mzc1MiwtMC4yNjQ1OSBjIDAuMDEyNywtMC4zMDgyMiAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NSAwLjA3NjQsMC4xOTA5OSAwLjA2NywwLjQxOTIgLTAuMDIwOSwwLjUwNzEyIC0wLjA4NzksMC4wODc5IC0wLjE1MDQ1LC0wLjA2ODMgLTAuMTM4OTYsLTAuMzQ3MjcgeiIKICAgICAgICAgaWQ9InBhdGg4NDgiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTAyLjY2MjMxLDE0OC4xMTAwOCBjIDAuMDEyNywtMC4zMDgyMiAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NSAwLjA3NjQsMC4xOTA5OSAwLjA2NywwLjQxOTIgLTAuMDIwOSwwLjUwNzEyIC0wLjA4NzksMC4wODc5IC0wLjE1MDQ2LC0wLjA2ODMgLTAuMTM4OTcsLTAuMzQ3MjcgeiBtIDAuMDY1OSwtMTIuNjc3OTUgYyAxMGUtNCwtMS4wMTg2NSAwLjA0OSwtMS40MDI5NCAwLjEwNjI4LC0wLjg1NCAwLjA1NzMsMC41NDg5NSAwLjA1NjMsMS4zODIzOSAtMC4wMDIsMS44NTIwOSAtMC4wNTg1LDAuNDY5NyAtMC4xMDUzMSwwLjAyMDYgLTAuMTA0MTQsLTAuOTk4MDkgeiBtIDM2Ljk5NjYzLC02Ljc0Njg4IGMgMCwtMC4zNjM4IDAuMDYsLTAuNTEyNjMgMC4xMzM0NSwtMC4zMzA3MiAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU1IDAsMC42NjE0NSAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NSwwLjAzMzEgLTAuMTMzNDUsLTAuMzMwNzMgeiBtIC03MC42NDM3NTMsLTEuNTg3NSBjIDAsLTAuMzYzOCAwLjA2MDA1LC0wLjUxMjYzIDAuMTMzNDUyLC0wLjMzMDcyIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTUgMCwwLjY2MTQ1IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ1MiwwLjAzMzEgLTAuMTMzNDUyLC0wLjMzMDczIHogbSAzNy4wNTE0NzMsLTEuNzE5NzkgYyAwLjAwNSwtMC40MzY1NiAwLjA2NDcsLTAuNTgzMTIgMC4xMzE3NywtMC4zMjU2OSAwLjA2NzEsMC4yNTc0MyAwLjA2MjcsMC42MTQ2MiAtMC4wMSwwLjc5Mzc1IC0wLjA3MjUsMC4xNzkxMyAtMC4xMjczNiwtMC4wMzE1IC0wLjEyMTk3LC0wLjQ2ODA2IHogbSAyNC44ODA2MywtOC44NjM1NCBjIDAsLTAuNTA5MzIgMC4wNTQ1LC0wLjcxNzY4IDAuMTIxMDEsLTAuNDYzMDIgMC4wNjY1LDAuMjU0NjYgMC4wNjY1LDAuNjcxMzggMCwwLjkyNjA0IC0wLjA2NjYsMC4yNTQ2NiAtMC4xMjEwMSwwLjA0NjMgLTAuMTIxMDEsLTAuNDYzMDIgeiBtIC01Mi45NTcxNTgsLTMuMzI5MzQgYyAwLjAxMjY5LC0wLjMwODIyIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODUgMC4wNzY0MywwLjE5MDk5IDAuMDY3MDMsMC40MTkyIC0wLjAyMDg5LDAuNTA3MTIgLTAuMDg3OTIsMC4wODc5IC0wLjE1MDQ1MywtMC4wNjgzIC0wLjEzODk2NCwtMC4zNDcyNyB6IG0gLTguOTY1MTQxLC04Ljk3Mzc4IGMgMC4wMDU0LC0wLjQzNjU3IDAuMDY0NjksLTAuNTgzMTMgMC4xMzE3NzMsLTAuMzI1NyAwLjA2NzA4LDAuMjU3NDMgMC4wNjI2NywwLjYxNDYyIC0wLjAwOTgsMC43OTM3NSAtMC4wNzI0OCwwLjE3OTEzIC0wLjEyNzM2MywtMC4wMzE1IC0wLjEyMTk3LC0wLjQ2ODA1IHogbSA3MC42MTMwNTksLTAuNTUxMjIgYyAwLjAxMjcsLTAuMzA4MjIgMC4wNzU0LC0wLjM3MDkyIDAuMTU5ODUsLTAuMTU5ODUgMC4wNzY0LDAuMTkwOTkgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjg0IC0wLjEzODk2LC0wLjM0NzI3IHoiCiAgICAgICAgIGlkPSJwYXRoODQ2IiAvPgogICAgICA8cGF0aAogICAgICAgICBzdHlsZT0iZmlsbC1vcGFjaXR5OjE7ZmlsbC1ydWxlOmV2ZW5vZGQ7c3Ryb2tlLXdpZHRoOjAuMjY0NTgzIgogICAgICAgICBkPSJtIDEwNi4xMDE5LDE0Ny41ODA5MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NiAwLjA3NjQsMC4xOTEgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjg0IC0wLjEzODk2LC0wLjM0NzI2IHogbSAtMy40MTg3LC0wLjM3NDgzIGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ1LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ1LDAuMDMzMSAtMC4xMzM0NSwtMC4zMzA3MyB6IG0gMC4wNDI5LC04LjQ2NjY3IGMgMCwtMC45NDU4OCAwLjA0OCwtMS4zMzI4NCAwLjEwNjcxLC0wLjg1OTg5IDAuMDU4NywwLjQ3Mjk0IDAuMDU4NywxLjI0Njg0IDAsMS43MTk3OSAtMC4wNTg3LDAuNDcyOTQgLTAuMTA2NzEsMC4wODYgLTAuMTA2NzEsLTAuODU5OSB6IG0gMzcuMDA4NTYsLTExLjI0NDc5IGMgMC4wMDUsLTAuNDM2NTYgMC4wNjQ3LC0wLjU4MzEzIDAuMTMxNzcsLTAuMzI1NjkgMC4wNjcxLDAuMjU3NDMgMC4wNjI3LDAuNjE0NjEgLTAuMDEsMC43OTM3NSAtMC4wNzI1LDAuMTc5MTMgLTAuMTI3MzYsLTAuMDMxNSAtMC4xMjE5NywtMC40NjgwNiB6IG0gLTMzLjYwMjA4LC0wLjc5Mzc1IGMgMC4wMDUsLTAuNDM2NTYgMC4wNjQ3LC0wLjU4MzEzIDAuMTMxNzcsLTAuMzI1NjkgMC4wNjcxLDAuMjU3NDMgMC4wNjI3LDAuNjE0NjEgLTAuMDEsMC43OTM3NSAtMC4wNzI1LDAuMTc5MTMgLTAuMTI3MzYsLTAuMDMxNSAtMC4xMjE5NywtMC40NjgwNiB6IG0gMTMuNTc0NDYsLTQuODk0NzkgYyAwLjI2MzM5LC0wLjI5MTA0IDAuNTM4NDIsLTAuNTI5MTcgMC42MTExOCwtMC41MjkxNyAwLjA3MjgsMCAtMC4wODMyLDAuMjM4MTMgLTAuMzQ2NiwwLjUyOTE3IC0wLjI2MzM5LDAuMjkxMDQgLTAuNTM4NDIsMC41MjkxNiAtMC42MTExOCwwLjUyOTE2IC0wLjA3MjgsMCAwLjA4MzIsLTAuMjM4MTIgMC4zNDY2LC0wLjUyOTE2IHogbSAtNTAuNTk1NjQ3LC0xNS44NzUgYyAwLC0wLjY1NDg1IDAuMDUxNiwtMC45MjI3NCAwLjExNDY2MywtMC41OTUzMSAwLjA2MzA2LDAuMzI3NDIgMC4wNjMwNiwwLjg2MzIgMCwxLjE5MDYyIC0wLjA2MzA3LDAuMzI3NDIgLTAuMTE0NjYzLDAuMDU5NSAtMC4xMTQ2NjMsLTAuNTk1MzEgeiBtIDcwLjYxMzQ2NywtMS4zMjI5MiBjIDAsLTAuMzYzOCAwLjA2LC0wLjUxMjYzIDAuMTMzNDUsLTAuMzMwNzMgMC4wNzM0LDAuMTgxOSAwLjA3MzQsMC40Nzk1NiAwLDAuNjYxNDYgLTAuMDczNCwwLjE4MTkgLTAuMTMzNDUsMC4wMzMxIC0wLjEzMzQ1LC0wLjMzMDczIHogbSAtMzYuMjY4OCwtMTMuMTE4OTIyIGMgMC4wMTI3LC0wLjMwODIyNSAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NTMgMC4wNzY0LDAuMTkwOTk2IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTE4IC0wLjA4NzksMC4wODc5MiAtMC4xNTA0NiwtMC4wNjgzNSAtMC4xMzg5NywtMC4zNDcyNjUgeiIKICAgICAgICAgaWQ9InBhdGg4NDQiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTA2LjEyMjc4LDE0Ni42NzY5MiBjIDAsLTAuMzYzOCAwLjA2MDEsLTAuNTEyNjMgMC4xMzM0NiwtMC4zMzA3MyAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU2IDAsMC42NjE0NiAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NiwwLjAzMzEgLTAuMTMzNDYsLTAuMzMwNzMgeiBtIC0zLjM3OTQsLTMuMTc1IGMgMCwtMS44MTkwMSAwLjA0MjMsLTIuNTYzMTUgMC4wOTM5LC0xLjY1MzY0IDAuMDUxNywwLjkwOTUgMC4wNTE3LDIuMzk3NzggMCwzLjMwNzI5IC0wLjA1MTcsMC45MDk1IC0wLjA5MzksMC4xNjUzNiAtMC4wOTM5LC0xLjY1MzY1IHogbSAzLjM5OTAxLC0xNS4zNDU4MyBjIDAsLTAuNTA5MzIgMC4wNTQ0LC0wLjcxNzY4IDAuMTIxLC0wLjQ2MzAyIDAuMDY2NSwwLjI1NDY2IDAuMDY2NSwwLjY3MTM4IDAsMC45MjYwNCAtMC4wNjY2LDAuMjU0NjYgLTAuMTIxLDAuMDQ2MyAtMC4xMjEsLTAuNDYzMDIgeiBtIDMzLjYxNjM3LC0yLjUxMzU0IGMgMC4wMDIsLTAuNzI3NjEgMC4wNTM0LC0wLjk5Mzg2IDAuMTE0MTksLTAuNTkxNjggMC4wNjA4LDAuNDAyMTggMC4wNTkyLDAuOTk3NSAtMC4wMDQsMS4zMjI5MiAtMC4wNjI4LDAuMzI1NDIgLTAuMTEyNTYsLTAuMDA0IC0wLjExMDU3LC0wLjczMTI0IHogbSAtNzAuNjI2OTcxLC00Ljg5NDggYyAwLC0xLjIzNjkyIDAuMDQ1NTgsLTEuNzQyOTQgMC4xMDEyODEsLTEuMTI0NDcgMC4wNTU3LDAuNjE4NDYgMC4wNTU3LDEuNjMwNDkgMCwyLjI0ODk1IC0wLjA1NTcsMC42MTg0NyAtMC4xMDEyODEsMC4xMTI0NSAtMC4xMDEyODEsLTEuMTI0NDggeiBtIDAsLTcuOTM3NSBjIDAsLTEuMjM2OTIgMC4wNDU1OCwtMS43NDI5NCAwLjEwMTI4MSwtMS4xMjQ0NyAwLjA1NTcsMC42MTg0NiAwLjA1NTcsMS42MzA0OSAwLDIuMjQ4OTUgLTAuMDU1NywwLjYxODQ3IC0wLjEwMTI4MSwwLjExMjQ1IC0wLjEwMTI4MSwtMS4xMjQ0OCB6IG0gOC45MjQyNzMsLTAuOTQ4MDggYyAwLjAxMjY5LC0wLjMwODIzIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODYgMC4wNzY0MywwLjE5MSAwLjA2NzAzLDAuNDE5MiAtMC4wMjA4OSwwLjUwNzEyIC0wLjA4NzkyLDAuMDg3OSAtMC4xNTA0NTMsLTAuMDY4MyAtMC4xMzg5NjQsLTAuMzQ3MjYgeiBtIC04Ljk5NTgzNCwtNC40OTc5MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTM5LC0wLjM3MDkyIDAuMTU5ODUzLC0wLjE1OTg1IDAuMDc2NDMsMC4xOTA5OSAwLjA2NzAzLDAuNDE5MTkgLTAuMDIwODksMC41MDcxMSAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI2IHogbSA3MC42OTg1MzIsLTEuMDM2MjkgYyAwLjAwMiwtMC43Mjc2IDAuMDUzNCwtMC45OTM4NSAwLjExNDE5LC0wLjU5MTY3IDAuMDYwOCwwLjQwMjE4IDAuMDU5MiwwLjk5NzQ5IC0wLjAwNCwxLjMyMjkyIC0wLjA2MjgsMC4zMjU0MiAtMC4xMTI1NiwtMC4wMDQgLTAuMTEwNTcsLTAuNzMxMjUgeiIKICAgICAgICAgaWQ9InBhdGg4NDIiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTA2LjE0NzczLDE0NS4yMjE3MSBjIDAuMDAzLC0wLjU4MjA4IDAuMDU3MSwtMC43ODkwNSAwLjEyMDQxLC0wLjQ1OTkzIDAuMDYzMywwLjMyOTEyIDAuMDYwOSwwLjgwNTM3IC0wLjAwNSwxLjA1ODMzIC0wLjA2NjIsMC4yNTI5NyAtMC4xMTgsLTAuMDE2MyAtMC4xMTUwNywtMC41OTg0IHogTSA4MS4yNzE1NTgsMTM1LjAzNTI1IGMgMCwtMC41MDkzMiAwLjA1NDQ1LC0wLjcxNzY4IDAuMTIxLC0wLjQ2MzAyIDAuMDY2NTUsMC4yNTQ2NyAwLjA2NjU1LDAuNjcxMzggMCwwLjkyNjA1IC0wLjA2NjU1LDAuMjU0NjYgLTAuMTIxLDAuMDQ2MyAtMC4xMjEsLTAuNDYzMDMgeiBtIDI0Ljg5MTQzMiwtNC42MzAyIGMgMC4wMDIsLTAuODczMTMgMC4wNTA5LC0xLjE5ODQyIDAuMTA5NzcsLTAuNzIyODcgMC4wNTg5LDAuNDc1NTUgMC4wNTc3LDEuMTg5OTIgLTAuMDAzLDEuNTg3NSAtMC4wNjA0LDAuMzk3NTggLTAuMTA4NTUsMC4wMDggLTAuMTA3MDcsLTAuODY0NjMgeiBtIDMzLjYzODMzLC0xNC40MTk4IGMgMCwtNC43Mjk0MiAwLjAzNTksLTYuNjY0MTkgMC4wNzk3LC00LjI5OTQ3IDAuMDQzOSwyLjM2NDcxIDAuMDQzOSw2LjIzNDI0IDAsOC41OTg5NSAtMC4wNDM4LDIuMzY0NzIgLTAuMDc5NywwLjQyOTk1IC0wLjA3OTcsLTQuMjk5NDggeiBNIDEwMS4yNjg5LDg5Ljg3NTM0NyBjIDAuMTkwOTksLTAuMDc2NDMgMC40MTkyLC0wLjA2NzAzIDAuNTA3MTIsMC4wMjA4OSAwLjA4NzksMC4wODc5MiAtMC4wNjg0LDAuMTUwNDUzIC0wLjM0NzI3LDAuMTM4OTY1IC0wLjMwODIzLC0wLjAxMjY5IC0wLjM3MDkyLC0wLjA3NTM5IC0wLjE1OTg1LC0wLjE1OTg1MyB6IgogICAgICAgICBpZD0icGF0aDg0MCIgLz4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDAuNjMwODQsMTUyLjU4MDg1IGMgLTAuNjkzNzk3LC0wLjEzNjcxIC0yNy4wNTY5NTMsLTE1LjM0MTU0IC0yOC42NzY3NTUsLTE2LjUzOTIgLTAuNjUyNDY0LC0wLjQ4MjQyIC0xLjQ4NTkwMSwtMS40MTIyMyAtMS44NTIwODMsLTIuMDY2MjQgbCAtMC42NjU3ODcsLTEuMTg5MTEgLTAuMDc0MTEsLTE1Ljk3NTQzIGMgLTAuMDY1MzIsLTE0LjA4MDczIC0wLjAyNTM3LC0xNi4wNDk4MiAwLjMzNjgyOSwtMTYuNjAyNjEgMC44ODUxNzMsLTEuMzUwOTQyIDAuOTA1MjEsLTEuMzQxMjkgMTkuNTc2NjY0LDkuNDI5OTQgMTAuOTY0MDQyLDYuMzI0OTYgMTIuNTMxNjAyLDcuNDU0MjQgMTMuMDc3NzgyLDkuNDIxMzIgMC4xODkwMiwwLjY4MDc2IDAuMjg1NTksNi4zNTQzMSAwLjI4NjU3LDE2LjgzNTYzIDAuMDAxLDE1LjQ1NzA5IC0wLjAxMDMsMTUuODIwNjMgLTAuNTI3NjksMTYuMzM4MDIgLTAuMjkxMDQsMC4yOTEwNCAtMC42MTg0NywwLjUxMjk1IC0wLjcyNzYxLDAuNDkzMTQgLTAuMTA5MTQsLTAuMDE5OCAtMC40NDgzNSwtMC4wODUzIC0wLjc1MzgxLC0wLjE0NTQ2IHogbSAtMTMuOTYxNjY5LC0xOS4wMDQ5IGMgLTAuMTI4MjAxLC0wLjMzNDA5IC00Ljc1NzYxMiwtMy4wMzg2MSAtNS4yMDEyODEsLTMuMDM4NjEgLTAuMTc4NjYyLDAgLTAuMjQ5MjAxLDEuMDAwMSAtMC4yMDA1MjIsMi44NDI5NyBsIDAuMDc1MSwyLjg0Mjk4IDIuNjQ1ODM0LDEuNTI0MjYgMi42NDU4MzMsMS41MjQyNiAwLjA3NTk0LC0yLjY5NTcgYyAwLjA0MTc2LC0xLjQ4MjY0IDAuMDIzMzYsLTIuODMyNzEgLTAuMDQwOSwtMy4wMDAxNiB6IG0gMC41MDU1NTgsLTMuMzY0OTggYyAwLjIwNTUzMywtMC42OTQ1OSAwLjYxODg3MSwtMS4wNTYxIDIuMTk3MzQsLTEuOTIxNzggMi40ODkyMTEsLTEuMzY1MTcgMi43ODQ4MTEsLTEuODk2ODIgMi42MzU1MDIsLTQuNzQwMDYgLTAuMDk0ODEsLTEuODA1NTMgLTAuMjU0Njg4LC0yLjQwNzUgLTEuMDIzMTE1LC0zLjg1MjI4IC0xLjExNjg5MSwtMi4wOTk5NyAtMy4xMTc5MjEsLTQuMTQzMzEgLTUuNzIyNzcxLC01Ljg0Mzc4IC0yLjA2MjQ0NywtMS4zNDYzOCAtNi4wMzgzNzMsLTIuODM4MjcgLTYuNzkyNjY1LC0yLjU0ODgyIC0wLjM1Nzc1OSwwLjEzNzI4IC0wLjQzMzg0NiwwLjU4NTU2IC0wLjQzMzg0NiwyLjU1NjA0IDAsMS4zMTQyNSAwLjA4OTMsMi4zOTE5MSAwLjE5ODQzNywyLjM5NDc5IDAuMTA5MTQxLDAuMDAzIDEuMDA2Mjc2LDAuMTIyMDUgMS45OTM2MzUsMC4yNjQ4MiAzLjQ4MzI1MSwwLjUwMzY2IDYuMDEwMDExLDIuNDY1NzggNi4wMTAwMTEsNC42NjcgMCwxLjI0Mzk1IC0wLjMyNjUzLDEuNTcwMTggLTIuODM4MTg4LDIuODM1NTkgLTEuNTkyOTcxLDAuODAyNTYgLTIuMjU5MywxLjgyNzkxIC0yLjEzNzc3MywzLjI4OTYxIDAuMDc3OSwwLjkzNjk1IDAuMTgyNDM5LDEuMDMyMjEgMi41OTQ3MTEsMi4zNjQ0NCAxLjM4MjQ0OCwwLjc2MzQ5IDIuNjM3NzQ4LDEuMzg4NTcgMi43ODk1NTUsMS4zODkwNyAwLjE1MTgxLDUuMmUtNCAwLjM4OTkzNSwtMC4zODQwOSAwLjUyOTE2NywtMC44NTQ2NCB6IG0gMTkuNzQ2MDYxLDIyLjA2ODIzIGMgLTAuNDM4MjUsLTAuNDM4MjUgLTAuNTExMjksLTEuMTUwODEgLTAuNjc3NTgsLTYuNjExMDYgLTAuMTAyNTEsLTMuMzY1OTIgLTAuMTA0ODYsLTEwLjYxNzg3IC0wLjAwNSwtMTYuMTE1NDQgMC4yMzg1NCwtMTMuMTU5OTYgLTAuNDA4MDIsLTExLjg1MTYgOC4zNTkzNSwtMTYuOTE1NzYgMi44MTAzNCwtMS42MjMyOSA4LjQ0MzQ2LC00Ljg4MDQgMTIuNTE4MDQsLTcuMjM4MDIgMTAuNzI0MTcsLTYuMjA1MTkgMTEuMTY1NjksLTYuMzk5MzQxIDEyLjEyOTgyLC01LjMzMzk4IDAuMzk3MjIsMC40Mzg5MiAwLjQzNzg5LDEuOTU4MTUgMC40Mzc4OSwxNi4zNTY3MyB2IDE1Ljg3Mjg2IGwgLTAuNzI3NjEsMS40MzY1MSBjIC0wLjQ2ODE4LDAuOTI0MzQgLTEuMTU5NDYsMS43NTQ0OSAtMS45Mzg4OCwyLjMyODM4IC0yLjI4MjMzLDEuNjgwNTEgLTI3LjgxNjgxLDE2LjMxMjU0IC0yOC44ODEwNSwxNi41NDk3MiAtMC41MTExNSwwLjExMzkyIC0wLjg2Nzc3LDAuMDE3MSAtMS4yMTQ3NywtMC4zMjk5NCB6IG0gMTguMTQ0MDcsLTE0LjAwMDIxIDAuODU5OSwtMC40MzEzMyB2IC0zLjAxODk2IC0zLjAxODk2IGwgLTEuMzg5MDcsMC43ODU4OSBjIC0wLjc2Mzk4LDAuNDMyMjMgLTEuOTg0MzcsMS4xMzYyOCAtMi43MTE5NywxLjU2NDU0IGwgLTEuMzIyOTIsMC43Nzg2NSAtMC4wNzQ5LDIuOTcyNDcgLTAuMDc0OSwyLjk3MjQ3IDEuOTI3MDIsLTEuMDg2NzIgYyAxLjA1OTg2LC0wLjU5NzcgMi4zMTM5OCwtMS4yODA4MiAyLjc4NjkyLC0xLjUxODA1IHogbSAtMS44MDU3MSwtNy4xMDc5MiBjIDIuNTI0NjMsLTEuNDY2MTUgMi41NTcxLC0xLjQ5Nzg2IDIuODY2NTIsLTIuODAwNDYgMC4xNzIxMywtMC43MjQ2MyAxLjE0NTQsLTIuNzA2NDUgMi4xNjI4MiwtNC40MDQwNCAyLjAzMzAzLC0zLjM5MjE3IDIuNTg2NTEsLTQuNzIxMDEgMi44MzAyOSwtNi43OTUyIDAuMzA3NDgsLTIuNjE2MTggLTAuODM0MzMsLTQuMzYxMTIgLTIuODUzNzEsLTQuMzYxMTIgLTIuMTE4MjMsMCAtNi4wNTcwNiwyLjE4NjEyIC05LjU1MDIxLDUuMzAwNTEgbCAtMS41MjEzNSwxLjM1NjQgdiAyLjU0MTEzIDIuNTQxMTEgbCAyLjQ3NjI1LC0yLjQyODM3IGMgMy4yNjYyLC0zLjIwMzA1IDUuNDEzMzYsLTQuMTg0MjUgNS44MTI5MSwtMi42NTYzNSAwLjIyOTA4LDAuODc2MDEgLTAuNDQyNjQsMi42MDQxIC0xLjkzODU5LDQuOTg3MjQgLTAuNzc2NDUsMS4yMzY5MyAtMS43NTQxNywyLjk4MDA0IC0yLjE3MjcxLDMuODczNTggLTAuODE4MTcsMS43NDY2OCAtMS4zNzExNSw0LjMyODUgLTAuOTI3MDksNC4zMjg1IDAuMTQzNzMsMCAxLjQxMDQyLC0wLjY2NzMyIDIuODE0ODcsLTEuNDgyOTMgeiBNIDEwMS40NTA4LDExMi41ODM5NSBDIDk5Ljc5OTk0OSwxMTEuNzk4NzYgNzUuNzExOTkyLDk3LjkzMDI1MiA3My4xMjc0MDcsOTYuMjc2OTEzIDcxLjczMDIxNiw5NS4zODMxNCA3MS42NTgyNTIsOTUuMjgzODYzIDcxLjczNTc0Niw5NC4zNTcwNzMgNzEuODM1ODM1LDkzLjE2MDA1MiA3MC45OTE5NDUsOTMuNzAwODg1IDg5LjAxMjEwOSw4My4yODQ5OCBsIDEzLjIyNTg5MSwtNy42NDQ3NSAxLjg2MTEyLC0wLjA5Mzk4IGMgMS43NzUxLC0wLjA4OTY0IDEuOTY0NzcsLTAuMDM5MjggNC4xMDQzMSwxLjA4OTczOSA0LjQ5ODA3LDIuMzczNTk3IDI3LjY2MzYsMTUuODgxMTIzIDI4LjM3MDgxLDE2LjU0MjY1OSAxLjI2NzM3LDEuMTg1NTE1IDAuODkxNDcsMi4wNTgwODIgLTEuNTM0MTksMy41NjEyOTEgLTIuNDUxODEsMS41MTk0MTQgLTI0Ljc2NzA3LDE0LjM5Mjc3MSAtMjYuNzgyNywxNS40NTA1NjEgLTIuNzgxMTIsMS40NTk1MSAtNC4zNzE4NCwxLjU1MTQ2IC02LjgwNjU1LDAuMzkzNDUgeiBtIDE0LjM3ODEyLC0xMi4zMDU1MiAzLjAwMTA0LC0xLjY4MjI2NCAtMC44ODQzNywtMC41MjcyMDkgYyAtMi4xNTgwNiwtMS4yODY0OTkgLTMuODg0MDMsLTIuMTkyMDM2IC00LjE3ODA2LC0yLjE5MjAzNiAtMC40MTU0NSwwIC02LjA5OTk2LDMuMTY3Mjk2IC02LjA5NTk2LDMuMzk2NTQ0IDAuMDA0LDAuMjQzMiA0LjI3MTIxLDIuNjI4OTQ1IDQuNzU5NDQsMi42NjEwOTUgMC4yMTgyOCwwLjAxNDQgMS43NDczNCwtMC43MzA4OCAzLjM5NzkxLC0xLjY1NjEzIHogbSAtOC42MTQ2NCwtMi45NzYxNjQgYyAwLjc1OTE1LC0wLjM4MDIyNSAyLjA2NjI1LC0xLjEwNDA1NiAyLjkwNDY4LC0xLjYwODUxIGwgMS41MjQ0MSwtMC45MTcxOTIgLTAuNTQ4MzgsLTAuNzcwMTI1IGMgLTAuNDk2NDIsLTAuNjk3MTUgLTAuNTMxNjEsLTEuMDAyMjc5IC0wLjM3MTM4LC0zLjIyMDEzNCAwLjE2MzU4LC0yLjI2NDM2OSAwLjEzMzI3LC0yLjUwNTE1NCAtMC40MDAwNywtMy4xNzc4IC0wLjMxNzM5LC0wLjQwMDI4NSAtMS4xMzM5MSwtMS4wNTQxOTkgLTEuODE0NDksLTEuNDUzMTQxIC0xLjA5NzE1LC0wLjY0MzEyOCAtMS41NTc0NSwtMC43MzYzMTkgLTQuMDYwNzIsLTAuODIyMTE0IC0yLjM4NjM4LC0wLjA4MTc5IC0zLjEwMjkyLC0wLjAwNDggLTQuNjMwMjA0LDAuNDk3Njg4IC0yLjIyNDgzOSwwLjczMTk0NyAtNC45MDQ3NDUsMi4wMzAxMzEgLTYuODMyODA2LDMuMzA5OTA1IC0xLjUzMDY0NiwxLjAxNTk4NCAtNC4xMDI1NjMsMy4yMjI3OCAtNC4xMDE3NTYsMy41MTk0NTEgNS4yOWUtNCwwLjE5NDIwNiAxLjk3MDE0MywxLjQ3NjgxNyAzLjM0MTQ0MSwyLjE3NTk0NyBsIDAuODkxODM3LDAuNDU0Njg0IDEuODkzOTksLTEuOTUzMjgyIGMgMi4zMTcwNywtMi4zODk2MDMgMy45NjYyNDIsLTMuMjg0OTUyIDYuMDI4ODk4LC0zLjI3MzE0IDIuMzU2OCwwLjAxMzUgMi40OTY0LDAuMjMxNDkyIDIuMjcwNTIsMy41NDU0NjggbCAtMC4xODk5MywyLjc4NjU1MiAwLjc5ODUzLDAuNzk4NTMxIGMgMC45Nzc1NSwwLjk3NzU0OCAxLjUyMjM2LDAuOTk1MjczIDMuMjk1NDMsMC4xMDcyMTIgeiIKICAgICAgICAgaWQ9InBhdGg4MzgiIC8+CiAgICA8L2c+CiAgPC9nPgo8L3N2Zz4K",
+ "properties": {
+ "number_of_entities": {
+ "name": "number_of_entities",
+ "title": "Number of Entities (Rows)",
+ "description": "How many rows will be created per run. Depending on your output dataset, this will result in different number of resources (Knowledge Graph),rows (CSV) or objects (JSON).",
+ "type": "string",
+ "parameterType": "Long",
+ "value": "10",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "number_of_values": {
+ "name": "number_of_values",
+ "title": "Number of Values (Columns)",
+ "description": "How many values are created per entity / row. Depending on your output dataset, this will result in different number of datatype properties (Knowledge Graph), columns (CSV) or attributes (JSON).",
+ "type": "string",
+ "parameterType": "Long",
+ "value": "5",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "string_length": {
+ "name": "string_length",
+ "title": "String Length",
+ "description": "How long (in characters) should each value be.",
+ "type": "string",
+ "parameterType": "Long",
+ "value": "16",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {
+ "random_function": {
+ "name": "random_function",
+ "title": "Random Function",
+ "description": "",
+ "type": "string",
+ "parameterType": "string",
+ "value": "token_urlsafe",
+ "advanced": true,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "property_namespace": {
+ "name": "property_namespace",
+ "title": "Property Namespace",
+ "description": "Output properties will have this namespace (following a number).",
+ "type": "string",
+ "parameterType": "string",
+ "value": "https://example.org/vocab/RandomValuePath",
+ "advanced": true,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "type_id": {
+ "name": "type_id",
+ "title": "Type Identifier",
+ "description": "Output entities will have this type identifier (IRI).",
+ "type": "string",
+ "parameterType": "string",
+ "value": "https://example.org/vocab/RandomValueRow",
+ "advanced": true,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "python",
+ "is_deprecated": false,
+ "tags": [
+ "WorkflowTask",
+ "PythonPlugin"
+ ],
+ "pluginType": "customtask",
+ "relatedPlugins": []
+ },
"cmem_plugin_shapes-plugin_shapes-ShapesPlugin": {
"pluginId": "cmem_plugin_shapes-plugin_shapes-ShapesPlugin",
"title": "Generate SHACL shapes from data",
@@ -3059,6 +3184,79 @@
"pluginType": "customtask",
"relatedPlugins": []
},
+ "jsonToFile": {
+ "pluginId": "jsonToFile",
+ "title": "JSON to File",
+ "categories": [
+ "Uncategorized"
+ ],
+ "main_category": "Uncategorized",
+ "description": "Writes a JSON string held in a field on each valid incoming entity to a file. Depending on the output mode, it produces one file per entity, packs all entities into a single ZIP archive, or merges them into a single JSON array file. Produces a file entity downstream, suitable for wiring into a file-backed dataset or any operator that consumes file entities.",
+ "markdownDocumentation": "## JSON to File\n\nThe JSON to File operator takes a JSON string held in a field on each incoming entity and writes it to a file. The\nresulting file is surfaced downstream as a file entity, so any operator that accepts file entities \u2014 a file-backed\ndataset, another file-processing operator \u2014 can pick it up.\n\nThe operator does not parse the JSON into structured entities. It validates that the value is well-formed JSON and then\nwrites the JSON value to the file. The content type of the produced file is set via the *MIME type* parameter and\ndefaults to *application/json*.\n\n## Input\n\nJSON to File accepts exactly one input. It iterates over every entity in that input, reads the JSON string from a\nfield on each entity, and validates it. What it then produces depends on the *Output mode*: in *file* mode (the\ndefault) one file per entity; in *zip* mode a single ZIP archive with one entry per entity; in *jsonArray* mode a\nsingle file holding all the JSON values merged into one JSON array. The output is surfaced as a stream of file\nentities for the downstream operator.\n\nWhich field holds the JSON string is controlled by the *Input path* parameter. When set, the operator reads the value\nat the given path expression. When left empty, it reads the value of the first property in the entity schema.\n\n## Invalid input\n\nValidation is per entity. An entity whose value is missing, empty, or not valid JSON is skipped and recorded as a\nwarning on the execution report, naming the entity and the reason; it produces no output. The remaining valid entities\nare written as usual, so a single malformed record no longer discards the whole batch. This applies in all three output\nmodes.\n\nWhen every entity is skipped, the operator still produces the mode's natural empty output: no files in *file* mode, an\nempty JSON array `[]` in *jsonArray* mode, and a ZIP archive with no entries in *zip* mode.\n\nConfiguration errors are not per-entity and still fail the task \u2014 for example, an input count other than one, or an\nunsupported output mode.\n\n## Output\n\nThe output of JSON to File is a stream of file entities. In *file* mode each file entity wraps a file holding the\nJSON value from one input entity. In *zip* mode the stream contains a single file entity whose backing file is a ZIP\narchive with one entry per input entity. In *jsonArray* mode the stream contains a single file entity backed by one\nfile holding a JSON array of all the input values. In *file* and *jsonArray* mode the MIME type is the value of the\n*MIME type* parameter; in *zip* mode a default *application/json* is overridden to *application/zip* (see *MIME type*\nbelow). Downstream operators or datasets that accept file entities consume the stream directly.\n\nWhen the output is wired to a file-backed dataset, the dataset writes the file's bytes into its own resource. The end\nresult is a file on disk \u2014 a JSON file per entity in *file* mode, a ZIP archive in *zip* mode, or a single JSON array\nfile in *jsonArray* mode.\n\n## Parameters\n\n**Input path** controls which field of the input entity holds the JSON string. When set to a Silk path expression\nsuch as */jsonContent*, the operator reads the value at that path. When left empty, the operator reads the value of\nthe first property in the entity schema.\n\n**MIME type** sets the content type of every produced file. Defaults to *application/json*. In *zip* mode, when this\nparameter is left at its default value, the executor overrides it to *application/zip* automatically; an explicit\nvalue is used as-is even in *zip* mode. In *file* and *jsonArray* mode the default *application/json* is correct and\nis not overridden.\n\n**Output property** wraps the JSON value in a JSON object under the given property key before writing. When set\nto *payload*, an input value of `{\"name\":\"Alice\"}` is written as `{\"payload\":{\"name\":\"Alice\"}}`. When left empty\n(default), the value is written as-is. The wrapping applies in all three output modes; in *jsonArray* mode each\nelement of the array is the wrapped form.\n\n**Output mode** selects what the operator produces. *file* (the default) writes one file per input entity. *zip* packs\nall input entities into a single ZIP file \u2014 one ZIP entry per entity, producing a single file entity whose backing\nfile is a ZIP archive. Entries are always named *entry-0.json*, *entry-1.json*, and so on, by position among valid\nentities. *jsonArray* merges all input entities into a single file\nholding one JSON array whose elements are the JSON values from each entity, in input order; there is always exactly\none output file.\n\n## Output mode examples\n\nIn *zip* mode: an upstream operator produces two entities, each with a JSON string in the *jsonContent* field. With\n*Input path* set to */jsonContent* and *Output mode* set to *zip*, JSON to File produces a single file entity backed\nby a ZIP archive containing two entries: *entry-0.json* and *entry-1.json*. The archive is written with a content type\nof *application/zip*. Wiring the output into a file-backed dataset writes the ZIP file to that dataset's resource.\n\nIn *jsonArray* mode: with two entities holding `{\"id\":1}` and `{\"id\":2}` and *Output mode* set to *jsonArray*, JSON to\nFile produces a single file containing the JSON array `[{\"id\":1},{\"id\":2}]`, with a content type of *application/json*.\n\n## Example\n\nAn upstream operator produces a single entity with the following JSON string in its *jsonContent* field. With\n*Input path* set to */jsonContent*, JSON to File reads from that field.\n\n```json\n{\n \"response\": {\n \"persons\": [\n { \"id\": \"1\", \"name\": \"Alice\" },\n { \"id\": \"2\", \"name\": \"Bob\" }\n ]\n }\n}\n```\n\nJSON to File validates the string and writes it to a file with a content type of *application/json*. The produced file\nentity can be wired into a downstream JSON dataset to persist the value as a file on disk, or fed into any other\noperator that accepts file entities.\n\nWith the `outputProperty` parameter set to `payload`, the same input is instead written as:\n\n```json\n{\n \"payload\": {\n \"response\": {\n \"persons\": [\n { \"id\": \"1\", \"name\": \"Alice\" },\n { \"id\": \"2\", \"name\": \"Bob\" }\n ]\n }\n }\n}\n```\n",
+ "pluginIcon": null,
+ "properties": {
+ "inputPath": {
+ "name": "inputPath",
+ "title": "Input path",
+ "description": "The Silk path expression of the input entity that contains the JSON string. If not set, the value of the first property in the entity schema will be taken.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "mimeType": {
+ "name": "mimeType",
+ "title": "Mime type",
+ "description": "MIME type of the produced file.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "application/json",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "outputMode": {
+ "name": "outputMode",
+ "title": "Output mode",
+ "description": "Output mode: \"One file per entity\" writes one file per entity, \"ZIP archive\" packs all entities into a single ZIP archive, \"Merged JSON array\" merges all entities into a single JSON array file.",
+ "type": "string",
+ "parameterType": "enumeration",
+ "value": "file",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "outputProperty": {
+ "name": "outputProperty",
+ "title": "Output property",
+ "description": "If set, the JSON value is wrapped in a JSON object under this property key before writing. For example, with outputProperty set to 'payload', the input {\"name\":\"Alice\"} is written as {\"payload\":{\"name\":\"Alice\"}}. When empty (default), the value is written as-is.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "WorkflowTask"
+ ],
+ "pluginType": "customtask",
+ "relatedPlugins": [
+ {
+ "id": "JsonParserOperator",
+ "description": "JSON to File writes the JSON string from each input entity to a file; Parse JSON parses the same kind of input into structured entities driven by a downstream schema."
+ }
+ ]
+ },
"cmem_plugin_kafka-ReceiveMessages": {
"pluginId": "cmem_plugin_kafka-ReceiveMessages",
"title": "Kafka Consumer (Receive Messages)",
@@ -3067,7 +3265,7 @@
],
"main_category": "Uncategorized",
"description": "Reads messages from a Kafka topic and saves it to a messages dataset (Consumer).",
- "markdownDocumentation": "\nThis workflow operator uses the Kafka Consumer API\nto receive messages from an [Apache Kafka](https://kafka.apache.org/) topic.\n\nMessages received from the topic will be generated as entities with the following\nflat schema:\n\n- **key** - the optional key of the message,\n- **content** - the message itself as plain text (use other operators, such as\n [Parse JSON](https://documentation.eccenca.com/latest/deploy-and-configure/configuration/dataintegration/plugin-reference/#parse-json) or [Parse XML](https://documentation.eccenca.com/latest/deploy-and-configure/configuration/dataintegration/plugin-reference/#parse-xml) to process\n complex message content),\n- **offset** - the given offset of the message in the topic,\n- **ts-production** - the timestamp when the message was written to the topic,\n- **ts-consumption** - the timestamp when the message was consumed from the topic.\n\nIn order to process the resulting entities, they have to run through a transformation.\n\nAs an alternate working mode, messages can be exported directly to a JSON or XML\ndataset if you know that the messages on your topic are valid JSON or XML documents\n(see Advanced Options > Messages Dataset).\n\nIn this case, a sample response from the consumer will appear as follows:\n\n\n Sample JSON Response\n\n```json\n[\n {\n \"message\": {\n \"key\": \"818432-942813-832642-453478\",\n \"headers\": {\n \"type\": \"ADD\"\n },\n \"content\": {\n \"location\": [\"Leipzig\"],\n \"obstacle\": {\n \"name\": \"Iron Bars\",\n \"order\": \"1\"\n }\n }\n }\n },\n {\n \"message\": {\n \"key\": \"887428-119918-570674-866526\",\n \"headers\": {\n \"type\": \"REMOVE\"\n },\n \"content\": {\n \"comments\": \"We can pass any json payload here.\"\n }\n }\n },\n {\n \"message\": {\n \"key\": \"TestKey\",\n \"tombstone\": true,\n \"headers\": {\n \"h1\": \"v1\",\n \"h2\": \"v2\"\n },\n \"content\": {\n \"will_be_ignored\": \"...\"\n }\n }\n }\n]\n```\n\n\n\n Sample XML Response\n\n```xml\n \n \n \n \n \n string\n \n \n \n \n \n \n string\n \n \n \n will be ignored\n \n```\n\n\n",
+ "markdownDocumentation": "\nThis workflow operator uses the Kafka Consumer API\nto receive messages from an [Apache Kafka](https://kafka.apache.org/) topic.\n\nMessages received from the topic will be generated as entities with the following\nflat schema:\n\n- **key** - the optional key of the message,\n- **content** - the message itself as plain text (use other operators, such as\n [Parse JSON](JsonParserOperator.md) or [Parse XML](XmlParserOperator.md) to process\n complex message content),\n- **offset** - the given offset of the message in the topic,\n- **ts-production** - the timestamp when the message was written to the topic,\n- **ts-consumption** - the timestamp when the message was consumed from the topic.\n\nIn order to process the resulting entities, they have to run through a transformation.\n\nAs an alternate working mode, messages can be exported directly to a JSON or XML\ndataset if you know that the messages on your topic are valid JSON or XML documents\n(see Advanced Options > Messages Dataset).\n\nIn this case, a sample response from the consumer will appear as follows:\n\n\n Sample JSON Response\n\n```json\n[\n {\n \"message\": {\n \"key\": \"818432-942813-832642-453478\",\n \"headers\": {\n \"type\": \"ADD\"\n },\n \"content\": {\n \"location\": [\"Leipzig\"],\n \"obstacle\": {\n \"name\": \"Iron Bars\",\n \"order\": \"1\"\n }\n }\n }\n },\n {\n \"message\": {\n \"key\": \"887428-119918-570674-866526\",\n \"headers\": {\n \"type\": \"REMOVE\"\n },\n \"content\": {\n \"comments\": \"We can pass any json payload here.\"\n }\n }\n },\n {\n \"message\": {\n \"key\": \"TestKey\",\n \"tombstone\": true,\n \"headers\": {\n \"h1\": \"v1\",\n \"h2\": \"v2\"\n },\n \"content\": {\n \"will_be_ignored\": \"...\"\n }\n }\n }\n]\n```\n\n\n\n Sample XML Response\n\n```xml\n \n \n \n \n \n string\n \n \n \n \n \n \n string\n \n \n \n will be ignored\n \n```\n\n\n",
"pluginIcon": null,
"properties": {
"message_dataset": {
@@ -4217,8 +4415,8 @@
"Uncategorized"
],
"main_category": "Uncategorized",
- "description": "Parses an incoming entity as a JSON dataset. Typically, it is used before a transformation task. Takes exactly one input of which only the first entity is processed.",
- "markdownDocumentation": "Parses an incoming entity as a JSON dataset. Typically, it is used before a transformation task. Takes exactly one input of which only the first entity is processed.\n",
+ "description": "Parses a JSON string held in a field on each incoming entity. Typically used before a transformation task. Takes exactly one input.",
+ "markdownDocumentation": "## Parse JSON\n\nParse JSON is a workflow operator that extracts structured data from a JSON string held in a field on incoming\nentities. It sits inside a pipeline between an upstream source and a downstream operator \u2014 typically a transformation\n\u2014 and turns the JSON content into entities ready for further processing.\n\nThe operator is useful whenever JSON arrives not as a file but as a string stored in a field: the result of an HTTP\nrequest, a column in a database record, a payload embedded in another dataset. Parse JSON consumes that string in place\nand produces entities for the rest of the pipeline.\n\n## Input\n\nParse JSON accepts exactly one input. It iterates over every entity in that input, extracts the JSON string from a\nfield on each entity, parses it, and produces output entities from its contents. The output entities from all input\nentities are concatenated into a single stream for the downstream operator.\n\nWhich field is used as the JSON source is controlled by the *Input path* parameter. When set, Parse JSON looks for the\nJSON string at the given path expression. When left empty, it reads the value of the first available field. If no value\nis found at the expected location, or if the field is empty, the operator raises an error and stops.\n\n## Output\n\nThe output of Parse JSON is a set of entities extracted from the parsed JSON structure. These entities are shaped by\nthree parameters: *Base path*, *URI suffix pattern*, and *Navigate into arrays*.\n\n**Base path** determines the starting point within the JSON document. When set to a path such as */Persons/Person*, only\nelements found at that location are read as entities; everything else in the document is ignored. When left empty, all\ndirect children of the root element are read.\n\n**URI suffix pattern** controls how the URIs of the output entities are constructed. The pattern is evaluated relative\nto the URI of the input entity: whatever suffix is specified gets appended to that URI. For example, a pattern of\n*/{id}* applied to an input entity with URI *http://example.org/record/42* produces URIs by appending the value of the\n*id* field \u2014 so an entity whose *id* is *7* receives URI *http://example.org/record/42/7*. When left empty, URIs are\ngenerated automatically.\n\n**Navigate into arrays** controls how JSON arrays are handled during path traversal. In JSON, an array is an anonymous\ncontainer with no name of its own \u2014 just a list of items. When a path expression crosses an array mid-way, it is\nambiguous whether the array itself or its contents is the intended target. This parameter resolves that ambiguity. When\nenabled \u2014 the default \u2014 the operator descends into arrays automatically, so a path like */Persons/Person* reaches the\nPerson elements directly even if Persons is an array. When disabled, the array is treated as an explicit step in the\npath: to reach the same Person elements, the path must be written as */Persons/#array/Person*.\n\nParse JSON supports the same path expressions as the JSON dataset, including wildcards for children and descendants,\nbackward paths, and special paths for hash IDs, key names, and array elements.\n\n## Schema\n\nBefore producing output entities, Parse JSON needs to know which fields to extract. The set of fields \u2014 the output\nschema \u2014 is requested by the downstream operator and reaches Parse JSON before any parsing happens. For each\nrequested field, Parse JSON evaluates its path expression against the parsed JSON, starting from the configured base\npath, and writes the resulting values onto the output entity. When the downstream operator requests a multi-entity\nschema, Parse JSON produces the root entities and the nested sub-entity tables in a single pass. In practice that\noperator is a transformation.\n\nParse JSON cannot be connected directly to a dataset. A dataset declares no fields to read, so Parse JSON has nothing\nto extract. Workflows that wire Parse JSON straight into a dataset fail at execution time with an error naming the\nmissing schema and asking for a downstream operator that declares one.\n\n## Example\n\nAn upstream operator produces a single entity with the following JSON string in its first field. Because *Input path*\nis not set, Parse JSON reads from that first field by default.\n\n```json\n{\n \"response\": {\n \"persons\": [\n { \"id\": \"1\", \"name\": \"Alice\", \"city\": \"Berlin\" },\n { \"id\": \"2\", \"name\": \"Bob\", \"city\": \"London\" }\n ]\n }\n}\n```\n\nWith *Base path* set to */response/persons*, Parse JSON navigates past the response wrapper and reads each element of\nthe persons array as a separate entity. The array is crossed automatically because *Navigate into arrays* is enabled.\nWith *URI suffix pattern* set to */{id}*, the two output entities receive URIs constructed by appending the value of\ntheir id field to the URI of the input entity.\n\nThe result is two entities \u2014 one for Alice, one for Bob \u2014 each with id, name, and city as fields.\n\nIf the upstream operator produces several entities, Parse JSON parses the JSON string in each one in turn and\nconcatenates the resulting entities into a single output stream.\n",
"pluginIcon": null,
"properties": {
"inputPath": {
@@ -4276,7 +4474,16 @@
"WorkflowTask"
],
"pluginType": "customtask",
- "relatedPlugins": []
+ "relatedPlugins": [
+ {
+ "id": "json",
+ "description": "Parse JSON and the JSON dataset share path syntax but not a content source: Parse JSON reads from a field value in an incoming entity, the JSON dataset from a file resource it opens directly."
+ },
+ {
+ "id": "jsonToFile",
+ "description": "Parse JSON parses a JSON string on each input entity into structured entities driven by a downstream schema; JSON to File writes the same kind of input to a file for downstream operators that read files."
+ }
+ ]
},
"XmlParserOperator": {
"pluginId": "XmlParserOperator",
@@ -5251,6 +5458,42 @@
"pluginType": "customtask",
"relatedPlugins": []
},
+ "cmem_plugin_random-SelectEntities": {
+ "pluginId": "cmem_plugin_random-SelectEntities",
+ "title": "Select random entities",
+ "categories": [
+ "Uncategorized"
+ ],
+ "main_category": "Uncategorized",
+ "description": "Select X random entities from an input dataset.",
+ "markdownDocumentation": "This workflow task selects X random entities from an input dataset\nusing the standard pseudo-random generator (reservoir sampling).\n\nThe task supports only flat entities. Hierarchical entities are ignored.\n",
+ "pluginIcon": "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjwhLS0gQ3JlYXRlZCB3aXRoIElua3NjYXBlIChodHRwOi8vd3d3Lmlua3NjYXBlLm9yZy8pIC0tPgoKPHN2ZwogICB3aWR0aD0iOTAuODU4NjA0bW0iCiAgIGhlaWdodD0iOTcuMjEwOG1tIgogICB2aWV3Qm94PSIwIDAgOTAuODU4NjA1IDk3LjIxMDgwMSIKICAgdmVyc2lvbj0iMS4xIgogICBpZD0ic3ZnNSIKICAgc29kaXBvZGk6ZG9jbmFtZT0iY3VzdG9tLnN2ZyIKICAgaW5rc2NhcGU6dmVyc2lvbj0iMS4xLjIgKGI4ZTI1YmU4LCAyMDIyLTAyLTA1KSIKICAgeG1sbnM6aW5rc2NhcGU9Imh0dHA6Ly93d3cuaW5rc2NhcGUub3JnL25hbWVzcGFjZXMvaW5rc2NhcGUiCiAgIHhtbG5zOnNvZGlwb2RpPSJodHRwOi8vc29kaXBvZGkuc291cmNlZm9yZ2UubmV0L0RURC9zb2RpcG9kaS0wLmR0ZCIKICAgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIgogICB4bWxuczpzdmc9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICA8c29kaXBvZGk6bmFtZWR2aWV3CiAgICAgaWQ9Im5hbWVkdmlldzgzNiIKICAgICBwYWdlY29sb3I9IiNmZmZmZmYiCiAgICAgYm9yZGVyY29sb3I9IiM2NjY2NjYiCiAgICAgYm9yZGVyb3BhY2l0eT0iMS4wIgogICAgIGlua3NjYXBlOnBhZ2VzaGFkb3c9IjIiCiAgICAgaW5rc2NhcGU6cGFnZW9wYWNpdHk9IjAuMCIKICAgICBpbmtzY2FwZTpwYWdlY2hlY2tlcmJvYXJkPSIwIgogICAgIGlua3NjYXBlOmRvY3VtZW50LXVuaXRzPSJtbSIKICAgICBzaG93Z3JpZD0iZmFsc2UiCiAgICAgaW5rc2NhcGU6em9vbT0iMi42OTMxODUxIgogICAgIGlua3NjYXBlOmN4PSIxNTkuMTA1MjkiCiAgICAgaW5rc2NhcGU6Y3k9IjE2OC43NTkyOSIKICAgICBpbmtzY2FwZTp3aW5kb3ctd2lkdGg9IjE5MjAiCiAgICAgaW5rc2NhcGU6d2luZG93LWhlaWdodD0iMTAyNyIKICAgICBpbmtzY2FwZTp3aW5kb3cteD0iMTcyOCIKICAgICBpbmtzY2FwZTp3aW5kb3cteT0iMjUiCiAgICAgaW5rc2NhcGU6d2luZG93LW1heGltaXplZD0iMCIKICAgICBpbmtzY2FwZTpjdXJyZW50LWxheWVyPSJzdmc1IgogICAgIGZpdC1tYXJnaW4tdG9wPSIxMCIKICAgICBmaXQtbWFyZ2luLWxlZnQ9IjEwIgogICAgIGZpdC1tYXJnaW4tcmlnaHQ9IjEwIgogICAgIGZpdC1tYXJnaW4tYm90dG9tPSIxMCIgLz4KICA8ZGVmcwogICAgIGlkPSJkZWZzMiIgLz4KICA8ZwogICAgIGlkPSJsYXllcjEiCiAgICAgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTU2Ljk1NDk3MSwtMTA2LjU0MjcxKSIKICAgICBzdHlsZT0iZmlsbC1ydWxlOmV2ZW5vZGQiPgogICAgPGcKICAgICAgIGlkPSJnODM2IgogICAgICAgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTIuMTAzODk2MSw0MS4wMjU5NzQpIgogICAgICAgc3R5bGU9ImZpbGwtcnVsZTpldmVub2RkIj4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDIuNzIwNzEsMTI1LjUxMDI1IGMgMCwtMC44MDAzNiAwLjA0OTYsLTEuMTI3NzggMC4xMTAyMiwtMC43Mjc2IDAuMDYwNiwwLjQwMDE4IDAuMDYwNiwxLjA1NTAzIDAsMS40NTUyMSAtMC4wNjA2LDAuNDAwMTggLTAuMTEwMjIsMC4wNzI4IC0wLjExMDIyLC0wLjcyNzYxIHogbSAzLjQwMjA3LC0zLjQzOTU4IGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ2LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ2LDAuMDMzMSAtMC4xMzM0NiwtMC4zMzA3MyB6IG0gLTI4LjA2NjcxOCwtNi41MDQzNCBjIDAuMDEyNjksLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MiwtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTY0LC0wLjM0NzI3IHogbSAyNi4xMDY3MTgsLTIuMTM3NTUgYyAwLjE4MTksLTAuMDczNCAwLjQ3OTU2LC0wLjA3MzQgMC42NjE0NiwwIDAuMTgxOSwwLjA3MzQgMC4wMzMxLDAuMTMzNDUgLTAuMzMwNzMsMC4xMzM0NSAtMC4zNjM4MSwwIC0wLjUxMjYzLC0wLjA2MDEgLTAuMzMwNzMsLTAuMTMzNDUgeiIKICAgICAgICAgaWQ9InBhdGg4NTIiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTM5LjcwMzk4LDEzMC4xMTg0MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NiAwLjA3NjQsMC4xOTEgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjgzIC0wLjEzODk2LC0wLjM0NzI2IHogbSAtMzYuOTc3ODcsLTEuNjk3NzUgYyAwLC0wLjk0NTg4IDAuMDQ4LC0xLjMzMjg0IDAuMTA2NzEsLTAuODU5ODkgMC4wNTg3LDAuNDcyOTQgMC4wNTg3LDEuMjQ2ODQgMCwxLjcxOTc5IC0wLjA1ODcsMC40NzI5NCAtMC4xMDY3MSwwLjA4NiAtMC4xMDY3MSwtMC44NTk5IHogbSAtMzMuNjY1ODgyLDAuNjM5NDEgYyAwLjAxMjcsLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MywtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI3IHogTSAxMDYuMTIyNzgsMTIzLjEyOSBjIDAsLTAuMzYzOCAwLjA2MDEsLTAuNTEyNjMgMC4xMzM0NiwtMC4zMzA3MiAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU1IDAsMC42NjE0NSAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NiwwLjAzMzEgLTAuMTMzNDYsLTAuMzMwNzMgeiBtIC0yOC4wNjY3MTgsLTguMzU2NDIgYyAwLjAxMjY5LC0wLjMwODIyIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODUgMC4wNzY0MywwLjE5MDk5IDAuMDY3MDMsMC40MTkyIC0wLjAyMDg5LDAuNTA3MTIgLTAuMDg3OTIsMC4wODc5IC0wLjE1MDQ1MywtMC4wNjgzIC0wLjEzODk2NCwtMC4zNDcyNyB6IG0gLTguOTk1ODM0LC0xMi40MzU0MSBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTM5LC0wLjM3MDkyIDAuMTU5ODUzLC0wLjE1OTg2IDAuMDc2NDMsMC4xOTEgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI2IHogbSAyNS45MDgyNzksLTkuMjM4Mzc0IGMgMC4zNDE3NzYsLTAuMzYzODAyIDAuNjgwOTQsLTAuNjYxNDU4IDAuNzUzNywtMC42NjE0NTggMC4wNzI3NiwwIC0wLjE0NzM0MSwwLjI5NzY1NiAtMC40ODkxMTcsMC42NjE0NTggLTAuMzQxNzc1LDAuMzYzODAyIC0wLjY4MDkzOSwwLjY2MTQ1OSAtMC43NTM3LDAuNjYxNDU5IC0wLjA3Mjc2LDAgMC4xNDczNDEsLTAuMjk3NjU3IDAuNDg5MTE3LC0wLjY2MTQ1OSB6IgogICAgICAgICBpZD0icGF0aDg1MCIgLz4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDIuNjYyMzEsMTQ4LjkwMzgzIGMgMC4wMTI3LC0wLjMwODIyIDAuMDc1NCwtMC4zNzA5MiAwLjE1OTg1LC0wLjE1OTg1IDAuMDc2NCwwLjE5MDk5IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTIgLTAuMDg3OSwwLjA4NzkgLTAuMTUwNDYsLTAuMDY4NCAtMC4xMzg5NywtMC4zNDcyNyB6IG0gMy40Mzk1OSwwIGMgMC4wMTI3LC0wLjMwODIyIDAuMDc1NCwtMC4zNzA5MiAwLjE1OTg1LC0wLjE1OTg1IDAuMDc2NCwwLjE5MDk5IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTIgLTAuMDg3OSwwLjA4NzkgLTAuMTUwNDUsLTAuMDY4NCAtMC4xMzg5NiwtMC4zNDcyNyB6IG0gLTMuMzcxNTIsLTE3LjA0MzU4IGMgMCwtMS4wOTE0IDAuMDQ2NywtMS41Mzc4OSAwLjEwMzc4LC0wLjk5MjE4IDAuMDU3MSwwLjU0NTcgMC4wNTcxLDEuNDM4NjcgMCwxLjk4NDM3IC0wLjA1NzEsMC41NDU3MSAtMC4xMDM3OCwwLjA5OTIgLTAuMTAzNzgsLTAuOTkyMTkgeiBtIC0zMy42NDkyNjMsLTMuNzA0MTYgYyAwLC0wLjM2MzggMC4wNjAwNSwtMC41MTI2MyAwLjEzMzQ1MiwtMC4zMzA3MyAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU2IDAsMC42NjE0NiAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NTIsMC4wMzMxIC0wLjEzMzQ1MiwtMC4zMzA3MyB6IG0gMzcuMDQxNjYzLC0zLjk2ODc1IGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ2LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ2LDAuMDMzMSAtMC4xMzM0NiwtMC4zMzA3MyB6IE0gNzguMDU2MDYyLDExMy45Nzg4MyBjIDAuMDEyNjksLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MiwtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODMgLTAuMTM4OTY0LC0wLjM0NzI3IHogbSAtOC45OTU4MzQsLTEwLjg0NzkxIGMgMC4wMTI3LC0wLjMwODIzIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTMsLTAuMTU5ODYgMC4wNzY0MywwLjE5MSAwLjA2NzAzLDAuNDE5MiAtMC4wMjA4OSwwLjUwNzEyIC0wLjA4NzkyLDAuMDg3OSAtMC4xNTA0NTMsLTAuMDY4MyAtMC4xMzg5NjMsLTAuMzQ3MjYgeiBtIDcwLjY0Mzc1MiwtMC4yNjQ1OSBjIDAuMDEyNywtMC4zMDgyMiAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NSAwLjA3NjQsMC4xOTA5OSAwLjA2NywwLjQxOTIgLTAuMDIwOSwwLjUwNzEyIC0wLjA4NzksMC4wODc5IC0wLjE1MDQ1LC0wLjA2ODMgLTAuMTM4OTYsLTAuMzQ3MjcgeiIKICAgICAgICAgaWQ9InBhdGg4NDgiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTAyLjY2MjMxLDE0OC4xMTAwOCBjIDAuMDEyNywtMC4zMDgyMiAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NSAwLjA3NjQsMC4xOTA5OSAwLjA2NywwLjQxOTIgLTAuMDIwOSwwLjUwNzEyIC0wLjA4NzksMC4wODc5IC0wLjE1MDQ2LC0wLjA2ODMgLTAuMTM4OTcsLTAuMzQ3MjcgeiBtIDAuMDY1OSwtMTIuNjc3OTUgYyAxMGUtNCwtMS4wMTg2NSAwLjA0OSwtMS40MDI5NCAwLjEwNjI4LC0wLjg1NCAwLjA1NzMsMC41NDg5NSAwLjA1NjMsMS4zODIzOSAtMC4wMDIsMS44NTIwOSAtMC4wNTg1LDAuNDY5NyAtMC4xMDUzMSwwLjAyMDYgLTAuMTA0MTQsLTAuOTk4MDkgeiBtIDM2Ljk5NjYzLC02Ljc0Njg4IGMgMCwtMC4zNjM4IDAuMDYsLTAuNTEyNjMgMC4xMzM0NSwtMC4zMzA3MiAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU1IDAsMC42NjE0NSAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NSwwLjAzMzEgLTAuMTMzNDUsLTAuMzMwNzMgeiBtIC03MC42NDM3NTMsLTEuNTg3NSBjIDAsLTAuMzYzOCAwLjA2MDA1LC0wLjUxMjYzIDAuMTMzNDUyLC0wLjMzMDcyIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTUgMCwwLjY2MTQ1IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ1MiwwLjAzMzEgLTAuMTMzNDUyLC0wLjMzMDczIHogbSAzNy4wNTE0NzMsLTEuNzE5NzkgYyAwLjAwNSwtMC40MzY1NiAwLjA2NDcsLTAuNTgzMTIgMC4xMzE3NywtMC4zMjU2OSAwLjA2NzEsMC4yNTc0MyAwLjA2MjcsMC42MTQ2MiAtMC4wMSwwLjc5Mzc1IC0wLjA3MjUsMC4xNzkxMyAtMC4xMjczNiwtMC4wMzE1IC0wLjEyMTk3LC0wLjQ2ODA2IHogbSAyNC44ODA2MywtOC44NjM1NCBjIDAsLTAuNTA5MzIgMC4wNTQ1LC0wLjcxNzY4IDAuMTIxMDEsLTAuNDYzMDIgMC4wNjY1LDAuMjU0NjYgMC4wNjY1LDAuNjcxMzggMCwwLjkyNjA0IC0wLjA2NjYsMC4yNTQ2NiAtMC4xMjEwMSwwLjA0NjMgLTAuMTIxMDEsLTAuNDYzMDIgeiBtIC01Mi45NTcxNTgsLTMuMzI5MzQgYyAwLjAxMjY5LC0wLjMwODIyIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODUgMC4wNzY0MywwLjE5MDk5IDAuMDY3MDMsMC40MTkyIC0wLjAyMDg5LDAuNTA3MTIgLTAuMDg3OTIsMC4wODc5IC0wLjE1MDQ1MywtMC4wNjgzIC0wLjEzODk2NCwtMC4zNDcyNyB6IG0gLTguOTY1MTQxLC04Ljk3Mzc4IGMgMC4wMDU0LC0wLjQzNjU3IDAuMDY0NjksLTAuNTgzMTMgMC4xMzE3NzMsLTAuMzI1NyAwLjA2NzA4LDAuMjU3NDMgMC4wNjI2NywwLjYxNDYyIC0wLjAwOTgsMC43OTM3NSAtMC4wNzI0OCwwLjE3OTEzIC0wLjEyNzM2MywtMC4wMzE1IC0wLjEyMTk3LC0wLjQ2ODA1IHogbSA3MC42MTMwNTksLTAuNTUxMjIgYyAwLjAxMjcsLTAuMzA4MjIgMC4wNzU0LC0wLjM3MDkyIDAuMTU5ODUsLTAuMTU5ODUgMC4wNzY0LDAuMTkwOTkgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjg0IC0wLjEzODk2LC0wLjM0NzI3IHoiCiAgICAgICAgIGlkPSJwYXRoODQ2IiAvPgogICAgICA8cGF0aAogICAgICAgICBzdHlsZT0iZmlsbC1vcGFjaXR5OjE7ZmlsbC1ydWxlOmV2ZW5vZGQ7c3Ryb2tlLXdpZHRoOjAuMjY0NTgzIgogICAgICAgICBkPSJtIDEwNi4xMDE5LDE0Ny41ODA5MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NiAwLjA3NjQsMC4xOTEgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjg0IC0wLjEzODk2LC0wLjM0NzI2IHogbSAtMy40MTg3LC0wLjM3NDgzIGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ1LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ1LDAuMDMzMSAtMC4xMzM0NSwtMC4zMzA3MyB6IG0gMC4wNDI5LC04LjQ2NjY3IGMgMCwtMC45NDU4OCAwLjA0OCwtMS4zMzI4NCAwLjEwNjcxLC0wLjg1OTg5IDAuMDU4NywwLjQ3Mjk0IDAuMDU4NywxLjI0Njg0IDAsMS43MTk3OSAtMC4wNTg3LDAuNDcyOTQgLTAuMTA2NzEsMC4wODYgLTAuMTA2NzEsLTAuODU5OSB6IG0gMzcuMDA4NTYsLTExLjI0NDc5IGMgMC4wMDUsLTAuNDM2NTYgMC4wNjQ3LC0wLjU4MzEzIDAuMTMxNzcsLTAuMzI1NjkgMC4wNjcxLDAuMjU3NDMgMC4wNjI3LDAuNjE0NjEgLTAuMDEsMC43OTM3NSAtMC4wNzI1LDAuMTc5MTMgLTAuMTI3MzYsLTAuMDMxNSAtMC4xMjE5NywtMC40NjgwNiB6IG0gLTMzLjYwMjA4LC0wLjc5Mzc1IGMgMC4wMDUsLTAuNDM2NTYgMC4wNjQ3LC0wLjU4MzEzIDAuMTMxNzcsLTAuMzI1NjkgMC4wNjcxLDAuMjU3NDMgMC4wNjI3LDAuNjE0NjEgLTAuMDEsMC43OTM3NSAtMC4wNzI1LDAuMTc5MTMgLTAuMTI3MzYsLTAuMDMxNSAtMC4xMjE5NywtMC40NjgwNiB6IG0gMTMuNTc0NDYsLTQuODk0NzkgYyAwLjI2MzM5LC0wLjI5MTA0IDAuNTM4NDIsLTAuNTI5MTcgMC42MTExOCwtMC41MjkxNyAwLjA3MjgsMCAtMC4wODMyLDAuMjM4MTMgLTAuMzQ2NiwwLjUyOTE3IC0wLjI2MzM5LDAuMjkxMDQgLTAuNTM4NDIsMC41MjkxNiAtMC42MTExOCwwLjUyOTE2IC0wLjA3MjgsMCAwLjA4MzIsLTAuMjM4MTIgMC4zNDY2LC0wLjUyOTE2IHogbSAtNTAuNTk1NjQ3LC0xNS44NzUgYyAwLC0wLjY1NDg1IDAuMDUxNiwtMC45MjI3NCAwLjExNDY2MywtMC41OTUzMSAwLjA2MzA2LDAuMzI3NDIgMC4wNjMwNiwwLjg2MzIgMCwxLjE5MDYyIC0wLjA2MzA3LDAuMzI3NDIgLTAuMTE0NjYzLDAuMDU5NSAtMC4xMTQ2NjMsLTAuNTk1MzEgeiBtIDcwLjYxMzQ2NywtMS4zMjI5MiBjIDAsLTAuMzYzOCAwLjA2LC0wLjUxMjYzIDAuMTMzNDUsLTAuMzMwNzMgMC4wNzM0LDAuMTgxOSAwLjA3MzQsMC40Nzk1NiAwLDAuNjYxNDYgLTAuMDczNCwwLjE4MTkgLTAuMTMzNDUsMC4wMzMxIC0wLjEzMzQ1LC0wLjMzMDczIHogbSAtMzYuMjY4OCwtMTMuMTE4OTIyIGMgMC4wMTI3LC0wLjMwODIyNSAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NTMgMC4wNzY0LDAuMTkwOTk2IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTE4IC0wLjA4NzksMC4wODc5MiAtMC4xNTA0NiwtMC4wNjgzNSAtMC4xMzg5NywtMC4zNDcyNjUgeiIKICAgICAgICAgaWQ9InBhdGg4NDQiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTA2LjEyMjc4LDE0Ni42NzY5MiBjIDAsLTAuMzYzOCAwLjA2MDEsLTAuNTEyNjMgMC4xMzM0NiwtMC4zMzA3MyAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU2IDAsMC42NjE0NiAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NiwwLjAzMzEgLTAuMTMzNDYsLTAuMzMwNzMgeiBtIC0zLjM3OTQsLTMuMTc1IGMgMCwtMS44MTkwMSAwLjA0MjMsLTIuNTYzMTUgMC4wOTM5LC0xLjY1MzY0IDAuMDUxNywwLjkwOTUgMC4wNTE3LDIuMzk3NzggMCwzLjMwNzI5IC0wLjA1MTcsMC45MDk1IC0wLjA5MzksMC4xNjUzNiAtMC4wOTM5LC0xLjY1MzY1IHogbSAzLjM5OTAxLC0xNS4zNDU4MyBjIDAsLTAuNTA5MzIgMC4wNTQ0LC0wLjcxNzY4IDAuMTIxLC0wLjQ2MzAyIDAuMDY2NSwwLjI1NDY2IDAuMDY2NSwwLjY3MTM4IDAsMC45MjYwNCAtMC4wNjY2LDAuMjU0NjYgLTAuMTIxLDAuMDQ2MyAtMC4xMjEsLTAuNDYzMDIgeiBtIDMzLjYxNjM3LC0yLjUxMzU0IGMgMC4wMDIsLTAuNzI3NjEgMC4wNTM0LC0wLjk5Mzg2IDAuMTE0MTksLTAuNTkxNjggMC4wNjA4LDAuNDAyMTggMC4wNTkyLDAuOTk3NSAtMC4wMDQsMS4zMjI5MiAtMC4wNjI4LDAuMzI1NDIgLTAuMTEyNTYsLTAuMDA0IC0wLjExMDU3LC0wLjczMTI0IHogbSAtNzAuNjI2OTcxLC00Ljg5NDggYyAwLC0xLjIzNjkyIDAuMDQ1NTgsLTEuNzQyOTQgMC4xMDEyODEsLTEuMTI0NDcgMC4wNTU3LDAuNjE4NDYgMC4wNTU3LDEuNjMwNDkgMCwyLjI0ODk1IC0wLjA1NTcsMC42MTg0NyAtMC4xMDEyODEsMC4xMTI0NSAtMC4xMDEyODEsLTEuMTI0NDggeiBtIDAsLTcuOTM3NSBjIDAsLTEuMjM2OTIgMC4wNDU1OCwtMS43NDI5NCAwLjEwMTI4MSwtMS4xMjQ0NyAwLjA1NTcsMC42MTg0NiAwLjA1NTcsMS42MzA0OSAwLDIuMjQ4OTUgLTAuMDU1NywwLjYxODQ3IC0wLjEwMTI4MSwwLjExMjQ1IC0wLjEwMTI4MSwtMS4xMjQ0OCB6IG0gOC45MjQyNzMsLTAuOTQ4MDggYyAwLjAxMjY5LC0wLjMwODIzIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODYgMC4wNzY0MywwLjE5MSAwLjA2NzAzLDAuNDE5MiAtMC4wMjA4OSwwLjUwNzEyIC0wLjA4NzkyLDAuMDg3OSAtMC4xNTA0NTMsLTAuMDY4MyAtMC4xMzg5NjQsLTAuMzQ3MjYgeiBtIC04Ljk5NTgzNCwtNC40OTc5MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTM5LC0wLjM3MDkyIDAuMTU5ODUzLC0wLjE1OTg1IDAuMDc2NDMsMC4xOTA5OSAwLjA2NzAzLDAuNDE5MTkgLTAuMDIwODksMC41MDcxMSAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI2IHogbSA3MC42OTg1MzIsLTEuMDM2MjkgYyAwLjAwMiwtMC43Mjc2IDAuMDUzNCwtMC45OTM4NSAwLjExNDE5LC0wLjU5MTY3IDAuMDYwOCwwLjQwMjE4IDAuMDU5MiwwLjk5NzQ5IC0wLjAwNCwxLjMyMjkyIC0wLjA2MjgsMC4zMjU0MiAtMC4xMTI1NiwtMC4wMDQgLTAuMTEwNTcsLTAuNzMxMjUgeiIKICAgICAgICAgaWQ9InBhdGg4NDIiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTA2LjE0NzczLDE0NS4yMjE3MSBjIDAuMDAzLC0wLjU4MjA4IDAuMDU3MSwtMC43ODkwNSAwLjEyMDQxLC0wLjQ1OTkzIDAuMDYzMywwLjMyOTEyIDAuMDYwOSwwLjgwNTM3IC0wLjAwNSwxLjA1ODMzIC0wLjA2NjIsMC4yNTI5NyAtMC4xMTgsLTAuMDE2MyAtMC4xMTUwNywtMC41OTg0IHogTSA4MS4yNzE1NTgsMTM1LjAzNTI1IGMgMCwtMC41MDkzMiAwLjA1NDQ1LC0wLjcxNzY4IDAuMTIxLC0wLjQ2MzAyIDAuMDY2NTUsMC4yNTQ2NyAwLjA2NjU1LDAuNjcxMzggMCwwLjkyNjA1IC0wLjA2NjU1LDAuMjU0NjYgLTAuMTIxLDAuMDQ2MyAtMC4xMjEsLTAuNDYzMDMgeiBtIDI0Ljg5MTQzMiwtNC42MzAyIGMgMC4wMDIsLTAuODczMTMgMC4wNTA5LC0xLjE5ODQyIDAuMTA5NzcsLTAuNzIyODcgMC4wNTg5LDAuNDc1NTUgMC4wNTc3LDEuMTg5OTIgLTAuMDAzLDEuNTg3NSAtMC4wNjA0LDAuMzk3NTggLTAuMTA4NTUsMC4wMDggLTAuMTA3MDcsLTAuODY0NjMgeiBtIDMzLjYzODMzLC0xNC40MTk4IGMgMCwtNC43Mjk0MiAwLjAzNTksLTYuNjY0MTkgMC4wNzk3LC00LjI5OTQ3IDAuMDQzOSwyLjM2NDcxIDAuMDQzOSw2LjIzNDI0IDAsOC41OTg5NSAtMC4wNDM4LDIuMzY0NzIgLTAuMDc5NywwLjQyOTk1IC0wLjA3OTcsLTQuMjk5NDggeiBNIDEwMS4yNjg5LDg5Ljg3NTM0NyBjIDAuMTkwOTksLTAuMDc2NDMgMC40MTkyLC0wLjA2NzAzIDAuNTA3MTIsMC4wMjA4OSAwLjA4NzksMC4wODc5MiAtMC4wNjg0LDAuMTUwNDUzIC0wLjM0NzI3LDAuMTM4OTY1IC0wLjMwODIzLC0wLjAxMjY5IC0wLjM3MDkyLC0wLjA3NTM5IC0wLjE1OTg1LC0wLjE1OTg1MyB6IgogICAgICAgICBpZD0icGF0aDg0MCIgLz4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDAuNjMwODQsMTUyLjU4MDg1IGMgLTAuNjkzNzk3LC0wLjEzNjcxIC0yNy4wNTY5NTMsLTE1LjM0MTU0IC0yOC42NzY3NTUsLTE2LjUzOTIgLTAuNjUyNDY0LC0wLjQ4MjQyIC0xLjQ4NTkwMSwtMS40MTIyMyAtMS44NTIwODMsLTIuMDY2MjQgbCAtMC42NjU3ODcsLTEuMTg5MTEgLTAuMDc0MTEsLTE1Ljk3NTQzIGMgLTAuMDY1MzIsLTE0LjA4MDczIC0wLjAyNTM3LC0xNi4wNDk4MiAwLjMzNjgyOSwtMTYuNjAyNjEgMC44ODUxNzMsLTEuMzUwOTQyIDAuOTA1MjEsLTEuMzQxMjkgMTkuNTc2NjY0LDkuNDI5OTQgMTAuOTY0MDQyLDYuMzI0OTYgMTIuNTMxNjAyLDcuNDU0MjQgMTMuMDc3NzgyLDkuNDIxMzIgMC4xODkwMiwwLjY4MDc2IDAuMjg1NTksNi4zNTQzMSAwLjI4NjU3LDE2LjgzNTYzIDAuMDAxLDE1LjQ1NzA5IC0wLjAxMDMsMTUuODIwNjMgLTAuNTI3NjksMTYuMzM4MDIgLTAuMjkxMDQsMC4yOTEwNCAtMC42MTg0NywwLjUxMjk1IC0wLjcyNzYxLDAuNDkzMTQgLTAuMTA5MTQsLTAuMDE5OCAtMC40NDgzNSwtMC4wODUzIC0wLjc1MzgxLC0wLjE0NTQ2IHogbSAtMTMuOTYxNjY5LC0xOS4wMDQ5IGMgLTAuMTI4MjAxLC0wLjMzNDA5IC00Ljc1NzYxMiwtMy4wMzg2MSAtNS4yMDEyODEsLTMuMDM4NjEgLTAuMTc4NjYyLDAgLTAuMjQ5MjAxLDEuMDAwMSAtMC4yMDA1MjIsMi44NDI5NyBsIDAuMDc1MSwyLjg0Mjk4IDIuNjQ1ODM0LDEuNTI0MjYgMi42NDU4MzMsMS41MjQyNiAwLjA3NTk0LC0yLjY5NTcgYyAwLjA0MTc2LC0xLjQ4MjY0IDAuMDIzMzYsLTIuODMyNzEgLTAuMDQwOSwtMy4wMDAxNiB6IG0gMC41MDU1NTgsLTMuMzY0OTggYyAwLjIwNTUzMywtMC42OTQ1OSAwLjYxODg3MSwtMS4wNTYxIDIuMTk3MzQsLTEuOTIxNzggMi40ODkyMTEsLTEuMzY1MTcgMi43ODQ4MTEsLTEuODk2ODIgMi42MzU1MDIsLTQuNzQwMDYgLTAuMDk0ODEsLTEuODA1NTMgLTAuMjU0Njg4LC0yLjQwNzUgLTEuMDIzMTE1LC0zLjg1MjI4IC0xLjExNjg5MSwtMi4wOTk5NyAtMy4xMTc5MjEsLTQuMTQzMzEgLTUuNzIyNzcxLC01Ljg0Mzc4IC0yLjA2MjQ0NywtMS4zNDYzOCAtNi4wMzgzNzMsLTIuODM4MjcgLTYuNzkyNjY1LC0yLjU0ODgyIC0wLjM1Nzc1OSwwLjEzNzI4IC0wLjQzMzg0NiwwLjU4NTU2IC0wLjQzMzg0NiwyLjU1NjA0IDAsMS4zMTQyNSAwLjA4OTMsMi4zOTE5MSAwLjE5ODQzNywyLjM5NDc5IDAuMTA5MTQxLDAuMDAzIDEuMDA2Mjc2LDAuMTIyMDUgMS45OTM2MzUsMC4yNjQ4MiAzLjQ4MzI1MSwwLjUwMzY2IDYuMDEwMDExLDIuNDY1NzggNi4wMTAwMTEsNC42NjcgMCwxLjI0Mzk1IC0wLjMyNjUzLDEuNTcwMTggLTIuODM4MTg4LDIuODM1NTkgLTEuNTkyOTcxLDAuODAyNTYgLTIuMjU5MywxLjgyNzkxIC0yLjEzNzc3MywzLjI4OTYxIDAuMDc3OSwwLjkzNjk1IDAuMTgyNDM5LDEuMDMyMjEgMi41OTQ3MTEsMi4zNjQ0NCAxLjM4MjQ0OCwwLjc2MzQ5IDIuNjM3NzQ4LDEuMzg4NTcgMi43ODk1NTUsMS4zODkwNyAwLjE1MTgxLDUuMmUtNCAwLjM4OTkzNSwtMC4zODQwOSAwLjUyOTE2NywtMC44NTQ2NCB6IG0gMTkuNzQ2MDYxLDIyLjA2ODIzIGMgLTAuNDM4MjUsLTAuNDM4MjUgLTAuNTExMjksLTEuMTUwODEgLTAuNjc3NTgsLTYuNjExMDYgLTAuMTAyNTEsLTMuMzY1OTIgLTAuMTA0ODYsLTEwLjYxNzg3IC0wLjAwNSwtMTYuMTE1NDQgMC4yMzg1NCwtMTMuMTU5OTYgLTAuNDA4MDIsLTExLjg1MTYgOC4zNTkzNSwtMTYuOTE1NzYgMi44MTAzNCwtMS42MjMyOSA4LjQ0MzQ2LC00Ljg4MDQgMTIuNTE4MDQsLTcuMjM4MDIgMTAuNzI0MTcsLTYuMjA1MTkgMTEuMTY1NjksLTYuMzk5MzQxIDEyLjEyOTgyLC01LjMzMzk4IDAuMzk3MjIsMC40Mzg5MiAwLjQzNzg5LDEuOTU4MTUgMC40Mzc4OSwxNi4zNTY3MyB2IDE1Ljg3Mjg2IGwgLTAuNzI3NjEsMS40MzY1MSBjIC0wLjQ2ODE4LDAuOTI0MzQgLTEuMTU5NDYsMS43NTQ0OSAtMS45Mzg4OCwyLjMyODM4IC0yLjI4MjMzLDEuNjgwNTEgLTI3LjgxNjgxLDE2LjMxMjU0IC0yOC44ODEwNSwxNi41NDk3MiAtMC41MTExNSwwLjExMzkyIC0wLjg2Nzc3LDAuMDE3MSAtMS4yMTQ3NywtMC4zMjk5NCB6IG0gMTguMTQ0MDcsLTE0LjAwMDIxIDAuODU5OSwtMC40MzEzMyB2IC0zLjAxODk2IC0zLjAxODk2IGwgLTEuMzg5MDcsMC43ODU4OSBjIC0wLjc2Mzk4LDAuNDMyMjMgLTEuOTg0MzcsMS4xMzYyOCAtMi43MTE5NywxLjU2NDU0IGwgLTEuMzIyOTIsMC43Nzg2NSAtMC4wNzQ5LDIuOTcyNDcgLTAuMDc0OSwyLjk3MjQ3IDEuOTI3MDIsLTEuMDg2NzIgYyAxLjA1OTg2LC0wLjU5NzcgMi4zMTM5OCwtMS4yODA4MiAyLjc4NjkyLC0xLjUxODA1IHogbSAtMS44MDU3MSwtNy4xMDc5MiBjIDIuNTI0NjMsLTEuNDY2MTUgMi41NTcxLC0xLjQ5Nzg2IDIuODY2NTIsLTIuODAwNDYgMC4xNzIxMywtMC43MjQ2MyAxLjE0NTQsLTIuNzA2NDUgMi4xNjI4MiwtNC40MDQwNCAyLjAzMzAzLC0zLjM5MjE3IDIuNTg2NTEsLTQuNzIxMDEgMi44MzAyOSwtNi43OTUyIDAuMzA3NDgsLTIuNjE2MTggLTAuODM0MzMsLTQuMzYxMTIgLTIuODUzNzEsLTQuMzYxMTIgLTIuMTE4MjMsMCAtNi4wNTcwNiwyLjE4NjEyIC05LjU1MDIxLDUuMzAwNTEgbCAtMS41MjEzNSwxLjM1NjQgdiAyLjU0MTEzIDIuNTQxMTEgbCAyLjQ3NjI1LC0yLjQyODM3IGMgMy4yNjYyLC0zLjIwMzA1IDUuNDEzMzYsLTQuMTg0MjUgNS44MTI5MSwtMi42NTYzNSAwLjIyOTA4LDAuODc2MDEgLTAuNDQyNjQsMi42MDQxIC0xLjkzODU5LDQuOTg3MjQgLTAuNzc2NDUsMS4yMzY5MyAtMS43NTQxNywyLjk4MDA0IC0yLjE3MjcxLDMuODczNTggLTAuODE4MTcsMS43NDY2OCAtMS4zNzExNSw0LjMyODUgLTAuOTI3MDksNC4zMjg1IDAuMTQzNzMsMCAxLjQxMDQyLC0wLjY2NzMyIDIuODE0ODcsLTEuNDgyOTMgeiBNIDEwMS40NTA4LDExMi41ODM5NSBDIDk5Ljc5OTk0OSwxMTEuNzk4NzYgNzUuNzExOTkyLDk3LjkzMDI1MiA3My4xMjc0MDcsOTYuMjc2OTEzIDcxLjczMDIxNiw5NS4zODMxNCA3MS42NTgyNTIsOTUuMjgzODYzIDcxLjczNTc0Niw5NC4zNTcwNzMgNzEuODM1ODM1LDkzLjE2MDA1MiA3MC45OTE5NDUsOTMuNzAwODg1IDg5LjAxMjEwOSw4My4yODQ5OCBsIDEzLjIyNTg5MSwtNy42NDQ3NSAxLjg2MTEyLC0wLjA5Mzk4IGMgMS43NzUxLC0wLjA4OTY0IDEuOTY0NzcsLTAuMDM5MjggNC4xMDQzMSwxLjA4OTczOSA0LjQ5ODA3LDIuMzczNTk3IDI3LjY2MzYsMTUuODgxMTIzIDI4LjM3MDgxLDE2LjU0MjY1OSAxLjI2NzM3LDEuMTg1NTE1IDAuODkxNDcsMi4wNTgwODIgLTEuNTM0MTksMy41NjEyOTEgLTIuNDUxODEsMS41MTk0MTQgLTI0Ljc2NzA3LDE0LjM5Mjc3MSAtMjYuNzgyNywxNS40NTA1NjEgLTIuNzgxMTIsMS40NTk1MSAtNC4zNzE4NCwxLjU1MTQ2IC02LjgwNjU1LDAuMzkzNDUgeiBtIDE0LjM3ODEyLC0xMi4zMDU1MiAzLjAwMTA0LC0xLjY4MjI2NCAtMC44ODQzNywtMC41MjcyMDkgYyAtMi4xNTgwNiwtMS4yODY0OTkgLTMuODg0MDMsLTIuMTkyMDM2IC00LjE3ODA2LC0yLjE5MjAzNiAtMC40MTU0NSwwIC02LjA5OTk2LDMuMTY3Mjk2IC02LjA5NTk2LDMuMzk2NTQ0IDAuMDA0LDAuMjQzMiA0LjI3MTIxLDIuNjI4OTQ1IDQuNzU5NDQsMi42NjEwOTUgMC4yMTgyOCwwLjAxNDQgMS43NDczNCwtMC43MzA4OCAzLjM5NzkxLC0xLjY1NjEzIHogbSAtOC42MTQ2NCwtMi45NzYxNjQgYyAwLjc1OTE1LC0wLjM4MDIyNSAyLjA2NjI1LC0xLjEwNDA1NiAyLjkwNDY4LC0xLjYwODUxIGwgMS41MjQ0MSwtMC45MTcxOTIgLTAuNTQ4MzgsLTAuNzcwMTI1IGMgLTAuNDk2NDIsLTAuNjk3MTUgLTAuNTMxNjEsLTEuMDAyMjc5IC0wLjM3MTM4LC0zLjIyMDEzNCAwLjE2MzU4LC0yLjI2NDM2OSAwLjEzMzI3LC0yLjUwNTE1NCAtMC40MDAwNywtMy4xNzc4IC0wLjMxNzM5LC0wLjQwMDI4NSAtMS4xMzM5MSwtMS4wNTQxOTkgLTEuODE0NDksLTEuNDUzMTQxIC0xLjA5NzE1LC0wLjY0MzEyOCAtMS41NTc0NSwtMC43MzYzMTkgLTQuMDYwNzIsLTAuODIyMTE0IC0yLjM4NjM4LC0wLjA4MTc5IC0zLjEwMjkyLC0wLjAwNDggLTQuNjMwMjA0LDAuNDk3Njg4IC0yLjIyNDgzOSwwLjczMTk0NyAtNC45MDQ3NDUsMi4wMzAxMzEgLTYuODMyODA2LDMuMzA5OTA1IC0xLjUzMDY0NiwxLjAxNTk4NCAtNC4xMDI1NjMsMy4yMjI3OCAtNC4xMDE3NTYsMy41MTk0NTEgNS4yOWUtNCwwLjE5NDIwNiAxLjk3MDE0MywxLjQ3NjgxNyAzLjM0MTQ0MSwyLjE3NTk0NyBsIDAuODkxODM3LDAuNDU0Njg0IDEuODkzOTksLTEuOTUzMjgyIGMgMi4zMTcwNywtMi4zODk2MDMgMy45NjYyNDIsLTMuMjg0OTUyIDYuMDI4ODk4LC0zLjI3MzE0IDIuMzU2OCwwLjAxMzUgMi40OTY0LDAuMjMxNDkyIDIuMjcwNTIsMy41NDU0NjggbCAtMC4xODk5MywyLjc4NjU1MiAwLjc5ODUzLDAuNzk4NTMxIGMgMC45Nzc1NSwwLjk3NzU0OCAxLjUyMjM2LDAuOTk1MjczIDMuMjk1NDMsMC4xMDcyMTIgeiIKICAgICAgICAgaWQ9InBhdGg4MzgiIC8+CiAgICA8L2c+CiAgPC9nPgo8L3N2Zz4K",
+ "properties": {
+ "number_of_entities": {
+ "name": "number_of_entities",
+ "title": "Number of Entities",
+ "description": "How many entities should be selected.",
+ "type": "string",
+ "parameterType": "Long",
+ "value": "10",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "python",
+ "is_deprecated": false,
+ "tags": [
+ "WorkflowTask",
+ "PythonPlugin"
+ ],
+ "pluginType": "customtask",
+ "relatedPlugins": []
+ },
"SendEMail": {
"pluginId": "SendEMail",
"title": "Send email",
@@ -5560,6 +5803,52 @@
"pluginType": "customtask",
"relatedPlugins": []
},
+ "setExecutionVariableOperator": {
+ "pluginId": "setExecutionVariableOperator",
+ "title": "Set execution variable",
+ "categories": [
+ "Variables"
+ ],
+ "main_category": "Variables",
+ "description": "Sets an execution variable to the first value of the (single) input and passes the input through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution.",
+ "markdownDocumentation": "Sets a single **execution-scope** template variable from this operator's input and passes the input through\nunchanged, so it can be inserted anywhere in a workflow chain.\n\nThe variable is written to the `execution` scope. A value set by this operator replaces an execution variable\nof the same name that was defined as a default on the workflow or provided when the run was started. This\noperator only takes effect while running inside a workflow execution, where all nodes share one\nexecution-variable holder.\n\nAny downstream node can read the variable as `{{execution.}}`. Referencing an execution variable that\nhas not been set fails.\n\nIt is the workflow-operator counterpart of the **Set execution variable** transformer (`setExecutionVariable`):\nuse this operator to pass a value between workflow nodes without embedding a transform.\n\n## Behaviour\n\n- Reads a value from the **first entity** of the input: the first value of **Source path**, or the entity's\n first value when no source path is given. Later entities are not consulted.\n- Writes it to the execution scope under `variableName`. If the input is empty or the first entity has no such\n value, the variable is left unchanged (a default or run-start override stays in place).\n- Forwards the input entities unchanged (connect the output to keep the chain going, or leave it unconnected to\n use this purely as a side-effecting node).\n",
+ "pluginIcon": null,
+ "properties": {
+ "variableName": {
+ "name": "variableName",
+ "title": "Variable name",
+ "description": "Name of the execution variable to set. It is written to the 'execution' scope and addressed downstream as 'execution.'.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "myVariable",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "sourcePath": {
+ "name": "sourcePath",
+ "title": "Source path",
+ "description": "Optional path/attribute of the input that supplies the value. If left empty, the first value of the input is used.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "WorkflowTask"
+ ],
+ "pluginType": "customtask",
+ "relatedPlugins": []
+ },
"cmem_plugin_parameters-ParametersPlugin": {
"pluginId": "cmem_plugin_parameters-ParametersPlugin",
"title": "Set or Overwrite parameters",
@@ -6067,14 +6356,14 @@
"Uncategorized"
],
"main_category": "Uncategorized",
- "description": "A task that executes a SPARQL Select query on a SPARQL enabled data source and outputs the SPARQL result. If the SPARQL source is defined on a specific graph, a FROM clause will be added to the query at execution time, except when there already exists a GRAPH or FROM clause in the query. FROM NAMED clauses are not injected.",
- "markdownDocumentation": "The SPARQL SELECT plugin is a task for executing SPARQL SELECT queries on the input RDF data source.\n\n## Description\n\nThe SPARQL Select query plugin is an example of a _RDF task_ or _operator_. Such a task can be used in a workflow,\nconnecting an input to an output. In this specific case, the _input_ is \u2014 in essence \u2014 a _SPARQL endpoint_ and the\n_output_ is the entity table containing the _SPARQL results_ of the SPARQL SELECT query execution.\n\nIn general terms, a [SPARQL 1.1 SELECT](https://www.w3.org/TR/sparql11-query/#select) query is supported. One of the\nsimplest examples is `SELECT * WHERE { ?s ?p ?o }`.\n\nThe [result limit](https://www.w3.org/TR/sparql11-query/#modResultLimit) can be specified for the SPARQL SELECT plugin\nitself, with the parameter `limit`. Additionally, a timeout can be specified with the parameter `sparqlTimeout`.\n\nAs usual, the SPARQL results contain both \"variables\" and \"bindings\", such as in\n[this example](https://www.w3.org/TR/sparql11-results-json/#json-result-object).\nThis tabular raw form is transformed into an _entity table_.\n\n### Internal Specifics\n\nIf the SPARQL source is defined on a specific graph, a `FROM` clause will be added to the query at execution time,\nexcept when there already exists a `GRAPH` or `FROM` clause in the query. `FROM NAMED` clauses are not injected.\n",
+ "description": "A task that executes a SPARQL Select query and outputs the SPARQL result.",
+ "markdownDocumentation": "The SPARQL SELECT plugin is a task for executing SPARQL SELECT queries on an RDF data source.\nIt can be used in a workflow, connecting an input to an output. A\n[SPARQL 1.1 SELECT](https://www.w3.org/TR/sparql11-query/#select) query is supported; the simplest example is\n`SELECT * WHERE { ?s ?p ?o }`.\n\n## Input and output\n\nThe _input_ depends on the configuration:\n\n- By default, the query is executed against the connected input, which must be a _SPARQL endpoint_\n (i.e. an RDF dataset).\n- When **Use fallback RDF dataset** (`useDefaultDataset`) is enabled, the query is executed against the\n fallback RDF dataset (as configured in `dataset.defaultRdf`) instead. The input port then depends on what\n the template references:\n - If the template references input entity properties (`input.entity.*`), the task accepts an entity input\n and generates one query per input entity.\n - If it references only parameters of the input task (`input.config.*`), an input connection is still\n required \u2014 it supplies the parameter values \u2014 but the query is rendered and executed only once.\n - If it references neither, the task has no input port.\n\nThe _output_ is an entity table built from the query's\n[SPARQL results](https://www.w3.org/TR/sparql11-results-json/#json-result-object): each projected variable becomes\na column, and each result binding becomes a row.\n\nThe [result size](https://www.w3.org/TR/sparql11-query/#modResultLimit) can be capped with the `limit` parameter,\nand a query timeout (in milliseconds) can be set via `sparqlTimeout`.\n\n## Automatic `FROM` clause injection\n\nIf the SPARQL source is defined on a specific graph, a `FROM` clause will be added to the query at execution time,\nexcept when there already exists a `GRAPH` or `FROM` clause in the query. `FROM NAMED` clauses are not injected.\n\n## Templating\n\nThe select query is rendered by a template engine before execution.\n[`Jinja`](https://jinja.palletsprojects.com/) is the default and is described below; for the deprecated `Simple`\nand `Velocity Engine` modes, see \"Legacy template engines\" at the end.\n\nJinja uses `{{ ... }}` for value expressions and `{% ... %}` for control flow such as conditionals.\n\n### Template variables\n\nThe following variables are available:\n\n- `input.config.`: a parameter of the task connected to the input port. `` is a parameter id\n of that task's plugin, e.g. `graph` on a SPARQL dataset.\n- `output.config.`: a parameter of the task the output is connected to.\n- `input.entity.`: the value(s) of the given property of the current input entity. Only available\n with **Use fallback RDF dataset** enabled, since only then the task receives input entities\n (see _Input and output_ above).\n- `project.`: a project-scoped template variable.\n- `global.`: a global template variable.\n\nA single-valued entity property is inserted as a plain string. A multi-valued property can be iterated with\n`{% for value in input.entity. %}`; inserting it directly concatenates all values without a\nseparator. Referencing a variable that is not available at execution time \u2014 an unknown parameter name or an\nentity property without a value \u2014 fails the query generation with an error.\n\nParameter, property and variable names must be valid Jinja identifiers (`[a-zA-Z_][a-zA-Z0-9_]*`);\nbracket-subscript access such as `input.entity[\"urn:prop:label\"]` is not supported.\n\nFor example, to query the named graph that is configured on the input dataset:\n\n```sparql\nSELECT * WHERE { GRAPH <{{ input.config.graph | validate_uri }}> { ?s ?p ?o } }\n```\n\n### Default scope\n\nThe `defaultScope` parameter declares one scope whose variables are additionally exposed at the top level of the\ntemplate context, so they can be referenced without the scope prefix. It defaults to `input.entity`, which means\na template may write `{{ property }}` as a shorthand for `{{ input.entity.property }}`:\n\n```\n{{ property }} \u2261 {{ input.entity.property }}\n```\n\nBoth forms resolve to the same value. Set `defaultScope` to the empty string to disable this aliasing and require\nevery variable to be addressed with its full scope.\n\n### Filters\n\nValues are inserted verbatim by default, so URI brackets (`<...>`) and quotation marks around literals must be\nwritten in the template. The following filters are provided to render values safely:\n\n- `validate_uri`: validates that the value is a valid absolute IRI and returns it unchanged. Throws a validation\n error otherwise. Wrap the output in `<...>` in the template.\n- `escape_literal`: escapes backslashes, quotes, newlines, carriage returns and tabs so the value can be used\n inside a short-form SPARQL string literal (`\"...\"` or `'...'`). No enclosing quotes are added.\n- `escape_multiline_literal`: escapes backslashes and breaks any run of three or more consecutive single or double\n quotes. Use for values that are wrapped in triple-quoted SPARQL literals (`\"\"\"...\"\"\"` or `'''...'''`).\n\nAll transformer plugins are also available as Jinja filters under their plugin id (for example `lowerCase`,\n`trim`, `urlEncode`).\n\n### Input schema inference\n\nThe input schema (the entity properties the task expects) is derived by scanning the raw template for\n`input.entity.` references (or bare references resolved via `defaultScope`). This scan operates\non the template text before rendering, so SPARQL line comments (`# ...`) are **not** stripped: a\ncommented-out line such as\n\n```sparql\n# {{ input.entity.property }}\n```\n\nwill still cause `property` to appear in the inferred input schema.\n\n### Output schema inference\n\nThe output schema is derived from the raw template by a heuristic, without rendering it. The heuristic takes\nthe projection between `SELECT` and the first `WHERE`, `FROM` or `{`, drops a leading `DISTINCT` / `REDUCED`,\nand then:\n\n- For `SELECT *`, collects every distinct `?var` token in the query.\n- Otherwise, collects each top-level `?var` and the trailing `AS ?alias` from parenthesised expressions\n (e.g. `(COUNT(?s) AS ?count)` yields `count`).\n\nEach variable becomes a string-typed path. If no variables can be detected (e.g. the projection is produced by\na Jinja expression), the output port is reported with an unknown schema.\n\n### Validation\n\nAt task creation, the Jinja template is checked against the available template variables:\n\n- Every `project.<...>` or `global.<...>` reference must resolve to a known variable, matched on the full\n scoped name (so e.g. `project.metaData.label` is looked up at that exact scope).\n- Every `input.<...>` or `output.<...>` reference must use `config` or `entity` as its second segment.\n\nBare references are resolved through `defaultScope` before applying the same rules. The template is not\nrendered and the resulting SPARQL is not parsed.\n\n### Legacy template engines\n\nIn addition to Jinja, two deprecated template engines are supported for backwards compatibility: `Simple`\nand [`Velocity Engine`](https://velocity.apache.org/engine/2.4.1/user-guide.html). Their syntax is identical\nto the one used by the `SPARQL Update operator` and is documented there.\n",
"pluginIcon": "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxzdmcKICAgICAgICB3aWR0aD0iMzJweCIKICAgICAgICBoZWlnaHQ9IjMycHgiCiAgICAgICAgdmlld0JveD0iMCAwIDMyIDMyIgogICAgICAgIHZlcnNpb249IjEuMSIKICAgICAgICBmaWxsPSJjdXJyZW50Q29sb3IiCiAgICAgICAgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIgogICAgICAgIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyIKICAgICAgICB4bWxuczpjYz0iaHR0cDovL2NyZWF0aXZlY29tbW9ucy5vcmcvbnMjIgogICAgICAgIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyI+CiAgICA8dGl0bGU+U1BBUlFMIFNlbGVjdCBxdWVyeTwvdGl0bGU+CiAgICA8cGF0aAogICAgICAgICAgICBkPSJtIDI3LDcgYSA0LDQgMCAwIDEgLTQsNCA0LDQgMCAwIDEgLTQsLTQgNCw0IDAgMCAxIDQsLTQgNCw0IDAgMCAxIDQsNCB6IiAvPgogICAgPHBhdGgKICAgICAgICAgICAgZD0iTSAxMiw2IEEgNCw0IDAgMCAxIDgsMTAgNCw0IDAgMCAxIDQsNiA0LDQgMCAwIDEgOCwyIDQsNCAwIDAgMSAxMiw2IFoiIC8+CiAgICA8cGF0aAogICAgICAgICAgICBkPSJtIDEyLDI2IGEgNCw0IDAgMCAxIC00LDQgNCw0IDAgMCAxIC00LC00IDQsNCAwIDAgMSA0LC00IDQsNCAwIDAgMSA0LDQgeiIgLz4KICAgIDxwYXRoCiAgICAgICAgICAgIGQ9Im0gMTEsMTYgYSAzLDMgMCAwIDEgLTMsMyAzLDMgMCAwIDEgLTMsLTMgMywzIDAgMCAxIDMsLTMgMywzIDAgMCAxIDMsMyB6IiAvPgogICAgPHBhdGgKICAgICAgICAgICAgZD0ibSAyNy42MDcxMjMsMjIuODIwMTQ1IGMgMC42NjYwOSwtMi4wNDQ0NDIgLTAuMTUyODk2LC00LjI3OTIyIC0xLjk4MjMwMSwtNS40MDkxMyAtMS44Mjk0MDQsLTEuMTI5OTExIC00LjE5NDM1OSwtMC44NjE2NTYgLTUuNzI0MjE0LDAuNjQ5MjkxIC0xLjUyOTg1NSwxLjUxMDk0OCAtMS44Mjc0OTgsMy44NzIzODQgLTAuNzIwNDIzLDUuNzE1Njk4IDEuMTA3MDc0LDEuODQzMzE0IDMuMzMxNDk1LDIuNjkwMDI3IDUuMzg0MDYyLDIuMDQ5NDEyIGwgMS45MDUxMzgsMy43MzkwNDIgMS4zODk5MDUsLTAuNzA4MTkyIC0xLjkwNTEzNywtMy43MzkwNDMgYyAwLjc3NTc1NiwtMC41NzU4NjEgMS4zNTMzODIsLTEuMzc4NTY4IDEuNjUyOTcsLTIuMjk3MDc4IHogbSAtNS4zOTI3NjIsMS41MTQ4MjkgYyAtMS42MzIwMzksLTAuNTMwMjgyIC0yLjUyNTE5LC0yLjI4MzE4OSAtMS45OTQ5MDksLTMuOTE1MjI4IDAuNTMwMjgyLC0xLjYzMjA0IDIuMjgzMTg5LC0yLjUyNTE5MSAzLjkxNTIyOSwtMS45OTQ5MDkgMS42MzIwMzksMC41MzAyODIgMi41MjUxOSwyLjI4MzE4OSAxLjk5NDkwOCwzLjkxNTIyOCAtMC4yNTQ2NSwwLjc4MzczMyAtMC44MTAyMDcsMS40MzQyMDcgLTEuNTQ0NDU1LDEuODA4MzI1IC0wLjczNDI0OCwwLjM3NDExOCAtMS41ODcwNCwwLjQ0MTIzNCAtMi4zNzA3NzMsMC4xODY1ODQgeiIgLz4KICAgIDxwYXRoCiAgICAgICAgICAgIGQ9Ik0gNy4yNSw1LjI1IFYgNiAyNi41IDI3LjI1IGggMS41IFYgMjYuNSA2IDUuMjUgWiIgLz4KICAgIDxwYXRoCiAgICAgICAgICAgIGQ9Ik0gMjMuMjQ0MTQxLDUuOTY4NzUgMjIuNjA1NDY5LDYuMzYxMzI4MSA3LjYwNTQ2ODcsMTUuNjExMzI4IDYuOTY4NzUsMTYuMDA1ODU5IDcuNzU1ODU5NCwxNy4yODEyNSA4LjM5NDUzMTMsMTYuODg4NjcyIDIzLjM5NDUzMSw3LjYzODY3MTkgMjQuMDMxMjUsNy4yNDQxNDA2IFoiIC8+CiAgICA8cGF0aAogICAgICAgICAgICBkPSJtIDcuNTcwMzEyNSwxNS4yNzkyOTcgLTAuNTQxMDE1NiwxLjQwMDM5IDAuNzAxMTcxOCwwLjI2OTUzMiAxMS4wMDAwMDAzLDQuMjUgMC42OTkyMTgsMC4yNzE0ODQgMC41NDEwMTYsLTEuNDAwMzkxIC0wLjcwMTE3MiwtMC4yNjk1MzEgLTEwLjk5OTk5OTcsLTQuMjUgeiIgLz4KICAgIDxtZXRhZGF0YT4KICAgICAgICA8cmRmOlJERj4KICAgICAgICAgICAgPGNjOldvcmsKICAgICAgICAgICAgICAgICAgICByZGY6YWJvdXQ9IiI+CiAgICAgICAgICAgICAgICA8ZGM6dGl0bGU+U1BBUlFMIFNlbGVjdCBxdWVyeTwvZGM6dGl0bGU+CiAgICAgICAgICAgIDwvY2M6V29yaz4KICAgICAgICA8L3JkZjpSREY+CiAgICA8L21ldGFkYXRhPgo8L3N2Zz4=",
"properties": {
"selectQuery": {
"name": "selectQuery",
"title": "Select query",
- "description": "A SPARQL 1.1 select query",
+ "description": "A SPARQL 1.1 select query. The query supports Jinja templating. Parameters of the connected input and output tasks can be accessed via 'input.config.' and 'output.config.'. Project and global template variables are available as 'project.' and 'global.'. Example: SELECT * WHERE { GRAPH <{{ input.config.graph }}> { ?s ?p ?o } }",
"type": "string",
"parameterType": "code-sparql",
"value": null,
@@ -6085,7 +6374,7 @@
"limit": {
"name": "limit",
"title": "Result limit",
- "description": "If set to a positive integer, the number of results is limited",
+ "description": "If set to a positive integer, the number of results is limited. The limit is applied per query: if one query is generated per input entity, it caps the results of each query, not the combined total.",
"type": "string",
"parameterType": "string",
"value": "",
@@ -6104,6 +6393,41 @@
"visibleInDialog": true,
"properties": {}
},
+ "useDefaultDataset": {
+ "name": "useDefaultDataset",
+ "title": "Use fallback RDF dataset",
+ "description": "If enabled, the query executes against the configured fallback RDF dataset (as configured in `dataset.defaultRdf`) when no RDF dataset is connected. If the query template references input entities, one query is generated per input entity.",
+ "type": "string",
+ "parameterType": "boolean",
+ "value": "false",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "templatingMode": {
+ "name": "templatingMode",
+ "title": "Templating mode",
+ "description": "The templating mode for the template engine.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "jinja",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "defaultScope": {
+ "name": "defaultScope",
+ "title": "Default scope",
+ "description": "Variables from this scope can be accessed without the scope prefix in Jinja. For example, with default scope 'input.entity', a template may reference '{{ property }}' instead of '{{ input.entity.property }}'. Leave empty to disable.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "input.entity",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {
"sparqlTimeout": {
"name": "sparqlTimeout",
"title": "SPARQL query timeout (ms)",
@@ -6111,13 +6435,18 @@
"type": "string",
"parameterType": "int",
"value": "0",
- "advanced": false,
+ "advanced": true,
"visibleInDialog": true,
"properties": {}
}
},
- "properties_advanced": {},
- "actions": {},
+ "actions": {
+ "showPrefixes": {
+ "label": "Show prefixes",
+ "description": "Shows the available namespace prefixes as a SPARQL header that can be copied into the query.",
+ "icon": null
+ }
+ },
"required": [
"selectQuery"
],
@@ -6147,13 +6476,13 @@
],
"main_category": "Uncategorized",
"description": "A task that outputs SPARQL Update queries for every entity from the input based on a SPARQL Update template. The output of this operator should be connected to the SPARQL datasets to which the results should be written.",
- "markdownDocumentation": "The SPARQL UPDATE query plugin is a task for outputting SPARQL UPDATE queries from the input RDF data source.\n\n## Description\n\nThe SPARQL Update query plugin is an example of a _task_. Notice well that this plugin is neither a _RDF task_\nnor a _RDF dataset_. This is in contrast to e.g. the SPARQL Select query and the SPARQL endpoint, respectively.\n\nMore specifically, this means the following: This plugin does not execute SPARQL queries of any sort, but _generates_\nthem. It generates [SPARQL Update](https://www.w3.org/TR/sparql11-update/) queries from a templating engine. In order to\n_execute_ these queries, we need to connect this task from an input into an output RDF dataset.\n\n## Templating\n\nThe SPARQL Update query plugin uses a template in order to construct and output SPARQL update queries.\nThere are two possible template engines supported by this plugin: a `Simple` engine and\n[`Velocity Engine`](https://velocity.apache.org/engine/2.4.1/user-guide.html).\nEach of these engines supports a different set of templating features, such as for example _variable interpolation_ with\nthe dollar sign (`$`), i.e. filling in input values via placeholders in the template.\n\n### Example of the `Simple` mode\n\n```\n DELETE DATA { ${} rdf:label ${\"PROP_FROM_ENTITY_SCHEMA2\"} }\n INSERT DATA { ${} rdf:label ${\"PROP_FROM_ENTITY_SCHEMA3\"} }\n```\n\nThis will insert the URI serialization of the property value `PROP_FROM_ENTITY_SCHEMA1` for the\n`${}` expression.\nFurthermore, it will insert a plain literal serialization for the property values `PROP_FROM_ENTITY_SCHEMA2` and\n`PROP_FROM_ENTITY_SCHEMA3` for the template literal expressions.\n\nIt is also possible to write something like `${\"PROP\"}^^` or `${\"PROP\"}@en`. In other words, we\ncan combine variable substitutions with fixed expressions to construct semi-flexible expressions within the template.\n\n### Example of the `Velocity Engine` mode\n\n```\n DELETE DATA { $row.uri(\"PROP_FROM_ENTITY_SCHEMA1\") rdf:label $row.plainLiteral(\"PROP_FROM_ENTITY_SCHEMA2\") }\n #if ( $row.exists(\"PROP_FROM_ENTITY_SCHEMA1\") )\n INSERT DATA { $row.uri(\"PROP_FROM_ENTITY_SCHEMA1\") rdf:label $row.plainLiteral(\"PROP_FROM_ENTITY_SCHEMA3\") }\n #end\n```\n\nInput values are accessible via various methods of the `row` variable (used with `$row`):\n\n- `$row.uri(inputPath: String)`: Renders an input value as **URI**. Throws an exception if the value isn't a valid URI.\n- `$row.plainLiteral(inputPath: String)`: Renders an input value as **plain literal**, i.e. it escapes problematic\n characters, etc.\n- `$row.rawUnsafe(inputPath: String)`: Renders an input value as is, i.e. **no escaping** is done.\n This should **only** be used if the input values can be trusted.\n- `$row.exists(inputPath: String)`: Returns `true` if a value for the input path **exists**, else `false`.\n\nThe methods `uri`, `plainLiteral` and `rawUnsafe` throw an exception if no input value is available for the given\ninput path.\n\nIn addition to input values, properties of the input and output tasks can be accessed via the `inputProperties` and\n`outputProperties` objects. The available keys in these objects are dynamic and correspond exactly to the configuration\nparameters of the tasks connected to the input and output ports of this operator.\n\n- To find the available keys for `$inputProperties`, check the parameter names of the task connected to the input port.\n- To find the available keys for `$outputProperties`, check the parameter names of the task connected to the output port.\n\nFor example, if the connected input task has a parameter named `graph`, you can access it as `$inputProperties.uri(\"graph\")`.\nSimilarly, if the connected output task has a parameter named `endpoint`, you can access it as `$outputProperties.uri(\"endpoint\")`.\n\nBoth `inputProperties` and `outputProperties` support the same methods as the `row` object:\n\n- `uri(inputPath: String)`\n- `plainLiteral(inputPath: String)`\n- `rawUnsafe(inputPath: String)`\n- `exists(inputPath: String)`\n\nFor more information about the Velocity Engine, visit http://velocity.apache.org.\n\n### Internal Specifics\n\nIn contrast to the SPARQL select operator, no `FROM` clause gets injected into the query.\n",
+ "markdownDocumentation": "The SPARQL UPDATE query plugin is a task for outputting SPARQL UPDATE queries from the input RDF data source.\n\n## Description\n\nThe SPARQL Update query plugin is an example of a _task_. Notice well that this plugin is neither a _RDF task_\nnor a _RDF dataset_. This is in contrast to e.g. the SPARQL Select query and the SPARQL endpoint, respectively.\n\nMore specifically, this means the following: This plugin does not execute SPARQL queries of any sort, but _generates_\nthem. It generates [SPARQL Update](https://www.w3.org/TR/sparql11-update/) queries from a templating engine. In order to\n_execute_ these queries, we need to connect this task from an input into an output RDF dataset.\n\n## Templating\n\nThe `sparqlUpdateOperator` plugin uses a **template** in order to construct and output SPARQL update queries.\nThree template engines are supported: `Jinja` (the default), `Simple`, and\n[`Velocity Engine`](https://velocity.apache.org/engine/2.4.1/user-guide.html).\nThe `Simple` and `Velocity Engine` modes are deprecated.\n\n### Example of the `Jinja` mode\n\n[Jinja](https://jinja.palletsprojects.com/) is the recommended template engine. It uses `{{ }}` for expressions and\n`{% %}` for control flow statements such as conditionals.\n\n```\nDELETE DATA { <{{ input.entity.subject | validate_uri }}> rdfs:label \"{{ input.entity.oldLabel | escape_literal }}\" } ;\n{% if input.entity.subject %}\n INSERT DATA { <{{ input.entity.subject | validate_uri }}> rdfs:label \"{{ input.entity.newLabel | escape_literal }}\" } ;\n{% endif %}\n```\n\nThe following variables are available:\n\n- `input.entity.`: the value of the given property on the current input entity.\n- `input.config.`: a parameter of the connected input task.\n- `output.config.`: a parameter of the connected output task.\n- `project.`: a project-scoped template variable.\n- `global.`: a global template variable.\n\nEntity property names must be valid Jinja identifiers (`[a-zA-Z_][a-zA-Z0-9_]*`); bracket-subscript access such as\n`input.entity[\"urn:prop:label\"]` is not supported.\n\nValues are inserted verbatim by default, so URI brackets (`<...>`) and quotation marks around literals must be\nwritten in the template. The following filters are provided to render values safely:\n\n- `validate_uri`: validates that the value is a valid absolute IRI and returns it unchanged. Throws a validation\n error otherwise. Wrap the output in `<...>` in the template.\n- `escape_literal`: escapes backslashes, quotes, newlines, carriage returns and tabs so the value can be used\n inside a short-form SPARQL string literal (`\"...\"` or `'...'`). No enclosing quotes are added.\n- `escape_multiline_literal`: escapes backslashes and breaks any run of three or more consecutive single or double\n quotes. Use for values that are wrapped in triple-quoted SPARQL literals (`\"\"\"...\"\"\"` or `'''...'''`).\n\nAll transformer plugins are also available as Jinja filters under their plugin id (for example `lowerCase`,\n`trim`, `urlEncode`).\n\n### Validation\n\nAt task creation, the template is checked against the available template variables. What is checked depends\non the selected templating mode:\n\n- `Jinja`:\n - Every `project.<...>` or `global.<...>` reference must resolve to a known variable, matched on the full\n scoped name (so e.g. `project.metaData.label` is looked up at that exact scope).\n - Every `input.<...>` or `output.<...>` reference must use `config` or `entity` as its second segment.\n - The template is not rendered and the resulting SPARQL is not parsed.\n- `Simple` / `Velocity Engine`:\n - The template is rendered once with placeholder values and the result must parse as a SPARQL Update query.\n - Templates that use `rawUnsafe` skip this parse check.\n\n### Example of the `Simple` mode (deprecated)\n\n```\n DELETE DATA { ${} rdf:label ${\"PROP_FROM_ENTITY_SCHEMA2\"} }\n INSERT DATA { ${} rdf:label ${\"PROP_FROM_ENTITY_SCHEMA3\"} }\n```\n\nThis will insert the URI serialization of the property value `PROP_FROM_ENTITY_SCHEMA1` for the\n`${}` expression.\nFurthermore, it will insert a plain literal serialization for the property values `PROP_FROM_ENTITY_SCHEMA2` and\n`PROP_FROM_ENTITY_SCHEMA3` for the template literal expressions.\n\nIt is also possible to write something like `${\"PROP\"}^^` or `${\"PROP\"}@en`. In other words, we\ncan combine variable substitutions with fixed expressions to construct semi-flexible expressions within the template.\n\n### Example of the `Velocity Engine` mode (deprecated)\n\n```\n DELETE DATA { $row.uri(\"PROP_FROM_ENTITY_SCHEMA1\") rdf:label $row.plainLiteral(\"PROP_FROM_ENTITY_SCHEMA2\") }\n #if ( $row.exists(\"PROP_FROM_ENTITY_SCHEMA1\") )\n INSERT DATA { $row.uri(\"PROP_FROM_ENTITY_SCHEMA1\") rdf:label $row.plainLiteral(\"PROP_FROM_ENTITY_SCHEMA3\") }\n #end\n```\n\nInput values are accessible via various methods of the `row` variable (used with `$row`):\n\n- `$row.uri(inputPath: String)`: Renders an input value as **URI**. Throws an exception if the value isn't a valid URI.\n- `$row.plainLiteral(inputPath: String)`: Renders an input value as **plain literal**, i.e. it escapes problematic\n characters, etc.\n- `$row.rawUnsafe(inputPath: String)`: Renders an input value as is, i.e. **no escaping** is done.\n This should **only** be used if the input values can be trusted.\n- `$row.exists(inputPath: String)`: Returns `true` if a value for the input path **exists**, else `false`.\n\nThe methods `uri`, `plainLiteral` and `rawUnsafe` throw an exception if no input value is available for the given\ninput path.\n\nIn addition to input values, properties of the input and output tasks can be accessed via the `inputProperties` and\n`outputProperties` objects. The available keys in these objects are dynamic and correspond exactly to the configuration\nparameters of the tasks connected to the input and output ports of this operator.\n\n- To find the available keys for `$inputProperties`, check the parameter names of the task connected to the input port.\n- To find the available keys for `$outputProperties`, check the parameter names of the task connected to the output port.\n\nFor example, if the connected input task has a parameter named `graph`, you can access it as `$inputProperties.uri(\"graph\")`.\nSimilarly, if the connected output task has a parameter named `endpoint`, you can access it as `$outputProperties.uri(\"endpoint\")`.\n\nBoth `inputProperties` and `outputProperties` support the same methods as the `row` object:\n\n- `uri(inputPath: String)`\n- `plainLiteral(inputPath: String)`\n- `rawUnsafe(inputPath: String)`\n- `exists(inputPath: String)`\n\nFor more information about the Velocity Engine, visit http://velocity.apache.org.\n\n### Internal Specifics\n\nIn contrast to the SPARQL select operator, no `FROM` clause gets injected into the query.\n",
"pluginIcon": "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxzdmcKICAgd2lkdGg9IjMycHgiCiAgIGhlaWdodD0iMzJweCIKICAgdmlld0JveD0iMCAwIDMyIDMyIgogICBmaWxsPSJjdXJyZW50Q29sb3IiCiAgIHZlcnNpb249IjEuMSIKICAgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIgogICB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiCiAgIHhtbG5zOmNjPSJodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9ucyMiCiAgIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyI+CiAgPHRpdGxlPlNQQVJRTCBVcGRhdGUgcXVlcnk8L3RpdGxlPgogIDxwYXRoCiAgICAgZD0iTSA0IDIgTCA0IDEwIEwgMTIgMTAgTCAxMiA4IEwgNy4wNzgxMjUgOCBDIDEwLjM5MTMwMiA0LjI5MTMyODIgMTUuNjUxODE2IDMuMDE0ODM3OSAyMC4yOTY4NzUgNC43OTEwMTU2IEMgMjQuOTQxOTM1IDYuNTY3MTkzMiAyOC4wMDY1NDQgMTEuMDI2OTM4IDI4IDE2IEwgMzAgMTYgQyAzMC4wMDkwMiAxMC4zMDY1NyAyNi41NjQ1OTkgNS4xNzcwNzExIDIxLjI5MTAxNiAzLjAzMTI1IEMgMTYuMDE3NDMzIDAuODg1NDI5IDkuOTY4Njg1OCAyLjE1MjEyNTUgNiA2LjIzNDM3NSBMIDYgMiBMIDQgMiB6IE0gMjMgMTMgQSAzIDMgMCAwIDAgMjAgMTYgQSAzIDMgMCAwIDAgMjAuMjQ2MDk0IDE3LjE5MTQwNiBMIDE2LjgyMjI2NiAyMC42MTUyMzQgQSAzIDMgMCAwIDAgMTUgMjAgQSAzIDMgMCAwIDAgMTIgMjMgQSAzIDMgMCAwIDAgMTUgMjYgQSAzIDMgMCAwIDAgMTcuMTI4OTA2IDI1LjExMzI4MSBMIDIyIDI2Ljk2ODc1IEEgMyAzIDAgMCAxIDIyIDI2Ljk3MDcwMyBBIDMgMyAwIDAgMCAyMiAyNyBBIDMgMyAwIDAgMCAyNSAzMCBBIDMgMyAwIDAgMCAyOCAyNyBBIDMgMyAwIDAgMCAyNS43MTI4OTEgMjQuMDg1OTM4IEwgMjQuODAyNzM0IDE4LjM5ODQzOCBBIDMgMyAwIDAgMCAyNiAxNiBBIDMgMyAwIDAgMCAyMyAxMyB6IE0gMiAxNiBDIDIuMDA4NzEgMjMuNzI4Mzc3IDguMjcxNjIzIDI5Ljk5MTI5MyAxNiAzMCBMIDE2IDI4IEMgOS4zNzU2NDQzIDI3Ljk5MjYxNSA0LjAwNzM4NDggMjIuNjI0MzU2IDQgMTYgTCAyIDE2IHogTSAyMS4xNzc3MzQgMTguMzg0NzY2IEEgMyAzIDAgMCAwIDIzIDE5IEEgMyAzIDAgMCAwIDIzLjM3Njk1MyAxOC45NzY1NjIgTCAyNC4xOTkyMTkgMjQuMTA5Mzc1IEEgMyAzIDAgMCAxIDI0LjE5NzI2NiAyNC4xMDkzNzUgQSAzIDMgMCAwIDAgMjIuMzk0NTMxIDI1LjUxMzY3MiBMIDE3Ljg5MjU3OCAyMy43OTg4MjggQSAzIDMgMCAwIDAgMTggMjMgQSAzIDMgMCAwIDAgMTcuNzUzOTA2IDIxLjgwODU5NCBMIDIxLjE3NzczNCAxOC4zODQ3NjYgeiAiIC8+CiAgPG1ldGFkYXRhPgogICAgPHJkZjpSREY+CiAgICAgIDxjYzpXb3JrCiAgICAgICAgIHJkZjphYm91dD0iIj4KICAgICAgICA8ZGM6dGl0bGU+U1BBUlFMIFVwZGF0ZSBxdWVyeTwvZGM6dGl0bGU+CiAgICAgIDwvY2M6V29yaz4KICAgIDwvcmRmOlJERj4KICA8L21ldGFkYXRhPgo8L3N2Zz4=",
"properties": {
"sparqlUpdateTemplate": {
"name": "sparqlUpdateTemplate",
"title": "SPARQL update query",
- "description": "The SPARQL UPDATE template for constructing SPARQL UPDATE queries for every entity from the input. The possible values for the template engine are `Simple` and `Velocity Engine`. See the general documentation of this plugin for further details on the features of each template engine.",
+ "description": "The SPARQL UPDATE template for constructing SPARQL UPDATE queries for every entity from the input. The possible values for the template engine are `Jinja` (default), `Simple` and `Velocity Engine`. See the general documentation of this plugin for further details on the features of each template engine.",
"type": "string",
"parameterType": "code-sparql",
"value": null,
@@ -6175,17 +6504,23 @@
"templatingMode": {
"name": "templatingMode",
"title": "Templating mode",
- "description": "The templating mode for the template engine. The possible values are `Simple` and `Velocity Engine`. See the general documentation of this plugin for further details on the features of each template engine.",
+ "description": "The templating mode for the template engine. See the general documentation of this plugin for further details on the features of each template engine.",
"type": "string",
- "parameterType": "enumeration",
- "value": "simple",
+ "parameterType": "string",
+ "value": "jinja",
"advanced": false,
"visibleInDialog": true,
"properties": {}
}
},
"properties_advanced": {},
- "actions": {},
+ "actions": {
+ "showPrefixes": {
+ "label": "Show prefixes",
+ "description": "Shows the available namespace prefixes as a SPARQL header that can be copied into the query.",
+ "icon": null
+ }
+ },
"required": [
"sparqlUpdateTemplate"
],
@@ -8298,15 +8633,27 @@
},
"inMemory": {
"pluginId": "inMemory",
- "title": "In-memory dataset",
+ "title": "In-memory Knowledge Graph",
"categories": [
"Embedded"
],
"main_category": "Embedded",
- "description": "A Dataset that holds all data in-memory.",
- "markdownDocumentation": "## 1. Purpose\n\nThe **in-memory dataset** is a small embedded RDF store that keeps all data **in memory** and exposes it via SPARQL. It is intended as a **temporary working graph** inside workflows, not as a large or persistent storage.\n\nTypical use cases:\n- Collecting intermediate results during a workflow run.\n- Storing small lookup graphs used by downstream operators.\n- Testing or prototyping workflows without configuring an external RDF store.\n\n## 2. Behaviour and lifecycle\n\n- The dataset maintains a single in-memory RDF model.\n- All read and write operations go through a SPARQL endpoint over this model.\n- Data exists only **in memory**:\n - It is not persisted to disk by this dataset.\n - After an application restart, the dataset contents are empty again.\n\nWithin a workflow:\n- The dataset can be used as both **input** and **output**:\n - Upstream operators can write triples/entities/links into it.\n - Downstream operators can read from it via SPARQL-based mechanisms.\n\n## 3. Reading data\n\n- When used as a **source**, the dataset exposes its data as a SPARQL endpoint.\n- Queries and retrievals behave like against a normal SPARQL dataset:\n - Entity retrieval, path/type discovery, sampling, etc. are executed via SPARQL.\n- There is no file backing this dataset; everything comes from what has been written into the in-memory model during the lifetime of the process.\n\n## 4. Writing data\n\nThe in-memory dataset accepts RDF data through:\n\n- **Entity sink**\n - Entities written by upstream components are converted to RDF triples and stored in the in-memory model.\n\n- **Link sink**\n - Links are written as RDF triples in the same model.\n\n- **Triple sink**\n - Triples are directly added to the in-memory model via SPARQL operations.\n\nAll three sinks ultimately write into the same in-memory graph; there is no separate physical storage per sink type.\n\n## 5. Configuration\n\n### Clear graph before workflow execution\n\n- **Parameter:** `Clear graph before workflow execution` (boolean)\n- **Default:** `true`\n\nBehaviour:\n\n- If **true**:\n - Before the dataset is used in a workflow execution, the graph is cleared (for writes via this dataset).\n - The workflow sees a **fresh, empty in-memory graph** at the start of the run.\n\n- If **false**:\n - Existing data in the in-memory graph is **preserved** when the workflow starts.\n - New data is added on top of whatever is already stored in the model.\n\nThis parameter controls whether the dataset behaves as a **fresh scratch graph per workflow run** or as a **longer-lived in-memory graph** within the lifetime of the running application.\n\n## 6. Limitations and recommendations\n\n- **Memory-bound**\n - All data is kept in memory; large graphs will increase memory usage and may impact performance.\n - For large or production RDF graphs, use an external RDF store and a SPARQL dataset instead.\n\n- **No persistence**\n - Contents are lost when the application/server is restarted.\n - Do not treat this dataset as long-term storage.\n\n- **Scope**\n - Best suited for:\n - small to medium intermediate results,\n - testing and prototyping,\n - temporary data that can be regenerated by re-running workflows.\n\n## 7. Example usage scenarios\n\n- Use as a **temporary integration graph**:\n - Multiple sources write into the in-memory dataset.\n - A downstream SPARQL-based operator queries the combined graph.\n\n- Use as a **scratch area for experimentation**:\n - Quickly test mapping or linking logic by writing output into the in-memory dataset.\n - Inspect the result via SPARQL without configuring an external endpoint.\n\n- Use as a **small lookup store**:\n - Preload a small set of reference triples (e.g. codes or mappings).\n - Let workflows query these during execution.\n",
+ "description": "A dataset that holds all data in-memory. In the default (workflow-scoped) mode, data is isolated per workflow execution and shared with nested workflows that reference the same dataset task. In application-scoped mode, data persists for the lifetime of the running process.",
+ "markdownDocumentation": "## 1. Purpose\n\nThe **in-memory dataset** is a small embedded RDF store that keeps all data **in memory** and exposes it via SPARQL. It is intended as a **temporary working graph** inside workflows, not as a large or persistent storage.\n\nTypical use cases:\n- Collecting intermediate results during a workflow run.\n- Storing small lookup graphs used by downstream operators.\n- Testing or prototyping workflows without configuring an external RDF store.\n\n## 2. Behaviour and lifecycle\n\nThe dataset maintains a single in-memory RDF model and exposes it via a SPARQL endpoint. Two lifecycle modes are available, controlled by the `workflowScoped` parameter:\n\n**Workflow-scoped mode** (default, `workflowScoped = true`):\n- A separate model is created for each workflow execution.\n- Concurrent workflow executions are fully isolated from each other.\n- A dataset task in a nested workflow shares the same model as the parent workflow for the same task identifier. Data written by the parent is available in the nested workflow and vice versa.\n- If the dataset is read from outside a workflow context, the data from the most recently started executor is returned.\n- When the workflow execution ends, the per-execution data is removed automatically.\n\n**Application-scoped mode** (`workflowScoped = false`):\n- A single shared model is created when the dataset is instantiated.\n- Data persists for the lifetime of the running application process.\n- All workflow executions share the same in-memory graph.\n- After an application restart, the dataset contents are empty again.\n\n## 3. Reading data\n\n- When used as a **source**, the dataset exposes its data as a SPARQL endpoint.\n- Queries and retrievals behave like against a normal SPARQL dataset:\n - Entity retrieval, path/type discovery, sampling, etc. are executed via SPARQL.\n- There is no file backing this dataset; everything comes from what has been written into the in-memory model during the lifetime of the process (application-scoped) or the workflow execution (workflow-scoped).\n\n## 4. Writing data\n\nThe in-memory dataset accepts RDF data through:\n\n- **Entity sink**\n - Entities written by upstream components are converted to RDF triples and stored in the in-memory model.\n\n- **Link sink**\n - Links are written as RDF triples in the same model.\n\n- **Triple sink**\n - Triples are directly added to the in-memory model via SPARQL operations.\n\nAll three sinks ultimately write into the same in-memory graph; there is no separate physical storage per sink type.\n\n## 5. Configuration\n\n### Workflow scoped \n\n- **Parameter:** `workflowScoped` (boolean)\n- **Default:** `true`\n\nWhen `true` (default, workflow-scoped mode):\n- Data is stored in a separate in-memory graph for each workflow execution.\n- Concurrent workflow executions are fully isolated from each other.\n- A dataset task in a nested workflow shares the same graph as the parent for the same task identifier. Data written by the parent is available in the nested workflow and vice versa.\n- If the dataset is read from outside a workflow context, the data of the workflow execution that most recently accessed this dataset is returned.\n- When the workflow execution ends, the per-execution data is removed automatically.\n\nWhen `false` (application-scoped mode):\n- Data persists in a single shared graph for the lifetime of the running process.\n- All workflow executions share the same graph.\n\n### Clear graph before workflow execution \n\n- **Parameter:** `clearGraphBeforeExecution` (boolean, **deprecated**)\n- **Default:** `false`\n\nThis parameter is deprecated. Use the **Clear dataset** operator in the workflow instead.\n\nBehaviour (application-scoped mode only):\n\n- If **true**:\n - Before the dataset is used in a workflow execution, the graph is cleared.\n - The workflow sees a **fresh, empty in-memory graph** at the start of the run.\n\n- If **false**:\n - Existing data in the in-memory graph is **preserved** when the workflow starts.\n - New data is added on top of whatever is already stored in the model.\n\nThis parameter has no effect when `workflowScoped = true` (the executor manages the lifecycle).\n\n## 6. Limitations and recommendations\n\n- **Memory-bound**\n - All data is kept in memory; large graphs will increase memory usage and may impact performance.\n - For large or production RDF graphs, use an external RDF store and a SPARQL dataset instead.\n - A size limit is enforced: once the estimated size of data written to the dataset exceeds the value of `org.silkframework.runtime.resource.Resource.maxInMemorySize`, the workflow fails with an error. This prevents the JVM from running out of heap memory.\n\n- **No persistence**\n - Contents are lost when the application/server is restarted.\n - Do not treat this dataset as long-term storage.\n\n- **SPARQL engine**\n - The dataset is backed by [Apache Jena](https://jena.apache.org/), exposed through a Jena in-memory SPARQL endpoint.\n\n- **No named-graph support**\n - Only the **default graph** is available. Writing triples into a named graph is not possible.\n\n- **Scope**\n - Best suited for:\n - small to medium intermediate results,\n - testing and prototyping,\n - temporary data that can be regenerated by re-running workflows.\n\n## 7. Example usage scenarios\n\n- Use as a **temporary integration graph** (application-scoped):\n - Multiple sources write into the in-memory dataset.\n - A downstream SPARQL-based operator queries the combined graph.\n\n- Use as a **scratch area for experimentation** (application-scoped):\n - Quickly test mapping or linking logic by writing output into the in-memory dataset.\n - Inspect the result via SPARQL without configuring an external endpoint.\n\n- Use as a **small lookup store** (application-scoped):\n - Preload a small set of reference triples (e.g. codes or mappings).\n - Let workflows query these during execution.\n\n- Use as a **workflow-local intermediate store** (workflow-scoped):\n - Multiple operators in a single workflow run write intermediate RDF results.\n - Downstream operators in the same run read from the dataset without affecting parallel runs.\n\n- Use in **nested workflows** (workflow-scoped):\n - A parent workflow writes data into a workflow-scoped dataset.\n - A nested sub-workflow reads and enriches the same data.\n - After the nested workflow completes, the parent can read the enriched result.\n",
"pluginIcon": null,
- "properties": {},
+ "properties": {
+ "workflowScoped": {
+ "name": "workflowScoped",
+ "title": "Workflow-scoped",
+ "description": "If true (default), data is isolated per workflow execution and cleared after the execution ends, sharing data with nested workflows that reference the same dataset task. If false, data persists for the lifetime of the application process.",
+ "type": "string",
+ "parameterType": "boolean",
+ "value": "true",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
"properties_advanced": {
"clearGraphBeforeExecution": {
"name": "clearGraphBeforeExecution",
@@ -8406,7 +8753,7 @@
],
"main_category": "File",
"description": "Read from or write to a JSON or JSON Lines file.",
- "markdownDocumentation": "Typically, this dataset is used to transform an JSON file to another format, e.g., to RDF.\n\n## Reading\n\nIn addition to plain JSON files, *JSON Lines* files can also be read.\n\nFor reading, the JSON dataset supports a number of special paths:\n- `#id` Is a special syntax for generating an id for a selected element. It can be used in URI patterns for entities which do not provide an identifier. Examples: `http://example.org/{#id}` or `http://example.org/{/pathToEntity/#id}`.\n- `#text` retrieves the text of the selected node.\n- The backslash can be used to navigate to the parent JSON node, e.g., `\\parent/key`. The name of the backslash key (here `parent`) is ignored.\n\n## Writing\n\nWhen writing JSON, all entities need to possess a unique URI. Writing multiple root entities with the same URI will result in multiple entries in the generated JSON. If multiple nested entities with the same URI are written, only the last entity with a given URI will be written.\n",
+ "markdownDocumentation": "Typically, this dataset is used to transform an JSON file to another format, e.g., to RDF.\n\n## Reading\n\nIn addition to plain JSON files, *JSON Lines* files can also be read.\n\nFor reading, the JSON dataset supports a number of special paths:\n- `#id` is a special syntax for generating a hash-based id for a selected element. It can be used in URI patterns for entities which do not provide an identifier. Examples: `http://example.org/{#id}` or `http://example.org/{/pathToEntity/#id}`. If no URI pattern is configured, the dataset instead generates the default entity URI from the JSON source location.\n- `#text` retrieves the text of the selected node.\n- The backslash can be used to navigate to the parent JSON node, e.g., `\\parent/key`. The name of the backslash key (here `parent`) is ignored.\n\n## Writing\n\nWhen writing JSON, all entities need to possess a unique URI. Writing multiple root entities with the same URI will result in multiple entries in the generated JSON. If multiple nested entities with the same URI are written, only the last entity with a given URI will be written.\n",
"pluginIcon": null,
"properties": {
"file": {
@@ -8511,7 +8858,12 @@
"Dataset"
],
"pluginType": "dataset",
- "relatedPlugins": []
+ "relatedPlugins": [
+ {
+ "id": "JsonParserOperator",
+ "description": "The JSON dataset is a pipeline source: it opens a file and needs no upstream input. Parse JSON is an operator: it reads a JSON string from a field value supplied by an upstream entity."
+ }
+ ]
},
"eccencaDataPlatform": {
"pluginId": "eccencaDataPlatform",
@@ -8527,10 +8879,10 @@
"graph": {
"name": "graph",
"title": "Graph",
- "description": "The URI of the named graph.",
+ "description": "The URI of the named graph. If left empty, the default graph is used.",
"type": "string",
"parameterType": "graph uri",
- "value": null,
+ "value": "",
"advanced": false,
"visibleInDialog": true,
"properties": {}
@@ -8660,9 +9012,7 @@
}
},
"actions": {},
- "required": [
- "graph"
- ],
+ "required": [],
"distanceMeasureRange": null,
"backendType": "native",
"is_deprecated": false,
@@ -8863,7 +9213,7 @@
],
"main_category": "Remote",
"description": "Neo4j graph",
- "markdownDocumentation": "\nSupports reading and writing Neo4j graphs. The following sections outline how graphs are generated and read back.\n\nFor more information about Neo4j, please refer to the [Neo4j documentation](https://neo4j.com/docs/).\n\n### Nodes\n\nFor each entity that is written to a Neo4j dataset, a _node_ will be created.\nA property `uri` will be added to each generated node, which holds the URI of the original entity.\nIn applications, the URI property should be used instead of the node identifiers, which are auto-generated in Neo4j and do not represent stable URIs.\n\nWhen reading nodes, the entity URIs will be generated based on that property.\nAt the moment, it's not supported to read nodes that do not provide a `uri` property.\n\n### Labels\n\n_Labels_ in Neo4j are used to group nodes into sets where all nodes that have a certain _label_ belongs to the same set.\nNeo4j _labels_ are comparable with _classes_ in RDF (not to be confused with labels in RDF).\n\nWhen writing entities to the Neo4j dataset, the following _labels_ will be added to each generated node:\n\n- For each entity _type_ (such as the _type_ set in a mapping), a _label_ will be added to the node in Neo4j.\n Since _types_ in eccenca DataIntegration are usually URIs, they will be converted according to the rules further down.\n- The _label_ as configured by the _label_ parameter on the Neo4j dataset itself.\n This is typically used to identify all entities that have been written by a certain Neo4j dataset specification in the project.\n For instance, if two Neo4j dataset specifications are added to a project - both writing to the same Neo4j database - different labels can be set to distinguish both sets of entities.\n In that respect it may be used to model a similar concept as _graphs_ in RDF.\n\n### Relationships\n\nA relationship connects two nodes in Neo4j.\nHierarchical mappings will generate relationships for all object mappings.\n\nRelationships can be addressed with property paths in mappings.\nAt the moment, only paths of length 1 are supported, i.e., it's not possible to use non-property paths.\n\n### Handling of URIs\n\nIn eccenca DataIntegration, URIs are typically used to uniquely identify classes and properties.\nWhile URIs are central in RDF, Neo4j does allow arbitrary names and does not have any special support for URIs.\n\nWhen generating Neo4j labels, properties and relationships, URIs will be shortened according to the following rules.\n- If a registered project prefix matches a URI, a name `{prefixName}_{localPart}` will be generated. For instance, `http://xmlns.com/foaf/0.1/name` will become `foaf_name`.\n Note that underscores (`_`) are used instead of colons (`:`) to separate the namespace and the local name.\n The reason is that colons are reserved in the Cypher query language and some tools don't escape properly and fail on databases that use colons in names.\n- If no project prefix matches a URI, the URI will be used verbatim. This will look ugly in Neo4j tools, so generally it's recommended to define prefixes for all used namespaces.\n\nWhen reading generated entities, the URIs of the classes and properties will be reconstructed based on the prefix table of the project. If the prefixes change between writing and reading, different URIs will be generated.\n\n### RDF vs. Neo4j terminology\n\nNeo4j uses a different terminology than RDF or description logic.\nFor users familiar with RDF, the following table shows the correspondent terms for some central concepts.\nThis is meant to help understanding and does not aim to provide a precise mapping as there are semantic differences between Neo4j and RDF.\n\n| RDF | Neo4j |\n| --- |--- |\n| resource | node |\n| class | label |\n| datatype property | property |\n| object property | relationship |\n| graph | Do not exist in Neo4j, but labels can be used to mimic graphs. |\n",
+ "markdownDocumentation": "\nSupports reading and writing Neo4j graphs. The following sections outline how graphs are generated and read back.\n\nFor more information about Neo4j, please refer to the [Neo4j documentation](https://neo4j.com/docs/).\n\n### Nodes\n\nFor each entity that is written to a Neo4j dataset, a _node_ will be created.\nA property `uri` will be added to each generated node, which holds the URI of the original entity.\nIn applications, the URI property should be used instead of the node identifiers, which are auto-generated in Neo4j and do not represent stable URIs.\n\nWhen reading nodes, the entity URIs will be generated based on that property.\nNodes that do not provide a `uri` property are read with a generated entity URI, which is not stable across reads.\n\n### Labels\n\n_Labels_ in Neo4j are used to group nodes into sets where all nodes that have a certain _label_ belongs to the same set.\nNeo4j _labels_ are comparable with _classes_ in RDF (not to be confused with labels in RDF).\n\nWhen writing entities to the Neo4j dataset, the following _labels_ will be added to each generated node:\n\n- For each entity _type_ (such as the _type_ set in a mapping), a _label_ will be added to the node in Neo4j.\n Since _types_ in eccenca DataIntegration are usually URIs, they will be converted according to the rules further down.\n- The _label_ as configured by the _label_ parameter on the Neo4j dataset itself.\n This is typically used to identify all entities that have been written by a certain Neo4j dataset specification in the project.\n For instance, if two Neo4j dataset specifications are added to a project - both writing to the same Neo4j database - different labels can be set to distinguish both sets of entities.\n In that respect it may be used to model a similar concept as _graphs_ in RDF.\n\n### Relationships\n\nA relationship connects two nodes in Neo4j.\nHierarchical mappings will generate relationships for all object mappings.\n\nRelationships can be addressed with property paths in mappings.\nAt the moment, only paths of length 1 are supported, i.e., it's not possible to use non-property paths.\n\n### Handling of URIs\n\nIn eccenca DataIntegration, URIs are typically used to uniquely identify classes and properties.\nWhile URIs are central in RDF, Neo4j does allow arbitrary names and does not have any special support for URIs.\n\nWhen generating Neo4j labels, properties and relationships, URIs will be shortened according to the following rules.\n- If a registered project prefix matches a URI, a name `{prefixName}_{localPart}` will be generated. For instance, `http://xmlns.com/foaf/0.1/name` will become `foaf_name`.\n Note that underscores (`_`) are used instead of colons (`:`) to separate the namespace and the local name.\n The reason is that colons are reserved in the Cypher query language and some tools don't escape properly and fail on databases that use colons in names.\n- If no project prefix matches a URI, the URI will be used verbatim. This will look ugly in Neo4j tools, so generally it's recommended to define prefixes for all used namespaces.\n\nWhen reading generated entities, the URIs of the classes and properties will be reconstructed based on the prefix table of the project. If the prefixes change between writing and reading, different URIs will be generated.\n\n### RDF vs. Neo4j terminology\n\nNeo4j uses a different terminology than RDF or description logic.\nFor users familiar with RDF, the following table shows the correspondent terms for some central concepts.\nThis is meant to help understanding and does not aim to provide a precise mapping as there are semantic differences between Neo4j and RDF.\n\n| RDF | Neo4j |\n| --- |--- |\n| resource | node |\n| class | label |\n| datatype property | property |\n| object property | relationship |\n| graph | Do not exist in Neo4j, but labels can be used to mimic graphs. |\n",
"pluginIcon": null,
"properties": {
"uri": {
@@ -8933,6 +9283,17 @@
"advanced": true,
"visibleInDialog": true,
"properties": {}
+ },
+ "writeBatchSize": {
+ "name": "writeBatchSize",
+ "title": "Write batch size",
+ "description": "The number of entities to write in a single Neo4j transaction. Reduce this value if you encounter Neo4j transaction memory limits (dbms.memory.transaction.total.max).",
+ "type": "string",
+ "parameterType": "int",
+ "value": "1000",
+ "advanced": true,
+ "visibleInDialog": true,
+ "properties": {}
}
},
"actions": {},
@@ -12416,6 +12777,50 @@
"pluginType": "transformer",
"relatedPlugins": []
},
+ "inputHash": {
+ "pluginId": "inputHash",
+ "title": "Combined input hash",
+ "categories": [
+ "Value"
+ ],
+ "main_category": "Value",
+ "description": "Calculates a single hash value covering all input values combined, across all input ports. Values are fed into the hash function in port order without any separator between them.",
+ "markdownDocumentation": "The **Combined input hash** operator produces exactly one hash value covering all input values combined, across all connected input ports. However many values arrive and however many ports are connected, the output is always a single string.\n\n## How combining works\n\nAll values from all input ports are fed sequentially into a single hash function \u2014 port 1 first, then port 2, and so on. Within each port, values are processed in the order they arrive. No separator is inserted between values or between ports. The hash covers the concatenated byte content of all values in that traversal order.\n\nThis means the result depends on both the content and the order of values. The same set of values in a different order produces a different hash. Connecting one port with values `[\"apple\", \"banana\"]` produces the same hash as connecting two ports with `[\"apple\"]` and `[\"banana\"]` respectively, because the bytes are fed in the same sequence either way.\n\n## Output\n\nThe output is a single lowercase hexadecimal string. The length depends on the algorithm: 64 characters for SHA-256, 32 for MD5, 40 for SHA-1, 96 for SHA-384, 128 for SHA-512. If the input is empty, the output is the hash of an empty message.\n\nValues are encoded as UTF-8 before hashing.\n\n## Algorithm parameter\n\nThe algorithm parameter selects the hash function. The default is SHA-256. The following algorithms from the [SPARQL 1.1 specification](https://www.w3.org/TR/sparql11-query/#func-hash) are supported:\n\n| SPARQL name | Java name | Notes |\n|-------------|-----------|-------|\n| MD5 | MD5 | Weak \u2014 vulnerable to collision attacks. Avoid for security-sensitive use. |\n| SHA1 | SHA-1 | Weak \u2014 deprecated for most security purposes. |\n| SHA256 | SHA-256 | Recommended default. |\n| SHA384 | SHA-384 | Stronger than SHA-256. |\n| SHA512 | SHA-512 | Strongest in the SPARQL set. |\n\nAdditional algorithms available on the JVM (such as SHA-512/256 and SHA-3 variants) are also accepted. The full list is JVM-dependent and visible in the algorithm parameter dropdown.\n\nNote that the Java names use hyphens (SHA-256, SHA-1) where SPARQL uses none (SHA256, SHA1). Both forms are accepted by this operator.\n\n## Examples\n\n**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values \"first\" and \"second\".\n\n---\n**A single input value produces one combined SHA-256 hash:**\n\n* Input values:\n 1. `[input value]`\n\n* Returns: `[f708c2afff0ed197e8551c4dd549ee5b848e0b407106cbdb8e451c8cd1479362]`\n\n\n---\n**Multiple values on one input are combined into a single hash:**\n\n* Input values:\n 1. `[apple, banana]`\n\n* Returns: `[5b692305517af54eb5ae12b9ff89eaf89e31f6a6ee208365886a18b81a2fc2f8]`\n\n\n---\n**Reversing the value order produces a different hash, confirming order-sensitivity:**\n\n* Input values:\n 1. `[banana, apple]`\n\n* Returns: `[d4183362b538440bb9a5f82359791c647280e6b657a1812f16f7bcc2b8f141ca]`\n\n\n---\n**Values from multiple ports are combined in port order, producing the same hash as the equivalent single-port sequence:**\n\n* Input values:\n 1. `[apple]`\n 2. `[banana]`\n\n* Returns: `[5b692305517af54eb5ae12b9ff89eaf89e31f6a6ee208365886a18b81a2fc2f8]`\n\n\n---\n**The algorithm parameter selects the hash function (MD5):**\n\n* Parameters\n * algorithm: `MD5`\n\n* Input values:\n 1. `[input value]`\n\n* Returns: `[cee963a28f70ee97751a85ef732e66dd]`\n\n\n---\n**The algorithm parameter selects the hash function (SHA-1):**\n\n* Parameters\n * algorithm: `SHA-1`\n\n* Input values:\n 1. `[apple]`\n\n* Returns: `[d0be2dc421be4fcd0172e5afceea3970e2f3d940]`\n\n\n---\n**The algorithm parameter selects the hash function (SHA-384):**\n\n* Parameters\n * algorithm: `SHA-384`\n\n* Input values:\n 1. `[apple]`\n\n* Returns: `[3d8786fcb588c93348756c6429717dc6c374a14f7029362281a3b21dc10250ddf0d0578052749822eb08bc0dc1e68b0f]`\n\n\n---\n**The algorithm parameter selects the hash function (SHA-512):**\n\n* Parameters\n * algorithm: `SHA-512`\n\n* Input values:\n 1. `[apple]`\n\n* Returns: `[844d8779103b94c18f4aa4cc0c3b4474058580a991fba85d3ca698a0bc9e52c5940feb7a65a3a290e17e6b23ee943ecc4f73e7490327245b4fe5d5efb590feb2]`\n\n\n---\n**Empty input produces the hash of an empty message:**\n\n* Input values:\n 1. `[]`\n\n* Returns: `[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855]`\n\n\n---\n**Empty algorithm string causes IllegalArgumentException:**\n\n* Parameters\n * algorithm: ``\n\n* Input values:\n 1. `[foo]`\n\n* Returns: `[]`\n* **Throws error:** `IllegalArgumentException`\n\n\n",
+ "pluginIcon": null,
+ "properties": {
+ "algorithm": {
+ "name": "algorithm",
+ "title": "Algorithm",
+ "description": "The hash algorithm to be used.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "SHA256",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "TransformOperator"
+ ],
+ "pluginType": "transformer",
+ "relatedPlugins": [
+ {
+ "id": "perValueHash",
+ "description": "The Per-value hash plugin hashes each input value independently and returns one hash per value, preserving cardinality. The Combined input hash plugin instead feeds all values into a single hash function, producing one combined hash regardless of input size."
+ },
+ {
+ "id": "mapWithDefaultInput",
+ "description": "One hash value is produced for the entire set of inputs by the Combined input hash plugin. The Map with default plugin instead keeps a value sequence and rewrites it position by position through the mapping, falling back to the second input where no mapping entry is found."
+ }
+ ]
+ },
"compareDates": {
"pluginId": "compareDates",
"title": "Compare dates",
@@ -12840,7 +13245,7 @@
"description": "Set date (e.g.YYYY-MM-DD) to convert currencies based on historic rates.",
"type": "string",
"parameterType": "string",
- "value": "2026-05-12",
+ "value": "2026-08-04",
"advanced": false,
"visibleInDialog": true,
"properties": {}
@@ -13587,6 +13992,52 @@
"pluginType": "transformer",
"relatedPlugins": []
},
+ "escape_multiline_literal": {
+ "pluginId": "escape_multiline_literal",
+ "title": "Escape SPARQL multiline literal",
+ "categories": [
+ "SPARQL"
+ ],
+ "main_category": "SPARQL",
+ "description": "Escapes a value so it can be safely used inside a SPARQL triple-quoted string literal (`\"\"\"...\"\"\"` or `'''...'''`). Escapes backslashes and breaks any run of three or more consecutive single or double quotes. Individual quotes and newlines are preserved. The returned value does not include enclosing quotation marks.",
+ "markdownDocumentation": "Escapes a value so it can be safely used inside a SPARQL triple-quoted string literal (`\"\"\"...\"\"\"` or `'''...'''`). Escapes backslashes and breaks any run of three or more consecutive single or double quotes. Individual quotes and newlines are preserved. The returned value does not include enclosing quotation marks.\n\n## Examples\n\n**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values \"first\" and \"second\".\n\n---\n**Example 1:**\n\n* Input values:\n 1. \n ```\n [simple\n value]\n ```\n\n* Returns: \n ```\n [simple\n value]\n ```\n\n\n---\n**Example 2:**\n\n* Input values:\n 1. `[with \"quote\"]`\n\n* Returns: `[with \"quote\"]`\n\n\n---\n**Example 3:**\n\n* Input values:\n 1. `[back\\slash]`\n\n* Returns: `[back\\\\slash]`\n\n\n---\n**Example 4:**\n\n* Input values:\n 1. `[triple \"\"\" quotes]`\n\n* Returns: `[triple \\\"\\\"\\\" quotes]`\n\n\n---\n**Example 5:**\n\n* Input values:\n 1. `[triple ''' quotes]`\n\n* Returns: `[triple \\'\\'\\' quotes]`\n\n\n",
+ "pluginIcon": null,
+ "properties": {},
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "TransformOperator"
+ ],
+ "pluginType": "transformer",
+ "relatedPlugins": []
+ },
+ "escape_literal": {
+ "pluginId": "escape_literal",
+ "title": "Escape SPARQL plain literal",
+ "categories": [
+ "SPARQL"
+ ],
+ "main_category": "SPARQL",
+ "description": "Escapes a value so it can be safely used inside a SPARQL short-form string literal. Escapes backslashes, quotes, newlines, carriage returns and tabs. The returned value does not include enclosing quotation marks.",
+ "markdownDocumentation": "Escapes a value so it can be safely used inside a SPARQL short-form string literal. Escapes backslashes, quotes, newlines, carriage returns and tabs. The returned value does not include enclosing quotation marks.\n\n## Examples\n\n**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values \"first\" and \"second\".\n\n---\n**Example 1:**\n\n* Input values:\n 1. `[simple value]`\n\n* Returns: `[simple value]`\n\n\n---\n**Example 2:**\n\n* Input values:\n 1. `[with \"quotes\"]`\n\n* Returns: `[with \\\"quotes\\\"]`\n\n\n---\n**Example 3:**\n\n* Input values:\n 1. `[back\\slash]`\n\n* Returns: `[back\\\\slash]`\n\n\n---\n**Example 4:**\n\n* Input values:\n 1. \n ```\n [line1\n line2]\n ```\n\n* Returns: `[line1\\nline2]`\n\n\n",
+ "pluginIcon": null,
+ "properties": {},
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "TransformOperator"
+ ],
+ "pluginType": "transformer",
+ "relatedPlugins": []
+ },
"TemplateTransformer": {
"pluginId": "TemplateTransformer",
"title": "Evaluate template",
@@ -14628,46 +15079,6 @@
"pluginType": "transformer",
"relatedPlugins": []
},
- "inputHash": {
- "pluginId": "inputHash",
- "title": "Input hash",
- "categories": [
- "Value"
- ],
- "main_category": "Value",
- "description": "Calculates the hash sum of the input values. Generates a single hash sum for all input values combined.",
- "markdownDocumentation": "Calculates the hash sum of the input values. Generates a single hash sum for all input values combined.\nThis operator supports using different hash algorithms from the [Secure Hash Algorithms family](https://en.wikipedia.org/wiki/Secure_Hash_Algorithms) (SHA, e.g. SHA256) and two algorithms from the [Message-Digest Algorithm family](https://en.wikipedia.org/wiki/MD5) (MD2 / MD5). Please be aware that some of these algorithms are not secure due the possibility of collision attacks and other attacks.\n\n## Examples\n\n**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values \"first\" and \"second\".\n\n---\n**Example 1:**\n\n* Input values:\n 1. `[input value]`\n\n* Returns: `[f708c2afff0ed197e8551c4dd549ee5b848e0b407106cbdb8e451c8cd1479362]`\n\n\n",
- "pluginIcon": null,
- "properties": {
- "algorithm": {
- "name": "algorithm",
- "title": "Algorithm",
- "description": "The hash algorithm to be used.",
- "type": "string",
- "parameterType": "string",
- "value": "SHA256",
- "advanced": false,
- "visibleInDialog": true,
- "properties": {}
- }
- },
- "properties_advanced": {},
- "actions": {},
- "required": [],
- "distanceMeasureRange": null,
- "backendType": "native",
- "is_deprecated": false,
- "tags": [
- "TransformOperator"
- ],
- "pluginType": "transformer",
- "relatedPlugins": [
- {
- "id": "mapWithDefaultInput",
- "description": "One hash value is produced for the entire set of inputs by the Input hash plugin. The Map with default plugin instead keeps a value sequence and rewrites it position by position through the mapping, falling back to the second input where no mapping entry is found."
- }
- ]
- },
"inputTaskAttributes": {
"pluginId": "inputTaskAttributes",
"title": "Input task attributes",
@@ -16662,6 +17073,46 @@
"pluginType": "transformer",
"relatedPlugins": []
},
+ "perValueHash": {
+ "pluginId": "perValueHash",
+ "title": "Per-value hash",
+ "categories": [
+ "Value"
+ ],
+ "main_category": "Value",
+ "description": "Hashes each input value independently and returns one hash per value. Accepts exactly one input port.",
+ "markdownDocumentation": "The **Per-value hash** operator hashes each input value independently and returns one hash per value. The output count always equals the input count \u2014 cardinality is preserved.\n\n## SPARQL alignment\n\nThis operator produces the same output as the SPARQL 1.1 hash functions applied per value. For a single input value, `SHA256(?x)` in SPARQL returns the same result as this operator with the default SHA256 algorithm.\n\n## Single-input constraint\n\nThe operator accepts exactly one input port. Connecting more than one port throws an `IllegalArgumentException`. This constraint exists because per-value hashing is defined relative to a single value sequence \u2014 combining values across ports would require choosing a port-merging strategy, which is the behaviour of the **Combined input hash** operator instead.\n\n## Output\n\nEach input value produces one lowercase hexadecimal hash string. The output order matches the input order. If the input is empty, the output is empty \u2014 no hash is produced.\n\nValues are encoded as UTF-8 before hashing.\n\n## Algorithm parameter\n\nThe algorithm parameter selects the hash function. The default is SHA-256. The five algorithms from the [SPARQL 1.1 specification](https://www.w3.org/TR/sparql11-query/#func-hash) are supported:\n\n| SPARQL name | Java name | Notes |\n|-------------|-----------|-------|\n| MD5 | MD5 | Weak \u2014 vulnerable to collision attacks. Avoid for security-sensitive use. |\n| SHA1 | SHA-1 | Weak \u2014 deprecated for most security purposes. |\n| SHA256 | SHA-256 | Recommended default. |\n| SHA384 | SHA-384 | Stronger than SHA-256. |\n| SHA512 | SHA-512 | Strongest in the SPARQL set. |\n\nAdditional algorithms available on the JVM are also accepted. The full list is JVM-dependent and visible in the algorithm parameter dropdown.\n\nNote that the Java names use hyphens (SHA-256, SHA-1) where SPARQL uses none (SHA256, SHA1). Both forms are accepted by this operator.\n\n## Contrast with Combined input hash\n\nThe **Combined input hash** operator feeds all values from all ports into a single hash function and returns one hash regardless of input size. Use it when you need a single fingerprint for a set of values taken together.\n\nUse **Per-value hash** when each value needs its own hash \u2014 for example, to hash a column of URIs independently, or to replicate `SHA256(?x)` in SPARQL.\n\n## Examples\n\n**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values \"first\" and \"second\".\n\n---\n**Single value produces one SHA-256 hash:**\n\n* Input values:\n 1. `[input value]`\n\n* Returns: `[f708c2afff0ed197e8551c4dd549ee5b848e0b407106cbdb8e451c8cd1479362]`\n\n\n---\n**Two values in, two independent hashes out \u2014 one per value, not a combined hash:**\n\n* Input values:\n 1. `[apple, banana]`\n\n* Returns: `[3a7bd3e2360a3d29eea436fcfb7e44c735d117c42d1c1835420b6b9942dd4f1b, b493d48364afe44d11c0165cf470a4164d1e2609911ef998be868d46ade3de4e]`\n\n\n---\n**The algorithm parameter selects the hash function (MD5), single value:**\n\n* Parameters\n * algorithm: `MD5`\n\n* Input values:\n 1. `[apple]`\n\n* Returns: `[1f3870be274f6c49b3e31a0c6728957f]`\n\n\n---\n**The algorithm parameter selects the hash function (MD5), multiple values:**\n\n* Parameters\n * algorithm: `MD5`\n\n* Input values:\n 1. `[apple, banana]`\n\n* Returns: `[1f3870be274f6c49b3e31a0c6728957f, 72b302bf297a228a75730123efef7c41]`\n\n\n---\n**The algorithm parameter selects the hash function (SHA-1):**\n\n* Parameters\n * algorithm: `SHA-1`\n\n* Input values:\n 1. `[apple]`\n\n* Returns: `[d0be2dc421be4fcd0172e5afceea3970e2f3d940]`\n\n\n---\n**The algorithm parameter selects the hash function (SHA-384):**\n\n* Parameters\n * algorithm: `SHA-384`\n\n* Input values:\n 1. `[apple]`\n\n* Returns: `[3d8786fcb588c93348756c6429717dc6c374a14f7029362281a3b21dc10250ddf0d0578052749822eb08bc0dc1e68b0f]`\n\n\n---\n**The algorithm parameter selects the hash function (SHA-512):**\n\n* Parameters\n * algorithm: `SHA-512`\n\n* Input values:\n 1. `[apple]`\n\n* Returns: `[844d8779103b94c18f4aa4cc0c3b4474058580a991fba85d3ca698a0bc9e52c5940feb7a65a3a290e17e6b23ee943ecc4f73e7490327245b4fe5d5efb590feb2]`\n\n\n---\n**Empty input produces empty output:**\n\n* Input values:\n 1. `[]`\n\n* Returns: `[]`\n\n\n---\n**Two input ports causes IllegalArgumentException:**\n\n* Input values:\n 1. `[foo]`\n 2. `[bar]`\n\n* Returns: `[]`\n* **Throws error:** `IllegalArgumentException`\n\n\n---\n**Invalid algorithm name causes NoSuchAlgorithmException:**\n\n* Parameters\n * algorithm: `NONEXISTENT`\n\n* Input values:\n 1. `[foo]`\n\n* Returns: `[]`\n* **Throws error:** `NoSuchAlgorithmException`\n\n\n---\n**Empty algorithm string causes IllegalArgumentException:**\n\n* Parameters\n * algorithm: ``\n\n* Input values:\n 1. `[foo]`\n\n* Returns: `[]`\n* **Throws error:** `IllegalArgumentException`\n\n\n",
+ "pluginIcon": null,
+ "properties": {
+ "algorithm": {
+ "name": "algorithm",
+ "title": "Algorithm",
+ "description": "The hash algorithm to be used.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "SHA256",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "TransformOperator"
+ ],
+ "pluginType": "transformer",
+ "relatedPlugins": [
+ {
+ "id": "inputHash",
+ "description": "The Combined input hash plugin produces one combined hash for all input values. The Per-value hash plugin instead hashes each value independently, preserving cardinality."
+ }
+ ]
+ },
"Excel_PERCENTILE": {
"pluginId": "Excel_PERCENTILE",
"title": "Percentile",
@@ -17150,6 +17601,53 @@
"pluginType": "transformer",
"relatedPlugins": []
},
+ "cmem_plugin_random-GenerateValues": {
+ "pluginId": "cmem_plugin_random-GenerateValues",
+ "title": "Random value",
+ "categories": [
+ "Uncategorized"
+ ],
+ "main_category": "Uncategorized",
+ "description": "Generates random values.",
+ "markdownDocumentation": null,
+ "pluginIcon": "data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjwhLS0gQ3JlYXRlZCB3aXRoIElua3NjYXBlIChodHRwOi8vd3d3Lmlua3NjYXBlLm9yZy8pIC0tPgoKPHN2ZwogICB3aWR0aD0iOTAuODU4NjA0bW0iCiAgIGhlaWdodD0iOTcuMjEwOG1tIgogICB2aWV3Qm94PSIwIDAgOTAuODU4NjA1IDk3LjIxMDgwMSIKICAgdmVyc2lvbj0iMS4xIgogICBpZD0ic3ZnNSIKICAgc29kaXBvZGk6ZG9jbmFtZT0iY3VzdG9tLnN2ZyIKICAgaW5rc2NhcGU6dmVyc2lvbj0iMS4xLjIgKGI4ZTI1YmU4LCAyMDIyLTAyLTA1KSIKICAgeG1sbnM6aW5rc2NhcGU9Imh0dHA6Ly93d3cuaW5rc2NhcGUub3JnL25hbWVzcGFjZXMvaW5rc2NhcGUiCiAgIHhtbG5zOnNvZGlwb2RpPSJodHRwOi8vc29kaXBvZGkuc291cmNlZm9yZ2UubmV0L0RURC9zb2RpcG9kaS0wLmR0ZCIKICAgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIgogICB4bWxuczpzdmc9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICA8c29kaXBvZGk6bmFtZWR2aWV3CiAgICAgaWQ9Im5hbWVkdmlldzgzNiIKICAgICBwYWdlY29sb3I9IiNmZmZmZmYiCiAgICAgYm9yZGVyY29sb3I9IiM2NjY2NjYiCiAgICAgYm9yZGVyb3BhY2l0eT0iMS4wIgogICAgIGlua3NjYXBlOnBhZ2VzaGFkb3c9IjIiCiAgICAgaW5rc2NhcGU6cGFnZW9wYWNpdHk9IjAuMCIKICAgICBpbmtzY2FwZTpwYWdlY2hlY2tlcmJvYXJkPSIwIgogICAgIGlua3NjYXBlOmRvY3VtZW50LXVuaXRzPSJtbSIKICAgICBzaG93Z3JpZD0iZmFsc2UiCiAgICAgaW5rc2NhcGU6em9vbT0iMi42OTMxODUxIgogICAgIGlua3NjYXBlOmN4PSIxNTkuMTA1MjkiCiAgICAgaW5rc2NhcGU6Y3k9IjE2OC43NTkyOSIKICAgICBpbmtzY2FwZTp3aW5kb3ctd2lkdGg9IjE5MjAiCiAgICAgaW5rc2NhcGU6d2luZG93LWhlaWdodD0iMTAyNyIKICAgICBpbmtzY2FwZTp3aW5kb3cteD0iMTcyOCIKICAgICBpbmtzY2FwZTp3aW5kb3cteT0iMjUiCiAgICAgaW5rc2NhcGU6d2luZG93LW1heGltaXplZD0iMCIKICAgICBpbmtzY2FwZTpjdXJyZW50LWxheWVyPSJzdmc1IgogICAgIGZpdC1tYXJnaW4tdG9wPSIxMCIKICAgICBmaXQtbWFyZ2luLWxlZnQ9IjEwIgogICAgIGZpdC1tYXJnaW4tcmlnaHQ9IjEwIgogICAgIGZpdC1tYXJnaW4tYm90dG9tPSIxMCIgLz4KICA8ZGVmcwogICAgIGlkPSJkZWZzMiIgLz4KICA8ZwogICAgIGlkPSJsYXllcjEiCiAgICAgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTU2Ljk1NDk3MSwtMTA2LjU0MjcxKSIKICAgICBzdHlsZT0iZmlsbC1ydWxlOmV2ZW5vZGQiPgogICAgPGcKICAgICAgIGlkPSJnODM2IgogICAgICAgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTIuMTAzODk2MSw0MS4wMjU5NzQpIgogICAgICAgc3R5bGU9ImZpbGwtcnVsZTpldmVub2RkIj4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDIuNzIwNzEsMTI1LjUxMDI1IGMgMCwtMC44MDAzNiAwLjA0OTYsLTEuMTI3NzggMC4xMTAyMiwtMC43Mjc2IDAuMDYwNiwwLjQwMDE4IDAuMDYwNiwxLjA1NTAzIDAsMS40NTUyMSAtMC4wNjA2LDAuNDAwMTggLTAuMTEwMjIsMC4wNzI4IC0wLjExMDIyLC0wLjcyNzYxIHogbSAzLjQwMjA3LC0zLjQzOTU4IGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ2LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ2LDAuMDMzMSAtMC4xMzM0NiwtMC4zMzA3MyB6IG0gLTI4LjA2NjcxOCwtNi41MDQzNCBjIDAuMDEyNjksLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MiwtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTY0LC0wLjM0NzI3IHogbSAyNi4xMDY3MTgsLTIuMTM3NTUgYyAwLjE4MTksLTAuMDczNCAwLjQ3OTU2LC0wLjA3MzQgMC42NjE0NiwwIDAuMTgxOSwwLjA3MzQgMC4wMzMxLDAuMTMzNDUgLTAuMzMwNzMsMC4xMzM0NSAtMC4zNjM4MSwwIC0wLjUxMjYzLC0wLjA2MDEgLTAuMzMwNzMsLTAuMTMzNDUgeiIKICAgICAgICAgaWQ9InBhdGg4NTIiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTM5LjcwMzk4LDEzMC4xMTg0MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NiAwLjA3NjQsMC4xOTEgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjgzIC0wLjEzODk2LC0wLjM0NzI2IHogbSAtMzYuOTc3ODcsLTEuNjk3NzUgYyAwLC0wLjk0NTg4IDAuMDQ4LC0xLjMzMjg0IDAuMTA2NzEsLTAuODU5ODkgMC4wNTg3LDAuNDcyOTQgMC4wNTg3LDEuMjQ2ODQgMCwxLjcxOTc5IC0wLjA1ODcsMC40NzI5NCAtMC4xMDY3MSwwLjA4NiAtMC4xMDY3MSwtMC44NTk5IHogbSAtMzMuNjY1ODgyLDAuNjM5NDEgYyAwLjAxMjcsLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MywtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI3IHogTSAxMDYuMTIyNzgsMTIzLjEyOSBjIDAsLTAuMzYzOCAwLjA2MDEsLTAuNTEyNjMgMC4xMzM0NiwtMC4zMzA3MiAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU1IDAsMC42NjE0NSAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NiwwLjAzMzEgLTAuMTMzNDYsLTAuMzMwNzMgeiBtIC0yOC4wNjY3MTgsLTguMzU2NDIgYyAwLjAxMjY5LC0wLjMwODIyIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODUgMC4wNzY0MywwLjE5MDk5IDAuMDY3MDMsMC40MTkyIC0wLjAyMDg5LDAuNTA3MTIgLTAuMDg3OTIsMC4wODc5IC0wLjE1MDQ1MywtMC4wNjgzIC0wLjEzODk2NCwtMC4zNDcyNyB6IG0gLTguOTk1ODM0LC0xMi40MzU0MSBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTM5LC0wLjM3MDkyIDAuMTU5ODUzLC0wLjE1OTg2IDAuMDc2NDMsMC4xOTEgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI2IHogbSAyNS45MDgyNzksLTkuMjM4Mzc0IGMgMC4zNDE3NzYsLTAuMzYzODAyIDAuNjgwOTQsLTAuNjYxNDU4IDAuNzUzNywtMC42NjE0NTggMC4wNzI3NiwwIC0wLjE0NzM0MSwwLjI5NzY1NiAtMC40ODkxMTcsMC42NjE0NTggLTAuMzQxNzc1LDAuMzYzODAyIC0wLjY4MDkzOSwwLjY2MTQ1OSAtMC43NTM3LDAuNjYxNDU5IC0wLjA3Mjc2LDAgMC4xNDczNDEsLTAuMjk3NjU3IDAuNDg5MTE3LC0wLjY2MTQ1OSB6IgogICAgICAgICBpZD0icGF0aDg1MCIgLz4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDIuNjYyMzEsMTQ4LjkwMzgzIGMgMC4wMTI3LC0wLjMwODIyIDAuMDc1NCwtMC4zNzA5MiAwLjE1OTg1LC0wLjE1OTg1IDAuMDc2NCwwLjE5MDk5IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTIgLTAuMDg3OSwwLjA4NzkgLTAuMTUwNDYsLTAuMDY4NCAtMC4xMzg5NywtMC4zNDcyNyB6IG0gMy40Mzk1OSwwIGMgMC4wMTI3LC0wLjMwODIyIDAuMDc1NCwtMC4zNzA5MiAwLjE1OTg1LC0wLjE1OTg1IDAuMDc2NCwwLjE5MDk5IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTIgLTAuMDg3OSwwLjA4NzkgLTAuMTUwNDUsLTAuMDY4NCAtMC4xMzg5NiwtMC4zNDcyNyB6IG0gLTMuMzcxNTIsLTE3LjA0MzU4IGMgMCwtMS4wOTE0IDAuMDQ2NywtMS41Mzc4OSAwLjEwMzc4LC0wLjk5MjE4IDAuMDU3MSwwLjU0NTcgMC4wNTcxLDEuNDM4NjcgMCwxLjk4NDM3IC0wLjA1NzEsMC41NDU3MSAtMC4xMDM3OCwwLjA5OTIgLTAuMTAzNzgsLTAuOTkyMTkgeiBtIC0zMy42NDkyNjMsLTMuNzA0MTYgYyAwLC0wLjM2MzggMC4wNjAwNSwtMC41MTI2MyAwLjEzMzQ1MiwtMC4zMzA3MyAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU2IDAsMC42NjE0NiAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NTIsMC4wMzMxIC0wLjEzMzQ1MiwtMC4zMzA3MyB6IG0gMzcuMDQxNjYzLC0zLjk2ODc1IGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ2LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ2LDAuMDMzMSAtMC4xMzM0NiwtMC4zMzA3MyB6IE0gNzguMDU2MDYyLDExMy45Nzg4MyBjIDAuMDEyNjksLTAuMzA4MjIgMC4wNzUzOSwtMC4zNzA5MiAwLjE1OTg1MiwtMC4xNTk4NSAwLjA3NjQzLDAuMTkwOTkgMC4wNjcwMywwLjQxOTIgLTAuMDIwODksMC41MDcxMiAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODMgLTAuMTM4OTY0LC0wLjM0NzI3IHogbSAtOC45OTU4MzQsLTEwLjg0NzkxIGMgMC4wMTI3LC0wLjMwODIzIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTMsLTAuMTU5ODYgMC4wNzY0MywwLjE5MSAwLjA2NzAzLDAuNDE5MiAtMC4wMjA4OSwwLjUwNzEyIC0wLjA4NzkyLDAuMDg3OSAtMC4xNTA0NTMsLTAuMDY4MyAtMC4xMzg5NjMsLTAuMzQ3MjYgeiBtIDcwLjY0Mzc1MiwtMC4yNjQ1OSBjIDAuMDEyNywtMC4zMDgyMiAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NSAwLjA3NjQsMC4xOTA5OSAwLjA2NywwLjQxOTIgLTAuMDIwOSwwLjUwNzEyIC0wLjA4NzksMC4wODc5IC0wLjE1MDQ1LC0wLjA2ODMgLTAuMTM4OTYsLTAuMzQ3MjcgeiIKICAgICAgICAgaWQ9InBhdGg4NDgiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTAyLjY2MjMxLDE0OC4xMTAwOCBjIDAuMDEyNywtMC4zMDgyMiAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NSAwLjA3NjQsMC4xOTA5OSAwLjA2NywwLjQxOTIgLTAuMDIwOSwwLjUwNzEyIC0wLjA4NzksMC4wODc5IC0wLjE1MDQ2LC0wLjA2ODMgLTAuMTM4OTcsLTAuMzQ3MjcgeiBtIDAuMDY1OSwtMTIuNjc3OTUgYyAxMGUtNCwtMS4wMTg2NSAwLjA0OSwtMS40MDI5NCAwLjEwNjI4LC0wLjg1NCAwLjA1NzMsMC41NDg5NSAwLjA1NjMsMS4zODIzOSAtMC4wMDIsMS44NTIwOSAtMC4wNTg1LDAuNDY5NyAtMC4xMDUzMSwwLjAyMDYgLTAuMTA0MTQsLTAuOTk4MDkgeiBtIDM2Ljk5NjYzLC02Ljc0Njg4IGMgMCwtMC4zNjM4IDAuMDYsLTAuNTEyNjMgMC4xMzM0NSwtMC4zMzA3MiAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU1IDAsMC42NjE0NSAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NSwwLjAzMzEgLTAuMTMzNDUsLTAuMzMwNzMgeiBtIC03MC42NDM3NTMsLTEuNTg3NSBjIDAsLTAuMzYzOCAwLjA2MDA1LC0wLjUxMjYzIDAuMTMzNDUyLC0wLjMzMDcyIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTUgMCwwLjY2MTQ1IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ1MiwwLjAzMzEgLTAuMTMzNDUyLC0wLjMzMDczIHogbSAzNy4wNTE0NzMsLTEuNzE5NzkgYyAwLjAwNSwtMC40MzY1NiAwLjA2NDcsLTAuNTgzMTIgMC4xMzE3NywtMC4zMjU2OSAwLjA2NzEsMC4yNTc0MyAwLjA2MjcsMC42MTQ2MiAtMC4wMSwwLjc5Mzc1IC0wLjA3MjUsMC4xNzkxMyAtMC4xMjczNiwtMC4wMzE1IC0wLjEyMTk3LC0wLjQ2ODA2IHogbSAyNC44ODA2MywtOC44NjM1NCBjIDAsLTAuNTA5MzIgMC4wNTQ1LC0wLjcxNzY4IDAuMTIxMDEsLTAuNDYzMDIgMC4wNjY1LDAuMjU0NjYgMC4wNjY1LDAuNjcxMzggMCwwLjkyNjA0IC0wLjA2NjYsMC4yNTQ2NiAtMC4xMjEwMSwwLjA0NjMgLTAuMTIxMDEsLTAuNDYzMDIgeiBtIC01Mi45NTcxNTgsLTMuMzI5MzQgYyAwLjAxMjY5LC0wLjMwODIyIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODUgMC4wNzY0MywwLjE5MDk5IDAuMDY3MDMsMC40MTkyIC0wLjAyMDg5LDAuNTA3MTIgLTAuMDg3OTIsMC4wODc5IC0wLjE1MDQ1MywtMC4wNjgzIC0wLjEzODk2NCwtMC4zNDcyNyB6IG0gLTguOTY1MTQxLC04Ljk3Mzc4IGMgMC4wMDU0LC0wLjQzNjU3IDAuMDY0NjksLTAuNTgzMTMgMC4xMzE3NzMsLTAuMzI1NyAwLjA2NzA4LDAuMjU3NDMgMC4wNjI2NywwLjYxNDYyIC0wLjAwOTgsMC43OTM3NSAtMC4wNzI0OCwwLjE3OTEzIC0wLjEyNzM2MywtMC4wMzE1IC0wLjEyMTk3LC0wLjQ2ODA1IHogbSA3MC42MTMwNTksLTAuNTUxMjIgYyAwLjAxMjcsLTAuMzA4MjIgMC4wNzU0LC0wLjM3MDkyIDAuMTU5ODUsLTAuMTU5ODUgMC4wNzY0LDAuMTkwOTkgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjg0IC0wLjEzODk2LC0wLjM0NzI3IHoiCiAgICAgICAgIGlkPSJwYXRoODQ2IiAvPgogICAgICA8cGF0aAogICAgICAgICBzdHlsZT0iZmlsbC1vcGFjaXR5OjE7ZmlsbC1ydWxlOmV2ZW5vZGQ7c3Ryb2tlLXdpZHRoOjAuMjY0NTgzIgogICAgICAgICBkPSJtIDEwNi4xMDE5LDE0Ny41ODA5MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NiAwLjA3NjQsMC4xOTEgMC4wNjcsMC40MTkyIC0wLjAyMDksMC41MDcxMiAtMC4wODc5LDAuMDg3OSAtMC4xNTA0NSwtMC4wNjg0IC0wLjEzODk2LC0wLjM0NzI2IHogbSAtMy40MTg3LC0wLjM3NDgzIGMgMCwtMC4zNjM4IDAuMDYwMSwtMC41MTI2MyAwLjEzMzQ1LC0wLjMzMDczIDAuMDczNCwwLjE4MTkgMC4wNzM0LDAuNDc5NTYgMCwwLjY2MTQ2IC0wLjA3MzQsMC4xODE5IC0wLjEzMzQ1LDAuMDMzMSAtMC4xMzM0NSwtMC4zMzA3MyB6IG0gMC4wNDI5LC04LjQ2NjY3IGMgMCwtMC45NDU4OCAwLjA0OCwtMS4zMzI4NCAwLjEwNjcxLC0wLjg1OTg5IDAuMDU4NywwLjQ3Mjk0IDAuMDU4NywxLjI0Njg0IDAsMS43MTk3OSAtMC4wNTg3LDAuNDcyOTQgLTAuMTA2NzEsMC4wODYgLTAuMTA2NzEsLTAuODU5OSB6IG0gMzcuMDA4NTYsLTExLjI0NDc5IGMgMC4wMDUsLTAuNDM2NTYgMC4wNjQ3LC0wLjU4MzEzIDAuMTMxNzcsLTAuMzI1NjkgMC4wNjcxLDAuMjU3NDMgMC4wNjI3LDAuNjE0NjEgLTAuMDEsMC43OTM3NSAtMC4wNzI1LDAuMTc5MTMgLTAuMTI3MzYsLTAuMDMxNSAtMC4xMjE5NywtMC40NjgwNiB6IG0gLTMzLjYwMjA4LC0wLjc5Mzc1IGMgMC4wMDUsLTAuNDM2NTYgMC4wNjQ3LC0wLjU4MzEzIDAuMTMxNzcsLTAuMzI1NjkgMC4wNjcxLDAuMjU3NDMgMC4wNjI3LDAuNjE0NjEgLTAuMDEsMC43OTM3NSAtMC4wNzI1LDAuMTc5MTMgLTAuMTI3MzYsLTAuMDMxNSAtMC4xMjE5NywtMC40NjgwNiB6IG0gMTMuNTc0NDYsLTQuODk0NzkgYyAwLjI2MzM5LC0wLjI5MTA0IDAuNTM4NDIsLTAuNTI5MTcgMC42MTExOCwtMC41MjkxNyAwLjA3MjgsMCAtMC4wODMyLDAuMjM4MTMgLTAuMzQ2NiwwLjUyOTE3IC0wLjI2MzM5LDAuMjkxMDQgLTAuNTM4NDIsMC41MjkxNiAtMC42MTExOCwwLjUyOTE2IC0wLjA3MjgsMCAwLjA4MzIsLTAuMjM4MTIgMC4zNDY2LC0wLjUyOTE2IHogbSAtNTAuNTk1NjQ3LC0xNS44NzUgYyAwLC0wLjY1NDg1IDAuMDUxNiwtMC45MjI3NCAwLjExNDY2MywtMC41OTUzMSAwLjA2MzA2LDAuMzI3NDIgMC4wNjMwNiwwLjg2MzIgMCwxLjE5MDYyIC0wLjA2MzA3LDAuMzI3NDIgLTAuMTE0NjYzLDAuMDU5NSAtMC4xMTQ2NjMsLTAuNTk1MzEgeiBtIDcwLjYxMzQ2NywtMS4zMjI5MiBjIDAsLTAuMzYzOCAwLjA2LC0wLjUxMjYzIDAuMTMzNDUsLTAuMzMwNzMgMC4wNzM0LDAuMTgxOSAwLjA3MzQsMC40Nzk1NiAwLDAuNjYxNDYgLTAuMDczNCwwLjE4MTkgLTAuMTMzNDUsMC4wMzMxIC0wLjEzMzQ1LC0wLjMzMDczIHogbSAtMzYuMjY4OCwtMTMuMTE4OTIyIGMgMC4wMTI3LC0wLjMwODIyNSAwLjA3NTQsLTAuMzcwOTIgMC4xNTk4NSwtMC4xNTk4NTMgMC4wNzY0LDAuMTkwOTk2IDAuMDY3LDAuNDE5MiAtMC4wMjA5LDAuNTA3MTE4IC0wLjA4NzksMC4wODc5MiAtMC4xNTA0NiwtMC4wNjgzNSAtMC4xMzg5NywtMC4zNDcyNjUgeiIKICAgICAgICAgaWQ9InBhdGg4NDQiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTA2LjEyMjc4LDE0Ni42NzY5MiBjIDAsLTAuMzYzOCAwLjA2MDEsLTAuNTEyNjMgMC4xMzM0NiwtMC4zMzA3MyAwLjA3MzQsMC4xODE5IDAuMDczNCwwLjQ3OTU2IDAsMC42NjE0NiAtMC4wNzM0LDAuMTgxOSAtMC4xMzM0NiwwLjAzMzEgLTAuMTMzNDYsLTAuMzMwNzMgeiBtIC0zLjM3OTQsLTMuMTc1IGMgMCwtMS44MTkwMSAwLjA0MjMsLTIuNTYzMTUgMC4wOTM5LC0xLjY1MzY0IDAuMDUxNywwLjkwOTUgMC4wNTE3LDIuMzk3NzggMCwzLjMwNzI5IC0wLjA1MTcsMC45MDk1IC0wLjA5MzksMC4xNjUzNiAtMC4wOTM5LC0xLjY1MzY1IHogbSAzLjM5OTAxLC0xNS4zNDU4MyBjIDAsLTAuNTA5MzIgMC4wNTQ0LC0wLjcxNzY4IDAuMTIxLC0wLjQ2MzAyIDAuMDY2NSwwLjI1NDY2IDAuMDY2NSwwLjY3MTM4IDAsMC45MjYwNCAtMC4wNjY2LDAuMjU0NjYgLTAuMTIxLDAuMDQ2MyAtMC4xMjEsLTAuNDYzMDIgeiBtIDMzLjYxNjM3LC0yLjUxMzU0IGMgMC4wMDIsLTAuNzI3NjEgMC4wNTM0LC0wLjk5Mzg2IDAuMTE0MTksLTAuNTkxNjggMC4wNjA4LDAuNDAyMTggMC4wNTkyLDAuOTk3NSAtMC4wMDQsMS4zMjI5MiAtMC4wNjI4LDAuMzI1NDIgLTAuMTEyNTYsLTAuMDA0IC0wLjExMDU3LC0wLjczMTI0IHogbSAtNzAuNjI2OTcxLC00Ljg5NDggYyAwLC0xLjIzNjkyIDAuMDQ1NTgsLTEuNzQyOTQgMC4xMDEyODEsLTEuMTI0NDcgMC4wNTU3LDAuNjE4NDYgMC4wNTU3LDEuNjMwNDkgMCwyLjI0ODk1IC0wLjA1NTcsMC42MTg0NyAtMC4xMDEyODEsMC4xMTI0NSAtMC4xMDEyODEsLTEuMTI0NDggeiBtIDAsLTcuOTM3NSBjIDAsLTEuMjM2OTIgMC4wNDU1OCwtMS43NDI5NCAwLjEwMTI4MSwtMS4xMjQ0NyAwLjA1NTcsMC42MTg0NiAwLjA1NTcsMS42MzA0OSAwLDIuMjQ4OTUgLTAuMDU1NywwLjYxODQ3IC0wLjEwMTI4MSwwLjExMjQ1IC0wLjEwMTI4MSwtMS4xMjQ0OCB6IG0gOC45MjQyNzMsLTAuOTQ4MDggYyAwLjAxMjY5LC0wLjMwODIzIDAuMDc1MzksLTAuMzcwOTIgMC4xNTk4NTIsLTAuMTU5ODYgMC4wNzY0MywwLjE5MSAwLjA2NzAzLDAuNDE5MiAtMC4wMjA4OSwwLjUwNzEyIC0wLjA4NzkyLDAuMDg3OSAtMC4xNTA0NTMsLTAuMDY4MyAtMC4xMzg5NjQsLTAuMzQ3MjYgeiBtIC04Ljk5NTgzNCwtNC40OTc5MiBjIDAuMDEyNywtMC4zMDgyMyAwLjA3NTM5LC0wLjM3MDkyIDAuMTU5ODUzLC0wLjE1OTg1IDAuMDc2NDMsMC4xOTA5OSAwLjA2NzAzLDAuNDE5MTkgLTAuMDIwODksMC41MDcxMSAtMC4wODc5MiwwLjA4NzkgLTAuMTUwNDUzLC0wLjA2ODQgLTAuMTM4OTYzLC0wLjM0NzI2IHogbSA3MC42OTg1MzIsLTEuMDM2MjkgYyAwLjAwMiwtMC43Mjc2IDAuMDUzNCwtMC45OTM4NSAwLjExNDE5LC0wLjU5MTY3IDAuMDYwOCwwLjQwMjE4IDAuMDU5MiwwLjk5NzQ5IC0wLjAwNCwxLjMyMjkyIC0wLjA2MjgsMC4zMjU0MiAtMC4xMTI1NiwtMC4wMDQgLTAuMTEwNTcsLTAuNzMxMjUgeiIKICAgICAgICAgaWQ9InBhdGg4NDIiIC8+CiAgICAgIDxwYXRoCiAgICAgICAgIHN0eWxlPSJmaWxsLW9wYWNpdHk6MTtmaWxsLXJ1bGU6ZXZlbm9kZDtzdHJva2Utd2lkdGg6MC4yNjQ1ODMiCiAgICAgICAgIGQ9Im0gMTA2LjE0NzczLDE0NS4yMjE3MSBjIDAuMDAzLC0wLjU4MjA4IDAuMDU3MSwtMC43ODkwNSAwLjEyMDQxLC0wLjQ1OTkzIDAuMDYzMywwLjMyOTEyIDAuMDYwOSwwLjgwNTM3IC0wLjAwNSwxLjA1ODMzIC0wLjA2NjIsMC4yNTI5NyAtMC4xMTgsLTAuMDE2MyAtMC4xMTUwNywtMC41OTg0IHogTSA4MS4yNzE1NTgsMTM1LjAzNTI1IGMgMCwtMC41MDkzMiAwLjA1NDQ1LC0wLjcxNzY4IDAuMTIxLC0wLjQ2MzAyIDAuMDY2NTUsMC4yNTQ2NyAwLjA2NjU1LDAuNjcxMzggMCwwLjkyNjA1IC0wLjA2NjU1LDAuMjU0NjYgLTAuMTIxLDAuMDQ2MyAtMC4xMjEsLTAuNDYzMDMgeiBtIDI0Ljg5MTQzMiwtNC42MzAyIGMgMC4wMDIsLTAuODczMTMgMC4wNTA5LC0xLjE5ODQyIDAuMTA5NzcsLTAuNzIyODcgMC4wNTg5LDAuNDc1NTUgMC4wNTc3LDEuMTg5OTIgLTAuMDAzLDEuNTg3NSAtMC4wNjA0LDAuMzk3NTggLTAuMTA4NTUsMC4wMDggLTAuMTA3MDcsLTAuODY0NjMgeiBtIDMzLjYzODMzLC0xNC40MTk4IGMgMCwtNC43Mjk0MiAwLjAzNTksLTYuNjY0MTkgMC4wNzk3LC00LjI5OTQ3IDAuMDQzOSwyLjM2NDcxIDAuMDQzOSw2LjIzNDI0IDAsOC41OTg5NSAtMC4wNDM4LDIuMzY0NzIgLTAuMDc5NywwLjQyOTk1IC0wLjA3OTcsLTQuMjk5NDggeiBNIDEwMS4yNjg5LDg5Ljg3NTM0NyBjIDAuMTkwOTksLTAuMDc2NDMgMC40MTkyLC0wLjA2NzAzIDAuNTA3MTIsMC4wMjA4OSAwLjA4NzksMC4wODc5MiAtMC4wNjg0LDAuMTUwNDUzIC0wLjM0NzI3LDAuMTM4OTY1IC0wLjMwODIzLC0wLjAxMjY5IC0wLjM3MDkyLC0wLjA3NTM5IC0wLjE1OTg1LC0wLjE1OTg1MyB6IgogICAgICAgICBpZD0icGF0aDg0MCIgLz4KICAgICAgPHBhdGgKICAgICAgICAgc3R5bGU9ImZpbGwtb3BhY2l0eToxO2ZpbGwtcnVsZTpldmVub2RkO3N0cm9rZS13aWR0aDowLjI2NDU4MyIKICAgICAgICAgZD0ibSAxMDAuNjMwODQsMTUyLjU4MDg1IGMgLTAuNjkzNzk3LC0wLjEzNjcxIC0yNy4wNTY5NTMsLTE1LjM0MTU0IC0yOC42NzY3NTUsLTE2LjUzOTIgLTAuNjUyNDY0LC0wLjQ4MjQyIC0xLjQ4NTkwMSwtMS40MTIyMyAtMS44NTIwODMsLTIuMDY2MjQgbCAtMC42NjU3ODcsLTEuMTg5MTEgLTAuMDc0MTEsLTE1Ljk3NTQzIGMgLTAuMDY1MzIsLTE0LjA4MDczIC0wLjAyNTM3LC0xNi4wNDk4MiAwLjMzNjgyOSwtMTYuNjAyNjEgMC44ODUxNzMsLTEuMzUwOTQyIDAuOTA1MjEsLTEuMzQxMjkgMTkuNTc2NjY0LDkuNDI5OTQgMTAuOTY0MDQyLDYuMzI0OTYgMTIuNTMxNjAyLDcuNDU0MjQgMTMuMDc3NzgyLDkuNDIxMzIgMC4xODkwMiwwLjY4MDc2IDAuMjg1NTksNi4zNTQzMSAwLjI4NjU3LDE2LjgzNTYzIDAuMDAxLDE1LjQ1NzA5IC0wLjAxMDMsMTUuODIwNjMgLTAuNTI3NjksMTYuMzM4MDIgLTAuMjkxMDQsMC4yOTEwNCAtMC42MTg0NywwLjUxMjk1IC0wLjcyNzYxLDAuNDkzMTQgLTAuMTA5MTQsLTAuMDE5OCAtMC40NDgzNSwtMC4wODUzIC0wLjc1MzgxLC0wLjE0NTQ2IHogbSAtMTMuOTYxNjY5LC0xOS4wMDQ5IGMgLTAuMTI4MjAxLC0wLjMzNDA5IC00Ljc1NzYxMiwtMy4wMzg2MSAtNS4yMDEyODEsLTMuMDM4NjEgLTAuMTc4NjYyLDAgLTAuMjQ5MjAxLDEuMDAwMSAtMC4yMDA1MjIsMi44NDI5NyBsIDAuMDc1MSwyLjg0Mjk4IDIuNjQ1ODM0LDEuNTI0MjYgMi42NDU4MzMsMS41MjQyNiAwLjA3NTk0LC0yLjY5NTcgYyAwLjA0MTc2LC0xLjQ4MjY0IDAuMDIzMzYsLTIuODMyNzEgLTAuMDQwOSwtMy4wMDAxNiB6IG0gMC41MDU1NTgsLTMuMzY0OTggYyAwLjIwNTUzMywtMC42OTQ1OSAwLjYxODg3MSwtMS4wNTYxIDIuMTk3MzQsLTEuOTIxNzggMi40ODkyMTEsLTEuMzY1MTcgMi43ODQ4MTEsLTEuODk2ODIgMi42MzU1MDIsLTQuNzQwMDYgLTAuMDk0ODEsLTEuODA1NTMgLTAuMjU0Njg4LC0yLjQwNzUgLTEuMDIzMTE1LC0zLjg1MjI4IC0xLjExNjg5MSwtMi4wOTk5NyAtMy4xMTc5MjEsLTQuMTQzMzEgLTUuNzIyNzcxLC01Ljg0Mzc4IC0yLjA2MjQ0NywtMS4zNDYzOCAtNi4wMzgzNzMsLTIuODM4MjcgLTYuNzkyNjY1LC0yLjU0ODgyIC0wLjM1Nzc1OSwwLjEzNzI4IC0wLjQzMzg0NiwwLjU4NTU2IC0wLjQzMzg0NiwyLjU1NjA0IDAsMS4zMTQyNSAwLjA4OTMsMi4zOTE5MSAwLjE5ODQzNywyLjM5NDc5IDAuMTA5MTQxLDAuMDAzIDEuMDA2Mjc2LDAuMTIyMDUgMS45OTM2MzUsMC4yNjQ4MiAzLjQ4MzI1MSwwLjUwMzY2IDYuMDEwMDExLDIuNDY1NzggNi4wMTAwMTEsNC42NjcgMCwxLjI0Mzk1IC0wLjMyNjUzLDEuNTcwMTggLTIuODM4MTg4LDIuODM1NTkgLTEuNTkyOTcxLDAuODAyNTYgLTIuMjU5MywxLjgyNzkxIC0yLjEzNzc3MywzLjI4OTYxIDAuMDc3OSwwLjkzNjk1IDAuMTgyNDM5LDEuMDMyMjEgMi41OTQ3MTEsMi4zNjQ0NCAxLjM4MjQ0OCwwLjc2MzQ5IDIuNjM3NzQ4LDEuMzg4NTcgMi43ODk1NTUsMS4zODkwNyAwLjE1MTgxLDUuMmUtNCAwLjM4OTkzNSwtMC4zODQwOSAwLjUyOTE2NywtMC44NTQ2NCB6IG0gMTkuNzQ2MDYxLDIyLjA2ODIzIGMgLTAuNDM4MjUsLTAuNDM4MjUgLTAuNTExMjksLTEuMTUwODEgLTAuNjc3NTgsLTYuNjExMDYgLTAuMTAyNTEsLTMuMzY1OTIgLTAuMTA0ODYsLTEwLjYxNzg3IC0wLjAwNSwtMTYuMTE1NDQgMC4yMzg1NCwtMTMuMTU5OTYgLTAuNDA4MDIsLTExLjg1MTYgOC4zNTkzNSwtMTYuOTE1NzYgMi44MTAzNCwtMS42MjMyOSA4LjQ0MzQ2LC00Ljg4MDQgMTIuNTE4MDQsLTcuMjM4MDIgMTAuNzI0MTcsLTYuMjA1MTkgMTEuMTY1NjksLTYuMzk5MzQxIDEyLjEyOTgyLC01LjMzMzk4IDAuMzk3MjIsMC40Mzg5MiAwLjQzNzg5LDEuOTU4MTUgMC40Mzc4OSwxNi4zNTY3MyB2IDE1Ljg3Mjg2IGwgLTAuNzI3NjEsMS40MzY1MSBjIC0wLjQ2ODE4LDAuOTI0MzQgLTEuMTU5NDYsMS43NTQ0OSAtMS45Mzg4OCwyLjMyODM4IC0yLjI4MjMzLDEuNjgwNTEgLTI3LjgxNjgxLDE2LjMxMjU0IC0yOC44ODEwNSwxNi41NDk3MiAtMC41MTExNSwwLjExMzkyIC0wLjg2Nzc3LDAuMDE3MSAtMS4yMTQ3NywtMC4zMjk5NCB6IG0gMTguMTQ0MDcsLTE0LjAwMDIxIDAuODU5OSwtMC40MzEzMyB2IC0zLjAxODk2IC0zLjAxODk2IGwgLTEuMzg5MDcsMC43ODU4OSBjIC0wLjc2Mzk4LDAuNDMyMjMgLTEuOTg0MzcsMS4xMzYyOCAtMi43MTE5NywxLjU2NDU0IGwgLTEuMzIyOTIsMC43Nzg2NSAtMC4wNzQ5LDIuOTcyNDcgLTAuMDc0OSwyLjk3MjQ3IDEuOTI3MDIsLTEuMDg2NzIgYyAxLjA1OTg2LC0wLjU5NzcgMi4zMTM5OCwtMS4yODA4MiAyLjc4NjkyLC0xLjUxODA1IHogbSAtMS44MDU3MSwtNy4xMDc5MiBjIDIuNTI0NjMsLTEuNDY2MTUgMi41NTcxLC0xLjQ5Nzg2IDIuODY2NTIsLTIuODAwNDYgMC4xNzIxMywtMC43MjQ2MyAxLjE0NTQsLTIuNzA2NDUgMi4xNjI4MiwtNC40MDQwNCAyLjAzMzAzLC0zLjM5MjE3IDIuNTg2NTEsLTQuNzIxMDEgMi44MzAyOSwtNi43OTUyIDAuMzA3NDgsLTIuNjE2MTggLTAuODM0MzMsLTQuMzYxMTIgLTIuODUzNzEsLTQuMzYxMTIgLTIuMTE4MjMsMCAtNi4wNTcwNiwyLjE4NjEyIC05LjU1MDIxLDUuMzAwNTEgbCAtMS41MjEzNSwxLjM1NjQgdiAyLjU0MTEzIDIuNTQxMTEgbCAyLjQ3NjI1LC0yLjQyODM3IGMgMy4yNjYyLC0zLjIwMzA1IDUuNDEzMzYsLTQuMTg0MjUgNS44MTI5MSwtMi42NTYzNSAwLjIyOTA4LDAuODc2MDEgLTAuNDQyNjQsMi42MDQxIC0xLjkzODU5LDQuOTg3MjQgLTAuNzc2NDUsMS4yMzY5MyAtMS43NTQxNywyLjk4MDA0IC0yLjE3MjcxLDMuODczNTggLTAuODE4MTcsMS43NDY2OCAtMS4zNzExNSw0LjMyODUgLTAuOTI3MDksNC4zMjg1IDAuMTQzNzMsMCAxLjQxMDQyLC0wLjY2NzMyIDIuODE0ODcsLTEuNDgyOTMgeiBNIDEwMS40NTA4LDExMi41ODM5NSBDIDk5Ljc5OTk0OSwxMTEuNzk4NzYgNzUuNzExOTkyLDk3LjkzMDI1MiA3My4xMjc0MDcsOTYuMjc2OTEzIDcxLjczMDIxNiw5NS4zODMxNCA3MS42NTgyNTIsOTUuMjgzODYzIDcxLjczNTc0Niw5NC4zNTcwNzMgNzEuODM1ODM1LDkzLjE2MDA1MiA3MC45OTE5NDUsOTMuNzAwODg1IDg5LjAxMjEwOSw4My4yODQ5OCBsIDEzLjIyNTg5MSwtNy42NDQ3NSAxLjg2MTEyLC0wLjA5Mzk4IGMgMS43NzUxLC0wLjA4OTY0IDEuOTY0NzcsLTAuMDM5MjggNC4xMDQzMSwxLjA4OTczOSA0LjQ5ODA3LDIuMzczNTk3IDI3LjY2MzYsMTUuODgxMTIzIDI4LjM3MDgxLDE2LjU0MjY1OSAxLjI2NzM3LDEuMTg1NTE1IDAuODkxNDcsMi4wNTgwODIgLTEuNTM0MTksMy41NjEyOTEgLTIuNDUxODEsMS41MTk0MTQgLTI0Ljc2NzA3LDE0LjM5Mjc3MSAtMjYuNzgyNywxNS40NTA1NjEgLTIuNzgxMTIsMS40NTk1MSAtNC4zNzE4NCwxLjU1MTQ2IC02LjgwNjU1LDAuMzkzNDUgeiBtIDE0LjM3ODEyLC0xMi4zMDU1MiAzLjAwMTA0LC0xLjY4MjI2NCAtMC44ODQzNywtMC41MjcyMDkgYyAtMi4xNTgwNiwtMS4yODY0OTkgLTMuODg0MDMsLTIuMTkyMDM2IC00LjE3ODA2LC0yLjE5MjAzNiAtMC40MTU0NSwwIC02LjA5OTk2LDMuMTY3Mjk2IC02LjA5NTk2LDMuMzk2NTQ0IDAuMDA0LDAuMjQzMiA0LjI3MTIxLDIuNjI4OTQ1IDQuNzU5NDQsMi42NjEwOTUgMC4yMTgyOCwwLjAxNDQgMS43NDczNCwtMC43MzA4OCAzLjM5NzkxLC0xLjY1NjEzIHogbSAtOC42MTQ2NCwtMi45NzYxNjQgYyAwLjc1OTE1LC0wLjM4MDIyNSAyLjA2NjI1LC0xLjEwNDA1NiAyLjkwNDY4LC0xLjYwODUxIGwgMS41MjQ0MSwtMC45MTcxOTIgLTAuNTQ4MzgsLTAuNzcwMTI1IGMgLTAuNDk2NDIsLTAuNjk3MTUgLTAuNTMxNjEsLTEuMDAyMjc5IC0wLjM3MTM4LC0zLjIyMDEzNCAwLjE2MzU4LC0yLjI2NDM2OSAwLjEzMzI3LC0yLjUwNTE1NCAtMC40MDAwNywtMy4xNzc4IC0wLjMxNzM5LC0wLjQwMDI4NSAtMS4xMzM5MSwtMS4wNTQxOTkgLTEuODE0NDksLTEuNDUzMTQxIC0xLjA5NzE1LC0wLjY0MzEyOCAtMS41NTc0NSwtMC43MzYzMTkgLTQuMDYwNzIsLTAuODIyMTE0IC0yLjM4NjM4LC0wLjA4MTc5IC0zLjEwMjkyLC0wLjAwNDggLTQuNjMwMjA0LDAuNDk3Njg4IC0yLjIyNDgzOSwwLjczMTk0NyAtNC45MDQ3NDUsMi4wMzAxMzEgLTYuODMyODA2LDMuMzA5OTA1IC0xLjUzMDY0NiwxLjAxNTk4NCAtNC4xMDI1NjMsMy4yMjI3OCAtNC4xMDE3NTYsMy41MTk0NTEgNS4yOWUtNCwwLjE5NDIwNiAxLjk3MDE0MywxLjQ3NjgxNyAzLjM0MTQ0MSwyLjE3NTk0NyBsIDAuODkxODM3LDAuNDU0Njg0IDEuODkzOTksLTEuOTUzMjgyIGMgMi4zMTcwNywtMi4zODk2MDMgMy45NjYyNDIsLTMuMjg0OTUyIDYuMDI4ODk4LC0zLjI3MzE0IDIuMzU2OCwwLjAxMzUgMi40OTY0LDAuMjMxNDkyIDIuMjcwNTIsMy41NDU0NjggbCAtMC4xODk5MywyLjc4NjU1MiAwLjc5ODUzLDAuNzk4NTMxIGMgMC45Nzc1NSwwLjk3NzU0OCAxLjUyMjM2LDAuOTk1MjczIDMuMjk1NDMsMC4xMDcyMTIgeiIKICAgICAgICAgaWQ9InBhdGg4MzgiIC8+CiAgICA8L2c+CiAgPC9nPgo8L3N2Zz4K",
+ "properties": {
+ "random_function": {
+ "name": "random_function",
+ "title": "Random Function",
+ "description": "",
+ "type": "string",
+ "parameterType": "string",
+ "value": "token_urlsafe",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ },
+ "string_length": {
+ "name": "string_length",
+ "title": "String Length",
+ "description": "How long (in characters) should each value be.",
+ "type": "string",
+ "parameterType": "Long",
+ "value": "16",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "python",
+ "is_deprecated": false,
+ "tags": [
+ "TransformOperator",
+ "PythonPlugin"
+ ],
+ "pluginType": "transformer",
+ "relatedPlugins": []
+ },
"Excel_RANK": {
"pluginId": "Excel_RANK",
"title": "Rank",
@@ -18233,6 +18731,41 @@
"pluginType": "transformer",
"relatedPlugins": []
},
+ "setExecutionVariable": {
+ "pluginId": "setExecutionVariable",
+ "title": "Set execution variable",
+ "categories": [
+ "Variables"
+ ],
+ "main_category": "Variables",
+ "description": "Sets an execution variable to the first value of the (single) input and passes the input values through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution.",
+ "markdownDocumentation": "Sets an execution variable to the first value of the (single) input and passes the input values through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution.\n",
+ "pluginIcon": null,
+ "properties": {
+ "variableName": {
+ "name": "variableName",
+ "title": "Variable name",
+ "description": "Name of the execution variable to set. It is written to the 'execution' scope and addressed downstream as 'execution.'.",
+ "type": "string",
+ "parameterType": "string",
+ "value": "myVariable",
+ "advanced": false,
+ "visibleInDialog": true,
+ "properties": {}
+ }
+ },
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "TransformOperator"
+ ],
+ "pluginType": "transformer",
+ "relatedPlugins": []
+ },
"Excel_SIGN": {
"pluginId": "Excel_SIGN",
"title": "Sign",
@@ -20286,6 +20819,30 @@
}
]
},
+ "validate_uri": {
+ "pluginId": "validate_uri",
+ "title": "Validate URI",
+ "categories": [
+ "Validation",
+ "SPARQL"
+ ],
+ "main_category": "Validation",
+ "description": "Validates that the input is a valid absolute IRI and returns it unchanged. Throws a validation error if the input is not a valid IRI.",
+ "markdownDocumentation": "Validates that the input is a valid absolute IRI and returns it unchanged. Throws a validation error if the input is not a valid IRI. \n\n## Examples\n\n**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values \"first\" and \"second\".\n\n---\n**Example 1:**\n\n* Input values:\n 1. `[http://example.org/entity1]`\n\n* Returns: `[http://example.org/entity1]`\n\n\n---\n**Example 2:**\n\n* Input values:\n 1. `[urn:example:1]`\n\n* Returns: `[urn:example:1]`\n\n\n---\n**Example 3:**\n\n* Input values:\n 1. `[not a uri]`\n\n* Returns: `[]`\n* **Throws error:** `ValidationException`\n\n\n---\n**Example 4:**\n\n* Input values:\n 1. `[]`\n\n* Returns: `[]`\n* **Throws error:** `ValidationException`\n\n\n",
+ "pluginIcon": null,
+ "properties": {},
+ "properties_advanced": {},
+ "actions": {},
+ "required": [],
+ "distanceMeasureRange": null,
+ "backendType": "native",
+ "is_deprecated": false,
+ "tags": [
+ "TransformOperator"
+ ],
+ "pluginType": "transformer",
+ "relatedPlugins": []
+ },
"Excel_VAR": {
"pluginId": "Excel_VAR",
"title": "Var",
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/view/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/view/index.md
new file mode 100644
index 000000000..c94b31c5e
--- /dev/null
+++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/view/index.md
@@ -0,0 +1,267 @@
+---
+title: "cmemc: Command Group - admin view"
+description: "List and update explore application view configurations."
+icon: octicons/cross-reference-24
+tags:
+ - cmemc
+---
+
+# admin view Command Group
+
+
+
+List and update explore application view configurations.
+
+This command group manages Explore (DataPlatform) application view configurations. Application view configurations control the behavior of specific Explore view profiles including companion services and other settings.
+
+
+## admin view list
+
+List explore application view configurations.
+
+```shell-session title="Usage"
+cmemc admin view list [OPTIONS]
+```
+
+
+
+
+Outputs a list of application view configurations from the Explore component. The default application view (id: 'default') is always listed first, followed by any custom application view configurations.
+
+Profile IDs can be used as a reference for the other commands of the `admin view` command group.
+
+
+
+??? info "Options"
+ ```text
+
+ --raw Outputs raw JSON.
+ --id-only Lists only profile IDs. This is useful for piping
+ the IDs into other commands.
+ --filter ... Filter application view configurations by one of
+ the following filter names and a corresponding
+ value: id, label.
+ ```
+
+## admin view export
+
+Export application view configurations to a JSON file.
+
+```shell-session title="Usage"
+$ cmemc admin view export [OPTIONS] [PROFILE_IDS]...
+```
+
+
+
+
+Application view configurations can be exported based on profile IDs, filters, or all at once. The exported JSON can be imported back using the `admin view import` command.
+
+```shell-session title="Example"
+$ cmemc admin view export --all
+```
+
+
+```shell-session title="Example"
+$ cmemc admin view export --all --output-file configs.json
+```
+
+
+```shell-session title="Example"
+$ cmemc admin view export --filter id my-view
+```
+
+
+```shell-session title="Example"
+$ cmemc admin view export my-view
+```
+
+
+
+
+??? info "Options"
+ ```text
+
+ -a, --all Export all application view configurations.
+ --filter ... Filter application view configurations by one
+ of the following filter names and a
+ corresponding value: id, label.
+ --output-file FILE Export to this file. Use '-' for stdout. If
+ specified, overrides --output-dir and
+ --filename-template.
+ --output-dir DIRECTORY The base directory where the export file will
+ be created. Ignored if --output-file is
+ specified. [default: .]
+ -t, --filename-template TEXT Template for the export file name. Possible
+ placeholders are (Jinja2): {{connection}}
+ (from the --connection option) and {{date}}
+ (the current date as YYYY-MM-DD). Ignored if
+ --output-file is specified. [default:
+ {{date}}-{{connection}}.view-configs.json]
+ --replace Replace an existing export file. This is a
+ dangerous option, so use it with care.
+ ```
+
+## admin view import
+
+Import application view configurations from a JSON file.
+
+```shell-session title="Usage"
+cmemc admin view import [OPTIONS] INPUT_FILE
+```
+
+
+
+
+This command imports application view configurations from a JSON file that was created using the `admin view export` command.
+
+If `--replace` is specified, existing configurations with the same profile ID will be updated. Otherwise, existing configurations will be skipped.
+
+!!! note
+ Importing the default application view configuration updates the project-level overrides stored in /api/conf/workspaces/projectDefault.
+
+
+```shell-session title="Example"
+cmemc admin view import configs.json
+```
+
+
+```shell-session title="Example"
+cmemc admin view import --replace configs.json
+```
+
+
+
+
+??? info "Options"
+ ```text
+
+ --replace Replace existing application view configurations. By default,
+ import will skip configurations that already exist.
+ --id TEXT Import the configuration under this profile ID instead of the
+ one stored in the file.
+ ```
+
+## admin view delete
+
+Delete custom application view configurations.
+
+```shell-session title="Usage"
+cmemc admin view delete [OPTIONS] [PROFILE_IDS]...
+```
+
+
+
+
+!!! warning
+ Application view configurations will be deleted without prompting.
+
+
+!!! note
+ The default application view configuration cannot be deleted. Use the `admin view list` command to list available application view configurations.
+
+
+
+
+??? info "Options"
+ ```text
+
+ -a, --all Delete all custom application view configurations.
+ This is a dangerous option, so use it with care.
+ --filter ... Filter application view configurations by one of
+ the following filter names and a corresponding
+ value: id, label.
+ ```
+
+## admin view create
+
+Create a new explore application view configuration.
+
+```shell-session title="Usage"
+cmemc admin view create [OPTIONS] PROFILE_ID
+```
+
+
+
+
+The new profile is created with its ID and label only. Use the `admin view update` command to set configuration values such as enableCompanion or module toggles.
+
+!!! note
+ Application view configurations can be listed with the `admin view list` command.
+
+
+
+
+??? info "Options"
+ ```text
+
+ --label TEXT Label for the application view configuration. Defaults to the
+ profile ID.
+ ```
+
+## admin view update
+
+Update a key in an existing explore application view configuration.
+
+```shell-session title="Usage"
+cmemc admin view update [OPTIONS] PROFILE_ID
+```
+
+
+
+
+Any configuration key can be updated, including nested module keys. All other fields are preserved.
+
+```shell-session title="Example"
+cmemc admin view update my-profile --key enableCompanion --value true
+```
+
+
+```shell-session title="Example"
+cmemc admin view update my-profile --key modules.marketplaceModuleConfiguration.enabled --value false
+```
+
+
+
+
+??? info "Options"
+ ```text
+
+ --key TEXT The configuration key to update. Supports nested keys using
+ 'a.b[i].c' notation, e.g.
+ modules.marketplaceModuleConfiguration.enabled. [required]
+ --value TEXT The new value. Parsed as JSON when possible (e.g. true, false,
+ 1), otherwise used as a plain string. [required]
+ ```
+
+## admin view inspect
+
+Inspect the configuration of an application view profile.
+
+```shell-session title="Usage"
+cmemc admin view inspect [OPTIONS] PROFILE_ID
+```
+
+
+
+
+For accessing nested configuration values, use the following notation: exploreGraphLists[4].comments[0]
+
+!!! note
+ Some shell environments require quotes around expressions with square brackets.
+
+
+Examples: cmemc admin view inspect my-profile
+
+cmemc admin view inspect my-profile `--key` enable
+
+cmemc admin view inspect my-profile `--key` "exploreGraphLists[4].comments[0]"
+
+
+
+
+??? info "Options"
+ ```text
+
+ --key TEXT Get a specific key only from the configuration.
+ --raw Outputs raw JSON.
+ ```
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md
index 0c2bab3f6..4814883cc 100644
--- a/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md
+++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md
@@ -37,6 +37,9 @@ The file name is optional and will be generated with by the template if absent.
option, so use it with care.
--type TEXT Type of the exported workspace file.
[default: xmlZip]
+ --without-userdata Do not export user-identifying metadata
+ (creation/modification timestamps and account
+ names) in the exported archive.
-t, --filename-template TEXT Template for the export file name. Possible
placeholders are (Jinja2): {{connection}}
(from the --connection option) and {{date}}
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/graph/insights/index.md b/docs/automate/cmemc-command-line-interface/command-reference/graph/insights/index.md
index f228757af..dfaccc9f2 100644
--- a/docs/automate/cmemc-command-line-interface/command-reference/graph/insights/index.md
+++ b/docs/automate/cmemc-command-line-interface/command-reference/graph/insights/index.md
@@ -121,7 +121,7 @@ After the update, the snapshot is hot-swapped.
following filter names and a corresponding
value: id, main-graph, status, affected-
graph, valid.
- -a, --all Delete all snapshots.
+ -a, --all Update all snapshots.
--wait Wait until snapshot creation is done.
--polling-interval INTEGER RANGE
How many seconds to wait between status
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/index.md b/docs/automate/cmemc-command-line-interface/command-reference/index.md
index 456fdfdef..939627609 100644
--- a/docs/automate/cmemc-command-line-interface/command-reference/index.md
+++ b/docs/automate/cmemc-command-line-interface/command-reference/index.md
@@ -48,6 +48,13 @@ tags:
| [admin user](admin/user/index.md) | [delete](admin/user/index.md#admin-user-delete) | Delete user accounts. |
| [admin user](admin/user/index.md) | [password](admin/user/index.md#admin-user-password) | Change the password of a user account. |
| [admin user](admin/user/index.md) | [open](admin/user/index.md#admin-user-open) | Open user in the browser. |
+| [admin view](admin/view/index.md) | [list](admin/view/index.md#admin-view-list) | List explore application view configurations. |
+| [admin view](admin/view/index.md) | [export](admin/view/index.md#admin-view-export) | Export application view configurations to a JSON file. |
+| [admin view](admin/view/index.md) | [import](admin/view/index.md#admin-view-import) | Import application view configurations from a JSON file. |
+| [admin view](admin/view/index.md) | [delete](admin/view/index.md#admin-view-delete) | Delete custom application view configurations. |
+| [admin view](admin/view/index.md) | [create](admin/view/index.md#admin-view-create) | Create a new explore application view configuration. |
+| [admin view](admin/view/index.md) | [update](admin/view/index.md#admin-view-update) | Update a key in an existing explore application view configuration. |
+| [admin view](admin/view/index.md) | [inspect](admin/view/index.md#admin-view-inspect) | Inspect the configuration of an application view profile. |
| [admin workspace](admin/workspace/index.md) | [export](admin/workspace/index.md#admin-workspace-export) | Export the complete workspace (all projects) to a ZIP file. |
| [admin workspace](admin/workspace/index.md) | [import](admin/workspace/index.md#admin-workspace-import) | Import the workspace from a file. |
| [admin workspace](admin/workspace/index.md) | [reload](admin/workspace/index.md#admin-workspace-reload) | Reload the workspace from the backend. |
@@ -90,6 +97,14 @@ tags:
| [graph validation](graph/validation/index.md) | [inspect](graph/validation/index.md#graph-validation-inspect) | List and inspect errors found with a validation process. |
| [graph validation](graph/validation/index.md) | [cancel](graph/validation/index.md#graph-validation-cancel) | Cancel a running validation process. |
| [graph validation](graph/validation/index.md) | [export](graph/validation/index.md#graph-validation-export) | Export a report of finished validations. |
+| [package](package/index.md) | [inspect](package/index.md#package-inspect) | Inspect the manifest of a package. |
+| [package](package/index.md) | [list](package/index.md#package-list) | List installed packages. |
+| [package](package/index.md) | [install](package/index.md#package-install) | Install packages. |
+| [package](package/index.md) | [uninstall](package/index.md#package-uninstall) | Uninstall installed packages. |
+| [package](package/index.md) | [export](package/index.md#package-export) | Export installed packages to package directories. |
+| [package](package/index.md) | [build](package/index.md#package-build) | Build a package archive from a package directory. |
+| [package](package/index.md) | [publish](package/index.md#package-publish) | Publish a package archive to the marketplace server. |
+| [package](package/index.md) | [search](package/index.md#package-search) | Search for available packages with a given search text. |
| [project](project/index.md) | [open](project/index.md#project-open) | Open projects in the browser. |
| [project](project/index.md) | [list](project/index.md#project-list) | List available projects. |
| [project](project/index.md) | [export](project/index.md#project-export) | Export projects to files. |
@@ -97,6 +112,7 @@ tags:
| [project](project/index.md) | [delete](project/index.md#project-delete) | Delete projects. |
| [project](project/index.md) | [create](project/index.md#project-create) | Create projects. |
| [project](project/index.md) | [reload](project/index.md#project-reload) | Reload projects from the workspace provider. |
+| [project](project/index.md) | [status](project/index.md#project-status) | Show task loading errors of projects. |
| [project file](project/file/index.md) | [list](project/file/index.md#project-file-list) | List available file resources. |
| [project file](project/file/index.md) | [delete](project/file/index.md#project-file-delete) | Delete file resources. |
| [project file](project/file/index.md) | [download](project/file/index.md#project-file-download) | Download file resources to the local file system. |
@@ -120,7 +136,7 @@ tags:
| [query](query/index.md) | [delete](query/index.md#query-delete) | Delete queries from a query catalog. |
| [vocabulary](vocabulary/index.md) | [open](vocabulary/index.md#vocabulary-open) | Open / explore a vocabulary graph in the browser. |
| [vocabulary](vocabulary/index.md) | [list](vocabulary/index.md#vocabulary-list) | Output a list of vocabularies. |
-| [vocabulary](vocabulary/index.md) | [install](vocabulary/index.md#vocabulary-install) | Install one or more vocabularies from the catalog. |
+| [vocabulary](vocabulary/index.md) | [install](vocabulary/index.md#vocabulary-install) | Install one or more vocabularies from the catalog (deprecated). |
| [vocabulary](vocabulary/index.md) | [uninstall](vocabulary/index.md#vocabulary-uninstall) | Uninstall one or more vocabularies. |
| [vocabulary](vocabulary/index.md) | [import](vocabulary/index.md#vocabulary-import) | Import a turtle file as a vocabulary. |
| [vocabulary cache](vocabulary/cache/index.md) | [update](vocabulary/cache/index.md#vocabulary-cache-update) | Reload / updates the data integration cache for a vocabulary. |
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/package/index.md b/docs/automate/cmemc-command-line-interface/command-reference/package/index.md
new file mode 100644
index 000000000..09b091d76
--- /dev/null
+++ b/docs/automate/cmemc-command-line-interface/command-reference/package/index.md
@@ -0,0 +1,229 @@
+---
+title: "cmemc: Command Group - package"
+description: "List, (un)install, export, create, or inspect packages."
+icon: material/shopping
+tags:
+ - cmemc
+ - Package
+---
+
+# package Command Group
+
+
+
+List, (un)install, export, create, or inspect packages.
+
+
+## package inspect
+
+Inspect the manifest of a package.
+
+```shell-session title="Usage"
+cmemc package inspect [OPTIONS] PACKAGE_PATH
+```
+
+
+
+
+
+??? info "Options"
+ ```text
+
+ --key TEXT Get a specific key only from the manifest.
+ --raw Outputs raw JSON.
+ ```
+
+## package list
+
+List installed packages.
+
+```shell-session title="Usage"
+cmemc package list [OPTIONS]
+```
+
+
+
+
+
+??? info "Options"
+ ```text
+
+ --filter ... Filter installed packages by one of the following
+ filter names and a corresponding value: type, name,
+ id.
+ --id-only Lists only package IDs. This is useful for piping
+ the IDs into other commands.
+ --raw Outputs raw JSON.
+ ```
+
+## package install
+
+Install packages.
+
+```shell-session title="Usage"
+cmemc package install [OPTIONS] [PACKAGE_ID]
+```
+
+
+
+
+This command installs a package either from the marketplace or from local package archives (.cpa) or package directories.
+
+If a local package is chosen which has unzipped project directories, the installation will handle the zipping silently. See the `package export` command for more information.
+
+
+
+??? info "Options"
+ ```text
+
+ -i, --input PATH Install a package from a package archive (.cpa) or
+ directory.
+ --replace Replace (overwrite) an existing package version or
+ package content, if present.
+ --no-cache Disable using cached package versions.
+ --ignore-lock Ignore and release the package lock file for this
+ operation. Use this to recover from a stale lock
+ left by an interrupted run. Dangerous under
+ concurrent access (it removes the lock other
+ processes rely on); use with care.
+ --version TEXT Specific version to install from the marketplace.
+ Defaults to the latest version.
+ --marketplace-url TEXT Base URL of the Marketplace - uses environment
+ variable ECCENCA_MARKETPLACE_URL if available.
+ [default: https://eccenca.market]
+ ```
+
+## package uninstall
+
+Uninstall installed packages.
+
+```shell-session title="Usage"
+cmemc package uninstall [OPTIONS] [PACKAGE_ID]
+```
+
+
+
+
+
+??? info "Options"
+ ```text
+
+ --ignore-lock Ignore and release the package lock file for this
+ operation. Use this to recover from a stale lock
+ left by an interrupted run. Dangerous under
+ concurrent access (it removes the lock other
+ processes rely on); use with care.
+ --filter ... Filter installed packages by one of the following
+ filter names and a corresponding value: type, name,
+ id.
+ -a, --all Uninstall all packages. This is a dangerous option,
+ so use it with care.
+ ```
+
+## package export
+
+Export installed packages to package directories.
+
+```shell-session title="Usage"
+$ cmemc package export [OPTIONS] [PACKAGE_ID]
+```
+
+
+
+
+
+??? info "Options"
+ ```text
+
+ --mime_type [text/turtle|text/turtle+pretty]
+ Choose the MIME type for graphs when
+ exporting packages. [default:
+ text/turtle+pretty]
+ --filter ... Filter installed packages by one of the
+ following filter names and a corresponding
+ value: type, name, id.
+ -a, --all Export all installed packages.
+ --output-dir DIRECTORY Create package directories in this base
+ directory. [default: .]
+ --replace Replace (overwrite) existing files, if
+ present.
+ --extract Extract the project files specified in the
+ manifest and replace the archive with
+ itsextracted directory. This is useful for
+ version controlled package directories.
+ ```
+
+## package build
+
+Build a package archive from a package directory.
+
+```shell-session title="Usage"
+cmemc package build [OPTIONS] PACKAGE_DIRECTORY
+```
+
+
+
+
+This command processes a package directory, validates its content including the manifest, and creates a versioned Corporate Memory Package Archive (.cpa) with the following naming convention: {package_id}-v{version}.cpa
+
+If the package contains an extracted project (directory) instead of a ZIP, it is zipped automatically in a temporary copy — the original package directory is never modified. The manifest still need to reference the project ZIP. See the `package export` command for more information.
+
+Package archives can be published to the marketplace using the `package publish` command.
+
+
+
+??? info "Options"
+ ```text
+
+ --version TEXT Set the package version.
+ --replace Replace package archive, if present.
+ --output-dir DIRECTORY Create the package archive in a specific directory.
+ [default: .]
+ ```
+
+## package publish
+
+Publish a package archive to the marketplace server.
+
+```shell-session title="Usage"
+cmemc package publish [OPTIONS] PACKAGE_ARCHIVE
+```
+
+
+
+
+
+??? info "Options"
+ ```text
+
+ --timeout INTEGER Timeout for marketplace requests.
+ --marketplace-url TEXT Base URL of the Marketplace - uses environment
+ variable ECCENCA_MARKETPLACE_URL if available.
+ [default: https://eccenca.market]
+ --marketplace-account TEXT Marketplace account - uses environment variable
+ ECCENCA_MARKETPLACE_ACCOUNT if available.
+ --marketplace-password TEXT Marketplace password - uses environment
+ variable ECCENCA_MARKETPLACE_PASSWORD if
+ available.
+ ```
+
+## package search
+
+Search for available packages with a given search text.
+
+```shell-session title="Usage"
+cmemc package search [OPTIONS] [SEARCH_TERMS]...
+```
+
+
+
+
+
+??? info "Options"
+ ```text
+
+ --raw Outputs raw JSON.
+ --marketplace-url TEXT Base URL of the Marketplace - uses environment
+ variable ECCENCA_MARKETPLACE_URL if available.
+ [default: https://eccenca.market]
+ ```
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/project/index.md b/docs/automate/cmemc-command-line-interface/command-reference/project/index.md
index 22b732e37..a9fb682fa 100644
--- a/docs/automate/cmemc-command-line-interface/command-reference/project/index.md
+++ b/docs/automate/cmemc-command-line-interface/command-reference/project/index.md
@@ -117,6 +117,10 @@ $ cmemc config list | parallel -I% cmemc -c % project export --all -t "dump/{{co
note that not all export types are
extractable.
--help-types Lists all possible export types.
+ --without-userdata Do not export user-identifying metadata
+ (creation/modification timestamps and account
+ names) in the exported archives or
+ directories.
```
## project import
@@ -233,3 +237,29 @@ This command reloads all tasks of a project from the workspace provider. This is
-a, --all Reload all projects
```
+
+## project status
+
+Show task loading errors of projects.
+
+```shell-session title="Usage"
+cmemc project status [OPTIONS] [PROJECT_IDS]...
+```
+
+
+
+
+This command checks the given projects (or all projects with the `--all` option) for task loading errors and outputs them as warnings.
+
+Use this to find out if your projects have tasks which could not be loaded, e.g. because a needed plugin is not installed.
+
+
+
+??? info "Options"
+ ```text
+
+ -a, --all Check all projects
+ --exit-1 Exit with code 1 if at least one project has task loading
+ errors.
+ --raw Outputs raw JSON of the task loading status.
+ ```
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md b/docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md
index 45c9ed5bf..706adf634 100644
--- a/docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md
+++ b/docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md
@@ -59,7 +59,7 @@ cmemc project variable get [OPTIONS] VARIABLE_ID
Use the ``--key`` option to specify which information you want to get.
!!! note
- Only the `value` key is always available on a project variable. Static value variables have no `template` key, and the `description` key is optional for both types of variables.
+ Only the `value` key is always available on a project variable. Static value variables have no `template` key, and the `description` key is optional for both types of variables. Use `--raw` to access all fields.
diff --git a/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md b/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md
index 3dc930188..9bf3966eb 100644
--- a/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md
+++ b/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md
@@ -59,7 +59,7 @@ Vocabularies are graphs (see `graph` command group) which consists of class and
## vocabulary install
-Install one or more vocabularies from the catalog.
+Install one or more vocabularies from the catalog (deprecated).
```shell-session title="Usage"
cmemc vocabulary install [OPTIONS] [IRIS]...
@@ -70,6 +70,10 @@ cmemc vocabulary install [OPTIONS] [IRIS]...
Vocabularies are identified by their graph IRI. Installable vocabularies can be listed with the vocabulary list command.
+!!! note
+ This command is deprecated. Vocabularies are now managed via packages; use the `package` command group instead.
+
+
??? info "Options"
@@ -92,6 +96,10 @@ cmemc vocabulary uninstall [OPTIONS] [IRIS]...
Vocabularies are identified by their graph IRI. Already installed vocabularies can be listed with the vocabulary list command.
+!!! note
+ This command is deprecated. Vocabularies are now managed via packages; use the `package` command group instead. Vocabularies which are managed by a package can not be uninstalled with this command.
+
+
??? info "Options"
diff --git a/docs/build/mapping-creator/index.md b/docs/build/mapping-creator/index.md
index a15409dcb..89bb070bb 100644
--- a/docs/build/mapping-creator/index.md
+++ b/docs/build/mapping-creator/index.md
@@ -7,6 +7,12 @@ tags:
---
# Mapping Creator
+!!! info "AI Disclaimer"
+
+ Mapping Creator uses AI-generated suggestions.
+ AI-generated content may be inaccurate or incomplete.
+ Please review all suggestions carefully before applying them.
+
## Configuration Info
A specific configuration is required to activate this feature.
diff --git a/docs/build/reference/customtask/.pages b/docs/build/reference/customtask/.pages
index cd2d435af..422a54c35 100644
--- a/docs/build/reference/customtask/.pages
+++ b/docs/build/reference/customtask/.pages
@@ -21,6 +21,7 @@ nav:
- "Execute Spark function": SparkFunction.md
- "Extract from PDF files": cmem_plugin_pdf_extract-pdf_extract-PdfExtract.md
- "Generate base36 IRDIs": cmem_plugin_irdi-workflow-irdi_plugin-IrdiPlugin.md
+ - "Generate random values": cmem_plugin_random-GenerateEntities.md
- "Generate SHACL shapes from data": cmem_plugin_shapes-plugin_shapes-ShapesPlugin.md
- "Get project files": getProjectFiles.md
- "Get workflow report": cmem_plugin_wfreports_get_report.md
@@ -28,6 +29,7 @@ nav:
- "Join tables": Merge.md
- "jq": cmem-plugin-jq-workflow.md
- "JQL query": cmem_plugin_jira-JqlQuery.md
+ - "JSON to File": jsonToFile.md
- "Kafka Consumer (Receive Messages)": cmem_plugin_kafka-ReceiveMessages.md
- "Kafka Producer (Send Messages)": cmem_plugin_kafka-SendMessages.md
- "List Nextcloud files": cmem_plugin_nextcloud-List.md
@@ -48,8 +50,10 @@ nav:
- "Search addresses": SearchAddresses.md
- "Search for Logs": cmem_plugin_logpoint-search_logs_task-RetrieveLogs.md
- "Search Vector Embeddings": cmem_plugin_pgvector-Search.md
+ - "Select random entities": cmem_plugin_random-SelectEntities.md
- "Send email": SendEMail.md
- "Send Mattermost messages": cmem_plugin_mattermost.md
+ - "Set execution variable": setExecutionVariableOperator.md
- "Set or Overwrite parameters": cmem_plugin_parameters-ParametersPlugin.md
- "Set parameters": setParameters.md
- "SHACL validation with pySHACL": shacl-pyshacl.md
diff --git a/docs/build/reference/customtask/DistinctBy.md b/docs/build/reference/customtask/DistinctBy.md
index 64fd9bcb7..49d149e76 100644
--- a/docs/build/reference/customtask/DistinctBy.md
+++ b/docs/build/reference/customtask/DistinctBy.md
@@ -1,6 +1,6 @@
---
title: "Distinct by"
-description: "Removes duplicated entities based on a user-defined path. Note that this operator does not retain the order of the entities. Since this operator accepts a flexible input schema, it can only be connected to operators that provide a non-flexible output schema. A typical way to achieve this is to place a transform operator before it, which produces a fixed output schema."
+description: "Removes duplicated entities based on user-defined paths. Duplicates can be resolved by keeping the first or last entity, or by keeping the entity with the minimum or maximum value of a compare path."
icon: octicons/cross-reference-24
tags:
- WorkflowTask
@@ -12,20 +12,88 @@ tags:
-Removes duplicated entities based on a user-defined path. Note that this operator does not retain the order of the entities.
+## 1. Introduction
-Since this operator accepts a flexible input schema, it can only be connected to operators that provide a non-flexible output schema.
-A typical way to achieve this is to place a transform operator before it, which produces a fixed output schema.
+The **Distinct by** operator removes duplicated entities based on one or more user-defined paths.
+
+All entities that share the same values at all **distinct paths** (one path per line) are considered duplicates of each other,
+and exactly one of them is kept according to the chosen **duplicate resolution strategy**.
+
+Note that this operator does not retain the order of the entities.
+
+## 2. Duplicate Resolution Strategies
+
+- **Keep first duplicate** – keeps the first entity encountered in the input.
+- **Keep last duplicate** – keeps the last entity encountered in the input.
+- **Keep duplicate with minimum value** – keeps the entity with the lowest value at the **compare path**.
+- **Keep duplicate with maximum value** – keeps the entity with the highest value at the **compare path**.
+
+The first and last strategies depend on the order of the input entities.
+The minimum and maximum strategies are independent of the input order and follow these rules:
+
+- Entities that have a value at the compare path win over entities that do not have one.
+- If an entity has multiple values at the compare path, its lowest (minimum strategy) or highest (maximum strategy) value is used for comparison.
+- On ties, the first encountered entity is kept.
+
+## 3. Compare Order
+
+The order used to compare values for the *Keep duplicate with minimum/maximum value* strategies can be configured:
+
+- `Autodetect` (default) – if both values are numbers, numerical order is used; otherwise, alphabetical order is used.
+- `Alphabetical` – values are always compared as strings.
+- `Numerical` – values are compared as decimal numbers; values that cannot be parsed never win a comparison.
+- `Integer` – values are compared as integers; values that cannot be parsed never win a comparison.
+
+## 4. Example
+
+Input:
+
+| key | value |
+|-----|-------|
+| A | 2 |
+| A | 1 |
+| B | 2 |
+| A | 3 |
+| B | 1 |
+
+Configuration:
+
+| Parameter | Value |
+|--------------------|-----------------------------------|
+| Distinct paths | `key` |
+| Resolve duplicates | Keep duplicate with minimum value |
+| Compare path | `value` |
+| Compare order | Autodetect (default) |
+
+Output:
+
+| key | value |
+|-----|-------|
+| A | 1 |
+| B | 1 |
+
+For each distinct `key`, only the entity with the lowest `value` is kept.
+
+## 5. Connecting the Operator
+
+Since this operator accepts a flexible input schema, it can only be connected to operators that provide a
+non-flexible output schema or an explicit schema, such as CSV datasets, which can be connected directly.
+For other inputs, a typical way to achieve this is to place a transform operator before it,
+which produces a fixed output schema.
+
+## 6. Technical Notes
+
+- Entities are buffered in a temporary disk-based store, so the operator also works on datasets that do not fit into memory.
## Parameter
-### Distinct path
+### Distinct paths
-Entities that share this path will be deduplicated.
+Entities that share the values of all these paths will be deduplicated. One path per line.
- ID: `distinctPath`
-- Datatype: `string`
+- Datatype: `multiline string`
- Default Value: `None`
@@ -38,6 +106,26 @@ Strategy to resolve duplicates.
- Datatype: `enumeration`
- Default Value: `keepLast`
+
+
+### Compare path
+
+Path whose value decides which duplicate is kept for the 'Keep duplicate with minimum/maximum value' strategies. Ignored otherwise.
+
+- ID: `comparePath`
+- Datatype: `string`
+- Default Value: `None`
+
+
+
+### Compare order
+
+Order used to compare values for the 'Keep duplicate with minimum/maximum value' strategies. Per default, if both values are numbers, numerical order is used for comparison. Otherwise, alphabetical order is used. Ignored for other strategies.
+
+- ID: `order`
+- Datatype: `enumeration`
+- Default Value: `Autodetect`
+
## Advanced Parameter
`None`
diff --git a/docs/build/reference/customtask/JsonParserOperator.md b/docs/build/reference/customtask/JsonParserOperator.md
index 968b7b255..8e2716a09 100644
--- a/docs/build/reference/customtask/JsonParserOperator.md
+++ b/docs/build/reference/customtask/JsonParserOperator.md
@@ -1,6 +1,6 @@
---
title: "Parse JSON"
-description: "Parses an incoming entity as a JSON dataset. Typically, it is used before a transformation task. Takes exactly one input of which only the first entity is processed."
+description: "Parses a JSON string held in a field on each incoming entity. Typically used before a transformation task. Takes exactly one input."
icon: octicons/cross-reference-24
tags:
- WorkflowTask
@@ -12,7 +12,89 @@ tags:
-Parses an incoming entity as a JSON dataset. Typically, it is used before a transformation task. Takes exactly one input of which only the first entity is processed.
+## Parse JSON
+
+Parse JSON is a workflow operator that extracts structured data from a JSON string held in a field on incoming
+entities. It sits inside a pipeline between an upstream source and a downstream operator — typically a transformation
+— and turns the JSON content into entities ready for further processing.
+
+The operator is useful whenever JSON arrives not as a file but as a string stored in a field: the result of an HTTP
+request, a column in a database record, a payload embedded in another dataset. Parse JSON consumes that string in place
+and produces entities for the rest of the pipeline.
+
+## Input
+
+Parse JSON accepts exactly one input. It iterates over every entity in that input, extracts the JSON string from a
+field on each entity, parses it, and produces output entities from its contents. The output entities from all input
+entities are concatenated into a single stream for the downstream operator.
+
+Which field is used as the JSON source is controlled by the *Input path* parameter. When set, Parse JSON looks for the
+JSON string at the given path expression. When left empty, it reads the value of the first available field. If no value
+is found at the expected location, or if the field is empty, the operator raises an error and stops.
+
+## Output
+
+The output of Parse JSON is a set of entities extracted from the parsed JSON structure. These entities are shaped by
+three parameters: *Base path*, *URI suffix pattern*, and *Navigate into arrays*.
+
+**Base path** determines the starting point within the JSON document. When set to a path such as */Persons/Person*, only
+elements found at that location are read as entities; everything else in the document is ignored. When left empty, all
+direct children of the root element are read.
+
+**URI suffix pattern** controls how the URIs of the output entities are constructed. The pattern is evaluated relative
+to the URI of the input entity: whatever suffix is specified gets appended to that URI. For example, a pattern of
+*/{id}* applied to an input entity with URI * produces URIs by appending the value of the
+*id* field — so an entity whose *id* is *7* receives URI *. When left empty, URIs are
+generated automatically.
+
+**Navigate into arrays** controls how JSON arrays are handled during path traversal. In JSON, an array is an anonymous
+container with no name of its own — just a list of items. When a path expression crosses an array mid-way, it is
+ambiguous whether the array itself or its contents is the intended target. This parameter resolves that ambiguity. When
+enabled — the default — the operator descends into arrays automatically, so a path like */Persons/Person* reaches the
+Person elements directly even if Persons is an array. When disabled, the array is treated as an explicit step in the
+path: to reach the same Person elements, the path must be written as */Persons/#array/Person*.
+
+Parse JSON supports the same path expressions as the JSON dataset, including wildcards for children and descendants,
+backward paths, and special paths for hash IDs, key names, and array elements.
+
+## Schema
+
+Before producing output entities, Parse JSON needs to know which fields to extract. The set of fields — the output
+schema — is requested by the downstream operator and reaches Parse JSON before any parsing happens. For each
+requested field, Parse JSON evaluates its path expression against the parsed JSON, starting from the configured base
+path, and writes the resulting values onto the output entity. When the downstream operator requests a multi-entity
+schema, Parse JSON produces the root entities and the nested sub-entity tables in a single pass. In practice that
+operator is a transformation.
+
+Parse JSON cannot be connected directly to a dataset. A dataset declares no fields to read, so Parse JSON has nothing
+to extract. Workflows that wire Parse JSON straight into a dataset fail at execution time with an error naming the
+missing schema and asking for a downstream operator that declares one.
+
+## Example
+
+An upstream operator produces a single entity with the following JSON string in its first field. Because *Input path*
+is not set, Parse JSON reads from that first field by default.
+
+```json
+{
+ "response": {
+ "persons": [
+ { "id": "1", "name": "Alice", "city": "Berlin" },
+ { "id": "2", "name": "Bob", "city": "London" }
+ ]
+ }
+}
+```
+
+With *Base path* set to */response/persons*, Parse JSON navigates past the response wrapper and reads each element of
+the persons array as a separate entity. The array is crossed automatically because *Navigate into arrays* is enabled.
+With *URI suffix pattern* set to */{id}*, the two output entities receive URIs constructed by appending the value of
+their id field to the URI of the input entity.
+
+The result is two entities — one for Alice, one for Bob — each with id, name, and city as fields.
+
+If the upstream operator produces several entities, Parse JSON parses the JSON string in each one in turn and
+concatenates the resulting entities into a single output stream.
## Parameter
@@ -58,3 +140,8 @@ Navigate into arrays automatically. If set to false, the `#array` path operator
## Advanced Parameter
`None`
+
+## Related Plugins
+
+- [json](../dataset/json.md) — Parse JSON and the JSON dataset share path syntax but not a content source: Parse JSON reads from a field value in an incoming entity, the JSON dataset from a file resource it opens directly.
+- [jsonToFile](jsonToFile.md) — Parse JSON parses a JSON string on each input entity into structured entities driven by a downstream schema; JSON to File writes the same kind of input to a file for downstream operators that read files.
diff --git a/docs/build/reference/customtask/clearDataset.md b/docs/build/reference/customtask/clearDataset.md
index 04ebd382c..6e745420a 100644
--- a/docs/build/reference/customtask/clearDataset.md
+++ b/docs/build/reference/customtask/clearDataset.md
@@ -12,7 +12,38 @@ tags:
-Clears the dataset that is connected to the output of this operator.
+Clears the dataset that is connected to the output of this operator, e.g. deletes all triples of a Knowledge Graph or removes the contents of a CSV file.
+
+The operator itself only emits a clear instruction. The **dataset node connected to its output** performs the physical clear when that node executes. Clearing a read-only dataset fails the workflow.
+
+## Execution order
+
+A dataset node fed by this operator (a "clear node") clears the dataset when the node executes. Its execution order relative to other nodes writing to the same dataset is undefined unless made explicit. Without an explicit order the clear may run before or after those writes. A clear that runs after them silently removes the just-written data. The workflow reports a warning when it detects this situation.
+
+The order is explicit if one of the following holds:
+
+- A (transitive) data-flow or dependency path connects the clear node and the writing node; the direction of the path determines which of the two runs first.
+- Clear and write happen on the same dataset node; the input port order decides: a clear input on an earlier port runs before data inputs on later ports.
+
+## Recipes
+
+### Clear, then write single dataset
+
+Example: a `Customers` dataset that is rebuilt from scratch on every run.
+
+Connect the output of this operator to the first input port of the `Customers` node and the output of the data-producing task to a later input port of the same node. The port order guarantees that the dataset is emptied before the new data is written. No dependency connections are needed.
+
+### Clear before write on separate nodes
+
+If the clear cannot share a node with the writes (for example because different branches of the workflow write to their own `Customers` nodes), place `Customers` on the canvas one more time and connect the output of this operator to it (the clear node). Then draw a *dependency* connection from the clear node to every node that `Customers` is written to, so all writes run after the clear.
+
+A dataset can also be cleared several times in one workflow (e.g. to reuse it freshly in a later phase), but then each clear node must be ordered against all writes this way. A clear that is left unordered may run after the writes and silently remove them.
+
+### Write before clear
+
+Example: a temporary `Staging` dataset that is filled and consumed during the workflow and should be left empty at the end.
+
+Draw a dependency connection from the node that writes to `Staging` to this operator (or to its clear node), so the clear runs after the write as the final step.
## Parameter
diff --git a/docs/build/reference/customtask/cmem_plugin_llm-ExecuteInstructions.md b/docs/build/reference/customtask/cmem_plugin_llm-ExecuteInstructions.md
index a10ae6072..7d136e10b 100644
--- a/docs/build/reference/customtask/cmem_plugin_llm-ExecuteInstructions.md
+++ b/docs/build/reference/customtask/cmem_plugin_llm-ExecuteInstructions.md
@@ -243,7 +243,7 @@ A list of messages comprising the conversation compatible with OpenAI chat compl
[
{
"role": "developer",
- "content": "You are a helpful assistant."
+ "content": "{{ developer_prompt }}"
},
{
"role": "user",
@@ -254,6 +254,16 @@ A list of messages comprising the conversation compatible with OpenAI chat compl
+### Developer Prompt Template
+
+The developer (system) prompt inserted at `{{ developer_prompt }}` in the Messages Template. Defines the LLM's role and behaviour. Leave the Messages Template's developer content hardcoded if this parameter is not needed.
+
+- ID: `developer_prompt_template`
+- Datatype: `code-jinja2`
+- Default Value: `You are a helpful assistant.`
+
+
+
### Output Format
Specifying the format that the model must output. Possible values are `TEXT` - Standard text output, `STRUCTURED_OUTPUT` - output follows a given schema. Add your schema as Pydantic model in the parameter below, `JSON_MODE` - a more basic version of the structured outputs feature where you have to add your structure to the prompt template.
diff --git a/docs/build/reference/customtask/cmem_plugin_random-GenerateEntities.md b/docs/build/reference/customtask/cmem_plugin_random-GenerateEntities.md
new file mode 100644
index 000000000..367f64dd7
--- /dev/null
+++ b/docs/build/reference/customtask/cmem_plugin_random-GenerateEntities.md
@@ -0,0 +1,92 @@
+---
+title: "Generate random values"
+description: "Generates entities with random values."
+icon: octicons/cross-reference-24
+tags:
+ - WorkflowTask
+ - PythonPlugin
+---
+
+# Generate random values
+
+
+
+!!! note inline end "Python Plugin"
+
+ This operator is part of a Python Plugin Package.
+ In order to use it, you need to install it,
+ e.g. with cmemc.
+
+This workflow task generates entities with random values.
+
+The plugin generates X entities with Y values, each value has a length of Z.
+
+All parameters can be configured with the parameters.
+
+Warning: Please note that high numbers in any of the parameters will result in more
+computational time as well as disk usage to save the entities.
+For example, while a configuration of 100 entities with 100 values / 100 characters
+results in a 1.4 MB CSV file (which is generated in milliseconds),
+a configuration of 1000 entities with 1000 values / 1000 characters will result
+already in a 1.3 GB CSV file.
+
+
+## Parameter
+
+### Number of Entities (Rows)
+
+How many rows will be created per run. Depending on your output dataset, this will result in different number of resources (Knowledge Graph),rows (CSV) or objects (JSON).
+
+- ID: `number_of_entities`
+- Datatype: `Long`
+- Default Value: `10`
+
+
+
+### Number of Values (Columns)
+
+How many values are created per entity / row. Depending on your output dataset, this will result in different number of datatype properties (Knowledge Graph), columns (CSV) or attributes (JSON).
+
+- ID: `number_of_values`
+- Datatype: `Long`
+- Default Value: `5`
+
+
+
+### String Length
+
+How long (in characters) should each value be.
+
+- ID: `string_length`
+- Datatype: `Long`
+- Default Value: `16`
+
+## Advanced Parameter
+
+### Random Function
+
+
+
+- ID: `random_function`
+- Datatype: `string`
+- Default Value: `token_urlsafe`
+
+
+
+### Property Namespace
+
+Output properties will have this namespace (following a number).
+
+- ID: `property_namespace`
+- Datatype: `string`
+- Default Value: `https://example.org/vocab/RandomValuePath`
+
+
+
+### Type Identifier
+
+Output entities will have this type identifier (IRI).
+
+- ID: `type_id`
+- Datatype: `string`
+- Default Value: `https://example.org/vocab/RandomValueRow`
diff --git a/docs/build/reference/customtask/cmem_plugin_random-SelectEntities.md b/docs/build/reference/customtask/cmem_plugin_random-SelectEntities.md
new file mode 100644
index 000000000..da69a5b15
--- /dev/null
+++ b/docs/build/reference/customtask/cmem_plugin_random-SelectEntities.md
@@ -0,0 +1,38 @@
+---
+title: "Select random entities"
+description: "Select X random entities from an input dataset."
+icon: octicons/cross-reference-24
+tags:
+ - WorkflowTask
+ - PythonPlugin
+---
+
+# Select random entities
+
+
+
+!!! note inline end "Python Plugin"
+
+ This operator is part of a Python Plugin Package.
+ In order to use it, you need to install it,
+ e.g. with cmemc.
+
+This workflow task selects X random entities from an input dataset
+using the standard pseudo-random generator (reservoir sampling).
+
+The task supports only flat entities. Hierarchical entities are ignored.
+
+
+## Parameter
+
+### Number of Entities
+
+How many entities should be selected.
+
+- ID: `number_of_entities`
+- Datatype: `Long`
+- Default Value: `10`
+
+## Advanced Parameter
+
+`None`
diff --git a/docs/build/reference/customtask/index.md b/docs/build/reference/customtask/index.md
index 24e586d63..59d10368b 100644
--- a/docs/build/reference/customtask/index.md
+++ b/docs/build/reference/customtask/index.md
@@ -24,7 +24,7 @@ A custom workflow task is an operator that can be used in a workflow.
| [Create Embeddings](cmem_plugin_llm-CreateEmbeddings.md) | Fetch and output LLM created embeddings from input entities. |
| [Create/Update Salesforce Objects](cmem_plugin_salesforce-workflow-operations-SobjectCreate.md) | Manipulate data in your organization's Salesforce account. |
| [Delete project files](deleteProjectFiles.md) | Removes file resources from the project based on a regular expression. |
- | [Distinct by](DistinctBy.md) | Removes duplicated entities based on a user-defined path. Note that this operator does not retain the order of the entities. Since this operator accepts a flexible input schema, it can only be connected to operators that provide a non-flexible output schema. A typical way to achieve this is to place a transform operator before it, which produces a fixed output schema. |
+ | [Distinct by](DistinctBy.md) | Removes duplicated entities based on user-defined paths. Duplicates can be resolved by keeping the first or last entity, or by keeping the entity with the minimum or maximum value of a compare path. |
| [Download file](downloadFile.md) | Downloads a file from a given URL. |
| [Download Nextcloud files](cmem_plugin_nextcloud-Download.md) | Download files from a given Nextcloud instance. |
| [Download Office 365 Files](cmem_plugin_office365-Download.md) | Download files from Microsoft OneDrive or Sites |
@@ -37,6 +37,7 @@ A custom workflow task is an operator that can be used in a workflow.
| [Execute Spark function](SparkFunction.md) | Applies a specified Scala function to a specified field. |
| [Extract from PDF files](cmem_plugin_pdf_extract-pdf_extract-PdfExtract.md) | Extract text and tables from PDF files |
| [Generate base36 IRDIs](cmem_plugin_irdi-workflow-irdi_plugin-IrdiPlugin.md) | Create unique ECLASS IRDIs. |
+ | [Generate random values](cmem_plugin_random-GenerateEntities.md) | Generates entities with random values. |
| [Generate SHACL shapes from data](cmem_plugin_shapes-plugin_shapes-ShapesPlugin.md) | Generate SHACL node and property shapes from a data graph |
| [Get project files](getProjectFiles.md) | Get file resources from the project. |
| [Get workflow report](cmem_plugin_wfreports_get_report.md) | Output a workflow execution report as a JSON file. |
@@ -44,6 +45,7 @@ A custom workflow task is an operator that can be used in a workflow.
| [Join tables](Merge.md) | Joins a set of inputs into a single table. Expects a list of entity tables and links. All entity tables are joined into the first entity table using the provided links. |
| [jq](cmem-plugin-jq-workflow.md) | Process a JSON document with a jq filter / program. |
| [JQL query](cmem_plugin_jira-JqlQuery.md) | Search and retrieve JIRA issues. |
+ | [JSON to File](jsonToFile.md) | Writes a JSON string held in a field on each valid incoming entity to a file. Depending on the output mode, it produces one file per entity, packs all entities into a single ZIP archive, or merges them into a single JSON array file. Produces a file entity downstream, suitable for wiring into a file-backed dataset or any operator that consumes file entities. |
| [Kafka Consumer (Receive Messages)](cmem_plugin_kafka-ReceiveMessages.md) | Reads messages from a Kafka topic and saves it to a messages dataset (Consumer). |
| [Kafka Producer (Send Messages)](cmem_plugin_kafka-SendMessages.md) | Reads a messages dataset and sends records to a Kafka topic (Producer). |
| [List Nextcloud files](cmem_plugin_nextcloud-List.md) | List directories and files from a given Nextcloud folder. |
@@ -54,7 +56,7 @@ A custom workflow task is an operator that can be used in a workflow.
| [Normalize units of measurement](ucumNormalizationTask.md) | Custom task that will substitute numeric values and pertaining unit symbols with a SI-system-unit normalized representation. |
| [OAuth2 Authentication](cmem_plugin_auth-workflow-auth-OAuth2.md) | Provide an OAuth2 access token for other tasks (via config port). |
| [Office 365 Upload Files](cmem_plugin_office365-Upload.md) | Upload files to OneDrive or a site Sharepoint |
- | [Parse JSON](JsonParserOperator.md) | Parses an incoming entity as a JSON dataset. Typically, it is used before a transformation task. Takes exactly one input of which only the first entity is processed. |
+ | [Parse JSON](JsonParserOperator.md) | Parses a JSON string held in a field on each incoming entity. Typically used before a transformation task. Takes exactly one input. |
| [Parse XML](XmlParserOperator.md) | Takes exactly one input and reads either the defined inputPath or the first value of the first entity as XML document. Then executes the given output entity schema similar to the XML dataset to construct the result entities. |
| [Parse YAML](cmem_plugin_yaml-parse.md) | Parses files, source code or input values as YAML documents. |
| [Pivot](Pivot.md) | The pivot operator takes data in separate rows, aggregates it and converts it into columns. |
@@ -64,15 +66,17 @@ A custom workflow task is an operator that can be used in a workflow.
| [Search addresses](SearchAddresses.md) | Looks up locations from textual descriptions using the configured geocoding API. Outputs results as RDF. |
| [Search for Logs](cmem_plugin_logpoint-search_logs_task-RetrieveLogs.md) | Search and retrieve logs from a Logpoint SIEM system with flexible schema output. |
| [Search Vector Embeddings](cmem_plugin_pgvector-Search.md) | Search for top-k metadata stored in Postgres Vector Store (PGVector). |
+ | [Select random entities](cmem_plugin_random-SelectEntities.md) | Select X random entities from an input dataset. |
| [Send email](SendEMail.md) | Sends an email using an SMTP server. |
| [Send Mattermost messages](cmem_plugin_mattermost.md) | Send messages to Mattermost channels and/or users. |
+ | [Set execution variable](setExecutionVariableOperator.md) | Sets an execution variable to the first value of the (single) input and passes the input through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution. |
| [Set or Overwrite parameters](cmem_plugin_parameters-ParametersPlugin.md) | Connect this task to a config port of another task in order to set or overwrite the parameter values of this task. |
| [Set parameters](setParameters.md) | Set and overwrite parameters of a task. |
| [SHACL validation with pySHACL](shacl-pyshacl.md) | Performs SHACL validation with pySHACL. |
| [SOQL query (Salesforce)](cmem_plugin_salesforce-SoqlQuery.md) | Executes a custom Salesforce Object Query (SOQL) to return sets of data your organization's Salesforce account. |
| [Spark SQL query](CustomSQLExecution.md) | Executes a custom SQL query on the first input Spark dataframe and returns the result as its output. |
| [SPARQL Construct query](sparqlCopyOperator.md) | A task that executes a SPARQL Construct query on a SPARQL enabled data source and outputs the SPARQL result. If the result should be written to the same RDF store it is read from, the SPARQL Update operator is preferable. |
- | [SPARQL Select query](sparqlSelectOperator.md) | A task that executes a SPARQL Select query on a SPARQL enabled data source and outputs the SPARQL result. If the SPARQL source is defined on a specific graph, a FROM clause will be added to the query at execution time, except when there already exists a GRAPH or FROM clause in the query. FROM NAMED clauses are not injected. |
+ | [SPARQL Select query](sparqlSelectOperator.md) | A task that executes a SPARQL Select query and outputs the SPARQL result. |
| [SPARQL Update query](sparqlUpdateOperator.md) | A task that outputs SPARQL Update queries for every entity from the input based on a SPARQL Update template. The output of this operator should be connected to the SPARQL datasets to which the results should be written. |
| [Split file](cmem_plugin_splitfile-plugin_splitfile-SplitFilePlugin.md) | Split a file into multiple parts with a specified size. |
| [SQL Update query](sqlUpdateQueryOperator.md) | A task that outputs SQL queries. The output of this operator should be connected to a remote SQL endpoint on which queries should be executed. |
diff --git a/docs/build/reference/customtask/jsonToFile.md b/docs/build/reference/customtask/jsonToFile.md
new file mode 100644
index 000000000..190cb0465
--- /dev/null
+++ b/docs/build/reference/customtask/jsonToFile.md
@@ -0,0 +1,177 @@
+---
+title: "JSON to File"
+description: "Writes a JSON string held in a field on each valid incoming entity to a file. Depending on the output mode, it produces one file per entity, packs all entities into a single ZIP archive, or merges them into a single JSON array file. Produces a file entity downstream, suitable for wiring into a file-backed dataset or any operator that consumes file entities."
+icon: octicons/cross-reference-24
+tags:
+ - WorkflowTask
+---
+
+# JSON to File
+
+
+
+
+
+## JSON to File
+
+The JSON to File operator takes a JSON string held in a field on each incoming entity and writes it to a file. The
+resulting file is surfaced downstream as a file entity, so any operator that accepts file entities — a file-backed
+dataset, another file-processing operator — can pick it up.
+
+The operator does not parse the JSON into structured entities. It validates that the value is well-formed JSON and then
+writes the JSON value to the file. The content type of the produced file is set via the *MIME type* parameter and
+defaults to *application/json*.
+
+## Input
+
+JSON to File accepts exactly one input. It iterates over every entity in that input, reads the JSON string from a
+field on each entity, and validates it. What it then produces depends on the *Output mode*: in *file* mode (the
+default) one file per entity; in *zip* mode a single ZIP archive with one entry per entity; in *jsonArray* mode a
+single file holding all the JSON values merged into one JSON array. The output is surfaced as a stream of file
+entities for the downstream operator.
+
+Which field holds the JSON string is controlled by the *Input path* parameter. When set, the operator reads the value
+at the given path expression. When left empty, it reads the value of the first property in the entity schema.
+
+## Invalid input
+
+Validation is per entity. An entity whose value is missing, empty, or not valid JSON is skipped and recorded as a
+warning on the execution report, naming the entity and the reason; it produces no output. The remaining valid entities
+are written as usual, so a single malformed record no longer discards the whole batch. This applies in all three output
+modes.
+
+When every entity is skipped, the operator still produces the mode's natural empty output: no files in *file* mode, an
+empty JSON array `[]` in *jsonArray* mode, and a ZIP archive with no entries in *zip* mode.
+
+Configuration errors are not per-entity and still fail the task — for example, an input count other than one, or an
+unsupported output mode.
+
+## Output
+
+The output of JSON to File is a stream of file entities. In *file* mode each file entity wraps a file holding the
+JSON value from one input entity. In *zip* mode the stream contains a single file entity whose backing file is a ZIP
+archive with one entry per input entity. In *jsonArray* mode the stream contains a single file entity backed by one
+file holding a JSON array of all the input values. In *file* and *jsonArray* mode the MIME type is the value of the
+*MIME type* parameter; in *zip* mode a default *application/json* is overridden to *application/zip* (see *MIME type*
+below). Downstream operators or datasets that accept file entities consume the stream directly.
+
+When the output is wired to a file-backed dataset, the dataset writes the file's bytes into its own resource. The end
+result is a file on disk — a JSON file per entity in *file* mode, a ZIP archive in *zip* mode, or a single JSON array
+file in *jsonArray* mode.
+
+## Parameters
+
+**Input path** controls which field of the input entity holds the JSON string. When set to a Silk path expression
+such as */jsonContent*, the operator reads the value at that path. When left empty, the operator reads the value of
+the first property in the entity schema.
+
+**MIME type** sets the content type of every produced file. Defaults to *application/json*. In *zip* mode, when this
+parameter is left at its default value, the executor overrides it to *application/zip* automatically; an explicit
+value is used as-is even in *zip* mode. In *file* and *jsonArray* mode the default *application/json* is correct and
+is not overridden.
+
+**Output property** wraps the JSON value in a JSON object under the given property key before writing. When set
+to *payload*, an input value of `{"name":"Alice"}` is written as `{"payload":{"name":"Alice"}}`. When left empty
+(default), the value is written as-is. The wrapping applies in all three output modes; in *jsonArray* mode each
+element of the array is the wrapped form.
+
+**Output mode** selects what the operator produces. *file* (the default) writes one file per input entity. *zip* packs
+all input entities into a single ZIP file — one ZIP entry per entity, producing a single file entity whose backing
+file is a ZIP archive. Entries are always named *entry-0.json*, *entry-1.json*, and so on, by position among valid
+entities. *jsonArray* merges all input entities into a single file
+holding one JSON array whose elements are the JSON values from each entity, in input order; there is always exactly
+one output file.
+
+## Output mode examples
+
+In *zip* mode: an upstream operator produces two entities, each with a JSON string in the *jsonContent* field. With
+*Input path* set to */jsonContent* and *Output mode* set to *zip*, JSON to File produces a single file entity backed
+by a ZIP archive containing two entries: *entry-0.json* and *entry-1.json*. The archive is written with a content type
+of *application/zip*. Wiring the output into a file-backed dataset writes the ZIP file to that dataset's resource.
+
+In *jsonArray* mode: with two entities holding `{"id":1}` and `{"id":2}` and *Output mode* set to *jsonArray*, JSON to
+File produces a single file containing the JSON array `[{"id":1},{"id":2}]`, with a content type of *application/json*.
+
+## Example
+
+An upstream operator produces a single entity with the following JSON string in its *jsonContent* field. With
+*Input path* set to */jsonContent*, JSON to File reads from that field.
+
+```json
+{
+ "response": {
+ "persons": [
+ { "id": "1", "name": "Alice" },
+ { "id": "2", "name": "Bob" }
+ ]
+ }
+}
+```
+
+JSON to File validates the string and writes it to a file with a content type of *application/json*. The produced file
+entity can be wired into a downstream JSON dataset to persist the value as a file on disk, or fed into any other
+operator that accepts file entities.
+
+With the `outputProperty` parameter set to `payload`, the same input is instead written as:
+
+```json
+{
+ "payload": {
+ "response": {
+ "persons": [
+ { "id": "1", "name": "Alice" },
+ { "id": "2", "name": "Bob" }
+ ]
+ }
+ }
+}
+```
+
+
+## Parameter
+
+### Input path
+
+The Silk path expression of the input entity that contains the JSON string. If not set, the value of the first property in the entity schema will be taken.
+
+- ID: `inputPath`
+- Datatype: `string`
+- Default Value: `None`
+
+
+
+### Mime type
+
+MIME type of the produced file.
+
+- ID: `mimeType`
+- Datatype: `string`
+- Default Value: `application/json`
+
+
+
+### Output mode
+
+Output mode: "One file per entity" writes one file per entity, "ZIP archive" packs all entities into a single ZIP archive, "Merged JSON array" merges all entities into a single JSON array file.
+
+- ID: `outputMode`
+- Datatype: `enumeration`
+- Default Value: `file`
+
+
+
+### Output property
+
+If set, the JSON value is wrapped in a JSON object under this property key before writing. For example, with outputProperty set to 'payload', the input {"name":"Alice"} is written as {"payload":{"name":"Alice"}}. When empty (default), the value is written as-is.
+
+- ID: `outputProperty`
+- Datatype: `string`
+- Default Value: `None`
+
+## Advanced Parameter
+
+`None`
+
+## Related Plugins
+
+- [JsonParserOperator](JsonParserOperator.md) — JSON to File writes the JSON string from each input entity to a file; Parse JSON parses the same kind of input into structured entities driven by a downstream schema.
diff --git a/docs/build/reference/customtask/setExecutionVariableOperator.md b/docs/build/reference/customtask/setExecutionVariableOperator.md
new file mode 100644
index 000000000..980eadc12
--- /dev/null
+++ b/docs/build/reference/customtask/setExecutionVariableOperator.md
@@ -0,0 +1,61 @@
+---
+title: "Set execution variable"
+description: "Sets an execution variable to the first value of the (single) input and passes the input through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution."
+icon: octicons/cross-reference-24
+tags:
+ - WorkflowTask
+---
+
+# Set execution variable
+
+
+
+
+
+Sets a single **execution-scope** template variable from this operator's input and passes the input through
+unchanged, so it can be inserted anywhere in a workflow chain.
+
+The variable is written to the `execution` scope. A value set by this operator replaces an execution variable
+of the same name that was defined as a default on the workflow or provided when the run was started. This
+operator only takes effect while running inside a workflow execution, where all nodes share one
+execution-variable holder.
+
+Any downstream node can read the variable as `{{execution.}}`. Referencing an execution variable that
+has not been set fails.
+
+It is the workflow-operator counterpart of the **Set execution variable** transformer (`setExecutionVariable`):
+use this operator to pass a value between workflow nodes without embedding a transform.
+
+## Behaviour
+
+- Reads a value from the **first entity** of the input: the first value of **Source path**, or the entity's
+ first value when no source path is given. Later entities are not consulted.
+- Writes it to the execution scope under `variableName`. If the input is empty or the first entity has no such
+ value, the variable is left unchanged (a default or run-start override stays in place).
+- Forwards the input entities unchanged (connect the output to keep the chain going, or leave it unconnected to
+ use this purely as a side-effecting node).
+
+
+## Parameter
+
+### Variable name
+
+Name of the execution variable to set. It is written to the 'execution' scope and addressed downstream as 'execution.'.
+
+- ID: `variableName`
+- Datatype: `string`
+- Default Value: `myVariable`
+
+
+
+### Source path
+
+Optional path/attribute of the input that supplies the value. If left empty, the first value of the input is used.
+
+- ID: `sourcePath`
+- Datatype: `string`
+- Default Value: `None`
+
+## Advanced Parameter
+
+`None`
diff --git a/docs/build/reference/customtask/sparqlSelectOperator.md b/docs/build/reference/customtask/sparqlSelectOperator.md
index 9625819a1..b2bb44891 100644
--- a/docs/build/reference/customtask/sparqlSelectOperator.md
+++ b/docs/build/reference/customtask/sparqlSelectOperator.md
@@ -1,6 +1,6 @@
---
title: "SPARQL Select query"
-description: "A task that executes a SPARQL Select query on a SPARQL enabled data source and outputs the SPARQL result. If the SPARQL source is defined on a specific graph, a FROM clause will be added to the query at execution time, except when there already exists a GRAPH or FROM clause in the query. FROM NAMED clauses are not injected."
+description: "A task that executes a SPARQL Select query and outputs the SPARQL result."
icon: octicons/cross-reference-24
tags:
- WorkflowTask
@@ -12,35 +12,150 @@ tags:
-The SPARQL SELECT plugin is a task for executing SPARQL SELECT queries on the input RDF data source.
+The SPARQL SELECT plugin is a task for executing SPARQL SELECT queries on an RDF data source.
+It can be used in a workflow, connecting an input to an output. A
+[SPARQL 1.1 SELECT](https://www.w3.org/TR/sparql11-query/#select) query is supported; the simplest example is
+`SELECT * WHERE { ?s ?p ?o }`.
-## Description
+## Input and output
-The SPARQL Select query plugin is an example of a _RDF task_ or _operator_. Such a task can be used in a workflow,
-connecting an input to an output. In this specific case, the _input_ is — in essence — a _SPARQL endpoint_ and the
-_output_ is the entity table containing the _SPARQL results_ of the SPARQL SELECT query execution.
+The _input_ depends on the configuration:
-In general terms, a [SPARQL 1.1 SELECT](https://www.w3.org/TR/sparql11-query/#select) query is supported. One of the
-simplest examples is `SELECT * WHERE { ?s ?p ?o }`.
+- By default, the query is executed against the connected input, which must be a _SPARQL endpoint_
+ (i.e. an RDF dataset).
+- When **Use fallback RDF dataset** (`useDefaultDataset`) is enabled, the query is executed against the
+ fallback RDF dataset (as configured in `dataset.defaultRdf`) instead. The input port then depends on what
+ the template references:
+ - If the template references input entity properties (`input.entity.*`), the task accepts an entity input
+ and generates one query per input entity.
+ - If it references only parameters of the input task (`input.config.*`), an input connection is still
+ required — it supplies the parameter values — but the query is rendered and executed only once.
+ - If it references neither, the task has no input port.
-The [result limit](https://www.w3.org/TR/sparql11-query/#modResultLimit) can be specified for the SPARQL SELECT plugin
-itself, with the parameter `limit`. Additionally, a timeout can be specified with the parameter `sparqlTimeout`.
+The _output_ is an entity table built from the query's
+[SPARQL results](https://www.w3.org/TR/sparql11-results-json/#json-result-object): each projected variable becomes
+a column, and each result binding becomes a row.
-As usual, the SPARQL results contain both "variables" and "bindings", such as in
-[this example](https://www.w3.org/TR/sparql11-results-json/#json-result-object).
-This tabular raw form is transformed into an _entity table_.
+The [result size](https://www.w3.org/TR/sparql11-query/#modResultLimit) can be capped with the `limit` parameter,
+and a query timeout (in milliseconds) can be set via `sparqlTimeout`.
-### Internal Specifics
+## Automatic `FROM` clause injection
If the SPARQL source is defined on a specific graph, a `FROM` clause will be added to the query at execution time,
except when there already exists a `GRAPH` or `FROM` clause in the query. `FROM NAMED` clauses are not injected.
+## Templating
+
+The select query is rendered by a template engine before execution.
+[`Jinja`](https://jinja.palletsprojects.com/) is the default and is described below; for the deprecated `Simple`
+and `Velocity Engine` modes, see "Legacy template engines" at the end.
+
+Jinja uses `{{ ... }}` for value expressions and `{% ... %}` for control flow such as conditionals.
+
+### Template variables
+
+The following variables are available:
+
+- `input.config.`: a parameter of the task connected to the input port. `` is a parameter id
+ of that task's plugin, e.g. `graph` on a SPARQL dataset.
+- `output.config.`: a parameter of the task the output is connected to.
+- `input.entity.`: the value(s) of the given property of the current input entity. Only available
+ with **Use fallback RDF dataset** enabled, since only then the task receives input entities
+ (see _Input and output_ above).
+- `project.`: a project-scoped template variable.
+- `global.`: a global template variable.
+
+A single-valued entity property is inserted as a plain string. A multi-valued property can be iterated with
+`{% for value in input.entity. %}`; inserting it directly concatenates all values without a
+separator. Referencing a variable that is not available at execution time — an unknown parameter name or an
+entity property without a value — fails the query generation with an error.
+
+Parameter, property and variable names must be valid Jinja identifiers (`[a-zA-Z_][a-zA-Z0-9_]*`);
+bracket-subscript access such as `input.entity["urn:prop:label"]` is not supported.
+
+For example, to query the named graph that is configured on the input dataset:
+
+```sparql
+SELECT * WHERE { GRAPH <{{ input.config.graph | validate_uri }}> { ?s ?p ?o } }
+```
+
+### Default scope
+
+The `defaultScope` parameter declares one scope whose variables are additionally exposed at the top level of the
+template context, so they can be referenced without the scope prefix. It defaults to `input.entity`, which means
+a template may write `{{ property }}` as a shorthand for `{{ input.entity.property }}`:
+
+```text
+{{ property }} ≡ {{ input.entity.property }}
+```
+
+Both forms resolve to the same value. Set `defaultScope` to the empty string to disable this aliasing and require
+every variable to be addressed with its full scope.
+
+### Filters
+
+Values are inserted verbatim by default, so URI brackets (`<...>`) and quotation marks around literals must be
+written in the template. The following filters are provided to render values safely:
+
+- `validate_uri`: validates that the value is a valid absolute IRI and returns it unchanged. Throws a validation
+ error otherwise. Wrap the output in `<...>` in the template.
+- `escape_literal`: escapes backslashes, quotes, newlines, carriage returns and tabs so the value can be used
+ inside a short-form SPARQL string literal (`"..."` or `'...'`). No enclosing quotes are added.
+- `escape_multiline_literal`: escapes backslashes and breaks any run of three or more consecutive single or double
+ quotes. Use for values that are wrapped in triple-quoted SPARQL literals (`"""..."""` or `'''...'''`).
+
+All transformer plugins are also available as Jinja filters under their plugin id (for example `lowerCase`,
+`trim`, `urlEncode`).
+
+### Input schema inference
+
+The input schema (the entity properties the task expects) is derived by scanning the raw template for
+`input.entity.` references (or bare references resolved via `defaultScope`). This scan operates
+on the template text before rendering, so SPARQL line comments (`# ...`) are **not** stripped: a
+commented-out line such as
+
+```sparql
+# {{ input.entity.property }}
+```
+
+will still cause `property` to appear in the inferred input schema.
+
+### Output schema inference
+
+The output schema is derived from the raw template by a heuristic, without rendering it. The heuristic takes
+the projection between `SELECT` and the first `WHERE`, `FROM` or `{`, drops a leading `DISTINCT` / `REDUCED`,
+and then:
+
+- For `SELECT *`, collects every distinct `?var` token in the query.
+- Otherwise, collects each top-level `?var` and the trailing `AS ?alias` from parenthesised expressions
+ (e.g. `(COUNT(?s) AS ?count)` yields `count`).
+
+Each variable becomes a string-typed path. If no variables can be detected (e.g. the projection is produced by
+a Jinja expression), the output port is reported with an unknown schema.
+
+### Validation
+
+At task creation, the Jinja template is checked against the available template variables:
+
+- Every `project.<...>` or `global.<...>` reference must resolve to a known variable, matched on the full
+ scoped name (so e.g. `project.metaData.label` is looked up at that exact scope).
+- Every `input.<...>` or `output.<...>` reference must use `config` or `entity` as its second segment.
+
+Bare references are resolved through `defaultScope` before applying the same rules. The template is not
+rendered and the resulting SPARQL is not parsed.
+
+### Legacy template engines
+
+In addition to Jinja, two deprecated template engines are supported for backwards compatibility: `Simple`
+and [`Velocity Engine`](https://velocity.apache.org/engine/2.4.1/user-guide.html). Their syntax is identical
+to the one used by the `SPARQL Update operator` and is documented there.
+
## Parameter
### Select query
-A SPARQL 1.1 select query
+A SPARQL 1.1 select query. The query supports Jinja templating. Parameters of the connected input and output tasks can be accessed via 'input.config.' and 'output.config.'. Project and global template variables are available as 'project.' and 'global.'. Example: SELECT * WHERE { GRAPH <{{ input.config.graph }}> { ?s ?p ?o } }
- ID: `selectQuery`
- Datatype: `code-sparql`
@@ -50,7 +165,7 @@ A SPARQL 1.1 select query
### Result limit
-If set to a positive integer, the number of results is limited
+If set to a positive integer, the number of results is limited. The limit is applied per query: if one query is generated per input entity, it caps the results of each query, not the combined total.
- ID: `limit`
- Datatype: `string`
@@ -68,6 +183,36 @@ An optional SPARQL dataset that can be used for example data, so e.g. the transf
+### Use fallback RDF dataset
+
+If enabled, the query executes against the configured fallback RDF dataset (as configured in `dataset.defaultRdf`) when no RDF dataset is connected. If the query template references input entities, one query is generated per input entity.
+
+- ID: `useDefaultDataset`
+- Datatype: `boolean`
+- Default Value: `false`
+
+
+
+### Templating mode
+
+The templating mode for the template engine.
+
+- ID: `templatingMode`
+- Datatype: `string`
+- Default Value: `jinja`
+
+
+
+### Default scope
+
+Variables from this scope can be accessed without the scope prefix in Jinja. For example, with default scope 'input.entity', a template may reference '{{ property }}' instead of '{{ input.entity.property }}'. Leave empty to disable.
+
+- ID: `defaultScope`
+- Datatype: `string`
+- Default Value: `input.entity`
+
+## Advanced Parameter
+
### SPARQL query timeout (ms)
SPARQL query timeout (select/update) in milliseconds. A value of zero means that there is no timeout set explicitly. If a value greater zero is specified this overwrites possible default timeouts.
@@ -76,11 +221,7 @@ SPARQL query timeout (select/update) in milliseconds. A value of zero means that
- Datatype: `int`
- Default Value: `0`
-## Advanced Parameter
-
-`None`
-
## Related Plugins
-- **sparqlEndpoint** — This plugin executes a SELECT query against a SPARQL endpoint; a SPARQL endpoint dataset in the workflow provides that endpoint. The SPARQL Update query plugin uses the same kind of dataset as a write target rather than a read source.
-- **sparqlUpdateOperator** — The SPARQL Update query plugin turns entity input into update statements that modify a SPARQL store; this plugin reads from the same kind of store by executing a SELECT query and returning the results as an entity table.
+- [sparqlEndpoint](../dataset/sparqlEndpoint.md) — This plugin executes a SELECT query against a SPARQL endpoint; a SPARQL endpoint dataset in the workflow provides that endpoint. The SPARQL Update query plugin uses the same kind of dataset as a write target rather than a read source.
+- [sparqlUpdateOperator](sparqlUpdateOperator.md) — The SPARQL Update query plugin turns entity input into update statements that modify a SPARQL store; this plugin reads from the same kind of store by executing a SELECT query and returning the results as an entity table.
diff --git a/docs/build/reference/customtask/sparqlUpdateOperator.md b/docs/build/reference/customtask/sparqlUpdateOperator.md
index c9d0ba22f..ba9750930 100644
--- a/docs/build/reference/customtask/sparqlUpdateOperator.md
+++ b/docs/build/reference/customtask/sparqlUpdateOperator.md
@@ -25,13 +25,62 @@ _execute_ these queries, we need to connect this task from an input into an outp
## Templating
-The SPARQL Update query plugin uses a template in order to construct and output SPARQL update queries.
-There are two possible template engines supported by this plugin: a `Simple` engine and
+The `sparqlUpdateOperator` plugin uses a **template** in order to construct and output SPARQL update queries.
+Three template engines are supported: `Jinja` (the default), `Simple`, and
[`Velocity Engine`](https://velocity.apache.org/engine/2.4.1/user-guide.html).
-Each of these engines supports a different set of templating features, such as for example _variable interpolation_ with
-the dollar sign (`$`), i.e. filling in input values via placeholders in the template.
+The `Simple` and `Velocity Engine` modes are deprecated.
-### Example of the `Simple` mode
+### Example of the `Jinja` mode
+
+[Jinja](https://jinja.palletsprojects.com/) is the recommended template engine. It uses `{{ }}` for expressions and
+`{% %}` for control flow statements such as conditionals.
+
+```text
+DELETE DATA { <{{ input.entity.subject | validate_uri }}> rdfs:label "{{ input.entity.oldLabel | escape_literal }}" } ;
+{% if input.entity.subject %}
+ INSERT DATA { <{{ input.entity.subject | validate_uri }}> rdfs:label "{{ input.entity.newLabel | escape_literal }}" } ;
+{% endif %}
+```
+
+The following variables are available:
+
+- `input.entity.`: the value of the given property on the current input entity.
+- `input.config.`: a parameter of the connected input task.
+- `output.config.`: a parameter of the connected output task.
+- `project.`: a project-scoped template variable.
+- `global.`: a global template variable.
+
+Entity property names must be valid Jinja identifiers (`[a-zA-Z_][a-zA-Z0-9_]*`); bracket-subscript access such as
+`input.entity["urn:prop:label"]` is not supported.
+
+Values are inserted verbatim by default, so URI brackets (`<...>`) and quotation marks around literals must be
+written in the template. The following filters are provided to render values safely:
+
+- `validate_uri`: validates that the value is a valid absolute IRI and returns it unchanged. Throws a validation
+ error otherwise. Wrap the output in `<...>` in the template.
+- `escape_literal`: escapes backslashes, quotes, newlines, carriage returns and tabs so the value can be used
+ inside a short-form SPARQL string literal (`"..."` or `'...'`). No enclosing quotes are added.
+- `escape_multiline_literal`: escapes backslashes and breaks any run of three or more consecutive single or double
+ quotes. Use for values that are wrapped in triple-quoted SPARQL literals (`"""..."""` or `'''...'''`).
+
+All transformer plugins are also available as Jinja filters under their plugin id (for example `lowerCase`,
+`trim`, `urlEncode`).
+
+### Validation
+
+At task creation, the template is checked against the available template variables. What is checked depends
+on the selected templating mode:
+
+- `Jinja`:
+ - Every `project.<...>` or `global.<...>` reference must resolve to a known variable, matched on the full
+ scoped name (so e.g. `project.metaData.label` is looked up at that exact scope).
+ - Every `input.<...>` or `output.<...>` reference must use `config` or `entity` as its second segment.
+ - The template is not rendered and the resulting SPARQL is not parsed.
+- `Simple` / `Velocity Engine`:
+ - The template is rendered once with placeholder values and the result must parse as a SPARQL Update query.
+ - Templates that use `rawUnsafe` skip this parse check.
+
+### Example of the `Simple` mode (deprecated)
```text
DELETE DATA { ${} rdf:label ${"PROP_FROM_ENTITY_SCHEMA2"} }
@@ -46,7 +95,7 @@ Furthermore, it will insert a plain literal serialization for the property value
It is also possible to write something like `${"PROP"}^^` or `${"PROP"}@en`. In other words, we
can combine variable substitutions with fixed expressions to construct semi-flexible expressions within the template.
-### Example of the `Velocity Engine` mode
+### Example of the `Velocity Engine` mode (deprecated)
```text
DELETE DATA { $row.uri("PROP_FROM_ENTITY_SCHEMA1") rdf:label $row.plainLiteral("PROP_FROM_ENTITY_SCHEMA2") }
@@ -95,7 +144,7 @@ In contrast to the SPARQL select operator, no `FROM` clause gets injected into t
### SPARQL update query
-The SPARQL UPDATE template for constructing SPARQL UPDATE queries for every entity from the input. The possible values for the template engine are `Simple` and `Velocity Engine`. See the general documentation of this plugin for further details on the features of each template engine.
+The SPARQL UPDATE template for constructing SPARQL UPDATE queries for every entity from the input. The possible values for the template engine are `Jinja` (default), `Simple` and `Velocity Engine`. See the general documentation of this plugin for further details on the features of each template engine.
- ID: `sparqlUpdateTemplate`
- Datatype: `code-sparql`
@@ -115,11 +164,11 @@ How many entities should be handled in a single update request.
### Templating mode
-The templating mode for the template engine. The possible values are `Simple` and `Velocity Engine`. See the general documentation of this plugin for further details on the features of each template engine.
+The templating mode for the template engine. See the general documentation of this plugin for further details on the features of each template engine.
- ID: `templatingMode`
-- Datatype: `enumeration`
-- Default Value: `simple`
+- Datatype: `string`
+- Default Value: `jinja`
## Advanced Parameter
@@ -127,5 +176,5 @@ The templating mode for the template engine. The possible values are `Simple` an
## Related Plugins
-- **sparqlEndpoint** — A SPARQL endpoint dataset in the workflow receives the update statements this plugin generates and executes them against the remote store.
-- **sparqlSelectOperator** — This plugin turns entity input into SPARQL Update statements that modify a store. The SPARQL Select query plugin reads from the same kind of store by executing a SELECT query and outputting the results as an entity table.
+- [sparqlEndpoint](../dataset/sparqlEndpoint.md) — A SPARQL endpoint dataset in the workflow receives the update statements this plugin generates and executes them against the remote store.
+- [sparqlSelectOperator](sparqlSelectOperator.md) — This plugin turns entity input into SPARQL Update statements that modify a store. The SPARQL Select query plugin reads from the same kind of store by executing a SELECT query and outputting the results as an entity table.
diff --git a/docs/build/reference/dataset/.pages b/docs/build/reference/dataset/.pages
index 6007de55c..d9a440848 100644
--- a/docs/build/reference/dataset/.pages
+++ b/docs/build/reference/dataset/.pages
@@ -10,7 +10,7 @@ nav:
- "Excel (Google Drive)": googlespreadsheet.md
- "Excel (OneDrive, Office365)": office365preadsheet.md
- "Hive database": Hive.md
- - "In-memory dataset": inMemory.md
+ - "In-memory Knowledge Graph": inMemory.md
- "Internal dataset": internal.md
- "Internal dataset (single graph)": LocalInternalDataset.md
- "JSON": json.md
diff --git a/docs/build/reference/dataset/eccencaDataPlatform.md b/docs/build/reference/dataset/eccencaDataPlatform.md
index f2cc457f0..4289d34e2 100644
--- a/docs/build/reference/dataset/eccencaDataPlatform.md
+++ b/docs/build/reference/dataset/eccencaDataPlatform.md
@@ -53,7 +53,7 @@ with a SQL database as a source dataset.
### Graph
-The URI of the named graph.
+The URI of the named graph. If left empty, the default graph is used.
- ID: `graph`
- Datatype: `graph uri`
@@ -171,6 +171,6 @@ Use streaming HTTP upload (chunked transfer encoding) instead of multipart file
## Related Plugins
-- **inMemory** — The Knowledge Graph plugin produces a durable graph in Corporate Memory's managed store; the In-memory dataset plugin produces a transient graph that is gone on restart. The two differ in persistence, not just scale: in-memory storage is not a lightweight version of the Knowledge Graph plugin.
-- **file** — The Knowledge Graph plugin writes to a named graph in Corporate Memory's managed store; the RDF file plugin writes to a file on disk. The two differ in where the data lives, not just how it is accessed.
-- **sparqlEndpoint** — Both the Knowledge Graph plugin and the SPARQL endpoint plugin use SPARQL. The difference is where the store lives: the Knowledge Graph plugin reads and writes a named graph in Corporate Memory's own store, while the SPARQL endpoint plugin connects to any external SPARQL endpoint.
+- [inMemory](inMemory.md) — The Knowledge Graph plugin produces a durable graph in Corporate Memory's managed store; the In-memory dataset plugin produces a transient graph that is gone on restart. The two differ in persistence, not just scale: in-memory storage is not a lightweight version of the Knowledge Graph plugin.
+- [file](file.md) — The Knowledge Graph plugin writes to a named graph in Corporate Memory's managed store; the RDF file plugin writes to a file on disk. The two differ in where the data lives, not just how it is accessed.
+- [sparqlEndpoint](sparqlEndpoint.md) — Both the Knowledge Graph plugin and the SPARQL endpoint plugin use SPARQL. The difference is where the store lives: the Knowledge Graph plugin reads and writes a named graph in Corporate Memory's own store, while the SPARQL endpoint plugin connects to any external SPARQL endpoint.
diff --git a/docs/build/reference/dataset/file.md b/docs/build/reference/dataset/file.md
index b42c7dfc0..4ec65e9dd 100644
--- a/docs/build/reference/dataset/file.md
+++ b/docs/build/reference/dataset/file.md
@@ -248,5 +248,5 @@ If the input resource is a ZIP file, files inside the file are filtered via this
## Related Plugins
-- **sparqlEndpoint** — The RDF file dataset loads a file into memory at read time and constrains output to N-Triples. The SPARQL endpoint dataset connects to a remote queryable store that handles queries and updates without those restrictions.
-- **inMemory** — The RDF file dataset writes to disk but only in N-Triples, and it loads the full file into memory to read; the in-memory dataset skips the filesystem entirely but discards all data when execution finishes.
+- [sparqlEndpoint](sparqlEndpoint.md) — The RDF file dataset loads a file into memory at read time and constrains output to N-Triples. The SPARQL endpoint dataset connects to a remote queryable store that handles queries and updates without those restrictions.
+- [inMemory](inMemory.md) — The RDF file dataset writes to disk but only in N-Triples, and it loads the full file into memory to read; the in-memory dataset skips the filesystem entirely but discards all data when execution finishes.
diff --git a/docs/build/reference/dataset/inMemory.md b/docs/build/reference/dataset/inMemory.md
index 7df496b8a..23ee6e81b 100644
--- a/docs/build/reference/dataset/inMemory.md
+++ b/docs/build/reference/dataset/inMemory.md
@@ -1,12 +1,12 @@
---
-title: "In-memory dataset"
-description: "A Dataset that holds all data in-memory."
+title: "In-memory Knowledge Graph"
+description: "A dataset that holds all data in-memory. In the default (workflow-scoped) mode, data is isolated per workflow execution and shared with nested workflows that reference the same dataset task. In application-scoped mode, data persists for the lifetime of the running process."
icon: octicons/cross-reference-24
tags:
- Dataset
---
-# In-memory dataset
+# In-memory Knowledge Graph
@@ -24,24 +24,29 @@ Typical use cases:
## 2. Behaviour and lifecycle
-- The dataset maintains a single in-memory RDF model.
-- All read and write operations go through a SPARQL endpoint over this model.
-- Data exists only **in memory**:
- - It is not persisted to disk by this dataset.
- - After an application restart, the dataset contents are empty again.
+The dataset maintains a single in-memory RDF model and exposes it via a SPARQL endpoint. Two lifecycle modes are available, controlled by the `workflowScoped` parameter:
-Within a workflow:
+**Workflow-scoped mode** (default, `workflowScoped = true`):
-- The dataset can be used as both **input** and **output**:
- - Upstream operators can write triples/entities/links into it.
- - Downstream operators can read from it via SPARQL-based mechanisms.
+- A separate model is created for each workflow execution.
+- Concurrent workflow executions are fully isolated from each other.
+- A dataset task in a nested workflow shares the same model as the parent workflow for the same task identifier. Data written by the parent is available in the nested workflow and vice versa.
+- If the dataset is read from outside a workflow context, the data from the most recently started executor is returned.
+- When the workflow execution ends, the per-execution data is removed automatically.
+
+**Application-scoped mode** (`workflowScoped = false`):
+
+- A single shared model is created when the dataset is instantiated.
+- Data persists for the lifetime of the running application process.
+- All workflow executions share the same in-memory graph.
+- After an application restart, the dataset contents are empty again.
## 3. Reading data
- When used as a **source**, the dataset exposes its data as a SPARQL endpoint.
- Queries and retrievals behave like against a normal SPARQL dataset:
- Entity retrieval, path/type discovery, sampling, etc. are executed via SPARQL.
-- There is no file backing this dataset; everything comes from what has been written into the in-memory model during the lifetime of the process.
+- There is no file backing this dataset; everything comes from what has been written into the in-memory model during the lifetime of the process (application-scoped) or the workflow execution (workflow-scoped).
## 4. Writing data
@@ -60,33 +65,60 @@ All three sinks ultimately write into the same in-memory graph; there is no sepa
## 5. Configuration
-### Clear graph before workflow execution
+### Workflow scoped
-- **Parameter:** `Clear graph before workflow execution` (boolean)
+- **Parameter:** `workflowScoped` (boolean)
- **Default:** `true`
-Behaviour:
+When `true` (default, workflow-scoped mode):
+
+- Data is stored in a separate in-memory graph for each workflow execution.
+- Concurrent workflow executions are fully isolated from each other.
+- A dataset task in a nested workflow shares the same graph as the parent for the same task identifier. Data written by the parent is available in the nested workflow and vice versa.
+- If the dataset is read from outside a workflow context, the data of the workflow execution that most recently accessed this dataset is returned.
+- When the workflow execution ends, the per-execution data is removed automatically.
+
+When `false` (application-scoped mode):
+
+- Data persists in a single shared graph for the lifetime of the running process.
+- All workflow executions share the same graph.
+
+### Clear graph before workflow execution
+
+- **Parameter:** `clearGraphBeforeExecution` (boolean, **deprecated**)
+- **Default:** `false`
+
+This parameter is deprecated. Use the **Clear dataset** operator in the workflow instead.
+
+Behaviour (application-scoped mode only):
- If **true**:
- - Before the dataset is used in a workflow execution, the graph is cleared (for writes via this dataset).
+ - Before the dataset is used in a workflow execution, the graph is cleared.
- The workflow sees a **fresh, empty in-memory graph** at the start of the run.
- If **false**:
- Existing data in the in-memory graph is **preserved** when the workflow starts.
- New data is added on top of whatever is already stored in the model.
-This parameter controls whether the dataset behaves as a **fresh scratch graph per workflow run** or as a **longer-lived in-memory graph** within the lifetime of the running application.
+This parameter has no effect when `workflowScoped = true` (the executor manages the lifecycle).
## 6. Limitations and recommendations
- **Memory-bound**
- All data is kept in memory; large graphs will increase memory usage and may impact performance.
- For large or production RDF graphs, use an external RDF store and a SPARQL dataset instead.
+ - A size limit is enforced: once the estimated size of data written to the dataset exceeds the value of `org.silkframework.runtime.resource.Resource.maxInMemorySize`, the workflow fails with an error. This prevents the JVM from running out of heap memory.
- **No persistence**
- Contents are lost when the application/server is restarted.
- Do not treat this dataset as long-term storage.
+- **SPARQL engine**
+ - The dataset is backed by [Apache Jena](https://jena.apache.org/), exposed through a Jena in-memory SPARQL endpoint.
+
+- **No named-graph support**
+ - Only the **default graph** is available. Writing triples into a named graph is not possible.
+
- **Scope**
- Best suited for:
- small to medium intermediate results,
@@ -95,22 +127,37 @@ This parameter controls whether the dataset behaves as a **fresh scratch graph p
## 7. Example usage scenarios
-- Use as a **temporary integration graph**:
+- Use as a **temporary integration graph** (application-scoped):
- Multiple sources write into the in-memory dataset.
- A downstream SPARQL-based operator queries the combined graph.
-- Use as a **scratch area for experimentation**:
+- Use as a **scratch area for experimentation** (application-scoped):
- Quickly test mapping or linking logic by writing output into the in-memory dataset.
- Inspect the result via SPARQL without configuring an external endpoint.
-- Use as a **small lookup store**:
+- Use as a **small lookup store** (application-scoped):
- Preload a small set of reference triples (e.g. codes or mappings).
- Let workflows query these during execution.
+- Use as a **workflow-local intermediate store** (workflow-scoped):
+ - Multiple operators in a single workflow run write intermediate RDF results.
+ - Downstream operators in the same run read from the dataset without affecting parallel runs.
+
+- Use in **nested workflows** (workflow-scoped):
+ - A parent workflow writes data into a workflow-scoped dataset.
+ - A nested sub-workflow reads and enriches the same data.
+ - After the nested workflow completes, the parent can read the enriched result.
+
## Parameter
-`None`
+### Workflow-scoped
+
+If true (default), data is isolated per workflow execution and cleared after the execution ends, sharing data with nested workflows that reference the same dataset task. If false, data persists for the lifetime of the application process.
+
+- ID: `workflowScoped`
+- Datatype: `boolean`
+- Default Value: `true`
## Advanced Parameter
@@ -124,5 +171,5 @@ This is deprecated, use the 'Clear dataset' operator instead to clear a dataset
## Related Plugins
-- **sparqlEndpoint** — Data in the in-memory dataset does not persist beyond the running process. The SPARQL endpoint dataset connects to an external store that persists independently, which means switching between them changes not just where the data lives but whether it survives execution at all.
-- **file** — Switching from the in-memory dataset to the RDF file dataset is not just adding persistence. The RDF file dataset loads the entire file into memory at read time and constrains output to N-Triples — neither of which the in-memory dataset does.
+- [sparqlEndpoint](sparqlEndpoint.md) — Data in the in-memory dataset does not persist beyond the running process. The SPARQL endpoint dataset connects to an external store that persists independently, which means switching between them changes not just where the data lives but whether it survives execution at all.
+- [file](file.md) — Switching from the in-memory dataset to the RDF file dataset is not just adding persistence. The RDF file dataset loads the entire file into memory at read time and constrains output to N-Triples — neither of which the in-memory dataset does.
diff --git a/docs/build/reference/dataset/index.md b/docs/build/reference/dataset/index.md
index 8e8cd9afa..9fa1f688d 100644
--- a/docs/build/reference/dataset/index.md
+++ b/docs/build/reference/dataset/index.md
@@ -26,7 +26,7 @@ Datasets are collections of data that can be read or written.
| [Excel (Google Drive)](googlespreadsheet.md) | Read data from a remote Google Spreadsheet. |
| [Excel (OneDrive, Office365)](office365preadsheet.md) | Read data from a remote onedrive or Office365 Spreadsheet. |
| [Hive database](Hive.md) | Read from or write to an embedded Apache Hive endpoint. |
- | [In-memory dataset](inMemory.md) | A Dataset that holds all data in-memory. |
+ | [In-memory Knowledge Graph](inMemory.md) | A dataset that holds all data in-memory. In the default (workflow-scoped) mode, data is isolated per workflow execution and shared with nested workflows that reference the same dataset task. In application-scoped mode, data persists for the lifetime of the running process. |
| [Internal dataset](internal.md) | Dataset for storing entities between workflow steps. The underlying dataset type can be configured using the `dataset.internal.*` configuration parameters. |
| [Internal dataset (single graph)](LocalInternalDataset.md) | Dataset for storing entities between workflow steps. This variant does use the same graph for all internal datasets in a workflow. The underlying dataset type can be configured using the `dataset.internal.*` configuration parameters. |
| [JSON](json.md) | Read from or write to a JSON or JSON Lines file. |
diff --git a/docs/build/reference/dataset/json.md b/docs/build/reference/dataset/json.md
index d71a2bc39..47e30695b 100644
--- a/docs/build/reference/dataset/json.md
+++ b/docs/build/reference/dataset/json.md
@@ -20,7 +20,7 @@ In addition to plain JSON files, *JSON Lines* files can also be read.
For reading, the JSON dataset supports a number of special paths:
-- `#id` Is a special syntax for generating an id for a selected element. It can be used in URI patterns for entities which do not provide an identifier. Examples: `http://example.org/{#id}` or `http://example.org/{/pathToEntity/#id}`.
+- `#id` is a special syntax for generating a hash-based id for a selected element. It can be used in URI patterns for entities which do not provide an identifier. Examples: `http://example.org/{#id}` or `http://example.org/{/pathToEntity/#id}`. If no URI pattern is configured, the dataset instead generates the default entity URI from the JSON source location.
- `#text` retrieves the text of the selected node.
- The backslash can be used to navigate to the parent JSON node, e.g., `\parent/key`. The name of the backslash key (here `parent`) is ignored.
@@ -108,3 +108,7 @@ If the input resource is a ZIP file, files inside the file are filtered via this
- ID: `zipFileRegex`
- Datatype: `string`
- Default Value: `^(?!.*[\/\\]\..*$|^\..*$).*\.jsonl?$`
+
+## Related Plugins
+
+- [JsonParserOperator](../customtask/JsonParserOperator.md) — The JSON dataset is a pipeline source: it opens a file and needs no upstream input. Parse JSON is an operator: it reads a JSON string from a field value supplied by an upstream entity.
diff --git a/docs/build/reference/dataset/neo4j.md b/docs/build/reference/dataset/neo4j.md
index 98f70b473..d00f5800b 100644
--- a/docs/build/reference/dataset/neo4j.md
+++ b/docs/build/reference/dataset/neo4j.md
@@ -24,7 +24,7 @@ A property `uri` will be added to each generated node, which holds the URI of th
In applications, the URI property should be used instead of the node identifiers, which are auto-generated in Neo4j and do not represent stable URIs.
When reading nodes, the entity URIs will be generated based on that property.
-At the moment, it's not supported to read nodes that do not provide a `uri` property.
+Nodes that do not provide a `uri` property are read with a generated entity URI, which is not stable across reads.
### Labels
@@ -136,3 +136,13 @@ This is deprecated, use the 'Clear dataset' operator instead to clear a dataset
- ID: `clearBeforeExecution`
- Datatype: `boolean`
- Default Value: `false`
+
+
+
+### Write batch size
+
+The number of entities to write in a single Neo4j transaction. Reduce this value if you encounter Neo4j transaction memory limits (dbms.memory.transaction.total.max).
+
+- ID: `writeBatchSize`
+- Datatype: `int`
+- Default Value: `1000`
diff --git a/docs/build/reference/dataset/sparqlEndpoint.md b/docs/build/reference/dataset/sparqlEndpoint.md
index 1e55842bf..8631f9a40 100644
--- a/docs/build/reference/dataset/sparqlEndpoint.md
+++ b/docs/build/reference/dataset/sparqlEndpoint.md
@@ -182,7 +182,7 @@ This is deprecated, use the 'Clear dataset' operator instead to clear a dataset
## Related Plugins
-- **inMemory** — The SPARQL endpoint dataset reads from and writes to a remote endpoint that retains its contents independently of the running process. The in-memory dataset does not persist data beyond the running process — the two are not alternatives for the same storage need.
-- **file** — The RDF file dataset loads its contents from a file into memory at read time and supports only N-Triples as output. The SPARQL endpoint dataset connects to a remote endpoint that handles queries and updates without loading the full dataset into process memory.
-- **sparqlUpdateOperator** — The SPARQL Update query plugin generates SPARQL Update statements from entity input using a template; the SPARQL endpoint dataset is what those statements are written to. One produces the queries, the other executes them against the endpoint.
-- **sparqlSelectOperator** — The SPARQL Select query plugin reads from a SPARQL endpoint dataset by executing a SELECT query against it; the SPARQL Update query plugin writes to the same kind of dataset by sending update statements to it. The two plugins sit on opposite ends of the same data flow.
+- [inMemory](inMemory.md) — The SPARQL endpoint dataset reads from and writes to a remote endpoint that retains its contents independently of the running process. The in-memory dataset does not persist data beyond the running process — the two are not alternatives for the same storage need.
+- [file](file.md) — The RDF file dataset loads its contents from a file into memory at read time and supports only N-Triples as output. The SPARQL endpoint dataset connects to a remote endpoint that handles queries and updates without loading the full dataset into process memory.
+- [sparqlUpdateOperator](../customtask/sparqlUpdateOperator.md) — The SPARQL Update query plugin generates SPARQL Update statements from entity input using a template; the SPARQL endpoint dataset is what those statements are written to. One produces the queries, the other executes them against the endpoint.
+- [sparqlSelectOperator](../customtask/sparqlSelectOperator.md) — The SPARQL Select query plugin reads from a SPARQL endpoint dataset by executing a SELECT query against it; the SPARQL Update query plugin writes to the same kind of dataset by sending update statements to it. The two plugins sit on opposite ends of the same data flow.
diff --git a/docs/build/reference/distancemeasure/date.md b/docs/build/reference/distancemeasure/date.md
index 3fb47e055..39770f468 100644
--- a/docs/build/reference/distancemeasure/date.md
+++ b/docs/build/reference/distancemeasure/date.md
@@ -148,4 +148,4 @@ If true, no distance value will be generated if months or days are missing (e.g.
## Related Plugins
-- **dateTime** — The date time metric plugin is the date metric plugin at a finer scale: it resolves differences to the second rather than the day, but does not tolerate partial dates that lack a month or day.
+- [dateTime](dateTime.md) — The date time metric plugin is the date metric plugin at a finer scale: it resolves differences to the second rather than the day, but does not tolerate partial dates that lack a month or day.
diff --git a/docs/build/reference/distancemeasure/dateTime.md b/docs/build/reference/distancemeasure/dateTime.md
index e43233d49..ee5fef491 100644
--- a/docs/build/reference/distancemeasure/dateTime.md
+++ b/docs/build/reference/distancemeasure/dateTime.md
@@ -30,4 +30,4 @@ Compares single values (as opposed to sequences of values). If multiple values a
## Related Plugins
-- **date** — Where the date time metric plugin demands full datetime values and measures in seconds, the date metric plugin works at day granularity and accepts year-only or year-month dates.
+- [date](date.md) — Where the date time metric plugin demands full datetime values and measures in seconds, the date metric plugin works at day granularity and accepts year-only or year-month dates.
diff --git a/docs/build/reference/distancemeasure/equality.md b/docs/build/reference/distancemeasure/equality.md
index a90a7f6fb..1ee6204c1 100644
--- a/docs/build/reference/distancemeasure/equality.md
+++ b/docs/build/reference/distancemeasure/equality.md
@@ -56,4 +56,4 @@ Compares single values (as opposed to sequences of values). If multiple values a
## Related Plugins
-- **inequality** — The inequality plugin is the logical inverse of string equality: it returns success when the values are different rather than equal.
+- [inequality](inequality.md) — The inequality plugin is the logical inverse of string equality: it returns success when the values are different rather than equal.
diff --git a/docs/build/reference/distancemeasure/greaterThan.md b/docs/build/reference/distancemeasure/greaterThan.md
index 3f82bb503..ea3afb32d 100644
--- a/docs/build/reference/distancemeasure/greaterThan.md
+++ b/docs/build/reference/distancemeasure/greaterThan.md
@@ -52,4 +52,4 @@ Reverse source and target inputs
## Related Plugins
-- **lowerThan** — The Lower than plugin is the logical inverse of Greater than: given the same inputs, it returns 1.0 exactly where Greater than returns 0.0.
+- [lowerThan](lowerThan.md) — The Lower than plugin is the logical inverse of Greater than: given the same inputs, it returns 1.0 exactly where Greater than returns 0.0.
diff --git a/docs/build/reference/distancemeasure/inequality.md b/docs/build/reference/distancemeasure/inequality.md
index d91addd51..ee62fa1ae 100644
--- a/docs/build/reference/distancemeasure/inequality.md
+++ b/docs/build/reference/distancemeasure/inequality.md
@@ -76,4 +76,4 @@ Compares single values (as opposed to sequences of values). If multiple values a
## Related Plugins
-- **equality** — The string equality plugin covers the complementary case: success when values are equal, where the inequality plugin would return failure.
+- [equality](equality.md) — The string equality plugin covers the complementary case: success when values are equal, where the inequality plugin would return failure.
diff --git a/docs/build/reference/distancemeasure/isSubstring.md b/docs/build/reference/distancemeasure/isSubstring.md
index aaaeb26b0..aee448fb4 100644
--- a/docs/build/reference/distancemeasure/isSubstring.md
+++ b/docs/build/reference/distancemeasure/isSubstring.md
@@ -36,5 +36,5 @@ Reverse source and target inputs
## Related Plugins
-- **startsWith** — The Starts With plugin tests a stricter condition: not only must the target appear in the source, but it must appear at the very start.
-- **substringDistance** — Containment and similarity are not the same measure. Is substring checks whether the source string appears anywhere inside the target and returns a binary result; Substring comparison scores the overall similarity between the two strings.
+- [startsWith](startsWith.md) — The Starts With plugin tests a stricter condition: not only must the target appear in the source, but it must appear at the very start.
+- [substringDistance](substringDistance.md) — Containment and similarity are not the same measure. Is substring checks whether the source string appears anywhere inside the target and returns a binary result; Substring comparison scores the overall similarity between the two strings.
diff --git a/docs/build/reference/distancemeasure/jaro.md b/docs/build/reference/distancemeasure/jaro.md
index 0d3307fe2..5d9f68fd0 100644
--- a/docs/build/reference/distancemeasure/jaro.md
+++ b/docs/build/reference/distancemeasure/jaro.md
@@ -32,5 +32,5 @@ Compares single values (as opposed to sequences of values). If multiple values a
## Related Plugins
-- **jaroWinkler** — The Jaro–Winkler distance plugin is a Jaro-style similarity measure that gives extra weight to shared prefixes. The Jaro distance metric plugin stays closer to the underlying common character and transposition signal without that prefix boost.
-- **metaphone** — The Metaphone plugin turns each input string into a phonetic key, reducing spelling variation before scoring. The Jaro distance metric plugin then compares those phonetic keys as ordinary strings.
+- [jaroWinkler](jaroWinkler.md) — The Jaro–Winkler distance plugin is a Jaro-style similarity measure that gives extra weight to shared prefixes. The Jaro distance metric plugin stays closer to the underlying common character and transposition signal without that prefix boost.
+- [metaphone](../transformer/Linguistic/metaphone.md) — The Metaphone plugin turns each input string into a phonetic key, reducing spelling variation before scoring. The Jaro distance metric plugin then compares those phonetic keys as ordinary strings.
diff --git a/docs/build/reference/distancemeasure/jaroWinkler.md b/docs/build/reference/distancemeasure/jaroWinkler.md
index d0f07dc8e..bc679d8df 100644
--- a/docs/build/reference/distancemeasure/jaroWinkler.md
+++ b/docs/build/reference/distancemeasure/jaroWinkler.md
@@ -32,4 +32,4 @@ Compares single values (as opposed to sequences of values). If multiple values a
## Related Plugins
-- **jaro** — The Jaro distance metric plugin provides the baseline Jaro similarity without emphasizing beginnings. The Jaro–Winkler distance plugin shifts the score toward shared prefixes, which can change rankings when many candidates start the same way.
+- [jaro](jaro.md) — The Jaro distance metric plugin provides the baseline Jaro similarity without emphasizing beginnings. The Jaro–Winkler distance plugin shifts the score toward shared prefixes, which can change rankings when many candidates start the same way.
diff --git a/docs/build/reference/distancemeasure/levenshtein.md b/docs/build/reference/distancemeasure/levenshtein.md
index 3881f75de..cb506db7a 100644
--- a/docs/build/reference/distancemeasure/levenshtein.md
+++ b/docs/build/reference/distancemeasure/levenshtein.md
@@ -102,4 +102,4 @@ The maximum character that is used for indexing
## Related Plugins
-- **levenshteinDistance** — The Levenshtein distance plugin counts the minimum edits needed to transform one string into the other. The normalized Levenshtein distance plugin divides that count by the length of the longer string, so the distance is comparable regardless of how long the strings are.
+- [levenshteinDistance](levenshteinDistance.md) — The Levenshtein distance plugin counts the minimum edits needed to transform one string into the other. The normalized Levenshtein distance plugin divides that count by the length of the longer string, so the distance is comparable regardless of how long the strings are.
diff --git a/docs/build/reference/distancemeasure/levenshteinDistance.md b/docs/build/reference/distancemeasure/levenshteinDistance.md
index 8d8e7b98a..407fda3a2 100644
--- a/docs/build/reference/distancemeasure/levenshteinDistance.md
+++ b/docs/build/reference/distancemeasure/levenshteinDistance.md
@@ -92,4 +92,4 @@ The maximum character that is used for indexing
## Related Plugins
-- **levenshtein** — Raw edit counts are not directly comparable across strings of different lengths. The normalized Levenshtein distance plugin addresses this by dividing the edit count by the length of the longer string, making the result length-independent.
+- [levenshtein](levenshtein.md) — Raw edit counts are not directly comparable across strings of different lengths. The normalized Levenshtein distance plugin addresses this by dividing the edit count by the length of the longer string, making the result length-independent.
diff --git a/docs/build/reference/distancemeasure/lowerThan.md b/docs/build/reference/distancemeasure/lowerThan.md
index 3df29aff0..1e62379d0 100644
--- a/docs/build/reference/distancemeasure/lowerThan.md
+++ b/docs/build/reference/distancemeasure/lowerThan.md
@@ -52,4 +52,4 @@ Reverse source and target inputs
## Related Plugins
-- **greaterThan** — The greater than plugin tests the same pair of values with the ordering flipped: it succeeds where the lower than plugin fails.
+- [greaterThan](greaterThan.md) — The greater than plugin tests the same pair of values with the ordering flipped: it succeeds where the lower than plugin fails.
diff --git a/docs/build/reference/distancemeasure/num.md b/docs/build/reference/distancemeasure/num.md
index 3fe7b4dee..f5ea9ee8e 100644
--- a/docs/build/reference/distancemeasure/num.md
+++ b/docs/build/reference/distancemeasure/num.md
@@ -46,4 +46,4 @@ The maximum number that is used for indexing
## Related Plugins
-- **numericEquality** — Numeric similarity measures how far apart two numbers are; Numeric equality asks only whether they match, with no in-between value.
+- [numericEquality](numericEquality.md) — Numeric similarity measures how far apart two numbers are; Numeric equality asks only whether they match, with no in-between value.
diff --git a/docs/build/reference/distancemeasure/numericEquality.md b/docs/build/reference/distancemeasure/numericEquality.md
index c85c5560e..422cfaaac 100644
--- a/docs/build/reference/distancemeasure/numericEquality.md
+++ b/docs/build/reference/distancemeasure/numericEquality.md
@@ -90,4 +90,4 @@ The range of tolerance in floating point number comparisons. Must be 0 or a non-
## Related Plugins
-- **num** — Matching and measuring distance are not the same operation. Numeric equality returns match or no match; Numeric similarity returns the actual distance between the two numbers.
+- [num](num.md) — Matching and measuring distance are not the same operation. Numeric equality returns match or no match; Numeric similarity returns the actual distance between the two numbers.
diff --git a/docs/build/reference/distancemeasure/startsWith.md b/docs/build/reference/distancemeasure/startsWith.md
index d3e22dce3..fb36a499e 100644
--- a/docs/build/reference/distancemeasure/startsWith.md
+++ b/docs/build/reference/distancemeasure/startsWith.md
@@ -56,5 +56,5 @@ The potential maximum length of the strings that must match. If the max length i
## Related Plugins
-- **isSubstring** — The Is Substring plugin checks whether the source appears anywhere inside the target, rather than whether the source begins with the target.
-- **substringDistance** — The result from the Starts with plugin is always binary — the source either begins with the target string or it does not — while Substring comparison quantifies the degree of similarity across the full string as a continuous score.
+- [isSubstring](isSubstring.md) — The Is Substring plugin checks whether the source appears anywhere inside the target, rather than whether the source begins with the target.
+- [substringDistance](substringDistance.md) — The result from the Starts with plugin is always binary — the source either begins with the target string or it does not — while Substring comparison quantifies the degree of similarity across the full string as a continuous score.
diff --git a/docs/build/reference/distancemeasure/substringDistance.md b/docs/build/reference/distancemeasure/substringDistance.md
index 66d3ace09..83d858bc1 100644
--- a/docs/build/reference/distancemeasure/substringDistance.md
+++ b/docs/build/reference/distancemeasure/substringDistance.md
@@ -36,5 +36,5 @@ The minimum length of a possible substring match.
## Related Plugins
-- **startsWith** — The Substring comparison plugin produces a continuous similarity score across the full string; Starts with reduces the comparison to a binary check on whether the source opens with the target.
-- **isSubstring** — The score from Substring comparison is continuous, reflecting overall string similarity; Is substring checks only whether the source appears anywhere inside the target, returning a binary result.
+- [startsWith](startsWith.md) — The Substring comparison plugin produces a continuous similarity score across the full string; Starts with reduces the comparison to a binary check on whether the source opens with the target.
+- [isSubstring](isSubstring.md) — The score from Substring comparison is continuous, reflecting overall string similarity; Is substring checks only whether the source appears anywhere inside the target, returning a binary result.
diff --git a/docs/build/reference/transformer/.pages b/docs/build/reference/transformer/.pages
index 95f5288de..476f8db4c 100644
--- a/docs/build/reference/transformer/.pages
+++ b/docs/build/reference/transformer/.pages
@@ -14,6 +14,7 @@ nav:
- "Numeric": Numeric
- "Parser": Parser
- "Replace": Replace
+ - "SPARQL": SPARQL
- "Selection": Selection
- "Sequence": Sequence
- "Substring": Substring
@@ -21,4 +22,5 @@ nav:
- "Tokenization": Tokenization
- "Uncategorized": Uncategorized
- "Validation": Validation
- - "Value": Value
\ No newline at end of file
+ - "Value": Value
+ - "Variables": Variables
\ No newline at end of file
diff --git a/docs/build/reference/transformer/Combine/concat.md b/docs/build/reference/transformer/Combine/concat.md
index da918cfd3..79c4a3606 100644
--- a/docs/build/reference/transformer/Combine/concat.md
+++ b/docs/build/reference/transformer/Combine/concat.md
@@ -170,5 +170,5 @@ Handle missing values as empty strings.
## Related Plugins
-* **concatPairwise** — Concatenate takes the Cartesian product of all inputs and produces one string per combination. Concatenate pairwise aligns values by position and produces one string per position, truncating to the shortest input.
-* **concatMultiValues** — Passing multiple values to a single input of Concatenate does not combine them — it multiplies the output. Concatenate multiple values is the plugin that collapses multiple values within an input into one string, producing exactly one result per input.
+* [concatPairwise](concatPairwise.md) — Concatenate takes the Cartesian product of all inputs and produces one string per combination. Concatenate pairwise aligns values by position and produces one string per position, truncating to the shortest input.
+* [concatMultiValues](concatMultiValues.md) — Passing multiple values to a single input of Concatenate does not combine them — it multiplies the output. Concatenate multiple values is the plugin that collapses multiple values within an input into one string, producing exactly one result per input.
diff --git a/docs/build/reference/transformer/Combine/concatMultiValues.md b/docs/build/reference/transformer/Combine/concatMultiValues.md
index 77c40517d..28118eb91 100644
--- a/docs/build/reference/transformer/Combine/concatMultiValues.md
+++ b/docs/build/reference/transformer/Combine/concatMultiValues.md
@@ -113,4 +113,4 @@ No description
## Related Plugins
-* **concat** — Concatenate multiple values collapses all values within each input into one string, preserving the boundary between inputs. Concatenate crosses that boundary — it takes one value from each input and produces all combinations, so the output grows with the number of inputs and values.
+* [concat](concat.md) — Concatenate multiple values collapses all values within each input into one string, preserving the boundary between inputs. Concatenate crosses that boundary — it takes one value from each input and produces all combinations, so the output grows with the number of inputs and values.
diff --git a/docs/build/reference/transformer/Combine/concatPairwise.md b/docs/build/reference/transformer/Combine/concatPairwise.md
index 7dc9429fc..7b39eec5d 100644
--- a/docs/build/reference/transformer/Combine/concatPairwise.md
+++ b/docs/build/reference/transformer/Combine/concatPairwise.md
@@ -82,5 +82,5 @@ Separator to be inserted between two concatenated strings. The text can contain
## Related Plugins
-* **concat** — Concatenate pairwise matches values by position and produces one combined string per position. Concatenate does not align by position — it produces every combination of values across inputs, so two inputs with three values each yield nine strings, not three.
-* **zip** — When inputs have unequal lengths, Concatenate pairwise drops the extra values from the longer input. Zip solves the same alignment problem for exactly two inputs but keeps them by substituting a configurable placeholder for each missing value.
+* [concat](concat.md) — Concatenate pairwise matches values by position and produces one combined string per position. Concatenate does not align by position — it produces every combination of values across inputs, so two inputs with three values each yield nine strings, not three.
+* [zip](zip.md) — When inputs have unequal lengths, Concatenate pairwise drops the extra values from the longer input. Zip solves the same alignment problem for exactly two inputs but keeps them by substituting a configurable placeholder for each missing value.
diff --git a/docs/build/reference/transformer/Combine/zip.md b/docs/build/reference/transformer/Combine/zip.md
index 9169eae51..d2f63d340 100644
--- a/docs/build/reference/transformer/Combine/zip.md
+++ b/docs/build/reference/transformer/Combine/zip.md
@@ -111,4 +111,4 @@ Separator to be inserted between two concatenated strings. The text can contain
## Related Plugins
-* **concatPairwise** — Zip handles unequal input lengths by padding, not truncating, and is constrained to exactly two inputs. Concatenate pairwise removes that constraint — it accepts any number of inputs — but resolves the length mismatch by stopping at the shortest.
+* [concatPairwise](concatPairwise.md) — Zip handles unequal input lengths by padding, not truncating, and is constrained to exactly two inputs. Concatenate pairwise removes that constraint — it accepts any number of inputs — but resolves the length mismatch by stopping at the shortest.
diff --git a/docs/build/reference/transformer/Conditional/ifMatchesRegex.md b/docs/build/reference/transformer/Conditional/ifMatchesRegex.md
index 3fcb39b81..5fa7b9b98 100644
--- a/docs/build/reference/transformer/Conditional/ifMatchesRegex.md
+++ b/docs/build/reference/transformer/Conditional/ifMatchesRegex.md
@@ -136,6 +136,6 @@ No description
## Related Plugins
-* **validateRegex** — A regular expression match drives both operators, but the outcome differs. The “If matches regex” plugin routes between alternative input values, while the “Validate regex” plugin draws an acceptance boundary by deciding whether the checked value is valid.
-* **regexSelect** — The Regex selection plugin marks match positions by emitting copies of a provided output value wherever a regular expression matches the checked value sequence. The If matches regex plugin uses a regular expression match as a branch decision between alternative input values rather than producing positional markers.
-* **regexExtract** — The Regex extract plugin returns the matching content from the input string, or the first capturing group if the regular expression contains capturing groups. The If matches regex plugin does not return matched content; it uses the match only to choose which of the provided input values becomes the output.
+* [validateRegex](../Validation/validateRegex.md) — A regular expression match drives both operators, but the outcome differs. The “If matches regex” plugin routes between alternative input values, while the “Validate regex” plugin draws an acceptance boundary by deciding whether the checked value is valid.
+* [regexSelect](../Selection/regexSelect.md) — The Regex selection plugin marks match positions by emitting copies of a provided output value wherever a regular expression matches the checked value sequence. The If matches regex plugin uses a regular expression match as a branch decision between alternative input values rather than producing positional markers.
+* [regexExtract](../Extract/regexExtract.md) — The Regex extract plugin returns the matching content from the input string, or the first capturing group if the regular expression contains capturing groups. The If matches regex plugin does not return matched content; it uses the match only to choose which of the provided input values becomes the output.
diff --git a/docs/build/reference/transformer/Date/compareDates.md b/docs/build/reference/transformer/Date/compareDates.md
index e1f6ec9bd..b75de551c 100644
--- a/docs/build/reference/transformer/Date/compareDates.md
+++ b/docs/build/reference/transformer/Date/compareDates.md
@@ -151,5 +151,5 @@ No description
## Related Plugins
-* **validateRegex** — The Compare dates plugin filters both inputs down to valid XSD date literals and returns 1 or 0 based on the comparator, returning 0 when one side contains no valid date at all. The Validate regex plugin enforces a pattern as a hard boundary and fails when a value does not conform, rather than turning invalid input into a 0 result.
-* **compareNumbers** — Compare dates applies ordering and equality comparators to XSD date literals and returns 1 or 0. Compare numbers applies the same comparators to doubles — the two plugins are not interchangeable, as each rejects the other's input type entirely.
+* [validateRegex](../Validation/validateRegex.md) — The Compare dates plugin filters both inputs down to valid XSD date literals and returns 1 or 0 based on the comparator, returning 0 when one side contains no valid date at all. The Validate regex plugin enforces a pattern as a hard boundary and fails when a value does not conform, rather than turning invalid input into a 0 result.
+* [compareNumbers](../Numeric/compareNumbers.md) — Compare dates applies ordering and equality comparators to XSD date literals and returns 1 or 0. Compare numbers applies the same comparators to doubles — the two plugins are not interchangeable, as each rejects the other's input type entirely.
diff --git a/docs/build/reference/transformer/Date/currentDate.md b/docs/build/reference/transformer/Date/currentDate.md
index 3264caa91..985de6cc3 100644
--- a/docs/build/reference/transformer/Date/currentDate.md
+++ b/docs/build/reference/transformer/Date/currentDate.md
@@ -25,4 +25,4 @@ Outputs the current date.
## Related Plugins
-- **parseDate** — Current date always outputs today's date, ignoring whatever values it receives. Parse date is input-driven: it reads a date from the input string and converts it according to a configured format.
+- [parseDate](parseDate.md) — Current date always outputs today's date, ignoring whatever values it receives. Parse date is input-driven: it reads a date from the input string and converts it according to a configured format.
diff --git a/docs/build/reference/transformer/Date/datetoTimestamp.md b/docs/build/reference/transformer/Date/datetoTimestamp.md
index ee849e4b6..a53e1d552 100644
--- a/docs/build/reference/transformer/Date/datetoTimestamp.md
+++ b/docs/build/reference/transformer/Date/datetoTimestamp.md
@@ -75,4 +75,4 @@ No description
## Related Plugins
-* **timeToDate** — The two plugins are inverses. Date to timestamp takes a date and outputs a Unix integer; Timestamp to date takes a Unix integer and outputs a date.
+* [timeToDate](timeToDate.md) — The two plugins are inverses. Date to timestamp takes a date and outputs a Unix integer; Timestamp to date takes a Unix integer and outputs a date.
diff --git a/docs/build/reference/transformer/Date/duration.md b/docs/build/reference/transformer/Date/duration.md
index cac1aacb0..51201567d 100644
--- a/docs/build/reference/transformer/Date/duration.md
+++ b/docs/build/reference/transformer/Date/duration.md
@@ -25,4 +25,4 @@ Computes the time difference between two data times.
## Related Plugins
-- **numberToDuration** — Duration is a measurement plugin: it takes a start and an end date and returns the interval between them as a duration. Number to duration is a construction plugin: it takes a number and builds a duration from it.
+- [numberToDuration](numberToDuration.md) — Duration is a measurement plugin: it takes a start and an end date and returns the interval between them as a duration. Number to duration is a construction plugin: it takes a number and builds a duration from it.
diff --git a/docs/build/reference/transformer/Date/durationInDays.md b/docs/build/reference/transformer/Date/durationInDays.md
index db8dd3ca1..1912829c4 100644
--- a/docs/build/reference/transformer/Date/durationInDays.md
+++ b/docs/build/reference/transformer/Date/durationInDays.md
@@ -25,4 +25,4 @@ Converts an xsd:duration to days.
## Related Plugins
-- **numberToDuration** — Duration in days extracts a day count from a duration. Number to duration is the reverse: it builds a duration from a number, and days is its default unit.
+- [numberToDuration](numberToDuration.md) — Duration in days extracts a day count from a duration. Number to duration is the reverse: it builds a duration from a number, and days is its default unit.
diff --git a/docs/build/reference/transformer/Date/durationInSeconds.md b/docs/build/reference/transformer/Date/durationInSeconds.md
index a07cad714..849edb55a 100644
--- a/docs/build/reference/transformer/Date/durationInSeconds.md
+++ b/docs/build/reference/transformer/Date/durationInSeconds.md
@@ -25,4 +25,4 @@ Converts an xsd:duration to seconds.
## Related Plugins
-- **numberToDuration** — Duration in seconds outputs a second count; Number to duration consumes one. Configured for seconds, Number to duration is the write operation to Duration in seconds' read.
+- [numberToDuration](numberToDuration.md) — Duration in seconds outputs a second count; Number to duration consumes one. Configured for seconds, Number to duration is the write operation to Duration in seconds' read.
diff --git a/docs/build/reference/transformer/Date/durationInYears.md b/docs/build/reference/transformer/Date/durationInYears.md
index a2b9f81e7..a6984f2b2 100644
--- a/docs/build/reference/transformer/Date/durationInYears.md
+++ b/docs/build/reference/transformer/Date/durationInYears.md
@@ -25,4 +25,4 @@ Converts an xsd:duration to years.
## Related Plugins
-- **numberToDuration** — Duration in years reduces a duration to a plain year count. Number to duration goes the other direction: it builds a duration from a year count when configured for years.
+- [numberToDuration](numberToDuration.md) — Duration in years reduces a duration to a plain year count. Number to duration goes the other direction: it builds a duration from a year count when configured for years.
diff --git a/docs/build/reference/transformer/Date/numberToDuration.md b/docs/build/reference/transformer/Date/numberToDuration.md
index 341012874..599772d18 100644
--- a/docs/build/reference/transformer/Date/numberToDuration.md
+++ b/docs/build/reference/transformer/Date/numberToDuration.md
@@ -31,7 +31,7 @@ No description
## Related Plugins
-- **durationInDays** — Number to duration and Duration in days form a round-trip: one builds a duration from days, the other extracts days from a duration.
-- **durationInSeconds** — Duration in seconds produces the finest-grained numeric output among the duration converters. Number to duration is its reverse when configured for seconds: it constructs a duration from a second count.
-- **durationInYears** — Duration in years extracts a year count from a duration value. Number to duration reconstructs the duration from that count.
-- **duration** — The two plugins produce durations by different means. Duration measures the gap between a start and an end date; Number to duration builds one from a time span expressed as a number.
+- [durationInDays](durationInDays.md) — Number to duration and Duration in days form a round-trip: one builds a duration from days, the other extracts days from a duration.
+- [durationInSeconds](durationInSeconds.md) — Duration in seconds produces the finest-grained numeric output among the duration converters. Number to duration is its reverse when configured for seconds: it constructs a duration from a second count.
+- [durationInYears](durationInYears.md) — Duration in years extracts a year count from a duration value. Number to duration reconstructs the duration from that count.
+- [duration](duration.md) — The two plugins produce durations by different means. Duration measures the gap between a start and an end date; Number to duration builds one from a time span expressed as a number.
diff --git a/docs/build/reference/transformer/Date/parseDate.md b/docs/build/reference/transformer/Date/parseDate.md
index 83106cc07..5fdd50824 100644
--- a/docs/build/reference/transformer/Date/parseDate.md
+++ b/docs/build/reference/transformer/Date/parseDate.md
@@ -146,4 +146,4 @@ Optional locale for the date format. If not set the system's locale will be used
## Related Plugins
-* **currentDate** — Parse date converts an input string to a date using a configured format. Current date ignores the input entirely and always outputs today's date.
+* [currentDate](currentDate.md) — Parse date converts an input string to a date using a configured format. Current date ignores the input entirely and always outputs today's date.
diff --git a/docs/build/reference/transformer/Date/timeToDate.md b/docs/build/reference/transformer/Date/timeToDate.md
index 903bf7fd9..22611137e 100644
--- a/docs/build/reference/transformer/Date/timeToDate.md
+++ b/docs/build/reference/transformer/Date/timeToDate.md
@@ -80,4 +80,4 @@ No description
## Related Plugins
-* **datetoTimestamp** — Timestamp to date converts a Unix integer to a date string; Date to timestamp is the reverse of that, converting a date string back to a Unix integer.
+* [datetoTimestamp](datetoTimestamp.md) — Timestamp to date converts a Unix integer to a date string; Date to timestamp is the reverse of that, converting a date string back to a Unix integer.
diff --git a/docs/build/reference/transformer/Extract/regexExtract.md b/docs/build/reference/transformer/Extract/regexExtract.md
index 0b4cd74be..cce3e7b29 100644
--- a/docs/build/reference/transformer/Extract/regexExtract.md
+++ b/docs/build/reference/transformer/Extract/regexExtract.md
@@ -174,7 +174,7 @@ If true, all matches are extracted. If false, only the first match is extracted
## Related Plugins
-* **regexReplace** — The Regex extract plugin returns what the regular expression matches, or the first capturing group if capturing groups exist. The Regex replace plugin returns the full input string after rewriting it by replacing every match with the configured replacement.
-* **regexSelect** — The Regex selection plugin does not return matched text at all. It emits copies of a provided output value at the positions where the checked values match the regular expressions, while the Regex extract plugin returns the matched substring or capturing-group content.
-* **ifMatchesRegex** — The If matches regex plugin uses the match only as a decision about which provided input value to return. The Regex extract plugin uses the match as the produced content, so the output is derived from the matched region rather than from branch inputs.
-* **validateRegex** — The Validate regex plugin keeps the original value only when the full value matches the configured regular expression and otherwise fails validation. The Regex extract plugin returns match-derived output and can return an empty result when nothing matches.
+* [regexReplace](../Replace/regexReplace.md) — The Regex extract plugin returns what the regular expression matches, or the first capturing group if capturing groups exist. The Regex replace plugin returns the full input string after rewriting it by replacing every match with the configured replacement.
+* [regexSelect](../Selection/regexSelect.md) — The Regex selection plugin does not return matched text at all. It emits copies of a provided output value at the positions where the checked values match the regular expressions, while the Regex extract plugin returns the matched substring or capturing-group content.
+* [ifMatchesRegex](../Conditional/ifMatchesRegex.md) — The If matches regex plugin uses the match only as a decision about which provided input value to return. The Regex extract plugin uses the match as the produced content, so the output is derived from the matched region rather than from branch inputs.
+* [validateRegex](../Validation/validateRegex.md) — The Validate regex plugin keeps the original value only when the full value matches the configured regular expression and otherwise fails validation. The Regex extract plugin returns match-derived output and can return an empty result when nothing matches.
diff --git a/docs/build/reference/transformer/Filter/filterByRegex.md b/docs/build/reference/transformer/Filter/filterByRegex.md
index 30e083bc0..ac817a7cd 100644
--- a/docs/build/reference/transformer/Filter/filterByRegex.md
+++ b/docs/build/reference/transformer/Filter/filterByRegex.md
@@ -41,4 +41,4 @@ No description
## Related Plugins
-- **regexSelect** — Filter by regex keeps or drops values from the input sequence based on full-string matching. Regex selection keeps the checked value out of the output and instead returns a pattern-list-shaped result filled with the provided output value where a pattern matches.
+- [regexSelect](../Selection/regexSelect.md) — Filter by regex keeps or drops values from the input sequence based on full-string matching. Regex selection keeps the checked value out of the output and instead returns a pattern-list-shaped result filled with the provided output value where a pattern matches.
diff --git a/docs/build/reference/transformer/Filter/removeDefaultStopWords.md b/docs/build/reference/transformer/Filter/removeDefaultStopWords.md
index d049a9c3f..b10a50bbe 100644
--- a/docs/build/reference/transformer/Filter/removeDefaultStopWords.md
+++ b/docs/build/reference/transformer/Filter/removeDefaultStopWords.md
@@ -56,4 +56,4 @@ If a different stop word list is needed, the Remove stop words plugin supports p
## Related Plugins
-* **removeRemoteStopWords** — The Remove remote stop words plugin performs stop word removal using a stop word list fetched from a remote URL, while the Remove default stop words plugin performs stop word removal using the built-in default list.
+* [removeRemoteStopWords](removeRemoteStopWords.md) — The Remove remote stop words plugin performs stop word removal using a stop word list fetched from a remote URL, while the Remove default stop words plugin performs stop word removal using the built-in default list.
diff --git a/docs/build/reference/transformer/Filter/removeEmptyValues.md b/docs/build/reference/transformer/Filter/removeEmptyValues.md
index 025b892fb..00233fc70 100644
--- a/docs/build/reference/transformer/Filter/removeEmptyValues.md
+++ b/docs/build/reference/transformer/Filter/removeEmptyValues.md
@@ -48,5 +48,5 @@ Removes empty values.
## Related Plugins
-* **removeValues** — Remove empty values removes only empty strings and has no parameters. Remove values is the configurable alternative, filtering out values that match words from a user-supplied blacklist.
-* **emptyValue** — Empty value produces what Remove empty values removes: an empty sequence. Remove empty values is selective; Empty value is unconditional.
+* [removeValues](removeValues.md) — Remove empty values removes only empty strings and has no parameters. Remove values is the configurable alternative, filtering out values that match words from a user-supplied blacklist.
+* [emptyValue](../Value/emptyValue.md) — Empty value produces what Remove empty values removes: an empty sequence. Remove empty values is selective; Empty value is unconditional.
diff --git a/docs/build/reference/transformer/Filter/removeRemoteStopWords.md b/docs/build/reference/transformer/Filter/removeRemoteStopWords.md
index 19f7ded7f..a7567adbe 100644
--- a/docs/build/reference/transformer/Filter/removeRemoteStopWords.md
+++ b/docs/build/reference/transformer/Filter/removeRemoteStopWords.md
@@ -78,4 +78,4 @@ RegEx for detecting words
## Related Plugins
-- **regexReplace** — The Remove remote stop words plugin removes tokens case-insensitively based on a stop word list loaded from a remote URL after splitting the input with the separator regex. The Regex replace plugin rewrites or deletes substrings based on a regular expression match in the string, which fits cases where the noise is defined by pattern rather than by a word list.
+- [regexReplace](../Replace/regexReplace.md) — The Remove remote stop words plugin removes tokens case-insensitively based on a stop word list loaded from a remote URL after splitting the input with the separator regex. The Regex replace plugin rewrites or deletes substrings based on a regular expression match in the string, which fits cases where the noise is defined by pattern rather than by a word list.
diff --git a/docs/build/reference/transformer/Filter/removeValues.md b/docs/build/reference/transformer/Filter/removeValues.md
index 43486df48..615a983f4 100644
--- a/docs/build/reference/transformer/Filter/removeValues.md
+++ b/docs/build/reference/transformer/Filter/removeValues.md
@@ -31,5 +31,5 @@ No description
## Related Plugins
-- **removeEmptyValues** — Remove values works from a blacklist: any value matching a word in that list is dropped. Remove empty values has no such list; it removes only empty strings.
-- **removeDuplicates** — The two plugins filter on different grounds. Remove values drops a value because of what it is; Remove duplicates drops a value because it already appeared earlier in the sequence.
+- [removeEmptyValues](removeEmptyValues.md) — Remove values works from a blacklist: any value matching a word in that list is dropped. Remove empty values has no such list; it removes only empty strings.
+- [removeDuplicates](../Normalize/removeDuplicates.md) — The two plugins filter on different grounds. Remove values drops a value because of what it is; Remove duplicates drops a value because it already appeared earlier in the sequence.
diff --git a/docs/build/reference/transformer/Linguistic/NYSIIS.md b/docs/build/reference/transformer/Linguistic/NYSIIS.md
index 20d1af232..90c1a6497 100644
--- a/docs/build/reference/transformer/Linguistic/NYSIIS.md
+++ b/docs/build/reference/transformer/Linguistic/NYSIIS.md
@@ -63,5 +63,5 @@ No description
## Related Plugins
-- **soundex** — The NYSIIS plugin encodes a name into a phonetic key, but it is not the same kind of key as Soundex. The Soundex plugin produces a fixed, coarse code, while NYSIIS keeps more structure so fewer distinct names collapse into the same bucket.
-- **metaphone** — The Metaphone plugin follows its own phonetic encoding path and returns a key that lives in a different rule space than NYSIIS. Switching between Metaphone and the NYSIIS plugin changes which spellings end up identical after encoding, not just how the encoded strings look.
+- [soundex](soundex.md) — The NYSIIS plugin encodes a name into a phonetic key, but it is not the same kind of key as Soundex. The Soundex plugin produces a fixed, coarse code, while NYSIIS keeps more structure so fewer distinct names collapse into the same bucket.
+- [metaphone](metaphone.md) — The Metaphone plugin follows its own phonetic encoding path and returns a key that lives in a different rule space than NYSIIS. Switching between Metaphone and the NYSIIS plugin changes which spellings end up identical after encoding, not just how the encoded strings look.
diff --git a/docs/build/reference/transformer/Linguistic/metaphone.md b/docs/build/reference/transformer/Linguistic/metaphone.md
index 0cb144ea6..d637de024 100644
--- a/docs/build/reference/transformer/Linguistic/metaphone.md
+++ b/docs/build/reference/transformer/Linguistic/metaphone.md
@@ -47,5 +47,5 @@ Illustrative examples:
## Related Plugins
-* **soundex** — The Metaphone plugin returns a phonetic encoding whose length depends on the input. The Soundex plugin returns a fixed four-character code, so it forces a much coarser normalization. This is not a similarity score; it is an encoding step.
-* **NYSIIS** — The Metaphone plugin and the NYSIIS plugin both produce phonetic encodings, but they do so under different encoding rules. The NYSIIS plugin also exposes a refined versus non-refined mode, while the Metaphone plugin has a single fixed encoding path.
+* [soundex](soundex.md) — The Metaphone plugin returns a phonetic encoding whose length depends on the input. The Soundex plugin returns a fixed four-character code, so it forces a much coarser normalization. This is not a similarity score; it is an encoding step.
+* [NYSIIS](NYSIIS.md) — The Metaphone plugin and the NYSIIS plugin both produce phonetic encodings, but they do so under different encoding rules. The NYSIIS plugin also exposes a refined versus non-refined mode, while the Metaphone plugin has a single fixed encoding path.
diff --git a/docs/build/reference/transformer/Linguistic/normalizeChars.md b/docs/build/reference/transformer/Linguistic/normalizeChars.md
index 517000e91..8b0d6e8e5 100644
--- a/docs/build/reference/transformer/Linguistic/normalizeChars.md
+++ b/docs/build/reference/transformer/Linguistic/normalizeChars.md
@@ -25,5 +25,5 @@ Replaces diacritical characters with non-diacritical ones (eg, ö -> o), plus so
## Related Plugins
-- **removeSpecialChars** — After Normalize chars, a string still contains its original punctuation, spaces, and symbols. Remove special chars removes all of those, keeping only letters, digits, and underscores.
-- **alphaReduce** — Normalize chars is a substitution-only plugin: it converts diacritics but does not remove anything. Strip non-alphabetic characters is a removal-only plugin: it strips digits and punctuation while keeping letters and spaces, but leaves diacritical letters in their original form.
+- [removeSpecialChars](../Normalize/removeSpecialChars.md) — After Normalize chars, a string still contains its original punctuation, spaces, and symbols. Remove special chars removes all of those, keeping only letters, digits, and underscores.
+- [alphaReduce](../Normalize/alphaReduce.md) — Normalize chars is a substitution-only plugin: it converts diacritics but does not remove anything. Strip non-alphabetic characters is a removal-only plugin: it strips digits and punctuation while keeping letters and spaces, but leaves diacritical letters in their original form.
diff --git a/docs/build/reference/transformer/Linguistic/soundex.md b/docs/build/reference/transformer/Linguistic/soundex.md
index 3e369b9af..d443f8c6a 100644
--- a/docs/build/reference/transformer/Linguistic/soundex.md
+++ b/docs/build/reference/transformer/Linguistic/soundex.md
@@ -89,5 +89,5 @@ No description
## Related Plugins
-* **metaphone** — The Soundex plugin turns a name into a short, fixed-format code built around the first letter and digit groups. The Metaphone plugin returns a letter-based phonetic key instead, so the output is not even the same kind of artifact.
-* **NYSIIS** — The Soundex plugin returns a short Soundex code, optionally in refined mode. The NYSIIS plugin returns a different phonetic key, and the refined flag is internal to each encoder rather than a shared setting that makes the two outputs compatible.
+* [metaphone](metaphone.md) — The Soundex plugin turns a name into a short, fixed-format code built around the first letter and digit groups. The Metaphone plugin returns a letter-based phonetic key instead, so the output is not even the same kind of artifact.
+* [NYSIIS](NYSIIS.md) — The Soundex plugin returns a short Soundex code, optionally in refined mode. The NYSIIS plugin returns a different phonetic key, and the refined flag is internal to each encoder rather than a shared setting that makes the two outputs compatible.
diff --git a/docs/build/reference/transformer/Normalize/alphaReduce.md b/docs/build/reference/transformer/Normalize/alphaReduce.md
index 7a1ce04f1..42625e045 100644
--- a/docs/build/reference/transformer/Normalize/alphaReduce.md
+++ b/docs/build/reference/transformer/Normalize/alphaReduce.md
@@ -25,5 +25,5 @@ Strips all non-alphabetic characters from a string. Spaces are retained.
## Related Plugins
-- **removeSpecialChars** — Strip non-alphabetic characters removes digits along with punctuation. Remove special chars keeps digits but does not preserve spaces. When numeric content in the string needs to survive, Remove special chars is the applicable plugin.
-- **normalizeChars** — Strip non-alphabetic characters removes digits and punctuation but does not normalize the letters it keeps; a diacritical letter in the input is a diacritical letter in the output. Normalize chars addresses that: it converts diacritical characters to ASCII equivalents, though it does not strip any content.
+- [removeSpecialChars](removeSpecialChars.md) — Strip non-alphabetic characters removes digits along with punctuation. Remove special chars keeps digits but does not preserve spaces. When numeric content in the string needs to survive, Remove special chars is the applicable plugin.
+- [normalizeChars](../Linguistic/normalizeChars.md) — Strip non-alphabetic characters removes digits and punctuation but does not normalize the letters it keeps; a diacritical letter in the input is a diacritical letter in the output. Normalize chars addresses that: it converts diacritical characters to ASCII equivalents, though it does not strip any content.
diff --git a/docs/build/reference/transformer/Normalize/capitalize.md b/docs/build/reference/transformer/Normalize/capitalize.md
index 1ea8056ef..2914b2113 100644
--- a/docs/build/reference/transformer/Normalize/capitalize.md
+++ b/docs/build/reference/transformer/Normalize/capitalize.md
@@ -60,5 +60,5 @@ No description
## Related Plugins
-* **lowerCase** — Capitalize raises only the first character, or the first of each word, leaving the rest of the string unchanged. Lower case converts every character, making it the right choice when the entire string needs to be normalized rather than just its initial character.
-* **upperCase** — Capitalize changes only the first character, leaving the rest of the string as-is. Upper case is the right plugin when every character needs to be raised, not just the initial one.
+* [lowerCase](lowerCase.md) — Capitalize raises only the first character, or the first of each word, leaving the rest of the string unchanged. Lower case converts every character, making it the right choice when the entire string needs to be normalized rather than just its initial character.
+* [upperCase](upperCase.md) — Capitalize changes only the first character, leaving the rest of the string as-is. Upper case is the right plugin when every character needs to be raised, not just the initial one.
diff --git a/docs/build/reference/transformer/Normalize/lowerCase.md b/docs/build/reference/transformer/Normalize/lowerCase.md
index e2622b132..6a89ffbcc 100644
--- a/docs/build/reference/transformer/Normalize/lowerCase.md
+++ b/docs/build/reference/transformer/Normalize/lowerCase.md
@@ -39,5 +39,5 @@ Converts a string to lower case.
## Related Plugins
-- **upperCase** — Lower case and Upper case apply the same exhaustive rule in opposite directions — every character is converted, none left unchanged. Upper case is the choice when the target is uniform all-caps.
-- **capitalize** — Lower case converts every character without exception. Capitalize is more selective: it uppercases only the first character of the string, leaving the rest unchanged.
+- [upperCase](upperCase.md) — Lower case and Upper case apply the same exhaustive rule in opposite directions — every character is converted, none left unchanged. Upper case is the choice when the target is uniform all-caps.
+- [capitalize](capitalize.md) — Lower case converts every character without exception. Capitalize is more selective: it uppercases only the first character of the string, leaving the rest unchanged.
diff --git a/docs/build/reference/transformer/Normalize/removeBlanks.md b/docs/build/reference/transformer/Normalize/removeBlanks.md
index 738378092..4122c7175 100644
--- a/docs/build/reference/transformer/Normalize/removeBlanks.md
+++ b/docs/build/reference/transformer/Normalize/removeBlanks.md
@@ -25,4 +25,4 @@ Remove whitespace from a string.
## Related Plugins
-- **trim** — Remove blanks removes only plain space characters and does so throughout the entire string regardless of position. Trim is the choice when only the surrounding whitespace needs to go and the internal structure must be preserved; unlike Remove blanks, it also handles tabs and newlines at the edges.
+- [trim](trim.md) — Remove blanks removes only plain space characters and does so throughout the entire string regardless of position. Trim is the choice when only the surrounding whitespace needs to go and the internal structure must be preserved; unlike Remove blanks, it also handles tabs and newlines at the edges.
diff --git a/docs/build/reference/transformer/Normalize/removeDuplicates.md b/docs/build/reference/transformer/Normalize/removeDuplicates.md
index a02ecb89f..58c646a0f 100644
--- a/docs/build/reference/transformer/Normalize/removeDuplicates.md
+++ b/docs/build/reference/transformer/Normalize/removeDuplicates.md
@@ -25,4 +25,4 @@ Removes duplicated values, making a value sequence distinct.
## Related Plugins
-- **removeValues** — Remove values is driven by a reference list — it drops every instance of a blacklisted word. Remove duplicates needs no such list: it keeps the first occurrence of each value and discards the rest, based solely on the input values repeating themselves.
+- [removeValues](../Filter/removeValues.md) — Remove values is driven by a reference list — it drops every instance of a blacklisted word. Remove duplicates needs no such list: it keeps the first occurrence of each value and discards the rest, based solely on the input values repeating themselves.
diff --git a/docs/build/reference/transformer/Normalize/removeSpecialChars.md b/docs/build/reference/transformer/Normalize/removeSpecialChars.md
index 9e5cd4224..634cdda3c 100644
--- a/docs/build/reference/transformer/Normalize/removeSpecialChars.md
+++ b/docs/build/reference/transformer/Normalize/removeSpecialChars.md
@@ -25,5 +25,5 @@ Remove special characters (including punctuation) from a string.
## Related Plugins
-- **normalizeChars** — Remove special chars keeps diacritical characters intact: because they qualify as Unicode letters, they are neither removed nor modified. Normalize chars is the right choice when diacritics need to be converted to their ASCII base forms rather than preserved as-is.
-- **alphaReduce** — The two plugins differ on digits and spaces: Remove special chars keeps digits and removes spaces; Strip non-alphabetic characters removes digits and keeps spaces. Strip non-alphabetic characters is the right tool when word spacing matters and digits do not belong in the output.
+- [normalizeChars](../Linguistic/normalizeChars.md) — Remove special chars keeps diacritical characters intact: because they qualify as Unicode letters, they are neither removed nor modified. Normalize chars is the right choice when diacritics need to be converted to their ASCII base forms rather than preserved as-is.
+- [alphaReduce](alphaReduce.md) — The two plugins differ on digits and spaces: Remove special chars keeps digits and removes spaces; Strip non-alphabetic characters removes digits and keeps spaces. Strip non-alphabetic characters is the right tool when word spacing matters and digits do not belong in the output.
diff --git a/docs/build/reference/transformer/Normalize/trim.md b/docs/build/reference/transformer/Normalize/trim.md
index 8d629a9b8..0fab72c14 100644
--- a/docs/build/reference/transformer/Normalize/trim.md
+++ b/docs/build/reference/transformer/Normalize/trim.md
@@ -25,4 +25,4 @@ Remove leading and trailing whitespaces.
## Related Plugins
-- **removeBlanks** — Trim removes all whitespace characters from the edges of a string: spaces, tabs, and newlines at the start or end are cleared, but the interior is left untouched. Remove blanks removes only plain space characters, but does so throughout the entire string, including the middle.
+- [removeBlanks](removeBlanks.md) — Trim removes all whitespace characters from the edges of a string: spaces, tabs, and newlines at the start or end are cleared, but the interior is left untouched. Remove blanks removes only plain space characters, but does so throughout the entire string, including the middle.
diff --git a/docs/build/reference/transformer/Normalize/upperCase.md b/docs/build/reference/transformer/Normalize/upperCase.md
index 62dc3c8e0..912032aea 100644
--- a/docs/build/reference/transformer/Normalize/upperCase.md
+++ b/docs/build/reference/transformer/Normalize/upperCase.md
@@ -25,5 +25,5 @@ Converts a string to upper case.
## Related Plugins
-- **lowerCase** — Upper case and Lower case are exact complements — one raises all characters, the other lowers them. Lower case is the choice when uniform lowercase is the target.
-- **capitalize** — Upper case raises every character to upper case. Capitalize raises only the first character of the string, leaving the rest in its original case.
+- [lowerCase](lowerCase.md) — Upper case and Lower case are exact complements — one raises all characters, the other lowers them. Lower case is the choice when uniform lowercase is the target.
+- [capitalize](capitalize.md) — Upper case raises every character to upper case. Capitalize raises only the first character of the string, leaving the rest in its original case.
diff --git a/docs/build/reference/transformer/Numeric/aggregateNumbers.md b/docs/build/reference/transformer/Numeric/aggregateNumbers.md
index f561c068d..cf05a2dfe 100644
--- a/docs/build/reference/transformer/Numeric/aggregateNumbers.md
+++ b/docs/build/reference/transformer/Numeric/aggregateNumbers.md
@@ -246,5 +246,5 @@ The aggregation operation to be applied to all values. One of `+`, `*`, `min`, `
## Related Plugins
-* **numOperation** — The Aggregate numbers plugin and the Numeric operation plugin both reduce a sequence of numeric inputs to one result, overlapping on addition and multiplication. Aggregate numbers ignores non-numeric values and adds min, max, and average, while Numeric operation fails on non-numeric input and adds subtraction and division.
-* **numReduce** — The silent discard of non-numeric values in Aggregate numbers is not a cleaning step — non-numeric characters within a value leave the entire value unparseable. Numeric reduce strips those characters from each value, exposing a digit sequence that Aggregate numbers can then include in its computation.
+* [numOperation](numOperation.md) — The Aggregate numbers plugin and the Numeric operation plugin both reduce a sequence of numeric inputs to one result, overlapping on addition and multiplication. Aggregate numbers ignores non-numeric values and adds min, max, and average, while Numeric operation fails on non-numeric input and adds subtraction and division.
+* [numReduce](numReduce.md) — The silent discard of non-numeric values in Aggregate numbers is not a cleaning step — non-numeric characters within a value leave the entire value unparseable. Numeric reduce strips those characters from each value, exposing a digit sequence that Aggregate numbers can then include in its computation.
diff --git a/docs/build/reference/transformer/Numeric/compareNumbers.md b/docs/build/reference/transformer/Numeric/compareNumbers.md
index 74d2d66a7..fcda09235 100644
--- a/docs/build/reference/transformer/Numeric/compareNumbers.md
+++ b/docs/build/reference/transformer/Numeric/compareNumbers.md
@@ -34,4 +34,4 @@ No description
## Related Plugins
-- **compareDates** — Compare numbers and Compare dates both return 1 or 0 by applying the same ordering and equality comparators across two input sets. Compare dates is not a date-aware extension of Compare numbers — each plugin accepts only its own value type, doubles for one and XSD date literals for the other.
+- [compareDates](../Date/compareDates.md) — Compare numbers and Compare dates both return 1 or 0 by applying the same ordering and equality comparators across two input sets. Compare dates is not a date-aware extension of Compare numbers — each plugin accepts only its own value type, doubles for one and XSD date literals for the other.
diff --git a/docs/build/reference/transformer/Numeric/extractPhysicalQuantity.md b/docs/build/reference/transformer/Numeric/extractPhysicalQuantity.md
index c9f5c7d52..ef4818fb7 100644
--- a/docs/build/reference/transformer/Numeric/extractPhysicalQuantity.md
+++ b/docs/build/reference/transformer/Numeric/extractPhysicalQuantity.md
@@ -64,5 +64,5 @@ If there are multiple matches, retrieve the value with the given index (zero-bas
## Related Plugins
-- **numOperation** — The Physical quantity extractor plugin turns number plus unit strings into plain numeric values in the configured base unit. The Numeric operation plugin is the arithmetic reducer once the inputs are already numbers, so unit parsing and calculation stay separate.
-- **formatNumber** — Extract physical quantity returns a plain numeric string in the base unit. Format number takes that value and renders it according to a decimal format pattern, controlling precision, digit grouping, and separators.
+- [numOperation](numOperation.md) — The Physical quantity extractor plugin turns number plus unit strings into plain numeric values in the configured base unit. The Numeric operation plugin is the arithmetic reducer once the inputs are already numbers, so unit parsing and calculation stay separate.
+- [formatNumber](formatNumber.md) — Extract physical quantity returns a plain numeric string in the base unit. Format number takes that value and renders it according to a decimal format pattern, controlling precision, digit grouping, and separators.
diff --git a/docs/build/reference/transformer/Numeric/formatNumber.md b/docs/build/reference/transformer/Numeric/formatNumber.md
index 34fb9adcb..2ea5885bb 100644
--- a/docs/build/reference/transformer/Numeric/formatNumber.md
+++ b/docs/build/reference/transformer/Numeric/formatNumber.md
@@ -147,4 +147,4 @@ No description
## Related Plugins
-* **extractPhysicalQuantity** — Format number requires a numeric input. If the source data contains quantity strings with embedded unit symbols, Extract physical quantity parses those strings and returns the numeric value in the base unit — the form that Format number can then render according to a decimal pattern.
+* [extractPhysicalQuantity](extractPhysicalQuantity.md) — Format number requires a numeric input. If the source data contains quantity strings with embedded unit symbols, Extract physical quantity parses those strings and returns the numeric value in the base unit — the form that Format number can then render according to a decimal pattern.
diff --git a/docs/build/reference/transformer/Numeric/numOperation.md b/docs/build/reference/transformer/Numeric/numOperation.md
index 222024bb9..d4d0c35cb 100644
--- a/docs/build/reference/transformer/Numeric/numOperation.md
+++ b/docs/build/reference/transformer/Numeric/numOperation.md
@@ -158,6 +158,6 @@ The operator to be applied to all values. One of `+`, `-`, `*`, `/`
## Related Plugins
-* **aggregateNumbers** — The Numeric operation plugin reduces all input numbers into one result using one arithmetic operator and fails when any value is not a number. The Aggregate numbers plugin also reduces to one result, but it ignores non-numeric values and shifts the operator set toward aggregation semantics such as minimum, maximum, and average.
-* **extractPhysicalQuantity** — The Extract physical quantity plugin converts number-and-unit text into base-unit numeric values as plain numeric output. The Numeric operation plugin combines those numeric values using one arithmetic operator across the operand sequence.
-* **numReduce** — Numeric reduce strips non-numeric characters from each value. Numeric operation then applies an arithmetic operator across the resulting values, since it throws on any input that is not a number.
+* [aggregateNumbers](aggregateNumbers.md) — The Numeric operation plugin reduces all input numbers into one result using one arithmetic operator and fails when any value is not a number. The Aggregate numbers plugin also reduces to one result, but it ignores non-numeric values and shifts the operator set toward aggregation semantics such as minimum, maximum, and average.
+* [extractPhysicalQuantity](extractPhysicalQuantity.md) — The Extract physical quantity plugin converts number-and-unit text into base-unit numeric values as plain numeric output. The Numeric operation plugin combines those numeric values using one arithmetic operator across the operand sequence.
+* [numReduce](numReduce.md) — Numeric reduce strips non-numeric characters from each value. Numeric operation then applies an arithmetic operator across the resulting values, since it throws on any input that is not a number.
diff --git a/docs/build/reference/transformer/Numeric/numReduce.md b/docs/build/reference/transformer/Numeric/numReduce.md
index cf5b476c1..08dba8895 100644
--- a/docs/build/reference/transformer/Numeric/numReduce.md
+++ b/docs/build/reference/transformer/Numeric/numReduce.md
@@ -60,6 +60,6 @@ No description
## Related Plugins
-* **regexReplace** — Numeric reduce is a zero-configuration specialization of Regex replace using a non-digit stripping pattern. Regex replace is the choice when the stripping rule is not strictly numeric.
-* **aggregateNumbers** — Numeric reduce strips non-numeric characters from each value. Aggregate numbers silently discards any value it cannot parse as a number, so values with embedded non-numeric characters are lost to the aggregation without this step.
-* **numOperation** — Numeric reduce strips non-numeric characters from each value, making each one parseable as a number. Numeric operation throws a validation exception on any input that cannot be parsed, rather than discarding it silently.
+* [regexReplace](../Replace/regexReplace.md) — Numeric reduce is a zero-configuration specialization of Regex replace using a non-digit stripping pattern. Regex replace is the choice when the stripping rule is not strictly numeric.
+* [aggregateNumbers](aggregateNumbers.md) — Numeric reduce strips non-numeric characters from each value. Aggregate numbers silently discards any value it cannot parse as a number, so values with embedded non-numeric characters are lost to the aggregation without this step.
+* [numOperation](numOperation.md) — Numeric reduce strips non-numeric characters from each value, making each one parseable as a number. Numeric operation throws a validation exception on any input that cannot be parsed, rather than discarding it silently.
diff --git a/docs/build/reference/transformer/Replace/map.md b/docs/build/reference/transformer/Replace/map.md
index 5cf124d63..c20a63f2a 100644
--- a/docs/build/reference/transformer/Replace/map.md
+++ b/docs/build/reference/transformer/Replace/map.md
@@ -72,5 +72,5 @@ Default if the map defines no value
## Related Plugins
-* **mapWithDefaultInput** — The Map plugin returns a fixed default string — set as a parameter — for any value not found in the map. Map with default replaces that fixed fallback with a second connected input, so the fallback can differ per value.
-* **replace** — The Map plugin matches the entire input value against a lookup table and substitutes the whole value on an exact match. Replace substitutes a search string wherever it appears within the value, without requiring the full value to match.
+* [mapWithDefaultInput](mapWithDefaultInput.md) — The Map plugin returns a fixed default string — set as a parameter — for any value not found in the map. Map with default replaces that fixed fallback with a second connected input, so the fallback can differ per value.
+* [replace](replace.md) — The Map plugin matches the entire input value against a lookup table and substitutes the whole value on an exact match. Replace substitutes a search string wherever it appears within the value, without requiring the full value to match.
diff --git a/docs/build/reference/transformer/Replace/mapWithDefaultInput.md b/docs/build/reference/transformer/Replace/mapWithDefaultInput.md
index e21c8f11c..134ec75e5 100644
--- a/docs/build/reference/transformer/Replace/mapWithDefaultInput.md
+++ b/docs/build/reference/transformer/Replace/mapWithDefaultInput.md
@@ -41,5 +41,5 @@ A map of values
## Related Plugins
-- **inputHash** — The Map with default plugin maps each input value and returns one output value per position, using the second input as the fallback when a value is not found in the map. The Input hash plugin returns one hash value for all input values combined, so the result is one combined identifier rather than a mapped value sequence.
-- **map** — Map with default takes its fallback values from a second input, one per input value, so the fallback can differ per value. If the fallback is a fixed string that applies to all unmapped values, the Map plugin accepts it as a parameter and requires only one input.
+- [inputHash](../Value/inputHash.md) — The Map with default plugin maps each input value and returns one output value per position, using the second input as the fallback when a value is not found in the map. The Input hash plugin returns one hash value for all input values combined, so the result is one combined identifier rather than a mapped value sequence.
+- [map](map.md) — Map with default takes its fallback values from a second input, one per input value, so the fallback can differ per value. If the fallback is a fixed string that applies to all unmapped values, the Map plugin accepts it as a parameter and requires only one input.
diff --git a/docs/build/reference/transformer/Replace/regexReplace.md b/docs/build/reference/transformer/Replace/regexReplace.md
index fbfe9c083..5b95c2618 100644
--- a/docs/build/reference/transformer/Replace/regexReplace.md
+++ b/docs/build/reference/transformer/Replace/regexReplace.md
@@ -144,7 +144,7 @@ The replacement of each match
## Related Plugins
-* **regexExtract** — The Regex replace plugin returns the full input string after rewriting every match with the replacement. The Regex extract plugin returns only what matched, or the first capturing group, so the output is match-derived content rather than a rewritten string.
-* **regexSelect** — The Regex replace plugin rewrites a string by substituting every match with the configured replacement, so the output stays a transformed version of the input text. The Regex selection plugin turns matching into positional markers by emitting a provided output value at the regex positions that match the checked value.
-* **validateRegex** — The Regex replace plugin rewrites a string by substituting every regex match and returns the rewritten value. The Validate regex plugin keeps the value only when the full value matches the regex and otherwise fails validation, so it serves as a format check before or after rewriting.
-* **replace** — The Replace plugin swaps one fixed substring for another everywhere it occurs. The Regex replace plugin does the same global substitution, but what counts as a hit is described by a regular expression.
+* [regexExtract](../Extract/regexExtract.md) — The Regex replace plugin returns the full input string after rewriting every match with the replacement. The Regex extract plugin returns only what matched, or the first capturing group, so the output is match-derived content rather than a rewritten string.
+* [regexSelect](../Selection/regexSelect.md) — The Regex replace plugin rewrites a string by substituting every match with the configured replacement, so the output stays a transformed version of the input text. The Regex selection plugin turns matching into positional markers by emitting a provided output value at the regex positions that match the checked value.
+* [validateRegex](../Validation/validateRegex.md) — The Regex replace plugin rewrites a string by substituting every regex match and returns the rewritten value. The Validate regex plugin keeps the value only when the full value matches the regex and otherwise fails validation, so it serves as a format check before or after rewriting.
+* [replace](replace.md) — The Replace plugin swaps one fixed substring for another everywhere it occurs. The Regex replace plugin does the same global substitution, but what counts as a hit is described by a regular expression.
diff --git a/docs/build/reference/transformer/Replace/replace.md b/docs/build/reference/transformer/Replace/replace.md
index f616b3ff0..d093beaf7 100644
--- a/docs/build/reference/transformer/Replace/replace.md
+++ b/docs/build/reference/transformer/Replace/replace.md
@@ -72,5 +72,5 @@ The replacement of each match
## Related Plugins
-* **regexReplace** — The Replace plugin substitutes a literal search string everywhere it occurs. The Regex replace plugin does the same kind of rewrite, but the match is defined by a regular expression rather than a fixed substring.
-* **map** — Replace performs in-place substitution of a substring, leaving the rest of the value intact. The Map plugin replaces entire values based on exact key matches and returns a configured default when no match is found.
+* [regexReplace](regexReplace.md) — The Replace plugin substitutes a literal search string everywhere it occurs. The Regex replace plugin does the same kind of rewrite, but the match is defined by a regular expression rather than a fixed substring.
+* [map](map.md) — Replace performs in-place substitution of a substring, leaving the rest of the value intact. The Map plugin replaces entire values based on exact key matches and returns a configured default when no match is found.
diff --git a/docs/build/reference/transformer/SPARQL/.pages b/docs/build/reference/transformer/SPARQL/.pages
new file mode 100644
index 000000000..44ad1fcdf
--- /dev/null
+++ b/docs/build/reference/transformer/SPARQL/.pages
@@ -0,0 +1,3 @@
+nav:
+ - "Escape SPARQL multiline literal": escape_multiline_literal.md
+ - "Escape SPARQL plain literal": escape_literal.md
\ No newline at end of file
diff --git a/docs/build/reference/transformer/SPARQL/escape_literal.md b/docs/build/reference/transformer/SPARQL/escape_literal.md
new file mode 100644
index 000000000..74b170230
--- /dev/null
+++ b/docs/build/reference/transformer/SPARQL/escape_literal.md
@@ -0,0 +1,69 @@
+---
+title: "Escape SPARQL plain literal"
+description: "Escapes a value so it can be safely used inside a SPARQL short-form string literal. Escapes backslashes, quotes, newlines, carriage returns and tabs. The returned value does not include enclosing quotation marks."
+icon: octicons/cross-reference-24
+tags:
+ - TransformOperator
+---
+
+# Escape SPARQL plain literal
+
+
+
+
+
+Escapes a value so it can be safely used inside a SPARQL short-form string literal. Escapes backslashes, quotes, newlines, carriage returns and tabs. The returned value does not include enclosing quotation marks.
+
+## Examples
+
+**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values "first" and "second".
+
+---
+**Example 1:**
+
+* Input values:
+ 1. `[simple value]`
+
+* Returns: `[simple value]`
+
+
+---
+**Example 2:**
+
+* Input values:
+ 1. `[with "quotes"]`
+
+* Returns: `[with \"quotes\"]`
+
+
+---
+**Example 3:**
+
+* Input values:
+ 1. `[back\slash]`
+
+* Returns: `[back\\slash]`
+
+
+---
+**Example 4:**
+
+* Input values:
+ 1.
+ ```text
+ [line1
+ line2]
+ ```
+
+* Returns: `[line1\nline2]`
+
+
+
+
+## Parameter
+
+`None`
+
+## Advanced Parameter
+
+`None`
diff --git a/docs/build/reference/transformer/SPARQL/escape_multiline_literal.md b/docs/build/reference/transformer/SPARQL/escape_multiline_literal.md
new file mode 100644
index 000000000..16f482038
--- /dev/null
+++ b/docs/build/reference/transformer/SPARQL/escape_multiline_literal.md
@@ -0,0 +1,82 @@
+---
+title: "Escape SPARQL multiline literal"
+description: "Escapes a value so it can be safely used inside a SPARQL triple-quoted string literal (`'''...'''` or `'''...'''`). Escapes backslashes and breaks any run of three or more consecutive single or double quotes. Individual quotes and newlines are preserved. The returned value does not include enclosing quotation marks."
+icon: octicons/cross-reference-24
+tags:
+ - TransformOperator
+---
+
+# Escape SPARQL multiline literal
+
+
+
+
+
+Escapes a value so it can be safely used inside a SPARQL triple-quoted string literal (`"""..."""` or `'''...'''`). Escapes backslashes and breaks any run of three or more consecutive single or double quotes. Individual quotes and newlines are preserved. The returned value does not include enclosing quotation marks.
+
+## Examples
+
+**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values "first" and "second".
+
+---
+**Example 1:**
+
+* Input values:
+ 1.
+ ```text
+ [simple
+ value]
+ ```
+
+* Returns:
+ ```text
+ [simple
+ value]
+ ```
+
+
+---
+**Example 2:**
+
+* Input values:
+ 1. `[with "quote"]`
+
+* Returns: `[with "quote"]`
+
+
+---
+**Example 3:**
+
+* Input values:
+ 1. `[back\slash]`
+
+* Returns: `[back\\slash]`
+
+
+---
+**Example 4:**
+
+* Input values:
+ 1. `[triple """ quotes]`
+
+* Returns: `[triple \"\"\" quotes]`
+
+
+---
+**Example 5:**
+
+* Input values:
+ 1. `[triple ''' quotes]`
+
+* Returns: `[triple \'\'\' quotes]`
+
+
+
+
+## Parameter
+
+`None`
+
+## Advanced Parameter
+
+`None`
diff --git a/docs/build/reference/transformer/Selection/regexSelect.md b/docs/build/reference/transformer/Selection/regexSelect.md
index c71e81441..88e2e40cd 100644
--- a/docs/build/reference/transformer/Selection/regexSelect.md
+++ b/docs/build/reference/transformer/Selection/regexSelect.md
@@ -95,7 +95,7 @@ No description
## Related Plugins
-* **regexExtract** — The Regex selection plugin returns the provided output value in a result sequence aligned with the regex list, filling only the positions whose pattern matches the checked value. The Regex extract plugin returns the matched substring itself, or the first capturing group, so the output is taken from the input text rather than from the provided output value.
-* **ifMatchesRegex** — The If matches regex plugin returns one of the provided branch inputs based on whether the checked value matches. The Regex selection plugin returns a result sequence aligned with the pattern list, placing the provided output value at every matching position.
-* **regexReplace** — The Regex replace plugin returns a rewritten string by replacing every match inside the input text with the configured replacement. The Regex selection plugin returns a result sequence aligned with the pattern list and fills each matching position with the provided output value.
-* **filterByRegex** — The Regex selection plugin keeps the checked value out of the output and instead returns a pattern-list-shaped result filled with the provided output value where a pattern matches. The Filter by regex plugin keeps or drops values from the input sequence based on full-string matching.
+* [regexExtract](../Extract/regexExtract.md) — The Regex selection plugin returns the provided output value in a result sequence aligned with the regex list, filling only the positions whose pattern matches the checked value. The Regex extract plugin returns the matched substring itself, or the first capturing group, so the output is taken from the input text rather than from the provided output value.
+* [ifMatchesRegex](../Conditional/ifMatchesRegex.md) — The If matches regex plugin returns one of the provided branch inputs based on whether the checked value matches. The Regex selection plugin returns a result sequence aligned with the pattern list, placing the provided output value at every matching position.
+* [regexReplace](../Replace/regexReplace.md) — The Regex replace plugin returns a rewritten string by replacing every match inside the input text with the configured replacement. The Regex selection plugin returns a result sequence aligned with the pattern list and fills each matching position with the provided output value.
+* [filterByRegex](../Filter/filterByRegex.md) — The Regex selection plugin keeps the checked value out of the output and instead returns a pattern-list-shaped result filled with the provided output value where a pattern matches. The Filter by regex plugin keeps or drops values from the input sequence based on full-string matching.
diff --git a/docs/build/reference/transformer/Substring/stripPostfix.md b/docs/build/reference/transformer/Substring/stripPostfix.md
index 04f17719f..69b3ff6eb 100644
--- a/docs/build/reference/transformer/Substring/stripPostfix.md
+++ b/docs/build/reference/transformer/Substring/stripPostfix.md
@@ -60,5 +60,5 @@ No description
## Related Plugins
-* **stripPrefix** — Strip postfix removes from the end; Strip prefix removes from the start. Both leave the value unchanged when the configured string is not found at the expected position.
-* **substring** — Strip postfix checks for a specific string at the end before removing it. Substring does not check content: a negative end index removes a fixed character count from the end unconditionally.
+* [stripPrefix](stripPrefix.md) — Strip postfix removes from the end; Strip prefix removes from the start. Both leave the value unchanged when the configured string is not found at the expected position.
+* [substring](substring.md) — Strip postfix checks for a specific string at the end before removing it. Substring does not check content: a negative end index removes a fixed character count from the end unconditionally.
diff --git a/docs/build/reference/transformer/Substring/stripPrefix.md b/docs/build/reference/transformer/Substring/stripPrefix.md
index 4a5f291d2..4e243c53c 100644
--- a/docs/build/reference/transformer/Substring/stripPrefix.md
+++ b/docs/build/reference/transformer/Substring/stripPrefix.md
@@ -60,5 +60,5 @@ No description
## Related Plugins
-* **stripPostfix** — Strip prefix removes a configured string from the start of the value, leaving it unchanged if the string is not found there. Strip postfix is the complement: it checks and removes from the end.
-* **substring** — Strip prefix removes a configured string from the start only if it is actually present. Substring removes by position: it skips the first N characters regardless of their content, so it will cut into the value even if the expected prefix is absent.
+* [stripPostfix](stripPostfix.md) — Strip prefix removes a configured string from the start of the value, leaving it unchanged if the string is not found there. Strip postfix is the complement: it checks and removes from the end.
+* [substring](substring.md) — Strip prefix removes a configured string from the start only if it is actually present. Substring removes by position: it skips the first N characters regardless of their content, so it will cut into the value even if the expected prefix is absent.
diff --git a/docs/build/reference/transformer/Substring/substring.md b/docs/build/reference/transformer/Substring/substring.md
index 6a909c243..b88b41892 100644
--- a/docs/build/reference/transformer/Substring/substring.md
+++ b/docs/build/reference/transformer/Substring/substring.md
@@ -163,6 +163,6 @@ If true, only strings will be accepted that are within the start and end indices
## Related Plugins
-* **stripPrefix** — Substring removes a fixed number of characters from the start regardless of their content. Strip prefix is more selective: it only removes from the start if the configured string is actually found there.
-* **stripPostfix** — Substring works by index: it removes a fixed count of trailing characters regardless of their content. Strip postfix is the alternative when the trailing portion is a known string; it checks for it and leaves the value unchanged if not found.
-* **untilCharacter** — Substring extracts by position: the start and end indices are fixed and apply to every input value regardless of its content. Until character extracts up to a specific character.
+* [stripPrefix](stripPrefix.md) — Substring removes a fixed number of characters from the start regardless of their content. Strip prefix is more selective: it only removes from the start if the configured string is actually found there.
+* [stripPostfix](stripPostfix.md) — Substring works by index: it removes a fixed count of trailing characters regardless of their content. Strip postfix is the alternative when the trailing portion is a known string; it checks for it and leaves the value unchanged if not found.
+* [untilCharacter](untilCharacter.md) — Substring extracts by position: the start and end indices are fixed and apply to every input value regardless of its content. Until character extracts up to a specific character.
diff --git a/docs/build/reference/transformer/Substring/untilCharacter.md b/docs/build/reference/transformer/Substring/untilCharacter.md
index 60363f83b..213695389 100644
--- a/docs/build/reference/transformer/Substring/untilCharacter.md
+++ b/docs/build/reference/transformer/Substring/untilCharacter.md
@@ -60,4 +60,4 @@ No description
## Related Plugins
-* **substring** — Until character adapts to the content of each value: the extracted portion ends wherever the target character appears. Substring does not adapt; it cuts at configured numeric indices that are the same for every input.
+* [substring](substring.md) — Until character adapts to the content of each value: the extracted portion ends wherever the target character appears. Substring does not adapt; it cuts at configured numeric indices that are the same for every input.
diff --git a/docs/build/reference/transformer/Tokenization/camelcasetokenizer.md b/docs/build/reference/transformer/Tokenization/camelcasetokenizer.md
index fcaf06c94..5031e49d4 100644
--- a/docs/build/reference/transformer/Tokenization/camelcasetokenizer.md
+++ b/docs/build/reference/transformer/Tokenization/camelcasetokenizer.md
@@ -48,4 +48,4 @@ Tokenizes a camel case string. That is it splits strings between a lower case ch
## Related Plugins
-* **tokenize** — When word boundaries are implicit in case rather than marked by a separator, camel case tokenizer is the right tool. Tokenize requires a separator to be present in the string — it cannot infer boundaries from case alone.
+* [tokenize](tokenize.md) — When word boundaries are implicit in case rather than marked by a separator, camel case tokenizer is the right tool. Tokenize requires a separator to be present in the string — it cannot infer boundaries from case alone.
diff --git a/docs/build/reference/transformer/Tokenization/tokenize.md b/docs/build/reference/transformer/Tokenization/tokenize.md
index 69d81227d..a2daed5c5 100644
--- a/docs/build/reference/transformer/Tokenization/tokenize.md
+++ b/docs/build/reference/transformer/Tokenization/tokenize.md
@@ -57,4 +57,4 @@ The regular expression used to split values.
## Related Plugins
-* **camelcasetokenizer** — A value written in camel case produces a single token under Tokenize, because there is no separator character to split on. Camel case tokenizer reads case transitions as boundaries and splits accordingly.
+* [camelcasetokenizer](camelcasetokenizer.md) — A value written in camel case produces a single token under Tokenize, because there is no separator character to split on. Camel case tokenizer reads case transitions as boundaries and splits accordingly.
diff --git a/docs/build/reference/transformer/Uncategorized/.pages b/docs/build/reference/transformer/Uncategorized/.pages
index 0815e8c5b..f8049f252 100644
--- a/docs/build/reference/transformer/Uncategorized/.pages
+++ b/docs/build/reference/transformer/Uncategorized/.pages
@@ -1,3 +1,4 @@
nav:
- "Convert currency values": cmem_plugin_currencies-transform.md
- - "jq": cmem-plugin-jq-transform.md
\ No newline at end of file
+ - "jq": cmem-plugin-jq-transform.md
+ - "Random value": cmem_plugin_random-GenerateValues.md
\ No newline at end of file
diff --git a/docs/build/reference/transformer/Uncategorized/cmem_plugin_currencies-transform.md b/docs/build/reference/transformer/Uncategorized/cmem_plugin_currencies-transform.md
index 41dc3a33e..106eeea21 100644
--- a/docs/build/reference/transformer/Uncategorized/cmem_plugin_currencies-transform.md
+++ b/docs/build/reference/transformer/Uncategorized/cmem_plugin_currencies-transform.md
@@ -73,7 +73,7 @@ Set date (e.g.YYYY-MM-DD) to convert currencies based on historic rates.
- ID: `date`
- Datatype: `string`
-- Default Value: `2026-05-12`
+- Default Value: `2026-08-04`
diff --git a/docs/build/reference/transformer/Uncategorized/cmem_plugin_random-GenerateValues.md b/docs/build/reference/transformer/Uncategorized/cmem_plugin_random-GenerateValues.md
new file mode 100644
index 000000000..e77d29774
--- /dev/null
+++ b/docs/build/reference/transformer/Uncategorized/cmem_plugin_random-GenerateValues.md
@@ -0,0 +1,44 @@
+---
+title: "Random value"
+description: "Generates random values."
+icon: octicons/cross-reference-24
+tags:
+ - TransformOperator
+ - PythonPlugin
+---
+
+# Random value
+
+
+
+!!! note inline end "Python Plugin"
+
+ This operator is part of a Python Plugin Package.
+ In order to use it, you need to install it,
+ e.g. with cmemc.
+
+Generates random values.
+
+## Parameter
+
+### Random Function
+
+
+
+- ID: `random_function`
+- Datatype: `string`
+- Default Value: `token_urlsafe`
+
+
+
+### String Length
+
+How long (in characters) should each value be.
+
+- ID: `string_length`
+- Datatype: `Long`
+- Default Value: `16`
+
+## Advanced Parameter
+
+`None`
diff --git a/docs/build/reference/transformer/Validation/.pages b/docs/build/reference/transformer/Validation/.pages
index a95f96c8f..a80388570 100644
--- a/docs/build/reference/transformer/Validation/.pages
+++ b/docs/build/reference/transformer/Validation/.pages
@@ -3,4 +3,5 @@ nav:
- "Validate date range": validateDateRange.md
- "Validate number of values": validateNumberOfValues.md
- "Validate numeric range": validateNumericRange.md
- - "Validate regex": validateRegex.md
\ No newline at end of file
+ - "Validate regex": validateRegex.md
+ - "Validate URI": validate_uri.md
\ No newline at end of file
diff --git a/docs/build/reference/transformer/Validation/validateDateAfter.md b/docs/build/reference/transformer/Validation/validateDateAfter.md
index bfb792690..91cee0abd 100644
--- a/docs/build/reference/transformer/Validation/validateDateAfter.md
+++ b/docs/build/reference/transformer/Validation/validateDateAfter.md
@@ -84,4 +84,4 @@ Allow both dates to be equal.
## Related Plugins
-* **validateDateRange** — Validate date after checks whether one input date is later than another. Validate date range instead tests a single date against a fixed interval with a configured minimum and maximum.
+* [validateDateRange](validateDateRange.md) — Validate date after checks whether one input date is later than another. Validate date range instead tests a single date against a fixed interval with a configured minimum and maximum.
diff --git a/docs/build/reference/transformer/Validation/validateDateRange.md b/docs/build/reference/transformer/Validation/validateDateRange.md
index 99075f86b..8cf47b710 100644
--- a/docs/build/reference/transformer/Validation/validateDateRange.md
+++ b/docs/build/reference/transformer/Validation/validateDateRange.md
@@ -41,4 +41,4 @@ Latest allowed data in YYYY-MM-DD
## Related Plugins
-- **validateDateAfter** — Validate date range checks a date against a configured earliest and latest date. Validate date after checks whether one input date is later than another.
+- [validateDateAfter](validateDateAfter.md) — Validate date range checks a date against a configured earliest and latest date. Validate date after checks whether one input date is later than another.
diff --git a/docs/build/reference/transformer/Validation/validateNumericRange.md b/docs/build/reference/transformer/Validation/validateNumericRange.md
index 2c4dd453a..aff4ab34c 100644
--- a/docs/build/reference/transformer/Validation/validateNumericRange.md
+++ b/docs/build/reference/transformer/Validation/validateNumericRange.md
@@ -41,4 +41,4 @@ Maximum allowed number
## Related Plugins
-- **compareNumbers** — Validate numeric range either passes a number through or throws, producing no output on violation. Compare numbers always produces a 1 or 0 regardless of which side is larger, so the downstream pipeline continues in either case.
+- [compareNumbers](../Numeric/compareNumbers.md) — Validate numeric range either passes a number through or throws, producing no output on violation. Compare numbers always produces a 1 or 0 regardless of which side is larger, so the downstream pipeline continues in either case.
diff --git a/docs/build/reference/transformer/Validation/validateRegex.md b/docs/build/reference/transformer/Validation/validateRegex.md
index 006ed0484..7369e2a6b 100644
--- a/docs/build/reference/transformer/Validation/validateRegex.md
+++ b/docs/build/reference/transformer/Validation/validateRegex.md
@@ -136,7 +136,7 @@ regular expression
## Related Plugins
-* **regexReplace** — Regex replace rewrites the input string by substituting every match and returns the rewritten value. Validate regex treats the pattern as a full-value check on the resulting string.
-* **ifMatchesRegex** — A regular expression match plays different roles here. The Validate regex plugin checks each value against the pattern and passes it through only when it fully matches. The If matches regex plugin uses the match to choose which provided branch value is returned.
-* **regexSelect** — Regex selection turns one checked value and a list of patterns into a result sequence aligned with that list, placing the provided output value wherever a pattern matches. Validate regex keeps the original value and treats the pattern as a full-value check.
-* **regexExtract** — Regex extract turns the match into output by returning the matched substring or the first capturing group. Validate regex leaves the value unchanged and only lets it through when the full value matches the pattern.
+* [regexReplace](../Replace/regexReplace.md) — Regex replace rewrites the input string by substituting every match and returns the rewritten value. Validate regex treats the pattern as a full-value check on the resulting string.
+* [ifMatchesRegex](../Conditional/ifMatchesRegex.md) — A regular expression match plays different roles here. The Validate regex plugin checks each value against the pattern and passes it through only when it fully matches. The If matches regex plugin uses the match to choose which provided branch value is returned.
+* [regexSelect](../Selection/regexSelect.md) — Regex selection turns one checked value and a list of patterns into a result sequence aligned with that list, placing the provided output value wherever a pattern matches. Validate regex keeps the original value and treats the pattern as a full-value check.
+* [regexExtract](../Extract/regexExtract.md) — Regex extract turns the match into output by returning the matched substring or the first capturing group. Validate regex leaves the value unchanged and only lets it through when the full value matches the pattern.
diff --git a/docs/build/reference/transformer/Validation/validate_uri.md b/docs/build/reference/transformer/Validation/validate_uri.md
new file mode 100644
index 000000000..1f5db9ab4
--- /dev/null
+++ b/docs/build/reference/transformer/Validation/validate_uri.md
@@ -0,0 +1,67 @@
+---
+title: "Validate URI"
+description: "Validates that the input is a valid absolute IRI and returns it unchanged. Throws a validation error if the input is not a valid IRI."
+icon: octicons/cross-reference-24
+tags:
+ - TransformOperator
+---
+
+# Validate URI
+
+
+
+
+
+Validates that the input is a valid absolute IRI and returns it unchanged. Throws a validation error if the input is not a valid IRI.
+
+## Examples
+
+**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values "first" and "second".
+
+---
+**Example 1:**
+
+* Input values:
+ 1. `[http://example.org/entity1]`
+
+* Returns: `[http://example.org/entity1]`
+
+
+---
+**Example 2:**
+
+* Input values:
+ 1. `[urn:example:1]`
+
+* Returns: `[urn:example:1]`
+
+
+---
+**Example 3:**
+
+* Input values:
+ 1. `[not a uri]`
+
+* Returns: `[]`
+* **Throws error:** `ValidationException`
+
+
+---
+**Example 4:**
+
+* Input values:
+ 1. `[]`
+
+* Returns: `[]`
+* **Throws error:** `ValidationException`
+
+
+
+
+## Parameter
+
+`None`
+
+## Advanced Parameter
+
+`None`
diff --git a/docs/build/reference/transformer/Value/.pages b/docs/build/reference/transformer/Value/.pages
index c8ebe0d15..b1a3c253a 100644
--- a/docs/build/reference/transformer/Value/.pages
+++ b/docs/build/reference/transformer/Value/.pages
@@ -1,10 +1,11 @@
nav:
+ - "Combined input hash": inputHash.md
- "Constant": constant.md
- "Constant URI": constantUri.md
- "Dataset parameter": datasetParameter.md
- "Default Value": defaultValue.md
- "Empty value": emptyValue.md
- - "Input hash": inputHash.md
+ - "Per-value hash": perValueHash.md
- "Random number": randomNumber.md
- "Read parameter": readParameter.md
- "ULID": cmem-plugin-ulid.md
diff --git a/docs/build/reference/transformer/Value/emptyValue.md b/docs/build/reference/transformer/Value/emptyValue.md
index f9fe3c4f4..ebb044924 100644
--- a/docs/build/reference/transformer/Value/emptyValue.md
+++ b/docs/build/reference/transformer/Value/emptyValue.md
@@ -25,4 +25,4 @@ Generates an empty value.
## Related Plugins
-- **removeEmptyValues** — Empty value always outputs an empty sequence, discarding all input. Remove empty values is selective: it passes non-empty strings through and drops only the empty ones.
+- [removeEmptyValues](../Filter/removeEmptyValues.md) — Empty value always outputs an empty sequence, discarding all input. Remove empty values is selective: it passes non-empty strings through and drops only the empty ones.
diff --git a/docs/build/reference/transformer/Value/inputHash.md b/docs/build/reference/transformer/Value/inputHash.md
index 1aa89f952..983a867cc 100644
--- a/docs/build/reference/transformer/Value/inputHash.md
+++ b/docs/build/reference/transformer/Value/inputHash.md
@@ -1,31 +1,156 @@
---
-title: "Input hash"
-description: "Calculates the hash sum of the input values. Generates a single hash sum for all input values combined."
+title: "Combined input hash"
+description: "Calculates a single hash value covering all input values combined, across all input ports. Values are fed into the hash function in port order without any separator between them."
icon: octicons/cross-reference-24
tags:
- TransformOperator
---
-# Input hash
+# Combined input hash
-Calculates the hash sum of the input values. Generates a single hash sum for all input values combined.
-This operator supports using different hash algorithms from the [Secure Hash Algorithms family](https://en.wikipedia.org/wiki/Secure_Hash_Algorithms) (SHA, e.g. SHA256) and two algorithms from the [Message-Digest Algorithm family](https://en.wikipedia.org/wiki/MD5) (MD2 / MD5). Please be aware that some of these algorithms are not secure due the possibility of collision attacks and other attacks.
+The **Combined input hash** operator produces exactly one hash value covering all input values combined, across all connected input ports. However many values arrive and however many ports are connected, the output is always a single string.
+
+## How combining works
+
+All values from all input ports are fed sequentially into a single hash function — port 1 first, then port 2, and so on. Within each port, values are processed in the order they arrive. No separator is inserted between values or between ports. The hash covers the concatenated byte content of all values in that traversal order.
+
+This means the result depends on both the content and the order of values. The same set of values in a different order produces a different hash. Connecting one port with values `["apple", "banana"]` produces the same hash as connecting two ports with `["apple"]` and `["banana"]` respectively, because the bytes are fed in the same sequence either way.
+
+## Output
+
+The output is a single lowercase hexadecimal string. The length depends on the algorithm: 64 characters for SHA-256, 32 for MD5, 40 for SHA-1, 96 for SHA-384, 128 for SHA-512. If the input is empty, the output is the hash of an empty message.
+
+Values are encoded as UTF-8 before hashing.
+
+## Algorithm parameter
+
+The algorithm parameter selects the hash function. The default is SHA-256. The following algorithms from the [SPARQL 1.1 specification](https://www.w3.org/TR/sparql11-query/#func-hash) are supported:
+
+| SPARQL name | Java name | Notes |
+|-------------|-----------|-------|
+| MD5 | MD5 | Weak — vulnerable to collision attacks. Avoid for security-sensitive use. |
+| SHA1 | SHA-1 | Weak — deprecated for most security purposes. |
+| SHA256 | SHA-256 | Recommended default. |
+| SHA384 | SHA-384 | Stronger than SHA-256. |
+| SHA512 | SHA-512 | Strongest in the SPARQL set. |
+
+Additional algorithms available on the JVM (such as SHA-512/256 and SHA-3 variants) are also accepted. The full list is JVM-dependent and visible in the algorithm parameter dropdown.
+
+Note that the Java names use hyphens (SHA-256, SHA-1) where SPARQL uses none (SHA256, SHA1). Both forms are accepted by this operator.
## Examples
**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values "first" and "second".
---
-**Example 1:**
+**A single input value produces one combined SHA-256 hash:**
+
+* Input values:
+ 1. `[input value]`
+
+* Returns: `[f708c2afff0ed197e8551c4dd549ee5b848e0b407106cbdb8e451c8cd1479362]`
+
+
+---
+**Multiple values on one input are combined into a single hash:**
+
+* Input values:
+ 1. `[apple, banana]`
+
+* Returns: `[5b692305517af54eb5ae12b9ff89eaf89e31f6a6ee208365886a18b81a2fc2f8]`
+
+
+---
+**Reversing the value order produces a different hash, confirming order-sensitivity:**
+
+* Input values:
+ 1. `[banana, apple]`
+
+* Returns: `[d4183362b538440bb9a5f82359791c647280e6b657a1812f16f7bcc2b8f141ca]`
+
+
+---
+**Values from multiple ports are combined in port order, producing the same hash as the equivalent single-port sequence:**
+
+* Input values:
+ 1. `[apple]`
+ 2. `[banana]`
+
+* Returns: `[5b692305517af54eb5ae12b9ff89eaf89e31f6a6ee208365886a18b81a2fc2f8]`
+
+
+---
+**The algorithm parameter selects the hash function (MD5):**
+
+* Parameters
+ * algorithm: `MD5`
-- Input values:
+* Input values:
1. `[input value]`
-- Returns: `[f708c2afff0ed197e8551c4dd549ee5b848e0b407106cbdb8e451c8cd1479362]`
+* Returns: `[cee963a28f70ee97751a85ef732e66dd]`
+
+
+---
+**The algorithm parameter selects the hash function (SHA-1):**
+
+* Parameters
+ * algorithm: `SHA-1`
+
+* Input values:
+ 1. `[apple]`
+
+* Returns: `[d0be2dc421be4fcd0172e5afceea3970e2f3d940]`
+
+
+---
+**The algorithm parameter selects the hash function (SHA-384):**
+
+* Parameters
+ * algorithm: `SHA-384`
+
+* Input values:
+ 1. `[apple]`
+
+* Returns: `[3d8786fcb588c93348756c6429717dc6c374a14f7029362281a3b21dc10250ddf0d0578052749822eb08bc0dc1e68b0f]`
+
+
+---
+**The algorithm parameter selects the hash function (SHA-512):**
+
+* Parameters
+ * algorithm: `SHA-512`
+
+* Input values:
+ 1. `[apple]`
+
+* Returns: `[844d8779103b94c18f4aa4cc0c3b4474058580a991fba85d3ca698a0bc9e52c5940feb7a65a3a290e17e6b23ee943ecc4f73e7490327245b4fe5d5efb590feb2]`
+
+
+---
+**Empty input produces the hash of an empty message:**
+
+* Input values:
+ 1. `[]`
+
+* Returns: `[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855]`
+
+
+---
+**Empty algorithm string causes IllegalArgumentException:**
+
+* Parameters
+ * algorithm: ``
+
+* Input values:
+ 1. `[foo]`
+
+* Returns: `[]`
+* **Throws error:** `IllegalArgumentException`
@@ -36,9 +161,9 @@ This operator supports using different hash algorithms from the [Secure Hash Alg
The hash algorithm to be used.
-- ID: `algorithm`
-- Datatype: `string`
-- Default Value: `SHA256`
+* ID: `algorithm`
+* Datatype: `string`
+* Default Value: `SHA256`
## Advanced Parameter
@@ -46,4 +171,5 @@ The hash algorithm to be used.
## Related Plugins
-- **mapWithDefaultInput** — One hash value is produced for the entire set of inputs by the Input hash plugin. The Map with default plugin instead keeps a value sequence and rewrites it position by position through the mapping, falling back to the second input where no mapping entry is found.
+* [perValueHash](perValueHash.md) — The Per-value hash plugin hashes each input value independently and returns one hash per value, preserving cardinality. The Combined input hash plugin instead feeds all values into a single hash function, producing one combined hash regardless of input size.
+* [mapWithDefaultInput](../Replace/mapWithDefaultInput.md) — One hash value is produced for the entire set of inputs by the Combined input hash plugin. The Map with default plugin instead keeps a value sequence and rewrites it position by position through the mapping, falling back to the second input where no mapping entry is found.
diff --git a/docs/build/reference/transformer/Value/perValueHash.md b/docs/build/reference/transformer/Value/perValueHash.md
new file mode 100644
index 000000000..fa9991eb0
--- /dev/null
+++ b/docs/build/reference/transformer/Value/perValueHash.md
@@ -0,0 +1,199 @@
+---
+title: "Per-value hash"
+description: "Hashes each input value independently and returns one hash per value. Accepts exactly one input port."
+icon: octicons/cross-reference-24
+tags:
+ - TransformOperator
+---
+
+# Per-value hash
+
+
+
+
+
+The **Per-value hash** operator hashes each input value independently and returns one hash per value. The output count always equals the input count — cardinality is preserved.
+
+## SPARQL alignment
+
+This operator produces the same output as the SPARQL 1.1 hash functions applied per value. For a single input value, `SHA256(?x)` in SPARQL returns the same result as this operator with the default SHA256 algorithm.
+
+## Single-input constraint
+
+The operator accepts exactly one input port. Connecting more than one port throws an `IllegalArgumentException`. This constraint exists because per-value hashing is defined relative to a single value sequence — combining values across ports would require choosing a port-merging strategy, which is the behaviour of the **Combined input hash** operator instead.
+
+## Output
+
+Each input value produces one lowercase hexadecimal hash string. The output order matches the input order. If the input is empty, the output is empty — no hash is produced.
+
+Values are encoded as UTF-8 before hashing.
+
+## Algorithm parameter
+
+The algorithm parameter selects the hash function. The default is SHA-256. The five algorithms from the [SPARQL 1.1 specification](https://www.w3.org/TR/sparql11-query/#func-hash) are supported:
+
+| SPARQL name | Java name | Notes |
+|-------------|-----------|-------|
+| MD5 | MD5 | Weak — vulnerable to collision attacks. Avoid for security-sensitive use. |
+| SHA1 | SHA-1 | Weak — deprecated for most security purposes. |
+| SHA256 | SHA-256 | Recommended default. |
+| SHA384 | SHA-384 | Stronger than SHA-256. |
+| SHA512 | SHA-512 | Strongest in the SPARQL set. |
+
+Additional algorithms available on the JVM are also accepted. The full list is JVM-dependent and visible in the algorithm parameter dropdown.
+
+Note that the Java names use hyphens (SHA-256, SHA-1) where SPARQL uses none (SHA256, SHA1). Both forms are accepted by this operator.
+
+## Contrast with Combined input hash
+
+The **Combined input hash** operator feeds all values from all ports into a single hash function and returns one hash regardless of input size. Use it when you need a single fingerprint for a set of values taken together.
+
+Use **Per-value hash** when each value needs its own hash — for example, to hash a column of URIs independently, or to replicate `SHA256(?x)` in SPARQL.
+
+## Examples
+
+**Notation:** List of values are represented via square brackets. Example: `[first, second]` represents a list of two values "first" and "second".
+
+---
+**Single value produces one SHA-256 hash:**
+
+* Input values:
+ 1. `[input value]`
+
+* Returns: `[f708c2afff0ed197e8551c4dd549ee5b848e0b407106cbdb8e451c8cd1479362]`
+
+
+---
+**Two values in, two independent hashes out — one per value, not a combined hash:**
+
+* Input values:
+ 1. `[apple, banana]`
+
+* Returns: `[3a7bd3e2360a3d29eea436fcfb7e44c735d117c42d1c1835420b6b9942dd4f1b, b493d48364afe44d11c0165cf470a4164d1e2609911ef998be868d46ade3de4e]`
+
+
+---
+**The algorithm parameter selects the hash function (MD5), single value:**
+
+* Parameters
+ * algorithm: `MD5`
+
+* Input values:
+ 1. `[apple]`
+
+* Returns: `[1f3870be274f6c49b3e31a0c6728957f]`
+
+
+---
+**The algorithm parameter selects the hash function (MD5), multiple values:**
+
+* Parameters
+ * algorithm: `MD5`
+
+* Input values:
+ 1. `[apple, banana]`
+
+* Returns: `[1f3870be274f6c49b3e31a0c6728957f, 72b302bf297a228a75730123efef7c41]`
+
+
+---
+**The algorithm parameter selects the hash function (SHA-1):**
+
+* Parameters
+ * algorithm: `SHA-1`
+
+* Input values:
+ 1. `[apple]`
+
+* Returns: `[d0be2dc421be4fcd0172e5afceea3970e2f3d940]`
+
+
+---
+**The algorithm parameter selects the hash function (SHA-384):**
+
+* Parameters
+ * algorithm: `SHA-384`
+
+* Input values:
+ 1. `[apple]`
+
+* Returns: `[3d8786fcb588c93348756c6429717dc6c374a14f7029362281a3b21dc10250ddf0d0578052749822eb08bc0dc1e68b0f]`
+
+
+---
+**The algorithm parameter selects the hash function (SHA-512):**
+
+* Parameters
+ * algorithm: `SHA-512`
+
+* Input values:
+ 1. `[apple]`
+
+* Returns: `[844d8779103b94c18f4aa4cc0c3b4474058580a991fba85d3ca698a0bc9e52c5940feb7a65a3a290e17e6b23ee943ecc4f73e7490327245b4fe5d5efb590feb2]`
+
+
+---
+**Empty input produces empty output:**
+
+* Input values:
+ 1. `[]`
+
+* Returns: `[]`
+
+
+---
+**Two input ports causes IllegalArgumentException:**
+
+* Input values:
+ 1. `[foo]`
+ 2. `[bar]`
+
+* Returns: `[]`
+* **Throws error:** `IllegalArgumentException`
+
+
+---
+**Invalid algorithm name causes NoSuchAlgorithmException:**
+
+* Parameters
+ * algorithm: `NONEXISTENT`
+
+* Input values:
+ 1. `[foo]`
+
+* Returns: `[]`
+* **Throws error:** `NoSuchAlgorithmException`
+
+
+---
+**Empty algorithm string causes IllegalArgumentException:**
+
+* Parameters
+ * algorithm: ``
+
+* Input values:
+ 1. `[foo]`
+
+* Returns: `[]`
+* **Throws error:** `IllegalArgumentException`
+
+
+
+
+## Parameter
+
+### Algorithm
+
+The hash algorithm to be used.
+
+* ID: `algorithm`
+* Datatype: `string`
+* Default Value: `SHA256`
+
+## Advanced Parameter
+
+`None`
+
+## Related Plugins
+
+* [inputHash](inputHash.md) — The Combined input hash plugin produces one combined hash for all input values. The Per-value hash plugin instead hashes each value independently, preserving cardinality.
diff --git a/docs/build/reference/transformer/Variables/.pages b/docs/build/reference/transformer/Variables/.pages
new file mode 100644
index 000000000..f9b8f14f4
--- /dev/null
+++ b/docs/build/reference/transformer/Variables/.pages
@@ -0,0 +1,2 @@
+nav:
+ - "Set execution variable": setExecutionVariable.md
\ No newline at end of file
diff --git a/docs/build/reference/transformer/Variables/setExecutionVariable.md b/docs/build/reference/transformer/Variables/setExecutionVariable.md
new file mode 100644
index 000000000..55d0943e8
--- /dev/null
+++ b/docs/build/reference/transformer/Variables/setExecutionVariable.md
@@ -0,0 +1,30 @@
+---
+title: "Set execution variable"
+description: "Sets an execution variable to the first value of the (single) input and passes the input values through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution."
+icon: octicons/cross-reference-24
+tags:
+ - TransformOperator
+---
+
+# Set execution variable
+
+
+
+
+
+Sets an execution variable to the first value of the (single) input and passes the input values through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution.
+
+
+## Parameter
+
+### Variable name
+
+Name of the execution variable to set. It is written to the 'execution' scope and addressed downstream as 'execution.'.
+
+- ID: `variableName`
+- Datatype: `string`
+- Default Value: `myVariable`
+
+## Advanced Parameter
+
+`None`
diff --git a/docs/build/reference/transformer/index.md b/docs/build/reference/transformer/index.md
index aec4bae2c..70301e880 100644
--- a/docs/build/reference/transformer/index.md
+++ b/docs/build/reference/transformer/index.md
@@ -39,6 +39,7 @@ Transform operators transform a one or more sequences of string values to a sequ
| [Coalesce (first non-empty input)](Selection/coalesce.md) | Selection | Forwards the first non-empty input, i.e. for which any value(s) exist. A single empty string is considered a value. |
| [Code](Excel/Excel_CODE.md) | Excel | Excel CODE(text): Returns a numeric code for the first character in a text string. Text is the text for which the code of the first character is to be found. |
| [Combin](Excel/Excel_COMBIN.md) | Excel | Excel COMBIN(count_1; count_2): Returns the number of combinations for a given number of objects. Count_1 is the total number of elements. Count_2 is the selected count from the elements. This is the same as the nCr function on a calculator. |
+ | [Combined input hash](Value/inputHash.md) | Value | Calculates a single hash value covering all input values combined, across all input ports. Values are fed into the hash function in port order without any separator between them. |
| [Compare dates](Date/compareDates.md) | Date | Compares two dates. |
| [Compare numbers](Numeric/compareNumbers.md) | Numeric | Compares the numbers of two sets. Returns 1 if the comparison yields true and 0 otherwise. If there are multiple numbers in both sets, the comparator must be true for all numbers. For instance, {1,2} < {2,3} yields 0 as not all numbers in the first set are smaller than in the second. |
| [Concatenate](Combine/concat.md) | Combine | Concatenates strings from multiple inputs. |
@@ -70,6 +71,8 @@ Transform operators transform a one or more sequences of string values to a sequ
| [Duration in years](Date/durationInYears.md) | Date | Converts an xsd:duration to years. |
| [Empty value](Value/emptyValue.md) | Value | Generates an empty value. |
| [Encode URL](Normalize/urlEncode.md) | Normalize | URL encodes the string. |
+ | [Escape SPARQL multiline literal](SPARQL/escape_multiline_literal.md) | SPARQL | Escapes a value so it can be safely used inside a SPARQL triple-quoted string literal (`"""..."""` or `'''...'''`). Escapes backslashes and breaks any run of three or more consecutive single or double quotes. Individual quotes and newlines are preserved. The returned value does not include enclosing quotation marks. |
+ | [Escape SPARQL plain literal](SPARQL/escape_literal.md) | SPARQL | Escapes a value so it can be safely used inside a SPARQL short-form string literal. Escapes backslashes, quotes, newlines, carriage returns and tabs. The returned value does not include enclosing quotation marks. |
| [Evaluate template](Template/TemplateTransformer.md) | Template | Evaluates a template. Input values can be addressed using the variables 'input1', 'input2', etc. Global variables are available in the 'global' scope, e.g., 'global.myVar'. |
| [Even](Excel/Excel_EVEN.md) | Excel | Excel EVEN(number): Rounds the given number up to the nearest even integer. |
| [Exact](Excel/Excel_EXACT.md) | Excel | Excel EXACT(text_1; text_2): Compares two text strings and returns TRUE if they are identical. This function is case- sensitive. Text_1 is the first text to compare. Text_2 is the second text to compare. |
@@ -95,7 +98,6 @@ Transform operators transform a one or more sequences of string values to a sequ
| [If exists](Conditional/ifExists.md) | Conditional | Accepts two or three inputs. If the first input provides a value, the second input is forwarded. Otherwise, the third input is forwarded (if present). |
| [If matches regex](Conditional/ifMatchesRegex.md) | Conditional | This transformer uses a regular expression as a matching condition, in order to distinguish which input to take. |
| [Input file attributes](Metadata/inputFileAttributes.md) | Metadata | Retrieves a metadata attribute from the input file (such as the file name). |
- | [Input hash](Value/inputHash.md) | Value | Calculates the hash sum of the input values. Generates a single hash sum for all input values combined. |
| [Input task attributes](Metadata/inputTaskAttributes.md) | Metadata | Retrieves individual attributes from the input task (such as the modified date) or the entire task as JSON. |
| [Int](Excel/Excel_INT.md) | Excel | Excel INT(number): Rounds the given number down to the nearest integer. |
| [Intercept](Excel/Excel_INTERCEPT.md) | Excel | Excel INTERCEPT(data_Y; data_X): Calculates the y-value at which a line will intersect the y-axis by using known x-values and y-values. Data_Y is the dependent set of observations or data. Data_X is the independent set of observations or data. Names, arrays or references containing numbers must be used here. Numbers can also be entered directly. |
@@ -148,6 +150,7 @@ Transform operators transform a one or more sequences of string values to a sequ
| [Parse SKOS term](Parser/SkosTypeParser.md) | Parser | Parses values from a SKOS ontology. |
| [Parse string](Parser/StringParser.md) | Parser | Parses string values. This is basically an identity function. |
| [Pearson](Excel/Excel_PEARSON.md) | Excel | Excel PEARSON(data_1; data_2): Returns the Pearson product moment correlation coefficient r. Data_1 is the array of the first data set. Data_2 is the array of the second data set. |
+ | [Per-value hash](Value/perValueHash.md) | Value | Hashes each input value independently and returns one hash per value. Accepts exactly one input port. |
| [Percentile](Excel/Excel_PERCENTILE.md) | Excel | Excel PERCENTILE(data; alpha): Returns the alpha-percentile of data values in an array. Data is the array of data. Alpha is the percentage of the scale between 0 and 1. |
| [Percentrank](Excel/Excel_PERCENTRANK.md) | Excel | Excel PERCENTRANK(data; value): Returns the percentage rank (percentile) of the given value in a sample. Data is the array of data in the sample. |
| [Pi](Excel/Excel_PI.md) | Excel | Excel PI(): Returns the value of PI to fourteen decimal places. |
@@ -161,6 +164,7 @@ Transform operators transform a one or more sequences of string values to a sequ
| [Radians](Excel/Excel_RADIANS.md) | Excel | Excel RADIANS(number): Converts the given number in degrees to radians. |
| [Rand](Excel/Excel_RAND.md) | Excel | Excel RAND(): Returns a random number between 0 and 1. |
| [Random number](Value/randomNumber.md) | Value | Generates a set of random numbers. |
+ | [Random value](Uncategorized/cmem_plugin_random-GenerateValues.md) | Uncategorized | Generates random values. |
| [Rank](Excel/Excel_RANK.md) | Excel | Excel RANK(value; data; type): Returns the rank of the given Value in a sample. Data is the array or range of data in the sample. Type (optional) is the sequence order, either ascending (0) or descending (1). |
| [Rate](Excel/Excel_RATE.md) | Excel | Excel RATE(NPER; PMT; PV; FV; type; guess): Returns the constant interest rate per period of an annuity. NPER is the total number of periods, during which payments are made (payment period). PMT is the constant payment (annuity) paid during each period. PV is the cash value in the sequence of payments. FV (optional) is the future value, which is reached at the end of the periodic payments. Type (optional) defines whether the payment is due at the beginning (1) or the end (0) of a period. Guess (optional) determines the estimated value of the interest with iterative calculation. |
| [Read parameter](Value/readParameter.md) | Value | Reads a parameter from a Java Properties file. |
@@ -189,6 +193,7 @@ Transform operators transform a one or more sequences of string values to a sequ
| [Roundup](Excel/Excel_ROUNDUP.md) | Excel | Excel ROUNDUP(number; count): Rounds the given number up. Count (optional) is the number of digits to which rounding up is to be done. If the count parameter is negative, only the whole number portion is rounded. It is rounded to the place indicated by the count. |
| [Search](Excel/Excel_SEARCH.md) | Excel | Excel SEARCH(find_text; text; position): Returns the position of a text segment within a character string. The start of the search can be set as an option. The search text can be a number or any sequence of characters. The search is not case-sensitive. The search supports regular expressions. Find_text is the text to be searched for. Text is the text where the search will take place. Position (optional) is the position in the text where the search is to start. |
| [Sequence values to indexes](Sequence/toSequenceIndex.md) | Sequence | Transforms the sequence of values to their respective indexes in the sequence. If there is more than one input, the values are numbered from the first input on and continued for the next inputs. Applied against an RDF source the order might not be deterministic. |
+ | [Set execution variable](Variables/setExecutionVariable.md) | Variables | Sets an execution variable to the first value of the (single) input and passes the input values through unchanged. The variable is written to the 'execution' scope and can be read downstream as 'execution.'. Only works while running inside a workflow execution. |
| [Sign](Excel/Excel_SIGN.md) | Excel | Excel SIGN(number): Returns the sign of the given number. The function returns the result 1 for a positive sign, -1 for a negative sign, and 0 for zero. |
| [Sin](Excel/Excel_SIN.md) | Excel | Excel SIN(number): Returns the sine of the given number (angle in radians). |
| [Sinh](Excel/Excel_SINH.md) | Excel | Excel SINH(number): Returns the hyperbolic sine of the given number (angle in radians). |
@@ -243,6 +248,7 @@ Transform operators transform a one or more sequences of string values to a sequ
| [Validate number of values](Validation/validateNumberOfValues.md) | Validation | Validates that the number of values lies in a specified range. |
| [Validate numeric range](Validation/validateNumericRange.md) | Validation | Validates if a number is within a specified range. |
| [Validate regex](Validation/validateRegex.md) | Validation | Validates if all values match a regular expression. |
+ | [Validate URI](Validation/validate_uri.md) | Validation | Validates that the input is a valid absolute IRI and returns it unchanged. Throws a validation error if the input is not a valid IRI. |
| [Var](Excel/Excel_VAR.md) | Excel | Excel VAR(number_1; number_2; ... number_30): Estimates the variance based on a sample. Number_1; number_2; ... number_30 are numerical values or ranges representing a sample based on an entire population. |
| [Vara](Excel/Excel_VARA.md) | Excel | Excel VARA(value_1; value_2; ... value_30): Estimates a variance based on a sample. The value of text is 0. Value_1; value_2; ... value_30 are values or ranges representing a sample derived from an entire population. Text has the value 0. |
| [Varp](Excel/Excel_VARP.md) | Excel | Excel VARP(Number_1; number_2; ... number_30): Calculates a variance based on the entire population. Number_1; number_2; ... number_30 are numerical values or ranges representing an entire population. |
diff --git a/docs/build/variables/index.md b/docs/build/variables/index.md
index c5ffcc4ee..a55d06ce0 100644
--- a/docs/build/variables/index.md
+++ b/docs/build/variables/index.md
@@ -31,14 +31,10 @@ The following scopes are available:
Project variables can only be used in the same project.
If a project is exported those will be exported as well.
-`Task variables (User-defined)` (`task.`)
+`Execution variables (User-defined)` (`execution.`)
-: They are defined by the user on an individual task and can only be used within that same task.
-Task variables can reference global and project variables in their templates.
-
-`Execution variables` (`execution.`)
-
-: They are not defined statically but provided for a single workflow run, either when the workflow execution is triggered or while the workflow is running.
+: They are defined by the user on an individual task or workflow as defaults and are only available while that task's execution is running.
+Each run can override them, and they can be changed while the run is in progress.
Build variables can be particularly useful in scenarios where multiple tasks or components within a system need access to the same data or configuration values.
Instead of repeating the same information in multiple places, project variables provide a centralized and reusable way to store and retrieve these values.
@@ -144,58 +140,46 @@ Type name as `email_ids`, in values we have updated all the email id’s of the
{ class="bordered" }
-## Task Variables
-
-While project variables are shared across all tasks of a project, task variables are defined on a single task and are only available within that same task.
-They are useful for values that are specific to one task and should not leak into the rest of the project.
-
-Task variables are managed in the same way as project variables, but from the configuration view of an individual task.
-Open a task (for example a dataset, transformation or workflow) and locate the **Task variables** widget.
-Click on :eccenca-item-add-artefact: to add a variable and provide a name, value and description in the same dialog used for project variables.
+## Execution Variables
-!!! note
+While project variables are shared across all tasks of a project, execution variables belong to a single task or workflow and exist only during its execution.
+They are referenced with the `execution.` prefix, for example `{{execution.myVariable}}`.
- The naming rules for task variables are the same as for project variables (letters, digits and underscores, not starting with a digit).
+Tasks (including workflows) have an **Execution variables** widget in its configuration view, managed in the same way as project variables:
+click on :eccenca-item-add-artefact: to add a variable and provide a name, value and description in the same dialog used for project variables.
+When an execution is started, these variables provide the default values of the execution scope.
+For a workflow run, the defaults come from the **workflow itself** — the execution variables of the operators and datasets inside the workflow are not used during a workflow run; they apply when such a task is executed directly.
-Task variables are referenced with the `task.` prefix, for example `{{task.myVariable}}`.
-In their templates they may themselves reference global and project variables, so a task variable can be composed from project-wide values.
+`{{execution.}}` resolves only from the execution scope — there is no fallback to other scopes.
+If `` has not been defined as a default, provided or set for the run, the reference cannot be resolved and the execution fails with an error.
+To base a default on a project or global variable, give the execution variable a *template* (e.g. `{{project.baseUrl}}/api`); it is resolved when the variable is saved, and the resulting value is used for each run.
!!! note
- Task variables are stored together with the task.
- When the task or its project is exported, the task variables are exported as well.
- They are not visible to or usable by other tasks.
+ Execution variables are resolved in templates that are evaluated **during execution**, for example the template of the template operator.
-## Execution Variables
-
-Execution variables are not defined statically in advance.
-Instead, they are provided for a single workflow run and are available to all tasks of that workflow during the run.
-They are referenced with the `execution.` prefix, for example `{{execution.myVariable}}`.
-
-!!! note "Execution scope fallback"
-
- When a template references `{{execution.}}` but `` has not been set directly in the execution scope, the value falls back to the variable of the same name in the `task`, then `project`, then `global` scope .
- A value that is set directly in the execution scope (provided when starting the workflow, or written during workflow execution) always takes precedence and suppresses the fallback.
- If the name is not defined in any of the execution, task, project or global scopes, the reference remains unbound and template evaluation fails.
+!!! note
- This makes execution variables convenient as overridable defaults: a workflow can reference `{{execution.}}` throughout, and unless a particular run overrides it, the value is taken from the task, project or global variable of the same name.
+ The execution variables of a task are stored together with the task (their default values, not any run-specific overrides).
+ When the task or its project is exported, they are exported as well.
+ The values of a running execution are never persisted and are not shared between runs.
-There are two ways to supply execution variables:
+Besides the defaults defined in the widget, there are two further ways to supply execution variables for a run:
### Passing execution variables when starting a workflow
-When a workflow execution is triggered via the REST API, execution variables can be provided in the JSON request body under the `workflowVariables` key as a simple name/value map.
+When a workflow execution is triggered via the REST API, execution variables can be provided in the JSON request body under the `executionVariables` key as a simple name/value map.
For example, executing a workflow with a single execution variable `testVar`:
```json
{
- "workflowVariables": {
+ "executionVariables": {
"testVar": "World"
}
}
```
-Each entry is added to the `execution` scope and can be referenced anywhere in the workflow as `{{execution.}}`.
+Each entry is set in the `execution` scope — overriding a default of the same name defined on the workflow — and can be referenced anywhere in the workflow as `{{execution.}}`.
For instance, an operator configured with the template `{{value}} {{execution.testVar}}` would resolve `execution.testVar` to `World` for that run.
### Setting execution variables during a workflow run
@@ -209,7 +193,6 @@ Two operators in the *Variables* category support this:
!!! note
Both operators only have an effect while running inside a workflow execution.
- Execution variables are scoped to a single workflow run; they are not persisted and are not shared between runs.
## Using Variables
diff --git a/docs/deploy-and-configure/configuration/label-resolution-and-full-text-search/index.md b/docs/deploy-and-configure/configuration/label-resolution-and-full-text-search/index.md
index 269761f39..439cadc5a 100644
--- a/docs/deploy-and-configure/configuration/label-resolution-and-full-text-search/index.md
+++ b/docs/deploy-and-configure/configuration/label-resolution-and-full-text-search/index.md
@@ -10,8 +10,8 @@ This resolution and, by extension, the full text search is configurable for diff
eccenca Explore backend (DataPlatform) offers three configuration options:
- `labelProperties` (line 2)
-- `languagePreferences` (line 5) and
-- `languagePreferencesAnyLangFallback` (line 8).
+- `languagePreferences` (line 6) and
+- `languagePreferencesAnyLangFallback` (line 10).
``` yaml linenums="1"
proxy:
@@ -28,42 +28,50 @@ These properties define not only which properties and languages should be consid
The retrieval process can be simplified to the following procedure:
-- When determining the label for a resource, the **property** is the primary criterion and the **language** the secondary one. In other words, the complete list of preferred languages is evaluated for the first property before Explore backend (DataPlatform) moves on to the next property.
-- Consequently, for a resource with the default settings above, the candidates are tried in this order:
- 1. An english value for `rdfs:label` is searched.
- 2. A literal of the property `rdfs:label` without a language tag is searched (which is why there is an entry `""`).
- 3. An english value of `skos:prefLabel` is searched.
- 4. A literal of the property `skos:prefLabel` without a language tag is searched.
- 5. If nothing is found and `languagePreferencesAnyLangFallback` is `true`, a value in any remaining language is used, again honoring the property precedence (see [Example](#example)).
- 6. If still nothing is found, Explore backend (DataPlatform) tries to create a prefixed URI, otherwise the last segment of the resource identifier is used.
+- When `languagePreferencesAnyLangFallback` is `true`, the **property order takes precedence over the language order**. For each property, Explore backend (DataPlatform) first tries the configured languages in their listed order and then any other language. Only if that property has no value does it continue with the next property.
+- Consequently, for a resource with the settings above, the candidates are tried in this order:
+ 1. An English, German, or untagged value for `rdfs:label` is searched, in that order.
+ 2. If none exists, an `rdfs:label` in any other language is used.
+ 3. The same language lookup is repeated for `skos:prefLabel` and then for `skos:notation`.
+ 4. If no configured property has a value, Explore backend (DataPlatform) tries to create a prefixed URI; otherwise, the last segment of the resource identifier is used.
Additionally, in case more than one label could be retrieved for the same property and language, for example by conflicting values, the alphabetically first entry is used.
-!!! note "Property precedence beats language precedence"
+!!! note "Any-language fallback preserves property precedence"
- Because the property is the primary criterion, a value of an earlier property is preferred over a better-matching language on a later property. This especially affects untagged literals: an entry `""` in `languagePreferences` matches a literal without a language tag, so if an earlier property carries such an untagged literal, it wins over a later property that has a value in a preferred language.
+ With `languagePreferencesAnyLangFallback: true`, a value in any language on an earlier property is preferred over a value in a configured language on a later property.
Consider the following configuration and resource:
``` yaml
proxy:
labelProperties:
- - "http://www.w3.org/2004/02/skos/core#notation"
- "http://www.w3.org/2000/01/rdf-schema#label"
- "http://www.w3.org/2004/02/skos/core#prefLabel"
+ - "http://www.w3.org/2004/02/skos/core#notation"
languagePreferences:
- "en"
- "de"
- ""
+ languagePreferencesAnyLangFallback: true
```
``` turtle
- :labelEn a owl:Class ;
- rdfs:label "label en"@en, "label de"@de ;
- skos:notation "notation" .
+ PREFIX dcterms:
+ PREFIX owl:
+ PREFIX rdf:
+ PREFIX rdfs:
+ PREFIX skos:
+ PREFIX xsd:
+
+
+ rdf:type owl:Class ;
+ rdfs:label "label es"@es ;
+ dcterms:modified "2026-03-10"^^xsd:date ;
+ skos:notation "notation" .
```
- The resolved label is `notation`: `skos:notation` is the first property, and although it has no `en` or `de` value, its untagged literal is matched by the `""` language preference, so the search never reaches `rdfs:label`. To make `rdfs:label` win here, either list it before `skos:notation` in `labelProperties`, or remove the `""` entry from `languagePreferences` (which lets `skos:notation` fall through to `rdfs:label "label en"@en`).
+ The resolved label is `label es`. Although Spanish is not listed in `languagePreferences`, `rdfs:label` is the first configured property. Its Spanish value is therefore selected by the any-language fallback before label resolution considers the untagged `skos:notation` value.
## Example
@@ -72,9 +80,9 @@ How labels are resolved is best explained using these default settings and some
``` turtle
:Resource1 rdfs:label "Leipzig"@en.
:Resource2 :someOtherProperty "Berlin"@en.
-:Resource3 rdfs:label "Stuttgart"@de
-:Resource4 rdfs:label "Hanover"@en
-:Resource4 rdfs:label "Another Label for Hanover"@en
+:Resource3 rdfs:label "Stuttgart"@fr.
+:Resource4 rdfs:label "Hanover"@en.
+:Resource4 rdfs:label "Another Label for Hanover"@en.
```
- For `:Resource1` the label will be `Leipzig` as the english `rdfs:label` will be retrieved.
diff --git a/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md b/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md
index ef58a094e..6aa008a85 100644
--- a/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md
+++ b/docs/deploy-and-configure/installation/scenario-k8s-deployment/index.md
@@ -17,7 +17,8 @@ provisioned cluster.
## Requirements
- Access credentials for the eccenca infrastructure (e.g. Docker Registry) → [contact us to get yours](https://eccenca.com/en/contact)
-- A GraphDB license ([free](https://www.ontotext.com/products/graphdb/) or commercial)
+- Triple store license (provided by eccenca)
+- Graph Insights license (optional, provided by eccenca)
- [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
- [Helm](https://helm.sh/docs/intro/install/)
- If deploying on K3D, download a [static binary](https://github.com/k3d-io/k3d/releases)
diff --git a/docs/deploy-and-configure/installation/scenario-local-installation/index.md b/docs/deploy-and-configure/installation/scenario-local-installation/index.md
index bd019b75f..e8e8cbf78 100644
--- a/docs/deploy-and-configure/installation/scenario-local-installation/index.md
+++ b/docs/deploy-and-configure/installation/scenario-local-installation/index.md
@@ -15,8 +15,8 @@ The code examples in this section assume that you have a POSIX-compliant shell (
- [docker](https://www.docker.com/) and [docker compose](https://docs.docker.com/compose/install/) (v2) installed locally
- [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) installed locally
- [jq](https://jqlang.github.io/jq/download/) installed locally
-- A GraphDB license ([free](https://www.ontotext.com/products/graphdb/) or commercial)
-- (optional) A Graph Insights license
+- Triple store license (provided by eccenca)
+- Graph Insights license (optional, provided by eccenca)
- make - build tool (apt-get install make) installed locally (don't use version 4.4.1 [→](https://savannah.gnu.org/bugs/?63650); e.g., 4.3 works well)
- At least 4 CPUs and 12GB of RAM (recommended: 16GB) dedicated to docker
@@ -26,7 +26,7 @@ Install the requirements
=== "Linux"
- Install all the needed packages:
+ Install all the needed packages:
```shell
sudo apt-get install -y curl gnupg2 \
diff --git a/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md b/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md
index f35d7de3e..c257fd950 100644
--- a/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md
+++ b/docs/deploy-and-configure/installation/scenario-single-node-cloud-installation/index.md
@@ -13,7 +13,8 @@ This page describes a docker-compose based orchestration running on a server ins
- A resolvable domain name to this server
- Terminal with ssh client installed locally
- An eccenca partner account for the docker registry as well as the release artifact area
-- A GraphDB license ([free](https://www.ontotext.com/products/graphdb/) or commercial)
+- Triple store license (provided by eccenca)
+- Graph Insights license (optional, provided by eccenca)
!!! Info
make - do not use version 4.4.1 [→](https://savannah.gnu.org/bugs/?63650)
diff --git a/docs/deploy-and-configure/system-architecture/Corporate-Memory-System-Architecture_26.2.drawio.png b/docs/deploy-and-configure/system-architecture/Corporate-Memory-System-Architecture_26.2.drawio.png
index 777ce2aa9..e16de4847 100644
Binary files a/docs/deploy-and-configure/system-architecture/Corporate-Memory-System-Architecture_26.2.drawio.png and b/docs/deploy-and-configure/system-architecture/Corporate-Memory-System-Architecture_26.2.drawio.png differ
diff --git a/docs/deploy-and-configure/system-architecture/index.md b/docs/deploy-and-configure/system-architecture/index.md
index a44006f63..715951e80 100644
--- a/docs/deploy-and-configure/system-architecture/index.md
+++ b/docs/deploy-and-configure/system-architecture/index.md
@@ -11,7 +11,7 @@ hide:
This page describes the overall system architecture of eccenca Corporate Memory and its components.
-
+
eccenca Corporate Memory consists of three core components:
diff --git a/docs/develop/.pages b/docs/develop/.pages
index a11a2dfb9..1d1dbc14b 100644
--- a/docs/develop/.pages
+++ b/docs/develop/.pages
@@ -2,6 +2,7 @@ nav:
- Develop: index.md
- Accessing Graphs with Java Applications: accessing-graphs-with-java-applications
- Python Plugins: python-plugins
+ - Marketplace Packages: packages
- cmempy - Python API: cmempy-python-api
- cmemc - Python Scripts: cmemc-scripts
- Build (DataIntegration) APIs: dataintegration-apis
diff --git a/docs/develop/index.md b/docs/develop/index.md
index c31d88f00..20505e37f 100644
--- a/docs/develop/index.md
+++ b/docs/develop/index.md
@@ -21,6 +21,12 @@ API documentation and programming recipes.
For Python developers, we offer a [Plugin SDK](python-plugins/index.md) as well as an API for accessing and manipulating Corporate Memory Instances ([cmem-cmempy](cmempy-python-api/index.md)).
+- :material-shopping: Marketplace Packages
+
+ ---
+
+ [Marketplace Packages](packages/index.md) bundle existing content (graphs, Build projects, dependencies, ...) into a single shareable artifact. [Create your own packages](packages/development/index.md) for easy distribution and reuse.
+
- :material-api: OpenAPI specification
---
diff --git a/docs/develop/packages/.pages b/docs/develop/packages/.pages
new file mode 100644
index 000000000..1f4c61800
--- /dev/null
+++ b/docs/develop/packages/.pages
@@ -0,0 +1,4 @@
+nav:
+ - Marketplace Packages: index.md
+ - Installation and Management: installation
+ - Development and Publication: development
diff --git a/docs/develop/packages/development/.pages b/docs/develop/packages/development/.pages
new file mode 100644
index 000000000..48c0722db
--- /dev/null
+++ b/docs/develop/packages/development/.pages
@@ -0,0 +1,3 @@
+nav:
+ - Development and Publication: index.md
+ - Tutorial: tutorial
diff --git a/docs/develop/packages/development/index.md b/docs/develop/packages/development/index.md
new file mode 100644
index 000000000..ebfd40356
--- /dev/null
+++ b/docs/develop/packages/development/index.md
@@ -0,0 +1,271 @@
+---
+title: "Marketplace Packages: Development and Publication"
+icon: material/code-json
+tags:
+ - Marketplace
+ - Package
+---
+# Development and Publication of Marketplace Packages
+
+## Introduction
+
+Marketplace Packages are archives that bundle content, functionality, and configuration from Corporate Memory for sharing and reuse.
+
+Each package has its own release cycle.
+Packages can be installed and uninstalled during runtime.
+
+In order to support the development and publication of Marketplace Packages, we published a [package-template](https://github.com/eccenca/cmem-package-template).
+Please have a look at this template to get started.
+
+This page gives an overview of the concepts you need to understand in order to develop packages.
+If you prefer to learn by doing, follow the [step-by-step tutorial](tutorial/index.md), which builds a package with a graph and a Build project from scratch.
+
+## Package Structure
+
+Use the [package-template](https://github.com/eccenca/cmem-package-template) to create the boilerplate for a package repository:
+
+```shell title="Create a package repository from the template"
+copier copy gh:eccenca/cmem-package-template my-package
+```
+
+The template asks for the following variables:
+
+`package_type`
+: `vocabulary` (default) or `project`, see [Metadata](#metadata).
+
+`package_id`
+: Unique package identifier in lowercase letters, numbers, and hyphens (e.g. `eccenca-supply-chain-vocab`).
+
+`package_name`
+: Human-readable package name (3 - 50 characters).
+
+`package_description`
+: Short description of the package (10 - 150 characters).
+
+`python_dependencies`
+: Comma-separated [Python plugin](../../python-plugins/index.md) dependencies (only asked for `project` packages).
+
+`vocab_dependencies`
+: Comma-separated dependencies on other Marketplace Packages (only asked for `project` packages).
+
+`github_page`
+: Optional URL of the package repository, used as the base for icons and the homepage link.
+
+The generated repository has two levels:
+the top level holds the generic package repository files (changelog, README, license, CI configuration, and a `Taskfile.yaml`), while the nested `{package_id}/` folder is the **package directory** - the actual package content plus its manifest.
+
+### License
+
+!!! info "No publication without license"
+
+ Packages without a license declaration cannot be published to a Corporate Memory Marketplace Server.
+
+Our template will bootstrap your package with an _Apache License 2.0 ([`Apache-2.0`](https://spdx.org/licenses/Apache-2.0.html))_.
+See if you need a different license.
+You can remove a license entirely; however, a package that does not declare a license cannot be published.
+
+### Manifest
+
+The `cpa-manifest.json` in the package directory is the central package definition.
+It contains all relevant package metadata and describes the package contents.
+It is used to present package details and contents to the `inspect` command, to install, configure and uninstall all parts of a package.
+
+#### Metadata
+
+`package_type`
+: `project`
+ : A package that may ship any content, mainly intended to contain Build projects, (instance/data) graphs, SHACL shapes, workspace configuration, query catalogs, etc.
+
+ `vocabulary`
+ : A package that is supposed to contribute vocabulary / ontology contents, such as `rdf:`, `org:`, `sso:`, etc. Such a package may contain multiple vocabularies / ontologies. Packaging related SHACL shapes is reasonable, too.
+
+`package_id`
+: Unique package identifier
+
+`package_version`
+: Semantic version identifier string of the package, but limited to proper releases.
+
+`metadata.name`
+: The package name in English.
+
+`metadata.description`
+: The package description in English.
+
+`metadata.license`
+: The [SPDX license identifier](https://spdx.org/licenses/) of the package, e.g. `Apache-2.0`.
+
+`metadata.comment`
+: A maintainer or publisher comment.
+
+`metadata.agents`
+: Publishers, authors, and contributors of the package.
+
+`metadata.urls`
+: Related links, e.g. the homepage or the issue tracker of the package.
+
+`metadata.tags`
+: Free-text tags used to categorize the package on a Marketplace Server.
+
+#### Files
+
+A package can contain graphs, Build projects, text files, and images.
+These contents are referenced in the `files` section of the `cpa-manifest.json`.
+
+##### Graphs
+
+Use the following structure to include a graph.
+`register_as_vocabulary` and `import_into` are optional instructions.
+We suggest to organize graphs in a respective sub-folder (here `graphs/`), but this is up to you:
+
+```json
+"files": [
+ …
+ {
+ "file_type": "graph",
+ "file_path": "graphs/file.ttl",
+ "graph_iri": "http://www.example.org/file/",
+ "register_as_vocabulary": true,
+ "import_into": [
+ "http://www.example.org/integration_graph/"
+ ]
+ },
+ …
+]
+```
+
+##### Projects
+
+Use the following structure to include a project.
+We suggest to organize projects in a respective sub-folder (here `projects/`), but this is up to you:
+
+```json
+"files": [
+ …
+ {
+ "file_type": "project",
+ "file_path": "projects/my-build-project.zip",
+ "project_id": "my-build-project"
+ },
+ …
+]
+```
+
+##### Texts and Images
+
+Text files and images describe the package itself rather than shipping content.
+The template declares `README.md`, `CHANGELOG.md`, and `LICENSE` this way; images are used to represent the package on a Marketplace Server:
+
+```json
+"files": [
+ …
+ {
+ "file_type": "text",
+ "file_path": "README.md",
+ "file_role": "readme"
+ },
+ {
+ "file_path": "icon.png",
+ "file_type": "image",
+ "file_role": "icon"
+ },
+ …
+]
+```
+
+#### Dependencies
+
+Dependencies to other packages or to Python plugins can be declared in the `copier copy` answers.
+The dependencies are added to the `cpa-manifest.json` as described in the next sections.
+
+##### Python Plugin Packages
+
+Use the following to declare a dependency to a Python plugin:
+
+```json
+"dependencies": [
+ …
+ {
+ "dependency_type": "python-package",
+ "pypi_id": "cmem-plugin-pyshacl"
+ },
+ …
+]
+```
+
+##### Marketplace Packages
+
+Use the following to declare a dependency to another Marketplace Package:
+
+```json
+"dependencies": [
+ …
+ {
+ "dependency_type": "marketplace-package",
+ "package_id": "w3c-rdfs-vocab"
+ }
+ …
+]
+```
+
+## Package Development Cycle
+
+!!! info "`cmemc package` reference"
+
+ The [cmemc package command group](../../../automate/cmemc-command-line-interface/command-reference/package/index.md)
+ contains all needed commands to support the complete package lifecycle.
+
+Some packages are simply wrapping existing artifacts into a managed structure (e.g. existing vocabulary/ontology).
+
+Most (solution) package development and evolution will be a back and forth between a package repository (making changes to `cpa-manifest.json` in terms of adding/removing dependencies, graph files, or Build project files) and a Corporate Memory (package development) instance.
+
+{ width="50%" }
+
+!!! tip "Task wrappers"
+
+ The generated package repository ships a `Taskfile.yaml` which wraps the commands below into `task import`, `task export`, `task build`, `task check`, `task delete`, and `task publish`.
+ The [tutorial](tutorial/index.md) uses these wrappers.
+
+### Install (local) Packages
+
+Use the following command to install a local package folder content (or built `.cpa` file) to a Corporate Memory (package development) instance.
+
+```shell
+cmemc package install --input PATH
+```
+
+Make changes to graphs, configuration, or Build projects as needed.
+Newly created or imported graphs or Build projects need to be registered in `cpa-manifest.json` so they will be fetched by `export`.
+
+### Export Contents into a Package
+
+Use the following command to export the file artifacts declared in `cpa-manifest.json` from a Corporate Memory (package development) instance to a local package folder.
+
+```shell
+cmemc package export PACKAGE_ID
+```
+
+Run this to initially populate package contents from a solution configuration. You can also use it to update contents after making changes on your Corporate Memory (package development) instance, capturing them for building and releasing as a Marketplace Package.
+
+For version controlled package directories, add `--extract` to store Build projects as extracted directories instead of ZIP archives (the manifest still references the ZIP; `build` and `install` zip it silently).
+
+### Inspect Packages
+
+Review and verify the contents of a package with the following command:
+
+```shell
+cmemc package inspect PACKAGE_PATH
+```
+
+### Build Packages
+
+During development you can install a package from a local path (plain folder or a `.cpa` package) using the `cmemc package install --input PATH` command.
+
+Use the `cmemc package build` command.
+This will build a package archive from a package directory.
+
+This command processes a package directory, validates its content including the manifest, and creates a versioned Corporate Memory package archive (`.cpa`) with the following naming convention: `{package_id}-v{version}.cpa`.
+
+### Publish Packages
+
+Package archives can be published to the Marketplace Server using the `cmemc package publish` command.
+After being published packages can be found and installed directly from the Marketplace Server (potential users do not need to have the local package folder or `.cpa` file available).
diff --git a/docs/develop/packages/development/tutorial/index.md b/docs/develop/packages/development/tutorial/index.md
new file mode 100644
index 000000000..fdf4720b4
--- /dev/null
+++ b/docs/develop/packages/development/tutorial/index.md
@@ -0,0 +1,157 @@
+---
+title: "Marketplace Packages: Development Tutorial"
+icon: material/school
+tags:
+ - Marketplace
+ - Package
+---
+# Tutorial: Develop your first Marketplace Package
+
+This tutorial walks you through a basic example of creating a new Marketplace Package, adding different types of content to it, and finally building it into a package archive ready for distribution.
+
+It is a "how to" and does not replace the full documentation of the [package-template](https://github.com/eccenca/cmem-package-template) repository.
+See [Development and Publication](../index.md) for the underlying concepts, and note that advanced scenarios such as publishing are only outlined at the end.
+
+## Initialize the Package Repository
+
+Follow the [template usage instructions](https://github.com/eccenca/cmem-package-template/tree/main#usage) to create a local package repository.
+For our example, we answer the template questions as follows:
+
+```shell title="copier copy gh:eccenca/cmem-package-template my-package-id"
+🎤 Type of package
+ Project Package
+🎤 Package ID (e.g., 'eccenca-supply-chain-vocab', 'w3c-org-vocab')
+ my-package-id
+🎤 Human-readable package name (e.g., 'My Awesome Vocabulary', 'My Great Project')
+ My own package
+🎤 Short description of the package (e.g., 'A vocabulary for ...', 'A project that ...')
+ My project and graphs
+🎤 Comma-separated Python package dependencies (e.g., 'cmem-plugin-pyshacl, cmem-plugin-llm')
+
+🎤 Comma-separated vocabulary or project dependencies (e.g., 'aksw-rut-vocab, my-other-project')
+
+🎤 github_page: This URL (e.g. https://github.com/user/repo) will be used as the base for icons and the homepage link. Leave blank if your package is not on github.
+
+```
+
+You should now have a folder with two levels of files:
+
+- Top level - generic package repository information such as the changelog, README, CI instructions, licensing, and the `Taskfile.yaml`.
+- Nested folder (`my-package-id`) - the package directory holding the actual package content, along with the `cpa-manifest.json` manifest.
+
+## Add Package Content
+
+The nested folder `my-package-id` represents your working directory for developing the package.
+
+To add content to the package, simply copy the files you want to add into this folder, or extract existing content from a live Corporate Memory instance into the working directory.
+
+!!! example "Extracting Corporate Memory content to add to the package"
+
+ ```shell
+ cmemc graph export https://my-company.org/queries/ --output-file my-package-id/queries.ttl
+
+ cmemc project export MyProject_78e981443900a761 --output-dir my-package-id
+ Export project 1/1: MyProject_78e981443900a761 to my-package-id/2026-07-08-unnamed-MyProject_78e981443900a761.project.zip ... done
+
+ mv my-package-id/2026-07-08-unnamed-MyProject_78e981443900a761.project.zip my-package-id/project.zip
+ ```
+
+## Declare the Files in the Manifest
+
+In order for the package to know about these added files, the `cpa-manifest.json` needs to be edited.
+
+The `"files": []` section of the manifest references the files the package needs to bundle.
+Complete information about the [package manifest can be found here](https://github.com/eccenca/cmem-package-template/tree/main#package-manifest), and more specifically [how to declare new files](https://github.com/eccenca/cmem-package-template/tree/main#adding-files).
+
+For our example, we add a query graph and a project file.
+Make sure each `file_path` is valid and relative to your package directory (the nested folder):
+
+```json title="my-package-id/cpa-manifest.json"
+"files": [
+ {…},
+ {
+ "file_path": "queries.ttl",
+ "file_type": "graph",
+ "graph_iri": "https://my-company.org/queries/",
+ "import_into": [],
+ "register_as_vocabulary": false
+ },
+ {
+ "file_path": "project.zip",
+ "file_type": "project",
+ "project_id": "MyProject_78e981443900a761"
+ }
+]
+```
+
+## Test your Package
+
+To ensure the package correctly detects your added files, you can try to import it into a Corporate Memory instance.
+
+The package template comes with a predefined `Taskfile.yaml` allowing you to wrap your development steps in single commands:
+
+```shell
+task: Available tasks for this project:
+* build: Build package archive
+* check: Run whole test suite
+* clean: Removes dist, *.cpa, ...
+* delete: Delete (uninstall) package from Corporate Memory
+* export: Export package content from Corporate Memory
+* import: Import (install) package to Corporate Memory
+* publish: Publish package archive to the marketplace
+```
+
+To tell the package system to take files from the local working directory and to import them into Corporate Memory, we use **task import**.
+An import always tries to uninstall a previously installed version of the same package first, to ensure it is correctly replaced.
+
+```shell
+task import
+task: [delete] cmemc package uninstall $package_id
+Package 'my-package-id' is not installed.
+task: [import] cmemc package install --input $package_dir
+Installing package 'my-package-id' from 'my-package-id' ... done
+```
+
+!!! warning "Importing duplicated content"
+
+ If you extracted already existing content from Corporate Memory, added it to your package with the same identifiers (graph IRIs, project IDs, ...), and try to import it back in the form of a new package, the instance might raise a `MarketplacePackagesImportError` due to conflicting elements, e.g. `Repository item 'https://my-company.org/queries/' already exists.`
+
+ In this case, you can simply delete the duplicated content inside Corporate Memory (make sure you have backups) before importing it back as package content.
+ The difference is that Corporate Memory now knows this content is part of a managed package, and will handle import/export of that file from now on.
+
+## Update the Package File Content
+
+If you make modifications to your package content in Corporate Memory, the files will not automatically sync back to your local working directory.
+
+To extract all updated content from Corporate Memory into your package working directory in a managed way, simply run **task export**:
+
+```shell
+task export
+```
+
+!!! warning "Exporting without installing first"
+
+ The platform can only export updated versions of package files that were imported at least once before. If you create new information directly in Corporate Memory that the package manifest does not yet declare, such as new graphs, you need to manually add them to your working directory and to your manifest, and then import them.
+
+ The rule of thumb is: if you need to make a structural change to your package that requires you to edit your manifest, then make sure to run `import` right after, to let Corporate Memory keep track of new files.
+
+ Adding a workflow inside a project is not impacted by this limitation, since it is part of the "project" that is managed and tracked by the package.
+
+## Build your Package
+
+To generate a `.cpa` file ready to be distributed and installed in different Corporate Memory instances, you can run **task build**.
+
+Make sure your local package folder is a git repository with a clean state - the task derives the package version from `git describe`, so the commit hash ends up in the archive name (e.g. `my-package-id-v0.0.0-4b7516f.cpa`).
+
+```shell
+task build
+```
+
+To check how this output `.cpa` file can be installed in different places, refer to the [Installation and Management](../../installation/index.md) section.
+
+!!! success "Next steps"
+
+ There are many improvements you can add to your package, such as declaring dependencies to other plugins or packages, to ensure your `.cpa` file can be installed with all its requirements everywhere. For that, you can refer to existing package examples, the [Development and Publication](../index.md) page, or the template documentation.
+
+ The final step is usually publishing a version of the package to a remote Marketplace Server, to avoid having to transfer the `.cpa` archive manually. This requires you to have publishing permissions on an eccenca Marketplace Server (either a public or private instance).
+ This can be done with **task publish**, either manually or from a CI runner.
diff --git a/docs/develop/packages/index.md b/docs/develop/packages/index.md
new file mode 100644
index 000000000..3e0008158
--- /dev/null
+++ b/docs/develop/packages/index.md
@@ -0,0 +1,54 @@
+---
+status: new
+title: "Marketplace Packages: Overview"
+icon: material/shopping
+tags:
+ - Marketplace
+ - Package
+hide:
+ - toc
+---
+
+# Marketplace Packages
+
+Starting with version 26.1, we support the creation and use of Marketplace Packages.
+
+Marketplace Packages bundle everything for a specific Corporate Memory–based solution or project into a single shareable, managed artifact:
+
+- Vocabularies / Ontologies
+- (SKOS) Taxonomies
+- (Instance / Data) Graphs
+- Build Projects
+- Dependencies on
+ - [python-plugins](../python-plugins/index.md)
+ - (other) Marketplace Packages
+
+This lets you share and reuse them across projects, teams, and different Corporate Memory instances.
+
+A Marketplace Package is distributed as a **C**orporate Memory **P**ackage **A**rchive (`.cpa` file), a zip-based archive which you can either hand over directly or publish to a Marketplace Server - a central repository which supports pushing and pulling packages.
+
+The lifecycle of a Corporate Memory Marketplace Package is shown in the following flowchart.
+
+{ width="50%" }
+
+The following pages give an overview of this feature:
+
+
+
+- :material-download-circle-outline: [Installation and Management](installation/index.md)
+
+ ---
+
+ Intended for Linked Data Experts, Deployment Engineers, and Corporate Memory Admins, this page outlines how to (un)install and manage Marketplace Packages, and where installed contents appear in Corporate Memory.
+
+ This section discusses the lifecycle commands and stages `search`, `install`, `list` and `uninstall`.
+
+- :material-code-json: [Development and Publication](development/index.md)
+
+ ---
+
+ Intended for Developers, Linked Data Experts, Consultants, and Partners, this page gives an overview of how to start developing and publish Marketplace Packages, followed by a [step-by-step tutorial](development/tutorial/index.md).
+
+ This section discusses the lifecycle commands and stages `copier copy`, _Package Definition and Release_, `inspect`, `install --input PATH` (from local), _Solution Development and Configuration_, `export`, `build`, and `publish`.
+
+
diff --git a/docs/develop/packages/installation/example-project.png b/docs/develop/packages/installation/example-project.png
new file mode 100644
index 000000000..753ed3baf
Binary files /dev/null and b/docs/develop/packages/installation/example-project.png differ
diff --git a/docs/develop/packages/installation/example-vocabulary.png b/docs/develop/packages/installation/example-vocabulary.png
new file mode 100644
index 000000000..013961a18
Binary files /dev/null and b/docs/develop/packages/installation/example-vocabulary.png differ
diff --git a/docs/develop/packages/installation/index.md b/docs/develop/packages/installation/index.md
new file mode 100644
index 000000000..6cdff1a82
--- /dev/null
+++ b/docs/develop/packages/installation/index.md
@@ -0,0 +1,101 @@
+---
+title: "Marketplace Packages: Installation and Management"
+icon: material/download-circle-outline
+tags:
+ - Marketplace
+ - Package
+---
+# Installation and Management of Marketplace Packages
+
+## Introduction
+
+Marketplace Packages can be installed directly from a Corporate Memory Marketplace Server (e.g. [https://eccenca.market](https://eccenca.market)), or from local **C**orporate Memory **P**ackage **A**rchives (`.cpa` files) and package directories.
+
+This page describes how to search, install, list, and uninstall Marketplace Packages using `cmemc`.
+
+!!! info "`cmemc package` reference"
+
+ The [cmemc package command group](../../../automate/cmemc-command-line-interface/command-reference/package/index.md)
+ contains all needed commands to support the complete package lifecycle.
+
+## Search Packages
+
+Use the following command to search a Marketplace Server for available packages:
+
+```shell title="Search the Marketplace Server"
+cmemc package search vocab
+```
+
+## Install Packages
+
+Use the following command to install a package from a Marketplace Server:
+
+```shell-session title="Install a package from the Marketplace Server"
+$ cmemc package install w3c-xsd-vocab
+Installing package 'w3c-xsd-vocab' from marketplace ... done
+```
+
+For installing local package archives (`.cpa` files) or package directories, use the `--input` option:
+
+```shell-session title="Install a package from a .cpa file"
+$ cmemc package install --replace --input my-package-v0.0.0-4b7516f.cpa
+Installing package 'my-package' from 'my-package-v0.0.0-4b7516f.cpa'
+done
+```
+
+!!! info "Replacing installed packages"
+
+ Use `--replace` to overwrite an already installed package version or package content.
+ Without this option, installing over existing content fails.
+
+## List Packages
+
+Use the following command to list all installed packages:
+
+```shell title="List installed packages"
+cmemc package list
+```
+
+To review the manifest of a package (installed, local directory, or `.cpa` file), use the `inspect` command:
+
+```shell title="Inspect a package manifest"
+cmemc package inspect my-package-v0.0.0-4b7516f.cpa
+```
+
+## Where Package Contents Appear
+
+Depending on the content types inside it, an installed package appears in different places in Corporate Memory, with each item (graph, project, workflow, ...) surfacing in its respective component.
+
+
+
+!!! info inline ""
+
+ 
+
+**Graphs** such as data graphs but also **Vocabularies** or **Shapes Catalogs** are listed in [**Explore > Graphs**](../../../explore-and-author/graph-exploration/index.md#graphs).
+
+
+
+
+
+!!! info inline ""
+
+ 
+
+**Projects** are imported into [**Build**](../../../build/introduction-to-the-user-interface/index.md#projects).
+When you install your first project package, Corporate Memory also creates a special project to store all installed files.
+This project is automatically managed by the package system, and removed once the last package is uninstalled.
+
+
+
+
+
+## Uninstall Packages
+
+Use the following command to uninstall a package:
+
+```shell title="Uninstall a package"
+cmemc package uninstall PACKAGE_ID
+```
+
+This removes all package contents from the Corporate Memory instance, including graphs and Build projects that were installed as part of the package.
diff --git a/docs/develop/packages/mmd.txt b/docs/develop/packages/mmd.txt
new file mode 100644
index 000000000..fc2ee5b30
--- /dev/null
+++ b/docs/develop/packages/mmd.txt
@@ -0,0 +1,13 @@
+# mermaid
+
+## live editor
+
+https://mermaid.live/edit#pako:eNqFU12L2zAQ_CuLIHVb7DSxL3YiykFJHnpwgaMfL636oNgbR0SWjCIdlwv575WcuA6FUvxg7WhmZ3clnUipKySUJEnClBVWIoWlNq023CKssdHmCGtu9mhbyUuEJ17ueY3wKLZYHkuJTHXa0egklLAUThBJXT_iM8qIQlThxtVRDJHdYYMBUeis4TKCM5xHI6YO1lutBK8Nb5LnlKnwdSAw0tutcBvSC60-bsw9VxV8QYn8gCF8K3XJ5TtGgB-g3dc3creR4rDDqqPd9NGT3WYgj8uW9_0NaeENLB8-LFdXhecMiq9aulCTL-8ZQlFLrbai7sR_T7HXN9gMCR7CSsprff-QCHUI84Ve9F2JXnYhOCXCzH6-_wVJch8mABRK3Qo04XdkKkDDlpe3WFqmQsMBDkVd8JAWQjEeF6p1Fp4-ffs8JPg_s2MMXvjiW7I3CbwlhY0Tshr8_TF4sL0cVmglAGHD-wxuLPR9A3uyHbDQPQ2j6MkkJrURFaHWOIxJg6bhISQnpsCPsbuQjFC_vF5JRpg6e1nL1Q-tm15ptKt3hG65PPjItdVwX_-gBlWFZqmdsoSmWZp1WQg9kRdCs2k-nswWxSy7mxaLfF7MYnIkdDpPx5NJUeR5kWZ5ls7PMXntfCfjYpIWd9lsvkizrJjli5hgJaw268tz7V7t-TfNwDqo
+
+## cli
+
+https://github.com/mermaid-js/mermaid-cli
+
+```sh
+mmdc -i mpp-lifecycle.mmd -o mpp-lifecycle.svg -b transparent
+```
diff --git a/docs/develop/packages/mpp-lifecycle.mmd b/docs/develop/packages/mpp-lifecycle.mmd
new file mode 100644
index 000000000..22960e9d9
--- /dev/null
+++ b/docs/develop/packages/mpp-lifecycle.mmd
@@ -0,0 +1,25 @@
+---
+#title: Corporate Memory Marketplace Package Lifecycle
+---
+%%{init: { 'logLevel': 'debug', 'theme': 'neutral' } }%%
+stateDiagram-v2
+
+
+state "Package Definition and Release (local)" as pkg
+state "Published (Marketplace)" as pub
+state ".cpa Package (local & CI/CD)" as cpa
+state "Solution Dev and Config (Corporate Memory)" as cmem
+state "Installed (Corporate Memory)" as ins
+%% state "Uninstalled" as uni
+
+[*] --> pkg : copier copy
+pkg --> pkg : inspect
+cpa --> cmem : install --input PATH
+pkg --> cmem : install --input PATH
+cmem --> pkg : export
+pkg --> cpa : build
+cpa --> pub : publish
+
+pub --> ins : install
+ins --> ins : list
+ins --> [*] : uninstall
diff --git a/docs/develop/packages/mpp-lifecycle.svg b/docs/develop/packages/mpp-lifecycle.svg
new file mode 100644
index 000000000..a6835805a
--- /dev/null
+++ b/docs/develop/packages/mpp-lifecycle.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/docs/develop/python-plugins/installation/index.md b/docs/develop/python-plugins/installation/index.md
index 7b4a73896..9cfd0f98c 100644
--- a/docs/develop/python-plugins/installation/index.md
+++ b/docs/develop/python-plugins/installation/index.md
@@ -7,7 +7,7 @@ tags:
---
# Installation and Usage of Python Plugins
-Plugins are a released as parts of Python packages.
+Plugins are released as parts of Python packages.
They can but do not need to be open-source and published on [pypi.org](https://pypi.org/search/?q=%22cmem-plugin-%22) (a widely used Python Package Index). One package can contain multiple plugins.
## Installation
diff --git a/docs/explore-and-author/companion/companion-anatomy.jpg b/docs/explore-and-author/companion/companion-anatomy.jpg
deleted file mode 100644
index 0fba7abc7..000000000
Binary files a/docs/explore-and-author/companion/companion-anatomy.jpg and /dev/null differ
diff --git a/docs/explore-and-author/companion/companion-anatomy.png b/docs/explore-and-author/companion/companion-anatomy.png
new file mode 100644
index 000000000..c9d13f8ed
Binary files /dev/null and b/docs/explore-and-author/companion/companion-anatomy.png differ
diff --git a/docs/explore-and-author/companion/index.md b/docs/explore-and-author/companion/index.md
index 424d16f9d..4a472f5b6 100644
--- a/docs/explore-and-author/companion/index.md
+++ b/docs/explore-and-author/companion/index.md
@@ -7,6 +7,12 @@ tags:
---
# Companion
+!!! info "AI Disclaimer"
+
+ Companion is an AI-based feature.
+ AI-generated content may be inaccurate or incomplete.
+ Please verify important information.
+
The Companion view enables you to interact with your data, graphs, vocabularies, resources and queries in a chat-like way.
## Configuration Info
@@ -28,7 +34,7 @@ For optimal results, we recommend using Anthropic or OpenAI frontier-level model
You can open the _Chat with Companion_ from any explore module via the :eccenca-application-ai-suggestion: Companion button in the top right of the application header, next to the :eccenca-application-useraccount: user menu.
-{ class="bordered" width="60%" }
+{ class="bordered" width="60%" }
In the companion sidebar, use:
diff --git a/docs/explore-and-author/graph-exploration/graph-insights/assets/settings.png b/docs/explore-and-author/graph-exploration/graph-insights/assets/settings.png
index a45c86e28..103a79cd4 100644
Binary files a/docs/explore-and-author/graph-exploration/graph-insights/assets/settings.png and b/docs/explore-and-author/graph-exploration/graph-insights/assets/settings.png differ
diff --git a/docs/explore-and-author/graph-exploration/graph-insights/features/application-settings.md b/docs/explore-and-author/graph-exploration/graph-insights/features/application-settings.md
index 6bb227fcc..e085e4c9a 100644
--- a/docs/explore-and-author/graph-exploration/graph-insights/features/application-settings.md
+++ b/docs/explore-and-author/graph-exploration/graph-insights/features/application-settings.md
@@ -10,6 +10,7 @@ Access global configuration via the top-left dropdown menu.
- **Technical details:** Toggles visibility of low-level metadata, including **IRIs** and **SPARQL queries**.
- **Tree visibility:** Hides empty or deactivated classes in the class tree.
+- **Content language:** Sets the preferred language for captions and descriptions. Options are dynamically populated from the current database and selecting a new language triggers an immediate data reload.
{ class="bordered" width="60%" }
diff --git a/docs/explore-and-author/graph-exploration/graph-insights/features/category-tree.md b/docs/explore-and-author/graph-exploration/graph-insights/features/category-tree.md
index d5fbe0477..8dab84584 100644
--- a/docs/explore-and-author/graph-exploration/graph-insights/features/category-tree.md
+++ b/docs/explore-and-author/graph-exploration/graph-insights/features/category-tree.md
@@ -33,7 +33,7 @@ The tree displays the hierarchical taxonomy of the dataset:
The tree is the primary tool to populate the canvas:
- **Drag and drop (Start):** Dragging a class to an empty area clears the canvas and starts a new exploration with a group of the selected class.
- *Alternatives: Double-click the class or use its context menu item `Start exploration with class`.*
+ - *Alternatives: Double-click the class or use its context menu item `Start exploration with class`.*
- **Drag and drop (Intersect):** Drag a class onto an **existing** group to apply an intersection filter (e.g., dragging `German Suppliers` onto `VIP Suppliers` restricts the group to resources with **both classes**).
---
diff --git a/docs/explore-and-author/graph-exploration/graph-insights/tutorial.md b/docs/explore-and-author/graph-exploration/graph-insights/tutorial.md
index 56be5bd9d..6a9b0ab3f 100644
--- a/docs/explore-and-author/graph-exploration/graph-insights/tutorial.md
+++ b/docs/explore-and-author/graph-exploration/graph-insights/tutorial.md
@@ -91,11 +91,11 @@ We define "VIP" as customers with the highest volume of incoming orders.
2. **Filter by country:** Open the menu of the `country` column from its header (using the menu dropdown or a right-click) and enter `USA` in the text field of the filter submenu.
3. **Identify VIPs:** Click the **Predecessors** column header to **Sort Descending**. This ranks customers by their incoming connection count (number of Orders).
-{ class="bordered" width="80%" }
+ { class="bordered" width="80%" }
-1. **Select and restrict:** Check the boxes for the top 5 customers and click `Restrict to selection`.
+4. **Select and restrict:** Check the boxes for the top 5 customers and click `Restrict to selection`.
-{ class="bordered" width="85%" }
+ { class="bordered" width="85%" }
### 3.2. Isolate UK Suppliers
@@ -163,11 +163,11 @@ Finally, we convert this visual insight into an actionable list for the procurem
- Right-click a specific high-value item (e.g., "Teatime Chocolate Biscuits") to open its context menu.
- Select `Flag all connected resources on the branch`.
-{ class="bordered" width="85%" }
+ { class="bordered" width="85%" }
-- Graph Insights highlights all resources on a specific high-risk traversal: The specific UK Supplier → The specific Biscuit → The specific VIP US Customers buying it.
+ - Graph Insights highlights all resources on a specific high-risk traversal: The specific UK Supplier → The specific Biscuit → The specific VIP US Customers buying it.
-{ class="bordered" width="85%" }
+ { class="bordered" width="85%" }
---
diff --git a/poetry.lock b/poetry.lock
index 4f4bfaf3c..01f088d26 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -762,6 +762,17 @@ enabler = ["pytest-enabler (>=2.2)"]
test = ["jaraco.test (>=5.4)", "pytest (>=6,!=8.1.*)", "zipp (>=3.17)"]
type = ["pytest-mypy"]
+[[package]]
+name = "iniconfig"
+version = "2.3.0"
+description = "brain-dead simple config-ini parsing"
+optional = false
+python-versions = ">=3.10"
+files = [
+ {file = "iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12"},
+ {file = "iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730"},
+]
+
[[package]]
name = "jinja2"
version = "3.1.6"
@@ -1394,6 +1405,21 @@ docs = ["furo (>=2024.8.6)", "proselint (>=0.14)", "sphinx (>=8.1.3)", "sphinx-a
test = ["appdirs (==1.4.4)", "covdefaults (>=2.3)", "pytest (>=8.3.4)", "pytest-cov (>=6)", "pytest-mock (>=3.14)"]
type = ["mypy (>=1.14.1)"]
+[[package]]
+name = "pluggy"
+version = "1.6.0"
+description = "plugin and hook calling mechanisms for python"
+optional = false
+python-versions = ">=3.9"
+files = [
+ {file = "pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746"},
+ {file = "pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3"},
+]
+
+[package.extras]
+dev = ["pre-commit", "tox"]
+testing = ["coverage", "pytest", "pytest-benchmark"]
+
[[package]]
name = "propcache"
version = "0.3.2"
@@ -1690,6 +1716,27 @@ files = [
[package.extras]
diagrams = ["jinja2", "railroad-diagrams"]
+[[package]]
+name = "pytest"
+version = "9.1.1"
+description = "pytest: simple powerful testing with Python"
+optional = false
+python-versions = ">=3.10"
+files = [
+ {file = "pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c"},
+ {file = "pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313"},
+]
+
+[package.dependencies]
+colorama = {version = ">=0.4", markers = "sys_platform == \"win32\""}
+iniconfig = ">=1.0.1"
+packaging = ">=22"
+pluggy = ">=1.5,<2"
+pygments = ">=2.7.2"
+
+[package.extras]
+dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "requests", "setuptools", "xmlschema"]
+
[[package]]
name = "python-dateutil"
version = "2.9.0.post0"
@@ -2209,4 +2256,4 @@ type = ["pytest-mypy"]
[metadata]
lock-version = "2.0"
python-versions = "^3.11"
-content-hash = "8dca9021c750c8ab63ab240ae05ccde92fb1998bcd1659ce19074d31610aefc9"
+content-hash = "b29564f7e517705f8e3acab382a355450ede6affa5d79e085c0200fe8284f134"
diff --git a/pyproject.toml b/pyproject.toml
index ff86d0d68..654e2fc25 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -35,6 +35,12 @@ jinja2 = "^3.1.6"
[tool.poetry.group.dev.dependencies]
linkcheckmd = "^1.4.0"
rumdl = "^0.0.194"
+pytest = "^9.1.1"
+
+[tool.pytest.ini_options]
+markers = [
+ "integration: needs a live DI/CMEM instance to run",
+]
[build-system]
requires = ["poetry-core"]
diff --git a/tests/test_update_di_reference.py b/tests/test_update_di_reference.py
index 330e4421a..d63f0c432 100644
--- a/tests/test_update_di_reference.py
+++ b/tests/test_update_di_reference.py
@@ -1,8 +1,226 @@
"""Test update DI references"""
-from tools.update_di_reference import get_plugin_descriptions
+import pytest
+from tools.update_di_reference import (
+ PluginDescription,
+ PluginReference,
+ RelatedPluginReferenceError,
+ get_plugin_descriptions,
+ resolve_related_plugin_links,
+ create_plugin_markdown,
+ build_plugin_paths,
+ validate_related_plugin_references,
+)
+
+@pytest.mark.integration
def test_get_plugin_descriptions():
"""Test get DI plugin descriptions"""
descriptions = get_plugin_descriptions()
- pass
+ assert set(descriptions.keys()) == {"customtask", "dataset", "distancemeasure", "transformer", "aggregator"}
+ assert len(descriptions["transformer"]) > 0
+ assert len(descriptions["dataset"]) > 0
+ for plugin_type, plugins in descriptions.items():
+ for plugin in plugins:
+ assert isinstance(plugin, PluginDescription)
+ assert plugin.pluginType == plugin_type
+ titles = [p.title.lower() for p in plugins]
+ assert titles == sorted(titles)
+
+
+def _make_plugin(plugin_id, plugin_type="transformer", main_category="Extract", related=None):
+ """Build a minimal valid PluginDescription for a test, filling in only what varies."""
+ return PluginDescription(
+ pluginId=plugin_id,
+ title=plugin_id,
+ categories=[main_category],
+ description="test plugin",
+ properties={},
+ actions={},
+ required=[],
+ backendType="scala",
+ pluginType=plugin_type,
+ relatedPlugins=related or [],
+ )
+
+
+@pytest.mark.parametrize(
+ "current_path, target_path, expected",
+ [
+ # same type, different category
+ ("transformer/Extract/regexExtract.md", "transformer/Replace/regexReplace.md", "../Replace/regexReplace.md"),
+ # different type entirely
+ ("transformer/Extract/regexExtract.md", "aggregator/average.md", "../../aggregator/average.md"),
+ # reverse: shallower page linking to a deeper one
+ ("aggregator/average.md", "transformer/Extract/regexExtract.md", "../transformer/Extract/regexExtract.md"),
+ # same directory
+ ("customtask/sparqlSelectOperator.md", "customtask/sparqlUpdateOperator.md", "sparqlUpdateOperator.md"),
+ ],
+)
+def test_resolve_related_plugin_links_path_shapes(current_path, target_path, expected):
+ ref = PluginReference(id="target")
+ plugin = _make_plugin("current", related=[ref])
+ plugin_paths = {"current": current_path, "target": target_path}
+ assert resolve_related_plugin_links(plugin, current_path, plugin_paths) == [(ref, expected)]
+
+
+def test_resolve_related_plugin_links_empty():
+ plugin = _make_plugin("regexExtract", related=[])
+ assert resolve_related_plugin_links(plugin, "transformer/Extract/regexExtract.md", {}) == []
+
+
+def test_resolve_related_plugin_links_single():
+ ref = PluginReference(id="regexReplace", description="replaces text")
+ plugin = _make_plugin("regexExtract", related=[ref])
+ plugin_paths = {"regexExtract": "transformer/Extract/regexExtract.md", "regexReplace": "transformer/Replace/regexReplace.md"}
+ resolved = resolve_related_plugin_links(plugin, "transformer/Extract/regexExtract.md", plugin_paths)
+ assert resolved == [(ref, "../Replace/regexReplace.md")]
+
+
+def test_resolve_related_plugin_links_multiple_preserves_order():
+ ref_a = PluginReference(id="regexReplace")
+ ref_b = PluginReference(id="regexSelect")
+ plugin = _make_plugin("regexExtract", related=[ref_a, ref_b])
+ plugin_paths = {
+ "regexExtract": "transformer/Extract/regexExtract.md",
+ "regexReplace": "transformer/Replace/regexReplace.md",
+ "regexSelect": "transformer/Selection/regexSelect.md",
+ }
+ resolved = resolve_related_plugin_links(plugin, "transformer/Extract/regexExtract.md", plugin_paths)
+ assert [ref for ref, _ in resolved] == [ref_a, ref_b]
+
+
+def test_resolve_related_plugin_links_unresolvable_raises():
+ ref = PluginReference(id="deprecatedPlugin")
+ plugin = _make_plugin("regexExtract", related=[ref])
+ with pytest.raises(RelatedPluginReferenceError, match="deprecatedPlugin"):
+ resolve_related_plugin_links(plugin, "transformer/Extract/regexExtract.md", {"regexExtract": "transformer/Extract/regexExtract.md"})
+
+
+def test_build_plugin_paths():
+ plugins = {
+ "transformer": [_make_plugin("regexExtract", plugin_type="transformer", main_category="Extract")],
+ "dataset": [_make_plugin("sparqlEndpoint", plugin_type="dataset")],
+ "customtask": [_make_plugin("deprecatedPlugin", plugin_type="customtask")],
+ }
+ paths = build_plugin_paths(plugins)
+ assert paths == {
+ "regexExtract": "transformer/Extract/regexExtract.md",
+ "sparqlEndpoint": "dataset/sparqlEndpoint.md",
+ }
+ assert "deprecatedPlugin" not in paths
+
+
+def test_validate_related_plugin_references_passes_when_all_resolve():
+ plugins = {
+ "transformer": [_make_plugin("regexExtract", related=[PluginReference(id="regexReplace")])],
+ }
+ plugin_paths = {
+ "regexExtract": "transformer/Extract/regexExtract.md",
+ "regexReplace": "transformer/Replace/regexReplace.md",
+ }
+ validate_related_plugin_references(plugins, plugin_paths)
+
+
+def test_validate_related_plugin_references_raises_on_unresolvable():
+ plugins = {
+ "transformer": [_make_plugin("regexExtract", related=[PluginReference(id="removedPlugin")])],
+ }
+ plugin_paths = {"regexExtract": "transformer/Extract/regexExtract.md"}
+ with pytest.raises(RelatedPluginReferenceError, match="removedPlugin"):
+ validate_related_plugin_references(plugins, plugin_paths)
+
+
+def test_validate_related_plugin_references_skips_deprecated_plugins():
+ plugins = {
+ "customtask": [_make_plugin("deprecatedPlugin", plugin_type="customtask", related=[PluginReference(id="removedPlugin")])],
+ }
+ validate_related_plugin_references(plugins, plugin_paths={})
+
+
+def test_create_plugin_markdown_writes_resolved_links(tmp_path):
+ plugin = _make_plugin(
+ "sparqlSelectOperator",
+ plugin_type="customtask",
+ related=[PluginReference(id="sparqlEndpoint", description="a SPARQL endpoint dataset")],
+ )
+ plugin_paths = {
+ "sparqlSelectOperator": "customtask/sparqlSelectOperator.md",
+ "sparqlEndpoint": "dataset/sparqlEndpoint.md",
+ }
+
+ create_plugin_markdown(plugin, tmp_path, plugin_paths)
+
+ written = (tmp_path / "customtask" / "sparqlSelectOperator.md").read_text()
+ assert "[sparqlEndpoint](../dataset/sparqlEndpoint.md)" in written
+ assert "**sparqlEndpoint**" not in written
+ assert "[sparqlEndpoint](../dataset/sparqlEndpoint.md) — a SPARQL endpoint dataset" in written
+
+
+@pytest.mark.parametrize(
+ "plugin_id, plugin_path, related_id, related_path, expected_link",
+ [
+ # same type, different category
+ ("regexExtract", "transformer/Extract/regexExtract.md", "regexReplace", "transformer/Replace/regexReplace.md", "../Replace/regexReplace.md"),
+ # reverse: shallower page linking to a deeper one
+ ("average", "aggregator/average.md", "regexExtract", "transformer/Extract/regexExtract.md", "../transformer/Extract/regexExtract.md"),
+ # same directory
+ ("sparqlSelectOperator", "customtask/sparqlSelectOperator.md", "sparqlUpdateOperator", "customtask/sparqlUpdateOperator.md", "sparqlUpdateOperator.md"),
+ ],
+)
+def test_create_plugin_markdown_link_path_shapes(tmp_path, plugin_id, plugin_path, related_id, related_path, expected_link):
+ plugin = _make_plugin(plugin_id, related=[PluginReference(id=related_id)])
+ plugin_paths = {plugin_id: plugin_path, related_id: related_path}
+
+ create_plugin_markdown(plugin, tmp_path, plugin_paths)
+
+ written = (tmp_path / plugin_path).read_text()
+ assert f"[{related_id}]({expected_link})" in written
+
+
+def test_create_plugin_markdown_writes_multiple_resolved_links(tmp_path):
+ plugin = _make_plugin(
+ "regexExtract",
+ plugin_type="transformer",
+ main_category="Extract",
+ related=[PluginReference(id="regexReplace"), PluginReference(id="regexSelect")],
+ )
+ plugin_paths = {
+ "regexExtract": "transformer/Extract/regexExtract.md",
+ "regexReplace": "transformer/Replace/regexReplace.md",
+ "regexSelect": "transformer/Selection/regexSelect.md",
+ }
+
+ create_plugin_markdown(plugin, tmp_path, plugin_paths)
+
+ written = (tmp_path / "transformer" / "Extract" / "regexExtract.md").read_text()
+ assert "[regexReplace](../Replace/regexReplace.md)" in written
+ assert "[regexSelect](../Selection/regexSelect.md)" in written
+
+
+def test_create_plugin_markdown_omits_description_when_absent(tmp_path):
+ plugin = _make_plugin(
+ "sparqlSelectOperator",
+ plugin_type="customtask",
+ related=[PluginReference(id="sparqlEndpoint")],
+ )
+ plugin_paths = {
+ "sparqlSelectOperator": "customtask/sparqlSelectOperator.md",
+ "sparqlEndpoint": "dataset/sparqlEndpoint.md",
+ }
+
+ create_plugin_markdown(plugin, tmp_path, plugin_paths)
+
+ written = (tmp_path / "customtask" / "sparqlSelectOperator.md").read_text()
+ assert "[sparqlEndpoint](../dataset/sparqlEndpoint.md)" in written
+ assert "[sparqlEndpoint](../dataset/sparqlEndpoint.md) —" not in written
+
+
+def test_create_plugin_markdown_omits_section_when_no_related_plugins(tmp_path):
+ plugin = _make_plugin("regexExtract", plugin_type="transformer", main_category="Extract", related=[])
+ plugin_paths = {"regexExtract": "transformer/Extract/regexExtract.md"}
+
+ create_plugin_markdown(plugin, tmp_path, plugin_paths)
+
+ written = (tmp_path / "transformer" / "Extract" / "regexExtract.md").read_text()
+ assert "Related Plugins" not in written
diff --git a/tools/templates/plugin.md b/tools/templates/plugin.md
index 3f9bb736f..e22400748 100644
--- a/tools/templates/plugin.md
+++ b/tools/templates/plugin.md
@@ -26,11 +26,11 @@ tags: {% for tag in plugin.tags %}
{{parameters_advanced if plugin.properties_advanced else "`None`"}}
-{%- if plugin.relatedPlugins %}
+{%- if related_plugins_resolved %}
## Related Plugins
-{% for ref in plugin.relatedPlugins -%}
-- **{{ ref.id }}**{% if ref.description %} — {{ ref.description }}{% endif %}
+{% for ref, link in related_plugins_resolved -%}
+- [{{ ref.id }}]({{ link }}){% if ref.description %} — {{ ref.description }}{% endif %}
{% endfor %}
{%- endif %}
diff --git a/tools/update_di_reference.py b/tools/update_di_reference.py
index 1f0d2c07c..6894509e0 100644
--- a/tools/update_di_reference.py
+++ b/tools/update_di_reference.py
@@ -1,6 +1,7 @@
"""Update DI Reference documentation"""
import json
+import posixpath
import re
from contextlib import suppress
from pathlib import Path
@@ -161,13 +162,43 @@ def get_plugin_descriptions() -> dict[str, list[PluginDescription]]:
plugins[type_id] = plugins_of_type
return plugins
-def create_plugin_markdown(plugin: PluginDescription, plugin_type: str, base_dir: Path) -> None:
+class RelatedPluginReferenceError(Exception):
+ """A relatedPlugins reference points at a plugin with no resolvable page."""
+
+
+class DuplicatePluginIdError(Exception):
+ """A plugin ID is not unique across the walked plugin types."""
+
+
+def resolve_related_plugin_links(plugin: PluginDescription, current_plugin_path: str, plugin_paths: dict[str, str]) -> list[tuple[PluginReference, str]]:
+ """Resolve each related plugin reference to a path relative to the current plugin's own page.
+
+ Raises if any reference points at a plugin with no resolvable page.
+ """
+ current_dir = posixpath.dirname(current_plugin_path)
+ resolved = []
+ unresolvable = []
+ for ref in plugin.relatedPlugins:
+ if ref.id not in plugin_paths:
+ unresolvable.append(ref.id)
+ continue
+ resolved.append((ref, posixpath.relpath(plugin_paths[ref.id], current_dir)))
+ if unresolvable:
+ raise RelatedPluginReferenceError(
+ f"Related plugin(s) {', '.join(unresolvable)} referenced by '{plugin.pluginId}' have no resolvable page"
+ )
+ return resolved
+
+def create_plugin_markdown(plugin: PluginDescription, base_dir: Path, plugin_paths: dict[str, str]) -> None:
"""Create markdown document from plugin description."""
if plugin.is_deprecated:
click.echo(f"Ignore deprecated plugin {plugin.pluginId}")
return
click.echo(f"Create reference documentation for {plugin.pluginId}")
+ current_plugin_path = plugin_paths[plugin.pluginId]
+ related_plugins_resolved = resolve_related_plugin_links(plugin, current_plugin_path, plugin_paths)
+
# create content
plugin_template = jinja_environment.get_template(f"plugin.md")
parameter_template = jinja_environment.get_template(f"parameter.md")
@@ -183,15 +214,11 @@ def create_plugin_markdown(plugin: PluginDescription, plugin_type: str, base_dir
plugin=plugin,
parameters=parameter_content.rstrip("\n"),
parameters_advanced=parameter_advanced_content.rstrip("\n"),
+ related_plugins_resolved=related_plugins_resolved,
)
- # create the file (incl. directory)
- if plugin.pluginType == "transformer":
- directory = base_dir / plugin_type / plugin.main_category
- else:
- directory = base_dir / plugin_type
- directory.mkdir(parents=True, exist_ok=True)
- file = directory / f"{plugin.pluginId}.md"
+ file = base_dir / current_plugin_path
+ file.parent.mkdir(parents=True, exist_ok=True)
with file.open("w", encoding="utf-8") as f:
f.write(content)
@@ -216,34 +243,33 @@ def create_umbrella_pages(plugins: dict[str, list[PluginDescription]], base_dir:
- "Transformers": transformer"""
f.write(content)
- for plugin_type in plugins:
- plugins_of_type = plugins[plugin_type]
- if plugin_type == "transformer":
+ for type_id, plugins_of_type in plugins.items():
+ if type_id == "transformer":
table_template = jinja_environment.get_template(f"operator_table_with_category.md")
else:
table_template = jinja_environment.get_template(f"operator_table.md")
# Create type-specific index.md file
- index_file = base_dir / f"{plugin_type}/index.md"
- index_template = jinja_environment.get_template(f"{plugin_type}_base.md")
+ index_file = base_dir / f"{type_id}/index.md"
+ index_template = jinja_environment.get_template(f"{type_id}_base.md")
items = table_template.render(plugins=plugins_of_type)
with index_file.open("w", encoding="utf-8") as f:
- click.echo(f"Create {plugin_type} index file in {index_file}")
+ click.echo(f"Create {type_id} index file in {index_file}")
f.write(index_template.render(items=items))
# Create the .pages files
- pages_file = base_dir / plugin_type / ".pages"
+ pages_file = base_dir / type_id / ".pages"
pages_content = "nav:\n - index.md"
- if plugin_type == "transformer":
+ if type_id == "transformer":
# transformer get separated per main_category
- categories = list(set([plugin.main_category for plugin in plugins[plugin_type]]))
+ categories = list(set([plugin.main_category for plugin in plugins[type_id]]))
categories.sort()
for category in categories:
pages_content += f'\n - "{category}": {category}'
- sub_pages_file = base_dir / plugin_type / category / ".pages"
+ sub_pages_file = base_dir / type_id / category / ".pages"
sub_pages_content = "nav:"
- category_plugins = [plugin for plugin in plugins[plugin_type] if plugin.main_category == category]
+ category_plugins = [plugin for plugin in plugins[type_id] if plugin.main_category == category]
for plugin in category_plugins:
sub_pages_content += f"\n - \"{plugin.title}\": {plugin.pluginId}.md"
with sub_pages_file.open("w", encoding="utf-8") as f:
@@ -258,6 +284,39 @@ def create_umbrella_pages(plugins: dict[str, list[PluginDescription]], base_dir:
click.echo(f"Create .pages file {pages_file}")
f.write(pages_content)
+def build_plugin_paths(plugins: dict[str, list[PluginDescription]]) -> dict[str, str]:
+ """Map every non-deprecated plugin's ID to the path its own page will have.
+
+ Deprecated plugins are excluded, since no page is ever generated for them.
+ """
+ plugin_paths: dict[str, str] = {}
+ for plugins_list in plugins.values():
+ for plugin in plugins_list:
+ if plugin.is_deprecated:
+ continue
+ if plugin.pluginType == "transformer":
+ plugin_paths[plugin.pluginId] = f"{plugin.pluginType}/{plugin.main_category}/{plugin.pluginId}.md"
+ else:
+ plugin_paths[plugin.pluginId] = f"{plugin.pluginType}/{plugin.pluginId}.md"
+ return plugin_paths
+
+def validate_related_plugin_references(plugins: dict[str, list[PluginDescription]], plugin_paths: dict[str, str]) -> None:
+ """Raise on every relatedPlugins reference with no resolvable page.
+
+ Deprecated plugins are skipped, since no page is ever generated for
+ them and their relatedPlugins is otherwise never inspected.
+ """
+ errors = []
+ for plugins_list in plugins.values():
+ for plugin in plugins_list:
+ if plugin.is_deprecated:
+ continue
+ try:
+ resolve_related_plugin_links(plugin, plugin_paths[plugin.pluginId], plugin_paths)
+ except RelatedPluginReferenceError as error:
+ errors.append(str(error))
+ if errors:
+ raise RelatedPluginReferenceError("\n".join(errors))
@click.command()
@click.option(
@@ -274,19 +333,23 @@ def update_di_reference(output_dir):
click.echo(f"Dump plugins descriptions to {(plugins_json := Path('data/plugins.json'))}")
plugins_dump: dict[str, dict] = {}
- for category, plugins_list in plugins.items():
+ for type_id, plugins_list in plugins.items():
for plugin in plugins_list:
if plugin.pluginId in plugins_dump:
- raise Exception(f"Duplicate plugin ID: {plugin.pluginId}")
+ raise DuplicatePluginIdError(f"Duplicate plugin ID: {plugin.pluginId}")
plugins_dump[plugin.pluginId] = plugin.model_dump()
+
+ plugin_paths = build_plugin_paths(plugins)
+ validate_related_plugin_references(plugins, plugin_paths)
+
plugins_json.write_text(json.dumps(plugins_dump, indent=2))
click.echo(f"Creating DI reference documentation in {basedir}")
# create directory structure
rmtree(basedir, ignore_errors=True)
basedir.mkdir(parents=True, exist_ok=True)
- for type_id in plugins:
+ for type_id, plugins_of_type in plugins.items():
Path(basedir / type_id).mkdir(parents=True, exist_ok=True)
- for plugin in plugins[type_id]:
- create_plugin_markdown(plugin, type_id, basedir)
+ for plugin in plugins_of_type:
+ create_plugin_markdown(plugin, basedir, plugin_paths)
create_umbrella_pages(plugins=plugins, base_dir=basedir)