Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file not shown.
88 changes: 0 additions & 88 deletions docs/modules/nifi/pages/usage_guide/writing-to-iceberg-tables.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -33,91 +33,3 @@ Please read on https://github.com/stackabletech/nifi-iceberg-bundle[its document
You don't need any special configs on the `NiFiCluster` in case you are using S3 and no Kerberos.

HDFS and Kerberos are also supported, please have a look at the https://github.com/stackabletech/nifi-operator/tree/main/tests/templates/kuttl/iceberg[Iceberg integration test] for that.

== NiFi 1

Starting with `1.19.0`, NiFi supports writing to Iceberg tables.

The following example shows an example NiFi setup using the Iceberg integration.

[source,yaml]
----
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
name: nifi
spec:
clusterConfig:
# ...
extraVolumes:
# Will be mounted at /stackable/userdata/nifi-hive-s3-config/
- name: nifi-hive-s3-config
secret:
secretName: nifi-hive-s3-config
---
apiVersion: v1
kind: Secret
metadata:
name: nifi-hive-s3-config
stringData:
core-site.xml: |
<configuration>
<property>
<name>fs.s3a.aws.credentials.provider</name>
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>

<property>
<name>fs.s3a.endpoint</name>
<value>http://minio:9000</value>
</property>

<property>
<name>fs.s3a.access.key</name>
<value>xxx</value>
</property>

<property>
<name>fs.s3a.secret.key</name>
<value>xxx</value>
</property>

<property>
<name>fs.s3a.path.style.access</name>
<value>true</value>
</property>

<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>false</value>
<description>Enables or disables SSL connections to S3.</description>
</property>
</configuration>
----

Please fill in the correct endpoint, access key and secret key for your S3 store, this is a classic Hadoop config file.

Use e.g. Trino to create a table for Nifi to write into using something like

[source,sql]
----
CREATE SCHEMA IF NOT EXISTS lakehouse.demo WITH (location = 's3a://lakehouse/demo/');

CREATE TABLE IF NOT EXISTS lakehouse.demo.test (
test varchar
);
----

In NiFi you need to create a `HiveCatalogService` first which allows you to access the Hive Metastore storing the Iceberg metadata.
Set `Hive Metastore URI` to something like `thrift://hive-iceberg.default.svc.cluster.local:9083`,
`Default Warehouse Location` to `s3a://lakehouse`
and `Hadoop Configuration Resources` to `/stackable/userdata/nifi-hive-s3-config/core-site.xml`.

Afterwards you can create the `PutIceberg` processor and configure the `HiveCatalogService`.
Also set `Catalog Namespace` to your schema name and the `Table Name`.

For the `File Format` it is recommended to use `PARQUET` or `ORC` rather than `AVRO` for performance reasons, but you can leave it empty or choose your desired format.

You should end up with the following `PutIceberg` processor:

image::put-iceberg-processor.png[]
3 changes: 1 addition & 2 deletions docs/modules/nifi/partials/supported-versions.adoc
Comment thread
sbernauer marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,8 @@
// Stackable Platform documentation.

* 2.9.0
* 2.7.2
* 2.7.2 (Deprecated)
** Please note that NiFi starting with 2.7.2 supports Iceberg, but only with S3 and Iceberg REST catalog (no Hive metastore or HDFS support). Please read on xref:nifi:usage_guide/writing-to-iceberg-tables.adoc[] for details.
* 2.6.0 (LTS)
* 1.28.1 (Deprecated)
For details on how to upgrade your NiFi version, refer to xref:nifi:usage_guide/updating.adoc[].
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,12 @@ metadata:
name: nifi
spec:
image:
{% if test_scenario['values']['nifi-v2'].find(",") > 0 %}
custom: "{{ test_scenario['values']['nifi-v2'].split(',')[1] }}"
productVersion: "{{ test_scenario['values']['nifi-v2'].split(',')[0] }}"
{% if test_scenario['values']['nifi'].find(",") > 0 %}
custom: "{{ test_scenario['values']['nifi'].split(',')[1] }}"
productVersion: "{{ test_scenario['values']['nifi'].split(',')[0] }}"
{% else %}
custom: null
productVersion: "{{ test_scenario['values']['nifi-v2'] }}"
productVersion: "{{ test_scenario['values']['nifi'] }}"
{% endif %}
pullPolicy: IfNotPresent
clusterConfig:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ data:
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-prometheus-nar",
"version": "1.28.1"
"version": "2.9.0"
},
"properties": {
"prometheus-reporting-task-metrics-endpoint-port": "8081",
Expand Down Expand Up @@ -190,7 +190,7 @@ data:
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-standard-nar",
"version": "1.28.1"
"version": "2.9.0"
},
"properties": {
"Regular Expression": "(?s)(^.*$)",
Expand Down Expand Up @@ -235,7 +235,7 @@ data:
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-standard-nar",
"version": "1.28.1"
"version": "2.9.0"
},
"properties": {
"HTTP Context Map": "fd3b17c4-0196-1000-0000-000076ed786c",
Expand Down Expand Up @@ -311,7 +311,7 @@ data:
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-standard-nar",
"version": "1.28.1"
"version": "2.9.0"
},
"properties": {
"multipart-request-max-size": "1 MB",
Expand Down Expand Up @@ -475,7 +475,7 @@ data:
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-http-context-map-nar",
"version": "1.28.1"
"version": "2.9.0"
},
"properties": {
"Request Expiration": "1 min",
Expand All @@ -488,7 +488,7 @@ data:
"bundle": {
"group": "org.apache.nifi",
"artifact": "nifi-standard-services-api-nar",
"version": "1.28.1"
"version": "2.9.0"
}
}
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,12 @@ metadata:
name: nifi
spec:
image:
{% if test_scenario['values']['nifi-v2'].find(",") > 0 %}
custom: "{{ test_scenario['values']['nifi-v2'].split(',')[1] }}"
productVersion: "{{ test_scenario['values']['nifi-v2'].split(',')[0] }}"
{% if test_scenario['values']['nifi'].find(",") > 0 %}
custom: "{{ test_scenario['values']['nifi'].split(',')[1] }}"
productVersion: "{{ test_scenario['values']['nifi'].split(',')[0] }}"
{% else %}
custom: null
productVersion: "{{ test_scenario['values']['nifi-v2'] }}"
productVersion: "{{ test_scenario['values']['nifi'] }}"
{% endif %}
pullPolicy: IfNotPresent
clusterConfig:
Expand Down
12 changes: 0 additions & 12 deletions tests/templates/kuttl/smoke_v1/20-assert.yaml

This file was deleted.

22 changes: 0 additions & 22 deletions tests/templates/kuttl/smoke_v1/20-install-zookeeper.yaml.j2

This file was deleted.

88 changes: 0 additions & 88 deletions tests/templates/kuttl/smoke_v1/30-install-nifi.yaml.j2

This file was deleted.

28 changes: 0 additions & 28 deletions tests/templates/kuttl/smoke_v1/50-install-test-nifi.yaml

This file was deleted.

Loading
Loading