feat: Implement generic database connection

[maltesander]

Description

Part of stackabletech/issues#238
Needs stackabletech/operator-rs#1200

• BREAKING: renamed metadata_storage_database -> metadata_database (consolidation)
• remove metadata storage database properties from product config properties yaml

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches) No, as we needed feat: Add DerbyConnection::jdbc_connection_details_for_network_access helper operator-rs#1200 and don't want to wait for an op-rs release

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[sbernauer]

I will have a look after stackabletech/superset-operator#722

[sbernauer]

Looks good, thanks!

[maltesander]

Release notes

BREAKING: Consolidated the metadata storage database CRD config to use the generic database connector mechanism.

The metadataStorageDatabase was renamed to metadataDatabase. The structure changed from:

PostgreSQL / MySQL

metadataStorageDatabase:
  dbType: postgresql # or mysql
  connString: jdbc:postgresql://postgresql-druid/druid
  host: postgresql-druid
  port: 5432
  credentialsSecret: druid-db-credentials

to:

metadataDatabase:
  postgresql: # or mysql
    host: postgresql-druid
    database: druid
    credentialsSecretName: druid-db-credentials

Derby

From:

metadataStorageDatabase:
  dbType: derby
  connString: jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
  host: localhost
  port: 1527

to:

metadataDatabase:
  derby: {}

Documentation

Not many changes, just example updates. E.g:

https://docs.stackable.tech/home/nightly/druid/getting_started/first_steps/#_druid

The merge-base changed after approval.

The merge-base changed after approval.

[sbernauer]

LGTM, but we should schedule a Jenkins run as soon as the images are build

[maltesander]

Test: https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/druid-operator-it-custom/80/

[maltesander]

Some failures (seem flaky), works locally:

--- PASS: kuttl (1554.29s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/oidc_druid-34.0.0_zookeeper-latest-3.9.4_s3-use-tls-false_openshift-false_oidc-use-tls-false (171.51s)
        --- PASS: kuttl/harness/ldap_druid-30.0.1_zookeeper-latest-3.9.4_opa-1.12.3_hadoop-latest-3.4.2_ldap-use-tls-false_ldap-no-bind-credentials-false_openshift-false (180.04s)
        --- PASS: kuttl/harness/oidc_druid-30.0.1_zookeeper-latest-3.9.4_s3-use-tls-true_openshift-false_oidc-use-tls-true (152.67s)
        --- PASS: kuttl/harness/ldap_druid-35.0.1_zookeeper-latest-3.9.4_opa-1.12.3_hadoop-latest-3.4.2_ldap-use-tls-true_ldap-no-bind-credentials-false_openshift-false (196.38s)
        --- PASS: kuttl/harness/oidc_druid-30.0.1_zookeeper-latest-3.9.4_s3-use-tls-true_openshift-false_oidc-use-tls-false (150.06s)
        --- PASS: kuttl/harness/oidc_druid-30.0.1_zookeeper-latest-3.9.4_s3-use-tls-false_openshift-false_oidc-use-tls-true (157.70s)
        --- PASS: kuttl/harness/oidc_druid-30.0.1_zookeeper-latest-3.9.4_s3-use-tls-false_openshift-false_oidc-use-tls-false (160.32s)
        --- PASS: kuttl/harness/oidc_druid-35.0.1_zookeeper-latest-3.9.4_s3-use-tls-false_openshift-false_oidc-use-tls-true (146.22s)
        --- PASS: kuttl/harness/ldap_druid-34.0.0_zookeeper-latest-3.9.4_opa-1.12.3_hadoop-latest-3.4.2_ldap-use-tls-true_ldap-no-bind-credentials-false_openshift-false (150.77s)
        --- PASS: kuttl/harness/overrides_druid-latest-35.0.1_zookeeper-latest-3.9.4_hadoop-latest-3.4.2_openshift-false (159.94s)
        --- PASS: kuttl/harness/ldap_druid-35.0.1_zookeeper-latest-3.9.4_opa-1.12.3_hadoop-latest-3.4.2_ldap-use-tls-false_ldap-no-bind-credentials-false_openshift-false (146.74s)
        --- PASS: kuttl/harness/oidc_druid-35.0.1_zookeeper-latest-3.9.4_s3-use-tls-true_openshift-false_oidc-use-tls-true (148.38s)
        --- PASS: kuttl/harness/ldap_druid-34.0.0_zookeeper-latest-3.9.4_opa-1.12.3_hadoop-latest-3.4.2_ldap-use-tls-false_ldap-no-bind-credentials-false_openshift-false (148.85s)
        --- PASS: kuttl/harness/oidc_druid-35.0.1_zookeeper-latest-3.9.4_s3-use-tls-true_openshift-false_oidc-use-tls-false (153.26s)
        --- PASS: kuttl/harness/ldap_druid-30.0.1_zookeeper-latest-3.9.4_opa-1.12.3_hadoop-latest-3.4.2_ldap-use-tls-true_ldap-no-bind-credentials-false_openshift-false (159.37s)
        --- PASS: kuttl/harness/oidc_druid-35.0.1_zookeeper-latest-3.9.4_s3-use-tls-false_openshift-false_oidc-use-tls-false (144.36s)
        --- PASS: kuttl/harness/oidc_druid-34.0.0_zookeeper-latest-3.9.4_s3-use-tls-true_openshift-false_oidc-use-tls-false (145.53s)
        --- PASS: kuttl/harness/oidc_druid-34.0.0_zookeeper-latest-3.9.4_s3-use-tls-true_openshift-false_oidc-use-tls-true (148.59s)
        --- PASS: kuttl/harness/oidc_druid-34.0.0_zookeeper-latest-3.9.4_s3-use-tls-false_openshift-false_oidc-use-tls-true (159.95s)
PASS