feat(gizmosql): add - #289
Conversation
| r2-account-id: | ||
| # The event stream analytics' cloud storage account identifier. | ||
| file: ~~/artifacts/secrets/r2-account-id.secret | ||
| r2-aws-credentials-access-key: | ||
| # The event stream analytics' cloud storage secret access key. | ||
| file: ~~/artifacts/secrets/r2-aws-credentials-access-key.secret | ||
| r2-aws-credentials-access-key-id: | ||
| # The event stream analytics' cloud storage access key identifier. | ||
| file: ~~/artifacts/secrets/r2-aws-credentials-access-key-id.secret |
There was a problem hiding this comment.
Let's check if it's possible to define these secrets on the shared src/development/compose.yaml or in the redpanda-connect compose file (where the credentials are used to write – writing comes before reading which gizmosql allows then). But as these credentials are used by multiple services, they should fit the shared compose file. This should be supported by dargstack already or added to it if not.
There was a problem hiding this comment.
Yeah should be doable with dargstack v4 then. I'll migrate the secrets then.
| # Tombstone messages (Debezium deletes) carry no payload and must be dropped. | ||
| - bloblang: | | ||
| root = if this.payload.after.or(this.after) == null { deleted() } | ||
| - mapping: | | ||
| root = this.payload.after.or(this.after) | ||
| root.operation = this.payload.op.or(this.op) |
There was a problem hiding this comment.
Not sure if this handles all tombstone messages correctly, I'd need to research a bit more.
| GF_DATABASE_PASSWORD__FILE: /run/secrets/postgres-role-service-grafana-password | ||
| GF_DATABASE_TYPE: postgres | ||
| GF_DATABASE_USER__FILE: /run/secrets/postgres-role-service-grafana-username | ||
| GF_INSTALL_PLUGINS: gizmodata-gizmosql-datasource |
There was a problem hiding this comment.
I think about removing all the provisioning code for Grafana and instead configure it in the live system. Provisioning codifies the initial setup and makes it more explicit in that sense but also is quite rigid and disallows changing the config in the live Grafana instance. As we want to persist configurations done in the Grafana UI anyway, we can safe ourselves from config being too rigid and extensive and just configure those connectors in the UI, I'd say.
2 participants
This pull request introduces an end-to-end event stream analytics pipeline, enabling Redpanda event data to be ingested, stored as Parquet files in cloud storage, and queried using Arrow Flight SQL via GizmoSQL. The changes add new services, configuration, and documentation to support this workflow.
Event Stream Analytics Pipeline
redpanda-connectservice, which ingests events from Redpanda, transforms the data, and writes Parquet files to S3-compatible cloud storage (R2). Includes configuration for S3 credentials and data schema.gizmosqlservice, which allows querying the event stream's Parquet lakehouse using Arrow Flight SQL. Includes secure credential handling and a custom entrypoint script for environment setup and templating.