feat(databricks_zerobus): Allow ingestion of variant data

[flaviofcruz]

Summary

Add support to the databricks_zerobus sink to push variant data. We rely on the zerobus SDK to give us variant annotations on the arrow batch schema, which we use to optionally iterate over the batch and rewrite it in a variant compatible way. Since we are using the serde_json crate to do the event -> batch conversion, we need to update the existing arrow batch when variants are detected. We have a variant plan which is used to cache and speed up arrow batch updates.

References

This is similar to #25890 but takes advantage of the new functionality in the Zerobus SDK.

Vector configuration

  sinks:
    zb:
      type: databricks_zerobus
      inputs: [demo]
      ingestion_endpoint: "https://ingest.dev.databricks.com"
      unity_catalog_endpoint: "https://workspace.cloud.databricks.com"
      table_name: "main.default.my_table"
      user_agent: "My Application"
      auth:
        strategy: oauth
        client_id: "${DATABRICKS_CLIENT_ID}"
        client_secret: "${DATABRICKS_CLIENT_SECRET}"

How did you test this PR?

This was tested with a Databricks table that uses variants. Ingestion worked fine and I was able to query the data.

Manually with a Databricks table containing a variant field.

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

Contributor Guidelines

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them. To catch issues early, add a pre-push hook (template) or run the following locally before pushing:
  • make fmt
  • make check-clippy (auto-fix with make clippy-fix)
  • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 11f07f526d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".