docs: outputs: add ZeroBus output plugin documentation#2537
docs: outputs: add ZeroBus output plugin documentation#2537mats16 wants to merge 4 commits intofluent:masterfrom
Conversation
📝 WalkthroughWalkthroughAdds a new Fluent Bit ZeroBus output plugin documentation page and a SUMMARY.md entry linking to it; documents prerequisites, configuration keys, build requirements, record transformation ordering, and example payloads for Databricks Zerobus ingestion. Changes
Sequence Diagram(s)(omitted — changes are documentation-only) Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Introduced new ZeroBus output plugin for sending logs to Databricks via the ZeroBus streaming ingestion interface. - Updated SUMMARY.md to include ZeroBus in the list of output plugins. - Provided detailed configuration parameters, usage examples, and record format transformations for the ZeroBus plugin. Signed-off-by: mats <mats.kazuki@gmail.com>
mats16
force-pushed
the
plugins/out_zerobus
branch
from
e387cd1 to
5dac612
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
pipeline/outputs/zerobus.md (3)
81-85: Consider rewording the ordered steps to reduce repeated sentence starts.The repeated “If …” pattern across consecutive steps is readable but slightly mechanical; a light rewrite would improve flow.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pipeline/outputs/zerobus.md` around lines 81 - 85, The bullet list is repetitive because each line starts with "If ..."; rephrase to vary sentence starts while preserving meaning by grouping related actions and using active phrasing: mention raw_log_key behavior (capture full original record as JSON and inject under configured key unless it exists), describe log_key behavior (include only specified keys), then mention time_key (inject RFC3339 timestamp with nanosecond precision unless key exists) and add_tag (inject Fluent Bit tag as _tag unless key exists); keep the examples (timestamp format) and the "unless a key with that name already exists" clause attached to each relevant item and reference the unique config names raw_log_key, log_key, time_key, and add_tag to locate the lines to edit.
23-23: Add a short secret-handling note forclient_secret.Please add guidance to avoid storing
client_secretin plaintext config files (for example, prefer environment variable substitution or secret-management integration).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pipeline/outputs/zerobus.md` at line 23, Add a short secret-handling note for the `client_secret` field advising not to store `client_secret` in plaintext configuration files; recommend using environment variable substitution (e.g., reading from ENV) or integrating with a secrets manager (Vault, AWS Secrets Manager, etc.), and mention limiting access and rotation as best practices so consumers of `client_secret` know secure handling expectations.
11-11: Use a technical docs link instead of the vendor homepage.Linking to the general Databricks homepage is less neutral/technical than linking to the relevant product documentation page for ZeroBus/ingestion setup.
Based on learnings: ensure Markdown stays technical and neutral, and avoid promotional links.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pipeline/outputs/zerobus.md` at line 11, The README sentence starting "The _ZeroBus_ output plugin lets you ingest log records into a Databricks table..." links to the Databricks homepage; replace that href with the official Databricks technical documentation for ZeroBus/streaming ingestion (the product docs page for ZeroBus or ingestion setup) so the link is neutral and technical. Edit pipeline/outputs/zerobus.md and update the Markdown link target in the sentence mentioning "_ZeroBus_" to point to the ZeroBus/streaming ingestion docs URL, keeping the visible text unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pipeline/outputs/zerobus.md`:
- Around line 81-85: The bullet list is repetitive because each line starts with
"If ..."; rephrase to vary sentence starts while preserving meaning by grouping
related actions and using active phrasing: mention raw_log_key behavior (capture
full original record as JSON and inject under configured key unless it exists),
describe log_key behavior (include only specified keys), then mention time_key
(inject RFC3339 timestamp with nanosecond precision unless key exists) and
add_tag (inject Fluent Bit tag as _tag unless key exists); keep the examples
(timestamp format) and the "unless a key with that name already exists" clause
attached to each relevant item and reference the unique config names
raw_log_key, log_key, time_key, and add_tag to locate the lines to edit.
- Line 23: Add a short secret-handling note for the `client_secret` field
advising not to store `client_secret` in plaintext configuration files;
recommend using environment variable substitution (e.g., reading from ENV) or
integrating with a secrets manager (Vault, AWS Secrets Manager, etc.), and
mention limiting access and rotation as best practices so consumers of
`client_secret` know secure handling expectations.
- Line 11: The README sentence starting "The _ZeroBus_ output plugin lets you
ingest log records into a Databricks table..." links to the Databricks homepage;
replace that href with the official Databricks technical documentation for
ZeroBus/streaming ingestion (the product docs page for ZeroBus or ingestion
setup) so the link is neutral and technical. Edit pipeline/outputs/zerobus.md
and update the Markdown link target in the sentence mentioning "_ZeroBus_" to
point to the ZeroBus/streaming ingestion docs URL, keeping the visible text
unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 3079b466-65a0-4ffc-b8c0-ec1c0a4981b3
📒 Files selected for processing (2)
SUMMARY.mdpipeline/outputs/zerobus.md
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pipeline/outputs/zerobus.md (1)
79-85: Consider varying sentence structure for better readability.The transformation steps are clear and well-ordered. However, all five steps begin with "If", which creates repetitive sentence structure. Consider rephrasing for variety while maintaining clarity.
📝 Example rewrite with varied sentence structure
Each log record is converted to a JSON object before ingestion. The plugin applies the following transformations in order: -1. If `raw_log_key` is set, the full original record is captured as a JSON string before any filtering. -2. If `log_key` is set, only the specified keys are included in the output record. -3. If `raw_log_key` is set, the captured JSON string is injected under the configured key (unless a key with that name already exists). -4. If `time_key` is set, a timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected (unless a key with that name already exists). -5. If `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists). +1. When `raw_log_key` is set, the full original record is captured as a JSON string before any filtering. +2. If `log_key` is set, only the specified keys are included in the output record. +3. The captured JSON string (if enabled) is injected under the configured `raw_log_key` (unless a key with that name already exists). +4. A timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected under `time_key` (unless disabled or a key with that name already exists). +5. When `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pipeline/outputs/zerobus.md` around lines 79 - 85, The steps all start with "If" creating repetitive phrasing; rewrite the five bullet sentences to vary sentence openings while keeping their order and meaning—e.g., start some with conditionals ("When `raw_log_key` is set..."), others with actions ("Capture the full original record as a JSON string..."), and some with clauses ("When `log_key` is set, include only the specified keys..."); preserve references to `raw_log_key`, `log_key`, `time_key`, `add_tag`, and the injected `_tag`, and retain the semantics about not overwriting existing keys and RFC 3339 nanosecond precision for `time_key`.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pipeline/outputs/zerobus.md`:
- Around line 79-85: The steps all start with "If" creating repetitive phrasing;
rewrite the five bullet sentences to vary sentence openings while keeping their
order and meaning—e.g., start some with conditionals ("When `raw_log_key` is
set..."), others with actions ("Capture the full original record as a JSON
string..."), and some with clauses ("When `log_key` is set, include only the
specified keys..."); preserve references to `raw_log_key`, `log_key`,
`time_key`, `add_tag`, and the injected `_tag`, and retain the semantics about
not overwriting existing keys and RFC 3339 nanosecond precision for `time_key`.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 1abe0133-81b8-4c9e-af56-09c97a8e2033
📒 Files selected for processing (2)
SUMMARY.mdpipeline/outputs/zerobus.md
✅ Files skipped from review due to trivial changes (1)
- SUMMARY.md
|
@mats16 can you link the code PR this output plugin doc is dependent on? Also the linting / vale errors need to be cleaned up. Note that you can fix the spelling issue with ZeroBus in the docs by using backticks around it. I'll review once these are addressed. |
eschabell
added
waiting-for-user
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
pipeline/outputs/zerobus.md (2)
11-11: Prefer a technical Databricks docs link over the marketing homepage.This sentence links to
https://www.databricks.com/; for docs tone and verifiability, link to the specific Databricks technical documentation page for Zerobus/streaming ingestion instead.Based on learnings: In the fluent-bit-docs repository, ensure all Markdown documentation remains technical and neutral. Do not include marketing-style content or promotional links, even when citing statistics. Prefer objective explanations, verifiable sources, and avoid promotional language across all .md files.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pipeline/outputs/zerobus.md` at line 11, Update the external link in the "_Zerobus_ output plugin" description to point to a technical Databricks documentation page for Zerobus/streaming ingestion instead of the marketing homepage (replace the existing https://www.databricks.com/ URL); locate the sentence beginning "The _Zerobus_ output plugin lets you ingest log records..." in zerobus.md and swap the link to a specific Databricks docs URL that documents Zerobus or streaming ingestion so the doc remains technical and verifiable.
31-53: Add a short secret-handling note nearclient_secretexamples.The examples are correct, but a one-line warning to avoid committing real
client_secretvalues (and to use env vars/secret managers) would improve security posture for copy/paste users.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pipeline/outputs/zerobus.md` around lines 31 - 53, Add a one-line secret-handling warning next to the client_secret example in the fluent-bit.yaml zerobus output block: mention the client_secret field by name and instruct users not to commit real secrets to source control and to use environment variables or a secret manager (e.g., reference client_secret and fluent-bit.yaml/zerobus output) so copy/paste users see the security guidance.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pipeline/outputs/zerobus.md`:
- Around line 81-85: The repeated sentence starts ("If …") in the transformation
steps should be rewritten to remove the lint trigger: update the five bullet
points that reference raw_log_key, log_key, time_key, and add_tag (and mention
of injected key `_tag`) so they use varied lead-ins like "When set, …", "Setting
`raw_log_key` …", or "Enabling `add_tag` …", and combine or rephrase where
logical (e.g., describe raw_log_key behavior once and then describe the
injection rule) while preserving the same semantics about capturing the original
record as JSON, including only specified keys for `log_key`, injecting captured
JSON under `raw_log_key` unless the key exists, adding an RFC 3339 nanosecond
timestamp for `time_key` unless the key exists, and injecting the Fluent Bit tag
as `_tag` when `add_tag` is enabled.
---
Nitpick comments:
In `@pipeline/outputs/zerobus.md`:
- Line 11: Update the external link in the "_Zerobus_ output plugin" description
to point to a technical Databricks documentation page for Zerobus/streaming
ingestion instead of the marketing homepage (replace the existing
https://www.databricks.com/ URL); locate the sentence beginning "The _Zerobus_
output plugin lets you ingest log records..." in zerobus.md and swap the link to
a specific Databricks docs URL that documents Zerobus or streaming ingestion so
the doc remains technical and verifiable.
- Around line 31-53: Add a one-line secret-handling warning next to the
client_secret example in the fluent-bit.yaml zerobus output block: mention the
client_secret field by name and instruct users not to commit real secrets to
source control and to use environment variables or a secret manager (e.g.,
reference client_secret and fluent-bit.yaml/zerobus output) so copy/paste users
see the security guidance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d75aed7d-874c-4392-9e0c-a9022d680d5a
📒 Files selected for processing (1)
pipeline/outputs/zerobus.md
mats16
force-pushed
the
plugins/out_zerobus
branch
from
55d4b6d to
fb04ddc
Compare
- Updated instances of "ZeroBus" to "Zerobus" for consistency throughout the documentation. - Ensured accurate representation of the Zerobus output plugin and its configuration parameters. Signed-off-by: mats <mats.kazuki@gmail.com>
- Revised the description to use "through" instead of "via" for improved readability. - Clarified the OAuth2 terminology to "OAuth 2.0" for consistency. - Ensured consistent language throughout the documentation regarding Zerobus and its configuration parameters. Signed-off-by: mats <mats.kazuki@gmail.com>
mats16
force-pushed
the
plugins/out_zerobus
branch
from
fb04ddc to
8fd9703
Compare
|
@eschabell Thanks! The actual plugin implementation PR is here: fluent/fluent-bit#11678 |
|
@eschabell Hi! Just checking in on this PR in case I can help move it forward. Also, if there’s anything I should update on the docs side to make review easier, I’d be happy to do that. For the corresponding Fluent Bit PR, is there anyone in particular I should mention or follow up with? I’d appreciate any guidance on who the best reviewer or maintainer would be there. Thanks! |
- Added a new section detailing the build requirements for the Zerobus output plugin. - Included instructions for installing the `zerobus-ffi` shared library and header. - Provided specific file placement instructions for successful compilation from source. Signed-off-by: mats <mats.kazuki@gmail.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pipeline/outputs/zerobus.md`:
- Around line 95-99: The listed transformation steps in
pipeline/outputs/zerobus.md repeat the same sentence start ("If..."), causing a
lint/style warning; rephrase the five bullet points to vary sentence structure
while preserving meaning: reference raw_log_key and log_key behavior (full
original record captured as JSON string vs. inclusion of specified keys),
explain that the captured JSON is injected under raw_log_key unless that key
exists, state that time_key injects an RFC3339 timestamp with nanosecond
precision unless the key exists, and note that enabling add_tag injects the
Fluent Bit tag as _tag unless that key exists; update the wording for each
bullet so they don't all begin with "If" (e.g., use active phrases like "When
set," "When configured," "The captured JSON is injected," "A timestamp is
added," "Enabling add_tag injects") to remove the repeated pattern.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 72f560f0-15dd-45ba-95af-8f3e6764eddc
📒 Files selected for processing (1)
pipeline/outputs/zerobus.md
| 1. If `raw_log_key` is set, the full original record is captured as a JSON string before any filtering. | ||
| 2. If `log_key` is set, only the specified keys are included in the output record. | ||
| 3. If `raw_log_key` is set, the captured JSON string is injected under the configured key (unless a key with that name already exists). | ||
| 4. If `time_key` is set, a timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected (unless a key with that name already exists). | ||
| 5. If `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists). |
There was a problem hiding this comment.
Fix the repeated sentence starts to resolve the lint warning.
The transformation steps repeatedly start with "If...", which triggers a style warning. This issue was flagged in a previous review and marked as addressed, but the current code still contains the repeated pattern. The PR objectives mention that linting errors need to be fixed.
✍️ Suggested rewording to eliminate repetition
-1. If `raw_log_key` is set, the full original record is captured as a JSON string before any filtering.
-2. If `log_key` is set, only the specified keys are included in the output record.
-3. If `raw_log_key` is set, the captured JSON string is injected under the configured key (unless a key with that name already exists).
-4. If `time_key` is set, a timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected (unless a key with that name already exists).
-5. If `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists).
+1. When `raw_log_key` is set, the full original record is captured as a JSON string before any filtering.
+2. When `log_key` is set, only the specified keys are included in the output record.
+3. The captured JSON string (from step 1) is then injected under the configured `raw_log_key` (unless a key with that name already exists).
+4. Next, when `time_key` is set, a timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected (unless a key with that name already exists).
+5. Finally, when `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists).📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| 1. If `raw_log_key` is set, the full original record is captured as a JSON string before any filtering. | |
| 2. If `log_key` is set, only the specified keys are included in the output record. | |
| 3. If `raw_log_key` is set, the captured JSON string is injected under the configured key (unless a key with that name already exists). | |
| 4. If `time_key` is set, a timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected (unless a key with that name already exists). | |
| 5. If `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists). | |
| 1. When `raw_log_key` is set, the full original record is captured as a JSON string before any filtering. | |
| 2. When `log_key` is set, only the specified keys are included in the output record. | |
| 3. The captured JSON string (from step 1) is then injected under the configured `raw_log_key` (unless a key with that name already exists). | |
| 4. Next, when `time_key` is set, a timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected (unless a key with that name already exists). | |
| 5. Finally, when `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists). |
🧰 Tools
🪛 LanguageTool
[style] ~97-~97: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...s are included in the output record. 3. If raw_log_key is set, the captured JSON...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~98-~98: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... key with that name already exists). 4. If time_key is set, a timestamp in RFC 3...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~99-~99: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... key with that name already exists). 5. If add_tag is enabled, the Fluent Bit ta...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@pipeline/outputs/zerobus.md` around lines 95 - 99, The listed transformation
steps in pipeline/outputs/zerobus.md repeat the same sentence start ("If..."),
causing a lint/style warning; rephrase the five bullet points to vary sentence
structure while preserving meaning: reference raw_log_key and log_key behavior
(full original record captured as JSON string vs. inclusion of specified keys),
explain that the captured JSON is injected under raw_log_key unless that key
exists, state that time_key injects an RFC3339 timestamp with nanosecond
precision unless the key exists, and note that enabling add_tag injects the
Fluent Bit tag as _tag unless that key exists; update the wording for each
bullet so they don't all begin with "If" (e.g., use active phrases like "When
set," "When configured," "The captured JSON is injected," "A timestamp is
added," "Enabling add_tag injects") to remove the repeated pattern.
2 participants
Summary
out_zerobusoutput plugin that sends logs to Databricks tables via the ZeroBus streaming ingestion interfaceendpoint,workspace_url,table_name,client_id,client_secret,add_tag,time_key,log_key,raw_log_key)Test plan
This pull request was AI-assisted by Claude.
Summary by CodeRabbit