docs: update Kafka MSK IAM authentication configuration

[kalavt]

• Replace deprecated aws_msk_iam and aws_msk_iam_cluster_arn parameters
• Add new rdkafka.sasl.mechanism=aws_msk_iam configuration method
• Add aws_region parameter for custom DNS/PrivateLink scenarios
• Update IAM permission examples for consumers and producers
• Add MSK Serverless example in input plugin
• Simplify documentation for user-friendly experience

Summary by CodeRabbit

• Documentation
  • Rewrote and expanded AWS MSK IAM authentication guidance for Kafka (corrected plugin context to output where applicable) with clearer prerequisites and a configuration-first flow.
  • Added concrete YAML and conf examples and a tabbed example workflow for easier reference.
  • Added explicit AWS credentials discovery steps, PrivateLink/custom DNS guidance, and notes on automatic configuration behavior.
  • Provided scoped IAM policy examples with ARN placeholders and substitution guidance.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Documentation for AWS MSK IAM authentication in the Kafka input and output docs was rewritten and expanded: sections reorganized, configuration parameters and examples added, credential guidance clarified, and a concrete IAM policy with explicit ARNs and placeholders provided.

Changes

Cohort / File(s)

Summary

AWS MSK IAM Documentation (inputs & outputs) pipeline/inputs/kafka.md, pipeline/outputs/kafka.md

Rewrote MSK IAM authentication guidance: changed header wording to “Starting with version 4.0.4…”, renamed "Build requirements" → "Prerequisites", added a configuration-parameters table (rdkafka.sasl.mechanism, aws_region), provided Fluent Bit YAML and .conf examples (including custom DNS/PrivateLink guidance), replaced generic credential guidance with an AWS credentials chain section and enumerated required IAM permissions, and substituted the broad example policy with a concrete JSON IAM policy scoped to explicit MSK ARNs using placeholders (REGION, ACCOUNT, CLUSTER_NAME, CLUSTER_UUID).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

• Verify IAM policy JSON and ARN placeholder formats and accuracy
• Confirm YAML and Fluent Bit .conf examples are syntactically correct
• Check consistency between input and output documents and links

Suggested labels

4.2.1

Poem

🐰 I hopped through docs to make things clear,

Replaced the vague with examples near,
ARNs and YAML neatly in a row,
Credentials lined up, ready to go,
A little carrot for the Kafka flow 🥕

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title accurately summarizes the main documentation update: changing Kafka MSK IAM authentication configuration. It is concise, specific, and directly reflects the primary changes across both input and output plugin documentation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (4)

pipeline/outputs/kafka.md (4)

5-5: Fix plugin reference: output doc incorrectly says “Kafka input plugin”.
Line 5 should refer to the Kafka output plugin (producer), not the input plugin.

-In Fluent Bit 4.0.4 and later, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
+In Fluent Bit 4.0.4 and later, the Kafka output plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.

---

27-27: Typo: librdfkafka → librdkafka.
Line 27 has a misspelling that may confuse readers searching for the right library knobs.

-Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to `1` are examples of recommended settings of `librdfkafka` properties.
+Setting `rdkafka.log.connection.close` to `false` and `rdkafka.request.required.acks` to `1` are examples of recommended settings of `librdkafka` properties.

---

48-58: Example config uses host (and a malformed IP) instead of brokers.
Line 56 has 192.1681.3:9092 (missing dot) and the key should match the documented parameter brokers.

   outputs:
     - name: kafka
       match: '*'
-      host: 192.1681.3:9092
+      brokers: 192.168.1.3:9092
       topics: test

---

191-191: Minor formatting: add space in “raw format”.

-#### Kafka configuration file with `raw`format
+#### Kafka configuration file with `raw` format

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3fea32e and 137bda0.

📒 Files selected for processing (2)

• pipeline/inputs/kafka.md (1 hunks)
• pipeline/outputs/kafka.md (1 hunks)

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pipeline/outputs/kafka.md (1)

5-6: Doc bug: “Kafka input plugin” should be “Kafka Producer output plugin” here.
This is in the producer/output page and is likely a copy/paste slip.

🧹 Nitpick comments (1)

pipeline/inputs/kafka.md (1)

149-165: Config parameters table/examples: consider explicitly stating whether aws_msk_iam_cluster_arn is no longer needed (and why).
Right now the table + examples imply rdkafka.sasl.mechanism=aws_msk_iam is sufficient, but users familiar with older docs will look for the removed ARN parameter and may be unsure what changed.

Also applies to: 173-183

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 137bda0 and 4db76b7.

📒 Files selected for processing (2)

• pipeline/inputs/kafka.md (1 hunks)
• pipeline/outputs/kafka.md (1 hunks)

🔇 Additional comments (2)

pipeline/inputs/kafka.md (1)

195-225: IAM permissions + ARN formats look solid and much safer (scoped resources + CLUSTER_UUID note).
The consumer actions list (DescribeGroup/AlterGroup) and the CLUSTER_UUID guidance are the right level of specificity.

pipeline/outputs/kafka.md (1)

347-374: Producer IAM policy example is clear and appropriately scoped.
Nice improvement over overly-broad Resource: "*" style examples.

[coderabbitai]

Actionable comments posted: 4

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9101aab and a767991.

📒 Files selected for processing (2)

• pipeline/inputs/kafka.md (1 hunks)
• pipeline/outputs/kafka.md (2 hunks)

🔇 Additional comments (2)

pipeline/inputs/kafka.md (1)

199-227: Approved: IAM permissions policy is now correct.

The policy correctly includes the cluster UUID in ARN paths and specifies all required actions for a consumer (Connect, DescribeTopic, ReadData, DescribeGroup, AlterGroup). The guidance to substitute REGION, ACCOUNT, CLUSTER_NAME, and CLUSTER_UUID is clear.

pipeline/outputs/kafka.md (1)

351-376: Approved: Producer IAM permissions policy is correct.

The policy correctly specifies producer-only actions (Connect, DescribeTopic, WriteData) with cluster and topic ARNs that include the CLUSTER_UUID segment. The guidance to substitute placeholders is clear.

[eschabell]

@alexakreizinger review request for you!

[alexakreizinger]

I want to give this my full attention, but I have to finish early for the day for an appointment so I'm setting a reminder to review it on Monday 😄