in_forward: handle retain metadata in upstream conf

[cosmo0920]

Fixes #11617.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

Release Notes

• New Features

  • Added retain_metadata_in_forward_mode configuration option to control metadata preservation during forward operations. When enabled (default: disabled), metadata including schema information and event attributes is retained when forwarding logs and signal events.

• Tests

  • Added comprehensive integration and runtime test scenarios to validate metadata retention behavior in forward mode, including upstream node configurations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f7e5e901-d241-43d5-a145-f33908028ca9

📥 Commits

Reviewing files that changed from the base of the PR and between 29deec9 and 76e1bbe.

📒 Files selected for processing (5)

• plugins/out_forward/forward.c
• tests/integration/scenarios/in_forward/config/in_opentelemetry_to_forward_receiver_retain_metadata.yaml
• tests/integration/scenarios/in_forward/tests/test_in_forward_001.py
• tests/runtime/data/forward/upstream_retain_metadata.conf
• tests/runtime/group_counter_semantics.c

---

📝 Walkthrough

Walkthrough

This PR implements support for the retain_metadata_in_forward_mode configuration option on upstream node configurations. The core plugin change reads this property and controls metadata retention during forward mode transmission. Comprehensive integration and unit tests validate the functionality across different configuration scenarios.

Changes

Cohort / File(s)	Summary
Core Implementation plugins/out_forward/forward.c	Added retain_metadata_in_forward_mode property parsing in config_set_properties() that sets fc->fwd_retain_metadata flag to control metadata retention during forward protocol transmission.
Integration Test Configuration tests/integration/scenarios/in_forward/config/in_opentelemetry_to_forward_receiver_retain_metadata.yaml	New test configuration with OpenTelemetry input piped to Forward output, enabling retain_metadata_in_forward_mode and send_options for end-to-end metadata retention validation.
Integration Test tests/integration/scenarios/in_forward/tests/test_in_forward_001.py	Added test_out_forward_logs_signal_e2e_with_forward_receiver_retain_metadata() to validate that forwarded records retain metadata structures including schema, OTLP attributes, and severity information.
Runtime Test Configuration tests/runtime/data/forward/upstream_retain_metadata.conf	New upstream configuration file with a node that enables retain_metadata_in_forward_mode and send_options for testing upstream-based configuration paths.
Runtime Test tests/runtime/group_counter_semantics.c	Added flb_test_forward_group_size_retain_metadata_upstream_node() to verify group size semantics when metadata retention is configured via upstream files.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

docs-required

Suggested reviewers

• edsiper
• koleini

Poem

🐰 Hops through the forward tunnels bright,
Metadata clutched, held tight,
Upstream nodes now understand,
Preserve the data, as they planned! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title "in_forward: handle retain metadata in upstream conf" clearly and specifically describes the main change: handling the retain_metadata_in_forward_mode parameter when declared in upstream NODE configuration blocks.
Linked Issues check	✅ Passed	The PR addresses issue #11617 by implementing the retain_metadata_in_forward_mode parameter handling in upstream NODE blocks, adding configuration support in forward.c, integration test coverage, and runtime tests validating metadata retention across forward hops.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to fix the retain_metadata_in_forward_mode upstream configuration issue: core config handling, integration test configuration/test, and runtime test configuration/test. No extraneous changes detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cosmo0920-handle-retain-metadata-in-upstream-conf

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[edsiper]

side question: why retain_metadata_in_forward_mode is not true by default ?

[cosmo0920]

side question: why retain_metadata_in_forward_mode is not true by default ?

There's a historical reason and performance reason.
The commit is here:
0f3c2a5

From the commit, when using retain_metadata_in_forward_mode is true, we need to transcode to fit logs type of transportation format of forward protocol. When not using this parameter, we needn't do that. So, I suppose the parameter not to be true by default.