opentelemetry: render unset AnyValue in OTLP JSON

[edsiper]

This fixes an OTLP JSON rendering failure in the Kafka output path when a valid OTLP logs payload contains an unset AnyValue in resource, scope, or record attributes.

Kafka format: otlp_json reuses the shared OTLP logs JSON renderer. The protobuf OTLP input accepts unset AnyValue fields and stores them internally as msgpack nil, but the renderer treated nil attribute values as conversion failures while building OTLP key/value arrays. That made the Kafka output drop the chunk with:

[error] [output:kafka:kafka.0] could not convert event chunk to OTLP JSON: -2

The renderer now emits unset attribute values as an empty OTLP AnyValue object ({}), while still omitting an absent/nil log body when the whole body candidate is nil.

This also adds Kafka integration coverage for:

• protobuf OTLP logs with unset resource, scope, and record attribute values;
• monolithic single-resource OTLP JSON payloads with otlp_logs_partition_by_resource: true;
• oversized Kafka messages proving the failure is reported as broker message-size rejection, not OTLP JSON conversion failure.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

tests/integration/scenarios/out_kafka/config/out_kafka_otlp_json_partition_by_resource_small_message_max.yaml

• Debug log output from testing the change

Before the fix, the new integration test reproduced the failure:

tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py::test_out_kafka_otlp_json_logs_preserves_unset_attribute_values -q

Result: failed waiting for Kafka output. Fluent Bit log contained:

[error] [output:kafka:kafka.0] could not convert event chunk to OTLP JSON: -2

After the fix:

tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py::test_out_kafka_otlp_json_logs_preserves_unset_attribute_values tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py::test_out_kafka_otlp_json_partition_by_resource_keeps_monolithic_resource_valid tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py::test_out_kafka_otlp_json_partition_by_resource_rejects_oversized_message -q

Result: 3 passed.

• Attached Valgrind output that shows no leaks or memory corruption was found

VALGRIND=1 VALGRIND_STRICT=1 tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py::test_out_kafka_otlp_json_logs_preserves_unset_attribute_values tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py::test_out_kafka_otlp_json_partition_by_resource_keeps_monolithic_resource_valid tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py::test_out_kafka_otlp_json_partition_by_resource_rejects_oversized_message -q

Result: 3 passed. Each focused valgrind log reported ERROR SUMMARY: 0 errors.

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [N/A] Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Bug Fixes

  • OTLP JSON conversion now properly handles null-like input values by converting them to empty objects instead of producing conversion failures.

• Tests

  • Added integration tests for OTLP-to-Kafka pipeline scenarios, including validation of unset attribute handling, resource partitioning behavior, and oversized message rejection.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ae7c1628-7899-4955-9e6c-9db0310e4be4

📥 Commits

Reviewing files that changed from the base of the PR and between 47db630 and 99fe6f7.

📒 Files selected for processing (4)

• src/opentelemetry/flb_opentelemetry_otlp_json.c
• tests/integration/scenarios/out_kafka/config/out_kafka_otlp_json_partition_by_resource_small_message_max.yaml
• tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py
• tests/integration/src/server/kafka_server.py

---

📝 Walkthrough

Walkthrough

This PR hardens OTLP JSON output conversion by treating null/nil message pack inputs as empty OTLP "Any" objects instead of returning NULL, adds helpers for monolithic OTLP log generation and log polling, introduces integration tests for unset value preservation and partition-by-resource behavior, and improves Kafka test server robustness against connection resets.

Changes

OTLP JSON Null-Value Handling and Validation

Layer / File(s)	Summary
Null-value handling in OTLP JSON converter src/opentelemetry/flb_opentelemetry_otlp_json.c	Adds empty_otlp_any_value() and updates msgpack_array_to_otlp_any_value(), msgpack_map_to_otlp_kv_array(), msgpack_map_to_otlp_kv_array_filtered(), and cfl_variant_to_otlp_any_value() to return OTLP Any objects for null/nil inputs instead of NULL.
Test support infrastructure for OTLP/Kafka scenarios tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py, tests/integration/scenarios/out_kafka/config/out_kafka_otlp_json_partition_by_resource_small_message_max.yaml, tests/integration/src/server/kafka_server.py	Adds time import, _build_monolithic_logs_payload() and _wait_for_log_text() helpers, Kafka YAML configuration for partition-by-resource with message size constraints, and ConnectionResetError handling in Kafka server test helper.
Integration tests for null preservation and partition-by-resource tests/integration/scenarios/out_kafka/tests/test_out_kafka_001.py	Validates that unset attribute values serialize as empty objects while other attributes remain intact, and verifies monolithic log payloads preserve single-resource validity under partition-by-resource; also asserts oversized payloads are rejected with appropriate logging.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

• cosmo0920

Poem

🐰 Null values hop no more to oblivion's door,
Now empty Any objects bloom at the core!
With tests to ensure preservation shines bright,
And partition-by-resource keeps payloads just right.
What once was a NULL now dances with grace,
In OTLP JSON's beautifully robust place! 🌟

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'opentelemetry: render unset AnyValue in OTLP JSON' directly and precisely describes the main technical change—converting unset null-like inputs into empty OTLP AnyValue objects in JSON rendering.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch kafka-otlp-json-size-6552

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}