out_azure_logs_ingestion: expose request payload metrics

[nourdouf]

Summary

Add request payload metrics to out_azure_logs_ingestion without changing request formatting, delivery, retry, timeout, or acknowledgement behavior.

The metrics are recorded immediately before each call to flb_http_do(), so they represent actual client-side HTTP attempts, including retries.

Metrics

Uncompressed payload size

fluentbit_azure_logs_ingestion_uncompressed_payload_size_bytes

Histogram containing the size of the formatted JSON payload before compression.

HTTP payload size

fluentbit_azure_logs_ingestion_http_payload_size_bytes

Histogram containing the actual HTTP request-body size:

• gzip bytes when compression succeeds;
• JSON bytes when compression is disabled or gzip fails.

Both histograms have these labels:

• name: Fluent Bit output instance or alias;
• dcr_id: Azure Data Collection Rule immutable ID.

Bucket boundaries include 200 KiB, 900,000 bytes, and 1 MiB, and extend through 16 MiB for visibility into oversized requests.

Metric initialization is best-effort. If metric allocation fails, the output logs a warning and continues sending logs.

Request rate

The HTTP histogram's _count series increments once per HTTP attempt. Request-attempt rate can be calculated with:

sum by (dcr_id) (
  rate(fluentbit_azure_logs_ingestion_http_payload_size_bytes_count[5m])
)

Compression ratio

The weighted compression ratio is the ratio between the HTTP and uncompressed histogram sums:

sum by (dcr_id) (
  rate(fluentbit_azure_logs_ingestion_http_payload_size_bytes_sum[5m])
)
/
sum by (dcr_id) (
  rate(fluentbit_azure_logs_ingestion_uncompressed_payload_size_bytes_sum[5m])
)

For example, a value of 0.12 means the HTTP request bodies are 12% of the original JSON size. Ratios above 1.0 are valid for small payloads where the gzip envelope adds bytes.

Validation

• Current Fluent Bit build passed.
• Azure Logs Ingestion integration test passed.
• The same test passed under strict macOS Leaks.
• Commit-prefix and DCO checks passed.
• The runtime patch applies with zero fuzz and builds on Fluent Bit v5.1.1 and v5.1.2.

The integration test covers a compressed request, a deterministic HTTP 500 retry followed by success, DCR labels, histogram counts and sums, compression-ratio derivation, and the 200 KiB bucket.

This pull request was developed with AI assistance.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[nourdouf]

Downstream batching compatibility note:

The redesigned batching implementation in #12374 has been materialized on a clean branch based directly on this PR's head 07de56883:

• branch: nourdouf/azure-logs-ingestion-batching-on-request-metrics
• head: 33ff6ace2

The batching-only diff applies with zero fuzz on top of this metrics patch, the stacked Fluent Bit tree builds, and focused payload-metrics/batching/retry/persistence/WAL tests pass (5 passed).

Packaging should keep this ordering: request metrics first, batching second. The batching layer retains these two histogram families unchanged and adds separate fixed-cardinality lifecycle metrics for buffered delivery.

[nourdouf]

Stacked-branch update: the batching-on-metrics head is now 179830c43. The follow-up guards empty worker callbacks used by fluent-bit --dry-run; stacked build, buffered config dry-run, and the previously reported five focused stacked tests pass.