out_es: add support for dynamic index using record accessor patterns

[SuRyaMane389]

This PR adds dynamic Elasticsearch index resolution in out_es using record accessor patterns on the index field (example: $myindex), enabling per-record index routing without duplicating output sections.

What changed:

• Added index record accessor support in ES output context:
  • Added ra_index field to output context.
  • Parse and initialize record accessor when index contains accessor syntax.
  • Release ra_index during plugin teardown.
• Updated bulk formatter logic:
  • When dynamic index is enabled, resolve target index per record at format time.
  • Preserve existing behavior for logstash_format, generate_id, id_key, suppress_type_name.
  • Fallback behavior: if accessor translation fails for a record, use configured static index and log warning.
• Added runtime test coverage:
  • Dynamic index basic behavior.
  • Dynamic index with suppress_type_name.
  • Dynamic index with id_key.
  • Dynamic index with generate_id.

Definitions:

• Record accessor pattern:
  • A field expression that extracts values from a record at runtime (for example, $myindex).
• Dynamic index:
  • The _index value in Elasticsearch bulk metadata is computed per event rather than fixed per output.
• Static fallback:
  • If accessor translation fails, plugin keeps pipeline alive by using configured default index.

Architectural impact:

• Scope-limited to output plugin out_es.
• No protocol/API contract changes outside ES bulk metadata composition.
• Backward compatibility preserved for existing static index and logstash-based index generation paths.
• Operational benefit: multi-tenant or domain-routed indexing from one output instance.

Example behavior:

• Input record:
  • {"message":"ok","myindex":"tenant-a-logs","key":"abc"}
• Configuration:
  • index $myindex
  • type _doc or suppress_type_name true
• Bulk action metadata generated:
  • {"create":{"_index":"tenant-a-logs","_type":"_doc"}}
  • or without type when suppress_type_name is enabled.

Example configuration file:

[SERVICE]
    Flush       1
    Log_Level   debug

[INPUT]
    Name        dummy
    Tag         app.test
    Dummy       {"message":"ok","myindex":"tenant-a-logs","key":"abc"}

[OUTPUT]
    Name        es
    Match       app.test
    Host        127.0.0.1
    Port        9200
    Index       $myindex
    Type        _doc
    Generate_ID true

Issue:
[N/A]

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Elasticsearch output can derive index names from record fields using pattern-based references.

• Bug Fixes

  • If dynamic index resolution fails, the plugin now warns and reliably falls back to the configured static index.

• Tests

  • Added runtime tests covering dynamic index selection, type suppression, custom ID, and auto-generated ID behaviors.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b5be885b-1c5c-4e5c-ad22-4ed8b45444af

📥 Commits

Reviewing files that changed from the base of the PR and between 958eee3 and 991fd7c.

📒 Files selected for processing (2)

• plugins/out_es/es.c
• plugins/out_es/es_conf.c

🚧 Files skipped from review as they are similar to previous changes (2)

• plugins/out_es/es_conf.c
• plugins/out_es/es.c

---

📝 Walkthrough

Walkthrough

Adds record-accessor-based dynamic index selection to the Elasticsearch output: config compiles an index accessor when the index contains $, formatting translates the index per-record and rebuilds bulk headers with proper cleanup on all paths, and runtime tests validate index/type/ID variations.

Changes

Dynamic index selection via record accessors

Layer / File(s)	Summary
Data structure definition plugins/out_es/es.h	The flb_elasticsearch struct adds a ra_index field to store the compiled record accessor for dynamic index patterns.
Configuration initialization and cleanup plugins/out_es/es_conf.c	Configuration compiles ctx->index into ctx->ra_index when it contains $; failure logs an error and aborts creation. Destroy frees ctx->ra_index when present.
Per-record dynamic index translation and formatting plugins/out_es/es.c	Formatting skips header pre-generation when ctx->ra_index is set, translates the index per-record via flb_ra_translate(), rebuilds the bulk header with the translated index (respecting suppress_type_name), and ensures the translated string is freed on all error and success paths.
Test callbacks and runtime tests tests/runtime/out_elasticsearch.c	Adds formatter verification callbacks and four runtime tests that configure index = "$myindex" and vary suppress_type_name, id_key, and generate_id; tests validate generated bulk headers and IDs and register in TEST_LIST.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I hopped through records, fetched the hidden name,
Built headers per-note, no memory left to blame,
Tests danced in chorus, indexes found their place,
Cleanups closed the loop, not a leak left in the space,
Hooray — dynamic routing, neat as carrot grace!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding support for dynamic index selection in the Elasticsearch output plugin using record accessor patterns.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 958eee31b6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Fall back when the index accessor resolves empty

When a record does not contain the configured accessor key, flb_ra_translate() is called with check disabled and returns an allocated empty SDS rather than NULL; this if (!ra_index) check is therefore skipped and the bulk metadata is emitted with _index":"" instead of using a safe fallback. Any mixed stream where one record lacks myindex will produce an Elasticsearch bulk item that the server rejects, so this should either use flb_ra_translate_check() or treat zero-length translations as missing.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Resolve record-accessor indexes before current-time formatting

With both Index $myindex and Current_Time_Index On, this new ra_index branch is never reached because the preceding current_time_index branch formats the literal ctx->index string. That means otherwise valid dynamic-index configurations that already enable current_time_index send _index":"$myindex" for every record instead of the per-record value.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Accept lowercase generated IDs in the new runtime test

The formatter generates IDs with %04x, which emits lowercase hexadecimal a-f, but this new check only permits uppercase A-F, digits, and dashes. As soon as the hash contains any lowercase hex letter, index_record_accessor_generate_id fails even though the generated _id is valid, breaking the added runtime coverage.

Useful? React with 👍 / 👎.