README-implementation

OpenTelemetry Filter Implementation Review
======================================================================

1  Overview
----------------------------------------------------------------------

The OpenTelemetry (OTel) filter for HAProxy provides distributed tracing,
metrics and logging capabilities.  It creates, propagates and exports spans,
metric instruments and log records that follow the OpenTelemetry specification.
The filter hooks into the HAProxy stream processing pipeline through the
filter API and maps HAProxy channel analyzer events to OpenTelemetry span
lifecycle operations, metric recordings and log-record emissions.

The implementation lives at the top of this addon checkout, consisting of header
files, C source files, a Makefile.mk fragment, and a set of test configurations
with runner scripts.


2  Directory Structure
----------------------------------------------------------------------

  haproxy-opentelemetry/
  |-- Makefile.mk        Build integration (loaded via EXTRA_MAKE)
  |-- include/
  |   |-- include.h      Master include (pulls all headers)
  |   |-- config.h       Build-time tunables (pool sizes, limits)
  |   |-- define.h       Utility macros (memory, strings, lists)
  |   |-- debug.h        Debug/logging infrastructure
  |   |-- filter.h       Filter return codes, alert macros
  |   |-- parser.h       Configuration keyword definitions
  |   |-- conf.h         Configuration data structures
  |   |-- conf_funcs.h   Generated init/free function macros
  |   |-- event.h        Event enumeration and data table
  |   |-- scope.h        Runtime span/context structures
  |   |-- pool.h         Memory pool helpers
  |   |-- http.h         HTTP header manipulation
  |   |-- otelc.h        Span context inject/extract wrappers
  |   |-- vars.h         HAProxy variable integration
  |   |-- util.h         String conversion, sample helpers
  |   |-- group.h        Group action (HAProxy rule integration)
  |   `-- cli.h          CLI command interface
  |-- src/
  |   |-- filter.c       Filter lifecycle and channel callbacks
  |   |-- parser.c       Configuration file parser
  |   |-- conf.c         Configuration structure init/free
  |   |-- event.c        Scope/span execution engine
  |   |-- scope.c        Runtime context and span management
  |   |-- http.c         HTTP header get/set/remove
  |   |-- otelc.c        C wrapper inject/extract bridge
  |   |-- vars.c         HAProxy variable read/write
  |   |-- pool.c         Pool alloc/free, trash buffers
  |   |-- util.c         Argument handling, sample conversion
  |   |-- group.c        Group action parsing and execution
  |   `-- cli.c          CLI command handlers
  `-- test/
      |-- copy-yml.sh    YAML configuration transformer
      |-- test-speed.sh  Performance benchmarking runner
      |-- run-sa.sh      Standalone test runner
      |-- run-fe-be.sh   Frontend-backend chain runner
      |-- run-ctx.sh     Context propagation test runner
      |-- run-cmp.sh     Comparison test runner
      |-- sa/            Standalone test configs
      |-- fe/            Frontend-only test configs
      |-- be/            Backend-only test configs
      |-- ctx/           Context propagation test configs
      |-- cmp/           Comparison test configs
      `-- empty/         Minimal/empty configuration test


3  Build System
----------------------------------------------------------------------

The Makefile.mk fragment is pulled in by the HAProxy top-level Makefile via the
EXTRA_MAKE variable, which lists addon directories whose Makefile.mk fragments
are included unconditionally.  The fragment detects the opentelemetry-c-wrapper
library via pkg-config or manual OTEL_INC/OTEL_LIB paths.

Build options:

  EXTRA_MAKE=<path> Addon directory containing Makefile.mk.  Listing this
                    directory is what enables the filter.  Multiple directories
                    may be passed, space-separated.
  OTEL_DEBUG=1      Compile with DEBUG_OTEL; links the _dbg variant of the
                    wrapper library and enables additional debug callbacks in
                    filter.c (stream_set_backend, http_headers, http_payload,
                    tcp_payload, etc.).
  OTEL_USE_VARS=1   Compile vars.c; enables USE_OTEL_VARS which allows span
                    context propagation via HAProxy transaction variables in
                    addition to HTTP headers.
  OTEL_INC=<path>   Manual include path for the C wrapper.
  OTEL_LIB=<path>   Manual library path for the C wrapper.
  OTEL_RUNPATH=1    Embed RPATH to the wrapper library.
  OTEL_STATIC=1     Pass --static to pkg-config so OTEL_LDFLAGS picks up the
                    full Libs.private chain; needed for static linking.

Compiled objects (11 always, 12 with OTEL_USE_VARS):

  cli.o  conf.o  event.o  filter.o  group.o  http.o  opentelemetry.o  parser.o
  pool.o  scope.o  util.o  [vars.o]


4  Configuration Parsing
----------------------------------------------------------------------

Configuration parsing is driven by parser.c.  The filter is declared in the
HAProxy configuration with:

  filter opentelemetry [id <name>] config <file>

The flt_otel_parse() function (parser.c) handles the "filter" line, creates an
flt_otel_conf structure, and delegates to parse_cfg() which loads the referenced
YAML/CFG file.  That file is parsed using temporary section registrations for
three section types:

  otel-instrumentation   ->  flt_otel_parse_cfg_instr()
  otel-group             ->  flt_otel_parse_cfg_group()
  otel-scope             ->  flt_otel_parse_cfg_scope()

After each section is fully parsed, a post-parse function validates the section
(e.g., flt_otel_post_parse_cfg_scope() checks that context injection is only
used on events that support it).

4.1  Instrumentation Section

  otel-instrumentation <name>
      config <file>
      log <target>
      debug-level <value>
      rate-limit <value>
      option { disabled | hard-errors | dontlog-normal }
      groups <name> ...
      scopes <name> ...
      acl <aclname> <criterion> ...

The instrumentation block defines global filter parameters: the YAML exporter
configuration file, logging, rate limiting, and references to groups and scopes.
Exactly one instrumentation block is allowed per filter instance.

4.2  Group Section

  otel-group <name>
      scopes <name> ...

Groups bundle multiple scopes under a single name for use with HAProxy
http-request/http-response rules via the "otel-group" action.  The group action
(group.c) parses the rule, resolves the scope references at check time, and
executes all referenced scopes when the rule fires.

4.3  Scope Section

  otel-scope <name>
      otel-event <event-name> [{ if | unless } <condition>]
      extract <name-prefix> [use-vars | use-headers]
      span <name> [parent <ref>] [link <ref>] [root] [kind <kind>]
        link { <ref> ... | <ref> attr <key> <sample> ... }
        attribute <key> <sample> ... [{ if | unless } <condition>]
        event <name> [time [<unit>] <sample>] <key> <sample> ... [{ if | unless } <condition>]
        baggage <key> <sample> ... [{ if | unless } <condition>]
        status <code> [<sample> ...] [{ if | unless } <condition>]
        inject <name-prefix> [use-vars] [use-headers]
      finish <name> ...
      instrument <type> <name> ... / instrument update <name> ...
      log-record <severity> [id <integer> event <name>] [time [<unit>] <sample>] [span <ref>] [attr <key> <sample>] ... <sample> ...
      acl <aclname> <criterion> ...
      otel-stop [{ if | unless } <condition>]

Each scope ties to a single HAProxy analyzer event (or none, if used only
through groups).  Scopes contain context extraction directives, span
definitions, metric instruments, log records, finish and otel-stop directives.

A span may specify:
  - A parent reference (another span or extracted context).
  - One or more links to other spans/contexts.  Inline link syntax allows one
    link on the span line; the standalone "link" keyword allows multiple.
  - The "root" flag marking it as the trace root.
  - Attributes, events, baggages and status evaluated from HAProxy sample
    expressions at runtime.
  - An inject directive to propagate the span context via HTTP headers and/or
    HAProxy variables.

4.4  Configuration Structure Initialization

All configuration structures are allocated and freed using macro-generated
functions from conf_funcs.h:

  FLT_OTEL_CONF_FUNC_INIT(type, id_field, extra_init)
  FLT_OTEL_CONF_FUNC_FREE(type, id_field, extra_free)

These macros produce flt_otel_conf_<type>_init() and _free() functions.
The init function:
  - Checks the identifier length against FLT_OTEL_ID_MAXLEN (64).
  - Checks for duplicate identifiers in the target list.
  - Allocates the structure with OTELC_CALLOC.
  - Copies the identifier with OTELC_STRDUP.
  - Appends to the head list.
  - Executes any extra initialization (e.g., LIST_INIT for sub-lists in the
    span structure).

The free function:
  - Executes any extra cleanup (e.g., destroying sub-lists).
  - Frees the identifier string.
  - Removes the node from its list.
  - Frees the structure.

The full init/free chain for all structures:

  flt_otel_conf          flt_otel_conf_init() / flt_otel_conf_free()
    flt_otel_conf_instr  generated via macro
      flt_otel_conf_ph   generated (for ph_groups, ph_scopes)
    flt_otel_conf_group  generated
      flt_otel_conf_ph   generated (for ph_scopes)
    flt_otel_conf_scope  generated
      flt_otel_conf_context    generated
      flt_otel_conf_span       generated
        flt_otel_conf_link       generated
        flt_otel_conf_sample     generated + _init_ex()
          flt_otel_conf_sample_expr  generated
      flt_otel_conf_instrument generated
      flt_otel_conf_log_record generated
        flt_otel_conf_sample     generated + _init_ex()
          flt_otel_conf_sample_expr  generated


5  Filter Lifecycle
----------------------------------------------------------------------

The filter registers its operations in the flt_otel_ops structure (filter.c)
and the keyword parser via INITCALL1 (parser.c).

5.1  Proxy-Level Initialization

  flt_otel_ops_init():
    - Registers CLI commands via flt_otel_cli_init().
    - Initializes the OpenTelemetry library via flt_otel_lib_init(): verifies
      the C wrapper version, resolves the absolute path of the YAML
      configuration file, calls otelc_init() to set up exporters, creates the
      tracer, meter and logger objects, and registers custom memory allocation
      and thread-id callbacks with the wrapper via otelc_ext_init().

  flt_otel_ops_check():
    - Validates that filter IDs are unique across all proxies.
    - Resolves group->scope and instrumentation->scope/group placeholder
      references to actual configuration structures (setting the ptr field
      and flag_used).
    - Warns about unused scopes, missing root spans, or multiple root spans.
    - Validates metric instruments: resolves update-form references to their
      matching create-form definitions, and rejects duplicate create-form names
      across scopes.
    - Computes the aggregated analyzer bitmask from all used scopes.

  flt_otel_ops_init_per_thread():
    - Starts the tracer, meter and logger background threads on first call.
    - Sets the FLT_CFG_FL_HTX flag to enable HTX stream filtering.

  flt_otel_ops_deinit():
    - Destroys the tracer, meter and logger.
    - Frees the entire configuration tree.
    - Calls otelc_deinit() to shut down the wrapper library.

5.2  Stream-Level Callbacks

  flt_otel_ops_attach():
    - Checks if the filter is globally disabled; returns IGNORE.
    - Applies rate limiting via ha_random32(); returns IGNORE if the random
      value exceeds the configured rate_limit.
    - Creates the runtime context (flt_otel_runtime_context_init) with a
      generated UUID and initialized span/context lists.
    - Sets pre_analyzers and post_analyzers bitmasks from the instrumentation's
      aggregated analyzer flags.  AN_REQ_WAIT_HTTP and AN_RES_WAIT_HTTP are
      placed in post_analyzers because those analyzers can only be used in the
      post_analyze callback.  AN_REQ_HTTP_TARPIT is excluded from pre_analyzers.

  flt_otel_ops_detach():
    - Frees the runtime context, which finishes all remaining active spans and
      destroys all remaining contexts.

  flt_otel_ops_check_timeouts():
    - Checks whether the idle-timeout timer has expired; if so, fires the
      on-idle-timeout event and reschedules the timer for the next interval.
    - Sets STRM_EVT_MSG on the stream's pending_events to ensure the filter is
      re-evaluated after a timeout.

5.3  Error Handling

Two helper functions manage errors:

  flt_otel_return_int() / flt_otel_return_void():
    - If the result indicates an error or an error string is set: in hard-error
      mode, the filter is disabled for the current stream (flag_disabled = 1)
      and the disabled counter is incremented atomically.  In soft-error mode,
      the error is merely logged.
    - The error string is always freed.
    - For int returns, FLT_OTEL_RET_OK is returned regardless, so the stream
      continues processing even after an error.


6  Event Processing (Channel Analyzers)
----------------------------------------------------------------------

The filter maps HAProxy channel analyzer callbacks to a table of named events
defined in event.h (FLT_OTEL_EVENT_DEFINES).

6.1  Event Table

Each event entry carries:
  - an_bit:           the HAProxy analyzer bit (AN_REQ_*, AN_RES_*)
  - an_name:          the analyzer bit name (e.g. "AN_REQ_FLT_HTTP_HDRS")
  - smp_opt_dir:      sample fetch direction (REQ or RES)
  - smp_val_fe/be:    valid sample fetch locations
  - flag_http_inject: whether span context can be injected into HTTP headers
                      at this point
  - name:             configuration event name (e.g. "on-frontend-http-request")

Events with an_bit == 0 are pseudo-events not tied to any channel
analyzer.  Two of them fire from stream lifecycle callbacks:
  - on-stream-start  (flt_otel_ops_stream_start, before channel processing)
  - on-stream-stop   (flt_otel_ops_stream_stop, after channel processing)

One fires periodically from the check_timeouts callback:
  - on-idle-timeout  (flt_otel_ops_check_timeouts, when stream is idle)

One fires from the stream_set_backend callback:
  - on-backend-set   (flt_otel_ops_stream_set_backend, when backend is assigned)

Four fire from HTTP lifecycle callbacks:
  - on-http-headers-request / on-http-headers-response (flt_otel_ops_http_headers)
  - on-http-end-request / on-http-end-response         (flt_otel_ops_http_end)
  - on-http-reply                                      (flt_otel_ops_http_reply)

But on-http-reply never actually fires on current HAProxy: nothing has invoked
its flt_http_reply() dispatcher since 2.2-dev8 (2020), so the filter callback
above is dead.  See README and README-configuration for the full explanation.

The remaining pseudo-events fire from channel start/end callbacks:
  - on-client-session-start / on-client-session-end
  - on-server-session-start / on-server-session-end
  - on-server-unavailable

The stream lifecycle events pass NULL for the channel argument, so
context injection/extraction via HTTP headers cannot be used.  Their
sample fetch direction is unconstrained (0xff), allowing both request
and response fetches.

6.2  Callback Flow

  stream_start(s, f):
    - Fires on-stream-start with chn=NULL.
    - Called when a new stream begins, before any channel processing.
    - Initializes the idle timer from the precomputed minimum idle_timeout in
      the instrumentation configuration.

  stream_set_backend(s, f, be):
    - Fires on-backend-set with chn=&s->req.
    - Called when a backend is assigned (skipped if frontend == backend).

  stream_stop(s, f):
    - Fires on-stream-stop with chn=NULL.
    - Called when a stream is destroyed, after all channel processing.

  check_timeouts(s, f):
    - Checks whether the idle-timeout timer has expired.
    - If expired, fires on-idle-timeout and reschedules the timer.

  channel_start_analyze(chn):
    - Enables the per-channel analyzers from pre_analyzers.
    - Fires on-client-session-start (request) or on-server-session-start
      (response).
    - Propagates the idle-timeout expiry to the channel's analyse_exp.

  channel_pre_analyze(chn, an_bit):
    - Looks up the event by an_bit in the event table.
    - Calls flt_otel_event_run() for the matching event.

  channel_post_analyze(chn, an_bit):
    - Same as pre_analyze but for post-analyzers (AN_REQ_WAIT_HTTP,
      AN_RES_WAIT_HTTP).

  channel_end_analyze(chn):
    - Fires on-client-session-end (request) or on-server-session-end (response).
    - For the request channel: if response analyzers were configured but
      none executed (server was unreachable), fires on-server-unavailable.

  http_headers(s, f, msg):
    - Fires on-http-headers-request or on-http-headers-response depending on
      msg->chn direction.

  http_end(s, f, msg):
    - Fires on-http-end-request or on-http-end-response depending on
      msg->chn direction.

  http_reply(s, f, status, msg):
    - Fires on-http-reply with chn=&s->res.
    - Never reached on current HAProxy: HAProxy dropped the flt_http_reply()
      call in 2.2-dev8 (commit 8dfeccf6d, 2020), so this callback has no caller.

6.3  Scope Execution

  flt_otel_event_run() (event.c):
    - Captures timestamps (CLOCK_MONOTONIC + CLOCK_REALTIME).
    - Updates the runtime context's executed-analyzers bitmask.
    - Iterates all scopes matching the event; calls flt_otel_scope_run() for
      each used scope.

  flt_otel_scope_run() (event.c):
    1. Evaluates the scope's ACL condition; if it fails:
       - If the scope contains a root span, disables the stream.
       - Returns without processing.
    2. Extracts contexts: for each configured extract directive, reads
       the span context from HTTP headers or HAProxy variables via
       flt_otel_scope_context_init().
    3. Processes spans: for each configured span:
       a. Calls flt_otel_scope_span_init() which either returns an existing
          scope_span (by name) or creates a new one with resolved parent
          reference.
       b. Resolves span links against the runtime context -- first searching
          active spans, then extracted contexts.  Unresolved links produce a
          NOTICE-level warning and are skipped.
       c. Evaluates attributes, events, baggages, and status from sample
          expressions via flt_otel_sample_add().
       d. Calls flt_otel_scope_run_span() which:
          - Creates the OTel span via tracer->start_span_with_options()
            (if not already started).
          - Adds all resolved links via span->add_link().
          - Sets baggage, attributes, events, and status.
          - Optionally injects the span context into HTTP headers and/or
            HAProxy variables.
    4. Processes metric instruments via flt_otel_scope_run_instrument(), which
       runs two passes: the first lazily creates create-form instruments using
       HA_ATOMIC_CAS for thread-safe one-time initialization; the second records
       measurements for update-form instruments, skipping any whose index is
       still negative (creation pending or not yet attempted).
    5. Emits log records via flt_otel_scope_run_log_record(), which iterates
       the scope's log-record list, skips entries below the logger's severity
       threshold, evaluates sample expressions into a body string, resolves
       the optional span reference, and emits the record via the logger.
    6. Marks spans listed in "finish" directives; a fired "otel-stop" marks
       every open span and context for completion.
    7. Calls flt_otel_scope_finish_marked() to end marked spans/contexts.
    8. Calls flt_otel_scope_free_unused() to remove finished and destroyed
       scope_span/scope_context entries from the runtime lists.
    9. If "otel-stop" fired, sets rt_ctx->flag_disabled = 1; the guard at the
       top of flt_otel_scope_run() then skips the stream's remaining scopes.


7  Runtime Data Structures
----------------------------------------------------------------------

7.1  Runtime Context (per stream)

  flt_otel_runtime_context:
    stream         Owning stream pointer.
    filter         Owning filter pointer.
    uuid[40]       Generated UUID v4 for the session.
    flag_harderr   Copied from instrumentation config.
    flag_disabled  Set when the filter encounters a hard error or ACL disables
                   processing.
    logging        Logging flags.
    analyzers      Bitmask of analyzers that have actually executed.
    idle_timeout   Idle timeout interval in milliseconds (0 = off).
    idle_exp       Tick at which the next idle timeout fires.
    spans          Linked list of flt_otel_scope_span.
    contexts       Linked list of flt_otel_scope_context.

7.2  Scope Span

  flt_otel_scope_span:
    id / id_len    Span operation name (borrowed from config).
    smp_opt_dir    Direction in which the span was created.
    flag_finish    Set by finish directives, cleared after ending.
    span           The OTel span object (NULL before start, NULL after
                   end_with_options).
    ref_span       Parent span pointer (resolved at init).
    ref_ctx        Parent context pointer (resolved at init).
    list           Chain in runtime_context.spans.

  flt_otel_scope_span_init() performs memoization: if a span with the same name
  already exists in rt_ctx->spans, it returns the existing entry.  This allows
  multiple scopes to contribute attributes/events to the same logical span.

7.3  Scope Context

  flt_otel_scope_context:
    id / id_len    Context name (borrowed from config).
    smp_opt_dir    Direction in which the context was extracted.
    flag_finish    Marks the context for destruction.
    context        The OTel span_context object.
    list           Chain in runtime_context.contexts.

  Similarly memoized: duplicate extraction of the same context name returns the
  existing entry.

7.4  Scope Data (per span per scope run, stack-allocated)

  flt_otel_scope_data:
    baggage        Key-value array for baggage items.
    attributes     Key-value array for span attributes.
    events         Linked list of flt_otel_scope_data_event (each with name
                   + key-value array).
    links          Linked list of flt_otel_scope_data_link (each with span
                   and/or context pointer).
    status         Status code and description string.

  Initialized at the start of each span processing block and freed at the end.
  The link entries hold borrowed pointers to OTel objects owned by the runtime
  context, so only the link nodes themselves are freed.

7.5  Span Finishing

  finish <name> / finish * / finish *req* / finish *res*

  The "finish" directive marks spans and contexts for completion:
    - "*" marks all.
    - "*req*" / "*res*" marks those created in the request/response direction
      respectively.
    - Otherwise, marks by exact name.

  flt_otel_scope_finish_marked() iterates all marked entries:
    - Spans are ended via span->end_with_options() which NULLs the span pointer.
    - Contexts are destroyed via context->destroy() which NULLs the context
      pointer.

  flt_otel_scope_free_unused() then removes entries with NULL span/context
  pointers from the runtime lists.  For contexts, associated HTTP headers
  and variables are also cleaned up.

  On stream detach (flt_otel_runtime_context_free), any remaining active spans
  are force-ended and all entries are freed.


8  Span Links
----------------------------------------------------------------------

Span links associate a span with other spans or contexts without establishing
a parent-child relationship.

8.1  Configuration

Two syntaxes are supported:

  Inline (one link per span declaration):
    span <name> [parent <ref>] link <ref> [root]

  Standalone (multiple links, or one link with attributes):
    link { <ref> [<ref> ...] | <ref> attr <key> <sample> ... }

The flt_otel_conf_link structure stores each link target name and, for the
single-link form, a list of attribute samples.  Duplicate link names within
the same span are rejected by the init macro's duplicate check.
The links list is initialized in flt_otel_conf_span_init() and destroyed in
flt_otel_conf_span_free().

8.2  Runtime Resolution

At scope execution time (event.c, flt_otel_scope_run), for each configured link:
  1. The name is searched in rt_ctx->spans (active scope_span entries).
     If found, the OTel span pointer is captured.
  2. If not found in spans, the name is searched in rt_ctx->contexts (extracted
     scope_context entries).  If found, the OTel span_context pointer is
     captured.
  3. If neither found, a NOTICE warning is logged and the link is skipped.
  4. A flt_otel_scope_data_link node is allocated and appended to the scope
     data's links list.

In flt_otel_scope_run_span(), all resolved links are applied via
span->add_link(span, link_span, link_context, NULL, 0).  The last two arguments
(attributes array and count) are NULL/0, meaning links carry no additional
attributes.


9  Context Propagation
----------------------------------------------------------------------

9.1  Extraction

  extract <name-prefix> [use-vars | use-headers]

Extracts an incoming trace context.  The prefix identifies the header name
pattern (for HTTP) or variable name pattern (for vars).

  - use-headers (default): flt_otel_http_headers_get() iterates HTX headers
    matching the prefix and builds an otelc_text_map.
  - use-vars: flt_otel_vars_get() reads HAProxy variables matching the prefix
    pattern.

The text map is passed to flt_otel_extract_http_headers() which uses the
C wrapper to reconstruct an otelc_span_context.

9.2  Injection

  inject <name-prefix> [use-vars] [use-headers]

Injects the current span's context into outgoing data.  Both storage types can
be used simultaneously.

  flt_otel_inject_http_headers() serializes the span context into an
  otelc_http_headers_writer which produces a text_map.  For each key-value pair:
    - use-headers: flt_otel_http_header_set() adds/replaces the header with the
      prefixed name.
    - use-vars: flt_otel_var_register() + flt_otel_var_set() stores the value
      in a HAProxy transaction variable with normalized name (dashes replaced
      with 'D', spaces with 'S', uppercase lowered; dots serve as component
      separators).


10  HTTP Header Manipulation
----------------------------------------------------------------------

  http.c provides three operations:

  flt_otel_http_headers_get(chn, prefix, prefix_len, err):
    Iterates the HTX message headers.  Headers whose name starts with the given
    prefix are collected into an otelc_text_map.  The prefix is stripped from
    the names in the returned map.

  flt_otel_http_header_set(chn, prefix, name, value, err):
    Removes any existing header matching "prefix" + "name", then adds a new
    header with the given value.  If name is NULL, all headers with the prefix
    are removed (bulk delete).

  flt_otel_http_headers_remove(chn, prefix, err):
    Convenience wrapper; removes all headers matching the prefix.


11  HAProxy Variable Integration
----------------------------------------------------------------------

Enabled with OTEL_USE_VARS=1.  Provides an alternative propagation mechanism
using HAProxy transaction-scoped variables.

Variable names are normalized: dashes and spaces are replaced with special
characters to comply with HAProxy variable naming rules.  A meta-variable
tracks the list of context variable names so they can be enumerated for
extraction.

Key functions:
  flt_otel_var_register()   Registers a variable with HAProxy.
  flt_otel_var_set()        Sets a variable value.
  flt_otel_vars_get()       Reads all context variables into a text_map for
                            extraction.
  flt_otel_vars_unset()     Removes all context variables.


12  Group Action Integration
----------------------------------------------------------------------

The "otel-group" HAProxy action allows triggering trace scopes from
tcp-request, tcp-response, http-request, http-response and
http-after-response rules:

  tcp-request         otel-group <filter-id> <group-name>
  tcp-response        otel-group <filter-id> <group-name>
  http-request        otel-group <filter-id> <group-name>
  http-response       otel-group <filter-id> <group-name>
  http-after-response otel-group <filter-id> <group-name>

  group.c implements:
    flt_otel_group_parse():   Parses the action arguments.
    flt_otel_group_check():   Resolves group and scope references.
    flt_otel_group_action():  At runtime, finds the OTel filter in the stream,
                              iterates all scopes in the group, and calls
                              flt_otel_scope_run() for each.


13  Memory Management
----------------------------------------------------------------------

  pool.c provides wrappers around HAProxy memory pools and standard
  allocation:

  flt_otel_pool_alloc()   Allocates from a pool (if non-NULL and the requested
                          size fits) or via calloc.
  flt_otel_pool_free()    Returns memory to the pool or frees it.
  flt_otel_pool_strndup() Duplicates a string via pool allocation.
  flt_otel_trash_alloc()  Acquires a trash buffer chunk.
  flt_otel_trash_free()   Releases a trash buffer chunk.

Four pool heads are registered for hot-path structures:
  - otel_scope_span       (scope.c)
  - otel_scope_context    (scope.c)
  - otel_runtime_context  (scope.c)
  - otel_span_context     (filter.c, used by the C wrapper via otelc_ext_init
                          callback)

The wrapper library's memory allocations are redirected through
flt_otel_mem_malloc() / flt_otel_mem_free() which use the otel_span_context
pool.  This ensures OTel objects benefit from HAProxy's pool allocator.


14  CLI Interface
----------------------------------------------------------------------

  cli.c registers commands under "flt-otel" for runtime control:
  - Setting the debug level.
  - Enabling/disabling the filter on the fly.

Logging can be independently controlled via the instrumentation's logging
flags (ON, NOLOGNORM).  Log output goes to the log servers configured in the
instrumentation block.


15  Debug Infrastructure
----------------------------------------------------------------------

When compiled with OTEL_DEBUG=1 (DEBUG_OTEL defined), the filter enables:

  - Additional flt_ops callbacks: stream_set_backend, deinit_per_thread,
    http_headers, http_payload, http_end, http_reset, http_reply, tcp_payload.
    In non-debug builds these are set to NULL.  (Note: stream_start and
    stream_stop are always registered because they fire the on-stream-start
    and on-stream-stop events.)

  - The OTELC_DBG() macro produces debug output at various levels.

  - flt_otel_scope_data_dump() dumps the complete scope data (baggage,
    attributes, events, links, status) for inspection.

  - Event usage counters (per-event htx_is_empty statistics) are maintained and
    printed at deinit.

  - Pool size information is printed at startup.

The debug level is a bitmask that can be adjusted at runtime via the CLI.


16  Test Infrastructure
----------------------------------------------------------------------

16.1  Test Scenarios

  sa    Standalone: comprehensive test exercising all request and response
        events, span links (both inline and standalone syntax), events with data
        capture, baggage, and the full span hierarchy from client session start
        to server session end.

  fe    Frontend-only: tests the request-side span chain with context injection
        into HTTP headers.

  be    Backend-only: tests context extraction from HTTP headers and
        response-side processing.  Designed to run as the backend of
        the fe/ test.

  ctx   Context propagation: deep nesting test that verifies context propagation
        via both HTTP headers and HAProxy variables.

  cmp   Comparison: simplified configuration made for comparison with other
        tracing implementations.

  empty Minimal: validates that an empty configuration (only the
        instrumentation block, no scopes) does not crash.

16.2  Test Runners

All runners are POSIX shell scripts (/bin/sh).  They accept an optional HAProxy
binary path and log to test/_logs/.

  run-sa.sh     Runs a single HAProxy instance with sa/ config.
  run-cmp.sh    Runs a single HAProxy instance with cmp/ config.
  test-speed.sh Runs performance benchmarks for one or all configurations.
  run-ctx.sh    Runs a single HAProxy instance with ctx/ config.
  run-fe-be.sh  Launches two HAProxy instances (frontend on port 10080, backend
                on port 11080) forming a trace propagation chain.  Handles
                graceful shutdown via SIGUSR1.

  copy-yml.sh   Transforms a template YAML configuration by replacing
                placeholders with test-specific values (service names, file
                suffixes, etc.).

16.3  Exporter Configuration

Each test directory contains an otel.yml file configuring three exporter types:
  - OTLP file exporter (writes traces to local files).
  - OTLP gRPC exporter (sends to localhost:4317).
  - OTLP HTTP exporter (sends to localhost:4318 in JSON format).


17  Notable Design Decisions
----------------------------------------------------------------------

  - Span memoization: flt_otel_scope_span_init() and
    flt_otel_scope_context_init() return existing entries if one with the
    same name already exists.  This allows multiple scopes to contribute data
    (attributes, events) to the same logical span across different analyzer
    events.

  - Lazy span creation: the OTel span object is created on first use in
    flt_otel_scope_run_span(), not at scope_span_init time.  This separates
    the span identity (name, parent reference) from the actual OTel resource.

  - Soft/hard error modes: in soft mode, errors are logged but the stream
    continues with tracing effectively abandoned for that span.  In hard mode,
    the filter disables itself for the rest of the stream.  Either way, stream
    processing is never interrupted by a tracing failure (FLT_OTEL_RET_OK is
    always returned).

  - Rate limiting uses a uint32 representation of a percentage
    (FLT_OTEL_FLOAT_U32), compared against ha_random32() for uniform
    distribution without floating-point at runtime.

  - Server-unavailable fallback: if the backend was never reached (no response
    analyzers executed), the on-server-unavailable event is fired at client
    session end to ensure all spans are properly closed.

  - Custom memory allocator: the C wrapper's allocations are routed through
    HAProxy memory pools via otelc_ext_init(), keeping OTel objects in the
    same allocation domain as the rest of the filter.

  - Thread integration: flt_otel_thread_id() returns the HAProxy tid, ensuring
    the wrapper's thread-local operations map to HAProxy worker threads.

  - Single dispatch stage per event: each scope event fires from one callback
    -- channel_pre_analyze or channel_post_analyze for analyzer-bound events,
    or a lifecycle/data callback otherwise.  Exposing both stages of every
    analyzer (an on-X-pre and an on-X-post) was considered and rejected: about
    half the events are not analyzer-bound and have no stage to split; the
    tarpit analyzer never finishes so it has no post stage, and the
    wait-for-HTTP analyzers have no parsed message in the pre stage; and the
    rest complete in a single pass, so their two stages fire microseconds
    apart.  Where a later view is genuinely needed -- the request body, for
    example, is buffered only after AN_REQ_HTTP_BODY -- an optional per-scope
    stage qualifier on the existing event name would be preferable to doubling
    the event table.  The otel-event directive would take an optional 'pre' or
    'post' keyword:

        otel-event on-http-body-request       # default stage (pre, as today)
        otel-event on-http-body-request post  # opt into the post stage

    defaulting to the current stage when omitted.  The filter would list such an
    analyzer in both the pre and post masks and run the scope only from the
    stage it asked for.


18  Tracer, Span and Metrics Internals
----------------------------------------------------------------------

This chapter describes the end-to-end lifecycle of the tracer and meter
objects, the runtime span management model, and the metric instrument
recording pipeline.

18.1  Tracer Provider Initialization

The tracer provider is set up during the proxy-level flt_otel_ops_init()
callback, which delegates to flt_otel_lib_init() (filter.c).
The initialization sequence is as follows:

  1. Version check: OTELC_IS_VALID_VERSION() verifies that the
     OpenTelemetry C wrapper library version matches the header files.

  2. Configuration path: the relative path from the "config" keyword in
     the instrumentation section is resolved to an absolute path using
     getcwd() + snprintf().

  3. SDK initialization: otelc_init(path, err) loads the YAML
     configuration file and sets up the SDK exporters, samplers,
     processors and metric readers.

  4. Tracer creation: otelc_tracer_create(err) allocates the tracer
     handle and stores it in instr->tracer.

  5. Meter creation: otelc_meter_create(err) allocates the meter handle
     and stores it in instr->meter.

  6. Logger creation: otelc_logger_create(err) allocates the logger
     handle and stores it in instr->logger.

  7. Extension callbacks: on success, otelc_ext_init() registers custom
     memory allocation (flt_otel_mem_malloc / flt_otel_mem_free) and
     thread-id (flt_otel_thread_id) callbacks so that OTel SDK objects
     use HAProxy memory pools and thread numbering.

  8. Log handler: otelc_log_set_handler() installs a callback that
     counts SDK diagnostic messages via the flt_otel_drop_cnt counter.

All three handles are stored in the flt_otel_conf_instr structure
(conf.h):

  struct flt_otel_conf_instr {
      ...
      struct otelc_tracer *tracer;  /* The OpenTelemetry tracer handle. */
      struct otelc_meter  *meter;   /* The OpenTelemetry meter handle. */
      struct otelc_logger *logger;  /* The OpenTelemetry logger handle. */
      ...
  };

18.2  Per-Thread Tracer, Meter and Logger Startup

The flt_otel_ops_init_per_thread() callback (filter.c) starts the
tracer, meter and logger background threads on the first call:

  if (!(fconf->flags & FLT_CFG_FL_HTX)) {
      retval = OTELC_OPS(conf->instr->tracer, start);
      if (retval != OTELC_RET_ERROR) {
          retval = OTELC_OPS(conf->instr->meter, start);
          ...
      }
      if (retval != OTELC_RET_ERROR) {
          retval = OTELC_OPS(conf->instr->logger, start);
          ...
      }
      fconf->flags |= FLT_CFG_FL_HTX;
  }

The FLT_CFG_FL_HTX flag ensures that start is called only once, even
when multiple proxies share the same filter configuration.  If any
start operation fails, the error string from the failing handle is
forwarded via FLT_OTEL_ALERT.

18.3  Tracer, Meter and Logger Shutdown

At proxy deinit (flt_otel_ops_deinit, filter.c), the tracer, meter
and logger are destroyed in a single call:

  otelc_deinit(&((*conf)->instr->tracer), &((*conf)->instr->meter), &((*conf)->instr->logger));

This flushes any pending spans, metric data and log records to the
configured exporters, then releases the SDK resources.  The full
configuration tree is freed immediately after via flt_otel_conf_free().

18.4  Span Lifecycle

Spans progress through four phases: identity allocation, OTel span
creation, data population, and completion.

18.4.1  Span Identity Allocation

When a scope containing a span definition executes for the first time,
flt_otel_scope_span_init() (scope.c) allocates a scope_span
entry from the otel_scope_span pool and inserts it into the runtime
context's spans list:

  retptr = flt_otel_pool_alloc(pool_head_otel_scope_span, ...);
  retptr->id          = id;       /* Borrowed from config. */
  retptr->id_len      = id_len;
  retptr->smp_opt_dir = dir;
  retptr->ref_span    = ref_span; /* Resolved parent span. */
  retptr->ref_ctx     = ref_ctx;  /* Resolved parent context. */
  LIST_INSERT(&(rt_ctx->spans), &(retptr->list));

The parent reference (ref_id) is resolved at this point by searching the
runtime context's spans list first, then the contexts list.  If the
parent name cannot be found in either list, an error is returned and the
span is not created.

Memoization: if a span with the same name already exists in
rt_ctx->spans, the existing entry is returned without allocation.  This
allows multiple scopes (across different analyzer events) to contribute
attributes, events and other data to the same logical span.

18.4.2  OTel Span Creation (Lazy)

The actual OTel span object is created lazily on first use in
flt_otel_scope_run_span() (event.c):

  if (span->span == NULL) {
      span->span = OTELC_OPS(conf->instr->tracer,
          start_span_with_options, span->id,
          span->ref_span, span->ref_ctx,
          ts_steady, ts_system, OTELC_SPAN_KIND_SERVER);
  }

The arguments are:

  span->id       The operation name (string identifier from config).
  span->ref_span The parent span pointer (NULL if root or no parent).
  span->ref_ctx  The parent span context (from extracted context).
  ts_steady      Monotonic timestamp (CLOCK_MONOTONIC) for duration.
  ts_system      Wall-clock timestamp (CLOCK_REALTIME) for events.
  OTELC_SPAN_KIND_SERVER  Fixed span kind for all HAProxy spans.

This separation between identity allocation and OTel creation means the
span name, parent references and pool entry exist before the OTel
resource is allocated.  Subsequent scope executions that reference the
same span name find the existing entry (via memoization) and add their
data to the already-created OTel span.

18.4.3  Span Data Population

After creation, flt_otel_scope_run_span() (event.c) populates
the span with data collected during scope execution:

  Links (event.c):
    Each resolved link is added via span->add_link(span, link_span,
    link_context, NULL, 0).  Links associate the span with other spans
    or contexts without establishing a parent-child relationship.  The
    last two arguments (attributes array and count) are always NULL/0.

  Baggage (event.c):
    span->set_baggage_kv_n(data->baggage.attr, data->baggage.cnt)
    sets key-value baggage items propagated across service boundaries.

  Attributes (event.c):
    span->set_attribute_kv_n(data->attributes.attr, data->attributes.cnt)
    sets key-value span attributes evaluated from HAProxy sample
    expressions.

  Events (event.c):
    For each event in data->events (iterated in reverse insertion order):
    span->add_event_kv_n(event->name, ts_system, event->attr, event->cnt)
    adds a named event with a wall-clock timestamp and key-value
    attributes.

  Status (event.c):
    span->set_status(data->status.code, data->status.description)
    sets the span's status code and description string.  Only one status
    per event is allowed.

18.4.4  Span Context Injection

After populating the span, if the configuration contains an "inject"
directive (conf_span->ctx_id is non-NULL), the span context is
serialized for downstream propagation (event.c).

  flt_otel_inject_http_headers() serializes the span context into an
  otelc_http_headers_writer, producing a text_map of key-value pairs.
  For each pair, depending on the ctx_flags:

    FLT_OTEL_CTX_USE_HEADERS:
      flt_otel_http_header_set() writes the header into the HTX message.

    FLT_OTEL_CTX_USE_VARS (requires OTEL_USE_VARS=1):
      flt_otel_var_register() + flt_otel_var_set() store the value
      in a HAProxy transaction variable.

  Both storage types can be used simultaneously on the same span.

18.4.5  Span Completion

Spans are ended through the marking mechanism described in chapter 7.5.
The actual end call in flt_otel_scope_finish_marked() (scope.c) is:

  OTELC_OPSR(span->span, end_with_options,
             ts_finish, OTELC_SPAN_STATUS_IGNORE, NULL);

The arguments are the monotonic timestamp, a status hint (IGNORE means
"do not override the status already set on the span"), and NULL for
error string.  After end_with_options returns, the OTELC_OPSR macro
NULLs the span pointer, making the entry eligible for removal by
flt_otel_scope_free_unused().

On stream detach, flt_otel_runtime_context_free() (scope.c)
force-ends any remaining active spans with the current monotonic
timestamp and frees all pool entries.

18.5  Metric Instruments

The filter supports the full set of OpenTelemetry metric instrument
types through a two-form configuration model: "create" form instruments
define the instrument, and "update" form instruments record measurements
against it.

18.5.1  Instrument Types

The following instrument types are available (parser.h):

    cnt_int         Counter (uint64)
    hist_int        Histogram (uint64)
    udcnt_int       UpDownCounter (int64)
    gauge_int       Gauge (int64)

Observable (asynchronous) instruments are not supported.  The OTel SDK invokes
their callbacks from an external background thread that is not a HAProxy
thread.  HAProxy sample fetches rely on internal per-thread-group state and
return incorrect results when called from a non-HAProxy thread.

Double-precision types are not supported because HAProxy sample fetches do not
return double values.

  Special:
    update          Update-form instrument (records measurements)

Each create-form instrument carries a description, unit, aggregation type,
sample expression list, and optional histogram bucket boundaries.  Each
update-form instrument carries a reference to its create-form counterpart
and an attribute key-value array for per-scope dimensions.

18.5.2  Instrument Configuration Structure

The flt_otel_conf_instrument structure (conf.h) holds:

  idx          Meter instrument index.  Initially set to
               OTELC_METRIC_INSTRUMENT_UNSET (-1).  Transitions to
               OTELC_METRIC_INSTRUMENT_PENDING (-2) during creation,
               then to the positive meter index on success.
  type         The otelc_metric_instrument_t type constant, or
               OTELC_METRIC_INSTRUMENT_UPDATE (0xff) for update-form.
  aggr_type    The otelc_metric_aggregation_type_t constant.
               Initially OTELC_METRIC_AGGREGATION_UNSET (-1).
  description  Instrument description string (create-form only).
  unit         Instrument unit string (create-form only).
  samples      List of sample expressions for the instrument value.
  bounds       Histogram bucket boundaries array (create-form only).
  bounds_num   Number of histogram bucket boundaries.
  attributes   List of flt_otel_conf_sample entries (update-form only).
  ref          Pointer to the create-form instrument (update-form only).

18.5.3  Meter Initialization and Startup

The meter handle is created alongside the tracer in flt_otel_lib_init()
(filter.c) via otelc_meter_create(err) and started per-thread
in flt_otel_ops_init_per_thread() (filter.c) via
OTELC_OPS(conf->instr->meter, start).  The meter background thread
handles periodic collection and export of metric data.

18.5.4  Instrument Creation and Recording

Metric instrument processing is performed by
flt_otel_scope_run_instrument() (event.c), which runs in two
passes during scope execution.

  Pass 1 -- Create-form instruments (event.c):

    Iterates all instruments in the scope.  For each create-form
    instrument whose idx is OTELC_METRIC_INSTRUMENT_UNSET:

    a. Thread-safe one-time creation: HA_ATOMIC_CAS transitions the idx
       from UNSET to PENDING.  If the CAS fails (another thread is
       already creating this instrument), the current thread skips it.

    b. Instrument creation: meter->create_instrument() is called with
       the instrument name, description, unit, type and callback data.
       On success, the returned index is stored atomically; on failure,
       the idx is reset to UNSET.  If the instrument has an explicit
       aggregation type or histogram bucket boundaries, meter->add_view()
       is called before instrument creation to register a view with the
       configured aggregation strategy and optional bounds.  When bounds
       are present without an explicit aggregation type, histogram
       aggregation is used automatically for backward compatibility.

  Pass 2 -- Update-form instruments (event.c):

    Iterates all instruments again.  For each update-form instrument:

    a. Reference validation: the ref pointer must be non-NULL (resolved
       at check time to the create-form instrument).

    b. Index check: if the create-form instrument's idx is still
       negative (UNUSED or PENDING), the measurement is skipped.

    c. Recording: flt_otel_scope_run_instrument_record() evaluates the
       sample expression, converts it to an otelc_value, and calls
       meter->update_instrument_kv_n(idx, &value, attr, attr_len).

18.5.5  Sample Evaluation for Metrics

The recording function flt_otel_scope_run_instrument_record()
(event.c) supports two evaluation paths:

  Standard path: evaluates sample_process() on the first expression in
  the create-form instrument's samples list, using the stream's backend,
  session and direction context.

  Log-format path: if sample->lf_used is set, allocates a temporary
  buffer of global.tune.bufsize, calls build_logline() to evaluate the
  log-format expression, and presents the result as an SMP_T_STR sample.

Both paths converge on flt_otel_sample_to_value(), which converts the
HAProxy sample data to an otelc_value.  Metric instruments require
numeric values (INT64); if the conversion produces
OTELC_VALUE_DATA (string), a warning is logged and the measurement is
rejected.


18.5.6  Instrument Lifecycle Summary

  Configuration time:
    idx = OTELC_METRIC_INSTRUMENT_UNSET (-1)
    type = instrument type constant

  First scope execution (any thread):
    idx transitions:  UNSET -> PENDING -> meter_index  (success)
                      UNSET -> PENDING -> UNSET       (failure)

  Subsequent scope executions:
    Create-form: skipped (idx is already a valid meter index).
    Update-form: evaluates samples and records via meter API.

  Shutdown:
    otelc_deinit() flushes and destroys tracer, meter and logger,
    including all registered instruments and their callbacks.


18.6  Log Records

The filter supports OpenTelemetry log records via the "log-record"
keyword inside otel-scope sections.  Each log record is emitted through
the OTel logger at a configured severity level, with an evaluated body,
optional span correlation and optional key-value attributes.

18.6.1  Log Record Configuration Structure

The flt_otel_conf_log_record structure (conf.h) holds:

  severity     The otelc_log_severity_t severity level.
  event_id     Optional numeric event identifier (int64).
  event_name   Optional event name string.
  span         Optional span reference name (resolved at runtime).
  time         Optional timestamp expression (single-entry conf_sample list).
  attributes   List of flt_otel_conf_sample entries for attributes.
  samples      List of sample expressions for the body.

The attributes list contains flt_otel_conf_sample entries, one per "attr"
keyword.  Each entry's key field holds the attribute name and its sample
expressions are evaluated at runtime, following the same two-path model
(bare sample or log-format) as span attributes.

The samples list contains exactly one flt_otel_conf_sample entry, which holds
either a list of bare sample expressions or a single log-format expression.  The
log-format path is selected when the value begins with the "%[" sequence; an
explicit '%[ ... ]' wrapper is then stripped before parsing so the inner string
may use HAProxy log-format aliases such as %ci, %t or %ft.

The time list, when populated, holds a single flt_otel_conf_sample produced by
the "time [<unit>] <sample>" keyword.  The unit selector (one of s, ms, us or
ns) is stored in the sample's extra field as an OTELC_VALUE_INT32; a default
of FLT_OTEL_TIME_UNIT_S is recorded when no unit is given.  The sample itself
follows the same parsing rules as any other flt_otel_conf_sample.

18.6.2  Log Record Emission

Log record processing is performed by flt_otel_scope_run_log_record()
(event.c), called from flt_otel_scope_run() after metric instrument
processing and before span finishing.

For each configured log record the function performs:

  1. Severity check: OTELC_OPS(logger, enabled, severity) tests whether
     the logger accepts records at this severity.  If not, the entry is
     skipped.  The threshold is controlled by the "min_severity" option
     in the YAML logs signal configuration.

  2. Attribute evaluation: each entry in the attributes list is evaluated via
     flt_otel_sample_add() into a temporary flt_otel_scope_data structure.
     The evaluated key-value array is passed to logger->log_span() and freed
     after emission.

  3. Body evaluation: the single sample entry is evaluated using one of
     two paths:

     Log-format path (sample->lf_used is true):
       A temporary buffer of global.tune.bufsize is allocated and
       build_logline() evaluates the log-format expression into it.

     Bare sample expression path:
       Each expression in sample->exprs is evaluated via
       sample_process() and converted to a string via
       flt_otel_sample_to_str().  Results are concatenated into a
       single buffer.

  4. Span resolution: if conf_log->span is non-NULL, the runtime
     context's spans list is searched for a scope_span with a matching
     name.  If found, the OTel span pointer is captured for correlation.
     A missing span is non-fatal -- a NOTICE warning is logged and the
     record is emitted without span correlation.

  5. Event timestamp selection: when conf_log->time is non-empty, its single
     sample is evaluated via flt_otel_sample_eval() in native mode.  A string
     result is coerced to int64 with otelc_value_strtonum(); on conversion
     failure the wall-clock ts is kept.  A successful int64 result is split
     into a struct timespec by the unit packed in the sample's extra field
     (s, ms, us or ns), and that timespec becomes the event timestamp.

  6. Emission: logger->log_span() is called with the severity, event_id,
     event_name, resolved span (or NULL), the event timestamp, the wall-clock ts
     as the observed timestamp, the evaluated attributes and the evaluated body
     string.

18.6.3  Logger Lifecycle Summary

  Proxy init (flt_otel_lib_init):
    otelc_logger_create() allocates the logger handle.

  Per-thread init (flt_otel_ops_init_per_thread):
    logger->start() launches the logger background thread.

  Scope execution (flt_otel_scope_run):
    flt_otel_scope_run_log_record() emits records via logger->log_span().

  Shutdown (flt_otel_ops_deinit):
    otelc_deinit() flushes pending log records and destroys the logger.