feat(eventhubs): add tracing and AMQP lifecycle observability - #7368
Open
Johnathan W (j7nw4r) wants to merge 18 commits into
Open
feat(eventhubs): add tracing and AMQP lifecycle observability#7368Johnathan W (j7nw4r) wants to merge 18 commits into
Johnathan W (j7nw4r) wants to merge 18 commits into
Conversation
Add eventhubs_tracing_test.cpp with the recording tracer, span, and attribute set doubles for the Event Hubs tracing work in issue 7336. The tests pin the TracingProvider option on ProducerClientOptions and ConsumerClientOptions, the eager tracer creation in both clients, the ProducerClient.Send span shape on a cancelled context and on a non-cancel exception, and the shape of the shared StartSpan helper for PartitionClient.ReceiveEvents. Tests 1 to 7 do not compile until the tracing source lands. The TracingProvider member does not exist yet, and the private header src/private/eventhubs_tracing.hpp does not exist yet. SendWithoutProviderIsUnchanged is a characterization test. It passes today. It guards the no-provider path against a null dereference after the source lands.
ProducerClientOptions and ConsumerClientOptions gain a TracingProvider field. Each client builds a TracingContextFactory from that field at construction time. When the field is empty, the factory has no tracer and the clients create no spans. ProducerClient::Send creates one Producer span named ProducerClient.Send. PartitionClient::ReceiveEvents creates one Client span named PartitionClient.ReceiveEvents. Both spans carry the messaging attributes, the az.namespace attribute, and the message count. Both spans record an exception event and an error status, then rethrow the original exception. A new private helper, src/private/eventhubs_tracing.hpp, holds the factory creation and the span shape, so the producer and the partition client emit the same attributes. EventDataBatch::NumberOfEvents is now const, and its mutex is mutable, because Send takes the batch by const reference. Trace-context propagation and span links stay deferred, because Azure Core has no API for them.
Run clang-format-11 over eventhubs_tracing_test.cpp. The declarations of AddAttributes and SetStatus on the recording span double broke the Validate Clang Format step in CI. The change is formatting only. No assertion, no test name, and no behavior changed.
ProducerClient::Send converted the batch to an AMQP message before it started the span. An empty batch makes that conversion throw, so the send failed before a span existed and nothing recorded the failure. The conversion now runs at the top of the try block, after the span starts. The existing catch clause adds the exception to the span and rethrows, the same as every other send failure.
The OpenTelemetry semantic conventions at schema 1.17.0 define messaging.batch.message_count as an int. The Event Hubs tracing helper records it as a string, and the test doubles could not show the difference, because they stringified every overload into one map. TestAttributeSet and TestSpan now record a type tag for each attribute. The tag names the overload that delivered the value. Three tests use it: the receive span count and the send span count must arrive through the uint64 overload, and SetMessageCount must stay quiet and safe when the factory has no tracer. The first two tests are red today. The third one is a characterization test that guards the null attribute set hazard.
The batch message count now goes out as an unsigned integer attribute, which matches the OpenTelemetry semantic conventions at schema 1.17.0. Before this change the helper wrote the count as a string. SetMessageCount takes the tracing context factory, because the typed attribute needs an attribute set. The helper guards the null attribute set on the no-tracer path, because the factory returns nullptr when it has no tracer and that pointer is not null safe.
The span helper built a std::string temporary for each attribute name before it added the attribute. Two of the names are longer than the libc++ small-string limit, so each call heap allocated even when the client had no tracer and dropped every attribute. The span helper now returns the tracing context immediately when the factory has no tracer. The message count helper keeps its own null attribute set guard, because PartitionClient::ReceiveEvents calls it directly. The remaining two allocations per operation come from azure-core and are tracked in issue 7340.
The SingleSpan helper called front() on the tracer list after a non-fatal EXPECT_EQ on the list size. When no tracer was created, that call dereferenced an empty std::list and the test process ended in a segmentation fault. The crash stopped the run, so the tests after the first affected one never reported at all. The helper now returns null when the tracer list is not the expected size, so a failure reports through the caller's ASSERT_NE assertion rather than a crash. This mirrors the guard that already protects the spans list two lines later.
The receive span of PartitionClient::ReceiveEvents has no offline coverage. A partition client comes only from a factory that opens a live AMQP link, so no unit test can reach the span, the message count, or the exception path. This test runs against a real namespace. It gives the consumer client a recording tracing provider, receives events from a partition, and makes sure the span name, the span kind, the messaging attributes, and the message count are correct. The count assertion compares the span attribute with the size of the vector that ReceiveEvents returned. The recording tracing test doubles move from eventhubs_tracing_test.cpp into a shared header, because the live test lives in a different file.
The two convenience Send overloads build the batch before they send it, and CreateBatch opens the connection, the session and the AMQP link. A bad host fails there. Only the batch overload starts a span today, so that failure unwinds with no span at all. Add two offline tests that send one event and a vector of three events against a fake connection string. Each test asserts one span named ProducerClient.Send with the kind Producer, the message count as a uint64 attribute, one recorded event and the Error status. Both fail today because the provider holds zero spans. Add a characterization test that keeps the no-provider path unchanged, and a live test that sends one event and asserts a single span. The live test catches a nested second span, which an offline test cannot reach because CreateBatch always throws offline.
The convenience Send overloads made the batch first, and the batch creation opens the connection, the session, and the AMQP link. A bad host or a failed authentication threw there, and the stack unwound before a span existed. A caller then saw no producer span for the failed send. Each public Send overload now starts one span, and one try/catch records the exception on that span. A new private method, SendBatchInSpan, holds the send body and starts no span, so one logical send still makes one span. The convenience overloads pass the span context to CreateBatch, which puts the link open under the span.
The tracing change added a TracingProvider field to ProducerClientOptions and ConsumerClientOptions, but no document told a user how to set it. The only repo document on tracing, doc/DistributedTracing.md, showed clientOptions.Telemetry.TracingProvider. That line does not compile for Event Hubs, because the two option structs do not derive from Azure::Core::_internal::ClientOptions and have no Telemetry field. - Adds a "Distributed tracing" section to the Event Hubs README with a snippet that builds an OpenTelemetryProvider and sets TracingProvider on both client options. - Lists the span names, the span kinds, and the attributes that the clients emit, and names the 1.17.0 semantic conventions schema. - States that the client creates no spans when TracingProvider is not set, because there is no global fallback provider. - Adds a note to doc/DistributedTracing.md for the clients that declare TracingProvider at the top level of their own options structure.
Identify Event Hubs AMQP connections, sessions, and links in lifecycle logs and record their failures, teardown, and recreation generations. Attribute scoped AMQP conditions to the failed protocol layer. Add child spans around link attachment, send disposition, and blocking receive calls so exporters can separate caller-visible SDK latency from AMQP transport latency. Document how to derive failure counters and latency histograms from logs and spans.
Johnathan W (j7nw4r)
force-pushed
the
feat/7336-eventhubs-observability
branch
from
August 21, 2026 18:21
2aa266d to
8e7d686
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds provider-neutral Event Hubs tracing and structured AMQP lifecycle observability.
Changes:
- Adds send, receive, and AMQP transport spans.
- Adds component identifiers, generations, and failure logs.
- Adds tracing tests and documentation.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
sdk/eventhubs/azure-messaging-eventhubs/test/ut/producer_client_test.cpp |
Tests live send tracing. |
sdk/eventhubs/azure-messaging-eventhubs/test/ut/eventhubs_tracing_test.cpp |
Tests tracing and lifecycle records. |
sdk/eventhubs/azure-messaging-eventhubs/test/ut/eventhubs_tracing_test_doubles.hpp |
Adds tracing test doubles. |
sdk/eventhubs/azure-messaging-eventhubs/test/ut/consumer_client_test.cpp |
Tests live receive tracing. |
sdk/eventhubs/azure-messaging-eventhubs/test/ut/CMakeLists.txt |
Registers tracing tests. |
sdk/eventhubs/azure-messaging-eventhubs/src/producer_client.cpp |
Instruments sends and producer lifecycle. |
sdk/eventhubs/azure-messaging-eventhubs/src/private/eventhubs_utilities.hpp |
Extends partition factory inputs. |
sdk/eventhubs/azure-messaging-eventhubs/src/private/eventhubs_tracing.hpp |
Declares tracing helpers. |
sdk/eventhubs/azure-messaging-eventhubs/src/private/eventhubs_diagnostics.hpp |
Declares lifecycle helpers. |
sdk/eventhubs/azure-messaging-eventhubs/src/partition_client.cpp |
Instruments receives and receiver lifecycle. |
sdk/eventhubs/azure-messaging-eventhubs/src/eventhubs_tracing.cpp |
Implements span creation and attributes. |
sdk/eventhubs/azure-messaging-eventhubs/src/eventhubs_diagnostics.cpp |
Implements structured lifecycle records. |
sdk/eventhubs/azure-messaging-eventhubs/src/consumer_client.cpp |
Propagates tracing and diagnostics state. |
sdk/eventhubs/azure-messaging-eventhubs/README.md |
Documents tracing and lifecycle logs. |
sdk/eventhubs/azure-messaging-eventhubs/inc/azure/messaging/eventhubs/producer_client.hpp |
Exposes producer tracing configuration. |
sdk/eventhubs/azure-messaging-eventhubs/inc/azure/messaging/eventhubs/partition_client.hpp |
Stores partition tracing metadata. |
sdk/eventhubs/azure-messaging-eventhubs/inc/azure/messaging/eventhubs/event_data_batch.hpp |
Makes event count const-accessible. |
sdk/eventhubs/azure-messaging-eventhubs/inc/azure/messaging/eventhubs/consumer_client.hpp |
Exposes consumer tracing configuration. |
sdk/eventhubs/azure-messaging-eventhubs/CMakeLists.txt |
Builds diagnostics and tracing sources. |
sdk/eventhubs/azure-messaging-eventhubs/CHANGELOG.md |
Records Event Hubs observability changes. |
sdk/core/azure-core-amqp/src/impl/uamqp/amqp/message_sender.cpp |
Improves sender failure logs. |
sdk/core/azure-core-amqp/src/impl/uamqp/amqp/message_receiver.cpp |
Improves receiver failure logs. |
sdk/core/azure-core-amqp/src/impl/uamqp/amqp/connection.cpp |
Improves connection failure logs. |
sdk/core/azure-core-amqp/CHANGELOG.md |
Records uAMQP logging changes. |
doc/DistributedTracing.md |
Documents top-level tracing options. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Johnathan W (j7nw4r)
marked this pull request as ready for review
August 21, 2026 19:53
|
Azure Pipelines: Successfully started running 3 pipeline(s). 7 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Sagar Patel (sagar0207)
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Direct metrics are not emitted because Azure Core does not expose a provider-neutral meter API. Failure counters and latency histograms can be derived from the lifecycle records and span durations.
Supersedes #7339.
Fixes #7336.
Verification
AzureCliCredentialsuccessfully sent and received an event through an existing Event Hub.az.eventhubs.amqp.component.id; the bug was fixed and the live test passed on rerun.EventHubsTracingTest.*: 22/22 passed on Rust AMQP.EventHubsTracingTest.*: 22/22 passed on uAMQP.clang-format-11andgit diff --checkpass.Checklist