Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions doc/DistributedTracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,13 @@ clientOptions.Telemetry.ApplicationId = "MyApplication";
ServiceClient myServiceClient(clientOptions);
```

Some clients do not derive their options from `Azure::Core::_internal::ClientOptions`, so those options have no `Telemetry` field. These clients declare a `TracingProvider` field at the top level of their own options structure. The Event Hubs clients work this way:

```cpp
Azure::Messaging::EventHubs::ProducerClientOptions producerOptions;
producerOptions.TracingProvider = provider;
```

## Distributed Tracing Service Integration

There are two steps needed to integrate Distributed Tracing with a Service Client.
Expand Down
2 changes: 2 additions & 0 deletions sdk/core/azure-core-amqp/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@

### Other Changes

- uAMQP connection and link failure logs now include the connection container ID or link name. Transitions to an error state use the warning log level.

## 1.0.0-beta.12 (2026-05-14)

### Features Added
Expand Down
6 changes: 4 additions & 2 deletions sdk/core/azure-core-amqp/src/impl/uamqp/amqp/connection.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -356,8 +356,10 @@ namespace Azure { namespace Core { namespace Amqp { namespace _detail {
// When the connection transitions into the error or end state, it is no longer pollable.
if (connection->m_options.EnableTrace)
{
Log::Stream(Logger::Level::Verbose)
<< "Connection " << connection->m_containerId << " state changed to " << newState;
Log::Stream(
newState == CONNECTION_STATE_ERROR ? Logger::Level::Warning : Logger::Level::Verbose)
<< "AMQP connection '" << connection->m_containerId << "' state changed to "
<< newState;
}
}
// Nothing polls the connection after this point, so every operation that
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -348,8 +348,11 @@ namespace Azure { namespace Core { namespace Amqp { namespace _detail {

if (receiver->m_options.EnableTrace)
{
Log::Stream(Logger::Level::Verbose)
<< "Message receiver state change " << oldState << " -> " << newState;
Log::Stream(
newState == MESSAGE_RECEIVER_STATE_ERROR ? Logger::Level::Warning
: Logger::Level::Verbose)
<< "Message receiver link '" << receiver->m_options.Name << "' state change " << oldState
<< " -> " << newState;
}
// If the message receiver isn't open, or if it's in the process of being destroyed, ignore
// this notification.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -248,8 +248,11 @@ namespace Azure { namespace Core { namespace Amqp { namespace _detail {
sender->m_currentState = MessageSenderStateFromLowLevel(newState);
if (sender->m_options.EnableTrace)
{
Log::Stream(Logger::Level::Verbose)
<< "Message sender state changed from " << oldState << " to " << newState << ".";
Log::Stream(
newState == MESSAGE_SENDER_STATE_ERROR ? Logger::Level::Warning
: Logger::Level::Verbose)
<< "Message sender link '" << sender->m_options.Name << "' state changed from "
<< oldState << " to " << newState << ".";
}
if (sender->m_events)
{
Expand Down
3 changes: 3 additions & 0 deletions sdk/eventhubs/azure-messaging-eventhubs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
- [[#7295]](https://github.com/Azure/azure-sdk-for-cpp/issues/7295) Connection-string authentication now works on the Rust AMQP backend. `ProducerClient` and `ConsumerClient` no longer throw when the caller passes a connection string.
- [[#7254]](https://github.com/Azure/azure-sdk-for-cpp/issues/7254) `ProducerClient::Send` now builds a new sender on each retry attempt. A failed attempt discards the sender, the session, and the connection for that partition, so the next attempt builds all three again and authenticates with a current token. A send that a link detach ended previously failed for the life of the client.
- [[#7254]](https://github.com/Azure/azure-sdk-for-cpp/issues/7254) `PartitionClient::ReceiveEvents` now attaches a new receiver after a link fault, and it starts after the last event that it gave the caller. So the caller sees no duplicate event and no lost event. A permanent condition, for example `amqp:link:stolen`, still reaches the caller at once. A call that already holds events gives them back and recovers on the next call.
- [[#7336]](https://github.com/Azure/azure-sdk-for-cpp/issues/7336) Added distributed tracing. `ProducerClientOptions` and `ConsumerClientOptions` gained a `TracingProvider` field. `ProducerClient::Send` and `PartitionClient::ReceiveEvents` create one span per call when a tracing provider is set. Child spans measure AMQP link attachment, send disposition, and blocking receive latency. AMQP lifecycle logs identify the client, partition, component, and component generation for failures and rebuilds.

### Breaking Changes

### Bugs Fixed

Expand Down
4 changes: 4 additions & 0 deletions sdk/eventhubs/azure-messaging-eventhubs/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,14 @@ set(
src/consumer_client.cpp
src/event_data.cpp
src/event_data_batch.cpp
src/eventhubs_diagnostics.cpp
src/eventhubs_tracing.cpp
src/eventhubs_utilities.cpp
src/partition_client.cpp
src/partition_client_models.cpp
src/private/eventhubs_constants.hpp
src/private/eventhubs_diagnostics.hpp
src/private/eventhubs_tracing.hpp
src/private/eventhubs_utilities.hpp
src/private/package_version.hpp
src/private/processor_load_balancer.hpp
Expand Down
94 changes: 92 additions & 2 deletions sdk/eventhubs/azure-messaging-eventhubs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,13 +246,102 @@ Azure::Messaging::EventHubs::PartitionClient partitionClient
auto events = partitionClient.ReceiveEvents(1);
```

## Distributed tracing

The `ProducerClient` and the `PartitionClient` create distributed tracing spans through the Azure Core tracing API. This package does not depend on opentelemetry-cpp. The application creates the OpenTelemetry tracer provider and links the `azure-core-tracing-opentelemetry` package.

To get the spans, set the `TracingProvider` field on the client options. The Event Hubs options structs declare this field at the top level:

```cpp
#include <azure/core/tracing/opentelemetry/opentelemetry.hpp>
#include <azure/messaging/eventhubs.hpp>

// Your Event Hubs namespace connection string is available in the Azure portal.
std::string connectionString = "<connection_string>";
std::string eventHubName = "<event_hub_name>";

// Use the opentelemetry-cpp tracer provider of the application.
opentelemetry::nostd::shared_ptr<opentelemetry::trace::TracerProvider> tracerProvider
= opentelemetry::trace::Provider::GetTracerProvider();

std::shared_ptr<Azure::Core::Tracing::TracerProvider> provider
= Azure::Core::Tracing::OpenTelemetry::OpenTelemetryProvider::Create(tracerProvider);

Azure::Messaging::EventHubs::ProducerClientOptions producerOptions;
producerOptions.TracingProvider = provider;
Azure::Messaging::EventHubs::ProducerClient producer(
connectionString, eventHubName, producerOptions);

Azure::Messaging::EventHubs::ConsumerClientOptions consumerOptions;
consumerOptions.TracingProvider = provider;
Azure::Messaging::EventHubs::ConsumerClient consumer(
connectionString,
eventHubName,
Azure::Messaging::EventHubs::DefaultConsumerGroup,
consumerOptions);
```

The clients create these operation spans:

| Span name | Span kind | Notes |
|---|---|---|
| `ProducerClient.Send` | Producer | One span for each `Send` call. The span covers all the retry attempts. The overloads that take events also create the batch inside the span. Batch and vector `ProducerClient.Send` operation spans receive the `messaging.batch.message_count` attribute. Single-event sends do not. |
| `PartitionClient.ReceiveEvents` | Client | One span for each `ReceiveEvents` call. |

The clients also create child spans around calls into the AMQP transport:

| Span name | What its duration measures |
|---|---|
| `ProducerClient.AmqpLink.Open` | Sender-link attachment or reattachment. On uAMQP, an initial attachment can also include lazy connection and session establishment. A retry can create another span. |
| `ProducerClient.AmqpSend` | The synchronous AMQP send through the service disposition. Every child span receives the internal batch count, including a single-event send. The `az.eventhubs.retry.attempt` attribute identifies the attempt. |
| `PartitionClient.AmqpLink.Open` | Receiver-link attachment or reattachment. On uAMQP, an initial attachment also includes lazy connection and session establishment. |
| `PartitionClient.AmqpReceive` | Time blocked in the AMQP transport waiting for a message or transport error. More than one can occur during one `ReceiveEvents` call. |

The operation span duration is total SDK latency as observed by the caller. Subtracting the non-overlapping child-span durations from the operation duration gives the time spent in SDK work, retry delay, and locally queued message processing. The AMQP child duration is the client-observed service round-trip boundary; it does not isolate processing time inside the Event Hubs service. A receive can complete from AMQP prefetch, so `PartitionClient.AmqpReceive` can be shorter than a network round trip.

The spans have these attributes. The names follow the OpenTelemetry semantic conventions version 1.17.0, which is the schema of the `azure-core-tracing-opentelemetry` package:

| Attribute | Value |
|---|---|
| `az.namespace` | `Microsoft.EventHub` |
| `messaging.system` | `eventhubs` |
| `messaging.destination.name` | The Event Hub name on send spans. |
| `messaging.source.name` | The Event Hub name on receive spans. |
| `messaging.operation` | `publish` on a send span, `receive` on a receive span. |
| `messaging.batch.message_count` | The number of events in the operation. Batch and vector `ProducerClient.Send` spans receive this attribute. Every `ProducerClient.AmqpSend` child span receives the internal batch count. A `PartitionClient.ReceiveEvents` span gets this attribute when the call is successful. |
| `net.peer.name` | The fully qualified namespace. |

AMQP child spans also have these Event Hubs diagnostic attributes:

| Attribute | Value |
|---|---|
| `az.eventhubs.client.id` | A unique ID for the producer or consumer. It includes the configured client name when one exists and a generated UUID. |
| `az.eventhubs.partition.id` | The partition ID, or `<gateway>` for the producer gateway link. |
| `az.eventhubs.amqp.component.type` | `link`. |
| `az.eventhubs.amqp.component.name` | The AMQP link name. |
| `az.eventhubs.amqp.component.id` | A unique ID composed from the client, partition, component generation, and type. |
| `az.eventhubs.amqp.component.generation` | Starts at 1 and increases when the component is recreated. |
| `az.eventhubs.retry.attempt` | The one-based retry or rebuild attempt, when applicable. |

The instrumentation scope is `azure-messaging-eventhubs-cpp` with the package version.

When the application does not set `TracingProvider`, the client creates no spans and records nothing. There is no global fallback provider.

For the OpenTelemetry provider setup, see [Distributed Tracing in the C++ SDK][distributed_tracing].

# Troubleshooting

## Logging

The EventHubs SDK client uses the [Azure SDK log message](https://github.com/Azure/azure-sdk-for-cpp/tree/main/sdk/core/azure-core#sdk-log-messages) functionality to
enable diagnostics.
The EventHubs SDK client uses the [Azure SDK log message](https://github.com/Azure/azure-sdk-for-cpp/tree/main/sdk/core/azure-core#sdk-log-messages) functionality to enable diagnostics.

AMQP lifecycle records start with `Event Hubs AMQP lifecycle:` and contain queryable `key='value'` fields. `client.id`, `partition.id`, `component.type`, `component.name`, `component.id`, and `component.generation` identify the exact connection, session, or link. The `event` field records creation, attachment, failure, discard, close, and recreation. Failure records use the warning level; successful recreations use the informational level; initial creation and normal close records use the verbose level.

Failure records use the AMQP error-condition namespace to attribute `amqp:connection:*` and `amqp:session:*` failures to the connection or session. Other failures are attributed to the link operation where the client observed them.

The producer gives every rebuilt connection, session, and link the same component generation. A receiver reattachment increases the link generation while keeping its owning connection and session. Low-level uAMQP connection and link failure records include the same connection container ID or link name, so they can be correlated with the Event Hubs lifecycle records.

Azure Core does not currently expose a provider-neutral metrics API to service libraries. Applications can derive failure and recreation counters from lifecycle records, and latency histograms from the operation and AMQP child span durations.


## Contributing
Expand Down Expand Up @@ -296,6 +385,7 @@ Azure SDK for C++ is licensed under the [MIT](https://github.com/Azure/azure-sdk
[producer_client]: https://azuresdkdocs.z19.web.core.windows.net/cpp/azure-messaging-eventhubs/latest/class_azure_1_1_messaging_1_1_event_hubs_1_1_producer_client.html

[source]: https://github.com/Azure/azure-sdk-for-cpp/tree/main/sdk/eventhubs
[distributed_tracing]: https://github.com/Azure/azure-sdk-for-cpp/blob/main/doc/DistributedTracing.md
[azure_identity_pkg]: https://azuresdkdocs.z19.web.core.windows.net/cpp/azure-identity/latest/index.html
[default_azure_credential]: https://azuresdkdocs.z19.web.core.windows.net/cpp/azure-identity/latest/index.html#defaultazurecredential

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@
#include <azure/core/diagnostics/logger.hpp>
#include <azure/core/http/policies/policy.hpp>
#include <azure/core/internal/diagnostics/log.hpp>
#include <azure/core/internal/tracing/service_tracing.hpp>
#include <azure/core/tracing/tracing.hpp>
namespace Azure { namespace Messaging { namespace EventHubs {
namespace _detail {
class EventHubsPropertiesClient;
Expand All @@ -41,6 +43,11 @@ namespace Azure { namespace Messaging { namespace EventHubs {
/** @brief Name of the consumer client. */
std::string Name{};

/**@brief The tracer provider used to create distributed tracing spans. When this field is
* empty, the client creates no spans.
*/
std::shared_ptr<Azure::Core::Tracing::TracerProvider> TracingProvider;

private:
// The friend declaration is needed so that ConsumerClient could access CppStandardVersion,
// and it is not a struct's public field like the ones above to be set non-programmatically.
Expand Down Expand Up @@ -238,6 +245,12 @@ namespace Azure { namespace Messaging { namespace EventHubs {
/// @brief The options used to configure the consumer client.
ConsumerClientOptions m_consumerClientOptions;

/// Correlates this client and its AMQP components across lifecycle logs and spans.
std::string m_clientIdentifier;

/// @brief The factory used to create the distributed tracing spans of this client.
Azure::Core::Tracing::_internal::TracingContextFactory m_tracingFactory;

void EnsureConnection(std::string const& partitionId, Azure::Core::Context const& context);
void EnsureSession(std::string const& partitionId, Azure::Core::Context const& context);
Azure::Core::Amqp::_internal::Connection CreateConnection(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ namespace Azure { namespace Messaging { namespace EventHubs {
*/
class EventDataBatch final {
private:
std::mutex m_rwMutex;
mutable std::mutex m_rwMutex;
std::string m_partitionId;
std::string m_partitionKey;
Azure::Nullable<std::uint64_t> m_maxBytes;
Expand Down Expand Up @@ -141,7 +141,7 @@ namespace Azure { namespace Messaging { namespace EventHubs {
/** @brief Gets the number of messages in the batch
*
*/
size_t NumberOfEvents()
size_t NumberOfEvents() const
{
std::lock_guard<std::mutex> lock(m_rwMutex);
return m_marshalledMessages.size();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,11 @@
#include <azure/core/amqp/internal/message_receiver.hpp>
#include <azure/core/datetime.hpp>
#include <azure/core/http/policies/policy.hpp>
#include <azure/core/internal/tracing/service_tracing.hpp>
#include <azure/core/nullable.hpp>

#include <cstdint>

namespace Azure { namespace Messaging { namespace EventHubs {
namespace _detail {
class PartitionClientFactory;
Expand Down Expand Up @@ -99,6 +102,13 @@ namespace Azure { namespace Messaging { namespace EventHubs {
/// The link name of the receiver. A rebuild reuses this name.
std::string m_receiverName;

/// Identifies the owning consumer and partition in lifecycle logs and spans.
std::string m_clientIdentifier;
std::string m_partitionId;

/// Increases after each successful receiver reattach.
std::uint64_t m_receiverGeneration{1};

/// The offset of the last event received. A rebuild starts just after it.
Azure::Nullable<std::string> m_lastReceivedOffset;

Expand All @@ -116,26 +126,45 @@ namespace Azure { namespace Messaging { namespace EventHubs {
*/
Azure::Core::Http::Policies::RetryOptions m_retryOptions{};

/// The factory used to create the distributed tracing spans of this client.
Azure::Core::Tracing::_internal::TracingContextFactory m_tracingFactory;

/// The name of the Event Hub.
std::string m_eventHubName;

/// The fully qualified namespace of the Event Hub.
std::string m_fullyQualifiedNamespace;

/** Creates a new PartitionClient
*
* @param messageReceiver Message Receiver for the partition client.
* @param session The AMQP session that carries the message receiver.
* @param partitionUrl The address of the partition.
* @param receiverName The link name of the message receiver.
* @param clientIdentifier identifies the owning consumer in lifecycle logs and spans.
* @param partitionId identifies the partition in lifecycle logs and spans.
* @param options options used to create the PartitionClient.
* @param retryOptions controls how many times we should retry an operation in response to being
* throttled or encountering a transient error.
* @param tracingFactory factory used to create the distributed tracing spans.
* @param eventHubName the name of the Event Hub.
* @param fullyQualifiedNamespace the fully qualified namespace of the Event Hub.
*/
PartitionClient(
Azure::Core::Amqp::_internal::MessageReceiver const& messageReceiver,
Azure::Core::Amqp::_internal::Session const& session,
std::string partitionUrl,
std::string receiverName,
std::string clientIdentifier,
std::string partitionId,
PartitionClientOptions options,
Core::Http::Policies::RetryOptions retryOptions);
Core::Http::Policies::RetryOptions retryOptions,
Azure::Core::Tracing::_internal::TracingContextFactory tracingFactory,
std::string eventHubName,
std::string fullyQualifiedNamespace);

/// Closes the faulted receiver and attaches a new one starting after the last offset.
void RebuildReceiver(Core::Context const& context);
void RebuildReceiver(std::uint64_t retryAttempt, Core::Context const& context);

std::string GetStartExpression(Models::StartPosition const& startPosition);
};
Expand Down
Loading
Loading