From b65fc9dc377b051dba1cac509f12a1f4a08cd2de Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Fri, 24 Apr 2026 22:27:29 +0530 Subject: [PATCH 1/6] Update pom.xml --- code/chapter09/payment/pom.xml | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/code/chapter09/payment/pom.xml b/code/chapter09/payment/pom.xml index 65a1c5b5..07d906b0 100644 --- a/code/chapter09/payment/pom.xml +++ b/code/chapter09/payment/pom.xml @@ -12,8 +12,8 @@ UTF-8 - 17 - 17 + 21 + 21 UTF-8 UTF-8 @@ -33,7 +33,7 @@ org.projectlombok lombok - 1.18.26 + 1.18.36 provided @@ -49,17 +49,26 @@ org.eclipse.microprofile microprofile - 6.1 + 7.1 pom provided + + + org.eclipse.microprofile.telemetry + microprofile-telemetry-api + pom + 2.1 + provided + + io.opentelemetry opentelemetry-api - 1.32.0 - compile + 1.48.0 + provided @@ -73,6 +82,16 @@ ${project.artifactId} + + + org.apache.maven.plugins + maven-compiler-plugin + 3.13.0 + + 21 + + + io.openliberty.tools @@ -90,4 +109,4 @@ - \ No newline at end of file + From 2a51acd85a330f4c779da93ccb6b0eaa1e0c94e5 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Fri, 24 Apr 2026 22:40:30 +0530 Subject: [PATCH 2/6] Update README.adoc Update README.adoc as per the mp telemetry 2.1 implementation --- code/chapter09/payment/README.adoc | 957 ++++++++--------------------- 1 file changed, 262 insertions(+), 695 deletions(-) diff --git a/code/chapter09/payment/README.adoc b/code/chapter09/payment/README.adoc index e3b65582..97144f17 100644 --- a/code/chapter09/payment/README.adoc +++ b/code/chapter09/payment/README.adoc @@ -7,91 +7,11 @@ toc::[] -This microservice is part of the Jakarta EE 10 and MicroProfile 6.1-based e-commerce application. It handles payment processing and transaction management. +This microservice is part of the Jakarta EE 10 and MicroProfile-based e-commerce application. It handles payment processing and transaction management. == Features * Payment transaction processing -* Dynamic configuration management via MicroProfile Config -* RESTful API endpoints with JSON support -* Custom ConfigSource implementation -* OpenAPI documentation -* **MicroProfile Fault Tolerance with Retry Policies** -* **Circuit Breaker protection for external services** -* **Fallback mechanisms for service resilience** -* **Bulkhead pattern for concurrency control** -* **Timeout protection for long-running operations** - -== MicroProfile Fault Tolerance Implementation - -The Payment Service implements comprehensive fault tolerance patterns using MicroProfile Fault Tolerance annotations: - -=== Retry Policies - -The service implements different retry strategies based on operation criticality: - -==== Payment Authorization Retry (@Retry) -* **Max Retries**: 3 attempts -* **Delay**: 1000ms with 500ms jitter -* **Max Duration**: 10 seconds -* **Retry On**: RuntimeException, WebApplicationException -* **Use Case**: Standard payment authorization with exponential backoff - -[source,java] ----- -@Retry( - maxRetries = 3, - delay = 2000, - maxDuration = 10000 - jitter = 500, - retryOn = {RuntimeException.class, WebApplicationException.class} -) ----- - -=== Circuit Breaker Protection - -Payment capture operations use circuit breaker pattern: - -[source,java] ----- -@CircuitBreaker( - failureRatio = 0.5, - requestVolumeThreshold = 4, - delay = 5000 -) ----- - -* **Failure Ratio**: 50% failure rate triggers circuit opening -* **Request Volume**: Minimum 4 requests for evaluation -* **Recovery Delay**: 5 seconds before attempting recovery - -=== Timeout Protection - -Operations with potential long delays are protected with timeouts: - -[source,java] ----- -@Timeout(value = 3000) ----- - -=== Bulkhead Pattern - -The bulkhead pattern limits concurrent requests to prevent system overload: - -[source,java] ----- -@Bulkhead(value = 5) ----- - -* **Concurrent Requests**: Limited to 5 concurrent requests -* **Excess Requests**: Rejected immediately instead of queuing -* **Use Case**: Protect service from traffic spikes and cascading failures - -=== Fallback Mechanisms - -All critical operations have fallback methods that provide graceful degradation: - -* **Payment Authorization Fallback**: Returns service unavailable with retry instructions == Endpoints @@ -125,8 +45,8 @@ All critical operations have fallback methods that provide graceful degradation: === Prerequisites -* JDK 17 or higher -* Maven 3.6.0 or higher +* JDK 21 or higher +* Maven 3.9.0 or higher === Local Development @@ -141,14 +61,6 @@ mvn liberty:run The server will start on port 9080 (HTTP) and 9081 (HTTPS). -=== Docker - -[source,bash] ----- -# Build and run with Docker -./run-docker.sh ----- - == Project Structure * `src/main/java/io/microprofile/tutorial/PaymentRestApplication.java` - Jakarta Restful web service application class @@ -159,374 +71,12 @@ The server will start on port 9080 (HTTP) and 9081 (HTTPS). * `src/main/resources/META-INF/services/` - Service provider configuration * `src/main/liberty/config/` - Liberty server configuration -== Custom ConfigSource - -The Payment Service implements a custom MicroProfile ConfigSource named `PaymentServiceConfigSource` that provides payment-specific configuration with high priority (ordinal: 600). - -=== Available Configuration Properties - -[cols="1,2,2", options="header"] -|=== -|Property -|Description -|Default Value - -|payment.gateway.endpoint -|Payment gateway endpoint URL -|https://api.paymentgateway.com -|=== - -=== Testing ConfigSource Endpoints - -You can test the ConfigSource endpoints using curl or any REST client: - -[source,bash] ----- -# Get current configuration -curl -s http://localhost:9080/payment/api/payment-config | json_pp - -# Update configuration property -curl -s -X POST -H "Content-Type: application/json" \ - -d '{"key":"payment.gateway.endpoint", "value":"https://new-api.paymentgateway.com"}' \ - http://localhost:9080/payment/api/payment-config | json_pp - -# Test payment processing with the configuration -curl -s -X POST -H "Content-Type: application/json" \ - -d '{"cardNumber":"4111111111111111", "cardHolderName":"Test User", "expiryDate":"12/25", "securityCode":"123", "amount":100.00}' \ - http://localhost:9080/payment/api/payment-config/process-example | json_pp - -# Test basic payment authorization -curl -s -X POST -H "Content-Type: application/json" \ - http://localhost:9080/payment/api/authorize | json_pp ----- - -=== Implementation Details - -The custom ConfigSource is implemented in the following classes: - -* `PaymentServiceConfigSource.java` - Implements the MicroProfile ConfigSource interface -* `PaymentConfig.java` - Utility class for accessing configuration properties - -Example usage in application code: - -[source,java] ----- -// Inject standard MicroProfile Config -@Inject -@ConfigProperty(name="payment.gateway.endpoint") -private String endpoint; - -// Or use the utility class -String gatewayUrl = PaymentConfig.getConfigProperty("payment.gateway.endpoint"); ----- - -The custom ConfigSource provides a higher priority (ordinal: 600) than system properties and environment variables, allowing for service-specific defaults while still enabling override via standard mechanisms. - -=== MicroProfile Config Sources - -MicroProfile Config uses a prioritized set of configuration sources. The payment service uses the following configuration sources in order of priority (highest to lowest): - -1. Custom ConfigSource (`PaymentServiceConfigSource`) - Ordinal: 600 -2. System properties - Ordinal: 400 -3. Environment variables - Ordinal: 300 -4. microprofile-config.properties file - Ordinal: 100 - -==== Updating Configuration Values - -You can update configuration properties through different methods: - -===== 1. Using the REST API (runtime) - -This uses the custom ConfigSource and persists only for the current server session: - -[source,bash] ----- -curl -X POST -H "Content-Type: application/json" \ - -d '{"key":"payment.gateway.endpoint", "value":"https://test-api.paymentgateway.com"}' \ - http://localhost:9080/payment/api/payment-config ----- - -===== 2. Using System Properties (startup) - -[source,bash] ----- -# Linux/macOS -mvn liberty:run -Dpayment.gateway.endpoint=https://sys-api.paymentgateway.com - -# Windows -mvn liberty:run "-Dpayment.gateway.endpoint=https://sys-api.paymentgateway.com" ----- - -===== 3. Using Environment Variables (startup) - -Environment variable names must follow the MicroProfile Config convention (uppercase with underscores): - -[source,bash] ----- -# Linux/macOS -export PAYMENT_GATEWAY_ENDPOINT=https://env-api.paymentgateway.com -mvn liberty:run - -# Windows PowerShell -$env:PAYMENT_GATEWAY_ENDPOINT="https://env-api.paymentgateway.com" -mvn liberty:run - -# Windows CMD -set PAYMENT_GATEWAY_ENDPOINT=https://env-api.paymentgateway.com -mvn liberty:run ----- - -===== 4. Using microprofile-config.properties File - -Edit the file at `src/main/resources/META-INF/microprofile-config.properties`: - -[source,properties] ----- -# Update the endpoint -payment.gateway.endpoint=https://config-api.paymentgateway.com ----- - -Then rebuild and restart the application: - -[source,bash] ----- -mvn clean package liberty:run ----- - -==== Testing Configuration Changes - -After changing a configuration property, you can verify it was updated by calling: - -[source,bash] ----- -curl http://localhost:9080/payment/api/payment-config ----- - -== Documentation - -=== OpenAPI - -The payment service automatically generates OpenAPI documentation using MicroProfile OpenAPI annotations. - -* OpenAPI UI: `http://localhost:9080/payment/api/openapi-ui/` -* OpenAPI JSON: `http://localhost:9080/payment/api/openapi` - -=== MicroProfile Config Specification - -For more information about MicroProfile Config, refer to the official documentation: - -* https://download.eclipse.org/microprofile/microprofile-config-3.1/microprofile-config-spec-3.1.html - === Related Resources * MicroProfile: https://microprofile.io/ * Jakarta EE: https://jakarta.ee/ * Open Liberty: https://openliberty.io/ -== Troubleshooting - -=== Common Issues - -==== Port Conflicts - -If you encounter a port conflict when starting the server, you can change the ports in the `pom.xml` file: - -[source,xml] ----- -9080 -9081 ----- - -==== ConfigSource Not Loading - -If the custom ConfigSource is not loading, check the following: - -1. Verify the service provider configuration file exists at: - `src/main/resources/META-INF/services/org.eclipse.microprofile.config.spi.ConfigSource` - -2. Ensure it contains the correct fully qualified class name: - `io.microprofile.tutorial.store.payment.config.PaymentServiceConfigSource` - -==== Deployment Errors - -For CWWKZ0004E deployment errors, check the server logs at: -`target/liberty/wlp/usr/servers/mpServer/logs/messages.log` - -== Testing Fault Tolerance Features - -=== Automated Test Scripts - -The Payment Service includes several test scripts to demonstrate and validate fault tolerance features: - -==== test-payment-basic.sh - -Basic functionality test to verify core payment operations: - -* Configuration retrieval -* Simple payment processing -* Error handling - -[source,bash] ----- -# Test basic payment operations -chmod +x test-payment-basic.sh -./test-payment-basic.sh ----- - -==== test-payment-retry.sh -Tests various retry scenarios with different triggers: - -* Normal payment processing (successful) -* Failed payment with retry (card ending in "0000") -* Verification with random failures -* Invalid input handling - -[source,bash] ----- -# Test retry scenarios -chmod +x test-payment-retry.sh -./test-payment-retry.sh ----- - -==== test-payment-concurrent-load.sh - -Tests the service under concurrent load: - -* Multiple simultaneous requests -* Observing thread handling -* Response time analysis - -[source,bash] ----- -# Test service under concurrent load -chmod +x test-payment-concurrent-load.sh -./test-payment-concurrent-load.sh ----- - -==== test-payment-async.sh - -Analyzes asynchronous processing behavior: - -* Response time measurement -* Thread utilization -* Future completion patterns - -[source,bash] ----- -# Analyze asynchronous processing -chmod +x test-payment-async.sh -./test-payment-async.sh ----- - -==== test-payment-bulkhead.sh -Demonstrates the bulkhead pattern by sending concurrent requests: - -* Concurrent request handling -* Bulkhead limit verification (5 requests) -* Rejection of excess requests -* Service recovery after load reduction - -[source,bash] ----- -# Test bulkhead functionality with concurrent requests -chmod +x test-payment-bulkhead.sh -./test-payment-bulkhead.sh ----- - -==== test-payment-async-analysis.sh - -Analyzes asynchronous processing behavior: - -* Response time measurement -* Thread utilization -* Future completion patterns - -[source,bash] ----- -# Analyze asynchronous processing -chmod +x test-payment-async-analysis.sh -./test-payment-async-analysis.sh ----- - -=== Running the Tests - -To run any of these test scripts: - -[source,bash] ----- -# Make the script executable -chmod +x test-payment-bulkhead.sh - -# Run the script -./test-payment-bulkhead.sh ----- - -You can also run all test scripts in sequence: - -[source,bash] ----- -# Run all test scripts -for script in test-payment-*.sh; do - echo "Running $script..." - chmod +x $script - ./$script - echo "----------------------------------------" - sleep 2 -done ----- - -== Configuration Properties - -=== Fault Tolerance Configuration - -The following properties can be configured via MicroProfile Config: - -[cols="1,2,2", options="header"] -|=== -|Property -|Description -|Default Value - -|payment.gateway.endpoint -|Payment gateway endpoint URL -|https://api.paymentgateway.com - -|payment.retry.maxRetries -|Maximum retry attempts for payment operations -|3 - -|payment.retry.delay -|Delay between retry attempts (milliseconds) -|1000 - -|payment.circuitbreaker.failureRatio -|Circuit breaker failure ratio threshold -|0.5 - -|payment.circuitbreaker.requestVolumeThreshold -|Minimum requests for circuit breaker evaluation -|4 - -|payment.timeout.duration -|Timeout duration for payment operations (milliseconds) -|3000 - -|payment.bulkhead.value -|Maximum concurrent requests for bulkhead -|5 -|=== - -== Fault Tolerance Implementation Details - -=== Server Configuration - -The MicroProfile Fault Tolerance feature is enabled in the Liberty server configuration: - -[source,xml] ----- -mpFaultTolerance ----- - === Code Implementation ==== PaymentService Class @@ -563,238 +113,14 @@ public PaymentResponse fallbackPaymentAuthorization(PaymentRequest request) { } ---- -=== Key Implementation Benefits - -==== 1. Resilience -- Service continues operating despite external service failures -- Automatic recovery from transient failures -- Protection against cascading failures - -==== 2. User Experience -- Reduced timeout errors through retry mechanisms -- Graceful degradation with meaningful error messages -- Improved service availability - -==== 3. Operational Excellence -- Configurable fault tolerance parameters -- Comprehensive logging and monitoring -- Clear separation of concerns between business logic and resilience - -==== 4. Enterprise Readiness -- Production-ready fault tolerance patterns -- Compliance with microservices best practices -- Integration with MicroProfile ecosystem - -== MicroProfile Fault Tolerance Patterns - -=== Retry Pattern - -The retry pattern allows the service to automatically retry failed operations: - -* **@Retry**: Automatically retries failed operations -* **Parameters**: maxRetries, delay, jitter, maxDuration, retryOn, abortOn -* **Use Case**: Transient failures in external service calls - -=== Circuit Breaker Pattern - -The circuit breaker pattern prevents cascading failures: - -* **@CircuitBreaker**: Tracks failure rates and opens circuit when threshold is reached -* **Parameters**: failureRatio, requestVolumeThreshold, delay -* **States**: Closed (normal), Open (failing), Half-Open (testing recovery) -* **Use Case**: Protect against downstream service failures - -=== Timeout Pattern - -The timeout pattern prevents operations from hanging indefinitely: - -* **@Timeout**: Sets maximum duration for operations -* **Parameters**: value, unit -* **Use Case**: Prevent indefinite waiting for slow external services - -=== Bulkhead Pattern - -The bulkhead pattern limits concurrent requests: - -* **@Bulkhead**: Sets maximum concurrent executions -* **Parameters**: value, waitingTaskQueue (for async) -* **Use Case**: Prevent system overload during traffic spikes - -=== Fallback Pattern - -The fallback pattern provides alternatives when operations fail: - -* **@Fallback**: Specifies alternative method when operation fails -* **Parameters**: fallbackMethod, applyOn, skipOn -* **Use Case**: Graceful degradation for failed operations - -== Fault Tolerance Best Practices - -=== Configuring Retry Policies - -When configuring retry policies, consider these best practices: - -* **Operation Criticality**: Use more aggressive retry policies for critical operations -* **Retry Delay**: Implement exponential backoff for external service calls -* **Jitter**: Add random jitter to prevent thundering herd problems -* **Max Duration**: Set an overall timeout to prevent excessive retries -* **Abort Conditions**: Define specific exceptions that should abort retry attempts - -=== Circuit Breaker Configuration - -For effective circuit breaker implementation: - -* **Failure Ratio**: Set appropriate threshold based on expected error rates (typically 0.3-0.5) -* **Request Volume**: Set minimum request count to prevent premature circuit opening -* **Recovery Delay**: Allow sufficient time for downstream services to recover -* **Monitoring**: Track circuit state transitions for operational visibility - -=== Bulkhead Strategies - -Choose the appropriate bulkhead strategy: - -* **Synchronous Bulkhead**: Limits concurrent executions for thread-constrained systems -* **Asynchronous Bulkhead**: Provides a waiting queue for manageable load spikes -* **Isolation Levels**: Consider using separate bulkheads for different types of operations - -=== Fallback Implementation - -Implement effective fallback mechanisms: - -* **Graceful Degradation**: Return partial results when possible -* **Meaningful Responses**: Provide clear error messages to clients -* **Operation Queuing**: Queue failed operations for later processing -* **Fallback Chain**: Implement multiple fallback levels for critical operations - -=== Combining Fault Tolerance Annotations - -When combining multiple fault tolerance annotations: - -* **Execution Order**: Understand the execution order (Fallback → Retry → CircuitBreaker → Timeout → Bulkhead) -* **Compatibility**: Ensure annotations work together as expected -* **Resource Impact**: Consider the resource impact of combined annotations -* **Testing**: Test all combinations of annotation behaviors - -== Troubleshooting Fault Tolerance Issues - -=== Common Fault Tolerance Issues - -==== 1. Ineffective Retry Policies - -**Symptoms**: -* Operations fail without retrying -* Excessive retries causing performance issues - -**Solutions**: -* Verify exceptions match retryOn parameter -* Check that delay and jitter are appropriate -* Ensure maxDuration allows sufficient time for retries - -==== 2. Circuit Breaker Problems - -**Symptoms**: -* Circuit opens too frequently -* Circuit never opens despite failures -* Circuit remains open indefinitely - -**Solutions**: -* Adjust failureRatio based on expected error rates -* Increase requestVolumeThreshold if premature opening occurs -* Verify that delay allows sufficient recovery time -* Ensure exceptions are properly handled - -==== 3. Timeout Issues - -**Symptoms**: -* Operations timeout too quickly -* Timeouts not triggering as expected - -**Solutions**: -* Adjust timeout duration based on operation complexity -* Ensure timeout is shorter than upstream timeouts -* Verify that timeout unit is properly specified - -==== 4. Bulkhead Restrictions - -**Symptoms**: -* Too many rejections during normal load -* Service overloaded despite bulkhead - -**Solutions**: -* Adjust bulkhead value based on resource capacity -* Consider using asynchronous bulkheads with waiting queue -* Implement client-side load balancing for better distribution - -==== 5. Fallback Failures - -**Symptoms**: -* Fallbacks not triggering despite failures -* Fallbacks throwing unexpected exceptions - -**Solutions**: -* Verify fallback method signature matches original method -* Ensure fallback method handles exceptions properly -* Check that fallback logic is fully tested - -=== Diagnosing with Metrics - -MicroProfile Metrics provides valuable insight into fault tolerance behavior: - -[source,bash] ----- -# Total number of retry attempts -curl https://localhost:9080/metrics?name=ft_retry_retries_total - -# Bulkhead calls total -curl http://localhost:9080/metrics?name=ft_bulkhead_calls_total - -# Timeout execution duration -curl http://localhost:9080/payment/metrics/application?name=ft_timeout_executionDuration_nanoseconds ----- - -=== Server Log Analysis - -Liberty server logs provide detailed information about fault tolerance operations: - -[source,bash] ----- -tail -f target/liberty/wlp/usr/servers/mpServer/logs/messages.log | grep -E "Retry|CircuitBreaker|Timeout|Bulkhead|Fallback" ----- - -Look for messages indicating: -* Retry attempts and success/failure -* Circuit breaker state transitions -* Timeout exceptions -* Bulkhead rejections -* Fallback method invocations - -== Resources and References - -=== MicroProfile Fault Tolerance Specification - -For detailed information about MicroProfile Fault Tolerance, refer to: - -* https://download.eclipse.org/microprofile/microprofile-fault-tolerance-4.0/microprofile-fault-tolerance-spec-4.0.html - -=== API Documentation - -* https://download.eclipse.org/microprofile/microprofile-fault-tolerance-4.0/apidocs/ - -=== Fault Tolerance Guides - -* https://openliberty.io/guides/microprofile-fallback.html -* https://openliberty.io/guides/retry-timeout.html -* https://openliberty.io/guides/circuit-breaker.html -* https://openliberty.io/guides/bulkhead.html - -=== Best Practices Resources +=== Resources * https://microprofile.io/ * https://www.ibm.com/docs/en/was-liberty/base?topic=liberty-microprofile-fault-tolerance == MicroProfile Telemetry Implementation -The Payment Service implements distributed tracing using MicroProfile Telemetry 1.1, which is based on OpenTelemetry standards. This enables end-to-end visibility of payment transactions across microservices and external dependencies. +The Payment Service implements distributed tracing, metrics, and logs export using MicroProfile Telemetry 2.1. This provides end-to-end visibility of payment transactions across microservices and external dependencies. === Telemetry Configuration @@ -807,8 +133,11 @@ The service is configured to send telemetry data to Jaeger, enabling comprehensi # MicroProfile Telemetry Configuration otel.service.name=payment-service otel.sdk.disabled=false -otel.metrics.exporter=none +otel.exporter.otlp.endpoint=http://localhost:4317 +otel.traces.exporter=otlp +otel.metrics.exporter=otlp otel.logs.exporter=none +otel.traces.sampler=parentbased_always_on ---- === Automatic Instrumentation @@ -831,20 +160,24 @@ For enhanced visibility, the Payment Service also implements manual instrumentat [source,java] ---- -private Tracer tracer; // Injected tracer for OpenTelemetry +@Inject +Tracer tracer; @PostConstruct public void init() { - // Programmatic tracer access - the correct approach - this.tracer = GlobalOpenTelemetry.getTracer("payment-service", "1.0.0"); - logger.info("Tracer initialized successfully"); + paymentAttemptsCounter = meter + .counterBuilder("payment.attempts") + .setDescription("Number of payment attempts by result") + .setUnit("1") + .build(); } // Create explicit span with business context Span span = tracer.spanBuilder("payment.process") - .setAttribute("payment.amount", paymentDetails.getAmount().toString()) + .setAttribute("payment.amount", paymentDetails.getAmount().doubleValue()) .setAttribute("payment.method", "credit_card") .setAttribute("payment.service", "payment-service") + .setAttribute("payment.status", "IN_PROGRESS") .startSpan(); try (io.opentelemetry.context.Scope scope = span.makeCurrent()) { @@ -856,13 +189,16 @@ try (io.opentelemetry.context.Scope scope = span.makeCurrent()) { } catch (Exception e) { // Record error details span.recordException(e); - span.setStatus(StatusCode.ERROR, e.getMessage()); + span.setAttribute("payment.status", "FAILED"); + span.setStatus(StatusCode.ERROR, e.getMessage()); throw e; } finally { span.end(); // Always end the span } ---- +MicroProfile Telemetry 2.1 also supports CDI injection for `io.opentelemetry.api.metrics.Meter`, which this service uses for custom counters. + === Key Telemetry Points The service captures telemetry at critical transaction points: @@ -890,16 +226,14 @@ Telemetry data can be viewed in Jaeger UI: [source,bash] ---- # Start Jaeger container (if not already running) -docker run --rm --name jaeger \ +docker run -d --name jaeger \ -p 16686:16686 \ -p 4317:4317 \ -p 4318:4318 \ - -p 5778:5778 \ - -p 9411:9411 \ - jaegertracing/jaeger:2.7.0 + jaegertracing/all-in-one:latest # Access Jaeger UI -open http://localhost:16686 +xdg-open http://localhost:16686 ---- In the Jaeger UI: @@ -914,9 +248,8 @@ If telemetry data is not appearing in Jaeger: 1. **Verify Jaeger is running** with OTLP ports exposed (4317, 4318) 2. **Check Liberty server configuration** in server.xml 3. **Validate application configuration** in microprofile-config.properties -4. **Ensure trace application is enabled** with `` -5. **Check network connectivity** between the service and Jaeger -6. **Inspect Liberty server logs** for telemetry-related messages +4. **Check network connectivity** between the service and Jaeger +5. **Inspect Liberty server logs** for telemetry-related messages === Testing Telemetry @@ -933,6 +266,240 @@ curl -X POST -H "Content-Type: application/json" \ curl -s http://localhost:16686/api/services ---- +=== Step-by-Step Verification + +Use the following process to verify MicroProfile Telemetry 2.1 behavior in the payment service. + +==== 1. Build and start the payment service + +NOTE: Before starting, stop any previously running server instance to avoid port conflicts: + +[source,bash] +---- +mvn liberty:stop +---- + +[source,bash] +---- +cd code/chapter09/payment +mvn clean package +mvn liberty:run +---- + +Expected result: + +* The service starts on `http://localhost:9080/payment/`. +* Health endpoint responds with `UP` at `http://localhost:9080/health`. + +==== 2. Verify telemetry configuration is active + +Confirm these values are present in `src/main/resources/META-INF/microprofile-config.properties`: + +* `otel.sdk.disabled=false` +* `otel.exporter.otlp.endpoint=http://localhost:4317` +* `otel.traces.exporter=otlp` +* `otel.metrics.exporter=otlp` +* `otel.logs.exporter=none` + +==== 3. Start Jaeger for trace ingestion + +Open a new terminal and run the following command to start Jaeger in a Docker container: +[source,bash] +---- +docker run -d --name jaeger \ + -p 16686:16686 \ + -p 4317:4317 \ + -p 4318:4318 \ + jaegertracing/all-in-one:latest +---- + +Expected result: + +* Jaeger UI available at `http://localhost:16686`. + +==== 4. Generate telemetry traffic manually + +Send requests to exercise traces, fault tolerance, and custom span attributes: + +[source,bash] +---- +# Authorization flow +curl -X POST "http://localhost:9080/payment/api/authorize?amount=100" + +# Payment details flow +curl -X POST "http://localhost:9080/payment/api/payments" \ + -H "Content-Type: application/json" \ + -d '{"cardNumber":"4111111111111111","cardHolderName":"Test User","expiryDate":"12/25","securityCode":"123","amount":75.50}' + +# Verification flow (creates transaction_id) +curl -X POST "http://localhost:9080/payment/api/verify" \ + -H "Content-Type: application/json" \ + -d '{"cardNumber":"4111111111111111","cardHolderName":"Test User","expiryDate":"12/25","securityCode":"123","amount":75.50}' + +# Circuit breaker path +curl "http://localhost:9080/payment/api/health/gateway" + +# Async + bulkhead path +curl -X POST "http://localhost:9080/payment/api/notify/PAY-12345?recipient=ops@example.com" +---- + +==== 5. Verify traces in Jaeger + +In Jaeger UI: + +1. Open `http://localhost:16686` in a browser. +2. In the *Search* panel, select service `payment-service`. +3. Set *Lookback* to `Last 15 min` (or a wider range if needed). +4. Click *Find Traces*. +5. Open one trace from `/payment/api/payments` or `/payment/api/verify`. +6. In the trace view, confirm you can see: +7. Root HTTP span for the inbound request. +8. Child/internal spans such as `payment.process`. +9. Span attributes including `payment.amount`, `payment.method`, and `payment.status`. +10. For failed requests, verify error state/events are visible on spans. +11. For `/payment/api/verify`, confirm trace includes verification workflow spans and request timing. + +If no traces appear in UI: + +1. Generate fresh traffic from Step 4. +2. Verify `otel.exporter.otlp.endpoint` points to the running Jaeger OTLP endpoint. +3. Re-run search with broader lookback window. + +CLI check (optional): + +[source,bash] +---- +curl -s http://localhost:16686/api/services +---- + +Expected result: + +* Response includes `payment-service`. + +==== 6. Verify metrics publication + +[source,bash] +---- +curl -s http://localhost:9080/metrics | grep -E "http_server_request_duration|ft_|jvm_" | head -n 40 +---- + +Expected result: + +* HTTP server metrics are present (`http_server_request_duration...`). +* Fault tolerance metrics are present (`ft_retry...`, `ft_circuitbreaker...`, `ft_bulkhead...`, `ft_timeout...`). +* JVM/runtime metrics are present (`jvm_...`, `memory_...`, `cpu_...`). + +==== 7. Verify logs and trace correlation signals + +Watch server logs while sending requests: + +[source,bash] +---- +tail -f target/liberty/wlp/usr/servers/mpServer/logs/messages.log +---- + +Expected result: + +* You see payment flow logs and fault tolerance events. +* If your telemetry backend supports OTLP logs, logs are exported through the configured OTLP endpoint. +* If the backend does not implement OTLP logs/metrics services, traces can still succeed while logs/metrics export reports `UNIMPLEMENTED`. + +==== 8. Verify OpenAPI endpoint used by this module + +[source,bash] +---- +curl -I http://localhost:9080/openapi +curl -I http://localhost:9080/openapi/ui/ +---- + +Expected result: + +* Both endpoints return HTTP `200`. + +==== 9. Clean up local verification environment + +[source,bash] +---- +# Stop Jaeger container +docker rm -f jaeger + +# Stop Liberty server (from mvn liberty:run terminal) +# Press Ctrl+C +---- + +=== Verifying Telemetry with Zipkin + +Use this flow when you want to validate MicroProfile OpenTelemetry traces in Zipkin instead of Jaeger. + +==== 1. Start Zipkin + +[source,bash] +---- +docker run -d --name zipkin \ + -p 9411:9411 \ + openzipkin/zipkin:latest +---- + +Expected result: + +* Zipkin UI is available at `http://localhost:9411`. + +==== 2. Run the payment service with Zipkin trace exporter + +From `code/chapter09/payment`, start Liberty with temporary environment overrides: + +[source,bash] +---- +OTEL_TRACES_EXPORTER=zipkin \ +OTEL_EXPORTER_ZIPKIN_ENDPOINT=http://localhost:9411/api/v2/spans \ +mvn liberty:run +---- + +Notes: + +* This overrides trace export for the current run only. +* Existing `otel.logs.exporter=none` remains valid for local verification. + +==== 3. Generate trace traffic + +[source,bash] +---- +curl -X POST "http://localhost:9080/payment/api/payments" \ + -H "Content-Type: application/json" \ + -d '{"cardNumber":"4111111111111111","cardHolderName":"Test User","expiryDate":"12/25","securityCode":"123","amount":75.50}' + +curl -X POST "http://localhost:9080/payment/api/verify" \ + -H "Content-Type: application/json" \ + -d '{"cardNumber":"4111111111111111","cardHolderName":"Test User","expiryDate":"12/25","securityCode":"123","amount":75.50}' +---- + +==== 4. Validate traces in Zipkin UI + +1. Open `http://localhost:9411`. +2. Select service `payment-service`. +3. Click *Run Query*. +4. Open a trace for `/payment/api/payments` or `/payment/api/verify`. +5. Verify spans include `payment.process` and `payment.verify`. +6. Confirm span tags include `payment.amount`, `payment.method`, and `payment.status`. + +Optional API check: + +[source,bash] +---- +curl -s http://localhost:9411/api/v2/services +---- + +Expected result: + +* Response includes `payment-service`. + +==== 5. Clean up Zipkin + +[source,bash] +---- +docker rm -f zipkin +---- + === Benefits of Telemetry Implementation 1. **End-to-End Transaction Visibility**: Follow payment flows across services @@ -940,4 +507,4 @@ curl -s http://localhost:16686/api/services 3. **Error Detection**: Quickly locate and diagnose failures 4. **Dependency Analysis**: Understand service dependencies and impacts 5. **Business Insights**: Correlate technical metrics with business outcomes -6. **Operational Excellence**: Improve MTTR and system reliability \ No newline at end of file +6. **Operational Excellence**: Improve MTTR and system reliability From e1ab4abf2549761d4eb3aaa2ddcb1961b6762d2b Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Fri, 24 Apr 2026 22:42:12 +0530 Subject: [PATCH 3/6] Update index.html --- code/chapter09/payment/src/main/webapp/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/code/chapter09/payment/src/main/webapp/index.html b/code/chapter09/payment/src/main/webapp/index.html index f7ba4adc..7d20f937 100644 --- a/code/chapter09/payment/src/main/webapp/index.html +++ b/code/chapter09/payment/src/main/webapp/index.html @@ -262,7 +262,7 @@

Links

MicroProfile Config, Fault Tolerance & Telemetry Demo | Payment Service

-

Powered by Open Liberty, MicroProfile 6.1 (Config 3.0, Fault Tolerance 4.0, Telemetry 1.1)

+

Powered by Open Liberty, MicroProfile 7.1 (Config 3.1, Fault Tolerance 4.1, Telemetry 2.1)

From 0641ea064e77630d46a7aac3f6653f24f39745a2 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Fri, 24 Apr 2026 22:47:17 +0530 Subject: [PATCH 4/6] Update server.xml --- code/chapter09/payment/src/main/liberty/config/server.xml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/code/chapter09/payment/src/main/liberty/config/server.xml b/code/chapter09/payment/src/main/liberty/config/server.xml index 2d5c7290..c680bc91 100644 --- a/code/chapter09/payment/src/main/liberty/config/server.xml +++ b/code/chapter09/payment/src/main/liberty/config/server.xml @@ -1,7 +1,7 @@ + microProfile-7.1 jakartaEE-10.0 - microProfile-6.1 restfulWS jsonp jsonb @@ -11,7 +11,6 @@ mpHealth mpMetrics mpTelemetry - mpOpenTracing mpFaultTolerance From a7904c4df4440fd4c4fd3d2ebaeb3cb6a7444255 Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Fri, 24 Apr 2026 22:49:01 +0530 Subject: [PATCH 5/6] Update PaymentResource.java --- .../payment/resource/PaymentResource.java | 73 +++++++++++++++++++ 1 file changed, 73 insertions(+) diff --git a/code/chapter09/payment/src/main/java/io/microprofile/tutorial/store/payment/resource/PaymentResource.java b/code/chapter09/payment/src/main/java/io/microprofile/tutorial/store/payment/resource/PaymentResource.java index ae9258fb..b48c0c07 100644 --- a/code/chapter09/payment/src/main/java/io/microprofile/tutorial/store/payment/resource/PaymentResource.java +++ b/code/chapter09/payment/src/main/java/io/microprofile/tutorial/store/payment/resource/PaymentResource.java @@ -11,8 +11,10 @@ import io.microprofile.tutorial.store.payment.service.PaymentService; import jakarta.enterprise.context.RequestScoped; import jakarta.inject.Inject; +import jakarta.ws.rs.GET; import jakarta.ws.rs.POST; import jakarta.ws.rs.Path; +import jakarta.ws.rs.PathParam; import jakarta.ws.rs.Produces; import jakarta.ws.rs.QueryParam; import jakarta.ws.rs.Consumes; @@ -22,10 +24,13 @@ import java.math.BigDecimal; import java.util.concurrent.CompletionStage; import java.util.UUID; +import java.util.logging.Logger; @RequestScoped @Path("/") public class PaymentResource { + + private static final Logger logger = Logger.getLogger(PaymentResource.class.getName()); @Inject @ConfigProperty(name = "payment.gateway.endpoint") @@ -131,4 +136,72 @@ public Response verifyPaymentWithTelemetry(PaymentDetails paymentDetails) } } + @GET + @Path("/health/gateway") + @Produces(MediaType.APPLICATION_JSON) + @Operation(summary = "Check gateway health", description = "Check payment gateway health with circuit breaker protection") + @APIResponses(value = { + @APIResponse(responseCode = "200", description = "Gateway is healthy"), + @APIResponse(responseCode = "503", description = "Gateway is unavailable") + }) + public Response checkGatewayHealth() { + try { + boolean healthy = paymentService.checkGatewayHealth(); + + if (healthy) { + return Response.ok() + .entity("{\"status\":\"healthy\",\"message\":\"Payment gateway is operational\"}") + .build(); + } + + return Response.status(Response.Status.SERVICE_UNAVAILABLE) + .entity("{\"status\":\"unhealthy\",\"message\":\"Payment gateway is not responding\"}") + .build(); + } catch (Exception e) { + logger.warning("Gateway health check failed: " + e.getMessage()); + return Response.status(Response.Status.SERVICE_UNAVAILABLE) + .entity("{\"status\":\"circuit_open\",\"message\":\"Circuit breaker is open - gateway appears to be down\"}") + .build(); + } + } + + @POST + @Path("/notify/{paymentId}") + @Produces(MediaType.APPLICATION_JSON) + @Operation(summary = "Send payment notification", description = "Send asynchronous payment notification with bulkhead protection") + @APIResponses(value = { + @APIResponse(responseCode = "200", description = "Notification sent or queued"), + @APIResponse(responseCode = "503", description = "Bulkhead rejected request") + }) + public CompletionStage sendNotification( + @PathParam("paymentId") String paymentId, + @QueryParam("recipient") String recipient + ) { + + if (recipient == null || recipient.isEmpty()) { + recipient = "default@example.com"; + } + + return paymentService.sendPaymentNotification(paymentId, recipient) + .thenApply(result -> { + logger.info("Notification result: " + result); + return Response.ok() + .entity("{\"status\":\"success\",\"message\":\"" + result + "\"}") + .build(); + }) + .exceptionally(ex -> { + logger.warning("Notification failed: " + ex.getMessage()); + + if (ex.getMessage() != null && ex.getMessage().contains("BulkheadException")) { + return Response.status(Response.Status.SERVICE_UNAVAILABLE) + .entity("{\"status\":\"rejected\",\"message\":\"Too many concurrent notifications - please try again later\"}") + .build(); + } + + return Response.status(Response.Status.INTERNAL_SERVER_ERROR) + .entity("{\"status\":\"error\",\"message\":\"Notification processing failed\"}") + .build(); + }); + } + } From 629787f4818103608946d3cdee35932fb20c555b Mon Sep 17 00:00:00 2001 From: Tarun Telang Date: Fri, 24 Apr 2026 23:03:15 +0530 Subject: [PATCH 6/6] Update microprofile-config.properties --- .../main/resources/META-INF/microprofile-config.properties | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/code/chapter09/payment/src/main/resources/META-INF/microprofile-config.properties b/code/chapter09/payment/src/main/resources/META-INF/microprofile-config.properties index d41c5ed4..5fea4a8e 100644 --- a/code/chapter09/payment/src/main/resources/META-INF/microprofile-config.properties +++ b/code/chapter09/payment/src/main/resources/META-INF/microprofile-config.properties @@ -13,5 +13,8 @@ io.microprofile.tutorial.store.payment.service.PaymentService/processPayment/Ret # MicroProfile Telemetry Configuration otel.service.name=payment-service otel.sdk.disabled=false -otel.metrics.exporter=none -otel.logs.exporter=none \ No newline at end of file +otel.exporter.otlp.endpoint=http://localhost:4317 +otel.traces.exporter=otlp +otel.metrics.exporter=otlp +otel.logs.exporter=none +otel.traces.sampler=parentbased_always_on