fix: Fixes for OTLP tracing#2268
Conversation
Recent updates to the opentelemetry crates have broken OTLP tracing, such that all spans are being filtered out. This is because opentelemetry no longer emits `code.namespace` metadata for spans, meaning CarbideSpanSampler would reject all traces. Fix this by explicitly setting an attribute `carbide.trace_root = true` for toplevel spans we want to emit, and have CarbideSpanSampler check for that attribute instead of the code module, which is error-prone, especially when we do any refactoring. Then we rely on ParentBased sampling to include all child spans of that root. Also, fix an issue where we'd begin a span from within StateController::Builder once at startup and never close it, leading to memory leaks as spans never got closed. Move the span setup to StateProcessor::run_single_iteration instead, so that the span closes at a natural time. Spans all seem to be properly closing themselves now.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
| physical_slot_number: None, | ||
| revision_id: None, | ||
| topology_id: None, | ||
| remediation_error: None, |
There was a problem hiding this comment.
Note, this was an error on the current main branch, had to include this so it all compiles. (Other PR's are probably having the same issues.)
|
@Matthias247 to have a look at the changes to where the root controller iteration span is emitted. (We probably need to overhaul the spans so that the individual object processors do their own spans, since a lot of refactoring has happened since these were set up.) |
1 participant
Description
Recent updates to the opentelemetry crates have broken OTLP tracing, such that all spans are being filtered out. This is because opentelemetry no longer emits
code.namespacemetadata for spans, meaning CarbideSpanSampler would reject all traces.Fix this by explicitly setting an attribute
carbide.trace_root = truefor toplevel spans we want to emit, and have CarbideSpanSampler check for that attribute instead of the code module, which is error-prone, especially when we do any refactoring. Then we rely on ParentBased sampling to include all child spans of that root.Also, fix an issue where we'd begin a span from within StateController::Builder once at startup and never close it, leading to memory leaks as spans never got closed. Move the span setup to StateProcessor::run_single_iteration instead, so that the span closes at a natural time. Spans all seem to be properly closing themselves now.
Type of Change
Related Issues (Optional)
#2267
Breaking Changes
Testing
Additional Notes