Conversation
gnurizen
force-pushed
the
otlp
branch
4 times, most recently
from
September 1, 2026 01:55
b19c12b to
45ec8d5
Compare
Groundwork for a second profile encoding. No behavior change; every existing
flag means what it did.
arrowReporter had grown to hold three separable concerns: the label pipeline,
the symbol uploader and metrics bridge, and the arrow encoding itself. Only the
last is specific to a wire format, so the other two move out where a second
backend can use them without either copying them or importing arrow.
reporter/labeler.go processLabeler: the label LRU, metadata providers,
relabel configs, external labels, and the three
disable flags. labelRetrievalResult now carries the
resource and sample label sets separately instead of
merging them, because a non-arrow encoding needs to
know which is which. The oomprof state it reads for
job="oomprof" becomes an atomic.Pointer, since it can
only be assigned after SetupWithReporter returns and
was previously written while readers ran.
reporter/shared.go execTracker, metricsBridge, reporterCounters, and the
logger()/tracer() accessors. reporterCounters holds
the format-neutral series; the four that describe
arrow write calls stay registered by New() so a
different backend does not publish permanently-zero
series.
reporter/frames.go classifyFrame, lifted out of appendLocationV2. It
returns a format-neutral description of a frame --
mapping file, build IDs, frame type, optional
function and line -- so an encoder consumes that
rather than reading libpf.Frame itself. The point is
that a fork bump changing frame semantics produces one
compile error instead of two silent divergences.
reporter/config.go Config, replacing reporter.New's 24 positional
parameters, plus newSharedReporterParts building the
pieces above the same way for any backend.
--remote-store-format supersedes --use-v2-schema. The bool was adequate for two
encodings and is not extensible; --use-v2-schema=false still maps onto
arrow-v1, so existing invocations are unaffected. The enum has two values here.
newLogProvider and newTracerProvider now take an exporter rather than a
*grpc.ClientConn, which is what lets a caller point the agent's own telemetry
somewhere other than the remote store.
Fixed while touching this code:
The vtproto codec was registered as a side effect of the remote-store dial.
It is a process-global replacement of gRPC's "proto" codec, so relying on
dial order is fragile; RegisterProtoCodec() now runs before any dial.
Tracer() dereferenced a nil provider despite its doc comment promising a
no-op fallback. Reachable through offline mode plus a probe config.
reporter.New's Start() error was discarded in main.go, and the arrow Start()
leaked its reporting context when offline-mode MkdirAll failed.
The three labelsForTID tests move to reporter/labeler_test.go against
processLabeler and pass unchanged, which is the assertion that the extraction
preserved behavior.
Selected with --remote-store-format=otlp, alongside the existing arrow v1 and v2 paths. Verified end to end against two independent OTLP/profiles receivers, both rendering symbolized flamegraphs from the agent's output. The conversion is ours rather than upstream's. Upstream's lives in reporter/internal/pdata, which is unimportable, and no extension point would have helped: pprofile.Profile carries exactly one SampleType, so a memory profile with four value axes needs four Profiles, and oomprof never passes through an origin at all; baseReporter also rejects the CUDA and GPU-PC origins before the encoding is reached. Owning it means every origin the tracer produces has a sample type, including GPU and memory. reporter/pprofile.go builds one Profiles per flush with a request-global dictionary, partitioning into Profiles by (sample type, unit, period type, period) so GPU-PC's per-PID period lands on its own Profile. It reserves mapping index 0 as the "no mapping" sentinel, because Location.MappingIndex has no presence flag. Memory frames get a special case: ReportMemoryTraces stashes the build ID in FunctionName and the executable path in SourceFile, which a generic classifier would emit as a function literally named after a build ID. Flags: --remote-store-format gains otlp. All three formats ride the existing remote-store connection, so OTLP inherits its TLS, auth, retry and client metrics rather than opening a second one. --remote-store-compression=gzip|none applies to the OTLP export only; the arrow paths compress inside their own IPC payload. gzip at level 1 rather than the level-6 default, since the CPU is spent on a node being profiled: level 1 runs at 192 MB/s against level 6's 108 MB/s for 6.6% of the ratio. gzip is the only choice because it is the only codec grpc-go ships on both sides; anything else fails at the first flush against a receiver that did not install it, which is not worth trading for a better ratio on a stream that runs at tens of KiB/s. --debuginfo-address points symbol upload at its own host, inheriting the remote store's credentials. Symbols travel the DebuginfoService protocol whatever encoding profiles use, so OTLP mode is not standalone, and flag validation says so rather than failing at the first upload. Also fixed here: service.name was set only from APMServiceName, so a system-wide profiler sent anonymous resources for nearly everything and consumers keying on it named every process by a hash. Now falls back to comm, then the executable's basename. The remote store was dialed whenever the agent was not in offline mode, which was right when "not offline" implied "has a remote store". --remote-store-format=otlp with --debuginfo-upload-disable and no address is a third case, and it died on "received empty target in Build()". The dial is now gated on a non-empty address, which also makes the OTLP path's grpcConn == nil check reachable instead of dead. Known gaps: OTLP has no field for aggregation temporality, so a consumer must derive delta from the sample type; off-CPU is wallclock on arrow and off_cpu on OTLP, which is still an open naming decision; and offline mode has no OTLP representation and is rejected at flag validation.
The upstream Go interpreter reads gopclntab out of the binary and resolves function, file and line in-agent. It was unconditionally disabled with no way to turn it on, because Parca symbolizes native Go frames server-side from uploaded debuginfo and doing it in both places pays for the same answer twice. That reasoning holds only when the backend has a symbolizer. Against one that does not, Go frames arrive as bare addresses and nothing later can name them, since the agent is the only place gopclntab was available. Off by default, so nothing changes for a Parca backend. On, it costs agent CPU and memory per Go process and needs no debuginfo upload at all. Independent of the profile encoding: it works the same on every --remote-store-format.
newProviders built an sdklog.LoggerProvider whenever Config.GRPCConn was non-nil, which is every non-offline run. --otlp-logging only controlled whether main.go attached the logrus hook to the other end, so the usual configuration ran a batch processor nothing wrote to and nothing read from. It is not free. The processor allocates on its own timer regardless of traffic, and an allocation profile over a 120s window put slices.Clone[[]sdk/log.Record] at 9% of all bytes allocated by the agent -- 140 to 156 MiB, roughly 1.3 MiB/s of garbage in a run that had logging switched off. Config gains ExportSelfLogs, set from --otlp-logging, and the logs branch of newProviders is gated on it. With it false the provider stays nil, Logger() hands out the OTel no-op it already falls back to, and shutdownProviders skips it -- all three paths existed for the offline case. Traces are deliberately untouched. A TracerProvider costs nothing until a span is created, and the probe service needs the one it gets today. Verified by running one binary both ways against an idle agent and diffing allocation profiles over 60s: with --otlp-logging the clone appears at 65.6 MiB, 6.3% of bytes; without it, no log.Record allocation at all.
Whole-agent measurements could not answer which encoder is cheaper. Under a realistic workload the agent's allocation profile is 45-53% eh_frame extraction, so an encoder difference of any plausible size is inside the noise, and the run-to-run variance in how many new executables a window happens to discover is larger than the effect being looked for. These benchmarks remove everything except the reporter. A .padata file named by PARCA_BENCH_CORPUS is decoded back into the ReportTraceEvent arguments that produced it, once, outside the timer; the benchmark replays them. With no corpus set the benchmarks skip, so `go test ./...` stays hermetic. The endpoint is a bufconn server with a raw codec and an unknown-service handler: it accepts any method, decodes nothing, and replies with an empty message. That detail is load-bearing. Registering the generated services instead charges the benchmark for the receiver's unmarshal, which is wildly asymmetric -- an arrow request is one bytes field, an OTLP request rebuilds every Sample, Stack and Location in pdata. It inflated OTLP by 41% (603,699 allocs/op against the 356,951 the agent actually pays) before the codec was swapped in. The client side stays real: real codec, real compression, real HTTP/2 framing. Two pairs of benchmarks. Encode* is a whole flush interval, accumulate plus serialize plus send. *Accumulate drops the flush, so the difference between them is what serialization costs. Reconstruction is faithful except for FileID: the arrow schema stores a mapping as (filename, build ID) and keeps no FileID, so one is synthesized by hashing that pair. The encoders use it as a dictionary key and as the htlhash build-ID fallback, both of which only need stability and distinctness, so cardinality is right even though the values are not real. TestBenchCorpusSanity guards the decode -- an off-by-one in the ListView offsets would produce empty stacks, which would benchmark nothing and look fast doing it.
internStack appended one location index per frame. Every Append on an empty Int32Slice regrows the backing array from zero, and at a mean stack depth of 33 that made pcommon.Int32Slice.Append the single largest allocator in the whole OTLP encode path -- 1.44M objects, 12.3% of everything allocated during accumulation, ahead of every pdata constructor. Collect the indices in a scratch slice reused across stacks, size the destination once, and append them together. The scratch is never aliased by the dictionary, which copies on Append. Measured with BenchmarkEncodeOTLP over a 20,000-sample corpus, 33.1 frames per sample, 231 PIDs, 83 mappings: allocs/op 356,951 -> 289,561 (-19%) B/op 31.9MB -> 28.5MB (-10%) ns/op 167.6 -> 159.1 (-5%) It does not close the gap to arrow v2, which is still about 1.6x cheaper on this corpus, but it is the largest single item and the cheapest to fix.
labelsForTID formats a CPU number and a thread ID on every sample, on the path both backends take. fmt.Sprint boxes each argument into an interface and goes through the reflection formatter; strconv.FormatUint does neither, and returns a shared string with no allocation at all for values below 100. Over 20,000 samples per iteration, 3 runs of 8: 30,140 allocations removed from each backend, -10% for OTLP and -23% for arrow v2, which allocates less to begin with. Wall time moves about 2%. A lookup table for larger values was tried and removed: it measured identically, because strconv already has the small-value fast path and thread IDs are mostly too large for any table worth keeping.
brancz
approved these changes
Sep 1, 2026
| case CompressionNone, "": | ||
| return nil, false, nil | ||
|
|
||
| case CompressionGzip: |
Member
There was a problem hiding this comment.
I think the agent needs to work together with a collector and in-cluster network traffic spending the cpu is not necessary. Doesn't hurt to have this.
Contributor
Author
There was a problem hiding this comment.
Yeah I made a mental note to turn this off in our local testing rig and it should probably go in our docs somewhere that none is preferred if you are just sending profiles to a sidecar pod over loopback.
|
|
||
| attrs := sample.AttributeIndices() | ||
| s.SampleLabels.Range(func(l labels.Label) { | ||
| switch l.Name { |
Member
There was a problem hiding this comment.
are these really the only 4 that semconv values exist for? Maybe it would be better if we integrated this mapping directly into the metadata providers directly so it's easier to maintain?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds OTLP/profiles as a third profile encoding alongside arrow v1 and v2,
selected with
--remote-store-format=otlp. Verified end to end against twoindependent OTLP/profiles receivers, both rendering symbolized flamegraphs.
Each commit builds and tests on its own:
reporter: extract the backend-independent parts of arrowReporter— purerefactor, no behavior change. The label pipeline, symbol uploader, metrics
bridge and frame classifier move out of
arrowReporter, andreporter.Newtakes a
Configinstead of 24 positional parameters.--remote-store-formatsupersedes the
--use-v2-schemabool, with two values at this point.reporter: add OTLP/profiles as a third export format— the encoder, plus--remote-store-compressionand--debuginfo-address. The conversion is oursrather than upstream's, because
pprofile.Profilecarries one SampleType andupstream rejects the GPU origins before the encoding is reached.
main: add --symbolize-go— independent of the encoding. Resolves Goframes from gopclntab in-agent, for backends with no symbolizer service. Off
by default.
reporter: only build the self-logs provider when logs are exported— thebatch processor was running in every non-offline configuration regardless of
--otlp-logging, allocating ~9% of the agent's total bytes.reporter: benchmark both encoders against a null gRPC endpoint— replaysa real
.padatacorpus through both paths so the encoders can be comparedwithout eh_frame extraction swamping the result.
reporter: append a stack's location indices in one call— the largestsingle allocator in the OTLP path; −19% allocations.
On the same 20,000-sample corpus arrow v2 is ~1.6x cheaper in CPU and allocates
2.2x fewer objects, while OTLP is ~2.4x smaller on the wire. Both are a fraction
of a percent of a core at realistic sample rates.
Two known gaps, both on the store side rather than the agent: off-CPU is
wallclockon arrow andoff_cpuon OTLP, which is a naming decision still tobe made; and OTLP has no field for aggregation temporality, so a consumer has to
derive delta from the sample type. Rationale for each change is in its commit
message.