Pure-Python opentelemetry-proto (drop the protobuf dependency) [1/4 RFC] by ocelotl · Pull Request #5503 · open-telemetry/opentelemetry-python

ocelotl · 2026-08-03T02:37:21Z

What

Replace the google.protobuf-generated message classes in opentelemetry-proto with a hand-written pure-Python protobuf implementation, and drop the protobuf dependency (protobuf>=5.0, <8.0) entirely.

The public opentelemetry.proto.* import namespace is preserved: real code lives under a private opentelemetry._proto package, and the existing opentelemetry.proto.* modules become thin re-export shims (from opentelemetry._proto... import *). Downstream code importing opentelemetry.proto.trace.v1.trace_pb2 etc. keeps working unchanged.

Why

protobuf ships a compiled C extension (upb) and enforces a narrow, moving version range. That causes real friction:

Injection / zero-code instrumentation conflicts — when the SDK is injected into an arbitrary target process (K8s operator/injector), the bundled protobuf can collide with the version the target app pins, producing hard-to-diagnose ABI/version errors. A pure-Python encoder is safe to inject.
Dependency-resolution friction — the >=5.0,<8.0 range regularly clashes with other libraries (gRPC stacks, ML tooling, cloud SDKs).
Wheel/ABI portability — no compiled component means it runs anywhere CPython runs.

The OTLP exporters only ever serialize protobuf; they never parse arbitrary messages. The only decode needed is the export-service response, which is empty. So a small, auditable encode-only implementation is sufficient — this is not a general-purpose protobuf runtime.

Scope of this PR

This is the base of a 4-PR stacked series that makes the whole OTLP export path dependency-light. Merge order:

opentelemetry-proto → pure-Python (this PR)
opentelemetry-exporter-otlp-proto-common → pure-Python backend
opentelemetry-exporter-otlp-proto-http → pure-Python + stdlib urllib (drops requests)
opentelemetry-exporter-otlp-proto-grpc → pure-Python gRPC (drops grpcio)

Because this org's fork model can't host intermediate base branches upstream, each PR targets main and its diff is cumulative (this PR's changes appear in all four). Isolated per-package diffs are viewable via the fork compare links noted in PRs 2–4.

Known gaps / discussion points (why this is a draft)

profiles signal not ported. The pure-Python impl currently covers common, resource, trace, metrics, logs and the collector service messages, but not opentelemetry.proto.profiles. That namespace is removed here and would need to be added before this could replace the current package for profiles users.
Codegen vs hand-maintenance. The message classes should ideally be generated from the same .proto sources the project vendors, so schema changes stay mechanical. This PR carries a hand-written implementation; agreeing on a generator is part of the discussion.
Structure. The _proto (real) + proto (shim) split is one option; collapsing into proto directly is another. Happy to go either way.
Conformance. Output should be byte-for-byte identical to the protobuf-generated serialization; a differential test against the current encoders would be worth adding.
CI will not be green as-is (tests and docs need adaptation) — this is intentionally an RFC to gauge interest in the approach.

The reference implementation this is derived from has been running in a downstream distribution.

Stack (merge in order): #5503 (proto) → #5504 (common) → #5505 (http) → #5506 (grpc)

opentelemetry-pr-dashboard · 2026-08-14T16:59:01Z

Pull request dashboard status

Waiting on the author · refreshed 2026-10-17 02:49 UTC

Move out of draft to request review.

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

ocelotl · 2026-09-07T12:51:23Z

WIP

herin049 · 2026-09-09T22:35:17Z

Hey @ocelotl, thanks for this -- I also like the idea of having a dependency-less protobuf/gRPC exporter. I have a few questions about the overall approach:

Is there a reason this can't be a separate package? Consumers that are fragile to additional dependencies can opt-in to utilizing this new package instead of the protobuf one, instead of forcing it on all consumers.
Can you include some more detailed information on the performance overhead this introduces compared to the native upb backend? I recall that you mention that the overhead is very significant when directly comparing serialization time, but is less significant in the context of the total export duration. While the latter point is a valid argument, I would still like to see how much of an increase in memory/CPU this would introduce in the context of the entire OTel Python SDK. For example if the relative memory/CPU increase across the SDK as a whole with these changes is <5%, it might be acceptable, but if it were to double the memory/CPU overhead of the SDK as a whole, I would personally see that as unacceptable.
I know I have raised this before, but have we considered using the Rust prost crate to create a native, dependency-less wheel to avoid the protobuf dependency. This would avoid the manual generation of message types, would eliminate the need to implement protobuf from scratch and would eliminate the performance downsides.
If we were to proceed with some variation of this approach, we would need to implement a protoc plugin to automatically generate all message types (i.e. similar to what is done for JSON) -- manually modifying the message types every time the protocol is updated isn't acceptable in my eyes.

xrmx · 2026-09-10T16:24:32Z

Another option would be to try protobuf-py. I've played a bit in converting the contrib opamp client to protobuf-py (python + rust) open-telemetry/opentelemetry-python-contrib#5006

aabmass · 2026-09-10T16:26:48Z

protobuf ships a compiled C extension (upb) and enforces a narrow, moving version range. That causes real friction:

https://protobuf.dev/support/cross-version-runtime-guarantee/ the range shouldn't be too narrow

herin049 · 2026-09-10T17:23:56Z

Another option would be to try protobuf-py. I've played a bit in converting the contrib opamp client to protobuf-py (python + rust) open-telemetry/opentelemetry-python-contrib#5006

I think the solution we're trying to implement here is to have an installable exporter that can never introduce dependency conflicts with user provided packages. Switching to protobuf-py would still leave possibilities of dependency conflicts (e.g. if user packages depend on different versions of protobuf-py) unless we vendored it directly into the exporters.

Having a Python native protobuf implementation is somewhat irrelevant here.

github-actions · 2026-10-03T03:37:25Z

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

Swap the google.protobuf-generated message classes for hand-written pure-Python encoders under opentelemetry._proto, keeping the public opentelemetry.proto.* import namespace via thin re-export shims. Removes the protobuf (and native upb) dependency entirely; only the serialize path used by the OTLP exporters is implemented (plus the empty export-service response decode).

Restore the parts of the protobuf message API that the unchanged otlp-proto-common/http/grpc components rely on, so replacing the generated classes is transparent to them: - add the top-level SpanFlags enum and nested Span.SpanKind / Status.StatusCode enums, with values also lifted onto the parent message class as protobuf does - add a Message base with value equality (by serialized bytes, matching proto3 default omission), repr, SerializePartialToString and FromString - support the metrics encoder's mutable API: as_int/as_double oneof setters on NumberDataPoint and Exemplar (with sfixed64 range checking), and auto-vivified gauge/sum/histogram/exponential_histogram/summary fields on Metric - accept a mapping for the Span.status message field - add the gRPC service Servicer and add_*Servicer_to_server helpers alongside the existing client Stub opentelemetry-proto (668 tests, including the byte-for-byte differential against protobuf) and otlp-proto-common (20 tests) pass.

A protoc plugin that generates the pure-Python, encode-only protobuf-wire message classes from the .proto files, so the hand-written classes in opentelemetry._proto can be generated instead of maintained by hand. Mirrors the architecture of opentelemetry-codegen-json: a plugin entry point (protoc-gen-pyproto / --pyproto_out), a generic CodeWriter, a wire-specific type table, and a descriptor-walking generator. Output goes to opentelemetry/_proto/<signal>/v1/<name>_pb2.py and imports the existing _pyprotobuf runtime. Proven for common/v1 and resource/v1: generated classes serialize byte-identically to both the hand-written classes and real protobuf across 21 constructed messages. Other signals, the mutable API (Metric auto-vivification, as_int/as_double setters), packed fields, maps, and gRPC service stubs remain to be covered.

The pure-Python proto classes only serialize. Tests that decode OTLP bytes compared the public classes to themselves, so they proved nothing. Generate a real protobuf reference set under opentelemetry.proto._test (gated by the new opentelemetry-proto[test] extra) and point the equivalence tests at it, so they now compare the pure-Python implementation against real protobuf. The conftest guard asserts the two sides are distinct implementations. Add scripts/proto_codegen_test.sh to regenerate the reference set.

The http span, log, and metrics test suites decode the exported OTLP bytes to assert on them. The pure-Python classes only serialize, so decode via the protobuf-backed opentelemetry.proto._test classes and compare against the same representation. The metrics batch-splitting unit tests keep the pure-Python classes, since _split_metrics_data produces them; give the pure-Python KeyValue and ResourceMetrics the same dict coercion protobuf constructors do so those tests build messages the same way.

Treat a None scalar as the proto3 default so serialization omits it, matching protobuf: the numeric field encoders omitted only an exact zero, so a None timestamp (from a span with no start time) raised in struct.pack and the grpc client reported 'Exception serializing request!'. Decode the log request with the protobuf-backed reference class before MessageToDict, which needs a real DESCRIPTOR.

Extend the opentelemetry-codegen-pyproto plugin to emit every signal completely and replace the hand-written message modules with its output. The generator now emits IntEnum enum classes with a proper import, drops the builtins-qualified annotations, lifts top-level enum values to module scope, exposes the global___<Name> aliases, and generates the mutable oneof API: auto-vivifying message-oneof properties, scalar-oneof setters that clear siblings and range-check integer widths, with SerializeToString reading the backing fields. The generated classes serialize byte-for-byte identically to real protobuf (the equivalence suite) and are a drop-in for the exporters. Add scripts/proto_codegen_pyproto.sh to regenerate them. This also adds the message types and enums the hand-written set omitted (TracesData, MetricsData, EntityRef, AggregationTemporality, DataPointFlags).

Add the SPDX header to the hand-written pure-Python proto modules and their tests. Add the 5503 changelog fragment. Let codespell accept the 'Nd' in the struct format string in the field encoders.

The pure-Python vs protobuf benchmarks used the pytest-benchmark fixture but lived under tests/, so the normal test env collected them and errored without that plugin. Move them to opentelemetry-proto/benchmarks with a benchmark-requirements.txt and a dedicated tox env, matching the other packages.

… set

…orts

This was referenced Aug 3, 2026

Pure-Python otlp-proto-common encoding (no protobuf) [2/4 RFC] #5504

Draft

Pure-Python otlp-proto-http over stdlib urllib (drop protobuf + requests) [3/4 RFC] #5505

Draft

Pure-Python otlp-proto-grpc (drop protobuf + grpcio) [4/4 RFC] #5506

Draft

ocelotl marked this pull request as ready for review August 7, 2026 23:07

ocelotl requested a review from a team as a code owner August 7, 2026 23:07

otelbot-python Bot added this to Python PR digest Aug 7, 2026

ocelotl force-pushed the pure-python-otlp-1-proto branch 2 times, most recently from c1d60de to 9989274 Compare August 12, 2026 14:24

opentelemetry-pr-dashboard Bot mentioned this pull request Aug 14, 2026

Pull Request Dashboard #5550

Open

ocelotl force-pushed the pure-python-otlp-1-proto branch from 9989274 to 1da6e50 Compare August 20, 2026 13:21

tammy-baylis-swi marked this pull request as draft August 20, 2026 16:07

ocelotl force-pushed the pure-python-otlp-1-proto branch 2 times, most recently from a6885cc to 6cd3ff2 Compare September 4, 2026 19:04

github-actions Bot added the Stale label Oct 3, 2026

ocelotl and others added 9 commits October 20, 2026 16:49

Add license headers, changelog fragment, and codespell exception

0169d20

Add the SPDX header to the hand-written pure-Python proto modules and their tests. Add the 5503 changelog fragment. Let codespell accept the 'Nd' in the struct format string in the field encoders.

ocelotl added 12 commits October 20, 2026 16:49

Apply ruff lint and format to the pure-Python proto changes

edfbffc

Lock the opentelemetry-proto test extra

e3a33dc

Decode json-common compatibility fixtures with the protobuf reference…

286ad0d

… set

Exclude generated protobuf modules from pylint

80d4b6a

Give the pure-Python message base an abstract serializer for pylint

bc6e20d

Ship type stubs for the proto reference set so lint resolves test imp…

2446143

…orts

Ignore the _proto class references in the docs nitpick check

54cbe2f

Decode OTLP test-server requests with the protobuf reference set

9fc9bdf

Let pylint resolve protobuf descriptor_pb2 dynamic members

4cd75c6

Resolve reference-set type stubs in the proto lint env

6635aea

Point the OTLP benchmark at the protobuf reference set

be0af7f

Use descriptive names in the proto tests and benchmark for pylint

abbba5e

ocelotl force-pushed the pure-python-otlp-1-proto branch from 6cd3ff2 to abbba5e Compare October 20, 2026 22:50

github-actions Bot removed the Stale label Oct 21, 2026

Navigation Menu

Repository navigation

Pure-Python opentelemetry-proto (drop the protobuf dependency) [1/4 RFC] - #5503

Pure-Python opentelemetry-proto (drop the protobuf dependency) [1/4 RFC]#5503
ocelotl wants to merge 21 commits into
open-telemetry:mainfrom
ocelotl:pure-python-otlp-1-proto

ocelotl commented Aug 3, 2026 •

edited

Loading

Uh oh!

opentelemetry-pr-dashboard Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

ocelotl commented Sep 7, 2026

Uh oh!

herin049 commented Sep 9, 2026

Uh oh!

xrmx commented Sep 10, 2026 •

edited

Loading

Uh oh!

aabmass commented Sep 10, 2026

Uh oh!

herin049 commented Sep 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ocelotl commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Scope of this PR

Known gaps / discussion points (why this is a draft)

Uh oh!

opentelemetry-pr-dashboard Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

ocelotl commented Sep 7, 2026

Uh oh!

herin049 commented Sep 9, 2026

Uh oh!

xrmx commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabmass commented Sep 10, 2026

Uh oh!

herin049 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ocelotl commented Aug 3, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Aug 14, 2026 •

edited

Loading

xrmx commented Sep 10, 2026 •

edited

Loading

herin049 commented Sep 10, 2026 •

edited

Loading