[BUG] func_otlp_grpc segfaults during a TLS case after ctest reports everything passing, failing the job

[thc1006]

Describe your environment

GitHub hosted Linux runners, main at 1c2b007a. Seen on two different maintainer mode jobs:

• CMake gcc 14 (maintainer mode, async)
• CMake clang 18 (maintainer mode, yaml config)

Steps to reproduce

I do not have a local reproduction. It shows up on CI, and both sightings below are on pull requests that do not touch the OTLP gRPC exporter.

What is the expected behavior?

ci/do_ci.sh reports the job's result from the tests. When ctest reports every test passing, the job passes.

What is the actual behavior?

The job fails with exit code 139 after ctest has already reported everything green. The crash is in the functional gRPC binary that runs after ctest, on a TLS case:

100% tests passed, 0 tests failed out of 1593
...
Running test cert-unreadable on secure https://localhost:4317 with server https
I0000 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A
./run_test_mode.sh: line 67: 34560 Segmentation fault (core dumped) "${TEST_BIN_DIR}/${TEST_EXECUTABLE}" --debug --m
##[error]Process completed with exit code 139

Two sightings, two days apart, two compilers:

PR	job	run
#4453	CMake gcc 14 (maintainer mode, async)	97488993921
#4337	CMake clang 18 (maintainer mode, yaml config)	97837662928

Additional context

Neither pull request can be the cause, and that is checkable rather than argued. functional/otlp/CMakeLists.txt links the crashing binary against two targets:

target_link_libraries(func_otlp_grpc ${CMAKE_THREAD_LIBS_INIT}
                      opentelemetry_trace opentelemetry_exporter_otlp_grpc)

#4337 changes only exporters/elasticsearch/, and grep -rn -i elasticsearch functional/ returns nothing. #4453 changes only exporters/otlp/src/otlp_http_client.cc and its test. Neither is in that binary's link closure.

For #4453 there is a second measurement: the head that passed this job and the head that failed it differ only in comments, 0 non-comment changed lines. One passed, one failed.

It is not constant. CMake clang 18 (maintainer mode, yaml config) was green on 11 of the last 12 main commits, so this reads as a race in the functional test or in the gRPC TLS teardown rather than something that reproduces on demand.

What it costs today is that a red job on a pull request has to be opened and read before it can be dismissed, and the log looks alarming because ctest is green immediately above it.

[marcalff]

I have seen it also in CI multiple times.

Example of log:

###############################################################
Starting otelcol --config otel-config-https-mtls.yaml
###############################################################

8140d8a97970de8bb9996bd75b89aac5b2b133aa9b732aa20a598b418a1afdeb
=====================================================================
Running test basic on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1788137886.112726   34358 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
I0000 00:00:1788137886.124213   34357 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
 - [E] [OTLP TRACE GRPC Exporter] Export() failed with status_code: "DEADLINE_EXCEEDED" error_message: "Deadline Exceeded"
 - [D] [OTLP GRPC Client] DEBUG: OtlpGrpcClient start to shutdown
TEST basic-insecure-https: PASSED
=====================================================================
Running test cert-invalid on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1788137887.153710   34364 ssl_transport_security.cc:1121] Could not load any root certificate.
E0000 00:00:1788137887.153746   34364 ssl_transport_security.cc:2995] Cannot load server root certificates.
E0000 00:00:1788137887.153763   34364 ssl_credentials.cc:213] Handshaker factory creation failed with TSI_INVALID_ARGUMENT
E0000 00:00:1788137887.154954   34367 endpoint_transport_client_channel_factory.cc:26] Failed to create channel args during subchannel creation: INTERNAL: Failed to create subchannel for secure name 'localhost:4317'; Got args: {grpc.client_channel_factory=0x5589f2cea3b0, grpc.default_authority=localhost:4317, grpc.experimental.event_engine=0x7f43d8004580, grpc.internal.channel_credentials=0x558a0d17bdc0, grpc.internal.client_channel_call_destination=0x5589f2ce9888, grpc.internal.stats_plugin_group=0x7f43d8003240, grpc.internal.subchannel_pool=0x558a0d1e4c60, grpc.internal.use_v3_stack=0, grpc.primary_user_agent=OTel-OTLP-Exporter-Cpp/1.29.0-dev grpc-c++/1.82.1, grpc.resource_quota=0x558a0d1d2060, grpc.server_uri=dns:///localhost:4317}
E0000 00:00:1788137887.154981   34367 endpoint_transport_client_channel_factory.cc:26] Failed to create channel args during subchannel creation: INTERNAL: Failed to create subchannel for secure name 'localhost:4317'; Got args: {grpc.client_channel_factory=0x5589f2cea3b0, grpc.default_authority=localhost:4317, grpc.experimental.event_engine=0x7f43d8002600, grpc.internal.channel_credentials=0x558a0d17bdc0, grpc.internal.client_channel_call_destination=0x5589f2ce9888, grpc.internal.stats_plugin_group=0x7f43d8003240, grpc.internal.subchannel_pool=0x558a0d1e4c60, grpc.internal.use_v3_stack=0, grpc.primary_user_agent=OTel-OTLP-Exporter-Cpp/1.29.0-dev grpc-c++/1.82.1, grpc.resource_quota=0x558a0d1d2060, grpc.server_uri=dns:///localhost:4317}
 - [E] [OTLP TRACE GRPC Exporter] Export() failed with status_code: "DEADLINE_EXCEEDED" error_message: "Deadline Exceeded"
 - [D] [OTLP GRPC Client] DEBUG: OtlpGrpcClient start to shutdown
TEST cert-invalid-insecure-https: PASSED
=====================================================================
Running test cert-unreadable on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1788137887.293349   34379 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
I0000 00:00:1788137887.302889   34378 ssl_transport_security.cc:2160] Handshake failed with error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed: unable to get local issuer certificate
 - [E] [OTLP TRACE GRPC Exporter] Export() failed with status_code: "DEADLINE_EXCEEDED" error_message: "Deadline Exceeded"
 - [D] [OTLP GRPC Client] DEBUG: OtlpGrpcClient start to shutdown
TEST cert-unreadable-insecure-https: PASSED
=====================================================================
Running test client-cert-not-found on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1788137888.477427   34389 ssl_transport_security_utils.cc:115] Corruption detected.
E0000 00:00:1788137888.477482   34389 ssl_transport_security_utils.cc:72] error:0A00045C:SSL routines::tlsv13 alert certificate required
E0000 00:00:1788137888.477488   34389 secure_endpoint.cc:269] Decryption error: TSI_DATA_CORRUPTED
E0000 00:00:1788137888.487649   34388 ssl_transport_security_utils.cc:115] Corruption detected.
E0000 00:00:1788137888.487675   34388 ssl_transport_security_utils.cc:72] error:0A00045C:SSL routines::tlsv13 alert certificate required
 - [D] [OTLP GRPC Client] DEBUG: OtlpGrpcClient start to shutdown
TEST client-key-not-found-insecure-https: PASSED
=====================================================================
Running test client-key-invalid on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1788137893.701509   34437 ssl_transport_security.cc:1173] Invalid private key.
E0000 00:00:1788137893.701572   34437 ssl_credentials.cc:213] Handshaker factory creation failed with TSI_INVALID_ARGUMENT
E0000 00:00:1788137893.702627   34442 endpoint_transport_client_channel_factory.cc:26] Failed to create channel args during subchannel creation: INTERNAL: Failed to create subchannel for secure name 'localhost:4317'; Got args: {grpc.client_channel_factory=0x5595b35d63b0, grpc.default_authority=localhost:4317, grpc.experimental.event_engine=0x7fe340003220, grpc.internal.channel_credentials=0x5595c0278360, grpc.internal.client_channel_call_destination=0x5595b35d5888, grpc.internal.stats_plugin_group=0x7fe3400045f0, grpc.internal.subchannel_pool=0x5595c02e8810, grpc.internal.use_v3_stack=0, grpc.primary_user_agent=OTel-OTLP-Exporter-Cpp/1.29.0-dev grpc-c++/1.82.1, grpc.resource_quota=0x5595c02d8b00, grpc.server_uri=dns:///localhost:4317}
E0000 00:00:1788137893.702654   34442 endpoint_transport_client_channel_factory.cc:26] Failed to create channel args during subchannel creation: INTERNAL: Failed to create subchannel for secure name 'localhost:4317'; Got args: {grpc.client_channel_factory=0x5595b35d63b0, grpc.default_authority=localhost:4317, grpc.experimental.event_engine=0x7fe3400025e0, grpc.internal.channel_credentials=0x5595c0278360, grpc.internal.client_channel_call_destination=0x5595b35d5888, grpc.internal.stats_plugin_group=0x7fe3400045f0, grpc.internal.subchannel_pool=0x5595c02e8810, grpc.internal.use_v3_stack=0, grpc.primary_user_agent=OTel-OTLP-Exporter-Cpp/1.29.0-dev grpc-c++/1.82.1, grpc.resource_quota=0x5595c02d8b00, grpc.server_uri=dns:///localhost:4317}
 - [E] [OTLP TRACE GRPC Exporter] Export() failed with status_code: "DEADLINE_EXCEEDED" error_message: "Deadline Exceeded"
 - [D] [OTLP GRPC Client] DEBUG: OtlpGrpcClient start to shutdown
TEST client-key-invalid-insecure-https: PASSED
=====================================================================
Running test client-key-unreadable on insecure http://localhost:4317 with server https
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1788137893.824038   34451 ssl_transport_security_utils.cc:115] Corruption detected.
E0000 00:00:1788137893.824081   34451 ssl_transport_security_utils.cc:72] error:0A00045C:SSL routines::tlsv13 alert certificate required
E0000 00:00:1788137893.824086   34451 secure_endpoint.cc:269] Decryption error: TSI_DATA_CORRUPTED
E0000 00:00:1788137893.836609   34451 ssl_transport_security_utils.cc:115] Corruption detected.
E0000 00:00:1788137893.836636   34451 ssl_transport_security_utils.cc:72] error:0A00045C:SSL routines::tlsv13 alert certificate required
E0000 00:00:1788137893.836640   34451 secure_endpoint.cc:269] Decryption error: TSI_DATA_CORRUPTED
double free or corruption (fasttop)
./run_test_mode.sh: line 46: 34447 Aborted                 (core dumped) "${TEST_BIN_DIR}/${TEST_EXECUTABLE}" --debug --mode ${SERVER_MODE} --cert-dir ${CERT_DIR} --endpoint ${TEST_ENDPOINT} ${T}
Error: Process completed with exit code 134.

The last part:

double free or corruption (fasttop)
./run_test_mode.sh: line 46: 34447 Aborted                 (core dumped) "${TEST_BIN_DIR}/${TEST_EXECUTABLE}" --debug --mode ${SERVER_MODE} --cert-dir ${CERT_DIR} --endpoint ${TEST_ENDPOINT} ${T}
Error: Process completed with exit code 134.

is the test crash.

[thc1006]

TL;DR: this looks like the same failure as #4273, and the upstream fix has already shipped. gRPC stopped registering OpenSSL's atexit handler in 1.83.0, and we pin 1.82.1 in third_party_release, so a gRPC bump should end it. I owe a correction there, because I said the opposite in July.

Thanks for the log. It is more useful than it might look, because it aborts with double free or corruption (fasttop) and exit 134, while the one in the description segfaults with exit 139. I think they are one bug rather than two. Both land after 100% tests passed, in the same binary, on a TLS case whose handshake fails, and the case that dies moves around: client-key-unreadable in yours, cert-unreadable in mine, cert-not-found in @dbarker's #4273. One heap corruption, reported by whichever check happens to notice it first, fits that better than three separate bugs.

The corruption is at process exit. OpenSSL's OPENSSL_cleanup atexit handler frees the library's global state while gRPC EventEngine threads are still using it. That part can be seen without waiting for a crash: an atexit handler registered before gRPC first touches OpenSSL runs after OpenSSL's own, since handlers run in reverse order of registration, so it reports the thread population at a point no earlier than OPENSSL_cleanup. On a small gRPC client with a failing handshake it prints

ATEXIT: live threads = 37 : ... event_engine (x33) timer_manager grpc_global_tim resolver-execut ...

so nothing has drained by then.

gRPC commit 784d4464b (2026-06-17, "Initialize openssl with no atexit handler", grpc/grpc#42608) changed init_openssl() in src/core/tsi/ssl_transport_security.cc to

OPENSSL_init_ssl(OPENSSL_INIT_NO_ATEXIT, nullptr);

so the handler is never registered and there is nothing left to race. The pull request shows as closed rather than merged, which is normal for changes that land through gRPC's internal import. Grepping that file at each tag:

gRPC tag	OPENSSL_INIT_NO_ATEXIT
v1.82.1, which we pin	absent
v1.82.2	absent
v1.83.0	present
v1.83.1	present

v1.82.2 came out on 2026-08-27 with six backports and none of them touch OpenSSL, so staying on the 1.82 line does not pick it up. third_party_release line 20 reads grpc=v1.82.1, and both jobs named in the description install from it through ci/install_thirdparty.sh --tags-file third_party_release.

The correction: in my July comment on #4273 I wrote that this would stop "once it ships in a gRPC release we can bump to". It had already shipped. 1.83.0 was tagged on 2026-07-22, four days before I wrote that. I had read the release notes, which do not mention it, instead of the code. Sorry for sending anyone down the wrong path.

On what to do next I would rather defer than push. Going from 1.82.1 to 1.83.x is a minor version step, and the two previous gRPC moves here (#4167 and #4255) were paired with protobuf, so the version choice looks like a maintainer call rather than something to slip in. If it is useful I am happy to open the third_party_release bump together with the matching submodule and MODULE.bazel updates and let CI report on 1.83.x, or to leave it with whoever normally drives those upgrades.

If a bump is not wanted yet, the smallest local workaround is to end func_grpc_main.cc with std::_Exit(rc) after flushing, so the test binary runs no atexit handler at all. I have not been able to put that in front of a run that actually fails, so I would rather fix the dependency than add it on a hunch.

Thanks @dbarker for the original report and the logs in #4273, and @marcalff for confirming here. Your log is what let me line the 134 and 139 cases up.