feat(profiling): [NO MERGE][2/n] support Python 3.15 for profiling by vlad-scherbich · Pull Request #17532 · DataDog/dd-trace-py

vlad-scherbich · 2026-04-15T03:10:08Z

https://datadoghq.atlassian.net/browse/PROF-14200

Description

Adds Python 3.15 support to the Continuous Profiler. CPython 3.15 made two breaking
internal ABI changes in the frame-reading hot path, requiring updates to both the native
C++ stack profiler (Echion) and the Python-side asyncio integration.

Changes

Native stack profiler (C++, stack/):

Updated frame.cc and cpython/tasks.h for two CPython 3.15 ABI breaks: PyFrameState
enum fully renumbered, FRAME_OWNED_BY_CSTACK removed from _frameowner
New compile-time test test_cpython_layout_contracts.cpp — static_asserts for every
CPython enum echion depends on; fires at build time if values drift in a future version
New runtime test test_frame_state_315.cpp — gtest suite for PyGen_yf under 3.15
frame states

Python asyncio integration (ddtrace/profiling/_asyncio.py):

Added hasattr guards for private asyncio APIs (_GatheringFuture, _wait,
_scheduled_tasks, _all_tasks) that have no stability contract
Tiered behavior: on Python ≤ 3.15 (validated versions) a missing internal raises
immediately; on ≥ 3.16 (where the asyncio policy system is being removed) it degrades
gracefully with a warning
AIDEV-TODO comments marking the asyncio policy system deprecation in 3.15
(CPython #127949, removal in 3.16) for the next port

Build and CI:

pyproject.toml, riotfile.py: Python 3.15 added to supported versions and profiling
venv matrix
GitLab CI and GitHub Actions: 3.15 added to profiling_native sanitizer matrix, package
build, testrunner, and version generation workflows
docker/.python-version: 3.15.0a7

Testing

Existing unit tests

All profiling riot test suites PASS

* profiling_native
* profiling::profile
* profiling::profile-memalloc

New ABI tests for the stack profiler

Validated locally against Python 3.15.0a7

# full extension rebuild
$ DD_PROFILING_NATIVE_TESTS=1 pip install -e . --force-reinstall --no-deps

# run ABI tests
$ TEST_DIR=ddtrace/internal/datadog/profiling/test
STACK_DIR=ddtrace/internal/datadog/profiling/stack
for t in test_frame_state_315 test_cpython_layout_contracts test_thread_span_links; do
  echo "=== $t ==="
  DYLD_LIBRARY_PATH=$STACK_DIR ./$TEST_DIR/$t
done
...

=== test_frame_state_315 ===
...
[  PASSED  ] 6 tests.

=== test_cpython_layout_contracts ===
...
[  PASSED  ] 2 tests.

=== test_thread_span_links ===
...
[  PASSED  ] 2 tests.

Risks

Free-threaded Python 3.15 (nogil): unsupported; planned as a follow-up
CPython 3.15 is not yet GA: alpha quality; CI will run in earnest once
pyo3/libdatadog unblocks and a stable 3.15 is available

Additional notes

Depends on chore: bump 'pyo3' to v0.28 libdatadog#1847

vlad-scherbich · 2026-04-17T12:44:27Z

@chatgpt-codex-connector please review

Copilot

Pull request overview

This PR extends dd-trace-py’s Continuous Profiler compatibility to Python 3.15 by updating native/Rust dependencies (notably pyo3) and adding new Echion/asyncio guardrails, along with CI/matrix expansions and new local compatibility tooling.

Changes:

Add Python 3.15 support signals and matrices across packaging/CI/riot, plus release notes and local verification scripts.
Update native components: pyo3 0.28 upgrade, libdatadog rev bump, and Echion frame/task handling adjustments for CPython 3.15 internals.
Introduce new Echion contract/runtime tests and temporarily xfail a couple of profiling tests on 3.15 while issues are investigated.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/profiling/test_uwsgi.py	Marks a known 3.15 uwsgi wall-time sampling issue as xfail.
tests/profiling/test_scheduler.py	Avoids real uploads during a logging-focused scheduler test by patching `ddup.upload`.
tests/profiling/collector/test_asyncio_idle.py	Temporarily xfails asyncio idle frame-capture validation on 3.15.
src/native/data_pipeline/mod.rs	Updates `TraceExporter` usage to use `NativeCapabilities`-parameterized exporter type.
src/native/Cargo.toml	Bumps pyo3/pyo3-ffi/pyo3-build-config to 0.28 and adds libdd-capabilities-impl; updates libdatadog git rev.
src/native/Cargo.lock	Lockfile updates reflecting pyo3/libdatadog dependency changes.
setup.py	Adds a 3.15 guard to set `PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1` as a build safety net.
scripts/verify_profiler_compatibility.py	Adds a profiler compatibility verification tool with baseline compare support.
scripts/run-profiling-tests	Adds a one-command runner to rebuild/check stack extension and run profiling suites for a target Python.
scripts/profiles/compatibility_baselines.json	Adds initial baselines for compatibility verification across 3.9–3.15.
riotfile.py	Adds Python 3.15 to supported matrices and sets pyo3 forward-compat env for builds.
releasenotes/notes/profiling-python315-support-5910a6a4e623716b.yaml	Announces Python 3.15 profiler support.
pyproject.toml	Extends supported Python range to `<3.16` and adds the 3.15 classifier.
ddtrace/profiling/_asyncio.py	Adds guards/warnings for private asyncio internals and future-version graceful degradation.
ddtrace/internal/datadog/profiling/stack/test/test_frame_state_315.cpp	Adds 3.15-specific frame-state tests around `PyGen_yf`.
ddtrace/internal/datadog/profiling/stack/test/test_cpython_layout_contracts.cpp	Adds compile-time/static enum contracts for CPython internals Echion depends on.
ddtrace/internal/datadog/profiling/stack/test/CMakeLists.txt	Registers the new Echion tests in the CMake test target list.
ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc	Switch-based handling for `_frameowner` changes (incl. 3.15 CSTACK removal), aiming to catch new enum values at build time.
ddtrace/internal/datadog/profiling/stack/echion/echion/cpython/tasks.h	Adds a 3.15-specific `PyGen_yf` implementation handling the new locked yield-from state (nogil builds).
ddtrace/internal/datadog/profiling/dd_wrapper/src/sample.cpp	Ensures `Python.h` is included first to avoid `_POSIX_C_SOURCE`/`_XOPEN_SOURCE` redefinition warnings-as-errors on 3.15+.
.gitlab/testrunner.yml	Adds 3.15 to the pyenv global list used in CI.
.gitlab/package.yml	Adds 3.15 to package build/test matrices and tags.
.gitlab/multi-os-tests.yml	Adds 3.15 to multi-OS test coverage.
.gitlab-ci.yml	Adds 3.15 to `profiling_native` matrices (sanitizers/valgrind/etc.).
.github/workflows/generate-supported-versions.yml	Adds Python 3.15 setup to version generation workflow.
.github/workflows/generate-package-versions.yml	Adds Python 3.15 setup to package-version generation workflow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7513bb8616

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Copilot

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

CPython 3.15 changed two bytecode operands in async/generator code paths: - YIELD_VALUE in SEND loops (await): 0 → 1 - RESUME after async gen yield-to-caller: 1 → 5 asyncs.py: new 3.15 branch with updated operands; ASYNC_HEAD and COROUTINE_ASSEMBLY structure identical to 3.14, ASYNC_GEN_ASSEMBLY yield_value updated in all three await loops (presend0/1/2) and resume updated in the yield-to-caller section. generators.py: new 3.15 branch identical to 3.14 — generator yields inside try blocks still use RESUME 1 in 3.15 (only unprotected yields changed to RESUME 5). context.py: extended 3.13 branch to cover 3.15 — no bytecode changes needed for context enter/exit/return assembly. All version guards bumped from >= (3,15) to >= (3,16). Validated on Python 3.15.0a7: coroutine, generator, async generator, and generator throw all pass.

The 3.13 INJECTION_ASSEMBLY (load_const/push_null/call/pop_top) is valid on 3.15 — no call convention changes were made. Bump the NotImplementedError guard from 3.15 to 3.16.

CPython 3.15 made two breaking internal ABI changes in the profiler hot path: 1. PyFrameState enum completely renumbered (FRAME_EXECUTING: 0 to 4, etc.) and FRAME_SUSPENDED_YIELD_FROM_LOCKED added for free-threaded builds. 2. FRAME_OWNED_BY_CSTACK removed from _frameowner enum. Native changes: - frame.cc: exhaustive switch on _frameowner (no default: so -Wswitch fires on new values); FRAME_OWNED_BY_CSTACK compiled out on 3.15+; three-tier is_entry assignment for 3.15+ / 3.12-3.14 / pre-3.12 - cpython/tasks.h: new PyGen_yf branch for 3.15 with renumbered frame states; FRAME_SUSPENDED_YIELD_FROM_LOCKED guarded by ifdef Py_GIL_DISABLED New C++ tests: - test_cpython_layout_contracts.cpp: static_assert for every CPython enum echion depends on -- fires at build time before any test runner - test_frame_state_315.cpp: gtest suite for PyGen_yf under all 3.15 frame states including the free-threaded FRAME_SUSPENDED_YIELD_FROM_LOCKED state Free-threaded Python 3.15 compiles correctly but is not yet validated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…17117) PR #17117 migrated detect_global_locks to riot (pys=select_pys()), which already picks up 3.15 via SUPPORTED_PYTHON_VERSIONS. Re-adding the deleted template was a rebase artifact from before that migration landed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

_GatheringFuture and _wait are asyncio internals with no stability contract. Wrap each with hasattr so removal/rename in a future CPython release degrades gracefully (task linking silently skipped) instead of AttributeError at profiler startup. Also adds AIDEV-TODO comments flagging the asyncio policy system deprecation in 3.15 (CPython #127949, removal in 3.16) for the next CPython port.

…asyncio internals On Python <= 3.15 (validated versions), missing _GatheringFuture or _wait is a real bug — raise RuntimeError immediately so it is never silently swallowed. On Python >= 3.16 (where the asyncio policy system is being removed and these internals may disappear), degrade gracefully with a log.warning so the profiler still starts with reduced task-link coverage.

…emoval Apply same tiered hasattr + RuntimeError/log.warning pattern to _scheduled_tasks and _all_tasks in _call_init_asyncio(). On Python <=3.15 (validated versions) a missing attribute raises immediately; on Python >=3.16 it degrades gracefully with a warning. _eager_tasks uses getattr fallback since it was added in 3.12 and absence is not an error.

Python 3.15's pyconfig.h defines _POSIX_C_SOURCE 202405L. If system headers (pulled in via libdatadog_helpers.hpp → features.h) are included first they define the older 200809L value, causing a -Werror redefinition on Linux aarch64 with Python 3.15 headers.

…3.15 builds Transitive deps (e.g. rpds-py) still ship pyo3 0.27.x which caps at Python 3.14. Setting this flag lets pyo3-build-config use the stable ABI instead of failing the version check. Harmless on 3.9-3.14.

The test was using the default tracer=ddtrace.tracer, causing ddup.upload() to flush any pending spans from prior tests. With no agent running, this logged a writer error that polluted caplog and broke the assertion.

Patch ddup.upload so the test only verifies scheduler logging and does not trigger a real upload attempt (which logs a writer connection error and pollutes caplog on Python 3.15 due to test ordering).

Stack profiler fails to collect wall-time samples in uwsgi worker subprocesses on Python 3.15. Mark xfail while this is investigated.

short_task frame capture not verified on Python 3.15 yet; under investigation.

… outside running loop on py3.15

1. Qualify release note with known xfails: uwsgi wall-time sampling and asyncio short-task frame capture are not yet fully verified on Python 3.15. 2. Fix asyncio import order in verify_profiler_compatibility.py: import ddtrace.profiling._asyncio first so the ModuleWatchdog callback is registered before asyncio enters sys.modules, then pop asyncio and re-import it so the callback is guaranteed to fire during the test. 3. Align run-profiling-tests usage comment with actual PYTHON_VERSION default ("3.15", resolved by pyenv to the installed patch version e.g. 3.15.0a7). # Conflicts: # scripts/verify_profiler_compatibility.py # Conflicts: # scripts/run-profiling-tests

….14+ PyFrame_GetBack was deprecated in Python 3.11 and removed in 3.14 (PEP 667). It is also absent from pyo3-ffi stable/limited-API bindings activated for Python 3.15 via PYO3_USE_ABI3_FORWARD_COMPATIBILITY. Gate behind not(any(Py_3_14, Py_LIMITED_API)); on 3.14+ advance_frame returns null_mut() stopping traversal after the top frame. Crash reports on 3.14+ show only the top frame until PyUnstable_Frame_GetBack lands in pyo3-ffi bindings.

`(3, 15)` was added to `SUPPORTED_PYTHON_VERSIONS` as part of the 3.15 migration, but the `select_pys` doctests still listed the pre-3.15 output and made `hatch run lint:riot` fail.

vlad-scherbich · 2026-04-20T19:45:50Z

Closing in favor of #17624, which better isolates profiling-only changes.

vlad-scherbich changed the base branch from main to vlad/ddtracepy-315-tracing-wrapping April 15, 2026 03:10

vlad-scherbich mentioned this pull request Apr 15, 2026

feat(tracing): [1/n] support Python 3.15 coroutine and generator wrapping #17446 #17531

Open

vlad-scherbich force-pushed the vlad/ddtracepy-upgrade-py-315-feature branch from d3934e3 to 6e57d4b Compare April 15, 2026 21:15

vlad-scherbich changed the base branch from vlad/ddtracepy-315-tracing-wrapping to main April 15, 2026 21:20

vlad-scherbich changed the base branch from main to vlad/ddtracepy-315-tracing-wrapping April 15, 2026 21:30

vlad-scherbich force-pushed the vlad/ddtracepy-315-tracing-wrapping branch from aa07e99 to 06e9017 Compare April 15, 2026 21:30

vlad-scherbich force-pushed the vlad/ddtracepy-upgrade-py-315-feature branch 2 times, most recently from f8bf205 to 5419900 Compare April 15, 2026 21:46

vlad-scherbich force-pushed the vlad/ddtracepy-315-tracing-wrapping branch 2 times, most recently from 4526b3b to 4d5a9de Compare April 17, 2026 01:06

vlad-scherbich force-pushed the vlad/ddtracepy-upgrade-py-315-feature branch 3 times, most recently from 4f37ced to 7513bb8 Compare April 17, 2026 12:40

vlad-scherbich requested a review from Copilot April 17, 2026 12:46

Copilot started reviewing on behalf of vlad-scherbich April 17, 2026 12:46 View session

This was referenced Apr 17, 2026

chore(profiling): [2.1/n] upgrade py-315 riot requirements #17596

Draft

docs(profiling): [3/n] add CPython version upgrade runbook and 3.15 analysis #17295

Draft

Copilot AI reviewed Apr 17, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Apr 17, 2026

View reviewed changes

Comment thread ddtrace/internal/datadog/profiling/stack/test/CMakeLists.txt Outdated

Comment thread ddtrace/internal/datadog/profiling/stack/test/test_cpython_layout_contracts.cpp

vlad-scherbich added the Profiling Continous Profling label Apr 17, 2026

vlad-scherbich force-pushed the vlad/ddtracepy-upgrade-py-315-feature branch from 43e778a to 2eae691 Compare April 17, 2026 20:43

vlad-scherbich requested a review from Copilot April 17, 2026 20:51

Copilot started reviewing on behalf of vlad-scherbich April 17, 2026 20:52 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread scripts/verify_profiler_compatibility.py Outdated

Comment thread scripts/verify_profiler_compatibility.py Outdated

Comment thread scripts/run-profiling-tests Outdated

Comment thread ddtrace/internal/datadog/profiling/stack/src/echion/frame.cc

vlad-scherbich force-pushed the vlad/ddtracepy-upgrade-py-315-feature branch 2 times, most recently from 444f513 to 4fe3e69 Compare April 18, 2026 20:55

vlad-scherbich added 3 commits April 19, 2026 21:40

chore: add release note for Python 3.15 wrapping support

931e4fa

fix(tracing): extend bytecode injection support to Python 3.15

373c441

The 3.13 INJECTION_ASSEMBLY (load_const/push_null/call/pop_top) is valid on 3.15 — no call convention changes were made. Bump the NotImplementedError guard from 3.15 to 3.16.

vlad-scherbich and others added 23 commits April 19, 2026 21:40

updated PY version gates

8f689cf

fix(profiling): isolate test_before_flush_failure from global tracer

87076c2

The test was using the default tracer=ddtrace.tracer, causing ddup.upload() to flush any pending spans from prior tests. With no agent running, this logged a writer error that polluted caplog and broke the assertion.

fix(profiling): isolate test_before_flush_failure from ddup.upload

2d0372f

Patch ddup.upload so the test only verifies scheduler logging and does not trigger a real upload attempt (which logs a writer connection error and pollutes caplog on Python 3.15 due to test ordering).

test(profiling): xfail uwsgi lazy-apps wall-time test on Python 3.15

a6cc3b1

Stack profiler fails to collect wall-time samples in uwsgi worker subprocesses on Python 3.15. Mark xfail while this is investigated.

test(profiling): xfail asyncio run frames test on Python 3.15

0a6b15c

short_task frame capture not verified on Python 3.15 yet; under investigation.

fix(profiling): remove usages of INVALID_FRAME from frame.cc

1277957

fix(profiling): guard asyncio task-link wrappers against RuntimeError…

60b09f4

… outside running loop on py3.15

address Copilot/Codex review comments on C++ test build

feca6ca

remove xfail from a uwsgi test

8a345b5

type - annotate

ef6259c

verified: asyncio short-task frame capture works

625d5fc

updated release note

c107f36

fix v3.15 tests

6502ca7

fix(riot): update select_pys doctests for Python 3.15

3a81fa8

`(3, 15)` was added to `SUPPORTED_PYTHON_VERSIONS` as part of the 3.15 migration, but the `select_pys` doctests still listed the pre-3.15 output and made `hatch run lint:riot` fail.

vlad-scherbich force-pushed the vlad/ddtracepy-315-tracing-wrapping branch from d379be4 to 8f689cf Compare April 20, 2026 01:41

vlad-scherbich force-pushed the vlad/ddtracepy-upgrade-py-315-feature branch from df58d18 to 3a81fa8 Compare April 20, 2026 01:41

vlad-scherbich force-pushed the vlad/ddtracepy-315-tracing-wrapping branch from 8f689cf to 4cada08 Compare April 20, 2026 19:37

vlad-scherbich closed this Apr 20, 2026

vlad-scherbich changed the title ~~feat(profiling): [2/n] support Python 3.15 for profiling~~ feat(profiling): [NO MERGE][2/n] support Python 3.15 for profiling Apr 20, 2026

vlad-scherbich mentioned this pull request Apr 20, 2026

chore(profiling): [2/n] support Python 3.15 for profiling (no free-threading) #17624

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(profiling): [NO MERGE][2/n] support Python 3.15 for profiling#17532

feat(profiling): [NO MERGE][2/n] support Python 3.15 for profiling#17532
vlad-scherbich wants to merge 32 commits intovlad/ddtracepy-315-tracing-wrappingfrom
vlad/ddtracepy-upgrade-py-315-feature

vlad-scherbich commented Apr 15, 2026 •

edited

Loading

Uh oh!

vlad-scherbich commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vlad-scherbich commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vlad-scherbich commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Testing

Existing unit tests

New ABI tests for the stack profiler

Risks

Additional notes

Uh oh!

vlad-scherbich commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vlad-scherbich commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vlad-scherbich commented Apr 15, 2026 •

edited

Loading