test(co): test for all snapshot IDs

[P403n1x87]

Description

We improve the testing by checking that all the snapshots referenced in span tags have been collected.

[github-actions]

CODEOWNERS have been resolved as:

tests/debugging/origin/test_span.py                                     @DataDog/debugger-python

[github-actions]

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 253 ± 3 ms.

The average import time from base is: 259 ± 3 ms.

The import time difference between this PR and base is: -5.6 ± 0.1 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 6.625 ms (2.62%)

ddtrace.bootstrap.sitecustomize 3.941 ms (1.56%)

ddtrace.bootstrap.preload 3.941 ms (1.56%)

ddtrace.internal.settings.dynamic_instrumentation 0.674 ms (0.27%)

ddtrace.internal.remoteconfig.worker 0.668 ms (0.26%)

ddtrace.internal.remoteconfig.client 0.597 ms (0.24%)

ddtrace 2.683 ms (1.06%)

ddtrace._logger 0.673 ms (0.27%)

ddtrace.internal.telemetry 0.673 ms (0.27%)

ddtrace.internal.telemetry.writer 0.673 ms (0.27%)

ddtrace.internal.utils.version 0.673 ms (0.27%)

ddtrace.trace 0.651 ms (0.26%)

ddtrace.internal._unpatched 0.028 ms (0.01%)

json 0.028 ms (0.01%)

json.decoder 0.028 ms (0.01%)

re 0.028 ms (0.01%)

enum 0.028 ms (0.01%)

types 0.028 ms (0.01%)

[pr-commenter]

Performance SLOs

Comparing candidate test/co-session-all-snapshots (5b73d86) with baseline main (2cc3c8b)

📈 Performance Regressions (1 suite)

📈 iastaspectsospath - 24/24

✅ ospathbasename_aspect

Time: ✅ 5.070µs (SLO: <10.000µs 📉 -49.3%) vs baseline: 📈 +17.9%

Memory: ✅ 40.010MB (SLO: <41.000MB -2.4%) vs baseline: +5.3%

---

✅ ospathbasename_noaspect

Time: ✅ 1.096µs (SLO: <10.000µs 📉 -89.0%) vs baseline: +0.9%

Memory: ✅ 39.911MB (SLO: <41.000MB -2.7%) vs baseline: +4.8%

---

✅ ospathjoin_aspect

Time: ✅ 6.203µs (SLO: <10.000µs 📉 -38.0%) vs baseline: ~same

Memory: ✅ 39.793MB (SLO: <41.000MB -2.9%) vs baseline: +4.7%

---

✅ ospathjoin_noaspect

Time: ✅ 2.294µs (SLO: <10.000µs 📉 -77.1%) vs baseline: ~same

Memory: ✅ 39.833MB (SLO: <41.000MB -2.8%) vs baseline: +4.7%

---

✅ ospathnormcase_aspect

Time: ✅ 3.567µs (SLO: <10.000µs 📉 -64.3%) vs baseline: +1.2%

Memory: ✅ 39.951MB (SLO: <41.000MB -2.6%) vs baseline: +5.2%

---

✅ ospathnormcase_noaspect

Time: ✅ 0.578µs (SLO: <10.000µs 📉 -94.2%) vs baseline: +0.6%

Memory: ✅ 39.892MB (SLO: <41.000MB -2.7%) vs baseline: +5.0%

---

✅ ospathsplit_aspect

Time: ✅ 4.905µs (SLO: <10.000µs 📉 -51.0%) vs baseline: +0.9%

Memory: ✅ 39.990MB (SLO: <41.000MB -2.5%) vs baseline: +5.3%

---

✅ ospathsplit_noaspect

Time: ✅ 1.610µs (SLO: <10.000µs 📉 -83.9%) vs baseline: +1.0%

Memory: ✅ 39.970MB (SLO: <41.000MB -2.5%) vs baseline: +5.1%

---

✅ ospathsplitdrive_aspect

Time: ✅ 3.757µs (SLO: <10.000µs 📉 -62.4%) vs baseline: +1.1%

Memory: ✅ 39.872MB (SLO: <41.000MB -2.8%) vs baseline: +4.8%

---

✅ ospathsplitdrive_noaspect

Time: ✅ 0.706µs (SLO: <10.000µs 📉 -92.9%) vs baseline: +0.3%

Memory: ✅ 39.872MB (SLO: <41.000MB -2.8%) vs baseline: +4.9%

---

✅ ospathsplitext_aspect

Time: ✅ 5.091µs (SLO: <10.000µs 📉 -49.1%) vs baseline: 📈 +10.0%

Memory: ✅ 39.813MB (SLO: <41.000MB -2.9%) vs baseline: +4.8%

---

✅ ospathsplitext_noaspect

Time: ✅ 1.375µs (SLO: <10.000µs 📉 -86.2%) vs baseline: +0.2%

Memory: ✅ 39.990MB (SLO: <41.000MB -2.5%) vs baseline: +4.9%

⚠️ Unstable Tests (1 suite)

⚠️ coreapiscenario - 10/10 (1 unstable)

⚠️ context_with_data_listeners

Time: ⚠️ 13.308µs (SLO: <20.000µs 📉 -33.5%) vs baseline: -0.1%

Memory: ✅ 34.564MB (SLO: <35.500MB -2.6%) vs baseline: +4.5%

---

✅ context_with_data_no_listeners

Time: ✅ 3.271µs (SLO: <10.000µs 📉 -67.3%) vs baseline: +0.4%

Memory: ✅ 34.583MB (SLO: <35.500MB -2.6%) vs baseline: +5.0%

---

✅ get_item_exists

Time: ✅ 0.584µs (SLO: <10.000µs 📉 -94.2%) vs baseline: +1.0%

Memory: ✅ 34.564MB (SLO: <35.500MB -2.6%) vs baseline: +4.8%

---

✅ get_item_missing

Time: ✅ 0.641µs (SLO: <10.000µs 📉 -93.6%) vs baseline: +1.2%

Memory: ✅ 34.583MB (SLO: <35.500MB -2.6%) vs baseline: +5.1%

---

✅ set_item

Time: ✅ 24.341µs (SLO: <30.000µs 📉 -18.9%) vs baseline: +0.9%

Memory: ✅ 34.544MB (SLO: <35.500MB -2.7%) vs baseline: +4.9%

✅ All Tests Passing (8 suites)

✅ errortrackingflasksqli - 6/6

✅ errortracking-enabled-all

Time: ✅ 2.067ms (SLO: <2.300ms 📉 -10.1%) vs baseline: -0.1%

Memory: ✅ 54.578MB (SLO: <56.500MB -3.4%) vs baseline: +5.2%

---

✅ errortracking-enabled-user

Time: ✅ 2.075ms (SLO: <2.250ms -7.8%) vs baseline: +0.3%

Memory: ✅ 54.441MB (SLO: <56.500MB -3.6%) vs baseline: +4.8%

---

✅ tracer-enabled

Time: ✅ 2.084ms (SLO: <2.300ms -9.4%) vs baseline: +0.6%

Memory: ✅ 54.539MB (SLO: <56.500MB -3.5%) vs baseline: +4.9%

---

✅ flasksimple - 18/18

✅ appsec-get

Time: ✅ 4.598ms (SLO: <4.750ms -3.2%) vs baseline: +0.3%

Memory: ✅ 64.053MB (SLO: <66.500MB -3.7%) vs baseline: +4.7%

---

✅ appsec-post

Time: ✅ 6.611ms (SLO: <6.750ms -2.1%) vs baseline: -0.1%

Memory: ✅ 64.389MB (SLO: <66.500MB -3.2%) vs baseline: +5.1%

---

✅ appsec-telemetry

Time: ✅ 4.598ms (SLO: <4.750ms -3.2%) vs baseline: +0.4%

Memory: ✅ 64.108MB (SLO: <66.500MB -3.6%) vs baseline: +5.0%

---

✅ debugger

Time: ✅ 1.858ms (SLO: <2.000ms -7.1%) vs baseline: +0.2%

Memory: ✅ 47.939MB (SLO: <49.500MB -3.2%) vs baseline: +5.0%

---

✅ iast-get

Time: ✅ 1.856ms (SLO: <2.000ms -7.2%) vs baseline: -0.3%

Memory: ✅ 44.417MB (SLO: <49.000MB -9.4%) vs baseline: +4.7%

---

✅ profiler

Time: ✅ 1.928ms (SLO: <2.100ms -8.2%) vs baseline: ~same

Memory: ✅ 48.449MB (SLO: <50.000MB -3.1%) vs baseline: +4.8%

---

✅ resource-renaming

Time: ✅ 3.375ms (SLO: <3.650ms -7.5%) vs baseline: +0.3%

Memory: ✅ 54.638MB (SLO: <56.000MB -2.4%) vs baseline: +4.8%

---

✅ tracer

Time: ✅ 3.365ms (SLO: <3.650ms -7.8%) vs baseline: +0.3%

Memory: ✅ 54.603MB (SLO: <56.500MB -3.4%) vs baseline: +4.9%

---

✅ tracer-native

Time: ✅ 3.360ms (SLO: <3.650ms -8.0%) vs baseline: -0.2%

Memory: ✅ 54.541MB (SLO: <60.000MB -9.1%) vs baseline: +4.6%

---

✅ flasksqli - 6/6

✅ appsec-enabled

Time: ✅ 3.954ms (SLO: <4.200ms -5.9%) vs baseline: -0.5%

Memory: ✅ 64.291MB (SLO: <66.000MB -2.6%) vs baseline: +4.8%

---

✅ iast-enabled

Time: ✅ 2.437ms (SLO: <2.800ms 📉 -13.0%) vs baseline: -0.4%

Memory: ✅ 60.359MB (SLO: <62.500MB -3.4%) vs baseline: +4.5%

---

✅ tracer-enabled

Time: ✅ 2.056ms (SLO: <2.250ms -8.6%) vs baseline: -0.3%

Memory: ✅ 54.460MB (SLO: <56.500MB -3.6%) vs baseline: +4.3%

---

✅ httppropagationinject - 16/16

✅ ids_only

Time: ✅ 21.032µs (SLO: <30.000µs 📉 -29.9%) vs baseline: +1.1%

Memory: ✅ 34.524MB (SLO: <35.500MB -2.7%) vs baseline: +4.7%

---

✅ with_all

Time: ✅ 27.919µs (SLO: <40.000µs 📉 -30.2%) vs baseline: +0.7%

Memory: ✅ 34.485MB (SLO: <35.500MB -2.9%) vs baseline: +4.7%

---

✅ with_dd_origin

Time: ✅ 24.779µs (SLO: <30.000µs 📉 -17.4%) vs baseline: ~same

Memory: ✅ 34.623MB (SLO: <35.500MB -2.5%) vs baseline: +4.9%

---

✅ with_priority_and_origin

Time: ✅ 24.974µs (SLO: <40.000µs 📉 -37.6%) vs baseline: +3.7%

Memory: ✅ 34.583MB (SLO: <35.500MB -2.6%) vs baseline: +5.1%

---

✅ with_sampling_priority

Time: ✅ 21.958µs (SLO: <30.000µs 📉 -26.8%) vs baseline: +5.9%

Memory: ✅ 34.623MB (SLO: <35.500MB -2.5%) vs baseline: +4.9%

---

✅ with_tags

Time: ✅ 26.076µs (SLO: <40.000µs 📉 -34.8%) vs baseline: +1.3%

Memory: ✅ 34.682MB (SLO: <35.500MB -2.3%) vs baseline: +5.2%

---

✅ with_tags_invalid

Time: ✅ 27.271µs (SLO: <40.000µs 📉 -31.8%) vs baseline: +0.2%

Memory: ✅ 34.642MB (SLO: <35.500MB -2.4%) vs baseline: +5.0%

---

✅ with_tags_max_size

Time: ✅ 26.582µs (SLO: <40.000µs 📉 -33.5%) vs baseline: +1.0%

Memory: ✅ 34.642MB (SLO: <35.500MB -2.4%) vs baseline: +5.3%

---

✅ iastaspectssplit - 12/12

✅ rsplit_aspect

Time: ✅ 1.398µs (SLO: <10.000µs 📉 -86.0%) vs baseline: +2.6%

Memory: ✅ 39.892MB (SLO: <41.000MB -2.7%) vs baseline: +4.8%

---

✅ rsplit_noaspect

Time: ✅ 0.583µs (SLO: <10.000µs 📉 -94.2%) vs baseline: +0.8%

Memory: ✅ 39.813MB (SLO: <41.000MB -2.9%) vs baseline: +4.5%

---

✅ split_aspect

Time: ✅ 1.369µs (SLO: <10.000µs 📉 -86.3%) vs baseline: +0.6%

Memory: ✅ 39.911MB (SLO: <41.000MB -2.7%) vs baseline: +5.0%

---

✅ split_noaspect

Time: ✅ 0.572µs (SLO: <10.000µs 📉 -94.3%) vs baseline: +0.2%

Memory: ✅ 39.813MB (SLO: <41.000MB -2.9%) vs baseline: +4.8%

---

✅ splitlines_aspect

Time: ✅ 1.376µs (SLO: <10.000µs 📉 -86.2%) vs baseline: +0.8%

Memory: ✅ 39.774MB (SLO: <41.000MB -3.0%) vs baseline: +4.9%

---

✅ splitlines_noaspect

Time: ✅ 0.583µs (SLO: <10.000µs 📉 -94.2%) vs baseline: -0.1%

Memory: ✅ 39.813MB (SLO: <41.000MB -2.9%) vs baseline: +4.8%

---

✅ packagesupdateimporteddependencies - 24/24

✅ import_many

Time: ✅ 155.159µs (SLO: <170.000µs -8.7%) vs baseline: -0.2%

Memory: ✅ 39.106MB (SLO: <41.000MB -4.6%) vs baseline: +4.3%

---

✅ import_many_cached

Time: ✅ 121.377µs (SLO: <130.000µs -6.6%) vs baseline: +0.4%

Memory: ✅ 39.151MB (SLO: <41.000MB -4.5%) vs baseline: +4.9%

---

✅ import_many_stdlib

Time: ✅ 1.630ms (SLO: <1.750ms -6.8%) vs baseline: -0.2%

Memory: ✅ 39.295MB (SLO: <41.000MB -4.2%) vs baseline: +4.8%

---

✅ import_many_stdlib_cached

Time: ✅ 0.986ms (SLO: <1.100ms 📉 -10.4%) vs baseline: +0.3%

Memory: ✅ 39.334MB (SLO: <41.000MB -4.1%) vs baseline: +4.9%

---

✅ import_many_unknown

Time: ✅ 836.228µs (SLO: <890.000µs -6.0%) vs baseline: +0.3%

Memory: ✅ 39.434MB (SLO: <41.000MB -3.8%) vs baseline: +5.2%

---

✅ import_many_unknown_cached

Time: ✅ 790.639µs (SLO: <870.000µs -9.1%) vs baseline: +0.2%

Memory: ✅ 39.323MB (SLO: <41.000MB -4.1%) vs baseline: +5.4%

---

✅ import_one

Time: ✅ 19.711µs (SLO: <30.000µs 📉 -34.3%) vs baseline: ~same

Memory: ✅ 39.093MB (SLO: <41.000MB -4.7%) vs baseline: +4.7%

---

✅ import_one_cache

Time: ✅ 6.253µs (SLO: <10.000µs 📉 -37.5%) vs baseline: -0.1%

Memory: ✅ 39.057MB (SLO: <41.000MB -4.7%) vs baseline: +3.9%

---

✅ import_one_stdlib

Time: ✅ 18.589µs (SLO: <20.000µs -7.1%) vs baseline: -0.4%

Memory: ✅ 39.076MB (SLO: <41.000MB -4.7%) vs baseline: +4.2%

---

✅ import_one_stdlib_cache

Time: ✅ 6.298µs (SLO: <10.000µs 📉 -37.0%) vs baseline: +0.4%

Memory: ✅ 39.101MB (SLO: <41.000MB -4.6%) vs baseline: +4.6%

---

✅ import_one_unknown

Time: ✅ 45.470µs (SLO: <50.000µs -9.1%) vs baseline: +0.1%

Memory: ✅ 39.166MB (SLO: <41.000MB -4.5%) vs baseline: +5.3%

---

✅ import_one_unknown_cache

Time: ✅ 6.355µs (SLO: <10.000µs 📉 -36.4%) vs baseline: +0.8%

Memory: ✅ 39.116MB (SLO: <41.000MB -4.6%) vs baseline: +5.0%

---

✅ ratelimiter - 12/12

✅ defaults

Time: ✅ 2.367µs (SLO: <10.000µs 📉 -76.3%) vs baseline: +0.2%

Memory: ✅ 34.603MB (SLO: <35.500MB -2.5%) vs baseline: +5.0%

---

✅ high_rate_limit

Time: ✅ 2.394µs (SLO: <10.000µs 📉 -76.1%) vs baseline: -0.9%

Memory: ✅ 34.524MB (SLO: <35.500MB -2.7%) vs baseline: +4.6%

---

✅ long_window

Time: ✅ 2.352µs (SLO: <10.000µs 📉 -76.5%) vs baseline: -0.5%

Memory: ✅ 34.505MB (SLO: <35.500MB -2.8%) vs baseline: +4.6%

---

✅ low_rate_limit

Time: ✅ 2.359µs (SLO: <10.000µs 📉 -76.4%) vs baseline: -0.3%

Memory: ✅ 34.603MB (SLO: <35.500MB -2.5%) vs baseline: +5.3%

---

✅ no_rate_limit

Time: ✅ 0.823µs (SLO: <10.000µs 📉 -91.8%) vs baseline: -0.5%

Memory: ✅ 34.564MB (SLO: <35.500MB -2.6%) vs baseline: +4.8%

---

✅ short_window

Time: ✅ 2.493µs (SLO: <10.000µs 📉 -75.1%) vs baseline: ~same

Memory: ✅ 34.623MB (SLO: <35.500MB -2.5%) vs baseline: +4.9%

---

✅ span - 26/26

✅ add-event

Time: ✅ 17.374ms (SLO: <22.500ms 📉 -22.8%) vs baseline: +0.7%

Memory: ✅ 36.466MB (SLO: <53.000MB 📉 -31.2%) vs baseline: +5.2%

---

✅ add-metrics

Time: ✅ 85.891ms (SLO: <93.500ms -8.1%) vs baseline: -1.2%

Memory: ✅ 40.793MB (SLO: <53.000MB 📉 -23.0%) vs baseline: +4.8%

---

✅ add-tags

Time: ✅ 139.246ms (SLO: <155.000ms 📉 -10.2%) vs baseline: -0.1%

Memory: ✅ 40.462MB (SLO: <53.000MB 📉 -23.7%) vs baseline: +5.0%

---

✅ get-context

Time: ✅ 16.059ms (SLO: <20.500ms 📉 -21.7%) vs baseline: ~same

Memory: ✅ 36.088MB (SLO: <53.000MB 📉 -31.9%) vs baseline: +4.4%

---

✅ is-recording

Time: ✅ 16.363ms (SLO: <20.500ms 📉 -20.2%) vs baseline: +0.8%

Memory: ✅ 36.130MB (SLO: <53.000MB 📉 -31.8%) vs baseline: +4.8%

---

✅ record-exception

Time: ✅ 35.642ms (SLO: <40.000ms 📉 -10.9%) vs baseline: +1.6%

Memory: ✅ 36.843MB (SLO: <53.000MB 📉 -30.5%) vs baseline: +4.7%

---

✅ set-status

Time: ✅ 17.945ms (SLO: <22.000ms 📉 -18.4%) vs baseline: +0.9%

Memory: ✅ 36.145MB (SLO: <53.000MB 📉 -31.8%) vs baseline: +4.5%

---

✅ start

Time: ✅ 16.103ms (SLO: <20.500ms 📉 -21.5%) vs baseline: +0.4%

Memory: ✅ 36.180MB (SLO: <53.000MB 📉 -31.7%) vs baseline: +4.7%

---

✅ start-finish

Time: ✅ 50.989ms (SLO: <52.500ms -2.9%) vs baseline: +0.6%

Memory: ✅ 34.564MB (SLO: <35.500MB -2.6%) vs baseline: +4.9%

---

✅ start-finish-telemetry

Time: ✅ 52.131ms (SLO: <54.500ms -4.3%) vs baseline: -0.1%

Memory: ✅ 34.544MB (SLO: <35.500MB -2.7%) vs baseline: +4.9%

---

✅ start-finish-traceid128

Time: ✅ 54.506ms (SLO: <57.000ms -4.4%) vs baseline: +1.3%

Memory: ✅ 34.485MB (SLO: <35.500MB -2.9%) vs baseline: +4.6%

---

✅ start-traceid128

Time: ✅ 16.550ms (SLO: <22.500ms 📉 -26.4%) vs baseline: +0.6%

Memory: ✅ 36.299MB (SLO: <53.000MB 📉 -31.5%) vs baseline: +4.8%

---

✅ update-name

Time: ✅ 16.479ms (SLO: <22.000ms 📉 -25.1%) vs baseline: +0.3%

Memory: ✅ 36.429MB (SLO: <53.000MB 📉 -31.3%) vs baseline: +4.9%

ℹ️ Scenarios Missing SLO Configuration (7 scenarios)

The following scenarios exist in candidate data but have no SLO thresholds configured:

• coreapiscenario-core_dispatch_listeners
• coreapiscenario-core_dispatch_no_listeners
• coreapiscenario-core_dispatch_with_results_listeners
• coreapiscenario-core_dispatch_with_results_no_listeners
• errortrackingflasksqli-baseline
• flasksimple-baseline
• flasksqli-baseline