util-genai: include reasoning tokens in the output token metric

[venkat-uk]

Description

InferenceInvocation treats reasoning tokens as part of the output token count on the
span, but not on the token usage metric.

_get_attributes() sets gen_ai.usage.output_tokens to output_tokens + thinking_tokens.
_get_metric_token_counts() recorded output_tokens alone, so the
gen_ai.client.token.usage histogram and the span disagreed for any response that carried
reasoning tokens. When only thinking_tokens was set the histogram recorded no output
point at all, while the span still reported the count.

This matters for cost: providers bill reasoning tokens as output tokens, and the GenAI
semantic conventions say gen_ai.usage.reasoning.output_tokens SHOULD be included in
gen_ai.usage.output_tokens. A dashboard built on the metric under-reported spend against
the same call the span reported correctly.

I moved the sum into _get_total_output_tokens() and used it in both places, so the two
paths cannot drift apart again. The comment on self.output_tokens already stated this
intent: "Output tokens will ultimately be the sum of normal output tokens and thinking
tokens."

Only InferenceInvocation carries thinking_tokens, so nothing else changes. Inputs are
untouched, and an invocation with no reasoning tokens records exactly what it recorded
before.

Fixes # (no issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Two new cases in util/opentelemetry-util-genai/tests/test_handler_metrics.py, each
  asserting the span attribute and the metric point together so the two can be compared
  directly:
  • test_stop_llm_token_metric_counts_reasoning_tokens_as_output: output_tokens=7,
    thinking_tokens=3. Before the change the span read 10 and the metric read 7. Both
    read 10 now.
  • test_stop_llm_token_metric_records_reasoning_tokens_alone: thinking_tokens=4 with
    no output_tokens. Before the change the span read 4 and gen_ai.client.token.usage
    was never emitted. Both read 4 now.

Both tests fail on 466ae4c without the source change.

Commands run locally on Python 3.12:

python -m pytest util/opentelemetry-util-genai/tests
233 passed (231 before, plus the 2 new cases)

ruff 0.16.0 check + format --check on the two changed files    # clean
pylint 4.0.5 --rcfile .pylintrc opentelemetry-util-genai       # 9.98/10, no message on the changed files
pyright 1.1.404 util/opentelemetry-util-genai                  # same result before and after the change

Does This PR Require a Core Repo Change?

• Yes. - Link to PR:
• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[linux-foundation-easycla]

• ❌ - login: @venkat-uk / name: Venkat Ramachandran. The commit (26ee363, 8ff4121) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Waiting on the author · refreshed 2026-08-22 13:17 UTC

Investigate required status check failures.

Status above doesn't look right?

• Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
• Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
• Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.