Skip to content

Python: [Bug]: BedrockChatClient and ClaudeAgent report input_token_count without cached prompt tokens #9988

Closed

Description

@lalith0407

Description

Same root cause as #9808 (AnthropicClient, fixed by #9859), in two more connectors that #9859 did not touch.

  • BedrockChatClient._parse_usage maps Converse inputTokens to input_token_count. With prompt caching, Bedrock reports cache reads and writes separately in cacheReadInputTokens / cacheWriteInputTokens and inputTokens covers only the uncached part (AWS prompt caching docs). totalTokens does include the cache, so on main the usage is internally inconsistent: input 12 + output 40 but total 9352.
  • ClaudeAgent builds input_token_count from the Claude Agent SDK ResultMessage.usage["input_tokens"], which is Anthropic's uncached-only field, and computes total_token_count = input + output, so both undercount.

input_token_count is the total input for the OpenAI and Gemini clients and is exported as gen_ai.usage.input_tokens, which the OTel GenAI semconv says should include cached tokens.

Observed Behavior (main 6fdeb2d)

Prompt = 12 uncached + 300 cache write + 9000 cache read:

[bedrock] input_token_count=12 total=9352 cache_read=9000 cache_write=300
[claude ] input_token_count=12 total=52 cache_read=9000 cache_write=300

Expected Behavior

input_token_count == 9312 for both; ClaudeAgent total_token_count == 9352. Cache fields unchanged.

Steps to Reproduce

Offline (stub Bedrock runtime, mocked ClaudeSDKClient, no keys):

repro.py
"""Offline repro (no AWS/Anthropic calls, no keys): Bedrock and ClaudeAgent report input_token_count
without cached prompt tokens. Prompt = 12 uncached + 300 cache write + 9000 cache read = 9312 tokens."""
import asyncio
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch

from agent_framework import Content, Message
from agent_framework_bedrock import BedrockChatClient

USAGE = {"inputTokens": 12, "outputTokens": 40, "totalTokens": 9352,
         "cacheReadInputTokens": 9000, "cacheWriteInputTokens": 300}


class Runtime:
    def converse(self, **kw: Any) -> dict[str, Any]:
        return {"modelId": kw["modelId"], "usage": USAGE,
                "output": {"completionReason": "end_turn",
                           "message": {"role": "assistant", "content": [{"text": "hi"}]}}}


async def bedrock() -> None:
    client = BedrockChatClient(model="anthropic.claude-sonnet", region="us-west-2", client=Runtime())  # type: ignore[arg-type]
    r = await client.get_response(messages=[Message(role="user", contents=[Content.from_text(text="hi")])])
    u = r.usage_details or {}
    print(f"[bedrock] input_token_count={u.get('input_token_count')} total={u.get('total_token_count')} "
          f"cache_read={u.get('cache_read_input_token_count')} cache_write={u.get('cache_creation_input_token_count')}")


async def claude() -> None:
    from claude_agent_sdk import AssistantMessage, ResultMessage, TextBlock
    from agent_framework_claude import ClaudeAgent

    async def gen():
        yield AssistantMessage(content=[TextBlock(text="hi")], model="claude-sonnet")
        yield ResultMessage(subtype="success", duration_ms=1, duration_api_ms=1, is_error=False, num_turns=1,
                            session_id="s", stop_reason="end_turn",
                            usage={"input_tokens": 12, "output_tokens": 40,
                                   "cache_creation_input_tokens": 300, "cache_read_input_tokens": 9000})
    m = MagicMock()
    for name in ("connect", "disconnect", "query", "set_model", "set_permission_mode"):
        setattr(m, name, AsyncMock())
    m.receive_response = MagicMock(return_value=gen())
    with patch("agent_framework_claude._agent.ClaudeSDKClient", return_value=m):
        r = await ClaudeAgent().run("hello")
    u = r.usage_details or {}
    print(f"[claude ] input_token_count={u.get('input_token_count')} total={u.get('total_token_count')} "
          f"cache_read={u.get('cache_read_input_token_count')} cache_write={u.get('cache_creation_input_token_count')}")


asyncio.run(bedrock()); asyncio.run(claude())
print("expected: input_token_count=9312 (12 + 300 + 9000) for both; claude total=9352")

Notes

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 2, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Nov 2, 2026
  3. github-actions commented on Nov 2, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    Bedrock’s bug is in python/packages/bedrock/agent_framework_bedrock/_chat_client.py::_parse_usage, while Claude’s is in python/packages/claude/agent_framework_claude/_agent.py ResultMessage handling around lines 981-1003. Providing nonzero uncached, cache-read, and cache-write token counts leaves cached tokens out of aggregate input usage and Claude total usage. Reproduce by running the two modified focused usage tests.

    • Failing test: python/packages/bedrock/tests/test_bedrock_client.py, python/packages/claude/tests/test_claude_agent.py
    • Files examined: python/packages/bedrock/agent_framework_bedrock/_chat_client.py, python/packages/claude/agent_framework_claude/_agent.py, python/packages/bedrock/tests/test_bedrock_client.py, python/packages/claude/tests/test_claude_agent.py
    • Tests run: test_parse_usage_surfaces_cache_tokens, TestClaudeAgentRun::test_run_captures_result_message_usage_and_finish_reason
  4. added
    agentsUsage: [Issues, PRs], Target: Single agent
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 2, 2026
  5. kimnamu commented on Nov 6, 2026

    @kimnamu
    Contributor

    Thanks lalith0407 for the exact numbers and the offline repro, they made the Bedrock half quick to check against the live API.

    Live Converse and ConverseStream in us-east-1 agree with the issue: inputTokens leaves out cache reads and writes while totalTokens includes them, on every model that returned cache fields. Grok 4.7, Kimi K3 and GPT-6 Luna report the same split without any cachePoint, so on main a repeated Grok prompt already shows input_token_count=0. With the #10003 source change applied, both live client runs below give input + output == total.

    BedrockChatClient, live, 2nd call main 1c3264b main + #10003
    Claude Haiku 4.5 + cachePoint, input_token_count 10 18901
    same, get_response(stream=True) 10 18902
    Grok 4.7, no cachePoint, input_token_count 0 3414
    input + output == total ❌ ✅
    total_token_count, cache fields Bedrock's values unchanged
    $ python raw_usage.py <model>[@auto] ...   # selected lines; 2 calls each over Converse (plain) and ConverseStream (stream)
    us.anthropic.claude-haiku-4-5-20251001-v1:0  plain   1st   in=10 out=5 cache_read=0 cache_write=18892 total=18907 total-(in+out)=18892 extra_keys=['cacheDetails']
    us.anthropic.claude-haiku-4-5-20251001-v1:0  stream  2nd   in=10 out=5 cache_read=18894 cache_write=0 total=18909 total-(in+out)=18894 extra_keys=[]
    us.anthropic.claude-sonnet-5-5               plain   2nd   in=13 out=4 cache_read=21093 cache_write=0 total=21110 total-(in+out)=21093 extra_keys=[]
    us.amazon.nova-lite-v1:0                     stream  2nd   in=7 out=3 cache_read=18559 cache_write=0 total=18569 total-(in+out)=18559 extra_keys=[]
    us.amazon.nova-2-lite-v1:0                   plain   1st   in=5 out=4 cache_read=0 cache_write=13440 total=13449 total-(in+out)=13440 extra_keys=[]
    us.xai.grok-4.7                              plain   2nd   in=0 out=16 cache_read=3413 cache_write=0 total=3429 total-(in+out)=3413 extra_keys=[]
    us.moonshotai.kimi-k3                        stream  2nd   in=7 out=16 cache_read=3471 cache_write=0 total=3494 total-(in+out)=3471 extra_keys=[]
    openai.gpt-oss-120b-1:0                      plain   2nd   in=3451 out=16 cache_read=None cache_write=None total=3467 total-(in+out)=0 extra_keys=[]
    us.openai.gpt-6-luna                         plain   1st   in=2 out=16 cache_read=0 cache_write=3390 total=3408 total-(in+out)=3390 extra_keys=[]
    us.deepseek.r1-v1:0                          stream  2nd   in=3385 out=16 cache_read=None cache_write=None total=3401 total-(in+out)=0 extra_keys=[]
    
    Probe scripts, full output, unit test red/green
    """Live Converse / ConverseStream usage with prompt caching: is inputTokens exclusive of cache tokens?"""
    import sys, uuid, boto3
    
    rt = boto3.client("bedrock-runtime", region_name="us-east-1")
    FILLER = " ".join(f"Fact {i}: item {i} weighs {i * 7 % 97} grams." for i in range(1, 1100))
    
    
    def call(model, system, stream):
        req = dict(modelId=model, system=system,
                   messages=[{"role": "user", "content": [{"text": "Reply with OK."}]}],
                   inferenceConfig={"maxTokens": 16})
        if not stream:
            return rt.converse(**req)["usage"]
        for ev in rt.converse_stream(**req)["stream"]:
            if "metadata" in ev:
                return ev["metadata"]["usage"]
    
    
    def show(model, mode, turn, u):
        keys = ("inputTokens", "outputTokens", "cacheReadInputTokens", "cacheWriteInputTokens", "totalTokens")
        i, o, r, w, t = (u.get(k) for k in keys)
        gap = None if t is None else t - (i or 0) - (o or 0)
        print(f"{model:44} {mode:7} {turn:5} in={i} out={o} cache_read={r} cache_write={w} total={t} "
              f"total-(in+out)={gap} extra_keys={sorted(set(u) - set(keys))}")
    
    
    for spec in sys.argv[1:]:
        model, _, flag = spec.partition("@")
        for stream in (False, True):
            system = [{"text": f"run {uuid.uuid4().hex[:8]}. " + (FILLER[: len(FILLER) // 4] if flag == "auto" else FILLER)}]
            if flag != "auto":
                system.append({"cachePoint": {"type": "default"}})
            for turn in ("1st", "2nd"):
                try:
                    show(model, "stream" if stream else "plain", turn, call(model, system, stream))
                except Exception as e:  # noqa: BLE001
                    print(f"{model:44} {'stream' if stream else 'plain':7} {turn:5} ERROR {type(e).__name__}: {str(e)[:160]}")
                    break
    $ python raw_usage.py us.anthropic.claude-haiku-4-5-20251001-v1:0 us.anthropic.claude-sonnet-5-5 us.amazon.nova-lite-v1:0 us.amazon.nova-2-lite-v1:0
    us.anthropic.claude-haiku-4-5-20251001-v1:0  plain   1st   in=10 out=5 cache_read=0 cache_write=18892 total=18907 total-(in+out)=18892 extra_keys=['cacheDetails']
    us.anthropic.claude-haiku-4-5-20251001-v1:0  plain   2nd   in=10 out=5 cache_read=18892 cache_write=0 total=18907 total-(in+out)=18892 extra_keys=[]
    us.anthropic.claude-haiku-4-5-20251001-v1:0  stream  1st   in=10 out=5 cache_read=0 cache_write=18894 total=18909 total-(in+out)=18894 extra_keys=['cacheDetails']
    us.anthropic.claude-haiku-4-5-20251001-v1:0  stream  2nd   in=10 out=5 cache_read=18894 cache_write=0 total=18909 total-(in+out)=18894 extra_keys=[]
    us.anthropic.claude-sonnet-5-5               plain   1st   in=13 out=5 cache_read=0 cache_write=21093 total=21111 total-(in+out)=21093 extra_keys=['cacheDetails']
    us.anthropic.claude-sonnet-5-5               plain   2nd   in=13 out=4 cache_read=21093 cache_write=0 total=21110 total-(in+out)=21093 extra_keys=[]
    us.anthropic.claude-sonnet-5-5               stream  1st   in=13 out=4 cache_read=0 cache_write=21093 total=21110 total-(in+out)=21093 extra_keys=['cacheDetails']
    us.anthropic.claude-sonnet-5-5               stream  2nd   in=13 out=4 cache_read=21093 cache_write=0 total=21110 total-(in+out)=21093 extra_keys=[]
    us.amazon.nova-lite-v1:0                     plain   1st   in=7 out=3 cache_read=0 cache_write=18558 total=18568 total-(in+out)=18558 extra_keys=[]
    us.amazon.nova-lite-v1:0                     plain   2nd   in=7 out=3 cache_read=0 cache_write=18558 total=18568 total-(in+out)=18558 extra_keys=[]
    us.amazon.nova-lite-v1:0                     stream  1st   in=7 out=3 cache_read=0 cache_write=18559 total=18569 total-(in+out)=18559 extra_keys=[]
    us.amazon.nova-lite-v1:0                     stream  2nd   in=7 out=3 cache_read=18559 cache_write=0 total=18569 total-(in+out)=18559 extra_keys=[]
    us.amazon.nova-2-lite-v1:0                   plain   1st   in=5 out=4 cache_read=0 cache_write=13440 total=13449 total-(in+out)=13440 extra_keys=[]
    us.amazon.nova-2-lite-v1:0                   plain   2nd   in=5 out=4 cache_read=0 cache_write=13440 total=13449 total-(in+out)=13440 extra_keys=[]
    us.amazon.nova-2-lite-v1:0                   stream  1st   in=5 out=4 cache_read=0 cache_write=13441 total=13450 total-(in+out)=13441 extra_keys=[]
    us.amazon.nova-2-lite-v1:0                   stream  2nd   in=5 out=4 cache_read=0 cache_write=13441 total=13450 total-(in+out)=13441 extra_keys=[]
    $ python raw_usage.py us.xai.grok-4.7@auto us.moonshotai.kimi-k3@auto openai.gpt-oss-120b-1:0@auto us.openai.gpt-6-luna@auto us.deepseek.r1-v1:0@auto
    us.xai.grok-4.7                              plain   1st   in=3413 out=16 cache_read=0 cache_write=0 total=3429 total-(in+out)=0 extra_keys=[]
    us.xai.grok-4.7                              plain   2nd   in=0 out=16 cache_read=3413 cache_write=0 total=3429 total-(in+out)=3413 extra_keys=[]
    us.xai.grok-4.7                              stream  1st   in=3414 out=16 cache_read=None cache_write=None total=3430 total-(in+out)=0 extra_keys=[]
    us.xai.grok-4.7                              stream  2nd   in=0 out=16 cache_read=3414 cache_write=0 total=3430 total-(in+out)=3414 extra_keys=[]
    us.moonshotai.kimi-k3                        plain   1st   in=3476 out=16 cache_read=0 cache_write=0 total=3492 total-(in+out)=0 extra_keys=[]
    us.moonshotai.kimi-k3                        plain   2nd   in=7 out=16 cache_read=3469 cache_write=0 total=3492 total-(in+out)=3469 extra_keys=[]
    us.moonshotai.kimi-k3                        stream  1st   in=3478 out=16 cache_read=None cache_write=None total=3494 total-(in+out)=0 extra_keys=[]
    us.moonshotai.kimi-k3                        stream  2nd   in=7 out=16 cache_read=3471 cache_write=0 total=3494 total-(in+out)=3471 extra_keys=[]
    openai.gpt-oss-120b-1:0                      plain   1st   in=3451 out=16 cache_read=None cache_write=None total=3467 total-(in+out)=0 extra_keys=[]
    openai.gpt-oss-120b-1:0                      plain   2nd   in=3451 out=16 cache_read=None cache_write=None total=3467 total-(in+out)=0 extra_keys=[]
    openai.gpt-oss-120b-1:0                      stream  1st   in=3450 out=16 cache_read=None cache_write=None total=3466 total-(in+out)=0 extra_keys=[]
    openai.gpt-oss-120b-1:0                      stream  2nd   in=3450 out=16 cache_read=None cache_write=None total=3466 total-(in+out)=0 extra_keys=[]
    us.openai.gpt-6-luna                         plain   1st   in=2 out=16 cache_read=0 cache_write=3390 total=3408 total-(in+out)=3390 extra_keys=[]
    us.openai.gpt-6-luna                         plain   2nd   in=2 out=16 cache_read=3390 cache_write=0 total=3408 total-(in+out)=3390 extra_keys=[]
    us.openai.gpt-6-luna                         stream  1st   in=2 out=5 cache_read=0 cache_write=3389 total=3396 total-(in+out)=3389 extra_keys=[]
    us.openai.gpt-6-luna                         stream  2nd   in=2 out=5 cache_read=3389 cache_write=0 total=3396 total-(in+out)=3389 extra_keys=[]
    us.deepseek.r1-v1:0                          plain   1st   in=3383 out=16 cache_read=None cache_write=None total=3399 total-(in+out)=0 extra_keys=[]
    us.deepseek.r1-v1:0                          plain   2nd   in=3383 out=16 cache_read=None cache_write=None total=3399 total-(in+out)=0 extra_keys=[]
    us.deepseek.r1-v1:0                          stream  1st   in=3385 out=16 cache_read=None cache_write=None total=3401 total-(in+out)=0 extra_keys=[]
    us.deepseek.r1-v1:0                          stream  2nd   in=3385 out=16 cache_read=None cache_write=None total=3401 total-(in+out)=0 extra_keys=[]
    
    """Live BedrockChatClient usage_details with a cached system prompt (Haiku 4.5, us-east-1)."""
    import asyncio, uuid
    from agent_framework import Message
    from agent_framework_bedrock import BedrockChatClient
    
    FILLER = " ".join(f"Fact {i}: item {i} weighs {i * 7 % 97} grams." for i in range(1, 1100))
    MODEL = "us.anthropic.claude-haiku-4-5-20251001-v1:0"
    
    
    async def main() -> None:
        client = BedrockChatClient(model=MODEL, region="us-east-1")
        for stream in (False, True):
            system = [{"text": f"run {uuid.uuid4().hex[:8]}. " + FILLER}, {"cachePoint": {"type": "default"}}]
            for turn in ("1st", "2nd"):
                msgs = [Message(role="user", contents=["Reply with OK."])]
                kw = {"client_kwargs": {"system": system}, "options": {"max_tokens": 16}}
                if stream:
                    r = await client.get_response(msgs, stream=True, **kw).get_final_response()
                else:
                    r = await client.get_response(msgs, **kw)
                u = r.usage_details or {}
                i, o, t = u.get("input_token_count"), u.get("output_token_count"), u.get("total_token_count")
                print(f"{'stream' if stream else 'plain':6} {turn} input_token_count={i} output={o} total={t} "
                      f"cache_read={u.get('cache_read_input_token_count')} cache_write={u.get('cache_creation_input_token_count')} "
                      f"input+output==total: {i + o == t}")
    
    
    asyncio.run(main())
    # client_auto.py: same get_response call on us.xai.grok-4.7 with no cachePoint, a quarter of FILLER, plain only
    # main 1c3264b0
    $ python client_usage.py
    plain  1st input_token_count=10 output=5 total=18906 cache_read=0 cache_write=18891 input+output==total: False
    plain  2nd input_token_count=10 output=5 total=18906 cache_read=18891 cache_write=0 input+output==total: False
    stream 1st input_token_count=10 output=5 total=18906 cache_read=0 cache_write=18891 input+output==total: False
    stream 2nd input_token_count=10 output=5 total=18906 cache_read=18891 cache_write=0 input+output==total: False
    $ python client_auto.py us.xai.grok-4.7
    us.xai.grok-4.7 1st input_token_count=3413 output=16 total=3429 cache_read=0 input+output==total: True
    us.xai.grok-4.7 2nd input_token_count=0 output=16 total=3429 cache_read=3413 input+output==total: False
    # main 1c3264b0 + #10003 source diff (685f0cb2)
    $ python client_usage.py
    plain  1st input_token_count=18901 output=5 total=18906 cache_read=0 cache_write=18891 input+output==total: True
    plain  2nd input_token_count=18901 output=5 total=18906 cache_read=18891 cache_write=0 input+output==total: True
    stream 1st input_token_count=18902 output=5 total=18907 cache_read=0 cache_write=18892 input+output==total: True
    stream 2nd input_token_count=18902 output=5 total=18907 cache_read=18892 cache_write=0 input+output==total: True
    $ python client_auto.py us.xai.grok-4.7
    us.xai.grok-4.7 1st input_token_count=3414 output=16 total=3430 cache_read=0 input+output==total: True
    us.xai.grok-4.7 2nd input_token_count=3414 output=16 total=3430 cache_read=3414 input+output==total: True
    
    # #10003 test change only, main source
    $ python -m pytest tests/test_bedrock_client.py -k parse_usage
    >       assert details["input_token_count"] == 21
    E       assert 10 == 21
    FAILED tests/test_bedrock_client.py::test_parse_usage_surfaces_cache_tokens
    1 failed, 1 passed, 68 deselected, 1 warning in 0.48s
    # + #10003 source change
    $ python -m pytest tests/ -m "not integration"
    105 passed, 1 deselected, 2 warnings in 2.48s
    

    I only checked the Bedrock half, not ClaudeAgent, and I'm leaving the fix with #10003. Claude (via Claude Code) assisted with the probes and this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

agentsUsage: [Issues, PRs], Target: Single agentpythonUsage: [Issues, PRs], Target: PythonreproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions