feat(scorers): add non_final_output_tokens metric robust to answer-verbosity variance

[prernakakkar-google]

What & why

token_consumption (input + output) is noisy for comparing agent efficiency
because output tokens carry high, presentation-driven variance: two runs can
invoke the identical tool calls yet differ widely in output because one writes
a terse summary while the other renders a full table. That variance makes
side-by-side efficiency comparison unreliable, especially for one-shot runs.

This adds non_final_output_tokens, a new deterministic scorer that reports
the output tokens spent on the path to the answer (reasoning + tool-call
emission) while excluding the final rendered response text. Per turn it computes
sum(candidates) − estimate(response) (clamped at 0), where the response-size
estimate reuses the existing len/4 chars-per-token heuristic from
mcp_tool_metrics.py. token_consumption is left unchanged — this is added as a
sibling, mirroring how tokens_processed / effective_billed_tokens were
introduced.

Tested

https://paste.googleplex.com/6298204256731136

[prernakakkar-google]

/gcbrun