Add E2E Normalized Interactivity x-axis metric as the agentic default / 新增端到端归一化交互性 X 轴指标并设为智能体默认 - #638
Open
cquil11 wants to merge 13 commits into
Open
Add E2E Normalized Interactivity x-axis metric as the agentic default / 新增端到端归一化交互性 X 轴指标并设为智能体默认#638cquil11 wants to merge 13 commits into
cquil11 wants to merge 13 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
cquil11
changed the title
…xis modes Retire the three experimental agentic x-axis modes and everything that existed solely to serve them, leaving Interactivity / E2E Latency / TTFT. - chart: drop the three mode buttons, DERIVED_X_MODE_SPECS and the derived-metric fetch/remap plumbing in ChartDisplay; the remaining modes apply to both scenario kinds, so the agentic-only button filter (and its `mounted` SSR guard) goes too - API/hook/db: delete /api/v1/derived-agentic-metrics, the useDerivedAgenticMetrics hook, and queries/derived-agentic-metrics.ts - aggregate_stats v6: drop normalizedSessionTimeS, p90PrefillTpsPerUser and normalizedE2e400; mergeProfileStatsUpgrade no longer carries them forward, and the backfill's profile-only fast path now covers every v3+ bundle instead of only v3 - drop the NORMALIZED_E2E_OUTPUT_TOKENS constant and the overlay suppression helper that existed only for Normalized E2E 中文：移除三个实验性智能体 X 轴模式（Normalized E2E、会话时长、 Prefill TPS / user）及其专属实现，仅保留交互性 / 端到端延迟 / TTFT。 同时删除对应的 API 路由、React Query hook 与数据库查询模块； aggregate_stats 升级至 v6，去掉三个已废弃字段，backfill 的 profile-only 快速路径扩展至所有 v3 及以上版本。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Introduces "OSL / E2EL" ("e2e interactivity") for agentic scenarios: the
per-request output-token rate INCLUDING the prefill wait,
OSL / (TTFT + generation time) ≈ 1 / (ITL + TTFT/OSL). Unlike plain
interactivity it cannot be inflated by delaying prefill.
- aggregate_stats v7: store `e2elPerOsl`, percentiles of the per-request
E2EL/OSL ratio (seconds per output token). The read path inverts, so
pXX OSL/E2EL = 1 / pXX(E2EL/OSL) — the slow-tail convention the ETL
already enforces for `*_intvty`
- API: /api/v1/derived-agentic-metrics returns p75/p90_osl_per_e2el
- chart: new mode, agentic default, listed first; fixed-seq is unchanged
and never shows the button
- overlays: suppressed in this mode (unofficial rows carry no persisted
per-request trace) with a caption disclaimer
- shared cypress helper interceptDerivedAgenticMetrics stubs the fetch
the default mode fires on mount; overlay specs switch to Interactivity
explicitly before asserting overlay points
中文：为智能体场景新增 "OSL / E2EL"（e2e interactivity）指标：每请求
输出 token 速率（含 prefill 等待），即 OSL /（TTFT + 生成时间）。
aggregate_stats 升级至 v7，存储每请求 E2EL/OSL 比值的分位数，读取时取
倒数以沿用 `*_intvty` 的慢尾约定；新增 API 字段与图表模式，并设为智能体
默认模式且置于最左侧；该模式下隐藏非官方运行覆盖层并显示提示文案。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cquil11
force-pushed
the
feat/agentic-osl-e2el-metric
branch
from
July 29, 2026 21:58
318b573 to
a65a098
Compare
cquil11
changed the base branch from
master
to
chore/remove-agentic-derived-x-axis-modes
July 29, 2026 21:58
cquil11
changed the title
The "switches back to Interactivity" test asserted a P75 axis label, but nothing in it selected p75 — it inherited the selector state from the Normalized E2E test that ran before it. Removing that test left the selector on p90 and the assertion failed. Assert P90 there (the default) and cover the p75 case in its own test that selects the percentile itself. 中文：「switches back to Interactivity」用例断言 P75 轴标签，但其自身 并未切换分位数，而是依赖此前 Normalized E2E 用例遗留的选择器状态。 该用例被移除后选择器停留在 p90，断言随之失败。现改为断言默认的 P90， 并将 p75 场景拆分为独立用例，由其自行切换分位数。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…entic-osl-e2el-metric
The percentile selector is page state shared across the whole suite, so leaving it on p75 made the following Interactivity test's P90 assertion depend on test order. Restore the default at the end of the test that changes it. 中文：分位数选择器是整个套件共享的页面状态，停留在 p75 会使后续 Interactivity 用例的 P90 断言依赖执行顺序。现在由改动它的用例在结束时 恢复默认值。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…red mode The reconcile effect ran before availability resolved the sequence. On a cold load it recorded the fixed-seq placeholder kind, then treated the switch to agentic as a user-driven kind change and snapped to the agentic default — so /inference?i_seq=agentic-traces&i_xmode=interactivity landed on OSL / E2EL instead. Gate the effect on `sequenceResolved`, matching the label effect above it, so the first run sees the real kind and the URL-restored mode survives. Found by Cursor Bugbot; reproduced against the live DB before and after. Pre-existing on master (it clobbered a URL-restored TTFT/E2E with interactivity there) — visible now because the agentic default differs. 中文：X 轴模式协调 effect 在 availability 解析出场景之前就已运行：冷启动 时它记录了固定序列占位场景，随后把切换到智能体场景误判为用户主动切换， 从而覆盖 URL 中的 `i_xmode`。现与其上方的 label effect 一致，改为在 `sequenceResolved` 之后才执行。该问题在 master 上已存在，因智能体默认模式 变更后才变得可见。新增回归用例（去掉修复即失败）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…2EL chart The mode hardcoded `upper_left` for every y-metric. That is right for throughput but inverts the frontier for cost and joules metrics, whose good direction is a LOWER corner — and Optimal Only filters off the same flag, so the wrong points were hidden on what is now the agentic default view. Derived modes render on the e2e chart definition (lower-x-is-better) while OSL / E2EL is higher-is-better, so the correct transform is a horizontal mirror of each configured corner, not a constant: upper_right → upper_left for throughput, lower_left → lower_right for cost/joules. Extracted as `derivedModeRoofline` and unit-tested against the real chart config — the mirrored e2e corner must equal the interactivity chart's corner for every y-metric, since both axes are higher-is-better. Found by Cursor Bugbot. Confirmed the tests fail with the hardcoded corner restored and pass with the mirror. 中文：该模式此前对所有 Y 轴指标硬编码 `upper_left`，对吞吐量正确，但会让 成本与能耗类指标的帕累托前沿方向反转（Optimal Only 也依赖同一标记， 导致隐藏了错误的点）。派生模式渲染在 e2e 图表定义之上（x 越小越好）， 而 OSL / E2EL 是越大越好，因此正确做法是对配置中的角位做水平镜像而非 固定取值。已抽出 `derivedModeRoofline` 并针对真实图表配置编写单元测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The derived-metrics read path only sees the profile blob, so it cannot recompute kvCacheUtil / prefixCacheHitRate — it carries them forward from the stale row. When the stale row had none (null stats, or a pre-v3 bundle), it still wrote nulls stamped at the current STATS_VERSION. That looks complete to everyone downstream: the backfill's candidate query matches on version and skips the row, and agentic-aggregates takes the fast path, so those server-derived fields stay null permanently. It also broke writeBackTraceReplayJsonb's documented contract that callers only persist COMPLETE payloads. Self-heal now only stamps the bundle when there are server-derived fields to preserve. Otherwise the row stays stale, costing one repeat profile parse and letting a reader that CAN see the server blob heal it fully. Pre-existing behavior carried over from before the metric split; found by Cursor Bugbot. Test asserts no UPDATE is issued for a null-stats row and fails if the guard is removed. 中文：派生指标读取路径只能看到 profile blob，无法重算 kvCacheUtil / prefixCacheHitRate，只能从旧行继承。当旧行没有这些字段时， 此前仍会写入 null 并打上当前 STATS_VERSION，使 backfill 跳过该行、 agentic-aggregates 走快速路径，导致这些字段永久为空。现仅在确实有服务端 字段可保留时才写回，否则保持 stale，交由能读取 server blob 的读取方修复。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Base automatically changed from
chore/remove-agentic-derived-x-axis-modes
to
master
July 30, 2026 03:48
…el-metric # Conflicts: # packages/app/cypress/e2e/ttft-x-axis-toggle.cy.ts # packages/app/src/app/api/v1/derived-agentic-metrics/route.ts # packages/app/src/components/inference/InferenceContext.tsx # packages/app/src/components/inference/hooks/useChartData.ts # packages/app/src/components/inference/types.ts # packages/app/src/components/inference/ui/ChartDisplay.tsx # packages/app/src/components/inference/utils.test.ts # packages/app/src/components/inference/utils.ts # packages/app/src/components/inference/utils/e2eFrontier.ts # packages/app/src/hooks/api/use-derived-agentic-metrics.ts # packages/db/src/backfill-aggregate-stats.ts # packages/db/src/etl/compute-aggregate-stats.test.ts # packages/db/src/etl/compute-aggregate-stats.ts # packages/db/src/queries/agentic-aggregates.test.ts # packages/db/src/queries/agentic-aggregates.ts # packages/db/src/queries/agentic-shared.ts # packages/db/src/queries/derived-agentic-metrics.test.ts # packages/db/src/queries/derived-agentic-metrics.ts
This comment has been minimized.
This comment has been minimized.
Renames the agentic x-axis metric everywhere it surfaces: the mode button, the chart heading, the x-axis label (still tok/s/user), the overlay-suppression disclaimer, the `i_xmode` value, data-testids, the `XAxisMode` literal, and the API/query fields (`p75/p90_osl_per_e2el` → `p75/p90_e2e_norm_intvty`, matching the existing `*_intvty` convention). Adds the Chinese labels the old name skipped: mode button, x-axis label, chart heading, and disclaimer now read 端到端归一化交互性 rather than falling back to English. The stored `e2elPerOsl` / `e2el_per_osl` bundle keys are deliberately unchanged — they name the raw persisted quantity (E2E latency per output token, s/tok), which is the reciprocal of the metric; the read path still inverts it. 中文：将智能体 X 轴指标 OSL / E2EL 全面重命名为 E2E Normalized Interactivity，涵盖模式按钮、图表标题、X 轴标签（单位仍为 tok/s/user）、 覆盖层提示文案、`i_xmode` 取值、data-testid、`XAxisMode` 字面量，以及 API 与查询字段（`p75/p90_osl_per_e2el` → `p75/p90_e2e_norm_intvty`， 与既有 `*_intvty` 命名约定一致）。 同时补齐此前缺失的中文文案：模式按钮、X 轴标签、图表标题与提示文案 现均显示"端到端归一化交互性"，不再回退英文。 存储层的 `e2elPerOsl` / `e2el_per_osl` 键名刻意保持不变：它们描述的是 持久化的原始量（每输出 token 的端到端延迟，单位 s/tok），即该指标的倒数， 读取路径仍会取倒数。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
functionstackx
changed the title
| // data forever after a bump — deriving the key from the constant means a | ||
| // STATS_VERSION bump automatically rolls the cache namespace. | ||
| /** Version-derived blob-cache key namespace (exported for the key-derivation test). */ | ||
| export const CACHE_KEY_PREFIX = `derived-agentic-metrics-v${STATS_VERSION}`; |
There was a problem hiding this comment.
Cache key test omits new route
Low Severity
derived-agentic-metrics exports CACHE_KEY_PREFIX for the shared version-derivation guard, but agentic-cache-keys.test.ts still only asserts the older agentic blob routes. The suite that exists to catch forever-stale blobOnly keys after a STATS_VERSION bump does not cover this new endpoint.
Reviewed by Cursor Bugbot for commit 2a2f2c4. Configure here.
csv-export-overlay visits an agentic scenario, so it now lands on E2E Normalized Interactivity, where unofficial-run overlays are suppressed — the `.unofficial-overlay-pt` assertion in `before()` could never pass and E2E shard 3 failed on both browsers. Applies the same fix the other overlay specs already use: stub the derived-metrics fetch, then switch to Interactivity before asserting overlay points. Pre-existing on this branch — shard 3 was already red on the master merge commit, before the rename. 中文：csv-export-overlay 访问的是智能体场景，因此会落到 E2E Normalized Interactivity 模式，而该模式会隐藏非官方运行覆盖层，导致 before() 中的 `.unofficial-overlay-pt` 断言永远无法通过，E2E shard 3 在两种浏览器下均失败。 此处采用与其他覆盖层测试一致的修复方式：先拦截派生指标请求，再切换到 交互性模式后断言覆盖层数据点。 该问题在本次重命名之前就已存在：master 合并提交上的 shard 3 已是红色。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Compute the true Pareto frontier once in E2E Normalized Interactivity space and reuse that exact winner set on E2E latency, Interactivity, and TTFT without a second local Pareto pass.\n\n中文：修复（智能体）：将端到端归一化交互性设为统一 Pareto 前沿，并让端到端延迟、交互性与 TTFT 横轴复用完全相同的优胜点集合，不再进行第二次局部 Pareto 筛选。
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f3a9fe4. Configure here.
| | 'upper_left' | ||
| | 'lower_left' | ||
| | 'lower_right' | ||
| | undefined; |
There was a problem hiding this comment.
Unsorted GPUGraph canonical rooflines
Medium Severity
canonicalFrontierPoints returns winners in input order, and GPUGraph no longer sorts them by x before drawing. The roofline path uses curveMonotoneX, which assumes monotonic x, so agentic GPU-comparison frontiers can render as crossed or distorted curves. ScatterGraph still sorts after the same helper.
Reviewed by Cursor Bugbot for commit f3a9fe4. Configure here.
| const chartCaption = ( | ||
| <> | ||
| <h2 className="text-lg font-semibold"> | ||
| {metricTitle(graph.chartDefinition, selectedYAxisMetric, locale)}{' '} |
There was a problem hiding this comment.
Derived fetch failure blanks all modes
High Severity
Every agentic x-axis now requires derivedMetrics, and when that value is missing the charts are wiped to empty arrays. Loading only shows a skeleton while the query is pending; on error derivedMetrics stays undefined, so E2E, Interactivity, and TTFT all stay blank even though they previously rendered from benchmark rows alone.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit f3a9fe4. Configure here.
|
Found 10 test failures on Blacksmith runners: Failures
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light.png)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds E2E Normalized Interactivity for agentic scenarios: the per-request output-token rate in tok/s/user measured including the prefill wait —
OSL / E2E latency,≈ 1/(ITL + TTFT/OSL). Unlike plain interactivity it cannot be inflated by delaying prefill.What changed
aggregate_statsv6 → v7 addse2elPerOsl, percentiles of the per-request E2EL/OSL ratio (seconds per output token)pXX E2E Normalized Interactivity = 1 / pXX(E2EL/OSL)— the slow-tail convention the ETL already enforces for*_intvty. The existing p75/p90 selector drives it/api/v1/derived-agentic-metricsreturnsp75/p90_e2e_norm_intvtyinterceptDerivedAgenticMetricscypress helper stubs the fetch the default mode fires on mount; overlay specs switch to Interactivity explicitly before asserting overlay pointsNaming
The rename covers every surface the term appears on: mode button, chart heading, x-axis label, overlay disclaimer, the
i_xmodeURL value,data-testids, theXAxisModeliteral (e2e-normalized-interactivity), and the API/query fields (p75/p90_osl_per_e2el→p75/p90_e2e_norm_intvty, matching the existing*_intvtyconvention).One deliberate exception: the stored
e2elPerOsl/e2el_per_oslbundle keys keep their names. They describe the raw persisted quantity — E2E latency per output token, in s/tok — which is the reciprocal of the metric; the read path inverts it. Naming them after the metric would misdescribe what's in the column.Why
Backfill
v7 needs a backfill (
db:backfill-aggregate-stats, profile-only fast path, ~20 min for the current row count). Rows also self-heal on read. The equivalent backfill was run against the target DB during validation; it will be re-run after merge to stamp v7. The rename does not touch stored keys, so it does not invalidate that run.Base
#644(removal of the three retired modes) is merged, andmasteris merged into this branch through2bfa3b60. No conflicts; the diff here is only the new metric.中文说明
为智能体场景新增 E2E Normalized Interactivity（端到端归一化交互性）：每请求输出 token 速率，单位 tok/s/user，且计入 prefill 等待，即
输出 token 数 / 端到端延迟，≈ 1/(ITL + TTFT/OSL)。与普通交互性不同，该指标无法通过延迟 prefill 刷分。改动：
aggregate_statsv6 → v7，新增e2elPerOsl（每请求 E2EL/OSL 比值的分位数，单位 s/tok）*_intvty的慢尾约定：读取时取倒数，pXX 端到端归一化交互性 = 1 / pXX(E2EL/OSL)p75/p90_e2e_norm_intvty命名： 重命名覆盖全部展示面与代码接口，包括模式按钮、图表标题、X 轴标签、覆盖层提示、
i_xmode取值、data-testid、XAxisMode字面量（e2e-normalized-interactivity）及 API 字段（p75/p90_osl_per_e2el→p75/p90_e2e_norm_intvty，与既有*_intvty约定一致）。唯一刻意保留的是存储层键名
e2elPerOsl/e2el_per_osl：它们描述的是持久化的原始量（每输出 token 的端到端延迟，单位 s/tok），即该指标的倒数，读取路径会取倒数。若按指标名重命名，反而会与列中实际存储的内容不符。原因： 1/TPOT 可通过延迟 prefill 刷分，本指标对等待计入惩罚；端到端延迟分位数受 OSL 主导，除以 OSL 后既计入 prefill 又不受输出长度支配。
回填： v7 需要执行
db:backfill-aggregate-stats（profile-only 快速路径，约 20 分钟），行也会在读取时自愈。本次重命名未改动存储键名，因此不影响此前已执行的回填。基线： #644 已合并，
master已合入本分支（至2bfa3b60），无冲突，本 PR 的 diff 仅包含新指标。🤖 Generated with Claude Code
Note
Medium Risk
Touches default agentic chart behavior, new cached API + JSONL parsing, and STATS_VERSION/backfill; changes are broad but covered by unit and E2E tests.
Overview
Introduces E2E Normalized Interactivity (
e2e-normalized-interactivity) for agentic charts: slow-tail tok/s/user from per-request E2EL/OSL in stored aiperf traces (pXX = 1 / pXX(E2EL/OSL)), exposed viaGET /api/v1/derived-agentic-metricsand persisted ase2elPerOslinaggregate_statsv7 (STATS_VERSIONbump).Agentic UX defaults to this mode (leftmost tab);
InferenceContextwaits for sequence resolution before reconciling x-axis soi_xmodeis not overwritten when agentic loads. Unofficial-run overlays are hidden in this mode with a caption disclaimer.Frontier logic replaces the old e2e-latency Pareto restriction with a single (E2E Normalized Interactivity, y) winner set stamped on all agentic axes; rooflines use mirrored e2e corners for higher-is-better x.
e2eFrontier.tsis removed in favor ofcanonicalFrontier.Cypress gains
interceptDerivedAgenticMetrics; specs that need overlays explicitly switch to Interactivity.Reviewed by Cursor Bugbot for commit f3a9fe4. Bugbot is set up for automated code reviews on this repo. Configure here.