Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,13 @@
.analysis-method summary { min-height: 44px; display: flex; align-items: center; cursor: pointer; color: var(--pw-text); }
.analysis-method dl { display: grid; grid-template-columns: 100px minmax(0, 1fr); gap: 8px 16px; }
.analysis-method dd { margin: 0; color: var(--pw-text); }
.analysis-continuation { margin-block: 16px; border-block: 1px solid var(--pw-line); padding-block: 12px; }
.analysis-continuation-summary { display: flex; align-items: center; flex-wrap: wrap; gap: 8px 16px; }
.analysis-continuation-summary strong { font-weight: 500; }
.analysis-continuation-summary > span { color: var(--pw-muted); }
.analysis-continuation-summary button { margin-inline-start: auto; }
.analysis-continuation .analysis-method { border: 0; margin: 0; padding: 0; }
.analysis-continuation .analysis-method > summary { display: list-item; list-style-position: inside; min-height: 36px; }
.analysis-spin { animation: analysis-request-spin 1s linear infinite; }
@keyframes analysis-request-spin { to { transform: rotate(360deg); } }
@media (prefers-reduced-motion: reduce) { .analysis-spin { animation: none; } }
Expand Down

Large diffs are not rendered by default.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions docs/development/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,11 @@ source, method and limitation paragraphs before adding another note.
every item. Use declared identities rather than extracting guessed labels
from prose. Keep unknowns, exclusions and denominators visible at the metric,
and preserve keyboard and touch access to the full evidence.
- When analysis has a successor, show one compact version/change summary and
the existing previous-version entry. Keep the new result separate from the
last valid result while pending or failed. Put detailed input differences and
changed judgments in the same optional disclosure; do not introduce another
setup page, repeated methodology paragraphs or automatic model execution.

For example, research metrics can show “19 episodes · purposive sample ·
uncalibrated” once, with a nearby explanation of yes/eligible counts and unknowns.
Expand Down
33 changes: 33 additions & 0 deletions docs/reference/trajectory-assessment.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,39 @@ Never impute missing dollars from an unverified model price.

## Producer and consumer boundary

### Continuing a frozen analysis

The trusted curator can register a complete successor snapshot using the
existing `benchmark trajectory-pack-register --base-pack-id <previous-pack>`.
The base must already belong to the same owner Goal and original Goal/case/run.
This records an immutable version link; it neither appends to the original
source nor starts a model. The Agent prepares this link with the input, rather
than asking the reader to enter internal identifiers.

The existing Goal analysis action becomes **Continue analysis**. Its inline
version summary identifies added, corrected and removed sources/episodes,
previous and current coverage, and changed judgments after completion. Previous
versions remain accessible through the same reader and browser history.
Reading, registering or refreshing a successor never starts analysis. A failed,
cancelled or concurrently running successor does not replace a valid base result.

The typed comparison reuses a qualified output only when literal excerpts,
source identities/paths/line positions, episode questions, eligibility,
objective, selection/window and analyst configuration are equivalent. Capture
revision/hash changes outside those excerpts do not themselves prove new
information. Explicit reuse requalifies every row against the current manifest
without starting a model; the original usage stays with its original job.
Missing or unreadable base evidence is not eligible for reuse.

All substantive additions, corrections, deletions or contract changes currently
trigger **full reassessment of the complete successor input**, including old
episodes that later evidence may affect. No heuristic certifies an unaffected
row and no prior model summary substitutes for original evidence. New metrics
are reduced from the complete new eligible pairs, including unknowns; old and
overlapping-window counts are never summed. This conservative continuation does
not claim tail-only execution, lower token cost or scientific uplift. Method or
configuration changes retain their original applicability boundaries.

Capability owner: existing `benchmark-toolkit`. Shared input validation, fixed
rubric, applicability identity and count reduction live in TypeScript at
`loopx/control_plane/capabilities/trajectory_assessment.ts`; browser-safe shapes
Expand Down
62 changes: 61 additions & 1 deletion examples/personal-workspace-browser/research-history.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ export const researchHistoryScenario = {
let analysisFixture = false, holdOldAnalysis = false, slowStarted, releaseOld, oldDelivered;
let holdCitation = false, failedCitation = false, citationStarted, releaseCitation;
let unavailableAnalysis = false, episodeMetadataLimit = null, comparisonFixture = false;
let analysisWrites = 0, localizedFixture = false;
let analysisWrites = 0, localizedFixture = false, continuationFixture = false, equivalentContinuation = false;
try {
context = await openWorkspacePage(browser, url, {collectCoverage, beforeGoto: async (_, page) => {
await page.route("**/api/chat/research-analysis**", async route => {
Expand All @@ -146,6 +146,22 @@ export const researchHistoryScenario = {
const selected = ["b", "c", "d"].find(id => query.get("pack_id") === id.repeat(64)) ?? "a";
if (holdOldAnalysis && selected === "a") {slowStarted(); await new Promise(resolve => {releaseOld = resolve;});}
const readback = comparisonFixture ? comparisonReadback() : analysisReadback(selected);
if (continuationFixture) {
const p = readback.pack, base = selected === 'b';
p.pack_id = (base ? 'b' : 'a').repeat(64);
p.analysis_key = (base ? '8' : '9').repeat(64);
p.scope = {...p.scope, revision: base ? 'original-capture' : 'later-evidence', window: base ? '09:00–12:00' : '09:00–14:00'};
p.result.scope = p.scope; p.result.analysis_key = p.analysis_key;
p.title = base ? 'Synthetic original analysis' : 'Synthetic analysis with later evidence';
if (!base) p.continuation = {
base_pack_id: 'b'.repeat(64), base_analysis_key: '8'.repeat(64), base_title: 'Synthetic original analysis',
base_scope: {...p.scope, revision: 'original-capture', window: '09:00–12:00'}, current_scope: p.scope,
sources: {added: ['follow-up'], revised: [], removed: []}, episodes: {added: [], revised: ['variant-alpha'], removed: []},
can_reuse: false, configuration_changed: false, objective_changed: false, base_available: true, base_episode_count: 4,
reassessed_episode_ids: p.episodes.map(e => e.id), new_pair_count: 0, removed_pair_count: 0,
verdict_changes: [{episode_id: 'variant-alpha', metric: 'benefit_test', before: 'yes', after: 'unknown'}],
};
}
for (const pack of [...readback.catalog.packs, readback.pack]) pack.config = {...pack.config, output_language: language};
if (readback.pack.result) {
readback.pack.result.output_language = language;
Expand All @@ -154,11 +170,20 @@ export const researchHistoryScenario = {
question: language === 'zh' ? `${episode.id} 是否检验原始目标?` : `Did ${episode.id} test the original objective?`}));
for (const assessment of readback.pack.result.assessments) assessment.reason = language === 'zh'
? '来源只记录了实验结果,缺少干预与目标瓶颈的联系。' : 'The source records the result but not the intervention’s link to the objective bottleneck.';
if (continuationFixture) for (const row of readback.pack.result.assessments) {
if (row.metric === 'benefit_test') row.reason = language === 'zh' ? '后续对照未覆盖原始任务收益,判断保留未知。' : 'The later control does not cover original-task utility; the judgment remains unknown.';
if (row.metric === 'evidence_adoption') row.reason = language === 'zh' ? '后续选择记录未标明采用版本,尚不能建立采用关系。' : 'The later selection does not identify an adopted revision, so adoption remains unestablished.';
}
readback.pack.result.limitations = [language === 'zh' ? '合成展示;不是科学校准结果。' : 'Synthetic presentation; not scientific calibration.'];
}
}
if (episodeMetadataLimit === 0) delete readback.pack.episodes;
else if (episodeMetadataLimit !== null) readback.pack.episodes = readback.pack.episodes.slice(-episodeMetadataLimit);
if (equivalentContinuation && selected === 'a') {
Object.assign(readback.pack.continuation, {can_reuse: true,
sources: {added: [], revised: [], removed: []}, episodes: {added: [], revised: [], removed: []}, reassessed_episode_ids: [], verdict_changes: []});
readback.pack.result = null; readback.pack.job = null;
}
await route.fulfill(unavailableAnalysis && selected === "a"
? {status: 503, json: {error: "Synthetic analysis read unavailable"}}
: {json: readback});
Expand Down Expand Up @@ -758,6 +783,41 @@ Path(sys.argv[1]).write_text(json.dumps({'goal_id':goal,'display_name':'Syntheti
await page.evaluate(key => localStorage.setItem(key, 'zh-CN'), 'loopx-pw-locale');
await page.reload({waitUntil: 'networkidle'});
await chineseAnalysis.locator('.analysis-episode h4').first().filter({hasText: 'variant-alpha 是否检验原始目标?'}).waitFor();
continuationFixture = true;
await page.reload({waitUntil: 'networkidle'});
// The inline version summary is a semantic section, not another page.
const version = chineseAnalysis.locator('.analysis-continuation');
await version.getByText('接续上一版', {exact: true}).waitFor();
await version.getByText(/已复核全部 4 片段/).waitFor();
assert.equal(await version.locator('details').getAttribute('open'), null);
await version.locator('summary').click();
assert.match(await version.innerText(), /original-capture → later-evidence/);
assert.match(await version.innerText(), /variant-alpha.*已做收益检验.*证据不足/);
await version.locator('summary').click();
await page.setViewportSize({width: 1440, height: 1000});
await chineseAnalysis.evaluate(element => element.scrollIntoView({block: 'start'}));
await page.screenshot({path: resolve(outputDir, 'research-analysis-continuation-desktop.png'), animations: 'disabled'});
const previousVersion = version.getByRole('button', {name: '查看上一版', exact: true});
await previousVersion.focus(); await previousVersion.press('Enter');
await chineseAnalysis.getByText('Synthetic original analysis', {exact: true}).waitFor();
assert.equal(await chineseAnalysis.locator('.analysis-continuation').count(), 0);
await page.goBack({waitUntil: 'networkidle'});
await version.getByText('接续上一版', {exact: true}).waitFor();
await page.reload({waitUntil: 'networkidle'});
await version.getByText('接续上一版', {exact: true}).waitFor();
await page.setViewportSize({width: 390, height: 844});
await chineseAnalysis.evaluate(element => element.scrollIntoView({block: 'start'}));
assert(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));
await page.screenshot({path: resolve(outputDir, 'research-analysis-continuation-mobile.png'), animations: 'disabled'});
await page.evaluate(key => localStorage.setItem(key, 'en'), 'loopx-pw-locale');
await page.reload({waitUntil: 'networkidle'});
await page.getByRole('button', {name: 'Previous version', exact: true}).waitFor();
equivalentContinuation = true;
await page.reload({waitUntil: 'networkidle'});
await page.getByRole('button', {name: 'Reuse analysis', exact: true}).first().waitFor();
await page.getByRole('button', {name: 'About deep analysis', exact: true}).click();
await page.getByText('This action reuses existing judgments without a model call; original usage stays with the previous version.', {exact: true}).waitFor();
await page.keyboard.press('Escape');
assert.equal(analysisWrites, 0, "Cached, reload, switch and keyboard reading never request execution");
assert.equal(api.turnRequests.length, 0, "Reading the graph cannot start an Agent");
return {note: "Three unchanged real checkpoint histories through production HTTP + packaged UI; per-Goal complete pagination, genuine negative evidence, no invented edges, reload/back/forward, mobile and failed refresh. Separate synthetic coincident-target regression; no benchmark qualification or model dispatch.",
Expand Down
27 changes: 27 additions & 0 deletions loopx/capabilities/benchmark_toolkit/trajectory_assessment.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ def execute(
retry: bool = False,
timeout_seconds: int = 900,
codex: str = "codex",
reuse_from: dict[str, Any] | None = None,
) -> dict[str, Any]:
# Re-derive the request at the I/O boundary; do not trust caller-supplied
# paths, prompts or schemas hidden in an otherwise valid prepared object.
Expand All @@ -84,6 +85,12 @@ def execute(
if prepared["analysis_key"] != verified["analysis_key"]:
raise ValueError("Analysis input/config key mismatch")
prepared = verified
if reuse_from is not None:
comparison = effect_runtime_result(
"benchmark.trajectory.continuation", {**prepared, "previous": reuse_from}
)
if not comparison["can_reuse"]:
raise ValueError("Changed evidence requires full reassessment")
directory = output_root.resolve() / prepared["analysis_key"]
directory.mkdir(parents=True, exist_ok=True, mode=0o700)
lock = directory / "execution.lock"
Expand Down Expand Up @@ -199,6 +206,26 @@ def execute(
"reused": False,
}
_write(directory / "receipt.json", receipt)
if reuse_from is not None:
# Same declared evidence, eligibility, scope and model contract.
# Requalify against the new exact manifest; never splice labels or
# add old metric totals to new ones. This path launches no child.
output = {key: reuse_from["output"][key] for key in
("assessments", "limitations", "episode_questions")
if key in reuse_from["output"]}
result = effect_runtime_result(
"benchmark.trajectory.qualify", {**prepared, "output": output}
)
_write(directory / "model-output.json", output)
_write(directory / "assessment.json", result)
receipt.update(
status="succeeded", ok=True, reused=True,
reused_from_analysis_key=reuse_from["analysis_key"],
analysis_wall_seconds=time.monotonic() - started,
**_result_proof(directory, prepared),
)
_write(directory / "receipt.json", receipt)
return receipt
with (
(directory / "prompt.txt").open(encoding="utf-8") as prompt,
(directory / "events.jsonl").open("w", encoding="utf-8") as events,
Expand Down
Loading
Loading