Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,15 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-

---

## [0.110.1] — 2026-07-09 — proposer portfolio export

### Added

- Exported the existing `compositeProposer` and `CompositeProposerOptions` from `@tangle-network/agent-eval/campaign`, so consumers can split one population budget across GEPA, SkillOpt, FAPO, memory, trace-analysis, or other proposers without copying the portfolio implementation.
- Added a packed-package import check for the export.

No proposer behavior changed.

## [0.108.1] — 2026-07-08 — public catalog docs patch

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion clients/python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "agent-eval-rpc"
version = "0.110.0"
version = "0.110.1"
description = "Python RPC client for @tangle-network/agent-eval — judge content against rubrics over HTTP or stdio RPC. Eval logic runs in the Node runtime; this package is a thin wire client."
readme = "README.md"
requires-python = ">=3.10"
Expand Down
2 changes: 1 addition & 1 deletion clients/python/src/agent_eval_rpc/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@
try:
__version__ = version("agent-eval-rpc")
except PackageNotFoundError:
__version__ = "0.110.0"
__version__ = "0.110.1"

__all__ = [
"Client",
Expand Down
2 changes: 1 addition & 1 deletion clients/python/uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 14 additions & 13 deletions docs/improvement-glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,25 +61,26 @@ Start at the top row and only move down when the row's *"reach for it when"* mat
| `gepaProposer` | You want the strong default: reflective full-surface prompt rewrites, grounded in findings, keeping a Pareto frontier of complementary winners. | prompt string | **Yes — `improve({ surface: 'prompt' })` default.** Proven live. |
| `skillOptProposer` | You are editing a structured `SKILL.md`/runbook and want small anchored add/delete/replace patches that preserve earlier rules. | skill/prompt string | **Yes — `improve({ surface: 'skills' })` default.** Not yet proven live. |
| `parameterSweepProposer` | The likely fix is a config knob, not words — `retrieval.k`, `temperature`, `max_tokens`. You give it candidate patches; it applies them to a JSON surface. | JSON config string | Yes, but you supply the candidate list. |
| `fapoProposer` | You want *evidence to decide when to escalate*: try prompt edits first, move to parameters, then to structural code — one scoped change per cycle, only escalating when the cheaper level is exhausted. **This is the composite/orchestration proposer** (there is no separate `compositeProposer`). | whatever its level proposers return | Exported; you wire the level proposers. |
| `fapoProposer` | You want *evidence to decide when to escalate*: try prompt edits first, move to parameters, then to structural code — one scoped change per cycle, only escalating when the cheaper level is exhausted. | whatever its level proposers return | Exported; you wire the level proposers. |
| `compositeProposer` | You want several proposers to share one candidate-generation budget in the same round. It allocates the population by declared weights, preserves member provenance, deduplicates surfaces, and isolates a member failure unless every member fails. | whatever its member proposers return | Exported; you wire the member proposers. |
| `aceProposer` | You are accumulating hard-won lessons into a playbook and must **never** summarize an old lesson away (append-only, provenance-tagged). | playbook string | Exported. |
| `memoryCurationProposer` | Same as ACE but you want a compact, deduped, re-ranked memory instead of append-only growth. | memory string | Exported. |
| `evolutionaryProposer` | You want blind population search (mutate → measure → select) with no reflection over findings — a cheap control or a baseline to beat. | any string | Exported. |
| `traceAnalystProposer` | Bench-only: race our trace-analysis evidence engine head-to-head inside `compareProposers`. | prompt string | Bench-only. |
| `haloProposer` | Bench-only: race the external `halo-engine` analysis against ours. | prompt string | Bench-only, external. |

Default path: `gepaProposer` for prompts; add `parameterSweepProposer` when a config knob is the suspect; wrap both in `fapoProposer` when you want the loop to decide *when* to escalate from words to knobs to code.
Default path: `gepaProposer` for prompts; add `parameterSweepProposer` when a config knob is the suspect; wrap levels in `fapoProposer` when the loop should decide *when* to escalate, or use `compositeProposer` when multiple proposer families must split one fixed population budget.

## Composing proposers — the only three ways, with runnable code
## Composing proposers — four distinct shapes

There is deliberately **no** `chainProposer`/`compositeProposer` primitive.
Composition happens in exactly three shapes:
Choose the shape that matches the experiment:

1. **Escalate** — `fapoProposer` wraps a prompt + parameter + structural proposer into one, spending the cheapest level first.
2. **Race** — `compareProposers` runs N proposers on one holdout and returns a per-proposer lift CI + pairwise "who won".
3. **Plug in** — hand any composed proposer to the gated loop via `runImprovementLoop({ proposer })`, or the one-liner `improve({ surface, generator })` in `@tangle-network/agent-runtime`.
1. **Portfolio** — `compositeProposer` splits one generation's population across member proposers by fixed weights and returns one provenance-labelled pool.
2. **Escalate** — `fapoProposer` wraps prompt + parameter + structural levels into one proposer and spends on the cheapest level until evidence says to escalate.
3. **Race** — `compareProposers` gives proposers separate loops, then re-scores their winners on one holdout and returns per-proposer lift intervals plus pairwise results.
4. **Plug in** — hand any proposer to `runImprovementLoop({ proposer })`, or use `improve({ surface, generator })` in `@tangle-network/agent-runtime`.

### 1 + 3 — compose by escalation, then run the gated loop
### 2 + 4 — compose by escalation, then run the improvement loop

```ts
import {
Expand All @@ -93,8 +94,8 @@ import {
const llm = { baseUrl: process.env.TANGLE_BASE_URL, apiKey: process.env.TANGLE_API_KEY }
const model = 'deepseek-v4-flash'

// Compose: prompt edits first (GEPA), escalate to a config knob only when
// prompt-level search plateaus. `fapoProposer` IS the composite proposer.
// Compose by escalation: prompt edits first (GEPA), then a config knob only
// when prompt-level search plateaus. This is distinct from a peer portfolio.
const proposer = fapoProposer({
scope: { allowedLevels: ['prompt', 'parameter'] }, // no structural/code tier here
promptProposer: gepaProposer({ llm, model, target: 'agent system prompt' }),
Expand Down Expand Up @@ -147,7 +148,7 @@ const out = await improve(profile, findings, {
if (out.shipped) deploy(out.profile) // out.lift is the held-out winner − baseline
```

### 2 — race proposers head-to-head for a lift CI
### 3 — race proposers head-to-head for a lift CI

```ts
import {
Expand Down Expand Up @@ -196,7 +197,7 @@ Every entrant is re-scored on the **same** holdout with the **same** judges, so
- Do not put eval logic inside a proposer — scoring lives in `dispatch` + `judges`, proposing lives in the proposer.
- Do not let a proposer read held-out judge scores — `ProposeContext` makes that a compile error on purpose; a proposer that games the acceptance axis is an oracle, not an optimizer.
- Do not read `lift` without `result.power`/MDE — a "+4" on a valset too small to detect +4 is noise wearing a number.
- Do not look for a `compositeProposer` `fapoProposer` is the composite; `compareProposers` is the race; `runImprovementLoop`/`improve` is the plug.
- Do not confuse `compositeProposer` with `fapoProposer`: the former allocates one fixed population across peers, while the latter escalates through ordered levels from cheaper to more structural changes.

### neutralizationGate — the placebo / content-causality control

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@tangle-network/agent-eval",
"version": "0.110.0",
"version": "0.110.1",
"description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
"homepage": "https://github.com/tangle-network/agent-eval#readme",
"repository": {
Expand Down
50 changes: 50 additions & 0 deletions scripts/verify-package-exports.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..')
const tempRoot = mkdtempSync(join(repoRoot, '.tmp-package-exports-'))

try {
verifyVersionLock()
const packDir = join(tempRoot, 'pack')
const unpackDir = join(tempRoot, 'unpack')
const appDir = join(tempRoot, 'app')
Expand Down Expand Up @@ -78,6 +79,10 @@ try {
if (!('analyzeBeliefPolicy' in beliefState)) {
throw new Error('missing belief-state export analyzeBeliefPolicy')
}
const campaign = await import('@tangle-network/agent-eval/campaign')
if (!('compositeProposer' in campaign)) {
throw new Error('missing campaign export compositeProposer')
}
`,
],
appDir,
Expand All @@ -86,6 +91,51 @@ try {
rmSync(tempRoot, { recursive: true, force: true })
}

function verifyVersionLock() {
const npmVersion = JSON.parse(readFileSync(join(repoRoot, 'package.json'), 'utf8')).version
const pyproject = readFileSync(join(repoRoot, 'clients/python/pyproject.toml'), 'utf8')
const pythonSource = readFileSync(
join(repoRoot, 'clients/python/src/agent_eval_rpc/__init__.py'),
'utf8',
)
const uvLock = readFileSync(join(repoRoot, 'clients/python/uv.lock'), 'utf8')
const pythonPackageVersion = matchVersion(
pyproject,
/^version\s*=\s*"([^"]+)"$/m,
'clients/python/pyproject.toml',
)
const pythonFallbackVersion = matchVersion(
pythonSource,
/except PackageNotFoundError:\s*\n\s*__version__ = "([^"]+)"/,
'clients/python/src/agent_eval_rpc/__init__.py',
)
const uvRootVersion = matchVersion(
uvLock,
/\[\[package\]\]\s*\nname = "agent-eval-rpc"\s*\nversion = "([^"]+)"/,
'clients/python/uv.lock',
)
const versions = {
npm: npmVersion,
pythonPackage: pythonPackageVersion,
pythonFallback: pythonFallbackVersion,
uvRoot: uvRootVersion,
}
const mismatched = Object.entries(versions).filter(([, version]) => version !== npmVersion)
if (mismatched.length > 0) {
throw new Error(
`release version mismatch: ${Object.entries(versions)
.map(([name, version]) => `${name}=${version}`)
.join(', ')}`,
)
}
}

function matchVersion(source, pattern, path) {
const match = source.match(pattern)
if (!match?.[1]) throw new Error(`could not read release version from ${path}`)
return match[1]
}

function run(command, args, cwd) {
const result = spawnSync(command, args, {
cwd,
Expand Down
4 changes: 4 additions & 0 deletions src/campaign/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,10 @@ export {
type SkillOptEpochRecord,
} from './presets/run-skill-opt'
export { type AceProposerOptions, aceProposer } from './proposers/ace'
export {
type CompositeProposerOptions,
compositeProposer,
} from './proposers/composite'
export { type EvolutionaryProposerOptions, evolutionaryProposer } from './proposers/evolutionary'
export {
extractFapoAttributionSignals,
Expand Down
30 changes: 26 additions & 4 deletions src/campaign/presets/run-optimization.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,12 @@ export interface RunOptimizationBaseOptions<TScenario extends Scenario, TArtifac
* generation's traces; findings-grounded proposers consume them. Opaque here;
* the proposer types its `TFindings`. Empty when no producer is wired. */
findings?: unknown[]
/** Per-generation findings producer. After each
* generation's candidates are scored, this is called with that generation's
* results; whatever it returns REPLACES `ctx.findings` for the NEXT
* generation's `propose()`, so the diagnosis is refreshed each round instead
/** Per-generation findings producer. Runs once on the BASELINE campaign
* (as `generation: -1`, the baseline convention) before generation 0
* proposes — so even a single-generation run proposes with trace context —
* and then after each generation's candidates are scored with that
* generation's results; whatever it returns REPLACES `ctx.findings` for the
* NEXT `propose()`, so the diagnosis is refreshed each round instead
* of being a static one-shot. Generic by design: the substrate does not
* import an analyst — the consumer plugs its trace-analyst registry / HALO
* here (reading the per-candidate `runDir` traces). When absent, findings
Expand Down Expand Up @@ -151,6 +153,26 @@ export async function runOptimization<TScenario extends Scenario, TArtifact>(
toParetoParent(opts.baselineSurface, winnerSurfaceHash, baselineCampaign, -1),
]

// Diagnose the BASELINE traces before generation 0 proposes. The
// between-generation producer call below only fires after gen g to feed gen
// g+1, so without this a single-generation run (maxGenerations = 1)
// proposes blind even though baseline traces exist. Baseline is
// `generation: -1` — the same convention the Pareto accumulator uses above.
// Skipped when the baseline produced no cells (dry/offline modes have no
// traces to analyze) or there is no generation 0 to feed; `propose()` then
// sees the static seed findings exactly as before.
if (opts.analyzeGeneration && opts.maxGenerations > 0 && baselineCampaign.cells.length > 0) {
const fresh = await opts.analyzeGeneration({
generation: -1,
runDir: `${opts.runDir}/baseline`,
candidates: [
{ surfaceHash: winnerSurfaceHash, campaign: baselineCampaign, composite: winnerComposite },
],
history,
})
if (Array.isArray(fresh)) currentFindings = fresh
}

for (let gen = 0; gen < opts.maxGenerations; gen++) {
// Decide: the proposer may stop early based on accumulated history.
if (proposer.decide?.({ history }).stop) break
Expand Down
11 changes: 6 additions & 5 deletions src/contract/self-improve.ts
Original file line number Diff line number Diff line change
Expand Up @@ -239,11 +239,12 @@ export interface SelfImproveOptions<TScenario extends Scenario, TArtifact> {
expectUsage?: 'assert' | 'warn' | 'off'

/**
* Per-generation findings producer. After each
* generation is scored, this is called; whatever it returns REPLACES the
* proposer's `findings` for the next generation's `propose()`. Plug a
* trace-analyst registry / HALO here. When absent, findings stay
* `opts.findings`.
* Per-generation findings producer. Runs once on the baseline campaign (as
* `generation: -1`) before generation 0 proposes — so single-generation runs
* propose with trace context — and again after each generation is scored;
* whatever it returns REPLACES the proposer's `findings` for the next
* `propose()`. Plug a trace-analyst registry / HALO here. When absent,
* findings stay `opts.findings`.
*/
analyzeGeneration?: RunOptimizationOptions<TScenario, TArtifact>['analyzeGeneration']

Expand Down
89 changes: 85 additions & 4 deletions tests/campaign/presets.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1414,12 +1414,93 @@ describe('runOptimization — analyzeGeneration feeds findings forward', () => {
})

expect(seen).toHaveLength(3)
// Gen 0 sees the static seed; gen 1 sees gen-0's produced finding; gen 2 gen-1's.
expect(seen[0]).toEqual([{ claim: 'seed' }])
// Gen 0 sees the baseline (gen -1) analysis — not the static seed; gen 1
// sees gen-0's produced finding; gen 2 gen-1's.
expect(seen[0]).toEqual([{ claim: 'gen--1 finding' }])
expect(seen[1]).toEqual([{ claim: 'gen-0 finding' }])
expect(seen[2]).toEqual([{ claim: 'gen-1 finding' }])
// Producer runs after gens 0 and 1, NOT the last (gen 2 has no next propose()).
expect(analyzed).toEqual([0, 1])
// Producer runs on the baseline (-1) and after gens 0 and 1, NOT the last
// (gen 2 has no next propose()).
expect(analyzed).toEqual([-1, 0, 1])
})

it("with generations=1, gen 0's propose() receives the baseline (gen -1) trace analysis", async () => {
const seen: unknown[][] = []
const proposer = {
kind: 'recorder',
async propose(ctx: ProposeContext) {
seen.push(ctx.findings)
return [{ surface: 'S-gen0', label: 'g0', rationale: 'r' }]
},
}
const result = await runOptimization<FakeScenario, FakeArtifact>({
scenarios: SCENARIOS,
baselineSurface: 'BASE',
dispatchWithSurface: async (surface) => ({ text: String(surface) }),
judges: [passJudge],
proposer,
populationSize: 1,
maxGenerations: 1,
promoteTopK: 1,
runDir,
seed: 1,
findings: [{ claim: 'seed' }],
analyzeGeneration: async ({ generation, runDir: analyzedDir, candidates, history }) => {
// The single propose() of the run is fed by a BASELINE analysis:
// generation -1 (the baseline convention), the baseline runDir, no
// history yet, and the baseline campaign as the sole candidate.
expect(generation).toBe(-1)
expect(analyzedDir).toBe(`${runDir}/baseline`)
expect(history).toEqual([])
expect(candidates).toHaveLength(1)
expect(candidates[0]!.surfaceHash).toBe(surfaceHash('BASE'))
// Marker derived from the baseline traces: the analyzed cells carry
// the artifact the BASELINE surface dispatched.
const artifacts = candidates[0]!.campaign.cells.map(
(c) => (c.artifact as FakeArtifact).text,
)
expect(artifacts).toEqual(['BASE', 'BASE'])
return [{ claim: 'baseline finding', from: artifacts.join(',') }]
},
})

// Gen 0 proposed WITH the baseline analysis (not the static seed), and the
// report is traceably derived from the baseline artifacts.
expect(seen).toEqual([[{ claim: 'baseline finding', from: 'BASE,BASE' }]])
expect(result.generations).toHaveLength(1)
})

it('skips the baseline analysis when the baseline produced no cells (nothing to analyze)', async () => {
const seen: unknown[][] = []
const analyzed: number[] = []
const proposer = {
kind: 'recorder',
async propose(ctx: ProposeContext) {
seen.push(ctx.findings)
return [{ surface: 'S-gen0', label: 'g0', rationale: 'r' }]
},
}
await runOptimization<FakeScenario, FakeArtifact>({
// No scenarios → the baseline campaign has zero cells (the dry shape) —
// the producer must NOT run on it, and propose() sees the static seed.
scenarios: [],
baselineSurface: 'BASE',
dispatchWithSurface: async (surface) => ({ text: String(surface) }),
judges: [passJudge],
proposer,
populationSize: 1,
maxGenerations: 1,
promoteTopK: 1,
runDir,
seed: 1,
findings: [{ claim: 'seed' }],
analyzeGeneration: async ({ generation }) => {
analyzed.push(generation)
return [{ claim: 'should not reach gen 0' }]
},
})
expect(analyzed).toEqual([])
expect(seen).toEqual([[{ claim: 'seed' }]])
})

it('without analyzeGeneration, findings stay the static seed every generation', async () => {
Expand Down
19 changes: 17 additions & 2 deletions tests/contract-self-improve.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -210,8 +210,6 @@ describe('selfImprove — forwarded loop knobs', () => {
let calls = 0
await selfImprove<S, A>({
...base,
// analyzeGeneration runs BETWEEN generations (its findings feed the next),
// so it needs ≥2 generations to fire at least once.
budget: { generations: 2, populationSize: 1 },
expectUsage: 'off',
analyzeGeneration: async () => {
Expand All @@ -221,4 +219,21 @@ describe('selfImprove — forwarded loop knobs', () => {
})
expect(calls).toBeGreaterThanOrEqual(1)
})

it('with generations=1, analyzeGeneration fires on the BASELINE (gen -1) so propose() is not blind', async () => {
const analyzed: number[] = []
await selfImprove<S, A>({
...base,
// Single generation — the common case. The producer analyzes the
// baseline traces first (gen -1) so gen 0 proposes with that report;
// the between-generation call is skipped (gen 0 is the last).
budget: { generations: 1, populationSize: 1 },
expectUsage: 'off',
analyzeGeneration: async ({ generation }) => {
analyzed.push(generation)
return [{ claim: 'baseline finding' }]
},
})
expect(analyzed).toEqual([-1])
})
})
Loading