diff --git a/README.md b/README.md index 76fe63e3..1223ec21 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,25 @@ # @tangle-network/agent-runtime -The engine Tangle's AI agents run on. It runs an agent — a **chat turn**, a **one-shot task**, or a **team of agents** working toward a goal — records every run, and uses those records to **measure and improve** agents against real pass/fail checks. +The engine Tangle's AI agents run on. It runs an agent as a **chat turn**, a **one-shot task**, or a **team of agents** working toward a goal, records every run, and uses those records to **measure and improve** agents against real pass/fail checks. -One loop, used three ways. Domain behavior (models, tools, knowledge) plugs in as adapters; the scoring statistics and the ship decision come from [`@tangle-network/agent-eval`](https://www.npmjs.com/package/@tangle-network/agent-eval); sandboxed execution from [`@tangle-network/sandbox`](https://www.npmjs.com/package/@tangle-network/sandbox). +One loop, used four common ways. Domain behavior (models, tools, knowledge) plugs in as adapters; the scoring statistics and the ship decision come from [`@tangle-network/agent-eval`](https://www.npmjs.com/package/@tangle-network/agent-eval); sandboxed execution from [`@tangle-network/sandbox`](https://www.npmjs.com/package/@tangle-network/sandbox). ```bash pnpm add @tangle-network/agent-runtime @tangle-network/agent-eval @tangle-network/sandbox ``` -**See it run in 30 seconds** (offline, no keys) — the one move everything else builds on, a driver reading a worker's output and composing the next step from it: +## Contents + +- [What you do with it](#what-you-do-with-it) +- [Run a chat turn](#run-a-chat-turn) +- [Supervise a team of agents](#supervise-a-team-of-agents) +- [Improve an agent](#improve-an-agent) +- [Improve a knowledge base](#improve-a-knowledge-base) +- [How it works](#how-it-works-the-short-version) +- [Examples](#examples) +- [Where to go next](#where-to-go-next) + +**See it run in 30 seconds** (offline, no keys): the one move everything else builds on, a driver reading a worker's output and composing the next step from it: ```bash pnpm tsx examples/driver-loop/driver-loop.ts @@ -18,9 +29,10 @@ pnpm tsx examples/driver-loop/driver-loop.ts | You want to… | Call | |---|---| -| Run a **chat turn** — what every product agent does in production | `handleChatTurn(...)` | +| Run a **chat turn** for a production product agent | `handleChatTurn(...)` | | Have one agent **supervise a team of agents** toward a goal | `supervise(profile, task, opts)` | | **Improve** an agent and prove the gain on fresh tasks | `improve(profile, findings, opts)` | +| **Improve** a knowledge base with agents, checks, and safe promotion | `runKnowledgeImprovementJob(...)` | ### Run a chat turn @@ -56,7 +68,7 @@ const result = await supervise( ### Improve an agent -`improve` optimizes one part of an agent (its prompt, skills, or code) and **only ships a change if it beats the current agent on tasks it never practiced on** — so registering an agent for self-improvement can never make it worse. +`improve` optimizes one part of an agent (its prompt, skills, or code) and **only ships a change if it beats the current agent on tasks it never practiced on**. Registering an agent for self-improvement cannot ship a worse candidate unless the caller supplies a bad measurement. ```ts import { improve } from '@tangle-network/agent-runtime' @@ -68,16 +80,36 @@ const { profile, shipped, lift } = await improve(baseProfile, findings, { }) ``` +### Improve a knowledge base + +`runKnowledgeImprovementJob` is the runtime-owned front door for KB, wiki, memory-backed, and RAG improvement jobs. It creates a candidate copy, runs supervised agents against it, checks readiness through `@tangle-network/agent-knowledge`, measures spend and timing, and promotes only when the candidate passes. + +```ts +import { runKnowledgeImprovementJob } from '@tangle-network/agent-runtime/knowledge' + +const result = await runKnowledgeImprovementJob({ + root: './kb', + goal: 'Improve support refund-policy knowledge', + readinessSpecs, + budget: { maxIterations: 8, maxTokens: 120_000, maxUsd: 10 }, + backend, +}) + +console.log(result.promoted, result.measurement.supervisedSpent) +``` + +Use it when the product needs one knob for "make this knowledge base better" instead of wiring `improveKnowledgeBase`, a runtime supervisor, candidate workspaces, readiness checks, and promotion tracking by hand. + ## How it works (the short version) -- **One agent, run two ways.** The same agent runs at "do the task" speed and at "get better at the task" speed. "Driver", "worker", and "coordinator" aren't separate types — they're roles one agent plays. -- **Everything is measured.** Every run is a trace: tokens, dollars, time, and a pass/fail score from a real check. "Better" is a number with a denominator, not a vibe — and "equally good but cheaper" is a result you can prove. -- **Improvement is gated.** A change ships only after it beats the current agent on fresh tasks no tuning step ever saw, with a statistical test — not a single lucky run. -- **The grader is honest.** Whatever gives feedback never sees the answer key, and scores are recomputed from the attempts actually run — an agent can't fabricate its own win. +- **One agent, run two ways.** The same agent runs at "do the task" speed and at "get better at the task" speed. "Driver", "worker", and "coordinator" are roles one agent plays, not separate types. +- **Everything is measured.** Every run is a trace: tokens, dollars, time, and a pass/fail score from a real check. "Better" is a number with a denominator, not a vibe, and "equally good but cheaper" is a result you can prove. +- **Improvement is gated.** A change ships only after it beats the current agent on fresh tasks no tuning step ever saw, with a statistical test, not a single lucky run. +- **The grader is honest.** Whatever gives feedback never sees the answer key, and scores are recomputed from the attempts actually run. An agent cannot fabricate its own win. ## Examples -Runnable, grouped by what they show — copy the one nearest your task: +Runnable, grouped by what they show. Copy the one nearest your task: | Do this | Example | |---|---| @@ -85,20 +117,21 @@ Runnable, grouped by what they show — copy the one nearest your task: | Drive a team of agents to a goal | [`supervise`](./examples/supervise) · [`recursive-supervisor`](./examples/recursive-supervisor) | | Benchmark strategies on your own domain | [`coding-benchmark`](./examples/coding-benchmark) | | Benchmark **harnesses × models** over a real task suite (the real WebCode dataset) | [`webcode-matrix`](./examples/webcode-matrix) | -| Render a **multi-profile leaderboard** — ranked board + score matrix + SVG/HTML charts, any domain | `leaderboard(records)` → `renderLeaderboardMarkdown` / `Svg` / `Html` | +| Render a **multi-profile leaderboard** with ranked board, score matrix, and SVG/HTML charts | `leaderboard(records)` → `renderLeaderboardMarkdown` / `Svg` / `Html` | | Trace + bill + effort-gate the WebCode benchmark (the Intelligence SDK) | [`intelligence-webcode`](./examples/intelligence-webcode) | | Self-improve an agent, gated on a held-out set | [`improve`](./examples/improve) · [`self-improving-coder`](./examples/self-improving-coder) | +| Improve a KB, wiki, or RAG corpus with runtime agents | [`docs/canonical-api.md`](./docs/canonical-api.md) | | Study coordination vs raw compute | [`ablation-suite`](./examples/ablation-suite) | -All 28 live in [`examples/`](./examples). +All 29 live in [`examples/`](./examples). ## Where to go next -- New here? [`docs/concepts.md`](./docs/concepts.md) — the mental model in plain terms. -- [`docs/canonical-api.md`](./docs/canonical-api.md) — find the primitive: "I want to ___ → use ___". -- [`docs/api/primitive-catalog.md`](./docs/api/primitive-catalog.md) — every export in one generated, never-stale list with its import path. Check it before building anything new. -- Import subpaths: the root export is the product surface (`handleChatTurn`, `improve`); deeper capabilities ship as subpaths — `/loops` (multi-agent + the loop kernel), `/mcp` (tool servers), `/intelligence` (observability drop-in), `/lifecycle`, `/agent`, `/profiles`, `/platform`, `/analyst-loop`, `/environment-provider`. -- [`docs/architecture.md`](./docs/architecture.md) — the design, end to end. -- [`bench/HARNESS.md`](./bench/HARNESS.md) — the experiment harness and how to run a benchmark. +- New here? [`docs/concepts.md`](./docs/concepts.md), the mental model in plain terms. +- [`docs/canonical-api.md`](./docs/canonical-api.md), find the primitive: "I want to ___ → use ___". +- [`docs/api/primitive-catalog.md`](./docs/api/primitive-catalog.md), every export in one generated, never-stale list with its import path. Check it before building anything new. +- Import subpaths: the root export is the product surface (`handleChatTurn`, `improve`); deeper capabilities ship as subpaths: `/loops` (multi-agent + the loop kernel), `/knowledge` (KB improvement), `/mcp` (tool servers), `/intelligence` (observability drop-in), `/lifecycle`, `/agent`, `/profiles`, `/platform`, `/analyst-loop`, `/environment-provider`. +- [`docs/architecture.md`](./docs/architecture.md), the design, end to end. +- [`bench/HARNESS.md`](./bench/HARNESS.md), the experiment harness and how to run a benchmark. **Contributing:** `pnpm i && pnpm test` gets you running; the full local gate is the [`package.json`](./package.json) scripts (`lint`, `typecheck`, `docs:check`). diff --git a/docs/canonical-api.md b/docs/canonical-api.md index 06b539fe..1149d8da 100644 --- a/docs/canonical-api.md +++ b/docs/canonical-api.md @@ -111,7 +111,7 @@ A general "loop" primitive is the single most common modelling error in this rep | Stand up a vertical agent in the eval loop | `defineAgent(manifest)` + `createSurfaceImprovementAdapter` — `/agent` | a per-vertical manifest parser, surface-validator, or bespoke `ImprovementAdapter` | | Turn intelligence/observation OFF (prove inference-only billing) | `withTangleIntelligence(agent, { effort: 'off' })` — `/intelligence` | a custom trace-wrapper or hand-rolled effort/tier config | | Fold **certified prompt additions into a system prompt you assemble yourself** (product chat routes) | `createCertifiedPromptSource({ target })` → `source.compose(base)` — `/intelligence` (cached, coalesced, fail-closed; `withCertifiedDelivery` rides the same source) | a module-scope cache + refresh-window + keep-last-known loop around `pullCertified` in product wiring | -| Improve a KB with runtime agents, candidate workspaces, readiness checks, and measured supervised spend | `runKnowledgeImprovementJob(options)` — `/knowledge` | hand-wiring `improveKnowledgeBase` + a supervised updater + a readiness callback in every product | +| Improve a KB with runtime agents, candidate workspaces, readiness checks, and measured supervised spend | `runKnowledgeImprovementJob(options)` from `/knowledge` | hand-wiring `improveKnowledgeBase` + a supervised updater + a readiness callback in every product | For the full export inventory (every primitive, its import path, its summary — generated, never stale), see `docs/api/primitive-catalog.md`; for per-symbol signatures, the per-module `docs/api/` pages. For the recursive atom (recursion · isolated-or-collaborative artifact · conserved budget · analysts) and the two-timescale architecture, see `docs/architecture.md`. For the genome→run→optimize→ship spine in depth, `docs/concepts.md` + `docs/learning-flywheel.md`. For the Intelligence SDK (Observe + the provable-OFF billing boundary), `docs/intelligence-sdk.md`.