dynamics

[pelikhan]

Bringing Dynamic Workflows to githubnext/rig

An implementation plan for adding Claude Agent SDK–style dynamic workflow orchestration to the rig harness

Date: 2026-07-28 · Target repo: githubnext/rig ("Harness as a Skill") · Reference feature: Claude Code dynamic workflows / Agent SDK Workflow tool

---

1. Research summary

1.1 What dynamic workflows are (Claude Code / Agent SDK)

A dynamic workflow is a plain JavaScript orchestration script that a runtime executes in an isolated background environment, spawning and coordinating many subagents. The defining property is that the script holds the plan: loops, branching, fan-out, and intermediate results live in ordinary script variables rather than in a model's context window. Only the final synthesized result returns to the conversation. This moves multi-agent coordination from model improvisation to deterministic code, which makes it repeatable, reviewable, diffable, and resumable.

The script surface is intentionally tiny. A saved workflow exports a meta block (name, description, and a phases list shown at approval time) followed by a body written in plain JavaScript with top-level await. The body has no imports and no direct filesystem or shell access; agents do all the reading, writing, and command execution, while the script only coordinates them. Input arrives through a global args value passed at invocation time. The body's return value becomes the run's result.

The runtime exposes a handful of primitives:

agent(prompt, opts) spawns one subagent and resolves with its result. Options include label and phase (for the progress UI), model and effort (per-call routing, e.g. a cheap fast model for mechanical listing work), isolation: "worktree" (each agent edits an isolated copy so parallel file writes don't collide), and schema (a JSON Schema that forces the subagent into a structured-output tool call with automatic retry on mismatch, so orchestration logic consumes validated JSON instead of parsing prose). A failed or skipped agent resolves to null rather than rejecting, so one dead worker never takes down the whole run — the script guards with if (!result) ....

pipeline(items, fn) runs fn once per item with streaming semantics: item N+1 enters a stage as soon as item N leaves it, with no barrier between stages. parallel(thunks) is the explicit barrier form — it runs thunks concurrently and waits for all of them; throwing thunks resolve to null. phase(name) and log(msg) structure the progress view.

The runtime adds the operational layer that makes this safe at scale: a concurrency cap (up to 16 concurrent agents, fewer on small machines), a hard cap of 1,000 agents per run, per-agent result caching that makes a stopped run resumable (completed agents return cached results; in-flight agents restart), live progress reporting (per-phase agent counts, token totals, elapsed time, drill-down into any agent's prompt, tool calls, and result), a "large workflow" advisory warning when a run schedules more than ~25 agents or its projected token total passes 1.5M, and a size guideline setting that biases how large a generated workflow should be. Every run's script is written to disk so it can be read, diffed, edited, and relaunched, and a good run can be saved as a reusable slash command that accepts structured args.

The quality patterns this enables — not just scale — are the interesting part: fan-out → reduce → synthesize, adversarial cross-verification of findings before reporting, loop-until-a-check-passes with a no-progress stop condition, and drafting a plan from several independent angles before committing.

1.2 What rig is today

rig is a minimal TypeScript agent harness designed to run inside sandboxed agentic workflows embedded in markdown (the gh-aw / "Continuous AI" family). Its core API is agent(spec) — which creates a typed agent function with instructions, an optional input schema, and an optional output schema built from declarative s.* helpers that serialize directly to JSON Schema. Prompt intents (p.read, p.bash, p.bashEach, p.write, p.writeOutput, p.glob, p.env, ...) declaratively bind workspace context into prompts. defineTool registers SDK-neutral custom tools. A per-turn addons middleware chain (express-style (context, next)) provides timeout, steering, and repair (retry on schema mismatch), plus oncePerAgent for adapter-level registration.

Crucially, rig is engine-agnostic: it runs against a minimal Agent interface (ask(prompt, {signal}) / close()), with configureAgent(factory) selecting an adapter — Copilot SDK (default), Anthropic, Codex, Gemini CLI, or pi-agent. Programs are TypeScript files (or inline stdin programs, or ```rig fences in markdown) executed by the skills/rig/rig.ts launcher, with --typecheck, --server, stdin input coercion, and RIG_DEBUG JSONL diagnostics.

1.3 Gap analysis

Rig already covers the worker half of the dynamic-workflow model remarkably well — arguably better in some respects (typed I/O end to end, engine adapters, tool registration, addon middleware). What it lacks is the orchestration and operations half:

Capability	Claude dynamic workflows	rig today
Single typed subagent call with schema + retry	agent(prompt, {schema}), auto retry	✅ agent(spec) + repair() addon
Streaming per-item fan-out	pipeline(items, fn)	❌ users hand-roll Promise.all
Concurrent barrier with null-on-failure	parallel(thunks)	❌ (Promise.all rejects on first failure)
Bounded concurrency (16-agent cap)	runtime-enforced	❌ unbounded
Total agent cap (runaway protection)	1,000/run	❌
Failure semantics: dead worker → null, run continues	built in	❌ exceptions propagate
Phases + labels + live progress (counts, tokens, elapsed)	/workflows view	❌ only RIG_DEBUG JSONL
Result cache → pause/resume within a run	per-agent journal	❌
meta block + saved, parameterized workflows (args)	.claude/workflows/	partial (markdown fences, stdin input)
Per-stage model/effort routing	model/effort per agent() call	✅ per-call model override exists
Workspace isolation for parallel edits	isolation: "worktree"	❌
Cost visibility / large-run warning	token totals + advisory threshold	❌ (engine-dependent)

The design consequence: rig should not clone the Claude script surface; it should add an orchestration layer on top of its existing typed agent() primitive, keeping rig's differentiators (typed schemas, engine adapters, addons, markdown embedding) intact.

---

2. Design goals and non-goals

Goals. Deterministic orchestration as code, with the run's plan living in the rig program rather than any model's context. Fan-out/reduce/synthesize, verification loops, and until-converged loops expressible in a few lines. Bounded resource use (concurrency + total-agent caps) safe for CI runners. Structured failure semantics where one failed agent yields null and the run continues. Journaled runs that are observable while running and resumable after interruption. Engine-agnostic operation across all five adapters, degrading gracefully where an engine can't report usage. Parameterized, saved workflows that slot into the existing markdown-fence and gh-aw distribution story.

Non-goals (v1). No interactive TUI like /workflows (rig runs headless in CI; observability is structured events + a renderer hook). No "Claude writes the script for you" planning step — in the gh-aw world the workflow author (human or an authoring agent) writes the rig program; rig provides the runtime. No cross-process resume beyond what the journal file gives for free. No git-worktree isolation in v1 (deferred; see §5, M5).

---

3. Proposed API

Everything below lives in a new module, rig/workflow, re-exported from the root. All primitives are plain functions operating on rig's existing typed agent functions, so TypeScript inference flows through unchanged.

3.1 The core: workflow(), run(), and WorkflowContext

import { agent, s, p } from "rig";
import { workflow } from "rig/workflow";

const listRoutes = agent({
  model: "nano",
  instructions: p`List every .ts file under src/routes/ using ${p.bash("find src/routes -name '*.ts'")}. Return only files that exist.`,
  output: s.object({ files: s.array(s.path) }),
});

const auditRoute = agent({
  input: s.object({ file: s.path }),
  instructions: p`Audit ${p.readInput("file")} for missing authentication checks.`,
  output: s.object({ findings: s.array(s.string), severity: s.enum("none", "low", "high") }),
});

export default workflow({
  meta: {
    name: "audit-routes",
    description: "Audit every route handler for missing auth checks",
    phases: ["Collect", "Audit", "Verify"],
  },
  input: s.object({ root: s.optional(s.path) }),
  async body({ phase, pipeline, call, log, input }) {
    phase("Collect");
    const found = await call(listRoutes, {});
    if (!found) return { findings: [], note: "collection failed" };

    phase("Audit");
    const audits = await pipeline(found.files, (file) =>
      call(auditRoute, { file }, { label: file })
    );

    phase("Verify");
    const confirmed = audits.filter((a) => a && a.severity !== "none");
    log(`Confirmed ${confirmed.length} findings`);
    return confirmed;
  },
});

Design decisions, and why:

Primitives arrive via a context object, not globals. Claude's runtime injects agent, pipeline, phase as globals into a sandboxed script; rig programs are ordinary typed TypeScript modules, so the workflow body receives a WorkflowContext destructured in the signature. This keeps everything importable, testable, and mockable, and gives the runtime a natural place to hang the run journal, the concurrency limiter, and cancellation.

call(agentFn, input, opts?) instead of a stringly agent(prompt). Rig's whole value proposition is typed agents; the workflow layer's spawn primitive takes an existing typed agent function plus typed input, and its return type is Awaited<ReturnType> | null. Options: label (progress display), model / maxTurns / timeout / effort (per-call overrides, forwarded to rig's existing override path), and cacheKey (see §3.4). For quick one-off prompts, call.text(prompt, opts?) accepts a string or p builder and returns string | null, matching the Claude ergonomic where a schema is overkill.

Null-on-failure is the contract. Inside body, call, pipeline, and parallel never reject for agent-level failures (engine error, exhausted repair retries, timeout, per-agent abort). They resolve to null and record the failure in the journal. Programming errors in the body itself (a TypeError in your reduce step) still throw and fail the run — silently swallowing those would make workflows undebuggable.

3.2 Fan-out primitives

pipeline<T, R>(items: readonly T[], fn: (item: T, i: number) => Promise<R>): Promise<(R | null)[]>
parallel<R>(thunks: (() => Promise<R>)[]): Promise<(R | null)[]>
until<S>(opts: { max: number; noProgressRounds?: number },
         step: (state: S | undefined, round: number) => Promise<{ state: S; done?: boolean; progressKey?: string }>): Promise<S>

pipeline is the workhorse: it schedules fn per item through the shared concurrency limiter with streaming semantics (no barrier between items — the moment a slot frees, the next item starts). parallel is the explicit barrier for stages that need all prior results at once. Results preserve input order; failures are null holes.

until codifies the third recurring dynamic-workflow shape — "keep fixing until the check passes or two rounds in a row make no progress" — because in practice this loop is easy to get subtly wrong (unbounded loops, missing convergence detection). progressKey is any string summarizing the round's outcome; when it repeats noProgressRounds times, the loop stops. This is a small utility, not a framework: users can always write a plain for loop.

3.3 The runtime: limits and configuration

import { runWorkflow } from "rig/workflow";

const result = await runWorkflow(myWorkflow, {
  args: { root: "src/routes" },
  limits: { concurrency: 8, maxAgents: 500, maxWallMs: 45 * 60_000 },
  journal: ".rig/runs/audit-routes",      // path or a RunStore implementation
  onEvent: (e) => process.stderr.write(JSON.stringify(e) + "\n"),
  signal: abortController.signal,
});

Defaults mirror the Claude runtime where sensible: concurrency defaults to min(16, max(2, cpuCount)); maxAgents defaults to 1,000 and, when exceeded, fails the run with a clear error naming the cap (runaway-loop protection). An advisory onEvent warning fires when scheduled agents pass a warnAgents threshold (default 25) — advisory, exactly like Claude's "Large workflow" flag, because CI budgets differ wildly. maxWallMs matters more for rig than for Claude Code because gh-aw runs execute inside GitHub Actions job timeouts.

The limiter is one shared semaphore for the whole run, not per-primitive, so nested pipeline-inside-pipeline can't multiply concurrency. To prevent the classic deadlock (a parent task holding a slot while awaiting children that need slots), only agent invocations acquire slots — pipeline/parallel bookkeeping does not.

3.4 Journal, caching, and resume

Every call is journaled to an append-only JSONL file (or a pluggable RunStore with load/append, mirroring the Agent SDK's SessionStore idea so gh-aw can later persist journals as workflow artifacts):

{"t":"agent_start","id":7,"key":"a41f…","phase":"Audit","label":"src/routes/user.ts","model":"small","ts":…}
{"t":"agent_done","id":7,"key":"a41f…","ok":true,"output":{…},"turns":2,"ms":48211,"usage":{"in":18234,"out":912}}

The cache key is a content hash of (agent spec fingerprint, rendered input, per-call overrides); cacheKey on call lets the author extend it when the same input should legitimately rerun (e.g. include a round number). On a rerun with the same journal path, runWorkflow replays: a call whose key has an agent_done record returns the cached output instantly; everything else runs live. This yields exactly Claude's resume semantics — completed agents are preserved, in-flight agents restart — and it works across process restarts for free, which Claude's same-session-only resume does not. Because replay is key-based rather than sequence-based, an edited script still reuses every result whose inputs didn't change, which makes iterate-on-the-orchestration cheap.

Determinism caveat, stated honestly in docs: agents are not deterministic, so replay guarantees reuse, not reproducibility. The journal also stores the workflow's meta and an args snapshot so a resumed run can refuse to reuse a journal created with different args (override with --force-resume).

3.5 Observability events

onEvent receives a discriminated union — run_start, phase_start, agent_start, agent_progress (forwarded turn count / partial info where the engine exposes it), agent_done, agent_failed, log, warning (large-run advisory, cap approach), run_done — each carrying phase, label, ids, timing, and usage where available. Two renderers ship in v1: a JSONL passthrough (the default; composes with RIG_DEBUG conventions and is what gh-aw log viewers will parse) and a minimal TTY line renderer (--progress on the launcher) showing per-phase counts and elapsed time. Token usage is best-effort per engine: the Anthropic and Copilot adapters can report it; Gemini-CLI/Codex adapters emit usage: null and the docs say so.

3.6 Launcher and markdown integration

The skills/rig/rig.ts launcher learns to recognize a default-exported workflow (in file, stdin, and ```rig fence modes) and route it through runWorkflow, with flags --journal <dir>, --resume, --concurrency N, --max-agents N, --progress, and args coerced from stdin exactly as today (string → input, object → schema-validated). --typecheck continues to work unchanged since workflows are ordinary modules. A --plan flag prints meta (name, description, phases) and exits — the headless analogue of Claude's approval screen, useful for gh-aw compile-time validation and PR review of workflow changes.

---

4. Architecture notes

Where code lives. src/workflow/ with context.ts (WorkflowContext factory), limiter.ts (semaphore), primitives.ts (pipeline, parallel, until), journal.ts (RunStore + JSONL impl + key hashing), events.ts, run.ts (runWorkflow), and renderers/. No changes to the Agent engine interface are required for M1–M3; the workflow layer sits entirely above agent(spec)'s existing invoke path. The one internal change: the agent invocation path must expose a structured completion record (ok/error, turns, timing, usage-if-available) to the caller instead of only debug logs — a small refactor that also benefits non-workflow users.

Cancellation. runWorkflow's signal fans out to every in-flight call via rig's existing per-call signal support; on abort, in-flight agents are closed, agent_failed{reason:"aborted"} is journaled (not cached as done), and run_done{status:"aborted"} fires. This is what makes pause/resume coherent.

Failure taxonomy. Distinguish agent_failed (→ null, run continues), run_failed (body threw, cap exceeded, wall clock exceeded), and journal_error (surfaced as warning; run continues without caching rather than dying — durability should never be the reason a 200-agent run is lost).

Engine parity. All primitives must behave identically across adapters; the test matrix (see §6) runs the workflow suite against a scripted fake engine plus at least the Anthropic and Copilot adapters. Adapters that reject rig tools (Codex, Gemini) impose no new constraint here since workflows don't require tools.

---

5. Milestones

M1 — Primitives and limits (the core feature). workflow(), runWorkflow(), call/call.text, pipeline, parallel, shared limiter, null-on-failure semantics, maxAgents/maxWallMs caps, phase/log as no-op-safe recorders. Internal refactor of the agent invoke path to return a structured completion record. Exit criteria: the three canonical shapes (fan-out audit, review-and-merge, research-and-synthesize) run green against the fake engine and one real adapter; concurrency verified never to exceed the cap under nested fan-out.

M2 — Journal and resume. JSONL RunStore, key hashing, replay-on-resume, args-snapshot guard, launcher --journal/--resume. Exit criteria: kill a 30-agent run mid-flight, resume, and observe only unfinished agents rerun; edit one agent's instructions and observe only its calls (and dependents) rerun.

M3 — Observability. Full event union, JSONL renderer, TTY --progress renderer, large-run advisory, per-engine usage where available. Exit criteria: a gh-aw Actions log of a workflow run is parseable into per-phase counts and totals by a 20-line script.

M4 — Packaging for the gh-aw world. meta handling in all launcher modes, --plan, ```rig fence workflows in markdown docs, until helper, sample programs (samples/6x-workflow-*.md: route audit, fix-until-green, deep-research-lite with cross-checking), README section, and a comparison doc mirroring the existing GenAIScript comparison. Exit criteria: a saved workflow file runs identically via file, stdin, and fence extraction; samples double as smoke tests.

M5 — Deferred hardening (post-v1). Workspace isolation for parallel writers (per-agent temp checkout or git worktree, opt-in isolation option on call); budget limits in USD where the engine prices are known; journal-as-artifact helpers for gh-aw; a verify() combinator packaging the adversarial cross-check pattern (N findings → M independent verifier agents → vote) once M1–M4 usage shows the pattern's real shape.

Sequencing rationale: M1 alone delivers the headline capability (deterministic scale with bounded resources); M2 is what makes large runs economically viable to iterate on; M3 is what makes them operable in CI; M4 is distribution. Isolation is deferred because most first-wave workflows are read-heavy (audits, research, review), and worktree semantics interact with gh-aw's sandbox in ways worth learning from real usage first.

---

6. Testing strategy

Unit tests (vitest, matching the repo's existing setup) cover the limiter under nested fan-out and cancellation, null-hole ordering in pipeline/parallel, until convergence and no-progress detection, key hashing stability across serialization order, and journal replay including the corrupted-line and args-mismatch paths. Integration tests run each sample workflow against a deterministic scripted engine (a configureAgent factory returning canned responses keyed by prompt hash — no network) and assert final results, event sequences, and journal contents; a small live-adapter smoke suite runs behind an env-gated flag. Property-style tests fuzz interleavings by randomizing the fake engine's latencies and asserting invariants: concurrency never exceeds the cap, every agent_start has a matching terminal event, resumed runs never re-execute a journaled-done key.

---

7. Risks and open questions

The main design risk is cache-key fidelity: if the agent-spec fingerprint misses something semantically significant (an addon that changes behavior, an engine switch), resume can silently reuse stale results — mitigated by including the engine identity and an addon-chain fingerprint in the key, and by making --resume opt-in rather than default. Second, usage reporting is uneven across engines, so cost features must degrade to "unknown" without breaking the advisory logic. Third, deep-copy semantics of cached outputs: journal replay returns parsed JSON, so outputs must be JSON-serializable — already true for schema'd agents, but call.text callers embedding class instances would break; the docs and types should enforce plain data. Open questions worth settling with maintainers before M1: whether workflow() should be the only entry point or whether a bare async-function-with-context export is also accepted in fence mode (lower ceremony for markdown embedding); whether journals belong under .rig/ or gh-aw's existing state directory conventions; and whether the phases list in meta should be validated against phase() calls at --typecheck/--plan time (cheap to do, catches drift, mirrors how Claude's approval screen derives its phase list).

---

Appendix: source map

Claude side — Dynamic workflows guide (script shape, meta, agent()/pipeline(), limits, resume, cost, size guidelines), Agent SDK TypeScript reference (AgentDefinition, options, message/event model, SessionStore precedent), plus community deep-dives documenting parallel() barrier semantics, null-on-failure, schema-retry, and pipeline-vs-parallel guidance (alexop.dev, developersdigest.tech, buildthisnow.com, claude-world.com). Rig side — githubnext/rig README: core API (agent, s.*, p.*, defineTool, addons, configureAgent), engine adapters and their constraints, launcher modes (file/stdin/fence, --typecheck, --server), debug-logging conventions, and the samples/docs layout this plan extends.