pipellm

Run pipelines of CLI agents (Claude Code, opencode) defined in YAML.

Install

pipx install -e .

Usage

pipellm check article.yaml
pipellm start article.yaml --var topic="sqlite WAL mode" > article.md
pipellm start article.yaml --vars-file topics.yaml
pipellm start notes.yaml --var doc=@notes.txt
pipellm start article.yaml --var topic="sqlite WAL mode" --parallel-limit 2
pipellm start article.yaml --var topic="sqlite WAL mode" --sequential
pipellm start article.yaml --var topic="sqlite WAL mode" --ignore-usage
pipellm start article.yaml --resume runs/article-20261027-173208
pipellm serve

start prints progress to stderr (job states, agent tool calls, a one-line preview of each output) and the final output to stdout. Every run is saved to runs/<pipeline>-<timestamp>/ (vars.json, jobs/<name>.txt|json, output.txt|json); jobs run again by back are saved as jobs/<name>.2.txt|json and so on.

Agents run without permission prompts (claude --permission-mode bypassPermissions, opencode --auto), so they use their tools without asking and Bash can run any command.

--vars-file is a YAML list of variable sets; the pipeline runs once per set.

--parallel-limit N runs at most N agents at once in the whole run, foreach items and child pipelines included, whatever their concurrency, so many agents don't hit the model or MCP servers at once. --sequential is --parallel-limit 1: foreach items run one at a time, in order.

Before starting a claude agent, pipellm checks the Claude subscription usage (what /usage in Claude Code shows), at most once a minute. At 97 % of the 5-hour window or 99 % of the weekly window it starts no more agents and prints waiting until 02:34 5h limit reset; the agents already running finish, and the run goes on after the reset. The process keeps running while it waits, and ctrl+c stops it as usual. The usage is read with the Claude Code login (~/.claude/.credentials.json, or the keychain on macOS); when that fails, for example with an API key, it prints usage limits not checked once and runs on. --ignore-usage skips the check.

Press ctrl+c to stop a running pipeline. The first press only asks; the second kills the running agents and ends the run. Jobs that finished stay saved in the run folder, and so do the finished items of a foreach job, each saved as soon as it is done; only the agents still running lose their progress. pipellm then prints the --resume command to continue. A hangup or kill stops the run at once, the same way.

--resume <rundir> continues a stopped or failed run instead of starting over. It retraces the run, reusing the saved output of every job it reaches, and runs agents again from the first job that has no saved output, so a run that died in job 5 re-runs from job 5, not job 1. If that job is a foreach job, its finished items are reused too and only the rest run again. back loops and next branches take the same path as before, because they are decided by the reused outputs. Variables come from the run's vars.json (--var overrides them), and new outputs are written into the same folder. --from <job> runs again from that job (for example after editing its prompt), reusing only what came before it. It saves to a new run folder and leaves the original as it was.

serve opens a web editor for the pipelines in the current directory. It prints a URL with an access token; anyone with the URL can run agents on this machine, so keep the default --host 127.0.0.1 unless you trust the network.

Pipeline

runner: claude             # default runner for all jobs (claude|opencode), default claude
model: sonnet              # default model for all jobs
effort: high               # default effort for all jobs (low|medium|high|xhigh|max)
vars:
  topic: null              # null means required
  audience: developers
jobs:
  - name: outline
    prompt: Plan an article about {{ vars.topic }}. Return section titles.
    schema:                # json schema, output becomes json instead of text
      type: array
      items: {type: string}
  - name: sections
    runner: opencode       # per-job, overrides the default
    model: openrouter/qwen/qwen3-coder
    effort: high
    foreach: jobs.outline  # one parallel agent per item
    concurrency: 3         # default 4
    retries: 3             # default 2
    max_failures: 1        # foreach items allowed to fail, default all but one
    prompt: Write the section {{ item }} for {{ vars.audience }}.
  - name: article
    tools: [Read, Bash]    # built-in tools, default none
    cwd: ../repo           # default current directory
    prompt: |
      Join these sections: {{ jobs.sections }}
      Reviewer notes on your previous version: {{ jobs.review.notes }}
  - name: review
    prompt: "Review this article: {{ jobs.article }}"
    schema:
      type: object
      properties: {revise: {type: boolean}, notes: {type: string}}
      required: [revise, notes]
    back:                  # go back and run again from an earlier job
      to: article          # this or an earlier job
      when: jobs.review.revise   # go back if truthy, default always
      max: 2               # required, the last run continues regardless
output: "{{ jobs.article }}"           # default last job output

Templates: {{ vars.x }}, {{ jobs.name }}, {{ item }} (inside foreach jobs), with nested access like {{ jobs.outline.0 }} or {{ item.title }}. Non-text values are inserted as JSON, except in output and pipeline vars, where a template that is a single reference, like "{{ item }}", keeps its value. prompt, runner, model, effort and cwd are templated, so model: "{{ vars.model }}" lets you choose the model per run.

A failed agent (an error, or no valid output for the schema) is run again up to retries times, waiting 5s, 10s, 20s and so on in between. A foreach item that still fails outputs null and the other items carry on; the job fails, and the run with it, only once more than max_failures items have failed, so by default only when every item does. max_failures: 0 stops at the first failed item. Any other job that fails after its retries ends the run. Costs of failed attempts are included in the totals.

Inside a back loop, prompts can also reference the job itself and later jobs of the loop, like {{ jobs.review.notes }} in article above. They hold the output of the previous pass and are empty on the first one.

stop ends the whole run as soon as the job finishes, if its when reference is truthy, printing reason (a template, default the when path) and exiting non-zero. Use it to bail out when a prerequisite is missing:

  - name: login
    tools: [Bash]
    prompt: "Run `gh auth status`. Set blocked true if it is not authenticated."
    schema:
      type: object
      properties: {blocked: {type: boolean}, reason: {type: string}}
      required: [blocked]
    stop:
      when: jobs.login.blocked          # stops when truthy
      reason: "{{ jobs.login.reason }}" # default: the when path

next jumps forward to a later job, skipping the jobs in between. Its entries are tried in order, the first whose when reference is truthy (default always) wins; if none does, the run continues with the following job. Skipped jobs render empty, so a job after the branches can reference all of them (see examples/triage.yaml):

  - name: triage
    prompt: "Classify this issue: {{ vars.issue }}"
    schema:
      type: object
      properties: {bug: {type: boolean}, feature: {type: boolean}}
      required: [bug, feature]
    next:
      - to: diagnose              # if
        when: jobs.triage.bug
      - to: design                # else if
        when: jobs.triage.feature
      - to: answer                # else
  - name: diagnose
    prompt: "Diagnose this bug: {{ vars.issue }}"
    next: [{to: reply}]           # skip the other branches
  - name: design
    prompt: "Design this feature: {{ vars.issue }}"
    next: [{to: reply}]
  - name: answer
    prompt: "Answer this question: {{ vars.issue }}"
  - name: reply
    prompt: "Reply to the author: {{ jobs.diagnose }}{{ jobs.design }}{{ jobs.answer }}"

Pipelines in pipelines

A job can run another pipeline instead of an agent, like a function: vars are its arguments and the job outputs the child's output (see examples/series.yaml). The path is relative to the parent file. Children can run pipelines of their own, but not themselves.

  - name: articles
    pipeline: review.yaml    # instead of prompt
    foreach: jobs.topics
    vars:                    # templates, a single reference keeps its value
      topic: "{{ item }}"
    on_stop: continue        # fail (default) or continue

A stop in the child fails the job by default, like a failed agent: a foreach item outputs null and counts toward max_failures, any other job ends the run. With on_stop: continue the job, or the foreach item, outputs the stop reason and the run carries on. Once a foreach job fails, its other running children are cancelled before their next job.

runner, model, effort, schema, tools, cwd and retries belong to the child's jobs; pass them as vars if the child should take them from the parent. Each child run is saved in the job's folder, jobs/<name>/ or jobs/<name>/<item>/ for foreach, and --resume continues inside it from the first child job without output.

Runners

runner picks the CLI agent that runs a job, per pipeline or per job.

• claude (default): model is a Claude Code model or alias like sonnet, effort is low|medium|high|xhigh|max.
• opencode: model is provider/model like openrouter/qwen/qwen3-coder, effort is appended to the model as provider/model#effort (provider-specific, like high, max, minimal). tools use the claude names above and become opencode permissions, everything else is denied: Read, Write, Edit, Glob, Grep, Bash, WebFetch, WebSearch, TodoWrite, Task, Skill. opencode has no schema flag, so the schema is appended to the prompt and the reply is parsed and validated against it; a reply that does not match is a failed attempt and is retried like any other.

To add a runner, write a module in pipellm/runners/ and register it in RUNNERS in pipellm/runners/__init__.py. The module defines EFFORTS (allowed efforts, None for any), TOOLS (supported tool names, None for any) and run(prompt, model, effort, schema, tools, cwd, log) returning (output, cost) and raising RunnerError(message, cost) on failure. pipellm/runners/base.py has Events, which starts the CLI so ctrl+c can stop it and yields its JSON events, and helpers for schema output by prompt.