Skip to content

cli.md

Latest commit

 

History

History
230 lines (184 loc) · 12.3 KB

cli.md

File metadata and controls

230 lines (184 loc) · 12.3 KB
Raw

Command-Line Interface

Installing the package installs one console script, evalrx (evalrx.cli:main), with eight subcommands. evalrx <command> --help is always the source of truth for flags; this page is the map of which command does what and how they chain together.

evalrx --help
evalrx <command> --help

-v / --verbose (before the subcommand) turns on EvalRX's internal stage-by-stage narration for any of them. Any command that prints a local UI address or a just-written report file (serve, dashboard, and the report.html every explore run writes) prints it as a clickable link in a terminal that supports OSC 8 hyperlinks (most modern ones do); it degrades to plain text automatically on a redirected/non-tty stream, or set EVALRX_NO_HYPERLINKS=1 to force plain text.

What this CLI does — and doesn't — run

explore/run-codebase only ever drive M2 (explore) → M3 (propose hypotheses) → M4-as---holdout-confirm — see the live-narration note under evalrx explore below for what that looks like. There is no CLI command for M1 (targeted probing) or M5 (repair/intervention). The full M1→M5 loop (VLDiagnoseLoop/AutoDiagnoseLoop) is Python-API only — see Quickstart — or run as a complete example:

  • Zero setup: evalvitals.github.io/evalrx/demo — two committed, real M1→M5 runs (VLM/ChartQA, ALM/MMAU), viewable in the browser with nothing installed.
  • Run one yourself: examples/m1_m5/deco_hallu runs the real chain end-to-end against a live VLM — needs a CUDA GPU, cached model weights, and a coding-agent CLI for the judge. Launch it with docker compose up from that directory (this repo's examples are Docker-only — see the example's own README.md), then evalrx serve outputs to view the result. It already runs with live M1-M5 terminal narration — same visual style as explore's, built from RunLoggerV2(..., narrate=True) — since its run.py opts in; do the same in your own RunLoggerV2(...) call to get it (see evalrx.eval_agent.narration.LoopNarrator).

The two jobs a subcommand does

Job Commands
Produce a run — turn result logs, a codebase, or a diagnosis loop into an output directory explore, run-codebase
View or export a finished run — everything below reads a run directory, none of them re-run the model serve, report, publish-report, dashboard, export-langfuse, backfill-langfuse

A "run directory" is either an explore output (exploratory_report.json + figures/ + tables/) or a loop run (run.json/M1..M5, or a logs/ folder holding those) — every viewing command accepts either shape.

Which viewer command do I want?

Four commands all read the same run directory and this is the part that actually confuses people, because two of them sound like the same thing:

Command What it produces When to reach for it
evalrx serve Starts a local FastAPI server and opens the dynamic React report UI (http://127.0.0.1:8501 by default) — the same interactive interface used throughout this repo's own examples. Nothing is written to disk beyond a small .evalrx-cache unless you also ask it to publish. Default choice. Point it at a run dir, or launch with none and drop a .zip on the page. --runs-root adds a runs panel to browse several experiments from one server.
evalrx report Renders the same React UI but bakes it into one portable, self-contained report.html — no server, no network. Sharing a result with someone who won't run a Python server, or archiving a run as a single file.
evalrx publish-report Compiles and caches the underlying ReportData + layout spec that both serve and report render, without launching or exporting anything. Pre-warming the cache for a run before serving it, or CI.
evalrx dashboard Deprecated. A dependency-free static-HTML server predating the React UI (evalrx/analysis/dashboard.py). Kept only for old explore outputs that never got a React-compatible report.html. Don't reach for this new — use serve.

serve/report/publish-report all take --source {auto,local,langfuse} + --trace-id: point them at a Langfuse trace instead of a local directory.

evalrx explore

Single-shot exploratory analysis (M2 + M3) over a results directory — no loop, no code. A local CLI coding agent writes and runs the analysis code; the host adjudicates the statistics and renders figures.

evalrx explore ./results \
  --backend codex \
  -q "What distinguishes failed cases from successful ones?" \
  --serve-report

By default this prints one live, aligned line per stage as it actually happens — the same M2 explore ············ … style the landing page's "simulated run" walkthrough shows, except every count and duration here is real (figures rendered, candidate signals found, hypotheses proposed, a repair-retry when the coding agent's first attempt needs fixing). Pass --no-narrate for plain output (e.g. when piping to a file); color is already skipped automatically for a non-tty stream or when NO_COLOR is set.

Flag Default Purpose
path (positional) — File or directory of JSON/JSONL results. Required.
-q, --question "Explore this dataset…" Natural-language analysis question for the coding agent.
--outcome-col auto-detect Target/outcome column name. Falls back to unsupervised EDA when none is found.
--out evalrx_explore_output Output directory for report/code/figures/tables.
--backend (--coder-provider) antigravity Coding-agent CLI: antigravity, codex, claude_code, opencode, gemini_cli, kimi_cli. Must be installed + authenticated separately.
--model (--coder-model) backend default Model name passed to the coding agent.
--max-rows / --max-files 2000 / 200 Sampling caps for large inputs.
--timeout-sec / --max-attempts 120 / 2 Per-attempt timeout and repair-retry budget.
--serve-report off Launch serve on the output when the run finishes.
--port auto Port for --serve-report.
--skill DIR (repeatable) — Agent-Skill directory to style agent-authored figures. Implies --allow-skills.
--no-skills off Skip the package's bundled skills (e.g. nature-figure).
--no-hypotheses off Skip M3 — stop after M2's takeaways.
--holdout-frac 0.0 Fraction held out before exploration (outcome-stratified, deterministic).
--holdout-confirm off Re-test frozen recipes/hypotheses on the held-out rows (confirm_report.json). Requires --holdout-frac > 0.
--judge-model claude-opus-4-8 Judge grading each hypothesis against the held-out table (only with --holdout-confirm).
--no-narrate off Don't print live per-stage terminal narration (see above).
--progress-path PATH — Also append durable JSONL progress events to PATH, for a separate workbench UI — independent of narration; both can run together.

--dashboard still exists as a deprecated alias for --serve-report.

evalrx run-codebase

For when you don't have result logs yet — only an evaluation/inference codebase. A CLI coding agent runs it inside an isolated copy (your original directory is never modified), harvests a records.json/.jsonl output, and hands it to the same explore (M2+M3) pipeline.

evalrx run-codebase ./my_eval_repo \
  --backend claude_code \
  -q "Where does the model fail and why?" \
  --out evalrx_run_codebase_output
Flag Default Purpose
path (positional) — Directory containing the codebase to run. Required.
-q, --question "Explore this dataset…" Also given to the run agent as task context.
--out evalrx_run_codebase_output Output directory for workspace/records/report/figures/tables.
--backend (--coder-provider) claude_code Same choices as explore. Used both to run the codebase and to explore it.
--records-name records.json Output-contract filename the run agent must write.
--timeout-sec / --max-attempts 1200 / 2 Budget for running the codebase itself.
--no-explore off Only run + harvest; skip the M2/M3 explore step.
--serve-report / --port off / auto Same as explore.
--no-narrate off Don't print live terminal narration (a RUN codebase line plus the explore narration above).

evalrx serve

evalrx serve outputs/qwen3.5-2b/chartqa
evalrx serve                          # start empty, drop a .zip on the page
evalrx serve --runs-root outputs/     # runs panel over every run under outputs/
Flag Default Purpose
run_dir (positional, optional) none Run directory. Omit to start with an empty drop-zone.
--port 8501 Loopback port.
--no-browser off Don't auto-open a browser (the default in a headless/remote session).
--source auto auto | local | langfuse. auto uses Langfuse only when --trace-id is set.
--trace-id — Langfuse trace id (required with --source langfuse).
--runs-root run_dir's parent Where the in-page runs panel looks for other experiments.

evalrx report

Exports the same UI serve shows, as one portable HTML file — no server needed to view it afterward.

evalrx report outputs/qwen3.5-2b/chartqa --out chartqa_report.html
Flag Default Purpose
run_dir (positional) outputs Run directory holding run.json/M1..M5, or a logs/ folder.
--example-dir — Root holding a benchmark/example's data/ manifest, for legacy runs.
--out, -o <run_dir>/report.html Output HTML path.
--embed-media representative representative | all | none — how much image/audio to inline.
--audio-bitrate 48k Transcodes inlined lossless audio to mono MP3 via ffmpeg (needs ffmpeg on PATH); none inlines the original bytes.
--no-audio off Skip audio transcoding entirely (forces --embed-media none).
--source / --trace-id local / — Same Langfuse switch as serve.

evalrx publish-report

Compiles and caches ReportData plus the validated json-render layout that serve/report both render — without launching a server or writing an HTML file. Useful for pre-warming a run's cache, or in CI.

evalrx publish-report outputs/qwen3.5-2b/chartqa

Takes run_dir (positional, default outputs), --example-dir, --source, --trace-id — same meanings as report.

evalrx dashboard (deprecated)

evalrx dashboard evalrx_explore_output

A dependency-free static-HTML server (evalrx/analysis/dashboard.py) predating the React report UI. It compiles a legacy report.html (evalrx/reporting/html_report.py) if one isn't already there and serves it with the standard-library http.server — no fastapi needed. Kept for old explore outputs; reach for serve for anything new.

Takes run_dir (positional) and --port.

evalrx export-langfuse

Maps a finished run (M1–M4, fixes, scores) onto Langfuse Traces, Spans, and Scores — either as a static JSON bundle or a live sync.

evalrx export-langfuse outputs/qwen3.5-2b/chartqa --out trace.json
evalrx export-langfuse outputs/qwen3.5-2b/chartqa --sync   # push to a live Langfuse server
Flag Default Purpose
run_dir (positional) outputs Run directory.
--out, -o <run_dir>/langfuse_trace.json Output JSON path (ignored with --sync).
--sync off Push directly to a live Langfuse server instead of writing a file.

evalrx backfill-langfuse

Queues an already-completed run for reliable Langfuse ingestion (an outbox pattern, distinct from export-langfuse --sync's direct push).

evalrx backfill-langfuse outputs/qwen3.5-2b/chartqa --dry-run

run_dir (positional, required) and --dry-run (inspect without writing an outbox) are the only flags. Prints trace=… events=… published=… pending=….