feat(brief): pre-execution contract negotiation with output evaluation and auto-retry#68
Open
Nivesh353 wants to merge 4 commits into
Open
feat(brief): pre-execution contract negotiation with output evaluation and auto-retry#68Nivesh353 wants to merge 4 commits into
Nivesh353 wants to merge 4 commits into
Conversation
…uccess criteria Adds a Planner/Evaluator negotiation loop that turns a task prompt into a concrete, versioned contract of binary pass/fail assertions before an agent runs, then grades the agent's actual output against that contract with auto-retry on failure. Wires it into the CLI (`gitagent brief`), the SDK (`runBriefOrchestration`/`runWithBrief`), and the scheduler.
shreyas-lyzr
reviewed
Jul 3, 2026
shreyas-lyzr
left a comment
Contributor
There was a problem hiding this comment.
Solid addition overall — the planner/evaluator negotiation loop is well-thought-out and the hardening in output-evaluator.ts (explicit temperature/maxTokens, distinct failure reasons) is good prior-work cleanup. Four issues worth addressing before merge, in rough priority order.
- schedule-runner.ts: revert runPrompt to its original one-arg signature, fold the brief suffix into the prompt internally so external SchedulerOptions implementers don't silently drop brief injection - orchestrator.ts: recompute version via nextVersion() on each regenerate attempt instead of `version + attempts`, which could skip version numbers - storage.ts: escape pipe characters in assertion/test text before writing the Markdown success-criteria table - approval.ts, report.ts, colors.ts: extract duplicated CATEGORY_COLORS into a shared colors.ts, fixing the missing isTTY guard in both copies
Schedules can now name an explicit brief via `briefPath` (auto-created on first run if missing), and scheduled jobs backed by a brief now go through the complete negotiate/run/evaluate/retry flow via runWithBrief instead of just having brief text spliced into the prompt. Falls back to the plain runPrompt callback, unchanged, when no brief applies — so existing SchedulerOptions implementers aren't affected.
The evaluator wrote "passed" before "evidence" in its JSON output, forcing it to commit to a verdict before reasoning through the assertion — causing cases where the evidence correctly concluded a pass but the recorded verdict stayed false. Reordering the schema (and AssertionResult type) so evidence/notes come first makes the model reason before it commits.
shreyas-lyzr
approved these changes
Jul 6, 2026
shreyas-lyzr
left a comment
Contributor
There was a problem hiding this comment.
All four issues from the previous review are addressed:
- runPrompt keeps its original one-argument signature; brief injection is absorbed into executeScheduledJob via runWithBrief, so no downstream implementors need to change.
- Version numbering on regenerate now calls nextVersion(agentDir, id) again each time rather than using version + attempts, so on-disk filenames and in-memory state stay consistent.
- Pipe characters in assertion/test text are escaped via escapeCell() before writing into the Markdown table.
- CATEGORY_COLORS extracted to a shared colors.ts with a proper isTTY guard; both approval.ts and report.ts import colorCategory from there.
The core feature is well-structured. Approving.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces Agent Brief — a negotiation step that runs before an agent
executes a task, producing a concrete, versioned "contract" of binary
pass/fail success criteria (assertions). Once the agent produces output,
a second evaluator grades the output against that contract and can trigger
automatic retries with targeted feedback on what failed.
This closes the gap where an agent's output "looks done" but silently
violates constraints (word count, tone, banned terms, structure) that were
never made explicit or checked.
What's new (
src/brief/, 13 files)planner.ts— LLM call that turns a task + agent's SOUL/RULES/DUTIESinto a
BriefDraft: task summary, ambiguities, binary assertions(format/content/quality/constraint/behavior/tone), a scoring rubric, and
estimated complexity. Assertion count is capped at 15 in code (backing up
the prompt's own "never exceed 15" instruction) to bound evaluator cost.
evaluator.ts— LLM call that reviews the Planner's draft itself(not the agent's output) for vague/unmeasurable assertions, missing
coverage, or contradictions with RULES.md, and returns critical/warning
issues.
negotiator.ts— drives Planner ⇄ Evaluator rounds until the draft isapproved or a round limit is hit.
output-evaluator.ts— after the agent runs, grades its actual outputagainst each approved assertion (binary, evidence-required, strict).
Hardened in this PR: explicit
constraints: { temperature: 0, maxTokens: 8000 }on the grading call (previously silently inherited the host agent's
possibly-too-small
max_tokens, causing truncated/unparseable JSON), plusthe failure path now distinguishes and reports why grading failed —
LLM call error, truncated response, or empty response — instead of a
generic
Unexpected end of JSON input.orchestrator.ts— top-level flow: negotiate → stale-check →approve/edit/regenerate/skip → save.
runner.ts(runWithBrief) — runs the agent with the brief injectedinto its system prompt, evaluates output, and auto-retries (default 2x)
with a prompt that lists exactly which assertions failed and why.
storage.ts— brief persistence: save/load/list/find/version/archiveunder
.gitagent/briefs/, content-hash based staleness keys.stale.ts— detects when SOUL/RULES/DUTIES changed since a brief wasapproved and reports which assertions are affected.
approval.ts— interactive CLI approval flow (approve/edit/regenerate/skip)and brief list/detail rendering.
editor.ts— opens a draft in$EDITORfor manual editing, validatesthe edited JSON against the schema.
injector.ts— serializes an approved brief into a system-prompt suffix.report.ts— renders the final pass/fail evaluation report.types.ts— shared types (Brief,BriefDraft,BriefAssertion,AssertionResult,OutputVerdict, etc).Integration changes (4 files)
src/loader.ts— adds optionalbrief?.modeltoAgentManifestso anagent.yamlcan pin a specific model for brief negotiation/evaluation,independent of the agent's own
model.preferred.src/index.ts— newgitagent briefCLI subcommand:gitagent brief "<task>"— negotiate + approve + run with evaluation--only— negotiate and save without running--brief-path <path>— run with an already-approved brief--list/--view <id>/--regenerate— manage saved briefs--model/-m,--dir/-d,--yes/-ysrc/exports.ts— exposes the brief API (runBriefOrchestration,generateBrief,runWithBrief, storage helpers, and all brief types) fromthe SDK's public surface.
src/schedule-runner.ts— scheduled jobs now look up an approved briefmatching the schedule's prompt and inject it into the run automatically,
warning if it's gone stale.
Why
Assertions are cheap to write and binary to check, which makes "did the
agent actually do what was asked" a testable question instead of a vibe
check — especially valuable for scheduled/unattended runs where nobody is
reading the output live.
Test plan
gitagent brief "write a 500-word post on X" --dir <agent>— negotiate,approve, confirm the assertion contract looks reasonable
gitagent brief --list/--view <id>render correctlywarning appears on next run
(
src/schedule-runner.ts)tsc --noEmitpasses