Skip to content

.NET: [Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate) #10171

Closed

Description

@lss680455-create

Problem

agent_framework_orchestrations ships five peer builder patterns — Sequential, Concurrent,
Handoff, GroupChat, Magentic (agent_framework_orchestrations/__init__.py:5-10,
python/packages/orchestrations/README.md:11-93). None of them represents opposing
positions
or produces a decision. Every pattern either merges outputs
(Concurrent), chains them (Sequential), routes control (Handoff), or runs an open-ended
conversation whose stopping point is "looks done" (GroupChat, Magentic).

The gap is visible in this repository today. The one debate-shaped artifact that exists is a
sample built out of a group chat:

  • python/samples/03-workflows/orchestrations/group_chat_philosophical_debate.py:226-234
    constructs GroupChatBuilder(...) with
    termination_condition=lambda messages: sum(1 for msg in messages if msg.role == "assistant") >= 10.
    There is no verdict, no ballot, no tie handling, and no separation between advocates and
    an adjudicator; the "conclusion" is simply the last thing the moderator says.
  • That sample is load-bearing enough that it is currently being repaired in
    PR #10085 (an
    intermediate_output_from argument bug). Hand-rolled orchestration in a sample is
    exactly the shape of code that drifts and has no unit tests.

Users who need a checked decision therefore re-implement the same state machine — openings
that cannot see peers, N rounds of rebuttal, a ballot, a tie rule, a structured verdict —
in prompts, and get non-reproducible results and no checkpointability. Multi-agent debate is
a well-established technique (Du et al., 2023, arXiv:2305.14325);
the framework has no first-class pattern for it.

Use Case and Evidence

  1. Release / policy review gates. Three reviewers with different mandates (safety, cost
    and latency, answer quality) argue over whether a change ships. The caller needs a
    machine-readable decision plus the recorded dissent, not a chat transcript it has to
    parse with an LLM afterwards. Today this is staffed by a group chat whose output shape
    varies run to run — unusable as a CI gate.
  2. Contested analytical questions. "Is this account takeover or a false positive?" Two
    or three hypotheses compete. Sequential cannot compare them, Concurrent produces two
    answers and no resolution, Magentic replans a task rather than judging positions.
    The user today writes a custom aggregator prompt and has no way to stop early, bound the
    cost, or resume after a failure.
  3. Auditable adjudication with human escalation. Regulated scenarios need the tie case
    to be explicit: either the judge decides (and the tie is recorded), or a human is asked,
    or the run reports "unresolved" instead of silently picking an answer.

A duplicate search across issues and pull requests found no existing request for a
debate/consensus orchestration pattern; the nearest items and why they are not this are
listed in Acknowledgements.

Desired Outcome

A DebateBuilder in agent_framework_orchestrations that:

  • takes N debaters plus one judge, runs a bounded sequence of
    independent openings → cross-examination rounds → adjudication;
  • returns the judge's decision as a structured value, with dissent and round/truncation
    metadata, on the workflow's terminal AgentResponse.value;
  • offers an explicit, documented tie policy instead of leaving ties to prompt luck;
  • behaves like the existing five builders for everything else: same termination_condition
    type, max_rounds cap, request-info/HITL flag, checkpointing, output_from /
    intermediate_output_from selection, streaming vs non-streaming output shape, and the
    same construction-time ValueError / TypeError semantics;
  • is strictly additive: no change to the existing five builders or to the workflow runtime.

Proposed Direction

Builder

DebateBuilder(
    *,
    name: str | None = None,
    participants: Sequence[SupportsAgentRun | Executor],  # debaters only, judge excluded
    judge: SupportsAgentRun | Executor,                   # adjudicator
    rounds: int = 2,                                      # cross-exam rounds after the openings
    voting: Literal["judge_only", "majority"] = "judge_only",
    tie_policy: Literal["judge_decides", "return_dissent", "escalate_to_human"] = "judge_decides",
    max_rounds: int | None = None,                        # hard cap; coerced to >= 1, as base does
    termination_condition: TerminationCondition | None = None,
    checkpoint_storage: CheckpointStorage | None = None,
    output_from=UNSET,                                    # same contract as the other builders
    intermediate_output_from=None,
)

Fluent methods, matching the existing builders' style
(_group_chat.py:822 with_termination_condition, :863 with_max_rounds,
:881 with_checkpointing, :913 with_request_info;
_handoff.py:956/:1005, _magentic.py:1568/:1616):

.with_termination_condition(cond) -> DebateBuilder
.with_max_rounds(n)               -> DebateBuilder
.with_checkpointing(storage)      -> DebateBuilder
.with_request_info(agents=None)   -> DebateBuilder
.build() -> Workflow

TerminationCondition is deliberately reused unchanged, not redefined:
Callable[[list[Message]], bool | Awaitable[bool]] (_base_group_chat_orchestrator.py:58).

Example

from agent_framework import AgentResponse
from agent_framework_orchestrations import DebateBuilder, DebateVerdict

workflow = (
    DebateBuilder(
        participants=[safety_reviewer, latency_reviewer, quality_reviewer],
        judge=arbiter,
        rounds=2,                        # openings + 2 cross-examination rounds
        voting="judge_only",
        tie_policy="judge_decides",
        max_rounds=8,
        checkpoint_storage=storage,
    )
    .with_request_info(agents=[safety_reviewer])   # optional HITL, same flag as other builders
    .build()
)

events = await workflow.run("Ship the model-card change this quarter? Answer yes or no, with reasons.")
response = events.get_outputs()[0]
assert isinstance(response, AgentResponse)
verdict: DebateVerdict | None = response.value
print(verdict.decision, verdict.confidence, [d.position for d in verdict.dissent])

(events.get_outputs() / AgentResponse usage mirrors
python/samples/03-workflows/orchestrations/concurrent_agents.py:76-84.)

Output schema

Terminal output is an AgentResponse (as with every other builder), whose value carries:

class DebateDissent(BaseModel):
    position: str          # the position not selected
    held_by: list[str]     # participant names that held it
    summary: str           # one or two sentences, written by the judge

class DebateVerdict(BaseModel):
    decision: str
    rationale: str
    confidence: float | None = None
    dissent: list[DebateDissent] = []
    rounds_completed: int
    truncated: bool = False        # max_rounds / termination_condition cut the debate short
    unresolved: bool = False       # tie_policy="return_dissent" and the tie did not break
    vote_counts: dict[str, int] | None = None   # only when voting="majority"

AgentResponse supports a structured value (agent_framework/_types.py:3729), and the
judge message is also appended to messages with author_name=judge, so text-only callers
and workflow.as_agent() keep working. Per-round debater turns are exposed as intermediate
output through the existing selection contract (python/packages/orchestrations/README.md:95-130),
not as a new event family in v1.

Checkpoint contract

State is stored through the shared OrchestrationState container
(_orchestration_state.py:33-54); pattern-specific metadata carries the phase
(opening / cross_exam / ballot / adjudicate), the round index, the per-debater
statements, and the tally. Any new envelope dataclass that crosses a checkpoint boundary
must be registered via register_checkpoint_type in __init__.py and covered by
tests/test_checkpoint_types.py — the restricted unpickler auto-trusts only the
agent_framework. prefix, not sibling distributions (__init__.py:117-146,
tests/test_checkpoint_types.py:1-15). This is a hard requirement, not a nicety.

Error semantics (same shape as the existing builders)

Condition Behaviour
participants=[], duplicate participant ids, judge also passed in participants ValueError at construction (cf. _sequential.py:144-163; existing tests tests/test_concurrent.py:46-56)
rounds < 1, unknown voting / tie_policy literal, max_rounds < 1 ValueError (rounds coerced to 1 like max_rounds, _base_group_chat_orchestrator.py:164)
participant is neither SupportsAgentRun nor Executor TypeError (mirrors _sequential.py:245)
output_from / intermediate_output_from overlap, unknown participant, wrong "all"/"all_other" use existing validation errors, unchanged (README.md:118-127)
a debater fails during a round recorded in the verdict; debate continues while ≥ 2 debaters produced statements, otherwise the orchestrator yields a completion message (not an exception)
judge output cannot be parsed bounded retry, then a completion message with value=None; the run still completes rather than raising

Failure-as-completion-message follows the base orchestrator's established behaviour
(_base_group_chat_orchestrator.py:353-371, :519-537) instead of inventing an exception
contract.

Semantics

Phases and roles. Round 0 is openings: every debater answers independently and never
sees a peer's message (fan-out is parallel, like ConcurrentBuilder's dispatcher/fan-out,
_concurrent.py:183-188). Round r is cross-examination: each debater receives the task
plus a bounded digest of opposing statements from round r−1 — not the full transcript — and
must defend or concede. The judge never speaks during the debate; it is invoked once at
adjudication with the task, the ordered transcript, and (if enabled) the tally. The judge is
required to be a separate agent and is rejected if it also appears in participants.

Voting.

  • judge_only (default): no ballot; the judge decides and writes the dissent. Debaters only
    need to be ordinary chat agents.
  • majority: after the last cross-exam round each debater submits a short ballot as
    structured output (this is the one mode that requires debater structured output; a
    ballot that fails to parse is dropped and counted in the tally metadata). A strict majority
    fixes the decision and the judge only writes the rationale and dissent; it does not
    override. No strict majority → tie_policy.

Ties and no-consensus.

  • judge_decides (default): the judge picks; the fact that the vote was tied stays visible
    in dissent and vote_counts, while unresolved remains False because a decision was
    produced.
  • return_dissent: no decision is forced; the verdict has unresolved=True, decision="",
    and every tied position in dissent. This is the option for pipelines that must not guess.
  • escalate_to_human: emits a request_info event (the existing HITL mechanism,
    _orchestration_request_info.py:187 AgentApprovalExecutor) carrying the tied options;
    the run resumes with the human's choice, which becomes the decision.

Budgets and truncation. rounds bounds the adversarial phase; max_rounds, if set,
caps total coordination rounds with the same coercion as the base class
(_base_group_chat_orchestrator.py:164, :498-517). Reaching a cap jumps to
adjudication
and sets truncated=True — a deliberate difference from GroupChat, where the
cap yields a generic completion message, because a truncated debate still has a useful
verdict. A user-supplied termination_condition is evaluated after each round and, when
true, also short-circuits to adjudication (truncated=True), never "return the raw
transcript".

Failure and cancellation. Debater failures are isolated per round; the debate degrades
to fewer stances and records it. Cancellation propagates as usual; because state is
checkpointed at phase boundaries, a cancelled run resumes at the start of the phase in
progress. No partial verdict is ever emitted.

HITL. .with_request_info(agents=[...]) reuses the existing flag verbatim: the workflow
pauses after the selected debaters' turns so a human can steer a stance
(_orchestration_request_info.py:19 resolve_request_info_filter). Judge-side intervention
is intentionally limited to the tie-escalation path in v1; a general "approve the verdict"
review step is listed as an open question rather than smuggled into the first cut.

Boundary against GroupChat and Magentic. GroupChat is an open-ended, orchestrator-chosen
speaking order with no distinction between advocate and decider and no structured decision;
use it for "let them talk". Debate is fixed-phase, bounded, adversarial-then-adjudicated, and
its contract is a verdict object. Magentic maintains task/progress ledgers and replans a
task; it answers "what should be done next", not "which position is right", and its
MagenticProgressLedger invariants have no place for dissent. Concurrent merges parallel
answers; it has no rebuttal round and no tie semantics.

Comparison with the five existing patterns

Pattern Control flow Rounds Structured decision Dissent / ties Checkpoint HITL
Sequential (_sequential.py:66) fixed chain 1 pass no no yes with_request_info
Concurrent (_concurrent.py:183) fan-out + aggregate 1 no no yes with_request_info
Handoff (_handoff.py:643) agents route control unbounded no no yes request/response
GroupChat (_group_chat.py:619) selector/orchestrator agent unbounded (max_rounds) optional (AgentOrchestrationOutput only drives the loop, _group_chat.py:263-283) no yes with_request_info
Magentic (_magentic.py:1418) manager + ledgers unbounded (max_round_count) no (task ledger) no yes plan review
Debate (proposed) openings → cross-exam → adjudicate bounded yes (DebateVerdict) yes (ballot, tie policy) yes with_request_info + tie escalation

Alternatives Considered

  1. GroupChatBuilder + custom selection_func + termination_condition. This is what the
    repository does today (the debate sample). Rejected: (a) openings cannot be made
    peer-blind — GroupChat broadcasts every message to every participant
    (_base_group_chat_orchestrator.py:410-440), so round 0 is not independent; (b) there is no
    ballot or tie rule; (c) termination by message count is the only cheap criterion
    (group_chat_philosophical_debate.py:228) and it is not a decision; (d) the terminal
    output is whatever the last speaker said, with no structured verdict to gate on.
  2. Ship it as a sample only. Rejected: the sample already exists and is the evidence of
    the problem. Tie handling, ballot parse failures, truncation, mid-debate
    checkpoint/restore and "no consensus" are exactly the behaviours that need unit tests and
    a stable contract; samples carry neither, and PR Python: Select all other participants in the group chat debate sample #10085 shows they are not maintained as
    contracts.
  3. ConcurrentBuilder + LLM aggregator callback. Closest workaround, but single-round:
    no rebuttal, no state machine, and the callback receives only
    list[AgentExecutorResponse] (_concurrent.py:139-150), so phase structure cannot be
    reconstructed. There is also no hook point between rounds.
  4. A debate mode on MagenticBuilder. Rejected: Magentic's documented contract is
    ledger-driven planning (_magentic.py:1440-1500); reusing it would either break those
    semantics or bolt a second state machine onto a class that already owns one. Larger diff,
    higher review risk, worse discoverability than a sibling builder.
  5. Handoff-based adversarial routing. Handoff routes control between agents; it has no
    notion of "everyone answers, then one decides", and no decision record.

Scope and Non-Goals

In scope

  • new _debate.py with DebateBuilder, a DebateOrchestrator(BaseGroupChatOrchestrator),
    and the DebateVerdict / DebateDissent models;
  • exports and checkpoint-type registration in agent_framework_orchestrations/__init__.py;
  • a new package-local feature index (FeatureIndex.ORCHESTRATION_DEBATE) in
    _feature_usage.py:6-13 plus its row in docs/specs/feature-usage-bit-registry.md:149
    (slots 37–47 are already reserved for orchestration growth);
  • unit tests, one sample, package/samples README entries.

Out of scope (explicit non-goals)

  • any change to the workflow runtime or to the existing five builders — no behaviour, no
    signature, no default changes;
  • .NET parity (Microsoft.Agents.AI.Workflows) — to be tracked separately if wanted;
  • shipping debate prompts or opinionated stances; the builder orchestrates, it does not
    prescribe persona text;
  • claim extraction, fact verification, citation checking, or judge-quality evaluation;
  • token-budget accounting, dynamic participant join/leave, LLM-judged "consensus reached"
    termination (that stays GroupChat territory);
  • relitigating the existing output_from / intermediate_output_from semantics.

Testing and Sample Plan

Unit tests run offline with stub agents/executors, following the existing pattern
(tests/test_concurrent.py:27-43 _FakeAgentExec, no model calls):

  • construction: empty participants, duplicate ids, judge in participants, rounds < 1,
    invalid voting/tie_policy, TypeError for a non-agent/executor, inherited
    output-selection validation errors;
  • phase order: openings are peer-blind and issued in parallel; cross-exam round r sees
    only round r−1; judge receives the full ordered transcript;
  • output contract: AgentResponse.value is a DebateVerdict; judge message present with
    author_name; intermediate_output_from=[...] surfaces debater turns as intermediate
    events and not as output events (extend
    tests/test_orchestration_intermediate_vs_terminal.py);
  • voting/ties: strict majority fixes the decision; no majority under each of the three
    tie policies; unparseable ballot is dropped and recorded;
  • budgets: max_rounds and an early-true termination_condition both jump to
    adjudication with truncated=True;
  • failures: one debater raising leaves the debate runnable; too few debaters yields the
    completion message; unparseable judge output after retries yields a completed run with
    value=None;
  • checkpointing: save mid-debate, rebuild with the same participant names, restore, finish;
    add the new envelopes to tests/test_checkpoint_types.py:56-75;
  • streaming: terminal output is a single AgentResponseUpdate, mirroring the base class's
    mode-aware completion (_base_group_chat_orchestrator.py:373-395);
  • feature index parity: the repo-wide validation test asserts exact table parity
    (docs/specs/feature-usage-bit-registry.md:282-284), so the new row must be added with the
    enum.

Sample: one file, e.g.
python/samples/03-workflows/orchestrations/debate_model_review.py — three reviewers with
different mandates, one judge, rounds=2, printing decision, confidence, dissent, and
demonstrating max_rounds truncation in a second run. Style follows the existing orchestration
samples (Foundry chat client, logging.basicConfig). Update
python/packages/orchestrations/README.md (pattern list + usage block) and
python/samples/03-workflows/orchestrations/README.md:56 (sample table; the "patterns like
Sequential, Concurrent, Handoff, GroupChat, and Magentic" sentence at :136).

Commands, from python/ (CONTRIBUTING.md:269-272):

uv run poe test -A -m "not integration"
uv run poe syntax
uv run poe mypy   # per-package task, python/packages/orchestrations/pyproject.toml:86-88

Compatibility

  • Additive only. New module, new public names, new feature index; no signature or
    behavioural change to the existing five builders, so the Griffe comparison has nothing to
    report. agent-framework-orchestrations is released
    (python/PACKAGE_STATUS.md:62), so the Griffe check applies to it (CONTRIBUTING.md:136-165)
    — additions are not breaking changes.
  • Lazy namespace parity. Public names are also added to the core lazy re-export list
    python/packages/core/agent_framework/orchestrations/__init__.py:21-69 and to
    __init__.pyi, so from agent_framework.orchestrations import DebateBuilder works like the
    other builders. This is the same stub/runtime parity issue currently being fixed in
    PR #10170, and is additive.
  • Checkpoints. Old checkpoints are unaffected: new types are registered in addition, and
    no existing registration changes (__init__.py:129-146).
  • Feature-usage bitmask. The new index takes an already-reserved orchestration slot
    (docs/specs/feature-usage-bit-registry.md:149), so no renumbering and no decoder change.
  • Hosting. agent-framework-hosting matches builders structurally via a zero-argument
    build() protocol (python/packages/hosting/agent_framework_hosting/_state.py:47-52), so
    DebateBuilder composes there without changes.

Language

Python (agent-framework-orchestrations). .NET parity deliberately not included.

Contribution Intent

I am willing to implement this (with tests and sample) after the direction is agreed.

AI Assistance Disclosure

Per AI_CONTRIBUTIONS.md:27-32, material AI assistance is disclosed here: repository
analysis, the API sketch, and the text of this issue were drafted with AI assistance. The
issue author owns the design and the implementation, and will answer review questions
directly. No prompts, credentials, or private data are included.

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I will wait for explicit maintainer agreement (ready-for-implementation or an
    equivalent comment, CONTRIBUTING.md:75-81) before starting a non-trivial change.

Duplicate check performed on 2026-11-08 across issues and pull requests (open and closed),
searching debate, consensus, voting, jury, panel, adjudicat, devil's advocate,
and in:title orchestration|builder|pattern. Results: no open or closed issue/PR proposes a
debate or consensus orchestration pattern; no assignee holds that direction. The nearest
items and why they are not this:

Open Questions for Maintainers

  1. Should an exhausted max_rounds jump to adjudication (truncated=True) as proposed, or
    keep GroupChat's generic completion message for consistency?
  2. Is voting="majority" (which makes debater structured output a requirement) wanted in the
    first cut, or should v1 ship judge_only only and add ballots later?
  3. Should a judge-verdict review step (request_info before the verdict becomes workflow
    output) be in scope, similar in spirit to Magentic's plan review
    (_magentic.py:1568)?
  4. Name: DebateBuilder / orchestration.debate (index 37) — or would maintainers prefer
    AdjudicationBuilder to keep "debate" out of the public API name?

Activity

  1. added
    .NETUsage: [Issues, PRs], Target: .Net
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 7, 2026
  2. changed the title [-][Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/-] [+]Python: [Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/+] on Nov 7, 2026
  3. changed the title [-]Python: [Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/-] [+].NET: [Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/+] on Nov 7, 2026
  4. added
    orchestrationUsage: [Issues, PRs], Target: multi-agent orchestration (high-level patterns)
    on Nov 7, 2026
  5. eavanvalkenburg commented on Nov 9, 2026

    @eavanvalkenburg
    Member

    We are open to exploring this as an experimental capability in the existing Python orchestration package. That is not yet a commitment to a stable DebateBuilder API or to implementation timing.

    Before choosing an API, we would like to compare naming options and see short sample code for several scenarios. Please consider both an LLM as judge and a human as judge, including whether adjudication is useful independently of a multi-round debate. The examples should make the differences between those variants clear.

    We will use planning to compare those options and the existing workflow-composition approach before agreeing on an implementation direction. Please hold off on a non-trivial implementation PR while that direction is still being worked out.

  6. eavanvalkenburg commented on Nov 9, 2026

    @eavanvalkenburg
    Member

    /plan Explore this as an experimental capability in the existing Python orchestration package, not a stable API commitment or authorization to implement. Produce design options before an implementation plan. Compare naming options such as DebateBuilder, AdjudicationBuilder, ReviewBuilder, and better alternatives; explain which names fit judging independent candidates versus multi-round debate. Include concise, explicitly proposed API snippets for several scenarios: (1) independent reviewers with an LLM as judge, (2) a human as judge using the existing request_info and checkpoint/resume primitives, and (3) bounded multi-round debate followed by adjudication. Compare each with a small custom workflow or sample built from current supported primitives, and identify the minimal reusable framework capability, if any, rather than assuming a new builder is required. Cover experimental-stage placement in agent-framework-orchestrations, outcome and unresolved-decision semantics, cancellation, checkpointing, and streaming compatibility without prematurely fixing a large verdict schema or policy inventory. Keep .NET parity out of scope. Do not implement changes or open an implementation PR; return naming alternatives, trade-offs, illustrative snippets, and the maintainer decisions needed next.

  7. github-actions commented on Nov 9, 2026

    @github-actions
    Contributor

    Summary

    The issue asks for a way to run opposing positions and get one recorded decision. Today agent_framework_orchestrations has Sequential, Concurrent, Handoff, GroupChat and Magentic. None has a judge or an explicit "unresolved" outcome. The only debate-like sample is group_chat_philosophical_debate.py, which stops after a fixed number of messages.

    The maintainer direction is to treat this as experimental, compare names and scenarios, and not commit to a stable DebateBuilder yet. Below is design exploration only. Nothing here has been run, and no changes are made.

    Recommendation

    Start with a sample, and add code only if the sample shows a gap. I confirmed from the repo that these pieces already exist:

    • ConcurrentBuilder fans out to N participants and has with_aggregator(...) (_concurrent.py). A judge can be the aggregator for "independent reviewers + judge".
    • ctx.request_info(...) (_workflow_context.py) pauses a workflow for a human reply. Checkpointing can resume it. See human-in-the-loop/ and handoff_with_tool_approval_checkpoint_resume.py.
    • with_request_info(...) already exists on the builders.
    • WorkflowBuilder edges, loops and fan-in can express a bounded round loop.
    • ExperimentalFeature is defined in core (agent_framework/_feature_stage.py).

    Scenario 1 is mostly covered already. Scenario 2 is a small custom executor. Scenario 3 is the only one where hand-written wiring is repetitive: openings that cannot see peers, N rebuttal rounds, a cap, and a judge.

    Likely minimal reusable capability: a "judged candidates" step, not a full debate builder. It would take N candidate outputs, call a judge, and return the judge's decision or an explicit "unresolved" outcome. Multi-round debate can then be a sample or a thin layer on top. This is a hypothesis for maintainers to confirm.

    Naming options

    Name Fits Concern
    DebateBuilder Multi-round, cross-examination Misleading for independent reviewers who never see each other
    AdjudicationBuilder Judge picks among independent candidates; also works as the final stage of a debate Does not suggest rounds
    ReviewBuilder Reviewers + verdict Overlaps with "request info / human review" and code review
    JudgedBuilder / PanelBuilder Panel of independent members plus a decider Less familiar
    DeliberationBuilder Rounds ending in a decision, with a human or LLM decider Vaguer

    Suggested split: AdjudicationBuilder (or PanelBuilder) for independent candidates plus a judge. Keep DebateBuilder for the multi-round case only if it is built later, possibly as AdjudicationBuilder plus a rounds option. One shared name would blur two different behaviours.

    Illustrative API sketches (proposed, not final)

    All three assume experimental placement. Parameter names are examples only.

    1. Independent reviewers, LLM judge

    workflow = AdjudicationBuilder(
        candidates=[safety, latency, quality],
        judge=arbiter_agent,
    ).build()

    Current alternative: ConcurrentBuilder(participants=[...]).with_aggregator(judge_callback).build(), where the callback calls the judge agent. This works today.

    2. Human judge

    workflow = AdjudicationBuilder(
        candidates=[safety, latency, quality],
        judge="human",        # emits a request_info(DecisionRequest) and waits
        checkpoint_storage=storage,
    ).build()

    Current alternative: a custom judge Executor that calls ctx.request_info(candidates_summary, str) and handles the reply in @response_handler. Resume then uses the existing checkpoint flow.

    3. Bounded debate, then adjudication

    workflow = DebateBuilder(
        debaters=[a, b],
        judge=arbiter_agent,
        rounds=2,
    ).build()

    Current alternative: a WorkflowBuilder loop with round-counter shared state and a final judge executor. It is verbose but supported. It is the best candidate for a sample, or for a builder if demand shows up.

    Semantics to settle (kept small)

    • Outcome: keep the output small. For example, the judge's answer plus a flag for "decided" versus "unresolved" and the candidate outputs. Leave out rich verdict, dissent and tie policy fields until real users ask for them.
    • Unresolved: a first-class result, not an exception or a silent pick. A human judge not answering is a pause, not "unresolved".
    • Cancellation: it should follow existing workflow cancellation. If cancelled mid-round, no verdict is emitted.
    • Checkpointing: the round index and candidate statements must be saved. Any new class that crosses a checkpoint must be registered with register_checkpoint_type (see orchestrations/__init__.py), with a test in tests/test_checkpoint_types.py.
    • Streaming: emit candidate and round outputs as intermediate output, and the verdict as the terminal output. This matches how the other builders use output_from / intermediate_output_from.
    • Placement: experimental in the orchestrations package, exported with the experimental stage marker. The enum lives in core, so a core change to add a feature ID may be needed. That is a maintainer decision.

    Decisions needed from maintainers

    1. Is a sample-first approach (scenarios 1–3 using current primitives) acceptable before any builder?
    2. If a builder is wanted, should the unit be judged independent candidates (AdjudicationBuilder / PanelBuilder), with debate layered on later, or one DebateBuilder covering both?
    3. Which name do you prefer from the table above?
    4. Should "human judge" be built into the reusable capability, or stay a documented pattern using request_info?
    5. Is adding a new experimental feature ID in core acceptable for this?
    6. What is the minimum outcome shape? Suggestion: decision text, decided/unresolved flag, and candidate outputs.

    Once these are answered, a concrete implementation plan (sample, tests, then optionally the builder) can be drafted.

    Generated by Issue Planning for #10171 · copilot · auto · ~21 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷

  8. lss680455-create commented on Nov 9, 2026

    @lss680455-create
    Author

    Understood on both points: this is design comparison only, and we are not opening a non-trivial implementation PR while the direction is being worked out. Nothing below changes code.

    We re-checked the assumptions in the /plan comment against main at 13956ad8c0c52538dc8fb5abae549019fbf2dc83 (2026-11-09), reading the sources rather than the docstrings alone. The snippets below are illustrative and syntax-checked, not executed end to end. Path shorthand: C = python/packages/core/agent_framework, O = python/packages/orchestrations/agent_framework_orchestrations, S = python/samples/03-workflows.

    1. Verification of the plan assumptions

    # Assumption in the plan comment Result Evidence
    1 ConcurrentBuilder fans out to N participants; with_aggregator lets a judge be the aggregator Confirmed O/_concurrent.py:183 class, :217-225 constructor, :276-331 with_aggregator, :83-136 and :196-198 default aggregator, :422 selection, :443-445 fan-out/fan-in wiring
    2 with_request_info already exists Confirmed, but it is not adjudication: it pauses after each selected agent participant so the caller can review and steer that participant's output O/_concurrent.py:333-364, docstring :338-350
    3 ctx.request_info pauses a workflow and checkpoints resume it Confirmed async def request_info(self, request_data: object, response_type: type, *, request_id: str | None = None) at C/_workflows/_workflow_context.py:404; @response_handler C/_workflows/_request_info_mixin.py:134; S/human-in-the-loop/concurrent_request_info.py:102-149,204-209; S/checkpoint/checkpoint_with_human_in_the_loop.py:130-163,238-271
    4 WorkflowBuilder edges, loops and fan-in can express a bounded round loop Partly. There is no loop primitive: a cycle is ordinary edges plus max_iterations C/_workflows/_workflow_builder.py:86-88 (max_iterations argument), :227 add_edge, :508 fan-in, :570-573 cycles explicitly permitted, :866 cap passed to the workflow; used as a loop in S/checkpoint/checkpoint_with_human_in_the_loop.py:194-204
    5 "Scenario 2 is a small custom executor" Confirmed, with two costs: the request and response types must be checkpoint-registered, and the resume loop stays caller code S/checkpoint/checkpoint_with_human_in_the_loop.py:105-173,288; O/__init__.py:135-138
    6 A new type crossing a checkpoint must be registered, with a test Confirmed O/__init__.py:129-146; python/packages/orchestrations/tests/test_checkpoint_types.py:56-93,133-158
    7 ExperimentalFeature lives in core, so a new feature ID may need a core change Corrected. Only the @experimental route needs a core enum member, and no orchestration builder uses it today; they mark usage through a package-local IntEnum plus mark_feature_used, which is not a core change C/_feature_stage.py:43-68,437; O/_feature_usage.py:6-13; O/_concurrent.py:419
    8 Scenario 1 is mostly covered Confirmed, with two gaps: the aggregator receives results only, so the judge prompt has to be reconstructed from a candidate's conversation, and there is no typed decided/unresolved outcome O/_concurrent.py:161-172; precedent for reading the prompt back: S/human-in-the-loop/concurrent_request_info.py:75-79
    9 Cancellation mid-round emits no verdict Unverified. We did not trace the cancellation path in this pass, and the sketches below do not test it —
    10 The existing debate sample "stops after a fixed number of messages" Fixed count confirmed for that sample; we did not verify that no other sample is debate-like S/orchestrations/group_chat_philosophical_debate.py:228,234

    Two of these change the design conversation. First, scenario 3 is not blocked by a missing loop API; the cost is bookkeeping (round state, peer visibility, cap, judge input), so it belongs in a sample before it belongs in a builder. Second, the feature-ID question is smaller than the plan comment implies: if you want the warning, yes, core gains an enum member; if you only want usage attribution, the existing package-local index is enough.

    2. Scenario A: independent reviewers, LLM judge

    Current primitives, roughly as a user would write it today:

    async def adjudicate(results: list[AgentExecutorResponse]) -> Verdict:
        # The aggregator gets results only (O/_concurrent.py:161), so the question
        # is recovered from a candidate's conversation, as the HITL sample does.
        question = next(m.text for m in reversed(results[0].full_conversation) if m.role == "user")
        candidates = [r.agent_response.text for r in results]
        transcript = "\n\n".join(f"[{i}] {c}" for i, c in enumerate(candidates))
        reply = await judge.run(f"Question: {question}\n\n{transcript}\n\nPick an index or say 'unresolved'.")
        return Verdict(decision=reply.text, decided=True, candidates=candidates)  # caller maps "unresolved" today
    
    workflow = ConcurrentBuilder(participants=[safety, latency, quality]).with_aggregator(adjudicate).build()

    If the judged-candidates step were a first-class option:

    workflow = (
        ConcurrentBuilder(participants=[safety, latency, quality], checkpoint_storage=storage)
        .with_judge(judge=arbiter, output_type=Verdict)  # proposed
        .build()
    )

    .with_judge has the same information the callback has above, so it can supply the question explicitly instead of making every user rediscover full_conversation; pass candidates in participant order; set decided=False when the reply is not a usable choice; and designate candidates as intermediate and the judge as terminal output, which _resolve_participant_output_config already supports via extra_output_executors (O/_concurrent.py:427-434). Fan-out, checkpointing and streaming are inherited unchanged.

    3. Scenario B: human judge

    @dataclass
    class VerdictRequest:
        question: str
        candidates: list[str]
    
    @dataclass
    class VerdictResponse:
        choice: int | None  # None means unresolved
    
    class HumanJudge(Executor):
        @handler
        async def ask(self, results: list[AgentExecutorResponse], ctx: WorkflowContext) -> None:
            await ctx.request_info(
                VerdictRequest(
                    question=next(m.text for m in reversed(results[0].full_conversation) if m.role == "user"),
                    candidates=[r.agent_response.text for r in results],
                ),
                VerdictResponse,
            )
    
        @response_handler
        async def decide(
            self, original_request: VerdictRequest, response: VerdictResponse, ctx: WorkflowContext[Never, Verdict]
        ) -> None:
            await ctx.yield_output(
                Verdict(
                    decision=original_request.candidates[response.choice] if response.choice is not None else "",
                    decided=response.choice is not None,
                    candidates=original_request.candidates,
                )
            )
    
    register_checkpoint_type(VerdictRequest)   # both types cross the checkpoint
    register_checkpoint_type(VerdictResponse)
    workflow = (
        ConcurrentBuilder(participants=[safety, latency, quality], checkpoint_storage=storage)
        .with_aggregator(HumanJudge(id="human_judge"))
        .build()
    )

    This is deliberately not with_request_info. That option pauses to review one participant's output before aggregation (O/_concurrent.py:338-350); here the human sees every candidate and produces the terminal verdict, and the caller resumes with workflow.run(stream=True, responses={request_id: VerdictResponse(...)}) as in S/checkpoint/checkpoint_with_human_in_the_loop.py:238-271.

    If the judge slot accepted an Executor, this pattern needs no builder support at all:

    # `judge` may be a plain Executor; the builder does not special-case it.
    workflow = (
        ConcurrentBuilder(participants=[safety, latency, quality], checkpoint_storage=storage)
        .with_judge(judge=HumanJudge(id="human_judge"), output_type=Verdict)  # proposed
        .build()
    )

    The one obligation that falls out: a builder wrapping this stage must not treat a pending request_info as completion, and must not swallow the event.

    4. Scenario C: bounded debate, then adjudication

    A hand-written round loop is graph wiring plus a coordinator; this is schematic, and the routing details are the fiddly part:

    class RoundCoordinator(Executor):
        @handler
        async def on_question(self, question: str, ctx: WorkflowContext[AgentExecutorRequest]) -> None:
            ctx.set_state("question", question)
            ctx.set_state("round", 0)  # ctx state survives a checkpoint; plain attrs do not
            for target in ("pro", "con"):
                await ctx.send_message(AgentExecutorRequest(messages=[Message("user", contents=[question])]), target_id=target)
    
        @handler
        async def on_round(self, results: list[AgentExecutorResponse], ctx: WorkflowContext) -> None:
            round_no = ctx.get_state("round") + 1
            ctx.set_state("round", round_no)
            if round_no < MAX_ROUNDS:
                openings = [r.agent_response.text for r in results]   # round 0 must not see these
                for target, peer in zip(("pro", "con"), reversed(openings)):
                    await ctx.send_message(self._rebuttal_prompt(ctx.get_state("question"), peer), target_id=target)
            else:
                await ctx.send_message(JudgeInput(question=ctx.get_state("question"), statements=round_statements), target_id="judge")
    
    wf = (
        WorkflowBuilder(max_iterations=2 * MAX_ROUNDS + 2, start_executor=coordinator,
                        output_from=[judge], intermediate_output_from=[coordinator])
        .add_fan_in_edges([debater_pro, debater_con], coordinator)   # cycle back into the coordinator
        .add_edge(coordinator, judge)
    )

    The four repeated costs are the round counter surviving checkpoints (ctx.set_state, or on_checkpoint_save for plain attributes, as in S/checkpoint/checkpoint_with_human_in_the_loop.py:165-173), keeping openings mutually invisible, the cap, and assembling judge input from per-round statements. If rounds were later folded into the candidate step:

    workflow = (
        AdjudicationBuilder(candidates=[debater_pro, debater_con], judge=arbiter,
                            rounds=2, checkpoint_storage=storage).build()  # proposed
    )

    We would only split out DebateBuilder if users need controls that rounds cannot express: per-round prompts, turn order, explicit rebuttal targets.

    5. Minimal reusable capability

    Option Surface Cost Risk
    1. Samples only none 3 samples, no API users copy plumbing; the question-recovery and decided/unresolved conventions stay undocumented
    2. ConcurrentBuilder.with_judge(...) one method, one outcome type small: reuses fan-out, aggregator, checkpoint, streaming, output designation slight growth of an existing builder; make it experimental if you prefer
    3. AdjudicationBuilder one class, one registration, one test row larger: new class, exports, checkpoint types, docs a second way to express fan-out + decide; harder to keep aligned with ConcurrentBuilder

    Our order: 1, then 2, then 3, with DebateBuilder only after multi-round demand is demonstrated. Samples first because scenario 3 is the only one with real boilerplate and a sample is the cheapest way to find out whether anyone hits it; with_judge next because it is the smallest change that removes the two sharp edges in scenario A without a new class; a standalone builder only if naming and discoverability turn out to matter more than surface area.

    6. Naming

    Name Verdict Why
    AdjudicationBuilder recommended Names the act and the outcome, fits independent candidates, and still describes the final stage of a debate
    PanelBuilder acceptable second Shorter, but names the input (a panel) rather than what happens
    DebateBuilder only for multi-round Implies exchange and rebuttal; misleading when candidates never see each other
    ReviewBuilder no Collides with the existing review meaning of with_request_info (O/_concurrent.py:342) and with code review
    JudgedBuilder no Past participle with no object; does not extend to a human judge
    DeliberationBuilder no Implies joint deliberation, which is the opposite of independent candidates; vague outcome

    7. Minimum outcome shape

    @dataclass
    class Verdict:
        decision: str          # chosen candidate text, or the judge's stated outcome
        decided: bool          # False means unresolved; not an exception, not a silent pick
        candidates: list[str]  # participant order

    No verdict schema beyond this: no confidence, dissent, vote counts or tie-policy fields. Tie handling stays a judge-prompt concern until users ask for it. Cancellation should follow the existing workflow path; we have not verified its event behaviour and will not design around it here. Checkpointing stores the round index in ctx.set_state and the verdict type is registered like any other; streaming labels candidates intermediate and the verdict output at build time, matching O/_concurrent.py:223-237 and C/_workflows/_workflow_context.py:354-366.

    8. Human judge placement

    Recommendation: documented pattern, not built into the builder. A human judge is an interaction protocol, not an orchestration topology; the pause, response routing and checkpoint plumbing already exist and are type-driven (request_info, @response_handler, register_checkpoint_type). Building it in would put package-owned request/response types into the checkpoint surface for a pattern that is roughly thirty lines, and it would not share semantics with the LLM judge beyond the Verdict shape. Letting the judge slot accept an Executor is enough.

    9. Your six decisions

    1. Sample-first: yes. We would contribute samples for all three scenarios as a small docs/samples PR if that is welcome.
    2. Unit: judged independent candidates first; rounds as an option later, not a second builder.
    3. Name: AdjudicationBuilder, PanelBuilder if you prefer shorter; DebateBuilder reserved for round-based controls.
    4. Human judge: documented pattern; not built in.
    5. New feature ID: not needed for samples. If the capability ships, the package-local FeatureIndex covers attribution without a core edit; @experimental is your call and does require a core enum member.
    6. Outcome shape: decision, decided, candidates, and nothing else for v1.

    These are yours to decide; we are not treating any of the above as settled. If sample-first is the path you pick, we can open a samples-only PR with the three scenarios and the human-judge pattern, and keep it out of the package. Until you agree on a direction we will not touch implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

.NETUsage: [Issues, PRs], Target: .NetorchestrationUsage: [Issues, PRs], Target: multi-agent orchestration (high-level patterns)pythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions