Repository navigation
Description
Problem
agent_framework_orchestrations ships five peer builder patterns — Sequential, Concurrent,
Handoff, GroupChat, Magentic (agent_framework_orchestrations/__init__.py:5-10,
python/packages/orchestrations/README.md:11-93). None of them represents opposing
positions or produces a decision. Every pattern either merges outputs
(Concurrent), chains them (Sequential), routes control (Handoff), or runs an open-ended
conversation whose stopping point is "looks done" (GroupChat, Magentic).
The gap is visible in this repository today. The one debate-shaped artifact that exists is a
sample built out of a group chat:
python/samples/03-workflows/orchestrations/group_chat_philosophical_debate.py:226-234
constructsGroupChatBuilder(...)with
termination_condition=lambda messages: sum(1 for msg in messages if msg.role == "assistant") >= 10.
There is no verdict, no ballot, no tie handling, and no separation between advocates and
an adjudicator; the "conclusion" is simply the last thing the moderator says.- That sample is load-bearing enough that it is currently being repaired in
PR #10085 (an
intermediate_output_fromargument bug). Hand-rolled orchestration in a sample is
exactly the shape of code that drifts and has no unit tests.
Users who need a checked decision therefore re-implement the same state machine — openings
that cannot see peers, N rounds of rebuttal, a ballot, a tie rule, a structured verdict —
in prompts, and get non-reproducible results and no checkpointability. Multi-agent debate is
a well-established technique (Du et al., 2023, arXiv:2305.14325);
the framework has no first-class pattern for it.
Use Case and Evidence
- Release / policy review gates. Three reviewers with different mandates (safety, cost
and latency, answer quality) argue over whether a change ships. The caller needs a
machine-readabledecisionplus the recorded dissent, not a chat transcript it has to
parse with an LLM afterwards. Today this is staffed by a group chat whose output shape
varies run to run — unusable as a CI gate. - Contested analytical questions. "Is this account takeover or a false positive?" Two
or three hypotheses compete. Sequential cannot compare them, Concurrent produces two
answers and no resolution, Magentic replans a task rather than judging positions.
The user today writes a custom aggregator prompt and has no way to stop early, bound the
cost, or resume after a failure. - Auditable adjudication with human escalation. Regulated scenarios need the tie case
to be explicit: either the judge decides (and the tie is recorded), or a human is asked,
or the run reports "unresolved" instead of silently picking an answer.
A duplicate search across issues and pull requests found no existing request for a
debate/consensus orchestration pattern; the nearest items and why they are not this are
listed in Acknowledgements.
Desired Outcome
A DebateBuilder in agent_framework_orchestrations that:
- takes N debaters plus one judge, runs a bounded sequence of
independent openings → cross-examination rounds → adjudication; - returns the judge's decision as a structured value, with dissent and round/truncation
metadata, on the workflow's terminalAgentResponse.value; - offers an explicit, documented tie policy instead of leaving ties to prompt luck;
- behaves like the existing five builders for everything else: same
termination_condition
type,max_roundscap, request-info/HITL flag, checkpointing,output_from/
intermediate_output_fromselection, streaming vs non-streaming output shape, and the
same construction-timeValueError/TypeErrorsemantics; - is strictly additive: no change to the existing five builders or to the workflow runtime.
Proposed Direction
Builder
DebateBuilder(
*,
name: str | None = None,
participants: Sequence[SupportsAgentRun | Executor], # debaters only, judge excluded
judge: SupportsAgentRun | Executor, # adjudicator
rounds: int = 2, # cross-exam rounds after the openings
voting: Literal["judge_only", "majority"] = "judge_only",
tie_policy: Literal["judge_decides", "return_dissent", "escalate_to_human"] = "judge_decides",
max_rounds: int | None = None, # hard cap; coerced to >= 1, as base does
termination_condition: TerminationCondition | None = None,
checkpoint_storage: CheckpointStorage | None = None,
output_from=UNSET, # same contract as the other builders
intermediate_output_from=None,
)Fluent methods, matching the existing builders' style
(_group_chat.py:822 with_termination_condition, :863 with_max_rounds,
:881 with_checkpointing, :913 with_request_info;
_handoff.py:956/:1005, _magentic.py:1568/:1616):
.with_termination_condition(cond) -> DebateBuilder
.with_max_rounds(n) -> DebateBuilder
.with_checkpointing(storage) -> DebateBuilder
.with_request_info(agents=None) -> DebateBuilder
.build() -> WorkflowTerminationCondition is deliberately reused unchanged, not redefined:
Callable[[list[Message]], bool | Awaitable[bool]] (_base_group_chat_orchestrator.py:58).
Example
from agent_framework import AgentResponse
from agent_framework_orchestrations import DebateBuilder, DebateVerdict
workflow = (
DebateBuilder(
participants=[safety_reviewer, latency_reviewer, quality_reviewer],
judge=arbiter,
rounds=2, # openings + 2 cross-examination rounds
voting="judge_only",
tie_policy="judge_decides",
max_rounds=8,
checkpoint_storage=storage,
)
.with_request_info(agents=[safety_reviewer]) # optional HITL, same flag as other builders
.build()
)
events = await workflow.run("Ship the model-card change this quarter? Answer yes or no, with reasons.")
response = events.get_outputs()[0]
assert isinstance(response, AgentResponse)
verdict: DebateVerdict | None = response.value
print(verdict.decision, verdict.confidence, [d.position for d in verdict.dissent])(events.get_outputs() / AgentResponse usage mirrors
python/samples/03-workflows/orchestrations/concurrent_agents.py:76-84.)
Output schema
Terminal output is an AgentResponse (as with every other builder), whose value carries:
class DebateDissent(BaseModel):
position: str # the position not selected
held_by: list[str] # participant names that held it
summary: str # one or two sentences, written by the judge
class DebateVerdict(BaseModel):
decision: str
rationale: str
confidence: float | None = None
dissent: list[DebateDissent] = []
rounds_completed: int
truncated: bool = False # max_rounds / termination_condition cut the debate short
unresolved: bool = False # tie_policy="return_dissent" and the tie did not break
vote_counts: dict[str, int] | None = None # only when voting="majority"AgentResponse supports a structured value (agent_framework/_types.py:3729), and the
judge message is also appended to messages with author_name=judge, so text-only callers
and workflow.as_agent() keep working. Per-round debater turns are exposed as intermediate
output through the existing selection contract (python/packages/orchestrations/README.md:95-130),
not as a new event family in v1.
Checkpoint contract
State is stored through the shared OrchestrationState container
(_orchestration_state.py:33-54); pattern-specific metadata carries the phase
(opening / cross_exam / ballot / adjudicate), the round index, the per-debater
statements, and the tally. Any new envelope dataclass that crosses a checkpoint boundary
must be registered via register_checkpoint_type in __init__.py and covered by
tests/test_checkpoint_types.py — the restricted unpickler auto-trusts only the
agent_framework. prefix, not sibling distributions (__init__.py:117-146,
tests/test_checkpoint_types.py:1-15). This is a hard requirement, not a nicety.
Error semantics (same shape as the existing builders)
| Condition | Behaviour |
|---|---|
participants=[], duplicate participant ids, judge also passed in participants |
ValueError at construction (cf. _sequential.py:144-163; existing tests tests/test_concurrent.py:46-56) |
rounds < 1, unknown voting / tie_policy literal, max_rounds < 1 |
ValueError (rounds coerced to 1 like max_rounds, _base_group_chat_orchestrator.py:164) |
participant is neither SupportsAgentRun nor Executor |
TypeError (mirrors _sequential.py:245) |
output_from / intermediate_output_from overlap, unknown participant, wrong "all"/"all_other" use |
existing validation errors, unchanged (README.md:118-127) |
| a debater fails during a round | recorded in the verdict; debate continues while ≥ 2 debaters produced statements, otherwise the orchestrator yields a completion message (not an exception) |
| judge output cannot be parsed | bounded retry, then a completion message with value=None; the run still completes rather than raising |
Failure-as-completion-message follows the base orchestrator's established behaviour
(_base_group_chat_orchestrator.py:353-371, :519-537) instead of inventing an exception
contract.
Semantics
Phases and roles. Round 0 is openings: every debater answers independently and never
sees a peer's message (fan-out is parallel, like ConcurrentBuilder's dispatcher/fan-out,
_concurrent.py:183-188). Round r is cross-examination: each debater receives the task
plus a bounded digest of opposing statements from round r−1 — not the full transcript — and
must defend or concede. The judge never speaks during the debate; it is invoked once at
adjudication with the task, the ordered transcript, and (if enabled) the tally. The judge is
required to be a separate agent and is rejected if it also appears in participants.
Voting.
judge_only(default): no ballot; the judge decides and writes the dissent. Debaters only
need to be ordinary chat agents.majority: after the last cross-exam round each debater submits a short ballot as
structured output (this is the one mode that requires debater structured output; a
ballot that fails to parse is dropped and counted in the tally metadata). A strict majority
fixes the decision and the judge only writes the rationale and dissent; it does not
override. No strict majority →tie_policy.
Ties and no-consensus.
judge_decides(default): the judge picks; the fact that the vote was tied stays visible
indissentandvote_counts, whileunresolvedremainsFalsebecause a decision was
produced.return_dissent: no decision is forced; the verdict hasunresolved=True,decision="",
and every tied position indissent. This is the option for pipelines that must not guess.escalate_to_human: emits arequest_infoevent (the existing HITL mechanism,
_orchestration_request_info.py:187AgentApprovalExecutor) carrying the tied options;
the run resumes with the human's choice, which becomes the decision.
Budgets and truncation. rounds bounds the adversarial phase; max_rounds, if set,
caps total coordination rounds with the same coercion as the base class
(_base_group_chat_orchestrator.py:164, :498-517). Reaching a cap jumps to
adjudication and sets truncated=True — a deliberate difference from GroupChat, where the
cap yields a generic completion message, because a truncated debate still has a useful
verdict. A user-supplied termination_condition is evaluated after each round and, when
true, also short-circuits to adjudication (truncated=True), never "return the raw
transcript".
Failure and cancellation. Debater failures are isolated per round; the debate degrades
to fewer stances and records it. Cancellation propagates as usual; because state is
checkpointed at phase boundaries, a cancelled run resumes at the start of the phase in
progress. No partial verdict is ever emitted.
HITL. .with_request_info(agents=[...]) reuses the existing flag verbatim: the workflow
pauses after the selected debaters' turns so a human can steer a stance
(_orchestration_request_info.py:19 resolve_request_info_filter). Judge-side intervention
is intentionally limited to the tie-escalation path in v1; a general "approve the verdict"
review step is listed as an open question rather than smuggled into the first cut.
Boundary against GroupChat and Magentic. GroupChat is an open-ended, orchestrator-chosen
speaking order with no distinction between advocate and decider and no structured decision;
use it for "let them talk". Debate is fixed-phase, bounded, adversarial-then-adjudicated, and
its contract is a verdict object. Magentic maintains task/progress ledgers and replans a
task; it answers "what should be done next", not "which position is right", and its
MagenticProgressLedger invariants have no place for dissent. Concurrent merges parallel
answers; it has no rebuttal round and no tie semantics.
Comparison with the five existing patterns
| Pattern | Control flow | Rounds | Structured decision | Dissent / ties | Checkpoint | HITL |
|---|---|---|---|---|---|---|
Sequential (_sequential.py:66) |
fixed chain | 1 pass | no | no | yes | with_request_info |
Concurrent (_concurrent.py:183) |
fan-out + aggregate | 1 | no | no | yes | with_request_info |
Handoff (_handoff.py:643) |
agents route control | unbounded | no | no | yes | request/response |
GroupChat (_group_chat.py:619) |
selector/orchestrator agent | unbounded (max_rounds) |
optional (AgentOrchestrationOutput only drives the loop, _group_chat.py:263-283) |
no | yes | with_request_info |
Magentic (_magentic.py:1418) |
manager + ledgers | unbounded (max_round_count) |
no (task ledger) | no | yes | plan review |
| Debate (proposed) | openings → cross-exam → adjudicate | bounded | yes (DebateVerdict) |
yes (ballot, tie policy) | yes | with_request_info + tie escalation |
Alternatives Considered
- GroupChatBuilder + custom
selection_func+termination_condition. This is what the
repository does today (the debate sample). Rejected: (a) openings cannot be made
peer-blind — GroupChat broadcasts every message to every participant
(_base_group_chat_orchestrator.py:410-440), so round 0 is not independent; (b) there is no
ballot or tie rule; (c) termination by message count is the only cheap criterion
(group_chat_philosophical_debate.py:228) and it is not a decision; (d) the terminal
output is whatever the last speaker said, with no structured verdict to gate on. - Ship it as a sample only. Rejected: the sample already exists and is the evidence of
the problem. Tie handling, ballot parse failures, truncation, mid-debate
checkpoint/restore and "no consensus" are exactly the behaviours that need unit tests and
a stable contract; samples carry neither, and PR Python: Select all other participants in the group chat debate sample #10085 shows they are not maintained as
contracts. ConcurrentBuilder+ LLM aggregator callback. Closest workaround, but single-round:
no rebuttal, no state machine, and the callback receives only
list[AgentExecutorResponse](_concurrent.py:139-150), so phase structure cannot be
reconstructed. There is also no hook point between rounds.- A
debatemode onMagenticBuilder. Rejected: Magentic's documented contract is
ledger-driven planning (_magentic.py:1440-1500); reusing it would either break those
semantics or bolt a second state machine onto a class that already owns one. Larger diff,
higher review risk, worse discoverability than a sibling builder. - Handoff-based adversarial routing. Handoff routes control between agents; it has no
notion of "everyone answers, then one decides", and no decision record.
Scope and Non-Goals
In scope
- new
_debate.pywithDebateBuilder, aDebateOrchestrator(BaseGroupChatOrchestrator),
and theDebateVerdict/DebateDissentmodels; - exports and checkpoint-type registration in
agent_framework_orchestrations/__init__.py; - a new package-local feature index (
FeatureIndex.ORCHESTRATION_DEBATE) in
_feature_usage.py:6-13plus its row indocs/specs/feature-usage-bit-registry.md:149
(slots 37–47 are already reserved for orchestration growth); - unit tests, one sample, package/samples README entries.
Out of scope (explicit non-goals)
- any change to the workflow runtime or to the existing five builders — no behaviour, no
signature, no default changes; - .NET parity (
Microsoft.Agents.AI.Workflows) — to be tracked separately if wanted; - shipping debate prompts or opinionated stances; the builder orchestrates, it does not
prescribe persona text; - claim extraction, fact verification, citation checking, or judge-quality evaluation;
- token-budget accounting, dynamic participant join/leave, LLM-judged "consensus reached"
termination (that stays GroupChat territory); - relitigating the existing
output_from/intermediate_output_fromsemantics.
Testing and Sample Plan
Unit tests run offline with stub agents/executors, following the existing pattern
(tests/test_concurrent.py:27-43 _FakeAgentExec, no model calls):
- construction: empty participants, duplicate ids, judge in participants,
rounds < 1,
invalidvoting/tie_policy,TypeErrorfor a non-agent/executor, inherited
output-selection validation errors; - phase order: openings are peer-blind and issued in parallel; cross-exam round r sees
only round r−1; judge receives the full ordered transcript; - output contract:
AgentResponse.valueis aDebateVerdict; judge message present with
author_name;intermediate_output_from=[...]surfaces debater turns asintermediate
events and not asoutputevents (extend
tests/test_orchestration_intermediate_vs_terminal.py); - voting/ties: strict majority fixes the decision; no majority under each of the three
tie policies; unparseable ballot is dropped and recorded; - budgets:
max_roundsand an early-truetermination_conditionboth jump to
adjudication withtruncated=True; - failures: one debater raising leaves the debate runnable; too few debaters yields the
completion message; unparseable judge output after retries yields a completed run with
value=None; - checkpointing: save mid-debate, rebuild with the same participant names, restore, finish;
add the new envelopes totests/test_checkpoint_types.py:56-75; - streaming: terminal output is a single
AgentResponseUpdate, mirroring the base class's
mode-aware completion (_base_group_chat_orchestrator.py:373-395); - feature index parity: the repo-wide validation test asserts exact table parity
(docs/specs/feature-usage-bit-registry.md:282-284), so the new row must be added with the
enum.
Sample: one file, e.g.
python/samples/03-workflows/orchestrations/debate_model_review.py — three reviewers with
different mandates, one judge, rounds=2, printing decision, confidence, dissent, and
demonstrating max_rounds truncation in a second run. Style follows the existing orchestration
samples (Foundry chat client, logging.basicConfig). Update
python/packages/orchestrations/README.md (pattern list + usage block) and
python/samples/03-workflows/orchestrations/README.md:56 (sample table; the "patterns like
Sequential, Concurrent, Handoff, GroupChat, and Magentic" sentence at :136).
Commands, from python/ (CONTRIBUTING.md:269-272):
uv run poe test -A -m "not integration"
uv run poe syntax
uv run poe mypy # per-package task, python/packages/orchestrations/pyproject.toml:86-88Compatibility
- Additive only. New module, new public names, new feature index; no signature or
behavioural change to the existing five builders, so the Griffe comparison has nothing to
report.agent-framework-orchestrationsisreleased
(python/PACKAGE_STATUS.md:62), so the Griffe check applies to it (CONTRIBUTING.md:136-165)
— additions are not breaking changes. - Lazy namespace parity. Public names are also added to the core lazy re-export list
python/packages/core/agent_framework/orchestrations/__init__.py:21-69and to
__init__.pyi, sofrom agent_framework.orchestrations import DebateBuilderworks like the
other builders. This is the same stub/runtime parity issue currently being fixed in
PR #10170, and is additive. - Checkpoints. Old checkpoints are unaffected: new types are registered in addition, and
no existing registration changes (__init__.py:129-146). - Feature-usage bitmask. The new index takes an already-reserved orchestration slot
(docs/specs/feature-usage-bit-registry.md:149), so no renumbering and no decoder change. - Hosting.
agent-framework-hostingmatches builders structurally via a zero-argument
build()protocol (python/packages/hosting/agent_framework_hosting/_state.py:47-52), so
DebateBuildercomposes there without changes.
Language
Python (agent-framework-orchestrations). .NET parity deliberately not included.
Contribution Intent
I am willing to implement this (with tests and sample) after the direction is agreed.
AI Assistance Disclosure
Per AI_CONTRIBUTIONS.md:27-32, material AI assistance is disclosed here: repository
analysis, the API sketch, and the text of this issue were drafted with AI assistance. The
issue author owns the design and the implementation, and will answer review questions
directly. No prompts, credentials, or private data are included.
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I will wait for explicit maintainer agreement (
ready-for-implementationor an
equivalent comment,CONTRIBUTING.md:75-81) before starting a non-trivial change.
Duplicate check performed on 2026-11-08 across issues and pull requests (open and closed),
searching debate, consensus, voting, jury, panel, adjudicat, devil's advocate,
and in:title orchestration|builder|pattern. Results: no open or closed issue/PR proposes a
debate or consensus orchestration pattern; no assignee holds that direction. The nearest
items and why they are not this:
- PR #10085 (open, no assignee) —
fixesintermediate_output_from="all"→"all_other"in the debate sample. It keeps the
GroupChat-based sample working; it adds no pattern, verdict, ballot, or tie rule. - Issue #3572 (open, no assignee) —
Handoff pattern support for the GitHub Copilot agent: provider/routing work, unrelated to
adjudication. - PR #10170 (open) — stub/runtime
parity for the lazyagent_framework.orchestrationsnamespace. Not a new pattern, but this
proposal follows its outcome for the export surface. - Issue Python: Are there plans to add workflow orchestration capabilities to DevUI? #1655 (closed) — DevUI orchestration UI, and .NET: [Bug]: Terminal Condition is not working in Magentic Orchestration #7595 / .NET: Python: Magentic orchestration stops after first query completion and does not continue to the next query #5657 — Magentic orchestration
bugs. Neither proposes a new pattern.
Open Questions for Maintainers
- Should an exhausted
max_roundsjump to adjudication (truncated=True) as proposed, or
keep GroupChat's generic completion message for consistency? - Is
voting="majority"(which makes debater structured output a requirement) wanted in the
first cut, or should v1 shipjudge_onlyonly and add ballots later? - Should a judge-verdict review step (
request_infobefore the verdict becomes workflow
output) be in scope, similar in spirit to Magentic's plan review
(_magentic.py:1568)? - Name:
DebateBuilder/orchestration.debate(index 37) — or would maintainers prefer
AdjudicationBuilderto keep "debate" out of the public API name?
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
[-][Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/-][+]Python: [Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/+][-]Python: [Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/-][+].NET: [Feature]: Add a DebateBuilder orchestration pattern (debate → vote → adjudicate)[/+]eavanvalkenburg commentedon Nov 9, 2026
We are open to exploring this as an experimental capability in the existing Python orchestration package. That is not yet a commitment to a stable
DebateBuilderAPI or to implementation timing.Before choosing an API, we would like to compare naming options and see short sample code for several scenarios. Please consider both an LLM as judge and a human as judge, including whether adjudication is useful independently of a multi-round debate. The examples should make the differences between those variants clear.
We will use planning to compare those options and the existing workflow-composition approach before agreeing on an implementation direction. Please hold off on a non-trivial implementation PR while that direction is still being worked out.
eavanvalkenburg commentedon Nov 9, 2026
/plan Explore this as an experimental capability in the existing Python orchestration package, not a stable API commitment or authorization to implement. Produce design options before an implementation plan. Compare naming options such as DebateBuilder, AdjudicationBuilder, ReviewBuilder, and better alternatives; explain which names fit judging independent candidates versus multi-round debate. Include concise, explicitly proposed API snippets for several scenarios: (1) independent reviewers with an LLM as judge, (2) a human as judge using the existing request_info and checkpoint/resume primitives, and (3) bounded multi-round debate followed by adjudication. Compare each with a small custom workflow or sample built from current supported primitives, and identify the minimal reusable framework capability, if any, rather than assuming a new builder is required. Cover experimental-stage placement in agent-framework-orchestrations, outcome and unresolved-decision semantics, cancellation, checkpointing, and streaming compatibility without prematurely fixing a large verdict schema or policy inventory. Keep .NET parity out of scope. Do not implement changes or open an implementation PR; return naming alternatives, trade-offs, illustrative snippets, and the maintainer decisions needed next.
github-actions commentedon Nov 9, 2026
Summary
The issue asks for a way to run opposing positions and get one recorded decision. Today
agent_framework_orchestrationshas Sequential, Concurrent, Handoff, GroupChat and Magentic. None has a judge or an explicit "unresolved" outcome. The only debate-like sample isgroup_chat_philosophical_debate.py, which stops after a fixed number of messages.The maintainer direction is to treat this as experimental, compare names and scenarios, and not commit to a stable
DebateBuilderyet. Below is design exploration only. Nothing here has been run, and no changes are made.Recommendation
Start with a sample, and add code only if the sample shows a gap. I confirmed from the repo that these pieces already exist:
ConcurrentBuilderfans out to N participants and haswith_aggregator(...)(_concurrent.py). A judge can be the aggregator for "independent reviewers + judge".ctx.request_info(...)(_workflow_context.py) pauses a workflow for a human reply. Checkpointing can resume it. Seehuman-in-the-loop/andhandoff_with_tool_approval_checkpoint_resume.py.with_request_info(...)already exists on the builders.WorkflowBuilderedges, loops and fan-in can express a bounded round loop.ExperimentalFeatureis defined in core (agent_framework/_feature_stage.py).Scenario 1 is mostly covered already. Scenario 2 is a small custom executor. Scenario 3 is the only one where hand-written wiring is repetitive: openings that cannot see peers, N rebuttal rounds, a cap, and a judge.
Likely minimal reusable capability: a "judged candidates" step, not a full debate builder. It would take N candidate outputs, call a judge, and return the judge's decision or an explicit "unresolved" outcome. Multi-round debate can then be a sample or a thin layer on top. This is a hypothesis for maintainers to confirm.
Naming options
DebateBuilderAdjudicationBuilderReviewBuilderJudgedBuilder/PanelBuilderDeliberationBuilderSuggested split:
AdjudicationBuilder(orPanelBuilder) for independent candidates plus a judge. KeepDebateBuilderfor the multi-round case only if it is built later, possibly asAdjudicationBuilderplus a rounds option. One shared name would blur two different behaviours.Illustrative API sketches (proposed, not final)
All three assume experimental placement. Parameter names are examples only.
1. Independent reviewers, LLM judge
Current alternative:
ConcurrentBuilder(participants=[...]).with_aggregator(judge_callback).build(), where the callback calls the judge agent. This works today.2. Human judge
Current alternative: a custom judge
Executorthat callsctx.request_info(candidates_summary, str)and handles the reply in@response_handler. Resume then uses the existing checkpoint flow.3. Bounded debate, then adjudication
Current alternative: a
WorkflowBuilderloop with round-counter shared state and a final judge executor. It is verbose but supported. It is the best candidate for a sample, or for a builder if demand shows up.Semantics to settle (kept small)
register_checkpoint_type(seeorchestrations/__init__.py), with a test intests/test_checkpoint_types.py.output_from/intermediate_output_from.Decisions needed from maintainers
AdjudicationBuilder/PanelBuilder), with debate layered on later, or oneDebateBuildercovering both?request_info?Once these are answered, a concrete implementation plan (sample, tests, then optionally the builder) can be drafted.
lss680455-create commentedon Nov 9, 2026
Understood on both points: this is design comparison only, and we are not opening a non-trivial implementation PR while the direction is being worked out. Nothing below changes code.
We re-checked the assumptions in the
/plancomment againstmainat13956ad8c0c52538dc8fb5abae549019fbf2dc83(2026-11-09), reading the sources rather than the docstrings alone. The snippets below are illustrative and syntax-checked, not executed end to end. Path shorthand:C=python/packages/core/agent_framework,O=python/packages/orchestrations/agent_framework_orchestrations,S=python/samples/03-workflows.1. Verification of the plan assumptions
ConcurrentBuilderfans out to N participants;with_aggregatorlets a judge be the aggregatorO/_concurrent.py:183class,:217-225constructor,:276-331with_aggregator,:83-136and:196-198default aggregator,:422selection,:443-445fan-out/fan-in wiringwith_request_infoalready existsO/_concurrent.py:333-364, docstring:338-350ctx.request_infopauses a workflow and checkpoints resume itasync def request_info(self, request_data: object, response_type: type, *, request_id: str | None = None)atC/_workflows/_workflow_context.py:404;@response_handlerC/_workflows/_request_info_mixin.py:134;S/human-in-the-loop/concurrent_request_info.py:102-149,204-209;S/checkpoint/checkpoint_with_human_in_the_loop.py:130-163,238-271WorkflowBuilderedges, loops and fan-in can express a bounded round loopmax_iterationsC/_workflows/_workflow_builder.py:86-88(max_iterationsargument),:227add_edge,:508fan-in,:570-573cycles explicitly permitted,:866cap passed to the workflow; used as a loop inS/checkpoint/checkpoint_with_human_in_the_loop.py:194-204S/checkpoint/checkpoint_with_human_in_the_loop.py:105-173,288;O/__init__.py:135-138O/__init__.py:129-146;python/packages/orchestrations/tests/test_checkpoint_types.py:56-93,133-158ExperimentalFeaturelives in core, so a new feature ID may need a core change@experimentalroute needs a core enum member, and no orchestration builder uses it today; they mark usage through a package-localIntEnumplusmark_feature_used, which is not a core changeC/_feature_stage.py:43-68,437;O/_feature_usage.py:6-13;O/_concurrent.py:419O/_concurrent.py:161-172; precedent for reading the prompt back:S/human-in-the-loop/concurrent_request_info.py:75-79S/orchestrations/group_chat_philosophical_debate.py:228,234Two of these change the design conversation. First, scenario 3 is not blocked by a missing loop API; the cost is bookkeeping (round state, peer visibility, cap, judge input), so it belongs in a sample before it belongs in a builder. Second, the feature-ID question is smaller than the plan comment implies: if you want the warning, yes, core gains an enum member; if you only want usage attribution, the existing package-local index is enough.
2. Scenario A: independent reviewers, LLM judge
Current primitives, roughly as a user would write it today:
If the judged-candidates step were a first-class option:
.with_judgehas the same information the callback has above, so it can supply the question explicitly instead of making every user rediscoverfull_conversation; pass candidates in participant order; setdecided=Falsewhen the reply is not a usable choice; and designate candidates asintermediateand the judge as terminaloutput, which_resolve_participant_output_configalready supports viaextra_output_executors(O/_concurrent.py:427-434). Fan-out, checkpointing and streaming are inherited unchanged.3. Scenario B: human judge
This is deliberately not
with_request_info. That option pauses to review one participant's output before aggregation (O/_concurrent.py:338-350); here the human sees every candidate and produces the terminal verdict, and the caller resumes withworkflow.run(stream=True, responses={request_id: VerdictResponse(...)})as inS/checkpoint/checkpoint_with_human_in_the_loop.py:238-271.If the judge slot accepted an
Executor, this pattern needs no builder support at all:The one obligation that falls out: a builder wrapping this stage must not treat a pending
request_infoas completion, and must not swallow the event.4. Scenario C: bounded debate, then adjudication
A hand-written round loop is graph wiring plus a coordinator; this is schematic, and the routing details are the fiddly part:
The four repeated costs are the round counter surviving checkpoints (
ctx.set_state, oron_checkpoint_savefor plain attributes, as inS/checkpoint/checkpoint_with_human_in_the_loop.py:165-173), keeping openings mutually invisible, the cap, and assembling judge input from per-round statements. If rounds were later folded into the candidate step:We would only split out
DebateBuilderif users need controls thatroundscannot express: per-round prompts, turn order, explicit rebuttal targets.5. Minimal reusable capability
ConcurrentBuilder.with_judge(...)AdjudicationBuilderConcurrentBuilderOur order: 1, then 2, then 3, with
DebateBuilderonly after multi-round demand is demonstrated. Samples first because scenario 3 is the only one with real boilerplate and a sample is the cheapest way to find out whether anyone hits it;with_judgenext because it is the smallest change that removes the two sharp edges in scenario A without a new class; a standalone builder only if naming and discoverability turn out to matter more than surface area.6. Naming
AdjudicationBuilderPanelBuilderDebateBuilderReviewBuilderwith_request_info(O/_concurrent.py:342) and with code reviewJudgedBuilderDeliberationBuilder7. Minimum outcome shape
No verdict schema beyond this: no confidence, dissent, vote counts or tie-policy fields. Tie handling stays a judge-prompt concern until users ask for it. Cancellation should follow the existing workflow path; we have not verified its event behaviour and will not design around it here. Checkpointing stores the round index in
ctx.set_stateand the verdict type is registered like any other; streaming labels candidatesintermediateand the verdictoutputat build time, matchingO/_concurrent.py:223-237andC/_workflows/_workflow_context.py:354-366.8. Human judge placement
Recommendation: documented pattern, not built into the builder. A human judge is an interaction protocol, not an orchestration topology; the pause, response routing and checkpoint plumbing already exist and are type-driven (
request_info,@response_handler,register_checkpoint_type). Building it in would put package-owned request/response types into the checkpoint surface for a pattern that is roughly thirty lines, and it would not share semantics with the LLM judge beyond theVerdictshape. Letting the judge slot accept anExecutoris enough.9. Your six decisions
AdjudicationBuilder,PanelBuilderif you prefer shorter;DebateBuilderreserved for round-based controls.FeatureIndexcovers attribution without a core edit;@experimentalis your call and does require a core enum member.decision,decided,candidates, and nothing else for v1.These are yours to decide; we are not treating any of the above as settled. If sample-first is the path you pick, we can open a samples-only PR with the three scenarios and the human-judge pattern, and keep it out of the package. Until you agree on a direction we will not touch implementation.