Skip to content

[WIP] .NET: Add Microsoft.Agents.AI.TypeSafe with Jev as an IDecisionClient and a tool (alpha) - #8859

Draft
Roger Barreto (rogerbarreto) wants to merge 18 commits into
microsoft:mainfrom
rogerbarreto:maf-jev-chatclient
Draft

Roger Barreto (rogerbarreto) wants to merge 18 commits into
microsoft:mainfrom
rogerbarreto:maf-jev-chatclient

Conversation

@rogerbarreto

@rogerbarreto Roger Barreto (rogerbarreto) commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Draft. The Microsoft.Extensions.AI (MEAI) decision API is still in review: the provider-neutral abstractions (Layer 1) in dotnet/extensions#7795, and decision interoperability (Layer 2) in dotnet/extensions#7796. Rather than wait, this PR ships a public copy of the proposed API in Microsoft.Agents.AI.Abstractions/MEAI, in the Microsoft.Extensions.AI namespace and marked MEAI001, mirroring its upstream placement in Microsoft.Extensions.AI.Abstractions. Code written against it compiles against the shipped API once the folder is deleted. Agent Framework did the same with earlier proposed MEAI types (#279, #426), removing them once MEAI shipped them (#427, #676).

Motivation & Context

Decision models answer typed questions about a state instead of writing text. A Choice picks one of a set of named alternatives, a Score places the state on an ordered rubric, and a Binary question gives the probability that the answer is yes. Every answer comes with probabilities, and a call is fast and inexpensive. That suits the classification, routing, and checking decisions that agents make on every turn. Several providers now offer them:

  • Jev, the TypeSafe System One model.
  • Azure OpenAI Decisions (/openai/v1/decisions with gpt-6-luna).
  • Gemini.

MEAI is standardizing them behind IDecisionClient, but that API has not shipped. This PR makes the proposed API usable from Agent Framework now:

  • Jev provider: JevDecisionClient, an IDecisionClient for Jev.
  • Samples: they use decisions at the agent level, the way the MEAI proposal presents them: as a loop judge, as an agent tool (AsAIFunction), and as a router (RoutingChatClient).
  • Jev as a tool: JevAIToolBuilder gives an agent the Jev API as a tool, with questions written by the agent's model.

There is no IChatClient for Jev. The MEAI proposal keeps a decision model as its own client next to the chat model, and the Python connector's chat client (#8592) does not carry over to that design.

The package is new and ships as alpha (Microsoft.Agents.AI.TypeSafe).

Description & Review Guide

  • What are the major changes?

    • The proposed MEAI decision API, public (src/Microsoft.Agents.AI.Abstractions/MEAI).
      • The files are those of Add Layer 2 decision interoperability dotnet/extensions#7796 at e87f1126c (which includes Python: resolve release tags against real package directories #7795), unchanged: IDecisionClient, DecisionRequest, Binary/Choice/Score questions and answers, DecisionDefinition<TResult> with its builder, and AsAIFunction.
      • They compile for every target of the package, including .NET Standard 2.0 and .NET Framework. Their 173 public symbols are in PublicAPI.Unshipped.txt for each target.
      • A folder .editorconfig keeps the upstream style and overload shapes from failing the build.
      • The README records the source commits, and the steps to replace the copy with the shipped API: delete it, forward the types, and suppress any shape difference.
    • JevDecisionClient, a public IDecisionClient for Jev (with JevDecisionClientOptions).
      • Binary, Choice, and Score questions become Jev Noul, Choice, and Score questions.
      • Requests are checked with the TypeSafe rules before sending. Results are checked against the request.
      • Answers declare TwoDecimalPlaces, because the live API rounds probabilities to two decimals.
      • Jev's confidence is in AdditionalProperties["confidence"], and the JevResult is the raw representation.
    • Contract types that mirror the official TypeScript SDK (@typesafe-ai/sdk) name for name: JevRequest, JevQuestion (JevChoiceQuestion, JevScoreQuestion, JevNoulQuestion), JevResult, JevResponse, JevUsage, and JevEntry. They are sent through an HTTP client on a System.ClientModel pipeline with the SDK's defaults: two retries on 408, 429, and 5xx, honoring retry-after-ms or Retry-After up to one minute, and a 10 second timeout per attempt.
    • Jev as a tool. JevAIToolBuilder builds an ask_system_one AIFunction whose arguments are a JevRequest written by the agent's model. With UseMapping and UseClient, another client can answer it. Where the application fixes the questions, the samples use DecisionDefinition<TResult>.AsAIFunction instead.
      • The contract types are public because the mapping hooks receive and return them, and the MEAI question types cannot carry everything a model may write: instructions and criteria as JSON, Choice labels without a description, and a Noul that describes only one outcome.
    • Samples. All of them use AIAgent.
      • typesafe/Agent_TypeSafe_Step01_LoopJudge: a LoopAgent whose judge is JevDecisionClient with a typed DecisionDefinition<Verdict>, one Binary probability per criterion. The evaluator stays in the sample (DelegateLoopEvaluator), not in the public API.
      • typesafe/Agent_TypeSafe_Step02_DecisionTool (new): a Foundry agent gets DecisionDefinition<TicketAnalysis>.AsAIFunction<SupportTicket>(jev). The application fixes the questions, and the agent passes only the ticket. The same tool is then answered by a local IDecisionClient.
      • typesafe/Agent_TypeSafe_Step03_DecisionRouting (new): an agent over RoutingChatClient.Create, where a Jev Choice routes each request to a fast or a reasoning deployment.
      • Agents/Agent_Step25_ClassificationWithJev: JevAIToolBuilder, with the TypeSafe API and with a local classifier.
      • All four are registered in eng/verify-samples, and are skipped when TYPESAFE_API_KEY is not set.
  • What is the impact of these changes?

    • New, additive alpha package and samples. No existing public API changes.
    • Microsoft.Agents.AI.Abstractions gains public types in the Microsoft.Extensions.AI namespace, all [Experimental("MEAI001")], until MEAI ships them. Replacing them with the shipped types will be breaking for code that uses them, as MEAI001 allows. Type forwards keep compiled applications loading the types. src/Shared/DiagnosticIds gains AIDecisions and UrlFormat for their attributes.
    • Strict results: a Jev result that is not a complete, correlated answer fails instead of reaching the caller, with InvalidOperationException for a missing or mismatched answer, and DecisionProtocolException for a distribution that leaves out a label.
  • What do you want reviewers to focus on?

    • Carrying the proposed MEAI API publicly in Microsoft.Agents.AI.Abstractions, rather than waiting or keeping it internal. Abstractions is where MEAI places it upstream, and every provider package depends on it, so an Azure OpenAI Decisions provider can implement the same IDecisionClient. Only one package may carry the copy, because a second package with the same public types would make them ambiguous.
    • API churn. The API review on Python: resolve release tags against real package directories #7795 asks to align DecisionResponse with ChatResponse (ModelId and ResponseId instead of DecisionProvenance, settable properties). It also asks to move the typed layer to Microsoft.Extensions.AI; if that happens, our copy of the typed layer moves to Microsoft.Agents.AI. JevDecisionClient uses a small part of the response, so a re-sync should stay small.
    • No IChatClient for Jev. The Python connector has one; .NET follows the MEAI proposal instead, which uses a decision model next to the chat model as a tool or a router.
    • Refusals. Azure OpenAI Decisions returns per-question refusals, which the proposal cannot represent yet (raised on Add experimental provider-neutral decision abstractions (Layer 1) dotnet/extensions#7795). Jev does not refuse, so this PR is not affected. An Azure provider will be.

Related Issue

No agreed issue tracks this .NET work yet. Related issues and PRs:

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: implementation, tests, samples, documentation, and this PR description.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

🤖 Generated with Claude Code

Adds an alpha package that exposes Jev, the TypeSafe System One
classification model, to agents as an AIFunction built with
JevAIToolBuilder. The function arguments are a JevRequest (state plus
typed choice, score, and noul questions) and the result is the
evaluator's JevResponse serialized as JSON.

- JevAIToolOptions derives from ClientPipelineOptions; the TypeSafe API
  is called through a System.ClientModel pipeline authenticated with an
  ApiKeyCredential, retrying 429/529 and capping Retry-After at one minute.
- UseEvaluator and UseClient let any other client answer requests, with
  input and output mappers; UseClient<TClient> resolves the client from
  the agent's services in a new scope per call.
- Requests from the model and evaluator responses are validated against
  the Jev contract before they reach the API or the model.
- Adds unit tests and the Agent_Step25_ClassificationWithJev sample.
… types with the TypeSafe SDK

Add an IChatClient for Jev, the TypeSafe System One model, ported from the
Python TypeSafe connector (agent-framework-typesafe):

- JevChatClient sends the conversation as the Jev state and answers the
  questions passed with ChatOptions.WithJevQuestions (carried by
  RawRepresentationFactory) or JevChatClientOptions.DefaultQuestions.
  GetJevResult reads the typed result from ChatResponse and AgentResponse.
- Closed-set function calling: internal route and argument questions choose
  the tool and each const, enum, Boolean, or enum-array argument, with
  presence questions for optional arguments, previous-call hints, and the same
  question IDs and limits as the Python connector.
  MaximumToolCallsPerTurn (default 1) ends the tool loop.
- Streaming, generative options, hosted tools, and non-text content are
  rejected with NotSupportedException. Handled tool approvals are left out of
  the state, and text sent with an approval continues the approved call's turn.

Align the tool contract with the official TypeScript SDK (@typesafe-ai/sdk):
JevEntry (text or JSON entries), JevResult and JevResponse types, SDK retry and
timeout defaults, JevHttpClient, and UseMapping instead of UseEvaluator.

Samples: Agent_With_TypeSafe (chat client, default questions, closed-set tools,
optional arguments, approvals), Agent_TypeSafe_Step01_LoopJudge (Jev as the
judge of a LoopAgent), and a local UseClient mapping in
Agent_Step25_ClassificationWithJev. The three samples are registered in
eng/verify-samples and run when TYPESAFE_API_KEY is set.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Chained local JSON-schema references are not resolved, causing otherwise supported function tools to be silently excluded.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds the alpha Microsoft.Agents.AI.TypeSafe package, integrating Jev as both an agent tool and an IChatClient, with closed-set function calling, validation, samples, and tests.

Changes:

  • Adds Jev contracts, HTTP pipeline, retry handling, tool builder, and chat client.
  • Adds closed-set tool compilation, result validation, and loop-judge integration.
  • Registers three samples and extensive unit coverage.
File Description
.github/​workflows/​dotnet-verify-samples.yml Supplies the TypeSafe API key to sample verification.
dotnet/​agent-framework-dotnet.slnx Registers package, tests, and samples.
dotnet/​agent-framework-src-and-tests.slnf Registers source and test projects.
dotnet/​eng/​verify-samples/​AgentsSamples.cs Adds verification definitions for all three samples.
dotnet/​samples/​AGENTS.md Documents the TypeSafe sample directory.
dotnet/​samples/​02-agents/​Agents/​README.md Lists the Jev classification sample.
dotnet/​samples/​02-agents/​Agents/​Agent_Step25_ClassificationWithJev/​Agent_Step25_ClassificationWithJev.csproj Defines the classification sample project.
dotnet/​samples/​02-agents/​Agents/​Agent_Step25_ClassificationWithJev/​KeywordClassifier.cs Implements the sample local classifier.
dotnet/​samples/​02-agents/​Agents/​Agent_Step25_ClassificationWithJev/​Program.cs Demonstrates Jev as a tool and custom mapping.
dotnet/​samples/​02-agents/​Agents/​Agent_Step25_ClassificationWithJev/​README.md Documents classification and dependency injection.
dotnet/​samples/​02-agents/​AgentProviders/​README.md Lists TypeSafe provider samples.
dotnet/​samples/​02-agents/​AgentProviders/​typesafe/​Agent_With_TypeSafe/​Agent_With_TypeSafe.csproj Defines the chat-client sample project.
dotnet/​samples/​02-agents/​AgentProviders/​typesafe/​Agent_With_TypeSafe/​Program.cs Demonstrates questions, tools, and approvals.
dotnet/​samples/​02-agents/​AgentProviders/​typesafe/​Agent_With_TypeSafe/​README.md Documents JevChatClient usage and limits.
dotnet/​samples/​02-agents/​AgentProviders/​typesafe/​Agent_TypeSafe_Step01_LoopJudge/​Agent_TypeSafe_Step01_LoopJudge.csproj Defines the loop-judge sample project.
dotnet/​samples/​02-agents/​AgentProviders/​typesafe/​Agent_TypeSafe_Step01_LoopJudge/​Program.cs Implements Jev-backed loop evaluation.
dotnet/​samples/​02-agents/​AgentProviders/​typesafe/​Agent_TypeSafe_Step01_LoopJudge/​README.md Documents the loop-judge pattern.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Microsoft.Agents.AI.TypeSafe.csproj Defines the new alpha package.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevAIToolBuilder.cs Builds Jev-backed or mapped AI functions.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevAIToolOptions.cs Configures the Jev tool and pipeline.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevChatClient.cs Implements Jev as an IChatClient.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevChatClientOptions.cs Configures questions, transport, and tool limits.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevChatOptionsExtensions.cs Passes Jev questions through chat options.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevChatRequestOptions.cs Defines per-request Jev options.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevChoiceQuestion.cs Defines Choice questions.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevChoiceResponse.cs Defines Choice responses.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevEntry.cs Represents text or structured JSON entries.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevNoulCriteria.cs Defines yes/no outcome descriptions.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevNoulQuestion.cs Defines Noul questions.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevNoulResponse.cs Defines Noul responses.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevQuestion.cs Defines the polymorphic question base.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevRequest.cs Defines System One requests.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevResponse.cs Defines the polymorphic response base.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevResult.cs Defines typed Jev results.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevResultExtensions.cs Extracts typed results from responses.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevScoreQuestion.cs Defines Score questions.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevScoreResponse.cs Defines Score responses.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​JevUsage.cs Defines token usage.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevAIFunction.cs Executes and validates Jev tool calls.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevApiRequest.cs Defines the HTTP request payload.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevArgumentKind.cs Enumerates supported argument strategies.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevArgumentPlan.cs Decodes tool arguments from answers.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevChatClientLog.cs Defines tool-exclusion logging.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevChatState.cs Serializes conversations and tracks turn boundaries.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevCompiledTool.cs Holds a compiled tool-routing plan.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevContractValidator.cs Validates requests and mapped results.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevEntryJsonConverter.cs Serializes JevEntry values.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevHttpClient.cs Sends System One HTTP requests.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevJsonContext.cs Provides source-generated JSON metadata.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevJsonUtilities.cs Provides schemas and JSON helpers.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevPipelineDefaults.cs Applies retry and timeout defaults.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevQuestionBudget.cs Limits internal routing questions.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevResponseClassifier.cs Classifies retriable HTTP responses.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevRetryPolicy.cs Implements TypeSafe retry behavior.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevToolCall.cs Represents a decoded tool call.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevToolCallCompiler.cs Compiles closed-set tool schemas into questions.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevToolCallPlan.cs Selects and decodes routed tools.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevToolSchema.cs Resolves and validates tool JSON schemas.
dotnet/​src/​Microsoft.Agents.AI.TypeSafe/​Core/​JevUnsupportedToolSchemaException.cs Represents excluded tool schemas.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​Microsoft.Agents.AI.TypeSafe.UnitTests.csproj Defines the unit-test project.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevAIFunctionTests.cs Tests tool invocation and validation.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevAIToolBuilderTests.cs Tests builder options and schemas.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevAgentIntegrationTests.cs Tests agent and dependency-injection integration.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevChatClientTests.cs Tests chat requests, options, and state.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevChatClientToolTests.cs Tests routing, limits, and approvals.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevChatTestData.cs Provides chat test fixtures.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevEntryTests.cs Tests entry conversion and equality.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevHttpClientTests.cs Tests HTTP, retry, and error behavior.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevLoopJudgeTests.cs Tests loop-judge integration.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevTestData.cs Provides shared request/result fixtures.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevTestHttpHandler.cs Provides a scripted HTTP handler.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​JevToolCallCompilerTests.cs Tests schema compilation and decoding.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​ListLoggerFactory.cs Captures logs for assertions.
dotnet/​tests/​Microsoft.Agents.AI.TypeSafe.UnitTests/​ScriptedChatClient.cs Provides scripted agent-model responses.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAF Automated Review — Iteration 1

Result: No findings
Scope: full PR (2 commit(s)): f3db78f43067, c4a10b376555
Model: gpt-5.6-sol-fast

Overview

This PR adds an isolated alpha package that exposes Jev both as a validated AIFunction and as an IChatClient, with source-generated serialization, bounded closed-set tool compilation, explicit unsupported-feature failures, and comprehensive unit coverage. Request and response contract validation, per-turn tool-call limits, reserved question IDs, scoped dependency lifetimes, bounded retries, and approval-flow tests provide concrete guardrails. No supported Critical, High, or Medium defect remains after reconciliation.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
No publishable findings remained after source verification for this scope.

@joslat

Jose Luis Latorre Millas (joslat) commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Hi Roger Barreto (@rogerbarreto) - amazing port and i am impressed you managed to put this into the IChatClient - well done!!

Thanks for porting the connector. Following the Python shape makes sense, and I've moved my own #8563 to draft in
its favour.

I've spent the last few weeks measuring how Jev behaves as a judge on hand-labelled golden sets (agentic quality,
safety, red-team and compliance cases). Three findings bear directly on the LoopJudge sample and its README. The
data and method are public; links are at the end.

1. The Noul threshold is where the error trade-off gets decided, so it should be a named, documented
parameter.
The sample treats a criterion as met when its Noul is above 0.5. On 298 labelled agentic cases,
moving one uniform threshold traded the two kinds of error against each other:

threshold accuracy false pass false fail
0.50 82.8% 27.3% 11.0%
0.55 87.9% 25.9% 4.2%
0.70 83.7% 15.6% 16.7%
0.90 80.5% 1.9% 29.5%

For a loop judge, a false pass means accepting an answer that still misses a criterion, while a false fail costs
one extra iteration. Which threshold is right depends on which of those the caller can afford. Suggestion: make the
threshold an explicit option in the sample (it is one line), and say in the README that 0.5 is not calibrated for
any particular task.

2. Keep criterion wording descriptive, not directional. We tested two ways of helping Jev with criteria it found
hard:

  • A neutral reference block that describes what is being judged moved accuracy on a violence set from 64% to
    96%, with 0 false passes. It was the best or joint-best arm on 10 of 16 sets.
  • Directional wording ("only mark met if …" or "lean met") saturated the model instead of calibrating it, with
    false fails up to 100% on some sets.

The sample's questions are already descriptive ("the assistant's latest response …"), which is the robust shape. A
sentence in the README telling users to keep them that way would save people from finding this out the hard way.
The eight reference blocks that won are published as DecisionReferences in AgentEval 0.42.0-beta, if they are
useful as a starting point: https://github.com/AgentEvalHQ/AgentEval/blob/main/src/AgentEval.Core/Decisions/DecisionReferences.cs

3. Safety-style criteria grade the content, not the agent's handling of it. When the state contains harmful
content, Jev tends to score whether that content is harmful, not whether the agent's response handled it well. A
correct refusal therefore often reads as "criterion not met": on EU AI Act prohibited-practice cases it agreed
with the labels 20% of the time, against 84% for a generative judge. A loop judge with a safety criterion would
keep iterating on an answer that was already a correct refusal. If safety criteria are an intended use, that is
worth a line in the README.

If it helps, my harness can give the LoopJudge sample's pattern an accuracy figure on these golden sets. I'm happy
to run it against this PR's JevChatClient and post the numbers here.

Evidence (all public, AgentEval repo):

…judge sample

- Follow chains of local $ref aliases when compiling tool schemas, bounded by
  the existing resolution depth, so an alias of a definition is routed instead
  of being excluded and a reference cycle excludes the tool.
- Check const and enum values against their declared type, as array members
  already were, so a tool whose values could only fail when called is
  excluded before Jev routes a call to it.
- Name the criterion threshold in the loop judge sample and document how to
  choose it, and how to word criteria for a judge.
@rogerbarreto

Copy link
Copy Markdown
Member Author

Jose Luis Latorre Millas (@joslat) thanks, data very useful, and thanks for moving #8563 to draft. Addressed in 3e7756e:

  1. Threshold. LoopJudge sample now has named CriterionThreshold (default 0.5) with trade-off comment. README new section "Writing criteria for a judge": 0.5 not calibrated for any task; higher → fewer false passes, more loop runs; pick from labeled examples by which error costs more.
  2. Wording. README: describe criterion, no directional wording ("only say yes if", "lean towards yes"); neutral description of what is judged helps more on hard criteria. Sample questions already descriptive.
  3. Safety. README: ask about response's handling, not topic; check on labeled examples incl. refusals before relying. Quick live probe on simple refusal (weapon request + refusal) scored correctly with response-scoped questions (handled appropriately 0.98, complied 0.01), so kept as guidance, not claim about Jev in general.

Harness offer: yes please. Numbers for LoopJudge pattern with JevChatClient on your golden sets very welcome here.

@rogerbarreto
Roger Barreto (rogerbarreto) marked this pull request as draft October 28, 2026 18:55
@rogerbarreto Roger Barreto (rogerbarreto) changed the title .NET: Add Microsoft.Agents.AI.TypeSafe with Jev as a tool and as an IChatClient (alpha) [WIP] .NET: Add Microsoft.Agents.AI.TypeSafe with Jev as a tool and as an IChatClient (alpha) Oct 28, 2026
@rogerbarreto

Roger Barreto (rogerbarreto) commented Oct 28, 2026 •

Copy link
Copy Markdown
Member Author

Assistant review assessment

Roger Barreto (@rogerbarreto), joslat's follow-up supplies corrected external benchmark figures and offers an adapter-specific run. This is community feedback, so no automatic implementation or response is authorized.

At 78d2948474e2ec594cf6ec107c6e4b7c92a0ffcc, the earlier sample requests are already covered: LoopJudge Program.cs:33-36,98 has the named threshold, and its README.md:49-64 explains calibration, descriptive criteria, and refusal handling. The follow-up acknowledges those changes. No remaining code defect is established. The external measurements and their applicability to JevChatClient have not been independently verified.

Recommendation: keep sample behavior/defaults unchanged; consider the offered adapter-specific benchmark only if useful to you. Existing PR-session ownership is verified; persistent Project ownership is not established. Decision requested: leave this informational, or pursue that benchmark follow-up? No external benchmark, code change, reply to the contributor, or thread resolution has been performed.

@Mehdi-Bl

Copy link
Copy Markdown

Roger Barreto (@rogerbarreto) I'm lost over this PR direction as this is not aligned with OpenAI decision API that was published and should be the primary integration target instead of JEV.
https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/decisions?tabs=python
Even the models in the samples are not the decision models we have.

@rogerbarreto

Copy link
Copy Markdown
Member Author

Roger Barreto (@rogerbarreto) The new comment raises an integration direction question, not a confirmed code defect. The official Azure OpenAI Decisions API is now published as an endpoint for typed predicate, choice, and score results, so the claim that a public API exists is accurate. The separate provider neutral decision abstractions this PR was waiting on remain open in dotnet/extensions #7795 and #7796.

The linked API does not by itself establish that it should replace Jev in this WIP. That choice would change the integration target and the models shown in the samples. Recommendation: keep the current draft unchanged until Roger decides whether to re-scope it toward Azure OpenAI Decisions. Roger, should this PR keep its current Jev direction, or should we revisit the integration target?

Copy the provider-neutral decision API proposed in dotnet/extensions#7795
(Layer 1) and microsoft#7796 (Layer 2) into src/Shared/Decisions, with every
top-level type made internal. Projects opt in with InjectSharedDecisions.
The namespace and the MEAI001 attributes are unchanged, so code built on
the copy compiles against the shipped types once the folder is deleted.

Microsoft.Agents.AI.TypeSafe compiles the copy.
JevDecisionClient implements the internal IDecisionClient copy over the
TypeSafe System One API. Binary, Choice, and Score questions map to Noul,
Choice, and Score questions; answers map back with complete, ordered
distributions, Jev's confidence as an additional property, and the
JevResult as the raw representation. Callers that hold Jev questions can
pass them through DecisionOptions.RawRepresentationFactory so JSON
instructions and criteria reach the API unchanged.

The Score test fixture now reports probabilities that agree with a
fractional score, as the decision API requires.
Move the conversation state, closed-set tool compiler, and turn logic of
JevChatClient into src/Shared/DecisionChat, built on the internal
IDecisionClient copy instead of the Jev contract types. Projects opt in
with InjectSharedDecisionChat, so other decision models (Azure OpenAI
Decisions, Gemini) can reuse the same tool calling and turn handling.

JevChatClient keeps its public surface and wire format: it validates and
projects its Jev questions, runs the engine over JevDecisionClient, and
describes the answers as the JevResult. Question IDs, limits, and wording
are unchanged. A response that is not a complete answer to the questions
now fails with InvalidOperationException.
Live TypeSafe responses round probabilities and confidence to two
decimal places, so a distribution can sum to 0.99 or 1.01. Declaring
HighPrecision made the decision API reject such answers with its 1e-6
tolerance. Jev answers now declare TwoDecimalPlaces, and a test covers
a rounded distribution that sums to 0.99.
# Conflicts:
#	dotnet/eng/MSBuild/Shared.props
…e package

Move the copy of the decision API proposed in dotnet/extensions#7795 and
microsoft#7796 from src/Shared/Decisions into Microsoft.Agents.AI.TypeSafe/MEAI,
restoring the upstream files verbatim, so IDecisionClient,
DecisionDefinition<TResult>, and AsAIFunction are public in the
Microsoft.Extensions.AI namespace and marked MEAI001, as proposed. This
follows how Agent Framework carried proposed MEAI types before they
shipped (microsoft#279, microsoft#426).

JevDecisionClient is now public, with an API key constructor and
JevDecisionClientOptions, so applications and samples can use Jev through
the proposed API today.
The LoopAgent judge now uses JevDecisionClient and a typed
DecisionDefinition<Verdict> with one Binary probability per criterion,
instead of JevChatClient, so the evaluator stays in the sample and any
IDecisionClient can judge. The answerer's first-answer instruction is
stricter, so the first answer reliably misses a criterion and the loop
runs twice, as the sample describes.
Agent_TypeSafe_Step02_DecisionTool gives a Foundry agent a decision as a
tool through DecisionDefinition<TResult>.AsAIFunction: the application
fixes the questions, the agent passes only the ticket, and Jev answers.
The same tool is then answered by a local IDecisionClient.

Agent_TypeSafe_Step03_DecisionRouting routes an agent's requests between
a fast and a reasoning deployment with RoutingChatClient.Create, with a
Jev Choice selecting the route.

Both follow the usage shown in dotnet/extensions#7796 and are registered
for sample verification.
@rogerbarreto Roger Barreto (rogerbarreto) changed the title [WIP] .NET: Add Microsoft.Agents.AI.TypeSafe with Jev as a tool and as an IChatClient (alpha) [WIP] .NET: Add Microsoft.Agents.AI.TypeSafe with Jev as an IDecisionClient, a tool, and an IChatClient (alpha) Nov 12, 2026
The issue routing test requires a route for every solution project, and
Microsoft.Agents.AI.TypeSafe had none.
….Abstractions

The copy now sits where MEAI places the API upstream
(Microsoft.Extensions.AI.Abstractions), in the one package every
provider depends on, so an Azure OpenAI Decisions provider can implement
the same public IDecisionClient. The files are unchanged; they compile
for every target of the package, including .NET Standard 2.0 and .NET
Framework, and their 173 public symbols are declared in
PublicAPI.Unshipped.txt for each target. RS0026 and RS0027 are silenced
for the copied overloads, whose shape the MEAI API review owns.

The README describes the switch to the shipped API: delete the copy,
add type forwards, and suppress any shape difference.
…Safe package

The MEAI decision API keeps a decision model as its own client, used as a tool
(DecisionDefinition.AsAIFunction) or a router (RoutingChatClient), not as an
IChatClient. JevChatClient answered questions through IChatClient, which needed
a provider-neutral engine to turn the conversation into a state and compile
closed-set tools into Choice questions. With the MEAI patterns available through
JevDecisionClient, the bridge no longer earns its public surface.

Removed: JevChatClient, JevChatClientOptions, JevChatOptionsExtensions,
JevChatRequestOptions, JevResultExtensions, the internal raw-question path of
JevDecisionClient, src/Shared/DecisionChat, and the Agent_With_TypeSafe sample.

The Jev wire types stay public: JevAIToolBuilder exposes them to its mapping
hooks, and MEAI's question types cannot carry JSON instructions and criteria,
undescribed Choice labels, or a Noul that describes one outcome.
@rogerbarreto Roger Barreto (rogerbarreto) changed the title [WIP] .NET: Add Microsoft.Agents.AI.TypeSafe with Jev as an IDecisionClient, a tool, and an IChatClient (alpha) [WIP] .NET: Add Microsoft.Agents.AI.TypeSafe with Jev as an IDecisionClient and a tool (alpha) Nov 13, 2026

This branch was successfully deployed

1 active and 1 inactive (outdated) deployments
github-app-auth — b3a1ba80 Deployed Nov 13, 2026 by rogerbarreto via add_label #27774
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs .NET Usage: [Issues, PRs], Target: .Net

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants