Skip to content

Python: .NET: Foundry Toolbox file_search returns source metadata but no answer citation annotations #10138

Open

Description

@mikael-ekholm

Observed Behavior

Using the .NET Foundry Toolbox integration, file_search retrieves passages with
resource._meta.file_id and title, and the agent streams its answer, but the
returned AgentResponseUpdate.Contents contain no AIAnnotations. A native
HostedFileSearchTool control using the same model, question and vector store
returns a CitationAnnotation.

This reproduces with Microsoft SDKs alone, before any Responses hosting converter.
There are no application dependencies, custom agent wrappers or output schemas.
The file identity metadata is present in the function result; I am not claiming
that an already-created answer citation was dropped by OutputConverter or MCP.

Expected Behavior

A supported way to retain Toolbox-managed file search and obtain structured
answer citations while streaming normal text, comparable to native file search.

Please confirm whether this is a defect in the .NET integration or an unsupported
capability. If applications are expected to map retrieved source identities into
answer annotations themselves, documentation and a streaming sample would help.
This report intentionally leaves the SDK-versus-service ownership question open.

Steps to Reproduce

  1. Use a Foundry project with a supported model deployment and Azure CLI credentials.

  2. Use a fully indexed vector store with a document answering the question. For
    independent reproduction, a document named fixture.md can contain:

    # Recovery guide
    The recovery timeout is 37 seconds.
    After a recovery timeout, wait 11 seconds before trying again.
  3. Configure a Toolbox containing a named file_search tool bound to that same store:

    {
      "type": "file_search",
      "name": "search_docs",
      "description": "Search recovery documentation.",
      "vector_store_ids": ["<VECTOR_STORE_ID>"],
      "max_num_results": 10
    }
  4. Save the two files below in an otherwise empty directory and set:

    $env:FOUNDRY_PROJECT_ENDPOINT = 'https://<account>.services.ai.azure.com/api/projects/<project>'
    $env:AZURE_AI_MODEL_DEPLOYMENT_NAME = '<model-deployment>'
    $env:REPRO_TOOLBOX = '<toolbox-name>'
    $env:REPRO_TOOL_NAME = 'search_docs'
    $env:REPRO_VECTOR_STORE = '<VECTOR_STORE_ID>'
    $env:REPRO_QUESTION = 'How long is the recovery timeout?'
    dotnet restore --source https://api.nuget.org/v3/index.json
    dotnet run -c Release --no-restore -- toolbox
    dotnet run -c Release --no-restore -- native
  5. Compare resourcesWithFileIds and annotations in the JSON summaries. Annotation
    counting inspects all framework content, including annotation-only updates.

Validation was performed on 2026-11-06 against an existing indexed documentation
store, not the synthetic fixture above. The model was gpt-5.4-mini; the question
was answerable from a retrieved release-note document. The same prompt/model/store
were used in both modes. Private documents, queries, endpoints and IDs are omitted.
Maintainers can substitute their own document and matching question.

Minimal Reproduction

ToolboxCitationRepro.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Agents.AI.Foundry.Hosting" Version="1.27.0-preview.261030.1" />
    <PackageReference Include="Azure.AI.Projects" Version="3.0.0-beta.6" />
  </ItemGroup>
</Project>

Program.cs:

using System.Text.Json;
using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry.Hosting;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Options;

#pragma warning disable MAAI001 // The Foundry Toolbox API is experimental.
if (args.Length != 1 || args[0] is not ("toolbox" or "native"))
    throw new ArgumentException("Usage: dotnet run -- toolbox|native");

using var timeout = new CancellationTokenSource(TimeSpan.FromMinutes(3));
var credential = new AzureCliCredential();
var toolboxOptions = new FoundryToolboxOptions();
await using var toolbox = new FoundryToolboxService(
    Options.Create(toolboxOptions), credential);
List<AITool> tools;
if (args[0] == "toolbox")
{
    toolboxOptions.ToolboxNames.Add(Required("REPRO_TOOLBOX"));
    await toolbox.StartAsync(timeout.Token);
    var name = Environment.GetEnvironmentVariable("REPRO_TOOL_NAME") ?? "search_docs";
    tools = toolbox.Tools.Where(t => t is AIFunction f && f.Name == name).ToList();
    if (tools.Count != 1) throw new InvalidOperationException("Expected exactly one matching search tool.");
}
else
{
    tools = [new HostedFileSearchTool { Inputs = [new HostedVectorStoreContent(Required("REPRO_VECTOR_STORE"))] }];
}

var agent = new AIProjectClient(new Uri(Required("FOUNDRY_PROJECT_ENDPOINT")), credential)
    .AsAIAgent(new ChatClientAgentOptions
    {
        Name = "citation-repro",
        ChatOptions = new ChatOptions
        {
            ModelId = Required("AZURE_AI_MODEL_DEPLOYMENT_NAME"),
            Instructions = "Search the documentation to answer the question. Cite the supporting source document.",
            Tools = tools,
            Reasoning = new ReasoningOptions { Effort = ReasoningEffort.Low }
        }
    });

var chunks = 0;
var characters = 0;
var calls = 0;
var results = 0;
var resourcesWithFileIds = 0;
var annotationTypes = new List<string>();
await foreach (var update in agent.RunStreamingAsync(
    Environment.GetEnvironmentVariable("REPRO_QUESTION") ?? "How long is the recovery timeout?",
    cancellationToken: timeout.Token))
{
    foreach (var content in update.Contents)
    {
        if (content.Annotations is { } annotations)
            annotationTypes.AddRange(annotations.Select(a => a.GetType().Name));
        if (content is TextContent { Text.Length: > 0 } text) { chunks++; characters += text.Text.Length; }
        if (content is FunctionCallContent) calls++;
        if (content is not FunctionResultContent result) continue;
        results++;
        var value = JsonSerializer.SerializeToElement(result.Result);
        while (value.ValueKind == JsonValueKind.String)
        {
            try { value = JsonSerializer.Deserialize<JsonElement>(value.GetString()!); }
            catch (JsonException) { break; }
        }
        if (value.ValueKind != JsonValueKind.Object
            || !value.TryGetProperty("content", out var blocks) || blocks.ValueKind != JsonValueKind.Array) continue;
        foreach (var block in blocks.EnumerateArray())
            if (block.TryGetProperty("resource", out var resource)
                && resource.TryGetProperty("_meta", out var meta)
                && meta.TryGetProperty("file_id", out var id) && id.ValueKind == JsonValueKind.String
                && meta.TryGetProperty("title", out _)) resourcesWithFileIds++;
    }
}
if (characters == 0 || (args[0] == "toolbox" && (results == 0 || resourcesWithFileIds == 0)))
    throw new InvalidOperationException("Retrieval/answer precondition failed; this run cannot demonstrate the citation gap.");
Console.WriteLine(JsonSerializer.Serialize(new
{
    mode = args[0], textChunks = chunks, textCharacters = characters,
    functionCalls = calls, functionResults = results, resourcesWithFileIds,
    annotations = annotationTypes.Count, annotationTypes
}));

static string Required(string name) => Environment.GetEnvironmentVariable(name)
    is { Length: > 0 } value ? value : throw new InvalidOperationException($"Set {name}.");

Error Messages and Stack Traces

No exception in either mode. Actual sanitized output:

{"mode":"toolbox","textChunks":47,"textCharacters":560,"functionCalls":1,"functionResults":1,"resourcesWithFileIds":10,"annotations":0,"annotationTypes":[]}
{"mode":"native","textChunks":39,"textCharacters":416,"functionCalls":0,"functionResults":0,"resourcesWithFileIds":0,"annotations":1,"annotationTypes":["CitationAnnotation"]}

These are individual runs, not a latency or quality benchmark. Native file search
runs server-side, so zero client function results in that control is expected.

Package Versions

Direct dependencies:

  • Microsoft.Agents.AI.Foundry.Hosting: 1.27.0-preview.261030.1
  • Azure.AI.Projects: 3.0.0-beta.6

Resolved dependencies:

  • Microsoft.Agents.AI: 1.27.0
  • Microsoft.Agents.AI.Foundry: 1.27.0-preview.261030.1
  • Microsoft.Extensions.AI.OpenAI: 10.10.1
  • OpenAI: 2.14.0
  • ModelContextProtocol: 2.2.0
  • Azure.AI.AgentServer.Core: 1.0.0-beta.30
  • Azure.AI.AgentServer.Responses: 1.0.0-beta.9

.NET Version

.NET 10, SDK 10.0.400.

Operating System

Windows.

Regression

Unknown. A separate application-level experiment also observed no native Toolbox
answer annotations with Foundry Hosting 1.21.0-preview.260911.1. That earlier result
is context, not an independent run of this minimal program on the older package.

Additional Context

FoundryToolboxService in the 1.27 release discovers McpClientTools and wraps them
as AIFunctions. The native control uses HostedFileSearchTool instead. These
paths differ in tool execution, so the observation may need a supported bridge
rather than simply another hosting annotation serializer fix.

Related reports checked:

Those do not establish a supported .NET Toolbox file-search answer-citation path.
The metadata shape observed here matches the File Search result shape described
in the official Toolbox documentation:
https://learn.microsoft.com/en-us/azure/foundry/agents/how-to/tools/toolbox

The program only consumes existing resources and creates no agent deployment,
Toolbox version or persistent application session. No implementation change is
proposed pending maintainer guidance.

Acknowledgements

  • I searched existing issues and did not find an exact duplicate.
  • The behavior was verified using the minimal program above; the alternative synthetic fixture has not been tested.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    .NETUsage: [Issues, PRs], Target: .Net
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 6, 2026
  2. changed the title [-].NET: Foundry Toolbox file_search returns source metadata but no answer citation annotations[/-] [+]Python: .NET: Foundry Toolbox file_search returns source metadata but no answer citation annotations[/+] on Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    .NETUsage: [Issues, PRs], Target: .NetpythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions