Skip to content

Python: tests/samples fails to collect because four test modules import samples that no longer exist #9761

Open

Description

Description

python/tests/samples can't be collected on main. Four of its six test modules import sample files that no longer exist:

$ uv run pytest tests/samples --collect-only -q
ERROR tests/samples/getting_started/test_agent_samples.py
ERROR tests/samples/getting_started/test_chat_client_samples.py
ERROR tests/samples/getting_started/test_threads_samples.py
ERROR tests/samples/hosting/test_toolbox_endpoint.py - FileNotFoundError: [Errno 2] No such file or directory: '.../responses/04_foundry_toolbox/main.py'
!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!

Pytest stops the whole session on collection errors. So the "Test OpenAI samples" and "Test Azure samples" steps in python-merge-tests.yml (uv run pytest tests/samples/ -m "openai" / -m "azure", gated on RUN_SAMPLES_TESTS) can't run any test, including the two modules that still collect.

Expected behavior

pytest tests/samples collects. The sample integration tests either target samples that exist or are removed.

Proposed fix

  • hosting/test_toolbox_endpoint.py: delete it. Its cases are covered by packages/foundry_hosting/tests/test_toolbox.py, and I'm porting the one extra case (several trailing slashes on the project endpoint). I have a PR for this part.
  • getting_started/*.py: this needs a maintainer decision. Either repoint the tests at the current 01-get-started / 02-agents samples, or delete them if the sample integration tests aren't maintained any more. I'm happy to do either.

Package Versions

main at c4de86d

Python Version

3.13

AI Assistance

Claude Code helped find this while checking which sample files the tests reference, and helped draft this report. I reproduced the collection errors locally.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 27, 2026
  2. eavanvalkenburg commented on Nov 2, 2026

    @eavanvalkenburg
    Member

    Could you provide a coverage map for the three remaining getting_started test modules, separating cases that map to current maintained samples from cases whose APIs/samples were intentionally removed? That would let us choose repointing versus retirement without silently losing supported-scenario coverage. Existing #9762 is explicitly the hosting-only part and does not resolve those three collection failures.

  3. Wilson0511 commented on Nov 11, 2026

    @Wilson0511
    Contributor

    Here is the coverage map for python/tests/samples on current main (@ b193c1b), built by parsing all 56 parametrized cases in the three getting_started modules and resolving each against the current samples/ tree.

    hosting/test_toolbox_endpoint.py — no longer testable as written

    The module tests resolve_toolbox_endpoint() in 04_foundry_toolbox/main.py and 06_files/main.py. Neither function exists anymore: after the #7405 rename the directories are responses/foundry_toolbox/ and responses/files/, and both main.py files were rewritten to build a FoundryToolbox(credential) inside create_agent() — no local endpoint-resolution helper. That logic now lives in the product package and is already covered by python/packages/foundry_hosting/tests/test_toolbox.py (_resolve_toolbox_endpoint, lines 42/329-350). Disposition: delete the module; coverage is subsumed by the package tests.

    getting_started modules — 56 param cases

    (A) 38 cases map to a maintained sample that exists today (each target verified to still define main):

    Old import family Count Current sample
    agents.openai.openai_responses_client_* 9 02-agents/providers/openai/client_{basic,reasoning,with_code_interpreter,with_explicit_settings,with_file_search,with_function_tools,with_local_mcp,with_session,with_web_search}
    agents.openai.openai_chat_client_* 6 02-agents/providers/openai/chat_completion_client_{basic,with_explicit_settings,with_function_tools,with_local_mcp,with_session,with_web_search}
    agents.azure_ai.azure_ai_* 5 02-agents/providers/foundry/foundry_agent_basic, foundry_chat_client_{with_code_interpreter,with_local_mcp,with_session} (_with_thread → with_session)
    agents.azure_openai.azure_responses_client_* 3 02-agents/providers/azure/openai_client_{basic,with_function_tools,with_session}
    agents.azure_openai.azure_chat_client_* 4 02-agents/providers/azure/openai_chat_completion_client_{basic,with_explicit_settings,with_function_tools,with_session}
    client.* (non-assistants) 6 02-agents/providers/{openai,azure}/…client_basic, foundry_chat_client_basic, chat_client/chat_response_cancellation
    threads.* 2 02-agents/conversations/{custom_history_provider,suspend_resume_session}
    agents.openai.azure_ai_basic etc. 3 (covered in the foundry/azure rows above)

    (B) 15 cases reference removed assistants clients — azure_assistants_* (6), openai_assistants_* (7), client.{azure,openai}_assistants_client (2). The assistants client family was removed in #4990 and has no maintained successor sample. Disposition: drop these param cases.

    (C) 3 cases have no current equivalent:

    • agents.azure_openai.azure_responses_client_with_code_interpreter and …_with_explicit_settings — the Azure Responses-API variants were never recreated under providers/azure/ (only openai_client_{basic,with_function_tools,with_session,with_structured_output} exist). Disposition: either drop, or cover via the plain-openai equivalents.
    • agents.azure_ai.azure_ai_with_function_tools — the test imports mixed_tools_example / tools_on_agent_level / tools_on_run_level; today's foundry_agent_with_function_tools.py defines a different surface. Disposition: re-point to foundry_agent_with_function_tools.main and rewrite the case, or drop.

    Caveat + proposal

    The map above is filename/symbol-level. Re-pointing (A) still needs each case's mocked-response list re-checked against the new samples' actual call sequences — the old recorded responses were written against the removed API shapes. I'm happy to take this issue: re-point the 38 (A) cases with refreshed mocks, drop the 15 (B) and 2–3 (C) cases per your preference, delete the hosting module, and get pytest tests/samples --collect-only green again.

    🤖 This analysis was prepared with AI assistance (ZCode); publishing was confirmed by the human author.

    Co-Authored-By: ZCode noreply@zcode.local

  4. added
    team-onlyImplementation is reserved for the Agent Framework team.
    and removed
    team-onlyImplementation is reserved for the Agent Framework team.
    on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions