Skip to content

test(server): add tool-state isolation smoke harness#27

Open
kira-ariaki wants to merge 1 commit into
basecompute:mainfrom
kira-ariaki:contrib/tool-state-smoke
Open

test(server): add tool-state isolation smoke harness#27
kira-ariaki wants to merge 1 commit into
basecompute:mainfrom
kira-ariaki:contrib/tool-state-smoke

Conversation

@kira-ariaki

Copy link
Copy Markdown

Summary

  • add a standard-library tool-call state smoke harness for sequential shared-prefix and concurrent requests
  • validate exact function names, complete JSON argument objects, sentinel identity, and finish reasons
  • add engine-free unit/integration tests with a local fake HTTP server
  • make the stronger check opt-in from the manual Server smoke workflow
  • run benchmark-tool unit tests in CI
  • update the manual workflow to the current lowercase release asset and basert-serve CLI names

Why

A one-shot chat completion cannot catch cross-request tool argument contamination. This harness sends distinct sentinels through a repeated tool schema and reports exact mismatches as structured FAIL records.

Failures are consistent with request-state leakage, but the harness remains diagnostic and does not claim an engine root cause by itself.

Refs #26.

Verification

  • python3 -m unittest discover -s benchmarks/tests -v — 17/17 passed
  • ruff check benchmarks/scripts/tool_state_smoke.py benchmarks/tests/test_tool_state_smoke.py
  • Python compile checks and workflow YAML parsing passed
  • live against BaseRT 0.1.7 + Qwen3.6-35B-A3B: correctly exits nonzero and identifies cross-request sentinel corruption in sequential, N=2, and N=4 cases
  • exact-argument tests reject missing or leaked extra fields; CLI tests reject malformed URLs and non-finite timeouts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant