Skip to content

Test the remote backends against llama-server - #426

Merged
mattt merged 2 commits into
mainfrom
mattt/llama-server-tests
Nov 18, 2026
Merged

mattt merged 2 commits into
mainfrom
mattt/llama-server-tests

Conversation

@mattt

@mattt mattt commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

This PR adds an integration suite that runs OpenAILanguageModel (Chat Completions and Responses), OpenResponsesLanguageModel, and AnthropicLanguageModel against a local llama-server, the HTTP server in llama.cpp. These backends take a base URL, so they work with it, but nothing showed that, and the README didn't mention it. The suite runs only when LLAMA_SERVER_BASE_URL is set, and its image tests only when LLAMA_SERVER_VISION_BASE_URL is set. It covers text and streamed responses with usage, instructions, multi-turn sessions, structured output, tool calling, reasoning, image input, temperature, maximum tokens, stop sequences, cancellation, and a prompt longer than the context.

Where llama-server differs from the API it imitates, the test marks the difference as a known issue instead of working around it. With tools and a response schema in one request, the Chat Completions and Responses endpoints allow only the schema, so the model can't call a tool. The Messages endpoint ignores output_config.format, and it has no way to turn off parallel tool calls. The README gets a llama-server section with an example and these limits.

@mattt
mattt merged commit c272a42 into main Nov 18, 2026
14 checks passed
@mattt
mattt deleted the mattt/llama-server-tests branch November 18, 2026 03:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant