Skip to content

fix(provider): OpenAI Compatible 模型验证误报'没有可用模型' / fix model validation false-negative for OpenAI-compatible providers - #779

Open
WenhuaXia wants to merge 1 commit into
AgentFlocks:mainfrom
WenhuaXia:fix/openai-compatible-model-validation
Open

WenhuaXia wants to merge 1 commit into
AgentFlocks:mainfrom
WenhuaXia:fix/openai-compatible-model-validation

Conversation

@WenhuaXia

Copy link
Copy Markdown

问题描述 / Bug Description

onboarding 配置本地模型(vLLM/Ollama 等 OpenAI Compatible provider)时,点击「保存并验证模型」永远报错 「该 Provider 没有可用的模型进行测试」(no_models_available),即使远程服务的 /v1/models 端点正常返回、模型也确实可用。

When configuring a local model (vLLM/Ollama, etc. via the OpenAI Compatible provider) in onboarding, clicking "save and validate" always reports "该 Provider 没有可用的模型进行测试" (no_models_available), even when the remote service's /v1/models endpoint returns models correctly.

根因 / Root Cause

flocks/provider/sdk/openai_compatible.py 的 get_models() 只返回 flocks.json 里手动配置的 _config_models,从不请求远程 /v1/models:

def get_models(self) -> List[ModelInfo]:
    """Return models from flocks.json (_config_models) only.
    OpenAI-compatible providers have no predefined model list; users add
    their own models via the model management UI.
    """
    return list(getattr(self, "_config_models", []))

而 catalog 中 openai-compatible 的预定义模型数为 0(对照 threatbook-cn-llm 有 11 个)。因此 isolated 验证时 probe_models 恒为空,在 ~14ms 内就返回 no_models_available —— 验证逻辑根本没接触到远程服务。

ThreatBook 不受影响,因为它的 catalog 自带模型列表,_prepare_isolated_provider 的 backfill 能补上。

修复 / Fix

在 flocks/server/routes/provider.py 的 isolated 验证分支中:若 probe_models 为空、且用户在表单里明确填了 model_id,则用该 id 合成一个最小 ModelInfo,让验证真正发起 chat 请求去校验该模型。

When probe_models is empty and the user explicitly supplied a model_id in the onboarding form, synthesize a minimal ModelInfo so the real chat call validates that model against the upstream instead of failing early.

  • 仅在 isolated(验证快照)路径生效,不写入 config,零副作用
  • 排除 ThreatBook provider(它们已有 backfill)
  • 仅在用户显式提供 model_id 时触发

验证 / Verification

直接调用 onboarding 使用的同一函数 _test_provider_credentials_impl(target: http://192.168.250.10:8000/v1, model Qwen3.8-27B, vLLM TP4×4):

修复前 / Before 修复后 / After
结果 / Result no_models_available 连接成功 (success)
延迟 / Latency 14ms(未连远程) 1015ms(真实推理)
回答 / Answer — Paris

远程服务健康:/v1/models 带 key 返回 Qwen3.8-27B。

测试计划 / Test Plan

  • 复现脚本验证:修复前 no_models_available(14ms)→ 修复后 连接成功(Paris, 1015ms)
  • 未改动 get_models() 的既有语义(手动模型管理流程不受影响)
  • ThreatBook 路径不受影响(显式排除)
  • 现有测试套件通过 / existing tests pass(纯后端逻辑改动)

风险评估 / Risk Assessment

Low — 改动仅 22 行,位于 isolated 验证分支,仅在 probe_models 为空且用户显式提供 model_id 时触发;合成的是临时 ModelInfo,不写入 flocks.json,不影响任何持久状态。


注: uv.lock 的本地环境漂移(aliyun mirror → pypi)已还原,未包含在本 PR。

…on false-negative for OpenAI-compatible providers

问题 / Problem:
onboarding 配置本地模型(vLLM/Ollama 等 OpenAI Compatible)时,
'保存并验证模型' 永远报错'该 Provider 没有可用的模型进行测试',
即使远程 /v1/models 正常返回且模型可用。
When configuring local models (vLLM/Ollama, etc.) during onboarding,
'save and validate' always reports 'No models available' even though
the remote /v1/models endpoint works.

根因 / Root cause:
openai_compatible.py 的 get_models() 只返回 flocks.json 里手动配置的
_config_models,从不请求远程 /v1/models。catalog 中 openai-compatible
的 models 列表为空,所以 isolated 验证时 probe_models 恒为空,
14ms 内就返回 no_models_available,根本没连远程。
get_models() only returns models manually configured in flocks.json
and never fetches remote /v1/models. The openai-compatible catalog
entry has zero predefined models, so probe_models is always empty
during isolated validation and it bails out with no_models_available
in ~14ms without ever contacting the upstream.

修复 / Fix:
在 isolated 验证分支中,若 probe_models 为空且用户在表单里明确填了
model_id,则用该 id 合成最小 ModelInfo,让验证真正发起 chat 请求。
In the isolated validation branch, when probe_models is empty and the
user explicitly supplied a model_id, synthesize a minimal ModelInfo so
the real chat call validates the model against the upstream.

验证 / Verified:
修复前: no_models_available (14ms,未连远程)
修复后: 连接成功, answer=Paris (1015ms,真实推理)
Before: no_models_available (14ms, never hit upstream)
After: success, answer=Paris (1015ms, real inference)
@WenhuaXia
WenhuaXia force-pushed the fix/openai-compatible-model-validation branch from bdb9a77 to 4576038 Compare October 13, 2026 11:03
WenhuaXia added a commit to WenhuaXia/flocks that referenced this pull request Oct 13, 2026
… to default LLM

问题 / Problem:
从模型管理页添加模型(非 onboarding 向导)后,provider 已配置、凭据已存,
但 default_models.llm 为空。/api/onboarding/status 要求'默认模型已设置
+ 凭据可用'才返回 completed=True,WebUI 在 completed=False 时每次登录
强制弹出初始化向导(前端 dismissed flag 已被测试显式忽略)。
结果:用户明明加了模型,每次登录还是被弹初始化。
Adding a model from the model-management page (not the onboarding
wizard) leaves default_models.llm empty. /api/onboarding/status only
returns completed=True when a default model is set AND credentials
exist, and the WebUI force-opens the setup wizard whenever
completed=False (the frontend dismissed flag is explicitly ignored).
So users who already added a model get the onboarding popup on
every login.

根因 / Root cause:
create_model (custom_provider.py) 只写 provider.{id}.models.{model_id},
从不调 set_default_model。只有 onboarding /apply 路径(1023-1057 行)
会设默认,而该路径此前被 AgentFlocks#779 的验证 bug 挡住,用户只能跳过 onboarding
后手动加模型,陷入'有模型无默认'的中间态。
create_model only upserts the model entry and never calls
set_default_model. Only the onboarding /apply path sets the default,
and that path was previously blocked by the validation bug in AgentFlocks#779 —
users skipped onboarding, added the model manually, and were left with
'a model but no default'.

修复 / Fix:
create_model 新增 _maybe_promote_first_model_to_default:当且仅当
(1) 该 provider 此前无模型(这是第一个模型)、(2) 全局尚无默认 LLM、
(3) 该 provider 已有可用凭据(secret 或 inline apiKey)时,自动把新模型
设为默认 LLM。
- 仅第一个模型才提升 → 不会在后续加第二/第三个模型时覆盖用户已选默认
- 无默认才提升 → 不覆盖 onboarding/其它 provider 已设的默认
- 有凭据才提升 → 与 status.completed 判定对齐,避免'设了默认但跑不通'
  的矛盾态(无凭据时不设,用户补 key 后在模型管理页一键设默认)
- 提升失败 try/except 兜底,绝不阻塞模型创建本身
create_model now calls _maybe_promote_first_model_to_default: the new
model is promoted to the default LLM only when (1) the provider had no
models before, (2) no global default LLM exists, and (3) the provider
already has usable credentials. Failures are swallowed — model creation
never blocks on default promotion.

验证 / Verified:
- 新增 5 个回归测试(首个模型+凭据→设默认;inline key→设默认;
  无凭据→不设;第二个模型→不覆盖;其它 provider 已有默认→不覆盖),
  8/8 passed
- 相邻模块 53 passed / 5 skipped,零回归
- 真实环境数据修复:将 Qwen3.8-27B 设为默认后 /api/onboarding/status
  completed=True,登录不再弹
- 8 regression tests: 8/8 passed; neighboring modules: 53 passed / 5 skipped
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant