Add dedicated VLM model configuration and wire it through multi-tenant VLM creation #10

Copilot · 2025-12-15T09:31:12Z

VLM model choice was hardcoded and tied to the text LLM model, preventing .env configuration and clear LLM/VLM separation.

Configuration
- Added required vlm_model to LLMConfig via VLM_MODEL (no fallback defaults).
- Added optional VLM_API_KEY and VLM_BASE_URL for vendor-independent VLM access; falls back to LLM credentials only when VLM-specific values are absent.
- Preserved vlm_timeout.
Multi-tenant behavior
- Vision model creation now uses the dedicated VLM model and supports VLM-specific key/base URL with consistent fallback.
- Tenant config merge includes VLM model/key/base URL to allow per-tenant overrides.
Environment template
- Introduced VLM_MODEL as mandatory and documented optional VLM_API_KEY/VLM_BASE_URL in env.example to expose VLM selection and vendor choice.

Example:

# .env
VLM_MODEL=seed-1-6-250615  # required, independent from LLM_MODEL
# VLM_API_KEY=...
# VLM_BASE_URL=...

# runtime config usage
config.llm.vlm_model        # used for vision model calls
config.llm.vlm_base_url     # VLM-specific endpoint when provided

Original prompt

This section details on the original issue you should resolve

<issue_title>VLM模型配置混乱，硬编码与.env配置不一致</issue_title>
<issue_description>## 问题

LLM_MODEL 在 .env 中配置为 Tongyi-Zhiwen/QwenLong-L1-32B（纯文本模型）

实际 VLM 模型硬编码在代码中为 seed-1-6-250615（多模态模型）

没有在 .env 中提供 VLM_MODEL 配置选项

影响

用户无法在 .env 中配置 VLM 模型

LLM 和 VLM 的区别不明显，容易混淆

不符合多租户配置的"配置即代码"原则

代码位置

/src/multi_tenant.py 第276-369行：VLM 函数硬编码使用 LLM 配置中的模型

/src/config.py：缺少 VLM_MODEL 配置字段

.env 第17行：只有 LLM_MODEL，没有 VLM_MODEL

建议修复

在 .env 中添加独立的 VLM_MODEL 配置项

在 config.py 中为 LLMConfig 添加 vlm_model 字段

更新多租户实例创建逻辑，使用 vlm_model 而不是 llm 的 model</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes VLM模型配置混乱，硬编码与.env配置不一致 #9

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Co-authored-by: BukeLy <19304666+BukeLy@users.noreply.github.com>

BukeLy

按照Comment的来改进

BukeLy · 2025-12-15T09:47:10Z

src/config.py

    api_key: str = Field(..., description="LLM API Key")
    base_url: str = Field(..., description="LLM API Base URL")
    model: str = Field(default="seed-1-6-250615", description="LLM Model Name")
+    vlm_model: str = Field(default="seed-1-6-250615", description="VLM Model Name")


这个还是默认回退到seed了,user不设置不一定用这个.直接去掉默认行为,不设置就报错

已改为必填且仅使用 VLM_MODEL 配置，不再回退默认值（e5548d2）。

BukeLy · 2025-12-15T09:47:41Z

src/config.py

+    def _load_vlm_model(cls, v):
+        """
+        优先读取独立的 VLM_MODEL（保持与 LLM 文本模型解耦）
+        回退顺序：
+        1) 显式传入值
+        2) 环境变量 VLM_MODEL（独立配置）
+        3) 环境变量 LLM_VLM_MODEL（前缀形式）
+        4) 默认值
+        """
+        return v or os.getenv("VLM_MODEL") or os.getenv("LLM_VLM_MODEL") or "seed-1-6-250615"


为什么要多级回退?我们只兼容一个配置,不引入额外的复杂性

已去掉多级回退，仅接受独立的 VLM_MODEL（e5548d2）。

BukeLy · 2025-12-15T09:48:14Z

src/multi_tenant.py


        # 从配置中提取参数（支持租户覆盖）
-        model = llm_config.get("model", self.ark_model)
+        model = llm_config.get("vlm_model", self.ark_vlm_model)


ark_model是什么?你还在用火山的命名方式吗?

已移除 ark 命名，统一使用 llm/vlm 前缀并保持含义清晰（e5548d2）。

BukeLy · 2025-12-15T09:49:00Z

env.example

 LLM_BASE_URL="https://ark.ap-southeast.bytepluses.com/api/v3"
 # 使用的模型名称
 LLM_MODEL=seed-1-6-250615
+# 用于多模态图片理解的 VLM 模型（与 LLM_MODEL 解耦）
+VLM_MODEL=seed-1-6-250615
 # LLM 供应商标识（ark/openai/claude）
 LLM_PROVIDER=ark
 # VLM 图片理解 API 超时时间（秒，默认 120 秒）


LLM和VLM难道不可以换供应商吗?难道一定要用LLM的Base URL吗?

已支持 VLM 独立的 API Key/Base URL（env 示例已添加 VLM_API_KEY/VLM_BASE_URL），不再强制复用 LLM Base URL（e5548d2）。

BukeLy · 2025-12-15T09:54:44Z

@copilot 修改Review出来的问题 #10 (review)

Co-authored-by: BukeLy <19304666+BukeLy@users.noreply.github.com>

Copilot

Pull request overview

This PR decouples VLM (Vision Language Model) configuration from text LLM configuration, enabling independent configuration of vision models through environment variables. Previously, VLM model selection was hardcoded and tied to LLM settings, preventing users from configuring different models or API endpoints for vision tasks.

Key changes:

Introduced dedicated VLM configuration fields (vlm_model, vlm_api_key, vlm_base_url) with fallback to LLM credentials when VLM-specific values are not provided
Updated multi-tenant manager to use VLM-specific configuration for vision model creation, replacing hardcoded model references
Enhanced environment template with VLM configuration options and documentation

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
src/config.py	Added required `vlm_model` field and optional `vlm_api_key`/`vlm_base_url` fields to LLMConfig; removed unused `os` import
src/multi_tenant.py	Renamed `ark_` variables to `llm_` for clarity; added VLM-specific variables with fallback logic; updated `_create_vision_model_func` to use dedicated VLM configuration
src/tenant_config.py	Extended `_merge_llm_config` to include VLM fields in tenant configuration merge, enabling per-tenant VLM overrides
env.example	Added VLM_MODEL as required configuration; documented optional VLM_API_KEY and VLM_BASE_URL with fallback behavior explanation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-15T10:16:34Z

src/config.py

+    vlm_model: str = Field(..., description="VLM Model Name", alias="VLM_MODEL")
+    vlm_api_key: Optional[str] = Field(default=None, description="VLM API Key", alias="VLM_API_KEY")
+    vlm_base_url: Optional[str] = Field(default=None, description="VLM API Base URL", alias="VLM_BASE_URL")


The VLM fields use explicit aliases (VLM_MODEL, VLM_API_KEY, VLM_BASE_URL) that don't follow the LLM_ prefix defined in the Config class. For these aliases to work correctly, the LLMConfig.Config class needs populate_by_name = True. Without this setting, pydantic-settings will not be able to load these environment variables. Other config classes in this file that use aliases (like DeepSeekOCRConfig at line 158) already include this setting.

Copilot · 2025-12-15T10:16:35Z

env.example

+# 用于多模态图片理解的 VLM 模型（必填，独立于 LLM_MODEL）
+VLM_MODEL=seed-1-6-250615


The documentation says VLM_MODEL is required (必填), but in the code at src/config.py:24, vlm_model uses Field(...) which makes it required at the pydantic validation level. However, the comment also says it should be "independent from LLM_MODEL" (独立于 LLM_MODEL), but the default value for LLM_MODEL and the example value for VLM_MODEL are both the same: seed-1-6-250615. This creates confusion about whether they should actually be different models. Consider clarifying whether VLM_MODEL can use the same model as LLM_MODEL (for models that support both text and vision), or if they must be different models.

Suggested change

# 用于多模态图片理解的 VLM 模型（必填，独立于 LLM_MODEL）

VLM_MODEL=seed-1-6-250615

# 用于多模态图片理解的 VLM 模型（必填，独立于 LLM_MODEL，需支持图片输入）

VLM_MODEL=seed-vlm-1-6-250615 # 示例：与 LLM_MODEL 不同，需为支持视觉的模型

Initial plan

b7b1960

Copilot AI assigned Copilot and BukeLy Dec 15, 2025

Copilot started work on behalf of BukeLy December 15, 2025 09:31 View session

拆分VLM模型配置

de4c36f

Co-authored-by: BukeLy <19304666+BukeLy@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add VLM model configuration to .env and code~~ Add dedicated VLM model configuration and wire it through multi-tenant VLM creation Dec 15, 2025

Copilot AI requested a review from BukeLy December 15, 2025 09:46

Copilot finished work on behalf of BukeLy December 15, 2025 09:46

BukeLy reviewed Dec 15, 2025

View reviewed changes

Copilot started work on behalf of BukeLy December 15, 2025 09:55 View session

Copilot AI and others added 2 commits December 15, 2025 09:59

完善VLM配置独立化

e5548d2

Co-authored-by: BukeLy <19304666+BukeLy@users.noreply.github.com>

优化VLM密钥回退一致性

bb21225

Co-authored-by: BukeLy <19304666+BukeLy@users.noreply.github.com>

Copilot AI requested a review from BukeLy December 15, 2025 10:03

Copilot finished work on behalf of BukeLy December 15, 2025 10:03

BukeLy approved these changes Dec 15, 2025

View reviewed changes

BukeLy marked this pull request as ready for review December 15, 2025 10:12

Copilot AI review requested due to automatic review settings December 15, 2025 10:12

Copilot started reviewing on behalf of BukeLy December 15, 2025 10:12 View session

BukeLy merged commit f7e2565 into main Dec 15, 2025
7 checks passed

BukeLy deleted the copilot/add-vlm-model-configuration branch December 15, 2025 10:13

Copilot AI reviewed Dec 15, 2025

View reviewed changes

		# 用于多模态图片理解的 VLM 模型（必填，独立于 LLM_MODEL）
		VLM_MODEL=seed-1-6-250615

Add dedicated VLM model configuration and wire it through multi-tenant VLM creation #10

Add dedicated VLM model configuration and wire it through multi-tenant VLM creation #10

Uh oh!

Conversation

Copilot AI commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

影响

代码位置

建议修复

Comments on the Issue (you are @copilot in this section)

Uh oh!

BukeLy left a comment

Choose a reason for hiding this comment

Uh oh!

BukeLy Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

BukeLy Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

BukeLy Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

BukeLy Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

BukeLy commented Dec 15, 2025

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Dec 15, 2025 •

edited

Loading