Skip to content

Add model configs: GLM, DeepSeek V4, Inkling, Laguna, Laguna NVFP4 #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rounakbende10 wants to merge 3 commits into
base: main
Choose a base branch
from
+143 −4
Open

Add model configs: GLM, DeepSeek V4, Inkling, Laguna, Laguna NVFP4 #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fb4fca6
Add model configs: GLM, DeepSeek V4, Inkling, Laguna, Laguna NVFP4
rounakbende10 Sep 3, 2026
1cdad6b
Add hardware-specific args and verify B200 DeepSeek/Inkling configs
rounakbende10 Sep 9, 2026
ddd797b
Also match gpu-b200-sxm-a platform variant for DeepSeek hardware args
rounakbende10 Sep 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/coding_agent_bench/models/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@
RedHatAI_Mistral_Small_4_119B_2603_NVFP4,
RedHatAI_NVIDIA_Nemotron_3_Super_120B_A12B_NVFP4,
RedHatAI_Qwen3_6_27B_FP8,
RedHatAI_GLM_5_2_FP8,
RedHatAI_DeepSeek_V4_Flash,
RedHatAI_DeepSeek_V4_Flash_NVFP4_FP8,
RedHatAI_Inkling_Small,
RedHatAI_Laguna_S_2_1,
poolside_Laguna_S_2_1_NVFP4,
)

MODEL_CONFIGS: list[type[ModelConfig]] = [
Expand All @@ -15,6 +21,12 @@
RedHatAI_Mistral_Small_4_119B_2603_NVFP4,
RedHatAI_NVIDIA_Nemotron_3_Super_120B_A12B_NVFP4,
RedHatAI_Qwen3_6_27B_FP8,
RedHatAI_GLM_5_2_FP8,
RedHatAI_DeepSeek_V4_Flash,
RedHatAI_DeepSeek_V4_Flash_NVFP4_FP8,
RedHatAI_Inkling_Small,
RedHatAI_Laguna_S_2_1,
poolside_Laguna_S_2_1_NVFP4,
]

MODEL_REGISTRY: dict[str, ModelConfig] = {cls.name: cls() for cls in MODEL_CONFIGS}
Expand Down
3 changes: 2 additions & 1 deletion src/coding_agent_bench/models/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
from abc import ABC

class ModelConfig(ABC):

name: str
image: str = "vllm/vllm-openai:v0.24.0"
args: list[str]
model_max_len: int
hardware_extra_args: dict[str, list[str]] = {}
default_args: list[str] = [
"--gpu-memory-utilization", "0.9",
"--async-scheduling",
Expand Down
125 changes: 124 additions & 1 deletion src/coding_agent_bench/models/configs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ class RedHatAI_NVIDIA_Nemotron_3_Super_120B_A12B_NVFP4(ModelConfig):
]

class RedHatAI_Qwen3_6_27B_FP8(ModelConfig):

name = "RedHatAI/Qwen3.6-27B-FP8"
model_max_len = 131072
Comment thread
taagarwa-rh marked this conversation as resolved.
args = [
Expand All @@ -90,3 +90,126 @@ class RedHatAI_Qwen3_6_27B_FP8(ModelConfig):
"--tool-call-parser", "qwen3_coder",
"--default-chat-template-kwargs", '{"enable_thinking": true}',
]


class RedHatAI_GLM_5_2_FP8(ModelConfig):
# Verified: 8x H200 141GB, concurrency 2.23x at 262K context

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is enough concurrency for us to consider using vLLM over OpenRouter for GLM 5.2. Let's keep it in the PR though for informational purposes. How does it do on 8x B200?

# Note: cannot fit 1M context on 8x H200 (needs 52.68 GiB KV, only 23.78 GiB available)

name = "RedHatAI/GLM-5.2-FP8"
model_max_len = 262144
args = [
"--model", "RedHatAI/GLM-5.2-FP8",
"--dtype", "auto",
"--max-model-len", "262144",
"--trust-remote-code",
Comment thread
taagarwa-rh marked this conversation as resolved.
"--kv-cache-dtype", "fp8",
"--enable-expert-parallel",
"--enable-auto-tool-choice",
"--reasoning-parser", "glm45",
"--tool-call-parser", "glm47",
]


class RedHatAI_DeepSeek_V4_Flash(ModelConfig):
# Verified: 8x H200 141GB, concurrency 9.91x at 1M context
# B200: 14.02x at 1M with mega_moe + attention_config.use_fp4_indexer_cache

name = "RedHatAI/DeepSeek-V4-Flash"
image = "vllm/vllm-openai:v0.27.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/DeepSeek-V4-Flash",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--block-size", "256",
"--enable-expert-parallel",
"--enable-auto-tool-choice",
"--tokenizer-mode", "deepseek_v4",
"--tool-call-parser", "deepseek_v4",
"--reasoning-parser", "deepseek_v4",

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash mentions a couple flags I don't see here:

  --attention_config.use_fp4_indexer_cache True \
  --moe-backend deep_gemm_mega_mo

Did you test these and find they are not needed?

@rounakbende10 rounakbende10 Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--moe-backend deep_gemm_mega_moe is B200-only (SM100) crashes on H200 (SM90). We tested on H200 initially, so excluded it. On B200 we did test with mega_moe and got 12.81x vs 9.91x on H200 without it. We can add it as a comment noting it's B200-only, or add a separate B200 config with it enabled. what do you think?

--attention_config.use_fp4_indexer_cache True is used only for NVFP4 variant. I will confirm once again with this flag if theres any change in concurrency

@taagarwa-rh taagarwa-rh Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh that is something I hadn't considered. Should we add a per-hardware args section to the ModelConfigs? E.g.

from coding_agent_bench.nebius_utils import B200, B200x8

class RedHatAI_DeepSeek_V4_Flash(ModelConfig):
    ...
    hardware_extra_args: dict[str, list[str]] = {
        B200.name: ["--moe-backend", "deep_gemm_mega_moe"],
        B200x8.name: ["--moe-backend", "deep_gemm_mega_moe"],
    }
    ...

Then somewhere when building the vLLM command it can reference

args += model_config.hardware_extra_args.get(hardware.name, [])

Regarding --attention_config.use_fp4_indexer_cache True, I'm seeing that listed on the FP8 model as well, so I think it's needed for both oh that's only on the B200 as well. That would be another one to add to the B200 specific args

@rounakbende10 rounakbende10 Sep 3, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on the hardware_extra_args approach. start_model already extracts the GPU preset from the instance so we can match on that. I'll implement it. DeepSeek gets --moe-backend deep_gemm_mega_moe on B200s and --attention_config.use_fp4_indexer_cache True

@rounakbende10 rounakbende10 Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--attention_config.use_fp4_indexer_cache True worked for both variants added in latest commit also it increased concurrency by +1x on each

]
hardware_extra_args = {
"gpu-b200-sxm": ["--moe-backend", "deep_gemm_mega_moe", "--attention_config.use_fp4_indexer_cache", "True"],
"gpu-b200-sxm-a": ["--moe-backend", "deep_gemm_mega_moe", "--attention_config.use_fp4_indexer_cache", "True"],
}


class RedHatAI_DeepSeek_V4_Flash_NVFP4_FP8(ModelConfig):
# Verified: 8x H200 141GB, concurrency 9.81x at 1M context
# B200: 13.60x at 1M with native FP4 + attention_config.use_fp4_indexer_cache. Marlin fallback on H200.

name = "RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8"
image = "vllm/vllm-openai:v0.27.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--block-size", "256",
"--enable-expert-parallel",
"--enable-auto-tool-choice",
"--tokenizer-mode", "deepseek_v4",
"--tool-call-parser", "deepseek_v4",
"--reasoning-parser", "deepseek_v4",

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment above

]
hardware_extra_args = {
"gpu-b200-sxm": ["--attention_config.use_fp4_indexer_cache", "True"],
"gpu-b200-sxm-a": ["--attention_config.use_fp4_indexer_cache", "True"],
}


class RedHatAI_Inkling_Small(ModelConfig):
# Verified: 8x H200 141GB, BF16, concurrency 13.22x at 1M context
# --tokenizer-mode inkling is a no-op (vLLM already defaults to it for this architecture)
# --kernel-config.enable_flashinfer_autotune=False breaks fp8 KV cache (AssertionError in FA4 kernel) — do not add

name = "RedHatAI/Inkling-Small"
image = "vllm/vllm-openai:v0.27.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/Inkling-Small",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--enable-auto-tool-choice",
"--tool-call-parser", "inkling",
"--reasoning-parser", "inkling",

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://recipes.vllm.ai/thinkingmachines/Inkling-Small?variant=bf16 has a couple flags I don't see here:

  --tokenizer-mode inkling \
  --kernel-config.enable_flashinfer_autotune=False \

Did you test these and find they are not needed?

@rounakbende10 rounakbende10 Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tokenizer mode is inkling by default doesn't matter if we mention or not.
--kernel-config.enable_flashinfer_autotune=False can't be used with kv cache dtype as fp8 we are 160MB short of memory to run on BF16 kv cache dtype for 8Xh200

]


class RedHatAI_Laguna_S_2_1(ModelConfig):
# Verified: 8x H200 141GB, BF16, max-model-len 1048576, concurrency 30.61x

name = "RedHatAI/Laguna-S-2.1"
model_max_len = 1048576
args = [
"--model", "RedHatAI/Laguna-S-2.1",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--enable-auto-tool-choice",
"--reasoning-parser", "poolside_v1",
"--tool-call-parser", "poolside_v1",
"--default-chat-template-kwargs", '{"enable_thinking": true}',
]

class poolside_Laguna_S_2_1_NVFP4(ModelConfig):
# Verified: 1x B200 183GB, NVFP4, max-model-len 1048576, concurrency 2.33x

@taagarwa-rh taagarwa-rh Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for testing this, same as above looks like this won't be enough concurrency so we'll prefer the BF16 version you added. No changes needed here. Thanks again for trying this, this will serve as justification for using the larger node to run this model


name = "poolside/Laguna-S-2.1-NVFP4"
model_max_len = 1048576
args = [
"--model", "poolside/Laguna-S-2.1-NVFP4",
"--dtype", "auto",
"--trust-remote-code",
"--kv-cache-dtype", "fp8",
"--enable-auto-tool-choice",
"--reasoning-parser", "poolside_v1",
"--tool-call-parser", "poolside_v1",
"--default-chat-template-kwargs", '{"enable_thinking": true}',
]


7 changes: 5 additions & 2 deletions src/coding_agent_bench/nebius_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -429,8 +429,10 @@ async def start_model(self, instance_name: str, model_name: str):
else:
raise e

# Get the number of gpus from resources (e.g. 1gpu-16vcpu-200gb)
resources = instance_details.get("spec", {}).get("resources", {}).get("preset")
# Get the number of gpus and platform from resources (e.g. 1gpu-16vcpu-200gb)
spec_resources = instance_details.get("spec", {}).get("resources", {})
resources = spec_resources.get("preset")
platform = spec_resources.get("platform", "")
if resources is None:
raise ValueError("Unable to get resources from instance details")
tensor_parallel_size = int(resources.split("-")[0].replace("gpu", ""))
Expand All @@ -449,6 +451,7 @@ async def start_model(self, instance_name: str, model_name: str):
]
docker_cmd += [model_config.image]
docker_cmd += model_config.args + model_config.default_args
docker_cmd += model_config.hardware_extra_args.get(platform, [])
docker_cmd += ["--tensor-parallel-size", str(tensor_parallel_size)]

hf_token = os.environ.get("HF_TOKEN", "")
Expand Down