fix(nv_build): retire the dead models and point the default at a served one - #391
Open
Mark2Mac wants to merge 2 commits into
Open
fix(nv_build): retire the dead models and point the default at a served one#391Mark2Mac wants to merge 2 commits into
Mark2Mac wants to merge 2 commits into
Conversation
The bundled registry gave z-ai/glm-5.2 a 1000000-token context window and no
output cap. model_info derives the output budget as
ctx * (1 - MAX_INPUT_TOKENS_PCT), so every request asked for 250000 output
tokens and the endpoint answered:
400 This model configuration accepts at most 202749 combined input and output
tokens. However, your request has 1249 input tokens and asks for 250000
output tokens (251249 tokens total).
202749 is quoted verbatim by the endpoint in that 400. With the entry corrected
the same scan completes with 4/4 LLM calls and the meta-analyzer applied.
An over-stated context window does not degrade gracefully: it zeroes the LLM
stage, and nothing in the error points at the registry. Under-stating is safe,
over-stating is not.
Limits may vary per account, which is now noted in the YAML.
Scope is deliberately one entry. The registry also names three models the
catalogue no longer serves, but removing them is coupled to DEFAULT_MODEL by an
invariant the suite already asserts ("nv_build's default model is in its
registry"), so that change travels with the default in a separate PR.
Refs NVIDIA#388
Signed-off-by: Mark2Mac <Mark2Mac@users.noreply.github.com>
…ed one Depends on NVIDIA#390. Three registry entries name models GET /v1/models does not serve: deepseek-v4-flash (410 Gone since 2026-08-07), deepseek-v4-pro, and glm-5.1. One of them is DEFAULT_MODEL, and another is the meta_analyzer slot override, so with no SKILLSPECTOR_MODEL set the out-of-the-box path failed every call. Removing them and retargeting the default is ONE change, not two. The suite already asserts the invariant that couples them, in test_constants: "nv_build's default model is in its registry — no warnings expected" Dropping the default from the registry while leaving it as the default breaks that test, and it is right to break: a default the registry does not describe gets its token budget from a guess, silently. Splitting these two edits was tried and abandoned for exactly this reason. The replacement is chosen for DETECTION, not latency, and that is the part worth arguing about. On a bait skill carrying credential exfiltration disguised as a synchronisation step, the fast served model (deepseek-v4-flash-0731, ~1.6 s/call) completed every call, reported no degradation, and returned a clean verdict. A confident false negative is the worst failure mode a security scanner has: it is the case where someone signs off. z-ai/glm-5.2 costs ~16 s per call on the same skill and returns CRITICAL. The meta_analyzer slot loses its override rather than gaining a new one. The aggregation pass does benefit from a stronger model, but naming a second model doubles the surface that can go stale — which is how the previous default rotted unnoticed. Happy to restore an override if preferred. Refs NVIDIA#388 Signed-off-by: Mark2Mac <Mark2Mac@users.noreply.github.com>
Mark2Mac
force-pushed
the
fix/nv-build-default-model
branch
from
August 18, 2026 09:21
554a247 to
315b0c2
Compare
Mark2Mac
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The other half of #388. Depends on #390 — merge that first.
Three registry entries name models
GET /v1/modelsdoes not serve:deepseek-ai/deepseek-v4-flashDEFAULT_MODELdeepseek-ai/deepseek-v4-prometa_analyzerslotz-ai/glm-5.1With no
SKILLSPECTOR_MODELset, the out-of-the-box path failed every call.Why this is one change and not two. Removing the dead entries and retargeting the default cannot be separated: the suite already asserts the invariant that couples them, in
test_constants.py—Dropping the default from the registry while leaving it as the default breaks that test, and it is right to break: a default the registry does not describe gets its token budget from a guess, silently. I attempted the split and abandoned it for this reason.
The part worth arguing about. The replacement is chosen for detection, not latency:
deepseek-v4-flash-0731z-ai/glm-5.2A confident false negative is the worst failure mode a security scanner has: it is the case where someone signs off. The fast model does not report uncertainty — it reports absence.
On dropping the slot override.
meta_analyzerloses its override rather than gaining a new one. The aggregation pass does benefit from a stronger model, but naming a second model doubles the surface that can go stale — which is how the previous default rotted unnoticed. Happy to restore one if you'd rather.1044 passed, ruff format and check clean, DCO signed.