fix(model): deterministic, type-filtered backend auto-detection (#9287)#10286
fix(model): deterministic, type-filtered backend auto-detection (#9287)#10286localai-bot wants to merge 3 commits into
Conversation
) When a model config declares no explicit `backend:`, Load() fell into a trial loop built by ranging the external-backends Go map (random order) with no filtering, returning the first backend whose gRPC LoadModel succeeded. An unrelated installed backend - e.g. the "opus" audio codec - could therefore win a GGUF/LLM model load, so a model that should run on llama.cpp wrongly tried to use opus. Extract the candidate selection into a pure, testable function SelectAutoLoadBackends that: - sorts the candidate list deterministically (no more map-order nondeterminism), and - for a `.gguf` model, filters to LLM-capable backends (via core/config.BackendCapabilities) and puts llama-cpp first, so an incompatible audio/codec/image backend can never win the trial loop. If filtering would leave zero candidates, the full sorted set is returned unchanged, so a previously-loadable model is never made unloadable. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
Heads up: this branch is now 427 commits behind
So a straight merge/rebase produces an import cycle:
To move forward this needs a real layering decision: either the auto-detect logic that needs |
…uto-detect The #9287 auto-detect change made pkg/model/autoload.go import core/config for the backend capability table. core/config already imports pkg/model (runtime_settings_registry.go uses model.DefaultWatchdogInterval), so this closed a core/config -> pkg/model -> core/config import cycle and broke the build and golangci-lint. Invert the dependency so the lower-level pkg/model no longer imports the higher-level core/config. pkg/model exposes RegisterLLMCapableBackendFunc and uses the registered predicate; core/config (which owns the capability table) registers it from an init(). The deterministic, GGUF-type-filtered selection behaviour is unchanged. When the predicate is unwired the GGUF filter is skipped, preserving the existing zero-candidate fallback. The unit test now injects a fake capability predicate so SelectAutoLoadBackends is exercised independently of the core/config table. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:opus-4.8 [Claude Code]
2 participants
Closes #9287
Problem
When a model config has no explicit
backend:,(*ModelLoader).Loadbuilt the auto-detect candidate list by ranging an unordered Go map of installed backends with no filtering, then loaded the first one whose gRPCLoadModelsucceeded. Every installed backend is registered there - including non-LLM ones like theopusaudio codec - so after installing such a backend it could win a GGUF/LLM load, sending the model to the wrong backend.Fix
New pure, unit-tested
SelectAutoLoadBackends(available, modelFile):.gguffiles, filters to LLM-capable backends (chat/completion/edit/embeddings usecases viacore/config.BackendCapabilities) withllama-cppfirst,Load()now calls this instead of ranging the map directly. (Verifiedpkg/model->core/configintroduces no import cycle.)Test plan
pkg/model/autoload_test.go(red -> green): given{opus, llama-cpp}+ a.gguf,opusis excluded andllama-cppis first; deterministic order; zero-candidate fallback returns the original set.go test ./pkg/model/... ./core/config/...green; scopedgolangci-lint --new-from-merge-baseclean.Follow-up (noted, not done)
Did not force
cfg.Backend = "llama-cpp"in the empty-backend GGUF hook (more blast radius on non-llama GGUFs); the candidate filter alone fixes the bug. A metadata-based GGUF architecture check is a possible refinement.Assisted-by: claude:claude-opus-4-8 [Claude Code]