Skip to content

refactor: add new module for backend-specific code#2308

Open
wbruna wants to merge 1 commit into
LostRuins:concedo_experimentalfrom
wbruna:kcpp_backend_checks
Open

refactor: add new module for backend-specific code#2308
wbruna wants to merge 1 commit into
LostRuins:concedo_experimentalfrom
wbruna:kcpp_backend_checks

Conversation

@wbruna

@wbruna wbruna commented Jul 4, 2026

Copy link
Copy Markdown

This is an initial attempt to replace build-time backend checks with runtime ones.

My main intention is to eventually be able to decouple all code from fixed ggml backends, which would give us the ability to have a distributed binary for any CPU instruction set, use more than one GPU backend type at the same time, etc. But even if that's not achievable (or desired!), I believe Koboldcpp can still benefit from changes in that direction: reduced build times, lower chance of building OK on one backend and failing on another, smaller binaries, etc.

I'm submitting very early to validate the approach before spending too much time on this, so right now it only replaces some easier parts of gpttype_adapter: checks for tensor splitting, BLAS support, etc. I've also refactored the tensor splitting check to avoid a bit of repetitive code.

The runtime backend tests are very similar to the ones used by stable-diffusion.cpp, identifying the backends by their names (with a bit of convenience code to be able to check several at once). Right now, I'm always checking the first device, so it should give the same results as the compile-time checks; but the API is already prepared to accept backend pointers.

There should be only one functional change: the check

    #if defined(GGML_USE_CUDA)
    mtmd_fa = LLAMA_FLASH_ATTN_TYPE_DISABLED; //kcpp: disabled in 1.102.2 as some headsizes break on turing
    #endif

also triggered for ROCm; so I've replaced it with a straight CUDA-only check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant