refactor: add new module for backend-specific code#2308
Open
wbruna wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an initial attempt to replace build-time backend checks with runtime ones.
My main intention is to eventually be able to decouple all code from fixed ggml backends, which would give us the ability to have a distributed binary for any CPU instruction set, use more than one GPU backend type at the same time, etc. But even if that's not achievable (or desired!), I believe Koboldcpp can still benefit from changes in that direction: reduced build times, lower chance of building OK on one backend and failing on another, smaller binaries, etc.
I'm submitting very early to validate the approach before spending too much time on this, so right now it only replaces some easier parts of
gpttype_adapter: checks for tensor splitting, BLAS support, etc. I've also refactored the tensor splitting check to avoid a bit of repetitive code.The runtime backend tests are very similar to the ones used by stable-diffusion.cpp, identifying the backends by their names (with a bit of convenience code to be able to check several at once). Right now, I'm always checking the first device, so it should give the same results as the compile-time checks; but the API is already prepared to accept backend pointers.
There should be only one functional change: the check
also triggered for ROCm; so I've replaced it with a straight CUDA-only check.