Skip to content

auto-fit: plan for integrated GPUs when no discrete GPU is present #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
danielhanchen wants to merge 1 commit into
base: master
Choose a base branch
from
Open

auto-fit: plan for integrated GPUs when no discrete GPU is present #51

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion src/core/backend_fit.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,28 @@ namespace sd::backend_fit {
static std::vector<Device> enumerate_gpu_devices(const sd::ggml_graph_cut::MaxVramAssignment& budgets,
bool include_other_devices) {
std::vector<Device> out;
// Like llama.cpp's default device list, integrated GPUs are planned for only when no discrete GPU
// exists, and only those of the first backend that reports one (an iGPU can appear once per backend).
bool has_discrete_gpu = false;
for (size_t i = 0; i < ggml_backend_dev_count(); ++i) {
if (ggml_backend_dev_type(ggml_backend_dev_get(i)) == GGML_BACKEND_DEVICE_TYPE_GPU) {
has_discrete_gpu = true;
break;
}
}
ggml_backend_reg_t igpu_reg = nullptr;
for (size_t i = 0; i < ggml_backend_dev_count(); ++i) {
ggml_backend_dev_t dev = ggml_backend_dev_get(i);
const auto type = ggml_backend_dev_type(dev);
if (type != GGML_BACKEND_DEVICE_TYPE_GPU &&
bool auto_igpu = false;
if (type == GGML_BACKEND_DEVICE_TYPE_IGPU && !include_other_devices && !has_discrete_gpu) {
ggml_backend_reg_t reg = ggml_backend_dev_backend_reg(dev);
if (igpu_reg != nullptr && reg != igpu_reg) {
continue;
}
auto_igpu = true;
}
if (type != GGML_BACKEND_DEVICE_TYPE_GPU && !auto_igpu &&
(!include_other_devices || type == GGML_BACKEND_DEVICE_TYPE_CPU)) {
continue;
}
Expand All @@ -142,6 +160,19 @@ namespace sd::backend_fit {
device.description = ggml_backend_dev_description(dev);
size_t free_bytes = 0, total_bytes = 0;
ggml_backend_dev_memory(dev, &free_bytes, &total_bytes);
if (auto_igpu) {
// An iGPU reports its share of the unified pool. Without a reading it cannot be planned for,
// so keep the default backend instead of falling back to CPU.
Comment on lines +163 to +165

@chatgpt-codex-connector chatgpt-codex-connector Bot Nov 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for shared RAM when planning iGPU parameter offloads

On UMA machines where GPU allocations consume available system RAM, admitting the iGPU to the existing planner treats GPU memory and RAM as independent pools: GPU placements reduce remaining, but never reduce ram_budget_bytes. For example, with 12 GiB free in the shared pool and 8/2/1 GiB diffusion/TE/VAE weights, compute_plan places diffusion on the GPU and TE/VAE in RAM. ModelManager::stage_tensors_to_compute_backend then allocates additional GPU copies of those offloaded weights while retaining their RAM backing, increasing physical memory consumption compared with the previous single-copy default placement. Near capacity this can force unnecessary segmentation, swapping, or allocation failures. Preserve shared-memory information in the device plan and account for GPU residency and staging against the same physical RAM budget before choosing CPU offloads.

Useful? React with 👍 / 👎.

if (free_bytes == 0 && total_bytes == 0) {
LOG_WARN("auto-fit: %s did not report memory; not planning for it", device.name.c_str());
continue;
}
// Vulkan's heap budget subtraction can wrap on iGPUs; the model manager reads that as total.
if (total_bytes > 0 && free_bytes > total_bytes) {
free_bytes = total_bytes;
}
igpu_reg = ggml_backend_dev_backend_reg(dev);
}
device.free_bytes = (int64_t)free_bytes;

float gib = budgets.default_gib;
Expand Down