Enhance performance for GpuToCpu for intel integrated GPU

[hmaarrfk]

I noticed serious slowdowns in copying from Intel integrated GPU -> CPU on systems that have both a intel CPU and GPU.

I'm using this

python -> wgpu-native -> wgpu-rs -> gpu-allocator

they seem to have moved to your allocator, and I am now seeing major slowdowns in copying data from the GPU to the CPU:

• Intel Core ultra 200 series
• Nvidia GPU available on the system. not sure if this matters
• Actually Using The Intel Integrated GPU

This little patch seems to resolve my problems.

An other solution was to use Non-temporal copy, but that is really niche and seems like a workaround.

I'm very rusty on memory profiling, so I had to entrust my claude..... but I do think there is a small reproducer in here (you will be better at reading this than me).

I do read enough code that we are inteligently trying to select memory of the right type. so I trust that this PR will be useful.

claude teaching me things every day, read at your own risk

Ready-to-paste GitHub issue

Repository: https://github.com/gfx-rs/wgpu

---

Title: Vulkan/ANV: MAP_READ buffers land on write-combined memory since v28 (gpu-allocator switch) - mapped readback ~87x slower on Intel iGPUs

---

Summary

Since wgpu v28.0.0, reading back a mapped MAP_READ buffer on Intel integrated
GPUs (Mesa ANV) is about 87x slower than it was on v27 - 0.29 GB/s instead of
25 GB/s, i.e. slower than the PCIe readback it is supposed to avoid.

The MAP_READ allocation now lands on a write-combined (HOST_COHERENT,
uncached) memory type instead of the HOST_CACHED type that ANV also exposes.
On write-combined memory CPU reads are uncached, so a plain memcpy out of the
mapped range runs at ~0.28 GB/s.

This is a regression from the Vulkan backend's switch from gpu-alloc to
gpu-allocator (#8158). Discrete NVIDIA cards are unaffected, because they expose a
memory type that is simultaneously HOST_VISIBLE | HOST_COHERENT | HOST_CACHED, so
gpu-allocator's preferred flag mask matches on the first try.

Bisected to a version boundary: v27 fast, v28.0.0 slow, v29.0.4 slow - same
machine, same driver, same source file.

Environment

GPU	Intel Arrow Lake iGPU - Intel(R) Graphics (ARL), IntegratedGpu
CPU	Intel Core Ultra 7 270K Plus
Driver	Intel open-source Mesa driver, Mesa 26.0.3-1ubuntu1 (ANV)
Backend	Vulkan
OS	Ubuntu 26.04 LTS, kernel 7.0.0-29-generic
rustc	1.97.1
wgpu	27.0.1 (good) vs 28.0.0 / 29.0.4 (bad)

The same regression has been observed on nine machines: every Intel Vulkan adapter
on wgpu >= 28 is slow, every NVIDIA adapter is fast, and Intel adapters on wgpu 27
are fast.

Measurements

8 MiB buffer, median of 9 copy_from_slice calls out of the mapped range, with
the destination pre-warmed so first-touch page faults are not timed. The same-sized
host-to-host memcpy in the second column is the RAM baseline measured in the same
process, so the ratio is self-normalising.

wgpu	mapped read	host memcpy	ratio	verdict
27.0.1	0.337 ms - 24.9 GB/s	0.334 ms - 25.1 GB/s	1.0x	ok
28.0.0	29.30 ms - 0.29 GB/s	0.338 ms - 24.8 GB/s	86.6x	regressed
29.0.4	29.29 ms - 0.29 GB/s	0.339 ms - 24.8 GB/s	86.4x	regressed

Three consecutive runs of each version varied by under 1%.

Root cause

ANV on parts without a CPU-shared LLC exposes no memory type that is both
HOST_COHERENT and HOST_CACHED:

=== Intel(R) Graphics (ARL) (INTEGRATED_GPU) ===
  type  0  heap 0  DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT     <- write-combined
  type  1  heap 0  DEVICE_LOCAL | HOST_VISIBLE | HOST_CACHED       <- cached, not coherent
  type  2  heap 0  DEVICE_LOCAL | PROTECTED
  type  3  heap 0  DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT
  type  4  heap 0  DEVICE_LOCAL | HOST_VISIBLE | HOST_CACHED
  heap  0  92.11 GiB  DEVICE_LOCAL

  gpu-allocator GpuToCpu preferred (HOST_VISIBLE|HOST_COHERENT|HOST_CACHED) -> None
  gpu-allocator GpuToCpu fallback  (HOST_VISIBLE|HOST_COHERENT)             -> Some(0)
  a cached host-visible type exists at                                      -> Some(1)

Measuring a CPU memcpy out of each host-visible type directly, with plain ash
and no wgpu involved, shows exactly what landing on the wrong one costs
(8 MiB, median of 9):

    type  0    30.202 ms     0.28 GB/s   [DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT]
    type  1     0.335 ms    25.06 GB/s   [DEVICE_LOCAL | HOST_VISIBLE | HOST_CACHED]
    type  3    30.364 ms     0.28 GB/s   [DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT]
    type  4     0.334 ms    25.09 GB/s   [DEVICE_LOCAL | HOST_VISIBLE | HOST_CACHED]

The ~90x gap in the wgpu numbers is exactly the gap between memory type 0 and
memory type 1. As a control, llvmpipe on the same machine exposes a single
HOST_VISIBLE | HOST_COHERENT | HOST_CACHED type, the preferred mask matches, and
there is no regression.

Why v28 changed behaviour

wgpu-hal 29.0.4 src/vulkan/device.rs:897 maps a read-mappable buffer to
gpu_allocator::MemoryLocation::GpuToCpu:

let location = match (is_cpu_read, is_cpu_write) {
    (true, true) => gpu_allocator::MemoryLocation::CpuToGpu,
    (true, false) => gpu_allocator::MemoryLocation::GpuToCpu,
    (false, true) => gpu_allocator::MemoryLocation::CpuToGpu,
    (false, false) => gpu_allocator::MemoryLocation::GpuOnly,
};

gpu-allocator 0.28.0 src/vulkan/mod.rs:810 then tries two hard flag masks,
and both of them require HOST_COHERENT:

MemoryLocation::GpuToCpu => {
    vk::MemoryPropertyFlags::HOST_VISIBLE
        | vk::MemoryPropertyFlags::HOST_COHERENT
        | vk::MemoryPropertyFlags::HOST_CACHED     // preferred -> no match on ANV
}
// ...
MemoryLocation::CpuToGpu | MemoryLocation::GpuToCpu => {
    vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT
}                                                  // required  -> type 0, write-combined

There is no third attempt, so HOST_CACHED-but-not-coherent memory is unreachable
for a MAP_READ buffer.

gpu-alloc, used up to v27, treated these as a priority ordering rather than a
requirement. wgpu-hal 27.0.1 src/vulkan/device.rs:1113 set
gpu_alloc::UsageFlags::DOWNLOAD ("Allocator will strongly prefer host-cached
memory"), and gpu-alloc 0.6.2 src/usage.rs:164 sorts candidates by

let host_cached: bool =
    flags.contains(Flags::HOST_CACHED) ^ usage.contains(UsageFlags::DOWNLOAD);

a preference, not a filter - so HOST_CACHED won and coherency was allowed to lose.

Suggested fix

For GpuToCpu, prefer HOST_VISIBLE | HOST_CACHED over
HOST_VISIBLE | HOST_COHERENT when the fully-preferred mask does not match. Reading
back is the entire point of the allocation, and cached-but-non-coherent serves that
far better than coherent-but-uncached.

This looks safe on the wgpu side: the Vulkan backend already supports non-coherent
mappings. wgpu-hal 29.0.4 src/vulkan/device.rs:993 records

let is_coherent = allocation
    .memory_properties()
    .contains(vk::MemoryPropertyFlags::HOST_COHERENT);

and flush_mapped_ranges / invalidate_mapped_ranges already issue
vkFlushMappedMemoryRanges / vkInvalidateMappedMemoryRanges. So the only thing
between ANV and the fast memory type is gpu-allocator's hard HOST_COHERENT
requirement in the fallback mask.

Two places it could be fixed:

1. In gpu-allocator: add a third attempt (HOST_VISIBLE | HOST_CACHED) for
   GpuToCpu, or make the selection a scored preference rather than a subset test.
2. In wgpu-hal: bypass MemoryLocation for MAP_READ buffers and select the
   memory type explicitly, picking a HOST_CACHED type when no coherent+cached type
   exists.

Reproducer

Two files, no shaders, no windowing. cargo run --release finishes in well under a
minute.

Cargo.toml:

[package]
name = "map-read-repro"
version = "0.1.0"
edition = "2021"

[dependencies]
# Change this to "27" to get the fast (gpu-alloc) Vulkan backend,
# or "29" / "28" to get the slow (gpu-allocator) one.
wgpu = "29"
pollster = "0.4"

[features]
# `Device::generate_allocator_report` only exists on wgpu >= 28.
alloc-report = []

[profile.release]
debug = false

build.rs (present only so the output states which wgpu it was built against):

// Bake the resolved `wgpu` version into the binary so the output is self-describing.
fn main() {
    let lock = std::fs::read_to_string("Cargo.lock").unwrap_or_default();
    let mut ver = "unknown".to_string();
    for pkg in lock.split("[[package]]") {
        if pkg.contains("name = \"wgpu\"\n") {
            for line in pkg.lines() {
                if let Some(v) = line.strip_prefix("version = ") {
                    ver = v.trim().trim_matches('"').to_string();
                }
            }
        }
    }
    println!("cargo:rustc-env=WGPU_VERSION={ver}");
    println!("cargo:rerun-if-changed=Cargo.lock");
}

src/main.rs:

//! Minimal reproducer: reading back a mapped MAP_READ buffer is ~90x slower
//! than plain RAM on Intel integrated GPUs (Vulkan/ANV) from wgpu v28 onward.
//!
//!     cargo run --release                 # wgpu version comes from Cargo.toml
//!     cargo run --release -F alloc-report # + Device::generate_allocator_report (wgpu >= 28)
//!
//! Exits 1 when the mapped read is more than 5x slower than a host memcpy of
//! the same size.

use std::time::Instant;

/// 8 MiB: big enough to be well out of L3, small enough to map instantly.
const BYTES: usize = 8 * 1024 * 1024;
const REPS: usize = 9;
/// Mapped memory is expected to be a bit slower than RAM; 5x is the alarm line.
const SLOWDOWN_LIMIT: f64 = 5.0;

fn stats(mut v: Vec<f64>) -> (f64, f64) {
    v.sort_by(|a, b| a.partial_cmp(b).unwrap());
    (v[v.len() / 2], v[0]) // (median, best)
}

fn gbps(bytes: usize, secs: f64) -> f64 {
    bytes as f64 / secs / 1e9
}

fn main() {
    // Default to Vulkan; override with WGPU_BACKEND=gl|metal|dx12.
    let backends = match std::env::var("WGPU_BACKEND").ok().as_deref() {
        Some("gl") => wgpu::Backends::GL,
        Some("metal") => wgpu::Backends::METAL,
        Some("dx12") => wgpu::Backends::DX12,
        _ => wgpu::Backends::VULKAN,
    };
    let mut desc = wgpu::InstanceDescriptor::new_without_display_handle();
    desc.backends = backends;
    let instance = wgpu::Instance::new(desc);
    let adapter = pollster::block_on(instance.request_adapter(&Default::default()))
        .expect("no adapter");
    let info = adapter.get_info();
    let (device, queue) = pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor {
        label: Some("map-read-repro"),
        ..Default::default()
    }))
    .expect("no device");

    println!("wgpu crate   : {}", env!("WGPU_VERSION"));
    println!("adapter      : {}", info.name);
    println!("backend      : {:?}  device_type: {:?}", info.backend, info.device_type);
    println!("driver       : {} {}", info.driver, info.driver_info);
    println!("buffer       : {} MiB\n", BYTES / (1024 * 1024));

    // ---- GPU buffer: COPY_DST | MAP_READ, filled from the host ----
    let buffer = device.create_buffer(&wgpu::BufferDescriptor {
        label: Some("readback"),
        size: BYTES as u64,
        usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
        mapped_at_creation: false,
    });
    let pattern: Vec<u8> = (0..BYTES).map(|i| (i % 251) as u8).collect();
    queue.write_buffer(&buffer, 0, &pattern);
    queue.submit([]);

    let slice = buffer.slice(..);
    let (tx, rx) = std::sync::mpsc::channel();
    slice.map_async(wgpu::MapMode::Read, move |r| {
        let _ = tx.send(r);
    });
    device.poll(wgpu::PollType::wait_indefinitely()).unwrap();
    rx.recv().unwrap().expect("map_async failed");
    let mapped = slice.get_mapped_range();

    // Warm the destination so first-touch page faults are not timed.
    let mut dst = vec![0u8; BYTES];
    dst.copy_from_slice(&mapped);
    let mut sink = 0u64;

    // ---- 1. read out of the mapped range ----
    let mut map_times = Vec::with_capacity(REPS);
    for _ in 0..REPS {
        let t = Instant::now();
        dst.copy_from_slice(&mapped);
        map_times.push(t.elapsed().as_secs_f64());
        sink += dst[BYTES - 1] as u64;
    }

    // ---- 2. same-sized host-to-host memcpy, as the RAM baseline ----
    let mut host_times = Vec::with_capacity(REPS);
    for _ in 0..REPS {
        let t = Instant::now();
        dst.copy_from_slice(&pattern);
        host_times.push(t.elapsed().as_secs_f64());
        sink += dst[BYTES - 1] as u64;
    }

    assert_eq!(dst[..64], pattern[..64], "readback mismatch (sink={sink})");

    let (map_med, map_best) = stats(map_times);
    let (host_med, host_best) = stats(host_times);

    println!(
        "mapped buffer -> host : {:8.3} ms median  ({:6.3} ms best)   {:7.2} GB/s",
        map_med * 1e3,
        map_best * 1e3,
        gbps(BYTES, map_med)
    );
    println!(
        "host memcpy (same size): {:8.3} ms median  ({:6.3} ms best)   {:7.2} GB/s",
        host_med * 1e3,
        host_best * 1e3,
        gbps(BYTES, host_med)
    );

    let ratio = map_med / host_med;
    println!("\nmapped read is {ratio:.1}x slower than plain RAM");

    #[cfg(feature = "alloc-report")]
    dump_allocator_report(&device);

    drop(mapped);
    buffer.unmap();

    if ratio > SLOWDOWN_LIMIT {
        println!("VERDICT: FAIL - mapped readback is on non-cached (write-combined) memory");
        std::process::exit(1);
    }
    println!("VERDICT: OK - mapped readback is on cached memory");
}

#[cfg(feature = "alloc-report")]
fn dump_allocator_report(device: &wgpu::Device) {
    let Some(report) = device.generate_allocator_report() else {
        println!("\n(no allocator report available on this backend)");
        return;
    };
    println!(
        "\nallocator report: {} allocations, {} reserved / {} allocated bytes",
        report.allocations.len(),
        report.total_reserved_bytes,
        report.total_allocated_bytes
    );
    for alloc in report.allocations.iter().filter(|a| a.size as usize >= BYTES) {
        println!("  {:?}  offset={}  size={}", alloc.name, alloc.offset, alloc.size);
    }
    println!("  (note: AllocatorReport carries no VkMemoryType index - see memtypes/ for that)");
}

Set wgpu = "27" in Cargo.toml for the good version. wgpu 27 and 28 took
InstanceDescriptor by reference and still derived Default, so those two builds
need one cosmetic hunk, unrelated to the regression:

-    let mut desc = wgpu::InstanceDescriptor::new_without_display_handle();
-    desc.backends = backends;
-    let instance = wgpu::Instance::new(desc);
+    let instance = wgpu::Instance::new(&wgpu::InstanceDescriptor {
+        backends,
+        ..Default::default()
+    });

Output

wgpu 29.0.4, cargo run --release --features alloc-report:

wgpu crate   : 29.0.4
adapter      : Intel(R) Graphics (ARL)
backend      : Vulkan  device_type: IntegratedGpu
driver       : Intel open-source Mesa driver Mesa 26.0.3-1ubuntu1
buffer       : 8 MiB

mapped buffer -> host :   29.250 ms median  (28.648 ms best)      0.29 GB/s
host memcpy (same size):    0.338 ms median  ( 0.333 ms best)     24.79 GB/s

mapped read is 86.5x slower than plain RAM

allocator report: 4 allocations, 67108864 reserved / 8913224 allocated bytes
  "readback"  offset=524816  size=8388656
VERDICT: FAIL - mapped readback is on non-cached (write-combined) memory

wgpu 27.0.1, same machine, same driver:

wgpu crate   : 27.0.1
adapter      : Intel(R) Graphics (ARL)
backend      : Vulkan  device_type: IntegratedGpu
driver       : Intel open-source Mesa driver Mesa 26.0.3-1ubuntu1
buffer       : 8 MiB

mapped buffer -> host :    0.300 ms median  ( 0.278 ms best)     27.94 GB/s
host memcpy (same size):    0.265 ms median  ( 0.254 ms best)     31.70 GB/s

mapped read is 1.1x slower than plain RAM
VERDICT: OK - mapped readback is on cached memory

The process exits non-zero when the mapped read is more than 5x slower than RAM.

Device::generate_allocator_report, added by the same PR, confirms the buffer is a
single ~8 MiB suballocation, but AllocatorReport carries no VkMemoryType index,
so it cannot show which memory type was chosen - which is why the ash dump above
is included. Exposing the memory type index on AllocationReport would make this
class of problem diagnosable from wgpu alone.

Memory-type dumper

The ash-only helper that produced the two dumps above (memtypes/, ash = "0.38"):

//! Dumps VkPhysicalDeviceMemoryProperties, so the memory types available to
//! the allocator are visible next to the timing numbers.

use ash::vk;
use std::time::Instant;

const BYTES: usize = 8 * 1024 * 1024;
const REPS: usize = 9;

fn flag_names(f: vk::MemoryPropertyFlags) -> String {
    let mut v = Vec::new();
    for (bit, name) in [
        (vk::MemoryPropertyFlags::DEVICE_LOCAL, "DEVICE_LOCAL"),
        (vk::MemoryPropertyFlags::HOST_VISIBLE, "HOST_VISIBLE"),
        (vk::MemoryPropertyFlags::HOST_COHERENT, "HOST_COHERENT"),
        (vk::MemoryPropertyFlags::HOST_CACHED, "HOST_CACHED"),
        (vk::MemoryPropertyFlags::LAZILY_ALLOCATED, "LAZILY_ALLOCATED"),
        (vk::MemoryPropertyFlags::PROTECTED, "PROTECTED"),
    ] {
        if f.contains(bit) {
            v.push(name);
        }
    }
    v.join(" | ")
}

fn main() {
    let entry = unsafe { ash::Entry::load() }.expect("no vulkan loader");
    let app = vk::ApplicationInfo::default().api_version(vk::API_VERSION_1_1);
    let ci = vk::InstanceCreateInfo::default().application_info(&app);
    let instance = unsafe { entry.create_instance(&ci, None) }.expect("create_instance");

    for pd in unsafe { instance.enumerate_physical_devices() }.unwrap() {
        let props = unsafe { instance.get_physical_device_properties(pd) };
        let name = props
            .device_name_as_c_str()
            .unwrap()
            .to_string_lossy()
            .to_string();
        println!("=== {name} ({:?}) ===", props.device_type);

        let mem = unsafe { instance.get_physical_device_memory_properties(pd) };
        for i in 0..mem.memory_type_count as usize {
            let t = mem.memory_types[i];
            println!(
                "  type {i:>2}  heap {}  {}",
                t.heap_index,
                flag_names(t.property_flags)
            );
        }
        for i in 0..mem.memory_heap_count as usize {
            let h = mem.memory_heaps[i];
            println!(
                "  heap {i:>2}  {:.2} GiB  {:?}",
                h.size as f64 / (1024.0 * 1024.0 * 1024.0),
                h.flags
            );
        }

        // What gpu-allocator's MemoryLocation::GpuToCpu asks for, in order.
        let hv = vk::MemoryPropertyFlags::HOST_VISIBLE;
        let hc = vk::MemoryPropertyFlags::HOST_COHERENT;
        let cached = vk::MemoryPropertyFlags::HOST_CACHED;
        let preferred = hv | hc | cached;
        let fallback = hv | hc;
        let pick = |want: vk::MemoryPropertyFlags| {
            (0..mem.memory_type_count as usize)
                .find(|&i| mem.memory_types[i].property_flags.contains(want))
        };
        println!("  gpu-allocator GpuToCpu preferred (HOST_VISIBLE|HOST_COHERENT|HOST_CACHED) -> {:?}", pick(preferred));
        println!("  gpu-allocator GpuToCpu fallback  (HOST_VISIBLE|HOST_COHERENT)             -> {:?}", pick(fallback));
        println!("  a cached host-visible type exists at                                      -> {:?}", pick(hv | cached));

        // Measure CPU read bandwidth out of every host-visible memory type, so
        // the cost of landing on the wrong one is visible directly.
        println!("\n  CPU read bandwidth out of each HOST_VISIBLE memory type ({} MiB, median of {}):", BYTES / (1024 * 1024), REPS);
        let device = match make_device(&instance, pd) {
            Some(d) => d,
            None => { println!("    (could not create a logical device)"); continue; }
        };
        for i in 0..mem.memory_type_count {
            let t = mem.memory_types[i as usize];
            if !t.property_flags.contains(hv) { continue; }
            match bench_memory_type(&device, i) {
                Some(secs) => println!(
                    "    type {i:>2}  {:8.3} ms  {:7.2} GB/s   [{}]",
                    secs * 1e3,
                    BYTES as f64 / secs / 1e9,
                    flag_names(t.property_flags)
                ),
                None => println!("    type {i:>2}  (allocation failed)"),
            }
        }
        unsafe { device.destroy_device(None) };
        println!();
    }

    unsafe { instance.destroy_instance(None) };
}

fn make_device(instance: &ash::Instance, pd: vk::PhysicalDevice) -> Option<ash::Device> {
    let qfams = unsafe { instance.get_physical_device_queue_family_properties(pd) };
    let qi = qfams
        .iter()
        .position(|q| q.queue_flags.contains(vk::QueueFlags::GRAPHICS | vk::QueueFlags::TRANSFER))
        .or_else(|| qfams.iter().position(|q| q.queue_flags.contains(vk::QueueFlags::TRANSFER)))?
        as u32;
    let prio = [1.0f32];
    let qci = [vk::DeviceQueueCreateInfo::default()
        .queue_family_index(qi)
        .queue_priorities(&prio)];
    let dci = vk::DeviceCreateInfo::default().queue_create_infos(&qci);
    unsafe { instance.create_device(pd, &dci, None) }.ok()
}

/// Allocate `BYTES` from `type_index`, map it, and time a memcpy out of it.
fn bench_memory_type(device: &ash::Device, type_index: u32) -> Option<f64> {
    let ai = vk::MemoryAllocateInfo::default()
        .allocation_size(BYTES as u64)
        .memory_type_index(type_index);
    let memory = unsafe { device.allocate_memory(&ai, None) }.ok()?;
    let ptr = unsafe {
        device.map_memory(memory, 0, BYTES as u64, vk::MemoryMapFlags::empty())
    }
    .ok()? as *mut u8;
    let src = unsafe { std::slice::from_raw_parts(ptr, BYTES) };

    // Write once so the pages are backed, then warm the destination.
    unsafe { std::ptr::write_bytes(ptr, 0x5a, BYTES) };
    let mut dst = vec![0u8; BYTES];
    dst.copy_from_slice(src);

    let mut times = Vec::with_capacity(REPS);
    for _ in 0..REPS {
        let t = Instant::now();
        dst.copy_from_slice(src);
        times.push(t.elapsed().as_secs_f64());
    }
    std::hint::black_box(&dst);
    unsafe {
        device.unmap_memory(memory);
        device.free_memory(memory, None);
    }
    times.sort_by(|a, b| a.partial_cmp(b).unwrap());
    Some(times[times.len() / 2])
}

xref: #260

[Jasper-Bekkers]

Odd, you wouldn't expect a driver to even hand out write combined memory when allocating read back memory. Might be worth also filing an ANV bug around this since it makes sense to investigate this together with them.

This also changes the contract we have a bit; which is that before folks didn't have to flush memory ranges at all since all memory we handed out was coherent.

[Jasper-Bekkers]

Can you ask fable to double check this, and to add asserts on the WC flag on your machine? If it triggers I would strongly suggest taking this upstream as we won't be the only consumer suffering from this performance problem.

Edit: let me read the claude report for a sec.

[Jasper-Bekkers]

@manon-traverse @MarijnS95 I think we should discuss what to do here: the bug report seems valid and correct (somehow we're handing out WC memory for readback).

However: switching this has some implications, primarily around the contract we have for "we never need to explicitly flush" and potentially also around our persistently mapped buffers. The way I see it there are 3 or 4 options.

1. We talk to the ANV driver folks about this and get them to hand out COHERENT|CACHED (like the official windows driver does by the looks of it). Quick reports: https://vulkan.gpuinfo.org/displayreport.php?id=51093#memory (windows) and https://vulkan.gpuinfo.org/displayreport.php?id=50814#memory (fedora). Note: these probably aren't the same GPU.
2. We accept this change as is (not ideal for breda)
3. We add a mode to gpu-allocator that lets consumers opt-in to incoherent memory with the remark that they're responsible for calling vkFlushMappedMemoryRanges
4. We ask them to one-off fix this in wgpu (might also not be ideal, since other users run into this issue too).

[Jasper-Bekkers]

Filed also an issue on ANV to see if we can get to the bottom of this a bit more https://gitlab.freedesktop.org/mesa/mesa/-/work_items/16136

[hmaarrfk]

Pretty easy to throw AI at problems when you have a 270k processor....

Does this patch make sense to you?

Details

commit 8f4c9439463efb62529a3e17bf1a0f10d8adfba3 (HEAD)
Author: Mark Harfouche <mark.harfouche@gmail.com>
Date:   Thu Aug 20 08:00:41 2026 -0400

    anv/i915: expose a cached+coherent memory type where the PAT supports it

    anv_i915_physical_device_init_memory_types() picks its integrated-GPU
    memory types based on devinfo->has_llc.  Everything from Xe-HP on sets
    has_llc = false (XEHP_FEATURES), so every integrated part since Meteor
    Lake falls into the branch written for Atom parts and gets exactly two
    host-visible types: coherent-but-write-combining, and cached-but-not-
    coherent.  No type advertises HOST_COHERENT | HOST_CACHED together.

    Applications that ask for cached readback memory the usual way -- prefer
    HOST_VISIBLE | HOST_COHERENT | HOST_CACHED, fall back to HOST_VISIBLE |
    HOST_COHERENT -- therefore never match the preferred set and land on the
    write-combining type, where CPU reads are roughly two orders of magnitude
    slower.  Measured on Arrow Lake-S (ARL, i915), 32 MiB GPU->CPU readback,
    median of 25, including any invalidate the type requires:

      type 0  HOST_VISIBLE | HOST_COHERENT                186.70 ms   0.18 GB/s
      type 1  HOST_VISIBLE | HOST_CACHED                    1.77 ms  18.98 GB/s
      type 2  HOST_VISIBLE | HOST_COHERENT | HOST_CACHED    1.37 ms  24.42 GB/s  (new)

    The new type is not just 135x the write-combining one, it also beats the
    cached-but-incoherent type, because it needs no invalidate at all.

    This is not a missing hardware capability.  These platforms have a PAT
    entry that is write-back on the CPU side and coherent on the GPU side
    (MTL/ARL: PAT 3, Xe2: PAT 1), anv_device_get_pat_entry() already maps
    ANV_BO_ALLOC_HOST_CACHED_COHERENT onto pat.cached_coherent, and the i915
    kmd backend already programs it via I915_GEM_CREATE_EXT_SET_PAT.  The
    combination was simply never advertised, so it was unreachable.  The xe
    kmd backend already exposes this type in the equivalent branch.

    Gate on the platform actually having that PAT entry rather than on
    has_llc, so the real Atom parts -- which have no set_pat uapi -- keep
    today's two types.

    Verified on ARL/i915 with no vkFlushMappedMemoryRanges or
    vkInvalidateMappedMemoryRanges issued for the new type, in both
    directions, with the CPU's cache lines for the destination deliberately
    dirtied before the GPU overwrote them: 20 x 8 MiB round trips through
    device-local memory compared byte for byte, plus 25 x 32 MiB GPU->CPU
    readbacks, no stale bytes.
mark@antpitta $ git diff
diff --git a/src/intel/vulkan/i915/anv_device.c b/src/intel/vulkan/i915/anv_device.c
index ecbe7663..93c35b73 100644
--- a/src/intel/vulkan/i915/anv_device.c
+++ b/src/intel/vulkan/i915/anv_device.c
@@ -221,6 +221,32 @@ anv_i915_physical_device_init_memory_types(struct anv_physical_device *device)
                           VK_MEMORY_PROPERTY_HOST_CACHED_BIT,
          .heapIndex = 0,
       };
+
+      /* Platforms from Xe-HP on report has_llc == false, so every integrated
+       * part since Meteor Lake lands here even though it is not an Atom and
+       * does have a PAT entry that is write-back on the CPU side and coherent
+       * on the GPU side (MTL/ARL: PAT 3, Xe2: PAT 1).
+       *
+       * anv_device_get_pat_entry() already maps ANV_BO_ALLOC_HOST_CACHED_COHERENT
+       * onto that entry, and i915_gem_create_uncached()/_ext() already program
+       * it, but no memory type advertised both bits, so the combination was
+       * unreachable.  Applications asking for cached readback memory had to
+       * settle for the write-combining type, where CPU reads are roughly two
+       * orders of magnitude slower.
+       *
+       * Expose the cached+coherent type where the PAT can back it.  Platforms
+       * without the set_pat uapi (the actual Atoms) keep the two types above.
+       */
+      if (device->info.has_set_pat_uapi &&
+          device->info.pat.cached_coherent.mmap == INTEL_DEVICE_INFO_MMAP_MODE_WB) {
+         device->memory.types[device->memory.type_count++] = (struct anv_memory_type) {
+            .propertyFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT |
+                             VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT |
+                             VK_MEMORY_PROPERTY_HOST_COHERENT_BIT |
+                             VK_MEMORY_PROPERTY_HOST_CACHED_BIT,
+            .heapIndex = 0,
+         };
+      }
    }

    return VK_SUCCESS;

[hmaarrfk]

Note

This comment was written by Claude, an AI agent, acting on @hmaarrfk's behalf and posted from his account. He asked it to reproduce the numbers in this PR, dig into @Jasper-Bekkers' questions, and report back. Everything below is collapsed so you can skip it — the short version is: the root cause is in ANV, not in gpu-allocator, and Mark is happy to drop this PR's patch in favour of a Mesa fix. All claims below were measured on his machine; the caveats are listed at the end. Mark reviewed the main analysis before it was posted; the addendum at the bottom was added afterwards at his request and he has not read it yet. Treat all of it as machine-generated and check anything load-bearing.

Full analysis: ANV never advertises COHERENT|CACHED on modern Intel iGPUs — 26-line Mesa patch, measurements, and answers to the questions raised above

Test system: Arrow Lake-S (8086:7d67, Intel(R) Graphics (ARL)), kernel 7.0.0-30, iGPU bound to i915, Mesa 26.0.3. Everything was measured with gpu-allocator + ash only — no wgpu in the loop.

1. The assert you asked for fires

@Jasper-Bekkers asked for an assert on the WC flag. Unmodified gpu-allocator 0.28.0, MemoryLocation::GpuToCpu:

properties     : DEVICE_LOCAL | HOST_VISIBLE | HOST_COHERENT
matching type# : [0]
HOST_CACHED    : false
HOST_COHERENT  : true
8 MiB, median of 9: 30.9 ms  ->  0.27 GB/s

So yes — readback memory is genuinely not HOST_CACHED. Mechanism is as described in the PR: the preferred mask HOST_VISIBLE | HOST_COHERENT | HOST_CACHED matches nothing, so it falls through to the required mask HOST_VISIBLE | HOST_COHERENT and lands on write-combining.

One correction to the PR text: the "type 1/4" framing is misleading. Types 3/4 are ANV's dynamic_visible duplicates used for descriptor buffers; they never appear in memory_type_bits for an ordinary TRANSFER_DST buffer. There is really only one cached type, type 1.

2. gpu-allocator did not regress — wgpu changed allocators

This is worth correcting because it changes who owns the bug. The selection logic in src/vulkan/mod.rs is byte-identical between 0.27.0 and 0.28.0 — git diff 0.27.0..0.28.0 -- src/vulkan/mod.rs is the no_std refactor and nothing else.

What changed is in wgpu-hal/Cargo.toml:

• v27.0.1 — Vulkan backend uses gpu-alloc
• v28.0.0 — Vulkan backend uses gpu-allocator

gpu-alloc ranks memory types with a weighted cost function instead of a preferred/required two-step (gpu-alloc/src/usage.rs):

// Prefer cached memory for downloads.
let host_cached = flags.contains(Flags::HOST_CACHED) ^ usage.contains(UsageFlags::DOWNLOAD);
// Prefer coherent for both uploads and downloads.
let host_coherent = flags.contains(Flags::HOST_COHERENT) ^ usage.intersects(UPLOAD | DOWNLOAD);

device_local as u32 * 8 + host_visible as u32 * 4 + host_cached as u32 * 2 + host_coherent as u32

HOST_CACHED is weighted 2, HOST_COHERENT is weighted 1. When a driver forces the choice, gpu-alloc takes cached and gpu-allocator takes coherent. Same hardware, opposite outcome — that is the entire wgpu 27→28 regression.

Relevant to the contract discussion: this means wgpu was already being handed non-coherent memory on this hardware before v28. The "callers never need to flush" invariant was not actually holding there either; it was just invisible because nothing checked.

3. It is not about dual-GPU systems

@hmaarrfk speculated above that dual-GPU was the trigger and offered to physically remove a card. That is not necessary — there is no NVIDIA Vulkan ICD installed on this machine at all (/usr/share/vulkan/icd.d contains only Mesa ICDs), so the discrete card never enters Vulkan enumeration, and the bug still reproduces. The trigger is Arrow Lake + i915.

4. Root cause: ANV has the capability and never advertises it

anv_i915_physical_device_init_memory_types() selects memory types from devinfo->has_llc. XEHP_FEATURES sets has_llc = false, and every Intel integrated part from Meteor Lake onward inherits it (MTL_CONFIG → XEHP_FEATURES; ARL is MTL_CONFIG). So modern iGPUs land in the branch whose comment reads:

/* The spec requires that we expose a host-visible, coherent memory
 * type, but Atom GPUs don't share LLC. ...
 */

That Atom branch is now the only branch modern integrated hardware can reach, and the has_llc branch — which does expose HOST_COHERENT | HOST_CACHED, and whose comment notes "The Intel Vulkan driver for Windows also advertises these memory types" — is effectively dead code for anything newer than Raptor Lake. That is why the Windows driver behaves differently, @Jasper-Bekkers: it isn't a different policy decision, it's that Linux/ANV is falling into a legacy branch.

The hardware supports the combination, and so does the rest of ANV:

• MTL_CONFIG defines .pat.cached_coherent = PAT_ENTRY(3, WB) — CPU write-back, GPU coherent. Xe2 has PAT_ENTRY(1, WB).
• anv_device_get_pat_entry() already maps ANV_BO_ALLOC_HOST_CACHED_COHERENT onto that entry for integrated platforms.
• anv_AllocateMemory() already derives those alloc flags generically from the memory type's propertyFlags.
• i915_gem_create() already programs it through I915_GEM_CREATE_EXT_SET_PAT.
• The xe KMD backend already advertises this exact type in its equivalent non-LLC branch.

Only the enumeration was missing. The capability was implemented and unreachable.

5. The fix, and what it measures

26 lines, gated on the platform genuinely having the PAT entry so real Atom parts are untouched (has_set_pat_uapi is set for ver > 12 || is_mtl_or_arl):

if (device->info.has_set_pat_uapi &&
    device->info.pat.cached_coherent.mmap == INTEL_DEVICE_INFO_MMAP_MODE_WB) {
   device->memory.types[device->memory.type_count++] = (struct anv_memory_type) {
      .propertyFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT |
                       VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT |
                       VK_MEMORY_PROPERTY_HOST_COHERENT_BIT |
                       VK_MEMORY_PROPERTY_HOST_CACHED_BIT,
      .heapIndex = 0,
   };
}

32 MiB GPU→CPU readback, median of 25, including any invalidate the type requires:

type	flags	inval ms	copy ms	total ms	GB/s
0	HOST_COHERENT (WC)	0.000	186.697	186.697	0.18
1	HOST_CACHED	0.431	1.337	1.768	18.98
2	HOST_COHERENT | HOST_CACHED (new)	0.000	1.374	1.374	24.42

The new type is 135× the write-combining type, and still ~29% faster than the cached-but-incoherent type, because it needs no invalidate at all.

Coherency was tested adversarially rather than assumed. For any type claiming HOST_COHERENT the test issues no vkFlushMappedMemoryRanges and no vkInvalidateMappedMemoryRanges, in both directions, and — the part that actually matters — it dirties the CPU's cache lines for the destination buffer before the GPU overwrites them, so stale lines would be caught. 20 × 8 MiB round trips through device-local memory compared byte for byte, plus 25 × 32 MiB readbacks with full-buffer verification. Zero stale bytes.

6. What this means for this PR

With the Mesa fix, gpu-allocator needs no change at all:

gpu-allocator	driver	chosen type	8 MiB readback
0.28.0 unmodified	stock Mesa	0 (WC)	0.27 GB/s
0.28.0 unmodified	patched Mesa	2 (cached+coherent)	10.98 GB/s
+ this PR	stock Mesa	1 (cached, incoherent)	8.63 GB/s
+ this PR	patched Mesa	2 (cached+coherent)	11.25 GB/s

Row 2 is the one that matters. The existing preferred mask matches the new type on the first try, so:

• the "we never need to explicitly flush" contract is preserved,
• persistently mapped buffers keep working unchanged,
• breda is unaffected,
• nobody opts into incoherent memory.

That is @Jasper-Bekkers' option 1, and it turned out to be a small driver patch rather than a negotiation. This PR's patch (option 2) does work, and is a safe no-op once a proper type exists — row 4 shows the new fallback simply never fires, because the preferred pass succeeds first. But it is no longer needed, and @hmaarrfk is happy to close this PR in favour of the Mesa fix.

If you would still like a defensive change in gpu-allocator for drivers that genuinely cannot offer both bits, the honest version is option 3 (explicit opt-in to incoherent memory), not a silent fallback — but there is no longer a known driver that needs it.

7. Caveats

• Measured only on ARL / i915 / Mesa 26.0.3. The gate should be correct for LNL, PTL and Xe2 by construction, but none of those — nor an actual Atom part, which is the case the gate exists to protect — were tested here. Worth stating plainly in any Mesa MR.
• Mesa was built and tested at tag mesa-26.0.3, not main, though main carries the same code in this path.
• No CTS run. The coherency testing above is hand-rolled and targeted, not dEQP-VK.memory.*.
• The xe KMD is also present on this machine and its ANV path already exposes the good type, so rebinding the iGPU from i915 to xe would sidestep this entirely — but that is a system-level change, unlike the driver patch.

⚠️ ADDENDUM — "why not just set has_llc = true for these parts?" (added after the comment above)

This is the obvious first reaction to the analysis above, so it is worth answering before anyone spends time on it. Short version: has_llc = false is correct, and flipping it would be both factually wrong and actively harmful. has_llc is being used to answer a question it no longer answers.

What LLC meant, and what changed

Through Raptor Lake the iGPU sat on the same ring interconnect as the CPU cores and was a peer client of the shared L3. A page could be CPU write-back cached and GPU-coherent simultaneously at zero cost — no snooping, both agents looked at the same cache. That is precisely what has_llc records, and why the has_llc branch can expose a single COHERENT | CACHED type.

From Meteor Lake on, Intel disaggregated the design: compute tile (cores + L3 on the ring), a separate GPU tile, and an SoC tile in the middle owning DRAM. The iGPU is no longer a ring client — reaching the CPU L3 would mean crossing two die boundaries. Intel's own i915 changes state the GT "can no longer allocate on LLC — only the CPU can." Arrow Lake-S keeps that layout.

So has_llc = false accurately describes the silicon. It is not a stale table entry.

The kernel is explicit that the coherency model changed, not just the cache

From include/drm-uapi/xe_drm.h:

On pre-MTL platforms ... there is always a shared-LLC (or is dgpu) so all GT memory accesses are coherent with CPU caches even with the caching mode set as uncached. ... On MTL+ this completely changes and the HW defines the coherency mode as part of the @pat_index, where incoherent GT access is possible.

Coherency went from a free side effect of a shared cache to an explicit per-page PAT attribute. has_llc = false correctly means "no shared cache". The bug is reading it as "therefore cached+coherent is impossible".

Flipping it would break unrelated code

has_llc has other consumers. The most damaging is anv_allocator.c:

/* In platforms with LLC we can promote all bos to cached+coherent for free */
if (device->info->has_llc && ((alloc_flags & not_allowed_promotion) == 0))
   alloc_flags |= ANV_BO_ALLOC_HOST_COHERENT;

Setting has_llc = true on ARL would silently promote every BO to coherent. That is not "for free" here — it forces snooped write-back on everything, including GPU-heavy resources the CPU never touches. Also iris_screen.c uses it for caps->resource_from_user_memory (a GL capability in a different driver), and i915/anv_device.c uses it to gate a kernel WC-mmap requirement check.

On 1-way vs 2-way coherency

Worth recording, since it looks alarming at first glance:

platform	pat.cached_coherent	has_llc
TGL / ADL / RPL	PAT 0 → WB, 2WAY	true
MTL / ARL	PAT 3 → WB, 1WAY	false
Xe2 (LNL / BMG)	PAT 1 → WB, 1WAY	false

The type this patch exposes is backed by a 1-way coherent entry, not the 2-way entry the older LLC parts use. That is not a shortfall — the kernel uAPI sets 1-way as exactly the required threshold:

For coherency the @pat_index needs to be at least 1way coherent when drm_xe_gem_create.cpu_caching is DRM_XE_GEM_CPU_CACHING_WB. The KMD will extract the coherency mode from the @pat_index and reject if there is a mismatch.

and it requires 1WAY-or-2WAY for userptr and imported dma-buf. So the MTL/ARL entry sits at the sanctioned bar and the kernel enforces it, which matches the empirical result reported above (CPU cache lines deliberately dirtied, GPU overwrite, no invalidate, zero stale bytes over ~800 MiB).

Honest gap: none of the sources reachable from this machine spell out what "1-way" means directionally — which agent snoops which. That is in Intel's BSpec 45101 / 71582, which is not publicly accessible, and gitlab.freedesktop.org is behind Anubis so work item 16136 could not be read either. The kernel treating 1-way as sufficient, plus the adversarial test passing, is strong evidence — but it is inference, not a quote from the spec. If anyone at Intel can confirm the directional semantics on the Mesa side, that would close it properly.

The framing that matters

Two properties that were equivalent through Raptor Lake decoupled at Meteor Lake:

• (a) the GPU shares the CPU's last-level cache → now false, correctly
• (b) memory can be simultaneously CPU-cached and GPU-coherent → still true, via PAT

ANV uses (a) as a proxy for (b). The patch stops proxying and tests (b) directly — hence the gate on has_set_pat_uapi && pat.cached_coherent.mmap == INTEL_DEVICE_INFO_MMAP_MODE_WB rather than on has_llc.

[Jasper-Bekkers]

I think it makes some sense - but do engage with the mesa developers directly and also dig into why they seemingly disabled it over bandwidth concerns. Your benchmark seems to indicate that it's actually /faster/ so it would be interesting to see what measurements the mesa developments did and how they differ from yours.