Steam Machine (Fremont, Navi 33): MES does not respond to msg=RESET, so a GPU hang escalates to a MODE1 reset, loses VRAM, and kills the desktop session

[BrandtChristian]

Your system information

• Steam client version: 1788652215 (2026-09-06)
• SteamOS version: 3.8.16, BUILD_ID 20260716.1, kernel 6.16.12-valve24.5-1-neptune-616-gb2f7cfe85e45
• Opted into Steam client beta?: No (steamdeck_stable)
• Opted into SteamOS beta?: No (branch rel)
• Have you checked for updates in Settings > System?: Yes. steamos-update check on 2026-09-10 reports "No update available" on the rel branch, so this is the current stable build.

Hardware: Valve Fremont revision 1, board revision 9 (product_family HawkPoint), BIOS
F7F0108 dated 2026-08-03. GPU is Navi 33, 1002:7481 rev c7, subsystem
Valve Software Device 1717, GC 11.0.2 (gfx1102). VBIOS 113-D7461000-101 /
022.003.001.037.000001, dated 2025/12/01. Mesa 26.0.0-devel (git-9cc9241790), RADV NAVI33,
Vulkan 1.4.330.

Happy to submit a full system report through Steam if that would help.

Firmware as loaded: MES 0x00000075, MES_KIQ 0x00000102, ME 0x0000094c, PFP 0x000009d8,
MEC 0x00000a1e, SMC 0x00525c00 (82.92.0).

Please describe your issue in as much detail as possible:

What I expected: when a game hangs the GPU, amdgpu resets the offending queue and kills only
that application. That is what the per-queue reset path exists to do.

What actually happens: the queue reset never succeeds, because MES does not respond to
msg=RESET. The driver falls back to a pipe reset, which CP firmware does not support, so it
escalates to a full MODE1 device reset. MODE1 loses VRAM, which invalidates the GPU context of
every process on the device. So a single hung game takes down the compositor and every other GPU
client with it, and the whole desktop session ends.

I have hit this twice with an identical signature, on 2026-09-03 and 2026-09-10, on the same
stable build with byte-identical kernel, firmware and VBIOS versions both times.

To be precise about which part I think is the bug: the pipe reset being unavailable is
expected. GFX11 KGQ pipe reset is implemented upstream but deliberately disabled pending ME
firmware support, so The CPFW hasn't support pipe reset yet. is intentional. The question is
the step before it. MES does not answer msg=RESET at all, so the MES queue reset, which is
the only graceful recovery path available on this part, never gets the chance to work. MES is
also unresponsive to msg=REMOVE_QUEUE later during teardown, which suggests MES was already
wedged rather than being broken by the reset attempt.

What I am asking, cheapest first:

1. Is the MES firmware shipped in the image current for GC 11.0.2? Loaded MES is 0x75, and
   gc_11_0_2_mes*.bin on this image is dated 2026-06-05 against an OS build of 2026-07-16. If a
   newer MES fixes the msg=RESET timeout, shipping it would close this with no driver change.
2. If MES is expected to be unresponsive on this part, can the escalation to a VRAM-losing
   MODE1 be avoided? The escalation is where the user-visible cost is. A reset that killed only
   the offending context would turn this from "the session dies" into "the game crashed".
3. Is the MES queue reset expected to work on Fremont at all?
4. Has this already changed on a newer branch? I am on rel and fully current, but Steam Machine recurrent random data-fabric sync-flood resets and missing Wi-Fi PCIe port after automatic reboot #2757
   shows a Fremont on 6.18.37-valve1 through 6.18.45-valve1 in July and August, against
   6.16.12-valve24.5 on stable here. If a newer kernel or MES firmware on another branch
   already addresses this, that is the answer and I am happy to stop here.

Possibly related Fremont-platform context, though it explicitly rules out amdgpu resets so it is
not the same fault: #2757.

Kernel log, the full reset sequence (dmesg -T, 2026-09-10)

The 2026-09-03 sequence is identical line for line apart from pids and sequence numbers.

[Thu Sep 10 14:36:15 2026] amdgpu 0000:03:00.0: amdgpu: Dumping IP State
[Thu Sep 10 14:36:15 2026] amdgpu 0000:03:00.0: amdgpu: Dumping IP State Completed
[Thu Sep 10 14:36:15 2026] amdgpu 0000:03:00.0: amdgpu: [drm] AMDGPU device coredump file has been created
[Thu Sep 10 14:36:16 2026] amdgpu 0000:03:00.0: amdgpu: ring gfx_0.0.0 timeout, signaled seq=7292300, emitted seq=7292304
[Thu Sep 10 14:36:16 2026] amdgpu 0000:03:00.0: amdgpu:  Process WoWClassic.exe pid 47714 thread dxvk-submit pid 47850
[Thu Sep 10 14:36:16 2026] amdgpu 0000:03:00.0: amdgpu: Starting gfx_0.0.0 ring reset
[Thu Sep 10 14:36:18 2026] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to msg=RESET
[Thu Sep 10 14:36:18 2026] [drm:amdgpu_mes_reset_legacy_queue [amdgpu]] *ERROR* failed to reset legacy queue
[Thu Sep 10 14:36:18 2026] amdgpu 0000:03:00.0: amdgpu: reset via MES failed and try pipe reset -110
[Thu Sep 10 14:36:18 2026] amdgpu 0000:03:00.0: amdgpu: The CPFW hasn't support pipe reset yet.
[Thu Sep 10 14:36:18 2026] amdgpu 0000:03:00.0: amdgpu: Ring gfx_0.0.0 reset failed
[Thu Sep 10 14:36:18 2026] amdgpu 0000:03:00.0: amdgpu: GPU reset begin!
[Thu Sep 10 14:36:21 2026] amdgpu 0000:03:00.0: amdgpu: MES failed to respond to msg=REMOVE_QUEUE
[Thu Sep 10 14:36:21 2026] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[Thu Sep 10 14:36:21 2026] [drm:gfx_v11_0_hw_fini [amdgpu]] *ERROR* failed to halt cp gfx
[Thu Sep 10 14:36:21 2026] amdgpu 0000:03:00.0: amdgpu: MODE1 reset
[Thu Sep 10 14:36:21 2026] amdgpu 0000:03:00.0: amdgpu: GPU mode1 reset
[Thu Sep 10 14:36:21 2026] amdgpu 0000:03:00.0: amdgpu: GPU smu mode1 reset
[Thu Sep 10 14:36:21 2026] amdgpu 0000:03:00.0: amdgpu: GPU reset succeeded, trying to resume
[Thu Sep 10 14:36:21 2026] [drm] PCIE GART of 512M enabled (table at 0x00000081FEB00000).
[Thu Sep 10 14:36:21 2026] [drm] VRAM is lost due to GPU reset!
[Thu Sep 10 14:36:22 2026] amdgpu 0000:03:00.0: amdgpu: GPU reset(1) succeeded!
[Thu Sep 10 14:36:22 2026] amdgpu 0000:03:00.0: [drm] device wedged, but recovered through reset

Recovery takes about 5.5 s of hard hang both times: 5.498 s on 2026-09-03 and 5.522 s on
2026-09-10, measured from the timeout line to device wedged, but recovered through reset.

What the VRAM loss kills (the user-visible part)

Within five seconds of the reset, from coredumpctl:

14:36:23  wowup-cf        SIGTRAP
14:36:24  chrome          SIGABRT
14:36:25  gwenview        SIGSEGV
14:36:26  plasmashell     SIGABRT
14:36:42  wine-preloader  SIGQUIT

Losing plasmashell ends the desktop session. Only the game was hung; everything else here was
an innocent GPU client whose context the reset invalidated.

Evidence that the hang is in shader work, with no VM fault

• The CP's last consumed dword on gfx_0.0.0 was 2860, which decodes to a NOP, so the ring was
  not blocked on a register wait. The stall is in work launched by a preceding
  INDIRECT_BUFFER.
• emitted seq minus signaled seq is exactly 4, which equals amdgpu.sched_hw_submission=4.
  The ring was saturated at its in-flight limit with nothing retiring.
• regGRBM_STATUS = 0xaf71002c: GUI_ACTIVE, CP_BUSY, SPI_BUSY, SX_BUSY, SC_BUSY,
  PA_BUSY, DB_BUSY set, with DB_CLEAN and CB_CLEAN clear and TA_BUSY clear. Active on
  GRBM_STATUS_SE0 and SE1 only. Waves resident, texture path idle, depth work outstanding and
  never draining.
• No VM fault. dmesg and the full journal contain zero page fault lines,
  regGCVM_L2_PROTECTION_FAULT_STATUS is 0x0, and every UTCL1_STATUS
  (CPF/CPC/CPG/IA/RLC/RMI/WD) is 0x0.
• Host telemetry sampled every 5 s shows power collapsing from 110 W to 62 W and junction from
  85 C to 74 C while gpu_busy_percent stayed at 100 and sclk stayed pinned at 2444 MHz, so the
  real onset is about 14:36:11, five seconds before the driver declared it. That matches
  amdgpu.lockup_timeout=5000 for gfx.

Ruled out: not thermal (junction peaked 88 C against a 100 C limit), no VRAM growth (flat 3968 to
4115 MiB over the preceding eight minutes), and no CPU, IO or memory pressure before the event
(every pressure and load maximum lands 40 to 60 s after it, during recovery). The same workload
had run for 90 minutes earlier the same day at higher peak power (132 W) and higher peak VRAM
(4245 MiB) without hanging.

One warning about the attached device coredump. Its Ring timed out details section prints
[gfxhub] Page fault observed with Faulty page starting at address: 0x0000000000000000 and
Protection fault status register: 0x0 on a hang that had no fault at all. That looks like an
unpopulated default being printed unconditionally, and it sends readers after a null dereference
that did not happen. Might be worth fixing separately.

The coredump is also truncated at 2,224,128 bytes, because the read is cut off by the very reset
it documents: cat of /sys/class/devcoredump/devcd1/data gets about six seconds before MODE1
completes and the device re-probe frees the node, after which every read returns 0 bytes. The cut
lands inside the trailing mes_kiq ring dump; the header, IP dump and the culprit gfx_0.0.0
ring are all complete and ahead of it.

This is not specific to my capture. SteamOS's own log submitter hits it too: issue #1312 contains
(udev-worker): devcd1: Process 'cat /sys/devices/virtual/devcoredump/devcd1/data > /var/lib/steamos-log-submitter/pending/devcoredump/4785' failed with exit code 1. So if there is
a supported way to capture a device coredump whole across a GPU reset, I would like to know it,
and it would presumably improve steamos-log-submitter's own captures at the same time.

Steps for reproducing this issue:

Not reliably reproducible. Twice in seven days, and one hang came 11 minutes into a boot while the
other came 6.7 hours in, so it is not tied to boot age. The exact ring stall offsets differ
between the two, so it is not one repeated draw either.

1. Boot to Desktop Mode (KDE Plasma on Wayland, kwin_wayland).
2. Launch World of Warcraft Classic through Lutris using GE-Proton11-6 with DXVK, and play at
   sustained near 100% GPU load (2350 to 2444 MHz, 100 to 118 W).
3. Have Chrome playing a YouTube video at the same time. (Probably incidental. Chrome was a
   casualty of the reset, not a cause, but it was present both times.)
4. Somewhere between 25 minutes and several hours in, gfx_0.0.0 times out and the sequence in
   the kernel log above follows. The session drops out to the Steam session.

I have persistent GPU crash telemetry running on this machine, so if there is a specific extra
capture that would help (register dumps, a particular debugfs node, RADV_DEBUG=hang shader
reports at the cost of GPU throughput), I can arm it and wait for the next occurrence.

valve-issue-attachments.zip

[Nephyrin]

The current preview build, 3.9.0, has a much newer 7.2-based kernel and may be worth re-testing this on. A fair bit of per-ring reset work has gone in since stable's current 6.16-based kernel.