[610.43.03] Unbounded kernel memory growth during sustained GStreamer video playback leads to global OOM (RTX 3090, Wayland)

[0fif]

NVIDIA Open GPU Kernel Modules Version

610.43.03, open kernel module, built by akmods against GCC 16.1.1

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Fedora 44

Kernel Release

7.1.4-200.fc44.x86_64

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA GeForce RTX 3090 (VBIOS 94.02.42.C0.05)

Describe the bug

Under sustained GPU compositing of video on Wayland, kernel-side memory grows without bound until the machine hits a global OOM. Userspace is not the consumer — at the moment of the OOM, every process on the system combined held ~213 MB of anonymous memory on a 32 GB machine, yet only 350 MB was free. The OOM killer was invoked from the driver's own KMS thread.

Environment

• GPU: GeForce RTX 3090 (VBIOS 94.02.42.C0.05)
• Driver: 610.43.03, open kernel module, built via akmods with GCC 16.1.1
• Kernel: 7.1.4-200.fc44.x86_64 (Fedora 44, stable)
• Desktop: GNOME Shell 50.3, Wayland
• CPU: AMD Ryzen 9 5950X / 32 GB RAM + 8 GB zram

Mem-Info at the second OOM (2026-07-21 12:10:48):

active_anon:33251 inactive_anon:20061 isolated_anon:0
 free:87485 free_pcp:308 free_cma:0
 slab_reclaimable:12516 slab_unreclaimable:4409875
 mapped:39961 shmem:74 pagetables:7434
Node 0 active_anon:133004kB inactive_anon:80244kB active_file:47856kB
 inactive_file:98976kB ... pagetables:29736kB sec_pagetables:2841472kB
 all_unreclaimable? yes
Node 0 Normal free:217048kB boost:155648kB min:217292kB low:247032kB
 present:30395392kB managed:29751464kB
36871 total pagecache pages
Free swap  = 6268820kB

Accounting:

slab_unreclaimable: 4409875 pages	16.8 GB
sec_pagetables: 2841472kB	2.8 GB
All userspace anon	213 MB
All page cache	144 MB
Free	350 MB
Unaccounted by any Mem-Info counter	~11 GB

Note free:217048kB sitting below min:217292kB, with all_unreclaimable? yes. There was nothing left to reclaim.

Trigger thread:

KMS thread invoked oom-killer: gfp_mask=0x40dc0(GFP_KERNEL|__GFP_ZERO|__GFP_COMP), order=3, oom_score_adj=100
CPU: 2 UID: 1000 PID: 4955 Comm: KMS thread Tainted: G           OE       7.1.4-200.fc44.x86_64 #1 PREEMPT(lazy)
Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE

PID 4955 is the nvidia_modeset kernel thread. nvidia is the only tainting module.

Two things I would like help interpreting:

1. sec_pagetables is 2.8 GB with no KVM guest and no VM of any kind running — no qemu/libvirt process exists on this machine. Idle value is currently 416 MB, also with no VM.
2. The Unreclaimable slab info: dump enumerates named caches summing to only 2.55 GB, against a slab_unreclaimable counter of 16.8 GB. The dump includes uvm_* caches (uvm_va_block_context_t, uvm_va_range_external_t, uvm_service_block_context_t, …). Largest named entries are randomized kmalloc buckets: kmalloc-rnd-03-8k 1053 MB, kmalloc-rnd-03-1k 606 MB, kmalloc-rnd-12-4k 396 MB.

Decode path — this should narrow it considerably:

During both OOM events, decoding was entirely in software. The only registered H.264 decoder at the time was openh264dec (rank marginal) — nvcodec reported 0 features with the warning CUDA library "libcuda.so.1" was not found.

This was a local misconfiguration, not a packaging gap, and I have since fixed it. The GStreamer plugin registry cache (~/.cache/gstreamer-1.0/registry.x86_64.bin) had been built at the exact minute the NVIDIA driver was installed, before libcuda.so.1 was in place, and cached nvcodec as having zero elements. Deleting the cache re-probed the plugin and nvh264dec now registers at rank primary + 1 (257), well above openh264dec.

I am flagging this because it means NVDEC was not involved in either OOM. Every frame was decoded on the CPU, uploaded, and composited through the driver's GL/present/KMS path. That is consistent with KMS thread invoking the OOM killer, and points at buffer upload/import and presentation rather than decode.

I have not yet re-run the workload with NVDEC active, so I cannot say whether hardware decode avoids the leak. I will follow up.

Consequence: 19 processes were killed across two events. Every one was a bystander — the largest held 22 MB, all 19 combined ~45 MB. The desktop lost its audio server, input method, file manager and terminal. Full list attached.

To Reproduce

Preconditions

• GeForce RTX 3090, driver 610.43.03 open kernel module
• GNOME Shell 50.3 on Wayland
• GStreamer 1.28.5 with gstreamer1-plugin-gtk4-0.15.2 (gtk4paintablesink)
• No hardware H.264 decoder registered at the time, so playbin autoplugged software decode (openh264dec). See the note in the bug description — this was a stale GStreamer registry cache on my side, since fixed. To reproduce the exact conditions, force software decode.
• Screen blanking disabled (I use the unblank GNOME extension), so the compositor keeps presenting for the whole run

Steps

• Install the GNOME extension Live Lock Screen 3.2.0 (https://github.com/nick-redwill/LiveLockScreen). It spawns a standalone gjs process running a GStreamer playbin into gtk4paintablesink, rendered as the lock-screen background.
• Set its video to a 3840x2160 60 fps H.264 file (mine: 75.3 s, 4.2 Mbit/s, yuv420p) and enable looping. Looping is implemented as a flushing Gst.Element.seek_simple back to zero on EOS.
• Lock the screen. Playback starts.
• Leave it locked and playing. Watch SUnreclaim and SecPageTables in /proc/meminfo climb. Baseline on my machine is SUnreclaim 331 MB / SecPageTables 416 MB.
• Global OOM follows.

Observed timeline (single boot)

23:25:48  gjs[10455]  pipeline created   \
23:28:57  gjs[14454]  pipeline created    |  six lock/unlock cycles in five
23:29:16  gjs[14772]  pipeline created    |  minutes; each previous player
23:30:06  gjs[14988]  pipeline created    |  SIGKILLed on unlock
23:30:21  gjs[15167]  pipeline created   /
23:30:52  gjs[15373]  pipeline created   <- screen re-locked, playback starts
23:42:43  FIRST OOM                      <- 12 minutes of sustained playback
12:10:48  SECOND OOM                     <- same PID, 12.8 h of playback
12:11:03  (user unlocks)

Over that 12.8 h run: roughly 613 flushing seeks and ~2.76 million frames decoded and presented.

I verified via the OOM task dumps that the earlier player processes were genuinely reaped — at 23:42:43 only PID 15373 was alive. This is not a userspace process leak.

Two candidate triggers, possibly both:

1. Churn — rapid create/destroy of GL/present pipelines (six in five minutes). Each flushing seek also tears down and recreates buffer pools, 613 times over the long run.
2. Duration — a single pipeline compositing continuously for 12.8 hours.

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

oom-evidence-trimmed.txt
oom-kernel-log.txt
oom-kills-summary.txt
system-info.txt

More Info

Expected: kernel-side memory attributable to the driver stays bounded during long-running compositing, or is reclaimable under pressure.

Actual: it grows until the kernel has nothing left to reclaim (all_unreclaimable? yes) and kills unrelated userspace processes. The workload holding this memory is a single video pipeline whose own userspace footprint is under 400 kB.

The attached nvidia-bug-report.log.gz was captured at baseline rather than during the leak. I can provide a leaked-state capture on request.

Happy to run a specific reproduction, collect /proc/slabinfo or debugfs output, test a patched module, or bisect driver versions. Tell me what would be most useful.