[Win11] Export exhausts all VRAM/RAM at 1080p/4K, causes unkillable freeze

[Stareven233]

Description

When exporting a ~16-minute recording on Windows 11 at either 1080p or 4K, Cap rapidly consumes all available VRAM and system RAM. This immediately causes the application to freeze or the entire display to go black. Critically, once this happens, the Cap process becomes completely unkillable (Task Manager and taskkill fail), forcing a hard reboot.

As a workaround, I attempted to manually concatenate the raw .mp4 and audio fragments into a single file, but discovered that the cursor overlay is not baked into the raw segments.

Export should complete within normal memory limits. If export fails, the app should gracefully crash and remain terminable without locking up the OS.

Is there any method to extract or render the original recording files with the mouse cursor included? I just need a reliable way to salvage these 16-minute recordings

Reproduction

1. Record a session for approximately 16 minutes on Windows 11.
2. Open the recording in Cap.
3. Start export at 1080p or 4K.

Additional Context

• Cap version: 0.4.84, 0.4.83, 0.4.71
• Operating system, version: win11 (26100.1.amd64fre.ge_release.240331-1435)
• Device (optional): AMD 5500GT, NVIDIA RTX 3060 12GB, 16GB DDR4

No any debug logs in "C:\Users{your username}\AppData\Roaming\so.cap.desktop\logs"

[linear]

CAP-693

[tembo]

This issue is similar to #1511 which reports the same symptom (excessive RAM usage during 4K export on a long recording). That issue describes a 30-minute recording causing 13GB RAM usage during 4K/30fps export on macOS.

For the export memory exhaustion, the key files involved are:

• crates/export/src/mp4.rs - the MP4 export pipeline with NV12 rendering
• crates/rendering/src/lib.rs - render_video_to_channel_nv12() which handles GPU frame rendering

The export uses a channel capacity of 8 frames at line 605 in mp4.rs. For 4K NV12, that's roughly 100MB of buffered frame data. The memory pressure likely comes from video decoding (the FRAME_CACHE_SIZE in decoder/mod.rs is 90 frames), combined with GPU memory for rendering at 4K resolution.

For salvaging your recordings: the raw .mp4 segments in your .cap recording folder do not contain baked-in cursor overlay - the cursor is rendered at export time using data from cursor.json in each segment folder. The cursor rendering happens in crates/rendering/src/layers/cursor.rs. If you want the cursor included, you'd need Cap's export pipeline to complete successfully.

As a potential workaround, you could try:

1. Export at 720p first to reduce memory pressure (4K uses ~4x more GPU memory per frame)
2. Try enabling "Force FFmpeg decoder" in settings (experimental) - this uses software decoding instead of MediaFoundation which may have different memory characteristics

The "unkillable process" behavior suggests GPU resource exhaustion causing driver-level hangs - this is concerning from a stability perspective and may warrant investigating memory-mapped GPU resource cleanup on export failure.