feat(linux): Add Vulkan video encoder for Linux by neatnoise · Pull Request #4603 · LizardByte/Sunshine

neatnoise · 2026-01-19T17:02:58Z

Description

This PR adds Vulkan video encoding support for Linux KMS capture, providing an alternative to VAAPI encoding. AI agent was used heavily for code generation, following the existing VAAPI encoder implementation as the reference pattern.

Changes:

Vulkan encoder: New FFmpeg-based Vulkan encoder (h264_vulkan, hevc_vulkan, av1_vulkan) with zero-copy DMA-BUF import via EGL interop - no GPU-to-CPU copies in the capture-encode pipeline
kms_vblank option: New config option to enable vblank-synchronized KMS capture, reducing tearing and improving frame pacing when display refresh rate exceeds stream FPS (fixes Screen Tearing/VSync Issue in KDE Plasma Wayland, Gnome Wayland #3189 - long-standing vblank sync issue on Linux KMS)
HiDPI fix: Fixed KMS framebuffer scaling to use physical framebuffer dimensions (fb->width/fb->height) instead of logical viewport dimensions, fixing capture on HiDPI displays
Web UI: Added Vulkan encoder selection and settings to the configuration interface
Documentation: Added docs for vulkan encoder and kms_vblank options in docs/configuration.md

Dependencies:

Requires corresponding PR in LizardByte/build-deps (ffmpeg-vulkan branch) to enable BUILD_FFMPEG_VULKAN option, which adds --enable-vulkan and Vulkan encoder support to FFmpeg builds.

Testing:

Tested on single GPU setup: AMD Radeon 780M (integrated)
Tested on multi-GPU setup: AMD Radeon 9070 XT (discrete) + AMD Radeon 780M (integrated)
Tested on Arch Linux KDE desktop. Requires vulkan-headers and vulkan-icd-loader packages to be installed (or similar on another distros).

Screenshot

Issues Fixed or Closed

Fixes Screen Tearing/VSync Issue in KDE Plasma Wayland, Gnome Wayland #3189

Roadmap Issues

Type of Change

Checklist

Code follows the style guidelines of this project
Code has been self-reviewed
Code has been commented, particularly in hard-to-understand areas
Code docstring/documentation-blocks for new or existing methods/components have been added or updated
Unit tests have been added or updated for any new or modified functionality

AI Usage

None: No AI tools were used in creating this PR
Light: AI provided minor assistance (formatting, simple suggestions)
Moderate: AI helped with code generation or debugging specific parts
Heavy: AI generated most or all of the code changes

ReenigneArcher

This is just a preliminary review. On top of the comments below, I think these files will probably need updates.

update dependencies:

src_assets/common/assets/web/public/assets/locale/en.json

cmake/compile_definitions/linux.cmake

docs/configuration.md

cmake/compile_definitions/linux.cmake

src/platform/linux/kmsgrab.cpp

ReenigneArcher · 2026-01-19T20:45:42Z

I will convert this to draft for now, please change it back when you're ready for a full review. Thanks!

neatnoise · 2026-01-19T23:39:38Z

Its ready

andygrundman · 2026-01-20T10:00:07Z

I know this PR is about encoding, but I saw changes to capture as well. Do you know if this could support variable framerate capture, similar to how capture works on Windows? In my limited experience Linux always captures at the stream framerate.

The ideal capture process would be capturing only when the image has changed (with duplicated frames subject to minimum_fps_target), as well as capturing using a frame interval timer requested by the client or based on the framerate. For example, an Xbox client works best if frames are captured on a timer with an interval of 59.94fps (60000/1001), a desktop client could just use a 60fps interval. One thing you do not want to do is base anything on the host's vsync interval.

ReenigneArcher · 2026-01-23T21:49:06Z

Its ready

I left a bunch of review comments.

neatnoise · 2026-01-26T11:13:11Z

I know this PR is about encoding, but I saw changes to capture as well. Do you know if this could support variable framerate capture, similar to how capture works on Windows? In my limited experience Linux always captures at the stream framerate.

The ideal capture process would be capturing only when the image has changed (with duplicated frames subject to minimum_fps_target), as well as capturing using a frame interval timer requested by the client or based on the framerate. For example, an Xbox client works best if frames are captured on a timer with an interval of 59.94fps (60000/1001), a desktop client could just use a 60fps interval. One thing you do not want to do is base anything on the host's vsync interval.

Yes, the kms_vblank option already behaves this way with variable frame rate similarly to Windows. It encodes that many frames that are generated by a game. It also respects fps frame rate which are set by a client (if the clients sets 60, it won't generate more than 60 fps with some minimal margin)

neatnoise · 2026-01-26T11:15:42Z

@ReenigneArcher I made proposed changes. I also added a tiling fix for RNDA 4 when Gnome DE is used. Would be helpful if somebody with Nvidia and Intel GPUs could test the new encoder

neatnoise · 2026-02-17T21:35:31Z

@ReenigneArcher I've tested with vulkan ffmpeg build https://github.com/LizardByte/build-deps/actions/runs/21850415771?pr=606 successfully.

Tweaked kms_vblank for the game Cronos The New Dawn meanwhile.

Tested in many games with various resolutions with and without vsync in-game and kms_vblank enabled and disabled. Tested with vulkan and vaapi encoders. It works solid.

ReenigneArcher · 2026-02-18T18:17:35Z

This PR needs to be rebased to pick up the right FFmpeg pre-build, before I can run CI.

Edit: actually the build-deps commit that added Vulkan hadn't been merged here yet: #4735

psyke83 · 2026-02-18T18:21:12Z

@neatnoise

Also, don't forget to verify compatibility with portalgrab when rebasing. This is enough to get started, but you might want to refine/verify:

diff --git a/src/platform/linux/portalgrab.cpp b/src/platform/linux/portalgrab.cpp
index 1f4e9062..eeaf1c92 100644
--- a/src/platform/linux/portalgrab.cpp
+++ b/src/platform/linux/portalgrab.cpp
@@ -29,6 +29,7 @@
 #include "src/platform/common.h"
 #include "src/video.h"
 #include "vaapi.h"
+#include "vulkan_encode.h"
 #include "wayland.h"
 
 namespace {
@@ -748,6 +749,7 @@ namespace portal {
         // On hybrid GPU systems (Intel+NVIDIA), DMA-BUFs come from the Intel GPU and cannot
         // be imported into CUDA, so we fall back to memory buffers in that case.
         bool use_dmabuf = n_dmabuf_infos > 0 && (mem_type == platf::mem_type_e::vaapi ||
+                                                 mem_type == platf::mem_type_e::vulkan ||
                                                  (mem_type == platf::mem_type_e::cuda && display_is_nvidia));
         if (use_dmabuf) {
           for (int i = 0; i < n_dmabuf_infos; i++) {
@@ -1226,6 +1228,9 @@ namespace portal {
         return va::make_avcodec_encode_device(width, height, n_dmabuf_infos > 0);
       }
 #endif
+      if (mem_type == platf::mem_type_e::vulkan) {
+        return vk::make_avcodec_encode_device_vram(width, height, 0, 0);
+      }
 
 #ifdef SUNSHINE_BUILD_CUDA
       if (mem_type == platf::mem_type_e::cuda) {
@@ -1368,7 +1373,7 @@ namespace portal {
 namespace platf {
   std::shared_ptr<display_t> portal_display(mem_type_e hwdevice_type, const std::string &display_name, const video::config_t &config) {
     using enum platf::mem_type_e;
-    if (hwdevice_type != system && hwdevice_type != vaapi && hwdevice_type != cuda) {
+    if (hwdevice_type != system && hwdevice_type != vaapi && hwdevice_type != vulkan && hwdevice_type != cuda) {
       BOOST_LOG(error) << "Could not initialize display with the given hw device type."sv;
       return nullptr;
     }

neatnoise · 2026-02-18T18:59:32Z

@XT-Martinez It looks like ffmpeg without built-in vulkan support, you need to build it with ffmpeg vulkan supported binaries (like from link above).

@psyke83 will do. I will add vulkan support also for portal grab. Actually I have my portal/pipeware implementation here https://github.com/neatnoise/Sunshine/tree/vulkan%2Bpipewire . So It will be similar.

neatnoise · 2026-02-19T19:39:27Z

Rebased to sunshine master (with portal support)
Added vulkan+portal support
Cleaned up to just add vulkan support - removed kms_vblank option (i might create a separate PR for it in the future)

The best results on my machine are portal+vulkan (vulkan settings low latency + vbr) and existing portal+vaapi. Portal seems to have a lower latency than KMS.

gschintgen · 2026-02-19T20:54:13Z

FWIW I found the vblank option quite interesting. (But I don't have much time right now to actually check it out.)

psyke83 · 2026-02-19T22:05:57Z

With the default settings (vbr, ll), the stream is massively violating the target bitrate (70Mbps target is saturating my wifi connection at 250Mbps+). I didn't notice this with prior testing.

psyke83 · 2026-02-19T23:41:54Z

This is the issue:

conn@ryzenl:~/work/sunshine$ ffmpeg 2>&1 -h encoder=hevc_vulkan | grep -A5 rc_mode 
  -rc_mode           <int>        E..V....... Select rate control type (from 0 to UINT32_MAX) (default auto)
     auto            4294967295   E..V....... Choose mode automatically based on parameters
     driver          0            E..V....... Driver-specific rate control
     cqp             1            E..V....... Constant quantizer mode
     cbr             2            E..V....... Constant bitrate mode
     vbr             4            E..V....... Variable bitrate mode
conn@ryzenl:~/work/sunshine$ ffmpeg 2>&1 -h encoder=h264_vulkan | grep -A5 rc_mode 
  -rc_mode           <int>        E..V....... Select rate control type (from 0 to UINT32_MAX) (default auto)
     auto            4294967295   E..V....... Choose mode automatically based on parameters
     driver          0            E..V....... Driver-specific rate control
     cqp             1            E..V....... Constant quantizer mode
     cbr             2            E..V....... Constant bitrate mode
     vbr             4            E..V....... Variable bitrate mode

Your current PR is assigning VBR as 3 erroneously. When set to 4, I'm not seeing the crazy bitrate overshooting.

src_assets/common/assets/web/config.html

neatnoise · 2026-02-20T00:11:02Z

This is the issue:

conn@ryzenl:~/work/sunshine$ ffmpeg 2>&1 -h encoder=hevc_vulkan | grep -A5 rc_mode 
  -rc_mode           <int>        E..V....... Select rate control type (from 0 to UINT32_MAX) (default auto)
     auto            4294967295   E..V....... Choose mode automatically based on parameters
     driver          0            E..V....... Driver-specific rate control
     cqp             1            E..V....... Constant quantizer mode
     cbr             2            E..V....... Constant bitrate mode
     vbr             4            E..V....... Variable bitrate mode
conn@ryzenl:~/work/sunshine$ ffmpeg 2>&1 -h encoder=h264_vulkan | grep -A5 rc_mode 
  -rc_mode           <int>        E..V....... Select rate control type (from 0 to UINT32_MAX) (default auto)
     auto            4294967295   E..V....... Choose mode automatically based on parameters
     driver          0            E..V....... Driver-specific rate control
     cqp             1            E..V....... Constant quantizer mode
     cbr             2            E..V....... Constant bitrate mode
     vbr             4            E..V....... Variable bitrate mode

Your current PR is assigning VBR as 3 erroneously. When set to 4, I'm not seeing the crazy bitrate overshooting.

Fixed

sonarqubecloud · 2026-02-20T00:14:34Z

Quality Gate failed

Failed conditions
11 New issues
9.6% Duplication on New Code (required ≤ 2%)
11 New Code Smells (required ≤ 0)
2 Duplicated Blocks on New Code (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

neatnoise changed the title ~~Add Vulkan video encoder for Linux (KMS capture)~~ feat(web): Add Vulkan video encoder for Linux Jan 19, 2026

neatnoise changed the title ~~feat(web): Add Vulkan video encoder for Linux~~ feat(linux): Add Vulkan video encoder for Linux Jan 19, 2026

ReenigneArcher requested changes Jan 19, 2026

View reviewed changes

ReenigneArcher marked this pull request as draft January 19, 2026 20:45

ReenigneArcher mentioned this pull request Feb 10, 2026

feat(linux): Add Vulkan video encoder support LizardByte/build-deps#606

Merged

21 tasks

This comment was marked as off-topic.

Sign in to view

Add Vulkan video encoder for Linux

5718d2d

neatnoise force-pushed the vulkan-pr branch from 4aa4cfb to 5718d2d Compare February 19, 2026 19:28

fix: correct Vulkan VBR rc_mode value from 3 to 4 per Vulkan spec

7c7432f

psyke83 reviewed Feb 20, 2026

View reviewed changes

src_assets/common/assets/web/config.html Outdated Show resolved Hide resolved

fix: correct VBR default in web config.html from 3 to 4

8053788

Uh oh!

Conversation

neatnoise commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes:

Dependencies:

Testing:

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

Checklist

AI Usage

Uh oh!

ReenigneArcher left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ReenigneArcher commented Jan 19, 2026

Uh oh!

neatnoise commented Jan 19, 2026

Uh oh!

andygrundman commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ReenigneArcher commented Jan 23, 2026

Uh oh!

neatnoise commented Jan 26, 2026

Uh oh!

neatnoise commented Jan 26, 2026

Uh oh!

neatnoise commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

ReenigneArcher commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

psyke83 commented Feb 18, 2026

Uh oh!

neatnoise commented Feb 18, 2026

Uh oh!

neatnoise commented Feb 19, 2026

Uh oh!

gschintgen commented Feb 19, 2026

Uh oh!

psyke83 commented Feb 19, 2026

Uh oh!

psyke83 commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

neatnoise commented Feb 20, 2026

Uh oh!

sonarqubecloud bot commented Feb 20, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Comments

neatnoise commented Jan 19, 2026 •

edited

Loading

ReenigneArcher left a comment •

edited

Loading

andygrundman commented Jan 20, 2026 •

edited

Loading

neatnoise commented Feb 17, 2026 •

edited

Loading

ReenigneArcher commented Feb 18, 2026 •

edited

Loading

psyke83 commented Feb 19, 2026 •

edited

Loading