Skip to content

fix(wayland): support DMA-BUF modifiers for wlroots capture#5132

Open
joeknock90 wants to merge 6 commits into
LizardByte:masterfrom
joeknock90:fix_wlroots_vulkan
Open

fix(wayland): support DMA-BUF modifiers for wlroots capture#5132
joeknock90 wants to merge 6 commits into
LizardByte:masterfrom
joeknock90:fix_wlroots_vulkan

Conversation

@joeknock90
Copy link
Copy Markdown

@joeknock90 joeknock90 commented May 17, 2026

Explicitly query supported DMA-BUF modifiers from the compositor and use them when creating GBM buffers. This fixes capture failures on Sway when using the Vulkan renderer, which rejects the implicit INVALID modifier.

Explicitly query supported DMA-BUF modifiers from the compositor and use them when creating GBM buffers. This fixes capture failures on Sway when using the Vulkan renderer, which rejects the implicit INVALID modifier.

DISCLAIMER: I am not a developer, programmer, or really any sort of smart person at all. I punched this issue into Gemini-CLI. I manually approved all of he changes it made with very little understanding of them, but did learn a little bit. Never thought I'd be a vibe coder 🤮

For full transparency, here is the prompt I used:

In this project something is broken when trying to stream a swaywm desktop using the WLR_RENDERER=vulkan option. According to some reports, this issues happens in Linux 7.0 but works as intended in Linux 6.18.25. According to the swaywm developers: "The wlroots capture method in Sunshine is using the implicit INVALID modifier, which is rejected in Vulkan."

I am currently testing my branch and it seems to be working well for me so far.

Screenshot

N/A

Issues Fixed or Closed

Fixes #4050

Roadmap Issues

N/A

Type of Change

  • feat: New feature (non-breaking change which adds functionality)
  • fix: Bug fix (non-breaking change which fixes an issue)
  • docs: Documentation only changes
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
  • refactor: Code change that neither fixes a bug nor adds a feature
  • perf: Code change that improves performance
  • test: Adding missing tests or correcting existing tests
  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Other changes that don't modify src or test files
  • revert: Reverts a previous commit
  • BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

  • Code follows the style guidelines of this project
  • Code has been self-reviewed
  • Code has been commented, particularly in hard-to-understand areas
  • Code docstring/documentation-blocks for new or existing methods/components have been added or updated
  • Unit tests have been added or updated for any new or modified functionality

AI Usage

  • None: No AI tools were used in creating this PR
  • Light: AI provided minor assistance (formatting, simple suggestions)
  • Moderate: AI helped with code generation or debugging specific parts
  • Heavy: AI generated most or all of the code changes

Explicitly query supported DMA-BUF modifiers from the compositor and use them
when creating GBM buffers. This fixes capture failures on Sway when using
the Vulkan renderer, which rejects the implicit INVALID modifier.
@neatnoise

This comment was marked as off-topic.

@joeknock90
Copy link
Copy Markdown
Author

This is continuing to work well for me. I'm not sure when I should remove the DRAFT designation. Any input from maintainers?

@ReenigneArcher
Copy link
Copy Markdown
Member

This is continuing to work well for me. I'm not sure when I should remove the DRAFT designation. Any input from maintainers?

If you think it's ready for review or close to being mergable, you can remove draft. Typically I don't really review anything that's in draft unless asked, just because there is so much to review already. And with that said, I don't really use Linux myself so would appreciate anyone of the other contributors also reviewing and/or testing the changes.

@joeknock90
Copy link
Copy Markdown
Author

would appreciate anyone of the other contributors also reviewing and/or testing the changes.

Considering I relied heavily on Gemini to create this that would certainly make me much more comfortable.

I'll take it out of draft though. I'm happy with the results personally, and if/when HDR support gets merged for Sway it would be nice to have this working as it relies on the sway's vulkan renderer.

@joeknock90 joeknock90 marked this pull request as ready for review May 19, 2026 23:21
@codecov
Copy link
Copy Markdown

codecov Bot commented May 20, 2026

Bundle Report

Bundle size has no change ✅

@codecov
Copy link
Copy Markdown

codecov Bot commented May 20, 2026

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.79%. Comparing base (87182c9) to head (096d9c0).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/platform/linux/wayland.cpp 0.00% 14 Missing ⚠️
src/platform/linux/wlgrab.cpp 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #5132     +/-   ##
=========================================
  Coverage   17.79%   17.79%             
=========================================
  Files         111      111             
  Lines       24143    24155     +12     
  Branches    10687    10692      +5     
=========================================
+ Hits         4296     4299      +3     
- Misses      15364    18297   +2933     
+ Partials     4483     1559   -2924     
Flag Coverage Δ
Archlinux 11.18% <0.00%> (-0.01%) ⬇️
FreeBSD-aarch64 ?
FreeBSD-amd64 13.31% <0.00%> (-0.02%) ⬇️
Homebrew-ubuntu-22.04 13.52% <0.00%> (-0.02%) ⬇️
Linux-AppImage 12.10% <0.00%> (-0.01%) ⬇️
Windows-AMD64 14.85% <ø> (ø)
Windows-ARM64 13.20% <ø> (+0.01%) ⬆️
macOS-arm64 18.86% <ø> (ø)
macOS-x86_64 18.34% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/platform/linux/wayland.h 0.00% <ø> (ø)
src/platform/linux/wlgrab.cpp 0.00% <0.00%> (ø)
src/platform/linux/wayland.cpp 2.96% <0.00%> (-0.14%) ⬇️

... and 66 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87182c9...096d9c0. Read the comment docs.

@ninele7
Copy link
Copy Markdown

ninele7 commented May 25, 2026

Can you provide more precise instructions to test this PR? For me build from pipeline doesn't work at all on headless sway instance on Arch. Both with and without Vulkan. Upstream builds work without Vulkan, but don't work with it.

Also it might be more beneficial to revive this PR: #4788 which implements more modern capture method.

@joeknock90
Copy link
Copy Markdown
Author

For my testing I cloned the pacman-repo repository and edited the PKGBUILD to use my repo and branch, then installed with makepkg and pacman.

I've been testing on headless sway as well, using the WLR_RENDERER=vulkan variable. I verified that the env variable is set in the headless sway session from the terminal.

@ReenigneArcher
Copy link
Copy Markdown
Member

Can you provide more precise instructions to test this PR? For me build from pipeline doesn't work at all on headless sway instance on Arch. Both with and without Vulkan. Upstream builds work without Vulkan, but don't work with it.

Also it might be more beneficial to revive this PR: #4788 which implements more modern capture method.

Go here -> https://github.com/LizardByte/Sunshine/actions/runs/26135435168?pr=5132#artifacts

Download and install the artifact for your system that starts with build-.

@ninele7
Copy link
Copy Markdown

ninele7 commented May 26, 2026

sunshine.log

Yes, I've installed build from PR pipelines, but it doesn't work both with and without Vulkan. I've attached the log. Is there any reason to try to build it from PKGBUILD of cloned repo?

There might be additional problems, because I'm running it inside of Proxmox container, but this setup works without Vulkan.

@teodorjuravlea
Copy link
Copy Markdown

teodorjuravlea commented May 26, 2026

I've tested the build from the PR pipeline (Archlinux + wlroots-git commit 7265a79e + sway-git commit e51f9d7) for about an hour with WLR_RENDERER=vulkan.
Seems to be working well, I didn't run into any issues.

Notably HDR still doesn't work, I think it's not yet supported by the wlroots headless backend based on this log from sway: [ERROR] [sway/config/output.c:369] Cannot enable HDR on output HEADLESS-1: BT2020 primaries not supported by output

@joeknock90
Copy link
Copy Markdown
Author

joeknock90 commented May 26, 2026

Notably HDR still doesn't work, I think it's not yet supported by the wlroots headless backend based on this log from sway: [ERROR] [sway/config/output.c:369] Cannot enable HDR on output HEADLESS-1: BT2020 primaries not supported by output

I didn't attempt HDR support in this PR. I'm not entirely sure it's possible in sway yet? I think sway-git might have added some support but I'm not sure It would work with the screen capture method being used. Someone can correct me if I'm wrong there.

@teodorjuravlea
Copy link
Copy Markdown

Notably HDR still doesn't work, I think it's not yet supported by the wlroots headless backend based on this log from sway: [ERROR] [sway/config/output.c:369] Cannot enable HDR on output HEADLESS-1: BT2020 primaries not supported by output

I didn't attempt HDR support in this PR. I'm not entirely sure it's possible in sway yet? I think sway-git might have added some support but I'm not sure It would work with the screen capture method being used. Someone can correct me if I'm wrong there.

Sway 1.12 just released with HDR support, but yeah, I don't think it's possible with the wlroots headless backend (the log I posted is just from Sway, before even starting Sunshine).
I haven't looked if the capture method supports HDR already, I would assume it's the same story so probably not.

I just mentioned it in case people were following this PR for this reason, I doubt anything can be done about it until wlroots supports it.

@ReenigneArcher
Copy link
Copy Markdown
Member

@joeknock90
Copy link
Copy Markdown
Author

I believe that clang-format was just looking for a space? Pushed and rebased.

@ReenigneArcher ReenigneArcher added the ai PR has signs of heavy ai usage (either indicated by user or assumed) label May 26, 2026
@sonarqubecloud
Copy link
Copy Markdown

❌ The last analysis has failed.

See analysis details on SonarQube Cloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai PR has signs of heavy ai usage (either indicated by user or assumed)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Linux - Broken output in Sway/wlroots with WLR_RENDERER=vulkan

5 participants