Add Wayland screencopy video capture - #14
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| "appsrc name=appsrc is-live=true format=time do-timestamp=true "+ | ||
| "caps=video/x-raw,format=BGRx,width=%d,height=%d,framerate=%d/1 "+ | ||
| "%s ! appsink name=appsink", screen.Width, screen.Height, fps, pipeline, | ||
| ), nil |
There was a problem hiding this comment.
Appsrc framerate conflicts with pipeline
High Severity
Wayland appsrc caps and wf-recorder both use screen.Rate, while the encoding chain from GetPipeline often forces a different framerate via VideoConfig.Fps (default "25"). Fixed appsrc caps cannot renegotiate against that capsfilter, so the default desktop rate (30) yields a not-negotiated pipeline and live video never starts.
Reviewed by Cursor Bugbot for commit 0cd3a36. Configure here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 5 potential issues.
There are 6 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cdbc229. Configure here.
| if code < 8 || code > 263 { | ||
| return 0, false | ||
| } | ||
| return uint16(code - 8), true |
There was a problem hiding this comment.
Broken Wayland keyboard mapping
High Severity
mapKey treats incoming values as X11 keycodes (code - 8), but the WebRTC client and X11 path send X11 keysyms (for example XK_a 0x61, XK_Escape 0xff1b). Most keys are rejected or emit the wrong evdev codes, so Wayland keyboard input fails for normal typing and shortcuts.
Reviewed by Cursor Bugbot for commit cdbc229. Configure here.
| for key := 0; key <= 0xff; key++ { | ||
| if err := input.ioctl(uiSetKeybit, uintptr(key)); err != nil { | ||
| return fmt.Errorf("enable uinput key %d: %w", key, err) | ||
| } |
There was a problem hiding this comment.
Mouse buttons never registered
High Severity
create only registers UI_SET_KEYBIT for codes 0..0xff, but mouse clicks emit BTN_LEFT/BTN_RIGHT/BTN_MIDDLE (0x110–0x112). Without those keybits, the kernel drops button events, so Wayland pointer clicks do not reach the compositor.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit cdbc229. Configure here.
|
|
||
| input, err := newWaylandInput(screenSize.Width, screenSize.Height) | ||
| if err != nil { | ||
| return manager.screenSize, err |
There was a problem hiding this comment.
Resize leaves compositor mismatched
Medium Severity
SetScreenSize runs the Wayland resize command before creating the new uinput device. If newWaylandInput fails afterward, the compositor is already resized while screenSize and the old device geometry stay unchanged, so capture and input stay out of sync with the display.
Reviewed by Cursor Bugbot for commit cdbc229. Configure here.
| manager.screenSize = screenSize | ||
| manager.emmiter.Emit("after_screen_size_change") | ||
| mu.Unlock() | ||
| oldInput.close() |
There was a problem hiding this comment.
Use-after-close input race
Medium Severity
Callers load manager.waylandInput without synchronization, while resize/shutdown swap and close the device, niling fd. A concurrent move/button/key on the old instance can reach emit with a nil fd and panic inside binary.Write.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit cdbc229. Configure here.
| if returnErr != nil { | ||
| manager.logger.Warn().Err(returnErr).Msg("Wayland scroll failed") | ||
| } | ||
| return |
There was a problem hiding this comment.
Ctrl-scroll ignored on Wayland
Medium Severity
The Wayland Scroll path drops the controlKey argument and only emits wheel events. The WebRTC client still sends Ctrl+wheel for zoom-style actions, so that behavior works on X11 and silently does nothing on Wayland.
Reviewed by Cursor Bugbot for commit cdbc229. Configure here.
1 participant
summary
not a serious PR, was just trying a long-running agent on investigating Wayland for browser image
wf-recorderexecutable viawlr-screencopy-unstable-v1BGRxframes into Neko’s existing GStreamer/WebRTC pipeline/dev/uinputfor pointer, keyboard, button, and scroll inputwlr-randrconfiguration
Enable:
NEKO_CAPTURE_VIDEO_WAYLAND=trueNEKO_DESKTOP_WAYLAND=trueThe runtime needs
wf-recorder,wlr-randr, and access to/dev/uinput. Output resizing defaults toHEADLESS-1and can be configured withNEKO_DESKTOP_WAYLAND_OUTPUTandNEKO_DESKTOP_WAYLAND_RESIZE_COMMAND.validation
go test ./...go vet ./...wf-recorder0.6.0 andwlr-randr0.5.0 successfully in the Ubuntu 22.04 image toolchainlimitations
The Wayland backend targets wlroots compositors exposing
wlr-screencopy-unstable-v1and output management. Clipboard, file chooser, and drag-and-drop paths remain X11-specific and are not enabled by this backend.Note
Medium Risk
Touches live WebRTC capture and remote input paths (subprocess recorder, uinput, pipeline lifecycle); behavior is gated off by default but misconfiguration or recorder failures could break sessions on Wayland deployments.
Overview
Adds an opt-in Wayland stack alongside the existing X11 defaults: separate flags for video capture (
capture.video.wayland) and desktop control (desktop.wayland).Video: When Wayland capture is enabled, pipelines use GStreamer
appsrcinstead ofximagesrc, with raw BGRx frames from a configurable recorder (defaultwf-recorderviawlr-screencopy). Customgst_pipelinestrings are rejected in this mode.StreamSinkManagergains an optional frame source that starts/stops with the pipeline and pushes intoappsrc.Desktop: With
desktop.wayland, startup skips X11 and drives pointer/keyboard through a/dev/uinputvirtual device (X11 keycodes mapped to Linux codes). Resolution changes callwlr-randr(output name and command configurable). Keyboard layout/modifiers, cursor image, and screenshots are no-ops or stubs on Wayland compared to X11.Configuration and docs are updated for the new capture and desktop options.
Reviewed by Cursor Bugbot for commit cdbc229. Bugbot is set up for automated code reviews on this repo. Configure here.