feat(macos): implement SCKit-based capture and prefer it to AVFoundation where available - #5511
feat(macos): implement SCKit-based capture and prefer it to AVFoundation where available#5511martona wants to merge 4 commits into
Conversation
|
The remaining QualityGate issue is not mine; it's from 4 months ago. I can certainly fix it but I would rather keep this PR on topic. |
|
Agree on the sonar issue. Could you fix the doxygen errors before I review this? https://app.readthedocs.org/projects/sunshinestream/builds/34086280/ You can expand the failed section and |
|
Done. Sorry I missed them last night. |
|
This PR compiles against the latest |
9204144 to
6adfca6
Compare
|
|
Sorry for the delay in getting to this. I don't know much about Apple so the following is according to GPT 5.6. Main10 capture uses an unsupported ScreenCaptureKit format. At display.mm:590, 10-bit sessions assign x420 to SCStreamConfiguration.pixelFormat. Apple documents only BGRA, l10r, 420v, and 420f as supported formats. Sunshine can reach this path through VideoToolbox’s P010/Main10 capability. Capture errors are then silently discarded at sc_capture.m:270, so the backend neither produces real frames nor falls back to AVFoundation. Keep 10-bit capture on AVFoundation or add a supported ScreenCaptureKit format/conversion path, and propagate permanent screenshot errors. Apple’s pixel-format contract. Additionally, can you add tests to cover the changes? At minimum, cover delayed packets, dropped PTS values, missing timestamps, and queue eviction. |
|
Re the Main10 / Minimal standalone probe calling macOS 26.5.2 (25F84), Apple M4, display 2048x1152:
So on this system Caveat: single machine, single OS version. I have not tested macOS 14.x or 15.x, which is where the question is still open given the The second part of the review stands regardless: sc_pixfmt_probe.m/*
* sc_pixfmt_probe.m
*
* Minimal standalone probe for LizardByte/Sunshine PR #5511.
*
* Question: what does SCScreenshotManager actually do when
* SCStreamConfiguration.pixelFormat is set to a format that Apple's docs do
* not list as supported? In particular 'x420'
* (kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange), which display.mm assigns
* for 10-bit sessions.
*
* Four possible outcomes, each leading to a different conclusion:
* 1. NSError is returned -> the review is right, needs a fix.
* 2. Buffer returned as 'x420' -> the format works despite the docs.
* 3. Buffer returned in another -> silent degradation: worse than an error,
* format the encoder gets an unexpected format.
* 4. Buffer returned as 'x420' -> format accepted but no real data. This is
* but all-zero content what "no frames, no error" would look like.
*
* Build:
* clang -fobjc-arc -framework Foundation -framework ScreenCaptureKit \
* -framework CoreMedia -framework CoreVideo -framework CoreGraphics \
* -o sc_pixfmt_probe sc_pixfmt_probe.m
*
* Run:
* ./sc_pixfmt_probe
*
* Requires Screen Recording permission for the terminal app launching it
* (System Settings > Privacy & Security > Screen Recording).
*/
#import <Foundation/Foundation.h>
#import <ScreenCaptureKit/ScreenCaptureKit.h>
#import <CoreMedia/CoreMedia.h>
#import <CoreVideo/CoreVideo.h>
#import <CoreGraphics/CoreGraphics.h>
static NSString *FourCC(OSType c) {
if (c == 0) return @"<none>";
char s[5] = {
(char) ((c >> 24) & 0xFF),
(char) ((c >> 16) & 0xFF),
(char) ((c >> 8) & 0xFF),
(char) (c & 0xFF),
0
};
for (int i = 0; i < 4; i++) {
if (s[i] < 32 || s[i] > 126) s[i] = '?';
}
return [NSString stringWithFormat:@"'%s' (0x%08X)", s, (unsigned) c];
}
static void probe(SCDisplay *display, OSType fmt, NSString *label) {
SCContentFilter *filter = [[SCContentFilter alloc] initWithDisplay:display
excludingWindows:@[]];
// Same configuration fields the PR sets in SCCapture.captureVideo.
SCStreamConfiguration *cfg = [[SCStreamConfiguration alloc] init];
cfg.width = display.width;
cfg.height = display.height;
cfg.pixelFormat = fmt;
cfg.showsCursor = YES;
cfg.captureResolution = SCCaptureResolutionBest;
cfg.preservesAspectRatio = YES;
printf("\n--- %s: requested %s\n", label.UTF8String, FourCC(fmt).UTF8String);
printf(" cfg.pixelFormat after assignment: %s\n",
FourCC(cfg.pixelFormat).UTF8String);
dispatch_semaphore_t sem = dispatch_semaphore_create(0);
// Same API the PR polls: SCScreenshotManager, not SCStream.
[SCScreenshotManager captureSampleBufferWithFilter:filter
configuration:cfg
completionHandler:^(CMSampleBufferRef sb, NSError *err) {
if (err) {
printf(" RESULT: NSError -> domain=%s code=%ld %s\n",
err.domain.UTF8String,
(long) err.code,
err.localizedDescription.UTF8String);
dispatch_semaphore_signal(sem);
return;
}
if (!sb) {
printf(" RESULT: no error, but sampleBuffer == NULL\n");
dispatch_semaphore_signal(sem);
return;
}
printf(" sampleBuffer valid: %s\n",
CMSampleBufferIsValid(sb) ? "yes" : "no");
CVImageBufferRef px = CMSampleBufferGetImageBuffer(sb);
if (!px) {
printf(" RESULT: sampleBuffer has no imageBuffer\n");
dispatch_semaphore_signal(sem);
return;
}
OSType got = CVPixelBufferGetPixelFormatType(px);
printf(" RESULT: buffer delivered with format %s\n",
FourCC(got).UTF8String);
printf(" dimensions: %zux%zu planar=%s planes=%zu\n",
CVPixelBufferGetWidth(px),
CVPixelBufferGetHeight(px),
CVPixelBufferIsPlanar(px) ? "yes" : "no",
CVPixelBufferIsPlanar(px) ? CVPixelBufferGetPlaneCount(px) : (size_t) 0);
printf(" matches requested: %s\n",
got == fmt ? "YES" : "NO <-- silent degradation");
// Outcome 4: format accepted but empty content (black frame).
// Sample plane 0 (Y for biplanar formats, the only plane for packed ones).
if (CVPixelBufferLockBaseAddress(px, kCVPixelBufferLock_ReadOnly) == kCVReturnSuccess) {
const uint8_t *base;
size_t bpr, h;
if (CVPixelBufferIsPlanar(px)) {
base = CVPixelBufferGetBaseAddressOfPlane(px, 0);
bpr = CVPixelBufferGetBytesPerRowOfPlane(px, 0);
h = CVPixelBufferGetHeightOfPlane(px, 0);
} else {
base = CVPixelBufferGetBaseAddress(px);
bpr = CVPixelBufferGetBytesPerRow(px);
h = CVPixelBufferGetHeight(px);
}
size_t nonzero = 0, total = 0;
// Sample 1 row out of 64; enough to detect an all-black frame.
// Note for 'x420': samples are 16-bit LE with the 10-bit value in the high
// bits, so the low byte is often zero on real images. Expect ~50-70% nonzero.
for (size_t row = 0; row < h; row += 64) {
const uint8_t *r = base + row * bpr;
for (size_t i = 0; i < bpr; i++) {
total++;
if (r[i] != 0) nonzero++;
}
}
CVPixelBufferUnlockBaseAddress(px, kCVPixelBufferLock_ReadOnly);
printf(" plane 0 content: %zu/%zu sampled bytes nonzero -> %s\n",
nonzero, total,
nonzero == 0 ? "BLACK FRAME <-- format accepted but no data" : "has image");
} else {
printf(" plane 0 content: could not lock buffer for reading\n");
}
dispatch_semaphore_signal(sem);
}];
// 10 s headroom: the first call can be slow while permission is being granted.
if (dispatch_semaphore_wait(sem,
dispatch_time(DISPATCH_TIME_NOW, 10ull * NSEC_PER_SEC)) != 0) {
printf(" RESULT: timeout, completion handler not invoked within 10 s\n");
}
}
int main(void) {
@autoreleasepool {
NSOperatingSystemVersion v =
[[NSProcessInfo processInfo] operatingSystemVersion];
printf("macOS %ld.%ld.%ld\n",
(long) v.majorVersion, (long) v.minorVersion, (long) v.patchVersion);
__block SCShareableContent *content = nil;
dispatch_semaphore_t sem = dispatch_semaphore_create(0);
[SCShareableContent getShareableContentWithCompletionHandler:
^(SCShareableContent *c, NSError *e) {
if (e) {
printf("getShareableContent failed: %s\n",
e.localizedDescription.UTF8String);
} else {
content = c;
}
dispatch_semaphore_signal(sem);
}];
dispatch_semaphore_wait(sem, DISPATCH_TIME_FOREVER);
if (!content || content.displays.count == 0) {
printf("No displays available. Check Screen Recording permission.\n");
return 1;
}
SCDisplay *display = content.displays.firstObject;
printf("display id=%u %zux%zu\n\n",
(unsigned) display.displayID,
(size_t) display.width,
(size_t) display.height);
// Documented baseline: what Sunshine uses today for 8-bit.
probe(display, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, @"420v 8-bit (documented)");
// The disputed one: what display.mm assigns for 10-bit sessions.
probe(display, kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, @"x420 10-bit (DISPUTED)");
// Documented 10-bit alternative, in case it is useful as a fallback path.
probe(display, kCVPixelFormatType_ARGB2101010LEPacked, @"l10r 10-bit (documented)");
// Control: should always work.
probe(display, kCVPixelFormatType_32BGRA, @"BGRA 8-bit (documented)");
printf("\nDone.\n");
}
return 0;
}Full output on macOS 26.5.2 / Apple M4 |
|
Glad to hear the PR is alive. I will address the issues raised - but @Optimiza, if you could hold off reviewing/testing until after I push changes that'd be great. Reason: the current implementation is limited to ~30fps (inherent >20ms latency in SCScreenshotManager, only one in-flight request). I have refactored the PR to use SCStream after all. The original issue (detections dropped) was worked around by using the method OBS uses: cadence set to 0.9*expected_fps. (Credit to Claude for digging this up.) Result: 10ms host-processing latency all-in, smooth 60fps (limited by my displays). It will never reach Windows' NVEnc 4ms number but it's close enough, and I'm very happy with it. I've been using the refactored version for over a week. Let me look at the 10-bit issue in detail and I'll push an update. |
|
Sounds good, I'll hold off until you push the changes and test everything together then, including the Sonoma/Intel path on the iMac. |
4 participants
Description
Replaces AVFoundation's AVCaptureScreenInput as the default macOS capture backend with a ScreenCaptureKit-based implementation (macOS 14+; AVFoundation remains as fallback for older systems or SCKit setup failures).
Capture runs through SCScreenshotManager polling paced to the session frame rate, rather than an SCStream.
Two reasons:
Polling a single consistent source resolves both, at capture latency measured equal to AVFoundation. Since polling goes through the zero-copy VideoToolbox path by default, there's no CPU cost to it.
Fixes
Testing
Tested on macOS 26 (Apple silicon, VM, Mac Studio and MBP): cursor reappearance, static-screen behavior, display reconfiguration, latency stat parity, extended interactive sessions at 1080p60/4K60 with hardware (VideoToolbox) and software (x264) encoders.
Screenshot
Issues Fixed or Closed
Closes #3433
Roadmap Issues
Type of Change
Checklist
AI Usage
See our AI usage policy.