fix(macos): use two H.264 VideoToolbox reference frames

[Elegy]

Summary

• Set max_ref_frames=2 for H.264 VideoToolbox.
• Keep parallel encoding enabled and leave the HEVC and AV1 settings unchanged.
• Avoid both the all-IDR output produced by one reference buffer and an M1-specific stall with automatic reference-buffer selection.

This is a follow-up to #5200 and the behavior reported in #5013.

Regression and A/B results

I reproduced the regression between these Sunshine revisions using identical Homebrew-mode builds and the same FFmpeg bundle:

• Known good: 3c7952b2dd7f1c8fe74249d0303e7f9d335daf8b
• First bad: 3ee4144a8123da467cc771acc1d449e98668a8e2 from fix(macos): drop max_ref_frames=1 for h264_videotoolbox and enable PARALLEL_ENCODING #5200

The first bad revision removed max_ref_frames=1 for H.264 VideoToolbox and enabled parallel encoding. Testing those changes independently produced this matrix:

H.264 reference buffers	Parallel encoding	Result
Automatic	Enabled	Frozen after the initial frame
Automatic	Disabled	Frozen after the initial frame
1	Disabled	Moving, about 15 FPS, 50% network drops, all-IDR output
1	Enabled	Moving, about 15 FPS, 50% network drops, all-IDR output
2	Enabled	30 to 33 incoming FPS, 0% network drops, requested IDRs plus normal P-frames

Two reference buffers preserve the bandwidth improvement from #5200 while restoring continuous H.264 delivery on this M1 host.

Test environment

• MacBook Air (MacBookAir10,1), Apple M1, 8 GB
• macOS 26.3.1, build 25D2128
• Moonlight Android debug client on an Amazon Fire TV Cube, 2nd generation, Fire OS 7.7.0.6
• 1280x720, 30 FPS, H.264, 5 Mbps, smoothest frame pacing
• Sunshine FEC 20%, minimum FPS target 30

The runtime matrix was measured with the same code change on Sunshine base 40ae6c800274afff664838cc48386e01fcffe2cd. This PR rebases that change onto current master.

Validation

• Applied the repository's pinned clang-format tooling.
• Built Sunshine.app from current master with ninja -C build sunshine.
• Built the complete test_sunshine target.
• EncoderVariants/EncoderTest.ValidateEncoder/videotoolbox passes.
• Full suite: 343 passed, 1 skipped, 3 unrelated macOS Retina mouse-coordinate tests failed.
• Live playback with the patched host sustained 30 to 33 FPS and 0% network drops with smooth video and system audio.

This is a draft because the value has only been validated on M1 hardware. Testing on Intel Macs and other Apple Silicon generations would be useful before merge.

[sonarqubecloud]

Quality Gate failed

Failed conditions
1 Duplicated Blocks on New Code (required ≤ 0)

See analysis details on SonarQube Cloud