Repository navigation
feat(camera): several cameras in the picture, with layout sections - #1026
Draft
christian-wr wants to merge 23 commits into
Draft
christian-wr wants to merge 23 commits into
christian-wr wants to merge 23 commits into
Conversation
Each camera of the webcams list (or the legacy single-camera fields) gets its own capture and encoder on the recording's T0. A camera that cannot be opened is dropped with an indexed webcam-unavailable warning; one whose samples fail mid-take is disabled on its own. recording-stopped keeps webcamPath and adds webcamPaths. MFEncoder now only balances an MFStartup it made: a dropped camera's never-initialized encoder ran an unmatched MFShutdown from its destructor and stopped every other encoder in the process.
A camera whose encoder initialize() or capture start() fails is now warned about with the indexed webcam-unavailable event and dropped, instead of ending the take; its encoder is finalized and its empty file removed. The screen encoder's failure stays fatal. mf_encoder_color_test pins the MFEncoder fix: finalizing a never-initialized encoder must leave a live one able to write and finalize.
Two webcams of the same model report the same name, and the browser id never matches a device path, so every such camera selected the first device and the second open failed as busy. The take now owns a claim set: each camera that opens adds its device (MF symbolic link or DirectShow DevicePath, normalized so both paths agree), and later cameras pick the best unclaimed match. The selection rule lives in device_selection.{h,cpp} with its own unit test.
A label already used by camera 1 or an earlier extra gets " (2)", " (3)" by occurrence, so unavailable and dropped cameras of the same model can be told apart.
The recorder caps at three extra cameras, but a validator that ships with a cap cannot be loosened later for older builds. The HUD and Electron still cap.
Two webcams of the same model share a name. When an extra carries a deviceId and camera 1 does not, the name says nothing about whether they are the same device, so the extra is no longer dropped as a duplicate of camera 1.
The helper deletes the file of a camera it drops at start, but ignored a failed DeleteFileW. It now logs a WARNING with the path and GetLastError, and Electron keeps the dropped cameras' paths so stop and discard remove a 0-byte stub left behind.
A camera the helper disables mid-take keeps its partial file in the take, but nobody was told. A camera whose file was kept (size > 0) yet is missing from recording-stopped.webcamPaths is now named in a "Stopped early" notice after the take, camera 1 included. Paths compare case-insensitively with either separator. Only an event that carries webcamPaths can say so, so the helper now prints the list, possibly empty, whenever a camera wrote a file of its own; an older helper or a missing event never produces the notice.
The checklist now notes that with several identical cameras plugged in the recorded ones follow Windows' enumeration order, adds a 1-vs-2-camera screen pacing comparison at 4K (getopenscreen#945) and a stopped-early check. The helper README says the camera index counts after entries without camPath are skipped, and the extras-need-camera-1 assumption (R6) is noted where links drop them.
A failed ReadSample was counted and retried forever, so an unplugged camera kept its file running to the end on its last picture and stayed in webcamPaths. The Media Foundation capture now latches lost on a device invalidated or hardware start failure, on end of stream during the take, or after a second of consecutive read failures; the DirectShow fallback latches it on EC_DEVICE_LOST (removal), EC_ERRORABORT or EC_STREAM_ERROR_STOPPED. The writer loop disables a lost camera like any mid-take failure, so its file ends at the loss and the app names it as stopped early.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The camera list matched a saved pick only by its id, while the recording request resolves a stale id by name. Both now go through resolveAdditionalCameraPicks, so the checked rows, the cap and the toggles follow the cameras the request carries.
A Full Camera section can now show a webcam tilted down onto the desk: "Desk view" in the inspector turns the camera 180°, switches the mirror off for that section (otherwise text on the page reads back to front) and shows the whole camera frame instead of the face crop. The timeline marks such sections with a rotate icon. The moment the camera is tilted is covered at both ends of the section: the whole camera picture is blurred and dimmed for the Full Camera grow (and the shrink), fading in and out around it, with a translated "Desk mode" label over it that can be switched off per section. Preview and export render it identically through the compositor's frame plan. - Rust: per-frame orientation (u/v bound swaps, crop bypass) and cover strength, a `LayerCB.cover` lane (HLSL, WGSL, MSL), the blur clamped to the camera's valid area so aligned decoder padding never bleeds in; the label is drawn at the cover strength of the same frame, so it fades exactly with the cover. - App: `rotation` / `mirror` / `deskLabel` on Full Camera regions (defaults are not stored), store and persistence, inspector controls, one generated label annotation per projected piece, translations in every locale. - Tests for each layer; WGSL is validated with naga on every host.
christian-wr
force-pushed
the
feat/multi-camera-layouts-pr
branch
from
October 7, 2026 10:31
f0bc3d1 to
8871923
Compare
Up to four recorded cameras can now be on screen together. Layout sections on the timeline choose a template — screen + small cameras, one camera full + small ones, two cameras side by side, or another camera full — and which camera goes into each place; the picture glides between sections. Each camera can be turned, mirrored, cropped and perspective-corrected. - Compositor: a list of camera layers per frame instead of one webcam rect, planned in `camera_layers.rs` (glide and fade between neighbouring sections, incl. a direct glide to and from an adjacent Full Camera section), extra camera decoders opened only for cameras a layout shows (preview and export, a camera that ends early never shortens the clip), a homography and a transparency lane in `LayerCB` (HLSL, WGSL, MSL), the corrected camera cover-fitted into its box; where a corrected camera reaches past its picture the layer is black, not transparent. - Editor: "Add layout" menu, a layout inspector (template, camera per place, and on/off switches per camera for screen + camera sections), PiP places movable and resizable in the preview, a "Cameras" section with per-camera settings. - Perspective: a calibration dialog with a loupe and a live rectified preview. "Detect markers" reads four printed ArUco markers (js-aruco2, MIT; OpenCV 4x4 dictionary codes, BSD-3 — both in THIRD-PARTY-NOTICES.md), fixes the desk plane from their known 40 mm squares (metric rectification, frame levelled with the camera, oriented by marker 0), and the user places a crop on the straightened desk (4:3, 16:9, 9:16, 1:1, free); the stored corners are a true rectangle on the desk. Four hand-placed corners remain available. - Main camera: any recorded camera can take camera 1's role (preset PiP, dual frame, vertical stack, Full Camera sections, effects). Implemented as a scene-only document swap; stored indices keep naming the physical camera, and one clip list (`buildCompositorClipList`) feeds scene, export dialog, CLI and preview. - Desk lane: a project desk camera (chosen in the Cameras section or resolved automatically, never the main camera) and desk sections added with D on their own lane, exclusive in time with Full Camera and layout sections; drawn through the layout path with a "Desk mode" label. A desk-camera default can be picked in the HUD device settings; a recording carries it to a new project. - Every change is one undo step; projects without the new fields render exactly as before. No schema bump.
christian-wr
force-pushed
the
feat/multi-camera-layouts-pr
branch
from
October 7, 2026 11:22
8871923 to
9af921e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Put several recorded cameras in the picture at once — the desk camera full with your face small on top, two cameras side by side, or the screen with two small cameras — and switch between such arrangements along the timeline with a glide.
withMainCamera); stored indices keep naming the physical camera, and one clip list (buildCompositorClipList) feeds scene, export dialog, CLI and preview.camera_layers.rs), extra camera decoders opened only for cameras a layout shows, and a homography + transparency lane appended toLayerCB(HLSL, WGSL and MSL agree); where a corrected camera reaches past its picture the layer is black, not see-through. Projects without the new fields render exactly as before.cameraLayoutRegions,cameraSettings,mainCamera,deskCamera,deskRegionsinlegacyEditor; the camera-full section of the main camera is a Full Camera region, every other camera's a layout row; no overlap across the camera lists (also for the AI agent's tools).js-aruco22.0.0 (MIT, pure JS) for marker detection; it embeds OpenCV's 4×4 dictionary codes (BSD-3). Both are listed inTHIRD-PARTY-NOTICES.md.Related issue
None — new feature.
Type of change
Release impact
Desktop impact
Screenshots / video
Can follow on request (desk camera full + face small, side by side, the calibration dialog).
Testing
npm run test(330 files, 4723 passed), bothtscconfigs,npm run lint(0 errors),npm run i18n:check.cargo test -p openscreen-compositor --libon Windows ARM64; Linux and macOS compositor tests incl. wgpu pixel tests passed in CI on a fork._electronon copies of a real Brio + built-in camera recording (projects folder hash-checked unchanged afterwards):scripts/editor-multicam-smoke.mjs);scripts/editor-desk-smoke.mjs);scripts/editor-main-camera-smoke.mjs);scripts/editor-pip-switches-smoke.mjs);scripts/hud-desk-camera-smoke.mjs).deskCamera: 1) and the new project; a desk section added with D showed it.Not covered yet (in the checklist): dual frame / vertical stack with a main camera, and other platforms in the running app. Translations other than English and German would benefit from a native speaker's look.