diff --git a/README.md b/README.md index ddf5aa05..d07ee11a 100644 --- a/README.md +++ b/README.md @@ -177,7 +177,11 @@ macOS the system's buttons and edges stay. It holds, from the left: - The health dot, the stream's worst grade at a glance. Click it for the worst problem first, then each destination, who is watching, and the encoder, and **Details** for the [Health](#health) window. Off air it reads - **Ready** once the show has a relay, and **No relay** until then. + **Ready** once the show has a relay, and **No relay** until then. While a + capture a live scene shows has [gone dark](#a-capture-gone-dark), or a + camera or broadcast it shows [stopped sending pictures](#a-picture-that-stalls), + and those are among the worst problems, it counts them, as **1 layer dark**, + **1 layer stalled**, or **2 layers dark or stalled**. - Record, which starts or stops a [recording](#recording) from any page, with the picks in Settings, Recording. Its dot pulses while recording, with how long it has been recording beside it. @@ -250,7 +254,10 @@ kind, its name, what it shows (and **linked ×3** when three layers show the source), and its eye, which shows or hides it. A lock shows on a locked row, and on the selected or pointed-at one to lock it. A dot says a source is not live, such as starting, stalled, offline, ended, or failed, with the words on -hover, and an error on the layer replaces what it shows, in red. Click a row +hover, and an error on the layer replaces what it shows, in red. A capture +[gone dark](#a-capture-gone-dark), or a camera or broadcast whose picture +[stalled](#a-picture-that-stalls), has an amber badge instead, which opens +what happened and the fix. Click a row to select its layer, drag it up or down its group to restack it, and right-click it for its menu: hide or show, lock or unlock, interact with a web page, bring it to the front or forward or send it backward or to the back, @@ -303,7 +310,7 @@ is named for it, with these sections: | Shape | Pick its outline, square, rounded, circle, or arch, give it a border inside that outline and the border's color, and give it a soft shadow, which follows the outline and is the layer's first drop shadow filter. | | Look | Remove its source's background, which every layer showing the source loses (make it independent to cut out here alone); grade its color through a LUT source; add, order, set, and remove its filters, copy them, or paste copied filters after its own; and set its blend mode. | | Shown in | See how many places show its source, **Linked in 3 scenes**, and **Make independent**, which points this layer at a copy of the source of its own, keeping its look, so editing that look no longer edits the others. A copy says what it copies. | -| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. For a role, pick the source filling it. | +| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. **Warn when it goes dark**, on by default for a camera, display, or window, watches it for [going dark](#a-capture-gone-dark). For a role, pick the source filling it. | Its **Animate** tab holds the layer's keyframes (see [Keyframes](#keyframes)). With no layer selected, the inspector shows the edited scene: rename it, @@ -347,7 +354,10 @@ the [hotkeys](#hotkeys), not in the scene file. monitor with every scene and the sources you pick live in one grid, as large as the window allows: the scene on air framed in blue and tagged ON AIR, the one cued (or, off air, the one the stream opens on) framed in amber, and a -blue dot on each source the live picture shows. Over the grid are the show's +blue dot on each source the live picture shows. A source +[gone dark](#a-capture-gone-dark) or [stalled](#a-picture-that-stalls), and +each scene showing it, carries the amber badge, and the source's picture says +so. Over the grid are the show's name, a clock in this computer's time zone, the stream's health, and how long it has been live. Click a scene's tile to take it, or cue it in Studio mode, as its pad does; its hotkey shows on the tile and works with the multiview in @@ -1231,6 +1241,8 @@ published independently takes no shape. `[video.]` sets a source's `background`, `crop`, and `filters`, wherever it is shown, and a layer's `crop`, `filters`, and `blend` its own. +`expect_dark = true` on a camera, display, or window says it is meant to go +dark at times, so the studio never flags it [gone dark](#a-capture-gone-dark). `background = "remove"` cuts the person out of a camera, display, window, image, media, broadcast, browser, or replay source's picture before its crop and filters, leaving the rest transparent. A segmentation model finds them in @@ -1685,6 +1697,54 @@ no past to replay: a page publishes no track of its own, so replay the program to show it. A replay source on program encodes each replay as its own track while it plays, like a camera. +## A capture gone dark + +A camera, display, or window can go black while its capture still runs: a +window closed, minimized, or switched to exclusive full screen, a screen +locked or showing protected video, a camera behind its shutter, or a +permission the system took away. Four times a second at most, and only when +it delivers a picture (a Linux screen capture repeats its last one while the +screen stands still, so it is read every time), the studio shrinks each +one's picture on the GPU to a grid of 64 by 36 tiles and reads the grid +back. A picture dark in every +tile but a mouse pointer's few, and all one color, is dark. + +For its first 3 seconds of dark the studio draws the capture's last picture +that was not dark in its place, on program and everywhere else but where its +background is removed, which shows the dark at once, so a moment +of black barely reaches viewers: the black shows until its picture is next +read, up to a quarter of a second, as each picture is drawn before it is +read, never held back for it. After that it shows the dark as it is, so a +stale picture never passes for live, and the capture's layer row, its +layer on the picture, and each scene pad showing it turn amber, and the +health dot counts it. The row's badge says what may have happened and offers +the fixes the studio can take: pick a picked window or display again in the +system picker, capture another listed window or the whole screen in its +place, or open a camera or display again. As with **Reload**, picking or +opening it again leaves its last picture on program until it delivers a new +one, even while the picker is open, so cut away first if that picture should +not show. Where only you can fix it, such as +a permission macOS took away or a game that refuses capture, it says what to +do and where. **It is meant to go dark** stops watching it, as +`expect_dark` in its `[video]` table, and **Warn when it goes dark** in the +inspector's Source section watches it again. + +A source published independently reaches viewers on its own track, which +shows the dark at once. + +### A picture that stalls + +A camera or broadcast can also stop sending pictures, so the program repeats +its last one: a camera unplugged or taken by another app, or a broadcast +whose sender stopped or lost its connection. A second without a new picture +counts. Its layer row reads **No new picture for 3 s**, and its layer on the +picture, each scene pad showing it, and the health dot turn amber as for a +capture gone dark. The badge offers to open the camera or subscribe to the +broadcast again, and says what to check where only you can: the camera's +cable and any other app using it, or the broadcast's sender. A window or +display that stands still is not stalled, as nothing new to draw sends no +new picture. + ## Health The Health window, from the health dot's **Details**, grades the stream every second on the thresholds moq's @@ -1695,7 +1755,8 @@ stuck, a source stalled (a camera or broadcast the program or a published scene draws, or a microphone or system audio that the audio scene on program or one a published scene carries plays: unmuted and held by it), a source one of them shows or plays failed or, as a media file that does not loop, -played to its end, a browser source one of them shows is uploaded from memory +played to its end, a camera, display, or window one of them shows +[went dark](#a-capture-gone-dark), a browser source one of them shows is uploaded from memory rather than copied on the GPU, or viewers are late, stalling, or cannot decode a track. A relay lost while the other serves is degraded, since viewers have nothing left to fall back to; with none serving diff --git a/justfile b/justfile index 248d7779..d6e4d77d 100644 --- a/justfile +++ b/justfile @@ -58,7 +58,7 @@ render-sw: mesa=$(nix build --inputs-from . --no-link --print-out-paths nixpkgs#mesa) icd=("$mesa"/share/vulkan/icd.d/lvp_icd.*.json) export VK_DRIVER_FILES="${icd[0]}" - cargo nextest run --locked -p moq-scene --lib --run-ignored only -E 'test(/^compositor::tests::/) | test(/^browser::layers::tests::/)' + cargo nextest run --locked -p moq-scene --lib --run-ignored only -E 'test(/^compositor::tests::/) | test(/^dark::tests::/) | test(/^browser::layers::tests::/)' cargo nextest run --locked -p moq-studio --features openh264 --run-ignored only -E '(test(/^engine::tests::/) - test(=engine::tests::the_virtual_camera_reaches_a_pipewire_app)) | test(=setup::tests::times_every_canvas)' cargo nextest run --locked -p moq-studio --features cef --run-ignored only -E 'binary(browser)' sh/pipewire-session.sh cargo nextest run --locked -p moq-studio --features pipewire --run-ignored only -E 'test(=engine::tests::the_virtual_camera_reaches_a_pipewire_app)' diff --git a/quest/m1/capture-health.md b/quest/m1/capture-health.md deleted file mode 100644 index e3032802..00000000 --- a/quest/m1/capture-health.md +++ /dev/null @@ -1,30 +0,0 @@ -# [M] A capture that went black says so - -## Goal - -When a camera, window, or display source goes black or stops sending -pictures, its layer carries an amber badge on the picture and in Layers, and -a popover says what happened in plain words and offers the fix: pick the -window again, capture the whole screen instead, or reopen the device. The -health dot counts it. See the `capturehealth` frame of -[`mock/later.html`](/quest/m0/ux/mock/later.html). - -## Plan - -A black capture is the commonest complaint about OBS, where the picture is -simply black and the cause is a forum search away. - -- A stalled camera or broadcast is already detected (no new picture for a - second). A window or display that stands still is not a fault, so those - are judged by content: a picture that is all one dark color for a - sustained time while its source claims to be live. -- The check reads a far smaller copy of the frame at a low rate, on the GPU. - Measure its cost per source. -- Each fix is an action the studio can take itself; where it cannot (a - permission was revoked, a game refuses capture), the popover says what the - streamer must do, and where. -- Recommended, not decided, what viewers see meanwhile: the last good - picture for a few seconds, then the black capture as it is, so a hiccup is - invisible but a stale frame never passes for live. The alternatives are - black at once, as today, or a placeholder the show chooses. -- A source that is meant to be black can dismiss the badge for good. diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index abc28065..a4613cfc 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -1917,6 +1917,12 @@ pub struct Video { /// Applied in order, before each item's own. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub filters: Vec, + /// Whether a camera, display, or window is meant to go all dark at + /// times, such as a camera whose shutter closes between segments: the + /// compositor never judges it gone dark, nor holds its last picture over + /// the dark. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + pub expect_dark: bool, } /// What happens to the background behind a person in a source's picture. @@ -2172,6 +2178,15 @@ impl Source { ) } + /// Whether the source is a camera, display, or window capture, which the + /// compositor watches for going dark; see [`Compositor::dark`](crate::Compositor::dark). + pub fn may_go_dark(&self) -> bool { + matches!( + self, + Self::Camera { .. } | Self::Display { .. } | Self::Window { .. } | Self::PickedDisplay | Self::PickedWindow + ) + } + /// Whether the source's picture can have its background removed: a /// picture of its own that may show a person. pub fn removes_background(&self) -> bool { diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index fe3739f0..12b40a98 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -15,6 +15,7 @@ use crate::collection::{ Background, Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Nested, Rect, Settled, Shape, Source, Transition, }; +use crate::dark::{Probe, Prober}; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; use crate::glide::{Glide, REFLOW, mixed, pair, stack}; use crate::matte::Matte; @@ -91,6 +92,9 @@ pub struct Compositor { nested: HashMap>, /// Lays out and paints overlays, started with the first one. overlays: Option, + /// Reduces captures' pictures to see if they went dark, built with the + /// first probe's first read. + prober: Option, /// The thumbnails shown, by what they show. thumbnails: BTreeMap, /// How many renders have run, which picks the thumbnails each draws. @@ -322,9 +326,26 @@ struct Layer { overlay: Option, /// Why the newest picture failed to draw, logged when it first does. error: Option, + /// Watches a camera, display, or window for going dark, unless it is + /// expected to. + probe: Option, + /// Whether `picture` is the probe's last good one, held over the dark. + holding: bool, } impl Layer { + /// Forget every picture from before a clear, so none may show, even if + /// the picture after it fails to draw: the newest, and the probe's last + /// good one and the read it has in flight. + fn clear(&mut self, device: &wgpu::Device, queue: &wgpu::Queue) { + self.cleared = true; + self.raw = None; + self.fresh = true; + if let Some(probe) = &mut self.probe { + *probe = Probe::new(device, queue); + } + } + /// Bring the picture drawn up to date with the newest one delivered, /// through its matte while `background` is removed. fn develop( @@ -424,6 +445,7 @@ impl Compositor { layers: HashMap::new(), nested: HashMap::new(), overlays: None, + prober: None, thumbnails: BTreeMap::new(), renders: 0, named: 0, @@ -609,6 +631,16 @@ impl Compositor { } } + /// How long the camera, display, or window called `source` has shown a + /// picture all one dark color while live, once that is at least + /// [`DARK`](crate::dark::DARK) at `now`. Until then the compositor draws + /// the last picture it showed that was not dark in its place, if any; + /// after, the dark picture as it is. Never for a source expected to go + /// dark, nor any other kind. + pub fn dark(&self, source: &str, now: Instant) -> Option { + self.layers.get(source)?.probe.as_ref()?.dark(now) + } + /// The width and height in pixels of `source`'s newest picture, a capture /// named as [`Collection::capture`] names it, once one is drawn. pub fn picture_size(&self, source: &str) -> Option<(u32, u32)> { @@ -1117,16 +1149,25 @@ impl Compositor { lut: None, overlay: None, error: None, + probe: None, + holding: false, }, None => continue, }; + let probed = source.may_go_dark() && !collection.video.get(name).is_some_and(|video| video.expect_dark); + match (&layer.probe, probed) { + (None, true) => layer.probe = Some(Probe::new(&self.device, &self.queue)), + (Some(_), false) => layer.probe = None, + _ => {} + } + // A feed started again, even between two renders, holds nothing + // from the one before. + if let Some(probe) = &mut layer.probe { + probe.follow(feed.generation()); + } if let Some(taken) = taken { if taken.cleared { - // Nothing from before the clear may show, even if the picture - // after it fails to draw. - layer.cleared = true; - layer.raw = None; - layer.fresh = true; + layer.clear(&self.device, &self.queue); } self.take(name, source, &mut layer, taken.picture, now); } @@ -1136,10 +1177,46 @@ impl Compositor { .map(|video| video.background) .unwrap_or_default(); layer.develop(name, background, &self.device, &self.queue, &self.effects); + let live = feed.status() == Status::Live; + self.watch(&mut layer, background, live, now); self.layers.insert(name.clone(), layer); } } + /// Have `layer`'s probe read its newest picture, and draw the probe's last + /// good picture in its place while it holds one over the dark. A picture + /// cut out of its background is never held, since the good one is not. + /// A picture is drawn before it is judged, so black shows until the next + /// read, up to [`INTERVAL`](crate::dark::INTERVAL) and its readback: + /// judging each one first would hold every capture's picture back by a + /// readback, or read every picture, and the maintainer took this bounded + /// flash over either on moq-dev/studio#192. + /// Without a probe, such as once its warning is turned off, or a picture, + /// it holds nothing. + fn watch(&mut self, layer: &mut Layer, background: Background, live: bool, now: Instant) { + let (Some(probe), Some(raw)) = (&mut layer.probe, &layer.raw) else { + if std::mem::take(&mut layer.holding) && background == Background::Keep { + layer.picture = layer.raw.as_ref().map(|raw| self.effects.picture(raw)); + } + return; + }; + let prober = self.prober.get_or_insert_with(|| Prober::new(&self.device)); + probe.step(prober, &self.effects, raw, live, now); + match probe.held(now).filter(|_| background == Background::Keep) { + Some(held) => { + layer.picture = Some(held.clone()); + layer.holding = true; + } + None if layer.holding => { + layer.holding = false; + if background == Background::Keep { + layer.picture = layer.raw.as_ref().map(|raw| self.effects.picture(raw)); + } + } + None => {} + } + } + /// Put `picture` from source `name` on the GPU as `layer`'s newest. An /// overlay that arrives now starts its CSS timeline at `now`. fn take(&mut self, name: &str, source: &Source, layer: &mut Layer, picture: source::Picture, now: Instant) { @@ -1151,8 +1228,14 @@ impl Compositor { source::Picture::Frame(frame) => { let renderer = match layer.renderer.as_mut() { Some(renderer) => Ok(renderer), - None => Renderer::new(&self.device, &self.queue, Default::default()) - .map(|renderer| layer.renderer.insert(renderer)), + None => { + // A probe copies each picture it reads. + let mut config = moq_video::render::Config::default(); + if source.may_go_dark() { + config.usage = wgpu::TextureUsages::COPY_SRC; + } + Renderer::new(&self.device, &self.queue, config).map(|renderer| layer.renderer.insert(renderer)) + } }; renderer .and_then(|renderer| renderer.render(&frame)) @@ -1192,6 +1275,9 @@ impl Compositor { layer.error = None; layer.raw = Some(texture); layer.fresh = true; + if let Some(probe) = &mut layer.probe { + probe.changed(); + } } Err(err) => { if layer.error.is_none() { @@ -3823,6 +3909,124 @@ mod tests { .await; } + /// A camera that goes dark for a moment draws its last good picture in + /// place of the dark, unless its background is removed; dark for + /// [`DARK`](crate::dark::DARK), it draws its own again and counts as + /// gone dark. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_moment_of_dark_draws_the_last_good_picture() { + let (device, queue) = gpu().await; + let mut compositor = Compositor::new(&device, &queue, collection().canvas).unwrap(); + let size = wgpu::Extent3d { + width: 64, + height: 36, + depth_or_array_layers: 1, + }; + let raw = device.create_texture(&wgpu::TextureDescriptor { + label: None, + size, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D2, + format: FORMAT, + usage: wgpu::TextureUsages::TEXTURE_BINDING | wgpu::TextureUsages::COPY_SRC | wgpu::TextureUsages::COPY_DST, + view_formats: &[], + }); + let show = |rgba: [u8; 4]| { + queue.write_texture( + raw.as_image_copy(), + &rgba.repeat(64 * 36), + wgpu::TexelCopyBufferLayout { + offset: 0, + bytes_per_row: Some(64 * 4), + rows_per_image: None, + }, + size, + ); + }; + let mut layer = Layer { + source: Source::Camera { id: None }, + renderer: None, + raw: Some(raw.clone()), + fresh: true, + cleared: false, + matte: None, + picture: None, + lut: None, + overlay: None, + error: None, + probe: Some(Probe::new(&device, &queue)), + holding: false, + }; + let start = Instant::now(); + let at = |offset: Duration| start.saturating_add(offset); + // Each picture is read, then judged on the next watch once the GPU is done. + let read = |compositor: &mut Compositor, layer: &mut Layer, rgba, when, background| { + show(rgba); + layer.probe.as_mut().unwrap().changed(); + compositor.watch(layer, background, true, when); + device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); + compositor.watch(layer, background, true, when); + }; + + read(&mut compositor, &mut layer, RED, at(Duration::ZERO), Background::Keep); + assert!(!layer.holding); + let dark = crate::dark::INTERVAL; + read(&mut compositor, &mut layer, BLACK, at(dark), Background::Keep); + assert!(layer.holding, "the last good picture is drawn"); + assert_eq!(layer.probe.as_ref().unwrap().dark(at(dark)), None); + + // A cut-out never shows the held picture, which has its background. + compositor.watch(&mut layer, Background::Remove, true, at(dark)); + assert!(!layer.holding); + compositor.watch(&mut layer, Background::Keep, true, at(dark)); + assert!(layer.holding); + + // Turning its warning off draws its own picture again at once, with + // no new picture to replace the held one. + let probe = layer.probe.take(); + compositor.watch(&mut layer, Background::Keep, true, at(dark)); + assert!(!layer.holding, "no probe holds a picture"); + assert!(layer.picture.is_some()); + layer.probe = probe; + compositor.watch(&mut layer, Background::Keep, true, at(dark)); + assert!(layer.holding); + + // A clear forgets the last good picture: black after it is drawn + // as it is, never the red from before it. + let mut cleared = Layer { + source: Source::Camera { id: None }, + renderer: None, + raw: Some(raw.clone()), + fresh: true, + cleared: false, + matte: None, + picture: None, + lut: None, + overlay: None, + error: None, + probe: Some(Probe::new(&device, &queue)), + holding: false, + }; + read(&mut compositor, &mut cleared, RED, at(Duration::ZERO), Background::Keep); + read(&mut compositor, &mut cleared, BLACK, at(dark), Background::Keep); + assert!(cleared.holding, "the red is held over the black"); + cleared.clear(&device, &queue); + compositor.watch(&mut cleared, Background::Keep, true, at(dark)); + assert!(!cleared.holding, "nothing from before the clear is held"); + assert!(cleared.picture.is_none(), "nothing from before the clear is drawn"); + cleared.raw = Some(raw.clone()); + read(&mut compositor, &mut cleared, BLACK, at(dark * 2), Background::Keep); + assert!(!cleared.holding, "the red from before the clear is never held again"); + + let gone = dark + crate::dark::DARK; + compositor.watch(&mut layer, Background::Keep, true, at(gone)); + assert!(!layer.holding, "the dark is drawn as it is"); + assert!(layer.picture.is_some()); + assert_eq!(layer.probe.as_ref().unwrap().dark(at(gone)), Some(crate::dark::DARK)); + } + /// A source that clears its picture, such as a media source played /// again, never shows the cut-out of a picture from before, even when its /// next picture overtakes the clear. @@ -3853,6 +4057,8 @@ mod tests { lut: None, overlay: None, error: None, + probe: None, + holding: false, }; let develop = |layer: &mut Layer| layer.develop("replay", Background::Remove, &device, &queue, &effects); let wait = |layer: &mut Layer| { diff --git a/rs/moq-scene/src/dark.rs b/rs/moq-scene/src/dark.rs new file mode 100644 index 00000000..624039b4 --- /dev/null +++ b/rs/moq-scene/src/dark.rs @@ -0,0 +1,636 @@ +//! Whether a capture went dark: its picture all one dark color while its +//! source claims to be live, such as a window that closed or changed how it +//! draws, a locked screen, or a camera behind its shutter. +//! +//! A camera that stops delivering already reads as stalled, but a display or +//! window that stands still is no fault, so a capture is judged by what it +//! shows. A [`Probe`] reads the newest picture at most every [`INTERVAL`], +//! on the GPU: a compute shader reduces it to a small grid of tiles, and only +//! the grid comes back, a frame or so later, to be judged on the CPU. +//! +//! Until a capture has been dark for [`DARK`], the probe holds the last +//! picture it judged not dark, for the compositor to draw in its place, so a +//! moment of black, such as a game switching modes, never reaches viewers. +//! After that the capture shows as it is, so a stale picture never passes for +//! live. + +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use kio::clock::Instant; + +use crate::effect::{Effects, Picture}; + +/// How often a probe reads a capture's picture, at most, and only when one +/// arrives: a camera or window that stops delivering is not read, though a +/// Linux screen capture repeats its last picture while it stands still, and +/// is read each time. +pub const INTERVAL: Duration = Duration::from_millis(250); + +/// How long a capture's picture stays dark before it counts as gone dark, +/// rather than a moment of black the last good picture covers. +pub const DARK: Duration = Duration::from_secs(3); + +/// The grid a picture is reduced to, as `dark.wgsl`'s `TILES`: about 30 +/// texels a side at 1080p. +const TILES: (u32, u32) = (64, 36); + +/// The bytes of the reduced grid, four a tile. +const BYTES: u64 = (TILES.0 * TILES.1 * 4) as u64; + +/// The brightest a dark picture's brightest channel gets, out of 255: +/// a camera's noise in the dark stays under it, and any lit pixel passes it. +const BRIGHTEST: u8 = 40; + +/// How many tiles of a dark picture may be lit anyway: a mouse pointer the +/// capture draws over the black spans up to four. +const LIT: usize = 4; + +/// How far apart a dark picture's tiles' mean colors get, out of 255: a dark +/// page with a line of text on it spreads further. +const SPREAD: u8 = 8; + +/// Whether a picture reduced to `tiles`, each RGBA with its mean color and +/// its brightest channel as alpha, is all one dark color. +fn judge(tiles: &[u8]) -> bool { + let mut low = [u8::MAX; 3]; + let mut high = [0u8; 3]; + let mut lit = 0; + for tile in tiles.as_chunks::<4>().0 { + if tile[3] > BRIGHTEST { + lit += 1; + if lit > LIT { + return false; + } + continue; + } + for channel in 0..3 { + low[channel] = low[channel].min(tile[channel]); + high[channel] = high[channel].max(tile[channel]); + } + } + (0..3).all(|channel| high[channel].saturating_sub(low[channel]) <= SPREAD) +} + +/// The pipeline every probe reduces its pictures with. +pub(crate) struct Prober { + pipeline: wgpu::ComputePipeline, + layout: wgpu::BindGroupLayout, +} + +impl Prober { + pub fn new(device: &wgpu::Device) -> Self { + let shader = device.create_shader_module(wgpu::include_wgsl!("dark.wgsl")); + let layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor { + label: Some("moq-scene dark"), + entries: &[ + wgpu::BindGroupLayoutEntry { + binding: 0, + visibility: wgpu::ShaderStages::COMPUTE, + ty: wgpu::BindingType::Texture { + sample_type: wgpu::TextureSampleType::Float { filterable: false }, + view_dimension: wgpu::TextureViewDimension::D2, + multisampled: false, + }, + count: None, + }, + wgpu::BindGroupLayoutEntry { + binding: 1, + visibility: wgpu::ShaderStages::COMPUTE, + ty: wgpu::BindingType::Buffer { + ty: wgpu::BufferBindingType::Storage { read_only: false }, + has_dynamic_offset: false, + min_binding_size: wgpu::BufferSize::new(BYTES), + }, + count: None, + }, + ], + }); + let pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor { + label: Some("moq-scene dark"), + bind_group_layouts: &[Some(&layout)], + immediate_size: 0, + }); + let pipeline = device.create_compute_pipeline(&wgpu::ComputePipelineDescriptor { + label: Some("moq-scene dark"), + layout: Some(&pipeline_layout), + module: &shader, + entry_point: Some("reduce"), + compilation_options: Default::default(), + cache: None, + }); + Self { pipeline, layout } + } +} + +/// A read under way: when its picture was taken, and the buffer's mapping +/// once the GPU is done. +struct Reading { + at: Instant, + mapped: Arc>>>, + /// Whether the capture stopped being live since, so what it read is + /// from before and judged neither dark nor good. + stale: bool, +} + +/// Watches one capture's pictures for going dark. +pub(crate) struct Probe { + device: wgpu::Device, + queue: wgpu::Queue, + tiles: wgpu::Buffer, + readback: wgpu::Buffer, + /// A copy of the picture last read, since the capture's own texture is + /// drawn over by its next picture before the read comes back. + read: Option, + /// The newest picture judged not dark, bound to draw. + good: Option<(wgpu::Texture, Picture)>, + reading: Option, + /// When the last read was taken. + last: Option, + /// Whether a picture arrived since the last read. + changed: bool, + /// When the first of the dark pictures read since the last good one was + /// taken. + dark: Option, + /// The [generation](crate::source::Feed::generation) of the feed it + /// watches, once it has watched one. + feed: Option, +} + +impl Probe { + pub fn new(device: &wgpu::Device, queue: &wgpu::Queue) -> Self { + let tiles = device.create_buffer(&wgpu::BufferDescriptor { + label: Some("moq-scene dark tiles"), + size: BYTES, + usage: wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC, + mapped_at_creation: false, + }); + let readback = device.create_buffer(&wgpu::BufferDescriptor { + label: Some("moq-scene dark tiles"), + size: BYTES, + usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ, + mapped_at_creation: false, + }); + Self { + device: device.clone(), + queue: queue.clone(), + tiles, + readback, + read: None, + good: None, + reading: None, + last: None, + changed: true, + dark: None, + feed: None, + } + } + + /// Watch the feed of `generation`: one restarted, such as opened or + /// picked again, has a new one, and nothing read from the feed before + /// may be held for it, as that may be another window. Nor is the picture + /// the layer still has from before read for it: only the new feed's own. + pub fn follow(&mut self, generation: u64) { + if self.feed.replace(generation).is_some_and(|was| was != generation) { + self.forget(); + self.changed = false; + } + } + + /// Forget the last good picture and what is being read, and that it is + /// dark. + fn forget(&mut self) { + self.dark = None; + self.good = None; + if let Some(reading) = &mut self.reading { + reading.stale = true; + } + } + + /// Note that the capture's texture holds a new picture. + pub fn changed(&mut self) { + self.changed = true; + } + + /// Judge the read that came back, if one did, and read `picture`, the + /// capture's newest, if it is new and the last read was at least + /// [`INTERVAL`] ago. `picture` needs `COPY_SRC`. + /// + /// A capture that is not `live`, such as one reopening, is not dark, and + /// forgets its last good picture and what it is reading, so nothing from + /// before it reopens is drawn for it after: a picked window picked again + /// may be another window. + pub fn step(&mut self, prober: &Prober, effects: &Effects, picture: &wgpu::Texture, live: bool, now: Instant) { + self.judge(effects); + if !live { + self.forget(); + return; + } + let due = self + .last + .is_none_or(|last| now.saturating_duration_since(last) >= INTERVAL); + if self.reading.is_none() && self.changed && due { + self.start(prober, picture, now); + } + } + + /// How long the capture has been dark, once that is at least [`DARK`]. + pub fn dark(&self, now: Instant) -> Option { + let since = now.saturating_duration_since(self.dark?); + (since >= DARK).then_some(since) + } + + /// The picture to draw in place of the capture's own: the last good one, + /// while the capture has been dark for less than [`DARK`]. + pub fn held(&self, now: Instant) -> Option<&Picture> { + let since = now.saturating_duration_since(self.dark?); + let (_, picture) = self.good.as_ref()?; + (since < DARK).then_some(picture) + } + + /// Copy `picture` and reduce the copy to tiles, read back once the GPU is + /// done. + fn start(&mut self, prober: &Prober, picture: &wgpu::Texture, now: Instant) { + let size = picture.size(); + let read = match self.read.take() { + Some(read) if read.size() == size && read.format() == picture.format() => read, + _ => self.device.create_texture(&wgpu::TextureDescriptor { + label: Some("moq-scene dark read"), + size, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D2, + format: picture.format(), + usage: wgpu::TextureUsages::COPY_DST | wgpu::TextureUsages::TEXTURE_BINDING, + view_formats: &[], + }), + }; + let bind = self.device.create_bind_group(&wgpu::BindGroupDescriptor { + label: Some("moq-scene dark"), + layout: &prober.layout, + entries: &[ + wgpu::BindGroupEntry { + binding: 0, + resource: wgpu::BindingResource::TextureView(&read.create_view(&Default::default())), + }, + wgpu::BindGroupEntry { + binding: 1, + resource: self.tiles.as_entire_binding(), + }, + ], + }); + let mut encoder = self.device.create_command_encoder(&Default::default()); + encoder.copy_texture_to_texture(picture.as_image_copy(), read.as_image_copy(), size); + { + let mut pass = encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { + label: Some("moq-scene dark"), + timestamp_writes: None, + }); + pass.set_pipeline(&prober.pipeline); + pass.set_bind_group(0, &bind, &[]); + pass.dispatch_workgroups(TILES.0, TILES.1, 1); + } + encoder.copy_buffer_to_buffer(&self.tiles, 0, &self.readback, 0, BYTES); + + self.queue.submit([encoder.finish()]); + let mapped = Arc::new(Mutex::new(None)); + let done = mapped.clone(); + // Called from a later submit or poll, once the GPU is done. + self.readback.map_async(wgpu::MapMode::Read, .., move |result| { + *done.lock().unwrap() = Some(result); + }); + self.read = Some(read); + self.reading = Some(Reading { + at: now, + mapped, + stale: false, + }); + self.last = Some(now); + self.changed = false; + } + + /// Judge the read under way, if it came back: a dark picture starts or + /// continues the dark, and any other ends it and becomes the good one. + fn judge(&mut self, effects: &Effects) { + let Some(reading) = &self.reading else { return }; + let Some(mapped) = reading.mapped.lock().unwrap().take() else { + return; + }; + let (at, stale) = (reading.at, reading.stale); + self.reading = None; + let dark = mapped.map_err(|err| err.to_string()).and_then(|()| { + let dark = match self.readback.slice(..).get_mapped_range() { + Ok(view) => Ok(judge(&view)), + Err(err) => Err(format!("{err:?}")), + }; + self.readback.unmap(); + dark + }); + let dark = match dark { + Ok(_) if stale => return, + Ok(dark) => dark, + Err(err) => { + tracing::warn!(%err, "cannot read a capture back to see if it went dark"); + return; + } + }; + if dark { + self.dark.get_or_insert(at); + return; + } + self.dark = None; + let read = self.read.take().expect("a read has its copy"); + let picture = effects.picture(&read); + // The good picture before becomes the next read's copy. + self.read = self.good.replace((read, picture)).map(|(texture, _)| texture); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// `tiles` tiles of one color, its brightest channel as given. + fn plain(rgb: [u8; 3], brightest: u8) -> Vec { + [rgb[0], rgb[1], rgb[2], brightest].repeat((TILES.0 * TILES.1) as usize) + } + + #[test] + fn black_is_dark() { + assert!(judge(&plain([0, 0, 0], 0))); + } + + /// Studio-range black, or a camera's noise behind its shutter. + #[test] + fn near_black_with_noise_is_dark() { + let mut tiles = plain([16, 16, 16], 30); + tiles[4..8].copy_from_slice(&[20, 18, 19, 38]); + assert!(judge(&tiles)); + } + + /// A pointer the capture draws over the black lights a few tiles, and + /// leaves it dark. + #[test] + fn a_pointer_on_black_is_dark() { + let mut tiles = plain([0, 0, 0], 0); + for tile in [100, 101, 164, 165] { + tiles[tile * 4..tile * 4 + 4].copy_from_slice(&[40, 40, 40, 255]); + } + assert!(judge(&tiles)); + } + + /// A little more lit than a pointer is not dark, however small each + /// light: a line of text, or the stars of a night sky. + #[test] + fn a_few_lit_pixels_are_not_dark() { + let mut tiles = plain([0, 0, 0], 0); + // Each tile's mean barely moves, but its brightest does. + for tile in [100, 300, 500, 700, 900] { + tiles[tile * 4..tile * 4 + 4].copy_from_slice(&[1, 1, 1, 255]); + } + assert!(!judge(&tiles)); + } + + /// A dim page with a line of dim text: every texel is dark, but not one + /// color. + #[test] + fn dim_text_is_not_dark() { + let mut tiles = plain([10, 10, 10], 12); + for tile in 64..96 { + tiles[tile * 4..tile * 4 + 4].copy_from_slice(&[24, 24, 24, 40]); + } + assert!(!judge(&tiles)); + } + + #[test] + fn a_bright_color_is_not_dark() { + assert!(!judge(&plain([200, 200, 200], 200))); + } + + async fn gpu() -> (wgpu::Device, wgpu::Queue) { + let instance = wgpu::Instance::default(); + let adapter = instance + .request_adapter(&wgpu::RequestAdapterOptions::default()) + .await + .expect("a GPU adapter"); + adapter + .request_device(&wgpu::DeviceDescriptor::default()) + .await + .expect("a GPU device") + } + + /// A capture's 1080p picture, as the compositor's renderer leaves it. + struct Capture { + queue: wgpu::Queue, + texture: wgpu::Texture, + } + + impl Capture { + const SIZE: (u32, u32) = (1920, 1080); + + fn new(device: &wgpu::Device, queue: &wgpu::Queue) -> Self { + let texture = device.create_texture(&wgpu::TextureDescriptor { + label: None, + size: wgpu::Extent3d { + width: Self::SIZE.0, + height: Self::SIZE.1, + depth_or_array_layers: 1, + }, + mip_level_count: 1, + sample_count: 1, + dimension: wgpu::TextureDimension::D2, + format: crate::FORMAT, + usage: wgpu::TextureUsages::TEXTURE_BINDING + | wgpu::TextureUsages::COPY_SRC + | wgpu::TextureUsages::COPY_DST, + view_formats: &[], + }); + Self { + queue: queue.clone(), + texture, + } + } + + /// Fill it with `rgb`, and a one-texel line of `line` across row 500. + fn show(&self, rgb: [u8; 3], line: Option<[u8; 3]>) { + let (width, height) = Self::SIZE; + let mut pixels = [rgb[0], rgb[1], rgb[2], 255].repeat((width * height) as usize); + if let Some(line) = line { + let row = (500 * width * 4) as usize; + for pixel in pixels[row..row + (width * 4) as usize].as_chunks_mut::<4>().0 { + pixel[..3].copy_from_slice(&line); + } + } + self.queue.write_texture( + self.texture.as_image_copy(), + &pixels, + wgpu::TexelCopyBufferLayout { + offset: 0, + bytes_per_row: Some(width * 4), + rows_per_image: None, + }, + self.texture.size(), + ); + } + } + + /// Read `capture` at `at`, and judge the read once the GPU is done. + fn read( + probe: &mut Probe, + prober: &Prober, + effects: &Effects, + device: &wgpu::Device, + capture: &Capture, + at: Instant, + ) { + probe.changed(); + probe.step(prober, effects, &capture.texture, true, at); + assert!(probe.reading.is_some(), "a read started"); + device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); + probe.step(prober, effects, &capture.texture, true, at); + assert!(probe.reading.is_none(), "the read came back"); + } + + /// The shader reads every texel: one line of light across a black 1080p + /// picture is enough to keep it from going dark. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn every_texel_counts() { + let (device, queue) = gpu().await; + let effects = Effects::new(&device, &queue, 64, 64); + let prober = Prober::new(&device); + let capture = Capture::new(&device, &queue); + let mut probe = Probe::new(&device, &queue); + let start = kio::clock::now(); + let at = |offset: Duration| start.saturating_add(offset); + + capture.show([0, 0, 0], Some([255, 255, 255])); + read(&mut probe, &prober, &effects, &device, &capture, at(Duration::ZERO)); + assert_eq!(probe.dark, None); + + capture.show([0, 0, 0], None); + read(&mut probe, &prober, &effects, &device, &capture, at(INTERVAL)); + assert_eq!(probe.dark, Some(at(INTERVAL))); + } + + /// A moment of dark shows the last good picture in its place; dark for + /// [`DARK`], the capture counts as gone dark and shows as it is; and a + /// picture again ends it. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn holds_the_last_good_picture_over_a_moment_of_dark() { + let (device, queue) = gpu().await; + let effects = Effects::new(&device, &queue, 64, 64); + let prober = Prober::new(&device); + let capture = Capture::new(&device, &queue); + let mut probe = Probe::new(&device, &queue); + let start = kio::clock::now(); + let at = |offset: Duration| start.saturating_add(offset); + + // Dark from its first picture: nothing good to hold. + capture.show([0, 0, 0], None); + read(&mut probe, &prober, &effects, &device, &capture, at(Duration::ZERO)); + assert!(probe.held(at(Duration::ZERO)).is_none()); + + let lit = INTERVAL; + capture.show([90, 120, 200], None); + read(&mut probe, &prober, &effects, &device, &capture, at(lit)); + assert!(probe.held(at(lit)).is_none() && probe.dark(at(lit)).is_none()); + + let dark = lit + INTERVAL; + capture.show([4, 4, 4], None); + read(&mut probe, &prober, &effects, &device, &capture, at(dark)); + let almost = dark + DARK - Duration::from_millis(1); + assert!(probe.held(at(almost)).is_some(), "the last good picture is held"); + assert_eq!(probe.dark(at(almost)), None); + assert!(probe.held(at(dark + DARK)).is_none(), "the dark shows"); + assert_eq!( + probe.dark(at(dark + DARK + Duration::from_secs(9))), + Some(DARK + Duration::from_secs(9)) + ); + + // Not read again without a new picture. + probe.step(&prober, &effects, &capture.texture, true, at(dark + INTERVAL / 2)); + assert!(probe.reading.is_none()); + + // A capture that stops being live, such as one reopening, is not dark, + // and forgets its last good picture: reopened dark, it holds nothing + // from before, which may be another window. + probe.step(&prober, &effects, &capture.texture, false, at(dark + DARK)); + assert_eq!(probe.dark(at(dark + DARK)), None); + let reopened = dark + DARK + INTERVAL; + capture.show([0, 0, 0], None); + read(&mut probe, &prober, &effects, &device, &capture, at(reopened)); + assert!(probe.held(at(reopened)).is_none(), "nothing from before it reopened"); + + let back = reopened + INTERVAL; + capture.show([90, 120, 200], None); + read(&mut probe, &prober, &effects, &device, &capture, at(back)); + assert_eq!(probe.dark(at(back + DARK)), None); + + // A read under way as it stops being live is from before, so it never + // becomes the picture held after. + let before = back + INTERVAL; + capture.show([200, 40, 40], None); + probe.changed(); + probe.step(&prober, &effects, &capture.texture, true, at(before)); + assert!(probe.reading.is_some(), "a read started"); + probe.step(&prober, &effects, &capture.texture, false, at(before)); + device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); + probe.step(&prober, &effects, &capture.texture, true, at(before)); + assert!(probe.reading.is_none(), "the read came back"); + let after = before + INTERVAL; + capture.show([0, 0, 0], None); + read(&mut probe, &prober, &effects, &device, &capture, at(after)); + assert!(probe.held(at(after)).is_none(), "the read from before is not held"); + } + + /// A feed started again, live before the compositor saw it stop, holds + /// nothing read from the feed before: not its last good picture, nor a + /// read under way. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_feed_started_again_holds_nothing_from_before() { + let (device, queue) = gpu().await; + let effects = Effects::new(&device, &queue, 64, 64); + let prober = Prober::new(&device); + let capture = Capture::new(&device, &queue); + let mut probe = Probe::new(&device, &queue); + let start = kio::clock::now(); + let at = |offset: Duration| start.saturating_add(offset); + + probe.follow(1); + capture.show([90, 120, 200], None); + read(&mut probe, &prober, &effects, &device, &capture, at(Duration::ZERO)); + probe.follow(1); + capture.show([0, 0, 0], None); + read(&mut probe, &prober, &effects, &device, &capture, at(INTERVAL)); + assert!( + probe.held(at(INTERVAL)).is_some(), + "the same feed holds its last good picture" + ); + + // A picture from before arrived, but was not read yet. + probe.changed(); + probe.follow(2); + assert!(probe.held(at(INTERVAL)).is_none(), "nothing from the feed before"); + assert_eq!(probe.dark(at(INTERVAL + DARK)), None); + // Live before its first picture, the new feed's layer still has the + // one from before, which is not read for it. + probe.step(&prober, &effects, &capture.texture, true, at(INTERVAL * 2)); + assert!(probe.reading.is_none(), "the picture from before is not read"); + + // A read under way as the feed starts again is from before. + let before = INTERVAL * 2; + capture.show([200, 40, 40], None); + probe.changed(); + probe.step(&prober, &effects, &capture.texture, true, at(before)); + assert!(probe.reading.is_some(), "a read started"); + probe.follow(3); + device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); + let after = before + INTERVAL; + capture.show([0, 0, 0], None); + read(&mut probe, &prober, &effects, &device, &capture, at(after)); + assert!(probe.held(at(after)).is_none(), "the read from before is not held"); + } +} diff --git a/rs/moq-scene/src/dark.wgsl b/rs/moq-scene/src/dark.wgsl new file mode 100644 index 00000000..628c1556 --- /dev/null +++ b/rs/moq-scene/src/dark.wgsl @@ -0,0 +1,45 @@ +// A capture's picture as a grid of tiles: each tile's mean color, and its +// brightest channel of any texel, so one lit pixel in a tile counts. One +// workgroup reduces each tile, its threads reading every texel of it once, +// side by side, so the reads are spread over the whole GPU. + +@group(0) @binding(0) var picture: texture_2d; +// Each tile, row by row, as packed RGBA8: the mean color, and the brightest +// channel as alpha. +@group(0) @binding(1) var tiles: array; + +// The grid, as dark.rs's TILES. +const TILES = vec2(64u, 36u); +const SIDE = 16u; + +var sums: array, 256>; + +@compute @workgroup_size(16, 16) +fn reduce(@builtin(workgroup_id) tile: vec3, @builtin(local_invocation_id) local: vec3, @builtin(local_invocation_index) index: u32) { + let size = textureDimensions(picture); + let start = tile.xy * size / TILES; + // A picture smaller than the grid gives a tile one texel at least. + let end = max(min((tile.xy + 1u) * size / TILES, size), start + 1u); + var sum = vec3(0.0); + var brightest = 0.0; + for (var y = start.y + local.y; y < end.y; y += SIDE) { + for (var x = start.x + local.x; x < end.x; x += SIDE) { + let color = textureLoad(picture, vec2(x, y), 0).rgb; + sum += color; + brightest = max(brightest, max(color.r, max(color.g, color.b))); + } + } + sums[index] = vec4(sum, brightest); + workgroupBarrier(); + for (var half = 128u; half > 0u; half >>= 1u) { + if index < half { + let other = sums[index + half]; + sums[index] = vec4(sums[index].rgb + other.rgb, max(sums[index].a, other.a)); + } + workgroupBarrier(); + } + if index == 0u { + let count = f32((end.x - start.x) * (end.y - start.y)); + tiles[tile.y * TILES.x + tile.x] = pack4x8unorm(vec4(sums[0].rgb / count, sums[0].a)); + } +} diff --git a/rs/moq-scene/src/grant.rs b/rs/moq-scene/src/grant.rs index ab7a4c8a..504ef66e 100644 --- a/rs/moq-scene/src/grant.rs +++ b/rs/moq-scene/src/grant.rs @@ -74,6 +74,14 @@ impl Grants { }) } + /// Forget the grant of the source called `source`, so it asks the picker + /// next time it opens, and stop its claims from writing one again. + pub(crate) fn forget(&self, source: &str) -> Result<(), Error> { + let mut owners = self.owners.lock().unwrap(); + owners.claims.remove(source); + self.change(|grants| grants.remove(source).is_some()) + } + /// Apply `change` to this scene file's table, writing the file back only /// when `change` says it changed something. fn change(&self, change: impl FnOnce(&mut BTreeMap) -> bool) -> Result<(), Error> { @@ -239,6 +247,22 @@ mod tests { assert_eq!(removed.get().unwrap(), None); } + #[test] + fn forgetting_one_keeps_the_rest() { + let dir = tempfile::tempdir().unwrap(); + let grants = Grants::new(dir.path().join("portal.toml"), Path::new("/scene.toml")); + let kept = grants.claim("Kept"); + let forgotten = grants.claim("Game"); + kept.set(Some("a")).unwrap(); + forgotten.set(Some("b")).unwrap(); + grants.forget("Game").unwrap(); + assert_eq!(kept.get().unwrap().as_deref(), Some("a")); + assert_eq!(forgotten.get().unwrap(), None); + // The capture it forgot, still closing, writes nothing back. + forgotten.set(Some("stale")).unwrap(); + assert_eq!(forgotten.get().unwrap(), None); + } + #[test] fn only_the_newest_claim_of_a_present_source_writes() { let dir = tempfile::tempdir().unwrap(); diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index 6055034f..e8d1f2ba 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -49,6 +49,7 @@ pub mod browser; pub mod clip; mod collection; mod compositor; +mod dark; mod edit; mod effect; pub mod file; diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index bdb66293..29d3b57b 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -15,7 +15,7 @@ use std::collections::{BTreeMap, BTreeSet, VecDeque}; use std::future::Future; use std::path::{Path, PathBuf}; use std::pin::Pin; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, Mutex}; use std::time::Duration; @@ -316,8 +316,8 @@ pub struct Sources { /// The sources on program, as last set. on_program: BTreeSet, /// The picked displays and windows the last sync ran, by name, whose - /// grants are kept, or `None` before the first. - picked: Option>, + /// grants are kept, each with its kind, or `None` before the first. + picked: Option>, } impl Sources { @@ -465,19 +465,29 @@ impl Sources { /// Forget the grant of each picked display or window `collection` no /// longer has, such as one removed or renamed, since a grant is a - /// credential. + /// credential, and of each whose kind changed, such as a window made a + /// display, since the portal refuses a grant of another kind. fn forget_grants(&mut self, collection: &Collection) { let Some(grants) = &self.context.grants else { return }; - let picked: BTreeSet = collection + let picked: BTreeMap = collection .sources .iter() .filter(|(_, source)| matches!(source, Source::PickedDisplay | Source::PickedWindow)) - .map(|(name, _)| name.clone()) + .map(|(name, source)| (name.clone(), source.clone())) .collect(); if self.picked.as_ref() == Some(&picked) { return; } - match grants.retain(&picked) { + let mut changed = self.picked.iter().flat_map(|before| { + picked + .iter() + .filter(|(name, source)| before.get(*name).is_some_and(|was| was != *source)) + .map(|(name, _)| name) + }); + let forgotten = changed + .try_for_each(|name| grants.forget(name)) + .and_then(|()| grants.retain(&picked.keys().cloned().collect())); + match forgotten { Ok(()) => self.picked = Some(picked), Err(err) => tracing::warn!(%err, "cannot forget the grants of removed sources"), } @@ -492,6 +502,16 @@ impl Sources { } } + /// Forget the grant of the display or window called `name` and + /// [restart](Self::restart) it, so it asks the system picker afresh. + pub fn pick_again(&mut self, name: &str) -> Result<(), Error> { + if let Some(grants) = &self.context.grants { + grants.forget(name)?; + } + self.restart(name); + Ok(()) + } + /// Stop every source, as removing it would: each [`Driver`] returns and /// its own track finishes cleanly with what it published. The mixes stay, /// and the next [`sync`](Self::sync) starts the sources again. @@ -557,9 +577,20 @@ pub struct Feed { silent: Arc, /// A broadcast source's playout clock, or the one a layer plays on. clock: Option, + /// Which start of its source this is, unique across feeds. + generation: u64, } +/// The next feed's [`Feed::generation`]. +static GENERATION: AtomicU64 = AtomicU64::new(0); + impl Feed { + /// Which start of its source this is: a feed restarted, or started again + /// under the same name, has a new one, so what was drawn from the one + /// before can be told apart and forgotten. + pub fn generation(&self) -> u64 { + self.generation + } /// What a replay of this broadcast source copies: the broadcast it plays, /// up to the newest picture it showed, which is what was just seen. /// @@ -2502,6 +2533,7 @@ fn open( Some((_, schedule)) => Some(schedule.clone()), None => clock.as_ref().map(sync::Producer::consume), }, + generation: GENERATION.fetch_add(1, Ordering::Relaxed), }; let driver = Driver { name: name.to_string(), @@ -3767,6 +3799,32 @@ mod tests { .await; } + /// A source started again has a feed of a new generation, so the + /// compositor forgets what it read from the one before. + #[tokio::test] + async fn a_restarted_feed_is_a_new_generation() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let mut collection = Collection::new(); + collection.sources.insert( + "red".into(), + Source::Color { + color: crate::Rgba([255, 0, 0, 255]), + }, + ); + drop(sources.sync(&collection)); + let first = sources.get("red").unwrap().generation(); + drop(sources.sync(&collection)); + assert_eq!(sources.get("red").unwrap().generation(), first, "a sync keeps the feed"); + sources.restart("red"); + drop(sources.sync(&collection)); + assert_ne!(sources.get("red").unwrap().generation(), first); + }) + .await; + } + #[tokio::test] async fn removing_or_renaming_a_picked_source_forgets_its_grant() { local(async { @@ -3794,6 +3852,18 @@ mod tests { collection.sources.insert("Renamed".into(), screen); drop(sources.sync(&collection)); assert!(!kept("Screen") && !kept("Window")); + + // A window made a display under its name, as "Capture the whole + // screen instead" does, asks the picker afresh. + collection.sources.insert("Game".into(), Source::PickedWindow); + drop(sources.sync(&collection)); + grants.claim("Game").set(Some("window")).unwrap(); + grants.claim("Renamed").set(Some("screen")).unwrap(); + assert!(kept("Game")); + collection.sources.insert("Game".into(), Source::PickedDisplay); + drop(sources.sync(&collection)); + assert!(!kept("Game"), "a grant of another kind is forgotten"); + assert!(kept("Renamed"), "the rest are kept"); }) .await; } diff --git a/rs/moq-studio/shots/dark.toml b/rs/moq-studio/shots/dark.toml new file mode 100644 index 00000000..082e2ff0 --- /dev/null +++ b/rs/moq-studio/shots/dark.toml @@ -0,0 +1,47 @@ +# The show `just shots` renders a capture gone dark on: a game's window under +# the host and a lower third. A view's build cannot capture, so the view says +# the window went dark, and it draws nothing over the black behind it. +version = 1 + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.game] +kind = "picked_window" + +[sources.host] +kind = "image" +path = "person.jpg" + +[sources.black] +kind = "color" +color = "#000000" + +[sources.bar] +kind = "color" +color = "#2563ebe6" + +[[scenes]] +name = "Gameplay" +items = [ + { source = "black", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "game", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "host", rect = { x = 1440, y = 690, width = 400, height = 300 } }, + { source = "bar", rect = { x = 80, y = 900, width = 700, height = 110 } }, +] + +[[scenes]] +name = "Chatting" +items = [ + { source = "black", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "host", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, +] + +[[scenes]] +name = "Break" +items = [ + { source = "black", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "bar", rect = { x = 460, y = 480, width = 1000, height = 120 } }, +] diff --git a/rs/moq-studio/shots/stalled.toml b/rs/moq-studio/shots/stalled.toml new file mode 100644 index 00000000..58f9ed49 --- /dev/null +++ b/rs/moq-studio/shots/stalled.toml @@ -0,0 +1,35 @@ +# The show `just shots` renders a stalled camera on: the host's camera beside +# a lower third. A view's build cannot capture, so the view says the camera +# stalled, and it draws nothing over the backdrop behind it. +version = 1 + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.host] +kind = "camera" + +[sources.backdrop] +kind = "color" +color = "#1e293b" + +[sources.bar] +kind = "color" +color = "#2563ebe6" + +[[scenes]] +name = "Interview" +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "host", rect = { x = 160, y = 90, width = 1600, height = 900 } }, + { source = "bar", rect = { x = 80, y = 900, width = 700, height = 110 } }, +] + +[[scenes]] +name = "Break" +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "bar", rect = { x = 460, y = 480, width = 1000, height = 120 } }, +] diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 088bf4de..7999791a 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -129,9 +129,68 @@ pub struct State { /// [`Collection::renames`] reads it, so what the window follows a scene by /// name with, such as a hotkey, follows it too. renamed: Vec>>, + /// Sources a view shows gone dark for so long, as no capture can be in + /// a view's headless build. + #[cfg(test)] + pub dark: BTreeMap, + /// Sources a view shows stalled for so long, as no camera or broadcast + /// can be in a view's headless build. + #[cfg(test)] + pub stalled: BTreeMap, +} + +/// Why a layer shows no new picture, once that counts: what its badge says. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Lost { + /// A camera, display, or window all dark this long. + Dark(Duration), + /// A camera or broadcast without a new picture this long, so its last + /// one repeats. + Stalled(Duration), +} + +impl Lost { + /// How long it has gone without a picture. + pub fn since(&self) -> Duration { + match self { + Self::Dark(since) | Self::Stalled(since) => *since, + } + } } impl State { + /// What stalled of the source `name`, a copy's what it copies, live at + /// `now`, once that counts; see [`source::Feed::stalled`]. + pub fn stalled(&self, name: &str, now: Instant) -> Option { + let capture = self.collection.capture(name); + #[cfg(test)] + if let Some(stalled) = self.stalled.get(capture) { + return Some(source::Stall::Picture(*stalled)); + } + self.sources.get(capture)?.stalled(now) + } + + /// Why `name`, a copy showing what it copies, shows no new picture at + /// `now`, once that counts: a picture stalled, else gone dark. + pub fn lost(&self, name: &str, now: Instant) -> Option { + if let Some(source::Stall::Picture(since)) = self.stalled(name, now) { + return Some(Lost::Stalled(since)); + } + self.dark(name, now).map(Lost::Dark) + } + + /// How long the camera, display, or window `name` shows, a copy showing + /// what it copies, has been all dark while live at `now`, once that is + /// long enough to count; see [`Compositor::dark`]. + pub fn dark(&self, name: &str, now: Instant) -> Option { + let capture = self.collection.capture(name); + #[cfg(test)] + if let Some(dark) = self.dark.get(capture) { + return Some(*dark); + } + self.compositor.dark(capture, now) + } + /// How the program's virtual camera, or the published scene `scene`'s, /// is doing, or `None` while it is off. pub fn camera(&self, scene: Option<&str>) -> Option { @@ -537,6 +596,14 @@ impl State { self.stale = true; } + /// Restart the display or window called `name`, asking the system picker + /// afresh rather than reopening what it was granted. + pub fn pick_again(&mut self, name: &str) -> anyhow::Result<()> { + self.sources.pick_again(name)?; + self.stale = true; + Ok(()) + } + /// After any change to the collection or its history, which the autosave /// writes once they settle. fn edited(&mut self) { @@ -1124,6 +1191,10 @@ impl Engine { stale: true, edits: autosave.edits(), renamed: Vec::new(), + #[cfg(test)] + dark: BTreeMap::new(), + #[cfg(test)] + stalled: BTreeMap::new(), }), autosave, stats, @@ -1471,7 +1542,7 @@ impl Engine { // Stalled sound matters only if an audio scene viewers hear plays it: // the one on program or one a published scene carries, not one only // listened to. A source that stopped matters either way. - let (stalled, stopped, uploaded) = { + let (stalled, dark, stopped, uploaded) = { let state = self.state.lock().unwrap(); let on_program = state.on_program(); let beside = state.beside(); @@ -1503,8 +1574,8 @@ impl Engine { let stalled = state .sources .iter() - .filter_map(|(name, feed)| { - let stall = feed.stalled(now)?; + .filter_map(|(name, _)| { + let stall = state.stalled(name, now)?; let live = match stall { source::Stall::Picture(_) => shown(name), source::Stall::Sound(_) => heard(name), @@ -1512,6 +1583,18 @@ impl Engine { live.any().then(|| (name.to_string(), stall, live)) }) .collect(); + let dark = state + .sources + .iter() + .filter_map(|(name, _)| { + // A stalled picture is reported as stalled, as its row says, never also as dark. + let Some(Lost::Dark(dark)) = state.lost(name, now) else { + return None; + }; + let live = shown(name); + live.any().then(|| (name.to_string(), dark, live)) + }) + .collect(); // A stinger's clip rests ended between stingers, unless a live scene // places it. let stinger = state.collection.transition.stinger(); @@ -1524,7 +1607,8 @@ impl Engine { shown: shown(name), heard: heard(name), }; - if !reach.any() { + // A dark or stalled source is reported so, as its layer's row says. + if !reach.any() || state.lost(name, now).is_some() { return None; } // A picture that fails to draw fails the source as the Sources panel shows it. @@ -1551,7 +1635,7 @@ impl Engine { moq_scene::browser::Paint::Gpu => None, }) .collect(); - (stalled, stopped, uploaded) + (stalled, dark, stopped, uploaded) }; let video = |video: &VideoStats, failed: Option| { let full = &video.renditions[Role::Canvas as usize]; @@ -1577,6 +1661,7 @@ impl Engine { broadcasts: std::iter::once(program).chain(published).collect(), grants: stats.grants(), stalled, + dark, stopped, uploaded, } @@ -3487,6 +3572,44 @@ mod tests { .expect("the studio closes"); } + /// A source on program whose picture went dark and then stalled is + /// reported once, as stalled, as its layer's row says; one dark only is + /// reported as dark. + #[tokio::test(flavor = "multi_thread")] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_lost_picture_is_reported_once() { + let dir = tempfile::tempdir().unwrap(); + let gpu = Gpu::new(wgpu::Backends::all()).await.expect("a GPU"); + let (autosave, _) = Autosave::open(dir.path().join("show.toml")).unwrap(); + let engine = start(gpu, Vec::new(), dir.path(), autosave, red_canvas()) + .await + .expect("the studio starts"); + // The program thread starts the sources on its first sync. + kio::clock::timeout(Duration::from_secs(10), async { + while engine.state.lock().unwrap().sources.get("red").is_none() { + kio::clock::sleep(Duration::from_millis(20)).await; + } + }) + .await + .expect("the source starts"); + let secs = Duration::from_secs; + engine.state.lock().unwrap().dark.insert("red".into(), secs(12)); + let local = engine.local(kio::clock::now()); + assert_eq!(local.dark.len(), 1, "{local:?}"); + assert!(local.stalled.is_empty(), "{local:?}"); + + engine.state.lock().unwrap().stalled.insert("red".into(), secs(4)); + let now = kio::clock::now(); + let local = engine.local(now); + assert!(local.dark.is_empty(), "{local:?}"); + assert_eq!(local.stalled.len(), 1, "{local:?}"); + assert_eq!( + engine.state.lock().unwrap().lost("red", now), + Some(Lost::Stalled(secs(4))) + ); + engine.stop(); + } + /// Without a relay, a recording encodes the program on its own and saves it /// as a regular MP4 holding its video and audio. The color source publishes /// a still, which has no file of its own. Recording a scene of another diff --git a/rs/moq-studio/src/health.rs b/rs/moq-studio/src/health.rs index 9370eede..a914e023 100644 --- a/rs/moq-studio/src/health.rs +++ b/rs/moq-studio/src/health.rs @@ -65,6 +65,12 @@ pub struct Report { pub sources: Vec, /// What is wrong, worst first, each naming its cause. pub problems: Vec, + /// How many cameras, displays, and windows a live broadcast shows went + /// dark, which the health dot counts. + pub dark: usize, + /// How many cameras and broadcasts a live broadcast shows stopped + /// sending pictures, which the health dot counts too. + pub stalled: usize, } /// One broadcast: who watches it, and how each of its tracks reaches @@ -218,6 +224,9 @@ pub struct Local { /// Sources whose stalled picture or sound a live broadcast carries, what /// stalled, and which broadcasts carry it. pub stalled: Vec<(String, Stall, Live)>, + /// Cameras, displays, and windows a live broadcast shows that have been + /// all dark this long, and which broadcasts show them. + pub dark: Vec<(String, Duration, Live)>, /// Sources a live broadcast shows or plays that ended or failed, and /// where. pub stopped: Vec<(String, Status, Reach)>, @@ -601,6 +610,12 @@ impl Sampler { viewers, tracks: &broadcast.tracks, }); + report.dark = local.dark.len(); + report.stalled = local + .stalled + .iter() + .filter(|(_, stall, _)| matches!(stall, Stall::Picture(_))) + .count(); report.problems = problems(&Facts { relays: &report.relays, local: &local, @@ -966,6 +981,16 @@ fn problems(facts: &Facts<'_>) -> Vec { format!("Source `{name}` stalled: {}, so {instead}.", without(*stall)), ); } + for (name, dark, live) in &local.dark { + add( + Verdict::Degraded, + format!( + "Source `{name}` has been all dark for {:.0} s, so {}.", + dark.as_secs_f64(), + clause(live, "shows", "show", "black where it was") + ), + ); + } for (name, status, reach) in &local.stopped { match status { Status::Ended => { @@ -1569,6 +1594,27 @@ mod tests { ); } + /// The dot counts the layers gone dark and those whose picture stalled, + /// but not a stalled sound, which no layer shows. + #[test] + fn counts_the_layers_without_a_picture() { + let program = Live { + program: true, + scenes: Vec::new(), + }; + let secs = Duration::from_secs; + let local = Local { + stalled: vec![ + ("camera".into(), Stall::Picture(secs(3)), program.clone()), + ("mic".into(), Stall::Sound(secs(2)), program.clone()), + ], + dark: vec![("game".into(), secs(12), program.clone())], + ..Default::default() + }; + let report = Sampler::new(None).sample(kio::clock::now(), local); + assert_eq!((report.dark, report.stalled), (1, 1)); + } + /// A scene published again under its name counts its frames afresh, /// rather than from the counters of the one before. #[test] @@ -1797,7 +1843,7 @@ mod tests { } /// Problems list worst first: an unreachable relay and a frozen or silent - /// source above a slow round trip. A frozen, silent, ended, or uploaded + /// source above a slow round trip. A frozen, silent, dark, ended, or uploaded /// source names the broadcasts that show or play it, including a published /// scene that alone plays it. #[test] @@ -1824,6 +1870,10 @@ mod tests { ("logo".into(), Stall::Picture(Duration::from_secs(3)), tall.clone()), ("mic".into(), Stall::Sound(Duration::from_secs(2)), tall.clone()), ], + dark: vec![ + ("game".into(), Duration::from_secs(12), program.clone()), + ("screen".into(), Duration::from_secs(3), both.clone()), + ], stopped: vec![ ("intro".into(), Status::Ended, reach(&program, &program)), ("outro".into(), Status::Ended, reach(&both, &both)), @@ -1858,6 +1908,8 @@ mod tests { "Source `logo` stalled: no new picture for 3 s, so scene `tall` repeats its last one.", "Source `mic` stalled: no new sound for 2 s, so scene `tall` plays silence for it.", "Source `music` failed: the file is gone.", + "Source `game` has been all dark for 12 s, so the program shows black where it was.", + "Source `screen` has been all dark for 3 s, so the program, scene `tall` and scene `wide` show black where it was.", "Source `intro` played to its end, so the program holds its last picture and plays no sound for it.", "Source `outro` played to its end, so the program, scene `tall` and scene `wide` hold its last picture and play no sound for it.", "Source `clip` played to its end, so the program holds its last picture and scene `tall` plays no sound for it.", diff --git a/rs/moq-studio/src/kit/widgets.rs b/rs/moq-studio/src/kit/widgets.rs index 8670d2f6..5320ed9f 100644 --- a/rs/moq-studio/src/kit/widgets.rs +++ b/rs/moq-studio/src/kit/widgets.rs @@ -1099,6 +1099,8 @@ pub struct Pad<'a> { refused: Option<&'a str>, /// What just failed on it, such as a rename. fault: Option<&'a str>, + /// Whether a capture it shows went dark. + dark: bool, width: f32, /// The thumbnail's shape, width over height: the show's. shape: f32, @@ -1115,6 +1117,7 @@ impl<'a> Pad<'a> { tag: None, refused: None, fault: None, + dark: false, width: 124.0, shape: 16.0 / 9.0, pose: None, @@ -1157,6 +1160,12 @@ impl<'a> Pad<'a> { self } + /// Badged in amber beside its name: a capture it shows went dark. + pub fn dark(mut self, dark: bool) -> Self { + self.dark = dark; + self + } + /// How wide it is, its picture following. pub fn width(mut self, width: f32) -> Self { self.width = width; @@ -1238,12 +1247,28 @@ impl Widget for Pad<'_> { painter.galley(cap.center() - text.size() / 2.0, text, Color32::from_white_alpha(0xbb)); } let ink = if self.refused.is_some() { DIM } else { TEXT }; - let galley = caps_galley(painter, self.name, super::display(14.0), ink, Some(rect.width() - 16.0)); + let badge = if self.dark { 20.0 } else { 0.0 }; + let galley = caps_galley( + painter, + self.name, + super::display(14.0), + ink, + Some(rect.width() - 16.0 - badge), + ); let at = pos2( rect.left() + 8.0, thumb.bottom() + (rect.bottom() - thumb.bottom() - galley.size().y) / 2.0, ); painter.galley(at, galley, TEXT); + if self.dark { + let center = pos2( + rect.right() - 16.0, + thumb.bottom() + (rect.bottom() - thumb.bottom()) / 2.0, + ); + let badge = Rect::from_center_size(center, vec2(16.0, 16.0)); + painter.circle_filled(center, 8.0, CUE); + Icon::Warning.paint(ui, badge, 10.0, ON_CUE); + } } match self.fault.or(self.refused) { Some(why) => tip(response, why, None), diff --git a/rs/moq-studio/src/shots.rs b/rs/moq-studio/src/shots.rs index 32a11e94..f70e5e51 100644 --- a/rs/moq-studio/src/shots.rs +++ b/rs/moq-studio/src/shots.rs @@ -110,6 +110,12 @@ enum View { Cued(Page), /// A scene, by name, open for editing off air on BUILD. OffAir(&'static str), + /// BUILD with the game's window gone dark selected, and what to do about + /// it open from its badge. + Dark, + /// BUILD with the host's camera stalled selected, and what to do about it + /// open from its badge. + Stalled, /// BUILD with no layer selected, after the guest `Ibis` appears on their /// own in the scene on program. Joined, @@ -118,6 +124,8 @@ enum View { /// with `Interview` cued. Multiview { live: bool, + /// Whether the host's picture stalled. + stalled: bool, }, /// BUILD's Add tab on the drop views' show, with a file dragged or /// dropped on the picture. @@ -280,6 +288,13 @@ views! { build_off_air_refused: Still, studio(View::OffAir("Vertical")); /// Too narrow for the banner beside the live picture, it goes under it. build_off_air_compact: Still, Open::Studio { view: View::OffAir("Break"), size: HALF, show: "show.toml", relay: false }; + /// A window gone dark: its row, the layer on the picture, its scene's + /// pad, and the health dot turn amber, and its badge opens what happened + /// and the fixes. + build_dark: Live, Open::Studio { view: View::Dark, size: SIZE, show: "dark.toml", relay: false }; + /// A camera that stopped sending pictures: the same amber badge, saying + /// its picture froze, and offering to open it again. + build_stalled: Live, Open::Studio { view: View::Stalled, size: SIZE, show: "stalled.toml", relay: false }; /// A logo dragged over the picture: where it lands, outlined and named /// for the kind of layer it becomes, and Add's row for that kind lit. build_drop: Still, dropping(Drop::Hover); @@ -371,11 +386,14 @@ views! { /// The multiview off air: every scene and the sources picked, the scene /// the stream opens on amber, and the clock, which keeps it from being /// still. - multiview: Live, studio(View::Multiview { live: false }); + multiview: Live, studio(View::Multiview { live: false, stalled: false }); + /// The multiview with the host's camera stalled: its tile and each scene + /// tile showing it carry the amber badge, and its picture says so. + multiview_stalled: Live, studio(View::Multiview { live: false, stalled: true }); /// The multiview on air in Studio mode: the scene on air framed in the /// accent, the cued one amber, and the sources the live picture shows /// dotted. - multiview_on_air: Live, relayed(View::Multiview { live: true }); + multiview_on_air: Live, relayed(View::Multiview { live: true, stalled: false }); } /// Draw the view `name` opens into `$SHOTS_DIR/.png`, twice if it is @@ -1437,8 +1455,11 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show studio.turn(*page); } View::OffAir(scene) => studio.edit_off_air(scene), + // The game's window, the back one of three over the black. + View::Dark => studio.build(Side::Layers, Inspect::Layer, Some(2)), + View::Stalled => studio.build(Side::Layers, Inspect::Layer, Some(1)), View::Joined => studio.build(Side::Layers, Inspect::Layer, None), - View::Multiview { live } => { + View::Multiview { live, .. } => { studio.multiview_alone(&["host", "highlight"]); if *live { studio.studio_mode(true); @@ -1457,6 +1478,14 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show } } let engine = started.get().expect("the studio started").clone(); + if let View::Dark = view { + let dark = Duration::from_secs(12); + engine.state.lock().unwrap().dark.insert("game".into(), dark); + } + if let View::Stalled | View::Multiview { stalled: true, .. } = view { + let stalled = Duration::from_secs(4); + engine.state.lock().unwrap().stalled.insert("host".into(), stalled); + } if let View::Joined = view { let mut state = engine.state.lock().unwrap(); let connect = url::Url::parse("https://relay.example/show").unwrap(); @@ -1472,7 +1501,7 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show | View::OnAirSetting(_) | View::Restart | View::ShowMenuLive - | View::Multiview { live: true } = view + | View::Multiview { live: true, .. } = view { controls.go_live().expect("going live starts"); runtime @@ -1523,8 +1552,34 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show harness.get_by_label("Audio scene Headphones").click_secondary(); harness.run_steps(4); } + View::Dark => { + // The health dot counts it once the stream is graded again. + runtime + .block_on(kio::clock::timeout(SETTLE, async { + while engine.health.lock().unwrap().dark == 0 { + kio::clock::sleep(Duration::from_millis(20)).await; + } + })) + .expect("the health dot counts the dark window"); + harness.run_steps(2); + harness.get_by_label("What happened to game").click(); + harness.run_steps(4); + } + View::Stalled => { + runtime + .block_on(kio::clock::timeout(SETTLE, async { + while engine.health.lock().unwrap().stalled == 0 { + kio::clock::sleep(Duration::from_millis(20)).await; + } + })) + .expect("the health dot counts the stalled camera"); + harness.run_steps(2); + harness.get_by_label("What happened to host").click(); + harness.run_steps(4); + } View::Menu => { // The row comes before the inspector's tab of the same name. + harness.get_all_by_label("host").next().unwrap().click_secondary(); harness.run_steps(4); } diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index bc38d72c..98a23733 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -19,7 +19,7 @@ use winit::keyboard::{KeyCode, PhysicalKey}; use crate::controls::{Action, Controls}; use crate::devices::{self, Listing, Pick}; -use crate::engine::{Engine, State}; +use crate::engine::{Engine, Lost, State}; use crate::fx::{Fx, Mark}; use crate::health::{self, Verdict}; use crate::hotkeys::{self, Bindings, Hotkeys, Reach}; @@ -31,6 +31,7 @@ use crate::{mix, record}; mod build; mod cue; +mod dark; mod drop; mod graph; mod inspector; @@ -831,14 +832,24 @@ impl Selection { blend, } = look; match self { - Self::Source(name) => draft.set_video( - name, - Some(Video { - background: background.unwrap_or_default(), - crop, - filters, - }), - ), + Self::Source(name) => { + // Not part of the look, so it stays as it was. + let expect_dark = draft + .collection() + .video + .get(name) + .is_some_and(|video| video.expect_dark); + draft.set_video( + name, + Some(Video { + background: background.unwrap_or_default(), + crop, + filters, + expect_dark, + }), + ) + } + Self::Item(layer, index) => { let item = draft .collection() diff --git a/rs/moq-studio/src/ui/build.rs b/rs/moq-studio/src/ui/build.rs index 94eafb8b..7724d50c 100644 --- a/rs/moq-studio/src/ui/build.rs +++ b/rs/moq-studio/src/ui/build.rs @@ -3,6 +3,7 @@ //! inspector on the right. use super::cue::Corner; +use super::dark::Mend; use super::shell::Snapshot; use super::*; @@ -147,6 +148,20 @@ impl Panels { // Docked on the live picture, away from the selected layer and its // toolbar, which a drag moves with it. let (width, height) = (canvas.width.max(1) as f32, canvas.height.max(1) as f32); + let now = kio::clock::now(); + let scale = response.rect.size() / egui::vec2(width, height); + for (_, item) in self.shown(&state) { + if item.visible + && let Some(lost) = state.lost(&item.source, now) + { + let at = egui::Rect::from_min_size( + response.rect.min + egui::vec2(item.rect.x, item.rect.y) * scale, + egui::vec2(item.rect.width, item.rect.height) * scale, + ); + super::dark::on_picture(ui, at.intersect(response.rect), &item.source, lost); + } + } + let selected = self.shown(&state).into_iter().find_map(|(picked, item)| { (Some(&picked) == self.selection.as_ref()).then(|| { [ @@ -786,6 +801,22 @@ impl Panels { result } LayerAction::Independent => make_independent(state, layer, index, &item), + LayerAction::Mend(mend) => { + let capture = collection.capture(&name).to_string(); + match mend { + Mend::PickAgain => state + .pick_again(&capture) + .with_context(|| format!("cannot pick `{capture}` again")), + Mend::Reopen => { + state.restart(&capture); + Ok(()) + } + Mend::Replace(source) => edit(state, format!("Change what `{capture}` captures"), None, |draft| { + draft.set_source(&capture, Some(source)) + }), + Mend::Expected => super::dark::expect(state, &capture, true), + } + } // The caller holds the state, so this copies from it rather than // through `copy`, which takes the lock. LayerAction::Copy => clip(ctx, state, &item).map(|said| self.say(Ok(said))), @@ -844,6 +875,8 @@ pub(super) enum LayerAction { }, /// Show a copy of its own of what it shows, so its look is its own. Independent, + /// Fix what it shows, which went dark or stalled. + Mend(Mend), Copy, Remove, } diff --git a/rs/moq-studio/src/ui/dark.rs b/rs/moq-studio/src/ui/dark.rs new file mode 100644 index 00000000..bb019d0a --- /dev/null +++ b/rs/moq-studio/src/ui/dark.rs @@ -0,0 +1,446 @@ +//! A camera, display, or window gone dark, or a camera or broadcast that +//! stopped sending pictures: the amber badge on its layer's row, over it on +//! the picture, and on its scenes' pads, and the popover that says in plain +//! words what happened and offers the fix. + +use egui::{Rect, Sense, Stroke, Vec2, pos2, vec2}; + +use super::*; + +/// What fixes a capture gone dark, as its popover offers. +#[derive(Clone, Debug, PartialEq)] +pub(super) enum Mend { + /// Ask the system picker afresh, forgetting the grant it holds. + PickAgain, + /// Open the device, or subscribe to the broadcast, again. + Reopen, + /// Capture this in its place, under the same name. + Replace(Source), + /// It is meant to go dark: stop watching it. + Expected, +} + +/// What a lost picture's popover says and offers, by its kind. +struct Advice { + title: &'static str, + /// What may have happened, and what viewers see. + cause: String, + /// What the streamer must do where the studio cannot, and where. + hint: Option, + /// Each fix the studio can take itself, the first the likeliest. + fixes: Vec<(&'static str, Mend)>, + /// Whether to offer the windows this machine lists, to capture one in + /// its place: a window chosen by id, which the system cannot ask again. + windows: bool, + /// Whether to offer to stop watching it: a capture meant to go dark. + expected: bool, +} + +/// What to say of `source`, called `name`, that `lost` its picture; `None` +/// for a kind never watched for it. +fn advice(name: &str, source: &Source, lost: Lost) -> Option { + match lost { + Lost::Dark(_) => dark(name, source), + Lost::Stalled(_) => stalled(name, source), + } +} + +/// What to say of `source`, called `name`, that stopped sending pictures. +fn stalled(name: &str, source: &Source) -> Option { + let again = ("Subscribe again", Mend::Reopen); + let advice = match source { + Source::Camera { .. } => Advice { + title: "This camera stopped sending pictures.", + cause: format!( + "`{name}` may have been unplugged, or another app may have taken it. Viewers see its last picture, frozen." + ), + hint: Some("If it stays frozen, check its cable, and quit any other app using the camera.".into()), + fixes: vec![("Open it again", Mend::Reopen)], + windows: false, + expected: false, + }, + Source::Broadcast { path } => Advice { + title: "This broadcast stopped sending pictures.", + cause: format!( + "`{name}`'s sender may have stopped, or lost its connection to the relay. Viewers see its last picture, frozen." + ), + hint: Some(format!( + "If it stays frozen, whoever sends `{path}` should check that it is still live and its connection keeps up." + )), + fixes: vec![again], + windows: false, + expected: false, + }, + Source::Layer { broadcast, .. } => Advice { + title: "This layer stopped sending pictures.", + cause: format!( + "`{name}` comes from the broadcast `{broadcast}`, whose sender may have stopped or lost its connection. Viewers see its last picture, frozen." + ), + hint: None, + fixes: vec![again], + windows: false, + expected: false, + }, + _ => return None, + }; + Some(advice) +} + +/// What to say of `source`, called `name`, gone dark. +fn dark(name: &str, source: &Source) -> Option { + let whole = ("Capture the whole screen instead", Mend::Replace(whole_screen())); + let advice = match source { + Source::PickedWindow | Source::Window { .. } => Advice { + title: "This window went black.", + cause: format!( + "`{name}` may have closed, been minimized, or changed how it draws. Viewers see black where it was." + ), + hint: Some(match () { + _ if cfg!(target_os = "macos") => SCREEN_PERMISSION.into(), + _ => "A game in exclusive full screen refuses capture: switch it to borderless or windowed in its display settings.".into(), + }), + fixes: match source { + Source::PickedWindow => vec![("Pick it again", Mend::PickAgain), whole], + _ => vec![whole], + }, + windows: !matches!(source, Source::PickedWindow), + expected: true, + }, + Source::PickedDisplay | Source::Display { .. } => Advice { + title: "This screen went black.", + cause: format!( + "`{name}` may be locked or asleep, or showing something that refuses capture, such as protected video. Viewers see black where it was." + ), + hint: cfg!(target_os = "macos").then(|| SCREEN_PERMISSION.into()), + fixes: match source { + Source::PickedDisplay => vec![("Pick it again", Mend::PickAgain)], + _ => vec![("Open it again", Mend::Reopen)], + }, + windows: false, + expected: true, + }, + Source::Camera { .. } => Advice { + title: "This camera went black.", + cause: format!( + "`{name}` may have its lens covered or its privacy shutter closed, or another app may have taken it. Viewers see black where it was." + ), + hint: match () { + _ if cfg!(target_os = "macos") => Some( + "If it stays black, macOS may have taken away the camera: allow kixel.live Studio in System Settings > Privacy & Security > Camera.", + ), + _ if cfg!(windows) => Some( + "If it stays black, Windows may have taken away the camera: allow desktop apps in Settings > Privacy & security > Camera.", + ), + _ => None, + } + .map(str::to_string), + fixes: vec![("Open it again", Mend::Reopen)], + windows: false, + expected: true, + }, + _ => return None, + }; + Some(advice) +} + +/// What macOS asks of a display or window capture that it stopped allowing. +const SCREEN_PERMISSION: &str = "If it stays black, macOS may have taken away screen recording: allow kixel.live Studio in System Settings > Privacy & Security > Screen & System Audio Recording."; + +/// What captures the whole screen here: the system picker on a Wayland +/// desktop, which captures no screen without it, else the main display. +fn whole_screen() -> Source { + if cfg!(target_os = "linux") && std::env::var_os("WAYLAND_DISPLAY").is_some() { + return Source::PickedDisplay; + } + Source::Display { id: None } +} + +/// Say whether the capture called `capture` is `expected` to go dark, so it +/// is never flagged, or is watched again. +pub(super) fn expect(state: &mut State, capture: &str, expected: bool) -> anyhow::Result<()> { + let video = state.collection().video.get(capture).cloned().unwrap_or_default(); + let video = Video { + expect_dark: expected, + ..video + }; + let label = match expected { + true => format!("Expect `{capture}` to go dark"), + false => format!("Warn when `{capture}` goes dark"), + }; + edit(state, label, None, |draft| draft.set_video(capture, Some(video))) +} + +/// "No picture for 12 s" for a capture gone dark, "No new picture for 3 s" +/// for one stalled: what its row says under its name. +pub(super) fn words(lost: Lost) -> String { + let what = match lost { + Lost::Dark(_) => "No picture", + Lost::Stalled(_) => "No new picture", + }; + format!("{what} for {:.0} s", lost.since().as_secs_f64()) +} + +/// What the health dot says of `dark` layers gone dark and `stalled` ones +/// stalled, such as "1 layer dark"; `None` for none. +pub(super) fn counted(dark: usize, stalled: usize) -> Option { + let layers = |count: usize| match count { + 1 => "1 layer".to_string(), + count => format!("{count} layers"), + }; + match (dark, stalled) { + (0, 0) => None, + (dark, 0) => Some(format!("{} dark", layers(dark))), + (0, stalled) => Some(format!("{} stalled", layers(stalled))), + (dark, stalled) => Some(format!("{} dark or stalled", layers(dark + stalled))), + } +} + +/// Whether the scene called `scene` draws a layer that `lost` its picture, +/// as `lost` says of a source: through the scenes it nests, and not one it +/// hides. +pub(super) fn draws_lost(collection: &Collection, scene: &str, lost: impl Fn(&str) -> bool) -> bool { + collection + .drawn([Layer::scene(scene)]) + .iter() + .any(|item| lost(&item.source)) +} + +/// The amber badge in `rect`: a warning sign on amber. +pub(super) fn badge(ui: &Ui, rect: Rect) { + let painter = ui.painter(); + painter.circle_filled(rect.center(), rect.width() / 2.0, kit::CUE); + Icon::Warning.paint(ui, rect, rect.width() * 0.6, kit::ON_CUE); +} + +/// The badge for the source `source`, called `name`, that `lost` its +/// picture, in `rect`: a click opens its popover. Whether it was clicked +/// open, and the fix asked for, if one was; with `windows`, the windows to +/// offer in its place, if listed. +pub(super) fn badge_button( + ui: &mut Ui, + rect: Rect, + id: egui::Id, + name: &str, + source: &Source, + lost: Lost, + windows: Option<&Result, String>>, +) -> (bool, Option) { + let response = ui.interact(rect, id, Sense::click()); + let label = format!("What happened to {name}"); + response.widget_info(|| egui::WidgetInfo::labeled(egui::WidgetType::Button, true, &label)); + badge(ui, rect); + let opened = response.clicked(); + let Some(advice) = advice(name, source, lost) else { + return (opened, None); + }; + let mut mend = None; + kit::popover(&response).width(300.0).show(|ui| { + mend = popover(ui, &advice, windows); + }); + (opened, mend) +} + +/// The popover's body: what happened, the fixes, and where the streamer acts +/// when the studio cannot. +fn popover(ui: &mut Ui, advice: &Advice, windows: Option<&Result, String>>) -> Option { + let mut mend = None; + ui.add_space(6.0); + ui.horizontal_top(|ui| { + ui.add(Icon::Warning.image(18.0, kit::CUE)); + ui.vertical(|ui| { + ui.add( + egui::Label::new( + egui::RichText::new(advice.title) + .font(kit::ui_strong(14.0)) + .color(kit::TEXT), + ) + .wrap(), + ); + ui.add(egui::Label::new(egui::RichText::new(&advice.cause).color(kit::DIM)).wrap()); + }); + }); + ui.add_space(8.0); + let wide = vec2(ui.available_width(), 30.0); + if advice.windows { + ui.menu_button("Pick another window", |ui| match windows { + Some(Ok(windows)) if !windows.is_empty() => { + for window in windows { + if ui.button(&window.name).clicked() { + mend = Some(Mend::Replace(window.source.clone())); + ui.close(); + } + } + } + Some(Ok(_)) => { + ui.label("No windows to capture"); + } + Some(Err(err)) => { + ui.colored_label(kit::ERR, err); + } + None => { + ui.spinner(); + } + }); + } + for (index, (label, fix)) in advice.fixes.iter().enumerate() { + // The likeliest fix stands out, as a light button. + let button = match index { + 0 if !advice.windows => egui::Button::new(egui::RichText::new(*label).color(kit::BG)).fill(kit::TEXT), + _ => egui::Button::new(*label), + }; + if ui.add(button.min_size(wide)).clicked() { + mend = Some(fix.clone()); + } + } + if let Some(hint) = &advice.hint { + ui.add_space(4.0); + ui.add(egui::Label::new(egui::RichText::new(hint).font(kit::ui(11.0)).color(kit::DIM)).wrap()); + } + if advice.expected { + ui.add_space(4.0); + let expected = ui + .add(egui::Button::new(egui::RichText::new("It is meant to go dark").color(kit::DIM)).frame(false)) + .on_hover_text("Stop watching it for going dark; its inspector watches it again"); + if expected.clicked() { + mend = Some(Mend::Expected); + } + } + mend +} + +/// Mark the layer called `name` at `at`, which `lost` its picture, on the +/// picture: an amber dashed outline, and its name on an amber pill in its +/// middle. +pub(super) fn on_picture(ui: &Ui, at: Rect, name: &str, lost: Lost) { + // Nothing spills onto what is beside it, such as a narrow multiview tile's + // neighbors. + let painter = ui.painter().with_clip_rect(at.intersect(ui.clip_rect())); + super::dashed(&painter, at.shrink(1.5), Stroke::new(2.0, kit::CUE)); + + let what = match lost { + Lost::Dark(_) => "no picture", + Lost::Stalled(_) => "no new picture", + }; + // Its name goes first where the pill would not fit, then the pill. + let pill = |text: String| { + let galley = painter.layout_no_wrap(text.to_uppercase(), kit::display(13.0), kit::ON_CUE); + let size = galley.size() + vec2(44.0, 14.0); + (size.x <= at.width() - 8.0 && size.y <= at.height() - 8.0).then_some((galley, size)) + }; + let Some((galley, size)) = pill(format!("{name} · {what}")).or_else(|| pill(what.to_string())) else { + return; + }; + let pill = Rect::from_center_size(at.center(), size); + painter.rect_filled(pill, size.y / 2.0, kit::CUE); + let icon = Rect::from_center_size(pos2(pill.left() + 18.0, pill.center().y), Vec2::splat(18.0)); + painter.circle_filled(icon.center(), 9.0, kit::ON_CUE); + Icon::Warning.paint(ui, icon, 11.0, kit::CUE); + painter.galley( + pos2(icon.right() + 6.0, pill.center().y - galley.size().y / 2.0), + galley, + kit::ON_CUE, + ); +} + +#[cfg(test)] +mod tests { + use super::*; + + const DARK: Lost = Lost::Dark(Duration::from_secs(12)); + const STALLED: Lost = Lost::Stalled(Duration::from_secs(3)); + + /// Every watched kind says what happened and offers a fix, and a window + /// or display the picker granted offers to pick it again first. + #[test] + fn every_watched_kind_has_advice() { + let kinds = [ + Source::Camera { id: None }, + Source::Display { id: None }, + Source::Window { id: "1".into() }, + Source::PickedDisplay, + Source::PickedWindow, + ]; + for source in kinds { + let advice = advice("game", &source, DARK).expect("advice"); + assert!(advice.cause.contains("`game`")); + assert!(!advice.fixes.is_empty() || advice.windows, "{source:?}"); + assert!(advice.expected, "{source:?}"); + } + let picked = advice("game", &Source::PickedWindow, DARK).unwrap(); + assert_eq!(picked.fixes[0].1, Mend::PickAgain); + let logo = Source::Image { + path: "logo.png".into(), + }; + assert!(advice("logo", &logo, DARK).is_none()); + assert!(advice("logo", &logo, STALLED).is_none()); + } + + /// Every kind whose picture can stall says so and offers to open or + /// subscribe again, and never to stop watching it. + #[test] + fn every_stalling_kind_has_advice() { + let kinds = [ + Source::Camera { id: None }, + Source::Broadcast { path: "guest".into() }, + Source::Layer { + broadcast: "remote".into(), + source: "cam".into(), + }, + ]; + for source in kinds { + let advice = advice("cam", &source, STALLED).expect("advice"); + assert!(advice.cause.contains("`cam`"), "{source:?}"); + assert!(advice.cause.contains("frozen"), "{source:?}"); + assert_eq!(advice.fixes[0].1, Mend::Reopen, "{source:?}"); + assert!(!advice.expected, "{source:?}"); + } + let broadcast = advice("cam", &Source::Broadcast { path: "guest".into() }, STALLED).unwrap(); + assert!(broadcast.hint.unwrap().contains("`guest`")); + // A window that stands still is not stalled. + assert!(advice("game", &Source::PickedWindow, STALLED).is_none()); + } + + #[test] + fn rows_say_how_long() { + assert_eq!(words(DARK), "No picture for 12 s"); + assert_eq!(words(STALLED), "No new picture for 3 s"); + } + + /// A scene pad is marked for a lost picture it draws, nested in a scene + /// it shows, but not for one it hides. + #[test] + fn a_pad_marks_what_it_draws() { + let fill = moq_scene::Rect::fill(&Canvas::default()); + let mut collection = Collection::new(); + collection.sources.insert("cam".into(), Source::Camera { id: None }); + collection + .sources + .insert("inner".into(), Source::Scene { scene: "Inner".into() }); + let mut inner = Scene::new("Inner"); + inner.items.push(Item::new("cam", fill)); + collection.scenes.push(inner); + let mut hidden = Scene::new("Hidden"); + hidden.items.push(Item { + visible: false, + ..Item::new("cam", fill) + }); + collection.scenes.push(hidden); + collection.scenes[0].items.push(Item::new("inner", fill)); + let outer = collection.scenes[0].name.clone(); + collection.validate().unwrap(); + let lost = |name: &str| name == "cam"; + assert!(draws_lost(&collection, "Inner", lost)); + assert!(draws_lost(&collection, &outer, lost), "through the scene it nests"); + assert!(!draws_lost(&collection, "Hidden", lost), "not what it hides"); + } + + /// The health dot names what the layers lost, and counts them. + #[test] + fn the_dot_counts_layers_without_a_picture() { + assert_eq!(counted(0, 0), None); + assert_eq!(counted(1, 0).as_deref(), Some("1 layer dark")); + assert_eq!(counted(2, 0).as_deref(), Some("2 layers dark")); + assert_eq!(counted(0, 1).as_deref(), Some("1 layer stalled")); + assert_eq!(counted(1, 1).as_deref(), Some("2 layers dark or stalled")); + } +} diff --git a/rs/moq-studio/src/ui/inspector.rs b/rs/moq-studio/src/ui/inspector.rs index e6e78cff..bead4aea 100644 --- a/rs/moq-studio/src/ui/inspector.rs +++ b/rs/moq-studio/src/ui/inspector.rs @@ -26,6 +26,8 @@ enum Asked { Reload(String), /// Publish the source independently, or stop. Publish(String, bool), + /// Watch the capture for going dark, or stop. + Warn(String, bool), /// Fill the role called this with this source, or leave it unfilled. Fill(String, Option), } @@ -452,6 +454,21 @@ impl Panels { asked = Some(Asked::Fill(name.clone(), fill)); } } + let capture = collection.capture(&name); + if collection.sources.get(capture).is_some_and(Source::may_go_dark) { + let mut warn = !collection.video.get(capture).is_some_and(|video| video.expect_dark); + if ui + .add(kit::Toggle::new(&mut warn).label("Warn when it goes dark")) + .on_hover_text( + "Badge it in amber when its picture stays all dark, holding its last picture over \ + a moment of dark. Turn off for a capture meant to go dark, such as a camera whose \ + shutter closes between segments.", + ) + .changed() + { + asked = Some(Asked::Warn(capture.to_string(), warn)); + } + } if source.publishes() { let mut published = collection.published.contains(&name); if ui @@ -510,6 +527,7 @@ impl Panels { state.restart(&name); Ok(()) } + Asked::Warn(capture, warn) => super::dark::expect(&mut state, &capture, !warn), Asked::Fill(role, source) => { let label = match &source { Some(source) => format!("Fill the role `{role}` with `{source}`"), diff --git a/rs/moq-studio/src/ui/layers.rs b/rs/moq-studio/src/ui/layers.rs index bb8b6c73..2cdf91e3 100644 --- a/rs/moq-studio/src/ui/layers.rs +++ b/rs/moq-studio/src/ui/layers.rs @@ -117,6 +117,7 @@ impl Panels { let now = kio::clock::now(); let mut action = None; let mut rows = Vec::new(); + self.devices.poll(); for index in (0..items.len()).rev() { let item = &items[index]; let picked = Picked { @@ -124,6 +125,14 @@ impl Panels { index, }; let selected = self.selection.as_ref() == Some(&picked); + let status = status_of(state, &item.source, now); + let capture = collection.sources.get(collection.capture(&item.source)); + // A window chosen by id offers the windows listed now in its place. + let window = matches!(status.lost, Some(Lost::Dark(_))) && matches!(capture, Some(Source::Window { .. })); + if window && self.devices.listing.is_none() && self.devices.pending.is_none() { + let refreshed = self.devices.refresh(&self.runtime).context("cannot list windows"); + self.fail(Spot::Add, refreshed); + } let fault = self.fault(&Spot::Layer(layer.clone(), item.source.clone())); let what = words(collection, &item.source); let what = match item.slot { @@ -135,11 +144,19 @@ impl Panels { item, kind: kind(collection, &item.source), words: what, - status: status_of(state, &item.source, now), + status, + capture, + windows: self.devices.listing.as_ref().map(|listing| &listing.windows), fault, selected, }; - let (response, asked) = row.show(ui, ui.id().with(("layer", layer.scene.as_deref(), index))); + let (response, asked, opened) = row.show(ui, ui.id().with(("layer", layer.scene.as_deref(), index))); + // Its popover, opened, lists the windows afresh: the listing kept + // may be from before the window went, or another opened. + if opened && window && self.devices.pending.is_none() { + let refreshed = self.devices.refresh(&self.runtime).context("cannot list windows"); + self.fail(Spot::Add, refreshed); + } if let Some(asked) = asked { action = Some((picked.clone(), asked)); } @@ -220,13 +237,18 @@ struct Row<'a> { kind: Icon, words: String, status: Health, + /// The source it captures, a copy's what it copies. + capture: Option<&'a Source>, + /// The windows this machine lists, once listed. + windows: Option<&'a Result, String>>, fault: Option<&'a str>, selected: bool, } impl Row<'_> { - /// Draw the row with `id`; its response and what its lock or eye asked. - fn show(self, ui: &mut Ui, id: egui::Id) -> (egui::Response, Option) { + /// Draw the row with `id`; its response, what its lock, eye, or badge + /// asked, and whether its badge was clicked open. + fn show(self, ui: &mut Ui, id: egui::Id) -> (egui::Response, Option, bool) { let size = vec2(ui.available_width(), ROW); let (rect, _) = ui.allocate_exact_size(size, Sense::hover()); let response = ui.interact(rect, id, Sense::click_and_drag()); @@ -265,9 +287,11 @@ impl Row<'_> { egui::TextStyle::Body, ); painter.galley(pos2(left, rect.top() + 8.0), galley, ink); - let (below, color) = match self.fault { - Some(fault) => (fault.to_string(), kit::ERR), - None => (self.words, kit::DIM), + let (below, color) = match (self.fault, self.status.lost) { + (Some(fault), _) => (fault.to_string(), kit::ERR), + // Its icon says what kind it is. + (None, Some(lost)) => (super::dark::words(lost), kit::CUE), + (None, None) => (self.words, kit::DIM), }; let small = egui::RichText::new(below).font(kit::ui(11.0)).color(color); let galley = egui::WidgetText::from(small).into_galley( @@ -279,10 +303,30 @@ impl Row<'_> { painter.galley(pos2(left, rect.top() + 26.0), galley, color); let mut asked = None; - // A source live and well needs no mark; the inspector says so. + let mut opened = false; + // A source live and well needs no mark; the inspector says so. One + // gone dark or stalled has a badge in its place, which opens what to do. let marked = self.status.color != kit::OK || self.fault.is_some(); + let lost = self.status.lost.filter(|_| self.fault.is_none()); + let dark = lost.is_some(); + let badge = Rect::from_center_size(dot.center() - vec2(4.0, 0.0), Vec2::splat(20.0)); + if let Some(lost) = lost + && let Some(capture) = self.capture + { + let mend; + (opened, mend) = super::dark::badge_button( + ui, + badge, + id.with("dark"), + &self.item.source, + capture, + lost, + self.windows, + ); + asked = mend.map(LayerAction::Mend); + } let status = ui.interact(dot, id.with("status"), Sense::hover()); - if marked { + if marked && !dark { let color = if self.fault.is_some() { kit::ERR } else { @@ -290,7 +334,17 @@ impl Row<'_> { }; painter.circle_filled(dot.center(), 3.5, color); } - if marked && (status.hovered() || self.fault.is_some()) { + // A failure besides, such as its track's, keeps its red dot beside + // the badge, with the words on hover. + if dark && self.status.color == kit::ERR { + let beside = Rect::from_center_size(pos2(badge.left() - 8.0, dot.center().y), Vec2::splat(12.0)); + painter.circle_filled(beside.center(), 3.5, kit::ERR); + let failed = ui.interact(beside, id.with("failed"), Sense::hover()); + if failed.hovered() { + failed.on_hover_text(self.status.lines.join("\n")); + } + } + if marked && !dark && (status.hovered() || self.fault.is_some()) { let mut lines = self.status.lines.clone(); if let Some(fault) = self.fault { lines.insert(0, fault.to_string()); @@ -318,7 +372,7 @@ impl Row<'_> { if ui.put(eye, IconButton::new(icon, name).small()).clicked() { asked = Some(LayerAction::Show(!self.item.visible)); } - (response, asked) + (response, asked, opened) } } @@ -358,6 +412,8 @@ impl egui::Widget for AddLayer { pub(super) struct Health { pub(super) color: Color32, pub(super) lines: Vec, + /// Why it shows no new picture, once that counts: gone dark or stalled. + pub(super) lost: Option, } /// How `name` is doing: live, starting, stalled, ended, or failed with its @@ -368,13 +424,12 @@ pub(super) fn status_of(state: &State, name: &str, now: kio::clock::Instant) -> // A clone is as live as what it clones. let capture = collection.capture(name); let feed = state.sources.get(capture); - let (mut color, first) = match ( - state.compositor.error(capture), - feed, - feed.and_then(|feed| feed.stalled(now)), - ) { + let dark = state.dark(name, now); + let lost = state.lost(name, now); + let (mut color, first) = match (state.compositor.error(capture), feed, state.stalled(name, now)) { (Some(err), _, _) => (kit::ERR, format!("Failed: {err}")), (None, _, Some(stall)) => (kit::CUE, format!("Stalled: {}", health::without(stall))), + (None, _, None) if let Some(dark) = dark => (kit::CUE, format!("All dark for {:.0} s", dark.as_secs_f64())), (None, Some(feed), None) => match feed.status() { Status::Starting => (kit::DIM, "Starting".to_string()), Status::Live => (kit::OK, "Live".to_string()), @@ -413,7 +468,7 @@ pub(super) fn status_of(state: &State, name: &str, now: kio::clock::Instant) -> None => {} }, } - Health { color, lines } + Health { color, lines, lost } } /// The icon for what `name` shows, by its source's kind, or the kind of what diff --git a/rs/moq-studio/src/ui/live.rs b/rs/moq-studio/src/ui/live.rs index 7898416d..887361e7 100644 --- a/rs/moq-studio/src/ui/live.rs +++ b/rs/moq-studio/src/ui/live.rs @@ -735,10 +735,12 @@ impl Panels { let on_air = self.engine.on_air(); let watched = self.engine.watched(); let (name, dot) = health_word(&self.engine); + let name = name.as_str(); let report = self.engine.health.lock().unwrap(); let mark = fx.on().then(|| crate::fx::Mark::new(ui.painter())); let text = if compact { "" } else { name }; let pill = ui.add(kit::Pill::new(text).dot(dot).clickable()); + let pill = kit::tip(pill, &format!("Health: {name}"), None); if let Some(mark) = mark { fx.health(&mark.around(pill.rect).inset(2.0)); @@ -1353,9 +1355,10 @@ impl Form { } } -/// The stream's health in a word, and its dot's color. A problem shows even -/// off air, such as a failed source. -pub(super) fn health_word(engine: &Engine) -> (&'static str, Color32) { +/// The stream's health in a word, and its dot's color, or how many layers +/// went dark or stalled when they are among the worst problems. A problem +/// shows even off air, such as a failed source. +pub(super) fn health_word(engine: &Engine) -> (String, Color32) { let live = engine.on_air().live().is_some(); let unwatched = engine.watched().is_empty(); let report = engine.health.lock().unwrap(); @@ -1363,12 +1366,24 @@ pub(super) fn health_word(engine: &Engine) -> (&'static str, Color32) { .problems .first() .map_or(Verdict::Healthy, |problem| problem.verdict); - match (worst, live, unwatched) { + let (word, dot) = match (worst, live, unwatched) { (Verdict::Unhealthy, ..) => ("Unhealthy", kit::ERR), (Verdict::Degraded, ..) => ("Degraded", kit::CUE), (_, false, true) => ("No relay", kit::DIM), (_, false, false) => ("Ready", kit::OK), (Verdict::Healthy, true, _) => ("Healthy", kit::OK), (Verdict::Unknown, true, _) => ("Starting", kit::DIM), - } + }; + // When layers gone dark or stalled are among the worst problems, the + // word counts them, as the streamer can see and fix them on BUILD. A + // stalled picture is unhealthy, a dark one degraded. + let among = match worst { + Verdict::Unhealthy => report.stalled > 0, + Verdict::Degraded => report.dark > 0, + _ => false, + }; + let counted = among + .then(|| super::dark::counted(report.dark, report.stalled)) + .flatten(); + (counted.unwrap_or_else(|| word.to_string()), dot) } diff --git a/rs/moq-studio/src/ui/multiview.rs b/rs/moq-studio/src/ui/multiview.rs index d220a22e..dd893d75 100644 --- a/rs/moq-studio/src/ui/multiview.rs +++ b/rs/moq-studio/src/ui/multiview.rs @@ -257,6 +257,8 @@ struct SourceTile { on_air: bool, /// Whether it has a tile. shown: bool, + /// Why it shows no new picture, if it went dark or stalled. + lost: Option, } /// A scene as its tile shows it. @@ -268,6 +270,8 @@ struct SceneTile { key: Option, tally: Tally, tag: Option<&'static str>, + /// Whether it draws a layer gone dark or stalled. + lost: bool, } /// What the multiview shows this frame, copied under one short lock. @@ -408,6 +412,7 @@ impl Inner { let collection = state.collection(); let program = state.compositor.program_scene(); let cued = state.cued(); + let now = kio::clock::now(); let scenes = collection .scenes .iter() @@ -422,6 +427,7 @@ impl Inner { key: self.bindings.get(&action).map(|binding| binding.shown(ctx)), tally, tag, + lost: super::dark::draws_lost(collection, name, |name| state.lost(name, now).is_some()), } }) .collect(); @@ -436,6 +442,7 @@ impl Inner { // The compositor names a copy by the source it copies. on_air: on_air.contains(collection.capture(name)), shown: self.saved.shows(&show, name, source), + lost: state.lost(name, now), }) .collect(); let canvas = state.compositor.canvas(); @@ -657,7 +664,7 @@ impl Inner { ui.add_space(8.0); on_air(ui, &self.engine); let (word, dot) = live::health_word(&self.engine); - ui.add(kit::Pill::new(word).dot(dot)); + ui.add(kit::Pill::new(&word).dot(dot)); }); picked } @@ -776,6 +783,10 @@ fn tile_ui( if refused.is_some() { painter.rect_filled(thumb, top, kit::BG.gamma_multiply(0.6)); } + // A source gone dark or stalled is marked on its picture, as on BUILD's. + if let Tile::Source(SourceTile { lost: Some(lost), .. }) = shows { + super::dark::on_picture(ui, thumb, name, *lost); + } painter.rect_stroke(rect, radius, outline, StrokeKind::Outside); // The name bar: the key or the source's kind, the name, and the tally's @@ -822,6 +833,17 @@ fn tile_ui( painter.galley(rect.center() - galley.size() / 2.0, galley, ink); right = rect.left() - 9.0 * scale; } + // The amber badge for a picture gone dark or stalled, as on a scene pad. + let lost = match shows { + Tile::Scene(scene) => scene.lost, + Tile::Source(source) => source.lost.is_some(), + }; + if lost { + let size = 18.0 * scale; + let badge = Rect::from_center_size(pos2(right - size / 2.0, middle), Vec2::splat(size)); + super::dark::badge(ui, badge); + right = badge.left() - 9.0 * scale; + } if let Tile::Source(source) = shows && source.on_air { diff --git a/rs/moq-studio/src/ui/shell.rs b/rs/moq-studio/src/ui/shell.rs index d1bab914..50433600 100644 --- a/rs/moq-studio/src/ui/shell.rs +++ b/rs/moq-studio/src/ui/shell.rs @@ -45,6 +45,8 @@ struct ScenePad { canvas: Canvas, /// Why it cannot go on program, if it cannot. refused: Option, + /// Whether a layer it shows went dark or stalled. + dark: bool, } /// A sound the stream carries, as its row shows it. @@ -177,6 +179,7 @@ impl Panels { let position = history.position(); let undo = position.checked_sub(1).and_then(|at| history.entries().nth(at)); let redo = history.entries().nth(position); + let now = kio::clock::now(); let scenes = collection .scenes .iter() @@ -184,6 +187,7 @@ impl Panels { name: scene.name.clone(), canvas: scene.canvas(collection.canvas), refused: collection.refuses(&scene.name), + dark: super::dark::draws_lost(collection, &scene.name, |name| state.lost(name, now).is_some()), }) .collect(); let stream = collection.program_audio().map(str::to_string); @@ -874,6 +878,7 @@ impl Panels { .tally(tally) .tag(tag) .refused(pad.refused.as_deref()) + .dark(pad.dark) .fault(fault) .width(width) .shape(shape);