diff --git a/README.md b/README.md index 46de5ea3..ddf5aa05 100644 --- a/README.md +++ b/README.md @@ -282,6 +282,14 @@ for the same menu, or the picture elsewhere to add a layer, paste, or set **Snap**, **Grid**, and the grid's spacing. Escape ends cropping, then the selection. +A layer in a [slot](#slots-and-roles) is outlined on the picture, dashed, with +a chip naming its role, or **Guest** or its slot, and a tag naming the source +it shows. It has no handles, nor Fit and Fill, since its slot places it: drag +it onto another slot, or drag a row of Layers onto a slot, to swap the two +layers' places. A row in a slot says **Main slot** or **Tile** before what it +shows. The row's menu puts a layer in the main slot or a tile, or sets it +free where it is drawn. + **Interact** passes the mouse and keyboard to the selected layer's web page alone, even where another layer is over it, while the picture says so; Escape or **Stop** ends it. @@ -291,16 +299,17 @@ is named for it, with these sections: | Section | Use it to | | --- | --- | -| Place | Type or drag its position and size, and its crop, in percent off each edge, fit or fill it, and lock it. | +| Place | Pick its [slot](#slots-and-roles), or none; type or drag its position and size, and its crop, in percent off each edge, fit or fill it, and lock it. A layer in a slot has no position or size of its own to type. | | Shape | Pick its outline, square, rounded, circle, or arch, give it a border inside that outline and the border's color, and give it a soft shadow, which follows the outline and is the layer's first drop shadow filter. | | Look | Remove its source's background, which every layer showing the source loses (make it independent to cut out here alone); grade its color through a LUT source; add, order, set, and remove its filters, copy them, or paste copied filters after its own; and set its blend mode. | | Shown in | See how many places show its source, **Linked in 3 scenes**, and **Make independent**, which points this layer at a copy of the source of its own, keeping its look, so editing that look no longer edits the others. A copy says what it copies. | -| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. | +| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. For a role, pick the source filling it. | Its **Animate** tab holds the layer's keyframes (see [Keyframes](#keyframes)). With no layer selected, the inspector shows the edited scene: rename it, see its size, mark it **Safe**, switch on **Publish** and pick the audio scene -it carries (see [Published scenes](#published-scenes)), and pick the show's +it carries (see [Published scenes](#published-scenes)), arrange its slots and +fill their roles (see [Slots and roles](#slots-and-roles)), and pick the show's transition and how long it takes, or for a stinger its media source and cut point (see [Transitions](#transitions)). A scene on program, or still leaving it in a transition, cannot be removed, even by an undo, until program cuts @@ -406,8 +415,8 @@ this run says so, and is not editable there. A change applies the next time you go live. A new relay or token is followed for broadcast sources and guests at once off air, and once the stream ends on air. The recordings folder (Settings, Recording) and the guests' room (Settings, Guests) are kept -in the same file, as `recordings`, `room`, `room_key`, `join_page`, and -`guest_expiry`. +in the same file, as `recordings`, `room`, `room_key`, `join_page`, +`guest_expiry`, and `guests_appear`. ### Settings @@ -421,7 +430,7 @@ setting costs. Escape or a click outside closes it. | Stream | Set the relay, its token, the failover relay, the broadcast's name, and on demand (see [Stream settings](#stream-settings)). | | Recording | Change or open the recordings folder, and pick the picture and the audio scene a recording holds, and whether each source records a file of its own too (see [Recording](#recording)). | | Hotkeys | Bind keys, by what they are for: going live, ending the stream, the panic key, and recording; Studio mode, TAKE, and AUTO; each scene; and replay and clips (see [Hotkeys](#hotkeys)). | -| Guests | Set the room guests join, its key, the join page, and how long a link lasts (see [Guests](#guests)). | +| Guests | Set the room guests join, its key, the join page, how long a link lasts, and whether guests appear on their own (see [Guests](#guests)). | | Advanced | Publish scenes as broadcasts of their own and sources independently, turn the virtual camera of the program or of each published scene on, pick how the window's [effects](#effects) move, and open the [scene graph](#scene-graph). | On air, a new canvas size or frame rate, or a published scene's new size, @@ -649,6 +658,57 @@ and speeding up through the rest; a picture stall leaves the sound playing, with the canvas showing the last picture meanwhile. Without audio, the picture plays 100 ms behind. The inspector's Source section shows how far behind each broadcast plays. +### Slots and roles + +A scene can lay out some of its layers by **slots** rather than by hand: one +**main** slot, the largest, and **tiles** beside it, in the order of their +layers. The scene's **arrangement** places them, each slot 16:9, +inside a margin, each picture covering its slot, cropped evenly to fit: + +| Arrangement | Places | +| --- | --- | +| Side column | The main slot on the left, the tiles stacked in a column on the right. | +| Bottom row | The main slot on top, the tiles in a row under it. | +| Grid | Every slot the same size, in as square a grid as fits them. | +| Picture in picture | The main slot fills the canvas, the tiles small, stacked up from its bottom-right corner. | + +A layer in a slot closes up when it has nothing to show: when it is hidden, +when its source is offline or failed, or when it shows a role nothing fills. +The others grow into its room, and they glide there over 0.4 s on program, as +a Move transition does, so a guest joining or leaving moves the rest rather +than cutting them. Layers set free, by their own `rect`, stay where they are, +so a scene can mix both: a backdrop behind the slots and a logo over them. + +A **role** is a source that names another, once for the whole show: +`Main`, the main picture, or `Camera`, the host's camera. A layer showing a +role shows its source, through that source's crop and filters, and filling +the role with another source, from the scene inspector's Slots or the +layer's Source section, changes it in every scene at once. A role filled by +nothing draws nothing, and its slot closes up. Removing the source filling a +role leaves the role unfilled. A role takes no look of its own, and is not +cloned or published; its layers take their own. + +The scene inspector's **Slots** section picks the arrangement, lists the +slots with the role or source in each and what fills each role, and adds a +**Main slot** or **Camera slot**, making the role if the show has none. Add, under +**Not in any scene**, puts a guest into the scene's next tile when it has +slots. + +**Appear on their own**, in Settings, Guests, has a guest whose camera comes +online take the next tile of the scene on program, and be heard on program, +as an edit you can undo. Up to four guests still in the room take tiles that +way, and more wait under Add. One who leaves keeps their tile, closed until +they are back, and frees their place: the next guest takes a new tile, or +once four tiles hold guests, a tile the studio gave one who has left. They +are no longer heard on program unless another layer shows them, and wait +under Add when back, as they appeared once already. A tile you gave a guest +yourself stays theirs, as does a copy of one, and the scene file marks the +studio's own with `auto = true`. A guest appears on their own once per link, +across restarts: after that, where they are shown is yours, so undoing their +appearance or removing their layer keeps them off when they reconnect, until +a new link. Since an appearance is your edit, it drops what you could redo. +Off, every guest waits under Add, as before. + ### Audio scenes Audio from every source is mixed whether or not its scene is live, and cuts @@ -929,10 +989,12 @@ before it expires, and is then renewed for as long as they stay, until you remove them. A guest's broadcasts become sources named after them (`alice`, `alice screen`) when they first appear, in no scene or audio scene: place them from BUILD's Add and send them to the audio scene on program on MIX -to put them on air. That includes a +to put them on air, or have them [appear on their own](#slots-and-roles) in +the next tile of the scene on program. That includes a guest invited before a restart whose link still works. A guest's source is saved to the scene file but is not your edit: it stays out of History, so undo -never removes it and a guest joining never loses what you could redo. A guest +never removes it and a guest joining never loses what you could redo, unless +they [appear on their own](#slots-and-roles). A guest who reconnects after you removed their source comes back under a new name (`alice 2`), since undoing the removal brings back the old one. **Remove from scenes** is your edit, so it undoes: it takes a guest, any clone @@ -940,9 +1002,9 @@ of their sources, and any mask reading them out of every scene and the top layer and mutes them. A signed token cannot be taken back, so it also stops renewing theirs and ends their return feed, and they are disconnected within `--guest-expiry`. That -holds across a restart, whichever scene file you open: removed guests are -kept beside the room key, as `room.guests.toml` for `room.jwk`, so its -directory must be writable. Undo puts them back in the scenes but not on +holds across a restart, whichever scene file you open: removed guests, and +those who appeared on their own, are kept beside the room key, as +`room.guests.toml` for `room.jwk`, so its directory must be writable. Undo puts them back in the scenes but not on renewal; **New link** lets them back in and renews them again, under a new generation, so a link from before the removal cannot read their renewals. @@ -996,6 +1058,10 @@ path = "logo.png" # relative to the scene file kind = "broadcast" path = "room/guest" +[sources.Main] # a role: the show's main picture, wherever a scene shows it +kind = "role" +source = "camera" # the source filling it; none draws nothing + [sources.alerts] kind = "browser" url = "https://example.com/alerts" @@ -1073,6 +1139,15 @@ items = [ { source = "main-view", rect = { x = 80, y = 80, width = 640, height = 360 } }, ] +[[scenes]] +name = "Guests" +arrangement = "column" # column (the default), row, grid, or inset +items = [ + { source = "Main", slot = "main" }, # a slot takes no rect + { source = "guest", slot = "tile" }, + { source = "logo", rect = { x = 1600, y = 40, width = 280, height = 120 } }, +] + [[scenes]] name = "Portrait" size = { width = 1080, height = 1920 } # its own size; the program's without one @@ -1110,7 +1185,7 @@ Source kinds are `camera`, `display`, and `microphone` (each with an optional (macOS), `image` (PNG or JPEG), `media` (a video or audio file; see [Media files](#media-files)), `color` (`#rrggbb` or `#rrggbbaa`), `broadcast`, `browser` (a `url`, and a `width` and `height` up to 4096), -`overlay` (its `html`, a `width` and `height` up to 4096, and whether it `restart`s, true by default), `clone` (with the `source` it shows), `lut` (a 3D Adobe or Resolve `.cube` +`overlay` (its `html`, a `width` and `height` up to 4096, and whether it `restart`s, true by default), `clone` (with the `source` it shows), `role` (with the `source` filling it, if any; see [Slots and roles](#slots-and-roles)), `lut` (a 3D Adobe or Resolve `.cube` file, which no layer places, but which grades what a `lut` filter names it in), `replay`, where REVIEW plays replays, `scene` (with the `scene` it draws), and `program` (see [Scene sources](#scene-sources)). BUILD's Add fills in @@ -1124,7 +1199,9 @@ at most. `safe = true` marks the scene the panic key cuts to, on one scene at most, which must be able to go on program. A scene's `audio` names the audio scene it carries while published, the -one on program without it. +one on program without it, and its `arrangement` how it places its slots: +`column`, the default, `row`, `grid`, or `inset` (see +[Slots and roles](#slots-and-roles)). `[[audio_scenes]]` lists the audio scenes, in the order MIX lists them, each with its `name`, the sources whose `sounds` it holds, and the other @@ -1140,7 +1217,8 @@ default), and `delay`, in every audio scene holding it. Each entry of a scene's `items` is one of its layers, back to front, and each `[[top]]` an always-on-top layer: the `source` it shows and its `rect` on the -canvas, in pixels, and optionally `visible = false`, `locked = true`, which +canvas, in pixels, or, in a scene, its `slot`, `main` (one per scene) or +`tile`, in place of a `rect`, and optionally `visible = false`, `locked = true`, which the picture's pointer passes through, and its `crop`, `filters`, `blend`, `shape`, and `keyframes`. diff --git a/quest/m1/auto-layout-check.md b/quest/m1/auto-layout-check.md new file mode 100644 index 00000000..4fa2e413 --- /dev/null +++ b/quest/m1/auto-layout-check.md @@ -0,0 +1,24 @@ +# [XS] Watch the slots reflow with real guests + +## Goal + +The maintainer has seen a scene laid out by slots reflow on a real show, with +guests joining and leaving mid-stream, and it reads as a deliberate glide +rather than a glitch. If it does not, the reflow's length, easing, or what +closes a slot changes until it does. + +## Plan + +The geometry, the glide's frames on the program clock, and a guest taking a +tile are tested in moq-scene and the studio, and BUILD's views show the +slots. Whether the motion reads well is a judgment no test makes, so it waits +on a person watching. + +## Waiting + +- Maintainer: run a show with a scene laid out by slots on program and + **Appear on their own** on (Settings, Guests), have two guests join from + the join page and one leave, and watch the program and a recording. An + agent cannot judge how motion reads to a viewer; the GPU tests already + check the frames. Done when the glide reads as intended, or the change it + needs is written into this Plan. diff --git a/quest/m1/auto-layout.md b/quest/m1/auto-layout.md deleted file mode 100644 index 2815e482..00000000 --- a/quest/m1/auto-layout.md +++ /dev/null @@ -1,37 +0,0 @@ -# [L] Scenes that lay themselves out - -## Goal - -A scene can be built from slots instead of fixed rectangles: a main slot, a -camera slot, and a row or grid of guest slots. A slot holds whatever fills -its role, and the scene reflows with a glide when a guest joins, a guest -leaves, or a source goes away. The `autolayout` frame of -[`mock/later.html`](/quest/m0/ux/mock/later.html) shows one scene before and -after a second guest joins. - -A show names its roles once (which camera is the main camera, which window is -the main picture), so a scene or a template built from roles survives a -change of hardware. - -## Plan - -- A slot layout is a new way to place a scene's items, beside today's `rect`, - in moq-scene and the scene file. A scene mixes both: a logo stays where it - was put while the guests reflow. Settle the file's shape here, keeping it - hand-editable. -- Roles map names to sources per show. An unfilled role draws nothing and - its slot closes up. -- A guest who joins fills the next guest slot of the scene on air only if - the show says guests appear on their own; otherwise they wait in Add, as - today. -- The reflow is a Move transition between the two arrangements, on the - program clock, so the stream and a recording show the same frames. -- BUILD shows slots as outlines with their role names, and dragging a layer - onto a slot assigns it. - -Human check: whether the reflow reads as intended rather than as a glitch, -with real guests joining mid-show. - -## Related - -- [First run](/quest/m0/ux/first-run.md) - its starting layouts become role-based templates diff --git a/quest/m1/auto-place.md b/quest/m1/auto-place.md index a2b6c79c..60245891 100644 --- a/quest/m1/auto-place.md +++ b/quest/m1/auto-place.md @@ -36,7 +36,10 @@ ready; local first with a hosted model as an opt-in tie-breaker). game is right from its first frame. - **Where it lives:** the analysis and the choice are UI-free, in moq-scene, tested on recorded frames with a paused clock. The move is the item's - rect animated as a Move transition eases. + rect animated as a Move transition eases: moq-scene's `Glide` + (`rs/moq-scene/src/glide.rs`), which glides a scene's slots when they + reflow, eases any list of items from one placement to the next on the + program clock, keyed by what changed. Measure, per AGENTS.md: the frame time and GPU copies the analysis adds, and that the encoder's load does not move. Should the signals prove too weak, a @@ -48,4 +51,3 @@ a judgment no test makes. ## Related - [moq-video's foreground app is pinned](/quest/m1/pin-capture-foreground.md) - when the app in front changes, and the key its slot is remembered under -- [Scenes that lay themselves out](/quest/m1/auto-layout.md) - slots chosen by role rather than by content diff --git a/quest/m1/dual-format.md b/quest/m1/dual-format.md index 1da4b73a..80065cfe 100644 --- a/quest/m1/dual-format.md +++ b/quest/m1/dual-format.md @@ -25,6 +25,10 @@ program source, cropped. This quest makes that one switch. - A portrait arrangement starts from the landscape one: the main picture cropped around its subject, the camera stacked under it. It is then the streamer's to move. +- A layer in a slot keeps no rect, so a scene laid out by slots needs only a + second `arrangement` for portrait (such as `portrait = { arrangement = + "row" }`), and the slots place themselves on the portrait canvas; only its + layers set free need a second rect. - Each format is its own encode, so it costs an encoder session and its bitrate; the checklist says whether the machine and the uplink carry both. - Where a layer is shown is said under the pictures and in the inspector, diff --git a/rs/moq-scene/src/arrange.rs b/rs/moq-scene/src/arrange.rs new file mode 100644 index 00000000..28ea892d --- /dev/null +++ b/rs/moq-scene/src/arrange.rs @@ -0,0 +1,364 @@ +//! Slot layouts: where a scene's [`Arrangement`] puts the items in its +//! slots, given which of them take one. + +use crate::collection::{Arrangement, Canvas, Collection, Crop, Item, Rect, Slot}; + +/// The shape of every slot: a camera's, a screen's, and a 1080p canvas's. +const ASPECT: f32 = 16.0 / 9.0; + +/// The margin around the slots, as a share of the canvas's shorter side. +const MARGIN: f32 = 0.04; + +/// The gap between two slots, as a share of the canvas's shorter side. +const GAP: f32 = 0.02; + +/// The widest the column of tiles grows, or the tallest the row, as a share +/// of the room for the slots, so the main item stays the largest. +const SIDE: f32 = 0.4; + +/// The widest an inset tile is, as a share of the canvas's width. +const INSET: f32 = 0.2; + +impl Arrangement { + /// Where it puts a main item, if there is one, and `tiles` tiles, in + /// order, on a `canvas`-sized picture, in whole pixels. + pub fn places(self, canvas: Canvas, main: bool, tiles: usize) -> (Option, Vec) { + places(self, canvas, main, tiles) + } +} + +/// Where `arrangement` puts a main item, if there is one, and `tiles` tiles, +/// in order, on a `canvas`-sized picture, in whole pixels. +fn places(arrangement: Arrangement, canvas: Canvas, main: bool, tiles: usize) -> (Option, Vec) { + let (width, height) = (canvas.width as f32, canvas.height as f32); + let short = width.min(height); + let (margin, gap) = ((short * MARGIN).round(), (short * GAP).round()); + let room = Rect { + x: margin, + y: margin, + width: (width - 2.0 * margin).max(0.0), + height: (height - 2.0 * margin).max(0.0), + }; + let (main, tiles) = match (main, tiles) { + (false, tiles) => (None, grid(room, tiles, gap)), + (true, 0) if arrangement == Arrangement::Inset => (Some(Rect::fill(&canvas)), Vec::new()), + (true, 0) => (Some(fit(room, ASPECT)), Vec::new()), + (true, tiles) => match arrangement { + Arrangement::Column => column(room, tiles, gap), + Arrangement::Row => row(room, tiles, gap), + Arrangement::Grid => { + let mut cells = grid(room, tiles + 1, gap); + let main = cells.remove(0); + (Some(main), cells) + } + Arrangement::Inset => inset(canvas, room, tiles, margin, gap), + }, + }; + (main.map(whole), tiles.into_iter().map(whole).collect()) +} + +/// The main item on the left and the tiles stacked on its right, the pair +/// centered in `room`. +fn column(room: Rect, tiles: usize, gap: f32) -> (Option, Vec) { + let count = tiles as f32; + let tall = (room.height - gap * (count - 1.0)) / count; + let wide = (tall * ASPECT).min((room.width - gap) * SIDE).max(0.0); + let tall = wide / ASPECT; + let main = fit( + Rect { + width: (room.width - wide - gap).max(0.0), + ..room + }, + ASPECT, + ); + let left = room.x + (room.width - main.width - gap - wide) / 2.0; + let main = Rect { x: left, ..main }; + let stack = tall * count + gap * (count - 1.0); + let top = room.y + (room.height - stack) / 2.0; + let x = left + main.width + gap; + let tiles = (0..tiles).map(|index| Rect { + x, + y: top + index as f32 * (tall + gap), + width: wide, + height: tall, + }); + (Some(main), tiles.collect()) +} + +/// The main item on top and the tiles in a row under it, the pair centered +/// in `room`. +fn row(room: Rect, tiles: usize, gap: f32) -> (Option, Vec) { + let count = tiles as f32; + let wide = (room.width - gap * (count - 1.0)) / count; + let tall = (wide / ASPECT).min((room.height - gap) * SIDE).max(0.0); + let wide = tall * ASPECT; + let main = fit( + Rect { + height: (room.height - tall - gap).max(0.0), + ..room + }, + ASPECT, + ); + let top = room.y + (room.height - main.height - gap - tall) / 2.0; + let main = Rect { y: top, ..main }; + let span = wide * count + gap * (count - 1.0); + let left = room.x + (room.width - span) / 2.0; + let y = top + main.height + gap; + let tiles = (0..tiles).map(|index| Rect { + x: left + index as f32 * (wide + gap), + y, + width: wide, + height: tall, + }); + (Some(main), tiles.collect()) +} + +/// The main item filling the canvas and the tiles stacked up from its +/// bottom-right corner, inside the margin. +fn inset(canvas: Canvas, room: Rect, tiles: usize, margin: f32, gap: f32) -> (Option, Vec) { + let count = tiles as f32; + let fits = (room.height - gap * (count - 1.0)) / count * ASPECT; + let wide = (canvas.width as f32 * INSET).min(fits).max(0.0); + let tall = wide / ASPECT; + let x = canvas.width as f32 - margin - wide; + let bottom = canvas.height as f32 - margin; + let tiles = (0..tiles).map(|index| Rect { + x, + y: bottom - tall - index as f32 * (tall + gap), + width: wide, + height: tall, + }); + (Some(Rect::fill(&canvas)), tiles.collect()) +} + +/// `count` equal cells filling as much of `room` as they can, row by row, +/// the last row centered and the grid centered in `room`. +fn grid(room: Rect, count: usize, gap: f32) -> Vec { + if count == 0 { + return Vec::new(); + } + // The number of columns that makes the cells largest. + let cell = |columns: usize| { + let rows = count.div_ceil(columns); + let wide = (room.width - gap * (columns as f32 - 1.0)) / columns as f32; + let tall = (room.height - gap * (rows as f32 - 1.0)) / rows as f32; + let wide = wide.min(tall * ASPECT).max(0.0); + (wide, wide / ASPECT) + }; + let columns = (1..=count) + .max_by(|a, b| cell(*a).0.total_cmp(&cell(*b).0).then(b.cmp(a))) + .expect("at least one cell"); + let rows = count.div_ceil(columns); + let (wide, tall) = cell(columns); + let block = tall * rows as f32 + gap * (rows as f32 - 1.0); + let top = room.y + (room.height - block) / 2.0; + (0..count) + .map(|index| { + let (row, column) = (index / columns, index % columns); + let across = (count - row * columns).min(columns) as f32; + let span = wide * across + gap * (across - 1.0); + let left = room.x + (room.width - span) / 2.0; + Rect { + x: left + column as f32 * (wide + gap), + y: top + row as f32 * (tall + gap), + width: wide, + height: tall, + } + }) + .collect() +} + +/// The largest rect of `aspect`, width over height, centered in `room`. +fn fit(room: Rect, aspect: f32) -> Rect { + let width = room.width.min(room.height * aspect).max(0.0); + let height = width / aspect; + Rect { + x: room.x + (room.width - width) / 2.0, + y: room.y + (room.height - height) / 2.0, + width, + height, + } +} + +/// `rect` with its edges on whole pixels, so a slot draws sharp and two meet +/// without a seam. +fn whole(rect: Rect) -> Rect { + let (left, top) = (rect.x.round(), rect.y.round()); + Rect { + x: left, + y: top, + width: (rect.x + rect.width).round() - left, + height: (rect.y + rect.height).round() - top, + } +} + +/// The crop that trims what `crop` leaves of a `size` picture evenly from two +/// opposite sides to the shape of `rect`, so it covers it unstretched. +pub(crate) fn cover(crop: Crop, size: (u32, u32), rect: &Rect) -> Crop { + let width = size.0 as f32 * (1.0 - crop.left - crop.right); + let height = size.1 as f32 * (1.0 - crop.top - crop.bottom); + Crop::covering(width / height, rect.width / rect.height) +} + +impl Crop { + /// The even crop that shows a picture shaped `shown` (width over height) + /// in a frame shaped `wanted` without stretching it; none for a shape + /// that is not a positive number. + pub fn covering(shown: f32, wanted: f32) -> Self { + if !shown.is_finite() || shown <= 0.0 || !wanted.is_finite() || wanted <= 0.0 { + return Self::default(); + } + if shown > wanted { + let trim = (1.0 - wanted / shown) / 2.0; + Self { + left: trim, + right: trim, + ..Self::default() + } + } else { + let trim = (1.0 - shown / wanted) / 2.0; + Self { + top: trim, + bottom: trim, + ..Self::default() + } + } + } +} + +impl Collection { + /// The items of the scene called `scene` as they are drawn, in order: + /// each in a slot placed by the scene's arrangement, its picture cropped + /// to cover the slot given its size from `size`, and hidden if it takes + /// no slot. An item takes one while it is visible, its role is filled, + /// and `present` says its source is there, asked by the source the item + /// shows through its role. `None` for a scene with no item in a slot, or + /// no such scene. + pub fn arranged( + &self, + scene: &str, + present: impl Fn(&str) -> bool, + size: impl Fn(&str) -> Option<(u32, u32)>, + ) -> Option> { + let scene = self.scene(scene)?; + if scene.items.iter().all(|item| item.slot.is_none()) { + return None; + } + // Asked once each, so counting and placing see the same answers. + let takes: Vec = (scene.items.iter()) + .map(|item| item.visible && !self.unfilled(&item.source) && present(self.resolve(&item.source))) + .collect(); + let slotted = |slot: Slot| { + (scene.items.iter().zip(&takes)).filter(move |(item, takes)| item.slot == Some(slot) && **takes) + }; + let main = slotted(Slot::Main).next().is_some(); + let tiles = slotted(Slot::Tile).count(); + let (main, tiles) = places(scene.arrangement, scene.canvas(self.canvas), main, tiles); + let mut tiles = tiles.into_iter(); + let arranged = scene.items.iter().zip(takes).map(|(item, takes)| { + let Some(slot) = item.slot else { + return item.clone(); + }; + let rect = match takes { + true if slot == Slot::Main => main, + true => tiles.next(), + false => None, + }; + let Some(rect) = rect else { + return Item { + visible: false, + ..item.clone() + }; + }; + let crop = match size(self.resolve(&item.source)) { + Some(size) => item.crop.then(&cover(self.crop(item), size, &rect)), + None => item.crop, + }; + Item { + rect, + crop, + ..item.clone() + } + }); + Some(arranged.collect()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const HD: Canvas = Canvas { + width: 1920, + height: 1080, + framerate: 30, + }; + + fn overlaps(a: &Rect, b: &Rect) -> bool { + a.x < b.x + b.width && b.x < a.x + a.width && a.y < b.y + b.height && b.y < a.y + a.height + } + + fn inside(rect: &Rect, canvas: Canvas) -> bool { + rect.x >= 0.0 + && rect.y >= 0.0 + && rect.x + rect.width <= canvas.width as f32 + && rect.y + rect.height <= canvas.height as f32 + } + + /// Every arrangement, on a wide and a tall canvas, with and without a + /// main item and with any number of tiles, keeps each slot on the + /// canvas, apart from the others, and the main item the largest; a + /// tile overlaps the main item only when it is inset in it. + #[test] + fn slots_stay_on_the_canvas_apart_and_the_main_one_largest() { + let tall = Canvas { + width: 1080, + height: 1920, + framerate: 30, + }; + for canvas in [HD, tall] { + for arrangement in Arrangement::ALL { + for main in [false, true] { + for count in 0..=9 { + let (big, tiles) = places(arrangement, canvas, main, count); + let on = format!("{arrangement:?} on {canvas:?}, main {main}, {count} tiles"); + assert_eq!(big.is_some(), main, "{on}"); + assert_eq!(tiles.len(), count, "{on}"); + for (index, tile) in tiles.iter().enumerate() { + assert!(inside(tile, canvas), "{on}: {tile:?}"); + assert!(tile.width > 0.0 && tile.height > 0.0, "{on}: {tile:?}"); + for other in &tiles[index + 1..] { + assert!(!overlaps(tile, other), "{on}: {tile:?} and {other:?}"); + } + if let Some(big) = big { + let area = |rect: &Rect| rect.width * rect.height; + // Equal grid cells may round a pixel apart. + assert!(area(&big) >= area(tile) * 0.99, "{on}: {big:?} and {tile:?}"); + let covers = arrangement == Arrangement::Inset; + assert_eq!(overlaps(&big, tile), covers, "{on}: {big:?} and {tile:?}"); + } + } + if let Some(big) = big { + assert!(inside(&big, canvas), "{on}: {big:?}"); + } + } + } + } + } + } + + /// The `autolayout` frame of the mock-up: a second guest shrinks the + /// column and grows the main picture, which keeps its place on the left. + #[test] + fn a_second_guest_shrinks_the_column_and_grows_the_main_picture() { + let (one, column) = places(Arrangement::Column, HD, true, 2); + let (two, longer) = places(Arrangement::Column, HD, true, 3); + let (one, two) = (one.unwrap(), two.unwrap()); + assert!(two.width > one.width, "{one:?} then {two:?}"); + assert!(longer[0].width < column[0].width); + assert!(column.iter().all(|tile| tile.x > one.x + one.width)); + for main in [one, two] { + assert!((main.width / main.height - ASPECT).abs() < 0.01, "{main:?}"); + } + } +} diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index c5c4be16..faae0a4d 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -7,7 +7,9 @@ use std::path::Path; use serde::{Deserialize, Serialize}; use crate::Error; -use crate::collection::{Audio, Collection, Filter, Item, Layer, Scene, Source, Video, validate_filters, versioned}; +use crate::collection::{ + Audio, Collection, Filter, Item, Layer, Rect, Scene, Slot, Source, Video, validate_filters, versioned, +}; use crate::edit::Draft; /// What a copy holds: items, or a filter chain. @@ -52,7 +54,11 @@ impl Clip { /// filters read. Relative image and LUT paths are resolved against `dir`, /// the absolute directory of the collection's scene file, so they name the /// same file wherever the clip is pasted. - pub fn items(collection: &Collection, items: Vec, dir: &Path) -> Self { + pub fn items(collection: &Collection, mut items: Vec, dir: &Path) -> Self { + // A copy is the host's, never the studio's to give away. + for item in &mut items { + item.auto = false; + } let mut names: Vec = items.iter().map(|item| item.source.clone()).collect(); names.extend(items.iter().flat_map(|item| &item.filters).filter_map(read)); @@ -62,7 +68,8 @@ impl Clip { audio: BTreeMap::new(), video: BTreeMap::new(), }; - // Each source brings what it clones and what its own filters read. + // Each source brings what it clones, what fills it if it is a role, + // and what its own filters read. while let Some(name) = names.pop() { let Some(source) = collection.sources.get(&name) else { continue; @@ -75,6 +82,7 @@ impl Clip { *path = dir.join(&path); } clip.sources.insert(name.clone(), source); + names.push(collection.resolve(&name).to_string()); names.push(collection.capture(&name).to_string()); if let Some(audio) = collection.audio.get(&name) { clip.audio.insert(name.clone(), audio.clone()); @@ -167,15 +175,23 @@ impl Draft { pub fn paste(&mut self, layer: &Layer, items: &Items, dir: &Path) -> Result<(), Error> { let mut names = BTreeMap::new(); let mut added = Vec::new(); - // What clones clone first, so each clone names what it was renamed to. - let (clones, origins): (Vec<_>, Vec<_>) = items - .sources - .iter() - .partition(|(_, source)| matches!(source, Source::Clone { .. })); - for (name, source) in origins.into_iter().chain(clones) { + // What clones clone first, then the clones, then the roles, which a + // clone may fill, so each names what it was renamed to. + let rank = |source: &Source| match source { + Source::Clone { .. } => 1, + Source::Role { .. } => 2, + _ => 0, + }; + let mut ordered: Vec<_> = items.sources.iter().collect(); + ordered.sort_by_key(|(_, source)| rank(source)); + for (name, source) in ordered { + let renamed = |source: &String| names.get(source).cloned().unwrap_or_else(|| source.clone()); let mut source = match source { Source::Clone { source } => Source::Clone { - source: names.get(source).cloned().unwrap_or_else(|| source.clone()), + source: renamed(source), + }, + Source::Role { source } => Source::Role { + source: source.as_ref().map(renamed), }, source => source.clone(), }; @@ -232,11 +248,19 @@ impl Draft { } for item in &items.items { let source = names.get(&item.source).cloned().unwrap_or_else(|| item.source.clone()); - let index = self + let held = self .collection() .items(layer) - .ok_or_else(|| Error::Conflict(format!("there is no {layer}")))? - .len(); + .ok_or_else(|| Error::Conflict(format!("there is no {layer}")))?; + let index = held.len(); + let main = held.iter().any(|item| item.slot == Some(Slot::Main)); + // Only a scene arranges slots, and it holds one main slot; a layer + // set free covers the canvas. + let (slot, rect) = match item.slot { + Some(_) if layer.scene.is_none() => (None, Rect::fill(&self.collection().canvas)), + Some(Slot::Main) if main => (Some(Slot::Tile), item.rect), + slot => (slot, item.rect), + }; let filters = rename(&item.filters); self.insert_item( layer, @@ -244,6 +268,8 @@ impl Draft { Item { source, filters, + slot, + rect, ..item.clone() }, )?; @@ -338,6 +364,17 @@ mod tests { assert_eq!(pasted.top, collection.scenes[0].items); } + /// A layer the studio placed on its own is the host's once copied. + #[test] + fn a_copy_is_the_hosts() { + let mut collection = collection(); + collection.scenes[0].items[0].auto = true; + let Clip::Items(items) = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)) else { + panic!("not items") + }; + assert!(!items.items[0].auto); + } + /// An animated item copies with its keyframes into a scene, and the top /// layer, which never goes on to play them, refuses it. #[test] @@ -360,6 +397,44 @@ mod tests { assert!(matches!(draft.finish(), Err(Error::Invalid(_)))); } + /// A layer in the main slot, showing a role filled by a clone, copies + /// with the role, the clone, and what it clones. Pasted back, it is + /// linked and takes a tile, the main slot being taken; on top, it is set + /// free over the canvas; into another collection, the role follows the + /// clone it names through a rename. + #[test] + fn a_role_in_a_slot_copies_with_what_fills_it() { + let mut collection = collection(); + let filled = Source::Role { + source: Some("cam faded".into()), + }; + collection.sources.insert("Main".into(), filled.clone()); + collection.scenes[0].items = vec![Item { + slot: Some(Slot::Main), + ..Item::new("Main", Rect::default()) + }]; + collection.validate().unwrap(); + let clip = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)); + let text = clip.to_toml().unwrap(); + + let pasted = paste(&collection, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.sources, collection.sources, "linked"); + assert_eq!(pasted.scenes[0].items[1].slot, Some(Slot::Tile)); + let pasted = paste(&collection, &Layer::top(), &text, DIR); + assert_eq!(pasted.top[0].slot, None); + assert_eq!(pasted.top[0].rect, Rect::fill(&collection.canvas)); + + let mut other = Collection::new(); + other.sources.insert("cam faded".into(), color([0, 255, 0, 255])); + let pasted = paste(&other, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.sources["cam faded 2"], Source::Clone { source: "cam".into() }); + let renamed = Source::Role { + source: Some("cam faded 2".into()), + }; + assert_eq!(pasted.sources["Main"], renamed, "the role follows the rename"); + assert_eq!(pasted.scenes[0].items[0].slot, Some(Slot::Main)); + } + /// Into another collection, a clip brings what its items show, renaming a /// source whose name is taken by another, and its clone follows the rename. #[test] diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 7f8b9eff..abc28065 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -249,14 +249,33 @@ impl Collection { None } - /// The source whose capture `name` shows: itself, or what it clones. + /// The source whose capture `name` shows: itself, what it clones, or, for + /// a role, what the source it names shows. pub fn capture<'a>(&'a self, name: &'a str) -> &'a str { + let name = self.resolve(name); match self.sources.get(name) { Some(Source::Clone { source }) => source, _ => name, } } + /// The source a [`Source::Role`] called `name` names, or `name` itself + /// for any other source and for a role nothing fills. + pub fn resolve<'a>(&'a self, name: &'a str) -> &'a str { + match self.sources.get(name) { + Some(Source::Role { source: Some(source) }) => source, + _ => name, + } + } + + /// Whether `name` is a role nothing fills, which draws nothing. + pub fn unfilled(&self, name: &str) -> bool { + matches!( + self.sources.get(self.resolve(name)), + Some(Source::Role { source: None }) + ) + } + /// What the source called `name` draws if it is a scene or program /// source, through a clone. pub fn nested(&self, name: &str) -> Option> { @@ -373,16 +392,18 @@ impl Collection { Ok(()) } - /// The filters `item` draws through: its source's, then its own. + /// The filters `item` draws through: its source's, through a role, then + /// its own. pub fn filters<'a>(&'a self, item: &'a Item) -> impl Iterator { - let source = self.video.get(&item.source).map(|video| video.filters.as_slice()); + let source = self.video.get(self.resolve(&item.source)); + let source = source.map(|video| video.filters.as_slice()); source.unwrap_or_default().iter().chain(&item.filters) } - /// What `item` shows of its source's picture: its source's crop, then its - /// own. + /// What `item` shows of its source's picture: its source's crop, through a + /// role, then its own. pub fn crop(&self, item: &Item) -> Crop { - let source = self.video.get(&item.source).map(|video| video.crop); + let source = self.video.get(self.resolve(&item.source)).map(|video| video.crop); source.unwrap_or_default().then(&item.crop) } @@ -407,6 +428,13 @@ impl Collection { .canvas(self.canvas) .validate() .map_err(|err| Error::Invalid(format!("scene `{}`: {err}", scene.name)))?; + let mains = scene.items.iter().filter(|item| item.slot == Some(Slot::Main)).count(); + if mains > 1 { + return Err(Error::Invalid(format!( + "scene `{}` puts {mains} layers in its main slot, which holds one", + scene.name + ))); + } if scene.published && (scene.name.contains('/') || moq_net::Path::new(&scene.name).is_err()) { return Err(Error::Invalid(format!( "scene `{}` is published, so its name is one segment of its broadcast's path: no `/`, and a `\\` only before another", @@ -444,6 +472,18 @@ impl Collection { } item.rect.validate(&layer, &item.source)?; let on = format!("`{}` in {layer}", item.source); + if item.slot.is_some() { + if layer.scene.is_none() { + return Err(Error::Invalid(format!( + "{on} is in a slot, but only a scene arranges slots" + ))); + } + if item.rect != Rect::default() { + return Err(Error::Invalid(format!( + "{on} is in a slot, which places it, so it takes no rect" + ))); + } + } item.shape .validate() .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; @@ -500,11 +540,26 @@ impl Collection { "`{name}` plays on `{broadcast}`, which is not a broadcast source" ))); } + if let Source::Role { source: Some(source) } = source { + match self.sources.get(source) { + Some(Source::Role { .. }) => { + return Err(Error::Invalid(format!( + "role `{name}` is filled by `{source}`, which is itself a role" + ))); + } + Some(filled) if filled.has_video() && !matches!(filled, Source::Lut { .. }) => {} + _ => { + return Err(Error::Invalid(format!( + "role `{name}` is filled by `{source}`, which is not a source with a picture" + ))); + } + } + } let Source::Clone { source } = source else { continue }; match self.sources.get(source) { - Some(Source::Clone { .. }) => { + Some(Source::Clone { .. } | Source::Role { .. }) => { return Err(Error::Invalid(format!( - "`{name}` clones `{source}`, which is itself a clone" + "`{name}` clones `{source}`, which is itself a clone or a role" ))); } Some(origin) if origin.has_video() => {} @@ -522,6 +577,11 @@ impl Collection { "video settings for `{name}`, which is not a source with a picture" ))); } + if matches!(self.sources[name], Source::Role { .. }) { + return Err(Error::Invalid(format!( + "video settings for `{name}`, a role, which shows its source's" + ))); + } if video.background == Background::Remove && !self.sources[name].removes_background() { return Err(Error::Invalid(format!( "`{name}` cannot have its background removed: only a camera, display, window, image, media, \ @@ -699,6 +759,11 @@ impl Collection { "{on} cannot be in a scene another scene draws, since viewers place it over the program" ))); } + if item.slot.is_some() { + return Err(Error::Invalid(format!( + "{on} cannot be in a slot, since viewers place it where its rect says" + ))); + } let keyframed = item.keyframes.iter().flat_map(|keyframe| &keyframe.filters); let mut filters = self.filters(item).chain(keyframed); if item.blend != Blend::Normal @@ -1009,6 +1074,9 @@ pub struct Scene { /// must be able to go on program. #[serde(default, skip_serializing_if = "is_false")] pub safe: bool, + /// Where its items in a [`Slot`] go. + #[serde(default, skip_serializing_if = "Arrangement::is_default")] + pub arrangement: Arrangement, /// What the scene draws, back to front. #[serde(default)] pub items: Vec, @@ -1025,6 +1093,7 @@ impl Scene { recorded: false, audio: None, safe: false, + arrangement: Arrangement::default(), items: Vec::new(), } } @@ -1075,6 +1144,44 @@ impl fmt::Display for Layer { } } +/// Where a scene places its items in a [`Slot`]: the main one large and +/// the tiles beside it, each shaped 16:9 and the picture in it cropped to +/// cover it unstretched. Without a main item, the tiles share the canvas as +/// a grid. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Arrangement { + /// The main item on the left, the tiles in a column on its right. + #[default] + Column, + /// The main item on top, the tiles in a row under it. + Row, + /// Every item in a grid of equal cells, the main item first. + Grid, + /// The main item filling the canvas, the tiles small in its bottom-right + /// corner. + Inset, +} + +impl Arrangement { + /// Every arrangement. + pub const ALL: [Self; 4] = [Self::Column, Self::Row, Self::Grid, Self::Inset]; + + fn is_default(&self) -> bool { + *self == Self::default() + } +} + +/// Which of its scene's slots an item fills. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Slot { + /// The largest. A scene has one at most. + Main, + /// One of the tiles beside it, in the order the scene lists them. + Tile, +} + /// What a [`Source::Scene`] or [`Source::Program`] draws. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] pub enum Nested<'a> { @@ -1094,8 +1201,22 @@ pub enum Nested<'a> { pub struct Item { /// The [`Collection::sources`] entry to draw. pub source: String, - /// Where on the canvas, stretched to fill it. + /// Where on the canvas, stretched to fill it. Empty, and left out of the + /// file, for an item in a [`slot`](Self::slot). + #[serde(default, skip_serializing_if = "Rect::is_empty")] pub rect: Rect, + /// The slot its scene's [`Arrangement`] places it in, rather than its + /// rect, which it then leaves out. The arrangement closes up a slot + /// whose item is hidden, a role nothing fills, or a source gone offline + /// or failed, and glides the others to their new places. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub slot: Option, + /// Whether the studio placed it on its own, as a guest appearing in the + /// next tile, rather than the host: once that guest leaves, the next to + /// appear may take it. Undo restores it with the rest of the item, and a + /// copy is the host's. + #[serde(default, skip_serializing_if = "is_false")] + pub auto: bool, /// Whether the item is drawn. A hidden item keeps its place. #[serde(default = "yes")] pub visible: bool, @@ -1132,6 +1253,8 @@ impl Item { Self { source: source.into(), rect, + slot: None, + auto: false, visible: true, locked: false, shape: Shape::default(), @@ -1152,6 +1275,8 @@ impl Item { filters.extend(offset.filters); Cow::Owned(Self { source: self.source.clone(), + slot: self.slot, + auto: self.auto, rect: Rect { x: self.rect.x + offset.x, y: self.rect.y + offset.y, @@ -1816,7 +1941,7 @@ impl Background { } /// A rectangle in canvas pixels, from the top-left corner. -#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)] +#[derive(Clone, Copy, Debug, Default, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct Rect { /// Left edge. @@ -1840,6 +1965,11 @@ impl Rect { } } + /// Whether it is the empty rect at the origin, which an item in a slot has. + pub fn is_empty(&self) -> bool { + *self == Self::default() + } + fn validate(&self, layer: &Layer, source: &str) -> Result<(), Error> { let finite = [self.x, self.y, self.width, self.height].iter().all(|v| v.is_finite()); if !finite || self.width < 0.0 || self.height < 0.0 { @@ -1953,6 +2083,16 @@ pub enum Source { #[serde(default = "yes", skip_serializing_if = "is_true")] restart: bool, }, + /// A role a show casts once, such as its main camera or main picture: it + /// shows the source it names, with that source's crop and filters, so a + /// scene built from roles survives a change of hardware. It draws + /// nothing while nothing fills it, and its slot closes up. Its sound is + /// its source's, not mixed a second time. + Role { + /// The source filling it: one with a picture, and not itself a role. + #[serde(default, skip_serializing_if = "Option::is_none")] + source: Option, + }, /// Another source's picture, captured once, with [`Collection::video`] /// filters of its own. Its sound is not mixed a second time. Clone { @@ -2018,10 +2158,18 @@ impl Source { } } + /// Whether it is a [`Source::Role`]. + pub fn is_role(&self) -> bool { + matches!(self, Self::Role { .. }) + } + /// Whether the source opens a capture, file, or subscription of its own, /// rather than showing another source's or a scene. pub fn opens(&self) -> bool { - !matches!(self, Self::Clone { .. } | Self::Scene { .. } | Self::Program) + !matches!( + self, + Self::Clone { .. } | Self::Role { .. } | Self::Scene { .. } | Self::Program + ) } /// Whether the source's picture can have its background removed: a @@ -3472,4 +3620,171 @@ background = "remove" assert!(bad.parse::().is_err(), "{bad}"); } } + + /// A scene of slots by hand: roles name the sources in them, an item in a + /// slot leaves its rect out, and a logo keeps its rect beside them. + const SLOTS: &str = r##"version = 1 + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.Main] +kind = "role" +source = "Game" + +[sources.Camera] +kind = "role" + +[sources.Game] +kind = "window" +id = "42" + +[sources.Logo] +kind = "color" +color = "#ffffff" + +[[scenes]] +name = "Guests" +arrangement = "row" + +[[scenes.items]] +source = "Main" +slot = "main" + +[[scenes.items]] +source = "Camera" +slot = "tile" + +[[scenes.items]] +source = "Logo" +rect = { x = 1700, y = 40, width = 160, height = 90 } +"##; + + #[test] + fn a_scene_of_slots_reads_and_writes_by_hand() { + let collection = Collection::from_toml(SLOTS).unwrap(); + let scene = collection.scene("Guests").unwrap(); + assert_eq!(scene.arrangement, Arrangement::Row); + assert_eq!(scene.items[0].slot, Some(Slot::Main)); + assert_eq!(scene.items[0].rect, Rect::default()); + assert_eq!(collection.resolve("Main"), "Game"); + assert_eq!(collection.capture("Main"), "Game"); + assert!(collection.unfilled("Camera")); + assert!(!collection.unfilled("Main")); + + let text = collection.to_toml().unwrap(); + assert_eq!(Collection::from_toml(&text).unwrap(), collection); + assert_eq!(text.matches("rect").count(), 1, "only the logo has a rect:\n{text}"); + assert!(text.contains("arrangement = \"row\""), "{text}"); + } + + #[test] + fn a_role_shows_its_source_through_that_source_s_crop_and_filters() { + let mut collection = Collection::from_toml(SLOTS).unwrap(); + let video = Video { + crop: Crop { + left: 0.5, + ..Crop::default() + }, + filters: vec![Filter::Opacity { amount: 0.5 }], + ..Video::default() + }; + collection.video.insert("Game".into(), video.clone()); + collection.validate().unwrap(); + let item = &collection.scenes[0].items[0]; + assert_eq!(collection.crop(item), video.crop); + assert_eq!(collection.filters(item).collect::>(), vec![&video.filters[0]]); + + let mut own = collection.clone(); + own.video.insert("Main".into(), video); + assert!(own.validate().is_err(), "a role has no video settings of its own"); + } + + #[test] + fn refuses_slots_and_roles_that_cannot_be_drawn() { + let collection = Collection::from_toml(SLOTS).unwrap(); + let refused = |change: &dyn Fn(&mut Collection), why: &str| { + let mut changed = collection.clone(); + change(&mut changed); + let err = changed.validate().unwrap_err().to_string(); + assert!(err.contains(why), "{why}: {err}"); + }; + refused( + &|c| c.scenes[0].items[1].rect = Rect::fill(&c.canvas), + "so it takes no rect", + ); + refused(&|c| c.scenes[0].items[1].slot = Some(Slot::Main), "which holds one"); + refused( + &|c| { + let mut item = c.scenes[0].items[0].clone(); + item.source = "Game".into(); + c.top.push(item); + }, + "only a scene arranges slots", + ); + refused( + &|c| { + let filled = Some("Camera".into()); + c.sources.insert("Main".into(), Source::Role { source: filled }); + }, + "itself a role", + ); + refused( + &|c| { + let filled = Some("Nowhere".into()); + c.sources.insert("Main".into(), Source::Role { source: filled }); + }, + "not a source with a picture", + ); + refused( + &|c| { + let cloned = "Main".into(); + c.sources.insert("Copy".into(), Source::Clone { source: cloned }); + }, + "a clone or a role", + ); + refused( + &|c| { + c.sources.insert("Cam".into(), Source::Camera { id: None }); + c.published.insert("Cam".into()); + c.scenes[0].items[1].source = "Cam".into(); + }, + "cannot be in a slot", + ); + } + + /// Each slot that takes no room closes up, the rest share the canvas, + /// and the picture in a slot is cropped to cover it unstretched. + #[test] + fn closed_slots_leave_their_room_to_the_rest() { + let mut collection = Collection::from_toml(SLOTS).unwrap(); + collection.sources.insert("Cam".into(), Source::Camera { id: None }); + let mut guest = collection.scenes[0].items[1].clone(); + guest.source = "Cam".into(); + collection.scenes[0].items.insert(2, guest); + collection.validate().unwrap(); + let all = |_: &str| true; + let square = |_: &str| Some((100, 100)); + let arranged = collection.arranged("Guests", all, square).unwrap(); + assert_eq!(arranged.len(), 4); + assert!(arranged[0].visible && !arranged[1].visible, "the unfilled role closes"); + assert_eq!(arranged[3], collection.scenes[0].items[3], "the logo stays"); + let (main, tiles) = Arrangement::Row.places(collection.canvas, true, 1); + assert_eq!(arranged[0].rect, main.unwrap()); + assert_eq!(arranged[2].rect, tiles[0]); + let crop = arranged[0].crop; + assert!( + crop.top > 0.0 && crop.top == crop.bottom && crop.left == 0.0, + "{crop:?}" + ); + + let away = |name: &str| name != "Cam"; + let arranged = collection.arranged("Guests", away, square).unwrap(); + assert!(!arranged[2].visible, "an absent source closes"); + let (main, _) = Arrangement::Row.places(collection.canvas, true, 0); + assert_eq!(arranged[0].rect, main.unwrap(), "the main slot takes the room"); + assert!(collection.arranged("Scene", all, square).is_none()); + } } diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index d784a7b6..fe3739f0 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -10,11 +10,13 @@ use kio::clock::Instant; use moq_video::render::Renderer; use crate::Error; +use crate::arrange::cover; use crate::collection::{ Background, Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Nested, Rect, Settled, Shape, Source, Transition, }; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; +use crate::glide::{Glide, REFLOW, mixed, pair, stack}; use crate::matte::Matte; use crate::overlay::{Overlay, Overlays}; use crate::source::{self, Feed, Sources, Status}; @@ -111,6 +113,17 @@ pub struct Compositor { /// Whether the last render drew layers over the program. layered: bool, switch: Option, + /// Each scene with items in slots, by name: where its arrangement put + /// them as of the last render, and their glide there. + reflows: HashMap, +} + +/// A scene's items in slots, arranged. +struct Reflow { + /// Its items where its arrangement puts them, in order. + arranged: Vec, + /// Its items as drawn, gliding to `arranged` when a slot opens or closes. + glide: Glide>, } /// The program and the scenes a transition draws whole on its way, as of the @@ -422,6 +435,7 @@ impl Compositor { program_shown: Duration::ZERO, layered: false, switch: None, + reflows: HashMap::new(), }) } @@ -602,6 +616,22 @@ impl Compositor { Some(picture.size()) } + /// The items of the scene called `scene` where its arrangement put them + /// as of the last [`render`](Self::render), in order, as + /// [`Collection::arranged`] gives them: `None` for a scene with no item + /// in a slot. What is drawn glides there when a slot opens or closes. + pub fn arranged(&self, scene: &str) -> Option<&[Item]> { + self.reflows.get(scene).map(|reflow| reflow.arranged.as_slice()) + } + + /// The items of the scene called `scene` where the last + /// [`render`](Self::render) drew them, in order: those in slots part way + /// along their glide while a reflow is under way, as + /// [`arranged`](Self::arranged) otherwise. + pub fn placed(&self, scene: &str) -> Option<&[Item]> { + self.reflows.get(scene).map(|reflow| reflow.glide.placed()) + } + /// Whether a transition is under way. pub fn transitioning(&self) -> bool { self.switch.is_some() @@ -664,6 +694,16 @@ impl Compositor { None => self.outputs.insert(scene, output), }; } + // Each scene keeps where its slots were, so two swapping names do + // not glide from each other's. + let reflows = std::mem::take(&mut self.reflows); + for (scene, reflow) in reflows { + match renames.get(&scene) { + Some(Some(to)) => self.reflows.insert(to.clone(), reflow), + Some(None) => None, + None => self.reflows.insert(scene, reflow), + }; + } } /// Drop each scene `collection` does not have from preview, program, a @@ -827,6 +867,9 @@ impl Compositor { } self.renders += 1; self.update(collection, sources, now); + // Every draw below reads the scenes as arranged. + let staged = self.reflow(collection, sources, now); + let collection = &*staged; self.restart(collection, preview_start, program_start); self.paint(collection, now); let thumbnails = self.thumbnail_targets(collection, now); @@ -1002,6 +1045,53 @@ impl Compositor { Ok(()) } + /// `collection` with each scene's items in slots where its arrangement + /// puts them at `now`, gliding there as a move transition does when a + /// slot opens or closes. A slot closes while its source is offline or + /// failed in `sources`, and the picture in it is cropped to cover it by + /// the size of its newest picture. + fn reflow<'a>(&mut self, collection: &'a Collection, sources: &Sources, now: Instant) -> Cow<'a, Collection> { + let present = |name: &str| { + let feed = sources.get(collection.capture(name)); + !feed.is_some_and(|feed| matches!(feed.status(), Status::Offline | Status::Failed(_))) + }; + let layers = &self.layers; + let size = |name: &str| match collection.nested(name) { + Some(Nested::Scene(scene)) => collection + .canvas(Some(scene)) + .map(|canvas| (canvas.width, canvas.height)), + Some(Nested::Program) => Some((collection.canvas.width, collection.canvas.height)), + None => { + let layer = layers.get(collection.capture(name))?; + layer.picture.as_ref().map(Picture::size) + } + }; + let mut staged = Cow::Borrowed(collection); + let mut reflows = HashMap::new(); + for (index, scene) in collection.scenes.iter().enumerate() { + let Some(arranged) = collection.arranged(&scene.name, present, size) else { + continue; + }; + let mut reflow = self.reflows.remove(&scene.name).unwrap_or_else(|| Reflow { + arranged: Vec::new(), + // Only the slots glide, so a layer set free that is being + // dragged follows the pointer. + glide: Glide::new(REFLOW, |item| item.slot.is_some()), + }); + // Only the slots start a glide: what each shows, whether it is + // open, and where. + let slotted = arranged.iter().filter(|item| item.slot.is_some()); + let key = slotted + .map(|item| (item.source.clone(), item.visible, item.rect)) + .collect(); + staged.to_mut().scenes[index].items = reflow.glide.show(key, &arranged, now).to_vec(); + reflow.arranged = arranged; + reflows.insert(scene.name.clone(), reflow); + } + self.reflows = reflows; + staged + } + /// Pull the newest picture from every source onto the GPU, and forget the /// sources the collection no longer has or has since edited. An overlay /// that arrives now starts its CSS timeline at `now`. @@ -1456,32 +1546,6 @@ fn stinger(feed: Option<&Feed>, cut: Duration, played: &mut Option, now: In }))) } -/// The crop that trims what `crop` leaves of a `size` picture evenly from two -/// opposite sides to `canvas`'s shape, so it covers the canvas unstretched. -fn cover(crop: Crop, size: (u32, u32), canvas: &Canvas) -> Crop { - let width = size.0 as f32 * (1.0 - crop.left - crop.right); - let height = size.1 as f32 * (1.0 - crop.top - crop.bottom); - let (shown, wanted) = (width / height, canvas.width as f32 / canvas.height as f32); - if !shown.is_finite() || shown <= 0.0 { - return Crop::default(); - } - if shown > wanted { - let trim = (1.0 - wanted / shown) / 2.0; - Crop { - left: trim, - right: trim, - ..Crop::default() - } - } else { - let trim = (1.0 - shown / wanted) / 2.0; - Crop { - top: trim, - bottom: trim, - ..Crop::default() - } - } -} - /// How far moving `amount` of `canvas`'s width or height in `direction` goes, /// in whole pixels so a moving scene stays sharp and two meet without a seam. fn offset(direction: Direction, amount: f32, canvas: &Rect) -> (f32, f32) { @@ -1541,52 +1605,6 @@ impl Pose { } } -/// `old`'s filters `progress` of the way to `new`'s, or `None` if their blend -/// modes differ or their filters do not pair up by kind, in order, as -/// [`Filter::toward`] mixes them. -fn mixed(old: &Item, new: &Item, progress: f32) -> Option> { - if old.blend != new.blend || old.filters.len() != new.filters.len() { - return None; - } - let pairs = old.filters.iter().zip(&new.filters); - pairs.map(|(old, new)| old.toward(new, progress)).collect() -} - -/// The order to draw a move's visible items in, back to front, as `lead` -/// stacks them: each slot is an item of `lead`, of `other`, or a pair of -/// both, as `lead_pairs` and `other_pairs` pair them. An unpaired item of -/// `other` goes just over the paired item it was over in `other`, or under -/// every slot if it was under every paired item. -fn stack( - lead: &[Cow<'_, Item>], - lead_pairs: &[Option], - other: &[Cow<'_, Item>], - other_pairs: &[Option], -) -> Vec<(Option, Option)> { - let mut over: HashMap, Vec> = HashMap::new(); - let mut under = None; - for index in (0..other.len()).filter(|&index| other[index].visible) { - match other_pairs[index] { - Some(_) => under = Some(index), - None => over.entry(under).or_default().push(index), - } - } - let mut unpaired = |under: Option| { - let items = over.remove(&under).unwrap_or_default(); - items.into_iter().map(|index| (None, Some(index))) - }; - - let mut slots: Vec<_> = unpaired(None).collect(); - for index in (0..lead.len()).filter(|&index| lead[index].visible) { - let pair = lead_pairs[index]; - slots.push((Some(index), pair)); - if pair.is_some() { - slots.extend(unpaired(pair)); - } - } - slots -} - /// Refuse an invalid canvas, or one larger than `device` can hold as a /// texture. fn fits(device: &wgpu::Device, canvas: Canvas) -> Result<(), Error> { @@ -1952,7 +1970,7 @@ impl Draw<'_> { }; let crop = self.collection.video.get(source).map(|video| video.crop); let item = Item { - crop: cover(crop.unwrap_or_default(), picture.size(), &canvas), + crop: cover(crop.unwrap_or_default(), picture.size(), &Rect::fill(&canvas)), ..Item::new(source, Rect::fill(&canvas)) }; let pose = Pose::of(self.collection, &item); @@ -2037,17 +2055,10 @@ impl Draw<'_> { let resting = items(to.0); let (leaving, entering) = (posed(from), posed(to)); - // Which item of the other scene each visible item pairs with. - let mut left = vec![None; leaving.len()]; - let mut entered = vec![None; entering.len()]; - for (index, item) in entering.iter().enumerate().filter(|(_, item)| item.visible) { - let pair = (0..leaving.len()) - .find(|&at| left[at].is_none() && leaving[at].visible && leaving[at].source == item.source); - if let Some(at) = pair { - left[at] = Some(index); - entered[index] = Some(at); - } - } + // Which item of the other scene each visible item pairs with: one + // showing the same source, through a role. + let same = |old: &Item, new: &Item| collection.resolve(&old.source) == collection.resolve(&new.source); + let (left, entered) = pair(&leaving, &entering, same); // Each slot is an index into `leaving`, `entering`, or both. let slots: Vec<(Option, Option)> = if progress < 0.5 { @@ -2260,7 +2271,7 @@ mod tests { use std::time::Duration; use super::*; - use crate::collection::{Blend, MaskMode, Rgba, Scene, Size, Video}; + use crate::collection::{Arrangement, Blend, MaskMode, Rgba, Scene, Size, Video}; use crate::source::test::{context, local, start, until, until_time}; /// A fade over a second. @@ -2971,6 +2982,184 @@ mod tests { .await; } + /// A 192x108 scene "slots" arranged in a column: red in the main slot, + /// then blue and green tiles. + fn slotted() -> Collection { + let mut collection = Collection::new(); + collection.canvas = Canvas { + width: 192, + height: 108, + framerate: 30, + }; + for (name, color) in [("red", RED), ("blue", BLUE), ("green", GREEN)] { + let color = Rgba(color); + collection.sources.insert(name.into(), Source::Color { color }); + } + let slot = |source: &str, slot| Item { + slot: Some(slot), + ..Item::new(source, Rect::default()) + }; + collection.scenes = vec![Scene { + items: vec![ + slot("red", crate::Slot::Main), + slot("blue", crate::Slot::Tile), + slot("green", crate::Slot::Tile), + ], + ..Scene::new("slots") + }]; + collection.validate().unwrap(); + collection + } + + /// The center of `rect`, in whole pixels. + fn center(rect: &Rect) -> (usize, usize) { + ( + (rect.x + rect.width / 2.0) as usize, + (rect.y + rect.height / 2.0) as usize, + ) + } + + /// A slot closing reflows the others with a glide on the program clock: + /// the program, which the encoder reads, shows the closed slot's item + /// half faded and the rest part way to their new places halfway through, + /// and the new arrangement once it is over. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_slot_closing_reflows_the_rest_with_a_glide() { + local(async { + let mut studio = Studio::new(slotted()).await; + studio.compositor.set_preview(Some("slots".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + let start = Instant::now(); + studio.render(start); + let before = studio.compositor.arranged("slots").unwrap().to_vec(); + let width = 192; + let program = studio.program(); + for (item, color) in before.iter().zip([RED, BLUE, GREEN]) { + let (x, y) = center(&item.rect); + assert_eq!( + texel(&program, width, x, y), + color, + "{} where it is arranged", + item.source + ); + } + + studio.collection.scenes[0].items[1].visible = false; + studio.render(start); + let after = studio.compositor.arranged("slots").unwrap().to_vec(); + assert!(!after[1].visible, "a hidden item takes no slot"); + assert_ne!(after[2].rect, before[2].rect, "green moves up"); + + studio.render(start.saturating_add(REFLOW / 2)); + let program = studio.program(); + let (x, y) = center(&before[1].rect); + let actual = texel(&program, width, x, y); + assert!( + near(actual, [0, 0, 128, 255]), + "blue half faded where it was: {actual:?}" + ); + let halfway = |from: f32, to: f32| (from + to) / 2.0; + let green = Rect { + x: halfway(before[2].rect.x, after[2].rect.x), + y: halfway(before[2].rect.y, after[2].rect.y), + width: halfway(before[2].rect.width, after[2].rect.width), + height: halfway(before[2].rect.height, after[2].rect.height), + }; + let (x, y) = center(&green); + assert_eq!(texel(&program, width, x, y), GREEN, "green on its way"); + + studio.render(start.saturating_add(REFLOW)); + let program = studio.program(); + let (x, y) = center(&after[2].rect); + assert_eq!(texel(&program, width, x, y), GREEN, "green where it lands"); + let (x, y) = center(&after[0].rect); + assert_eq!(texel(&program, width, x, y), RED); + let (x, y) = center(&before[1].rect); + assert_ne!(texel(&program, width, x, y), BLUE, "blue gone"); + assert_eq!(studio.program(), studio.preview(), "the preview glides alike"); + }) + .await; + } + + /// Two slotted scenes swapping names each keep where their slots were, + /// so neither glides from the other's arrangement. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn scenes_swapping_names_keep_their_own_slots() { + local(async { + let mut collection = slotted(); + let row = Scene { + name: "row".into(), + arrangement: crate::Arrangement::Row, + ..collection.scenes[0].clone() + }; + collection.scenes.push(row); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + studio.render(start); + + let mut draft = studio.collection.draft("Swap"); + draft.rename_scene(0, "swapping").unwrap(); + draft.rename_scene(1, "slots").unwrap(); + draft.rename_scene(0, "row").unwrap(); + let edit = draft.finish().unwrap(); + studio.compositor.rename(&studio.collection.renames(&edit)); + studio.collection.apply(&edit).unwrap(); + studio.render(start.saturating_add(REFLOW / 2)); + for scene in ["slots", "row"] { + let arranged = studio.compositor.arranged(scene).unwrap().to_vec(); + let placed = studio.compositor.placed(scene).unwrap().to_vec(); + assert_eq!(placed, arranged, "`{scene}` is where it was, with no glide"); + } + }) + .await; + } + + /// A slot whose source is offline is closed, so a guest who leaves + /// takes no room, and an unfilled role draws nothing and takes none. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn an_offline_source_and_an_unfilled_role_take_no_slot() { + local(async { + let mut collection = slotted(); + let guest = Source::Broadcast { path: "guest".into() }; + collection.sources.insert("guest".into(), guest); + collection.sources.insert("main".into(), Source::Role { source: None }); + collection.scenes[0].items[0].source = "main".into(); + collection.scenes[0].items[1].source = "guest".into(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("slots".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let arranged = studio.compositor.arranged("slots").unwrap().to_vec(); + assert!(!arranged[0].visible, "an unfilled role takes no slot"); + assert!(!arranged[1].visible, "an offline guest takes no slot"); + let (_, alone) = Arrangement::Column.places(studio.collection.canvas, false, 1); + assert_eq!(arranged[2].rect, alone[0]); + let (x, y) = center(&arranged[2].rect); + assert_eq!(texel(&studio.program(), 192, x, y), GREEN, "green has the canvas"); + + let role = Source::Role { + source: Some("red".into()), + }; + studio.collection.sources.insert("main".into(), role); + let filled = Instant::now(); + studio.render(filled); + studio.render(filled.saturating_add(REFLOW)); + let arranged = studio.compositor.arranged("slots").unwrap().to_vec(); + assert!(arranged[0].visible, "a filled role takes the main slot"); + let (x, y) = center(&arranged[0].rect); + assert_eq!( + texel(&studio.program(), 192, x, y), + RED, + "it shows the source filling it" + ); + }) + .await; + } + /// Keyframes sliding an item in from off the canvas's left over a second. fn slide_in() -> Vec { vec![ @@ -3501,16 +3690,19 @@ mod tests { height: 1920, framerate: 30, }; - let crop = cover(Crop::default(), (1920, 1080), &portrait); + let crop = cover(Crop::default(), (1920, 1080), &Rect::fill(&portrait)); let kept = 1.0 - crop.left - crop.right; assert!((kept - (1080.0 / 1920.0) / (1920.0 / 1080.0)).abs() < 1e-6, "{crop:?}"); assert_eq!((crop.left, crop.top, crop.bottom), (crop.right, 0.0, 0.0)); let landscape = Canvas::default(); - let crop = cover(Crop::default(), (1080, 1920), &landscape); + let crop = cover(Crop::default(), (1080, 1920), &Rect::fill(&landscape)); assert_eq!((crop.left, crop.right), (0.0, 0.0)); assert!(crop.top > 0.0 && crop.top == crop.bottom, "{crop:?}"); - assert_eq!(cover(Crop::default(), (1920, 1080), &landscape), Crop::default()); + assert_eq!( + cover(Crop::default(), (1920, 1080), &Rect::fill(&landscape)), + Crop::default() + ); // A source cropped to a square is trimmed from what it leaves. let square = Crop { @@ -3518,7 +3710,7 @@ mod tests { right: 0.21875, ..Crop::default() }; - let crop = cover(square, (1920, 1080), &landscape); + let crop = cover(square, (1920, 1080), &Rect::fill(&landscape)); assert!((crop.top - 0.21875).abs() < 1e-6, "{crop:?}"); } diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index d523b92a..25893a56 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -600,7 +600,8 @@ impl Draft { /// Remove the source called `name` and everything that refers to it: its /// clones, the items showing either, their audio and video settings, the /// audio scenes holding its sound, the filters reading it, such as a mask, - /// and a stinger playing it, which becomes the default fade. + /// and a stinger playing it, which becomes the default fade. A role it + /// fills is left unfilled. pub fn remove_source(&mut self, name: &str) -> Result<(), Error> { let clones: Vec = self .collection @@ -612,6 +613,17 @@ impl Draft { for clone in clones { self.remove_source(&clone)?; } + // A role it fills is left unfilled, its slots closed up. + let roles: Vec = self + .collection + .sources + .iter() + .filter(|(_, source)| matches!(source, Source::Role { source: Some(source) } if source == name)) + .map(|(role, _)| role.clone()) + .collect(); + for role in roles { + self.set_source(&role, Some(Source::Role { source: None }))?; + } let placed: Vec<(Layer, usize)> = self .collection @@ -1248,13 +1260,18 @@ mod tests { } /// Removing a source removes its clones, every item showing either, and - /// their settings, and one undo puts all of it back. + /// their settings, leaves a role it fills unfilled, and one undo puts all + /// of it back. #[test] fn removing_a_source_cascades_and_undoes_whole() { let mut collection = collection(); collection .sources .insert("red copy".into(), Source::Clone { source: "red".into() }); + let role = Source::Role { + source: Some("red".into()), + }; + collection.sources.insert("main".into(), role); collection.video.insert( "red copy".into(), Video { @@ -1275,7 +1292,8 @@ mod tests { let mut draft = collection.draft("Remove red"); draft.remove_source("red").unwrap(); commit(&mut collection, &mut history, draft, None); - assert!(collection.sources.is_empty()); + let unfilled = Source::Role { source: None }; + assert_eq!(collection.sources.values().collect::>(), vec![&unfilled]); assert!(collection.video.is_empty()); assert!(collection.top.is_empty()); assert!(collection.scenes[0].items.is_empty()); diff --git a/rs/moq-scene/src/glide.rs b/rs/moq-scene/src/glide.rs new file mode 100644 index 00000000..a361d549 --- /dev/null +++ b/rs/moq-scene/src/glide.rs @@ -0,0 +1,379 @@ +//! A glide between two arrangements of one run of items: a +//! [`Transition::Move`](crate::Transition::Move) within a scene, such as its +//! slots reflowing, on the program clock. + +use std::borrow::Borrow; +use std::collections::HashMap; +use std::time::Duration; + +use kio::clock::Instant; + +use crate::collection::{Ease, Filter, Item, Rect, mix}; + +/// How long a reflow takes. +pub(crate) const REFLOW: Duration = Duration::from_millis(400); + +/// A run of items shown as their arrangement changes: once the caller's +/// `K`, naming what moves the items, changes, those `glides` says it moves +/// glide from where they were shown to where they now are, as a move +/// transition takes them; the rest show as they are at once, so one being +/// dragged follows the pointer. +pub(crate) struct Glide { + duration: Duration, + glides: fn(&Item) -> bool, + /// The arrangement shown, or `None` before the first. + key: Option, + /// What was shown when the glide under way started, and when. + from: Option<(Vec, Instant)>, + /// What the last [`show`](Self::show) showed. + shown: Vec, + /// Where the last [`show`](Self::show) drew each item it was given. + placed: Vec, +} + +impl Glide { + /// A glide taking `duration` for the items `glides` picks, from the + /// second arrangement shown on. + pub(crate) fn new(duration: Duration, glides: fn(&Item) -> bool) -> Self { + Self { + duration, + glides, + key: None, + from: None, + shown: Vec::new(), + placed: Vec::new(), + } + } + + /// Each item the last [`show`](Self::show) was given, in its order, where + /// it was drawn: part way along its glide while one is under way. + pub(crate) fn placed(&self) -> &[Item] { + &self.placed + } + + /// The items to draw at `now` for `to`, arranged as `key` names: `to` + /// itself, unless `key` changed within the last glide's time, when they + /// are part way there from what was shown, as [`between`] mixes them. + /// The first arrangement shows at once. + pub(crate) fn show(&mut self, key: K, to: &[Item], now: Instant) -> &[Item] { + if self.key.as_ref() != Some(&key) { + if self.key.is_some() { + self.from = Some((std::mem::take(&mut self.shown), now)); + } + self.key = Some(key); + } + let progress = self.from.as_ref().and_then(|(_, start)| { + let linear = now.saturating_duration_since(*start).as_secs_f32() / self.duration.as_secs_f32(); + (linear < 1.0).then(|| Ease::InOut.apply(linear)) + }); + let same = |a: &Item, b: &Item| a.source == b.source; + let glides = self.glides; + (self.shown, self.placed) = match (progress, &self.from) { + (Some(progress), Some((from, _))) => ( + between(from, to, progress, same, glides), + placed(from, to, progress, same, glides), + ), + _ => { + self.from = None; + (to.to_vec(), to.to_vec()) + } + }; + &self.shown + } +} + +/// `progress` of the way from `from` to `to`, two arrangements of the same +/// run, as a move transition draws them: each visible item of `to` paired by +/// `same` with the first visible item of `from` no earlier one took glides +/// from its rect, crop, and shape there to its own, its filters' settings +/// mixing; the rest of `from` fade out and the rest of `to` fade in where +/// they are. They stack as `from` does until halfway, then as `to` does. +/// A paired item `glides` leaves out shows as it is in `to`. Only visible +/// items are returned. +pub(crate) fn between( + from: &[Item], + to: &[Item], + progress: f32, + same: impl Fn(&Item, &Item) -> bool, + glides: impl Fn(&Item) -> bool, +) -> Vec { + let (left, entered) = pair(from, to, same); + let slots: Vec<(Option, Option)> = if progress < 0.5 { + stack(from, &left, to, &entered) + } else { + let slots = stack(to, &entered, from, &left); + slots.into_iter().map(|(to, from)| (from, to)).collect() + }; + let faded = |item: &Item, opacity: f32| { + let mut item = item.clone(); + item.filters.push(Filter::Opacity { amount: opacity }); + item + }; + slots + .into_iter() + .map(|slot| match slot { + (Some(at), None) => faded(&from[at], 1.0 - progress), + (None, Some(index)) => faded(&to[index], progress), + (Some(_), Some(index)) if !glides(&to[index]) => to[index].clone(), + (Some(at), Some(index)) => { + let (old, new) = (&from[at], &to[index]); + let look = match mixed(old, new, progress) { + Some(filters) => Item { filters, ..new.clone() }, + // A look that cannot be mixed changes as the stacking does. + None if progress < 0.5 => Item { + keyframes: new.keyframes.clone(), + ..old.clone() + }, + None => new.clone(), + }; + Item { + rect: toward(&old.rect, &new.rect, progress), + crop: old.crop.toward(&new.crop, progress), + shape: old.shape.toward(&new.shape, progress), + ..look + } + } + (None, None) => unreachable!("a slot holds an item"), + }) + .collect() +} + +/// Each item of `to` where [`between`] draws it at `progress`: one paired +/// with an item of `from` and picked by `glides` part way from its place +/// there, the rest where they are. +fn placed( + from: &[Item], + to: &[Item], + progress: f32, + same: impl Fn(&Item, &Item) -> bool, + glides: impl Fn(&Item) -> bool, +) -> Vec { + let (_, entered) = pair(from, to, same); + let paired = to.iter().zip(entered); + paired + .map(|(new, at)| match at { + Some(at) if glides(new) => Item { + rect: toward(&from[at].rect, &new.rect, progress), + crop: from[at].crop.toward(&new.crop, progress), + shape: from[at].shape.toward(&new.shape, progress), + ..new.clone() + }, + _ => new.clone(), + }) + .collect() +} + +/// `progress` of the way from `from` to `to`. +fn toward(from: &Rect, to: &Rect, progress: f32) -> Rect { + Rect { + x: mix(from.x, to.x, progress), + y: mix(from.y, to.y, progress), + width: mix(from.width, to.width, progress), + height: mix(from.height, to.height, progress), + } +} + +/// Which item of `entering` each visible item of `leaving` pairs with, and +/// the other way: each visible item of `entering` takes the first visible +/// item of `leaving` that `same` matches and no earlier one took. +pub(crate) fn pair, B: Borrow>( + leaving: &[A], + entering: &[B], + same: impl Fn(&Item, &Item) -> bool, +) -> (Vec>, Vec>) { + let mut left = vec![None; leaving.len()]; + let mut entered = vec![None; entering.len()]; + for (index, item) in entering.iter().enumerate() { + let item = item.borrow(); + if !item.visible { + continue; + } + let pair = (0..leaving.len()).find(|&at| { + let old = leaving[at].borrow(); + left[at].is_none() && old.visible && same(old, item) + }); + if let Some(at) = pair { + left[at] = Some(index); + entered[index] = Some(at); + } + } + (left, entered) +} + +/// `old`'s filters `progress` of the way to `new`'s, or `None` if their blend +/// modes differ or their filters do not pair up by kind, in order, as +/// [`Filter::toward`] mixes them. An opacity one has past the other's last +/// filter, as a glide under way fades an item, mixes with full opacity. +pub(crate) fn mixed(old: &Item, new: &Item, progress: f32) -> Option> { + if old.blend != new.blend { + return None; + } + let opaque = Filter::Opacity { amount: 1.0 }; + let padded = |filters: &[Filter], to: usize| -> Option> { + let mut filters = filters.to_vec(); + match to.checked_sub(filters.len())? { + 0 => {} + 1 => filters.push(opaque.clone()), + _ => return None, + } + Some(filters) + }; + let len = old.filters.len().max(new.filters.len()); + let (from, to) = (padded(&old.filters, len)?, padded(&new.filters, len)?); + if old.filters.len() != new.filters.len() { + let longer = if old.filters.len() > new.filters.len() { + &from + } else { + &to + }; + if !matches!(longer.last(), Some(Filter::Opacity { .. })) { + return None; + } + } + let pairs = from.iter().zip(&to); + pairs.map(|(old, new)| old.toward(new, progress)).collect() +} + +/// The order to draw a move's visible items in, back to front, as `lead` +/// stacks them: each slot is an item of `lead`, of `other`, or a pair of +/// both, as `lead_pairs` and `other_pairs` pair them. An unpaired item of +/// `other` goes just over the paired item it was over in `other`, or under +/// every slot if it was under every paired item. +pub(crate) fn stack, B: Borrow>( + lead: &[A], + lead_pairs: &[Option], + other: &[B], + other_pairs: &[Option], +) -> Vec<(Option, Option)> { + let mut over: HashMap, Vec> = HashMap::new(); + let mut under = None; + for index in (0..other.len()).filter(|&index| other[index].borrow().visible) { + match other_pairs[index] { + Some(_) => under = Some(index), + None => over.entry(under).or_default().push(index), + } + } + let mut unpaired = |under: Option| { + let items = over.remove(&under).unwrap_or_default(); + items.into_iter().map(|index| (None, Some(index))) + }; + + let mut slots: Vec<_> = unpaired(None).collect(); + for index in (0..lead.len()).filter(|&index| lead[index].borrow().visible) { + let pair = lead_pairs[index]; + slots.push((Some(index), pair)); + if pair.is_some() { + slots.extend(unpaired(pair)); + } + } + slots +} + +#[cfg(test)] +mod tests { + use super::*; + + fn at(source: &str, x: f32) -> Item { + Item::new( + source, + Rect { + x, + y: 0.0, + width: 100.0, + height: 100.0, + }, + ) + } + + fn opacity(item: &Item) -> f32 { + match item.filters.last() { + Some(Filter::Opacity { amount }) => *amount, + _ => 1.0, + } + } + + #[test] + fn a_joining_item_fades_in_while_the_others_glide() { + let from = vec![at("main", 0.0), at("cam", 500.0)]; + let to = vec![at("main", 0.0), at("cam", 300.0), at("guest", 600.0)]; + let half = between(&from, &to, 0.5, |a, b| a.source == b.source, |_| true); + assert_eq!(half.len(), 3); + assert_eq!(half[1].rect.x, 400.0); + assert_eq!(half[2].source, "guest"); + assert_eq!(opacity(&half[2]), 0.5); + assert_eq!(opacity(&half[0]), 1.0); + } + + #[test] + fn a_leaving_item_fades_out_where_it_was() { + let from = vec![at("cam", 0.0), at("guest", 600.0)]; + let to = vec![at("cam", 300.0)]; + let quarter = between(&from, &to, 0.25, |a, b| a.source == b.source, |_| true); + let guest = quarter.iter().find(|item| item.source == "guest").unwrap(); + assert_eq!(guest.rect.x, 600.0); + assert_eq!(opacity(guest), 0.75); + } + + /// A glide that starts while one is under way starts from what it showed, + /// so nothing jumps: a guest half faded in carries on from half. + #[test] + fn a_glide_under_way_restarts_from_what_it_showed() { + let start = kio::clock::now(); + let mut glide = Glide::new(Duration::from_millis(400), |_| true); + let one = vec![at("cam", 0.0)]; + let two = vec![at("cam", 0.0), at("guest", 600.0)]; + let three = vec![at("cam", 200.0), at("guest", 600.0)]; + assert_eq!(glide.show(1, &one, start), one.as_slice()); + // It starts with the first draw that sees the new arrangement. + let joined = glide.show(2, &two, start).to_vec(); + assert_eq!(opacity(&joined[1]), 0.0); + let halfway = start.saturating_add(Duration::from_millis(200)); + let half = glide.show(2, &two, halfway).to_vec(); + assert_eq!(opacity(&half[1]), 0.5); + assert_eq!(glide.placed(), two.as_slice(), "nothing moves, so each is where it is"); + // The arrangement changes again at once: the guest goes on from half. + let again = glide.show(3, &three, halfway).to_vec(); + assert_eq!(opacity(&again[1]), 0.5); + assert_eq!(again[0].rect.x, 0.0); + let quarter = halfway.saturating_add(Duration::from_millis(100)); + let drawn = glide + .show(3, &three, quarter) + .iter() + .find(|item| item.source == "cam") + .unwrap() + .rect; + assert_eq!( + glide.placed()[0].rect, + drawn, + "placed where it is drawn, in the order given" + ); + assert!(drawn.x > 0.0 && drawn.x < 200.0); + let later = glide + .show(3, &three, halfway.saturating_add(Duration::from_millis(400))) + .to_vec(); + assert_eq!(later, three); + } + + /// An item the glide leaves out, such as a layer set free being dragged, + /// shows where it is now while the others glide. + #[test] + fn an_item_left_out_shows_where_it_is() { + let start = kio::clock::now(); + let mut glide = Glide::new(REFLOW, |item| item.source != "logo"); + let one = vec![at("cam", 0.0), at("logo", 0.0)]; + let two = vec![at("cam", 200.0), at("logo", 100.0)]; + glide.show(1, &one, start); + glide.show(2, &two, start); + let half = glide.show(2, &two, start.saturating_add(REFLOW / 2)).to_vec(); + assert_eq!(half[1].rect.x, 100.0, "the logo is where it is"); + assert!(half[0].rect.x > 0.0 && half[0].rect.x < 200.0, "the camera glides"); + assert_eq!(glide.placed()[1].rect.x, 100.0); + } + + #[test] + fn the_first_arrangement_shows_at_once() { + let mut glide = Glide::new(REFLOW, |_| true); + let items = vec![at("cam", 0.0)]; + assert_eq!(glide.show((), &items, kio::clock::now()), items.as_slice()); + } +} diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index 722c199c..6055034f 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -44,6 +44,7 @@ //! source that stalls repeats its last picture rather than holding up the //! canvas. +mod arrange; pub mod browser; pub mod clip; mod collection; @@ -51,6 +52,7 @@ mod compositor; mod edit; mod effect; pub mod file; +mod glide; mod grant; pub mod layout; mod lut; @@ -65,8 +67,8 @@ pub mod source; pub use clip::Clip; pub use collection::{ - Audio, AudioScene, Background, Blend, Canvas, Collection, Crop, Direction, Ease, Filter, Item, Keyframe, Layer, - MaskMode, Nested, Rect, Rgba, Scene, Shape, Size, Source, Transition, Video, + Arrangement, Audio, AudioScene, Background, Blend, Canvas, Collection, Crop, Direction, Ease, Filter, Item, + Keyframe, Layer, MaskMode, Nested, Rect, Rgba, Scene, Shape, Size, Slot, Source, Transition, Video, }; pub use compositor::{Compositor, Detail, FORMAT, Output, Thumbnail}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 69c4df6b..bdb66293 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -922,8 +922,8 @@ impl Driver { let plays = self.plays.clone().expect("a replay source has plays"); self.replays(plays, listing).await } - Source::Clone { .. } | Source::Scene { .. } | Source::Program => { - unreachable!("a clone or scene opens no feed") + Source::Clone { .. } | Source::Role { .. } | Source::Scene { .. } | Source::Program => { + unreachable!("a clone, role, or scene opens no feed") } device => self.device(device, listing).await, } @@ -972,6 +972,7 @@ impl Driver { | Source::Browser { .. } | Source::Overlay { .. } | Source::Clone { .. } + | Source::Role { .. } | Source::Lut { .. } | Source::Replay | Source::Scene { .. } @@ -2635,6 +2636,7 @@ impl Source { | Source::Browser { .. } | Source::Overlay { .. } | Source::Clone { .. } + | Source::Role { .. } | Source::Lut { .. } | Source::Scene { .. } | Source::Program => false, diff --git a/rs/moq-studio/shots/slots.toml b/rs/moq-studio/shots/slots.toml new file mode 100644 index 00000000..7dd901dc --- /dev/null +++ b/rs/moq-studio/shots/slots.toml @@ -0,0 +1,63 @@ +# The show the slot views render on: a scene laid out by slots, its main +# picture and camera cast as roles, a guest in a tile, and a logo placed by +# hand over them. Still sources only, so each view draws the same picture. +version = 1 + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.Main] +kind = "role" +source = "game" + +[sources.Camera] +kind = "role" +source = "host" + +[sources.game] +kind = "color" +color = "#3b1d6e" + +[sources.host] +kind = "image" +path = "person.jpg" + +# Guests, as stills: copies of the host, each tinted its own way. +[sources.Dana] +kind = "clone" +source = "host" + +[sources.Ibis] +kind = "clone" +source = "host" + +[sources.backdrop] +kind = "color" +color = "#101828" + +[sources.logo] +kind = "color" +color = "#f59e0b" + +[video.Dana] +filters = [{ kind = "hue_rotate", degrees = 140.0 }] + +[video.Ibis] +filters = [{ kind = "hue_rotate", degrees = 260.0 }] + +[[scenes]] +name = "Guests" +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "Main", slot = "main" }, + { source = "Camera", slot = "tile" }, + { source = "Dana", slot = "tile" }, + { source = "logo", rect = { x = 24, y = 1016, width = 104, height = 40 } }, +] + +[[scenes]] +name = "Just chatting" +arrangement = "inset" +items = [{ source = "Camera", slot = "main" }, { source = "Dana", slot = "tile" }] diff --git a/rs/moq-studio/src/controls.rs b/rs/moq-studio/src/controls.rs index bc818a69..f3382dca 100644 --- a/rs/moq-studio/src/controls.rs +++ b/rs/moq-studio/src/controls.rs @@ -171,6 +171,7 @@ impl Default for Replay { impl Controls { pub fn new(engine: Arc, stream: Stream, runtime: tokio::runtime::Handle) -> Self { + engine.set_guests_appear(stream.guests_appear()); Self { engine, runtime, @@ -369,6 +370,7 @@ impl Controls { /// now if off air, or once the stream ends; says whether they wait for it. /// Otherwise a guests' room that failed is tried again, off air. pub fn set_stream(self: &Arc, stream: Stream, followed_changed: bool) -> bool { + self.engine.set_guests_appear(stream.guests_appear()); { // Marked with the settings it belongs to, so a snapshot never // holds the new relays without knowing they wait to be followed. diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index c8ce20e6..088bf4de 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -107,6 +107,11 @@ pub struct State { /// How many times the collection was reloaded from its files, so the UI /// can let go of what a reload may have removed. pub reloads: u64, + /// How many times an edit the UI did not make, such as a guest appearing + /// on their own, inserted or replaced a layer of a scene, and the count + /// each scene last did so at, so the UI can let go of a layer there whose + /// index it may have given to another. See [`shifted`](Self::shifted). + pub shifts: (u64, BTreeMap), /// How many times the preview was taken to program, by a cut or a /// transition, so the UI can mark each however it was fired. pub takes: u64, @@ -161,6 +166,14 @@ impl State { Ok(()) } + /// Note that an edit the UI did not make inserted or replaced a layer of + /// `scene`. + pub fn shifted(&mut self, scene: &str) { + let (count, scenes) = &mut self.shifts; + *count += 1; + scenes.insert(scene.to_string(), *count); + } + /// Apply `draft`, a change the user did not make, such as a guest joining, /// without recording it: it is not undone, and drops nothing that could be /// redone. See [`History::apply_outside`] for what it may change. @@ -967,6 +980,9 @@ pub struct Engine { /// The path of every room followed this run, so the panic key still mutes /// a guest added from a room left since. rooms: Mutex>, + /// Whether a guest who joins takes the next tile of the scene on program, + /// as the show's settings say. + guests_appear: std::sync::atomic::AtomicBool, /// Each guest's return feed, once the studio's session in the room dials. feeds: Mutex>, /// What the program's broadcasts share, on air or not. @@ -1100,6 +1116,7 @@ impl Engine { history, dropped, reloads: 0, + shifts: (0, BTreeMap::new()), takes: 0, sources, compositor, @@ -1123,6 +1140,7 @@ impl Engine { recorder, room: Mutex::default(), rooms: Mutex::default(), + guests_appear: Default::default(), feeds: Mutex::default(), program, subscribe, @@ -1195,6 +1213,17 @@ impl Engine { self.rooms.lock().unwrap().clone() } + /// Whether a guest who joins takes the next tile of the scene on program + /// on their own, rather than waiting under Add. + pub fn guests_appear(&self) -> bool { + self.guests_appear.load(std::sync::atomic::Ordering::Relaxed) + } + + /// Set whether guests appear on their own, as the show's settings say. + pub fn set_guests_appear(&self, appear: bool) { + self.guests_appear.store(appear, std::sync::atomic::Ordering::Relaxed); + } + /// Keep `room` among the [rooms](Self::rooms) Panic mutes the guests of, /// as one the settings named, even if it could not be joined: a guest of /// it already in a scene may still play through the relays followed. @@ -5450,6 +5479,230 @@ mod tests { .expect("the studio closes"); } + /// A guest whose camera comes online in the room takes the next tile of + /// the scene on program once the show says guests appear on their own, as + /// an edit the host can undo, which a reconnect leaves undone; before + /// that, for their screen, once the host removed them, and while four + /// guests there fill the tiles, they wait under Add. + #[tokio::test(flavor = "multi_thread")] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_guest_appears_in_the_next_tile_once_the_show_says_so() { + use crate::guests::{Joining, Room, tests}; + + let dir = tempfile::tempdir().unwrap(); + let gpu = Gpu::new(wgpu::Backends::all()).await.expect("a GPU"); + let (autosave, _) = Autosave::open(dir.path().join("show.toml")).unwrap(); + let mut loaded = red_canvas(); + let scene = loaded.collection.scenes[0].name.clone(); + let main = &mut loaded.collection.scenes[0].items[0]; + main.slot = Some(moq_scene::Slot::Main); + main.rect = moq_scene::Rect::default(); + let engine = start(gpu, Vec::new(), dir.path(), autosave, loaded) + .await + .expect("the studio starts"); + let connect = url::Url::parse("https://relay.example/show").unwrap(); + let page = url::Url::parse("https://studio.example/join/").unwrap(); + let key = tests::key("show/guests", &["**"], &["**"]); + let room = Room::new(&connect, "guests", key, page, tests::EXPIRY).unwrap(); + engine.join(Some(Joining::offline(room))).unwrap(); + let items = |engine: &Engine| -> Vec<(String, Option)> { + let state = engine.state.lock().unwrap(); + let items = &state.collection().scene(&scene).unwrap().items; + items.iter().map(|item| (item.source.clone(), item.slot)).collect() + }; + let joined = |name: &'static str| { + let engine = engine.clone(); + kio::clock::timeout(TIMEOUT, async move { + while !engine.state.lock().unwrap().collection().sources.contains_key(name) { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }) + }; + + let _bob = engine + .subscribe + .route("guests/bob/camera.hang", Default::default()) + .unwrap(); + joined("bob").await.expect("bob joins"); + assert_eq!( + items(&engine), + [("red".into(), Some(moq_scene::Slot::Main))], + "bob waits under Add" + ); + + engine.set_guests_appear(true); + let _screen = engine + .subscribe + .route("guests/alice/screen.hang", Default::default()) + .unwrap(); + joined("alice screen").await.expect("alice's screen joins"); + let _camera = engine + .subscribe + .route("guests/alice/camera.hang", Default::default()) + .unwrap(); + kio::clock::timeout(TIMEOUT, async { + while items(&engine).len() < 2 { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("alice appears"); + let tiled = [ + ("red".into(), Some(moq_scene::Slot::Main)), + ("alice".into(), Some(moq_scene::Slot::Tile)), + ]; + assert_eq!(items(&engine), tiled, "alice's camera alone, in a tile"); + let undone = engine.state.lock().unwrap().undo().unwrap(); + assert_eq!(undone, Some(format!("`alice` joins `{scene}`"))); + assert_eq!(items(&engine).len(), 1, "an undo takes her out"); + assert!( + engine.state.lock().unwrap().collection().sources.contains_key("alice"), + "and leaves her under Add" + ); + + // Removed from scenes, a guest whose link still works waits under Add. + engine.room().unwrap().stop_renewing("carol").unwrap(); + let _carol = engine + .subscribe + .route("guests/carol/camera.hang", Default::default()) + .unwrap(); + joined("carol").await.expect("carol joins"); + assert_eq!(items(&engine).len(), 1, "a removed guest takes no tile"); + + let shown = |name: &'static str| { + let engine = engine.clone(); + kio::clock::timeout(TIMEOUT, async move { + while !items(&engine).iter().any(|(source, _)| source == name) { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }) + }; + let offline = |name: &'static str| { + let engine = engine.clone(); + kio::clock::timeout(TIMEOUT, async move { + while engine.room().unwrap().participants()[name].camera { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }) + }; + // The host took her off, so rejoining leaves her under Add. + drop(_camera); + offline("alice").await.expect("alice leaves"); + let _camera = engine + .subscribe + .route("guests/alice/camera.hang", Default::default()) + .unwrap(); + kio::clock::timeout(TIMEOUT, async { + while !engine.room().unwrap().participants()["alice"].camera { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("alice is back"); + assert_eq!(items(&engine).len(), 1, "an undone appearance stays undone"); + + // Four guests there fill the tiles; one leaving frees a place, which + // the next takes in his tile, so his coming back makes no fifth. + let mut cameras = Vec::new(); + for name in ["dave", "erin", "frank", "ivy"] { + let path = format!("guests/{name}/camera.hang"); + cameras.push(engine.subscribe.route(path, Default::default()).unwrap()); + shown(name).await.expect("a guest appears"); + } + let _gus = engine + .subscribe + .route("guests/gus/camera.hang", Default::default()) + .unwrap(); + joined("gus").await.expect("gus joins"); + assert!( + !items(&engine).iter().any(|(source, _)| source == "gus"), + "a fifth waits" + ); + // A mix of the host's own holding dave keeps him once his tile goes. + { + let mut state = engine.state.lock().unwrap(); + assert!( + state + .collection() + .heard(moq_scene::AudioScene::DEFAULT) + .contains("dave"), + "dave is heard on program" + ); + let mut scenes = state.collection().audio_scenes.clone(); + let mut headphones = moq_scene::AudioScene::new("Headphones"); + headphones.sounds.insert("dave".into()); + scenes.push(headphones); + let mut draft = state.collection().draft("Headphones"); + draft.set_audio_scenes(scenes).unwrap(); + // And the host hid dave's tile, which the next guest takes shown. + let scene = state.collection().scenes[0].name.clone(); + let at = (state.collection().scenes[0].items.iter()) + .position(|item| item.source == "dave") + .unwrap(); + let hidden = moq_scene::Item { + visible: false, + ..state.collection().scenes[0].items[at].clone() + }; + draft.set_item(&moq_scene::Layer::scene(&scene), at, hidden).unwrap(); + state.commit(draft, None).unwrap(); + } + drop(cameras.remove(0)); + offline("dave").await.expect("dave leaves"); + let shifts = engine.state.lock().unwrap().shifts.0; + let _hal = engine + .subscribe + .route("guests/hal/camera.hang", Default::default()) + .unwrap(); + shown("hal").await.expect("hal takes the place dave left"); + { + let state = engine.state.lock().unwrap(); + let shifted = state.shifts.1.get(&scene).is_some_and(|at| *at > shifts); + assert!( + shifted, + "hal's appearance shifted the scene, so the UI lets go of a layer there" + ); + let hal = state.collection().scenes[0] + .items + .iter() + .find(|item| item.source == "hal"); + assert!( + hal.is_some_and(|hal| hal.visible && hal.auto), + "shown, in a tile of his own" + ); + } + assert!( + !items(&engine).iter().any(|(source, _)| source == "dave"), + "in dave's tile" + ); + { + let state = engine.state.lock().unwrap(); + let heard = |audio: &str| state.collection().heard(audio).contains("dave"); + assert!(!heard(moq_scene::AudioScene::DEFAULT), "nor is he heard on program"); + assert!(heard("Headphones"), "but still in the host's own mix"); + } + let _dave = engine + .subscribe + .route("guests/dave/camera.hang", Default::default()) + .unwrap(); + kio::clock::timeout(TIMEOUT, async { + while !engine.room().unwrap().participants()["dave"].camera { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("dave is back"); + let guests = items(&engine) + .iter() + .filter(|(_, slot)| *slot == Some(moq_scene::Slot::Tile)) + .count(); + assert_eq!(guests, 4, "dave back waits under Add, the tiles full"); + + engine.stop(); + kio::clock::timeout(TIMEOUT, engine.close()) + .await + .expect("the studio closes"); + } + /// A follow that can set up none of the show's relays, such as a relay /// and a failover on one fixed `--connect-bind` port, keeps the sessions /// followed before, and with them the broadcast sources and guests. diff --git a/rs/moq-studio/src/guests.rs b/rs/moq-studio/src/guests.rs index e8a63776..323ec38e 100644 --- a/rs/moq-studio/src/guests.rs +++ b/rs/moq-studio/src/guests.rs @@ -30,10 +30,10 @@ use std::time::Duration; use anyhow::Context; use moq_net::Path; -use moq_scene::{Collection, Draft, Filter, History, Item, Layer, Source, Video}; +use moq_scene::{Collection, Draft, Filter, History, Item, Layer, Slot, Source, Video}; use url::Url; -use crate::engine::Engine; +use crate::engine::{Engine, State}; /// The domain `.github/workflows/join.yml` deploys `js/join` to. const JOIN_PAGE: &str = "https://join.studio.moq.dev/"; @@ -54,6 +54,9 @@ const RETURN: &str = ".return"; /// guest's return feed shows. The room's announcements hide it. pub const PROGRAM: &str = ".program"; +/// The most guests a scene's tiles take on their own; more wait under Add. +pub const MAX_GUESTS: usize = 4; + /// The shortest `--guest-expiry`: half of it must leave room for a renewal to /// reach the guest and the relay before the token it replaces expires. const MIN_EXPIRY: Duration = Duration::from_secs(60); @@ -152,8 +155,8 @@ impl Config { } } -/// Where the guests removed from scenes, and how often each was let back in, -/// are kept: `room.guests.toml` for `room.jwk`. Beside the key, since the +/// Where the guests removed from scenes, how often each was let back in, and +/// who appeared on their own are kept: `room.guests.toml` for `room.jwk`. Beside the key, since the /// tokens they bound are the key's, whichever scene file the studio opens. fn removals_path(key: &std::path::Path) -> PathBuf { let mut name = key.file_stem().unwrap_or_default().to_os_string(); @@ -171,6 +174,10 @@ struct Removals { /// Each guest's [`Participant::generation`], when not zero. #[serde(default)] generation: BTreeMap, + /// The guests who [appeared](Participant::appeared) on their own since + /// their newest link. + #[serde(default, skip_serializing_if = "BTreeSet::is_empty")] + appeared: BTreeSet, } /// The studio's room: who was invited, and what each guest publishes. @@ -211,6 +218,10 @@ pub struct Participant { /// How often a new link let them back in after a removal. Their renewals are /// published under it, so a token from before cannot read them. pub generation: u64, + /// Whether they appeared on their own in a tile since their newest link, + /// across restarts, after which where they are shown is the host's to say: + /// a reconnect never undoes the host taking them off. + pub appeared: bool, } impl Participant { @@ -313,6 +324,9 @@ impl Room { for (identity, generation) in removals.generation { participants.entry(identity).or_default().generation = generation; } + for identity in removals.appeared { + participants.entry(identity).or_default().appeared = true; + } self.removals = Some(path); Ok(self) } @@ -342,11 +356,13 @@ impl Room { let generation = participants .iter() .filter(|(_, participant)| participant.generation > 0); + let appeared = participants.iter().filter(|(_, participant)| participant.appeared); let removals = Removals { removed: removed.map(|(identity, _)| identity.clone()).collect(), generation: generation .map(|(identity, participant)| (identity.clone(), participant.generation)) .collect(), + appeared: appeared.map(|(identity, _)| identity.clone()).collect(), }; let context = || format!("writing {}", path.display()); let dir = match path.parent() { @@ -384,6 +400,7 @@ impl Room { anyhow::ensure!(!name.starts_with('.'), "a guest's name cannot start with `.`"); let mut participants = self.participants.lock().unwrap(); let removed = participants.get(name).is_some_and(|participant| participant.removed); + let appeared = participants.get(name).is_some_and(|participant| participant.appeared); let generation = participants.get(name).map_or(0, |participant| participant.generation); // Refused before anything changes, as a name that cannot be a guest. let (token, expires) = self.sign(name, generation + u64::from(removed))?; @@ -404,6 +421,10 @@ impl Room { .finish(); link.set_fragment(Some(&fragment)); let invite = Invite { link, expires }; + // A new link may appear on its own again. + if appeared { + self.change(&mut participants, name, |participant| participant.appeared = false)?; + } participants.entry(name.to_string()).or_default().invite = Some(invite.clone()); drop(participants); self.changed.send_replace(()); @@ -615,6 +636,14 @@ impl Room { Ok(*draft.collection() != before) } + /// The source in `collection` playing the room-relative broadcast `path`. + fn source_at(&self, collection: &Collection, path: &Path) -> Option { + let played = self.path.join(path).ok()?.to_string(); + let mut sources = collection.sources.iter(); + let found = sources.find(|(_, source)| matches!(source, Source::Broadcast { path } if *path == played)); + found.map(|(name, _)| name.clone()) + } + /// The sources in `collection` that show the guest called `identity`. pub fn sources_of(&self, collection: &Collection, identity: &str) -> Vec { self.guest_sources(collection, identity) @@ -647,6 +676,177 @@ impl Room { fn guest_of(&self, source: &Source) -> Option<(Path, Kind)> { guest_in(&self.path, source) } + + /// Whether `source` plays a broadcast of a guest in the room, announced + /// now. + fn here(&self, source: &Source) -> bool { + let Some((identity, kind)) = self.guest_of(source) else { + return false; + }; + let participants = self.participants.lock().unwrap(); + participants.get(identity.as_str()).is_some_and(|guest| match kind { + Kind::Camera => guest.camera, + Kind::Screen => guest.screen, + }) + } +} + +/// Where `source` would take a tile of `scene`: after its last layer in a +/// slot, so a logo stays in front. `None` for a scene with no slots, or one +/// showing it already, under its name or another playing the same broadcast. +pub fn tile(collection: &Collection, scene: &str, source: &str) -> Option { + let items = &collection.scene(scene)?.items; + let last = items.iter().rposition(|item| item.slot.is_some())?; + let broadcast = |name: &str| match collection.sources.get(collection.capture(name)) { + Some(Source::Broadcast { path }) => Some(path), + _ => None, + }; + let wanted = broadcast(source); + let showing = |item: &Item| { + collection.capture(&item.source) == source || wanted.is_some() && broadcast(&item.source) == wanted + }; + if items.iter().any(showing) { + return None; + } + Some(last + 1) +} + +/// How many guests in `room` the tiles of `scene` show: a guest who left +/// keeps their tile, closed until they are back or [`place`] gives it to +/// another, but not their place among the guests. +fn tiled(collection: &Collection, room: &Room, scene: &str) -> usize { + let items = collection.scene(scene).map_or(&[][..], |scene| &scene.items); + let guests = items.iter().filter(|item| { + let shown = collection.sources.get(collection.capture(&item.source)); + item.slot == Some(Slot::Tile) && shown.is_some_and(|shown| room.here(shown)) + }); + guests.count() +} + +/// Where in its scene the next guest to appear on their own goes. +enum Place { + /// A new tile after the others. + Next, + /// The tile at this index the studio gave a guest who has since left, + /// in place of them. + Taken(usize), +} + +/// Where in `scene` the next guest to appear on their own goes: a new tile +/// while the tiles hold fewer than [`MAX_GUESTS`] guests of the room, here +/// or not, and once they hold that many the [auto](Item::auto) tile of one +/// who has left, so one coming back cannot make a fifth. A tile the host +/// gave a guest stays theirs, even one who once appeared on their own; with +/// none to take there is no place. +fn place(collection: &Collection, room: &Room, scene: &str) -> Option { + let items = &collection.scene(scene)?.items; + let guests: Vec<(usize, &Item, &Source)> = (items.iter().enumerate()) + .filter(|(_, item)| item.slot == Some(Slot::Tile)) + .filter_map(|(index, item)| Some((index, item, collection.sources.get(collection.capture(&item.source))?))) + .filter(|(_, _, shown)| room.guest_of(shown).is_some()) + .collect(); + if guests.len() < MAX_GUESTS { + return Some(Place::Next); + } + let left = guests + .into_iter() + .find(|(_, item, shown)| item.auto && !room.here(shown)); + left.map(|(index, ..)| Place::Taken(index)) +} + +/// The audio scenes with the guest shown by the tile at `index` of `scene` +/// no longer heard on program, as [`appear`] had them, once the studio gives +/// that tile to another, so they are not heard there unseen when back. +/// `None` while a layer elsewhere, in a scene or the top layer, still shows +/// them. The other mixes stay as the host set them. +fn unheard(collection: &Collection, scene: &str, index: usize) -> Option> { + let gone = collection.capture(&collection.scene(scene)?.items.get(index)?.source); + let others = collection.scenes.iter().flat_map(|other| { + let elsewhere = move |(at, _): &(usize, &Item)| other.name != scene || *at != index; + other.items.iter().enumerate().filter(elsewhere).map(|(_, item)| item) + }); + if others + .chain(&collection.top) + .any(|item| collection.capture(&item.source) == gone) + { + return None; + } + let mut scenes = collection.audio_scenes.clone(); + scenes.iter_mut().find(|audio| audio.program)?.sounds.remove(gone); + Some(scenes) +} + +/// Show the guest playing from `source` in the next tile of `scene`, the +/// one on program, and hear them on program, as one edit the host can undo, +/// if [`tile`] finds one and the tiles show fewer than [`MAX_GUESTS`] +/// guests still in `room`, marking them [appeared](Participant::appeared). +/// Once the tiles hold four guests, here or not, they take the tile one who +/// appeared on their own and left had, in place of them, or wait under Add +/// with no such tile; see [`place`]. Says whether it did. +pub fn appear(state: &mut State, room: &Room, scene: Option<&str>, source: &str) -> anyhow::Result { + let Some(scene) = scene else { return Ok(false) }; + if tiled(state.collection(), room, scene) >= MAX_GUESTS { + return Ok(false); + } + let Some(next) = tile(state.collection(), scene, source) else { + return Ok(false); + }; + let layer = Layer::scene(scene); + let Some(place) = place(state.collection(), room, scene) else { + return Ok(false); + }; + let mut draft = state.collection().draft(format!("`{source}` joins `{scene}`")); + // A tile of their own, shown, with nothing of the look the host gave + // whoever had it before. + let item = Item { + slot: Some(Slot::Tile), + auto: true, + ..Item::new(source, moq_scene::Rect::default()) + }; + match place { + Place::Taken(index) => { + if let Some(scenes) = unheard(state.collection(), scene, index) { + draft.set_audio_scenes(scenes)?; + } + draft.set_item(&layer, index, item)?; + } + Place::Next => draft.insert_item(&layer, next, item)?, + } + if state.collection().sources.get(source).is_some_and(Source::has_audio) { + draft.hear(source)?; + } + let identity = state + .collection() + .sources + .get(source) + .and_then(|shown| room.guest_of(shown)); + // Kept before they go on, so a latch that cannot be saved never leaves + // them on air with nothing keeping them off once the host undoes it. + let latch = |appeared: bool| match &identity { + Some((identity, _)) => { + let mut participants = room.participants.lock().unwrap(); + room.change(&mut participants, identity.as_str(), |participant| { + participant.appeared = appeared + }) + } + None => Ok(()), + }; + latch(true)?; + if let Err(err) = state.commit(draft, None) { + if let Err(undone) = latch(false) { + // Forgotten here even so, so they may appear on their next + // join, and the next write of the file forgets it there too. + if let Some((identity, _)) = &identity + && let Some(guest) = room.participants.lock().unwrap().get_mut(identity.as_str()) + { + guest.appeared = false; + } + tracing::warn!(error = format!("{undone:#}"), "cannot forget a guest's appearance"); + } + return Err(err); + } + state.shifted(scene); + Ok(true) } /// Whether `source` plays a guest's camera and microphone, or their screen, in @@ -722,20 +922,40 @@ pub async fn run(engine: Arc, room: Arc, origin: moq_net::origin:: }; let path = announce.prefix; let mut state = engine.state.lock().unwrap(); - let Some((name, source)) = room.observe(&path, online, state.collection(), state.history()) else { - continue; - }; - // Not the host's edit, so it stays out of what they undo and redo. It - // is in no scene or audio scene, so the guest reaches the program only - // once the host places them or puts their sound in an audio scene. - let mut draft = state.collection().draft(format!("Add guest `{name}`")); - let added = draft - .set_source(&name, Some(source)) - .map_err(anyhow::Error::from) - .and_then(|()| state.apply_outside(draft)); - match added { - Ok(()) => tracing::info!(source = %name, %path, "a guest joined"), - Err(err) => tracing::error!(source = %name, error = format!("{err:#}"), "cannot add a guest"), + if let Some((name, source)) = room.observe(&path, online, state.collection(), state.history()) { + // Not the host's edit, so it stays out of what they undo and redo. + // It is in no scene or audio scene, so the guest reaches the + // program only once the host places them or puts their sound in an + // audio scene, or they appear on their own below. + let mut draft = state.collection().draft(format!("Add guest `{name}`")); + let added = draft + .set_source(&name, Some(source)) + .map_err(anyhow::Error::from) + .and_then(|()| state.apply_outside(draft)); + match added { + Ok(()) => tracing::info!(source = %name, %path, "a guest joined"), + Err(err) => tracing::error!(source = %name, error = format!("{err:#}"), "cannot add a guest"), + } + } + // A guest the host removed, or one who appeared once already, stays + // where the host put them until a new link. + let camera = guest(&path).is_some_and(|(identity, kind)| { + let participants = room.participants.lock().unwrap(); + let settled = participants + .get(identity.as_str()) + .is_some_and(|guest| guest.removed || guest.appeared); + kind == Kind::Camera && !settled + }); + if online && camera && engine.guests_appear() { + let Some(name) = room.source_at(state.collection(), &path) else { + continue; + }; + let program = state.compositor.program_scene().map(str::to_string); + match appear(&mut state, &room, program.as_deref(), &name) { + Ok(true) => tracing::info!(source = %name, "a guest appeared on their own"), + Ok(false) => {} + Err(err) => tracing::error!(source = %name, error = format!("{err:#}"), "cannot show a guest"), + } } } tracing::warn!("the room's announcements ended"); @@ -1203,6 +1423,7 @@ pub(crate) mod tests { screen: false, removed: false, generation: 0, + appeared: false, } ); @@ -1484,7 +1705,8 @@ pub(crate) mod tests { /// A removed guest stays unrenewed when the studio restarts, whichever /// scene file it opens, until a new link, whose generation a restart keeps - /// too: the state is the room key's, kept beside it. + /// too, and so does a guest having appeared on their own: the state is + /// the room key's, kept beside it. #[test] fn a_removal_outlasts_a_restart() { let dir = tempfile::tempdir().unwrap(); @@ -1534,6 +1756,126 @@ pub(crate) mod tests { std::fs::remove_dir(&file).unwrap(); room.stop_renewing("alice").unwrap(); assert!(start().participants()["alice"].removed); + + // Appearing on their own outlasts a restart too, until a new link. + let room = start(); + let mut participants = room.participants.lock().unwrap(); + room.change(&mut participants, "bob", |participant| participant.appeared = true) + .unwrap(); + drop(participants); + let room = start(); + assert!(room.participants()["bob"].appeared, "kept across a restart"); + room.invite("bob").unwrap(); + let cleared = start().participants().get("bob").is_none_or(|bob| !bob.appeared); + assert!(cleared, "a new link clears it"); + } + + /// A guest takes the tile after the scene's last layer in a slot, so a + /// logo stays in front; the tiles take four guests on their own, and the + /// host more. A scene with no slots, or showing them already, takes none. + #[test] + fn a_guest_takes_the_next_tile_until_four() { + let room = room("https://relay.example/show", key("show/guests", &["**"], &["**"])).unwrap(); + let mut collection = Collection::new(); + collection + .sources + .insert("Camera".into(), Source::Role { source: None }); + let observe = |collection: &Collection, guest: &str, online: bool| { + let path = format!("{guest}/camera.hang"); + room.observe(&Path::new(&path).unwrap(), online, collection, &History::new()); + }; + let logo = Source::Color { + color: moq_scene::Rgba([255, 255, 255, 255]), + }; + collection.sources.insert("logo".into(), logo); + for guest in ["alice", "bob", "carol", "dave", "erin"] { + let path = format!("guests/{guest}/camera.hang"); + collection.sources.insert(guest.into(), Source::Broadcast { path }); + } + let slot = |source: &str| Item { + slot: Some(Slot::Tile), + ..Item::new(source, moq_scene::Rect::default()) + }; + let fill = moq_scene::Rect::fill(&collection.canvas); + collection.scenes[0].items = vec![slot("Camera"), Item::new("logo", fill)]; + collection.validate().unwrap(); + + assert_eq!(tile(&collection, "Scene", "alice"), Some(1)); + let path = "guests/alice/camera.hang".to_string(); + collection.sources.insert("Alice".into(), Source::Broadcast { path }); + collection.scenes[0].items.insert(1, slot("Alice")); + assert_eq!(tile(&collection, "Scene", "alice"), None, "shown under another name"); + collection.scenes[0].items.remove(1); + collection.sources.remove("Alice"); + for guest in ["alice", "bob", "carol", "dave"] { + observe(&collection, guest, true); + assert!(tiled(&collection, &room, "Scene") < MAX_GUESTS); + let at = tile(&collection, "Scene", guest).unwrap(); + collection.scenes[0].items.insert(at, slot(guest)); + assert_eq!(tile(&collection, "Scene", guest), None, "{guest} is shown"); + } + assert_eq!(collection.scenes[0].items.last().unwrap().source, "logo"); + assert_eq!( + tiled(&collection, &room, "Scene"), + MAX_GUESTS, + "four guests fill the tiles" + ); + observe(&collection, "bob", false); + assert_eq!( + tiled(&collection, &room, "Scene"), + MAX_GUESTS - 1, + "one who left frees a place" + ); + assert_eq!(tile(&collection, "Scene", "erin"), Some(5), "the host can add a fifth"); + // The host placed bob, so his tile stays his, even once he has + // appeared on his own elsewhere. + room.participants.lock().unwrap().get_mut("bob").unwrap().appeared = true; + assert!( + place(&collection, &room, "Scene").is_none(), + "a tile the host gave bob stays his" + ); + collection.scenes[0].items[2].auto = true; + let Some(Place::Taken(index)) = place(&collection, &room, "Scene") else { + panic!("the next guest takes the tile the studio gave bob"); + }; + assert_eq!(index, 2); + observe(&collection, "bob", true); + assert!(place(&collection, &room, "Scene").is_none(), "nor while he is back"); + observe(&collection, "bob", false); + + collection.scenes[0].items.retain(|item| item.slot.is_none()); + assert_eq!(tile(&collection, "Scene", "erin"), None, "a scene with no slots"); + } + + /// A guest whose tile the studio gives away is no longer heard on + /// program, unless a layer elsewhere still shows them, and stays in the + /// host's other mixes. + #[test] + fn a_guest_whose_tile_is_taken_is_heard_only_where_shown() { + let mut collection = Collection::new(); + let path = "guests/dave/camera.hang".to_string(); + collection.sources.insert("dave".into(), Source::Broadcast { path }); + let tile = Item { + slot: Some(Slot::Tile), + auto: true, + ..Item::new("dave", moq_scene::Rect::default()) + }; + collection.scenes[0].items = vec![tile]; + let mut headphones = moq_scene::AudioScene::new("Headphones"); + headphones.sounds.insert("dave".into()); + collection.audio_scenes[0].sounds.insert("dave".into()); + collection.audio_scenes.push(headphones); + collection.validate().unwrap(); + + let scenes = unheard(&collection, "Scene", 0).expect("shown nowhere else"); + assert!(!scenes[0].sounds.contains("dave"), "not on program"); + assert!(scenes[1].sounds.contains("dave"), "still in the headphones"); + + let fill = moq_scene::Rect::fill(&collection.canvas); + let mut other = moq_scene::Scene::new("Other"); + other.items.push(Item::new("dave", fill)); + collection.scenes.push(other); + assert!(unheard(&collection, "Scene", 0).is_none(), "shown in another scene"); } /// A guest joining is not the host's edit: what the host undid can still diff --git a/rs/moq-studio/src/shots.rs b/rs/moq-studio/src/shots.rs index da148c11..32a11e94 100644 --- a/rs/moq-studio/src/shots.rs +++ b/rs/moq-studio/src/shots.rs @@ -110,6 +110,9 @@ enum View { Cued(Page), /// A scene, by name, open for editing off air on BUILD. OffAir(&'static str), + /// BUILD with no layer selected, after the guest `Ibis` appears on their + /// own in the scene on program. + Joined, /// The multiview, as its own window draws it, with the host and the /// highlight picked beside the guests. If `live`, on air in Studio mode /// with `Interview` cued. @@ -174,6 +177,17 @@ fn relayed(view: View) -> Open { } } +/// The studio on the slot views' show, at the mock-ups' size, turned to +/// `view`. +fn slotted(view: View) -> Open { + Open::Studio { + view, + size: SIZE, + show: "slots.toml", + relay: false, + } +} + /// The studio on the MIX views' show, `size` large, turned to MIX in `view` /// with the strip of the sound called `opened` open. fn mixing(view: MixView, opened: Option<&'static str>, size: egui::Vec2) -> Open { @@ -244,6 +258,17 @@ views! { /// BUILD in a window snapped beside a game: the inspector is a tab beside /// Layers and Add. build_compact: Still, Open::Studio { view: BUILD, size: HALF, show: "show.toml", relay: false }; + /// A scene laid out by slots: each slot outlined with the role or guest + /// in it, and the inspector's arrangements and slots, the main picture + /// and camera filled by roles. + build_slots: Still, slotted(View::Build { side: Side::Layers, inspect: Inspect::Layer, index: None }); + /// The same scene once a second guest appears on their own: the column + /// shrinks to take them and the main picture grows, as the `autolayout` + /// mock-up frame has it. + build_slots_joined: Still, slotted(View::Joined); + /// The camera's layer selected in its slot: no handles to resize it, its + /// slot in the inspector, and the source filling its role. + build_slots_layer: Still, slotted(View::Build { side: Side::Layers, inspect: Inspect::Layer, index: Some(2) }); /// BUILD in Studio mode edits the scene on air, with NEXT docked on it. build_studio: Still, studio(View::Cued(Page::Build)); /// A scene open off air: the frame turns amber, and the live picture @@ -934,6 +959,119 @@ fn a_layer_menu_edits_its_layer() { quit(&engine, &runtime); } +/// A Layers row dragged onto a slot on the picture takes that slot, unless +/// a guest appears in the scene mid-drag, whose new tile may have taken the +/// row's index: the drop then does nothing rather than move another layer. +/// It draws no view. +#[test] +#[ignore = "draws on a GPU; run with `just shots`"] +fn a_row_dropped_on_a_slot_after_a_guest_appears_moves_nothing() { + let (_show, runtime, gpu) = lavapipe(); + let _entered = runtime.enter(); + let fixture = Path::new(env!("CARGO_MANIFEST_DIR")); + std::fs::copy(fixture.join("shots/slots.toml"), "show.toml").unwrap(); + let (mut harness, started, _) = window( + start(launch(&gpu), stream(None), &runtime), + &gpu, + &runtime, + SIZE, + Default::default(), + ); + harness.step(); + let App::Studio(studio) = &mut harness.state_mut().app else { + panic!("the studio starts on its first frame") + }; + studio.build(Side::Layers, Inspect::Layer, None); + let engine = started.get().expect("the studio started").clone(); + runtime + .block_on(kio::clock::timeout(SETTLE, settle(&mut harness, &engine))) + .expect("the sources and pictures settle"); + let guests = || { + let state = engine.state.lock().unwrap(); + state.collection().scene("Guests").unwrap().items.clone() + }; + let slot_of = |items: &[moq_scene::Item], source: &str| { + items + .iter() + .find(|item| item.source == source) + .and_then(|item| item.slot) + }; + let pointer = |harness: &mut Harness<'_, Window>, pos: egui::Pos2, pressed: Option| { + let event = match pressed { + None => egui::Event::PointerMoved(pos), + Some(pressed) => egui::Event::PointerButton { + pos, + button: egui::PointerButton::Primary, + pressed, + modifiers: Default::default(), + }, + }; + harness.input_mut().events.push(event); + harness.run_steps(2); + }; + // The logo's row, and the main slot on the picture. + let row = egui::pos2(160.0, 217.0); + let main = egui::pos2(574.0, 400.0); + let grab = |harness: &mut Harness<'_, Window>| { + pointer(harness, row, None); + pointer(harness, row, Some(true)); + pointer(harness, row + egui::vec2(0.0, 12.0), None); + pointer(harness, row + egui::vec2(40.0, 40.0), None); + }; + let release = |harness: &mut Harness<'_, Window>| { + pointer(harness, main, None); + pointer(harness, main, Some(false)); + pointer(harness, egui::pos2(720.0, 880.0), None); + }; + assert_eq!(slot_of(&guests(), "logo"), None, "the logo starts free"); + + // Dropped on the main slot, the logo takes it. + grab(&mut harness); + release(&mut harness); + assert_eq!( + slot_of(&guests(), "logo"), + Some(moq_scene::Slot::Main), + "the row's drop put the logo in the main slot" + ); + harness.get_by_label_contains("Undo: ").click(); + harness.run_steps(2); + assert_eq!(slot_of(&guests(), "logo"), None, "undone"); + let before = guests(); + + // Ibis appears mid-drag, in a tile at the logo's index. + grab(&mut harness); + { + let mut state = engine.state.lock().unwrap(); + let connect = url::Url::parse("https://relay.example/show").unwrap(); + let page = url::Url::parse("https://studio.example/join/").unwrap(); + let key = crate::guests::tests::key("show/room", &["**"], &["**"]); + let room = crate::guests::Room::new(&connect, "room", key, page, crate::guests::tests::EXPIRY).unwrap(); + let appeared = crate::guests::appear(&mut state, &room, Some("Guests"), "Ibis").unwrap(); + assert!(appeared, "Ibis takes a tile"); + } + // Drawn in its tile, the others glided to theirs, still held. + runtime + .block_on(kio::clock::timeout(SETTLE, settle(&mut harness, &engine))) + .expect("the new tile settles"); + release(&mut harness); + let after = guests(); + assert_eq!(after.len(), before.len() + 1); + assert_eq!( + slot_of(&after, "Ibis"), + Some(moq_scene::Slot::Tile), + "Ibis keeps the tile" + ); + assert_eq!( + slot_of(&after, "Main"), + Some(moq_scene::Slot::Main), + "Main keeps its slot" + ); + assert_eq!(slot_of(&after, "logo"), None, "the logo stays free"); + + drop(harness); + quit(&engine, &runtime); +} + /// Studio mode, driven as a person does: its switch turns it on, a pad then /// cues its scene rather than taking it, and TAKE puts it on air. NEXT's Edit /// opens the cued scene off air on BUILD, Back to live keeps it cued, and @@ -1299,6 +1437,7 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show studio.turn(*page); } View::OffAir(scene) => studio.edit_off_air(scene), + View::Joined => studio.build(Side::Layers, Inspect::Layer, None), View::Multiview { live } => { studio.multiview_alone(&["host", "highlight"]); if *live { @@ -1318,6 +1457,15 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show } } let engine = started.get().expect("the studio started").clone(); + if let View::Joined = view { + let mut state = engine.state.lock().unwrap(); + let connect = url::Url::parse("https://relay.example/show").unwrap(); + let page = url::Url::parse("https://studio.example/join/").unwrap(); + let key = crate::guests::tests::key("show/room", &["**"], &["**"]); + let room = crate::guests::Room::new(&connect, "room", key, page, crate::guests::tests::EXPIRY).unwrap(); + let appeared = crate::guests::appear(&mut state, &room, Some("Guests"), "Ibis").unwrap(); + assert!(appeared, "Ibis takes a tile"); + } if let View::OnAir | View::EndStream | View::OnAirHealth diff --git a/rs/moq-studio/src/stream.rs b/rs/moq-studio/src/stream.rs index 8462d541..da556640 100644 --- a/rs/moq-studio/src/stream.rs +++ b/rs/moq-studio/src/stream.rs @@ -45,6 +45,10 @@ pub struct Saved { /// How long a join link works. #[serde(with = "humantime_serde")] pub guest_expiry: Option, + /// Whether a guest who joins takes the next tile of the scene on + /// program, rather than waiting under Add. + #[serde(skip_serializing_if = "std::ops::Not::not")] + pub guests_appear: bool, } impl Saved { @@ -249,6 +253,11 @@ impl Stream { } } + /// Whether guests appear in the scene on program on their own. + pub fn guests_appear(&self) -> bool { + self.saved.guests_appear + } + /// The guests' room, as the flags and the show set it. pub fn guests(&self) -> guests::Config { self.guests_with(&self.saved) diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index dff44596..bc38d72c 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -43,6 +43,7 @@ mod multiview; mod review; mod settings; mod shell; +mod slots; pub(crate) use layout::fit; use levels::Levels; @@ -277,6 +278,7 @@ impl Studio { message: None, faults: Vec::new(), reloads: 0, + shifts: 0, closing: false, levels: Levels::default(), graph: graph::View::default(), @@ -709,6 +711,8 @@ struct Panels { faults: Vec, /// The engine's reloads seen so far. reloads: u64, + /// The engine's [shifts](engine::State::shifts) seen so far. + shifts: u64, /// Whether Close show was clicked, until the window has closed the show. closing: bool, /// Each sound's and audio scene's recent levels, for the meters. @@ -994,7 +998,11 @@ impl Panels { Some(scene) => self.edited(state) != Some(scene.as_str()), None => self.off_air(state).is_some(), }); - if gone { + // A layer of its scene that an edit of the engine's own, such as a + // guest appearing, inserted or replaced may have taken its index. + let shifted = shifted_since(self.selection.as_ref(), &state.shifts, self.shifts); + self.shifts = state.shifts.0; + if gone || shifted { self.selection = None; self.drag = None; self.moving = None; @@ -1870,14 +1878,24 @@ fn color_ui(ui: &mut Ui, color: &mut moq_scene::Rgba) -> egui::Response { response } +/// Whether an edit the UI did not make inserted or replaced a layer of the +/// scene `selection` is in after the UI saw `seen` of the engine's +/// [shifts](engine::State::shifts). +fn shifted_since(selection: Option<&Picked>, shifts: &(u64, BTreeMap), seen: u64) -> bool { + let (count, scenes) = shifts; + let scene = selection.and_then(|picked| picked.layer.scene.as_ref()); + *count != seen && scene.is_some_and(|scene| scenes.get(scene).is_some_and(|at| *at > seen)) +} + /// The source and video settings of a clone of `name`. A clone of a clone /// shows the same capture, so it clones that. It copies the crop and filters, -/// but not a removed background: the clone already shows the cut-out. +/// but not a removed background: the clone already shows the cut-out. For a +/// role, it copies the source filling it, with that look. fn clone_of(collection: &Collection, name: &str) -> (Source, Option