diff --git a/README.md b/README.md index d07ee11a..7721ba61 100644 --- a/README.md +++ b/README.md @@ -383,12 +383,15 @@ window, so there the desktop picks where it opens. - **Where it goes:** the show's relay and its failover relay, each with a switch and its state, such as *Connected, 18 ms away*. A relay switched off - is left out of this stream. + is left out of this stream. A show going out in both formats says each + relay carries both broadcasts. - **Checks**, filled in as they finish: whether each camera has a picture, whether a microphone's level moves, whether the encoder keeps up with the canvas (timed as [setup](#first-launch) times it), and whether the connection to the first relay switched on carries the program (measured as - setup measures it; published scenes and sources share what is left). A failed check is one sentence and its fix, + setup measures it; published scenes and sources share what is left). A show + going out in both formats needs about twice of each: two frames' encoding in + a frame's time, and two videos' bitrate. A failed check is one sentence and its fix, such as **Send 720p instead**, which changes the canvas, or **Check again**. - **GO LIVE TO** the number of destinations switched on. A failed check never stops you; only having none switched on does. @@ -436,7 +439,7 @@ setting costs. Escape or a click outside closes it. | Pane | Use it to | | --- | --- | -| Show | Pick the format, landscape or portrait, which swaps the canvas's sides; the canvas size as 720p, 1080p, 1440p, or 4K, with the upload each wants, or **Custom** for exact numbers; the frame rate as 30, 60, or **Custom**; and, under **Scene sizes**, each scene's own size (see [Published scenes](#published-scenes)). Saved in the scene file, and undone like any edit. | +| Show | Pick the format, landscape or portrait, which swaps the canvas's sides; **Both at once**, which sends a portrait picture beside a landscape one (see [Both formats at once](#both-formats-at-once)); the canvas size as 720p, 1080p, 1440p, or 4K, with the upload each wants, or **Custom** for exact numbers; the frame rate as 30, 60, or **Custom**; and, under **Scene sizes**, each scene's own size (see [Published scenes](#published-scenes)). Saved in the scene file, and undone like any edit. | | Stream | Set the relay, its token, the failover relay, the broadcast's name, and on demand (see [Stream settings](#stream-settings)). | | Recording | Change or open the recordings folder, and pick the picture and the audio scene a recording holds, and whether each source records a file of its own too (see [Recording](#recording)). | | Hotkeys | Bind keys, by what they are for: going live, ending the stream, the panic key, and recording; Studio mode, TAKE, and AUTO; each scene; and replay and clips (see [Hotkeys](#hotkeys)). | @@ -719,6 +722,52 @@ appearance or removing their layer keeps them off when they reconnect, until a new link. Since an appearance is your edit, it drops what you could redo. Off, every guest waits under Add, as before. +### Both formats at once + +A landscape show can go out in portrait too, for phones, beside the +landscape picture for desks and TVs: switch on **Both at once** in Settings, +Show. Each scene of the show's size then has two arrangements of the same +layers, and the studio draws, encodes, and sends both pictures. A scene of its +own size, such as a published portrait scene, stays one picture, in its own +shape, so it cannot go on program meanwhile. A scene given the show's size as +its own follows the show's size instead when you turn it on. + +Turning it on is one edit you can undo, and starts each portrait arrangement +from the landscape one. The largest picture, covering a quarter of the canvas +or more, is the main one: it is cropped to the top 60% of the portrait canvas, +with the largest camera stacked under it, cropped to fill the rest, or fitted +there keeping its outline if it has a round or rounded shape. Without a +camera, the main picture covers the whole portrait canvas. A backdrop covering +the landscape canvas covers the portrait one, cropped. What sits on the main +picture, such as a lower third or a logo, keeps its place on it. A layer set +anywhere else keeps its place on the turned canvas, scaled to fit, until you +move it in portrait. A scene laid out by slots arranges them in a row in +portrait, or in a grid or inset if that is how it arranges them in landscape, +and picks its own portrait arrangement in BUILD's scene inspector. A layer +showing another scene shows that scene's landscape picture in portrait too, +placed and cropped like any other landscape source. + +LIVE shows both pictures side by side, each in its tally frame, with its +format, size, and what its video sends under it. BUILD edits one picture at a +time: **Landscape** or **Portrait** under the picture picks which, and the other +shows small beside it, where a click edits it instead. A drag, Fit, Fill, the +inspector's X, Y, W, H, and crop all place the layer in the picture edited, +and leave its place in the other alone. A layer in a slot has one crop in both. +Each scene's pad shows the pair, and Studio mode's NEXT card shows both +pictures of the scene cued. + +A layer is shown in both pictures, or only in one: **Format** in its +inspector, and the chip under the pictures naming the selected layer, picks +**Both**, **Landscape**, or **Portrait**. A layer shown only in landscape keeps +no portrait place. Switching **Both at once** off removes every layer shown +only in portrait and forgets every portrait place and arrangement, as one +edit. + +Only a landscape canvas turns: picking Portrait in Settings, Show, switches +Both off. The portrait picture goes out as its own broadcast, at +`/portrait` (see [Output](#output)), so a published scene cannot +be named `portrait` meanwhile. + ### Audio scenes Audio from every source is mixed whether or not its scene is live, and cuts @@ -1158,6 +1207,15 @@ items = [ { source = "logo", rect = { x = 1600, y = 40, width = 280, height = 120 } }, ] +[[scenes]] +name = "Game" +portrait = { arrangement = "row" } # its slots in portrait, while the show goes out in both +items = [ + # Its place in the portrait picture, while the show goes out in both. + { source = "game", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1152 }, crop = { left = 0.2, right = 0.2 } } }, + { source = "logo", rect = { x = 1600, y = 40, width = 280, height = 120 }, format = "landscape" }, # shown only in landscape +] + [[scenes]] name = "Portrait" size = { width = 1080, height = 1920 } # its own size; the program's without one @@ -1225,6 +1283,16 @@ default), and `delay`, in every audio scene holding it. `published = ["camera", "mic"]`, a top-level key, names the sources [published independently](#published-independently). +`portrait = true`, a top-level key, sends a portrait picture beside the +landscape one (see [Both formats at once](#both-formats-at-once)). Meanwhile, +in a scene of the show's size or `[[top]]`, a layer's `format`, `landscape` +or `portrait`, shows it only in that picture, both without one, and its +`portrait`, a `rect` and `crop`, places it in the portrait picture, where it +sits on the landscape canvas, fitted to the turned one, without one. A layer +in a slot, or shown only in landscape, takes no `portrait` place, and a +scene's `portrait = { arrangement = "..." }` arranges its slots there, `row` +without one. + Each entry of a scene's `items` is one of its layers, back to front, and each `[[top]]` an always-on-top layer: the `source` it shows and its `rect` on the canvas, in pixels, or, in a scene, its `slot`, `main` (one per scene) or @@ -1319,7 +1387,12 @@ under the broadcast's path, while on air. A canvas under 360 lines has no half size. Each published scene publishes the same way at `/`, at its own size and half of it, its tracks sharing the connection's bandwidth -with the program's, and its own encode of the audio scene it carries. +with the program's, and its own encode of the audio scene it carries. A show +going out in both formats publishes its portrait picture the same way at +`/portrait`, at the canvas turned and half of it, with its own +encode of the audio scene on program. The sources published independently +are drawn into it, with no layer layout of their own. Each relay carries both broadcasts, +and GO LIVE's checks time the encoder and size the connection for both. Timestamps come from the studio's own clock, so a source that stalls repeats its last picture instead of holding up the stream. @@ -1394,6 +1467,7 @@ encode engine for them all. The Health window counts the sessions open: | The program's canvas size | 1 while on air | | Its half size | 1 while someone watches it | | Each published scene's own size and half size | as the program's | +| The portrait picture's, while the show goes out in both | as the program's | | The recording's own size | 1 while recording | | Each captured source or media file | 1 while used, as [above](#source-tracks) | @@ -1771,8 +1845,8 @@ Below that it shows: watches, since the relay pulls nothing otherwise. Source tracks never reach the relay, so they are left out of the media. -Then, for the program's broadcast and each published scene's, each with its -own viewers: +Then, for the program's broadcast, the portrait picture's, and each published +scene's, each with its own viewers: - **Watching**: the sessions subscribed to it, from the relays' `.stats` broadcasts. A relay shares those only with a connection allowed to diff --git a/quest/m0/destinations.md b/quest/m0/destinations.md index 2176be9f..e4b3af68 100644 --- a/quest/m0/destinations.md +++ b/quest/m0/destinations.md @@ -78,6 +78,14 @@ the go-live sheet and live control the studio has, which land them for MoQ relays. This quest adds its destinations to that sheet rather than building a second way to start a stream. +Proposed 2026-11-17 by +[Landscape and portrait at once](https://github.com/moq-dev/studio/pull/200), +waiting on the maintainer there: a show going out in both formats has two +broadcasts, the program's and `/portrait`. MoQ relays carry both, +and each destination here names the one it takes, landscape by default, so a +phone-first platform takes the portrait picture. That picture is another +encoder-sharing key: two destinations taking it share its encoder. + ## Required - [moq's egress dial changes are pinned](/quest/m0/pin-egress-dial.md) - RTMPS that keeps `tcUrl`, and a dial policy the caller supplies diff --git a/quest/m0/ux/README.md b/quest/m0/ux/README.md index 270c6ebc..4d277941 100644 --- a/quest/m0/ux/README.md +++ b/quest/m0/ux/README.md @@ -63,7 +63,7 @@ shows those pictures beside the frames they implement. from capture, dockable into spare room but never docked by default (rejected: a chat column in the main window; leaving chat to the browser). - **Formats:** ✅ one format per show in m0, landscape or portrait, and the - layout follows it. [Both at once](/quest/m1/dual-format.md) is m1 (rejected: + layout follows it. [Both at once](https://github.com/moq-dev/studio/pull/200) is m1 (rejected: designing every m0 page for two pictures). - **Layout follows the picture:** ✅ the picture is larger than any other part of the window but the title bar, as large as the window allows, and diff --git a/quest/m1/auto-place.md b/quest/m1/auto-place.md index 60245891..84519b42 100644 --- a/quest/m1/auto-place.md +++ b/quest/m1/auto-place.md @@ -41,6 +41,15 @@ ready; local first with a hosted model as an opt-in tie-breaker). reflow, eases any list of items from one placement to the next on the program clock, keyed by what changed. +- **In each format:** while the show goes out in landscape and portrait at + once ([moq-dev/studio#200](https://github.com/moq-dev/studio/pull/200)), + the layer keeps out of the way in each picture on its own. The analysis + reads each picture, the slots are each canvas's own, and the slot chosen + is the layer's place in that picture: its `rect` in landscape, its + `portrait` place in portrait, which moq-scene's `Item::placed` sets. A + layer shown in one format only is placed in that one alone. Off in + portrait, it keeps the place turning **Both at once** on gave it. + Measure, per AGENTS.md: the frame time and GPU copies the analysis adds, and that the encoder's load does not move. Should the signals prove too weak, a small on-device model is a later quest, not a hosted one. diff --git a/quest/m1/dual-format-check.md b/quest/m1/dual-format-check.md new file mode 100644 index 00000000..13f16f69 --- /dev/null +++ b/quest/m1/dual-format-check.md @@ -0,0 +1,26 @@ +# [XS] Watch one show on a phone and a desk + +## Goal + +The maintainer has watched one show going out in both formats on a phone and +on a desktop at once, and each reads as made for its screen: the portrait +picture's main picture is cropped around what matters, the camera stacked +under it is not cut off, and overlays sit where a phone viewer sees them. If +it does not, how turning **Both at once** on places the portrait arrangement +changes until it does. + +## Plan + +[Landscape and portrait at once](https://github.com/moq-dev/studio/pull/200) +draws, encodes, and publishes the portrait picture at `/portrait`, +and its GPU and engine tests check the frames and the broadcast. Whether the +derived arrangement reads well on a real phone, and whether a viewer's player +picks the broadcast for its screen, are judgments no test makes. + +## Waiting + +- Maintainer: switch on **Both at once** in Settings, Show, on a show with a + game or a screen as its main picture and a camera, go live to a relay, + and watch `` on a desktop and `/portrait` on a phone. + Done when both read as intended, or the change the derivation needs is + written into this Plan. diff --git a/quest/m1/dual-format-encodes.md b/quest/m1/dual-format-encodes.md new file mode 100644 index 00000000..7c7c65d1 --- /dev/null +++ b/quest/m1/dual-format-encodes.md @@ -0,0 +1,32 @@ +# [S] Time two program encodes for real + +## Goal + +Setup and the go-live sheet say whether this machine keeps up with a show +going out in landscape and portrait at once from two encodes timed together, +not from one encode's time doubled. The numbers come from a real GPU encoder +with its session cap, such as NVENC on `nixos`'s RTX 3070 Ti, beside the +software encoders. + +## Plan + +Decided 2026-11-17 (maintainer, on +[moq-dev/studio#200](https://github.com/moq-dev/studio/pull/200)): ✅ +estimate two encodes as twice one until this lands (rejected: timing two +encodes in that PR). + +- `setup::Encoding::keeps_up_all` and `total_of` double one encode's frame + time and bitrate. Time the landscape and the turned canvas encoding at once + instead, as `setup::time` times one, and keep the doubled estimate only + where two sessions cannot open. +- A check that opens two hardware sessions holds both while it runs, so it + must finish before going live opens the program's, as the one-encode check + already does. +- Measure on `nixos` (NVENC, session cap 8, with CARLA and a transcode job + holding some) and on a software encoder: frame time and encoder load for + one encode and for two, per AGENTS.md. Record whether doubling over- or + under-estimates, and by how much. + +## Hosts + +- linux - the RTX 3070 Ti on `nixos` diff --git a/quest/m1/dual-format.md b/quest/m1/dual-format.md deleted file mode 100644 index 80065cfe..00000000 --- a/quest/m1/dual-format.md +++ /dev/null @@ -1,39 +0,0 @@ -# [L] Landscape and portrait at once - -## Goal - -A show can go out in both formats together. Each scene has a landscape and a -portrait arrangement of the same layers, the LIVE page shows the two -pictures side by side, each in its tally frame, and a scene pad takes both. -A layer says where it is shown: both, landscape, or portrait. The go-live -sheet sends each destination one format. See the `dual` frame of -[`mock/later.html`](/quest/m0/ux/mock/later.html). - -## Plan - -Decided 2026-11-09 (maintainer, /quest-plan): ✅ one format per show in m0 -and both at once here (rejected: designing every m0 page for two pictures; -never planning it). - -The engine can already do it the long way: a scene has its own size, can be -published as a broadcast of its own, and can draw the program through a -program source, cropped. This quest makes that one switch. - -- Settle the model here: one scene with an arrangement per format, or two - scenes linked by name. The first keeps a rename, a reorder, and a take from - ever going out of step. -- A portrait arrangement starts from the landscape one: the main picture - cropped around its subject, the camera stacked under it. It is then the - streamer's to move. -- A layer in a slot keeps no rect, so a scene laid out by slots needs only a - second `arrangement` for portrait (such as `portrait = { arrangement = - "row" }`), and the slots place themselves on the portrait canvas; only its - layers set free need a second rect. -- Each format is its own encode, so it costs an encoder session and its - bitrate; the checklist says whether the machine and the uplink carry both. -- Where a layer is shown is said under the pictures and in the inspector, - never on the layer, so nothing covers either picture. -- The pages follow: BUILD edits one arrangement at a time with the other - beside it, and Studio mode cues both. - -Human check: a phone and a desktop viewer watching the same show. diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index faae0a4d..f0bf54ee 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -136,6 +136,13 @@ impl Clip { }) .chain(drawn) .collect(), + // Copied from a show going out in both formats, its items + // may say where each picture shows them, which pasting + // keeps or drops as the destination does. + portrait: items + .items + .iter() + .any(|item| item.format.is_some() || item.portrait.is_some()), ..Collection::new() } .validate()? @@ -262,6 +269,11 @@ impl Draft { slot => (slot, item.rect), }; let filters = rename(&item.filters); + // Where it shows in portrait means nothing to a layer drawn in one + // format, and a slot places it there too. + let dual = self.collection().dual(layer); + let format = item.format.filter(|_| dual); + let portrait = item.portrait.filter(|_| dual && slot.is_none()); self.insert_item( layer, index, @@ -270,6 +282,8 @@ impl Draft { filters, slot, rect, + format, + portrait, ..item.clone() }, )?; @@ -364,6 +378,32 @@ mod tests { assert_eq!(pasted.top, collection.scenes[0].items); } + /// A layer copied from a show going out in both formats reads back with + /// its portrait place, which a paste into a show going out in landscape + /// alone drops. + #[test] + fn a_layer_with_a_portrait_place_copies() { + let mut collection = collection(); + collection.portrait = true; + let item = &mut collection.scenes[0].items[0]; + item.portrait = Some(crate::Placement { + rect: Rect::fill(&collection.canvas.turned()), + crop: Default::default(), + }); + item.format = Some(crate::Format::Portrait); + collection.validate().unwrap(); + let clip = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)); + let text = clip.to_toml().unwrap(); + let pasted = paste(&collection, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.scenes[0].items[1], collection.scenes[0].items[0]); + let mut landscape = collection.clone(); + landscape.portrait = false; + landscape.scenes[0].items.clear(); + let pasted = paste(&landscape, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.scenes[0].items[0].portrait, None); + assert_eq!(pasted.scenes[0].items[0].format, None); + } + /// A layer the studio placed on its own is the host's once copied. #[test] fn a_copy_is_the_hosts() { diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index a4613cfc..8289e270 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -11,6 +11,7 @@ use serde::{Deserialize, Serialize}; use serde_with::{DeserializeFromStr, SerializeDisplay}; use crate::Error; +use crate::format::{Format, Placement, Portrait}; /// Everything a studio composes: the program's canvas, the sources, and the /// scenes that place them. @@ -24,6 +25,12 @@ pub struct Collection { /// The program's size and frame rate: the size of every scene that sets /// none, and the frame rate every scene is drawn at. pub canvas: Canvas, + /// Whether the show goes out in portrait too, beside the landscape + /// canvas: every scene of the program's size drawn again on the canvas + /// turned on its side, as its [`Portrait`] arrangement and each item's + /// [`portrait`](Item::portrait) place it. Only a landscape canvas turns. + #[serde(default, skip_serializing_if = "is_false")] + pub portrait: bool, /// How the preview is taken to program, short of a cut. #[serde(default)] pub transition: Transition, @@ -77,6 +84,7 @@ impl Collection { pub fn new() -> Self { Self { canvas: Canvas::default(), + portrait: false, transition: Transition::default(), sources: BTreeMap::new(), audio: BTreeMap::new(), @@ -221,9 +229,9 @@ impl Collection { } /// Whether the scene called `scene` can go on program: it is the - /// program's size, and draws no program source, which it would draw into - /// itself. Refused with the reason otherwise, or for a scene that does - /// not exist. + /// program's size, following it while the show goes out in portrait too, + /// and draws no program source, which it would draw into itself. Refused + /// with the reason otherwise, or for a scene that does not exist. pub fn takes(&self, scene: &str) -> Result<(), Error> { match self.refuses(scene) { Some(why) => Err(Error::Invalid(why)), @@ -243,6 +251,13 @@ impl Collection { canvas.width, canvas.height, self.canvas.width, self.canvas.height )); } + // Only a scene that follows the program's size has a portrait picture. + if self.portrait && self.scene(scene).is_some_and(|scene| scene.size.is_some()) { + return Some(format!( + "`{scene}` has a size of its own, so it has no portrait picture and cannot go on program while \ + the show goes out in portrait too" + )); + } if self.draws_program(scene) { return Some(format!("`{scene}` draws the program, so it cannot go on program")); } @@ -287,21 +302,33 @@ impl Collection { } /// The visible items `layers` draw, then those of each scene they draw - /// through a scene source, and so on, each scene once. + /// through a scene source, and so on, each scene once. While the show + /// goes out in both, `layers` draw an item shown in either picture, and a + /// scene source shows the landscape picture, so the scenes they draw only + /// the items shown in landscape. pub fn drawn(&self, layers: impl IntoIterator) -> Vec<&Item> { + self.drawn_through(layers, self) + } + + /// The visible items `layers` draw, then those of each scene they draw + /// through a scene source as `nested` arranges it, and so on, each scene + /// once, such as the portrait picture's, whose scene sources show the + /// landscape picture's. + pub fn drawn_through<'a>(&'a self, layers: impl IntoIterator, nested: &'a Self) -> Vec<&'a Item> { let mut items: Vec<&Item> = layers .into_iter() .flat_map(|layer| self.items(&layer).unwrap_or_default()) .filter(|item| item.visible) .collect(); - let mut nested = BTreeSet::new(); + let mut walked = BTreeSet::new(); let mut next = 0; while let Some(item) = items.get(next) { - if let Some(Nested::Scene(scene)) = self.nested(&item.source) - && nested.insert(scene) - && let Some(scene) = self.scene(scene) + if let Some(Nested::Scene(scene)) = nested.nested(&item.source) + && walked.insert(scene) + && let Some(scene) = nested.scene(scene) { - items.extend(scene.items.iter().filter(|item| item.visible)); + let shown = |item: &&Item| item.visible && item.shown_in(Format::Landscape); + items.extend(scene.items.iter().filter(shown)); } next += 1; } @@ -622,6 +649,7 @@ impl Collection { self.validate_nesting()?; self.validate_published()?; + self.validate_formats()?; // Checked once the scenes it draws are, so a panic never finds out. if let Some(safe) = self.safe() && let Some(why) = self.refuses(safe) @@ -1077,6 +1105,11 @@ pub struct Scene { /// Where its items in a [`Slot`] go. #[serde(default, skip_serializing_if = "Arrangement::is_default")] pub arrangement: Arrangement, + /// How it is arranged in the portrait picture, while the show goes out in + /// [portrait](Collection::portrait). Only a scene of the program's size + /// has one. + #[serde(default, skip_serializing_if = "Portrait::is_default")] + pub portrait: Portrait, /// What the scene draws, back to front. #[serde(default)] pub items: Vec, @@ -1094,6 +1127,7 @@ impl Scene { audio: None, safe: false, arrangement: Arrangement::default(), + portrait: Portrait::default(), items: Vec::new(), } } @@ -1167,7 +1201,7 @@ impl Arrangement { /// Every arrangement. pub const ALL: [Self; 4] = [Self::Column, Self::Row, Self::Grid, Self::Inset]; - fn is_default(&self) -> bool { + pub(crate) fn is_default(&self) -> bool { *self == Self::default() } } @@ -1211,6 +1245,15 @@ pub struct Item { /// or failed, and glides the others to their new places. #[serde(default, skip_serializing_if = "Option::is_none")] pub slot: Option, + /// The one format it is shown in, or `None` for both, while the show goes + /// out in [portrait](Collection::portrait). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub format: Option, + /// Where it is drawn in the portrait picture, or `None` where it sits on + /// the landscape one, scaled to fit, as [`Placement::fitted`] has it. An + /// item in a slot has none, since the portrait arrangement places it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub portrait: Option, /// Whether the studio placed it on its own, as a guest appearing in the /// next tile, rather than the host: once that guest leaves, the next to /// appear may take it. Undo restores it with the rest of the item, and a @@ -1254,6 +1297,8 @@ impl Item { source: source.into(), rect, slot: None, + format: None, + portrait: None, auto: false, visible: true, locked: false, @@ -1276,6 +1321,8 @@ impl Item { Cow::Owned(Self { source: self.source.clone(), slot: self.slot, + format: self.format, + portrait: self.portrait, auto: self.auto, rect: Rect { x: self.rect.x + offset.x, @@ -1873,7 +1920,7 @@ impl Crop { } } - fn validate(&self) -> Result<(), String> { + pub(crate) fn validate(&self) -> Result<(), String> { let edges = [self.left, self.top, self.right, self.bottom]; if edges.iter().any(|edge| !(0.0..1.0).contains(edge)) || self.left + self.right >= 1.0 @@ -1976,7 +2023,7 @@ impl Rect { *self == Self::default() } - fn validate(&self, layer: &Layer, source: &str) -> Result<(), Error> { + pub(crate) fn validate(&self, layer: &Layer, source: &str) -> Result<(), Error> { let finite = [self.x, self.y, self.width, self.height].iter().all(|v| v.is_finite()); if !finite || self.width < 0.0 || self.height < 0.0 { return Err(Error::Invalid(format!( @@ -3466,6 +3513,29 @@ background = "remove" assert!(collection.draws_program("A")); assert!(collection.draws_program("C")); assert!(!collection.draws_program("Scene")); + + // While the show goes out in both, a layer of a scene drawn whole is + // drawn in either picture, but a scene shown through a scene source + // shows its landscape picture alone. + let turned = Rect::fill(&collection.canvas.turned()); + let portrait_only = |source| Item { + format: Some(Format::Portrait), + portrait: Some(Placement { + rect: turned, + crop: Crop::default(), + }), + ..Item::new(source, fill) + }; + collection.portrait = true; + collection.scenes[1].items.push(portrait_only("camera")); + collection.scenes[3].items.push(portrait_only("guest")); + collection.validate().unwrap(); + let drawn: Vec<&str> = collection + .drawn([Layer::scene("A")]) + .into_iter() + .map(|item| item.source.as_str()) + .collect(); + assert_eq!(drawn, ["b", "b", "camera", "logo", "c", "background"]); } /// A scene's own size is one the encoder can take, a published scene is diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 12b40a98..38b474cd 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -17,6 +17,7 @@ use crate::collection::{ }; use crate::dark::{Probe, Prober}; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; +use crate::format::Format; use crate::glide::{Glide, REFLOW, mixed, pair, stack}; use crate::matte::Matte; use crate::overlay::{Overlay, Overlays}; @@ -74,9 +75,12 @@ pub struct Compositor { queue: wgpu::Queue, effects: Effects, - program: Program, - /// The scene in preview, at its own size, as of the last render. - preview: Target, + /// The landscape picture, or the only one. + view: View, + /// The portrait picture, while the show goes out in portrait: the same + /// scenes on program and in preview, through the same transition, each + /// arranged as its portrait arrangement says. + portrait: Option, /// Whether the next render draws the preview, which it skips while no /// one looks. previewing: bool, @@ -86,10 +90,6 @@ pub struct Compositor { outputs: BTreeMap, layers: HashMap, - /// The textures scenes drawn through scene sources are drawn into, by - /// scene: as many as the last render drew it at different moments, - /// reused by the next. - nested: HashMap>, /// Lays out and paints overlays, started with the first one. overlays: Option, /// Reduces captures' pictures to see if they went dark, built with the @@ -114,14 +114,70 @@ pub struct Compositor { preview_shown: Duration, /// How long the program scene had been on program as of the last render. program_shown: Duration, + switch: Option, +} + +/// One picture the show goes out in, as of the last render: its program, its +/// preview, and the scenes they draw. +struct View { + program: Program, + /// The scene in preview, at its own size. + preview: Target, + /// The textures scenes drawn through scene sources are drawn into, by + /// scene: as many as the last render drew it at different moments, + /// reused by the next. + nested: HashMap>, /// Whether the last render drew layers over the program. layered: bool, - switch: Option, /// Each scene with items in slots, by name: where its arrangement put /// them as of the last render, and their glide there. reflows: HashMap, } +impl View { + fn new(device: &wgpu::Device, effects: &Effects, canvas: Canvas) -> Self { + Self { + program: Program::new(device, effects, canvas), + preview: Target::new(device, effects, canvas, "moq-scene preview"), + nested: HashMap::new(), + layered: false, + reflows: HashMap::new(), + } + } + + /// Draw at `canvas` from the next render on: a program of a new size + /// gets new textures of that size, blank until drawn. + fn fit(&mut self, device: &wgpu::Device, effects: &Effects, canvas: Canvas) { + if self.program.canvas.size() == canvas.size() { + self.program.canvas = canvas; + } else { + self.program = Program::new(device, effects, canvas); + } + } + + /// The program as viewers see it: with the layers of each source + /// published independently over it, while it has any showing. + fn layered(&self) -> &Target { + match &self.program.layered { + Some(layered) if self.layered => layered, + _ => &self.program.program, + } + } + + /// Keep where each scene's slots were through `renames`, so two scenes + /// swapping names do not glide from each other's. + fn rename(&mut self, renames: &BTreeMap>) { + let reflows = std::mem::take(&mut self.reflows); + for (scene, reflow) in reflows { + match renames.get(&scene) { + Some(Some(to)) => self.reflows.insert(to.clone(), reflow), + Some(None) => None, + None => self.reflows.insert(scene, reflow), + }; + } + } +} + /// A scene's items in slots, arranged. struct Reflow { /// Its items where its arrangement puts them, in order. @@ -226,6 +282,9 @@ pub enum Thumbnail { /// The source called this: its newest picture through its own crop and /// filters, or what a scene or program source shows. Source(String), + /// The scene called this as the portrait picture shows it, while the + /// show goes out in portrait and the scene is the program's size. + Portrait(String), } /// A thumbnail named by [`Compositor::set_thumbnails`]. @@ -434,8 +493,8 @@ impl Compositor { let effects = Effects::new(device, queue, canvas.width, canvas.height); Ok(Self { - program: Program::new(device, &effects, canvas), - preview: Target::new(device, &effects, canvas, "moq-scene preview"), + view: View::new(device, &effects, canvas), + portrait: None, previewing: true, previewed: false, outputs: BTreeMap::new(), @@ -443,7 +502,6 @@ impl Compositor { queue: queue.clone(), effects, layers: HashMap::new(), - nested: HashMap::new(), overlays: None, prober: None, thumbnails: BTreeMap::new(), @@ -455,15 +513,13 @@ impl Compositor { program_start: None, preview_shown: Duration::ZERO, program_shown: Duration::ZERO, - layered: false, switch: None, - reflows: HashMap::new(), }) } /// The program's canvas. pub fn canvas(&self) -> Canvas { - self.program.canvas + self.view.program.canvas } /// Draw `collection`'s program and scenes at their sizes from the next @@ -475,7 +531,8 @@ impl Compositor { pub fn fit(&mut self, collection: &Collection) -> Result<(), Error> { let program = collection.canvas; let scenes = collection.scenes.iter().map(|scene| scene.canvas(program)); - let canvases: Vec = std::iter::once(program).chain(scenes).collect(); + let portrait = collection.portrait_canvas(); + let canvases: Vec = std::iter::once(program).chain(portrait).chain(scenes).collect(); for canvas in &canvases { fits(&self.device, *canvas)?; } @@ -483,11 +540,15 @@ impl Compositor { let width = canvases.iter().map(|canvas| canvas.width).max().unwrap_or_default(); let height = canvases.iter().map(|canvas| canvas.height).max().unwrap_or_default(); self.effects.fit(width, height); - if self.program.canvas.size() == program.size() { - self.program.canvas = program; - } else { - self.program = Program::new(&self.device, &self.effects, program); - } + self.view.fit(&self.device, &self.effects, program); + self.portrait = match (portrait, self.portrait.take()) { + (None, _) => None, + (Some(canvas), Some(mut view)) => { + view.fit(&self.device, &self.effects, canvas); + Some(view) + } + (Some(canvas), None) => Some(View::new(&self.device, &self.effects, canvas)), + }; Ok(()) } @@ -496,12 +557,20 @@ impl Compositor { /// size, with the pictures of their [layer layout](crate::layout) over /// both. pub fn preview(&self) -> &wgpu::Texture { - &self.preview.texture + &self.view.preview.texture } /// The canvas the preview was last drawn at. pub fn preview_canvas(&self) -> Canvas { - self.preview.canvas + self.view.preview.canvas + } + + /// The scene in preview as the portrait picture shows it, as + /// [`preview`](Self::preview) is, with the canvas it was last drawn at, + /// while the show goes out in portrait. + pub fn portrait_preview(&self) -> Option<(&wgpu::Texture, Canvas)> { + let view = self.portrait.as_ref()?; + Some((&view.preview.texture, view.preview.canvas)) } /// Whether the last [`render`](Self::render) drew the preview. @@ -520,7 +589,7 @@ impl Compositor { /// `moq_video::render::export::Exporter::copy` takes to hand it to an /// encoder. pub fn program(&self) -> &wgpu::Texture { - &self.program.program.texture + &self.view.program.program.texture } /// The program as viewers see it, as of the last [`render`](Self::render): @@ -528,10 +597,19 @@ impl Compositor { /// [`program`](Self::program) itself while it has none showing. Created /// as the program is. pub fn layered(&self) -> &wgpu::Texture { - match &self.program.layered { - Some(layered) if self.layered => &layered.texture, - _ => self.program(), - } + &self.view.layered().texture + } + + /// The portrait picture as viewers see it, as of the last + /// [`render`](Self::render), with its canvas, while the show goes out in + /// portrait: the scene on program, or the transition under way, as its + /// portrait arrangement places it. It carries no + /// [layer layout](crate::layout), so the pictures of sources published + /// independently are drawn into it, as in [`layered`](Self::layered). + /// Created as the program is. + pub fn portrait(&self) -> Option<(&wgpu::Texture, Canvas)> { + let view = self.portrait.as_ref()?; + Some((&view.layered().texture, view.program.canvas)) } /// How long the program scene had been on program as of the last @@ -649,19 +727,32 @@ impl Compositor { } /// The items of the scene called `scene` where its arrangement put them - /// as of the last [`render`](Self::render), in order, as - /// [`Collection::arranged`] gives them: `None` for a scene with no item - /// in a slot. What is drawn glides there when a slot opens or closes. - pub fn arranged(&self, scene: &str) -> Option<&[Item]> { - self.reflows.get(scene).map(|reflow| reflow.arranged.as_slice()) + /// in `format`'s picture as of the last [`render`](Self::render), in + /// order, as [`Collection::arranged`] gives them for + /// [`Collection::formatted`]: `None` for a scene with no item in a slot, + /// or a picture not drawn. What is drawn glides there when a slot opens or + /// closes. + pub fn arranged(&self, format: Format, scene: &str) -> Option<&[Item]> { + let reflow = self.view_of(format)?.reflows.get(scene)?; + Some(reflow.arranged.as_slice()) } /// The items of the scene called `scene` where the last - /// [`render`](Self::render) drew them, in order: those in slots part way - /// along their glide while a reflow is under way, as - /// [`arranged`](Self::arranged) otherwise. - pub fn placed(&self, scene: &str) -> Option<&[Item]> { - self.reflows.get(scene).map(|reflow| reflow.glide.placed()) + /// [`render`](Self::render) drew them in `format`'s picture, in order: + /// those in slots part way along their glide while a reflow is under + /// way, as [`arranged`](Self::arranged) otherwise. + pub fn placed(&self, format: Format, scene: &str) -> Option<&[Item]> { + let reflow = self.view_of(format)?.reflows.get(scene)?; + Some(reflow.glide.placed()) + } + + /// The picture of `format`: the portrait one, or the landscape one, or + /// the only one. + fn view_of(&self, format: Format) -> Option<&View> { + match format { + Format::Landscape => Some(&self.view), + Format::Portrait => self.portrait.as_ref(), + } } /// Whether a transition is under way. @@ -726,15 +817,9 @@ impl Compositor { None => self.outputs.insert(scene, output), }; } - // Each scene keeps where its slots were, so two swapping names do - // not glide from each other's. - let reflows = std::mem::take(&mut self.reflows); - for (scene, reflow) in reflows { - match renames.get(&scene) { - Some(Some(to)) => self.reflows.insert(to.clone(), reflow), - Some(None) => None, - None => self.reflows.insert(scene, reflow), - }; + self.view.rename(renames); + if let Some(portrait) = &mut self.portrait { + portrait.rename(renames); } } @@ -863,7 +948,8 @@ impl Compositor { /// Draw the program, the preview unless no one looks at it, and each /// output being read as of `now`, from the newest picture each of - /// `sources` has. + /// `sources` has, and the portrait picture's program and preview too while + /// the show goes out in portrait. /// /// A source whose picture fails to draw keeps its last one and reports why /// through [`error`](Self::error); this fails only when a canvas itself @@ -872,7 +958,7 @@ impl Compositor { // A scene grown since the last `fit` would overrun what it sized. let (width, height) = self.effects.fitted(); let grown = collection.scenes.iter().find(|scene| { - let canvas = scene.canvas(self.program.canvas); + let canvas = scene.canvas(self.view.program.canvas); canvas.width > width || canvas.height > height }); if let Some(scene) = grown { @@ -881,6 +967,12 @@ impl Compositor { scene.name ))); } + let turned = collection.portrait_canvas(); + if turned.map(|canvas| canvas.size()) != self.portrait.as_ref().map(|view| view.program.canvas.size()) { + return Err(Error::Invalid( + "the portrait picture changed since the compositor was fit".into(), + )); + } // Before the pictures are pulled, so a stinger's clip starting now is // cleared in this frame rather than showing where its last play ended. @@ -890,7 +982,7 @@ impl Compositor { for output in self.outputs.values_mut() { output.start.get_or_insert(now); } - let framerate = self.program.canvas.framerate; + let framerate = self.view.program.canvas.framerate; for thumb in self.thumbnails.values_mut() { let period = u64::from((framerate / thumb.detail.rate.max(1)).max(1)); thumb.due = !thumb.drawn || (self.renders + thumb.phase).is_multiple_of(period); @@ -899,12 +991,24 @@ impl Compositor { } self.renders += 1; self.update(collection, sources, now); - // Every draw below reads the scenes as arranged. - let staged = self.reflow(collection, sources, now); + // Every draw below reads the scenes as arranged, in its own picture. + let staged = reflow( + &mut self.view.reflows, + &self.layers, + collection.formatted(Format::Landscape), + collection, + sources, + now, + ); + let turned = self.portrait.as_mut().map(|view| { + let formatted = collection.formatted(Format::Portrait); + reflow(&mut view.reflows, &self.layers, formatted, collection, sources, now) + }); let collection = &*staged; - self.restart(collection, preview_start, program_start); - self.paint(collection, now); - let thumbnails = self.thumbnail_targets(collection, now); + let turned = turned.as_deref(); + self.restart(collection, turned, preview_start, program_start); + self.paint(collection, turned, now); + let thumbnails = self.thumbnail_targets(collection, turned, now); let preview_since = now.saturating_duration_since(preview_start); self.preview_shown = preview_since; @@ -918,30 +1022,14 @@ impl Compositor { (switch, from, progress) }); - // Each texture at its scene's size: a scene that does not exist, or no - // scene, at the program's. - let program_canvas = self.program.canvas; - let canvas_of = |scene: Option<&str>| collection.canvas(scene).unwrap_or(program_canvas); - let previewing = self.previewing; - let preview_canvas = canvas_of(preview.0); - // The top layer goes over a preview of what could go on program. - let preview_top = preview_canvas.size() == program_canvas.size(); - if previewing { - resize( - &mut self.preview, - &self.device, - &self.effects, - preview_canvas, - "moq-scene preview", - ); - } + let program_canvas = self.view.program.canvas; let mut outputs = Vec::new(); for (scene, output) in &mut self.outputs { output.drawn = output.draw; if !output.draw { continue; } - let canvas = canvas_of(Some(scene)); + let canvas = collection.canvas(Some(scene)).unwrap_or(program_canvas); match &mut output.target { Some(target) => resize(target, &self.device, &self.effects, canvas, "moq-scene output"), None => output.target = Some(Target::new(&self.device, &self.effects, canvas, "moq-scene output")), @@ -953,177 +1041,30 @@ impl Compositor { )); } - // Each texture's scenes and top layer, and the scenes they draw. - let mut roots = Vec::new(); - if previewing { - roots.extend(preview.0.map(|scene| (Place::scene(scene), preview.1))); - if preview_top { - roots.push((Place::top(), Duration::ZERO)); - } - } - for On(scene, since) in on_program(program, switch) { - roots.extend(scene.map(|scene| (Place::scene(scene), since))); - } - roots.push((Place::top(), Duration::ZERO)); - for (On(scene, since), _) in &outputs { - roots.extend(scene.map(|scene| (Place::scene(scene), *since))); - } - for (shows, since, _) in &thumbnails { - if let Shows::Scene(scene) = shows { - roots.push((Place::scene(*scene), *since)); - } - } - // A source's thumbnail draws it whole, as an item of its own. - let wholes: Vec<(Item, Duration)> = thumbnails - .iter() - .filter_map(|(shows, since, target)| match shows { - Shows::Source(source) => Some((Item::new(*source, Rect::fill(&target.canvas)), *since)), - _ => None, - }) - .collect(); - let roots = roots - .iter() - .map(|(place, since)| (collection.items(place).unwrap_or_default(), *since)); - let wholes = wholes.iter().map(|(item, since)| (std::slice::from_ref(item), *since)); - let nesting = Nesting::of(collection, roots.chain(wholes)); - let targets = nested_targets(&mut self.nested, &self.device, &self.effects, collection, &nesting); - - // The pictures viewers draw over the program, which the preview draws - // over its scene too, so both show what viewers would see. - let program_layers = crate::layout::pictures(collection, program.0, true); - let preview_layers = crate::layout::pictures(collection, preview.0, preview_top); - self.layered = program_layers.iter().any(|item| item.visible); - if self.layered && self.program.layered.is_none() { - let target = Target::new(&self.device, &self.effects, program_canvas, "moq-scene layered"); - self.program.layered = Some(target); - } - + let (landscape, portrait): (Vec<_>, Vec<_>) = thumbnails + .into_iter() + .partition(|(format, ..)| *format == Format::Landscape); let mut frame = self.effects.frame(); - let mut draw = Draw { - collection, + let mut pass = Pass { + device: &self.device, effects: &mut self.effects, - library: Library(&self.layers), + layers: &self.layers, frame: &mut frame, - size: program_canvas, - prepared: HashMap::new(), - nesting: &nesting, - scenes: HashMap::new(), - program: None, + previewing: self.previewing, + preview, + program, + switch, }; - // What the program shows is drawn first, then the program, then what - // shows it, then everything else. - let nests = nesting.order.iter().zip(&targets).enumerate(); - let (late, early): (Vec<_>, Vec<_>) = nests.partition(|(_, (nest, _))| nest.program); - for (index, (nest, target)) in early { - draw.nest(index, nest, target); - } - draw.draw_program(&self.program, program, switch); - draw.program = Some(self.program.program.picture.clone()); - for (index, (nest, target)) in late { - draw.nest(index, nest, target); - } - if let Some(layered) = self.program.layered.as_ref().filter(|_| self.layered) { - draw.size = program_canvas; - draw.target(layered); - let canvas = Rect::fill(&program_canvas); - draw.picture(layered, &self.program.program, canvas, canvas, 1.0); - draw.layers(layered, &program_layers, program_since); - } - if previewing { - draw.draw_preview(&self.preview, preview, preview_top); - draw.layers(&self.preview, &preview_layers, preview_since); - } - for (on, target) in &outputs { - draw.size = target.canvas; - draw.target(target); - draw.scene(target, *on); - } - for (shows, since, target) in &thumbnails { - draw.size = target.canvas; - let fill = Rect::fill(&target.canvas); - match *shows { - Shows::Scene(scene) => { - draw.target(target); - draw.scene(target, On(Some(scene), *since)); - } - Shows::Program => { - // What viewers see, as the Program panel shows it: with the - // published layers over it. - let program = self.program.layered.as_ref().filter(|_| self.layered); - draw.target(target); - draw.picture(target, program.unwrap_or(&self.program.program), fill, fill, 1.0); - } - Shows::Preview(_) => { - draw.target(target); - draw.picture(target, &self.preview, fill, fill, 1.0); - } - Shows::Output(scene) => { - let output = outputs.iter().find(|(on, _)| on.0 == Some(scene)); - let (_, output) = output.expect("an output drawn this render"); - draw.target(target); - draw.picture(target, output, fill, fill, 1.0); - } - Shows::Source(source) => { - draw.frame - .target(&target.picture, target.canvas, wgpu::Color::TRANSPARENT); - draw.source(target, source, *since); - } - Shows::Nothing => unreachable!("a thumbnail showing nothing is not drawn"), - } + pass.draw(&mut self.view, collection, collection, &outputs, &strip(landscape)); + if let (Some(view), Some(turned)) = (&mut self.portrait, turned) { + pass.draw(view, turned, collection, &[], &strip(portrait)); } - self.previewed = previewing; + self.previewed = self.previewing; self.effects.submit(frame); Ok(()) } - /// `collection` with each scene's items in slots where its arrangement - /// puts them at `now`, gliding there as a move transition does when a - /// slot opens or closes. A slot closes while its source is offline or - /// failed in `sources`, and the picture in it is cropped to cover it by - /// the size of its newest picture. - fn reflow<'a>(&mut self, collection: &'a Collection, sources: &Sources, now: Instant) -> Cow<'a, Collection> { - let present = |name: &str| { - let feed = sources.get(collection.capture(name)); - !feed.is_some_and(|feed| matches!(feed.status(), Status::Offline | Status::Failed(_))) - }; - let layers = &self.layers; - let size = |name: &str| match collection.nested(name) { - Some(Nested::Scene(scene)) => collection - .canvas(Some(scene)) - .map(|canvas| (canvas.width, canvas.height)), - Some(Nested::Program) => Some((collection.canvas.width, collection.canvas.height)), - None => { - let layer = layers.get(collection.capture(name))?; - layer.picture.as_ref().map(Picture::size) - } - }; - let mut staged = Cow::Borrowed(collection); - let mut reflows = HashMap::new(); - for (index, scene) in collection.scenes.iter().enumerate() { - let Some(arranged) = collection.arranged(&scene.name, present, size) else { - continue; - }; - let mut reflow = self.reflows.remove(&scene.name).unwrap_or_else(|| Reflow { - arranged: Vec::new(), - // Only the slots glide, so a layer set free that is being - // dragged follows the pointer. - glide: Glide::new(REFLOW, |item| item.slot.is_some()), - }); - // Only the slots start a glide: what each shows, whether it is - // open, and where. - let slotted = arranged.iter().filter(|item| item.slot.is_some()); - let key = slotted - .map(|item| (item.source.clone(), item.visible, item.rect)) - .collect(); - staged.to_mut().scenes[index].items = reflow.glide.show(key, &arranged, now).to_vec(); - reflow.arranged = arranged; - reflows.insert(scene.name.clone(), reflow); - } - self.reflows = reflows; - staged - } - /// Pull the newest picture from every source onto the GPU, and forget the /// sources the collection no longer has or has since edited. An overlay /// that arrives now starts its CSS timeline at `now`. @@ -1295,8 +1236,15 @@ impl Compositor { /// do. Program, outputs, and preview share one timeline, so a preview /// never restarts an overlay program or an output shows, through the scene /// a transition leaves, the top layer, or a stinger's masks included, nor - /// does an output going on restart one program shows. - fn restart(&mut self, collection: &Collection, preview_start: Instant, program_start: Instant) { + /// does an output going on restart one program shows. Program and preview + /// are both pictures', `collection` and `turned`, the portrait one. + fn restart( + &mut self, + collection: &Collection, + turned: Option<&Collection>, + preview_start: Instant, + program_start: Instant, + ) { let restarts = |layer: &Layer| matches!(layer.source, Source::Overlay { restart: true, .. }); if !self .layers @@ -1305,15 +1253,23 @@ impl Compositor { { return; } - let program = self.drawn(collection, self.program_scene.as_deref().map(Some)); - let on_air = self.on_air_sources(collection); - let preview = self.drawn(collection, self.preview_scene.as_deref().map(Some)); + let (mut program, mut on_air, mut preview) = (HashSet::new(), HashSet::new(), HashSet::new()); + for picture in std::iter::once(collection).chain(turned) { + program.extend(self.drawn(picture, collection, self.program_scene.as_deref().map(Some))); + on_air.extend(self.on_air_in(picture, collection)); + preview.extend(self.drawn(picture, collection, self.preview_scene.as_deref().map(Some))); + } // One not drawn shows nothing, so it leaves the overlay to the preview. let outputs: Vec<(HashSet<&str>, Instant)> = self .outputs .iter() .filter(|(_, output)| output.draw) - .filter_map(|(scene, output)| Some((self.drawn(collection, [Some(scene.as_str())]), output.start?))) + .filter_map(|(scene, output)| { + Some(( + self.drawn(collection, collection, [Some(scene.as_str())]), + output.start?, + )) + }) .collect(); for (name, layer) in &mut self.layers { let name = name.as_str(); @@ -1344,9 +1300,9 @@ impl Compositor { /// where it moved since its last paint. One drawn nowhere waits: its CSS /// runs on the program clock, so it lands where it should whenever it is /// drawn again. - fn paint(&mut self, collection: &Collection, now: Instant) { + fn paint(&mut self, collection: &Collection, turned: Option<&Collection>, now: Instant) { let Some(overlays) = &self.overlays else { return }; - let shown = self.shown(collection); + let shown = self.shown(collection, turned); let painting: Vec<_> = self .layers .iter() @@ -1385,21 +1341,31 @@ impl Compositor { /// The sources whose pictures the next render draws: each visible item's /// of preview, program, and each output drawn, through the scenes they - /// draw, a stinger's clip under way, and each mask's, through clones. - fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { - let mut shown = self.on_air_sources(collection); - if self.previewing { - shown.extend(self.drawn(collection, [self.preview_scene.as_deref(), None])); + /// draw, in each picture, `collection` and `turned`, the portrait one, a + /// stinger's clip under way, and each mask's, through clones. + fn shown<'a>(&self, collection: &'a Collection, turned: Option<&'a Collection>) -> HashSet<&'a str> { + let mut shown = HashSet::new(); + let landscape = collection; + for picture in std::iter::once(collection).chain(turned) { + shown.extend(self.on_air_in(picture, landscape)); + if self.previewing { + shown.extend(self.drawn(picture, landscape, [self.preview_scene.as_deref(), None])); + } } let outputs = self.outputs.iter().filter(|(_, output)| output.draw); - shown.extend(self.drawn(collection, outputs.map(|(scene, _)| Some(scene.as_str())))); + shown.extend(self.drawn(collection, landscape, outputs.map(|(scene, _)| Some(scene.as_str())))); let thumbnails = self.thumbnails.iter().filter(|(_, thumb)| thumb.due); for (thumbnail, _) in thumbnails { - match self.shows(collection, thumbnail) { - Shows::Scene(scene) => shown.extend(self.drawn(collection, [Some(scene)])), + let (format, shows) = self.shows(collection, turned, thumbnail); + let collection = match format { + Format::Portrait => turned.unwrap_or(collection), + Format::Landscape => collection, + }; + match shows { + Shows::Scene(scene) => shown.extend(self.drawn(collection, landscape, [Some(scene)])), Shows::Source(source) => { if let Some(Nested::Scene(scene)) = collection.nested(source) { - shown.extend(self.drawn(collection, [Some(scene)])); + shown.extend(self.drawn(landscape, landscape, [Some(scene)])); } shown.insert(collection.capture(source)); let video = collection.video.get(source); @@ -1415,49 +1381,69 @@ impl Compositor { shown } - /// What `thumbnail` shows this render: a scene on program, in preview - /// while it is drawn, or drawn for an output is sampled from its texture. - fn shows<'a>(&self, collection: &'a Collection, thumbnail: &Thumbnail) -> Shows<'a> { - let scene = match thumbnail { - Thumbnail::Scene(scene) => scene, + /// What `thumbnail` shows this render, and in which picture: a scene on + /// program, in preview while it is drawn, or drawn for an output is + /// sampled from its texture. A scene in portrait shows nothing unless + /// `turned`, the portrait picture, is drawn and draws it. + fn shows<'a>( + &self, + collection: &'a Collection, + turned: Option<&'a Collection>, + thumbnail: &Thumbnail, + ) -> (Format, Shows<'a>) { + let (format, collection, scene) = match thumbnail { + Thumbnail::Scene(scene) => (Format::Landscape, collection, scene), + Thumbnail::Portrait(scene) => { + let turned = turned.filter(|turned| turned.dual(&Place::scene(scene.as_str()))); + let Some(turned) = turned else { + return (Format::Portrait, Shows::Nothing); + }; + (Format::Portrait, turned, scene) + } Thumbnail::Source(source) => { - return match collection.sources.get_key_value(source) { + let shows = match collection.sources.get_key_value(source) { Some((source, _)) => Shows::Source(source), None => Shows::Nothing, }; + return (Format::Landscape, shows); } }; let Some(scene) = collection.scene(scene).map(|scene| scene.name.as_str()) else { - return Shows::Nothing; + return (format, Shows::Nothing); }; - if self.program_scene.as_deref() == Some(scene) { + let shows = if self.program_scene.as_deref() == Some(scene) { Shows::Program } else if self.previewing && self.preview_scene.as_deref() == Some(scene) { Shows::Preview(scene) - } else if self.outputs.get(scene).is_some_and(|output| output.draw) { + } else if format == Format::Landscape && self.outputs.get(scene).is_some_and(|output| output.draw) { Shows::Output(scene) } else { Shows::Scene(scene) - } + }; + (format, shows) } - /// What each thumbnail due this render shows, from how long ago it was - /// named, and the texture it draws into, sized to what it shows. One - /// showing nothing lets its texture go. + /// What each thumbnail due this render shows, in which picture, from how + /// long ago it was named, and the texture it draws into, sized to what it + /// shows. One showing nothing lets its texture go. fn thumbnail_targets<'a>( &mut self, collection: &'a Collection, + turned: Option<&'a Collection>, now: Instant, - ) -> Vec<(Shows<'a>, Duration, Target)> { - let due: Vec<(Thumbnail, Shows)> = self + ) -> Vec<(Format, Shows<'a>, Duration, Target)> { + let due: Vec<(Thumbnail, (Format, Shows))> = self .thumbnails .iter() .filter(|(_, thumb)| thumb.due) - .map(|(thumbnail, _)| (thumbnail.clone(), self.shows(collection, thumbnail))) + .map(|(thumbnail, _)| (thumbnail.clone(), self.shows(collection, turned, thumbnail))) .collect(); - let program = self.program.canvas; let mut jobs = Vec::with_capacity(due.len()); - for (thumbnail, shows) in due { + for (thumbnail, (format, shows)) in due { + let (collection, program) = match (format, turned, &self.portrait) { + (Format::Portrait, Some(turned), Some(view)) => (turned, view.program.canvas), + _ => (collection, self.view.program.canvas), + }; let thumb = self.thumbnails.get_mut(&thumbnail).expect("listed above"); let since = now.saturating_duration_since(thumb.start.unwrap_or(now)); let canvas = match shows { @@ -1495,17 +1481,35 @@ impl Compositor { _ => Target::sized(&self.device, &self.effects, canvas, size, "moq-scene thumbnail"), }; thumb.target = Some(target.clone()); - jobs.push((shows, since, target)); + jobs.push((format, shows, since, target)); } jobs } /// The sources whose pictures program draws: each visible item's in the - /// scene on program, the scene a transition leaves, and the top layer, - /// through the scenes they draw, a stinger's clip under way, and each - /// mask's, through clones. + /// scene on program, the scene a transition leaves, and the top layer, in + /// each picture the show goes out in, through the scenes they draw, a + /// stinger's clip under way, and each mask's, through clones. pub fn on_air_sources<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { - let mut shown = self.drawn(collection, self.on_air().map(Some).chain([None])); + let pictures: Vec<_> = match collection.portrait { + true => Format::ALL.iter().map(|format| collection.formatted(*format)).collect(), + false => vec![Cow::Borrowed(collection)], + }; + let mut shown = HashSet::new(); + for picture in &pictures { + // Named as `collection` names them, rather than as the pictures do. + let names = self.on_air_in(picture, &pictures[0]).into_iter(); + let names = names.filter_map(|name| collection.sources.get_key_value(name)); + shown.extend(names.map(|(name, _)| name.as_str())); + } + shown + } + + /// The sources whose pictures program draws in one picture, `collection` + /// as it places its scenes and `nested`, the landscape picture, the + /// scenes they draw, as [`on_air_sources`](Self::on_air_sources) says. + fn on_air_in<'a>(&self, collection: &'a Collection, nested: &'a Collection) -> HashSet<&'a str> { + let mut shown = self.drawn(collection, nested, self.on_air().map(Some).chain([None])); // Drawn through its source's filters alone. if let Some((name, _)) = self.stinger().and_then(|name| collection.sources.get_key_value(name)) { shown.insert(name); @@ -1520,18 +1524,19 @@ impl Compositor { } /// The sources whose pictures `scenes` draw, `None` being the top layer: - /// each visible item's, through the scenes they draw, and each mask's, - /// through clones. + /// each visible item's, through the scenes they draw as `nested`, the + /// landscape picture, arranges them, and each mask's, through clones. fn drawn<'a, 'b>( &self, collection: &'a Collection, + nested: &'a Collection, scenes: impl IntoIterator>, ) -> HashSet<&'a str> { let places = scenes.into_iter().map(|scene| Place { scene: scene.map(str::to_string), }); let mut shown = HashSet::new(); - for item in collection.drawn(places) { + for item in collection.drawn_through(places, nested) { shown.insert(collection.capture(&item.source)); masks(collection, &mut shown, collection.filters(item)); } @@ -1539,6 +1544,237 @@ impl Compositor { } } +/// `thumbnails` without the picture each is in. +fn strip<'a>(thumbnails: Vec<(Format, Shows<'a>, Duration, Target)>) -> Vec<(Shows<'a>, Duration, Target)> { + let thumbnails = thumbnails.into_iter(); + thumbnails + .map(|(_, shows, since, target)| (shows, since, target)) + .collect() +} + +/// What a render draws in each picture: the same scenes on program and in +/// preview, onto that picture's textures. +struct Pass<'a, 's> { + device: &'a wgpu::Device, + effects: &'a mut Effects, + layers: &'a HashMap, + frame: &'a mut Frame, + previewing: bool, + preview: On<'s>, + program: On<'s>, + switch: Option<(&'a Switch, On<'s>, f32)>, +} + +impl Pass<'_, '_> { + /// Draw `view`'s program, its preview unless no one looks at it, + /// `outputs`, and `thumbnails`, from `collection` as that picture places + /// its scenes, and the scenes they draw through scene sources as `nested`, + /// the landscape picture, does: a scene source shows a 16:9 picture in + /// both, as the place its layer has in each is shaped for. + fn draw( + &mut self, + view: &mut View, + collection: &Collection, + nested: &Collection, + outputs: &[(On, Target)], + thumbnails: &[(Shows, Duration, Target)], + ) { + let (preview, program, switch) = (self.preview, self.program, self.switch); + // Each texture at its scene's size: a scene that does not exist, or no + // scene, at the program's. + let program_canvas = view.program.canvas; + let canvas_of = |scene: Option<&str>| collection.canvas(scene).unwrap_or(program_canvas); + let previewing = self.previewing; + let preview_canvas = canvas_of(preview.0); + // The top layer goes over a preview of what could go on program. + let preview_top = preview_canvas.size() == program_canvas.size(); + if previewing { + resize( + &mut view.preview, + self.device, + self.effects, + preview_canvas, + "moq-scene preview", + ); + } + + // Each texture's scenes and top layer, and the scenes they draw. + let mut roots = Vec::new(); + if previewing { + roots.extend(preview.0.map(|scene| (Place::scene(scene), preview.1))); + if preview_top { + roots.push((Place::top(), Duration::ZERO)); + } + } + for On(scene, since) in on_program(program, switch) { + roots.extend(scene.map(|scene| (Place::scene(scene), since))); + } + roots.push((Place::top(), Duration::ZERO)); + for (On(scene, since), _) in outputs { + roots.extend(scene.map(|scene| (Place::scene(scene), *since))); + } + for (shows, since, _) in thumbnails { + if let Shows::Scene(scene) = shows { + roots.push((Place::scene(*scene), *since)); + } + } + // A source's thumbnail draws it whole, as an item of its own. + let wholes: Vec<(Item, Duration)> = thumbnails + .iter() + .filter_map(|(shows, since, target)| match shows { + Shows::Source(source) => Some((Item::new(*source, Rect::fill(&target.canvas)), *since)), + _ => None, + }) + .collect(); + let roots = roots + .iter() + .map(|(place, since)| (collection.items(place).unwrap_or_default(), *since)); + let wholes = wholes.iter().map(|(item, since)| (std::slice::from_ref(item), *since)); + let nesting = Nesting::of(nested, roots.chain(wholes)); + let targets = nested_targets(&mut view.nested, self.device, self.effects, nested, &nesting); + + // The pictures viewers draw over the program, which the preview draws + // over its scene too, so both show what viewers would see. + let program_layers = crate::layout::pictures(collection, program.0, true); + let preview_layers = crate::layout::pictures(collection, preview.0, preview_top); + view.layered = program_layers.iter().any(|item| item.visible); + if view.layered && view.program.layered.is_none() { + let target = Target::new(self.device, self.effects, program_canvas, "moq-scene layered"); + view.program.layered = Some(target); + } + + let mut draw = Draw { + collection, + nested, + prepared_nested: HashMap::new(), + effects: self.effects, + library: Library(self.layers), + frame: self.frame, + size: program_canvas, + prepared: HashMap::new(), + nesting: &nesting, + scenes: HashMap::new(), + program: None, + }; + // What the program shows is drawn first, then the program, then what + // shows it, then everything else. + let nests = nesting.order.iter().zip(&targets).enumerate(); + let (late, early): (Vec<_>, Vec<_>) = nests.partition(|(_, (nest, _))| nest.program); + for (index, (nest, target)) in early { + draw.nest(index, nest, target); + } + draw.draw_program(&view.program, program, switch); + draw.program = Some(view.program.program.picture.clone()); + for (index, (nest, target)) in late { + draw.nest(index, nest, target); + } + if let Some(layered) = view.program.layered.as_ref().filter(|_| view.layered) { + draw.size = program_canvas; + draw.target(layered); + let canvas = Rect::fill(&program_canvas); + draw.picture(layered, &view.program.program, canvas, canvas, 1.0); + draw.layers(layered, &program_layers, program.1); + } + if previewing { + draw.draw_preview(&view.preview, preview, preview_top); + draw.layers(&view.preview, &preview_layers, preview.1); + } + for (on, target) in outputs { + draw.size = target.canvas; + draw.target(target); + draw.scene(target, *on); + } + for (shows, since, target) in thumbnails { + draw.size = target.canvas; + let fill = Rect::fill(&target.canvas); + match *shows { + Shows::Scene(scene) => { + draw.target(target); + draw.scene(target, On(Some(scene), *since)); + } + Shows::Program => { + // What viewers see, as the Program panel shows it: with the + // published layers over it. + draw.target(target); + draw.picture(target, view.layered(), fill, fill, 1.0); + } + Shows::Preview(_) => { + draw.target(target); + draw.picture(target, &view.preview, fill, fill, 1.0); + } + Shows::Output(scene) => { + let output = outputs.iter().find(|(on, _)| on.0 == Some(scene)); + let (_, output) = output.expect("an output drawn this render"); + draw.target(target); + draw.picture(target, output, fill, fill, 1.0); + } + Shows::Source(source) => { + draw.frame + .target(&target.picture, target.canvas, wgpu::Color::TRANSPARENT); + draw.source(target, source, *since); + } + Shows::Nothing => unreachable!("a thumbnail showing nothing is not drawn"), + } + } + } +} + +/// `collection` with each scene's items in slots where its arrangement puts +/// them at `now`, gliding there as a move transition does when a slot opens +/// or closes, following `reflows`, one picture's. A slot closes while its +/// source is offline or failed in `sources`, and the picture in it is cropped +/// to cover it by the size of its newest picture in `layers`. +fn reflow<'a>( + reflows: &mut HashMap, + layers: &HashMap, + mut staged: Cow<'a, Collection>, + nested: &Collection, + sources: &Sources, + now: Instant, +) -> Cow<'a, Collection> { + let collection: &Collection = &staged; + let present = |name: &str| { + let feed = sources.get(collection.capture(name)); + !feed.is_some_and(|feed| matches!(feed.status(), Status::Offline | Status::Failed(_))) + }; + // A scene source shows the landscape picture in both, at its size. + let size = |name: &str| match collection.nested(name) { + Some(Nested::Scene(scene)) => nested.canvas(Some(scene)).map(|canvas| (canvas.width, canvas.height)), + Some(Nested::Program) => Some((collection.canvas.width, collection.canvas.height)), + None => { + let layer = layers.get(collection.capture(name))?; + layer.picture.as_ref().map(Picture::size) + } + }; + let mut kept = HashMap::new(); + let mut placed = Vec::new(); + for (index, scene) in collection.scenes.iter().enumerate() { + let Some(arranged) = collection.arranged(&scene.name, present, size) else { + continue; + }; + let mut reflow = reflows.remove(&scene.name).unwrap_or_else(|| Reflow { + arranged: Vec::new(), + // Only the slots glide, so a layer set free that is being + // dragged follows the pointer. + glide: Glide::new(REFLOW, |item| item.slot.is_some()), + }); + // Only the slots start a glide: what each shows, whether it is + // open, and where. + let slotted = arranged.iter().filter(|item| item.slot.is_some()); + let key = slotted + .map(|item| (item.source.clone(), item.visible, item.rect)) + .collect(); + placed.push((index, reflow.glide.show(key, &arranged, now).to_vec())); + reflow.arranged = arranged; + kept.insert(scene.name.clone(), reflow); + } + for (index, items) in placed { + staged.to_mut().scenes[index].items = items; + } + *reflows = kept; + staged +} + /// `target` at `canvas`'s size, made anew if it was another, labeled `label`. fn resize(target: &mut Target, device: &wgpu::Device, effects: &Effects, canvas: Canvas, label: &str) { if target.canvas.size() == canvas.size() { @@ -1870,6 +2106,13 @@ fn moment(collection: &Collection, scene: &str, since: Duration) -> Option { collection: &'a Collection, + /// How the scenes drawn through scene sources are arranged: as + /// `collection` is, or the landscape picture for the portrait one. + nested: &'a Collection, + /// `prepared` for the scenes drawn through scene sources, while they are + /// arranged unlike `collection`, so a scene both drawn whole and through + /// a scene source does not share an item placed differently in each. + prepared_nested: HashMap<(Place, usize, Settled), Option>, effects: &'a mut Effects, library: Library<'a>, frame: &'a mut Frame, @@ -1892,10 +2135,19 @@ impl Draw<'_> { /// Draw the scene `nest` names, the `index`th of the nesting, into /// `target`, transparent where it draws nothing, for items to show. fn nest(&mut self, index: usize, nest: &Nest, target: &Target) { + let root = std::mem::replace(&mut self.collection, self.nested); + let apart = !std::ptr::eq(root, self.nested); + if apart { + std::mem::swap(&mut self.prepared, &mut self.prepared_nested); + } self.size = target.canvas; self.frame.target(&target.picture, self.size, wgpu::Color::TRANSPARENT); self.layer(target, Place::scene(nest.scene), nest.since); self.scenes.insert(index, target.picture.clone()); + if apart { + std::mem::swap(&mut self.prepared, &mut self.prepared_nested); + } + self.collection = root; } /// Draw `preview` showing the scene `on`, under the top layer if `top`. @@ -2931,6 +3183,217 @@ mod tests { .await; } + /// An 8x4 show going out in portrait too, at 4x8. Scene "a": red filling + /// each picture, blue over the landscape one's right half alone, and + /// green over the portrait one's bottom half alone. Scene "b": blue + /// filling the landscape picture, fitted in the portrait one. + fn dual() -> Collection { + let mut collection = collection(); + collection.canvas = Canvas { + width: 8, + height: 4, + framerate: 30, + }; + collection.portrait = true; + collection + .sources + .insert("green".into(), Source::Color { color: Rgba(GREEN) }); + let rect = |x, y, width, height| Rect { x, y, width, height }; + let placed = |rect| { + Some(crate::Placement { + rect, + crop: Crop::default(), + }) + }; + collection.scenes = vec![ + Scene { + items: vec![ + Item { + portrait: placed(rect(0.0, 0.0, 4.0, 8.0)), + ..Item::new("red", rect(0.0, 0.0, 8.0, 4.0)) + }, + Item { + format: Some(Format::Landscape), + ..Item::new("blue", rect(4.0, 0.0, 4.0, 4.0)) + }, + Item { + format: Some(Format::Portrait), + portrait: placed(rect(0.0, 4.0, 4.0, 4.0)), + ..Item::new("green", rect(0.0, 0.0, 8.0, 4.0)) + }, + ], + ..Scene::new("a") + }, + Scene { + items: vec![Item::new("blue", rect(0.0, 0.0, 8.0, 4.0))], + ..Scene::new("b") + }, + ]; + collection.validate().unwrap(); + collection + } + + /// Each picture draws the scene on program as it is arranged there, a + /// layer shown in one format only in that one, and a cut and a fade go + /// through both at once. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn both_pictures_follow_the_program() { + local(async { + let mut studio = Studio::new(dual()).await; + studio.compositor.fit(&studio.collection).unwrap(); + let portrait = |studio: &mut Studio| { + let (texture, canvas) = studio.compositor.portrait().unwrap(); + assert_eq!((canvas.width, canvas.height), (4, 8)); + let texture = texture.clone(); + studio.read(&texture) + }; + let start = Instant::now(); + studio.compositor.set_preview(Some("a".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(start); + let landscape = studio.program(); + assert_eq!(pixel(&landscape, 1, 1), RED); + assert_eq!(pixel(&landscape, 6, 1), BLUE, "shown in landscape"); + let turned = portrait(&mut studio); + assert_eq!(texel(&turned, 4, 1, 1), RED); + assert_eq!(texel(&turned, 4, 1, 6), GREEN, "shown in portrait"); + assert!(!turned.as_chunks::<4>().0.contains(&BLUE), "shown in landscape alone"); + + // "b" fills the landscape picture and is fitted, 4x2, across the + // middle of the portrait one. + studio.compositor.set_preview(Some("b".into())); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); + studio.render(start.saturating_add(Duration::from_millis(500))); + let turned = portrait(&mut studio); + let fading = texel(&turned, 4, 1, 6); + assert!((126..=129).contains(&fading[1]) && fading[0] == 0, "{fading:?}"); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(start.saturating_add(Duration::from_millis(600))); + let turned = portrait(&mut studio); + assert_eq!(texel(&turned, 4, 1, 3), BLUE); + assert_eq!(texel(&turned, 4, 1, 1), BLACK); + assert_eq!(texel(&turned, 4, 1, 6), BLACK); + }) + .await; + } + + /// A layer showing a scene of the program's size shows its landscape + /// picture in the portrait one too, the 16:9 picture its portrait place + /// is shaped for, rather than the scene's portrait arrangement squeezed + /// into it. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_scene_source_shows_the_landscape_picture_in_portrait() { + local(async { + let mut collection = dual(); + collection + .sources + .insert("scene a".into(), Source::Scene { scene: "a".into() }); + let rect = |x, y, width, height| Rect { x, y, width, height }; + collection.scenes.push(Scene { + items: vec![Item { + portrait: Some(crate::Placement { + rect: rect(0.0, 3.0, 4.0, 2.0), + crop: Crop::default(), + }), + ..Item::new("scene a", rect(0.0, 0.0, 8.0, 4.0)) + }], + ..Scene::new("c") + }); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.fit(&studio.collection).unwrap(); + studio.compositor.set_preview(Some("c".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let landscape = studio.program(); + assert_eq!(pixel(&landscape, 1, 1), RED); + assert_eq!(pixel(&landscape, 6, 1), BLUE); + let (texture, _) = studio.compositor.portrait().unwrap(); + let texture = texture.clone(); + let turned = studio.read(&texture); + // "a" in landscape: red, blue over its right half. + assert_eq!(texel(&turned, 4, 0, 3), RED); + assert_eq!(texel(&turned, 4, 3, 4), BLUE); + assert_eq!(texel(&turned, 4, 1, 1), BLACK); + assert!(!turned.as_chunks::<4>().0.contains(&GREEN), "a's portrait-only layer"); + }) + .await; + } + + /// A slot showing a scene in the portrait picture is cropped for the + /// landscape picture the scene source shows there. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_slot_showing_a_scene_crops_its_landscape_picture_in_portrait() { + local(async { + let mut collection = dual(); + collection + .sources + .insert("scene b".into(), Source::Scene { scene: "b".into() }); + collection.scenes.push(Scene { + items: vec![Item { + slot: Some(crate::Slot::Main), + ..Item::new("scene b", Rect::default()) + }], + ..Scene::new("s") + }); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.fit(&studio.collection).unwrap(); + studio.render(Instant::now()); + let turned = studio.collection.formatted(Format::Portrait); + let landscape = turned.arranged("s", |_| true, |_| Some((8, 4))).unwrap(); + let portrait = turned.arranged("s", |_| true, |_| Some((4, 8))).unwrap(); + assert_ne!(landscape, portrait); + let arranged = studio.compositor.arranged(Format::Portrait, "s").unwrap(); + assert_eq!(arranged, landscape.as_slice()); + }) + .await; + } + + /// A scene's portrait thumbnail draws it as the portrait picture + /// arranges it, and the portrait preview the scene in preview. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_portrait_thumbnail_and_preview_draw_the_portrait_arrangement() { + local(async { + let mut studio = Studio::new(dual()).await; + studio.compositor.fit(&studio.collection).unwrap(); + let detail = Detail { side: 8, rate: 30 }; + studio.compositor.set_thumbnails([ + (Thumbnail::Portrait("a".into()), detail), + (Thumbnail::Scene("a".into()), detail), + ]); + studio.compositor.set_preview(Some("a".into())); + studio.render(Instant::now()); + let texture = studio + .compositor + .thumbnail(&Thumbnail::Portrait("a".into())) + .unwrap() + .clone(); + assert_eq!((texture.width(), texture.height()), (4, 8)); + let thumbnail = studio.read(&texture); + assert_eq!(texel(&thumbnail, 4, 1, 1), RED); + assert_eq!(texel(&thumbnail, 4, 1, 6), GREEN); + let texture = studio + .compositor + .thumbnail(&Thumbnail::Scene("a".into())) + .unwrap() + .clone(); + assert_eq!((texture.width(), texture.height()), (8, 4)); + let (texture, _) = studio.compositor.portrait_preview().unwrap(); + let texture = texture.clone(); + let preview = studio.read(&texture); + assert_eq!(texel(&preview, 4, 1, 6), GREEN); + }) + .await; + } + /// Halfway through a slide to the left, eased to half the canvas, the /// outgoing scene's right half shows on the left and the incoming scene's /// left half on the right; a wipe leaves the outgoing scene in place and @@ -3118,7 +3581,7 @@ mod tests { studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); let start = Instant::now(); studio.render(start); - let before = studio.compositor.arranged("slots").unwrap().to_vec(); + let before = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); let width = 192; let program = studio.program(); for (item, color) in before.iter().zip([RED, BLUE, GREEN]) { @@ -3133,7 +3596,7 @@ mod tests { studio.collection.scenes[0].items[1].visible = false; studio.render(start); - let after = studio.compositor.arranged("slots").unwrap().to_vec(); + let after = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); assert!(!after[1].visible, "a hidden item takes no slot"); assert_ne!(after[2].rect, before[2].rect, "green moves up"); @@ -3195,8 +3658,8 @@ mod tests { studio.collection.apply(&edit).unwrap(); studio.render(start.saturating_add(REFLOW / 2)); for scene in ["slots", "row"] { - let arranged = studio.compositor.arranged(scene).unwrap().to_vec(); - let placed = studio.compositor.placed(scene).unwrap().to_vec(); + let arranged = studio.compositor.arranged(Format::Landscape, scene).unwrap().to_vec(); + let placed = studio.compositor.placed(Format::Landscape, scene).unwrap().to_vec(); assert_eq!(placed, arranged, "`{scene}` is where it was, with no glide"); } }) @@ -3219,7 +3682,7 @@ mod tests { studio.compositor.set_preview(Some("slots".into())); studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); - let arranged = studio.compositor.arranged("slots").unwrap().to_vec(); + let arranged = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); assert!(!arranged[0].visible, "an unfilled role takes no slot"); assert!(!arranged[1].visible, "an offline guest takes no slot"); let (_, alone) = Arrangement::Column.places(studio.collection.canvas, false, 1); @@ -3234,7 +3697,7 @@ mod tests { let filled = Instant::now(); studio.render(filled); studio.render(filled.saturating_add(REFLOW)); - let arranged = studio.compositor.arranged("slots").unwrap().to_vec(); + let arranged = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); assert!(arranged[0].visible, "a filled role takes the main slot"); let (x, y) = center(&arranged[0].rect); assert_eq!( @@ -3561,7 +4024,7 @@ mod tests { ..Default::default() }, ); - let shown = studio.compositor.shown(&collection); + let shown = studio.compositor.shown(&collection, None); assert!(shown.contains("sting") && shown.contains("red"), "{shown:?}"); sting(&mut studio, Filter::Brightness { amount: 0.0 }); @@ -5112,7 +5575,7 @@ mod tests { assert_eq!(pixel(&program, 3, 4), BLUE, "the red scene's corner"); assert_eq!(pixel(&program, 6, 6), BLUE, "under it"); assert_eq!(pixel(&program, 0, 0), RED, "shown twice"); - assert_eq!(studio.compositor.nested.get("red").map(Vec::len), Some(1)); + assert_eq!(studio.compositor.view.nested.get("red").map(Vec::len), Some(1)); studio.compositor.set_preview(Some("blue".into())); studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); @@ -5120,7 +5583,7 @@ mod tests { let program = studio.program(); assert_eq!(pixel(&program, 1, 1), RED, "through the nested scene"); assert_eq!(pixel(&program, 6, 1), BLUE); - let drawn: Vec<&String> = studio.compositor.nested.keys().collect(); + let drawn: Vec<&String> = studio.compositor.view.nested.keys().collect(); assert_eq!(drawn, ["corner"], "only what is drawn"); }) .await; @@ -5304,12 +5767,12 @@ mod tests { assert_eq!((pixel(&program, 5, 1), pixel(&program, 6, 1)), (BLUE, BLACK)); let preview = studio.preview(); assert_eq!((pixel(&preview, 1, 1), pixel(&preview, 2, 1)), (BLUE, BLACK)); - assert_eq!(studio.compositor.nested["blue"].len(), 2, "one per moment"); + assert_eq!(studio.compositor.view.nested["blue"].len(), 2, "one per moment"); studio.render(at(2000)); assert_eq!(pixel(&studio.program(), 7, 1), BLUE); assert_eq!(pixel(&studio.preview(), 7, 1), BLUE); - assert_eq!(studio.compositor.nested["blue"].len(), 1, "settled alike"); + assert_eq!(studio.compositor.view.nested["blue"].len(), 1, "settled alike"); }) .await; } @@ -5849,6 +6312,30 @@ mod bench { .await; } + /// What going out in portrait too costs, over the program and the same + /// scene in preview: landscape only, both formats at once, and the long + /// way before it, a published portrait scene placing every source. + #[tokio::test] + #[ignore = "a benchmark; run by hand on a GPU"] + async fn both_formats() { + local(async { + println!("sources case frame passes copies"); + for sources in [4, 16] { + let landscape = collection(sources, 0, sources); + let mut both = landscape.clone(); + both.portrait = true; + both.validate().unwrap(); + let published = collection(sources, 1, sources); + let scene = landscape.scenes[0].name.clone(); + for (case, collection) in [("landscape", landscape), ("both", both), ("published", published)] { + let (frame, (passes, copies)) = cost(collection, &scene).await; + println!("{sources:>7} {case:<10} {frame:>7.2?} {passes:>6} {copies:>6}"); + } + } + }) + .await; + } + /// What drawing sources published independently as layers costs, over the /// program and the same scene in preview: none, then the last of the /// sources, then the last four. diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index 25893a56..20abc3fe 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -72,6 +72,8 @@ pub enum Change { Transition { before: Transition, after: Transition }, /// [`Collection::canvas`]. Canvas { before: Canvas, after: Canvas }, + /// [`Collection::portrait`]. + Portrait { before: bool, after: bool }, /// [`Collection::audio_scenes`], whole. AudioScenes { before: Vec, @@ -127,6 +129,10 @@ impl Change { before: after, after: before, }, + Self::Portrait { before, after } => Self::Portrait { + before: after, + after: before, + }, Self::AudioScenes { before, after } => Self::AudioScenes { before: after, after: before, @@ -145,6 +151,7 @@ impl Change { | Self::Item { .. } | Self::Transition { .. } | Self::Canvas { .. } + | Self::Portrait { .. } | Self::AudioScenes { .. } => None, } } @@ -160,6 +167,7 @@ impl Change { Self::Item { before, after, .. } => before == after, Self::Transition { before, after } => before == after, Self::Canvas { before, after } => before == after, + Self::Portrait { before, after } => before == after, Self::AudioScenes { before, after } => before == after, } } @@ -261,6 +269,9 @@ impl Change { (Self::Canvas { before, after }, Self::Canvas { before: b, after: a }) if after == *b => { Self::Canvas { before, after: *a } } + (Self::Portrait { before, after }, Self::Portrait { before: b, after: a }) if after == *b => { + Self::Portrait { before, after: *a } + } (Self::AudioScenes { before, after }, Self::AudioScenes { before: b, after: a }) if after == *b => { Self::AudioScenes { before, @@ -317,6 +328,13 @@ impl Change { collection.canvas = *after; Ok(()) } + Self::Portrait { before, after } => { + if collection.portrait != *before { + return Err(conflict("whether the show goes out in portrait changed".into())); + } + collection.portrait = *after; + Ok(()) + } Self::AudioScenes { before, after } => { if collection.audio_scenes != *before { return Err(conflict("the audio scenes changed".into())); diff --git a/rs/moq-scene/src/format.rs b/rs/moq-scene/src/format.rs new file mode 100644 index 00000000..7f5c0633 --- /dev/null +++ b/rs/moq-scene/src/format.rs @@ -0,0 +1,867 @@ +//! Landscape and portrait at once: while a show goes out in +//! [portrait](Collection::portrait), each scene of the program's size has a +//! portrait arrangement of the same layers, drawn on the canvas turned on its +//! side. + +use std::borrow::Cow; + +use serde::{Deserialize, Serialize}; + +use crate::Error; +use crate::collection::{Arrangement, Canvas, Collection, Crop, Item, Layer, Rect, Scene, Size, Source}; +use crate::edit::{Change, Draft}; + +/// Where the portrait picture goes out, beneath the program's broadcast, so +/// no scene published while the show goes out in portrait takes the name. +pub const PORTRAIT: &str = "portrait"; + +/// How much of the portrait canvas's height the main picture takes when a +/// camera is stacked under it. +const MAIN_SHARE: f32 = 0.6; + +/// How much of the landscape canvas an item covers to count as a backdrop, +/// which covers the portrait canvas too. +const BACKDROP: f32 = 0.9; + +/// One of the two shapes a show goes out in. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Format { + /// Wider than tall: the canvas as it is. + Landscape, + /// Taller than wide: the canvas turned on its side. + Portrait, +} + +impl Format { + /// Both, landscape first. + pub const ALL: [Self; 2] = [Self::Landscape, Self::Portrait]; + + /// The other one. + pub fn other(self) -> Self { + match self { + Self::Landscape => Self::Portrait, + Self::Portrait => Self::Landscape, + } + } + + /// Its name, capitalized, as a label says it. + pub fn name(self) -> &'static str { + match self { + Self::Landscape => "Landscape", + Self::Portrait => "Portrait", + } + } +} + +/// Where an item is drawn in the portrait picture. +#[derive(Clone, Copy, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Placement { + /// Where on the portrait canvas, stretched to fill it. + pub rect: Rect, + /// Trimmed from what its source's crop leaves, in place of the item's own + /// crop. + #[serde(default, skip_serializing_if = "Crop::is_none")] + pub crop: Crop, +} + +impl Placement { + /// Where an item at `rect` on the `landscape` canvas sits on the portrait + /// one without a place of its own there: as far across and down, scaled by + /// as much as the narrower side shrinks, so it keeps its shape and fits. + pub fn fitted(rect: Rect, crop: Crop, landscape: Canvas) -> Self { + let (from, to) = (Rect::fill(&landscape), Rect::fill(&landscape.turned())); + Self::moved(rect, crop, &from, &to) + } + + /// `rect` moved from on `from` to on `to` as [`fitted`](Self::fitted) + /// moves it from one canvas to the other: its center as far across and + /// down, and scaled to fit, uncropped. + fn moved(rect: Rect, crop: Crop, from: &Rect, to: &Rect) -> Self { + let scale = (to.width / from.width).min(to.height / from.height); + let (width, height) = (rect.width * scale, rect.height * scale); + let center = ( + to.x + (rect.x + rect.width / 2.0 - from.x) / from.width * to.width, + to.y + (rect.y + rect.height / 2.0 - from.y) / from.height * to.height, + ); + Self { + rect: Rect { + x: (center.0 - width / 2.0).round(), + y: (center.1 - height / 2.0).round(), + width: width.round(), + height: height.round(), + }, + crop, + } + } +} + +/// How a scene is arranged in the portrait picture. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Portrait { + /// Where its items in a slot go: in a row under the main one unless set, + /// since a column beside it leaves a tall canvas mostly empty. + #[serde(default = "row", skip_serializing_if = "is_row")] + pub arrangement: Arrangement, +} + +impl Default for Portrait { + fn default() -> Self { + Self { + arrangement: Arrangement::Row, + } + } +} + +impl Portrait { + pub(crate) fn is_default(&self) -> bool { + *self == Self::default() + } +} + +fn row() -> Arrangement { + Arrangement::Row +} + +fn is_row(arrangement: &Arrangement) -> bool { + *arrangement == Arrangement::Row +} + +impl Canvas { + /// The canvas turned on its side: its width and height swapped. + pub fn turned(self) -> Self { + Self { + width: self.height, + height: self.width, + framerate: self.framerate, + } + } +} + +impl Item { + /// Whether it is shown in `format`'s picture. + pub fn shown_in(&self, format: Format) -> bool { + self.format.is_none_or(|only| only == format) + } + + /// Where it is drawn in `format`'s picture, in a show whose landscape + /// canvas is `landscape`: its rect and crop, or its portrait place, or + /// failing that where it sits on the landscape picture, fitted. + pub fn placement(&self, format: Format, landscape: Canvas) -> Placement { + let landscaped = Placement { + rect: self.rect, + crop: self.crop, + }; + match format { + Format::Landscape => landscaped, + Format::Portrait => self + .portrait + .unwrap_or_else(|| Placement::fitted(self.rect, self.crop, landscape)), + } + } + + /// The item drawn at `placement` in `format`'s picture, as it was in the + /// other. An item in a slot has one crop in both, which this sets. + pub fn placed(self, format: Format, placement: Placement) -> Self { + match format { + Format::Portrait if self.slot.is_none() => Self { + portrait: Some(placement), + ..self + }, + _ => Self { + rect: placement.rect, + crop: placement.crop, + ..self + }, + } + } + + /// The item as `format`'s picture draws it at rest, in a show whose + /// landscape canvas is `landscape`: at its place there, and hidden unless + /// shown in it. An item in a slot stays where the arrangement puts it. + pub fn portrayed(&self, format: Format, landscape: Canvas) -> Self { + let mut item = self.clone(); + if item.slot.is_none() { + let placement = item.placement(format, landscape); + item.rect = placement.rect; + item.crop = placement.crop; + } + item.visible &= item.shown_in(format); + item + } + + /// Whether it says anything about the portrait picture. + fn formats(&self) -> bool { + self.format.is_some() || self.portrait.is_some() + } +} + +impl Collection { + /// The portrait picture's canvas, while the show goes out in portrait. + pub fn portrait_canvas(&self) -> Option { + self.portrait.then(|| self.canvas.turned()) + } + + /// The canvas `format`'s picture is drawn on: the program's, or turned. + pub fn canvas_of(&self, format: Format) -> Canvas { + match format { + Format::Landscape => self.canvas, + Format::Portrait => self.canvas.turned(), + } + } + + /// Whether `layer` is drawn in both pictures: the top layer, or a scene of + /// the program's size, while the show goes out in portrait. + pub fn dual(&self, layer: &Layer) -> bool { + self.portrait + && match &layer.scene { + None => true, + Some(scene) => self.scene(scene).is_some_and(|scene| scene.size.is_none()), + } + } + + /// The collection as `format`'s picture draws it, while the show goes out + /// in portrait: in portrait, on the canvas turned, each scene of the + /// program's size arranged as its portrait arrangement says and each item + /// at its portrait place; in either, an item shown only in the other is + /// hidden. A scene of its own size is drawn as it is. Itself otherwise. + pub fn formatted(&self, format: Format) -> Cow<'_, Self> { + if !self.portrait { + return Cow::Borrowed(self); + } + if format == Format::Landscape { + let hides = self + .layers() + .any(|(_, items)| items.iter().any(|item| !item.shown_in(format))); + if !hides { + return Cow::Borrowed(self); + } + } + let landscape = self.canvas; + let mut view = self.clone(); + view.canvas = self.canvas_of(format); + for scene in view.scenes.iter_mut().filter(|scene| scene.size.is_none()) { + if format == Format::Portrait { + scene.arrangement = scene.portrait.arrangement; + } + for item in &mut scene.items { + *item = item.portrayed(format, landscape); + } + } + for item in &mut view.top { + *item = item.portrayed(format, landscape); + } + Cow::Owned(view) + } + + /// The picture `layer` is edited in when `format` is asked for: `format` + /// while the layer is drawn in both, landscape otherwise. + pub fn format_of(&self, layer: &Layer, format: Format) -> Format { + match self.dual(layer) { + true => format, + false => Format::Landscape, + } + } + + /// Check the portrait picture's rules: only a landscape canvas turns, and + /// only a scene of the program's size, or the top layer, says anything + /// about it, an item in a slot or shown only in landscape taking no place + /// there; nor does a scene publish where it goes out. + pub(crate) fn validate_formats(&self) -> Result<(), Error> { + if self.portrait && self.canvas.width <= self.canvas.height { + return Err(Error::Invalid(format!( + "the show goes out in portrait too, which turns a landscape canvas, not {}x{}", + self.canvas.width, self.canvas.height + ))); + } + for scene in &self.scenes { + let why = match (self.portrait, scene.size) { + (false, _) => "the show does not go out in portrait", + (true, Some(_)) => "it has a size of its own, so it is drawn in one format", + (true, None) => "", + }; + if why.is_empty() { + continue; + } + if !scene.portrait.is_default() { + return Err(Error::Invalid(format!( + "scene `{}` has a portrait arrangement, but {why}", + scene.name + ))); + } + if let Some(item) = scene.items.iter().find(|item| item.formats()) { + return Err(Error::Invalid(format!( + "`{}` in scene `{}` says where it shows in portrait, but {why}", + item.source, scene.name + ))); + } + } + if !self.portrait + && let Some(item) = self.top.iter().find(|item| item.formats()) + { + return Err(Error::Invalid(format!( + "always-on-top `{}` says where it shows in portrait, but the show does not go out in portrait", + item.source + ))); + } + for (layer, items) in self.layers() { + for item in items { + let Some(placement) = &item.portrait else { continue }; + let on = format!("`{}` in {layer}", item.source); + if item.slot.is_some() { + return Err(Error::Invalid(format!( + "{on} is in a slot, which places it in portrait too, so it takes no portrait place" + ))); + } + if item.format == Some(Format::Landscape) { + return Err(Error::Invalid(format!( + "{on} is shown only in landscape, so it takes no portrait place" + ))); + } + placement.rect.validate(&layer, &item.source)?; + placement + .crop + .validate() + .map_err(|err| Error::Invalid(format!("{on} in portrait: {err}")))?; + } + } + if self.portrait + && let Some(scene) = self + .scenes + .iter() + .find(|scene| scene.published && scene.name == PORTRAIT) + { + return Err(Error::Invalid(format!( + "scene `{}` is published where the portrait picture goes out; rename it to publish it", + scene.name + ))); + } + Ok(()) + } + + /// The items of a layer as a show turning to portrait first places them: + /// the main picture, the largest covering a quarter of the canvas or more, + /// cropped to the top of the portrait canvas, or all of it without a + /// camera, and the largest camera stacked under it, keeping its shape if + /// it has an outline; a backdrop covering the landscape canvas covers the + /// portrait one, cropped. What sits on the main picture keeps its place + /// on it. The rest sit where they are, fitted, until moved there. + fn turned_on(&self, items: &[Item]) -> Vec { + let landscape = self.canvas; + let portrait = landscape.turned(); + let whole = Rect::fill(&portrait); + let canvas = landscape.width as f32 * landscape.height as f32; + let area = |item: &Item| covered(&item.rect, &landscape) / canvas; + let free: Vec = (0..items.len()) + .filter(|&at| { + let item = &items[at]; + item.slot.is_none() && item.visible && item.shown_in(Format::Portrait) && area(item) > 0.0 + }) + .collect(); + let largest = |among: &mut dyn Iterator| { + // The frontmost of two as large, which is the one seen. + among.max_by(|a, b| area(&items[*a]).total_cmp(&area(&items[*b])).then(a.cmp(b))) + }; + let main = largest(&mut free.iter().copied()).filter(|&at| area(&items[at]) >= 0.25); + let camera = largest(&mut free.iter().copied().filter(|&at| { + let source = self.sources.get(self.capture(&items[at].source)); + Some(at) != main && matches!(source, Some(Source::Camera { .. })) && area(&items[at]) < BACKDROP + })); + + let mut placed = items.to_vec(); + if let Some(main) = main { + let top = match camera { + Some(_) => Rect { + height: (portrait.height as f32 * MAIN_SHARE).round(), + ..whole + }, + None => whole, + }; + placed[main].portrait = Some(covering(&items[main], top)); + // What sits on the main picture, such as a lower third, keeps + // its place on it, rather than landing on the camera. + let on = items[main].rect; + for &at in free.iter().filter(|&&at| Some(at) != Some(main) && Some(at) != camera) { + let item = &items[at]; + let center = ( + item.rect.x + item.rect.width / 2.0, + item.rect.y + item.rect.height / 2.0, + ); + let inside = + (on.x..on.x + on.width).contains(¢er.0) && (on.y..on.y + on.height).contains(¢er.1); + if camera.is_some() && inside && area(item) < BACKDROP { + placed[at].portrait = Some(Placement::moved(item.rect, item.crop, &on, &top)); + } + } + if let Some(camera) = camera { + let under = Rect { + y: top.height, + height: whole.height - top.height, + ..whole + }; + let item = &items[camera]; + placed[camera].portrait = Some(match item.shape.is_square() { + true => covering(item, under), + false => fitting(item, under), + }); + } + } + for &at in &free { + let item = &items[at]; + let source = self.sources.get(self.capture(&item.source)); + // A page or overlay is mostly see-through, so it is fitted whole + // rather than cropped. + let see_through = matches!(source, Some(Source::Browser { .. } | Source::Overlay { .. })); + if Some(at) != main && Some(at) != camera && area(item) >= BACKDROP && !see_through { + placed[at].portrait = Some(covering(item, whole)); + } + } + placed + } +} + +impl Scene { + /// The scene at `size`, its own, or the program's for `None`. A scene of + /// its own size is drawn in one picture, so it keeps no portrait + /// arrangement or place, and its layers shown only in portrait are + /// removed, as turning portrait off removes them. + pub fn sized(mut self, size: Option) -> Self { + if size.is_some() { + self.single(); + } + Self { size, ..self } + } + + /// Forget the portrait picture: its arrangement, each layer's place + /// there, and the layers shown only there. + fn single(&mut self) { + self.portrait = Portrait::default(); + self.items.retain(|item| item.shown_in(Format::Landscape)); + for item in &mut self.items { + item.format = None; + item.portrait = None; + } + } +} + +impl Draft { + /// Send the show out in portrait too, or no longer. Turned on, each + /// layer's portrait arrangement starts from its landscape one: the main + /// picture cropped to the top, the camera stacked under it, and slots in + /// a row unless they were in a grid or inset. A scene given the program's + /// size as its own follows the program's instead, so it has a portrait + /// picture and can stay on program. Turned off, the layers shown only in + /// portrait are removed and every portrait place dropped. + pub fn set_portrait(&mut self, on: bool) -> Result<(), Error> { + let before = self.collection().portrait; + if before == on { + return Ok(()); + } + let collection = self.collection().clone(); + let program = (collection.canvas.width, collection.canvas.height); + for (index, scene) in collection.scenes.iter().enumerate() { + let pinned = on && scene.size.is_some_and(|size| (size.width, size.height) == program); + if scene.size.is_some() && !pinned { + continue; + } + let mut after = Scene { + size: None, + ..scene.clone() + }; + match on { + true => { + after.items = collection.turned_on(&scene.items); + if matches!(scene.arrangement, Arrangement::Grid | Arrangement::Inset) { + after.portrait.arrangement = scene.arrangement; + } + } + false => after.single(), + } + if after != *scene { + self.set_scene(index, after)?; + } + } + let top = Layer::top(); + match on { + true => { + for (index, item) in collection.turned_on(&collection.top).into_iter().enumerate() { + if item != collection.top[index] { + self.set_item(&top, index, item)?; + } + } + } + false => { + for (index, item) in collection.top.iter().enumerate().rev() { + if !item.shown_in(Format::Landscape) { + self.remove_item(&top, index)?; + } else if item.formats() { + let item = Item { + format: None, + portrait: None, + ..item.clone() + }; + self.set_item(&top, index, item)?; + } + } + } + } + self.change(Change::Portrait { before, after: on }) + } +} + +/// How much of `canvas` `rect` covers, in square pixels. +fn covered(rect: &Rect, canvas: &Canvas) -> f32 { + let width = (rect.x + rect.width).min(canvas.width as f32) - rect.x.max(0.0); + let height = (rect.y + rect.height).min(canvas.height as f32) - rect.y.max(0.0); + width.max(0.0) * height.max(0.0) +} + +/// `item` covering `rect`, its picture cropped evenly to `rect`'s shape on +/// top of its own crop, so it shows unstretched as it did. +fn covering(item: &Item, rect: Rect) -> Placement { + let shown = item.rect.width / item.rect.height; + let wanted = rect.width / rect.height; + Placement { + rect, + crop: item.crop.then(&Crop::covering(shown, wanted)), + } +} + +/// `item` as large as fits in `rect`, keeping its shape, centered. +fn fitting(item: &Item, rect: Rect) -> Placement { + let shape = item.rect.width / item.rect.height; + let width = rect.width.min(rect.height * shape); + let height = width / shape; + Placement { + rect: Rect { + x: (rect.x + (rect.width - width) / 2.0).round(), + y: (rect.y + (rect.height - height) / 2.0).round(), + width: width.round(), + height: height.round(), + }, + crop: item.crop, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::collection::{Rgba, Scene, Shape, Size, Slot}; + + fn color() -> Source { + Source::Color { + color: Rgba([255, 0, 0, 255]), + } + } + + /// A 1920x1080 show: a game filling the canvas, a camera in its corner, + /// and a logo, in scene "Game"; and a slotted scene "Guests". + fn show() -> Collection { + let mut collection = Collection::new(); + collection.sources.insert("game".into(), color()); + collection.sources.insert("camera".into(), Source::Camera { id: None }); + collection.sources.insert("logo".into(), color()); + let fill = Rect::fill(&collection.canvas); + let corner = Rect { + x: 1440.0, + y: 720.0, + width: 480.0, + height: 270.0, + }; + let logo = Rect { + x: 1600.0, + y: 40.0, + width: 280.0, + height: 120.0, + }; + let slotted = |slot| Item { + slot: Some(slot), + ..Item::new("camera", Rect::default()) + }; + collection.scenes = vec![ + Scene { + items: vec![ + Item::new("game", fill), + Item::new("camera", corner), + Item::new("logo", logo), + ], + ..Scene::new("Game") + }, + Scene { + items: vec![slotted(Slot::Main), slotted(Slot::Tile)], + ..Scene::new("Guests") + }, + ]; + collection.validate().unwrap(); + collection + } + + fn turned(collection: &Collection) -> Collection { + let mut draft = collection.draft("Both"); + draft.set_portrait(true).unwrap(); + let edit = draft.finish().unwrap(); + let mut turned = collection.clone(); + turned.apply(&edit).unwrap(); + turned + } + + #[test] + fn turning_on_crops_the_main_picture_over_the_camera() { + let turned = turned(&show()); + assert!(turned.portrait); + let items = &turned.scenes[0].items; + let game = items[0].portrait.unwrap(); + assert_eq!( + game.rect, + Rect { + x: 0.0, + y: 0.0, + width: 1080.0, + height: 1152.0 + } + ); + // The 16:9 game cropped evenly to the 1080x1152 band, unstretched. + let kept = 1.0 - game.crop.left - game.crop.right; + assert!((1920.0 * kept / 1080.0 - 1080.0 / 1152.0).abs() < 1e-3, "{game:?}"); + assert!((game.crop.left - game.crop.right).abs() < 1e-6); + let camera = items[1].portrait.unwrap(); + assert_eq!( + camera.rect, + Rect { + x: 0.0, + y: 1152.0, + width: 1080.0, + height: 768.0 + } + ); + // The logo on the game keeps its place on it, above the camera. + assert_eq!( + items[2].portrait.unwrap().rect, + Rect { + x: 900.0, + y: 73.0, + width: 158.0, + height: 68.0 + } + ); + turned.validate().unwrap(); + } + + /// A backdrop as large as the game, behind it, is not the main picture: + /// the game in front is, and the backdrop covers the portrait canvas. + #[test] + fn the_frontmost_of_two_as_large_is_the_main_picture() { + let mut show = show(); + let fill = show.scenes[0].items[0].rect; + show.scenes[0].items.insert(0, Item::new("logo", fill)); + let turned = turned(&show); + let items = &turned.scenes[0].items; + assert_eq!(items[0].portrait.unwrap().rect, Rect::fill(&show.canvas.turned())); + assert_eq!(items[1].portrait.unwrap().rect.height, 1152.0); + } + + #[test] + fn a_round_camera_keeps_its_shape() { + let mut show = show(); + show.scenes[0].items[1].rect = Rect { + x: 1500.0, + y: 700.0, + width: 300.0, + height: 300.0, + }; + show.scenes[0].items[1].shape = Shape { + corners: [0.5; 4], + ..Shape::default() + }; + let turned = turned(&show); + let camera = turned.scenes[0].items[1].portrait.unwrap(); + assert_eq!(camera.rect.width, camera.rect.height); + assert_eq!(camera.rect.width, 768.0); + assert_eq!(camera.rect.y, 1152.0); + assert_eq!(camera.rect.x, 156.0); + } + + #[test] + fn without_a_camera_the_main_picture_covers_the_canvas() { + let mut show = show(); + show.scenes[0].items.remove(1); + let turned = turned(&show); + let game = turned.scenes[0].items[0].portrait.unwrap(); + assert_eq!(game.rect, Rect::fill(&show.canvas.turned())); + } + + #[test] + fn slots_go_in_a_row_unless_gridded() { + let mut show = show(); + show.scenes[1].arrangement = Arrangement::Grid; + let mut copy = show.scenes[1].clone(); + copy.name = "Row".into(); + copy.arrangement = Arrangement::Column; + show.scenes.push(copy); + let turned = turned(&show); + assert_eq!(turned.scenes[1].portrait.arrangement, Arrangement::Grid); + assert_eq!(turned.scenes[2].portrait.arrangement, Arrangement::Row); + assert!(turned.scenes[1].items.iter().all(|item| item.portrait.is_none())); + } + + #[test] + fn a_scene_of_its_own_size_is_left_alone() { + let mut show = show(); + show.scenes[0].size = Some(Size { + width: 1080, + height: 1920, + }); + let turned = turned(&show); + assert_eq!(turned.scenes[0], show.scenes[0]); + } + + /// A scene of its own size has no portrait picture, so it does not go on + /// program while the show goes out in both, even at the program's size; + /// one given the program's size before turning on follows it instead. + #[test] + fn only_a_scene_following_the_program_goes_on_in_both() { + let mut show = show(); + let program = Size { + width: 1920, + height: 1080, + }; + show.scenes[0].size = Some(program); + show.takes("Game").unwrap(); + let mut turned = turned(&show); + assert_eq!( + turned.scenes[0].size, None, + "pinned at the program's size, so it follows it" + ); + assert!(turned.scenes[0].items[0].portrait.is_some()); + turned.takes("Game").unwrap(); + + turned.scenes[0] = turned.scenes[0].clone().sized(Some(program)); + turned.validate().unwrap(); + let why = turned.refuses("Game").unwrap(); + assert!(why.contains("size of its own"), "{why}"); + } + + #[test] + fn the_portrait_view_places_each_item_there() { + let mut turned = turned(&show()); + turned.scenes[0].items[2].format = Some(Format::Landscape); + let view = turned.formatted(Format::Portrait); + assert_eq!(view.canvas, turned.canvas.turned()); + let items = &view.scenes[0].items; + assert_eq!(items[0].rect, turned.scenes[0].items[0].portrait.unwrap().rect); + assert!(!items[2].visible, "shown only in landscape"); + assert_eq!(view.scenes[1].arrangement, Arrangement::Row); + // The landscape view hides nothing shown in landscape, so it is the + // collection itself. + assert!(matches!(turned.formatted(Format::Landscape), Cow::Borrowed(_))); + turned.scenes[0].items[2].format = Some(Format::Portrait); + let view = turned.formatted(Format::Landscape); + assert!(!view.scenes[0].items[2].visible); + assert!(view.scenes[0].items[0].visible); + } + + #[test] + fn fitted_keeps_its_place_and_shape() { + let landscape = Canvas::default(); + let logo = Rect { + x: 1600.0, + y: 40.0, + width: 280.0, + height: 120.0, + }; + let fitted = Placement::fitted(logo, Crop::default(), landscape); + // 9/16 the size, its center as far across and down. + assert_eq!(fitted.rect.width, 158.0); + assert_eq!(fitted.rect.height, 68.0); + let center = fitted.rect.x + fitted.rect.width / 2.0; + assert!((center / 1080.0 - 1740.0 / 1920.0).abs() < 1e-3); + let center = fitted.rect.y + fitted.rect.height / 2.0; + assert!((center / 1920.0 - 100.0 / 1080.0).abs() < 1e-3); + } + + #[test] + fn turning_off_drops_the_portrait_picture() { + let mut turned = turned(&show()); + turned.scenes[0].items.push(Item { + format: Some(Format::Portrait), + ..Item::new("logo", Rect::fill(&turned.canvas)) + }); + turned.top.push(Item { + format: Some(Format::Portrait), + ..Item::new("logo", Rect::fill(&turned.canvas)) + }); + turned.top.push(Item { + format: Some(Format::Landscape), + ..Item::new("logo", Rect::fill(&turned.canvas)) + }); + turned.validate().unwrap(); + let mut draft = turned.draft("Landscape"); + draft.set_portrait(false).unwrap(); + let edit = draft.finish().unwrap(); + let mut off = turned.clone(); + off.apply(&edit).unwrap(); + assert_eq!(off.scenes, show().scenes); + assert_eq!(off.top.len(), 1); + assert_eq!(off.top[0].format, None); + // Undone whole. + off.apply(&edit.inverse()).unwrap(); + assert_eq!(off, turned); + } + + #[test] + fn refuses_what_the_portrait_picture_cannot_draw() { + let refused = |edit: &dyn Fn(&mut Collection)| { + let mut collection = turned(&show()); + edit(&mut collection); + collection.validate().unwrap_err().to_string() + }; + assert!(refused(&|c| c.canvas = c.canvas.turned()).contains("turns a landscape canvas")); + assert!(refused(&|c| c.scenes[1].items[0].portrait = Some(Placement::default())).contains("in a slot")); + assert!(refused(&|c| c.scenes[0].items[0].format = Some(Format::Landscape)).contains("only in landscape")); + assert!( + refused(&|c| { + c.portrait = false; + }) + .contains("does not go out in portrait") + ); + assert!( + refused(&|c| { + c.scenes[0].name = PORTRAIT.into(); + c.scenes[0].published = true; + }) + .contains("rename it") + ); + } + + /// A scene given its own size is drawn in one picture, so it forgets the + /// portrait one in the same edit, which validates. + #[test] + fn a_scene_given_its_own_size_forgets_the_portrait_picture() { + let mut turned = turned(&show()); + turned.scenes[0].items[2].format = Some(Format::Portrait); + turned.validate().unwrap(); + let size = Size { + width: 1080, + height: 1920, + }; + let sized = turned.scenes[0].clone().sized(Some(size)); + assert_eq!(sized.size, Some(size)); + assert_eq!(sized.items.len(), 2, "the portrait-only logo removed"); + assert!(sized.items.iter().all(|item| !item.formats())); + turned.scenes[0] = sized; + turned.validate().unwrap(); + } + + #[test] + fn round_trips_through_the_scene_file() { + let mut turned = turned(&show()); + turned.scenes[0].items[2].format = Some(Format::Landscape); + turned.scenes[0].items[2].portrait = None; + let text = turned.to_toml().unwrap(); + assert!(text.contains("portrait = true"), "{text}"); + assert!(text.contains("format = \"landscape\""), "{text}"); + assert_eq!(Collection::from_toml(&text).unwrap(), turned); + } +} diff --git a/rs/moq-scene/src/layout.rs b/rs/moq-scene/src/layout.rs index 95a0ec7f..0f0d1064 100644 --- a/rs/moq-scene/src/layout.rs +++ b/rs/moq-scene/src/layout.rs @@ -53,8 +53,16 @@ impl Layout { /// The layout of `collection` with the scene called `scene` on program /// since `start` on the media clock, or no scene for `None`. pub fn new(collection: &Collection, scene: Option<&str>, start: Duration) -> Self { + // As the landscape picture places them, which is the one the layout + // goes with: the portrait picture flattens them and has none. let pictures = pictures(collection, scene, true) .into_iter() + .map(|item| Item { + visible: item.visible && item.shown_in(crate::Format::Landscape), + format: None, + portrait: None, + ..item + }) .map(Picture::new) .collect(); let heard = collection.program_audio().map(|audio| collection.heard(audio)); @@ -324,6 +332,27 @@ mod tests { assert_eq!(top.pictures.len(), 1, "only the top layer's"); } + /// In a show going out in both formats, the layout places each picture + /// as the landscape picture does, which it goes with: hidden there when + /// shown in portrait alone, and saying nothing of the portrait one. + #[test] + fn lays_out_the_landscape_picture_of_both() { + let mut collection = collection(); + collection.portrait = true; + let cam = &mut collection.scenes[0].items[1]; + cam.format = Some(crate::Format::Portrait); + cam.portrait = Some(crate::Placement { + rect: rect(4.0), + crop: Default::default(), + }); + collection.validate().unwrap(); + let layout = Layout::new(&collection, Some("live"), Duration::ZERO); + let cam = &layout.pictures[0].item; + assert!(!cam.visible, "shown in portrait alone"); + assert_eq!((cam.format, cam.portrait), (None, None)); + assert_eq!(cam.rect, rect(0.0)); + } + /// Each source's own crop and filters fold into its picture, a clone's /// its own rather than the source's it shows. #[test] diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index e8d1f2ba..53436089 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -53,6 +53,7 @@ mod dark; mod edit; mod effect; pub mod file; +mod format; mod glide; mod grant; pub mod layout; @@ -73,6 +74,7 @@ pub use collection::{ }; pub use compositor::{Compositor, Detail, FORMAT, Output, Thumbnail}; pub use edit::{Change, Draft, Edit, History}; +pub use format::{Format, PORTRAIT, Placement, Portrait}; pub use grant::Grants; pub use scale::Scaler; diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 29d3b57b..61e531fe 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -313,6 +313,10 @@ pub struct Sources { feeds: BTreeMap, /// Each audio scene's mix, by its name. mixes: BTreeMap, + /// The program's audio scene with its sources published independently + /// mixed in too, by its name, while the show goes out in portrait and + /// one of them is heard: see [`whole`](Self::whole). + whole: Option<(String, Mix)>, /// The sources on program, as last set. on_program: BTreeSet, /// The picked displays and windows the last sync ran, by name, whose @@ -327,6 +331,7 @@ impl Sources { context, feeds: BTreeMap::new(), mixes: BTreeMap::new(), + whole: None, on_program: BTreeSet::new(), picked: None, } @@ -370,11 +375,11 @@ impl Sources { /// [`Collection::validate`] refuses. pub fn sync(&mut self, collection: &Collection) -> Vec { self.forget_grants(collection); - let mixes = &self.mixes; + let (mixes, whole) = (&self.mixes, &self.whole); self.feeds.retain(|name, feed| { let keep = collection.sources.get(name) == Some(&feed.source); if !keep { - leave(mixes, feed); + leave(mixes, whole, feed); } keep }); @@ -390,10 +395,22 @@ impl Sources { .collect(); for name in orphaned { let feed = self.feeds.remove(&name).expect("listed above"); - leave(&self.mixes, &feed); + leave(&self.mixes, &self.whole, &feed); } // A mix ends once its last handle drops, which may be a reader's. self.mixes.retain(|name, _| collection.audio_scene(name).is_some()); + // The portrait picture carries no layout, so its sound has the + // sources published independently mixed in, when any is heard. + let whole = collection + .portrait_canvas() + .and(collection.program_audio()) + .filter(|audio| { + let heard = collection.heard(audio); + heard.iter().any(|name| collection.published.contains(*name)) + }); + if self.whole.as_ref().map(|(name, _)| name.as_str()) != whole { + self.whole = whole.map(|name| (name.to_string(), self.context.bus.mix())); + } for scene in &collection.audio_scenes { if !self.mixes.contains_key(&scene.name) { self.mixes.insert(scene.name.clone(), self.context.bus.mix()); @@ -458,6 +475,17 @@ impl Sources { mix.unroute(strip); } } + if let Some((scene, mix)) = &self.whole { + let routed = heard + .get(scene.as_str()) + .is_some_and(|heard| heard.contains(name.as_str())); + if mix.routes(strip) != routed { + match routed { + true => mix.route(strip), + false => mix.unroute(strip), + } + } + } } } drivers @@ -498,7 +526,7 @@ impl Sources { /// every mix at once, as an edited source does. pub fn restart(&mut self, name: &str) { if let Some(feed) = self.feeds.remove(name) { - leave(&self.mixes, &feed); + leave(&self.mixes, &self.whole, &feed); } } @@ -517,7 +545,7 @@ impl Sources { /// and the next [`sync`](Self::sync) starts the sources again. pub fn stop(&mut self) { for feed in std::mem::take(&mut self.feeds).into_values() { - leave(&self.mixes, &feed); + leave(&self.mixes, &self.whole, &feed); } } @@ -532,6 +560,17 @@ impl Sources { self.mixes.get(scene) } + /// The sound of a picture that carries no layer layout, such as the + /// portrait one: the program's audio scene, as [`mix`](Self::mix) has + /// it, with the sources published independently that it hears mixed in, + /// which reach the landscape picture's viewers by their own tracks. + pub fn whole(&self, collection: &Collection) -> Option<&Mix> { + match &self.whole { + Some((_, mix)) => Some(mix), + None => self.mix(collection.program_audio()?), + } + } + /// Whether the audio scene called `scene` plays the source called `name`: /// it has sound, is not muted, and the audio scene reaches it. pub fn plays(&self, scene: &str, name: &str) -> bool { @@ -548,11 +587,11 @@ impl Sources { } } -/// Take a stopping `feed` out of every one of `mixes`, so it leaves nothing -/// behind in them, nor plays over a replacement. -fn leave(mixes: &BTreeMap, feed: &Feed) { +/// Take a stopping `feed` out of every one of `mixes`, and `whole`, so it +/// leaves nothing behind in them, nor plays over a replacement. +fn leave(mixes: &BTreeMap, whole: &Option<(String, Mix)>, feed: &Feed) { let Some(strip) = &feed.strip else { return }; - for mix in mixes.values() { + for mix in mixes.values().chain(whole.iter().map(|(_, mix)| mix)) { mix.unroute(strip); } } @@ -4669,6 +4708,46 @@ mod media_tests { .await; } + /// A sound published independently reaches the landscape picture's + /// viewers by its own track, so the program's mix leaves it out; the + /// portrait picture, which carries no layout, has it mixed in. + #[tokio::test(start_paused = true)] + async fn the_portrait_picture_hears_what_is_published_independently() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let mut collection = media(dir.path(), &[("tone.webm", TONE, true, false)]); + collection.published.insert("tone.webm".into()); + collection.validate().unwrap(); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let strip = sources.get("tone.webm").unwrap().strip().unwrap().clone(); + let program = sources.mix(crate::AudioScene::DEFAULT).unwrap(); + assert!(!program.routes(&strip), "by its own track"); + assert!( + std::ptr::eq(sources.whole(&collection).unwrap(), program), + "without portrait, the program's" + ); + + collection.portrait = true; + collection.validate().unwrap(); + assert!(sources.sync(&collection).is_empty()); + let whole = sources.whole(&collection).unwrap(); + assert!(whole.routes(&strip), "mixed into the portrait picture's"); + assert!(!sources.mix(crate::AudioScene::DEFAULT).unwrap().routes(&strip)); + + // Restarted, it leaves that mix too, and comes back in. + sources.restart("tone.webm"); + assert!(!sources.whole(&collection).unwrap().routes(&strip)); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let strip = sources.get("tone.webm").unwrap().strip().unwrap().clone(); + assert!(sources.whole(&collection).unwrap().routes(&strip)); + }) + .await; + } + /// A file the studio cannot read fails its source, saying why, and only a /// media source plays from its start. #[tokio::test] diff --git a/rs/moq-studio/shots/dual.toml b/rs/moq-studio/shots/dual.toml new file mode 100644 index 00000000..85e3cac6 --- /dev/null +++ b/rs/moq-studio/shots/dual.toml @@ -0,0 +1,75 @@ +# The show the dual-format views render on: one show going out in landscape +# and portrait at once. Each scene of the program's size has a portrait +# arrangement of the same layers: the game cropped into the top of the +# portrait picture with the camera under it, as turning Both on places them, +# a logo shown only in landscape, and a scene laid out by slots in a row in +# portrait. Still sources only, so each view draws the same picture. +version = 1 +portrait = true + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.Main] +kind = "role" +source = "game" + +[sources.Camera] +kind = "role" +source = "host" + +[sources.game] +kind = "color" +color = "#3b1d6e" + +[sources.host] +kind = "image" +path = "person.jpg" + +[sources.Dana] +kind = "clone" +source = "host" + +[sources.backdrop] +kind = "color" +color = "#101828" + +[sources.bar] +kind = "color" +color = "#2563ebe6" + +[sources.logo] +kind = "color" +color = "#f59e0b" + +[video.Dana] +filters = [{ kind = "hue_rotate", degrees = 140.0 }] + +[[scenes]] +name = "Gameplay" +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1920 } } }, + { source = "game", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1152 } } }, + { source = "host", rect = { x = 1440, y = 600, width = 400, height = 400 }, crop = { left = 0.21875, right = 0.21875 }, portrait = { rect = { x = 0, y = 1152, width = 1080, height = 768 }, crop = { left = 0.1, right = 0.1 } } }, + { source = "bar", rect = { x = 80, y = 860, width = 900, height = 120 }, portrait = { rect = { x = 45, y = 948, width = 506, height = 68 } } }, + { source = "logo", rect = { x = 1760, y = 40, width = 120, height = 120 }, format = "landscape" }, +] + +[[scenes]] +name = "Guests" +portrait = { arrangement = "row" } +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1920 } } }, + { source = "Main", slot = "main" }, + { source = "Camera", slot = "tile" }, + { source = "Dana", slot = "tile" }, +] + +[[scenes]] +name = "Break" +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1920 } } }, + { source = "bar", rect = { x = 460, y = 480, width = 1000, height = 120 } }, +] diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 7999791a..eccde6af 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -732,7 +732,8 @@ pub struct Stats { /// hold. pub recording: Arc, /// Each published scene's video, by the scene's name, while it is - /// published. + /// published, and the portrait picture's, as [`moq_scene::PORTRAIT`], + /// while the show goes out in both. published: Mutex>>, /// The video encoders open: the program's, each published scene's, the /// recording's, and the sources'. @@ -1794,13 +1795,20 @@ impl Engine { // On air, each published scene's line starts once it is // published, in a broadcast of its own, and ends once it no - // longer is. + // longer is, as does the portrait picture's while the show + // goes out in portrait. let published = published(&state.collection); + let portrait = state.collection.portrait_canvas(); + let extras: Vec<(String, Canvas)> = published + .iter() + .cloned() + .chain(portrait.map(|canvas| (moq_scene::PORTRAIT.to_string(), canvas))) + .collect(); if let Some(air) = &mut air { let gone: Vec = air .lines .keys() - .filter(|line| !published.iter().any(|(name, _)| name == *line)) + .filter(|line| !extras.iter().any(|(name, _)| name == *line)) .cloned() .collect(); for name in gone { @@ -1809,7 +1817,7 @@ impl Engine { } self.stats.published.lock().unwrap().remove(&name); } - for (name, canvas) in &published { + for (name, canvas) in &extras { let started = !air.lines.contains_key(name); let extras = &air.extras; let line = air.lines.entry(name.clone()).or_insert_with(|| { @@ -1820,11 +1828,20 @@ impl Engine { .inspect_err(|err| video.fail(format!("{err:#}"))) .ok() }); - // Its audio follows the audio scene it carries from now on. + // Its audio follows the audio scene it carries from now on: + // the portrait picture's, the program's, with the sources + // published independently in it, as it carries no layout. + let portrait = portrait.is_some() && name == moq_scene::PORTRAIT; + let mix = match portrait { + true => state.sources.whole(&state.collection), + false => state + .collection + .published_audio(name) + .and_then(|audio| state.sources.mix(audio)), + }; if (synced || started) && let Some((_, broadcast)) = line - && let Some(audio) = state.collection.published_audio(name) - && let Some(mix) = state.sources.mix(audio) + && let Some(mix) = mix { broadcast.audio.set(mix.clone()); } @@ -1886,6 +1903,7 @@ impl Engine { } let program = (compositor.program().clone(), compositor.canvas()); let texture = |scene: &str| compositor.output(scene).and_then(moq_scene::Output::texture).cloned(); + let portrait = compositor.portrait().map(|(texture, canvas)| (texture.clone(), canvas)); Drawn { published: published .into_iter() @@ -1893,6 +1911,9 @@ impl Engine { let texture = texture(&name); (name, texture, canvas) }) + .chain( + portrait.map(|(texture, canvas)| (moq_scene::PORTRAIT.to_string(), Some(texture), canvas)), + ) .collect(), recorded: match recorded { Some(scene) => (texture(&scene), recorded_canvas), @@ -2629,7 +2650,9 @@ type Lines = BTreeMap>; struct Drawn { /// The program, at its canvas. program: (wgpu::Texture, Canvas), - /// Each published scene by name, unless not drawn, at its canvas. + /// Each published scene by name, unless not drawn, at its canvas, and + /// the portrait picture, as [`moq_scene::PORTRAIT`], while the show goes + /// out in portrait. published: Vec<(String, Option, Canvas)>, /// What recordings hold, unless not drawn, at its canvas. recorded: (Option, Canvas), @@ -4488,6 +4511,75 @@ mod tests { assert_eq!(engine.stats.sessions.open(), 0); } + /// A show going out in portrait too sends its portrait picture as a + /// broadcast of its own beneath the program's, turned on its side, with + /// the program's audio scene, holding an encode session of its own; + /// turning portrait off ends it, giving the session back. + #[tokio::test(flavor = "multi_thread")] + #[ignore = "needs a Vulkan device and an H.264 encoder; run by just render-sw"] + async fn the_portrait_picture_has_its_own_broadcast() { + let _ = moq_tokio::crypto::install_default(); + let fixture = moq_relay::test_relay().await.expect("test relay"); + let ready = fixture.relay.ready(); + let relay = fixture.relay.cluster().origin.consume(); + tokio::spawn(fixture.relay.run()); + ready.wait().await.expect("relay ready"); + + let mut client = moq_tokio::connect::Config::default(); + client.tls.fingerprint = vec![fixture.fingerprint.clone()]; + let client = client.init(Default::default()).expect("client"); + + let dir = tempfile::tempdir().unwrap(); + let gpu = Gpu::new(wgpu::Backends::all()).await.expect("a GPU"); + let (autosave, _) = Autosave::open(dir.path().join("show.toml")).unwrap(); + let relays = vec![Relay { + client, + url: fixture.url.clone(), + }]; + let mut loaded = red_canvas(); + // Red fills the landscape picture, and is fitted across the middle of + // the portrait one. + loaded.collection.portrait = true; + let engine = start(gpu, relays, dir.path(), autosave, loaded) + .await + .expect("the studio starts"); + + let path = format!("studio/{}", moq_scene::PORTRAIT); + let (broadcast, catalogs, catalog) = routed(&relay, &path).await; + assert!(catalog.audio.enabled().next().is_some(), "its audio"); + let (video, config) = catalog + .video + .renditions + .iter() + .find(|(_, config)| config.coded_width == Some(360)) + .expect("the canvas turned"); + assert_eq!(config.coded_height, Some(640)); + let (video, config) = (video.clone(), config.clone()); + let mut pictures = subscribe(&broadcast, &video, &config).await; + red(&mut pictures, moq_video::Size::new(360, 640)) + .await + .expect("the portrait picture arrives"); + assert_eq!(engine.stats.sessions.open(), 2, "one full size each"); + + let track = served(&broadcast, &video).await; + { + let mut state = engine.state.lock().unwrap(); + let mut draft = state.collection().draft("Landscape"); + draft.set_portrait(false).unwrap(); + state.commit(draft, None).unwrap(); + } + ended(catalogs, track).await; + kio::clock::timeout(TIMEOUT, broadcast.closed()) + .await + .expect("the portrait broadcast ends"); + assert_eq!(engine.stats.sessions.open(), 1, "its session given back"); + let stopping = engine.clone(); + tokio::task::spawn_blocking(move || stopping.stop()).await.unwrap(); + kio::clock::timeout(TIMEOUT, engine.close()) + .await + .expect("the studio closes"); + } + /// What a viewer of one broadcast takes in, frame by frame as it arrives, /// each stamped with its timestamp: the color at the center of each /// picture, and the loudest channel of each sample. diff --git a/rs/moq-studio/src/kit/widgets.rs b/rs/moq-studio/src/kit/widgets.rs index 5320ed9f..1426215b 100644 --- a/rs/moq-studio/src/kit/widgets.rs +++ b/rs/moq-studio/src/kit/widgets.rs @@ -1104,6 +1104,9 @@ pub struct Pad<'a> { width: f32, /// The thumbnail's shape, width over height: the show's. shape: f32, + /// The scene's portrait thumbnail and its shape, drawn at the right of + /// the landscape one, while the show goes out in both. + portrait: Option<(Option, f32)>, pose: Option, } @@ -1120,10 +1123,21 @@ impl<'a> Pad<'a> { dark: false, width: 124.0, shape: 16.0 / 9.0, + portrait: None, pose: None, } } + /// Beside the landscape thumbnail, the portrait one of `shape`, once + /// there is one, for a scene going out in both. The pad's own + /// [`shape`](Self::shape) is then the pair's. + pub fn portrait(mut self, picture: Option, shape: f32) -> Self { + if shape.is_finite() && shape > 0.0 { + self.portrait = Some((picture, shape)); + } + self + } + /// The scene's thumbnail, once there is one, fit to the pad whatever its /// shape. pub fn picture(mut self, picture: Option) -> Self { @@ -1215,9 +1229,18 @@ impl Widget for Pad<'_> { painter.rect(rect, 10, fill, Stroke::new(2.0, border), StrokeKind::Inside); let thumb = Rect::from_min_size(rect.min + vec2(4.0, 4.0), picture); painter.rect_filled(thumb, 7, BG); - if let Some(texture) = self.picture { - let fit = texture.size * (thumb.size() / texture.size).min_elem(); - let at = Rect::from_center_size(thumb.center(), fit); + // The landscape picture, and the portrait one at its right. + let mut pictures = vec![(thumb, self.picture)]; + if let Some((texture, shape)) = self.portrait { + let wide = (thumb.height() * shape).min(thumb.width() / 2.0); + let right = thumb.with_min_x(thumb.right() - wide); + pictures[0].0 = thumb.with_max_x(right.left() - 3.0); + pictures.push((right, texture)); + } + for (room, texture) in pictures { + let Some(texture) = texture else { continue }; + let fit = texture.size * (room.size() / texture.size).min_elem(); + let at = Rect::from_center_size(room.center(), fit); egui::Image::new(texture).corner_radius(7).paint_at(ui, at); } if self.refused.is_some() { diff --git a/rs/moq-studio/src/setup.rs b/rs/moq-studio/src/setup.rs index ec7ba09b..61ba1b75 100644 --- a/rs/moq-studio/src/setup.rs +++ b/rs/moq-studio/src/setup.rs @@ -85,7 +85,20 @@ impl Encoding { /// Whether the encoder keeps up with `canvas` in real time, within [`PACE`]. pub(crate) fn keeps_up(&self, canvas: Canvas) -> bool { - self.frame <= budget(canvas) + self.keeps_up_all(canvas, 1) + } + + /// Whether it keeps up with `encodes` pictures of `canvas` at once, such + /// as a show's landscape and portrait ones: each costs about a frame of + /// the one timed, as a turned picture has as many pixels. + pub(crate) fn keeps_up_all(&self, canvas: Canvas, encodes: u32) -> bool { + self.frame * encodes <= budget(canvas) + } + + /// What `encodes` pictures at this bitrate send, each a broadcast with + /// its own audio. + pub(crate) fn total_of(&self, encodes: u32) -> Rate { + Rate::from_bps(self.total().as_bps() * u64::from(encodes)) } /// How many times its [`budget`] a frame of `canvas` took, scaled to compare @@ -934,6 +947,14 @@ mod tests { assert_eq!(pick(&encoders, None), CANVASES[2], "the default does not keep up"); } + /// The portrait picture is a broadcast of its own, audio included. + #[test] + fn both_formats_send_the_audio_twice() { + let encoding = encoders(DEFAULTS)[1].1.clone().unwrap(); + assert_eq!(encoding.total_of(1), encoding.total()); + assert_eq!(encoding.total_of(2), Rate::from_kbps(2 * (4354 + 128))); + } + #[test] fn proposes_the_closest_to_keeping_up_when_none_does() { let encoders = timed([100, 80, 60, 40, 30]); diff --git a/rs/moq-studio/src/shots.rs b/rs/moq-studio/src/shots.rs index f70e5e51..c577ca51 100644 --- a/rs/moq-studio/src/shots.rs +++ b/rs/moq-studio/src/shots.rs @@ -130,6 +130,34 @@ enum View { /// BUILD's Add tab on the drop views' show, with a file dragged or /// dropped on the picture. Drop(Drop), + /// On the dual-format show, `page` with the layer of the scene on + /// program at `index`, front first, selected, BUILD editing `format`'s + /// picture, in Studio mode with the scene `cued` cued, if any, and the + /// scene `off_air` open off air, if any. + Dual { + page: Page, + format: moq_scene::Format, + index: Option, + cued: Option<&'static str>, + off_air: Option<&'static str>, + }, +} + +/// The studio on the dual-format views' show, at the mock-ups' size, turned +/// to `page` with the camera's layer selected, editing `format`'s picture. +fn dual(page: Page, format: moq_scene::Format) -> Open { + Open::Studio { + view: View::Dual { + page, + format, + index: Some(2), + cued: None, + off_air: None, + }, + size: SIZE, + show: "dual.toml", + relay: false, + } } /// A file over the picture, at the pointer. @@ -306,6 +334,34 @@ views! { build_drop_refused: Still, dropping(Drop::Refused); /// The logo dropped lands where it was let go, at its own size, selected. build_dropped: Still, dropping(Drop::Placed); + /// A show going out in both formats: the landscape and portrait + /// pictures side by side, each in its tally frame with its size under + /// it, where the selected camera is shown between them, and each pad + /// showing the pair, as the `dual` mock-up frame has it. + live_dual: Still, dual(Page::Live, moq_scene::Format::Landscape); + /// It in a half-size window: the chip of where the camera is shown keeps + /// clear of what is said under the landscape picture, without its words. + live_dual_compact: Still, Open::Studio { view: View::Dual { page: Page::Live, format: moq_scene::Format::Landscape, index: Some(2), cued: None, off_air: None }, size: HALF, show: "dual.toml", relay: false }; + /// Studio mode on it: NEXT shows the cued scene in both formats. + live_dual_studio: Still, Open::Studio { view: View::Dual { page: Page::Live, format: moq_scene::Format::Landscape, index: None, cued: Some("Guests"), off_air: None }, size: SIZE, show: "dual.toml", relay: false }; + /// BUILD on it: the landscape picture edited, the portrait one small + /// beside it, and under them the tabs that pick which and where the + /// selected camera is shown, which the inspector says too. + build_dual: Still, dual(Page::Build, moq_scene::Format::Landscape); + /// BUILD on it in a half-size window: the chip keeps clear of the tabs, + /// or leaves the choice to the inspector where there is no room. + build_dual_compact: Still, Open::Studio { view: View::Dual { page: Page::Build, format: moq_scene::Format::Landscape, index: Some(2), cued: None, off_air: None }, size: HALF, show: "dual.toml", relay: false }; + /// BUILD editing the portrait picture: the camera's handles at its + /// portrait place, the landscape picture small beside it. + build_dual_portrait: Still, dual(Page::Build, moq_scene::Format::Portrait); + /// A scene laid out by slots on it, open off air in portrait: its own + /// arrangement there, a row, which the inspector picks. + build_dual_slots: Still, Open::Studio { view: View::Dual { page: Page::Build, format: moq_scene::Format::Portrait, index: None, cued: None, off_air: Some("Guests") }, size: SIZE, show: "dual.toml", relay: false }; + /// Settings, Show on it: Both at once on. + settings_show_dual: Still, Open::Studio { view: View::Setting(Setting::Show), size: SIZE, show: "dual.toml", relay: false }; + /// GO LIVE's sheet on it: both broadcasts to each relay, and the encoder + /// and connection checked for both pictures. + going_dual: Live, Open::Studio { view: View::Going, size: SIZE, show: "dual.toml", relay: true }; /// MIX: a strip per sound and per audio scene, with their sends, and a /// small live picture in the sound's place. mix: Still, mixing(MixView::Strips, None, SIZE); @@ -1467,6 +1523,24 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show } } View::Drop(_) => studio.build(Side::Add, Inspect::Layer, None), + View::Dual { + page, + format, + index, + cued, + off_air, + } => { + studio.build(Side::Layers, Inspect::Layer, *index); + studio.edit_format(*format); + if let Some(cued) = cued { + studio.studio_mode(true); + studio.cue(cued); + } + studio.turn(*page); + if let Some(scene) = off_air { + studio.edit_off_air(scene); + } + } View::Review { picked, .. } => { studio.turn(Page::Review); studio.seed_shelf(shelf()); diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 98a23733..56b6eed9 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -12,8 +12,8 @@ use egui::{Color32, Ui}; use moq_scene::browser::Paint; use moq_scene::source::{Feed, Status}; use moq_scene::{ - Background, Blend, Canvas, Clip, Collection, Crop, Direction, Draft, Ease, Filter, Item, Keyframe, Layer, MaskMode, - Rect, Scene, Source, Thumbnail, Transition, Video, + Background, Blend, Canvas, Clip, Collection, Crop, Direction, Draft, Ease, Filter, Format, Item, Keyframe, Layer, + MaskMode, Placement, Rect, Scene, Source, Thumbnail, Transition, Video, }; use winit::keyboard::{KeyCode, PhysicalKey}; @@ -33,6 +33,7 @@ mod build; mod cue; mod dark; mod drop; +mod formats; mod graph; mod inspector; mod layers; @@ -232,6 +233,10 @@ impl Studio { program, program_drawn: Canvas::default(), preview, + portrait: None, + portrait_drawn: None, + portrait_preview: None, + format: Format::Landscape, editing: None, framed, page: Page::default(), @@ -332,6 +337,8 @@ impl Studio { let mut renderer = render.renderer.write(); for registered in [&panels.program, &panels.preview] .into_iter() + .chain(&panels.portrait) + .chain(&panels.portrait_preview) .chain(panels.thumbnails.values()) { renderer.free_texture(®istered.id); @@ -346,6 +353,11 @@ impl Studio { self.panels.page = page; } + /// Edit `format`'s picture on BUILD, as its tab under the picture would. + pub(crate) fn edit_format(&mut self, format: Format) { + self.panels.format = format; + } + /// Turn to MIX in `view`, with the strip of the sound called `opened` /// open, as clicking its name would. pub(crate) fn mix(&mut self, view: mixer::View, opened: Option<&str>) { @@ -574,6 +586,26 @@ impl Registered { } } + /// Show `texture` in `registered` from now on, while there is one, + /// registering it when it appears and giving it back when it goes, as the + /// portrait picture does when the show stops going out in portrait. + pub(crate) fn follow_while( + registered: &mut Option, + render: &eframe::egui_wgpu::RenderState, + texture: Option<&wgpu::Texture>, + ) { + match (registered.as_mut(), texture) { + (Some(shown), Some(texture)) => shown.follow(render, texture), + (None, Some(texture)) => *registered = Some(Self::new(render, texture)), + (Some(_), None) => { + if let Some(gone) = registered.take() { + render.renderer.write().free_texture(&gone.id); + } + } + (None, None) => {} + } + } + /// Show `texture` from now on, if a resize replaced the one shown. pub(crate) fn follow(&mut self, render: &eframe::egui_wgpu::RenderState, texture: &wgpu::Texture) { if self.texture == *texture { @@ -612,6 +644,15 @@ struct Panels { /// The cued scene, or the one open off air, which the compositor draws /// while there is one. preview: Registered, + /// The portrait picture as viewers see it, while the show goes out in + /// portrait. + portrait: Option, + /// Its size this frame, as its texture was drawn, while there is one. + portrait_drawn: Option, + /// The preview in portrait, while the show goes out in portrait. + portrait_preview: Option, + /// The picture BUILD edits a scene drawn in both in. + format: Format, /// The scene open for editing off air on BUILD, while it is the one cued. editing: Option, /// Whether the title bar is the window's frame, which moves, resizes, @@ -743,6 +784,23 @@ impl Picked { collection.items(&self.layer)?.get(self.index) } + /// The layer at rest in `format`'s picture, at its place there, if it is + /// still there: in the landscape one for a layer drawn in one. + fn rest(&self, collection: &Collection, format: Format) -> Option { + let format = collection.format_of(&self.layer, format); + Some(self.item(collection)?.portrayed(format, collection.canvas)) + } + + /// The layer placed at `rect` with `crop` in `format`'s picture, as it + /// is in the other, if it is still there and shown in that picture. + fn placed(&self, collection: &Collection, format: Format, rect: Rect, crop: Crop) -> Option { + let format = collection.format_of(&self.layer, format); + let item = self.item(collection)?.clone(); + // Not shown there, it has no place there. + item.shown_in(format) + .then(|| item.placed(format, Placement { rect, crop })) + } + /// What its look edits. fn look(&self) -> Selection { Selection::Item(self.layer.clone(), self.index) @@ -2131,7 +2189,7 @@ const REACH: f32 = 8.0; /// than `dragged`: what a dragged layer snaps to. A scene open off air still /// snaps to the always-on-top ones: its picture draws them, as the program /// will, though BUILD leaves them to the program to edit. -fn others(state: &State, scene: Option<&str>, dragged: &Picked, since: Duration) -> Vec { +fn others(state: &State, scene: Option<&str>, dragged: &Picked, since: Duration, format: Format) -> Vec { let collection = state.collection(); // The top layer goes over a scene of the program's size, as the // compositor draws it. @@ -2141,7 +2199,7 @@ fn others(state: &State, scene: Option<&str>, dragged: &Picked, since: Duration) let mut rects = Vec::new(); let layers = scene.map(Layer::scene).into_iter().chain(top.then(Layer::top)); for shown in layers { - let items = slots::placed(state, &shown); + let items = slots::placed(state, &shown, format); for (at, item) in items.iter().enumerate() { if item.visible && !(shown == dragged.layer && at == dragged.index) { rects.push(item.posed(since).rect); diff --git a/rs/moq-studio/src/ui/build.rs b/rs/moq-studio/src/ui/build.rs index 7724d50c..784136e6 100644 --- a/rs/moq-studio/src/ui/build.rs +++ b/rs/moq-studio/src/ui/build.rs @@ -44,14 +44,70 @@ impl Panels { kit::wash(ui.painter(), middle); let mut room = middle.shrink(28.0); let mut shape = snapshot.shape; - if let Some(cued) = snapshot.cued.as_ref().filter(|cued| cued.off_air) { + let off_air = snapshot.cued.as_ref().filter(|cued| cued.off_air); + if let Some(cued) = off_air { room.min.y = self.off_air_header(ui, cued, room, snapshot.shape); shape = cued.shape(); } - self.stage(ui, fx, snapshot, super::layout::fit(room, shape)); + // A scene drawn in both: the picture edited large, the other small + // beside it, and under them which is edited and where the selected + // layer is shown. + let dual = { + let state = self.engine.state.lock().unwrap(); + self.edits_both(&state) + }; + let pictures = match (dual, off_air) { + (Some(portrait), Some(cued)) => self.portrait_preview.as_ref().map(|portrait_preview| { + let landscape = (self.preview.id, cued.canvas); + (landscape, (portrait_preview.id, portrait)) + }), + (Some(portrait), None) => self + .portrait + .as_ref() + .map(|registered| ((self.program.id, self.program_drawn), (registered.id, portrait))), + (None, _) => None, + }; + let Some((landscape, portrait)) = pictures else { + self.format = Format::Landscape; + self.stage(ui, fx, snapshot, super::layout::fit(room, shape)); + self.build_keys(ui); + return; + }; + let (edited, other) = match self.format { + Format::Landscape => (landscape, portrait), + Format::Portrait => (portrait, landscape), + }; + let gap = super::formats::GAP; + room.max.y -= super::formats::UNDER; + // The other picture a third as tall as the edited one. + let shapes = [super::shell::shape(edited.1), super::shell::shape(other.1) / 3.0]; + let [main, beside] = super::layout::pair(room, shapes, gap); + let small = egui::Rect::from_min_size( + beside.min, + egui::vec2(beside.width(), beside.width() / super::shell::shape(other.1)), + ); + self.stage(ui, fx, snapshot, main); + self.other_picture(ui, small, other.0, other.1); + // Under both pictures: the tabs at the left, the chip at the right. + let under = egui::Rect::from_min_max( + egui::pos2(main.left(), main.bottom() + 18.0), + egui::pos2(main.right().max(small.right()), main.bottom() + 48.0), + ); + let tabs = self.format_tabs(ui, under); + self.shown_in(ui, under.with_min_x(tabs.right() + 12.0), None); self.build_keys(ui); } + /// The portrait picture's canvas while BUILD edits a scene drawn in both, + /// the one open off air or else the one on program. + pub(super) fn edits_both(&self, state: &State) -> Option { + let collection = state.collection(); + let scene = self.edited(state)?; + collection + .portrait_canvas() + .filter(|_| collection.dual(&Layer::scene(scene))) + } + /// The page's keys, unless a field or a page has the keyboard: A adds a /// layer, and Escape leaves cropping, then the selection. fn build_keys(&mut self, ui: &Ui) { @@ -98,10 +154,23 @@ impl Panels { }; (canvas, tag) }); - let (canvas, id, frame) = match off_air { - Some((canvas, tag)) => (canvas, self.preview.id, TallyFrame::new(Tally::Cue).tag(tag)), + // The portrait picture, while BUILD edits it. + let portrait = match self.format { + Format::Portrait => self.edits_both(&state).zip(match off_air { + Some(_) => self.portrait_preview.as_ref().map(|registered| registered.id), + None => self.portrait.as_ref().map(|registered| registered.id), + }), + Format::Landscape => None, + }; + if portrait.is_none() { + self.format = Format::Landscape; + } + let (canvas, id, frame) = match (off_air, portrait) { + (Some((_, tag)), Some((canvas, id))) => (canvas, id, TallyFrame::new(Tally::Cue).tag(tag)), + (Some((canvas, tag)), None) => (canvas, self.preview.id, TallyFrame::new(Tally::Cue).tag(tag)), // It is the live picture, so its frame says whether it is on air. - None => (self.program_drawn, self.program.id, self.program_frame()), + (None, Some((canvas, id))) => (canvas, id, self.program_frame()), + (None, None) => (self.program_drawn, self.program.id, self.program_frame()), }; let builder = egui::UiBuilder::new().max_rect(at.expand(10.0)); if self.interacting { @@ -120,7 +189,7 @@ impl Panels { // `interacted` found the page through the selection, so one is. let picked = self.selection.clone().expect("interacting with the selected layer"); let only = Some((&picked.layer, picked.index)); - let staged = super::slots::staged(&state, scene.map(|(scene, _)| scene)); + let staged = super::slots::staged(&state, scene.map(|(scene, _)| scene), self.format); let delivered = self.interact.update(ui, image, &staged, scene, only); deliver(&state.sources, delivered); self.rebind(&state.sources); @@ -211,7 +280,7 @@ impl Panels { let layers = self.scene(state).ok().into_iter().chain(top); layers .flat_map(|layer| { - let items = super::slots::placed(state, &layer); + let items = super::slots::placed(state, &layer, self.format); let posed = items.into_iter().map(move |item| item.posed(since).into_owned()); posed.enumerate().map(move |(index, item)| { let picked = Picked { @@ -282,10 +351,10 @@ impl Panels { let picked = shown[index].0.clone(); self.select(Some(picked.clone())); let collection = state.collection(); - let resting = picked.item(collection).cloned(); + let resting = picked.rest(collection, self.format); if let Some(resting) = resting { let drawn = match resting.slot { - Some(_) => super::slots::drawn(state, &picked.layer, picked.index), + Some(_) => super::slots::drawn(state, &picked.layer, picked.index, self.format), None => None, }; let drawn = drawn.unwrap_or((resting.rect, resting.crop)); @@ -359,7 +428,7 @@ impl Panels { let point = view.point(pos); // Whole canvas pixels, so an unsnapped drag saves round numbers. let delta = [(point[0] - drag.origin[0]).round(), (point[1] - drag.origin[1]).round()]; - let item = drag.picked.item(state.collection()).cloned(); + let item = drag.picked.rest(state.collection(), self.format); let held = ui.input(|input| input.modifiers); let (rect, crop) = match self.cropping { true => { @@ -385,7 +454,7 @@ impl Panels { height: shown.height - rest.height, }, }; - let others = others(state, scene.as_deref(), &drag.picked, since); + let others = others(state, scene.as_deref(), &drag.picked, since, self.format); snap.apply(drag.grab, rect, &others, &canvas) }); let rect = drag.hints.as_ref().map_or(rect, |snapped| snapped.rect); @@ -408,9 +477,11 @@ impl Panels { let label = format!("{verb} `{}`", item.source); spot = Some(Spot::Layer(drag.picked.layer.clone(), item.source.clone())); let picked = &drag.picked; - result = edit(state, label, drag.gesture, |draft| { - draft.set_item(&picked.layer, picked.index, Item { rect, crop, ..item }) - }); + if let Some(placed) = picked.placed(state.collection(), self.format, rect, crop) { + result = edit(state, label, drag.gesture, |draft| { + draft.set_item(&picked.layer, picked.index, placed) + }); + } } } if !response.dragged() { @@ -432,7 +503,7 @@ impl Panels { if let Some((picked, item)) = selected.filter(|(_, item)| item.visible) { if self.cropping { let collection = state.collection(); - let resting = picked.item(collection).map_or(item.crop, |item| item.crop); + let resting = picked.rest(collection, self.format).map_or(item.crop, |item| item.crop); view.whole(&painter, &item.rect, &resting); } // A layer in a slot has no size of its own to drag. @@ -521,9 +592,8 @@ impl Panels { } }); let Some(item) = picked - .item(state.collection()) + .rest(state.collection(), self.format) .filter(|item| !item.locked && item.slot.is_none()) - .cloned() else { return (None, Ok(())); }; @@ -541,8 +611,11 @@ impl Panels { }; let label = format!("Move `{}`", item.source); let spot = Spot::Layer(picked.layer.clone(), item.source.clone()); + let Some(moved) = picked.placed(state.collection(), self.format, rect, item.crop) else { + return (None, Ok(())); + }; let result = edit(state, label, Some(gesture), |draft| { - draft.set_item(&picked.layer, picked.index, Item { rect, ..item }) + draft.set_item(&picked.layer, picked.index, moved) }); (Some(spot), result) } @@ -675,22 +748,35 @@ impl Panels { "`{name}` is in a slot, which places it; set it free to place it yourself" )), LayerAction::Slot(slot) => super::slots::set_slot(state, picked, slot), - LayerAction::Fit => { - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); - let rect = match aspect(state, &item, true) { - Some(aspect) => snap::fit(aspect, &canvas), - None => Rect::fill(&canvas), + LayerAction::Fit | LayerAction::Fill => { + // In the picture BUILD edits. + let format = collection.format_of(layer, self.format); + let canvas = match collection.dual(layer) { + true => collection.canvas_of(format), + false => collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas), }; - set(state, format!("Fit `{name}`"), Item { rect, ..item }) - } - LayerAction::Fill => { - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); - let crop = match aspect(state, &item, false) { - Some(aspect) => snap::fill(aspect, &canvas), - None => Crop::default(), + if !item.shown_in(format) { + let why = format!("`{name}` is not shown in the {} picture", format.name().to_lowercase()); + return (spot, Err(anyhow::anyhow!(why))); + } + let rest = item.portrayed(format, collection.canvas); + let (label, rect, crop) = match action { + LayerAction::Fit => { + let rect = match aspect(state, &rest, true) { + Some(aspect) => snap::fit(aspect, &canvas), + None => Rect::fill(&canvas), + }; + (format!("Fit `{name}`"), rect, rest.crop) + } + _ => { + let crop = match aspect(state, &rest, false) { + Some(aspect) => snap::fill(aspect, &canvas), + None => Crop::default(), + }; + (format!("Fill with `{name}`"), Rect::fill(&canvas), crop) + } }; - let rect = Rect::fill(&canvas); - set(state, format!("Fill with `{name}`"), Item { rect, crop, ..item }) + set(state, label, item.placed(format, Placement { rect, crop })) } LayerAction::Shape(corners) => { let shape = moq_scene::Shape { corners, ..item.shape }; @@ -736,13 +822,22 @@ impl Panels { }; let count = collection.items(&to).map_or(0, <[Item]>::len); // The always-on-top layers never go on, so they have no - // keyframes to play, nor slots; it stays where it was drawn. - let (rect, crop) = super::slots::drawn(state, layer, index).unwrap_or((item.rect, item.crop)); + // keyframes to play, nor slots; it stays where it was drawn, + // in each picture it goes to. + let drawn = |format| super::slots::drawn(state, layer, index, format).unwrap_or((item.rect, item.crop)); + let (rect, crop) = drawn(Format::Landscape); + let dual = collection.dual(&to) && collection.dual(layer); + let portrait = dual.then(|| { + let (rect, crop) = drawn(Format::Portrait); + Placement { rect, crop } + }); let moved = Item { keyframes: if top { Vec::new() } else { item.keyframes.clone() }, slot: None, rect, crop, + format: item.format.filter(|_| dual), + portrait: portrait.filter(|_| item.shown_in(Format::Portrait)), ..item.clone() }; let label = match top { diff --git a/rs/moq-studio/src/ui/cue.rs b/rs/moq-studio/src/ui/cue.rs index 1aca9a96..f68183d2 100644 --- a/rs/moq-studio/src/ui/cue.rs +++ b/rs/moq-studio/src/ui/cue.rs @@ -13,6 +13,8 @@ pub(super) struct Cued { pub(super) refused: Option, /// Its size, which its picture is drawn at. pub(super) canvas: Canvas, + /// Its portrait picture's size, while it goes out in both. + pub(super) portrait: Option, /// Whether it is open for editing off air, rather than only cued. pub(super) off_air: bool, /// Why viewers see it anyway, if they do, such as being published. @@ -24,10 +26,22 @@ pub(super) struct Cued { impl Cued { /// Its picture's shape, width over height. pub(super) fn shape(&self) -> f32 { - self.canvas.width as f32 / self.canvas.height.max(1) as f32 + super::shell::shape(self.canvas) + } + + /// The shape of its pictures side by side, while it goes out in both, + /// with room between them, or else of its picture. + fn card_shape(&self) -> f32 { + match self.portrait { + Some(portrait) => super::layout::pair_shape([self.shape(), super::shell::shape(portrait)], CARD_GAP), + None => self.shape(), + } } } +/// The gap between the NEXT card's pictures, as a share of their height. +const CARD_GAP: f32 = 0.06; + /// A corner of the live picture, where the NEXT card docks. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) enum Corner { @@ -76,7 +90,7 @@ impl Panels { let inner = width - 2.0 * PADDING; // The rest of the card is about 120 points tall; the picture gets what // the live picture leaves. - let tall = (inner / cued.shape()) + let tall = (inner / cued.card_shape()) .min(picture.height() - 2.0 * INSET - 120.0) .max(24.0); let (header, buttons, hint) = (24.0, 40.0, 14.0); @@ -140,11 +154,20 @@ impl Panels { // The cued scene as it will go on air, framed in amber. let frame = Rect::from_min_size(pos2(card.left() + PADDING, y), vec2(inner, tall)); painter.rect_filled(frame, 8, kit::BG); - let shown = vec2(cued.canvas.width as f32, cued.canvas.height as f32); - let fit = shown * (frame.shrink(3.0).size() / shown).min_elem(); - egui::Image::new((self.preview.id, fit)) - .corner_radius(6) - .paint_at(ui, Rect::from_center_size(frame.center(), fit)); + // Both of its pictures, side by side, while it goes out in both. + let room = frame.shrink(3.0); + let pictures = match (cued.portrait, &self.portrait_preview) { + (Some(portrait), Some(registered)) => { + let shapes = [cued.shape(), super::shell::shape(portrait)]; + let gap = CARD_GAP * room.height(); + let [wide, tall] = super::layout::pair(room, shapes, gap); + vec![(self.preview.id, wide), (registered.id, tall)] + } + _ => vec![(self.preview.id, super::layout::fit(room, cued.shape()))], + }; + for (id, at) in pictures { + egui::Image::new((id, at.size())).corner_radius(6).paint_at(ui, at); + } ui.painter() .rect_stroke(frame, 8, Stroke::new(2.0, kit::CUE), egui::StrokeKind::Inside); y = frame.bottom() + 8.0; diff --git a/rs/moq-studio/src/ui/drop.rs b/rs/moq-studio/src/ui/drop.rs index 2c498927..212f9feb 100644 --- a/rs/moq-studio/src/ui/drop.rs +++ b/rs/moq-studio/src/ui/drop.rs @@ -53,6 +53,9 @@ impl Arrival { struct Landing { /// The layers it joins. layer: Layer, + /// The picture it lands in, BUILD's while it edits a scene drawn in both, + /// and that picture's canvas. + format: Format, canvas: Canvas, /// Its center, in canvas pixels. at: [f32; 2], @@ -538,7 +541,11 @@ impl Panels { fn landing(&self, state: &State, at: Option<[f32; 2]>) -> Result { let layer = self.scene(state)?; let collection = state.collection(); - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); + let format = collection.format_of(&layer, self.format); + let canvas = match collection.dual(&layer) { + true => collection.canvas_of(format), + false => collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas), + }; let center = [canvas.width as f32 / 2.0, canvas.height as f32 / 2.0]; let onto = at.and_then(|at| { let shown = self.shown(state); @@ -548,6 +555,7 @@ impl Panels { }); Ok(Landing { layer, + format, canvas, at: at.unwrap_or(center), onto, @@ -738,13 +746,36 @@ impl Panels { let layer = &landing.layer; let heard = source.has_audio(); let index = state.collection().items(layer).map_or(0, <[Item]>::len); + // Landed in the portrait picture, it sits there; in the landscape + // one, it lands centered at the same size until moved there. One that + // landed in portrait before its layers stopped being drawn in both, + // such as while its file was read, lands centered in their one. + let collection = state.collection(); + let centered = |canvas: Canvas, rect: Rect| { + let center = [canvas.width as f32 / 2.0, canvas.height as f32 / 2.0]; + land(canvas, [rect.width, rect.height], center) + }; + let item = rect.map(|rect| match (landing.format, collection.dual(layer)) { + (Format::Landscape, _) => Item::new(name.clone(), rect), + (Format::Portrait, true) => Item { + portrait: Some(Placement { + rect, + crop: Crop::default(), + }), + ..Item::new(name.clone(), centered(collection.canvas, rect)) + }, + (Format::Portrait, false) => { + let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); + Item::new(name.clone(), centered(canvas, rect)) + } + }); edit(state, format!("Drop `{name}` {}", place(layer)), None, |draft| { draft.set_source(&name, Some(source))?; if heard { draft.hear(&name)?; } - if let Some(rect) = rect { - draft.insert_item(layer, index, Item::new(name.clone(), rect))?; + if let Some(item) = item { + draft.insert_item(layer, index, item)?; } Ok(()) })?; diff --git a/rs/moq-studio/src/ui/formats.rs b/rs/moq-studio/src/ui/formats.rs new file mode 100644 index 00000000..81e2502a --- /dev/null +++ b/rs/moq-studio/src/ui/formats.rs @@ -0,0 +1,215 @@ +//! A show going out in landscape and portrait at once: what is said under +//! its two pictures, where a layer is shown, and which picture BUILD edits. + +use egui::{Align2, Sense, Stroke, StrokeKind, pos2, vec2}; +use moq_net::bandwidth::Rate; + +use super::*; + +/// Room between the two pictures on LIVE and BUILD. +pub(super) const GAP: f32 = 32.0; +/// Room under them for what is said about them. +pub(super) const UNDER: f32 = 48.0; + +/// Where a layer may be shown, as the chip and the inspector offer it: +/// `None` for both pictures. +pub(super) fn shown_in_options() -> [(Option, &'static str); 3] { + [ + (None, "Both"), + (Some(Format::Landscape), "Landscape"), + (Some(Format::Portrait), "Portrait"), + ] +} + +/// `item` shown only in `format`, or in both for `None`. Shown only in +/// landscape, it keeps no place in portrait. +pub(super) fn shown(item: &Item, format: Option) -> Item { + let portrait = match format { + Some(Format::Landscape) => None, + _ => item.portrait, + }; + Item { + format, + portrait, + ..item.clone() + } +} + +/// The edit's label for showing `name` as [`shown`] does. +pub(super) fn shown_label(name: &str, format: Option) -> String { + match format { + None => format!("Show `{name}` in both pictures"), + Some(format) => format!("Show `{name}` only in {}", format.name().to_lowercase()), + } +} + +/// Under a picture, in `rect`: its format, its size, and what its video sent +/// over the last interval, while on air. The part of `rect` left free +/// between them. +pub(super) fn format_label( + ui: &Ui, + rect: egui::Rect, + format: Format, + canvas: Canvas, + sent: Option, +) -> egui::Rect { + let painter = ui.painter(); + let mid = rect.center().y; + mark(painter, pos2(rect.left() + 7.0, mid), format, kit::DIM); + let name = kit::caps_galley(painter, format.name(), kit::display_bold(11.5), kit::TEXT, None); + let x = rect.left() + 20.0; + let width = name.size().x; + painter.galley(pos2(x, mid - name.size().y / 2.0), name, kit::TEXT); + let size = painter.text( + pos2(x + width + 10.0, mid), + Align2::LEFT_CENTER, + format!("{} × {}", canvas.width, canvas.height), + kit::ui(12.0), + kit::DIM, + ); + let mut free = rect.with_min_x(size.right() + 16.0); + if let Some(sent) = sent { + let text = painter.text( + pos2(rect.right(), mid), + Align2::RIGHT_CENTER, + crate::health::rate(sent), + kit::ui(12.0), + kit::DIM, + ); + painter.circle_filled(pos2(text.left() - 9.0, mid), 3.5, kit::OK); + free.max.x = text.left() - 24.0; + } + free +} + +/// A small outline of `format`'s shape, centered at `at`, in `ink`. +pub(super) fn mark(painter: &egui::Painter, at: egui::Pos2, format: Format, ink: Color32) { + let size = match format { + Format::Landscape => vec2(13.0, 9.0), + Format::Portrait => vec2(8.0, 12.0), + }; + let rect = egui::Rect::from_center_size(at, size); + painter.rect_stroke(rect, 2, Stroke::new(1.3, ink), StrokeKind::Inside); +} + +impl Panels { + /// In `rect`, under the pictures, centered on `center` as near as it fits, + /// or at its right for `None`: where the selected layer is shown, both + /// pictures, landscape, or portrait, for a layer drawn in both. Picking + /// another shows it there. Without its words where `rect` is too narrow + /// for them, and not at all where it is too narrow for the choices, which + /// the inspector offers too. + pub(super) fn shown_in(&mut self, ui: &mut Ui, rect: egui::Rect, center: Option) { + let engine = self.engine.clone(); + let mut state = engine.state.lock().unwrap(); + self.follow_reloads(&state); + let Some(picked) = self.selection.clone() else { return }; + let collection = state.collection(); + if !collection.dual(&picked.layer) { + return; + } + let Some(item) = picked.item(collection).cloned() else { + return; + }; + let options = shown_in_options(); + let font = kit::ui_strong(12.5); + let words = format!("{} is shown in", item.source); + let words = ui.painter().layout_no_wrap(words, kit::ui(12.0), kit::DIM); + let choices: f32 = options + .iter() + .map(|(_, name)| { + ui.painter() + .layout_no_wrap(name.to_string(), font.clone(), kit::TEXT) + .size() + .x + 28.0 + }) + .sum::() + + 4.0; + let full = 12.0 + words.size().x + 10.0 + choices + 4.0; + let (width, words) = match full <= rect.width() { + true => (full, Some(words)), + false => (choices + 8.0, None), + }; + if width > rect.width() { + return; + } + let size = vec2(width, 42.0); + let x = match center { + Some(center) => (center - size.x / 2.0).clamp(rect.left(), rect.right() - size.x), + None => rect.right() - size.x, + }; + let chip = egui::Rect::from_min_size(pos2(x, rect.center().y - 21.0), size); + ui.painter() + .rect(chip, 10, kit::P1, Stroke::new(1.0, kit::LINE), StrokeKind::Inside); + if let Some(words) = words { + let at = pos2(chip.left() + 12.0, chip.center().y - words.size().y / 2.0); + ui.painter().galley(at, words, kit::DIM); + } + let segmented = egui::Rect::from_min_size( + pos2(chip.right() - 4.0 - choices, chip.top() + 4.0), + vec2(choices, chip.height() - 8.0), + ); + let mut format = item.format; + ui.put(segmented, kit::Segmented::new(&mut format, &options)); + if format == item.format { + return; + } + let name = item.source.clone(); + let shown = shown(&item, format); + let spot = Spot::Layer(picked.layer.clone(), name.clone()); + let result = edit(&mut state, shown_label(&name, format), None, |draft| { + draft.set_item(&picked.layer, picked.index, shown) + }); + drop(state); + self.fail(spot, result); + } + + /// BUILD's choice of the picture it edits, a scene drawn in both: the + /// landscape one or the portrait one, at the left of `rect`; where it is. + pub(super) fn format_tabs(&mut self, ui: &mut Ui, rect: egui::Rect) -> egui::Rect { + let options = [(Format::Landscape, "Landscape"), (Format::Portrait, "Portrait")]; + let mut format = self.format; + let at = egui::Rect::from_min_size(rect.left_center() - vec2(0.0, 17.0), vec2(rect.width(), 34.0)); + let tabs = ui.scope_builder(egui::UiBuilder::new().max_rect(at), |ui| { + ui.add(kit::Segmented::new(&mut format, &options)) + .on_hover_text("The picture to edit: each places the layers on its own") + .rect + }); + self.format = format; + tabs.inner + } + + /// The picture BUILD is not editing, small in `rect`, `id` drawn on + /// `canvas`, with its format under it. A click edits it instead. + pub(super) fn other_picture(&mut self, ui: &mut Ui, rect: egui::Rect, id: egui::TextureId, canvas: Canvas) { + let other = self.format.other(); + let response = ui + .put( + rect, + egui::Image::new((id, rect.size())) + .corner_radius(6) + .sense(Sense::click()), + ) + .on_hover_text(format!("Edit the {} picture", other.name().to_lowercase())) + .on_hover_cursor(egui::CursorIcon::PointingHand); + let ink = match response.hovered() { + true => kit::FAINT, + false => kit::LINE, + }; + ui.painter() + .rect_stroke(rect, 6, Stroke::new(1.5, ink), StrokeKind::Outside); + let under = egui::Rect::from_min_size(rect.left_bottom() + vec2(0.0, 8.0), vec2(rect.width(), 18.0)); + let painter = ui.painter(); + mark(painter, pos2(under.left() + 7.0, under.center().y), other, kit::DIM); + painter.text( + pos2(under.left() + 18.0, under.center().y), + Align2::LEFT_CENTER, + format!("{} × {}", canvas.width, canvas.height), + kit::ui(11.0), + kit::DIM, + ); + if response.clicked() { + self.format = other; + } + } +} diff --git a/rs/moq-studio/src/ui/inspector.rs b/rs/moq-studio/src/ui/inspector.rs index bead4aea..f0434b07 100644 --- a/rs/moq-studio/src/ui/inspector.rs +++ b/rs/moq-studio/src/ui/inspector.rs @@ -124,7 +124,51 @@ impl Panels { asked = Some(Asked::Act(LayerAction::Slot(slot))); } } - let mut rect = item.rect; + // Where it is shown, and its place in the picture BUILD edits, for a + // layer drawn in both. + let format = self.format; + if collection.dual(&picked.layer) { + let mut shown = item.format; + ui.horizontal(|ui| { + ui.label("Format"); + let options = super::formats::shown_in_options(); + ui.add(kit::Segmented::new(&mut shown, &options)) + .on_hover_text("Which pictures show it: both, or only the landscape or the portrait one"); + }); + if shown != item.format { + asked = Some(Asked::Item( + super::formats::shown(&item, shown), + super::formats::shown_label(&name, shown), + None, + )); + } + if item.slot.is_none() && item.shown_in(format) { + ui.label( + egui::RichText::new(format!("Its place in the {} picture", format.name().to_lowercase())) + .color(kit::DIM) + .font(kit::ui(11.0)), + ); + } + } + let rest = picked.rest(collection, format).unwrap_or_else(|| item.clone()); + let placed = |rect, crop| { + picked + .placed(collection, format, rect, crop) + .unwrap_or_else(|| item.clone()) + }; + let mut rect = rest.rect; + // A layer the picture BUILD edits never shows has no place there. + let placeable = item.shown_in(collection.format_of(&picked.layer, format)); + if !placeable { + ui.label( + egui::RichText::new(format!( + "Not shown in the {} picture; place it in the other", + format.name().to_lowercase() + )) + .color(kit::DIM) + .font(kit::ui(11.0)), + ); + } if item.slot.is_some() { ui.label( egui::RichText::new("Placed by the scene's arrangement; drag it onto another slot to swap them") @@ -132,7 +176,7 @@ impl Panels { .font(kit::ui(11.0)), ); } - if item.slot.is_none() { + if item.slot.is_none() && placeable { ui.horizontal(|ui| { ui.spacing_mut().item_spacing.x = 4.0; let width = ((ui.available_width() - 12.0) / 4.0).max(40.0); @@ -153,28 +197,26 @@ impl Panels { for (verb, response) in responses { if let Some(gesture) = changed(&response, &mut self.gestures) { let label = format!("{verb} `{name}`"); - asked = Some(Asked::Item(Item { rect, ..item.clone() }, label, gesture)); + asked = Some(Asked::Item(placed(rect, rest.crop), label, gesture)); } } }); } - let mut crop = item.crop; - ui.horizontal(|ui| { - ui.label(egui::RichText::new("Crop").color(kit::DIM)); - if let Some(gesture) = crop_ui(ui, &mut crop, &mut self.gestures) { - asked = Some(Asked::Item( - Item { crop, ..item.clone() }, - format!("Crop `{name}`"), - gesture, - )); - } - }) - .response - .on_hover_text("Percent of the picture trimmed from the left, top, right, and bottom"); + let mut crop = rest.crop; + if placeable { + ui.horizontal(|ui| { + ui.label(egui::RichText::new("Crop").color(kit::DIM)); + if let Some(gesture) = crop_ui(ui, &mut crop, &mut self.gestures) { + asked = Some(Asked::Item(placed(rest.rect, crop), format!("Crop `{name}`"), gesture)); + } + }) + .response + .on_hover_text("Percent of the picture trimmed from the left, top, right, and bottom"); + } ui.horizontal(|ui| { - let fits = match item.slot { - Some(_) => &[][..], - None => &[ + let fits = match (item.slot, placeable) { + (Some(_), _) | (_, false) => &[][..], + (None, true) => &[ ( "Fit", LayerAction::Fit, diff --git a/rs/moq-studio/src/ui/layout.rs b/rs/moq-studio/src/ui/layout.rs index be810095..b0b2066a 100644 --- a/rs/moq-studio/src/ui/layout.rs +++ b/rs/moq-studio/src/ui/layout.rs @@ -259,6 +259,28 @@ pub fn fit(rect: Rect, shape: f32) -> Rect { Rect::from_center_size(rect.center(), size.max(Vec2::ZERO)) } +/// The landscape and portrait pictures of a show going out in both, of +/// `shapes` (width over height), side by side `gap` apart: as tall as fits +/// in `rect`, and centered in it. +pub fn pair(rect: Rect, shapes: [f32; 2], gap: f32) -> [Rect; 2] { + let [landscape, portrait] = shapes; + let tall = rect + .height() + .min((rect.width() - gap).max(0.0) / (landscape + portrait)) + .max(0.0); + let wide = tall * (landscape + portrait) + gap; + let min = pos2(rect.center().x - wide / 2.0, rect.center().y - tall / 2.0); + let first = Rect::from_min_size(min, vec2(tall * landscape, tall)); + let second = Rect::from_min_size(pos2(first.right() + gap, min.y), vec2(tall * portrait, tall)); + [first, second] +} + +/// The shape, width over height, of the pair [`pair`] places, with a gap +/// `gap` of their height between them. +pub fn pair_shape(shapes: [f32; 2], gap: f32) -> f32 { + shapes[0] + shapes[1] + gap +} + fn area(rect: Rect) -> f32 { rect.width().max(0.0) * rect.height().max(0.0) } @@ -286,6 +308,23 @@ mod tests { ); } + /// Both pictures of a show going out in both fit side by side, as tall + /// as the room allows, with the gap between them. + #[test] + fn a_pair_fits_side_by_side() { + let room = Rect::from_min_size(pos2(0.0, 0.0), vec2(1000.0, 400.0)); + let [landscape, portrait] = pair(room, [LANDSCAPE, PORTRAIT], 20.0); + assert!(room.contains_rect(landscape.union(portrait))); + assert_eq!(landscape.height(), portrait.height()); + assert!((portrait.left() - landscape.right() - 20.0).abs() < 1e-3); + assert!((landscape.width() / landscape.height() - LANDSCAPE).abs() < 1e-3); + assert!((portrait.width() / portrait.height() - PORTRAIT).abs() < 1e-3); + // Too narrow for the full height, the pair is as wide as the room. + let narrow = Rect::from_min_size(pos2(0.0, 0.0), vec2(600.0, 400.0)); + let [landscape, portrait] = pair(narrow, [LANDSCAPE, PORTRAIT], 20.0); + assert!((portrait.right() - landscape.left() - 600.0).abs() < 1e-3); + } + /// For every size from the smallest up, of every shape of picture and /// count of sounds: every part is inside the window, none overlaps /// another, the picture is inside its stage and larger than every part diff --git a/rs/moq-studio/src/ui/live.rs b/rs/moq-studio/src/ui/live.rs index 887361e7..1b664b21 100644 --- a/rs/moq-studio/src/ui/live.rs +++ b/rs/moq-studio/src/ui/live.rs @@ -124,6 +124,9 @@ impl Going { /// stream for the uplink. struct Measuring { task: Task>, + /// How many pictures its ceiling was sized for, so turning a second on + /// measures again, past what both send. + encodes: u32, /// Dropping it ends the measurement. _stop: Option>, } @@ -294,7 +297,8 @@ impl Panels { return; } ctx.request_repaint_after(Duration::from_millis(100)); - let (show, program, canvas) = { + let broadcast = self.controls.stream.lock().unwrap().broadcast().to_string(); + let (show, program, canvas, both) = { let state = self.engine.state.lock().unwrap(); let path = self.engine.autosave.path(); let show = path @@ -303,7 +307,15 @@ impl Panels { .to_string_lossy() .into_owned(); let program = state.compositor.program_scene().map(str::to_string); - (show, program, state.collection().canvas) + let collection = state.collection(); + // Each relay carries both pictures, each its own broadcast. + let both = collection.portrait.then(|| { + format!( + "Both formats go to each relay: landscape at {broadcast}, portrait at {broadcast}/{}.", + moq_scene::PORTRAIT + ) + }); + (show, program, collection.canvas, both) }; // Not while going live, which keeps the sheet waiting on it, and // whose encoders a new check would compete with: once it ends. @@ -334,6 +346,10 @@ impl Panels { ui.add_space(10.0); heading(ui, "Where it goes"); let count = self.destinations(ui, &mut settings); + if let Some(both) = &both { + ui.add_space(4.0); + ui.label(egui::RichText::new(both).font(kit::ui(12.0)).color(kit::DIM)); + } ui.add_space(10.0); heading(ui, "Checks"); for (index, check) in checks.iter().enumerate() { @@ -405,8 +421,15 @@ impl Panels { /// Collect the checks that finished, and start measuring the connection /// once the encoder says what it sends. fn advance_going(&mut self) { + let encodes = match self.engine.state.lock().unwrap().collection().portrait { + true => 2, + false => 1, + }; let Some(going) = &mut self.going else { return }; going.encoder.poll(); + if going.uplink.as_ref().is_some_and(|uplink| uplink.encodes < encodes) { + going.uplink = None; + } if going.uplink.is_none() && let Task::Done(encoder) = &going.encoder { @@ -414,11 +437,15 @@ impl Panels { Ok(encoding) => { // Up to past what it sends, so a connection that carries // it says so rather than stopping at its rate. - let ceiling = encoding.total().scaled(1.0 / setup::headroom()); - measure(&self.controls, &self.runtime, ceiling) + let ceiling = encoding.total_of(encodes).scaled(1.0 / setup::headroom()); + Measuring { + encodes, + ..measure(&self.controls, &self.runtime, ceiling) + } } Err(_) => Measuring { task: Task::Done(Err("nothing encodes, so nothing would be sent".into())), + encodes, _stop: None, }, }); @@ -621,7 +648,13 @@ impl Panels { }, }); - // The encoder keeps up with the canvas. + // The encoder keeps up with the canvas: twice over, a picture in each + // format, while the show goes out in both. + let encodes = if collection.portrait { 2 } else { 1 }; + let formats = match encodes { + 1 => String::new(), + _ => " in both formats".to_string(), + }; let canvas = going.canvas; let send = smaller(canvas).map(|smaller| (format!("Send {} instead", named(smaller)), Fix::Send(smaller))); let encoding = match &going.encoder { @@ -637,11 +670,11 @@ impl Panels { level: None, fixes: Vec::new(), }, - Task::Done(Ok(encoding)) if encoding.keeps_up(canvas) => Check { + Task::Done(Ok(encoding)) if encoding.keeps_up_all(canvas, encodes) => Check { mark: Outcome::Passed, icon: Icon::Encoder, name: "Encoder", - said: format!("Keeps up with {}", named(canvas)), + said: format!("Keeps up with {}{formats}", named(canvas)), level: None, fixes: Vec::new(), }, @@ -650,8 +683,8 @@ impl Panels { icon: Icon::Encoder, name: "Encoder", said: format!( - "The encoder takes {} ms a frame, too slow for {}. Viewers would see it stutter.", - encoding.frame.as_millis(), + "The encoder takes {} ms a frame, too slow for {}{formats}. Viewers would see it stutter.", + (encoding.frame * encodes).as_millis(), named(canvas) ), level: None, @@ -696,14 +729,18 @@ impl Panels { fixes: Vec::new(), }, (Some(Task::Done(Ok(carries))), Some(encoding)) => { - let needs = encoding.total(); + let needs = encoding.total_of(encodes); + let sending = match encodes { + 1 => "the program", + _ => "the program in both formats", + }; match carries.scaled(setup::headroom()) >= needs { true => Check { mark: Outcome::Passed, icon: Icon::Uplink, name: "Connection", said: format!( - "Carries {}, and the program needs {}", + "Carries {}, and {sending} needs {}", health::rate(*carries), health::rate(needs) ), @@ -715,7 +752,7 @@ impl Panels { icon: Icon::Uplink, name: "Connection", said: format!( - "The connection carries {} and the program needs {}. Viewers would see it stall.", + "The connection carries {} and {sending} needs {}. Viewers would see it stall.", health::rate(*carries), health::rate(needs) ), @@ -1107,6 +1144,7 @@ fn measure(controls: &Controls, runtime: &tokio::runtime::Handle, ceiling: Rate) }); Measuring { task: task.unwrap_or_else(|err| Task::Done(Err(format!("cannot start the measurement: {err}")))), + encodes: 1, _stop: Some(stop), } } diff --git a/rs/moq-studio/src/ui/settings.rs b/rs/moq-studio/src/ui/settings.rs index 8048f512..3aedcb58 100644 --- a/rs/moq-studio/src/ui/settings.rs +++ b/rs/moq-studio/src/ui/settings.rs @@ -250,9 +250,10 @@ impl Panels { /// Settings, Show: the format, the canvas size and frame rate by name or /// by number, and each scene's own size. fn show_pane(&mut self, ui: &mut Ui) { - let (current, scenes) = { + let (current, scenes, both) = { let state = self.engine.state.lock().unwrap(); - (state.collection().canvas, state.collection().scenes.clone()) + let collection = state.collection(); + (collection.canvas, collection.scenes.clone(), collection.portrait) }; let shown = self.canvas.unwrap_or(current); let mut wanted = None; @@ -261,7 +262,7 @@ impl Panels { row( ui, "Format", - "One format for the whole show. The layout of this window follows it.", + "The shape of the show's picture. The layout of this window follows it.", |ui| { ui.spacing_mut().item_spacing.x = CARD_GAP; // Right to left, so Landscape comes first. @@ -280,6 +281,22 @@ impl Panels { }, ); + let mut on = both; + row( + ui, + "Both at once", + "Also send a portrait picture, for phones, as its own broadcast beside the landscape one. Each \ + scene gets a portrait arrangement of its layers, and the machine encodes both.", + |ui| { + let toggle = ui + .add_enabled(!portrait, kit::Toggle::new(&mut on).label("Landscape and portrait")) + .on_disabled_hover_text("A portrait show turns to landscape first, then adds portrait"); + if toggle.changed() { + self.set_both(on); + } + }, + ); + let named = named_size(current).filter(|_| self.canvas.is_none()); let mut size = named; let sizes: Vec<(Option, &str)> = SIZES @@ -514,20 +531,44 @@ impl Panels { "Set the canvas to {}x{} at {} fps", canvas.width, canvas.height, canvas.framerate ); - edit(&mut state, label, None, |draft| draft.set_canvas(canvas)) + // Only a landscape canvas goes out in portrait too. + let turning_off = canvas.height >= canvas.width && state.collection().portrait; + let result = edit(&mut state, label, None, |draft| { + if canvas.height >= canvas.width { + draft.set_portrait(false)?; + } + draft.set_canvas(canvas) + }); + // As turning Both off does, it removes the layers shown only in + // portrait, which may move the selected layer's index. + if result.is_ok() && turning_off { + self.deselect_item(); + } + result } Restart::Scene(name, size) => match state.collection().scenes.iter().position(|scene| scene.name == name) { Some(index) => { let scene = state.collection().scenes[index].clone(); + // The program's size, while the show goes out in portrait + // too, follows the program's, so the scene keeps its + // portrait picture and can still go on program. + let program = state.collection().canvas; + let size = size.filter(|size| { + !state.collection().portrait || (size.width, size.height) != (program.width, program.height) + }); let label = match size { Some(size) => format!("Size scene `{name}` {}x{}", size.width, size.height), None => format!("Size scene `{name}` as the program"), }; let result = edit(&mut state, label, None, |draft| { - draft.set_scene(index, Scene { size, ..scene }) + draft.set_scene(index, scene.sized(size)) }); if result.is_ok() { self.sizes.remove(&name); + // Given its own size, its layers shown only in portrait go. + if size.is_some() { + self.deselect_item(); + } } result } @@ -538,6 +579,24 @@ impl Panels { self.check(result); } + /// Send a portrait picture too, or stop, as an edit undo takes back. + fn set_both(&mut self, on: bool) { + let engine = self.engine.clone(); + let mut state = engine.state.lock().unwrap(); + let label = match on { + true => "Go out in landscape and portrait", + false => "Go out in landscape only", + }; + let result = edit(&mut state, label.to_string(), None, |draft| draft.set_portrait(on)); + drop(state); + // Turned off, the layers shown only in portrait go, which may move + // the selected layer's index. + if result.is_ok() && !on { + self.deselect_item(); + } + self.check(result); + } + /// Every edit, newest first; pick one to undo or redo to just after it. pub(super) fn history(&mut self, ui: &mut Ui) { let engine = self.engine.clone(); diff --git a/rs/moq-studio/src/ui/shell.rs b/rs/moq-studio/src/ui/shell.rs index 50433600..26678f59 100644 --- a/rs/moq-studio/src/ui/shell.rs +++ b/rs/moq-studio/src/ui/shell.rs @@ -33,16 +33,40 @@ pub(super) struct Snapshot { pub(super) takes: u64, /// The program's shape, width over height. pub(super) shape: f32, + /// The portrait picture's size as drawn, while the show goes out in both. + pub(super) portrait: Option, /// The scene cued, or open off air. pub(super) cued: Option, /// Whether Studio mode is on: a scene's pad cues it. pub(super) studio: bool, } +impl Snapshot { + /// A pad's thumbnail's shape, width over height: the program's, or, while + /// the show goes out in both, the pair's. + pub(super) fn pad_shape(&self) -> f32 { + match self.portrait { + Some(portrait) => super::layout::pair_shape([self.shape, shape(portrait)], PAIR_GAP), + None => self.shape, + } + } +} + +/// The shape of a picture of `canvas`, width over height. +pub(super) fn shape(canvas: Canvas) -> f32 { + canvas.width as f32 / canvas.height.max(1) as f32 +} + +/// The gap between a pair's pictures in a thumbnail, as a share of their +/// height. +const PAIR_GAP: f32 = 0.04; + /// A scene as its pad shows it. struct ScenePad { name: String, canvas: Canvas, + /// Whether it goes out in portrait too, so its pad shows both pictures. + dual: bool, /// Why it cannot go on program, if it cannot. refused: Option, /// Whether a layer it shows went dark or stalled. @@ -119,8 +143,13 @@ impl Panels { if let Some(render) = frame.wgpu_render_state() { self.program.follow(render, state.compositor.layered()); self.preview.follow(render, state.compositor.preview()); + let portrait = state.compositor.portrait(); + Registered::follow_while(&mut self.portrait, render, portrait.map(|(texture, _)| texture)); + let preview = state.compositor.portrait_preview(); + Registered::follow_while(&mut self.portrait_preview, render, preview.map(|(texture, _)| texture)); self.follow_thumbnails(render, &state.compositor); } + self.portrait_drawn = state.compositor.portrait().map(|(_, canvas)| canvas); // Redraw at the canvas rate, so the picture and the pads are live. let framerate = state.compositor.canvas().framerate; ctx.request_repaint_after(Duration::from_secs(1) / framerate); @@ -186,6 +215,7 @@ impl Panels { .map(|scene| ScenePad { name: scene.name.clone(), canvas: scene.canvas(collection.canvas), + dual: collection.dual(&Layer::scene(&scene.name)), refused: collection.refuses(&scene.name), dark: super::dark::draws_lost(collection, &scene.name, |name| state.lost(name, now).is_some()), }) @@ -209,6 +239,9 @@ impl Panels { let cued = cued.map(|name| Cued { refused: collection.refuses(&name), canvas: collection.canvas(Some(name.as_str())).unwrap_or(collection.canvas), + portrait: collection + .portrait_canvas() + .filter(|_| collection.dual(&Layer::scene(&name))), off_air: self.editing.as_ref() == Some(&name), live: state.seen(&name), transition: transition_words(&collection.transition), @@ -230,7 +263,8 @@ impl Panels { sounds, stream, takes: state.takes, - shape: self.program_drawn.width as f32 / self.program_drawn.height.max(1) as f32, + shape: shape(self.program_drawn), + portrait: self.portrait_drawn.filter(|_| self.portrait.is_some()), cued, studio: self.controls.studio(), } @@ -241,7 +275,13 @@ impl Panels { /// messages over them. pub(super) fn window(&mut self, ui: &mut Ui, fx: &mut Fx, snapshot: &Snapshot) { let window = ui.max_rect(); - let (size, shape, sounds) = (window.size(), snapshot.shape, snapshot.sounds.len()); + let (size, mut shape, sounds) = (window.size(), snapshot.shape, snapshot.sounds.len()); + // LIVE shows both pictures of a show going out in both, and room for + // what is said under them, at about the height they are drawn. + if let (Page::Live, Some(portrait)) = (self.page, snapshot.portrait) { + let pair = super::layout::pair_shape([shape, self::shape(portrait)], super::formats::GAP / 600.0); + shape = pair * 600.0 / (600.0 + super::formats::UNDER); + } let layout = match self.page { // MIX puts a live picture in the sound's slot. Page::Mix => Layout::monitor(size, shape, sounds), @@ -273,7 +313,10 @@ impl Panels { match self.page { Page::Live => { kit::wash(ui.painter(), stage); - let picture = self.on_air(ui, Some(fx), layout.picture).rect; + let picture = match snapshot.portrait { + Some(portrait) => self.both_on_air(ui, fx, layout, portrait), + None => self.on_air(ui, Some(fx), layout.picture).rect, + }; if let Some(cued) = &snapshot.cued { self.next(ui, fx, cued, picture, Corner::TopRight); } @@ -288,8 +331,20 @@ impl Panels { /// The live picture in its tally frame, filling `picture`. pub(super) fn on_air(&mut self, ui: &mut Ui, fx: Option<&mut Fx>, picture: Rect) -> egui::Response { - let id = self.program.id; - let canvas = self.program_drawn; + let (id, canvas) = (self.program.id, self.program_drawn); + self.framed(ui, fx, picture, id, canvas) + } + + /// The picture `id` of `canvas` in the program's tally frame, filling + /// `picture`. + fn framed( + &self, + ui: &mut Ui, + fx: Option<&mut Fx>, + picture: Rect, + id: egui::TextureId, + canvas: Canvas, + ) -> egui::Response { let builder = UiBuilder::new().max_rect(picture.expand(10.0)); let frame = self.program_frame(); ui.scope_builder(builder, |ui| { @@ -298,6 +353,52 @@ impl Panels { .inner } + /// Both live pictures of a show going out in both, each in its tally + /// frame, side by side in the stage, with each one's format, size, and + /// what it sends under it, and where the selected layer is shown between + /// them. The landscape picture's rect, which the NEXT card docks in. + fn both_on_air(&mut self, ui: &mut Ui, fx: &mut Fx, layout: &Layout, portrait: Canvas) -> Rect { + let room = layout.stage.shrink(20.0); + let room = room.with_max_y(room.bottom() - super::formats::UNDER); + let [wide, tall] = super::layout::pair(room, [shape(self.program_drawn), shape(portrait)], super::formats::GAP); + self.on_air(ui, Some(fx), wide); + let Some(id) = self.portrait.as_ref().map(|registered| registered.id) else { + return wide; + }; + self.framed(ui, Some(fx), tall, id, portrait); + let sent = self.sent(); + let under = |picture: Rect| { + Rect::from_min_size( + pos2(picture.left(), picture.bottom() + 18.0), + vec2(picture.width(), 24.0), + ) + }; + let free = super::formats::format_label(ui, under(wide), Format::Landscape, self.program_drawn, sent[0]); + super::formats::format_label(ui, under(tall), Format::Portrait, portrait, sent[1]); + // Centered under the landscape picture, clear of what is said there. + self.shown_in(ui, free, Some(wide.center().x)); + wide + } + + /// What the landscape and portrait broadcasts' video sent over the last + /// interval, while on air. + fn sent(&self) -> [Option; 2] { + let report = self.engine.health.lock().unwrap(); + let video = |name: Option<&str>| { + let broadcast = report + .broadcasts + .iter() + .find(|broadcast| broadcast.name.as_deref() == name)?; + let tracks = broadcast.tracks.iter().filter(|track| track.label.starts_with("video")); + let sent: Vec = tracks + .filter_map(|track| track.sent) + .map(|sent| sent.as_bps()) + .collect(); + (!sent.is_empty()).then(|| moq_net::bandwidth::Rate::from_bps(sent.into_iter().sum())) + }; + [video(None), video(Some(moq_scene::PORTRAIT))] + } + /// The title bar: the show's menu, undo and redo, the pages, the health /// dot, record, the live control, Settings, and the window's buttons. Its /// empty room moves the window. @@ -661,10 +762,12 @@ impl Panels { return; } // As wide as fits the strip's height, at the show's shape. - let tall = kit::Pad::height(PAD, snapshot.shape); + let shape = snapshot.pad_shape(); + let pad = PAD * shape.max(1.0) / snapshot.shape.max(1.0); + let tall = kit::Pad::height(pad, shape); let width = match tall > room.height() { - true => ((room.height() - 33.0) * snapshot.shape + 8.0).floor().max(48.0), - false => PAD, + true => ((room.height() - 33.0) * shape + 8.0).floor().max(48.0), + false => pad, }; egui::ScrollArea::horizontal().id_salt("scenes").show(ui, |ui| { ui.horizontal_centered(|ui| { @@ -763,7 +866,7 @@ impl Panels { if index > snapshot.scenes.len() { return None; } - let size = vec2(width, kit::Pad::height(width, snapshot.shape)); + let size = vec2(width, kit::Pad::height(width, snapshot.pad_shape())); let (rect, response) = ui.allocate_exact_size(size, Sense::click()); let response = kit::tip(response, "Add a scene", None); let ink = if response.hovered() { kit::TEXT } else { kit::FAINT }; @@ -775,7 +878,7 @@ impl Panels { let cued = snapshot.cued.as_ref().is_some_and(|cued| cued.name == pad.name); let live = self.engine.on_air().live().is_some(); let (tally, tag) = tally(on_air, live, cued); - let response = self.pad(ui, pad, tally, tag, width, snapshot.shape); + let response = self.pad(ui, pad, tally, tag, width, snapshot); let click = match snapshot.studio { true => PadAction::Cue(pad.name.clone()), false => PadAction::Take(pad.name.clone()), @@ -854,7 +957,8 @@ impl Panels { } /// A scene's pad, `width` wide, with its thumbnail once the compositor - /// has drawn it, which it draws while the pad is on screen. + /// has drawn it, which it draws while the pad is on screen, and its + /// portrait one beside it while the show goes out in both. fn pad( &mut self, ui: &mut Ui, @@ -862,14 +966,25 @@ impl Panels { tally: Tally, tag: Option<&str>, width: f32, - shape: f32, + snapshot: &Snapshot, ) -> egui::Response { + let (shape, portrait) = (snapshot.pad_shape(), snapshot.portrait.filter(|_| pad.dual)); let thumbnail = Thumbnail::Scene(pad.name.clone()); let size = vec2(pad.canvas.width as f32, pad.canvas.height as f32); let picture = self .thumbnails .get(&thumbnail) .map(|registered| egui::load::SizedTexture::new(registered.id, size)); + // The portrait picture beside it, for a scene going out in both. + let second = portrait.map(|canvas| { + let thumbnail = Thumbnail::Portrait(pad.name.clone()); + let size = vec2(canvas.width as f32, canvas.height as f32); + let picture = self + .thumbnails + .get(&thumbnail) + .map(|registered| egui::load::SizedTexture::new(registered.id, size)); + (thumbnail, picture, self::shape(canvas)) + }); let key = self.key(&Action::Scene(pad.name.clone()), ui.ctx()); let fault = self.fault(&Spot::Pad(pad.name.clone())); let widget = kit::Pad::new(&pad.name) @@ -882,9 +997,14 @@ impl Panels { .fault(fault) .width(width) .shape(shape); + let widget = match &second { + Some((_, picture, shape)) => widget.portrait(*picture, *shape), + None => widget, + }; let response = ui.add(widget); if ui.is_rect_visible(response.rect) { self.thumbnails_shown.insert(thumbnail); + self.thumbnails_shown.extend(second.map(|(thumbnail, ..)| thumbnail)); } response } @@ -1353,20 +1473,30 @@ impl Panels { // Every published scene, encoding only on air. let encoding = stats.published(); - let published: Vec<(String, String)> = { + // And the portrait picture, while the show goes out in both, which + // carries the program's audio. + let published: Vec<(String, String, bool)> = { let state = self.engine.state.lock().unwrap(); let collection = state.collection(); let scenes = collection.scenes.iter().filter(|scene| scene.published); + let portrait = collection.portrait.then(|| { + let audio = collection.program_audio().unwrap_or_default().to_string(); + (moq_scene::PORTRAIT.to_string(), audio, true) + }); scenes .map(|scene| { let audio = collection.published_audio(&scene.name).unwrap_or_default().to_string(); - (scene.name.clone(), audio) + (scene.name.clone(), audio, false) }) + .chain(portrait) .collect() }; - for (name, audio) in published { + for (name, audio, portrait) in published { ui.separator(); - ui.strong(format!("Scene `{name}`")); + match portrait { + true => ui.strong("The portrait picture"), + false => ui.strong(format!("Scene `{name}`")), + }; if let Some(live) = on_air.live() { ui.label(format!("Broadcast {}/{name}, with audio scene `{audio}`", live.path)); } diff --git a/rs/moq-studio/src/ui/slots.rs b/rs/moq-studio/src/ui/slots.rs index e1aee325..3fedecb8 100644 --- a/rs/moq-studio/src/ui/slots.rs +++ b/rs/moq-studio/src/ui/slots.rs @@ -7,15 +7,21 @@ use moq_scene::{Arrangement, Crop, Slot}; use super::layers::section; use super::*; -/// `layer`'s layers where they are drawn, in order: those of a scene in -/// slots where the last render drew them, hidden while their slot is +/// `layer`'s layers where they are drawn in `format`'s picture, in order: +/// each at its place there, hidden unless shown there, and those of a scene +/// in slots where the last render drew them, hidden while their slot is /// closed. -pub(super) fn placed(state: &State, layer: &Layer) -> Vec { - let items = state.collection().items(layer).unwrap_or_default(); - let arranged = layer.scene.as_deref().and_then(|scene| state.compositor.placed(scene)); +pub(super) fn placed(state: &State, layer: &Layer, format: Format) -> Vec { + let collection = state.collection(); + let format = collection.format_of(layer, format); + let items = collection.items(layer).unwrap_or_default(); + let items: Vec = (items.iter()) + .map(|item| item.portrayed(format, collection.canvas)) + .collect(); + let arranged = (layer.scene.as_deref()).and_then(|scene| state.compositor.placed(format, scene)); // One frame behind an edit, until the next render arranges it. let Some(arranged) = arranged.filter(|arranged| arranged.len() == items.len()) else { - return items.to_vec(); + return items; }; let placed = items.iter().zip(arranged).map(|(item, arranged)| match item.slot { Some(_) => Item { @@ -28,35 +34,44 @@ pub(super) fn placed(state: &State, layer: &Layer) -> Vec { placed.collect() } -/// The collection with the scene called `scene` as the last render drew -/// it, cover crops and a glide under way included, so a page in a slot -/// takes input where it is drawn. -pub(super) fn staged<'a>(state: &'a State, scene: Option<&str>) -> std::borrow::Cow<'a, Collection> { +/// The collection as `format`'s picture draws it, with the scene called +/// `scene` as the last render drew it there, cover crops and a glide under +/// way included, so a page in a slot takes input where it is drawn. +pub(super) fn staged<'a>(state: &'a State, scene: Option<&str>, format: Format) -> std::borrow::Cow<'a, Collection> { let collection = state.collection(); + let format = scene.map_or(format, |scene| collection.format_of(&Layer::scene(scene), format)); let found = scene.and_then(|scene| { let index = collection.scenes.iter().position(|found| found.name == scene)?; - let arranged = state.compositor.placed(scene)?; + let arranged = state.compositor.placed(format, scene)?; (arranged.len() == collection.scenes[index].items.len()).then_some((index, arranged)) }); - let Some((index, arranged)) = found else { - return std::borrow::Cow::Borrowed(collection); - }; - let mut staged = collection.clone(); - staged.scenes[index].items = arranged.to_vec(); - std::borrow::Cow::Owned(staged) + let mut staged = collection.formatted(format); + if let Some((index, arranged)) = found { + staged.to_mut().scenes[index].items = arranged.to_vec(); + } + staged } -/// Where the layer at `index` of `layer` is drawn and what it shows of its -/// picture, cover crop included, so setting it free keeps how it looks; its -/// own place for a layer set free or not yet arranged. -pub(super) fn drawn(state: &State, layer: &Layer, index: usize) -> Option<(Rect, Crop)> { +/// Where the layer at `index` of `layer` is drawn in `format`'s picture and +/// what it shows of its picture, cover crop included, so setting it free +/// keeps how it looks; its own place there for a layer set free or not yet +/// arranged. +pub(super) fn drawn(state: &State, layer: &Layer, index: usize, format: Format) -> Option<(Rect, Crop)> { let collection = state.collection(); - let item = collection.items(layer)?.get(index)?; - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); + let format = collection.format_of(layer, format); + let item = collection + .items(layer)? + .get(index)? + .portrayed(format, collection.canvas); + let item = &item; + let canvas = match collection.dual(layer) { + true => collection.canvas_of(format), + false => collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas), + }; let arranged = layer .scene .as_deref() - .and_then(|scene| state.compositor.placed(scene)) + .and_then(|scene| state.compositor.placed(format, scene)) .filter(|arranged| arranged.len() == collection.items(layer).map_or(0, <[Item]>::len)) .and_then(|arranged| arranged.get(index)) .filter(|arranged| item.slot.is_some() && !arranged.rect.is_empty()); @@ -167,24 +182,37 @@ pub(super) fn assign(state: &mut State, dragged: &Picked, target: &Picked) -> an let taking = Item { slot: held.slot, rect: Rect::default(), + portrait: None, ..moved.clone() }; - // Set free in place of `moved`, cropped to its shape rather than - // stretched. - let (rect, crop) = match moved.slot { - Some(_) => (Rect::default(), held.crop), + // Set free in place of `moved`, in each picture, cropped to its shape + // there rather than stretched. + let free = |format: Format| { + let landscape = state.collection().canvas; + let Placement { rect, .. } = moved.placement(format, landscape); + let (from, crop) = drawn(state, &target.layer, target.index, format).unwrap_or((rect, held.crop)); + let shape = |rect: &Rect| rect.width / rect.height; + let crop = crop.then(&Crop::covering(shape(&from), shape(&rect))); + Placement { rect, crop } + }; + let leaving = match moved.slot { + Some(_) => Item { + slot: moved.slot, + rect: Rect::default(), + ..held + }, None => { - let (from, crop) = drawn(state, &target.layer, target.index).unwrap_or((moved.rect, held.crop)); - let shape = |rect: &Rect| rect.width / rect.height; - (moved.rect, crop.then(&Crop::covering(shape(&from), shape(&moved.rect)))) + let dual = state.collection().dual(&target.layer); + let landscape = free(Format::Landscape); + Item { + slot: None, + rect: landscape.rect, + crop: landscape.crop, + portrait: (dual && held.shown_in(Format::Portrait)).then(|| free(Format::Portrait)), + ..held + } } }; - let leaving = Item { - slot: moved.slot, - rect, - crop, - ..held - }; edit(state, label, None, |draft| { draft.set_item(&target.layer, target.index, taking)?; draft.set_item(&dragged.layer, dragged.index, leaving) @@ -206,9 +234,18 @@ pub(super) fn set_slot(state: &mut State, picked: &Picked, slot: Option) - return Ok(()); } let name = item.source.clone(); + // Set free where it is drawn in each picture. + let drawn = |format| drawn(state, &picked.layer, picked.index, format).unwrap_or((item.rect, item.crop)); let (rect, crop) = match slot { Some(_) => (Rect::default(), item.crop), - None => drawn(state, &picked.layer, picked.index).unwrap_or((item.rect, item.crop)), + None => drawn(Format::Landscape), + }; + let portrait = match slot { + None if collection.dual(&picked.layer) && item.shown_in(Format::Portrait) => { + let (rect, crop) = drawn(Format::Portrait); + Some(Placement { rect, crop }) + } + _ => None, }; let main = match slot { Some(Slot::Main) => collection @@ -239,6 +276,7 @@ pub(super) fn set_slot(state: &mut State, picked: &Picked, slot: Option) - slot, rect, crop, + portrait, ..item }, ) @@ -255,10 +293,11 @@ fn arrangement_name(arrangement: Arrangement) -> &'static str { } } -/// A button showing `arrangement`'s slots, a main item and three tiles, lit -/// while `on`. -fn arrangement_button(ui: &mut Ui, arrangement: Arrangement, on: bool) -> egui::Response { - let size = egui::vec2(58.0, 33.0); +/// A button showing `arrangement`'s slots on `canvas`, a main item and +/// three tiles, lit while `on`. +fn arrangement_button(ui: &mut Ui, arrangement: Arrangement, canvas: Canvas, on: bool) -> egui::Response { + // As tall as a landscape one, so a row of either lines up. + let size = egui::vec2(33.0 * canvas.width as f32 / canvas.height as f32, 33.0); let (rect, response) = ui.allocate_exact_size(size + egui::vec2(8.0, 8.0), egui::Sense::click()); response.widget_info(|| { egui::WidgetInfo::selected( @@ -280,7 +319,6 @@ fn arrangement_button(ui: &mut Ui, arrangement: Arrangement, on: bool) -> egui:: false => egui::Stroke::new(1.0, kit::LINE), }; painter.rect(rect, 6.0, fill, stroke, egui::StrokeKind::Inside); - let canvas = Canvas::default(); let picture = egui::Rect::from_center_size(rect.center(), size); let scale = size.x / canvas.width as f32; let (main, tiles) = arrangement.places(canvas, true, 3); @@ -311,12 +349,17 @@ enum Asked { impl Panels { /// The Slots section of the scene inspector for the scene at `index`: - /// how it arranges its slots, each slot and the role or source in it, - /// and a slot to add for the main picture or the camera; the result of - /// what it asked for, if it asked. + /// how it arranges its slots in the picture BUILD edits, each slot and + /// the role or source in it, and a slot to add for the main picture or + /// the camera; the result of what it asked for, if it asked. pub(super) fn slots(&mut self, ui: &mut Ui, state: &mut State, index: usize) -> Option> { let collection = state.collection(); let scene = collection.scenes[index].clone(); + let format = collection.format_of(&Layer::scene(&scene.name), self.format); + let (arranged, canvas) = match format { + Format::Landscape => (scene.arrangement, Canvas::default()), + Format::Portrait => (scene.portrait.arrangement, Canvas::default().turned()), + }; let room = self.engine.room(); let room = room.as_ref().map(|room| room.path()); let mut asked = None; @@ -328,10 +371,17 @@ impl Panels { .color(kit::DIM) .font(kit::ui(11.0)), ); + if collection.dual(&Layer::scene(&scene.name)) { + ui.label( + egui::RichText::new(format!("Arranged in {}", format.name().to_lowercase())) + .color(kit::DIM) + .font(kit::ui(11.0)), + ); + } ui.horizontal_wrapped(|ui| { ui.spacing_mut().item_spacing.x = 6.0; for arrangement in Arrangement::ALL { - if arrangement_button(ui, arrangement, scene.arrangement == arrangement).clicked() { + if arrangement_button(ui, arrangement, canvas, arranged == arrangement).clicked() { asked = Some(Asked::Arrange(arrangement)); } } @@ -408,16 +458,21 @@ impl Panels { let asked = asked?; Some(match asked { - Asked::Arrange(arrangement) => edit( - state, - format!( + Asked::Arrange(arrangement) => { + let label = format!( "Arrange `{}` as {}", scene.name, arrangement_name(arrangement).to_lowercase() - ), - None, - |draft| draft.set_scene(index, Scene { arrangement, ..scene }), - ), + ); + let scene = match format { + Format::Landscape => Scene { arrangement, ..scene }, + Format::Portrait => Scene { + portrait: moq_scene::Portrait { arrangement }, + ..scene + }, + }; + edit(state, label, None, |draft| draft.set_scene(index, scene)) + } Asked::Fill(role, source) => { let label = match &source { Some(source) => format!("Fill the role `{role}` with `{source}`"),