diff --git a/README.md b/README.md index 30ec323..fc4cd32 100644 --- a/README.md +++ b/README.md @@ -306,7 +306,7 @@ is named for it, with these sections: | Section | Use it to | | --- | --- | -| Place | Pick its [slot](#slots-and-roles), or none; type or drag its position and size, and its crop, in percent off each edge, fit or fill it, and lock it. A layer in a slot has no position or size of its own to type. | +| Place | Pick its [slot](#slots-and-roles), or none; type or drag its position and size, and its crop, in percent off each edge, fit or fill it, lock it, and switch on **Keep out of the way** (see [Keeping out of the way](#keeping-out-of-the-way)). A layer in a slot has no position or size of its own to type, and keeps out of the way only when set free. | | Shape | Pick its outline, square, rounded, circle, or arch, give it a border inside that outline and the border's color, and give it a soft shadow, which follows the outline and is the layer's first drop shadow filter. | | Look | Remove its source's background, which every layer showing the source loses (make it independent to cut out here alone); grade its color through a LUT source; add, order, set, and remove its filters, copy them, or paste copied filters after its own; and set its blend mode. | | Shown in | See how many places show its source, **Linked in 3 scenes**, and **Make independent**, which points this layer at a copy of the source of its own, keeping its look, so editing that look no longer edits the others. A copy says what it copies. | @@ -718,6 +718,40 @@ across restarts: after that, where they are shown is yours, so undoing their appearance or removing their layer keeps them off when they reconnect, until a new link. Since an appearance is your edit, it drops what you could redo. Off, every guest waits under Add, as before. +### Keeping out of the way + +A scene's layer with **Keep out of the way** on, in the inspector's Place +section, moves itself off what matters in the picture under it, such as a +facecam off a game's minimap or a slide's text. A few times a second the +studio draws the layers under it, scaled far down, and reduces them on the +GPU to a grid: where the picture moves, where it is dense with edges that hold +still, as text is, and what holds still while the rest moves, as a HUD does. +Each can be switched off under **Stays clear of**. Nothing is uploaded: the +choice is made on this machine, from the picture alone. + +It keeps to one of eight places, the canvas's corners and edge middles, +inside a margin of a twentieth of the canvas's shorter side, that its size +fits in, taking the one that covers the least of what matters, a corner over +a middle that covers as much. It chooses a second after its scene goes on, +about two seconds after the content under it changes kind (another layout, or +another app), and otherwise moves only once its place has been the worst of them for +10 seconds, never for a single busy moment. It glides there as a move +transition eases, in 0.4 s unless **Glide** says otherwise. Its rect in the +scene file gives its size, and where it rests until it first moves. + +While it is selected, BUILD shows what it saw: what matters glowing, tagged +**MOVING**, **TEXT**, or **HUD**, the places it may go, dashed, and where it +last moved from, with what it left there and an arrow to where it went. Drag +it, nudge it, or type its X or Y, and it stays where it was put until the +content changes, as it does where a redo or a reload of the scene file +moves it, or an undo moves it anywhere but back before such a move. Undo +that move and it goes back to where it was, such as the corner it had +glided to. A resize keeps it free to move. Switched off, it stays where it +is drawn, and undoing that puts it back keeping clear where it was. Only a +scene's layer set free of its [slots](#slots-and-roles) keeps out of the +way, and not one showing a source +[published independently](#published-independently), which viewers place +themselves; putting one in a slot switches it off. ### Audio scenes @@ -1230,7 +1264,12 @@ Each entry of a scene's `items` is one of its layers, back to front, and each canvas, in pixels, or, in a scene, its `slot`, `main` (one per scene) or `tile`, in place of a `rect`, and optionally `visible = false`, `locked = true`, which the picture's pointer passes through, and its `crop`, `filters`, `blend`, -`shape`, and `keyframes`. +`shape`, `keyframes`, and `keep_clear`. + +A scene's layer with `keep_clear = {}` [keeps out of the +way](#keeping-out-of-the-way) of motion, text, and a HUD; `motion = false`, +`text = false`, or `hud = false` in it ignores one, and `glide` sets how long +a move takes, `"400ms"` unless it says otherwise, up to `"5s"`. A layer's `shape` cuts it to an outline, before its filters, so a drop shadow follows it: `corners`, how round each corner is (top-left, top-right, @@ -1438,7 +1477,8 @@ published, holding an encode session as one on program does. Every player must reproduce how it is placed, so a layer showing it takes no filter but opacity, no blend mode, and no shape, nor do its source's filters or -its keyframes, and no scene that another scene draws can place it. A source with +its keyframes, it does not keep out of the way, and no scene that another scene +draws can place it. A source with its background removed cannot be published, since H.264 carries no transparency and its track would show the background. diff --git a/quest/m1/auto-place-check.md b/quest/m1/auto-place-check.md new file mode 100644 index 0000000..a9da4ea --- /dev/null +++ b/quest/m1/auto-place-check.md @@ -0,0 +1,28 @@ +# [XS] Live with a camera that keeps out of the way + +## Goal + +The maintainer has streamed a few times with a facecam set to **Keep out of +the way** over real games and slides, and it moves when it should and no +more often. If it does not, the timings or weights it chooses by change until +it does. + +## Plan + +The analysis, the choice, and the glide are tested in moq-scene on synthetic +reads and on the GPU, and BUILD's `build_keep_clear` view shows what it saw. +How often it moves on real content is a judgment no test makes. The knobs are +constants in `rs/moq-scene/src/place.rs`: `SETTLE` (1 s before a choice), +`WORST` (10 s in the worst place before it moves anyway), `BETTER` (how much +clearer a place must be), `APART` and `CHANGING` (what counts as the content +changing kind), `WEIGHT` (motion 0.5, text 0.8, HUD 1.0), and `MIDDLE` (a +corner's lead over an edge middle). + +## Waiting + +- Maintainer: put a camera over a game and over a slide deck with **Keep out + of the way** on (inspector, Place), stream or record a few sessions, and + note each move that was wrong, late, or too frequent. An agent cannot judge + how a move reads to a viewer or which part of a real game matters; the + tests already check the frames and the rules. Done when it moves as + intended, or the change it needs is written into this Plan. diff --git a/quest/m1/auto-place.md b/quest/m1/auto-place.md index 6024589..0b0a94e 100644 --- a/quest/m1/auto-place.md +++ b/quest/m1/auto-place.md @@ -1,53 +1,43 @@ -# [L] A camera that keeps out of the way +# [S] A camera that keeps out of the way remembers each app ## Goal -A layer with **Keep out of the way** on moves itself so it does not cover what -matters in the picture under it: a game's minimap, a slide's text, the window -the streamer is working in. It chooses among a few slots along the edges, -moves only when the content changes, and glides there. The `autoplace` frame -of [`mock/later.html`](/quest/m0/ux/mock/later.html) shows the slots, what it -saw, and the move. +A layer with **Keep out of the way** on knows the app in front: it chooses +again the moment another app comes to the front, rather than waiting to see +the picture change, and the place it chose for an app is remembered per user, +so a known game is right from its first frame. -The decision is made on the machine. Nothing is uploaded. +The decision stays on the machine. Nothing is uploaded. ## Plan +The rest of the quest landed: moq-scene's `place` module reads the picture +under the layer on the GPU a few times a second, weighs motion, still edges +(text), and still edges while the rest moves (a HUD), and keeps the layer in +the corner or edge middle that covers the least, gliding there; BUILD shows +what it saw, and the inspector its settings (`keep_clear` in the scene file). +What remains needs moq to report the app in front. + +- **When it moves:** a notice that the app in front changed starts a new + choice at once, as the content changing kind does now + (`Placer::see` setting `looking`), for the display capture under the + layer. +- **Memory:** the slot chosen for each app, keyed by the app moq reports, + kept per user with the hotkeys and snap settings, not in the scene file. + A remembered slot is where the layer starts when that app comes to the + front, before it has looked; a move by hand or a new choice updates it. + Decided 2026-11-09 (maintainer, /quest-plan): โœ… local only (rejected: a hosted decision model asked on each content change, cheap but it uploads thumbnails of the streamer's screen and its image input is not production -ready; local first with a hosted model as an opt-in tie-breaker). - -- **What it reads:** the composited picture under the layer, scaled far down - on the GPU a few times a second: where it moves, where it is dense with - edges and text, and what stays put while the rest moves, which is a HUD. - The inspector says which of these it stays clear of, and each can be - switched off. It reads neither the pointer nor the focused window, since - moq reports neither a pointer nor a window's place on its display. -- **What it decides:** the slot whose area matters least, among the corners - and edge middles that fit the layer, inside the canvas's margins. -- **When it moves:** when the scene changes, when the content under it - changes kind (another layout, or another app in front where - [moq reports it](/quest/m1/pin-capture-foreground.md)), or when its slot has been the - worst for a sustained time. Never on a single busy frame, and never while - the streamer is dragging it. A manual move wins and holds until the content - changes. -- **Memory:** the slot chosen for an app is remembered per user, so a known - game is right from its first frame. -- **Where it lives:** the analysis and the choice are UI-free, in moq-scene, - tested on recorded frames with a paused clock. The move is the item's - rect animated as a Move transition eases: moq-scene's `Glide` - (`rs/moq-scene/src/glide.rs`), which glides a scene's slots when they - reflow, eases any list of items from one placement to the next on the - program clock, keyed by what changed. - -Measure, per AGENTS.md: the frame time and GPU copies the analysis adds, and -that the encoder's load does not move. Should the signals prove too weak, a -small on-device model is a later quest, not a hosted one. - -Human check: living with it for a few streams. Whether it moves too often is -a judgment no test makes. +ready; local first with a hosted model as an opt-in tie-breaker). Should the +signals prove too weak, a small on-device model is a later quest, not a +hosted one. -## Related +## Required - [moq-video's foreground app is pinned](/quest/m1/pin-capture-foreground.md) - when the app in front changes, and the key its slot is remembered under + +## Related + +- [Living with it](/quest/m1/auto-place-check.md) - whether it moves too often, judged on real streams diff --git a/quest/m1/item-identity.md b/quest/m1/item-identity.md new file mode 100644 index 0000000..e7d8e2e --- /dev/null +++ b/quest/m1/item-identity.md @@ -0,0 +1,50 @@ +# [M] Each layer has an identity the studio keeps while it runs + +## Goal + +Each item of a scene and of the top layer carries a runtime identity: a new +item gets a new one, an edit that changes an item keeps its own, undo and +redo bring back the one it had, and the scene file never saves it. What the +compositor keeps about a layer between frames is matched to its item by +that identity, not by where it sits, so adding, removing, reordering, or +duplicating layers never hands one layer's state to another. + +## Plan + +Keep out of the way (`rs/moq-scene/src/place.rs`) keeps a placer per layer: +what it saw, the slot it chose, and whether a move by hand holds it. With no +identity on an item, `compositor::follow` matches placers to items by +source, then saved rect, then index, after each edit. Two layers of the same +source at the same rect, as Duplicate makes them, can still trade placers +when a layer is added under both. Auto-layout's glide pairs items by source +for the same reason. + +- **The identity:** a counter-made id on `Item`, outside serde, that + `Item::new`, loading, and every edit that adds an item as a new one + assign: BUILD's Duplicate (`LayerAction::Duplicate`), paste, a drop on the + picture (`ui/drop.rs`), an Add, and a guest joining. `set_item` and the + history keep the id of the item they change, so an undo restores it. +- **Equality:** two collections that differ only in ids are equal, so + autosave, reload, and every change check behave as now. A reload assigns + fresh ids, then matches them to the items before by the same rule + `follow` uses, so a reload of an unchanged file keeps every placer. +- **Users:** the compositor's keepers key by id; `follow` keeps only the + reload match. The slots' glide (`rs/moq-scene/src/glide.rs`, which pairs + items by source) pairs by id where both sides have one. +- **Undo across layers:** a placer whose layer leaves its scene, as Keep + always on top does, stays parked under its id + while the history can bring the layer back, so the undo puts it back where + it was, as an undo of switching keep-clear off does now (#194's decision + 11). Today such an undo starts a fresh placer at the saved rect. +- **Tests:** duplicates of one camera keep their own placers across an + insert under both, a reorder, an undo of a removal, an undo of Keep always + on top, and a reload of an unchanged file. + +Decided 2026-11-17 (maintainer, on +[#194](https://github.com/moq-dev/studio/pull/194#issuecomment-6944247792)): +โœ… a runtime identity, as its own quest (rejected: in #194, more matching +heuristics in #194). + +## Related + +- [A camera that keeps out of the way remembers each app](/quest/m1/auto-place.md) - another user of the placers this keys diff --git a/quest/m1/readback.md b/quest/m1/readback.md new file mode 100644 index 0000000..85c14a2 --- /dev/null +++ b/quest/m1/readback.md @@ -0,0 +1,32 @@ +# [S] One readback for the compositor's small GPU reads + +## Goal + +The compositor reads a few small pictures back from the GPU, a few times a +second each, through one shared helper: the dark-capture probe +(`rs/moq-scene/src/dark.rs`) and the keep-clear look +(`rs/moq-scene/src/place.rs`) each keep only their shader and what they +make of the tiles, and neither stalls a frame. + +## Plan + +The two landed separately with the same pattern: a compute pass reducing a +picture to a grid of tiles (64x36 for the probe, 32x18 for the look), at +most every 250 ms, its result copied to a buffer +mapped with `map_async` and read on a later frame, a failed or late read +dropped. + +- **Shared:** the rate cap (`due`), the storage and readback buffers sized + to the largest grid, the copy after the pass, the map in flight and its + poll, and discarding a read whose source or feed changed meanwhile. +- **Kept apart:** each pipeline and its bindings. `dark.wgsl` reduces the + capture texture alone to mean color and brightest channel; `place.wgsl` + compares the items drawn under a layer with the last read, so it keeps + two pictures that ping-pong. +- Their tests (`dark::tests::`, `compositor::tests::a_camera_keeping_clear_glides_off_what_moves`, + `a_moment_of_dark_draws_the_last_good_picture`) pass unchanged under + `just render-sw`, and the frame time on the AMD iGPU does not grow. + +Decided 2026-11-17 (maintainer, on +[#194](https://github.com/moq-dev/studio/pull/194#issuecomment-6941200435)): +โœ… one helper once both land (rejected: sharing in #194, before #192 lands). diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index a4613cf..cfe21d8 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -491,7 +491,26 @@ impl Collection { .validate() .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; self.validate_chain(&item.filters, &on)?; + if let Some(keep) = &item.keep_clear + && keep.glide > KeepClear::LONGEST + { + return Err(Error::Invalid(format!( + "{on} glides for {:?} to keep clear, longer than the {:?} a glide may take", + keep.glide, + KeepClear::LONGEST + ))); + } + if item.keep_clear.is_some() && item.slot.is_some() { + return Err(Error::Invalid(format!( + "{on} keeps clear in a slot, but its scene's arrangement places it; set it free first" + ))); + } if layer.scene.is_none() { + if item.keep_clear.is_some() { + return Err(Error::Invalid(format!( + "{on} keeps clear, but only a scene's item has a picture under it to keep clear of" + ))); + } if !item.keyframes.is_empty() { return Err(Error::Invalid(format!( "{on} has keyframes, but an always-on-top layer never goes on to start them" @@ -731,8 +750,9 @@ impl Collection { /// Refuse an item showing a source published independently that the /// layer layout cannot carry: one blended or filtered past opacity, which - /// not every player can reproduce, or one in a scene another scene draws, - /// which the layout cannot place. + /// not every player can reproduce, one keeping clear, which moves where + /// no player sees, or one in a scene another scene draws, which the + /// layout cannot place. fn validate_published(&self) -> Result<(), Error> { let nested: BTreeSet<&str> = self .sources @@ -759,6 +779,11 @@ impl Collection { "{on} cannot be in a scene another scene draws, since viewers place it over the program" ))); } + if item.keep_clear.is_some() { + return Err(Error::Invalid(format!( + "{on} cannot keep out of the way, since viewers place it where the scene file does" + ))); + } if item.slot.is_some() { return Err(Error::Invalid(format!( "{on} cannot be in a slot, since viewers place it where its rect says" @@ -1245,6 +1270,12 @@ pub struct Item { /// layer never goes on. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub keyframes: Vec, + /// Whether it moves itself off what matters in the picture under it, and + /// what that is. Its rect gives its size, and where it rests until it + /// first moves; the [`Compositor`](crate::Compositor) chooses where it + /// is drawn. Only a scene's item keeps clear. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub keep_clear: Option, } impl Item { @@ -1262,6 +1293,7 @@ impl Item { filters: Vec::new(), blend: Blend::Normal, keyframes: Vec::new(), + keep_clear: None, } } @@ -1291,6 +1323,7 @@ impl Item { filters, blend: self.blend, keyframes: Vec::new(), + keep_clear: self.keep_clear, }) } @@ -1319,6 +1352,61 @@ impl Item { } } +/// What an item keeping out of the way stays clear of in the picture under +/// it, and how it moves. +/// +/// It chooses among the corners and edge middles of the canvas, inside a +/// margin, and moves when the content under it changes, or when its place +/// has been the worst of them for a while, never for a single busy moment. +/// A move by hand holds until the content changes. The choice is made on +/// this machine from the picture alone. +#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct KeepClear { + /// Whether it stays clear of where the picture moves. + #[serde(default = "yes", skip_serializing_if = "is_true")] + pub motion: bool, + /// Whether it stays clear of text and other dense detail. + #[serde(default = "yes", skip_serializing_if = "is_true")] + pub text: bool, + /// Whether it stays clear of what holds still while the rest moves, such + /// as a game's HUD or minimap. + #[serde(default = "yes", skip_serializing_if = "is_true")] + pub hud: bool, + /// How long it takes to glide to a new place. + #[serde( + default = "KeepClear::default_glide", + with = "humantime_serde", + skip_serializing_if = "KeepClear::is_default_glide" + )] + pub glide: Duration, +} + +impl KeepClear { + /// The longest glide, so a typo cannot park an item mid-canvas for minutes. + pub const LONGEST: Duration = Duration::from_secs(5); + + fn default_glide() -> Duration { + Duration::from_millis(400) + } + + fn is_default_glide(glide: &Duration) -> bool { + *glide == Self::default_glide() + } +} + +impl Default for KeepClear { + /// Clear of everything it can see, gliding for 0.4 s. + fn default() -> Self { + Self { + motion: true, + text: true, + hud: true, + glide: Self::default_glide(), + } + } +} + /// Where a moment falls among an item's keyframes: holding the first, between /// two at that moment, or holding the last. Keyframes at the same moment jump, /// so the first keyframe's moment is not the same as before it. @@ -3271,6 +3359,87 @@ background = "remove" assert_eq!(filters(500), Filter::Invert { amount: 1.0 }); } + /// An item keeps clear with `keep_clear = {}`, of everything it can see, + /// writing only what differs from that. + #[test] + fn reads_and_writes_keep_clear() { + let text = r#" +version = 1 + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.camera] +kind = "camera" + +[[scenes]] +name = "Game" +items = [ + { source = "camera", rect = { x = 0, y = 0, width = 320, height = 180 }, keep_clear = {} }, + { source = "camera", rect = { x = 0, y = 0, width = 320, height = 180 }, keep_clear = { hud = false, glide = "1s" } }, +] +"#; + let collection = Collection::from_toml(text).unwrap(); + let items = &collection.scenes[0].items; + assert_eq!(items[0].keep_clear, Some(KeepClear::default())); + let partly = KeepClear { + hud: false, + glide: Duration::from_secs(1), + ..KeepClear::default() + }; + assert_eq!(items[1].keep_clear, Some(partly)); + let saved = collection.to_toml().unwrap(); + assert!( + saved.contains("[scenes.items.keep_clear]\nhud = false\nglide = \"1s\"\n"), + "{saved}" + ); + assert_eq!(Collection::from_toml(&saved).unwrap(), collection); + } + + #[test] + fn refuses_keeping_clear_where_it_cannot() { + let mut collection = example(); + collection.scenes[0].items[1].keep_clear = Some(KeepClear::default()); + collection.validate().unwrap(); + collection.scenes[0].items[1].keep_clear = Some(KeepClear { + glide: Duration::from_secs(6), + ..KeepClear::default() + }); + assert!( + matches!(collection.validate(), Err(Error::Invalid(_))), + "a glide too long" + ); + collection.scenes[0].items[1].keep_clear = None; + collection.top[0].keep_clear = Some(KeepClear::default()); + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "the top layer"); + collection.top[0].keep_clear = None; + collection.validate().unwrap(); + collection.scenes[0].items[1].keep_clear = Some(KeepClear::default()); + collection.scenes[0].items[1].slot = Some(Slot::Tile); + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "in a slot"); + collection.scenes[0].items[1].slot = None; + collection.validate().unwrap(); + + // A player of the layer layout places a published source where the + // file does, so it cannot move. + let mut published = Collection::new(); + published.sources.insert("cam".into(), Source::Camera { id: None }); + published.published.insert("cam".into()); + published.scenes = vec![Scene { + items: vec![Item { + keep_clear: Some(KeepClear::default()), + ..Item::new("cam", Rect::fill(&published.canvas)) + }], + ..Scene::new("Main") + }]; + assert!( + matches!(published.validate(), Err(Error::Invalid(_))), + "a published source" + ); + } + #[test] fn refuses_keyframes_it_cannot_play() { let at = |millis| Keyframe::new(Duration::from_millis(millis)); diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 12b40a9..99cff98 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -20,6 +20,7 @@ use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; use crate::glide::{Glide, REFLOW, mixed, pair, stack}; use crate::matte::Matte; use crate::overlay::{Overlay, Overlays}; +use crate::place::{GRAB, Look, Placement, Placer, Reducer}; use crate::source::{self, Feed, Sources, Status}; /// The format of every canvas texture: gamma-encoded RGBA, what @@ -117,11 +118,46 @@ pub struct Compositor { /// Whether the last render drew layers over the program. layered: bool, switch: Option, + /// Each scene item keeping clear, by its scene and place in it. + keepers: HashMap<(String, usize), Keeper>, + /// Each scene item that stopped keeping clear, by its scene and place in + /// it, for as long as it is there and keeps clear of nothing. + parked: HashMap<(String, usize), Parked>, + /// The item keeping clear a gesture has in hand, by its scene, place, and + /// source, until when, unless the gesture says so again. + grabbed: Option<((String, usize, String), Instant)>, + /// Reduces the pictures under them, made with the first. + reducer: Option, /// Each scene with items in slots, by name: where its arrangement put /// them as of the last render, and their glide there. reflows: HashMap, } +/// A scene item keeping clear: where it goes, and its reads of the picture +/// under it. +struct Keeper { + /// The source it shows: an item showing another is another keeper. + source: String, + /// The rect the scene file gave its item, as of the last render, which + /// tells it from another showing the same source. + rect: Rect, + /// Whether the last render read its picture: its scene drawn and it + /// shown. + watched: bool, + placer: Placer, + look: Look, +} + +/// A scene item that stopped keeping clear: where it went, held since, so an +/// undo that turns keeping clear back on puts it back. +struct Parked { + /// The source it shows. + source: String, + /// The rect the scene file gave its item, as of the last render. + rect: Rect, + placer: Placer, +} + /// A scene's items in slots, arranged. struct Reflow { /// Its items where its arrangement puts them, in order. @@ -290,11 +326,11 @@ impl Switch { /// A texture drawn into at a canvas's coordinates, and bound to sample: /// canvas-sized, or smaller for a thumbnail. #[derive(Clone)] -struct Target { +pub(crate) struct Target { /// The canvas drawn, at the program's frame rate. - canvas: Canvas, - texture: wgpu::Texture, - picture: Picture, + pub canvas: Canvas, + pub texture: wgpu::Texture, + pub picture: Picture, } /// One source's newest picture or table on the GPU. @@ -457,6 +493,10 @@ impl Compositor { program_shown: Duration::ZERO, layered: false, switch: None, + keepers: HashMap::new(), + parked: HashMap::new(), + grabbed: None, + reducer: None, reflows: HashMap::new(), }) } @@ -541,6 +581,75 @@ impl Compositor { self.program_shown } + /// Where the `index`th item of the scene called `scene`, one showing + /// `source` that keeps clear, was drawn as of the last + /// [`render`](Self::render), and why: `None` for any other item, or before + /// a render placed it. + pub fn placement(&self, scene: &str, index: usize, source: &str) -> Option<&Placement> { + let keeper = self.keepers.get(&(scene.to_string(), index))?; + (keeper.source == source).then(|| keeper.placer.placement()) + } + + /// Note the item a gesture has in hand as of `now`, such as a drag, as + /// its scene, index, and source, or `None` once let go: an item keeping + /// clear does not choose a place while in hand. Each frame of the gesture + /// says so again; one that stops saying so, as when the window is hidden + /// mid-drag, lets go after [`GRAB`]. + pub fn grab(&mut self, item: Option<(&str, usize, &str)>, now: Instant) { + self.grabbed = item.map(|(scene, index, source)| { + let key = (scene.to_string(), index, source.to_string()); + (key, now.saturating_add(GRAB)) + }); + } + + /// Note that the `index`th item of the scene called `scene`, one showing + /// `source` that keeps clear, was moved by hand: it stays where the scene + /// file puts it until the content under it changes. A resize is no move. + pub fn hold(&mut self, scene: &str, index: usize, source: &str) { + if let Some(keeper) = self.keepers.get_mut(&(scene.to_string(), index)) + && keeper.source == source + { + keeper.placer.hold(); + } + } + + /// Note that a redo or a reload saved the `index`th item of the scene + /// called `scene`, one showing `source` that keeps clear, at `saved`: it + /// holds there, as [`hold`](Self::hold) says. + pub fn redone(&mut self, scene: &str, index: usize, source: &str, saved: Rect) { + if let Some(keeper) = self.keepers.get_mut(&(scene.to_string(), index)) + && keeper.source == source + { + keeper.placer.redone(saved); + } + } + + /// Note that an undo saved the `index`th item of the scene called + /// `scene`, one showing `source` that keeps clear, at `saved` as of + /// `now`: back where it was saved before the move by hand that held it, + /// or before keeping clear was switched off, it goes back to where it was + /// then, such as the slot it had glided to. Anywhere else, it holds, as + /// [`hold`](Self::hold) says. + pub fn undone(&mut self, scene: &str, index: usize, source: &str, saved: Rect, now: Instant) { + let key = (scene.to_string(), index); + if !self.keepers.contains_key(&key) && self.parked.get(&key).is_some_and(|parked| parked.source == source) { + let parked = self.parked.remove(&key).expect("found above"); + let keeper = Keeper { + source: parked.source, + rect: saved, + watched: true, + placer: parked.placer, + look: Look::new(&self.device, &self.queue), + }; + self.keepers.insert(key.clone(), keeper); + } + if let Some(keeper) = self.keepers.get_mut(&key) + && keeper.source == source + { + keeper.placer.undone(saved, now); + } + } + /// Keep the scenes `outputs` names as outputs from now on, each with /// whether the next [`render`](Self::render) draws it, as it does while an /// output reads it. Each plays its keyframes from the first render after @@ -899,8 +1008,10 @@ impl Compositor { } self.renders += 1; self.update(collection, sources, now); - // Every draw below reads the scenes as arranged. - let staged = self.reflow(collection, sources, now); + // Every draw below reads the items keeping clear where they are + // placed, and the scenes as arranged around them. + let placed = self.place(collection, now); + let staged = self.reflow(&placed, sources, now); let collection = &*staged; self.restart(collection, preview_start, program_start); self.paint(collection, now); @@ -1071,12 +1182,138 @@ impl Compositor { Shows::Nothing => unreachable!("a thumbnail showing nothing is not drawn"), } } + + // The picture under each item keeping clear, read a few times a + // second while its scene is drawn: on program, in preview, for an + // output, or through a scene source. + let mut drawn: HashMap<&str, Duration> = HashMap::new(); + let roots = on_program(program, switch) + .into_iter() + .chain(previewing.then_some(preview)) + .chain(outputs.iter().map(|(on, _)| *on)); + for On(scene, since) in roots { + if let Some(scene) = scene { + drawn.entry(scene).or_insert(since); + } + } + for nest in &nesting.order { + drawn.entry(nest.scene).or_insert(nest.since); + } + let mut reads = Vec::new(); + for ((scene, index), keeper) in &mut self.keepers { + keeper.watched = false; + let Some(&since) = drawn.get(scene.as_str()) else { + continue; + }; + keeper.placer.shown(since, now); + let Some(shown) = collection.scene(scene) else { continue }; + // While its slots glide, a scene draws only its visible items, so + // one keeping clear is found by what it shows and where it is. + let mine = |item: &Item| item.keep_clear.is_some() && item.source == keeper.source; + let rect = keeper.placer.placement().rect; + let at = match shown.items.get(*index) { + Some(item) if mine(item) => Some(*index), + _ => shown.items.iter().position(|item| mine(item) && item.rect == rect), + }; + let Some(at) = at.filter(|&at| shown.items[at].visible) else { + continue; + }; + keeper.watched = true; + if !keeper.look.due(now) { + continue; + } + let canvas = canvas_of(Some(scene)); + let target = keeper.look.target(draw.effects, canvas); + draw.size = canvas; + draw.target(&target); + draw.under(&target, Place::scene(scene.as_str()), at, since); + reads.push((scene.clone(), *index)); + } self.previewed = previewing; self.effects.submit(frame); + if !reads.is_empty() { + let reducer = self.reducer.get_or_insert_with(|| Reducer::new(&self.device)); + for key in reads { + let keeper = self.keepers.get_mut(&key).expect("read above"); + keeper.look.start(reducer, now); + } + } Ok(()) } + /// `collection` with each scene item keeping clear where its placer puts + /// it at `now`, from what the reads of the picture under it that came + /// back show. A keeper whose item stopped keeping clear is parked, held + /// where it was, until the item keeps clear again or is gone; one whose + /// item is gone is forgotten. + fn place<'a>(&mut self, collection: &'a Collection, now: Instant) -> Cow<'a, Collection> { + let mut staged = Cow::Borrowed(collection); + let keeping = |item: &Item| item.keep_clear.is_some(); + let (mut kept, left) = follow(std::mem::take(&mut self.keepers), collection, keeping, |keeper| { + (keeper.source.as_str(), keeper.rect) + }); + let mut parked = std::mem::take(&mut self.parked); + parked.extend(left.into_iter().map(|(key, mut keeper)| { + keeper.placer.hold(); + let parked = Parked { + source: keeper.source, + rect: keeper.rect, + placer: keeper.placer, + }; + (key, parked) + })); + (self.parked, _) = follow( + parked, + collection, + |item| !keeping(item), + |parked| (parked.source.as_str(), parked.rect), + ); + let mut keepers = HashMap::new(); + for (number, scene) in collection.scenes.iter().enumerate() { + let canvas = scene.canvas(collection.canvas); + for (index, item) in scene.items.iter().enumerate() { + let Some(keep) = &item.keep_clear else { continue }; + let key = (scene.name.clone(), index); + let mut keeper = kept.remove(&key).unwrap_or_else(|| Keeper { + source: item.source.clone(), + rect: item.rect, + watched: false, + placer: Placer::new(now), + look: Look::new(&self.device, &self.queue), + }); + match keeper.look.finish() { + Some(Ok((tiles, at))) => keeper.placer.see(&tiles, at), + Some(Err(err)) => tracing::warn!( + scene = %scene.name, + source = %item.source, + %err, + "cannot read the picture under an item keeping clear" + ), + None => {} + } + keeper.rect = item.rect; + keeper.placer.watch(keeper.watched, now); + let grabbed = self.grabbed.as_ref().is_some_and(|((at, number, source), until)| { + *at == scene.name && *number == index && *source == item.source && now < *until + }); + keeper.placer.grab(grabbed); + let rect = keeper.placer.place(item.rect, keep, canvas, now); + if rect != item.rect { + staged.to_mut().scenes[number].items[index].rect = rect; + } + keepers.insert(key, keeper); + } + for (index, item) in scene.items.iter().enumerate() { + if let Some(parked) = self.parked.get_mut(&(scene.name.clone(), index)) { + parked.rect = item.rect; + } + } + } + self.keepers = keepers; + staged + } + /// `collection` with each scene's items in slots where its arrangement /// puts them at `now`, gliding there as a move transition does when a /// slot opens or closes. A slot closes while its source is offline or @@ -1996,8 +2233,15 @@ impl Draw<'_> { /// item, a source with no picture yet, or a filter whose source has nothing /// yet draws nothing. fn layer(&mut self, target: &Target, place: Place, since: Duration) { + self.under(target, place, usize::MAX, since); + } + + /// Draw `place`'s items under its `index`th onto `target`, as + /// [`layer`](Self::layer) draws them: what that item covers. + fn under(&mut self, target: &Target, place: Place, index: usize, since: Duration) { let items = self.collection.items(&place).unwrap_or_default(); - for (index, item) in items.iter().enumerate().filter(|(_, item)| item.visible) { + let under = items.iter().enumerate().take(index); + for (index, item) in under.filter(|(_, item)| item.visible) { let key = (place.clone(), index, self.settled(item, since)); self.item(target, key, &item.posed(since), since); } @@ -2324,7 +2568,7 @@ impl Target { } /// A target drawing `canvas` into `size` pixels, labeled `label`. - fn sized(device: &wgpu::Device, effects: &Effects, canvas: Canvas, size: (u32, u32), label: &str) -> Self { + pub fn sized(device: &wgpu::Device, effects: &Effects, canvas: Canvas, size: (u32, u32), label: &str) -> Self { let texture = device.create_texture(&wgpu::TextureDescriptor { label: Some(label), size: wgpu::Extent3d { @@ -2352,6 +2596,60 @@ impl Target { } } +/// Each of something by the scene and place of the item it is for. +type Placed = HashMap<(String, usize), T>; + +/// `old`, the keepers, each under where its item that `keeps` now is, so it +/// goes on from where it was placed, and the rest. A keeper shows the same +/// source; of those, it is first the one in the same scene saved at the same +/// rect, at the same index or the nearest, as a layer added or removed under +/// it shifts it; then one in the same scene at the same index or the +/// nearest, as an edit to its rect leaves it; then the same in a scene no +/// longer there, as a rename leaves it. +fn follow( + mut old: Placed, + collection: &Collection, + keeps: impl Fn(&Item) -> bool, + shows: impl Fn(&T) -> (&str, Rect), +) -> (Placed, Placed) { + let mut kept = HashMap::new(); + if old.is_empty() { + return (kept, old); + } + let mut missing: Vec<((String, usize), &Item)> = Vec::new(); + for scene in &collection.scenes { + for (index, item) in scene.items.iter().enumerate() { + if keeps(item) { + missing.push(((scene.name.clone(), index), item)); + } + } + } + let gone = |scene: &str| collection.scene(scene).is_none(); + for (same_scene, same_rect) in [(true, true), (true, false), (false, true), (false, false)] { + missing.retain(|(key, item)| { + let found = old + .iter() + .filter(|((scene, _), keeper)| { + let (source, rect) = shows(keeper); + source == item.source + && (!same_rect || rect == item.rect) + && match same_scene { + true => *scene == key.0, + false => gone(scene), + } + }) + .map(|(old_key, _)| old_key) + .min_by_key(|(_, index)| index.abs_diff(key.1)) + .cloned(); + let Some(found) = found else { return true }; + let keeper = old.remove(&found).expect("found above"); + kept.insert(key.clone(), keeper); + false + }); + } + (kept, old) +} + #[cfg(test)] mod tests { use std::time::Duration; @@ -2546,6 +2844,237 @@ mod tests { actual.iter().zip(expected).all(|(a, e)| a.abs_diff(e) <= 1) } + /// A 320x180 scene "busy" with a camera keeping clear over the bottom + /// right, where a block moves: a checkerboard in the top right and lines + /// of "text" in the bottom left, on black. + fn busy() -> Collection { + let mut collection = Collection::new(); + collection.canvas = Canvas { + width: 320, + height: 180, + framerate: 30, + }; + for (name, color) in [("black", BLACK), ("white", [255; 4]), ("camera", BLUE)] { + let color = Rgba(color); + collection.sources.insert(name.into(), Source::Color { color }); + } + let rect = |x, y, width, height| Rect { x, y, width, height }; + let mut items = vec![Item::new("black", Rect::fill(&collection.canvas))]; + for row in 0..9 { + for column in (row % 2..10).step_by(2) { + let at = rect(232.0 + 8.0 * column as f32, 4.0 + 8.0 * row as f32, 8.0, 8.0); + items.push(Item::new("white", at)); + } + } + for line in 0..13 { + items.push(Item::new("white", rect(8.0, 130.0 + 3.0 * line as f32, 100.0, 1.0))); + } + items.push(Item::new("white", rect(250.0, 120.0, 40.0, 40.0))); + items.push(Item { + keep_clear: Some(crate::KeepClear::default()), + ..Item::new("camera", rect(247.0, 107.0, 64.0, 64.0)) + }); + collection.scenes = vec![Scene { + items, + ..Scene::new("busy") + }]; + collection.validate().unwrap(); + collection + } + + /// The compositor reads the picture under a camera keeping clear on the + /// GPU, sees the block moving under it, and glides it to the corner that + /// covers nothing, on program and in preview alike. It stays there as a + /// layer is added under it, and as an undo switches keeping clear back + /// on. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_camera_keeping_clear_glides_off_what_moves() { + local(async { + let mut studio = Studio::new(busy()).await; + studio.compositor.set_preview(Some("busy".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + let camera = studio.collection.scenes[0].items.len() - 1; + let block = camera - 1; + let saved = studio.collection.scenes[0].items[camera].rect; + let start = Instant::now(); + let mut now = start; + let mut moved = None; + for frame in 0..40u32 { + studio.collection.scenes[0].items[block].rect.x = 240.0 + 6.0 * (frame % 7) as f32; + studio.render(now); + studio.device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); + let placement = studio.compositor.placement("busy", camera, "camera").unwrap(); + if placement.slot.is_some() { + moved = Some(now); + break; + } + assert_eq!(placement.rect, saved, "it stays put while it looks"); + now = now.saturating_add(Duration::from_millis(100)); + } + let moved = moved.expect("it chose a slot"); + assert!( + moved.saturating_duration_since(start) >= crate::place::SETTLE, + "it looks before it moves" + ); + let placement = studio.compositor.placement("busy", camera, "camera").unwrap().clone(); + assert_eq!(placement.slot, Some(crate::place::Slot::TopLeft)); + let left = placement.left.unwrap(); + assert_eq!(left.rect, saved); + assert_eq!(left.over, Some(crate::place::Signal::Motion)); + let heat = &placement.heat; + let (column, row) = (heat.columns - 2, 1); + let (matters, signal) = heat.tiles[row * heat.columns + column]; + assert!(matters > 0.5, "the checkerboard matters: {matters}"); + assert_eq!(signal, crate::place::Signal::Text, "still edges in a still picture"); + assert_eq!(heat.tiles[heat.columns * 4 + 4].0, 0.0, "black does not"); + + let glide = crate::KeepClear::default().glide; + studio.render(moved.saturating_add(glide / 2)); + let half = studio.compositor.placement("busy", camera, "camera").unwrap().rect; + assert!(half.x < saved.x && half.x > 9.0, "on its way: {half:?}"); + studio.render(moved.saturating_add(glide)); + let to = crate::place::Slot::TopLeft + .rect(64.0, 64.0, studio.collection.canvas) + .unwrap(); + assert_eq!(studio.compositor.placement("busy", camera, "camera").unwrap().rect, to); + let program = studio.program(); + assert_eq!(texel(&program, 320, 41, 41), BLUE, "the camera where it went"); + assert_eq!(texel(&program, 320, 279, 139), [255; 4], "the block uncovered"); + assert_eq!(program, studio.preview(), "the preview shows it alike"); + + // A layer added under it shifts it, but it stays where it went. + let added = Item { + visible: false, + ..studio.collection.scenes[0].items[0].clone() + }; + studio.collection.scenes[0].items.insert(0, added); + let camera = camera + 1; + let after = moved.saturating_add(glide * 2); + studio.render(after); + let placement = studio.compositor.placement("busy", camera, "camera").unwrap(); + assert_eq!(placement.rect, to, "it goes on from where it was placed"); + assert_eq!(placement.slot, Some(crate::place::Slot::TopLeft)); + + // Switched off, as the inspector does, holding it, and the switch + // undone before the next render: it is still in its slot. + studio.compositor.hold("busy", camera, "camera"); + let item = &mut studio.collection.scenes[0].items[camera]; + let keep = item.keep_clear.take(); + item.rect = to; + let item = &mut studio.collection.scenes[0].items[camera]; + item.keep_clear = keep; + item.rect = saved; + studio.compositor.undone("busy", camera, "camera", saved, after); + let after = after.saturating_add(Duration::from_millis(100)); + studio.render(after); + let placement = studio.compositor.placement("busy", camera, "camera").unwrap(); + assert_eq!((placement.rect, placement.held), (to, false), "undone before it drew"); + + // Switched off and drawn so, it stays where it is drawn; an undo of + // that puts it back in its slot, keeping clear. + studio.compositor.hold("busy", camera, "camera"); + let item = &mut studio.collection.scenes[0].items[camera]; + let keep = item.keep_clear.take(); + item.rect = to; + let after = after.saturating_add(Duration::from_millis(100)); + studio.render(after); + assert!(studio.compositor.placement("busy", camera, "camera").is_none()); + let item = &mut studio.collection.scenes[0].items[camera]; + item.keep_clear = keep; + item.rect = saved; + studio.compositor.undone("busy", camera, "camera", saved, after); + let after = after.saturating_add(Duration::from_millis(100)); + studio.render(after); + let placement = studio.compositor.placement("busy", camera, "camera").unwrap(); + assert_eq!(placement.rect, to, "back where it was drawn"); + assert_eq!(placement.slot, Some(crate::place::Slot::TopLeft)); + assert!(!placement.held); + + // Moved by hand, it stays where the scene file puts it. + studio.compositor.hold("busy", camera, "camera"); + let dragged = Rect { x: 200.0, ..saved }; + studio.collection.scenes[0].items[camera].rect = dragged; + studio.render(after.saturating_add(Duration::from_millis(100))); + let placement = studio.compositor.placement("busy", camera, "camera").unwrap(); + assert!(placement.held); + assert_eq!(placement.rect, dragged); + + // Switched off where it is held, which leaves its rect as it is, and + // that undone: it is still held. + let after = after.saturating_add(Duration::from_millis(200)); + studio.compositor.hold("busy", camera, "camera"); + studio.collection.scenes[0].items[camera].keep_clear = None; + studio.render(after); + studio.collection.scenes[0].items[camera].keep_clear = keep; + studio.compositor.undone("busy", camera, "camera", dragged, after); + let after = after.saturating_add(Duration::from_millis(100)); + studio.render(after); + let placement = studio.compositor.placement("busy", camera, "camera").unwrap(); + assert_eq!((placement.rect, placement.held), (dragged, true), "still held"); + }) + .await; + } + + /// Two layers keeping clear of the same camera keep their own keepers as + /// a layer added under both shifts them, as an edit to one's rect, and as + /// their scene is renamed. + #[test] + fn keepers_follow_their_items() { + let rect = |x| Rect { + x, + y: 0.0, + width: 10.0, + height: 10.0, + }; + let keeping = |x| Item { + keep_clear: Some(crate::KeepClear::default()), + ..Item::new("camera", rect(x)) + }; + let mut collection = Collection::new(); + collection.scenes = vec![Scene { + items: vec![keeping(0.0), keeping(50.0)], + ..Scene::new("main") + }]; + let named = |name: &str, index| (name.to_string(), index); + fn shows<'a>(keeper: &'a (String, Rect, &'static str)) -> (&'a str, Rect) { + (keeper.0.as_str(), keeper.1) + } + let keeping_clear = |item: &Item| item.keep_clear.is_some(); + let mut keepers = HashMap::from([ + (named("main", 0), ("camera".to_string(), rect(0.0), "first")), + (named("main", 1), ("camera".to_string(), rect(50.0), "second")), + ]); + let which = |keepers: &HashMap<(String, usize), (String, Rect, &'static str)>, scene, index| { + keepers.get(&named(scene, index)).map(|keeper| keeper.2) + }; + + collection.scenes[0].items.insert(0, Item::new("camera", rect(90.0))); + keepers = follow(keepers, &collection, keeping_clear, shows).0; + assert_eq!(which(&keepers, "main", 1), Some("first")); + assert_eq!(which(&keepers, "main", 2), Some("second")); + + // Moved, the second still has its own. + collection.scenes[0].items[2].rect = rect(70.0); + keepers = follow(keepers, &collection, keeping_clear, shows).0; + assert_eq!(which(&keepers, "main", 1), Some("first")); + assert_eq!(which(&keepers, "main", 2), Some("second")); + keepers.get_mut(&named("main", 2)).unwrap().1 = rect(70.0); + + collection.scenes[0].name = "renamed".into(); + keepers = follow(keepers, &collection, keeping_clear, shows).0; + assert_eq!(which(&keepers, "renamed", 1), Some("first")); + assert_eq!(which(&keepers, "renamed", 2), Some("second")); + + // One that stops keeping clear has no keeper; its is left over. + collection.scenes[0].items[1].keep_clear = None; + let left; + (keepers, left) = follow(keepers, &collection, keeping_clear, shows); + assert_eq!(keepers.len(), 1); + assert_eq!(which(&keepers, "renamed", 2), Some("second")); + assert_eq!(which(&left, "renamed", 1), Some("first")); + } + /// A source published independently is left out of the program and drawn /// over it as its layer, as it is over the scene in preview, wherever its /// scene stacks it; without one showing, the layered program is the diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index e8d1f2b..997f492 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -60,6 +60,7 @@ mod lut; mod matte; mod media; mod overlay; +pub mod place; mod playout; pub mod publish; pub mod replay; @@ -69,7 +70,7 @@ pub mod source; pub use clip::Clip; pub use collection::{ Arrangement, Audio, AudioScene, Background, Blend, Canvas, Collection, Crop, Direction, Ease, Filter, Item, - Keyframe, Layer, MaskMode, Nested, Rect, Rgba, Scene, Shape, Size, Slot, Source, Transition, Video, + KeepClear, Keyframe, Layer, MaskMode, Nested, Rect, Rgba, Scene, Shape, Size, Slot, Source, Transition, Video, }; pub use compositor::{Compositor, Detail, FORMAT, Output, Thumbnail}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-scene/src/place.rs b/rs/moq-scene/src/place.rs new file mode 100644 index 0000000..432cfe4 --- /dev/null +++ b/rs/moq-scene/src/place.rs @@ -0,0 +1,1608 @@ +//! Where an item that keeps out of the way goes. +//! +//! A few times a second, the compositor draws the picture under the item, +//! scaled far down, and a compute shader reduces it, with the read before, to +//! a grid of tiles: how much of each moved, how much of it is edges, as text +//! and a HUD are, and how much of those held still. Only the grid comes back, +//! a frame or so later. A [`Placer`] weighs it into what matters where, and +//! keeps the item in the [`Slot`] that covers the least of it: it chooses +//! when its scene goes on and when the content under it changes, and moves +//! otherwise only once its place has been the worst for a while, gliding +//! there. A move by hand holds until the content changes. + +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use kio::clock::Instant; + +use crate::collection::{Canvas, Item, KeepClear, Rect, mix}; +use crate::compositor::Target; +use crate::effect::Effects; +use crate::glide::Glide; + +/// How often the picture under an item is read, at most. +pub const INTERVAL: Duration = Duration::from_millis(250); + +/// How long a gesture holds an item keeping clear in hand after it last +/// said so: a window that stops drawing mid-drag, as a hidden one does, +/// lets go this long after. +pub const GRAB: Duration = Duration::from_millis(500); + +/// The longest side of the picture read, in pixels: text a 1080p canvas +/// shows at a readable size still has edges at this size. +const SIDE: u32 = 512; + +/// A tile's side in pixels of the picture read, as `place.wgsl`'s workgroup. +const TILE: u32 = 16; + +/// The bytes of the largest grid, four a tile. +const BYTES: u64 = ((SIDE / TILE) * (SIDE / TILE) * 4) as u64; + +/// How long what matters lingers once it stops, so one busy moment is never +/// a reason to move. +const LINGER: Duration = Duration::from_secs(2); + +/// How long the layout of the picture is averaged over to tell a new kind of +/// content from the one before, which is averaged over [`USUAL`]. +const RECENT: Duration = Duration::from_millis(500); + +/// How long the layout the content usually has is averaged over. +const USUAL: Duration = Duration::from_secs(6); + +/// How far apart the recent and usual layouts get, per tile on average out of +/// 1 for its brightness and its edges, once the content changed kind. +const APART: f32 = 0.12; + +/// How long the layouts stay [`APART`] before the content counts as changed: +/// a flash or a fast pan passes sooner. +const CHANGING: Duration = Duration::from_secs(1); + +/// How long an item looks, once its scene goes on or the content under it +/// changes, before it chooses: long enough to see what moves. +pub const SETTLE: Duration = Duration::from_secs(1); + +/// How long its place must be the worst before it moves without the content +/// changing. +pub const WORST: Duration = Duration::from_secs(10); + +/// How much less a place must matter than where the item is for it to move, +/// on a tile's 0 to 1 scale, so it never wavers between two alike. +const BETTER: f32 = 0.1; + +/// How close two places' costs are to count as alike, past the rounding of +/// summing them tile by tile. +const ALIKE: f32 = 1e-4; + +/// What a slot along an edge's middle costs over a corner covering as much: +/// the middle of an edge is where captions and the game's own bars go. +const MIDDLE: f32 = 0.05; + +/// How much each signal weighs at full strength: a HUD that holds still is +/// what viewers look for, and motion is everywhere in a game. +const WEIGHT: [f32; 3] = [0.5, 0.8, 1.0]; + +/// A place along the canvas's edges an item can keep to. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Slot { + TopLeft, + Top, + TopRight, + Left, + Right, + BottomLeft, + Bottom, + BottomRight, +} + +impl Slot { + /// Every slot, corners first. + pub const ALL: [Self; 8] = [ + Self::TopLeft, + Self::TopRight, + Self::BottomLeft, + Self::BottomRight, + Self::Top, + Self::Left, + Self::Right, + Self::Bottom, + ]; + + /// Whether it is a corner, which a middle slot yields to when both cover + /// as much. + pub fn corner(self) -> bool { + matches!( + self, + Self::TopLeft | Self::TopRight | Self::BottomLeft | Self::BottomRight + ) + } + + /// Its column and row in a 3x3 grid over the canvas, from the top left. + pub fn cell(self) -> (u8, u8) { + match self { + Self::TopLeft => (0, 0), + Self::Top => (1, 0), + Self::TopRight => (2, 0), + Self::Left => (0, 1), + Self::Right => (2, 1), + Self::BottomLeft => (0, 2), + Self::Bottom => (1, 2), + Self::BottomRight => (2, 2), + } + } + + /// Where an item `width` by `height` sits in it on `canvas`, inside the + /// [`margin`], or `None` if it does not fit there. + pub fn rect(self, width: f32, height: f32, canvas: Canvas) -> Option { + let margin = margin(canvas); + let along = |cell: u8, length: f32, size: f32| match cell { + 0 => margin, + 1 => ((length - size) / 2.0).round(), + _ => length - margin - size, + }; + let (w, h) = (canvas.width as f32, canvas.height as f32); + let fits = width > 0.0 && height > 0.0 && width + 2.0 * margin <= w && height + 2.0 * margin <= h; + let (column, row) = self.cell(); + fits.then(|| Rect { + x: along(column, w, width), + y: along(row, h, height), + width, + height, + }) + } +} + +/// How far in from the canvas's edges a slot keeps, in pixels: a twentieth +/// of its shorter side. +pub fn margin(canvas: Canvas) -> f32 { + (canvas.width.min(canvas.height) as f32 / 20.0).round() +} + +/// What makes part of a picture matter. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Signal { + /// It moves. + Motion, + /// It is dense with edges that hold still, as text is. + Text, + /// It holds still while the rest moves, as a game's HUD or minimap does. + Hud, +} + +impl Signal { + const ALL: [Self; 3] = [Self::Motion, Self::Text, Self::Hud]; + + /// Whether `keep` stays clear of it. + fn kept(self, keep: &KeepClear) -> bool { + match self { + Self::Motion => keep.motion, + Self::Text => keep.text, + Self::Hud => keep.hud, + } + } +} + +/// What matters in the picture under an item, as a grid of tiles over its +/// canvas. +#[derive(Clone, Debug, Default, PartialEq)] +pub struct Heat { + /// Tiles across. + pub columns: usize, + /// Tiles down. + pub rows: usize, + /// Each tile, row by row: how much it matters, from 0 to 1, and the + /// signal it matters most for. + pub tiles: Vec<(f32, Signal)>, + /// Where each tile is on the canvas, as fractions of its width and height + /// at the tile edges: `columns + 1` across, then `rows + 1` down. + pub edges: (Vec, Vec), +} + +impl Heat { + /// The tile at `column`, `row` on the canvas, as fractions of its size. + pub fn tile(&self, column: usize, row: usize) -> (f32, f32, f32, f32) { + let (across, down) = &self.edges; + (across[column], down[row], across[column + 1], down[row + 1]) + } +} + +/// Where an item keeping clear is drawn, and why, as of the last render. +#[derive(Clone, Debug, PartialEq)] +pub struct Placement { + /// Where it is drawn, part way along a glide while one is under way. + pub rect: Rect, + /// The slot it is in or gliding to, or `None` while it stays where it + /// was put. + pub slot: Option, + /// Whether it stays where the streamer moved it, until the content under + /// it changes. A move by hand says so at once, before the next render. + pub held: bool, + /// Where it last moved from, and what it left there. + pub left: Option, + /// The places it may go: each slot it fits in. + pub slots: Vec<(Slot, Rect)>, + /// What matters under it, once it has looked. + pub heat: Heat, +} + +impl Placement { + /// Where the item is drawn, saved at `saved`: where it was placed, or, + /// held, where it is saved, which an edit since the last render, such as + /// a nudge, may have moved. + pub fn drawn(&self, saved: Rect) -> Rect { + match self.held { + true => saved, + false => self.rect, + } + } +} + +/// Where an item moved from. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct Left { + /// Where it was. + pub rect: Rect, + /// What mattered most there, if anything did. + pub over: Option, +} + +/// One read of the picture under an item: each tile's share of pixels that +/// moved, that are edges, and that are edges that held still, and its +/// brightness, each out of 255. +#[derive(Clone, Debug, PartialEq)] +pub(crate) struct Tiles { + /// The picture's size in pixels, which the tiles cover [`TILE`] at a + /// time, those at the right and bottom only as far as it goes. + pub size: (u32, u32), + /// Whether there was a read before to compare with, without which + /// nothing moved. + pub compared: bool, + pub tiles: Vec<[u8; 4]>, +} + +impl Tiles { + const MOVED: usize = 0; + const EDGES: usize = 1; + const STILL: usize = 2; + const LIGHT: usize = 3; +} + +/// The fraction `amount` of the way to `full`, at most 1. +fn ramp(amount: f32, full: f32) -> f32 { + (amount / full).clamp(0.0, 1.0) +} + +/// How much of the way to a new value an average kept over `over` goes when +/// a value arrives `dt` after the last. +fn weight(dt: Duration, over: Duration) -> f32 { + 1.0 - (-dt.as_secs_f32() / over.as_secs_f32()).exp() +} + +/// What the reads of the picture under an item show over time. +#[derive(Default)] +struct Sight { + /// The picture size the reads are of; a read of another starts over. + size: (u32, u32), + /// Each tile's signals, lingering, in [`Signal::ALL`]'s order. + signals: Vec<[f32; 3]>, + /// Each tile's brightness and edges, recently and usually. + recent: Vec<[f32; 2]>, + usual: Vec<[f32; 2]>, + /// Since when the recent and usual layouts have been [`APART`]. + apart: Option, + /// When the last read was taken. + last: Option, + /// How many reads compared with the one before since it started over. + compared: u32, + /// Bumped with every read, so what is drawn from it is redrawn. + generation: u64, +} + +impl Sight { + /// Take in a read taken `at`, returning whether the content changed kind. + fn see(&mut self, read: &Tiles, at: Instant) -> bool { + self.generation += 1; + let count = read.tiles.len(); + if read.size != self.size || self.signals.len() != count { + *self = Self { + size: read.size, + generation: self.generation, + ..Self::default() + }; + } + let unit = |value: u8| f32::from(value) / 255.0; + let moving = match read.compared { + true => read.tiles.iter().map(|tile| unit(tile[Tiles::MOVED])).sum::() / count.max(1) as f32, + false => 0.0, + }; + // Edges that hold still are text, and a HUD while the rest moves; + // edges that move are motion. A first read, with nothing to compare, + // takes every edge for still. + let hud = ramp(moving - 0.03, 0.1); + let raw = read.tiles.iter().map(|tile| { + let (moved, still) = match read.compared { + true => (unit(tile[Tiles::MOVED]), unit(tile[Tiles::STILL])), + false => (0.0, unit(tile[Tiles::EDGES])), + }; + [ramp(moved, 0.3), ramp(still, 0.2), ramp(still, 0.15) * hud] + }); + let layout = read + .tiles + .iter() + .map(|tile| [unit(tile[Tiles::LIGHT]), unit(tile[Tiles::EDGES])]); + + let Some(last) = self.last.replace(at) else { + self.signals = raw.collect(); + self.recent = layout.collect(); + self.usual = self.recent.clone(); + self.compared = u32::from(read.compared); + return false; + }; + let dt = at.saturating_duration_since(last); + self.compared += u32::from(read.compared); + let linger = weight(dt, LINGER); + for (signals, raw) in self.signals.iter_mut().zip(raw) { + for (signal, raw) in signals.iter_mut().zip(raw) { + // A signal rises at once and lingers as it falls, so what + // matters counts from the first read that sees it. + *signal = if raw > *signal { raw } else { mix(*signal, raw, linger) }; + } + } + let (recent, usual) = (weight(dt, RECENT), weight(dt, USUAL)); + let mut distance = 0.0; + for ((now, usually), layout) in self.recent.iter_mut().zip(&mut self.usual).zip(layout) { + for channel in 0..2 { + now[channel] = mix(now[channel], layout[channel], recent); + usually[channel] = mix(usually[channel], layout[channel], usual); + distance += (now[channel] - usually[channel]).abs(); + } + } + distance /= count.max(1) as f32; + if distance < APART { + self.apart = None; + return false; + } + let apart = *self.apart.get_or_insert(at); + if at.saturating_duration_since(apart) < CHANGING { + return false; + } + // What it usually looks like is what it looks like now, and what + // mattered in the content before matters no more. + self.usual = self.recent.clone(); + self.signals = self.recent.iter().map(|_| [0.0; 3]).collect(); + self.apart = None; + self.compared = 0; + true + } + + /// Forget what was seen, as when the scene goes on again. + fn forget(&mut self) { + *self = Self { + generation: self.generation + 1, + ..Self::default() + }; + } + + /// How much each tile matters to an item keeping clear as `keep` says, + /// and for what most, or `None` before any read. + fn heat(&self, keep: &KeepClear) -> Option { + self.last?; + let tiles = self + .signals + .iter() + .map(|signals| { + let weighed = Signal::ALL.map(|signal| { + let index = signal as usize; + let weight = if signal.kept(keep) { WEIGHT[index] } else { 0.0 }; + (signals[index] * weight, signal) + }); + weighed + .into_iter() + .max_by(|a, b| a.0.total_cmp(&b.0)) + .expect("three signals") + }) + .collect(); + let (width, height) = self.size; + let fractions = |length: u32| -> Vec { + let tiles = length.div_ceil(TILE); + (0..=tiles) + .map(|tile| (tile * TILE).min(length) as f32 / length as f32) + .collect() + }; + Some(Heat { + columns: width.div_ceil(TILE) as usize, + rows: height.div_ceil(TILE) as usize, + tiles, + edges: (fractions(width), fractions(height)), + }) + } +} + +/// How much `rect` on `canvas` covers of what matters in `heat`: the mean of +/// its tiles, each by how much of it the rect covers, and what mattered most +/// under it. +fn covered(heat: &Heat, rect: &Rect, canvas: Canvas) -> (f32, Option) { + let (w, h) = (canvas.width as f32, canvas.height as f32); + let (left, top) = (rect.x / w, rect.y / h); + let (right, bottom) = ((rect.x + rect.width) / w, (rect.y + rect.height) / h); + let area = (right - left) * (bottom - top); + if area <= 0.0 { + return (0.0, None); + } + let mut sum = 0.0; + let mut most = (0.0, None); + for row in 0..heat.rows { + for column in 0..heat.columns { + let (x0, y0, x1, y1) = heat.tile(column, row); + let overlap = (x1.min(right) - x0.max(left)).max(0.0) * (y1.min(bottom) - y0.max(top)).max(0.0); + if overlap <= 0.0 { + continue; + } + let (matters, signal) = heat.tiles[row * heat.columns + column]; + sum += matters * overlap; + if matters > most.0 { + most = (matters, Some(signal)); + } + } + } + (sum / area, most.1) +} + +/// Where an item is meant to be. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum At { + /// Where its saved rect puts it, before it first chooses. + Rest, + /// Where the streamer moved it, until the content changes. + Held, + /// In a slot. + Slot(Slot), +} + +/// What a move by hand held an item from, so an undo back there can put it +/// back. +#[derive(Clone, Copy, Debug, PartialEq)] +struct Before { + /// Where its scene file put it. + saved: Rect, + /// Where it was meant to be, such as the slot it had glided to. + at: At, + /// Whether it was looking before choosing. + looking: bool, +} + +/// Chooses where one item keeping clear goes, from what reads of the picture +/// under it show. +pub(crate) struct Placer { + sight: Sight, + at: At, + /// When it last started looking before choosing: when it began, its + /// scene went on, or the content under it changed. + looking: Option, + /// Since when its place has been the worst. + worst: Option, + /// How long its scene had been on as of the last read, which tells it + /// going on again. + since: Option, + /// The item gliding from one place to the next as a move transition + /// eases, keyed by how many times it moved. + glide: Glide, + /// How long [`glide`](Self::glide) takes. + glides: Duration, + moves: u64, + /// Where it was meant to be as of the last [`place`](Self::place), and + /// after how many moves: another place with no move, as a resize gives, + /// is taken at once, even part way along a glide. + aimed: Option<(Rect, u64)>, + /// Whether its picture is read: its scene drawn and the item shown. It + /// chooses only while it is. + watched: bool, + /// Whether a gesture has it in hand, such as a drag: it chooses only once + /// let go. + grabbed: bool, + /// Where a glide under way stopped when a gesture took it in hand: it + /// stays there, at the size it is meant to have, until let go, then + /// glides on from there. + paused: Option, + placement: Placement, + /// Where its scene file put it as of the last [`place`](Self::place). + saved: Option, + /// What the move by hand that holds it, or last held it, held it from. + before: Option, + /// What [`Sight::generation`] and settings the heat was drawn from. + heated: Option<(u64, KeepClear)>, +} + +impl Placer { + /// A placer that starts looking at `now`. + pub fn new(now: Instant) -> Self { + let nowhere = Rect { + x: 0.0, + y: 0.0, + width: 0.0, + height: 0.0, + }; + Self { + sight: Sight::default(), + at: At::Rest, + looking: Some(now), + worst: None, + since: None, + glide: Glide::new(KeepClear::default().glide, |_| true), + glides: KeepClear::default().glide, + moves: 0, + aimed: None, + watched: true, + grabbed: false, + paused: None, + placement: Placement { + rect: nowhere, + slot: None, + held: false, + left: None, + slots: Vec::new(), + heat: Heat::default(), + }, + saved: None, + before: None, + heated: None, + } + } + + /// Take in a read of the picture under the item, taken `at`. + pub fn see(&mut self, read: &Tiles, at: Instant) { + if self.sight.see(read, at) { + self.looking = Some(at); + } + } + + /// Note that the item's scene is drawn, on for `since`, as of `now`: one + /// that went on again since the last is new content to choose for. + pub fn shown(&mut self, since: Duration, now: Instant) { + if self.since.is_some_and(|last| since < last) { + self.sight.forget(); + self.looking = Some(now); + } + self.since = Some(since); + } + + /// Moved by hand, such as dragged or nudged: it stays where the scene + /// file has it until the content under it changes, and a choice it was + /// about to make is dropped. A resize alone is no move: it keeps its + /// place. The first move of a hold remembers where it was, for + /// [`undone`](Self::undone). + pub fn hold(&mut self) { + if self.at != At::Held + && let Some(saved) = self.saved + { + self.before = Some(Before { + saved, + at: self.at, + looking: self.looking.is_some(), + }); + } + self.at = At::Held; + self.placement.held = true; + self.placement.slot = None; + self.looking = None; + self.worst = None; + self.glide = Glide::new(self.glides, |_| true); + self.aimed = None; + self.paused = None; + } + + /// Moved by an undo, saved at `saved` as of `now`: back where its scene + /// file put it before the move by hand that held it, it goes back to + /// where it was meant to be then, such as the slot it had glided to, so + /// the undo puts back what the streamer saw. Anywhere else, it holds, as + /// after a move by hand. + pub fn undone(&mut self, saved: Rect, now: Instant) { + // What another undo or a redo before the next place starts from. + self.saved = Some(saved); + match self.before.take() { + Some(before) if before.saved == saved => { + self.at = before.at; + self.placement.held = false; + self.placement.slot = match before.at { + At::Slot(slot) => Some(slot), + _ => None, + }; + self.looking = before.looking.then_some(now); + self.worst = None; + self.glide = Glide::new(self.glides, |_| true); + self.aimed = None; + self.paused = None; + } + before => { + self.hold(); + // What it was held from is still what an undo further back + // returns to. + self.before = before; + } + } + } + + /// Moved by an edit already made, such as a redo or a reload, that saved + /// it at `saved`: it holds, as [`hold`](Self::hold) says, and another + /// undo or redo before the next [`place`](Self::place) starts from + /// there. + pub fn redone(&mut self, saved: Rect) { + self.hold(); + self.saved = Some(saved); + } + + /// Note whether its picture is read as of `now`: while its scene is not + /// drawn or the item is hidden it sees nothing new, so it neither + /// chooses nor counts its place the worst, and it looks again once it is + /// read again, unless a move by hand holds it, which only the content + /// changing ends. + pub fn watch(&mut self, watched: bool, now: Instant) { + if watched && !self.watched && self.at != At::Held { + self.looking = Some(now); + } + if !watched { + self.worst = None; + } + self.watched = watched; + } + + /// Note whether a gesture has it in hand, such as a drag that moves, + /// resizes, or crops it: while it does, it neither chooses nor counts its + /// place the worst, and once let go it goes on as the gesture left it, a + /// move holding it. + pub fn grab(&mut self, grabbed: bool) { + if grabbed { + self.worst = None; + } + self.grabbed = grabbed; + } + + /// Where it is as of the last [`place`](Self::place). + pub fn placement(&self) -> &Placement { + &self.placement + } + + /// Where to draw the item at `now`, saved at `saved` and keeping clear as + /// `keep` says on `canvas`. + pub fn place(&mut self, saved: Rect, keep: &KeepClear, canvas: Canvas, now: Instant) -> Rect { + self.saved = Some(saved); + let slots: Vec<(Slot, Rect)> = Slot::ALL + .into_iter() + .filter_map(|slot| Some((slot, slot.rect(saved.width, saved.height, canvas)?))) + .collect(); + if let At::Slot(slot) = self.at + && !slots.iter().any(|(fits, _)| *fits == slot) + { + self.at = At::Rest; + } + + let generation = (self.sight.generation, *keep); + if self.heated != Some(generation) { + self.heated = Some(generation); + self.placement.heat = self.sight.heat(keep).unwrap_or_default(); + } + if self.watched && !self.grabbed && self.sight.last.is_some() && !slots.is_empty() { + self.choose(saved, &slots, keep, canvas, now); + } + + let target = self.target(saved, &slots); + // Taken in hand part way along a glide, as a crop started then is, it + // stops where it was last drawn, so it stays under the pointer, however + // late the next render comes; let go, it glides on from there. + let gliding = self.aimed.is_some_and(|(aimed, _)| aimed != self.placement.rect); + if self.grabbed && gliding && self.paused.is_none() { + self.paused = Some(self.placement.rect); + } + if !self.grabbed + && let Some(paused) = self.paused.take() + { + let at = Rect { + width: target.width, + height: target.height, + ..paused + }; + self.glide = Glide::new(self.glides, |_| true); + self.glide.show(self.moves, &[shown(at)], now); + self.moves += 1; + } + let rect = match self.paused { + Some(paused) => { + self.aimed = Some((target, self.moves)); + Rect { + width: target.width, + height: target.height, + ..paused + } + } + None => { + // Another target with no move, as a resize gives, shows at + // once, even part way along a glide. + if self + .aimed + .is_some_and(|(aimed, moves)| aimed != target && moves == self.moves) + { + self.glide = Glide::new(self.glides, |_| true); + } + self.aimed = Some((target, self.moves)); + self.glide.show(self.moves, &[shown(target)], now); + self.glide.placed()[0].rect + } + }; + self.placement.rect = rect; + self.placement.slot = match self.at { + At::Slot(slot) => Some(slot), + _ => None, + }; + self.placement.held = self.at == At::Held; + self.placement.slots = slots; + rect + } + + /// Where the item is meant to be, gliding or not. + fn target(&self, saved: Rect, slots: &[(Slot, Rect)]) -> Rect { + match self.at { + At::Slot(slot) => slots + .iter() + .find(|(fits, _)| *fits == slot) + .map_or(saved, |(_, rect)| *rect), + At::Rest | At::Held => saved, + } + } + + /// Move to the slot that matters least, if the time has come. + fn choose(&mut self, saved: Rect, slots: &[(Slot, Rect)], keep: &KeepClear, canvas: Canvas, now: Instant) { + let heat = &self.placement.heat; + let cost = |slot: Option, rect: &Rect| { + let (covered, over) = covered(heat, rect, canvas); + let middle = if slot.is_some_and(|slot| !slot.corner()) { + MIDDLE + } else { + 0.0 + }; + (covered + middle, over) + }; + let costs: Vec<(Slot, f32)> = slots + .iter() + .map(|(slot, rect)| (*slot, cost(Some(*slot), rect).0)) + .collect(); + let lowest = costs.iter().map(|(_, cost)| *cost).fold(f32::INFINITY, f32::min); + // Of slots that cover alike, the first: a corner, from the top left. + let (best, _) = *costs + .iter() + .find(|(_, cost)| *cost <= lowest + ALIKE) + .expect("a slot fits"); + let current = match self.at { + At::Slot(slot) => Some(slot), + _ => None, + }; + let here = self.target(saved, slots); + let (cost_here, over) = cost(current, &here); + let better = lowest + BETTER <= cost_here && current != Some(best); + + let looked = self + .looking + .is_some_and(|since| now.saturating_duration_since(since) >= SETTLE && self.sight.compared >= 2); + if looked { + self.looking = None; + self.worst = None; + if self.at == At::Held { + self.at = At::Rest; + } + if better { + self.go(best, here, over, keep, now); + } + return; + } + if self.looking.is_some() || self.at == At::Held { + return; + } + let worst = costs.iter().all(|(_, cost)| *cost <= cost_here + ALIKE); + if !(worst && better) { + self.worst = None; + return; + } + let since = *self.worst.get_or_insert(now); + if now.saturating_duration_since(since) >= WORST { + self.worst = None; + self.go(best, here, over, keep, now); + } + } + + /// Glide to `slot` from `here`, over what mattered most there. + fn go(&mut self, slot: Slot, here: Rect, over: Option, keep: &KeepClear, now: Instant) { + self.placement.left = Some(Left { rect: here, over }); + self.at = At::Slot(slot); + if keep.glide != self.glides { + // Another length: a glide of that length, from where it is. + self.glides = keep.glide; + self.glide = Glide::new(keep.glide, |_| true); + self.glide.show(self.moves, &[shown(self.placement.rect)], now); + } + self.moves += 1; + } +} + +/// An item at `rect`, standing for the one keeping clear in its glide. +fn shown(rect: Rect) -> Item { + Item::new("keeping clear", rect) +} + +/// The pipeline every read of the picture under an item goes through. +pub(crate) struct Reducer { + pipeline: wgpu::ComputePipeline, + layout: wgpu::BindGroupLayout, +} + +impl Reducer { + pub fn new(device: &wgpu::Device) -> Self { + let shader = device.create_shader_module(wgpu::include_wgsl!("place.wgsl")); + let picture = |binding| wgpu::BindGroupLayoutEntry { + binding, + visibility: wgpu::ShaderStages::COMPUTE, + ty: wgpu::BindingType::Texture { + sample_type: wgpu::TextureSampleType::Float { filterable: false }, + view_dimension: wgpu::TextureViewDimension::D2, + multisampled: false, + }, + count: None, + }; + let layout = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor { + label: Some("moq-scene place"), + entries: &[ + picture(0), + picture(1), + wgpu::BindGroupLayoutEntry { + binding: 2, + visibility: wgpu::ShaderStages::COMPUTE, + ty: wgpu::BindingType::Buffer { + ty: wgpu::BufferBindingType::Storage { read_only: false }, + has_dynamic_offset: false, + min_binding_size: wgpu::BufferSize::new(BYTES), + }, + count: None, + }, + ], + }); + let pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor { + label: Some("moq-scene place"), + bind_group_layouts: &[Some(&layout)], + immediate_size: 0, + }); + let pipeline = device.create_compute_pipeline(&wgpu::ComputePipelineDescriptor { + label: Some("moq-scene place"), + layout: Some(&pipeline_layout), + module: &shader, + entry_point: Some("reduce"), + compilation_options: Default::default(), + cache: None, + }); + Self { pipeline, layout } + } +} + +/// A read under way: when it was taken, its picture's size, whether it had +/// one before to compare with, and the buffer's mapping once the GPU is done. +struct Reading { + at: Instant, + size: (u32, u32), + compared: bool, + mapped: Arc>>>, +} + +/// One item's reads of the picture under it. +pub(crate) struct Look { + device: wgpu::Device, + queue: wgpu::Queue, + /// The pictures read, by turns: the next read draws into the one the read + /// before last did, and compares with the other. + pictures: [Option; 2], + /// Which of `pictures` the next read draws into. + next: usize, + tiles: wgpu::Buffer, + readback: wgpu::Buffer, + reading: Option, + /// When the last read was taken. + last: Option, +} + +impl Look { + pub fn new(device: &wgpu::Device, queue: &wgpu::Queue) -> Self { + let tiles = device.create_buffer(&wgpu::BufferDescriptor { + label: Some("moq-scene place tiles"), + size: BYTES, + usage: wgpu::BufferUsages::STORAGE | wgpu::BufferUsages::COPY_SRC, + mapped_at_creation: false, + }); + let readback = device.create_buffer(&wgpu::BufferDescriptor { + label: Some("moq-scene place tiles"), + size: BYTES, + usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ, + mapped_at_creation: false, + }); + Self { + device: device.clone(), + queue: queue.clone(), + pictures: [None, None], + next: 0, + tiles, + readback, + reading: None, + last: None, + } + } + + /// Whether a read may start at `now`: none is under way, and the last + /// was at least [`INTERVAL`] ago. + pub fn due(&self, now: Instant) -> bool { + self.reading.is_none() + && self + .last + .is_none_or(|last| now.saturating_duration_since(last) >= INTERVAL) + } + + /// What the next read draws the picture under the item into, `canvas` + /// scaled down to at most [`SIDE`] a side. + pub fn target(&mut self, effects: &Effects, canvas: Canvas) -> Target { + let scale = (SIDE as f32 / canvas.width.max(canvas.height) as f32).min(1.0); + let side = |length: u32| ((length as f32 * scale).round() as u32).max(1); + let size = (side(canvas.width), side(canvas.height)); + let picture = &mut self.pictures[self.next]; + match picture { + Some(target) if (target.texture.width(), target.texture.height()) == size => { + target.canvas = canvas; + target.clone() + } + _ => { + // A picture of another size has nothing to compare with. + let other = &mut self.pictures[1 - self.next]; + if other + .as_ref() + .is_some_and(|other| (other.texture.width(), other.texture.height()) != size) + { + *other = None; + } + let target = Target::sized(&self.device, effects, canvas, size, "moq-scene place"); + self.pictures[self.next].insert(target).clone() + } + } + } + + /// Reduce the picture the frame drew into [`target`](Self::target), once + /// submitted, with the one before, and read the tiles back. + pub fn start(&mut self, reducer: &Reducer, now: Instant) { + let read = self.pictures[self.next].as_ref().expect("drawn into"); + let before = self.pictures[1 - self.next].as_ref(); + let compared = before.is_some(); + let view = |target: &Target| target.texture.create_view(&Default::default()); + let bind = self.device.create_bind_group(&wgpu::BindGroupDescriptor { + label: Some("moq-scene place"), + layout: &reducer.layout, + entries: &[ + wgpu::BindGroupEntry { + binding: 0, + resource: wgpu::BindingResource::TextureView(&view(read)), + }, + wgpu::BindGroupEntry { + binding: 1, + resource: wgpu::BindingResource::TextureView(&view(before.unwrap_or(read))), + }, + wgpu::BindGroupEntry { + binding: 2, + resource: self.tiles.as_entire_binding(), + }, + ], + }); + let size = (read.texture.width(), read.texture.height()); + let mut encoder = self.device.create_command_encoder(&Default::default()); + { + let mut pass = encoder.begin_compute_pass(&wgpu::ComputePassDescriptor { + label: Some("moq-scene place"), + timestamp_writes: None, + }); + pass.set_pipeline(&reducer.pipeline); + pass.set_bind_group(0, &bind, &[]); + pass.dispatch_workgroups(size.0.div_ceil(TILE), size.1.div_ceil(TILE), 1); + } + let bytes = u64::from(size.0.div_ceil(TILE) * size.1.div_ceil(TILE) * 4); + encoder.copy_buffer_to_buffer(&self.tiles, 0, &self.readback, 0, bytes); + self.queue.submit([encoder.finish()]); + + let mapped = Arc::new(Mutex::new(None)); + let done = mapped.clone(); + // Called from a later submit or poll, once the GPU is done. + self.readback.map_async(wgpu::MapMode::Read, .., move |result| { + *done.lock().unwrap() = Some(result); + }); + self.reading = Some(Reading { + at: now, + size, + compared, + mapped, + }); + self.last = Some(now); + self.next = 1 - self.next; + } + + /// The read under way, once it came back, and when it was taken. + pub fn finish(&mut self) -> Option> { + let reading = self.reading.as_ref()?; + let mapped = reading.mapped.lock().unwrap().take()?; + let Reading { at, size, compared, .. } = self.reading.take().expect("checked above"); + let count = (size.0.div_ceil(TILE) * size.1.div_ceil(TILE)) as usize; + let read = mapped.map_err(|err| err.to_string()).and_then(|()| { + let tiles = match self.readback.slice(..).get_mapped_range() { + Ok(view) => Ok(view.as_chunks::<4>().0[..count].to_vec()), + Err(err) => Err(format!("{err:?}")), + }; + self.readback.unmap(); + tiles + }); + Some(read.map(|tiles| (Tiles { size, compared, tiles }, at))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const CANVAS: Canvas = Canvas { + width: 1920, + height: 1080, + framerate: 30, + }; + + /// The picture size a 1080p canvas is read at: 32 by 18 tiles. + const SIZE: (u32, u32) = (512, 288); + + /// A read where every tile is `tile`, except those `marked` returns + /// another for, by column and row. + fn read(tile: [u8; 4], marked: impl Fn(usize, usize) -> Option<[u8; 4]>) -> Tiles { + let mut tiles = Vec::new(); + for row in 0..18 { + for column in 0..32 { + tiles.push(marked(column, row).unwrap_or(tile)); + } + } + Tiles { + size: SIZE, + compared: true, + tiles, + } + } + + /// A game: moving everywhere, with a minimap that holds still in the top + /// right corner. + fn game(column: usize, row: usize) -> Option<[u8; 4]> { + (column >= 26 && row < 7).then_some([0, 60, 50, 90]) + } + + const MOVING: [u8; 4] = [120, 10, 0, 80]; + + /// A slide: still, dark, with text along its bottom left. + fn slide(column: usize, row: usize) -> Option<[u8; 4]> { + (column < 14 && row >= 13).then_some([0, 70, 70, 200]) + } + + const STILL: [u8; 4] = [0, 0, 0, 20]; + + fn camera() -> Rect { + Rect { + x: 1574.0, + y: 54.0, + width: 288.0, + height: 288.0, + } + } + + /// A placer fed `frames` reads a quarter second apart from `start`, + /// placing an item saved at `saved` after each; the time after the last. + fn run(placer: &mut Placer, saved: Rect, start: Instant, reads: impl IntoIterator) -> Instant { + let mut now = start; + for read in reads { + placer.see(&read, now); + placer.place(saved, &KeepClear::default(), CANVAS, now); + now = now.saturating_add(INTERVAL); + } + now + } + + #[test] + fn slots_keep_inside_the_margin() { + let margin = margin(CANVAS); + assert_eq!(margin, 54.0); + let top_left = Slot::TopLeft.rect(288.0, 288.0, CANVAS).unwrap(); + assert_eq!((top_left.x, top_left.y), (54.0, 54.0)); + let bottom = Slot::Bottom.rect(288.0, 288.0, CANVAS).unwrap(); + assert_eq!((bottom.x, bottom.y), (816.0, 738.0)); + let right = Slot::Right.rect(288.0, 288.0, CANVAS).unwrap(); + assert_eq!((right.x, right.y), (1578.0, 396.0)); + // Too tall to fit between the margins. + assert_eq!(Slot::TopLeft.rect(288.0, 1000.0, CANVAS), None); + assert_eq!(Slot::TopLeft.rect(0.0, 100.0, CANVAS), None); + } + + #[test] + fn a_rect_covers_the_tiles_under_it_by_area() { + let mut sight = Sight::default(); + sight.see(&read(STILL, slide), Instant::now()); + let heat = sight.heat(&KeepClear::default()).unwrap(); + assert_eq!((heat.columns, heat.rows), (32, 18)); + let text = Slot::BottomLeft.rect(288.0, 288.0, CANVAS).unwrap(); + let (covered_text, over) = covered(&heat, &text, CANVAS); + // Text weighs 0.8, over the part of the slot below the 13th row. + assert!((covered_text - 0.8 * 246.0 / 288.0).abs() < 1e-3, "{covered_text}"); + assert_eq!(over, Some(Signal::Text)); + let clear = Slot::TopRight.rect(288.0, 288.0, CANVAS).unwrap(); + assert_eq!(covered(&heat, &clear, CANVAS), (0.0, None)); + } + + /// Edges that hold still while the rest moves are a HUD; the same edges + /// in a still picture are text. + #[test] + fn still_edges_are_a_hud_only_while_the_rest_moves() { + let start = Instant::now(); + let mut sight = Sight::default(); + sight.see(&read(MOVING, game), start); + sight.see(&read(MOVING, game), start.saturating_add(INTERVAL)); + let heat = sight.heat(&KeepClear::default()).unwrap(); + assert_eq!(heat.tiles[30].1, Signal::Hud); + assert_eq!(heat.tiles[0].1, Signal::Motion); + + let mut sight = Sight::default(); + let still = |column, row| game(column, row).map(|tile| [0, tile[1], tile[2], tile[3]]); + sight.see(&read(STILL, still), start); + sight.see(&read(STILL, still), start.saturating_add(INTERVAL)); + let heat = sight.heat(&KeepClear::default()).unwrap(); + assert_eq!(heat.tiles[30].1, Signal::Text); + } + + /// Over a game's minimap, it waits a moment to see what moves, then + /// glides to the corner that covers the least, easing as a move + /// transition does. + #[test] + fn it_glides_off_a_hud_once_it_has_looked() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + assert_eq!(placer.place(saved, &KeepClear::default(), CANVAS, start), saved); + // Not yet: it has not looked for long enough. + let now = run(&mut placer, saved, start, (0..3).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, None); + let now = run(&mut placer, saved, now, (0..2).map(|_| read(MOVING, game))); + let placement = placer.placement().clone(); + assert_eq!(placement.slot, Some(Slot::TopLeft)); + let left = placement.left.unwrap(); + assert_eq!(left.rect, saved); + assert_eq!(left.over, Some(Signal::Hud)); + + let moved = now.saturating_sub(INTERVAL); + let to = Slot::TopLeft.rect(288.0, 288.0, CANVAS).unwrap(); + let glide = KeepClear::default().glide; + let half = placer.place(saved, &KeepClear::default(), CANVAS, moved.saturating_add(glide / 2)); + assert_eq!(half.x, (saved.x + to.x) / 2.0, "halfway, eased in and out"); + let landed = placer.place(saved, &KeepClear::default(), CANVAS, moved.saturating_add(glide)); + assert_eq!(landed, to); + } + + /// It glides for as long as its settings say, or jumps. + #[test] + fn it_glides_as_long_as_it_is_set_to() { + let to = Slot::TopLeft.rect(288.0, 288.0, CANVAS).unwrap(); + for glide in [Duration::ZERO, Duration::from_millis(1500)] { + let keep = KeepClear { + glide, + ..KeepClear::default() + }; + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let mut now = start; + let mut moved = None; + for _ in 0..8 { + placer.see(&read(MOVING, game), now); + placer.place(saved, &keep, CANVAS, now); + if placer.placement().slot.is_some() { + moved = Some(now); + break; + } + now = now.saturating_add(INTERVAL); + } + let moved = moved.expect("it chose a slot"); + let first = placer.placement().rect; + match glide.is_zero() { + true => assert_eq!(first, to, "a jump lands at once"), + false => { + assert_eq!(first, saved, "a glide starts where it was"); + let half = placer.place(saved, &keep, CANVAS, moved.saturating_add(glide / 2)); + assert_eq!(half.x, (saved.x + to.x) / 2.0, "halfway at half its length"); + let landed = placer.place(saved, &keep, CANVAS, moved.saturating_add(glide)); + assert_eq!(landed, to); + } + } + } + } + + /// In hand, as a drag has it, it does not choose, though the time to has + /// come; let go, it does. + #[test] + fn it_chooses_only_once_let_go() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + placer.grab(true); + let now = run(&mut placer, saved, start, (0..12).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, None, "in hand, it stays"); + placer.grab(false); + let now = run(&mut placer, saved, now, [read(MOVING, game)]); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + let _ = now; + } + + /// Taken in hand part way along a glide, as a crop started then is, it + /// stops where it is drawn; let go, it glides on from there to its slot. + #[test] + fn a_glide_pauses_while_in_hand() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..5).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + let moved = now.saturating_sub(INTERVAL); + let glide = KeepClear::default().glide; + let to = Slot::TopLeft.rect(288.0, 288.0, CANVAS).unwrap(); + + let half = moved.saturating_add(glide / 2); + let at = placer.place(saved, &KeepClear::default(), CANVAS, half); + assert!(at.x < saved.x && at.x > to.x, "on its way: {at:?}"); + placer.grab(true); + for later in [half, half.saturating_add(glide), half.saturating_add(glide * 4)] { + let held = placer.place(saved, &KeepClear::default(), CANVAS, later); + assert_eq!(held, at, "in hand, it stays where it is drawn"); + } + + placer.grab(false); + let released = half.saturating_add(glide * 4); + assert_eq!(placer.place(saved, &KeepClear::default(), CANVAS, released), at); + let on = placer.place(saved, &KeepClear::default(), CANVAS, released.saturating_add(glide / 2)); + assert!(on.x < at.x && on.x > to.x, "it glides on: {on:?}"); + let landed = placer.place(saved, &KeepClear::default(), CANVAS, released.saturating_add(glide)); + assert_eq!(landed, to); + } + + /// Taken in hand part way along a glide, as last drawn, it stops there + /// though the next render comes after the glide would have landed, as on + /// a slow canvas. + #[test] + fn a_glide_pauses_where_last_drawn() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..5).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + let moved = now.saturating_sub(INTERVAL); + let glide = KeepClear::default().glide; + let half = moved.saturating_add(glide / 2); + let at = placer.place(saved, &KeepClear::default(), CANVAS, half); + placer.grab(true); + let late = half.saturating_add(glide * 2); + assert_eq!(placer.place(saved, &KeepClear::default(), CANVAS, late), at); + } + + /// A resize part way along a glide takes its new size at once, in the + /// slot it was gliding to. + #[test] + fn a_resize_while_it_glides_lands_at_once() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..5).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + let resized = Rect { + width: 200.0, + height: 200.0, + ..saved + }; + let rect = placer.place(resized, &KeepClear::default(), CANVAS, now); + assert_eq!(Some(rect), Slot::TopLeft.rect(200.0, 200.0, CANVAS)); + } + + /// Unwatched, as when its scene is not drawn or it is hidden, it neither + /// chooses nor counts toward [`WORST`], and it looks again once watched. + #[test] + fn it_chooses_only_while_watched() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + placer.watch(false, start); + let now = run(&mut placer, saved, start, (0..8).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, None, "unwatched, it stays"); + let later = now.saturating_add(WORST * 2); + placer.place(saved, &KeepClear::default(), CANVAS, later); + assert_eq!(placer.placement().slot, None); + + placer.watch(true, later); + placer.place(saved, &KeepClear::default(), CANVAS, later); + assert_eq!(placer.placement().slot, None, "it looks again first"); + let now = run(&mut placer, saved, later, (0..8).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + + // Moved by hand, then hidden and shown over the same game: it stays + // where it was put. + let dragged = Rect { x: 1600.0, ..saved }; + placer.hold(); + let now = run(&mut placer, dragged, now, (0..4).map(|_| read(MOVING, game))); + placer.watch(false, now); + let now = run(&mut placer, dragged, now, (0..4).map(|_| read(MOVING, game))); + placer.watch(true, now); + let now = run(&mut placer, dragged, now, (0..40).map(|_| read(MOVING, game))); + let placement = placer.placement(); + assert!(placement.held, "still held"); + assert_eq!(placement.rect, dragged); + let _ = now; + } + + /// One busy read is never a reason to move: what it covers lingers, but + /// only a place that stays the worst for [`WORST`] moves it. + #[test] + fn a_busy_moment_does_not_move_it() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = Slot::TopLeft.rect(288.0, 288.0, CANVAS).unwrap(); + let now = run(&mut placer, saved, start, (0..8).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, None, "already clear"); + + // Something flares under it for one read. + let flare = |column, row| { + (column < 10 && row < 8) + .then_some([255, 200, 200, 200]) + .or(game(column, row)) + }; + let now = run(&mut placer, saved, now, [read(MOVING, flare)]); + let now = run(&mut placer, saved, now, (0..80).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, None); + assert_eq!(placer.placement().rect, saved); + + // Something that stays there moves it, once it has been the worst + // place for long enough. + let reads = (WORST.as_millis() / INTERVAL.as_millis()) as usize; + let now = run(&mut placer, saved, now, (0..reads - 4).map(|_| read(MOVING, flare))); + assert_eq!(placer.placement().slot, None, "not yet"); + run(&mut placer, saved, now, (0..8).map(|_| read(MOVING, flare))); + assert_ne!(placer.placement().slot, None); + assert_eq!(placer.placement().left.unwrap().over, Some(Signal::Hud)); + } + + /// A move by hand holds, however much it covers, until the content under + /// it changes; then it chooses again. + #[test] + fn a_move_by_hand_holds_until_the_content_changes() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..6).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + + // Dragged back over the minimap. + let dragged = Rect { x: 1600.0, ..saved }; + placer.hold(); + assert_eq!(placer.place(dragged, &KeepClear::default(), CANVAS, now), dragged); + let now = run(&mut placer, dragged, now, (0..80).map(|_| read(MOVING, game))); + let placement = placer.placement(); + assert!(placement.held); + assert_eq!(placement.rect, dragged); + + // A slide replaces the game: it looks again, and leaves the text alone. + let mut now = now; + while placer.placement().held { + now = run(&mut placer, dragged, now, [read(STILL, slide)]); + assert!(now.saturating_duration_since(start) < Duration::from_secs(60)); + } + let now = run(&mut placer, dragged, now, (0..8).map(|_| read(STILL, slide))); + let placement = placer.placement(); + assert!(!placement.held); + // Its own place covers no text, so nothing is clearly better. + assert_eq!(placement.slot, None); + assert_eq!(placement.rect, dragged); + let _ = now; + } + + /// A move by hand while it is still looking drops the choice it was about + /// to make: it stays where it was put, over the minimap, as the same game + /// plays on. + #[test] + fn a_move_by_hand_while_it_looks_drops_the_choice() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..2).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, None, "still looking"); + // Moved by hand: only the caller can say so. + placer.hold(); + let now = run(&mut placer, saved, now, (0..40).map(|_| read(MOVING, game))); + let placement = placer.placement(); + assert!(placement.held); + assert_eq!(placement.slot, None); + assert_eq!(placement.rect, saved); + let _ = now; + } + + /// A move by hand says it is held at once, before the next place, so + /// moves that come faster than renders, as nudges on a slow canvas do, + /// each start from where the last saved it; an undo says it is not. + #[test] + fn a_move_by_hand_is_held_at_once() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..6).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + let nudged = Rect { x: 20.0, ..saved }; + placer.hold(); + let placement = placer.placement(); + assert_eq!((placement.held, placement.slot), (true, None)); + assert_eq!(placement.drawn(nudged), nudged); + placer.undone(saved, now); + let placement = placer.placement(); + assert_eq!((placement.held, placement.slot), (false, Some(Slot::TopLeft))); + assert_eq!(placement.drawn(saved), placement.rect); + } + + /// An undo back to where it was saved before a move by hand puts back + /// where it was meant to be then, the slot it had glided to, at once, + /// though moves since held it; a redo holds it again. An undo to anywhere + /// else holds, as does one with no move by hand to undo. + #[test] + fn an_undo_of_a_move_by_hand_puts_back_its_place() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let keep = KeepClear::default(); + let now = run(&mut placer, saved, start, (0..6).map(|_| read(MOVING, game))); + let now = now.saturating_add(keep.glide); + let to = Slot::TopLeft.rect(288.0, 288.0, CANVAS).unwrap(); + assert_eq!(placer.place(saved, &keep, CANVAS, now), to); + + // Dragged over the minimap, then on, a move each. + let first = Rect { x: 1000.0, ..saved }; + let second = Rect { x: 1600.0, ..saved }; + for dragged in [first, second] { + placer.hold(); + assert_eq!(placer.place(dragged, &keep, CANVAS, now), dragged); + } + let now = run(&mut placer, second, now, (0..8).map(|_| read(MOVING, game))); + + placer.undone(first, now); + assert_eq!( + placer.place(first, &keep, CANVAS, now), + first, + "back one move, it holds" + ); + assert!(placer.placement().held); + placer.undone(saved, now); + assert_eq!(placer.place(saved, &keep, CANVAS, now), to, "back in its slot, at once"); + let placement = placer.placement(); + assert_eq!((placement.slot, placement.held), (Some(Slot::TopLeft), false)); + let now = run(&mut placer, saved, now, (0..40).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().rect, to, "it stays as the game plays on"); + + // Redone, it holds; undone again, it is back. + placer.hold(); + assert_eq!(placer.place(first, &keep, CANVAS, now), first); + assert!(placer.placement().held); + placer.undone(saved, now); + assert_eq!(placer.place(saved, &keep, CANVAS, now), to); + + // Redone and undone twice before it is placed again, as on a slow + // canvas, it is still back. + for _ in 0..2 { + placer.redone(first); + placer.undone(saved, now); + } + assert_eq!(placer.place(saved, &keep, CANVAS, now), to); + assert!(!placer.placement().held); + + // An undo of an edit that held nothing holds. + placer.undone(first, now); + assert_eq!(placer.place(first, &keep, CANVAS, now), first); + assert!(placer.placement().held); + } + + /// A rect saved elsewhere is no move by hand unless the caller says so, + /// as a resize from the top left saves one: it stays in its corner. + #[test] + fn a_rect_saved_elsewhere_is_no_move_by_hand() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..6).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + let resized = Rect { + x: 1600.0, + y: 80.0, + width: 262.0, + height: 262.0, + }; + let now = run(&mut placer, resized, now, (0..8).map(|_| read(MOVING, game))); + let placement = placer.placement(); + assert!(!placement.held); + assert_eq!(placement.slot, Some(Slot::TopLeft)); + let _ = now; + } + + /// A resize is no move by hand: it stays in its corner at its new size. + #[test] + fn a_resize_keeps_its_place() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..6).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + let resized = Rect { + width: 200.0, + height: 200.0, + ..saved + }; + let now = run(&mut placer, resized, now, (0..8).map(|_| read(MOVING, game))); + let placement = placer.placement(); + assert!(!placement.held); + assert_eq!(placement.slot, Some(Slot::TopLeft)); + assert_eq!(Some(placement.rect), Slot::TopLeft.rect(200.0, 200.0, CANVAS)); + let _ = now; + } + + /// The content changing kind is a new choice: a camera in the clear + /// corner of a slide moves when a game puts its minimap there. + #[test] + fn new_content_is_a_new_choice() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let now = run(&mut placer, saved, start, (0..8).map(|_| read(STILL, slide))); + assert_eq!(placer.placement().slot, None, "the top right is clear of text"); + let mut now = now; + for _ in 0..16 { + now = run(&mut placer, saved, now, [read(MOVING, game)]); + if placer.placement().slot.is_some() { + break; + } + } + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + assert!( + now.saturating_duration_since(start) < Duration::from_secs(2) + CHANGING + SETTLE + INTERVAL * 4, + "soon after the change, not the worst-place wait" + ); + } + + /// What it is told not to stay clear of does not matter to it. + #[test] + fn a_signal_switched_off_does_not_matter() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + let keep = KeepClear { + hud: false, + text: false, + ..KeepClear::default() + }; + let mut now = start; + for _ in 0..8 { + placer.see(&read(MOVING, game), now); + placer.place(saved, &keep, CANVAS, now); + now = now.saturating_add(INTERVAL); + } + assert_eq!(placer.placement().slot, None); + } + + /// Going on again is a new choice too, made from what it sees then. + #[test] + fn the_scene_going_on_again_looks_again() { + let start = Instant::now(); + let mut placer = Placer::new(start); + let saved = camera(); + placer.shown(Duration::from_secs(5), start); + let now = run(&mut placer, saved, start, (0..8).map(|_| read(STILL, slide))); + assert_eq!(placer.placement().slot, None); + placer.shown(Duration::ZERO, now); + run(&mut placer, saved, now, (0..6).map(|_| read(MOVING, game))); + assert_eq!(placer.placement().slot, Some(Slot::TopLeft)); + } +} diff --git a/rs/moq-scene/src/place.wgsl b/rs/moq-scene/src/place.wgsl new file mode 100644 index 0000000..206b4aa --- /dev/null +++ b/rs/moq-scene/src/place.wgsl @@ -0,0 +1,76 @@ +// The picture under an item keeping clear, as a grid of tiles, against the +// read before: each tile's share of pixels whose brightness moved, that are +// edges, and that are edges that held still, and its mean brightness. One +// workgroup reduces each tile, a thread to a pixel. + +@group(0) @binding(0) var picture: texture_2d; +// The read before, or this one when there was none. +@group(0) @binding(1) var before: texture_2d; +// Each tile, row by row, as packed RGBA8: moved, edges, still, brightness. +@group(0) @binding(2) var tiles: array; + +// A tile's side, as place.rs's TILE. +const SIDE = 16u; +// How far brightness steps to a neighbor at an edge, out of 1. +const EDGE = 0.1; +// How far a pixel's brightness changes when it moved. +const MOVED = 0.06; +// How little an edge's brightness changes when it held still. +const STILL = 0.02; + +var moved: atomic; +var edges: atomic; +var still: atomic; +var light: atomic; + +fn luma(texture: texture_2d, at: vec2, size: vec2) -> f32 { + let color = textureLoad(texture, clamp(at, vec2(0), size - 1), 0).rgb; + return dot(color, vec3(0.2126, 0.7152, 0.0722)); +} + +// How far brightness steps from `at` to its right and lower neighbors, the +// larger of the two. +fn rise(texture: texture_2d, at: vec2, size: vec2, here: f32) -> f32 { + let right = luma(texture, at + vec2(1, 0), size); + let below = luma(texture, at + vec2(0, 1), size); + return max(abs(right - here), abs(below - here)); +} + +@compute @workgroup_size(16, 16) +fn reduce( + @builtin(workgroup_id) tile: vec3, + @builtin(num_workgroups) grid: vec3, + @builtin(global_invocation_id) pixel: vec3, + @builtin(local_invocation_index) index: u32, +) { + let size = vec2(textureDimensions(picture)); + let at = vec2(pixel.xy); + if all(at < size) { + let now = luma(picture, at, size); + let then = luma(before, at, size); + let change = abs(now - then); + if change > MOVED { + atomicAdd(&moved, 1u); + } + if rise(picture, at, size, now) > EDGE { + atomicAdd(&edges, 1u); + if change < STILL && rise(before, at, size, then) > EDGE { + atomicAdd(&still, 1u); + } + } + atomicAdd(&light, u32(round(now * 255.0))); + } + workgroupBarrier(); + if index == 0u { + let start = vec2(tile.xy * SIDE); + let end = min(start + vec2(i32(SIDE)), size); + let count = f32((end.x - start.x) * (end.y - start.y)); + let shares = vec4( + f32(atomicLoad(&moved)), + f32(atomicLoad(&edges)), + f32(atomicLoad(&still)), + f32(atomicLoad(&light)) / 255.0, + ) / count; + tiles[tile.y * grid.x + tile.x] = pack4x8unorm(shares); + } +} diff --git a/rs/moq-studio/shots/keep.toml b/rs/moq-studio/shots/keep.toml new file mode 100644 index 0000000..8fea252 --- /dev/null +++ b/rs/moq-studio/shots/keep.toml @@ -0,0 +1,78 @@ +# The show the Keep out of the way view renders: a game, as an overlay whose +# road rolls and sun bobs on the program clock, under a facecam that keeps +# out of the way. It starts over the minimap, and leaves it. +version = 1 + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.facecam] +kind = "image" +path = "person.jpg" + +[sources.game] +kind = "overlay" +width = 1920 +height = 1080 +html = """ + +
+
+
+
HP 82 / 100
+
+
+
+
+
+
+
+
+
+
+
+
+
24 / 120
+
KIXELATEDRanked grind, day 12
+""" + +[[scenes]] +name = "Gameplay" +items = [ + { source = "game", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + { source = "facecam", rect = { x = 1578, y = 54, width = 288, height = 288 }, crop = { left = 0.21875, right = 0.21875 }, shape = { corners = [ + 0.5, + 0.5, + 0.5, + 0.5, + ], border = 6 }, keep_clear = {} }, +] + +[[audio_scenes]] +name = "Program" +program = true diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 7999791..c5bf581 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -285,12 +285,40 @@ impl State { } else { entries[from..position].iter().map(|&edit| edit.clone()).collect() }; + let moved = moved(&self.collection, &collection); self.adopt(collection, &made)?; + match position < from { + true => self.undone(&moved), + false => self.hold(&moved), + } self.history = history; self.edited(); Ok(()) } + /// Note each layer keeping clear in `moved`, as [`moved`] lists them, as + /// moved by hand: a redo or a reload put it elsewhere, so it stays where + /// the scene file has it until the content under it changes. + fn hold(&mut self, moved: &[Moved]) { + for moved in moved { + self.compositor + .redone(&moved.scene, moved.index, &moved.source, moved.rect); + } + } + + /// Note each layer keeping clear in `moved`, as [`moved`] lists them, as + /// put back by an undo: one back where it was saved before a move by hand + /// held it, or before keeping clear was switched off, goes back to where + /// it was then, such as the slot it had glided to, so the undo puts back what the streamer saw; any other + /// holds, as [`hold`](Self::hold) says. + fn undone(&mut self, moved: &[Moved]) { + let now = kio::clock::now(); + for moved in moved { + self.compositor + .undone(&moved.scene, moved.index, &moved.source, moved.rect, now); + } + } + /// Make `collection`, which the edits `made` led to from the current one, /// the one drawn, once the compositor fits its scenes' sizes. The scenes /// in preview and on program follow the renames and removals `made` holds, @@ -680,7 +708,9 @@ impl autosave::Document for State { } = loaded; // A scene in preview or on program that the file no longer has, such as // one renamed by hand, leaves it until another is picked. + let moved = moved(&self.collection, &collection); self.adopt(collection, &[])?; + self.hold(&moved); self.history = history; self.dropped = dropped; self.reloads += 1; @@ -689,6 +719,44 @@ impl autosave::Document for State { } } +/// A scene item keeping clear that an edit saved elsewhere, as [`moved`] +/// finds it. +#[derive(Clone, Debug, PartialEq)] +struct Moved { + scene: String, + index: usize, + source: String, + /// Where it is saved now. + rect: moq_scene::Rect, +} + +/// Each scene item keeping clear in `after` that `before` saved elsewhere or +/// had not keeping clear: the same scene's item at the same index showing +/// the same source, at another x or y or switched back on. Its keeper is +/// still under that place, as the compositor follows items only as it +/// renders. +fn moved(before: &Collection, after: &Collection) -> Vec { + let mut moved = Vec::new(); + for scene in &after.scenes { + let Some(was) = before.scene(&scene.name) else { continue }; + for (index, item) in scene.items.iter().enumerate() { + let Some(old) = was.items.get(index) else { continue }; + if item.keep_clear.is_some() + && old.source == item.source + && (old.keep_clear.is_none() || (old.rect.x, old.rect.y) != (item.rect.x, item.rect.y)) + { + moved.push(Moved { + scene: scene.name.clone(), + index, + source: item.source.clone(), + rect: item.rect, + }); + } + } + } + moved +} + /// Refuse edits, each read as its [`Collection::renames`], that remove a /// scene `on_air` program draws, so a slip cannot put black on air, that /// leave one drawing the program in `collection`, which it would draw into @@ -3419,6 +3487,135 @@ mod tests { } } + /// A layer keeping clear that an undo, a redo, or a reload saved + /// elsewhere, or switched keeping clear back on, is listed as moved; one + /// only resized, another source, or one that does not keep clear is not. + #[test] + fn an_undo_or_a_reload_moves_by_hand() { + let mut before = red_canvas().collection; + let rect = moq_scene::Rect { + x: 400.0, + y: 200.0, + width: 100.0, + height: 100.0, + }; + let keeping = moq_scene::Item { + keep_clear: Some(moq_scene::KeepClear::default()), + ..moq_scene::Item::new("red", rect) + }; + before.scenes[0] + .items + .extend([keeping.clone(), keeping.clone(), keeping]); + before.scenes[0].items.push(moq_scene::Item::new("red", rect)); + before.scenes[0].items.push(moq_scene::Item::new("red", rect)); + let main = before.scenes[0].name.clone(); + let mut after = before.clone(); + let items = &mut after.scenes[0].items; + items[1].rect.x = 300.0; + items[2].rect.width = 50.0; + items[3].source = "blue".into(); + items[3].rect.x = 300.0; + items[4].rect.x = 300.0; + items[5].keep_clear = Some(moq_scene::KeepClear::default()); + let moved = |index, rect| Moved { + scene: main.clone(), + index, + source: "red".into(), + rect, + }; + let elsewhere = moq_scene::Rect { x: 300.0, ..rect }; + assert_eq!(super::moved(&before, &after), [moved(1, elsewhere), moved(5, rect)]); + } + + /// An undo of a move by hand puts a layer keeping clear back as it was + /// before the move, and a redo holds it again. An undo of an edit made + /// in the collection rather than by a gesture, which held nothing, holds + /// it where it puts it. + #[tokio::test(flavor = "multi_thread")] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn an_undo_puts_back_a_layer_keeping_clear() { + let dir = tempfile::tempdir().unwrap(); + let mut loaded = red_canvas(); + let rect = moq_scene::Rect { + x: 400.0, + y: 200.0, + width: 100.0, + height: 100.0, + }; + loaded.collection.scenes[0].items.push(moq_scene::Item { + keep_clear: Some(moq_scene::KeepClear::default()), + ..moq_scene::Item::new("red", rect) + }); + let main = loaded.collection.scenes[0].name.clone(); + let gpu = Gpu::new(wgpu::Backends::all()).await.expect("a GPU"); + let (autosave, _) = Autosave::open(dir.path().join("show.toml")).unwrap(); + let engine = start(gpu, Vec::new(), dir.path(), autosave, loaded) + .await + .expect("the studio starts"); + let placed = |at: moq_scene::Rect, held: bool| { + let engine = &engine; + let main = &main; + async move { + loop { + let placement = engine + .state + .lock() + .unwrap() + .compositor + .placement(main, 1, "red") + .cloned(); + if placement.is_some_and(|placement| placement.rect == at && placement.held == held) { + return; + } + kio::clock::sleep(Duration::from_millis(10)).await; + } + } + }; + kio::clock::timeout(TIMEOUT, placed(rect, false)) + .await + .expect("placed where it is saved"); + + let commit = |to: moq_scene::Rect, by_hand: bool| { + let mut state = engine.state.lock().unwrap(); + if by_hand { + state.compositor.hold(&main, 1, "red"); + } + let collection = state.collection(); + let item = collection.scenes[0].items[1].clone(); + let mut draft = collection.draft("Move `red`"); + let layer = moq_scene::Layer::scene(main.as_str()); + draft.set_item(&layer, 1, moq_scene::Item { rect: to, ..item }).unwrap(); + state.commit(draft, None).unwrap(); + }; + + let moved = moq_scene::Rect { x: 200.0, ..rect }; + commit(moved, true); + kio::clock::timeout(TIMEOUT, placed(moved, true)) + .await + .expect("a move by hand holds it"); + engine.state.lock().unwrap().undo().unwrap(); + kio::clock::timeout(TIMEOUT, placed(rect, false)) + .await + .expect("the undo puts it back as it was"); + engine.state.lock().unwrap().redo().unwrap(); + kio::clock::timeout(TIMEOUT, placed(moved, true)) + .await + .expect("the redo holds it"); + engine.state.lock().unwrap().undo().unwrap(); + kio::clock::timeout(TIMEOUT, placed(rect, false)) + .await + .expect("undone again, it is back"); + + commit(moved, false); + kio::clock::timeout(TIMEOUT, placed(moved, false)) + .await + .expect("an edit is no move by hand"); + engine.state.lock().unwrap().undo().unwrap(); + kio::clock::timeout(TIMEOUT, placed(rect, true)) + .await + .expect("the undo holds it"); + } + /// Frames the recording's own encoder encoded. fn recorded(engine: &Engine) -> u64 { engine.stats.recording.renditions[Role::Record as usize] diff --git a/rs/moq-studio/src/shots.rs b/rs/moq-studio/src/shots.rs index f70e5e5..7873bc2 100644 --- a/rs/moq-studio/src/shots.rs +++ b/rs/moq-studio/src/shots.rs @@ -110,6 +110,9 @@ enum View { Cued(Page), /// A scene, by name, open for editing off air on BUILD. OffAir(&'static str), + /// BUILD with the front layer selected, once it kept out of the way: + /// moved off what mattered and landed. + KeepClear, /// BUILD with the game's window gone dark selected, and what to do about /// it open from its badge. Dark, @@ -288,6 +291,13 @@ views! { build_off_air_refused: Still, studio(View::OffAir("Vertical")); /// Too narrow for the banner beside the live picture, it goes under it. build_off_air_compact: Still, Open::Studio { view: View::OffAir("Break"), size: HALF, show: "show.toml", relay: false }; + /// A facecam keeping out of the way of a game: what matters glowing on the + /// picture, the places it may go, where it was and the glide to where it + /// went, the legend under the picture, and its settings in the inspector. + build_keep_clear: Live, Open::Studio { view: View::KeepClear, size: SIZE, show: "keep.toml", relay: false }; + /// The same in the smallest window: the legend's labels shortened to fit + /// beside the note, or the note left out. + build_keep_clear_compact: Live, Open::Studio { view: View::KeepClear, size: crate::ui::MIN_SIZE, show: "keep.toml", relay: false }; /// A window gone dark: its row, the layer on the picture, its scene's /// pad, and the health dot turn amber, and its badge opens what happened /// and the fixes. @@ -1435,6 +1445,7 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show View::Mix(view, opened) => studio.mix(*view, *opened), View::MixMenu => studio.mix(MixView::Strips, None), View::Build { side, inspect, index } => studio.build(*side, *inspect, *index), + View::KeepClear => studio.build(Side::Layers, Inspect::Layer, Some(0)), View::Menu => studio.build(Side::Layers, Inspect::Layer, Some(1)), View::Setting(setting) | View::OnAirSetting(setting) => studio.open(*setting), View::Restart | View::CustomCanvas => studio.open(Setting::Show), @@ -1540,6 +1551,29 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show } })) .expect("the room fails to join at launch"), + View::KeepClear => runtime + .block_on(kio::clock::timeout(SETTLE, async { + loop { + let landed = { + let state = engine.state.lock().unwrap(); + let item = &state.collection().scenes[0].items[1]; + let placement = state.compositor.placement("Gameplay", 1, &item.source); + placement.is_some_and(|placement| { + let slot = placement + .slot + .and_then(|slot| placement.slots.iter().find(|(fits, _)| *fits == slot)); + placement.left.is_some() && slot.is_some_and(|(_, rect)| *rect == placement.rect) + }) + }; + if landed { + break; + } + harness.step(); + kio::clock::sleep(Duration::from_millis(20)).await; + } + harness.run_steps(2); + })) + .expect("the facecam moves off the minimap and lands"), View::OnAirHealth => { let pill = ["Healthy", "Starting", "Degraded", "Unhealthy"] .into_iter() diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 98a2373..82f7afc 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -35,6 +35,7 @@ mod dark; mod drop; mod graph; mod inspector; +mod keep; mod layers; mod layout; mod levels; @@ -256,6 +257,7 @@ impl Studio { duration: Transition::default().duration().unwrap_or_default(), cut: STINGER_CUT, drag: None, + legend: false, renaming: None, title_needs: 0.0, renaming_audio: None, @@ -663,6 +665,10 @@ struct Panels { cut: Duration, /// The layer a drag on the BUILD page's picture holds. drag: Option, + /// Whether the BUILD page's picture leaves room for the keep-clear + /// legend under it, which a drag leaves as it found it, so the picture + /// under the pointer neither moves nor resizes mid-gesture. + legend: bool, /// The layer row being dragged to a new place, by its index. moving: Option, /// The scene being renamed, by index, and the name typed so far. @@ -2144,7 +2150,7 @@ fn others(state: &State, scene: Option<&str>, dragged: &Picked, since: Duration) let items = slots::placed(state, &shown); for (at, item) in items.iter().enumerate() { if item.visible && !(shown == dragged.layer && at == dragged.index) { - rects.push(item.posed(since).rect); + rects.push(keep::placed(state, &shown, at, item).posed(since).rect); } } } diff --git a/rs/moq-studio/src/ui/build.rs b/rs/moq-studio/src/ui/build.rs index 7724d50..fe3d8b3 100644 --- a/rs/moq-studio/src/ui/build.rs +++ b/rs/moq-studio/src/ui/build.rs @@ -48,7 +48,20 @@ impl Panels { room.min.y = self.off_air_header(ui, cued, room, snapshot.shape); shape = cued.shape(); } - self.stage(ui, fx, snapshot, super::layout::fit(room, shape)); + // A drag that selects a layer keeping clear leaves the picture where + // it was until let go, so the pointer stays over what it moves. + if self.drag.is_none() { + self.legend = self.keeping_clear(); + } + let keeping = self.legend; + if keeping { + room.max.y -= keep::LEGEND; + } + let at = super::layout::fit(room, shape); + self.stage(ui, fx, snapshot, at); + if keeping { + keep::legend(ui, at); + } self.build_keys(ui); } @@ -121,7 +134,10 @@ impl Panels { let picked = self.selection.clone().expect("interacting with the selected layer"); let only = Some((&picked.layer, picked.index)); let staged = super::slots::staged(&state, scene.map(|(scene, _)| scene)); + // A layer keeping clear takes the pointer where it is drawn. + let staged = keep::drawn(&state, staged); let delivered = self.interact.update(ui, image, &staged, scene, only); + drop(staged); deliver(&state.sources, delivered); self.rebind(&state.sources); let stop = interacting_banner(ui, image, &page); @@ -179,6 +195,17 @@ impl Panels { } } + /// Whether the selected layer keeps out of the way, so the stage explains + /// what it saw under the picture. + fn keeping_clear(&self) -> bool { + let Some(picked) = &self.selection else { + return false; + }; + let state = self.engine.state.lock().unwrap(); + let item = picked.item(state.collection()); + item.is_some_and(|item| item.visible && keep::placement(&state, &picked.layer, picked.index, item).is_some()) + } + /// The web page the selected layer shows, while it is a visible layer of /// the picture that shows one. fn interacted(&self, state: &State) -> Option { @@ -204,7 +231,8 @@ impl Panels { /// The layers the picture edits, front last, each with its place: the /// scene's, then the always-on-top ones over it, unless the scene is open /// off air, as they are drawn on program, each in its slot where the - /// scene's arrangement put it. + /// scene's arrangement put it, and each keeping clear where it was + /// placed. pub(super) fn shown(&self, state: &State) -> Vec<(Picked, Item)> { let (_, since) = self.edited_since(state); let top = self.off_air(state).is_none().then(Layer::top); @@ -282,7 +310,10 @@ impl Panels { let picked = shown[index].0.clone(); self.select(Some(picked.clone())); let collection = state.collection(); - let resting = picked.item(collection).cloned(); + // A layer keeping clear moves from where it was placed. + let resting = picked + .item(collection) + .map(|item| keep::placed(state, &picked.layer, picked.index, item)); if let Some(resting) = resting { let drawn = match resting.slot { Some(_) => super::slots::drawn(state, &picked.layer, picked.index), @@ -408,6 +439,15 @@ impl Panels { let label = format!("{verb} `{}`", item.source); spot = Some(Spot::Layer(drag.picked.layer.clone(), item.source.clone())); let picked = &drag.picked; + // A move by hand holds a layer keeping clear where it is put; + // a resize or a crop leaves it where it rests. + let rect = match verb { + "Move" => { + keep::hold(state, &picked.layer, picked.index, &item); + rect + } + _ => keep::resized(state, &picked.layer, picked.index, &item, rect), + }; result = edit(state, label, drag.gesture, |draft| { draft.set_item(&picked.layer, picked.index, Item { rect, crop, ..item }) }); @@ -416,6 +456,12 @@ impl Panels { if !response.dragged() { self.drag = None; } + // A layer keeping clear stays put while in hand, which each frame + // says again, at least four times a lease, whatever the canvas rate. + keep::grab(state, self.drag.as_ref().map(|drag| &drag.picked)); + if self.drag.is_some() { + ui.ctx().request_repaint_after(moq_scene::place::GRAB / 4); + } if response.has_focus() && self.drag.is_none() && result.is_ok() { (spot, result) = self.nudge(ui, state, response.id); } @@ -427,6 +473,15 @@ impl Panels { super::slots::outlines(&painter, &view, state, room, &shown, target); let selected = shown.iter().find(|(picked, _)| Some(picked) == self.selection.as_ref()); let accent = ui.visuals().selection.stroke.color; + // What a layer keeping clear saw, under its handles, unless it is + // being dragged, which holds it where it is put. + if let Some((picked, item)) = selected.filter(|(_, item)| item.visible && self.drag.is_none()) + && let Some(resting) = picked.item(state.collection()) + && let Some(placement) = keep::placement(state, &picked.layer, picked.index, resting) + && let Some(keep) = resting.keep_clear + { + keep::overlay(&painter, &view, placement, &item.shape, keep.glide); + } // A hidden layer has no handles, as the picture cannot grab it, and a // locked one only its outline. if let Some((picked, item)) = selected.filter(|(_, item)| item.visible) { @@ -527,6 +582,8 @@ impl Panels { else { return (None, Ok(())); }; + // A layer keeping clear moves from where it was placed. + let item = keep::placed(state, &picked.layer, picked.index, &item); if delta == [0.0; 2] { return (None, Ok(())); } @@ -541,6 +598,7 @@ impl Panels { }; let label = format!("Move `{}`", item.source); let spot = Spot::Layer(picked.layer.clone(), item.source.clone()); + keep::hold(state, &picked.layer, picked.index, &item); let result = edit(state, label, Some(gesture), |draft| { draft.set_item(&picked.layer, picked.index, Item { rect, ..item }) }); @@ -736,11 +794,15 @@ impl Panels { }; let count = collection.items(&to).map_or(0, <[Item]>::len); // The always-on-top layers never go on, so they have no - // keyframes to play, nor slots; it stays where it was drawn. + // keyframes to play, nor slots, nor a picture under them to + // keep clear of; it stays where it was drawn. + let placed = keep::placed(state, layer, index, &item).rect; let (rect, crop) = super::slots::drawn(state, layer, index).unwrap_or((item.rect, item.crop)); + let rect = if item.slot.is_none() { placed } else { rect }; let moved = Item { keyframes: if top { Vec::new() } else { item.keyframes.clone() }, slot: None, + keep_clear: if top { None } else { item.keep_clear }, rect, crop, ..item.clone() diff --git a/rs/moq-studio/src/ui/inspector.rs b/rs/moq-studio/src/ui/inspector.rs index bead4ae..934d90a 100644 --- a/rs/moq-studio/src/ui/inspector.rs +++ b/rs/moq-studio/src/ui/inspector.rs @@ -95,6 +95,10 @@ impl Panels { } let name = item.source.clone(); let mut asked = None; + // Whether the edit asked for is a move by hand, or switches keeping + // clear off, which leaves it where it is drawn as a move would. + let mut moved = false; + let kept = item.keep_clear; section(ui, "Place"); if picked.layer.scene.is_some() { @@ -124,7 +128,10 @@ impl Panels { asked = Some(Asked::Act(LayerAction::Slot(slot))); } } - let mut rect = item.rect; + // Where the picture draws it: a layer keeping clear where it was + // placed, which a move by hand starts from. + let placement = keep::placement(&state, &picked.layer, picked.index, &item); + let mut rect = placement.map_or(item.rect, |placement| placement.drawn(item.rect)); if item.slot.is_some() { ui.label( egui::RichText::new("Placed by the scene's arrangement; drag it onto another slot to swap them") @@ -153,6 +160,17 @@ impl Panels { for (verb, response) in responses { if let Some(gesture) = changed(&response, &mut self.gestures) { let label = format!("{verb} `{name}`"); + // A resize keeps where a layer keeping clear rests, so it + // stays free to move. + let rect = match verb { + "Resize" => Rect { + x: item.rect.x, + y: item.rect.y, + ..rect + }, + _ => rect, + }; + moved = verb == "Move"; asked = Some(Asked::Item(Item { rect, ..item.clone() }, label, gesture)); } } @@ -204,6 +222,34 @@ impl Panels { asked = Some(Asked::Act(LayerAction::Lock(locked))); } }); + // Only a scene's layer has a picture under it to keep clear of, and + // viewers place a published source where the scene file does. + let published = collection.published.contains(collection.capture(&item.source)); + if picked.layer.scene.is_some() + && item.slot.is_none() + && !published + && let Some((keep_clear, label)) = keep::inspector(ui, item.keep_clear, placement) + { + let label = format!("{label}: `{name}`"); + // Switched off, it stays where it is drawn, as keeping it always + // on top does, rather than going back over what it moved off. + let rect = match (keep_clear, placement) { + (None, Some(placement)) => { + moved = true; + placement.drawn(item.rect) + } + _ => item.rect, + }; + asked = Some(Asked::Item( + Item { + keep_clear, + rect, + ..item.clone() + }, + label, + None, + )); + } section(ui, "Shape"); if let Some(corners) = shapes(ui, item.shape.corners) { @@ -505,9 +551,19 @@ impl Panels { let Some(asked) = asked else { return }; let result = match asked { - Asked::Item(item, label, gesture) => edit(&mut state, label, gesture, |draft| { - draft.set_item(&picked.layer, picked.index, item) - }), + Asked::Item(item, label, gesture) => { + if moved { + // Held as it was before the edit, so an undo puts it back. + let was = Item { + keep_clear: kept, + ..item.clone() + }; + keep::hold(&mut state, &picked.layer, picked.index, &was); + } + edit(&mut state, label, gesture, |draft| { + draft.set_item(&picked.layer, picked.index, item) + }) + } Asked::Look(selection, look, label, gesture) => { edit(&mut state, label, gesture, |draft| selection.set_look(draft, look)) } diff --git a/rs/moq-studio/src/ui/keep.rs b/rs/moq-studio/src/ui/keep.rs new file mode 100644 index 0000000..5081633 --- /dev/null +++ b/rs/moq-studio/src/ui/keep.rs @@ -0,0 +1,586 @@ +//! Keep out of the way: where the picture draws a layer that moves itself +//! off what matters, what it saw there, and its settings in the inspector. + +use std::borrow::Cow; +use std::time::Duration; + +use egui::{Color32, Pos2, Stroke}; +use moq_scene::place::{Placement, Signal, Slot}; +use moq_scene::{KeepClear, Shape}; + +use super::*; + +/// What matters on screen, as the stage shows it. +const HEAT: Color32 = Color32::from_rgb(0x4d, 0xe3, 0xff); + +/// The glides the inspector offers. +const GLIDES: [Duration; 5] = [ + Duration::ZERO, + Duration::from_millis(200), + Duration::from_millis(400), + Duration::from_millis(800), + Duration::from_millis(1500), +]; + +/// The legend's height under the stage, with its gap. +pub(super) const LEGEND: f32 = 30.0; + +/// How the last render placed `item`, the `index`th of `layer`, while it +/// keeps clear. +pub(super) fn placement<'a>(state: &'a State, layer: &Layer, index: usize, item: &Item) -> Option<&'a Placement> { + item.keep_clear?; + let scene = layer.scene.as_deref()?; + state.compositor.placement(scene, index, &item.source) +} + +/// `item`, the `index`th of `layer`, resting where the picture draws it: a +/// layer keeping clear where it was placed. +pub(super) fn placed(state: &State, layer: &Layer, index: usize, item: &Item) -> Item { + match placement(state, layer, index, item) { + Some(placement) => Item { + rect: placement.drawn(item.rect), + ..item.clone() + }, + None => item.clone(), + } +} + +/// `rect`, a new rect for `item` from a gesture other than a move, such as a +/// resize, to save: a layer placed off where it rests keeps resting where it +/// did, so the resize leaves it free to keep clear. +pub(super) fn resized(state: &State, layer: &Layer, index: usize, item: &Item, rect: Rect) -> Rect { + match placement(state, layer, index, item) { + Some(placement) if placement.drawn(item.rect) != item.rect => Rect { + x: item.rect.x, + y: item.rect.y, + ..rect + }, + _ => rect, + } +} + +/// Note that `item`, the `index`th of `layer`, was moved by hand: a layer +/// keeping clear stays where it was put until the content under it changes. +pub(super) fn hold(state: &mut State, layer: &Layer, index: usize, item: &Item) { + if item.keep_clear.is_some() + && let Some(scene) = layer.scene.as_deref() + { + state.compositor.hold(scene, index, &item.source); + } +} + +/// Note the layer a gesture on the picture has in hand, if any: a layer +/// keeping clear does not choose a place while in hand. +pub(super) fn grab(state: &mut State, picked: Option<&Picked>) { + let collection = state.collection(); + let grabbed = picked.and_then(|picked| { + let scene = picked.layer.scene.clone()?; + let item = picked.item(collection)?; + item.keep_clear?; + Some((scene, picked.index, item.source.clone())) + }); + let grabbed = grabbed + .as_ref() + .map(|(scene, index, source)| (scene.as_str(), *index, source.as_str())); + state.compositor.grab(grabbed, kio::clock::now()); +} + +/// `staged`, the collection as the picture draws it, with each layer +/// keeping clear where it was placed, so a pointer over the picture finds +/// what it sees. +pub(super) fn drawn<'a>(state: &State, staged: Cow<'a, Collection>) -> Cow<'a, Collection> { + let collection = state.collection(); + let mut drawn = staged; + for (number, scene) in collection.scenes.iter().enumerate() { + for (index, item) in scene.items.iter().enumerate() { + if item.keep_clear.is_none() { + continue; + } + let Some(placement) = state.compositor.placement(&scene.name, index, &item.source) else { + continue; + }; + // A scene staged one frame behind an edit may lack it. + let staged = drawn.scenes.get(number).and_then(|staged| staged.items.get(index)); + let rect = placement.drawn(item.rect); + if staged.is_some_and(|staged| staged.source == item.source && staged.rect != rect) { + drawn.to_mut().scenes[number].items[index].rect = rect; + } + } + } + drawn +} + +fn slot_name(slot: Slot) -> &'static str { + match slot { + Slot::TopLeft => "Top left", + Slot::Top => "Top", + Slot::TopRight => "Top right", + Slot::Left => "Left", + Slot::Right => "Right", + Slot::BottomLeft => "Bottom left", + Slot::Bottom => "Bottom", + Slot::BottomRight => "Bottom right", + } +} + +/// Its chip in the inspector, its tag on the stage, and what a layer left +/// over it. +fn signal_names(signal: Signal) -> (&'static str, &'static str, &'static str) { + match signal { + Signal::Motion => ("Motion", "MOVING", "the action"), + Signal::Text => ("Text", "TEXT", "the text"), + Signal::Hud => ("HUD", "HUD", "the HUD"), + } +} + +/// The inspector's Keep out of the way, for a layer keeping clear as `keep` +/// says, placed as `placement` says: its switch and, while on, what it stays +/// clear of, where it is, and how it glides. The settings asked for, with a +/// label, if any changed. +pub(super) fn inspector( + ui: &mut Ui, + keep: Option, + placement: Option<&Placement>, +) -> Option<(Option, &'static str)> { + let mut asked = None; + ui.horizontal(|ui| { + ui.label("Keep out of the way"); + ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| { + let mut on = keep.is_some(); + if ui + .add(kit::Toggle::new(&mut on)) + .on_hover_text( + "Move off what matters in the picture under it: motion, text, and a HUD that holds still. \ + Decided on this machine.", + ) + .changed() + { + let label = if on { + "Keep out of the way" + } else { + "Stop keeping out of the way" + }; + asked = Some((on.then(KeepClear::default), label)); + } + ui.label( + egui::RichText::new("moves off what matters") + .color(kit::DIM) + .font(kit::ui(11.0)), + ); + }); + }); + let Some(mut keep) = keep else { return asked }; + ui.add_space(4.0); + egui::Frame::new() + .fill(kit::P2) + .stroke(Stroke::new(1.0, kit::LINE)) + .corner_radius(kit::RADIUS) + .inner_margin(10.0) + .show(ui, |ui| { + ui.set_width(ui.available_width()); + ui.spacing_mut().item_spacing.y = 9.0; + ui.label( + egui::RichText::new("Stays clear of") + .color(kit::DIM) + .font(kit::ui(12.0)), + ); + ui.horizontal_wrapped(|ui| { + ui.spacing_mut().item_spacing.x = 6.0; + for signal in [Signal::Motion, Signal::Text, Signal::Hud] { + let on = match signal { + Signal::Motion => &mut keep.motion, + Signal::Text => &mut keep.text, + Signal::Hud => &mut keep.hud, + }; + let (name, _, _) = signal_names(signal); + if chip(ui, name, on).clicked() { + asked = Some((Some(keep), "Change what it stays clear of")); + } + } + }); + ui.horizontal(|ui| { + ui.spacing_mut().item_spacing.x = 10.0; + let (slot, held) = placement.map_or((None, false), |placement| (placement.slot, placement.held)); + cells(ui, slot); + let (name, note) = match (slot, held) { + (Some(slot), _) => (slot_name(slot), "Corners first"), + (None, true) => ("Where you put it", "Until the content changes"), + (None, false) => ("Where it was put", "Until a corner is clearer"), + }; + ui.vertical(|ui| { + ui.spacing_mut().item_spacing.y = 2.0; + ui.label(egui::RichText::new(name).font(kit::ui(12.0))); + ui.label(egui::RichText::new(note).color(kit::FAINT).font(kit::ui(10.5))); + }); + ui.with_layout(egui::Layout::right_to_left(egui::Align::Center), |ui| { + let mut glide = keep.glide; + egui::ComboBox::from_id_salt("keep clear glide") + .selected_text(format!("Glide ยท {}", seconds(glide))) + .width(96.0) + .show_ui(ui, |ui| { + for choice in GLIDES { + let text = match choice.is_zero() { + true => "Jump".to_string(), + false => seconds(choice), + }; + ui.selectable_value(&mut glide, choice, text); + } + }); + if glide != keep.glide { + keep.glide = glide; + asked = Some((Some(keep), "Change how it glides")); + } + }); + }); + ui.label( + egui::RichText::new("Moves only when the content changes.") + .color(kit::DIM) + .font(kit::ui(12.0)), + ); + }); + asked +} + +/// `duration` in seconds, as the glide reads. +fn seconds(duration: Duration) -> String { + format!("{:.1} s", duration.as_secs_f32()) +} + +/// A chip that is on or off, ticked while on. +fn chip(ui: &mut Ui, text: &str, on: &mut bool) -> egui::Response { + let galley = ui + .painter() + .layout_no_wrap(text.into(), kit::ui_medium(12.0), kit::TEXT); + let size = egui::vec2(10.0 + 14.0 + 5.0 + galley.size().x + 10.0, 26.0); + let (rect, mut response) = ui.allocate_exact_size(size, egui::Sense::click()); + if response.clicked() { + *on = !*on; + response.mark_changed(); + } + let lit = *on; + response.widget_info(|| egui::WidgetInfo::selected(egui::WidgetType::Checkbox, true, lit, text)); + let painter = ui.painter(); + let (fill, stroke, ink) = match lit { + true => (kit::ACCENT.gamma_multiply(0.14), kit::ACCENT, kit::TEXT), + false => (Color32::TRANSPARENT, kit::LINE, kit::DIM), + }; + let stroke = match response.hovered() { + true => stroke.gamma_multiply(1.3), + false => stroke, + }; + painter.rect(rect, 6, fill, Stroke::new(1.0, stroke), egui::StrokeKind::Inside); + let mark = egui::Rect::from_center_size(egui::pos2(rect.left() + 17.0, rect.center().y), egui::Vec2::splat(14.0)); + if lit { + Icon::Check.paint(ui, mark, 13.0, kit::TEXT); + } + let at = egui::pos2(mark.right() + 5.0, rect.center().y - galley.size().y / 2.0); + ui.painter().galley(at, galley, ink); + response +} + +/// The slots as a 3x3 grid, `slot` lit and the middle, where it never goes, +/// dashed. +fn cells(ui: &mut Ui, slot: Option) { + let (cell, gap) = (egui::vec2(17.0, 11.0), 3.0); + let size = egui::vec2(cell.x * 3.0 + gap * 2.0, cell.y * 3.0 + gap * 2.0); + let (rect, _) = ui.allocate_exact_size(size, egui::Sense::hover()); + let painter = ui.painter(); + let lit = slot.map(Slot::cell); + for row in 0..3u8 { + for column in 0..3u8 { + let min = rect.min + egui::vec2(f32::from(column) * (cell.x + gap), f32::from(row) * (cell.y + gap)); + let at = egui::Rect::from_min_size(min, cell); + let grey = Color32::from_rgb(0x3d, 0x3d, 0x4b); + match (column, row) { + (1, 1) => super::dashed(painter, at, Stroke::new(1.0, grey)), + _ if lit == Some((column, row)) => drop(painter.rect_filled(at, 2, kit::ACCENT)), + _ => drop(painter.rect_filled(at, 2, grey)), + } + } + } +} + +/// What a layer keeping clear saw and did, over the stage: what matters on +/// screen, glowing and tagged by kind; the slots it may go to, dashed in its +/// shape; and where it last moved from, with an arrow to where it went. +pub(super) fn overlay(painter: &egui::Painter, view: &View, placement: &Placement, shape: &Shape, glide: Duration) { + let heat = &placement.heat; + let screen = |x: f32, y: f32| view.rect.min + egui::vec2(x * view.rect.width(), y * view.rect.height()); + glow(painter, heat, screen); + // A tag for each kind of thing that matters, in the middle of where it + // matters most: the tile whose neighbors matter most for it with it. + let mut tags: Vec<(Signal, f32, Pos2)> = Vec::new(); + let mattering = |column: usize, row: usize, kind: Signal| { + let (matters, signal) = heat.tiles[row * heat.columns + column]; + if signal == kind { matters } else { 0.0 } + }; + for row in 0..heat.rows { + for column in 0..heat.columns { + let (matters, signal) = heat.tiles[row * heat.columns + column]; + if matters < 0.35 { + continue; + } + let near = |at: usize, length: usize| at.saturating_sub(1)..(at + 2).min(length); + let around: f32 = near(row, heat.rows) + .flat_map(|row| near(column, heat.columns).map(move |column| (column, row))) + .map(|(column, row)| mattering(column, row, signal)) + .sum(); + let (x0, y0, x1, y1) = heat.tile(column, row); + let center = screen((x0 + x1) / 2.0, (y0 + y1) / 2.0); + match tags.iter_mut().find(|(kind, _, _)| *kind == signal) { + Some(tag) if tag.1 < around => *tag = (signal, around, center), + Some(_) => {} + None => tags.push((signal, around, center)), + } + } + } + + let dashed = Stroke::new(1.5, Color32::from_white_alpha(128)); + for (slot, rect) in &placement.slots { + // Its own slot shows as it, and the one it left as that. + let left = placement.left.is_some_and(|left| left.rect == *rect); + if Some(*slot) == placement.slot || left { + continue; + } + let outline = outline(view.screen(rect), shape); + painter.extend(egui::Shape::dashed_line(&outline, dashed, 5.0, 4.0)); + } + + for (signal, _, at) in tags { + let (_, tag, _) = signal_names(signal); + let galley = painter.layout_no_wrap(tag.into(), kit::display_bold(10.0), Color32::from_rgb(0xc9, 0xf6, 0xff)); + let rect = egui::Rect::from_center_size(at, galley.size() + egui::vec2(10.0, 6.0)); + painter.rect_filled(rect, 3, Color32::from_rgba_unmultiplied(0x06, 0x22, 0x2b, 0xd9)); + painter.galley(rect.min + egui::vec2(5.0, 3.0), galley, Color32::WHITE); + } + // Its notes over the tags, which they name again where they meet. + if let Some(left) = placement.left { + moved(painter, view, placement, left, shape, glide); + } +} + +/// Where a layer keeping clear moved from, what it left there, and an arrow +/// to where it went. +fn moved( + painter: &egui::Painter, + view: &View, + placement: &Placement, + left: moq_scene::place::Left, + shape: &Shape, + glide: Duration, +) { + let was = view.screen(&left.rect); + let outline_was = outline(was, shape); + painter.add(egui::Shape::convex_polygon( + outline_was.clone(), + Color32::from_white_alpha(30), + Stroke::NONE, + )); + painter.extend(egui::Shape::dashed_line( + &outline_was, + Stroke::new(2.0, Color32::WHITE), + 6.0, + 4.0, + )); + let to = view.screen(&placement_target(placement)); + if was.center().distance(to.center()) > 4.0 { + arrow(painter, was, to); + if !glide.is_zero() { + let mid = was.center() + (to.center() - was.center()) / 2.0; + note(painter, mid, &format!("{} glide", seconds(glide))); + } + } + let text = match left.over { + Some(signal) => format!("Was here, over {}", signal_names(signal).2), + None => "Was here".to_string(), + }; + note(painter, egui::pos2(was.center().x, was.top() - 14.0), &text); +} + +/// What matters, as a soft glow: each corner of the tiles as bright as the +/// tiles around it, blended between. +fn glow(painter: &egui::Painter, heat: &moq_scene::place::Heat, screen: impl Fn(f32, f32) -> Pos2) { + let (columns, rows) = (heat.columns, heat.rows); + if columns == 0 || rows == 0 { + return; + } + let (across, down) = &heat.edges; + let mut mesh = egui::Mesh::default(); + for (row, y) in down.iter().enumerate() { + for (column, x) in across.iter().enumerate() { + let mut sum = 0.0; + let mut count = 0.0; + for (c, r) in [ + (column, row), + (column.wrapping_sub(1), row), + (column, row.wrapping_sub(1)), + (column.wrapping_sub(1), row.wrapping_sub(1)), + ] { + if c < columns && r < rows { + sum += heat.tiles[r * columns + c].0; + count += 1.0; + } + } + let matters = sum / count; + // Faint for what barely matters, so the picture shows through. + let alpha = ((matters - 0.08) / 0.62).clamp(0.0, 1.0) * 0.55; + mesh.colored_vertex(screen(*x, *y), HEAT.gamma_multiply(alpha)); + } + } + let stride = (columns + 1) as u32; + for row in 0..rows as u32 { + for column in 0..columns as u32 { + let at = row * stride + column; + mesh.add_triangle(at, at + 1, at + stride); + mesh.add_triangle(at + 1, at + stride + 1, at + stride); + } + } + painter.add(egui::Shape::mesh(mesh)); +} + +/// Where a layer keeping clear is going: its slot, or where it is. +fn placement_target(placement: &Placement) -> Rect { + let slot = placement + .slot + .and_then(|slot| placement.slots.iter().find(|(fits, _)| *fits == slot)); + slot.map_or(placement.rect, |(_, rect)| *rect) +} + +/// A note in a dark pill, centered on `at`, kept on the picture. +fn note(painter: &egui::Painter, at: Pos2, text: &str) { + let galley = painter.layout_no_wrap(text.into(), kit::ui_medium(11.5), kit::TEXT); + let size = galley.size() + egui::vec2(16.0, 10.0); + let mut rect = egui::Rect::from_center_size(at, size); + let clip = painter.clip_rect().shrink(4.0); + rect = rect.translate(egui::vec2( + (clip.left() - rect.left()).max(0.0) - (rect.right() - clip.right()).max(0.0), + (clip.top() - rect.top()).max(0.0) - (rect.bottom() - clip.bottom()).max(0.0), + )); + painter.rect( + rect, + 6, + Color32::from_black_alpha(204), + Stroke::new(1.0, Color32::from_white_alpha(51)), + egui::StrokeKind::Inside, + ); + painter.galley(rect.min + egui::vec2(8.0, 5.0), galley, kit::TEXT); +} + +/// A curved arrow from the edge of `from` to the edge of `to`. +fn arrow(painter: &egui::Painter, from: egui::Rect, to: egui::Rect) { + let (start, end) = (from.center(), to.center()); + let along = end - start; + let length = along.length(); + let direction = along / length; + // Out of each rect, so the arrow runs between them. + let start = start + direction * (from.width().min(from.height()) / 2.0 + 4.0); + let end = end - direction * (to.width().min(to.height()) / 2.0 + 6.0); + let normal = egui::vec2(-direction.y, direction.x); + let control = start + (end - start) / 2.0 + normal * (length * 0.25); + let points: Vec = (0..=24) + .map(|step| { + let t = step as f32 / 24.0; + let (a, b) = (start + (control - start) * t, control + (end - control) * t); + a + (b - a) * t + }) + .collect(); + let stroke = Stroke::new(2.5, Color32::WHITE); + let shadow = Stroke::new(4.5, Color32::from_black_alpha(90)); + painter.add(egui::Shape::line(points.clone(), shadow)); + painter.add(egui::Shape::line(points.clone(), stroke)); + let tip = points[points.len() - 1]; + let back = (points[points.len() - 3] - tip).normalized() * 12.0; + let side = egui::vec2(-back.y, back.x) * 0.5; + painter.add(egui::Shape::line( + vec![tip + back + side, tip, tip + back - side], + stroke, + )); +} + +/// The outline of `shape` around `rect` on screen, as points round it. +fn outline(rect: egui::Rect, shape: &Shape) -> Vec { + let side = rect.width().min(rect.height()); + let radius = |corner: f32| (corner.clamp(0.0, 0.5) * side).max(0.0); + // Top left, top right, bottom right, bottom left, each with the angle its + // arc starts at. + let corners = [ + (rect.left_top(), egui::vec2(1.0, 1.0), std::f32::consts::PI), + (rect.right_top(), egui::vec2(-1.0, 1.0), 1.5 * std::f32::consts::PI), + (rect.right_bottom(), egui::vec2(-1.0, -1.0), 0.0), + (rect.left_bottom(), egui::vec2(1.0, -1.0), 0.5 * std::f32::consts::PI), + ]; + let mut points = Vec::new(); + for ((corner, inward, start), round) in corners.into_iter().zip(shape.corners) { + let radius = radius(round); + let center = corner + inward * radius; + for step in 0..=8 { + let angle = start + step as f32 / 8.0 * std::f32::consts::FRAC_PI_2; + points.push(center + egui::vec2(angle.cos(), angle.sin()) * radius); + } + } + points.push(points[0]); + points +} + +/// The legend under the stage while a layer keeping clear is selected: what +/// the overlay's marks mean, and that the decision stays on this machine. +pub(super) fn legend(ui: &Ui, below: egui::Rect) { + let painter = ui.painter(); + let y = below.bottom() + 18.0; + let font = kit::ui(12.0); + let mut x = below.left() + 4.0; + let mark = |painter: &egui::Painter, x: f32, kind: u8| { + let center = egui::pos2(x + 7.0, y); + match kind { + 0 => { + painter.circle_filled(center, 7.0, HEAT.gamma_multiply(0.25)); + painter.circle_filled(center, 4.0, HEAT); + } + 1 => { + let ring: Vec = (0..=24) + .map(|step| { + let angle = step as f32 / 24.0 * std::f32::consts::TAU; + center + egui::vec2(angle.cos(), angle.sin()) * 6.5 + }) + .collect(); + painter.extend(egui::Shape::dashed_line( + &ring, + Stroke::new(1.5, Color32::from_white_alpha(150)), + 2.5, + 2.0, + )); + } + _ => { + let square = egui::Rect::from_center_size(center, egui::Vec2::splat(13.0)); + painter.rect_stroke(square, 3, Stroke::new(2.0, kit::ACCENT), egui::StrokeKind::Inside); + } + } + }; + let right = below.right() - 4.0; + let laid = |labels: [&str; 3]| { + let galleys = labels.map(|text| painter.layout_no_wrap(text.into(), font.clone(), kit::DIM)); + let width: f32 = galleys.iter().map(|galley| 21.0 + galley.size().x + 18.0).sum(); + (galleys, width - 18.0) + }; + // Narrow, as BUILD is in a small window, the labels shorten. + let (galleys, width) = laid(["What matters on screen", "Places it may go", "Where it went"]); + let galleys = match x + width <= right { + true => galleys, + false => laid(["What matters", "May go", "Went"]).0, + }; + for (kind, galley) in galleys.into_iter().enumerate() { + mark(painter, x, kind as u8); + let width = galley.size().x; + painter.galley(egui::pos2(x + 21.0, y - galley.size().y / 2.0), galley, kit::DIM); + x += 21.0 + width + 18.0; + } + // The note goes where it fits beside them. + let galley = painter.layout_no_wrap("Decided on this machine".into(), font, kit::DIM); + let at = egui::pos2(right - galley.size().x, y - galley.size().y / 2.0); + if at.x - 20.0 < x { + return; + } + let icon = egui::Rect::from_center_size(egui::pos2(at.x - 12.0, y), egui::Vec2::splat(14.0)); + painter.galley(at, galley, kit::DIM); + Icon::Panic.paint(ui, icon, 13.0, kit::DIM); +} diff --git a/rs/moq-studio/src/ui/slots.rs b/rs/moq-studio/src/ui/slots.rs index e1aee32..bb52456 100644 --- a/rs/moq-studio/src/ui/slots.rs +++ b/rs/moq-studio/src/ui/slots.rs @@ -12,18 +12,25 @@ use super::*; /// closed. pub(super) fn placed(state: &State, layer: &Layer) -> Vec { let items = state.collection().items(layer).unwrap_or_default(); + // A layer set free that keeps clear, where it was placed. + let free = |index, item| super::keep::placed(state, layer, index, item); let arranged = layer.scene.as_deref().and_then(|scene| state.compositor.placed(scene)); // One frame behind an edit, until the next render arranges it. let Some(arranged) = arranged.filter(|arranged| arranged.len() == items.len()) else { - return items.to_vec(); + return items + .iter() + .enumerate() + .map(|(index, item)| free(index, item)) + .collect(); }; - let placed = items.iter().zip(arranged).map(|(item, arranged)| match item.slot { + let placed = items.iter().zip(arranged).enumerate(); + let placed = placed.map(|(index, (item, arranged))| match item.slot { Some(_) => Item { rect: arranged.rect, visible: item.visible && arranged.visible, ..item.clone() }, - None => item.clone(), + None => free(index, item), }); placed.collect() } @@ -164,9 +171,11 @@ pub(super) fn assign(state: &mut State, dragged: &Picked, target: &Picked) -> an }; anyhow::ensure!(held.slot.is_some(), "`{}` is in no slot", held.source); let label = format!("Put `{}` in the slot of `{}`", moved.source, held.source); + // In a slot, the arrangement places it, so it no longer keeps clear. let taking = Item { slot: held.slot, rect: Rect::default(), + keep_clear: None, ..moved.clone() }; // Set free in place of `moved`, cropped to its shape rather than @@ -239,6 +248,8 @@ pub(super) fn set_slot(state: &mut State, picked: &Picked, slot: Option) - slot, rect, crop, + // In a slot, the arrangement places it. + keep_clear: item.keep_clear.filter(|_| slot.is_none()), ..item }, )