Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
ded721f
chore(quest): claim quest/m1/auto-place.md
kixelated Nov 17, 2026
850cb08
feat: a layer that keeps out of the way
kixelated Nov 17, 2026
92e4375
feat: refuse keeping clear where it cannot, document it, and leave th…
kixelated Nov 17, 2026
4790fbd
chore: views for the PR body (deleted next)
kixelated Nov 17, 2026
4928053
chore: drop the PR body's views
kixelated Nov 17, 2026
042a548
Merge remote-tracking branch 'origin/main' into quest/m1/auto-place
kixelated Nov 17, 2026
cb95bde
chore(quest): one readback for the compositor's small GPU reads
kixelated Nov 17, 2026
0867795
fix: a move by hand says so, and a layer keeping clear follows its item
kixelated Nov 17, 2026
91cb8e2
fix: keepers tell alike layers apart, and a reloaded move holds
kixelated Nov 17, 2026
07eabcd
Merge remote-tracking branch 'origin/main' into quest/m1/auto-place
kixelated Nov 17, 2026
61cce15
fix: only a move by hand, an undo, or a reload holds a layer keeping …
kixelated Nov 17, 2026
f20002d
Merge origin/main (auto-layout, #191) into quest/m1/auto-place
kixelated Nov 17, 2026
c5157a7
refactor: a layer keeping clear glides with glide::Glide
kixelated Nov 17, 2026
7a9ab71
fix: a hidden layer keeping clear waits, a resize mid-glide lands, an…
kixelated Nov 17, 2026
5c8730e
fix: a layer held by hand stays held when shown again
kixelated Nov 17, 2026
db67621
chore: the compact view for the PR body (deleted next)
kixelated Nov 17, 2026
a76c6aa
chore: drop the PR body's compact view
kixelated Nov 17, 2026
6ab2c58
Merge remote-tracking branch 'origin/main' into quest/m1/auto-place
kixelated Nov 17, 2026
795e450
fix: a layer keeping clear stays put while a drag has it in hand
kixelated Nov 17, 2026
60b4dd5
fix: a drag's hold on a layer keeping clear lapses if not renewed
kixelated Nov 17, 2026
442404b
fix: a drag renews its hold four times a lease, at any canvas rate
kixelated Nov 17, 2026
5316d38
Merge remote-tracking branch 'origin/main' into quest/m1/auto-place
kixelated Nov 17, 2026
cb39347
fix: switching keep-clear off leaves the layer where it is drawn
kixelated Nov 17, 2026
3c0b485
fix: a glide under way pauses while a gesture has the layer in hand
kixelated Nov 17, 2026
07704f8
Merge remote-tracking branch 'origin/main' into quest/m1/auto-place
kixelated Nov 17, 2026
6a538ce
feat: an undo of a move by hand puts back where a layer keeping clear…
kixelated Nov 17, 2026
67d6446
feat: an undo of switching keep-clear off puts the layer back where i…
kixelated Nov 17, 2026
30c0f42
fix: an undo puts a layer keeping clear back whenever it renders
kixelated Nov 17, 2026
94ceaeb
fix: nudges between renders add up, and a drag keeps the picture still
kixelated Nov 17, 2026
da40b65
Merge remote-tracking branch 'origin/main' into quest/m1/auto-place
kixelated Nov 17, 2026
827f503
quest: readback no longer waits on capture-health, which landed
kixelated Nov 17, 2026
f5bedbc
Merge remote-tracking branch 'origin/main' into quest/m1/auto-place
kixelated Nov 17, 2026
ad6568c
fix: a glide taken in hand stops where it was last drawn, however late
kixelated Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 43 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ is named for it, with these sections:

| Section | Use it to |
| --- | --- |
| Place | Pick its [slot](#slots-and-roles), or none; type or drag its position and size, and its crop, in percent off each edge, fit or fill it, and lock it. A layer in a slot has no position or size of its own to type. |
| Place | Pick its [slot](#slots-and-roles), or none; type or drag its position and size, and its crop, in percent off each edge, fit or fill it, lock it, and switch on **Keep out of the way** (see [Keeping out of the way](#keeping-out-of-the-way)). A layer in a slot has no position or size of its own to type, and keeps out of the way only when set free. |
| Shape | Pick its outline, square, rounded, circle, or arch, give it a border inside that outline and the border's color, and give it a soft shadow, which follows the outline and is the layer's first drop shadow filter. |
| Look | Remove its source's background, which every layer showing the source loses (make it independent to cut out here alone); grade its color through a LUT source; add, order, set, and remove its filters, copy them, or paste copied filters after its own; and set its blend mode. |
| Shown in | See how many places show its source, **Linked in 3 scenes**, and **Make independent**, which points this layer at a copy of the source of its own, keeping its look, so editing that look no longer edits the others. A copy says what it copies. |
Expand Down Expand Up @@ -718,6 +718,40 @@ across restarts: after that, where they are shown is yours, so undoing their
appearance or removing their layer keeps them off when they reconnect, until
a new link. Since an appearance is your edit, it drops what you could redo.
Off, every guest waits under Add, as before.
### Keeping out of the way

A scene's layer with **Keep out of the way** on, in the inspector's Place
section, moves itself off what matters in the picture under it, such as a
facecam off a game's minimap or a slide's text. A few times a second the
studio draws the layers under it, scaled far down, and reduces them on the
GPU to a grid: where the picture moves, where it is dense with edges that hold
still, as text is, and what holds still while the rest moves, as a HUD does.
Each can be switched off under **Stays clear of**. Nothing is uploaded: the
choice is made on this machine, from the picture alone.

It keeps to one of eight places, the canvas's corners and edge middles,
inside a margin of a twentieth of the canvas's shorter side, that its size
fits in, taking the one that covers the least of what matters, a corner over
a middle that covers as much. It chooses a second after its scene goes on,
about two seconds after the content under it changes kind (another layout, or
another app), and otherwise moves only once its place has been the worst of them for
10 seconds, never for a single busy moment. It glides there as a move
transition eases, in 0.4 s unless **Glide** says otherwise. Its rect in the
scene file gives its size, and where it rests until it first moves.

While it is selected, BUILD shows what it saw: what matters glowing, tagged
**MOVING**, **TEXT**, or **HUD**, the places it may go, dashed, and where it
last moved from, with what it left there and an arrow to where it went. Drag
it, nudge it, or type its X or Y, and it stays where it was put until the
content changes, as it does where a redo or a reload of the scene file
moves it, or an undo moves it anywhere but back before such a move. Undo
that move and it goes back to where it was, such as the corner it had
glided to. A resize keeps it free to move. Switched off, it stays where it
is drawn, and undoing that puts it back keeping clear where it was. Only a
scene's layer set free of its [slots](#slots-and-roles) keeps out of the
way, and not one showing a source
[published independently](#published-independently), which viewers place
themselves; putting one in a slot switches it off.

### Audio scenes

Expand Down Expand Up @@ -1230,7 +1264,12 @@ Each entry of a scene's `items` is one of its layers, back to front, and each
canvas, in pixels, or, in a scene, its `slot`, `main` (one per scene) or
`tile`, in place of a `rect`, and optionally `visible = false`, `locked = true`, which
the picture's pointer passes through, and its `crop`, `filters`, `blend`,
`shape`, and `keyframes`.
`shape`, `keyframes`, and `keep_clear`.

A scene's layer with `keep_clear = {}` [keeps out of the
way](#keeping-out-of-the-way) of motion, text, and a HUD; `motion = false`,
`text = false`, or `hud = false` in it ignores one, and `glide` sets how long
a move takes, `"400ms"` unless it says otherwise, up to `"5s"`.

A layer's `shape` cuts it to an outline, before its filters, so a drop shadow
follows it: `corners`, how round each corner is (top-left, top-right,
Expand Down Expand Up @@ -1438,7 +1477,8 @@ published, holding an encode session as one on program does.

Every player must reproduce how it is placed, so a layer showing it takes no
filter but opacity, no blend mode, and no shape, nor do its source's filters or
its keyframes, and no scene that another scene draws can place it. A source with
its keyframes, it does not keep out of the way, and no scene that another scene
draws can place it. A source with
its background removed cannot be published, since H.264 carries no
transparency and its track would show the background.

Expand Down
28 changes: 28 additions & 0 deletions quest/m1/auto-place-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# [XS] Live with a camera that keeps out of the way

## Goal

The maintainer has streamed a few times with a facecam set to **Keep out of
the way** over real games and slides, and it moves when it should and no
more often. If it does not, the timings or weights it chooses by change until
it does.

## Plan

The analysis, the choice, and the glide are tested in moq-scene on synthetic
reads and on the GPU, and BUILD's `build_keep_clear` view shows what it saw.
How often it moves on real content is a judgment no test makes. The knobs are
constants in `rs/moq-scene/src/place.rs`: `SETTLE` (1 s before a choice),
`WORST` (10 s in the worst place before it moves anyway), `BETTER` (how much
clearer a place must be), `APART` and `CHANGING` (what counts as the content
changing kind), `WEIGHT` (motion 0.5, text 0.8, HUD 1.0), and `MIDDLE` (a
corner's lead over an edge middle).

## Waiting

- Maintainer: put a camera over a game and over a slide deck with **Keep out
of the way** on (inspector, Place), stream or record a few sessions, and
note each move that was wrong, late, or too frequent. An agent cannot judge
how a move reads to a viewer or which part of a real game matters; the
tests already check the frames and the rules. Done when it moves as
intended, or the change it needs is written into this Plan.
70 changes: 30 additions & 40 deletions quest/m1/auto-place.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,43 @@
# [L] A camera that keeps out of the way
# [S] A camera that keeps out of the way remembers each app

## Goal

A layer with **Keep out of the way** on moves itself so it does not cover what
matters in the picture under it: a game's minimap, a slide's text, the window
the streamer is working in. It chooses among a few slots along the edges,
moves only when the content changes, and glides there. The `autoplace` frame
of [`mock/later.html`](/quest/m0/ux/mock/later.html) shows the slots, what it
saw, and the move.
A layer with **Keep out of the way** on knows the app in front: it chooses
again the moment another app comes to the front, rather than waiting to see
the picture change, and the place it chose for an app is remembered per user,
so a known game is right from its first frame.

The decision is made on the machine. Nothing is uploaded.
The decision stays on the machine. Nothing is uploaded.

## Plan

The rest of the quest landed: moq-scene's `place` module reads the picture
under the layer on the GPU a few times a second, weighs motion, still edges
(text), and still edges while the rest moves (a HUD), and keeps the layer in
the corner or edge middle that covers the least, gliding there; BUILD shows
what it saw, and the inspector its settings (`keep_clear` in the scene file).
What remains needs moq to report the app in front.

- **When it moves:** a notice that the app in front changed starts a new
choice at once, as the content changing kind does now
(`Placer::see` setting `looking`), for the display capture under the
layer.
- **Memory:** the slot chosen for each app, keyed by the app moq reports,
kept per user with the hotkeys and snap settings, not in the scene file.
A remembered slot is where the layer starts when that app comes to the
front, before it has looked; a move by hand or a new choice updates it.

Decided 2026-11-09 (maintainer, /quest-plan): ✅ local only (rejected: a
hosted decision model asked on each content change, cheap but it uploads
thumbnails of the streamer's screen and its image input is not production
ready; local first with a hosted model as an opt-in tie-breaker).

- **What it reads:** the composited picture under the layer, scaled far down
on the GPU a few times a second: where it moves, where it is dense with
edges and text, and what stays put while the rest moves, which is a HUD.
The inspector says which of these it stays clear of, and each can be
switched off. It reads neither the pointer nor the focused window, since
moq reports neither a pointer nor a window's place on its display.
- **What it decides:** the slot whose area matters least, among the corners
and edge middles that fit the layer, inside the canvas's margins.
- **When it moves:** when the scene changes, when the content under it
changes kind (another layout, or another app in front where
[moq reports it](/quest/m1/pin-capture-foreground.md)), or when its slot has been the
worst for a sustained time. Never on a single busy frame, and never while
the streamer is dragging it. A manual move wins and holds until the content
changes.
- **Memory:** the slot chosen for an app is remembered per user, so a known
game is right from its first frame.
- **Where it lives:** the analysis and the choice are UI-free, in moq-scene,
tested on recorded frames with a paused clock. The move is the item's
rect animated as a Move transition eases: moq-scene's `Glide`
(`rs/moq-scene/src/glide.rs`), which glides a scene's slots when they
reflow, eases any list of items from one placement to the next on the
program clock, keyed by what changed.

Measure, per AGENTS.md: the frame time and GPU copies the analysis adds, and
that the encoder's load does not move. Should the signals prove too weak, a
small on-device model is a later quest, not a hosted one.

Human check: living with it for a few streams. Whether it moves too often is
a judgment no test makes.
ready; local first with a hosted model as an opt-in tie-breaker). Should the
signals prove too weak, a small on-device model is a later quest, not a
hosted one.

## Related
## Required

- [moq-video's foreground app is pinned](/quest/m1/pin-capture-foreground.md) - when the app in front changes, and the key its slot is remembered under

## Related

- [Living with it](/quest/m1/auto-place-check.md) - whether it moves too often, judged on real streams
50 changes: 50 additions & 0 deletions quest/m1/item-identity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# [M] Each layer has an identity the studio keeps while it runs

## Goal

Each item of a scene and of the top layer carries a runtime identity: a new
item gets a new one, an edit that changes an item keeps its own, undo and
redo bring back the one it had, and the scene file never saves it. What the
compositor keeps about a layer between frames is matched to its item by
that identity, not by where it sits, so adding, removing, reordering, or
duplicating layers never hands one layer's state to another.

## Plan

Keep out of the way (`rs/moq-scene/src/place.rs`) keeps a placer per layer:
what it saw, the slot it chose, and whether a move by hand holds it. With no
identity on an item, `compositor::follow` matches placers to items by
source, then saved rect, then index, after each edit. Two layers of the same
source at the same rect, as Duplicate makes them, can still trade placers
when a layer is added under both. Auto-layout's glide pairs items by source
for the same reason.

- **The identity:** a counter-made id on `Item`, outside serde, that
`Item::new`, loading, and every edit that adds an item as a new one
assign: BUILD's Duplicate (`LayerAction::Duplicate`), paste, a drop on the
picture (`ui/drop.rs`), an Add, and a guest joining. `set_item` and the
history keep the id of the item they change, so an undo restores it.
- **Equality:** two collections that differ only in ids are equal, so
autosave, reload, and every change check behave as now. A reload assigns
fresh ids, then matches them to the items before by the same rule
`follow` uses, so a reload of an unchanged file keeps every placer.
- **Users:** the compositor's keepers key by id; `follow` keeps only the
reload match. The slots' glide (`rs/moq-scene/src/glide.rs`, which pairs
items by source) pairs by id where both sides have one.
- **Undo across layers:** a placer whose layer leaves its scene, as Keep
always on top does, stays parked under its id
while the history can bring the layer back, so the undo puts it back where
it was, as an undo of switching keep-clear off does now (#194's decision
11). Today such an undo starts a fresh placer at the saved rect.
- **Tests:** duplicates of one camera keep their own placers across an
insert under both, a reorder, an undo of a removal, an undo of Keep always
on top, and a reload of an unchanged file.

Decided 2026-11-17 (maintainer, on
[#194](https://github.com/moq-dev/studio/pull/194#issuecomment-6944247792)):
✅ a runtime identity, as its own quest (rejected: in #194, more matching
heuristics in #194).

## Related

- [A camera that keeps out of the way remembers each app](/quest/m1/auto-place.md) - another user of the placers this keys
32 changes: 32 additions & 0 deletions quest/m1/readback.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# [S] One readback for the compositor's small GPU reads

## Goal

The compositor reads a few small pictures back from the GPU, a few times a
second each, through one shared helper: the dark-capture probe
(`rs/moq-scene/src/dark.rs`) and the keep-clear look
(`rs/moq-scene/src/place.rs`) each keep only their shader and what they
make of the tiles, and neither stalls a frame.

## Plan

The two landed separately with the same pattern: a compute pass reducing a
picture to a grid of tiles (64x36 for the probe, 32x18 for the look), at
most every 250 ms, its result copied to a buffer
mapped with `map_async` and read on a later frame, a failed or late read
dropped.

- **Shared:** the rate cap (`due`), the storage and readback buffers sized
to the largest grid, the copy after the pass, the map in flight and its
poll, and discarding a read whose source or feed changed meanwhile.
- **Kept apart:** each pipeline and its bindings. `dark.wgsl` reduces the
capture texture alone to mean color and brightest channel; `place.wgsl`
compares the items drawn under a layer with the last read, so it keeps
two pictures that ping-pong.
- Their tests (`dark::tests::`, `compositor::tests::a_camera_keeping_clear_glides_off_what_moves`,
`a_moment_of_dark_draws_the_last_good_picture`) pass unchanged under
`just render-sw`, and the frame time on the AMD iGPU does not grow.

Decided 2026-11-17 (maintainer, on
[#194](https://github.com/moq-dev/studio/pull/194#issuecomment-6941200435)):
✅ one helper once both land (rejected: sharing in #194, before #192 lands).
Loading