diff --git a/README.md b/README.md index 1494ffcd..698f1a80 100644 --- a/README.md +++ b/README.md @@ -1232,12 +1232,16 @@ plus 1 for each captured source it shows that program does not. When the program encodes on hardware, so do the sources, and a session the GPU refuses fails that source's track with the encoder's reason on its layer rather than falling back to software. The source keeps -drawing in the studio's own scenes. The program opens its session before any -source, even with on demand, so the sources past the cap fail rather than -the program; so does each scene published in the scene file at startup. A -scene published later, past the cap, publishes without video and says why in -Settings' Show tab. When the program encodes in software, a source takes hardware -where it can and software otherwise. +drawing in the studio's own scenes. The program comes first: when the GPU +refuses the program or a published scene an encoder, such as going live while +the sources on program hold every session, the studio closes the encoders those +sources keep only for a replay of their recent past and opens the program's in +their sessions. Each such source reopens once a session is given back, such as +when the stream ends, its replay starting over; one whose track something reads, +such as a recording, keeps its encoder. A scene published past the cap with no +replay left to close publishes without video and says why in Settings' Show +tab. When the program encodes in software, a source takes hardware where it can +and software otherwise. ### Published independently diff --git a/quest/m0/program-encoder-priority.md b/quest/m0/program-encoder-priority.md deleted file mode 100644 index af6f51d7..00000000 --- a/quest/m0/program-encoder-priority.md +++ /dev/null @@ -1,29 +0,0 @@ -# [S] The program's encoder before the sources' - -## Goal - -Going live gets the program its hardware encode sessions even when the -sources on program took every one the GPU allows while off air. Today the -program opens its encoders at GO LIVE, after the sources' replay encoders -(`rs/moq-scene/src/publish.rs`, which encode a source shown on program so a -replay has its recent past) already hold theirs, so on a GPU with a session -cap, such as a GeForce card's, a show with many sources on program goes live -on the CPU encoder or not at all. Before going live from the window, the -program opened first, at launch. - -## Plan - -Raised by the review of -[moq-dev/studio#175](https://github.com/moq-dev/studio/pull/175). The go-live -sheet's Encoder check already opens the same encoder with the sources' -sessions held, so the failure shows before the confirm; this makes it not -happen. `Sessions` counts encoders but cannot know a cap, which a driver -reports only by refusing an open. - -Decided by the maintainer: when the program's encoder cannot open, release -the replay encoders of the sources on program, open the program's, then let -each source reopen as sessions allow, its replay starting over (rejected: -opening the program's encoders when the show opens and holding them idle off -air, at one or two sessions and their memory for the whole session; leaving -it to the go-live sheet's Encoder check alone). A replay is a convenience and -the program is what viewers see; the cost falls only on a machine at its cap. diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index a2482534..c7610ac4 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -134,6 +134,16 @@ pub enum Error { #[error(transparent)] Net(#[from] moq_net::Error), + /// Every encode session [`Sessions::capped`](publish::Sessions::capped) + /// allows is open. + #[error("all {0} encode sessions are open")] + Sessions(usize), + + /// A replay encoder may not open while the program reclaims their + /// sessions. + #[error("the program is reclaiming the replay encoders' sessions")] + Reclaimed, + /// A browser source's page failed, or its helper process stopped. #[error("browser: {0}")] Browser(String), diff --git a/rs/moq-scene/src/publish.rs b/rs/moq-scene/src/publish.rs index 59df52f3..622f27e1 100644 --- a/rs/moq-scene/src/publish.rs +++ b/rs/moq-scene/src/publish.rs @@ -12,7 +12,6 @@ //! something in the studio subscribes to it, such as a recording. use std::collections::{BTreeMap, BTreeSet}; -use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::{Arc, Mutex}; use serde::{Deserialize, Serialize}; @@ -64,29 +63,152 @@ impl Section { /// The catalog of the program's broadcast, or of the sources' local one. pub type Catalog = moq_mux::catalog::Producer; -/// How many video encoders are open. Each holds a session on a hardware -/// encoder, which a GPU caps, such as at 8 on a GeForce card. -#[derive(Clone, Debug, Default)] -pub struct Sessions(Arc); +/// The video encoders open, each holding a session on a hardware encoder, +/// which a GPU caps, such as at 8 on a GeForce card, refusing an open past it. +/// +/// The program's encoders come first: one the GPU refuses +/// [reclaims](Self::reclaim) the sessions of the sources' replay encoders, +/// those open only so a replay of a source on program has its recent past. +#[derive(Clone, Debug)] +pub struct Sessions(Arc>); + +#[derive(Clone, Copy, Debug, Default)] +struct State { + open: usize, + /// Those of `open` held by a replay encoder, which the program may take. + spare: usize, + /// How many encoders have closed, so one refused waits for any given back + /// after it tried, however many are open by then. + closed: u64, + /// The program's reclaims under way: while any is, a replay encoder + /// closes and none opens. + reclaims: usize, + /// Refuse an open past this many, as a capped GPU does. + cap: Option, +} + +impl Default for Sessions { + fn default() -> Self { + Self(Arc::new(tokio::sync::watch::Sender::new(State::default()))) + } +} impl Sessions { + /// Sessions that refuse an encoder past `cap` open at once, as a GPU with + /// that cap does, to stand in for one on a machine without it. + pub fn capped(cap: usize) -> Self { + let state = State { + cap: Some(cap), + ..Default::default() + }; + Self(Arc::new(tokio::sync::watch::Sender::new(state))) + } + + /// Whether these stand in for a capped GPU, made by [`capped`](Self::capped). + pub fn is_capped(&self) -> bool { + self.0.borrow().cap.is_some() + } + /// How many encoders are open now. pub fn open(&self) -> usize { - self.0.load(Ordering::Relaxed) + self.0.borrow().open + } + + /// How many of those are the sources' replay encoders, which a + /// [reclaim](Self::reclaim) would close. + pub fn spare(&self) -> usize { + self.0.borrow().spare + } + + /// Close every replay encoder and open none until the returned guard + /// drops, so the program's encoder can open in their sessions. A replay + /// starts over once its encoder reopens, which waits for a session given + /// back if the GPU refuses it. + pub fn reclaim(&self) -> Reclaim { + self.0.send_modify(|state| state.reclaims += 1); + Reclaim(self.clone()) + } + + /// Count a session before its encoder asks the GPU for one, so the program + /// sees a replay's session, `spare`, while it is still opening. A replay's + /// is refused while a reclaim is under way. + fn hold(&self, spare: bool) -> Result { + let mut refused = None; + self.0.send_if_modified(|state| match state.cap { + _ if spare && state.reclaims > 0 => { + refused = Some(Error::Reclaimed); + false + } + Some(cap) if state.open >= cap => { + refused = Some(Error::Sessions(cap)); + false + } + _ => { + state.open += 1; + state.spare += usize::from(spare); + true + } + }); + match refused { + Some(err) => Err(err), + None => Ok(Session { + sessions: self.clone(), + spare, + opened: false, + }), + } } +} + +/// The program's claim on the replay encoders' sessions, from +/// [`Sessions::reclaim`] until dropped. +pub struct Reclaim(Sessions); - fn hold(&self) -> Session { - self.0.fetch_add(1, Ordering::Relaxed); - Session(self.clone()) +impl Reclaim { + /// Wait until every replay encoder has closed. + pub async fn closed(&self) { + let mut state = (self.0).0.subscribe(); + // Fails only once the sender drops, which this holds. + let _ = state.wait_for(|state| state.spare == 0).await; + } +} + +impl Drop for Reclaim { + fn drop(&mut self) { + (self.0).0.send_modify(|state| state.reclaims -= 1); } } /// One open encoder's place in its [`Sessions`], given back when dropped. -struct Session(Sessions); +pub struct Session { + sessions: Sessions, + /// Held by a replay encoder. + spare: bool, + /// An encoder opened in it, rather than failing to: only then does it + /// give a session back that one refused could take. + opened: bool, +} + +impl Session { + fn set_spare(&mut self, spare: bool) { + if self.spare != spare { + self.spare = spare; + self.sessions.0.send_modify(|state| match spare { + true => state.spare += 1, + false => state.spare -= 1, + }); + } + } +} impl Drop for Session { fn drop(&mut self) { - (self.0).0.fetch_sub(1, Ordering::Relaxed); + let Self { spare, opened, .. } = *self; + self.sessions.0.send_modify(|state| { + state.open -= 1; + state.spare -= usize::from(spare); + state.closed += u64::from(opened); + }); } } @@ -664,13 +786,36 @@ impl Encoding { self.producer.as_ref().expect("open until it ends").demand() } - /// Encode with `encoder` from here on, after draining the one it replaces. - pub fn open(&mut self, encoder: E) -> Result<(), Error> { + /// Drain the open encoder, then count a session for the next one before it + /// asks the GPU, `spare` when only a replay would use it, to + /// [open](Self::open) it in. Fails when [capped](Sessions::capped) + /// sessions are all open, or, for a spare one, while the program reclaims. + pub fn hold(&mut self, spare: bool) -> Result { + self.drain()?; + self.sessions.hold(spare) + } + + /// Encode with `encoder`, in the `session` [held](Self::hold) for it, from + /// here on. + pub fn open(&mut self, mut session: Session, encoder: E) -> Result<(), Error> { self.drain()?; - self.encoder = Some((encoder, self.sessions.hold())); + session.opened = true; + self.encoder = Some((encoder, session)); Ok(()) } + /// Whether the open encoder is a replay's, which the program may reclaim. + fn spare(&self) -> bool { + self.encoder.as_ref().is_some_and(|(_, session)| session.spare) + } + + /// Mark the open encoder as a replay's, or not. + fn set_spare(&mut self, spare: bool) { + if let Some((_, session)) = &mut self.encoder { + session.set_spare(spare); + } + } + /// Encode `frame` and publish what the encoder hands back. /// /// # Panics @@ -712,11 +857,18 @@ impl Encoding { /// Close the encoder, publishing what it still holds. pub fn drain(&mut self) -> Result<(), Error> { - if let (Some((mut encoder, _session)), Some(producer)) = (self.encoder.take(), &mut self.producer) { - encoder.flush()?; - publish_ready(&mut encoder, producer)?; - } - Ok(()) + let (Some((mut encoder, session)), Some(producer)) = (self.encoder.take(), &mut self.producer) else { + return Ok(()); + }; + let drained = encoder + .flush() + .map_err(Error::from) + .and_then(|()| publish_ready(&mut encoder, producer)); + // The GPU takes the session back before it counts as given back, so a + // replay that the count wakes to reopen finds it free. + drop(encoder); + drop(session); + drained } /// Mark a gap in the track after [draining](Self::drain) the encoder, which @@ -793,6 +945,14 @@ pub(crate) struct Video<'a> { /// Where the capture's private timeline starts on the studio clock, in /// microseconds, taken at the first frame unless [anchored](Self::anchor). epoch: Option, + /// The encoder closed for the program's [reclaim](Sessions::reclaim), + /// until it reopens. + reclaimed: bool, + /// How many encoders had closed before the GPU refused a reclaimed one + /// reopening, which waits until another closes. Not how many were open: + /// another replay's session the GPU already holds counts only once its + /// open returns, so the refused one could wait for fewer than ever are. + refused: Option, } impl<'a> Video<'a> { @@ -825,6 +985,8 @@ impl<'a> Video<'a> { config, clock: listing.studio.clock(), epoch: None, + reclaimed: false, + refused: None, }) } @@ -857,7 +1019,7 @@ impl<'a> Video<'a> { }; frame.timestamp = moq_net::Timestamp::from_micros((epoch + captured) as u64).map_err(moq_video::Error::from)?; - if !self.used() { + if !self.wanted() { return self.release(); } @@ -871,9 +1033,35 @@ impl<'a> Video<'a> { self.track.drain()?; } if !self.track.is_open() { - self.track.open(moq_video::encode::Encoder::new(&self.config)?)?; - self.granted = None; + let closed = self.track.sessions.0.borrow().closed; + let spare = !self.track.watched(); + let opened = self.track.hold(spare).and_then(|session| { + let encoder = moq_video::encode::Encoder::new(&self.config)?; + self.track.open(session, encoder) + }); + match opened { + Ok(()) => { + self.reclaimed = false; + self.refused = None; + self.granted = None; + } + // A reclaim began since it was wanted: it stays closed for the + // program, and reopens once the reclaim ends. + Err(Error::Reclaimed) => { + self.reclaimed = true; + return Ok(()); + } + // Its session went to the program, so it waits for one given + // back rather than failing. + Err(err) if self.reclaimed => { + tracing::warn!(source = %self.claim.name, %err, "waiting for an encode session"); + self.refused = Some(closed); + return Ok(()); + } + Err(err) => return Err(err), + } } + self.track.set_spare(!self.track.watched()); // Published independently, it encodes at what the connection grants. let grant = self .grants @@ -907,23 +1095,44 @@ impl<'a> Video<'a> { self.on_program.borrow().shown.contains(&self.claim.name) || self.track.watched() } - /// Wait until whether the source is used disagrees with whether its - /// encoder is open, so a source that delivers nothing new, such as a - /// still screen or a media file that ended, can act without waiting on a - /// picture: [`release`] the encoder when it is open and nothing uses the - /// source, or bring a picture when it is closed and the source goes into - /// use. + /// Whether the encoder is to be open: while the source is used, but not + /// for a replay alone while the program reclaims its session, nor while + /// waiting for a session given back since the GPU refused one. + fn wanted(&self) -> bool { + let watched = self.track.watched(); + if !watched && !self.on_program.borrow().shown.contains(&self.claim.name) { + return false; + } + let sessions = self.track.sessions.0.borrow(); + (watched || sessions.reclaims == 0) && self.refused.is_none_or(|closed| sessions.closed > closed) + } + + /// Whether the encoder is not as [`wanted`](Self::wanted): open or closed + /// when it should not be, or open and marked a replay's while read, or + /// the other way around, which a reclaim would misjudge. + fn unsettled(&self) -> bool { + let wanted = self.wanted(); + wanted != self.track.is_open() || (wanted && self.track.spare() == self.track.watched()) + } + + /// Wait until the encoder is not as wanted, so a source that delivers + /// nothing new, such as a still screen or a media file that ended, can + /// act without waiting on a picture: [`release`] the encoder when it is + /// open, or bring a picture when it is closed and wanted. /// /// [`release`]: Self::release pub(crate) async fn settle(&self) { let mut demand = self.track.demand(); let mut on_program = self.on_program.clone(); on_program.mark_unchanged(); - while self.used() == self.track.is_open() { + let mut sessions = self.track.sessions.0.subscribe(); + while !self.unsettled() { // The first look at the demand reports it as it is now. tokio::select! { changed = on_program.changed() => if changed.is_err() { std::future::pending::<()>().await }, wanted = demand.wanted_changed() => if wanted.is_err() { std::future::pending::<()>().await }, + // Never fails: the track holds the sender. + _ = sessions.changed() => {} } } } @@ -943,13 +1152,20 @@ impl<'a> Video<'a> { self.epoch = None; } - /// Close the encoder while nothing uses the source, marking the gap, which - /// gives the encode session back. + /// Close the encoder while it is not [wanted](Self::wanted), marking the + /// gap, which gives the encode session back, or else mark it a replay's + /// or not, as whether its track is read says. pub(crate) fn release(&mut self) -> Result<(), Error> { - if self.track.is_open() && !self.used() { - self.track.discontinuity()?; + if !self.track.is_open() { + return Ok(()); } - Ok(()) + if self.wanted() { + self.track.set_spare(!self.track.watched()); + return Ok(()); + } + // Still used, so it closed for the program. + self.reclaimed = self.used(); + self.track.discontinuity() } /// Start a group with the next picture, as a keyframe, such as where a @@ -1128,6 +1344,14 @@ mod tests { (publish, catalog, consumer) } + /// A session an encoder opened in, such as the program's, standing in for + /// one held on the GPU. + fn opened(sessions: &Sessions) -> Session { + let mut session = sessions.hold(false).unwrap(); + session.opened = true; + session + } + /// On program with only `name`'s picture shown. fn shown(name: &str) -> OnProgram { OnProgram { @@ -1600,6 +1824,75 @@ mod tests { .await; } + /// On a GPU at its session cap, the program reclaims a replay encoder's + /// session, even from a still screen that delivers no picture, and the + /// replay waits for a session given back rather than failing, then starts + /// over. One whose track is read keeps its encoder. + #[tokio::test(start_paused = true)] + async fn the_program_reclaims_a_replays_session() { + local(async { + let clock = moq_mux::Clock::new(); + let sessions = Sessions::capped(1); + let publish = Publish::new(clock, moq_video::encode::Kind::Auto, sessions.clone()).unwrap(); + let claim = publish.claim("display"); + let listing = Listing::new(&claim, &publish).unwrap(); + let framerate = moq_video::FrameRate::new(30, 1).unwrap(); + let size = moq_video::Size::new(64, 48); + let mut video = Video::new(&listing, size, framerate, None).await.unwrap(); + let wait = std::time::Duration::from_secs(1); + let settled = async |video: &Video<'_>| kio::clock::timeout(wait, video.settle()).await; + + publish.set_on_program(shown("display")); + video.encode(&mut gray(size, 0)).unwrap(); + assert_eq!((sessions.open(), sessions.spare()), (1, 1), "a replay's"); + assert!(sessions.hold(false).is_err(), "at the cap"); + + let reclaim = sessions.reclaim(); + settled(&video).await.expect("asked to close without a picture"); + video.release().unwrap(); + kio::clock::timeout(wait, reclaim.closed()) + .await + .expect("the replay closed"); + assert_eq!(sessions.open(), 0); + video.encode(&mut gray(size, 33)).unwrap(); + assert_eq!(sessions.open(), 0, "none opens while the program reclaims"); + + let program = opened(&sessions); + drop(reclaim); + settled(&video).await.expect("wanted again"); + video.encode(&mut gray(size, 66)).unwrap(); + assert!(!video.is_open(), "refused, so it waits"); + assert!(settled(&video).await.is_err(), "until a session is given back"); + // Given back and taken again before the replay looks, as by another + // replay reopening first: it still tries, and waits again. + drop(program); + let sibling = opened(&sessions); + settled(&video) + .await + .expect("a session given back, even if taken again"); + video.encode(&mut gray(size, 80)).unwrap(); + assert!(!video.is_open(), "refused again, so it waits"); + assert!(settled(&video).await.is_err(), "until another is given back"); + drop(sibling); + settled(&video).await.expect("a session given back"); + video.encode(&mut gray(size, 100)).unwrap(); + assert_eq!((sessions.open(), sessions.spare()), (1, 1), "the replay starts over"); + + // Read, such as by a recording, it is no replay's alone. + let snapshot = listing.catalog.snapshot(); + let name = snapshot.video.renditions.keys().next().unwrap(); + let _track = publish.consume().track(name).unwrap().subscribe(None).await.unwrap(); + let reclaim = sessions.reclaim(); + settled(&video).await.expect("marked read without a picture"); + video.release().unwrap(); + kio::clock::timeout(wait, reclaim.closed()) + .await + .expect("nothing to close"); + assert_eq!((sessions.open(), sessions.spare()), (1, 0), "kept"); + }) + .await; + } + /// A source off program encodes only while its track is subscribed, such /// as by a recording, holding no encoder between subscribers. #[tokio::test] @@ -1836,6 +2129,106 @@ mod tests { } } + /// A replay's session counts as spare from before its encoder asks the GPU, + /// so the program, on another thread, reclaims it and waits for it to + /// close, and none starts opening once a reclaim is under way. + #[tokio::test(start_paused = true)] + async fn a_replay_is_counted_before_it_opens() { + let sessions = Sessions::default(); + let opening = sessions.hold(true).unwrap(); + assert_eq!((sessions.open(), sessions.spare()), (1, 1), "a replay's, still opening"); + + let reclaim = sessions.reclaim(); + assert!(matches!(sessions.hold(true), Err(Error::Reclaimed)), "no replay starts"); + let program = sessions.hold(false).expect("the program's opens"); + let wait = std::time::Duration::from_millis(10); + assert!( + kio::clock::timeout(wait, reclaim.closed()).await.is_err(), + "the opening replay is waited for" + ); + // Refused by the GPU, say: it gives back nothing one refused could take, + // so a replay waiting for a session does not retry on its own failure. + drop(opening); + assert_eq!(sessions.0.borrow().closed, 0, "nothing given back"); + kio::clock::timeout(wait, reclaim.closed()).await.expect("then closed"); + drop(reclaim); + drop(sessions.hold(true).expect("a replay opens again")); + drop(program); + } + + /// Records how many sessions are counted open as it closes. + struct Closing { + encoder: moq_video::encode::Encoder, + sessions: Sessions, + open: Arc>>, + } + + impl Drop for Closing { + fn drop(&mut self) { + *self.open.lock().unwrap() = Some(self.sessions.open()); + } + } + + impl Encode for Closing { + fn encode(&mut self, frame: &moq_video::Frame) -> Result<(), moq_video::Error> { + self.encoder.encode(frame) + } + + fn receive(&mut self) -> Result, moq_video::Error> { + self.encoder.receive() + } + + fn flush(&mut self) -> Result<(), moq_video::Error> { + self.encoder.flush() + } + + fn cut(&mut self) -> Result<(), moq_video::Error> { + self.encoder.cut() + } + + fn set_bitrate(&mut self, bitrate: moq_net::bandwidth::Rate) -> Result<(), moq_video::Error> { + self.encoder.set_bitrate(bitrate) + } + } + + /// An encoder closes before its session counts as given back, so a replay + /// that the count wakes to reopen finds the GPU's session free. + #[tokio::test] + async fn an_encoder_closes_before_its_session_is_given_back() { + let (publish, _catalog, _consumer) = studio(); + let claim = publish.claim("program"); + let listing = Listing::new(&claim, &publish).unwrap(); + let size = moq_video::Size::new(64, 48); + let framerate = moq_video::FrameRate::new(30, 1).unwrap(); + let mut config = moq_video::encode::Config::new(size.width, size.height, framerate); + config.kind = moq_video::encode::Kind::Named("openh264".into()); + let rendition = config.probe().await.unwrap(); + let reserved = listing.catalog.reserve(); + let track = listing + .broadcast + .unique_track(".avc3", reserved.track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let producer = moq_video::encode::Producer::new(track, reserved, rendition).unwrap(); + let sessions = Sessions::default(); + let mut track = Encoding::new(producer, sessions.clone()); + let open = Arc::new(Mutex::new(None)); + let session = track.hold(false).unwrap(); + track + .open( + session, + Closing { + encoder: moq_video::encode::Encoder::new(&config).unwrap(), + sessions: sessions.clone(), + open: open.clone(), + }, + ) + .unwrap(); + track.encode(&gray(size, 0)).unwrap(); + track.drain().unwrap(); + assert_eq!(*open.lock().unwrap(), Some(1), "still counted as it closed"); + assert_eq!(sessions.open(), 0, "given back after"); + } + /// A pipelined encoder's last pictures land before a pause's gap rather /// than after it, the picture after the gap opens a group of its own, and /// the pictures it still holds as the track ends are published before the @@ -1866,13 +2259,17 @@ mod tests { let mut reader = moq_mux::container::Consumer::new(subscriber, container); let mut track = Encoding::new(producer, Sessions::default()); + let session = track.hold(false).unwrap(); let encoder = moq_video::encode::Encoder::new(&config).unwrap(); track - .open(Pipelined { - encoder, - held: Default::default(), - flushed: false, - }) + .open( + session, + Pipelined { + encoder, + held: Default::default(), + flushed: false, + }, + ) .unwrap(); for millis in [0, 33, 66] { track.encode(&gray(size, millis)).unwrap(); diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 834a4006..69c4df6b 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -2057,6 +2057,17 @@ impl Driver { /// own track gets no more pictures, so its encoder is released here once /// nothing uses the source, giving the encode session back. async fn ended(&self, cues: &mut watch::Receiver, own: &mut Own<'_, '_>) { + // The feed is gone, which stops this driver anyway. + if self.serve(own, cues.changed()).await.is_err() { + std::future::pending::<()>().await; + } + } + + /// Wait for `until`, meanwhile releasing the source's open encoder once + /// nothing wants it, such as when the program reclaims its session, rather + /// than at the next picture, which may be far off or never come. + async fn serve(&self, own: &mut Own<'_, '_>, until: impl Future) -> T { + let mut until = std::pin::pin!(until); loop { let settled = async { match &own.track { @@ -2065,13 +2076,7 @@ impl Driver { } }; tokio::select! { - changed = cues.changed() => { - // The feed is gone, which stops this driver anyway. - if changed.is_err() { - std::future::pending::<()>().await; - } - return; - } + done = &mut until => return done, () = settled => { if let Some(video) = &mut own.track && let Err(err) = video.release() @@ -2161,6 +2166,9 @@ impl Driver { let end = std::cell::Cell::new(begin); let last = |until: Instant| end.set(end.get().max(until)); + // Once every sound is handed over, which the pictures wait for after + // their last, still giving the encoder back if the program asks. + let sounded = Notify::new(); let pictures = async { let Some((consumer, config)) = &mut video else { return Ok(()); @@ -2168,17 +2176,18 @@ impl Driver { let mut previous = None; while let Some(mut frame) = picture.take() { let due = at(frame.timestamp)?; - ahead(due).await; + self.serve(own, ahead(due)).await; last(due.saturating_add(previous.map_or(Duration::ZERO, |previous| due - previous))); previous = Some(due); self.encode(own, &mut frame, config, due).await?; shows.queue.borrow_mut().push_back((due, play, frame)); shows.handed.notify_one(); - picture = next_picture(consumer).await?; + picture = self.serve(own, next_picture(consumer)).await?; } + self.serve(own, sounded.notified()).await; Ok::<_, Error>(()) }; - let sounds = async { + let handing = async { let (Some(consumer), Some(sound)) = (&mut audio, &mut sound) else { return Ok(()); }; @@ -2214,9 +2223,14 @@ impl Driver { } Ok::<_, Error>(()) }; + let sounds = async { + let handed = handing.await; + sounded.notify_one(); + handed + }; tokio::try_join!(pictures, sounds)?; let end = end.get(); - ahead(end).await; + self.serve(own, ahead(end)).await; Ok(end) } @@ -4056,6 +4070,39 @@ mod media_tests { .await; } + /// A playing file on program gives its replay encoder's session back as + /// soon as the program reclaims it, not at its next picture, which may be + /// far off, or never come while its sound plays on. + #[tokio::test(start_paused = true)] + async fn a_playing_media_file_gives_its_session_back_at_once() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (mut context, _origin) = context(dir.path()); + let sessions = crate::publish::Sessions::capped(1); + let publish = + crate::publish::Publish::new(moq_mux::Clock::new(), moq_video::encode::Kind::Auto, sessions.clone()) + .unwrap(); + publish.set_on_program(crate::publish::OnProgram { + shown: ["clip.mkv".to_string()].into(), + ..Default::default() + }); + context.publish = Some(publish); + let mut sources = Sources::new(context); + let collection = media(dir.path(), &[("clip.mkv", CLIP, false, false)]); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + until(|| (sessions.spare() == 1).then_some(())).await; + + // Its pictures come every 100 ms. + let reclaim = sessions.reclaim(); + kio::clock::timeout(Duration::from_millis(10), reclaim.closed()) + .await + .expect("given back before the next picture"); + assert_eq!(sessions.open(), 0); + }) + .await; + } + /// The file a stinger plays opens blank and silent, waiting for its first /// play, and each play reports when it began and ended, never a play it /// cut off. diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index be973fe6..37f4ddca 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -39,7 +39,7 @@ use anyhow::Context; use bytes::Bytes; use kio::clock::Instant; use moq_net::bandwidth; -use moq_scene::publish::{Encoding, Sessions}; +use moq_scene::publish::{Encoding, Reclaim, Sessions}; use moq_scene::source::{self, Driver, Sources, Status}; use moq_scene::{Canvas, Collection, Compositor, Draft, Edit, History, Scaler}; use moq_video::render::export::{Exporter, Pending, Route}; @@ -860,6 +860,8 @@ pub struct Program { epoch: Duration, /// Where the local sources keep their own tracks, on the same clock. sources: moq_scene::publish::Publish, + /// [`Renditions::reclaim`]. + reclaim: Option, /// The sources published independently, whose tracks the broadcast on air /// carries beside the program's. published: tokio::sync::watch::Sender>, @@ -1002,6 +1004,8 @@ pub struct Launch { pub recordings: PathBuf, /// The room guests join, if any. pub room: Option>, + /// Where the video encoders count the encode sessions they hold. + pub sessions: Sessions, } impl Engine { @@ -1013,7 +1017,12 @@ impl Engine { /// /// Call inside the tokio runtime that serves the relay connections. pub async fn start(launch: Launch, autosave: Autosave, loaded: autosave::Loaded) -> anyhow::Result> { - let Launch { gpu, recordings, room } = launch; + let Launch { + gpu, + recordings, + room, + sessions, + } = launch; let (bus, driver) = moq_audio::bus::Bus::new(Default::default())?; let subscribe = moq_tokio::origin::spawn(); // Every relay feeds `subscribe`, so a guest either one routes is seen. @@ -1033,7 +1042,10 @@ impl Engine { dropped, } = loaded; let canvas = collection.canvas; - let stats = Arc::new(Stats::default()); + let stats = Arc::new(Stats { + sessions, + ..Default::default() + }); // The program clock's start, from which the audio clock advances the // bus too, so the sources place their sound on it. @@ -1893,22 +1905,23 @@ async fn publish(gpu: &Gpu, stats: &Arc, canvas: Canvas, start: Instant) // that source loudly instead of falling to software. When the program could // not use hardware, such as NVENC beside an integrated GPU it cannot read, a // source may still reach it from the CPU. - let kind = match Probe::new(gpu, canvas, false).await { - Ok(probe) => match &probe.rungs[0].config.kind { - moq_video::encode::Kind::Named(name) if name == "openh264" => moq_video::encode::Kind::Auto, - _ => moq_video::encode::Kind::Hardware, - }, + let kind = match Probe::open(gpu, canvas, false).await { + Ok(probe) if !probe.hardware() => moq_video::encode::Kind::Auto, + Ok(_) => moq_video::encode::Kind::Hardware, Err(err) => { stats.video.fail(format!("{err:#}")); moq_video::encode::Kind::Hardware } }; + let capped = matches!(kind, moq_video::encode::Kind::Hardware) || stats.sessions.is_capped(); + let reclaim = capped.then(|| stats.sessions.clone()); let sources = moq_scene::publish::Publish::new(clock.clone(), kind, stats.sessions.clone())?; let recorded_video = Renditions { gpu: gpu.clone(), broadcast: recorder.clone(), timebase: recorded.timebase(), record: true, + reclaim: reclaim.clone(), bandwidth: moq_net::bandwidth::Allocator::unlimited(), on_demand: true, live: false, @@ -1925,6 +1938,7 @@ async fn publish(gpu: &Gpu, stats: &Arc, canvas: Canvas, start: Instant) clock, epoch, sources: sources.clone(), + reclaim, published, recording, recorded, @@ -1988,6 +2002,11 @@ struct Renditions { /// full quality whatever the grant, rather than the published full and /// half sizes. record: bool, + /// Where the program reclaims the sources' replay encoders' sessions when + /// the GPU refuses it the hardware encoder it found at launch, or `None` + /// when it found none and the sessions are not + /// [capped](Sessions::capped), so the sources' encoders hold none either. + reclaim: Option, /// The connection's bandwidth, which each published rendition follows its /// share of. bandwidth: bandwidth::Allocator, @@ -2006,10 +2025,10 @@ struct Renditions { impl Renditions { /// Probe for `canvas`'s renditions and [`start`](Self::start) them, waiting - /// for the full size's encoder to open, as at startup, so it holds its - /// encode session before any source opens one. + /// for the full size's encoder to open, so a canvas the GPU cannot encode + /// fails here, even after reclaiming the sources' replay encoders. async fn open(&self, canvas: Canvas) -> anyhow::Result { - let probe = Probe::new(&self.gpu, canvas, self.record).await?; + let probe = Probe::new(self, canvas).await?; let (ladder, opened) = self.start(probe)?; opened.await.context("the encode thread stopped")??; Ok(ladder) @@ -2041,6 +2060,7 @@ impl Renditions { .collect(); let (opened, open) = tokio::sync::oneshot::channel(); let mut opened = Some(opened); + let mut reclaim = probe.reclaim; let number = self.video.rendition(); let mut rungs = Vec::with_capacity(probe.rungs.len()); for probed in probe.rungs { @@ -2090,8 +2110,19 @@ impl Renditions { // One more than the exporter's pool, for a skip behind a full pool. let (frames, pending) = mpsc::sync_channel(Exporter::DEPTH + 1); + // Live, the canvas size holds its session from the start, so a + // source past the GPU's session cap is the one that fails. Without + // a relay nobody can watch it, so it opens nothing. + let opened = match role { + Role::Canvas if self.live => opened.take(), + _ => None, + }; let job = Job { sessions: self.stats.sessions.clone(), + reclaim_from: self.reclaim.clone(), + // Kept from the probe until the canvas size opens, so no source + // takes back a session meanwhile. + reclaim: opened.is_some().then(|| reclaim.take()).flatten(), video: self.video.clone(), role, rendition: (role == Role::Canvas).then_some(number), @@ -2102,13 +2133,6 @@ impl Renditions { // The canvas size holds its session while it skips frames. hold: role == Role::Canvas, }; - // Live, the canvas size holds its session from the start, so a - // source past the GPU's session cap is the one that fails. Without - // a relay nobody can watch it, so it opens nothing. - let opened = match role { - Role::Canvas if self.live => opened.take(), - _ => None, - }; let thread = std::thread::Builder::new() .name("encode".into()) .spawn(move || encode(job, opened, pending))?; @@ -2138,6 +2162,8 @@ struct Probe { rungs: Vec, /// Why the encoder refused a rendition the program does without. refused: Vec<(Role, String)>, + /// The sources' replay encoders' sessions, reclaimed for these encoders. + reclaim: Option, } /// One rendition of a [`Probe`]. @@ -2149,11 +2175,41 @@ struct Probed { } impl Probe { + /// [`open`](Self::open) `renditions` of `canvas`. When the GPU refuses + /// them the hardware encoder found at launch, as one at its session cap + /// does, the program comes first: this closes the sources' replay + /// encoders, which replay a source's recent past, and probes again in + /// their sessions, holding them until the full size's encoder opens. + async fn new(renditions: &Renditions, canvas: Canvas) -> anyhow::Result { + let probe = Self::open(&renditions.gpu, canvas, renditions.record).await; + let Some(sessions) = renditions.reclaim.as_ref().filter(|sessions| sessions.spare() > 0) else { + return probe; + }; + if probe.as_ref().is_ok_and(Self::hardware) { + return probe; + } + match &probe { + Ok(_) => tracing::warn!("closing the sources' replay encoders for the program's hardware encoder"), + Err(err) => tracing::warn!(%err, "closing the sources' replay encoders for the program's encoder"), + } + drop(probe); + let reclaim = sessions.reclaim(); + reclaim.closed().await; + let mut probe = Self::open(&renditions.gpu, canvas, renditions.record).await?; + probe.reclaim = Some(reclaim); + Ok(probe) + } + + /// Whether the canvas size took a hardware encoder. + fn hardware(&self) -> bool { + !matches!(&self.rungs[0].config.kind, moq_video::encode::Kind::Named(name) if name == "openh264") + } + /// Open a throwaway encoder of each of `canvas`'s renditions on what `gpu` /// exports, the published sizes or, if `record`, the recording's alone, /// which also settles whether the encoder reads the program on the GPU. /// The first picks the encoder, which the other uses too. - async fn new(gpu: &Gpu, canvas: Canvas, record: bool) -> anyhow::Result { + async fn open(gpu: &Gpu, canvas: Canvas, record: bool) -> anyhow::Result { let framerate = canvas.framerate(); let size = moq_video::Size::new(canvas.width, canvas.height); let roles: Vec<(Role, moq_video::Size)> = match record { @@ -2190,15 +2246,18 @@ impl Probe { video, }); } - Ok(Self { rungs, refused }) + Ok(Self { + rungs, + refused, + reclaim: None, + }) } /// [`new`](Self::new) on a thread of its own, off the program clock, once /// the encode threads `ending` have closed their sessions. fn spawn( - gpu: Gpu, + renditions: Renditions, canvas: Canvas, - record: bool, ending: Vec>, ) -> std::io::Result>> { std::thread::Builder::new().name("probe".into()).spawn(move || { @@ -2207,7 +2266,7 @@ impl Probe { let _ = thread.join(); } let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build()?; - runtime.block_on(Self::new(&gpu, canvas, record)) + runtime.block_on(Self::new(&renditions, canvas)) }) } } @@ -2331,7 +2390,7 @@ impl Line { } if self.probing.is_none() && !self.settled { let ending = std::mem::take(&mut self.ending); - match Probe::spawn(renditions.gpu.clone(), self.canvas, renditions.record, ending) { + match Probe::spawn(renditions.clone(), self.canvas, ending) { Ok(probe) => self.probing = Some((self.canvas, probe)), Err(err) => { self.settled = true; @@ -2466,6 +2525,8 @@ struct Extras { on_demand: bool, live: bool, stats: Arc, + /// [`Renditions::reclaim`]. + reclaim: Option, /// Where each published scene's broadcast is graded, while live. graded: Option, } @@ -2555,6 +2616,7 @@ impl Extras { broadcast, timebase: catalog.timebase(), record: false, + reclaim: self.reclaim.clone(), bandwidth: self.bandwidth.clone(), on_demand: self.on_demand, live: self.live, @@ -2713,6 +2775,10 @@ enum Video { /// What an encode thread runs. struct Job { sessions: Sessions, + /// [`Renditions::reclaim`]. + reclaim_from: Option, + /// The sources' replay encoders' sessions, held until the encoder opens. + reclaim: Option, /// How its canvas's video is doing. video: Arc, /// Which of the program's renditions it is. @@ -2733,6 +2799,8 @@ struct Job { /// One rendition's encoder and track, doing what its gate decided. struct Encoder { track: Encoding, + /// [`Renditions::reclaim`]. + reclaim_from: Option, config: moq_video::encode::Config, joins: Joins, /// Where to report each encoder opened as its canvas's, under the @@ -2751,8 +2819,25 @@ struct Encoder { } impl Encoder { - /// Open an encoder, returning its backend's name. + /// Open an encoder. When the GPU refuses it one, the program comes first: + /// this closes the sources' replay encoders and opens it in their + /// sessions. fn open(&mut self) -> anyhow::Result<()> { + match self.try_open() { + Err(err) if let Some(sessions) = self.reclaim_from.clone().filter(|sessions| sessions.spare() > 0) => { + tracing::warn!(err = %format!("{err:#}"), "closing the sources' replay encoders for the program's"); + let reclaim = sessions.reclaim(); + // Each source closes its own at its next picture, or at once + // when it delivers none. + pollster::block_on(reclaim.closed()); + self.try_open() + } + result => result, + } + } + + fn try_open(&mut self) -> anyhow::Result<()> { + let session = self.track.hold(false)?; let mut encoder = moq_video::encode::Encoder::new(&self.config)?; // Asked of a fresh encoder, whose first picture is a keyframe anyway, so // the cut it queues costs nothing. @@ -2771,7 +2856,7 @@ impl Encoder { if let Some((stats, rendition)) = &self.report { stats.opened_encoder(*rendition, encoder.name()); } - self.track.open(encoder)?; + self.track.open(session, encoder)?; Ok(()) } @@ -2841,6 +2926,8 @@ impl Encoder { fn encode(job: Job, opened: Option>>, video: mpsc::Receiver