Repository navigation
[feature request] native frame-at-timestamp export (no external ffmpeg dependency for downstream tools) #2059
Description
Problem
cap export already renders a full video natively, but there's no way to pull
a single still frame at a given timestamp (or read a segment's exact duration)
without shelling out to a separately-installed ffmpeg/ffprobe. Any tool
built on top of Cap that needs still frames — thumbnailing, a step-by-step
doc generator, a diffing/QA tool — ends up carrying that whole external
dependency itself, with all its fragility: PATH lookup, version drift,
platform-specific installs, and (what actually happened to us) a Homebrew
library-version mismatch silently breaking frame extraction with no clear
error.
Concrete case
cap-tools (a Python CLI built on Cap) has a capt guide command that
extracts a screenshot per detected click from a .cap recording's
display.mp4. It shelled out to ffmpeg -ss <t> -i display.mp4 -vframes 1
per click and ffprobe for segment duration. On 2026-07-31, a routine
brew upgrade on a dev machine left the installed ffmpeg linked against a
now-missing libSvtAv1Enc.3.dylib, and every call silently produced zero
frames — the tool's own error handling didn't even catch it as an ffmpeg
problem, since subprocess.run still spawns the process, it just exits
non-zero. The realistic fix for us was vendoring
PyAV (FFmpeg statically compiled into the Python
wheel) to remove the external dependency entirely — but that's real
duplicated effort (and duplicated FFmpeg-linking risk) that every Cap-based
tool has to solve independently, when Cap's own CLI already has a native,
often hardware-accelerated decode path (cap export --force-ffmpeg-decoder
exists specifically as an opt-out of the platform decoder, implying the
default path isn't shelling out to system ffmpeg at all).
Proposal
Add a way to pull one or more still frames — and/or just a duration query —
through the same native pipeline cap export already uses, so no consumer
of Cap recordings needs its own vendored or system video toolchain just to
get a thumbnail.
cap export <project.cap> --frames-at 3.5,7.2,12.0 --out-dir frames/ --json
cap project duration <project.cap> --json # or fold into `project validate`'s existing output
- Frame output could reuse
--format's existing container knowledge (jpg/png)
and--resolutionfor downscaling — same flags ascap exportalready has. - Duration is probably the cheaper win alone:
cap project validatealready
reads the project; adding adurationSecondsfield to its JSON output would
let tools dropffprobeeven without full frame-export support.
Why now
Cap's CLI surface is clearly meant to be the foundation other tools build on
(cap agents install, the MCP server, the whole "designed to be driven by
automation and AI agents" framing in cap --help). Every downstream tool
that wants a still frame currently has to solve the exact fragility we just
hit, independently, on their own.
Status
No PR yet — wanted to check appetite and naming/shape preference first.
Happy to prototype against crates/export if this is a direction you'd take
a contribution on.
Activity
linear commentedon Jul 31, 2026
CAP-771
Post export-frames feature request; recreate lost workflows.mdx PR draft
tembo commentedon Jul 31, 2026
Good write-up. A chunk of what you're asking for already exists in the CLI today, which is worth knowing before shaping the PR — the real gaps are CLI surface and discoverability, not a missing native capability.
Frame-at-timestamp already works:
cap export-previewThere's already a top-level
cap export-previewcommand that renders a single still frame at a given timestamp through the exact same native pipeline ascap export— sameExporterBasebuilder, same decoders, sameFrameRenderer:ExportPreviewinapps/cli/src/export.rs—--frame-time <f64>,--settings-json,--force-ffmpeg-decoderrender_previewincrates/export/src/preview.rs— buildsExporterBase, callssegment_media.decoders.get_frames(segment_time, ...), renders viaFrameRenderer::render_immediate, encodes to JPEG.So today a downstream tool can already pull a thumbnail with zero external ffmpeg by running
cap export-preview foo.cap --frame-time 3.5 --settings-json '{"fps":60,"resolution_base":{"x":1920,"y":1080},"compression_bpp":0.15}'and base64-decoding thejpeg_base64field from the JSON result (ExportPreviewResult).What's missing vs your proposal, and would make a clean PR:
--out-dir. Adding a file-output mode (reusing--format's jpg/png knowledge +--resolution) is a thin layer on top ofrender_preview.--frames-at a,b,cbatch. Noterender_previewrebuildsExporterBase(the heavy part — wgpu device/adapter setup + decoder spawn) per call, so a multi-frame mode should build once and looprender_preview_with_baseN times to avoid re-paying setup per timestamp.--settings-jsoninstead of the friendly--format/--resolution/--qualityflagscap exportalready exposes —settings_from_flagsinexport.rsis the pattern to copy.cap guide --json(the agent capability manifest). The guide listsexportbut omitsexport-preview, so it's effectively undiscoverable for agents — fixing that is a one-liner inguide.rsand probably the highest-value piece on its own.Duration is already computed natively (in-process, no ffprobe)
cap_rendering::get_duration(crates/rendering/src/lib.rs:1844) already computes project duration fromProjectRecordingsMeta::duration()+get_source_duration(). The source duration is read viaVideo::newusingffmpeg::format::input— i.e. the Rust FFmpeg library bindings, not a systemffprobesubprocess.ExportPreviewResult.total_framesis already derived from it (preview.rs:245).So your "cheaper win alone" instinct is right and very small:
cap project validate(project.rs:275) andcap project inspect(project.rs:52) already loadRecordingMeta+ProjectConfiguration— they'd just need to buildProjectRecordingsMetafrom the studio meta and callget_durationto emit adurationSecondsfield in the existingValidationReport/ProjectInspectionJSON. That alone lets every downstream tool dropffprobewithout touching frame export.One clarification on
--force-ffmpeg-decoderYour read is correct, and worth extending: that flag selects the in-process FFmpeg software decoder vs the platform hardware decoder (AVAssetReader on macOS / MediaFoundation on Windows), with automatic fallback to FFmpeg if the platform decoder fails (
decoder/mod.rs:628). Both paths go throughcap_video_decode— neither shells out to a separately-installed systemffmpeg/ffprobebinary. Cap links FFmpeg as a Rust library. So the "remove the external dependency" property you want already holds for any consumer that uses Cap's pipeline; the fragility you hit is purely from going around Cap to call system ffmpeg directly, which this feature would let tools stop doing.Suggested scope split
Given the above, this splits cleanly into two independent PRs:
durationSecondstoproject validate/inspectJSON + documentexport-previewincap guide. Tiny, high leverage, unblocks theffprobedrop immediately.--frames-at/--out-dir/--formatto a frame-export mode reusingrender_previewwith a sharedExporterBase. Bigger, butcrates/exportis exactly the right place as you suggested.Happy to help review a prototype against
crates/exportif you go ahead.Record maintainer response on export-frames issue (CapSoftware/Cap#2059)
Prefer cap export-preview for frame extraction; PyAV as automatic fal…
kylebrodeur commentedon Jul 31, 2026
Thanks, this resolved what I needed.
I integrated
cap export-previewdirectly into capt-tools as the primary path for frame extraction, decodingjpeg_base64straight to a file. Kept my old PyAV extraction as an automatic fallback for machines without Cap installed. Tested live against a real recording, both with and without cap on PATH, correct resolution either way.Turns out I didn't need the duration/ffprobe fix myself. PyAV already gives me that with no ffmpeg dependency, so it was never actually blocking me. Still think it'd be worth having
durationSecondsbuilt into Cap natively though, so other tools don't have to reach for their own workaround.The batch/file-output mode is worth doing too, along with the guide manifest listing. I'd consider picking up either of these at some point.
sloemo01 commentedon Oct 14, 2026
Taking the duration slice from tembo's split above:
durationSecondsincap project validate/inspectoutput, plusexport-previewlisted incap guide. I'll open a PR that references this issue. @kylebrodeur if you're still planning to pick one of these up yourself, say the word and I'll leave it.sloemo01 commentedon Oct 14, 2026
Opened #2556 for the duration slice:
durationSecondsincap project validate/inspectoutput, andexport-previewlisted incap guide.feat(cli): add durationSeconds to project validate and inspect
feat(cli): add durationSeconds to project validate and inspect