Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ Most editors keep a project in a format built for their own app, not for you or
- **Transitions**: put a dissolve, a fade through black or white, a wipe, a slide, a blur or a pixelate on any cut, from the mark between two clips on the timeline, or on every cut at once. A transition is centred on its cut and moves no clip, so titles and music stay where you put them, and the preview draws each one frame for frame as it exports.
- **Captions**: switch them on for the whole video, pick one of twelve languages and one style. The words come from each clip's transcript, so captions follow every trim, split and reorder.
- **AI, in one conversation**: add your clips and say what you want beside the cut, in your own words: it builds the cut from them ("a 30 second video from my clips") or changes the one you have. It remembers the conversation, so "shorter" or "a square one too" knows what it refers to, and it asks back instead of guessing when a request could mean two things. A request that would watch a lot of footage says how much first and, on a yes, runs in the background: keep editing or close the tab, and it never lands on top of a change you made meanwhile. The model calls a fixed set of checked operations, so the edit is always a valid document. One request, one undo.
- **Clips from a long video**: "5 vertical clips with a strong hook from this interview" reads what is said, picks the moments that stand on their own, opens each on its strongest line, and makes each its own 9:16 project with captions, framed on the person speaking. Shots it is unsure how to frame keep the whole picture, and it tells you which. "Frame it for vertical" does the same for the cut you have.
- **Projects hold videos**: a project is your footage and every video you make from it (the full cut, short clips, a teaser), each with its own timeline, shapes and export, listed together in the project.
- **Clips from a long video**: "5 vertical clips with a strong hook from this interview" reads what is said, picks the moments that stand on their own, opens each on its strongest line, and makes each its own 9:16 video inside the project, with captions, framed on the person speaking. Shots it is unsure how to frame keep the whole picture, and it tells you which. "Frame it for vertical" does the same for the cut you have.
- **AI assist**: Auto-cut watches several clips together and assembles the strongest sequence for what the video is for, captions included; Clean up trims one clip down to its good parts.
- **Export to MP4** in draft, standard or high quality, on Clawnify's managed edit service.
- **Share by link**: one link per project plays an export to anyone you send it to, with no sign-in, and a Download button. The link stays on the export you shared until you move it to a newer one, so a draft never reaches viewers; turning it off ends it for everyone.
Expand Down
29 changes: 26 additions & 3 deletions agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,16 +231,39 @@ tell the person it is running: the editor shows its progress. Read `GET
/api/projects/{id}/ask` for how it ended, and pass on what `said` says,
including any shot it was unsure how to frame.

### Projects hold videos

A project is its footage plus every video made from it. The row you create
with `POST /api/projects` is the project and its first video; it holds the
footage, highlights and conversation. Every other video (a cut-down, a
teaser) is a row of its own with `parent_id` set to it, with its own
timeline, formats, exports, share link and comments, and is left out of `GET
/api/projects` (which counts them in `videos`). `GET
/api/projects/{id}/videos` lists a project's videos from any of them; `POST
/api/projects/{id}/videos { name, from? }` makes a new one, empty (9:16) or a
copy of `from`; `PUT /api/projects/{id} { parent_id }` moves a project with no
footage or videos of its own in. Deleting the project deletes its videos;
deleting a video deletes only that video.

### Putting an AI change back

Every change `/instruct` makes is recorded as a job with the cut before and
after it, and is written only over the cut it read (otherwise it comes back
`stale`, kept aside, and the cut stays as the person left it). `POST
/api/projects/{id}/ask/jobs/{job}/revert` puts the cut back as it was, while
the cut is still what that change made (`revertible: true`); after a later
edit it answers 409, and the person undoes it in the editor instead.

### Clips from a long video, and framing for vertical

"5 vertical clips with a strong hook from this interview" is one request:
`/instruct` reads the video's transcript, picks the moments that stand on
their own (about 45 seconds unless asked otherwise, at most 8), opens each on
its strongest line (played first when it comes later in the moment), and makes
each one **its own project**: 1080×1920 with captions on, framed on the person
speaking. The project the request was sent to is not changed. The finished
each one **its own video in the project**: 1080×1920 with captions on, framed on the person
speaking. The video the request was sent to is not changed. The finished
message, and the job while it runs, list them in `made: [{ id, name }]`; open
one at `/edits/{id}`. "Frame it for vertical" frames the project's own cut the
one at `/edits/{id}`. "Frame it for vertical" frames the video it was sent to the
same way, in its 9:16 version (added when missing). Shots the framing is unsure
of keep the whole picture, and the message says how many: a person checks them
in the editor's AI reframe. Both need the video on the media service (an
Expand Down
9 changes: 9 additions & 0 deletions schema.sql
Original file line number Diff line number Diff line change
Expand Up @@ -469,3 +469,12 @@ ALTER TABLE ask_messages ADD COLUMN job_id TEXT;

-- Projects an AI message made (clips cut from a long video), as JSON [{id, name}].
ALTER TABLE ask_messages ADD COLUMN made TEXT;

-- When an applied AI change was put back to the cut before it (POST …/ask/jobs/{job}/revert).
ALTER TABLE ask_jobs ADD COLUMN reverted_at TEXT;

-- Videos of one project: every edit made from a project's footage (a cut-down,
-- a teaser) is its own row, pointing at the row that holds the footage, its
-- highlights and its conversation. NULL: a project (and its first video).
ALTER TABLE edit_projects ADD COLUMN parent_id TEXT;
CREATE INDEX IF NOT EXISTS idx_edit_projects_parent ON edit_projects(parent_id);
7 changes: 5 additions & 2 deletions src/client/app.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,22 @@ function useRouter() {
* it.
*/
function HostNav({ active, path, navigate }: { active: string; path: string; navigate: (to: string) => void }) {
const [projects, setProjects] = useState<Omit<EditProject, "edl">[]>([]);
const [projects, setProjects] = useState<(Omit<EditProject, "edl"> & { video_ids?: string[] })[]>([]);
useEffect(() => {
fetch("/api/projects")
.then((r) => (r.ok ? (r.json() as Promise<Omit<EditProject, "edl">[]>) : []))
.then(setProjects)
.catch(() => {});
reportLocation(window.location.pathname + window.location.search);
}, [path]);
// A project's other videos are inside it: opening one keeps the project marked.
const owner = projects.find((p) => p.video_ids?.some((v) => `edits/${v}` === active));
const marked = owner ? `edits/${owner.id}` : active || "home";
return (
<AppNav
title="Video"
icon="video"
active={active || "home"}
active={marked}
groups={[
{ items: [{ id: "home", label: "Projects", icon: "video", href: "/", home: true }, { id: "highlights", label: "Highlights", icon: "sparkles", href: "/highlights" }] },
{
Expand Down
21 changes: 18 additions & 3 deletions src/client/assistant.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ export interface AskJob {
error: string | null;
/** Clip projects made so far, while it runs. */
made?: Made[];
/** An applied change that can still be put back. */
revertible?: boolean;
reverted_at?: string | null;
}

const ACTIVE: AskJobStatus[] = ["queued", "running"];
Expand Down Expand Up @@ -129,8 +132,12 @@ export function useAsk(projectId: string, getEdl: () => Edl, onApplied: (edl: Ed
}, [active, refresh]);

const act = useCallback(
async (jobId: string, action: "run" | "cancel" | "apply") => {
const out = await api.send<{ result_edl?: Edl | null }>("POST", `/api/projects/${projectId}/ask/jobs/${jobId}/${action}`);
async (jobId: string, action: "run" | "cancel" | "apply" | "revert") => {
const out = await api.send<{ result_edl?: Edl | null; base_edl?: Edl | null }>("POST", `/api/projects/${projectId}/ask/jobs/${jobId}/${action}`);
// Put back: the cut before the change returns, as one undo step here too.
if (action === "revert" && out.base_edl && canonical(latest.current.getEdl()) !== canonical(out.base_edl)) {
latest.current.onApplied(out.base_edl);
}
if (action === "run") seenActive.current.add(jobId);
if (action === "apply" && out.result_edl) {
handled.current.add(jobId);
Expand Down Expand Up @@ -313,7 +320,7 @@ export function AssistantPanel({
}
};

const doAct = async (jobId: string, action: "run" | "cancel" | "apply") => {
const doAct = async (jobId: string, action: "run" | "cancel" | "apply" | "revert") => {
setActing(`${jobId}:${action}`);
setActErr("");
try {
Expand Down Expand Up @@ -375,6 +382,14 @@ export function AssistantPanel({
</div>
);
}
if (job.revertible && !saving) {
return (
<button onClick={() => doAct(job.id, "revert")} disabled={acting !== null} className={`${btnGhost} mt-1 h-7 text-fine`} title="Put the cut back as it was before this change">
{busy("revert") ? <Loader2 className="w-3.5 h-3.5 animate-spin" /> : <RotateCcw className="w-3.5 h-3.5" />} Put back
</button>
);
}
if (job.reverted_at) return <p className="mt-1 text-fine text-faint">Put back.</p>;
if (job.status === "stale" || notTaken.has(job.id)) {
return (
<div className="mt-2 flex gap-1.5">
Expand Down
44 changes: 30 additions & 14 deletions src/client/edit.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { ReframeDialog } from "./reframe";
import { AssistantPanel, JobStrip, canonical, useAsk } from "./assistant";
import { VideoSwitcher, useProjectVideos } from "./videos";
import { embedded, openChat, useChatCommands, useChatContext, useHostChanges } from "@clawnify/app/client";
// Footage edit projects — the timeline editor over EDL documents.
//
Expand Down Expand Up @@ -321,6 +322,8 @@ type RailTab = "media" | "highlights" | "audio" | "text" | "captions" | "comment

export interface EditProject {
id: string;
/** The project this video belongs to; null for a project's first video. */
parent_id?: string | null;
name: string;
edl: Edl;
brief: string;
Expand Down Expand Up @@ -799,6 +802,8 @@ type ProjectSummary = Omit<EditProject, "edl" | "brief"> & {
cover_media: string | null;
/** Clips taken from the project's Drive folders. */
footage: number;
/** How many videos the project holds, its first included. */
videos?: number;
};

/** The frame a project is recognised by: its opening shot, or a blank tile. */
Expand Down Expand Up @@ -907,8 +912,8 @@ export function ProjectsHome({ navigate }: { navigate: (to: string) => void }) {
)}
</h1>
<p className="text-body-sm text-muted mt-0.5">
One project is one video: upload your footage, trim it, put the clips in order, add
text and music, and export it to MP4.
A project holds your footage and every video you make from it: the full cut,
short clips, a teaser. Each video has its own timeline, shapes and export.
</p>
</div>
{projects && projects.length > 0 && (
Expand Down Expand Up @@ -966,7 +971,7 @@ export function ProjectsHome({ navigate }: { navigate: (to: string) => void }) {
<div className="text-fine text-faint mt-0.5">
{removing?.id === p.id
? `Deleting${removing.left ? `, ${removing.left} clips to go` : ""}`
: `Edited ${fmtDate(p.updated_at)}${p.footage ? ` · ${p.footage} source ${p.footage === 1 ? "video" : "videos"}` : ""}`}
: `Edited ${fmtDate(p.updated_at)}${(p.videos ?? 1) > 1 ? ` · ${p.videos} videos` : ""}${p.footage ? ` · ${p.footage} source ${p.footage === 1 ? "video" : "videos"}` : ""}`}
</div>
</div>
</button>
Expand All @@ -988,7 +993,9 @@ export function ProjectsHome({ navigate }: { navigate: (to: string) => void }) {
<ConfirmDialog
title={`Delete “${confirmDel.name}”?`}
body={
confirmDel.footage
(confirmDel.videos ?? 1) > 1
? `The project, its ${confirmDel.videos} videos and their exports are deleted. Shared highlights and videos used by other projects are kept.`
: confirmDel.footage
? "The project and its export history are deleted. Shared highlights and videos used by them or other projects are kept. Unused Drive-folder copies are removed; originals in Google Drive are kept."
: "The project and its export history go with it. Your footage stays in the media library."
}
Expand Down Expand Up @@ -1041,12 +1048,10 @@ export function EditRoute({ id, navigate }: { id: string; navigate: (to: string)
setAssets(null);
setFootage(null);
setErr("");
Promise.all([
api.get<EditProject>(`/api/projects/${id}`),
api.get<Asset[]>("/api/assets"),
// A first look only: the panel's own reads move the footage on.
api.get<FootageList>(`/api/projects/${id}/footage?logs=0&step=0`),
])
Promise.all([api.get<EditProject>(`/api/projects/${id}`), api.get<Asset[]>("/api/assets")])
// The footage is the project's, whichever of its videos is open. A
// first look only: the panel's own reads move the footage on.
.then(async ([p, a]) => [p, a, await api.get<FootageList>(`/api/projects/${p.parent_id ?? p.id}/footage?logs=0&step=0`)] as const)
.then(([p, a, f]) => {
if (!current) return;
setProject(p);
Expand Down Expand Up @@ -1089,7 +1094,8 @@ export function EditRoute({ id, navigate }: { id: string; navigate: (to: string)
initial={project}
initialAssets={assets}
initialFootage={footage}
onOpenHighlights={() => navigate(`/edits/${id}/highlights`)}
projectId={project.parent_id ?? project.id}
onOpenHighlights={() => navigate(`/edits/${project.parent_id ?? project.id}/highlights`)}
onOpenProject={(other) => navigate(`/edits/${other}`)}
/>
);
Expand All @@ -1101,14 +1107,17 @@ export function EditEditor({
initialFootage,
onOpenHighlights,
onOpenProject,
projectId,
}: {
initial: EditProject;
initialAssets: Asset[];
initialFootage: FootageList;
/** Opens the footage's highlights. */
onOpenHighlights?: () => void;
/** Opens another project, such as a clip the AI made. */
/** Opens another video, such as a clip the AI made. */
onOpenProject?: (id: string) => void;
/** The project this video belongs to: its footage and highlights. */
projectId?: string;
}) {
const [name, setName] = useState(initial.name);
const [brief, setBrief] = useState(initial.brief ?? "");
Expand Down Expand Up @@ -1228,6 +1237,12 @@ export function EditEditor({
[commit],
);
const ask = useAsk(initial.id, () => edlRef.current, applyAi);
const videos = useProjectVideos(initial.id);
// Clips the AI makes become videos of this project: count them as they come.
const madeCount = (ask.messages ?? []).reduce((n, m) => n + (m.made?.length ?? 0), 0) + ask.jobs.reduce((n, j) => n + (j.made?.length ?? 0), 0);
useEffect(() => {
if (madeCount) void videos.reload();
}, [madeCount, videos.reload]);

// Inside Clawnify the conversation is the dashboard's chat. It knows which
// project is open, offers its requests in the "/" menu, and calls this app's
Expand Down Expand Up @@ -1548,7 +1563,7 @@ export function EditEditor({
<input
value={name}
onChange={(e) => setName(e.target.value)}
aria-label="Project name"
aria-label="Video name"
className="min-w-24 flex-1 lg:flex-none lg:w-64 bg-transparent text-heading-3 outline-none rounded-sm h-7 px-1.5 -mx-1.5 hover:bg-surface-sunken focus:bg-surface focus:shadow-edge"
/>
<span
Expand All @@ -1558,6 +1573,7 @@ export function EditEditor({
>
{saveState === "saved" ? "Saved" : saveState === "saving" ? "Saving…" : saveState}
</span>
<VideoSwitcher videoId={initial.id} videos={videos.videos} reload={videos.reload} onOpen={onOpenProject} />
<div className="flex-1" />
<button
onClick={undo}
Expand Down Expand Up @@ -1658,7 +1674,7 @@ export function EditEditor({
{/* three-panel middle */}
<div className="flex-1 flex min-h-0">
<LeftPanel
projectId={initial.id}
projectId={projectId ?? initial.id}
initialFootage={initialFootage}
onOpenHighlights={onOpenHighlights}
pane={pane}
Expand Down
Loading
Loading