How AI video clipping actually works: the four stages between a link and a clip

"AI clipping" is a label on a pipeline, and the pipeline is the same shape in every tool: transcribe the audio, read the transcript for good moments, crop each one to vertical, render with captions. The tools differ in how well each stage is done and how honestly the failures are reported.

This is what each stage is, built from ours. Where we give numbers they are measured on our own production runs, not quoted from anyone's marketing.

Stage 1: Transcription

The audio is pulled out of the video and turned into text with a timestamp on every word. Everything downstream reads this, so it is the ceiling on the quality of the whole run.

Two things matter here that the output doesn't show you.

Word timestamps, not sentence timestamps. A clip boundary has to land on a real pause. If the transcript only knows where sentences are to within a second, every cut is up to a second off, and you get the mid-sentence cuts that are the single most common complaint about this category of tool.

Proper nouns. A speech model has never heard your guest's surname or your product's name. It will substitute something phonetically nearby, and that substitution ends up burned into the captions. We hand the transcriber the video's title as a hint, which fixes the cases where the name appears there and does nothing for the rest.

We run transcription locally on our own machines rather than through a paid API. It costs compute time but no per-minute fee, which is a large part of why we can offer flat pricing where others sell credits.

Stage 2: Finding the moments

A language model reads the transcript and returns a list of start and end times, each with a title, a reason, and a score. This is the stage that is the product; everything else is plumbing.

What separates a good implementation from a bad one:

It classifies the video first. The unit of a clip is different for a podcast (a complete thought), a comedy upload (setup, punchline, and the reaction after it) and a lecture (one insight). A model that doesn't know which it's reading produces punchlines without setups on comedy and mid-argument fragments on podcasts. We ask for the classification explicitly before any picks.

It reads more than words. A transcript cannot show laughter, a gasp, or thirty seconds of chaos with no speech in it, which on reaction content is exactly where the clip is. We compute an energy level from the audio waveform every half second and weave it into what the model reads: lines get tagged as loud, and gaps become "(loud reaction — no speech)". On a real upload, the loudest spike landed precisely on the punchline.

It processes long videos in windows. A 90-minute transcript is a lot of text to hold in one call, and the results get worse as it grows: the model loses the thread and favours the start and end. We split anything long into 15-minute windows with an overlap, pick within each, then merge. This also turned out to fix a reliability problem: before windowing, one in three of our jobs over 30 minutes failed outright, because a single call on that much text ran long enough for the connection to drop.

Its scores mean something. Most tools score every clip in the 80s, which is a ranking of nothing. We wrote about why that happens and how much of it we fixed. The honest answer was "partly".

The model matters. We moved this stage to a larger, more expensive model than the one we started with, for a few cents more per video, because a cheaper model read "open on a hook" as "start at the punchline". Clip judgment is the product; it's the wrong place to save money.

Stage 3: Reframing to 9:16

A 16:9 frame becomes vertical by keeping a 9:16 column of it, which is only 56% of the width. Where that column sits decides whether the speaker is in the clip.

The naive version centres it. In a two-person shot, centre is the gap between them; the reported symptom was a clip of a knee and a pair of shoes with the speaker out of frame.

The better version detects faces and puts the column on them. But a version that follows a face every frame reads as a handheld camera, and one that locks a single position for the clip reads as a security camera when the subject moves. Getting this right took us four attempts, and the details are their own post. Short version: move only where the source cuts, land on the best face for the new shot, and never drift within one.

Stage 4: Captions and render

Word-timed captions from the transcript, burned in, one encode per clip. This stage is mostly engineering, but it has one failure worth knowing about because it looks like a captioning bug and isn't.

If the video and audio for a clip are fetched separately, and they are when a tool fetches only the seconds it needs rather than the whole file, they don't start at the same instant. On one real clip the audio window began 5.65 seconds before the video window. Mux the two naively and the sound runs that far ahead of the picture, and since captions follow the transcript, they follow the audio. The captions look wrong; the alignment is what's broken. We measure this now by cross-correlating the rendered audio against the source; the acceptable offset is zero.

How to tell which stage broke

When a clip is wrong, the symptom points at the stage:

What you see	Which stage
Caption spells a name wrong, consistently	Transcription
Clip starts or ends mid-thought	Detection (or transcription timestamps)
Clip is 8 seconds long	Detection: found the punchline, lost the setup
Speaker is out of frame, or the crop drifts	Reframing
Captions ahead of or behind the voice	Render: A/V alignment
Every clip scores 85–89	Detection: the scoring rubric has no floor
Soft, blurry output from a link import	Ingest: the tool fetched 360p

That last row is worth its own sentence. Cloud tools fetch from datacenter addresses, and YouTube serves those a 360p stream. A 360p source scaled to 1080x1920 is a 5x upscale. If link imports look worse than uploads on any tool, that's what happened, and no amount of downstream work recovers it.

What this costs to run

For anyone curious why pricing across this category looks the way it does: per video, transcription is compute time (no fee for us, a per-minute fee for tools using a transcription API), moment detection is a few cents of language-model calls, and the render is compute. Measured on our own infrastructure, a 60-minute upload producing 8 clips costs us under €0.10 all in. The expensive part, when there is one, is fetching HD video from platforms that don't want to serve it to you, and we only fetch the seconds we clip to keep that bounded.

None of that is a reason to meter customers by the minute. It's just arithmetic that most tools would rather you didn't do.