How auto-reframe to 9:16 works, and why the first three versions of ours were wrong

A 16:9 video is 1920 pixels wide. A 9:16 clip made from it at full height is 608 pixels wide. Everything outside that column is gone. So auto-reframe is not really a question of how to crop; it's a question of where, sixty times a second, for every clip.

We've shipped four different answers to that question. The first three were wrong in ways that are instructive, so this is all four, including the one that's live.

Version 0: centred

The default. ffmpeg's crop filter, no arguments, middle of the frame.

It's correct for a single person sitting in the middle of shot, which is how most talking-head YouTube videos are framed, and it's how every tool starts. It is wrong for everything else. In a two-person podcast shot the middle of the frame is the gap between the two people. The first real complaint we got described a clip showing "a knee and a pair of shoes" while the speaker sat out of frame to the left.

Worth knowing: a tool that says it reframes and produces clips like that is probably running this version, and a tool that produces clips like that intermittently is probably running a smarter version that silently falls back to this one when its face detector fails. Ours did exactly that for a while, because of a missing data file in the build, and the only symptom was that every clip reported "centred". Silent fallbacks are the recurring villain of this whole category.

Version 1: track the face

Detect faces once a second, put the crop on the largest one, smooth the motion so it doesn't jitter.

This fixed the two-shot. It introduced a new problem: the crop followed people. A speaker who leans, gestures, or just shifts in their chair produced a crop that drifted after them, and a drifting crop reads as a handheld camera. Smoothing helped (a dead zone so small movements are ignored, easing so moves start and stop gently, a speed cap so a bad detection can't fling the frame across the shot) but the complaint that came back was still "the camera is going from left to right".

The specific failure was cuts. When the source video cuts from a wide shot to a close-up, the face jumps from one part of the frame to another instantly. A smoothed tracker eases across that jump over a second or so, and the viewer sees a pan that isn't in the original. On one clip we counted 42 of those reframes, most of them landing mid-shot rather than on the cut, and every one of them was a visible camera move the source didn't have.

Version 2: lock it

The obvious over-correction: pick one position per clip and hold it.

No drift, no pans. And it's wrong the moment the subject moves, because a locked window on a moving person reads as a security camera. The complaint this time was "the frame is now stuck". A tracked crop is wrong because it moves when it shouldn't; a locked crop is wrong because it doesn't move when it should. Both are the same mistake from opposite sides: treating "should the crop move" as a property of the clip rather than of the shot.

Version 3: per shot, snapped at the cut

What's live now, and what the rest of the market converged on too:

Find the cuts, exactly. ffmpeg's scene detector reports every cut to the frame (1.15s, 2.083s, 3.933s…). This matters: inferring cuts from once-a-second face samples locates them only to the nearest second, and a reframe that lands a half-second after a cut is a visible pan, while one that lands on the cut is invisible, because the whole picture changed anyway.

Frame each shot independently. Within a shot, detect faces across all its samples and choose the one crop position that keeps the most face coverage for the whole shot. Then hold it. A shot whose faces all sit within a small band gets a perfectly still crop, which is what a locked-off camera should look like.

Move only at cuts, and move instantly. At each cut, jump to the new shot's position with no easing. The source already cut, so a simultaneous reframe reads as part of the edit rather than as camera movement. This is the whole insight: the viewer accepts any reframe that coincides with a cut and notices every reframe that doesn't.

When a shot has no face, don't inherit the last one. A title card, a wide shot of an empty room, a screen share: fall back to the clip-wide best position, not the previous shot's. Inheriting is how our crop ended up parked on a white chat panel for six seconds; that position was chosen for a completely different shot.

When two people share a shot, stack them. Webcam chats, split-screen calls, anything with two persistent, separated faces: instead of choosing one, crop both and stack them vertically, 9:8 over 9:8. Every major tool does some version of this now. The trap we found by rendering rather than reasoning: the crop size has to be capped by the distance between the two faces, or the two panels overlap each other's content.

What it doesn't do

Active-speaker switching. The tempting feature is "cut to whoever's talking". Nobody credibly does this on uploaded flat files, and we don't claim to. It needs either separate audio tracks per person or lip-movement detection that's reliable at webcam resolution, and neither is there yet. Stacking both people is the honest version.

Tiny faces. Face detection runs on a downscaled frame for speed. A webcam-chat participant who is 30 pixels tall at that scale is at the edge of what the detector sees, and when it misses, the shot falls back to centred. Running detection at higher resolution is the next change on the list; it costs time per clip.

Anything without a face. Gameplay, slides, product shots. The crop centres, which is right about half the time.

How to check a tool's reframing

Whatever tool you're evaluating, three tests, in order of how often they fail:

1. A two-person shot. If the crop lands between them, it's centred.
2. A clip with a cut in it. If the crop pans around the cut instead of jumping at it, it's tracking with smoothing, and it'll do that on every cut.
3. A single speaker who leans. If the crop follows them, it'll read as handheld across a whole video. If it doesn't move at all on a big move, it's locked.

A good result on all three is a crop that's still within shots and invisible between them, which is what a human editor would produce and what nobody notices when it's right.