Auto Speaker Focus & Multicam (Frame Whoever Is Talking)

About the Auto Speaker Focus & Multicam (Frame Whoever Is Talking)

A podcast or an interview filmed with a camera on each person needs an edit that shows whoever is talking — and a wide shot of one camera needs a crop that follows them, above all for vertical Reels, Shorts and TikTok. This page does both on your own device. With several cameras (up to four, each with its own sound), it lines them up by their sound, works out from the microphones who is talking — the camera whose sound is clearly the loudest — and cuts to that camera, with the wide shot when several people talk at once. With one wide shot, it finds the faces and follows the one whose mouth moves while there is speech, framed close, medium or wide in the shape you choose.

You check every switch before anything is made: a preview made of your own videos plays the edit, a timeline shows who is talking and who is on, and a button per camera or person switches from the moment you press it. The result is an MP4 with the sound of one camera or an automatic mix in which the talker’s microphone leads. Detection only chooses a camera or a crop; nobody is recognised or identified, and nothing is uploaded.

The free preview on this page is made of the moments around the switches — 4-second windows, up to 10 seconds in all — with a MySmartCoPilot watermark. With a Pro pass, or after unlocking this one result, you get the whole video.

Common uses

• Video podcasts filmed with a camera (or phone) on each person
• Interviews and panel talks filmed in one wide shot, cut for vertical Reels, Shorts and TikTok
• Webinar and meeting recordings with each speaker on their own camera
• Turning a horizontal conversation into a square or 4:5 post that follows the speaker

How it decides who is talking

• Several cameras. Each camera’s sound is measured every 100 ms as momentary loudness (ITU-R BS.1770, a 400 ms window), and speech is found with an energy detector. The loudest microphones are first brought to the same level of loud speech, so a quieter microphone is not at a disadvantage; then the camera whose sound is louder than every other camera’s by the microphone separation (6 dB unless you change it) shows its person. Within that margin several people are talking: the wide shot, or the person who was on.
• Lining up. The rises of the sound level in two cameras happen at the same moments, so the offset is where they match best (a cross-correlation of their onsets, to a hundredth of a second). It is shown for every camera and can be typed in.
• One wide shot. Faces are found five times a second with a face detector that runs on your device, and each face’s mouth area is compared with the next frame. While there is speech, the face whose mouth moves the most, over two seconds, is followed.
• The edit. A lowest-cost path through the evidence, with a price on every switch (set by switching), shots of at least the shortest shot, and each cut moved onto the start of the new talker’s speech. Your own switches are kept exactly.

Framing and sound

For a crop, the frame is sized from the face — head and shoulders (close), upper body (medium) or as much as fits (wide) — with the eyes about a third of the way down, kept still while the face hardly moves and following it smoothly (a One-Euro filter) when it does. With cameras in another shape, each camera’s main face is kept in its crop. The automatic mix gives each microphone its share of the sound by how loud it is at that moment (gain sharing), so the talker’s microphone leads and the room sound and bleed of the others drop; a limiter keeps the peaks under −1 dBFS.

Recording tips

• A microphone close to each person (a clip-on or a desk microphone into each camera) gives the clearest decisions; cameras that only use their built-in microphones from across the room hear everyone almost equally — lower the separation, or check more switches by hand.
• A clap at the start makes lining up quick and certain.
• For one wide shot, keep faces turned roughly towards the camera and well lit.

Limitations

• Automatic switching can pick the wrong person when people talk over each other or a microphone hears the whole room: check the timeline before you make the video.
• For one wide shot, faces must face the camera and be at least about a twentieth of the picture’s width; people in profile or turned away are not followed.
• One recording with a separate microphone on each of its sound tracks is not split; add one video per camera instead.
• Up to 60 minutes and four cameras on a computer, and 10 minutes and two cameras on a phone. The video is encoded again (H.264 MP4).
• The cameras’ sound is lined up as a whole: a camera whose clock drifts in a long recording can slowly go out of step.

Privacy

On this device; its files download once

Everything happens in your browser. What you enter or open here is not uploaded or stored by MySmartCoPilot. The video engine, and for faces the face-detection engine (about 13 MB, once), download from MySmartCoPilot the first time they are needed. Your videos are never uploaded.

Open-source components

This page can load these open-source components, each only when a task needs it:

• It uses FFmpeg under the LGPLv2.1, through the Mediabunny add-ons: licence and source code.
• It uses the LAME MP3 encoder under the LGPL, through the Mediabunny MP3 add-on: licence and source code.
• It uses Mediabunny under the MPL 2.0: licence and source code.

You may replace them with your own builds. Their licences and the credits of every library the site uses are on the open-source credits page.