How to design sound and voice for an AI video
For AI agents: find the right skill in one call —
curl -s "https://varg.ai/how-to/api/search?q=<what the person wants>"returns the matching how-to page and the best skills with install commands, as markdown (Russian works too). Full index: https://varg.ai/how-to/llms.txt. Every page has a markdown twin: add.mdto its path. Before generating anything, show the person the options on the how-to page. From a terminal:curl -fsSL https://varg.ai/install | shinstalls the vargai CLI (https://varg.ai/how-to/cli.md):vargai searchandvargai install <slug>with no key, every varg model aftervargai login.
A video brief goes in (type, platform, length, who speaks); a sound plan, cast voices, recorded lines, music, effects and a mix checked by script come out. On the same voice lines the designed soundtrack passed 8 of 8 checks, while the plain one had the music 6.6 dB louder than the voice where words live.
Step 4 of 10 in How to make a video with an AI agent in ten steps. Before: How to write a video script with an AI agent. Next: How to design characters and props for AI video.
For your agent
Install the skill, then paste the prompt.
npx skills add vargHQ/skills --skill video-sound
Use the video-sound skill. You are the sound designer, voice director and music editor for this video.
Brief: {what the video is, type (launch / cinematic / music video / trailer or ad / UGC / explainer), platform,
length, who speaks, budget cap in dollars}
You cannot hear, so plan in writing, measure everything with numbers, and never say you heard something.
Stop where it says STOP: after the sound plan and lines, after the audition page, and before delivery.
The full prompt, which works without the skill installed, is under "The full prompt" below.
The short version
- Sound before picture. Write the sound plan and record the voice first; the word times set the cuts. Pick only 3-6 moments that get an accent, and at most 5 layers in any 5 s (Walter Murch).
- Few effects. Measured pro launch films leave about 85% of hard cuts without a sound of their own and use about 2 whooshes per film. Every effect needs a visible cause.
- Direct the voice with the model's own tags. On
eleven_v4the bare line ran 231 words a minute; with punctuation and breath tags it ran 194, and the pause after the first sentence grew from 0.36 s to 0.92 s. No tag was read aloud. - Cast from candidates. The hardest line in 3-5 voices, 2 takes each, measured and put on one page; a person listens and picks.
- Music models keep tempo, not structure. ElevenLabs Music held 120 BPM in 3 of 3 takes and made the planned silence in 0 of 3, so make the stop, the drop and the exact length in the edit.
- Measure every file. 1 of 5 generated effects came back silent. Impacts land by their start, whooshes by their peak.
- Mix by cue sheet, check by script. Voice 10 dB or more over the music where words live (300 Hz-4 kHz), effects within one frame, cuts on the beat, no dead air, -14 LUFS and -1 dBTP for social.
- A person listens at three STOPs: the plan and lines, the voice audition, and the final mix on headphones and a phone. An LLM judge shortlists takes; it never picks by taste.
Sound and voice for AI video
Guide · November 3, 2026
How film sound designers, trailer editors, radio producers and voice directors make sound, turned into a routine an AI agent can follow: real-sounding voices, dialogue with pauses and breaths, effects on the right frame, music mapped to the beats, and a mix checked by script.
| Number | What it measures |
|---|---|
| ~85% | of hard cuts in pro launch films get no sound of their own |
| 231 → 194 | words a minute: bare text vs breath-tagged on Eleven v4 |
| 0 of 3 | music takes made the planned silence and hit |
| 8/8 vs 5/8 | checks passed: designed vs plain soundtrack |
The short answer
Make the sound before the picture, use fewer sounds than feels natural, and let a person's ears decide what numbers can't. An agent can't hear, so it plans in writing, measures every file, and stops for a listener at three points. This routine works for any video, whether made in code (HyperFrames, Remotion) or generated (Seedance, Kling, Veo).
- Brief. Video type, platform and its loudness target, length, who speaks, music or not. The type decides what leads: music (launch, music video), the voice (explainer, UGC), or the room and its silences (cinematic).
- Sound plan before any picture. One row per beat: voice, music, effects, ambience, planned silence. For each music cue: start, stop, emotion, "why now". Pick only 3-6 moments that get an accent, and at most 5 layers in any 5 s (Walter Murch).
- Lines for the ear. Short spoken sentences, one breath each, behaviour instead of emotion words. Pauses, breaths and small actions go in the TTS model's own tags. Cut every line the picture already says. STOP: the person approves.
- Cast from candidates. The hardest line in 3-5 voices, 2 takes each, measured and put on one page. STOP: the person listens and picks.
- Record every line, transcribe it, re-render the wrong ones. The word times set the cuts.
- Music. Prompt genre, mood, instruments, BPM, key and era, and narrate the arrangement. Measure what came back, fit it to length on downbeats, and make the stop and the drop in the edit.
- Effects only where something happens on screen. Measure every file. Impacts land by their start, whooshes by their peak. Room tone runs under everything.
- Mix from a cue sheet and check it by script: voice 10 dB or more over the music where words live, effects within one frame, cuts on the beat, no dead air, -14 LUFS and -1 dBTP for social.
- STOP: the person listens on headphones and a phone, then you fix their notes.
What the evidence and our tests say most clearly:
- Few effects. Measured pro launch films leave about 85% of hard cuts without a sound of their own and use about 2 whooshes per film.
- Bare text sounds rushed. On
eleven_v4the bare line ran 231 words a minute. With punctuation and breath tags it ran 194, and the pause after the first sentence grew from 0.36 s to 0.92 s. No tag was read aloud. - Music models keep tempo, not structure. ElevenLabs Music held 120 BPM in 3 of 3 takes and made the planned silence in 0 of 3.
- Measure every file. 1 of 5 generated effects came back silent.
- The routine is checkable. On the same voice lines, the designed launch soundtrack passed 8 of 8 checks. The plain one had the music 6.6 dB louder than the voice where words live.
The guide covers six video types: launch films, cinematic AI films, music videos, trailers and ads, UGC and talking characters, and narrated explainers and code demos. Those six cover nearly every video an agent is asked to make, and each puts a different layer in front. Everything ships as the video-sound skill, a varg how to card and one copy-paste prompt (see "The skill and the copy-paste prompt").
Who we learned from
Every rule in this guide comes from a person with checkable credits, an official model guide, or our own measured test. We did not invent any of it. We read 81 interviews and essays by film, trailer, radio and ad sound people, profiled 57 candidates on X and kept 20, and re-read the 160 earlier sound tips from X.
| Who | Credits | What we took |
|---|---|---|
| Walter Murch | Editor and sound designer: Apocalypse Now, The Godfather, The English Patient; 3 Oscars | At most 5 layers in any 5 s; silence 3-4 times a film; pre-laps; music "like drugs" |
| Randy Thom | Director of sound design, Skywalker Sound: Forrest Gump, Cast Away | Point-of-view sound; show a sound once, then keep it off screen; let characters listen |
| Ben Burtt | Star Wars, E.T., WALL-E | Real recordings over synths; R2-D2 is 50% human voice |
| Richard King | 4 Oscars: Dunkirk, Inception, The Dark Knight | A little silence before every big hit |
| Johnnie Burn | Oscar 2024, The Zone of Interest | A sound about 0.5 s before the reaction; distance takes bass and top end |
| Mark Mangini | Oscars for Mad Max: Fury Road and Dune | Spotting sessions; the sound redesign alone raised test scores 8 points |
| Craig Berkey, Skip Lievsay, Carter Burwell | No Country for Old Men | Underplay before a shock; ambience as score; music hidden behind wind |
| Mac Smith | Emmy-winning supervising sound editor, Skywalker Sound | Bring sound in at the script; don't over-clean dialogue |
| Michael May, NPR Training | Scored features for NPR and Studio 360 | Beds that leave room for the voice; cut music by whole bars |
| Jessica Hansen, NPR | NPR announcer and voice coach | A full breath at the start of a thought |
| Khris Brown | Voice director, launched the LucasArts voice department | Cast the voice that makes you feel; no voice beats a bad voice |
| Paul Machliss | Editor, Baby Driver (BAFTA) | A 100-minute radio play before picture; don't edit the song |
| Craig Mellish | Ken Burns' editor, 2 Emmy noms for sound | A radio cut first; final music from day one |
| Derek Lieu | Game trailer editor | Duck music 4 frames down, 20-30 frames up; no hard audio cuts |
| Tommy Malatesta, Adam Gonzales | AV Squad: A24 trailers | Play a sound before its picture; music changes after each piece of information |
| Todd Yellin, Lon Bender | Netflix "ta-dum" | A sonic logo under 3 s, built from 4 layers, tested blind |
| Henry Daubrez | Google Flow filmmaker in residence | Give AI lines room to breathe; clone a clean line and re-dub |
| PJ Ace | AI ad director, CEO Genre.ai | NO MUSIC block, sounds in angle brackets, a DELIVERY line per line |
| Uncanny Harry | BAFTA-nominated TV director | Act the line yourself and transfer the performance |
| Ian Shepherd | Mastering engineer, loudness specialist | -1 dBTP ceiling; platforms normalise, so don't over-squash |
| Alex Knickerbocker | Studio sound editor and re-recording mixer | -14 is a web number; TV is -24 LKFS |
| bruno (tvnxty) | Sound designer for product launch videos | Not every movement gets a sound; UI sounds fit in 0.5 s |
What experts liked or saved could not be read. X has hidden all likes since June 2024, and twitter_user_likes returned nothing for all 14 experts we tried; bookmarks are private. Their own posts and the pages they wrote are the evidence.
Four popular accounts in the earlier X tip bank were dropped as sources. Each has more saves than likes on a small following, plus credits we could not verify: @second_bassoon (the most-saved "why AI dialogue sounds fake" thread), @beechinour (dBFS mix ranges), @Voxyz_ai and @Asteri_eth. Where their advice matches a verified expert, we cite the expert instead.
Voices: cast, direct, make them sound real
An AI voice sounds fake for four reasons, and each has a known fix. Cast by listening to candidates on your hardest line. Direct with the syntax the model documents. Give lines room. Put the voice in a room.
| Tell | Who names it | Fix |
|---|---|---|
| Rushed: too many words for the clip | Henry Daubrez: "pushing 20 lines of text through a 30s generation, the model will speed through the delivery" | Fewer lines per clip; a 4 s shot holds one or two short exchanges (OpenAI Sora guide) |
| Too clean: no room, no breath | Randy Thom: re-recorded dialogue "almost always" lacks "the life" of the original | Room tone under every line; ElevenLabs audio_effects can place the voice in a room (docs) |
| Over-cleaned | Mac Smith, Skywalker: cleaning "softening consonants" hurts intelligibility | Don't denoise generated voice; leave quiet breaths in (iZotope RX) |
| No soul in a synthetic character | Ben Burtt: synth R2-D2 "didn't seem to have a soul"; the final was 50% human | Start from a human performance for robots and creatures, then process it |
Make candidates, let a person pick
An agent can't hear, so it makes and measures candidates and a person chooses. Voice director Khris Brown casts the voice that makes you "feel something personal", and says "Better no voice at all than poor voice."
- Write the casting note as a role, a listener and 2-3 precise adjectives ("caring nurse", "to a scared child", "reluctant, certain"). Never write "conversational"; write a situation: "toss off that last phrase like it's something you've discussed a million times" (voice actors on Game Developer).
- Shortlist 3-5 voices. Premade voices by age, accent and use case, or design new ones. ElevenLabs Voice Design returns 3 voices per generation for the price of one preview (docs). Its prompt format:
Native <Language>. <Gender>, <Age range>. <Quality>. Persona: <2-5 words>. Emotion: <2-3 adjectives>.Keep guidance scale low (their examples run 20-40%), and make the preview line fit the character. Gemini wants a 1-2 sentence voice description (docs). - Audition the hardest line, 2 takes per voice, on one page. For two speakers pick contrasting voices: in Hume's blind test mixed-gender pairs scored 4.65 for telling speakers apart, same-gender pairs 2.87-3.44 (Hume).
- Screen with numbers, then listen on headphones and a phone speaker. Our
audition.pyrenders the takes, measures length, words per minute and pauses, and builds the page with a Pick button. - Lock it: voice id, model and settings go in the brief. Re-render only the lines that are wrong.
No expert publishes a casting chart by video type. The documented starting points: ElevenLabs' "Movie Trailer Voice" design example at 20% guidance; a phone-mic, no-music audio line for UGC (prompt, 151k views); Seedance builds a voice from how the character looks, so the character sheet casts the voice; ElevenLabs has no child voices by policy (help).
Direct it: the exact syntax per model
| Model (varg id) | Pause | Breath, reaction | Delivery | Watch out |
|---|---|---|---|---|
ElevenLabs v4 / v3 (eleven_v4) | ..., a line break, [pause] | [sighs] [exhales] [laughs] [clears throat] [inhales deeply] | [whispers] [sarcastic] [curious], or a stage note: [Low, steady voice, restrained urgency] | No SSML <break>; only tags you can hear (no [standing]); narrative cues like "she asked, trembling" are read aloud (guide) |
Gemini 3.8 TTS (gemini_3_8_flash_tts) | <short pause> <long pause> | <breath> <sigh> <cough> <chuckle> and 30 more | short style: "whispered urgently" | Long "Director's Notes" blocks cause voice drift; split the turn when the emotion changes (docs) |
| Google Cloud Gemini-TTS | [short pause] \250 ms, [medium pause] \500 ms, [long pause] 1000+ ms | [laughing] | style prompt | Emotion tags like [scared] are spoken aloud (docs) |
| Fish Audio S2 / Drama 3 | [pause] [long pause] | [inhale] [exhale] [sigh] [shaky breath] | [barely holding it together, forcing a smile], <emphasis>...</emphasis> | A tag changes everything after it, so place it at the turn (Fish) |
OpenAI gpt-4o-mini-tts | in instructions | in instructions | fields: Voice Affect, Tone, Pacing, Emotion, Pronunciation, Pauses (openai.fm) | Shuts down 2027-01-06 |
| Hume Octave | [pause], trailing_silence | under 100 characters: "frightened, rushed" | Rate goes in speed, not words |
On ElevenLabs, stability 0.30-0.50 gives emotional delivery that can wobble and 0.60-0.85 gives steady delivery that can go flat (docs). Spell out numbers, symbols and URLs before TTS, and fix names with IPA in slashes on v4.
What the direction did, measured
We ran one line in four versions on eleven_v4, 2 voices × 2 takes each (16 takes, $0.24 in all). Bare text was rushed. Each layer of direction slowed it and opened the pauses. No tag was read aloud, and all 16 transcripts matched the script.

Same line, four versions: bare, + punctuation, + breaths and sniffs, + emotion tags (voice: Jessica). The four versions in a row (a still from the clip); waveform on top, spectrogram below.
| Version | Text sent | Length | Words/min | Pause after "break it" | Loudness |
|---|---|---|---|---|---|
| Bare | I didn't mean to break it. I was only trying to fix the light before you got home. I'm sorry. | 5.7 s | 231 | 0.36 s | -26.5 LUFS |
| + punctuation | ... and — | 6.5 s | 207 | 0.51 s | -26.8 |
| + bodily actions | [exhales] [sniffs] [swallows] between sentences | 8.7 s | 194 | 0.92 s | -29.8 |
| + emotion tags | adds [quietly, ashamed] and [voice breaking] | 8.0 s | 200 | 0.89 s | -31.6 |
The "quietly" tag made the read 5 dB softer, so re-level lines after directing them. The same four versions in a male voice (Liam):

Same line, four versions (voice: Liam). The four versions in a row (a still from the clip); waveform on top, spectrogram below.
Keep one voice across shots
- Fixed voice block per character, repeated in every prompt: age, exact accent and what it is not, timbre, then a DELIVERY line for this moment. PJ Ace's: "27-year-old male, light casual working-class British accent, street-level London-adjacent ESTUARY, NOT cockney" (post).
- Clone and re-dub. Daubrez takes one clean line from the video model, loops it to about 10 s, clones it, mutes the baked audio and re-voices every line (post).
- Kling: bind the voice to the character element and don't describe its tone again (guide).
- Seedance voice reference: sources disagree. Kavan's method (via PJ Ace, May) wraps the voice in a black-screen video; PJ Ace in November uploads an ElevenLabs v4 mp3. Test both on your model version.
- Perform it yourself: act the line on video and prompt "Transfer the performance, movement and audio from Video 1 onto the the character Image 1. Do not add any music" (Uncanny Harry).
Consent
Clone only voices you have the right to use. ElevenLabs Professional clones are limited to your own verified voice; Instant clones require you to confirm consent (docs). Gemini voice replication needs a consent recording from the same speaker. YouTube asks for an AI label when a real person seems to say something they didn't (policy).
Dialogue and narration: write for the ear, then leave room
Write fewer, shorter, spoken lines, record them before any picture, and let pauses, breaths and small actions carry the feeling. Adjectives in the prompt do less than a sigh in the right place.
Build the voice track first
Ken Burns' editor Craig Mellish screens a "radio cut", narration and voices only, before cutting picture (interview). Baby Driver began as a 100-minute radio play (Paul Machliss). For an agent, this means: voice lines first, word timestamps from them, and every cut lands on a word or a pause, never inside a word.
Write for the ear
- Plain spoken words, one idea per sentence, contractions. "the car", not "the vehicle" (NPR's Marielle Segarra).
- Write to the breath. "Have an idea of how much your host can actually say in one breath, and write to their breath." Alternate short and long sentences; long vowels and l, m, n, r slow the read (CBC's Chris Wodskou).
- Round numbers. "nearly two thousand people", not "1,988 people" (Media Helping Media).
- Cut words. "Having written a commentary, the first thing to do is to carefully see what you can get rid of" (David Attenborough). Randy Thom warns that wall-to-wall dialogue leaves no room for sound.
- Write disfluency in when realism matters. Google's TTS docs: write the text "as a real spoken transcript", e.g. "Oh uh yeah I think... hm, so that's interesting" (docs).
Pauses, breaths and small actions
- Give lines room. Fewer lines per generation. After a key line, hold 1-3 s with only ambience so the character listens (Randy Thom).
- Keep the breath before a thought, don't stuff every gap. NPR's voice coach: "When you start a thought with a full, supported breath, your speech has more energy" (NPR). Thom: gaps filled with "grunts, groans, and breathing" kill the track.
- Prompt behaviour, not emotion. "Never prompt emotions. Prompt behavior. Type "she is afraid" and you get the cliche version of afraid." (PJ Ace). In TTS, a reaction tag does the same job: "[sigh] [sad] I just don't know anymore. The sigh grounds the emotion physically" (Fish Audio).
- Cut-offs and trailing off. A double dash means someone is cut off; an ellipsis means the line trails away (John August). ElevenLabs dialogue: end the first line with a dash and start the next with
[jumping in]or[overlapping]. Gemini puts a listener's "oh hmm" inside the speaker's turn:So the launch is Thursday |oh hmm| Are we actually ready? - Record the exchange together. Actors who hear each other sound more real (Jesse Harlin). For TTS, generate a two-person exchange in one dialogue call (ElevenLabs Text to Dialogue takes up to 10 voices and 2,000 characters), or pass
previous_textandnext_textwhen you stitch separate takes. - A sound before the reaction. When a character reacts to a sound, Johnnie Burn puts the sound about half a second before the reaction (interview).
A line sheet the agent keeps, one row per line:
| id | who | text exactly as sent | direction | after | shot |
| L01 | MIA | [exhales] I didn't mean to break it... | ashamed, to her brother | 0.9 s | s02 |
| L02 | MIA | [sniffs] I was only trying to fix the light — before you got home. | quieter, avoids his eyes | 0.6 s | s02 |
| L03 | LEO | [jumping in] —You were trying to fix it? | dry, a small smile | 1.5 s hold | s03 |
Dialogue inside video models
| Model | Write the line like this | Sound rules |
|---|---|---|
| Seedance 2.x | {Really, sis?} then DELIVERY: understated, mildly incredulous... No shouting. (PJ Ace) | Seedance 2.5 always outputs audio (Together AI). Open with "NO MUSIC... Use only diegetic sound effects, ambience, dialogue, and foley written inside <>" and list the sounds in <...>. Two clipped lines across a cut beat one long line (fal) |
| Kling 3.0 | [Black-suited Agent, raspy, deep voice]: "Don't move." with the action before the line; Immediately, between speakers or the lines merge (fal / Kling rules) | 10 s clips and at most 2 speakers work best (Kling 2.6 guide) |
| Veo 3.1 | Man: (Hand on his hunting knife) "That's no ordinary bear." | SFX: and Ambient noise: as separate cues; timestamped segments for multi-shot (Google Cloud) |
| Sora 2 | a Dialogue: block under the visual prose, one label per speaker | A 4 s shot holds one or two short exchanges |
| Any model, lip-sync later | prompt "the character should be speaking naturally" so there is mouth movement (sync.) | Feed the lip-sync model one clean voice, no music |
If the native voice drifts or sounds off, keep the take and replace the voice: clone a clean line, re-voice, re-sync (Daubrez), or run the clip audio through a voice changer to one target voice (MetaPuppet). Check every generated line against the script by transcript: models add gibberish words.
Sound effects: few, exact, on the right frame
Pros use far fewer effects than agents do. In measured launch films about 85% of hard cuts have no sound of their own and there are about 2 whooshes per film (Cinetic). earshot found that a strong accent lands on only 4-25% of cuts (study of 8 films). Every effect needs a visible cause, lands by its loudest point, and sits in its own band. "Few and exact: every effect has a cause on screen, and too many read as a template" (HyperFrames).
What gets a sound, and when
| On screen | Sound | Timing | Source |
|---|---|---|---|
| A real cursor click | short click | 10-30 ms before the visible press | Cinetic, earshot (measured) |
| Text typing in | high-passed noise ticks at 3-12 kHz, nothing under 2 kHz, about 87 ms apart, level and timing jittered | with each character | earshot (measured) |
| A badge, chip or number landing | low thump at 80-150 Hz with a faint tick | 25-40 ms after its cut | earshot (measured) |
| A real scene change | one soft whoosh; big whooshes only for big moments | peak within about ±150 ms of the cut | earshot (measured), Nate Curtiss |
| The payoff or the logo | riser into impact, once per film | the music stops on a bar line, 0.2-0.7 s of near-silence, then one soft element on the wordmark, a 0.5-2.5 s tail | Cinetic |
Sources disagree on the lead: HyperFrames puts a peak one frame after its picture, OpenMontage starts a whoosh 10-20 ms before, earshot and Cinetic measured onsets 100-450 ms ahead with the apex on the visual peak. All agree the loudest point belongs on the event. Trailer editor Derek Lieu adds that a hit two frames off sometimes looks better, so measure first, then let a person nudge.
Layer a scene the way film sound designers do
- At most 5 layers in any 5 s, spread across dialogue, music and different kinds of effects; two-and-a-half of one kind blur (Walter Murch).
- Room tone under everything. "10 seconds of room tone is IMPORTANT AF!" (Braden Whiteside, animation studio sound designer). Dead digital silence between lines sounds like a fault.
- Show a sound once, then keep it off screen. After one close shot of the machine, its sound can carry the rest of the scene (Randy Thom).
- Distance changes the sound. Far sounds lose bass and top end and gain reflections; Johnnie Burn adds a quieter, echoing copy wherever the frame shows a wall (interview).
- Build hero sounds from layers of real sources. "Any sound is made up of four sounds, generally" (Lon Bender on the Netflix ta-dum, 20k). Lucas told Ben Burtt to record motors and animals rather than reach for synths. Mangini sweetened Fury Road's trucks with whale moans.
- Keep effects readable. Stagger hits so no two big ones start on the same frame and put them in different bands (Gary Rydstrom). UI and transition sounds stay out of 300 Hz-1 kHz, where the voice and the music's chords live (earshot).
Prompt and generate effects
ElevenLabs SFX v3 (eleven_sound_effects_v3 on varg): "describe the action, setting, atmosphere and performance you want rather than a broad sound category": "Footsteps on gravel approaching slowly". Chain events with "then". Counts like "triangle hit 4 times" are not followed, so make one hit and repeat it yourself. Set loop for ambience beds, and raise prompt_influence above its 0.3 default for literal UI sounds (docs).
We generated five effects ($0.07). One came back unusable:
| Prompt | Asked | Came back |
|---|---|---|
| "Single crisp click on a laptop trackpad, close-mic'd, dry, quiet room, one-shot" | 1 s | near silence, peak -54 dBFS |
| "Loud, crisp computer mouse click, close-up, one-shot" | model picks | 0.48 s, peak -9.3 dBTP: usable |
| "Fast airy whoosh that rises and swells into a hit, clean and modern, close, one-shot" | 2 s | 0.40 s of silent lead-in, loudest point at 1.21 s |
| "Deep cinematic impact: a punchy low boom with a short sub-bass drop and a soft airy tail, one-shot" | 3 s | starts at 0.00 s, 2.6 s of sound |
| "Quiet modern studio room tone, soft air conditioning hum..." (loop) | 10 s | -53.5 LUFS, needed +18 dB |
So measure every effect before placing it, and place a whoosh by its peak: start = hit time - 1.21 s.
Inside video models. Veo: SFX: thunder cracks in the distance and Ambient noise: the quiet hum of a starship bridge as separate cues (Google Cloud). Kling: [Object: Wooden Door] suddenly [Action: Slams] + [Sound Effect: Bang] (Kling 2.6 guide). Seedance: list every sound in <...> and ban generic transition sounds: "Also no "whoosh" or "zoom" sound effects" (PJ Ace). For a phone-video look: "phone-mic quality... No music, narrator, or cinematic SFX".
Foley from the picture. MMAudio works on 8 s windows and syncs to the centre square of the frame; it can mumble speech-like sounds and its weights are non-commercial (README). A video-only foley model with no text "stays almost silent" (Heather Cooper), so describe the sounds too.
Library or code? HyperFrames retrieves only CC0 and CC BY sounds and bans synthesized tones. earshot and Cinetic synthesize sounds in code, yet list "code-synthesized beeps, sine pings" among the cheap tells and ask for a loudness-matched human blind test. Use code-made sounds for UI ticks and pops; use recordings or generated sounds for anything that should feel physical. Check licences: Freesound CC BY-NC bans money-making use; Sonniss GDC sounds may not be used to train AI.
The cheap tells (audit before delivery)
From earshot's checklist (source): code beeps and sine pings, one sample machine-gunned, a whoosh on every cut, a boom on every logo, reverb on tiny sounds, random pitch on typing, clicks at cuts, music chopped mid-note, an audible ambience loop, a different room on every sound, an ending that stops mid-note instead of a tail into silence.
Music: decide it, map it to the beats, finish it in the edit
Answer genre, mood, instruments, tempo and era yourself, plan where the music enters, stops and hits, then measure what came back. Make the stops, the drop and the exact length in the edit. In our test the model kept 120 BPM in 3 of 3 takes but never made the planned silence.
How much music, and where
- Dose it. Murch: "approach music in films the way you would approach administering drugs" (Web of Stories). In grounded drama the No Country team hid music "behind wind or car sounds" (Carter Burwell). Creator editor Hayden Hillier-Smith goes the other way: "I couldn't make anything without nailing the perfect soundtrack first" (Musicbed). Genre decides.
- Take it away once. Keep music under most of a piece, then drop it for the one serious moment. "When you fade music out in a piece, the silence has the subtle effect of spotlighting whatever comes directly after" (NPR's Michael May). Composer Jon Brantingham: sudden silence "can be the strongest kind of hit. Use it with caution" (guide).
- Beds that leave room for a voice. "Look for music that feels "incomplete," that has left sonic room for another element"; no vocals, nothing in the voice's range like trumpets. Aim at subtle moods, "pensive" or "hopeful", not "sad" or "happy" (May).
- Spot every cue. For each: start, stop, emotion, intensity, perspective, and "why this, why now?" (composers Coole and Nettleship). Pick only a few moments to catch; James Horner warns that dozens of "catches" kill a melody.
- Music first, or picture first. Trailer editor Derek Lieu: "You should do audio first." When dialogue sets the timing, cut the performances first and generate music to fit (Burak Tuyan).
Tempo, length and the beat map
- Tempo feel matters more than the number: "kick drums hitting on every beat or a repeated one-eighth piano note can make a track feel faster" (music supervisor Julia Trainor).
- Fast tempos push a slow scene forward; slow tempos "freeze time" and magnify small details (Brantingham).
- Measured pro launch films: music under 89-100% of the film, 108-136 BPM, median 123 (earshot).
- Work in bars: one bar of 4/4 lasts 240 / BPM seconds, so 2 s at 120 BPM. "120 BPM, 7 bars, something happens on every beat" is a launch-film brief with 21,165 saves (@twoclipping). In story pieces, cut on only one or two big changes per sequence or it "feels like a music video" (Noam Kroll).
- Pick the tempo that fits the hit points: Brantingham tests 117-123 BPM in 0.25 steps and keeps the one with the fewest frames off.
The beat map of our 20 s launch example, 120 BPM, one bar = 2 s:
| Time | Bars | Music | Voice and picture | Effects |
|---|---|---|---|---|
| 0-8 s | 1-4 | just a muted pulse and hi-hat | L01 at 0.5 s: "Your week... is already full." | click on the 4.0 s downbeat cut |
| 8-12 s | 5-6 | warm sub bass enters | L02 at 8.25 s rides the bass entry | |
| 12-13 s | half a bar | stopped on the bar line; room tone stays | the build holds | whoosh swells, peak at 13.0 s |
| 13 s | 7 | re-enters on the downbeat | logo lands | impact on 13.0 s |
| 14-20 s | 7-10 | groove, then rings out | L03 at 14.0 s: "This is Tide. Your week, handed back." |

Watch: the designed launch soundtrack, sound on (MP4)
The designed launch soundtrack: orange bars are voice lines, white lines effects, grey lines cuts. Sound on: waveform on top, spectrogram below; orange bars are voice lines, white lines effects, grey lines cuts.
Prompt the music model
| Model (varg id) | What works | Known failures |
|---|---|---|
ElevenLabs Music v2.5 (eleven_music_v2_5) | Answer the five questions: "French house, 122 BPM, filtered disco sample chops, sidechained bass...". Narrate the arrangement: "start with just a shuffled drum loop, add a warm sub bassline after four bars, then bring in...". "Without the just, the model fills the silence." BPM and key as numbers. The ElevenLabs API also takes a composition plan of up to 30 sections, each with its own duration (guide) | Naming an artist or song returns an error |
Lyria 3.5 (lyria_3_5) | [Genre & style] + [Mood] + [Instrumentation] + [Tempo & rhythm] + [Vocal style & language] + [Lyrics]; timed sections like [0:12 - 0:24] Verse 1: ... Intensity: 3/10; "Build to a beat drop at 12 seconds" (Google) | No negative prompt on 3.5; in our October run it ignored the asked length (20 s asked, about 90 s back) |
| Suno v6 | Use the lyrics field as a timeline of bracketed events: [A heavy, distorted 808 sub-bass drops in] ... [Fade to silence] (INK); Exclude field for unwanted parts | Every model before v6 is retired, so older tag tricks need re-testing; tempo can speed up across a song (composer Tomoya Kinoshita); an unwanted sound often comes from one style word, so delete it |
Mureka 9.5 (mureka_9_5) | instrumental from a text description up to 1,024 characters | The model picks the length |
| ACE-Step 1.5 (open) | bpm, key and duration as parameters, not in the caption | "Final result might be 118 or 122, not exactly 120" |
Our test on eleven_music_v2_5, 20 s asked, a detailed prompt written from the ElevenLabs guide, 3 takes at $0.05 each, measured with sound.py beats:
| Instruction | Followed |
|---|---|
| 120 BPM | 3 of 3 (120.0, 120.05, 119.97) |
| "start with just ... then bring in a warm sub bass" after four bars | 2 of 3 |
| "half a bar of complete silence", then an impact and a kick | 0 of 3 |
| 20 s length | 2 of 3 (one came back 29 s) |
librosa's own tempo estimate read 117.45 BPM for these 120 BPM takes, and its downbeat guess was half a bar off. The bass entry showed the real bar line. The average beat gap gives the true tempo; check downbeats against where instruments enter.
Edit music to length
- Cut whole bars, downbeat to downbeat (May). Cut whole sections, then remove the double downbeat at the splice (Brent Pierce).
- Button the ending, don't fade. Move the song's real ending to where the picture ends (Danny Greer).
sound.py fitdoes this: it keeps the opening and the real ending and removes bars from the middle. - If the picture is choreographed to the song, don't cut the song. "If you cut 2 seconds, the whole rest of the scene will suddenly drift" (Machliss). If the action runs long, loop a clean riff and hide the seam under effects.
- Trailer cues run in three acts with gaps between sections as edit points, and the ending does not resolve (Dan Graham, Sound On Sound). The music moves to a new section right after each piece of information lands (AV Squad).
- Under narration, drop the vocals:
demucs --two-stems=vocalskeeps the accompaniment. Convert MP3 to WAV before beat analysis, since decoders shift MP3 timing (All-In-One).
Licences
| Service | Commercial use |
|---|---|
| ElevenLabs Music | online and offline commercial use on paid plans, except film, TV, radio and studio games (terms); no artist, song or label names in prompts |
| Suno | only songs downloaded on a paid plan; free-plan songs belong to Suno; 20 or 60 downloads a month; fully AI music is not copyrightable in the US (help) |
| Udio | downloads disabled since the UMG deal |
| Lyria | Google claims no ownership but may make similar music for others; SynthID watermark |
| Mureka API | paid calls carry commercial authorization |
| TikTok ads | use the Commercial Music Library, not any trending song |
Mix and master: voice first, measured, checked on a phone
Mix the voice first and keep the music 10-20 dB under it where words live. Keep room tone under everything. Master social video to about -14 LUFS with true peak at or below -1 dBTP, then listen on a phone speaker. Evidence that sound adds views comes mostly from platforms that sell sound-on ads, so make the video work muted too.
Loudness targets
| Where it plays | Integrated loudness | True peak | Source |
|---|---|---|---|
| YouTube | no published number; turns loud videos down, never quiet ones up; -14 LUFS is the common reading | -1 dBTP | Ian Shepherd, YouTube Help |
| TikTok, Instagram Reels | no official spec; practitioners deliver -14 LUFS | -1 dBTP | OpenMontage |
| Spotify | -14 LUFS | -1 dBTP, or -2 if the master is louder than -14 | Spotify for Artists |
| Apple Podcasts | -16 LKFS ±1 | -1 dBFS | Apple |
| Streaming speech (AES TD1008) | -18 LUFS speech, -16 music | Ian Shepherd | |
| European broadcast | -23 LUFS | -1 dBTP | EBU R128 |
| US TV | -24 LKFS | Alex Knickerbocker | |
| Netflix | -27 LKFS ±2, measured on dialogue | -2 dBTP | Netflix spec v1.5 |
| Audiobook (ACX) | -23 to -18 dB RMS; noise floor under -60 dB | -3 dB peak | ACX |
Mastering engineer Ian Shepherd's advice: don't squash a mix to chase a number, since platforms normalise; keep -1 dBTP because "Lossy encoding for streaming or mp3 can also generate extra peak information above zero". earshot found that 6 of 7 pro launch films clip after AAC decoding, so don't copy their peaks. ffmpeg's loudnorm defaults to -24 LUFS and -2 dBTP, so always pass your own target.
Levels, ducking, space
- How far under the voice. Sources span 7-25 dB. WCAG AAA asks for background "at least 20 decibels lower" than speech (W3C). OpenMontage says 18-20 dB. HyperFrames' engine sets a bed under narration at 0.12, about -18 dB. The BBC once proposed music 4 dB lower than feels right; re-recording mixers called a blanket cut "a serious creative issue" (Broadcast). Start at 10 dB in 300 Hz-4 kHz and go deeper for dense explainers.
- Duck fast down, slow up. Derek Lieu fades the music down in about 4 frames as a line starts and back up over 20-30 frames (blog). HyperFrames bridges gaps under 0.6 s so the bed doesn't pump between words.
- Carve, don't only lower. Dip the bands the voice uses: "you can use your equalizer to pull out the high frequencies of the music so it doesn't clash with voice but stays present" (NPR's May). HyperFrames carves by default. Music editor Jim Harrison lowers only the stem that clashes. Murch sometimes removes the music for the one line that must land: "you do not hear it go away".
- Effects over the bed. In measured pro launch films, hits and clicks sit 13-17 dB over the quiet bed between beats (earshot). Agent prompts that keep every effect at or under the music make them vanish.
- Voice by symptom (HyperFrames mixing): rumble, high-pass at 80 Hz; boomy, -4 dB at 200 Hz; words unclear, +2.5 dB at 3 kHz or duck the music; harsh, -3 dB at 3.2 kHz; limiter last. A 4-6 dB level fall across a sentence is normal speech; levelling it "is heard as robotic".
- Place it. Dialogue in the centre; music and ambience wide (Murch). Check mono on a single speaker (Production Expert); listen on a phone, earbuds and a TV (BAFTA-winning mixer George Foulgham).
- No hard edges. "Pretty much, none of your audio clips should have a hard cut at the beginning or end" (Lieu). Ease into silence over a second or more, or it sounds like a fault; Murch uses full silence "three or four times in every film".
Designed vs plain, measured
Same three voice lines, both mastered to -14 LUFS, graded by sound.py check:
| Check | Designed | Plain |
|---|---|---|
| Loudness, true peak | -14.2 LUFS, -1.5 dBTP | -14.0 LUFS, -1.5 dBTP |
| Voice above music, 300 Hz-4 kHz, worst line | +15.9 dB | -6.6 dB: the music is louder than the voice |
| Effects on their moments | 3 of 3 within one frame | no effects |
| Cuts on the beat | 4 of 4 | 1 of 4 (44-125 ms off) |
| Dead air | none; room tone runs through the stop | 1.2 s of digital silence at the end |

Watch: the plain version of the launch soundtrack, sound on (MP4)
The plain version: vague music at full level, no ducking, no effects, no room tone. Sound on: waveform on top, spectrogram below; orange bars are voice lines, white lines effects, grey lines cuts.
Does sound add views?
No public study measures watch time for the same video with and without sound design. Every number we found was paid for by a platform that sells sound-on ads, or is a survey:
| Finding | Who and how |
|---|---|
| 88% of TikTok users call sound essential; 73% stop for ads with audio | TikTok-commissioned Kantar survey, 2021 (report) |
| 2.4x more attention to ads on sight-and-sound platforms | Nielsen lab study for Google, 400 people, 2016 (PDF) |
| About 80% of Reels watched with sound on | EssenceMediacom for Meta, sponsored, 2024 (Campaign) |
| 92% watch mobile video with sound off; captions lifted ad recall 8% | Verizon Media and Publicis survey, 2019 (Next TV) |
| A brand's own sonic cues beat borrowed music and celebrities | Ipsos meta-analysis of 2,000+ ads, independent (Ipsos) |
| Fury Road's preview score rose 8 points after a sound-only redesign | Mark Mangini, one film (BTL) |
| "I've seen multiple videos fail because the music was too loud, the music was too chaotic, or the sound came through distorted" | Mario Joos, retention director for MrBeast and others; opinion (post) |
What follows from it: build for sound on TikTok and Reels, caption for the sound-off viewer, own a recognisable sound instead of borrowing a song, and never let music bury words.
Doing it with code: the agent's sound toolkit
An agent can't hear, so give it data and checks. Keep every sound's time in a cue sheet, build the mix from it with a script, grade the mix with numbers, and render a picture of the sound to look at. The same steps serve a 100% code video (HyperFrames, Remotion) and a generated film (Seedance, Kling, Veo). "Agents can't listen. Flag music endings and loop points for a human ear-check" (Elie Steinbock's launch-film prompt).
The pipeline
- Voice first. Render the lines, get word timings from the same call (ElevenLabs
/with-timestamps) or a transcript, and let them size the timeline. Remotion's skill: generate TTS per scene, thencalculateMetadatasizes the composition to the audio (Remotion skills). - Music. Generate or pick, then
sound.py beatsfor tempo and bar lines,sound.py fitto cut it to length on downbeats, and snap the picture's cuts to the grid. - Effects. List the on-screen events, generate or retrieve one sound per event, measure each one, and give it an
attime (andalign: peakfor whooshes and risers). - Mix and grade.
sound.py mix cues.jsonplaces everything, ducks the music under the voice band, adds room tone and masters to the platform.sound.py checkgrades it and exits non-zero on a failure. - Look, then let a person listen.
sound.py previewmakes a video of the waveform and spectrogram with a playhead, voice lines, effects and cuts marked. A person listens on headphones and a phone. Thensound.py muxputs the mix on the picture. - Generated films: Seedance 2.5 always outputs audio. Keep good native dialogue, strip the rest, and rebuild music and effects on top; re-voice drifting lines and lip-sync with a clean, single voice.
The two scripts in the skill
| Command | What it does |
|---|---|
sound.py measure FILE | loudness, true peak, loudness range, silences |
sound.py beats MUSIC -o beats.json | BPM from the average beat gap, beats, estimated downbeats, strongest hits, quiet spans (librosa) |
sound.py fit MUSIC --to 30 -o bed.wav | keeps the opening and the real ending, removes whole bars from the middle |
sound.py mix cues.json | builds the mix and stems from the cue sheet; fast-down, slow-up ducking in the voice band; 30 ms fades on every clip; master to target with a 4x-oversampled limiter |
sound.py check MIX --cues cues.json | pass or fail: loudness, true peak, length, voice above music in 300 Hz-4 kHz, effects within one frame of their moment, cuts on the beat, no dead air, mono fold-down |
sound.py sheet / preview | the sound as a picture, or a video with a playhead |
audition.py plan / render / score / page | the same lines in several voices: renders through the varg API, measures pace, pauses and pitch spread, builds a page with a Pick button |
{ "duration": 20.0, "fps": 30, "platform": "youtube", "duck": { "depth_db": 8 },
"beats": "music/bed_beats.json", "cuts": [4.0, 8.0, 13.0, 16.0],
"silences": [{ "start": 12.0, "end": 13.0, "why": "stop before the logo hit" }],
"tracks": [
{ "id": "L01", "role": "voice", "file": "vo/L01.mp3", "start": 0.5 },
{ "id": "bed_a", "role": "music", "file": "music/bed.mp3", "start": 0, "length": 12.0, "gain_db": -4 },
{ "id": "bed_b", "role": "music", "file": "music/bed.mp3", "start": 13.0, "trim_start": 12.0, "length": 7.0, "gain_db": -4, "fade_out": 1.5 },
{ "id": "room", "role": "ambience", "file": "sfx/room.mp3", "loop": true, "gain_db": 18 },
{ "id": "whoosh", "role": "sfx", "file": "sfx/whoosh.mp3", "at": 13.0, "align": "peak" },
{ "id": "impact", "role": "sfx", "file": "sfx/impact.mp3", "start": 13.0 } ] }
Code-video specifics
- HyperFrames: "A film over about 15 s that tells something" gets a music bed and sounds on its visible actions (sound.md). Audio lives in the composition as
<audio>elements with a volume automation lane.npx hyperframes audio <file> --jsonreports where a sound peaks. One narrator per series: HyperFrames had to re-voice three docs films that fell back to a different voice. - Remotion: animate volume per frame with
interpolate();@remotion/sfxships attribution-free effects normalised to a -3 dB peak (docs). - Browser-only builds: render synthesized sounds with
Tone.Offline; browsers play nothing until a click, so add one quiet play button.
ffmpeg traps
amixhalves every input unless you setnormalize=0(FFmpeg).- Put
afadebeforeadelay, and useadelay=1500:all=1: withoutall=1a stereo file is delayed on the left channel only. loudnormdefaults to -24 LUFS and resamples to 192 kHz: setI,TPand-ar 48000, and run two passes with the measured values (Kyle Swanson, its author).- Convert MP3 to WAV before beat analysis. Whisper
.enmodels silently translate non-English speech (HyperFrames).
Casting candidates by code
- ElevenLabs Voice Design returns 3 previews per call; the same seed with the same inputs reproduces a voice.
- digitalsamba's toolkit scores 3 takes per sentence by pitch spread minus a penalty for pace outside 140-170 wpm, and never stretches a take below 0.9x. A flat clone comes from a flat reference clip, so fix the reference (CLAUDE.md).
- Reject bad takes with judges: word error rate from Whisper-large-v3 and WavLM speaker similarity (seed-tts-eval), UTMOS naturalness in two lines (SpeechMOS), DNSMOS
-pto catch a second voice. A 28-model TTS benchmark pulled its own quality score because it did not track human ranking: "Human votes remain the preference ground truth" (tts-bench).
Open models worth knowing
| Job | Model | Licence note |
|---|---|---|
| Narration, free and offline | Kokoro-82M | Apache-2.0; no cloning |
| Cloning | Chatterbox (27.5k stars) | MIT; watermark on every output |
| Two-speaker dialogue | Dia | Apache-2.0; a fixed seed does not hold the voice (open issue #109) |
| Directed delivery | Fish Audio Drama 3 | research licence, not OSI |
| Foley from video | MMAudio, HunyuanVideo-Foley | MMAudio weights are non-commercial |
| Songs | ACE-Step 1.5, YuE | MIT, Apache-2.0 |
| Stems | Demucs v4 | MIT |
| Word timings | whisperX, faster-whisper | BSD, MIT |
| Beats and downbeats | Beat This!, All-In-One | MIT |
F5-TTS, MusicGen, MMAudio, Higgs Audio v3 and madmom's model files are non-commercial: check before a paid project.
Ranking takes with an LLM judge, without bias
Use an LLM judge to shortlist takes and check the words, never to measure levels or to pick by taste. Write one yes/no question per known failure, let a script answer every question a script can, and compute the winner in code. Compare takes two at a time in both orders, and let a person make the final pick. In our test a vague "which sounds better?" judge chose the plain mix over the designed one every time it answered. It also picked confidently between two identical takes.
The criteria: one question per failure, one checker per question
Modern TTS "stopped failing on artifacts. It fails on subtle things now: prosody, breath placement, expressive overshoot" (harrrshall). So each criterion names one of those failures and has one checker. A criterion is clear when "two domain experts would independently reach the same pass/fail verdict" (Anthropic).
| What | Passes when | Checked by | Why that checker |
|---|---|---|---|
| Voice: words | the words match the script line, nothing added, dropped or said wrong | script (transcript diff), then the judge on names | Gemini judged pronunciation with "negligible hallucinations, leveraging Gemini's robust ASR" (EmergentTTS-Eval). A transcript lifts audio-judge pronunciation accuracy from 46% to 63% (AudioJudge) |
| Voice: tags | no stage direction ("sigh", "exhales") spoken as a word | script (transcript), the judge as backup | Speech models sometimes "directly read the words 'sigh' out loud" (Chiang et al.). It happened 0 times in 16 in our Eleven v4 test |
| Voice: ending | the last syllable is not cut off | the judge, on that take alone | Elie Steinbock's judge scores "pacing, mispronunciations, odd stresses and clipped endings" (post) |
| Voice: pace | 140-170 words a minute for narration (dialogue may sit outside on purpose) | script (audition.py score) | Audio judges told which of two clips was faster 47-55% of the time (AudioJudge). The window is from digitalsamba's pick_takes |
| Voice: pauses | each planned pause is there and at least as long as the cue says | script (silence detection) | Gemini judges were weaker on pause length (EmergentTTS-Eval); ours got 3 of 4 |
| Voice: pitch | pitch spread of at least 2 semitones | script (audition.py score) | "a spread under 2 semitones reads flat" (digitalsamba pick_takes) |
| Voice: delivery | sounds like the brief ("apologising, not reading") | the judge in pairs, both orders, as a shortlist; a person picks | The best audio judge "still only achieves less than 70% agreement with human preferences" on naturalness (SpeechJudge) |
| Dialogue in video | each line said by the right character, in order, nothing extra | the judge on that clip alone, with MM:SS times | Gemini's documented audio uses include transcription and speaker diarization (Gemini docs) |
| Lip sync | mouths match the words | a person, or a judge at a raised frame rate | Gemini samples video at 1 frame a second by default, and "fast action sequences might lose detail" (Gemini video docs) |
| Music: tempo, length | BPM and length match the plan | script (sound.py beats) | right 3 of 3 times in our test |
| Music: quiet part | the planned quiet section is there | script (loudness over time) | the model delivered it 0 of 3 times in our test |
| Music: vocals | no vocals in an instrumental bed | the judge: "describe what you hear", then a text check | "most are speech-trained and return a transcript, not the music" (wolfram) |
| Music: fit | fits the video's mood | a person | A music-quality predictor reached 0.926 rank correlation over whole systems but 0.633 per clip (MusicEval). Audiobox Aesthetics predicted people's pick 62.3% of the time; chance is 50% (Zhang et al.) |
| Effect: there and on time | not silent, its peak within 1 frame of the cue | script (sound.py check) | 1 of 5 effects came back silent in our test |
| Effect: identity | it is the sound asked for | the judge: describe first, compare in code | Audio LLMs "tend to give affirmative answers" to "is there a sound of X?" (Kuan et al.) |
| Mix: levels | platform loudness ±1 LUFS, true peak, voice 10 dB or more above music in 300 Hz-4 kHz during every line | script (sound.py check) | Judges kept 73-93% of verdicts unchanged at 1 dB signal-to-noise (AudioJudge): they hear words through noise you want flagged. Ours misread the plain mix's levels |
| Mix: feel | the soundtrack carries the video | a person | no judge in our sources was validated for this |
Our test: an audio judge on our own soundtracks
We used varg's analyze_video, which runs Gemini 3.8 Flash on a video and hears its soundtrack: 30 calls, $1.50. Each pair played as one video with neutral "CLIP A" / "CLIP B" cards, in both orders. We scored it against measured truth on three pairs. The first was the designed vs the plain launch mix. The second was a bare vs a breath-tagged take, for two voices. The control was two takes of the same line.
| Setup | Mix: designed vs plain | Voices: bare vs tagged (2 pairs) | Control: two near-identical takes |
|---|---|---|---|
| Vague question, 2 runs x 2 orders | the plain mix every time it answered (3 of 3; once no answer) | the tagged take 7 of 8; once "B" in both orders (the slot, not the take) | a confident pick every time; take 2 in run 1, take 1 in run 2 |
| 5 yes/no questions, winner by a fixed rule | right in one order, tie in the other | right in one order each, ties in the other | tie in both orders |
| The same questions, each clip alone | designed 4 of 5 answers right, plain 1 of 5 | pause question 3 of 4 right | not needed |
- Its reasons are not evidence. On the control it explained "the slightly mechanical delivery of Clip A". It reported a 54-second timeline for a 41.5-second video, and "chuckling" that is not there.
- The first clip leaks into the second. 6 of 30 answers about the same clip changed with its position. Both pause errors came when the bare take played right after a tagged one.
- The rubric removed every wrong winner, and decided nothing on its own. Each real pair was right in one order and a tie in the other, so under the both-orders rule all three are ties. That is safe, but the ranking then has to come from the scripts and a person.
- It may hear less than you think. Gemini's docs say audio inside a video is processed at 1 Kbps on one channel, against 16 Kbps for an audio file, and all channels are mixed to mono. That may explain the misread levels. It also means it cannot judge stereo width.
Twelve rules for an unbiased judge
- Write criteria from real failures, one judge per failure. "In practice, we recommend building a separate judge for each failure mode" (Hamel Husain). Grade a batch of real takes before you write the prompt: people's criteria change as they grade (Shankar et al.). Lotte Verheyden says 10-20 labelled cases are enough to build a judge (post); Eugene Yan found 20 too noisy and labels 50-100 (AlignEval).
- Ask yes/no questions, never for a 1-5 score. "The difference between adjacent points (like 3 vs 4) is subjective" (Husain and Shankar). Splitting criteria into yes/no checklist questions raised agreement across judge models by 0.45 (CheckEval). Each question names one thing and what to ignore: "The response cites at least one source document. Ignore formatting issues" (Verheyden).
- Ask the judge to describe, not to confirm. Audio models answer "yes" to "is there a sound of X?" when X is missing (Kuan et al.). Ask for a transcript and a list of non-word sounds with MM:SS times, and check them in code. Allow "unknown": "give the LLM a way out" (Anthropic). End the prompt as Woody does: "Don't call the sound good. Report what the checks show" (post).
- Reason first, but don't expect reasoning to remove bias. "Ask the judge to output its explanation prior to (instead of after) outputting the score" (Cameron Wolfe). For audio, a chain-of-thought rubric "just inverted the bias from always-B to always-A" (harrrshall).
- Compute the winner in code from the answers. Never make "which is better" the deciding question. In our test that change alone removed every wrong pick.
- Compare two at a time, in both orders, and count only verdicts that agree. Off-the-shelf audio models "pick whichever clip is in slot B, then write a confident rationale for why B was better" (harrrshall). The median model flips 45% of decisive pairs (Lech Mazur). Position bias grows when there are three or four options in one prompt (CALM), so never rank five takes in one call. "Call the judge twice with swapped positions. If evaluation results are inconsistent, then return False" (Jerry Liu). Google's autorater swaps half its pairs for this (Vertex AI).
- Judge each clip alone for yes/no checks; stitch pairs into one file. In our test the first clip leaked into answers about the second. For pairs, one file with both takes beat two uploads: pronunciation accuracy rose from 46% to 66% (AudioJudge).
- Make the takes comparable and hide where they came from. Every audio judge AudioJudge tested preferred the longer clip when people rated the two equal. Voice timbre shifts emotion verdicts (EmergentTTS-Eval), so judge delivery in the same voice. Judges also score "refined" answers higher without saying why (CALM). Name files CLIP A / CLIP B, never "v2" or "tagged".
- Split dimensions. Three narrow audio judges (words, delivery, quality) with a majority vote reached 0.912 rank correlation with people (AudioJudge). One judge scoring a voice "kept inferring tone from the words, inventing vocal cues that weren't there" and correlated 0.62 with people; scoring content and voice apart reached 0.78 (Neethu Joy).
- Repeat, vote, and send unstable calls to a person. A majority of 5 runs helped; averaging the 5 runs or keeping the best did not (Gu et al.). Sending only the 20% most order-unstable pairs to people matched human accuracy at 39% lower cost (Wang et al.).
- Judge with a different model family, or prove it is safe. One judge "marked 38% of its own failed unit tests as passed, vs 1.9% for other models' code" (Sword). Anthropic's docs call a different grading model best practice (docs). Husain and Shankar say the same model "is usually fine" if it matches human labels. Either way, the human-label check in rule 12 decides.
- Measure the judge against people, and against people's own agreement. Validate on about 100 labelled examples per failure, with both passes and fails (Husain). Report how many defects it catches, apart from how many good takes it passes. One judge passed over 95% of good outputs but caught only 30-60% of the bad ones (Eugene Yan). Use Cohen's kappa, not correlation: the same judge scored kappa 0.3-0.5 but correlation 0.8-0.9 (Yan). A judge "can agree with itself every time and still be wrong in the same direction every time" (DAIR.AI). Aim at the human ceiling, not 100%. Expert annotators agreed with the majority 71.7% of the time (Wang et al.). About 70% of TTS pairs had full or weak human agreement (SpeechJudge). People agreed on dialogue style at only r = 0.253 (Chiang et al.). Pin the judge model and log raw outputs. Check every claim it makes: "Video and audio judges hallucinate problems, so verify every claim against frames" (Steinbock).
Ranking five takes, step by step
- Filter by script. Drop any take whose transcript misses or adds words, reads a tag aloud, clips, or falls outside the pace window.
- Check each remaining take alone with the prompt below. A take passes when every answer is "no" and its sounds list has the breath the cue asks for. A take with an "unknown" goes to a person.
- Compare the survivors two at a time, in both orders, one question per call. Use one stitched file per pair. A win counts only when both orders agree; anything else is a tie. Five takes make 10 pairs, so 20 calls per question: about $1 at 5 credits a call on
analyze_video. - Rank by wins. Send pairs that split to a person.
- A person listens to the top 2-3 on headphones and on a phone, and picks. Their pick and reason become labels for the next check of the judge (rule 12).
You are checking ONE voice take. The script line is: "<line exactly as written>".
Listen to the whole take. Do not compare it with anything and do not say whether it is good.
1. Write down exactly what you hear, word for word.
2. List every sound that is not a word (breath, sigh, sniff, laugh, click, noise) with its MM:SS time, or "none".
Then for each question write one short reason, then answer yes, no or unknown (unknown when the audio does not tell you):
Q1 Is any word missing, added or changed compared with the script line?
Q2 Is a stage direction spoken as a word (for example "sigh", "exhales", "quietly")?
Q3 Is any name or brand word said wrongly? Give its MM:SS.
Q4 Is the last syllable cut off?
Ignore the accent, the voice's gender and any music. Don't call the take good. Report what you hear.
Return JSON: {"heard": "...", "sounds": [{"t": "MM:SS", "what": "..."}],
"Q1": {"reason": "...", "answer": "yes|no|unknown"}, "Q2": {...}, "Q3": {...}, "Q4": {...}}
This file holds two takes of the same line: "CLIP A", a short tone, then "CLIP B".
Question: <one narrow question, e.g. "Which take sounds more like someone apologising than reading?">
First describe each take in one sentence about this question only. Then answer A, B or tie.
Answer tie if you are not sure. Return JSON: {"A": "...", "B": "...", "answer": "A|B|tie"}
Run the second prompt again with the takes swapped. A win counts only when both runs name the same take.
The video-sound skill runs steps 2-4 with scripts/judge_takes.py. prompt prints both prompts, and check turns a one-take reply into pass, fail or review in code. make builds level-matched blind pair files in both orders, and tally counts a win only when both orders agree. The criteria, rules and test are in references/judge.md (vargHQ/skills #95). All 143 judge findings with their quotes (73 from X, 70 from papers and docs) and the raw judge replies are in vargHQ/aeo #42.
Variations by video type
The routine stays the same; what changes is who leads. Music leads a launch film and a music video, the voice leads an explainer and a UGC clip, the room and the silences lead a cinematic film, and a trailer alternates between them.
| Type | Leads | Music | Effects | Signature move |
|---|---|---|---|---|
| Launch / product film | music | under nearly all of it, 108-136 BPM | 3-6 accents: reveal, key UI action, logo | the music stops on a bar line before the logo |
| Cinematic AI film | room and silence | little, often hidden | off-screen, at real distances | silence before the big hit, 3-4 times a film |
| Music video | the song | the song, unedited | few, under the song | picture cut to the song, never the reverse |
| Trailer / ad | alternates | three acts with gaps | built from the film's own sounds | a line, then a hit |
| UGC / talking character | voice | none, or the room's | the room and the hands | phone-mic sound that matches the camera |
| Narrated explainer / code demo | voice | a carved bed | one big hit every 7-10 s | every cut on a word or a pause |
Launch and product films (code-made)
- Pick the track first and cut to it. Music runs under 89-100% of measured pro launch films, median 123 BPM (earshot). Lock tempo and bar count before animating.
- Accents only where they matter: reveal, logo, key UI action. "Sound design is not about about making every movement audible. It's about deciding what deserves the viewer's attention"; UI sounds must fit about 0.5 s (bruno, launch-film sound designer). In a dense product video, "music and sound design must never fight it for attention".
- No silent cold open: pro films start sound by 0.13 s, at body level (earshot).
- End on the logo the Cinetic way: music stops or filters down on a bar line, 0.2-0.7 s of near-silence, one soft element on the wordmark, a 0.5-2.5 s tail. A sonic logo stays under about 3 s (Netflix's "it had to be really short") and can open with a short burst that "closes off whatever sound was there before" (Intel's composer Walter Werzowa, Globe and Mail).
- One narrator for the whole series, at 145-155 words per minute with music about -31 LUFS under it (HyperFrames' docs voice).
Cinematic AI films
- Generate with natural sound only, add music in post. "NO MUSIC NATURAL SOUND ONLY"; "By removing AI music from the generation, he gets the sounds of fire crackling, footsteps, and sword slashes" (Kavan Cardoza's method via PJ Ace).
- Choose whose ears we are in. "Most of the great sound sequences in films are "pov" sequences" (Randy Thom). Build each place from 2-3 ambience layers that come and go; on No Country "the winds were the score" (Craig Berkey).
- Build tension by taking sound away. Underplay backgrounds and footsteps before a shock; a string of small off-screen sounds leads to the reveal (Berkey). "Loud sounds like explosions are more startling and effective if they're preceded by a little silence" (Richard King). A sudden drop to silence signals danger (Murch).
- Music like a drug. Hide drones behind wind; for a tragic beat try cheerful, indifferent source music (Chion's "anempathetic" music). Make one theme in many versions rather than many themes (Ken Burns' team records "10, 15, 20 versions of it").
- Voices. The AI feature now in US cinemas used eleven human voice actors, four composers and a foley artist (Gossip Goblin). Without actors: clone a clean line and re-dub, and give each character a fixed voice block.
Music videos
- The song is the clock. Name the uploaded audio as the master timing source: "Synchronize cuts, breath, gaze, gestures and type to the actual audio. Do not invent a different song or singer" (prompt).
- Don't edit the song; cut the picture to it. Machliss kept the song "complete and unedited" about 95% of the time. Action takes longer than you plan: "you put four bars of music but it takes six".
- Feed lip-sync a clean vocal. Separate the vocal stem with Demucs; "Background music, crowd noise, and overlapping conversations degrade lip sync accuracy" (sync.). When a performer should not sing, say so: "no miming the lyrics, no lip-sync" (sandmrk).
- Long songs in chunks. Split into 15 s pieces that overlap by 1 s so the clips stitch cleanly (techhalla).
Trailers and ads
- Three acts, gaps between them, an ending that doesn't resolve (Dan Graham, Gothic Storm). A line of dialogue "is followed immediately by a sound effect or big beat in the music cue that punctuates the story beats" (Derek Lieu).
- Make the rhythm from the film's own sounds. Mark Woollen built The Social Network trailer's pulse from a head against a chalkboard (NPR). AV Squad builds drums from effects, plays a sound before its picture and repeats it one measure later with the picture, and moves the music on after each piece of information.
- Ads: pick your moments. "You really have to pick your moments to shine and pick your moments to get out of the way" (Kip Smedley). Finish the longest cut first, then the 30 s and the 6-7 s punch line (Tom Disher). A brand's own sound beats a borrowed song (Ipsos).
UGC and talking characters
- Match the sound to the camera. "AUDIO: Natural Japanese dialogue with accurate lip-sync, phone-mic quality, footsteps, traffic... No music, narrator, or cinematic SFX" (prompt, 151k views).
- Generate the voice separately, then put its exact script into the video prompt, so the model mouths what you recorded (ZssBecker, 3,135 saves). Lock the audio over the whole clip.
- Short lines per shot: one 3-6 word line per 2-3 s shot, and "No dialogue, breathing and distant gym sounds" where nobody talks (Alpha Mom).
- Speech clarity matters for retention; an accent does not (Mario Joos). Label realistic AI people as required by the platform.
Narrated explainers and code demos
- Record the narration first and cut to the words. Every cut lands on a word or a pause.
- A carved bed, deeper ducking. Up to 22 dB of ducking for complex educational topics (OpenMontage); streaming speech targets -18 LUFS (AES).
- Sparse hits on words. "only about 1 big hit every 7-10 s, each one downloaded and trimmed so its peak lands on its word" and "The narrator never says a sound word; the SFX carry every sound" (finnardley).
- Mark the end of the intro with a sound or a change of tone so viewers stop judging and stay (Mario Joos).
The skill and the copy-paste prompt
Everything above ships as the video-sound skill (GitHub, merged in vargHQ/skills #79) and as a varg how to card (varg.ai/how-to/skills/video-sound). The card is #1 in how-to search for "make my AI voice sound real" and "add sound design to my video".
| Agent | Install |
|---|---|
| Claude Code | claude plugin marketplace add vargHQ/skills && claude plugin install varg@varg |
| Codex | npx skills add vargHQ/skills --skill video-sound -a codex -y |
| Cursor | npx skills add vargHQ/skills --skill video-sound -a cursor -y |
| Gemini CLI | npx skills add vargHQ/skills --skill video-sound -a gemini-cli -y |
What is in it: SKILL.md (the routine with STOPs), scripts/sound.py and scripts/audition.py, one reference file per section of this guide, references/evidence.md with all 642 findings and their quotes, and examples/launch-20s/, which rebuilds the designed and plain soundtracks from this guide with one fetch.sh. It hands off to video-script (lines and the sound column of script.md) and character-passport (each character's voice block).
The full prompt
The whole skill as one prompt, for any agent with a shell. Replace the brief line:
You are the sound designer, voice director and music editor for this video. You cannot hear, so you plan in writing,
measure everything with numbers, and a person listens at the STOPs. Never say you heard something.
Brief: {what the video is, type (launch / cinematic / music video / trailer or ad / UGC / explainer), platform,
length, who speaks, budget cap in dollars}
If the video-sound skill is installed, follow it. Otherwise fetch its scripts and references first:
https://github.com/vargHQ/skills/tree/main/video-sound
1. Sound plan before picture. Write sound/plan.md: one row per beat with time, picture, voice line, music, effects,
ambience and planned silence. For each music cue: start, stop, emotion, intensity, why now. Pick only 3-6 moments
that get an accent and one silence. Never more than 5 sound layers in any 5 seconds. Say what leads: music,
voice, or room and silence.
2. Lines for the ear. Write sound/lines.md: id, speaker, text exactly as sent, direction, pause after, shot. Plain
spoken words, one idea per sentence, one breath per sentence; cut every line the picture already says. Direct
with behaviour, not emotion words. Mark pauses, breaths and small actions with the model's documented tags
(Eleven v4: ..., line breaks, [pause], [sighs], [exhales], [whispers], a bracketed stage note; no SSML <break>).
STOP: show plan.md and lines.md.
3. Cast from candidates. Render the hardest line in 3-5 voices, 2 takes each. Measure every take (length, words
per minute, pauses, pitch spread; transcript must match) and build one audition page. Quote the price first.
STOP: I listen and pick.
4. Record all lines in the chosen voice. Transcribe with word timestamps, re-render only wrong lines, keep quiet
breaths. Every cut lands on a word boundary or a pause.
5. Music. Prompt genre, mood, instruments, BPM and key as numbers, and era; narrate the arrangement ("start with just
... then bring in ..."); never name an artist or song. Measure what came back, fit it to length on downbeats
keeping the real ending, and make stops, drops and exact length in the edit.
6. Effects only for events with a visible cause; about 2 whooshes per film. Measure every file; align impacts by
onset and whooshes by their loudest point. Room tone under everything.
7. Mix and check from sound/cues.json: music ducked hardest in 300 Hz-4 kHz, fast down and slow up, faded edges,
mastered to -14 LUFS / -1 dBTP for social. Fix every failure: loudness, true peak, voice 10 dB over music in the
speech band, effects within one frame, cuts on the beat, no dead air, mono safe. Audit the cheap tells.
8. STOP: make a preview video of the sound (waveform, spectrogram, playhead) and the muxed video. I listen on
headphones and a phone. Apply my notes, re-check, deliver.
Never: an effect on every cut, music louder than words, lyrics under narration, a fade-out ending, dead digital
silence, cloning a voice without consent, claiming you listened.
Sources
All 642 findings, each with its verbatim quote, author, credentials and link, are in the skill's references/evidence.md. The raw research, the six reports, the expert roster with engagement flags and the test files are in vargHQ/aeo #22. The pages this guide quotes most:
Film, trailer, ad and radio craft
- Walter Murch: Transom interview, FTII interview, "Stretching Sound to Help the Mind See", worldizing, Web of Stories
- Randy Thom, Designing a Movie for Sound; Ben Burtt and J.W. Rinzler on NPR; Gary Rydstrom in Holman's Surround Sound
- No Country for Old Men team on Designing Sound and Carter Burwell; Richard King on Inception and silence; Mark Mangini on Fury Road; Johnnie Burn on The Zone of Interest; Michel Chion, Audio-Vision ch. 1
- Paul Machliss on Baby Driver; Craig Mellish on Ken Burns' films; Derek Lieu's editing tidbits; AV Squad via Jonny Elwyn; Dan Graham in Sound On Sound; Netflix ta-dum on Twenty Thousand Hertz; A Sound Effect on ads
- NPR Training on music in audio stories and voice; CBC's writing for radio; Khris Brown on voice direction
Model and platform docs
- ElevenLabs: TTS best practices, Eleven v4, Voice Design, Sound effects, Music best practices, Music terms
- Google: Gemini speech generation, Lyria prompt guide, Veo 3.1 prompting; Kling 3.0 and 2.6 audio guides; Seedance 2.5 on Together; Fish Audio tags; openai.fm
- Loudness: EBU R128, Netflix mix spec, Spotify, Apple Podcasts, Ian Shepherd on online loudness and YouTube, WCAG 1.4.7, ffmpeg loudnorm
Agent tools and repos
- HyperFrames sound.md and mixing.md; earshot's launch-film study and motion SFX study; Cinetic sound; OpenMontage sound design; Remotion skills; digitalsamba toolkit
Practitioners on X (verified credits): Henry Daubrez, PJ Ace, Uncanny Harry, Mac Smith, Gossip Goblin, MetaPuppet, bruno (tvnxty), Mario Joos
LLM-as-a-judge
- Audio and speech judges: AudioJudge, SpeechJudge, EmergentTTS-Eval, Chiang et al., speaking-style judges, Kuan et al., audio hallucination, MusicEval, Zhang et al., music preference, Gemini audio and video docs
- Bias and rubrics: CALM, CheckEval, Wang et al., fair evaluators, Gu et al., LLM-as-a-judge survey, Shankar et al., who validates the validators, Vertex AI autorater
- Calibration: Hamel Husain (judge guide, evals FAQ with Shreya Shankar), Eugene Yan (LLM evaluators, AlignEval), Anthropic (agent evals, grading docs)
- On X: Hamel Husain, Lotte Verheyden, harrrshall, Lech Mazur, Jerry Liu, Cameron Wolfe, Neethu Joy, Sword, DAIR.AI, Elie Steinbock, Woody, wolfram
Our tests (2026-11-03, about $0.60 on varg): the voice line test, the music prompt test, five generated effects and the designed vs plain launch soundtrack. Files and numbers: RESULTS.md; the soundtrack rebuilds from examples/launch-20s.
Go deeper
- The skill: video-sound (
npx skills add vargHQ/skills --skill video-sound) - The designed version of this guide, with every picture and clip: https://varg.ai/how-to/design-sound-and-voice-for-ai-video
- Every claim of the ten steps traced to its source: the evidence bank
All 10 steps
The route, with the one line that hands it to an agent: How to make a video with an AI agent in ten steps
- How to find references for an AI video
- How to make a moodboard for an AI video
- How to write a video script with an AI agent
- How to design sound and voice for an AI video (this page)
- How to design characters and props for AI video
- How to previz a video in Three.js or Blender
- How to animate a previz with AI video models
- How to edit a video with HyperFrames and AI agents
- How to judge an AI video: measure, don't grade
- How to keep an AI video project in GitHub