How to design sound and voice for an AI video

For AI agents: find the right skill in one call — curl -s "https://varg.ai/how-to/api/search?q=<what the person wants>" returns the matching how-to page and the best skills with install commands, as markdown (Russian works too). Full index: https://varg.ai/how-to/llms.txt. Every page has a markdown twin: add .md to its path. Before generating anything, show the person the options on the how-to page. From a terminal: curl -fsSL https://varg.ai/install | sh installs the vargai CLI (https://varg.ai/how-to/cli.md): vargai search and vargai install <slug> with no key, every varg model after vargai login.

A video brief goes in (type, platform, length, who speaks); a sound plan, cast voices, recorded lines, music, effects and a mix checked by script come out. On the same voice lines the designed soundtrack passed 8 of 8 checks, while the plain one had the music 6.6 dB louder than the voice where words live.

Step 4 of 10 in How to make a video with an AI agent in ten steps. Before: How to write a video script with an AI agent. Next: How to design characters and props for AI video.

For your agent

Install the skill, then paste the prompt.

npx skills add vargHQ/skills --skill video-sound
Use the video-sound skill. You are the sound designer, voice director and music editor for this video.
Brief: {what the video is, type (launch / cinematic / music video / trailer or ad / UGC / explainer), platform,
length, who speaks, budget cap in dollars}
You cannot hear, so plan in writing, measure everything with numbers, and never say you heard something.
Stop where it says STOP: after the sound plan and lines, after the audition page, and before delivery.

The full prompt, which works without the skill installed, is under "The full prompt" below.

The short version

Sound and voice for AI video

Guide · November 3, 2026

How film sound designers, trailer editors, radio producers and voice directors make sound, turned into a routine an AI agent can follow: real-sounding voices, dialogue with pauses and breaths, effects on the right frame, music mapped to the beats, and a mix checked by script.

NumberWhat it measures
~85%of hard cuts in pro launch films get no sound of their own
231 → 194words a minute: bare text vs breath-tagged on Eleven v4
0 of 3music takes made the planned silence and hit
8/8 vs 5/8checks passed: designed vs plain soundtrack

The short answer

Make the sound before the picture, use fewer sounds than feels natural, and let a person's ears decide what numbers can't. An agent can't hear, so it plans in writing, measures every file, and stops for a listener at three points. This routine works for any video, whether made in code (HyperFrames, Remotion) or generated (Seedance, Kling, Veo).

  1. Brief. Video type, platform and its loudness target, length, who speaks, music or not. The type decides what leads: music (launch, music video), the voice (explainer, UGC), or the room and its silences (cinematic).
  2. Sound plan before any picture. One row per beat: voice, music, effects, ambience, planned silence. For each music cue: start, stop, emotion, "why now". Pick only 3-6 moments that get an accent, and at most 5 layers in any 5 s (Walter Murch).
  3. Lines for the ear. Short spoken sentences, one breath each, behaviour instead of emotion words. Pauses, breaths and small actions go in the TTS model's own tags. Cut every line the picture already says. STOP: the person approves.
  4. Cast from candidates. The hardest line in 3-5 voices, 2 takes each, measured and put on one page. STOP: the person listens and picks.
  5. Record every line, transcribe it, re-render the wrong ones. The word times set the cuts.
  6. Music. Prompt genre, mood, instruments, BPM, key and era, and narrate the arrangement. Measure what came back, fit it to length on downbeats, and make the stop and the drop in the edit.
  7. Effects only where something happens on screen. Measure every file. Impacts land by their start, whooshes by their peak. Room tone runs under everything.
  8. Mix from a cue sheet and check it by script: voice 10 dB or more over the music where words live, effects within one frame, cuts on the beat, no dead air, -14 LUFS and -1 dBTP for social.
  9. STOP: the person listens on headphones and a phone, then you fix their notes.

What the evidence and our tests say most clearly:

The guide covers six video types: launch films, cinematic AI films, music videos, trailers and ads, UGC and talking characters, and narrated explainers and code demos. Those six cover nearly every video an agent is asked to make, and each puts a different layer in front. Everything ships as the video-sound skill, a varg how to card and one copy-paste prompt (see "The skill and the copy-paste prompt").

Who we learned from

Every rule in this guide comes from a person with checkable credits, an official model guide, or our own measured test. We did not invent any of it. We read 81 interviews and essays by film, trailer, radio and ad sound people, profiled 57 candidates on X and kept 20, and re-read the 160 earlier sound tips from X.

WhoCreditsWhat we took
Walter MurchEditor and sound designer: Apocalypse Now, The Godfather, The English Patient; 3 OscarsAt most 5 layers in any 5 s; silence 3-4 times a film; pre-laps; music "like drugs"
Randy ThomDirector of sound design, Skywalker Sound: Forrest Gump, Cast AwayPoint-of-view sound; show a sound once, then keep it off screen; let characters listen
Ben BurttStar Wars, E.T., WALL-EReal recordings over synths; R2-D2 is 50% human voice
Richard King4 Oscars: Dunkirk, Inception, The Dark KnightA little silence before every big hit
Johnnie BurnOscar 2024, The Zone of InterestA sound about 0.5 s before the reaction; distance takes bass and top end
Mark ManginiOscars for Mad Max: Fury Road and DuneSpotting sessions; the sound redesign alone raised test scores 8 points
Craig Berkey, Skip Lievsay, Carter BurwellNo Country for Old MenUnderplay before a shock; ambience as score; music hidden behind wind
Mac SmithEmmy-winning supervising sound editor, Skywalker SoundBring sound in at the script; don't over-clean dialogue
Michael May, NPR TrainingScored features for NPR and Studio 360Beds that leave room for the voice; cut music by whole bars
Jessica Hansen, NPRNPR announcer and voice coachA full breath at the start of a thought
Khris BrownVoice director, launched the LucasArts voice departmentCast the voice that makes you feel; no voice beats a bad voice
Paul MachlissEditor, Baby Driver (BAFTA)A 100-minute radio play before picture; don't edit the song
Craig MellishKen Burns' editor, 2 Emmy noms for soundA radio cut first; final music from day one
Derek LieuGame trailer editorDuck music 4 frames down, 20-30 frames up; no hard audio cuts
Tommy Malatesta, Adam GonzalesAV Squad: A24 trailersPlay a sound before its picture; music changes after each piece of information
Todd Yellin, Lon BenderNetflix "ta-dum"A sonic logo under 3 s, built from 4 layers, tested blind
Henry DaubrezGoogle Flow filmmaker in residenceGive AI lines room to breathe; clone a clean line and re-dub
PJ AceAI ad director, CEO Genre.aiNO MUSIC block, sounds in angle brackets, a DELIVERY line per line
Uncanny HarryBAFTA-nominated TV directorAct the line yourself and transfer the performance
Ian ShepherdMastering engineer, loudness specialist-1 dBTP ceiling; platforms normalise, so don't over-squash
Alex KnickerbockerStudio sound editor and re-recording mixer-14 is a web number; TV is -24 LKFS
bruno (tvnxty)Sound designer for product launch videosNot every movement gets a sound; UI sounds fit in 0.5 s

What experts liked or saved could not be read. X has hidden all likes since June 2024, and twitter_user_likes returned nothing for all 14 experts we tried; bookmarks are private. Their own posts and the pages they wrote are the evidence.

Four popular accounts in the earlier X tip bank were dropped as sources. Each has more saves than likes on a small following, plus credits we could not verify: @second_bassoon (the most-saved "why AI dialogue sounds fake" thread), @beechinour (dBFS mix ranges), @Voxyz_ai and @Asteri_eth. Where their advice matches a verified expert, we cite the expert instead.

Voices: cast, direct, make them sound real

An AI voice sounds fake for four reasons, and each has a known fix. Cast by listening to candidates on your hardest line. Direct with the syntax the model documents. Give lines room. Put the voice in a room.

TellWho names itFix
Rushed: too many words for the clipHenry Daubrez: "pushing 20 lines of text through a 30s generation, the model will speed through the delivery"Fewer lines per clip; a 4 s shot holds one or two short exchanges (OpenAI Sora guide)
Too clean: no room, no breathRandy Thom: re-recorded dialogue "almost always" lacks "the life" of the originalRoom tone under every line; ElevenLabs audio_effects can place the voice in a room (docs)
Over-cleanedMac Smith, Skywalker: cleaning "softening consonants" hurts intelligibilityDon't denoise generated voice; leave quiet breaths in (iZotope RX)
No soul in a synthetic characterBen Burtt: synth R2-D2 "didn't seem to have a soul"; the final was 50% humanStart from a human performance for robots and creatures, then process it

Make candidates, let a person pick

An agent can't hear, so it makes and measures candidates and a person chooses. Voice director Khris Brown casts the voice that makes you "feel something personal", and says "Better no voice at all than poor voice."

  1. Write the casting note as a role, a listener and 2-3 precise adjectives ("caring nurse", "to a scared child", "reluctant, certain"). Never write "conversational"; write a situation: "toss off that last phrase like it's something you've discussed a million times" (voice actors on Game Developer).
  2. Shortlist 3-5 voices. Premade voices by age, accent and use case, or design new ones. ElevenLabs Voice Design returns 3 voices per generation for the price of one preview (docs). Its prompt format: Native <Language>. <Gender>, <Age range>. <Quality>. Persona: <2-5 words>. Emotion: <2-3 adjectives>. Keep guidance scale low (their examples run 20-40%), and make the preview line fit the character. Gemini wants a 1-2 sentence voice description (docs).
  3. Audition the hardest line, 2 takes per voice, on one page. For two speakers pick contrasting voices: in Hume's blind test mixed-gender pairs scored 4.65 for telling speakers apart, same-gender pairs 2.87-3.44 (Hume).
  4. Screen with numbers, then listen on headphones and a phone speaker. Our audition.py renders the takes, measures length, words per minute and pauses, and builds the page with a Pick button.
  5. Lock it: voice id, model and settings go in the brief. Re-render only the lines that are wrong.

No expert publishes a casting chart by video type. The documented starting points: ElevenLabs' "Movie Trailer Voice" design example at 20% guidance; a phone-mic, no-music audio line for UGC (prompt, 151k views); Seedance builds a voice from how the character looks, so the character sheet casts the voice; ElevenLabs has no child voices by policy (help).

Direct it: the exact syntax per model

Model (varg id)PauseBreath, reactionDeliveryWatch out
ElevenLabs v4 / v3 (eleven_v4)..., a line break, [pause][sighs] [exhales] [laughs] [clears throat] [inhales deeply][whispers] [sarcastic] [curious], or a stage note: [Low, steady voice, restrained urgency]No SSML <break>; only tags you can hear (no [standing]); narrative cues like "she asked, trembling" are read aloud (guide)
Gemini 3.8 TTS (gemini_3_8_flash_tts)<short pause> <long pause><breath> <sigh> <cough> <chuckle> and 30 moreshort style: "whispered urgently"Long "Director's Notes" blocks cause voice drift; split the turn when the emotion changes (docs)
Google Cloud Gemini-TTS[short pause] \250 ms, [medium pause] \500 ms, [long pause] 1000+ ms[laughing]style promptEmotion tags like [scared] are spoken aloud (docs)
Fish Audio S2 / Drama 3[pause] [long pause][inhale] [exhale] [sigh] [shaky breath][barely holding it together, forcing a smile], <emphasis>...</emphasis>A tag changes everything after it, so place it at the turn (Fish)
OpenAI gpt-4o-mini-ttsin instructionsin instructionsfields: Voice Affect, Tone, Pacing, Emotion, Pronunciation, Pauses (openai.fm)Shuts down 2027-01-06
Hume Octave[pause], trailing_silenceunder 100 characters: "frightened, rushed"Rate goes in speed, not words

On ElevenLabs, stability 0.30-0.50 gives emotional delivery that can wobble and 0.60-0.85 gives steady delivery that can go flat (docs). Spell out numbers, symbols and URLs before TTS, and fix names with IPA in slashes on v4.

What the direction did, measured

We ran one line in four versions on eleven_v4, 2 voices × 2 takes each (16 takes, $0.24 in all). Bare text was rushed. Each layer of direction slowed it and opened the pauses. No tag was read aloud, and all 16 transcripts matched the script.

Waveform and spectrogram of the same line in four versions, voice Jessica: bare, with punctuation, with breaths and sniffs, with emotion tags

Same line, four versions: bare, + punctuation, + breaths and sniffs, + emotion tags (voice: Jessica). The four versions in a row (a still from the clip); waveform on top, spectrogram below.

VersionText sentLengthWords/minPause after "break it"Loudness
BareI didn't mean to break it. I was only trying to fix the light before you got home. I'm sorry.5.7 s2310.36 s-26.5 LUFS
+ punctuation... and —6.5 s2070.51 s-26.8
+ bodily actions[exhales] [sniffs] [swallows] between sentences8.7 s1940.92 s-29.8
+ emotion tagsadds [quietly, ashamed] and [voice breaking]8.0 s2000.89 s-31.6

The "quietly" tag made the read 5 dB softer, so re-level lines after directing them. The same four versions in a male voice (Liam):

Waveform and spectrogram of the same line in four versions, voice Liam

Same line, four versions (voice: Liam). The four versions in a row (a still from the clip); waveform on top, spectrogram below.

Keep one voice across shots

Consent

Clone only voices you have the right to use. ElevenLabs Professional clones are limited to your own verified voice; Instant clones require you to confirm consent (docs). Gemini voice replication needs a consent recording from the same speaker. YouTube asks for an AI label when a real person seems to say something they didn't (policy).

Dialogue and narration: write for the ear, then leave room

Write fewer, shorter, spoken lines, record them before any picture, and let pauses, breaths and small actions carry the feeling. Adjectives in the prompt do less than a sigh in the right place.

Build the voice track first

Ken Burns' editor Craig Mellish screens a "radio cut", narration and voices only, before cutting picture (interview). Baby Driver began as a 100-minute radio play (Paul Machliss). For an agent, this means: voice lines first, word timestamps from them, and every cut lands on a word or a pause, never inside a word.

Write for the ear

Pauses, breaths and small actions

A line sheet the agent keeps, one row per line:

| id  | who  | text exactly as sent                                   | direction                    | after  | shot |
| L01 | MIA  | [exhales] I didn't mean to break it...                  | ashamed, to her brother      | 0.9 s  | s02  |
| L02 | MIA  | [sniffs] I was only trying to fix the light — before you got home. | quieter, avoids his eyes | 0.6 s | s02 |
| L03 | LEO  | [jumping in] —You were trying to fix it?               | dry, a small smile           | 1.5 s hold | s03 |

Dialogue inside video models

ModelWrite the line like thisSound rules
Seedance 2.x{Really, sis?} then DELIVERY: understated, mildly incredulous... No shouting. (PJ Ace)Seedance 2.5 always outputs audio (Together AI). Open with "NO MUSIC... Use only diegetic sound effects, ambience, dialogue, and foley written inside <>" and list the sounds in <...>. Two clipped lines across a cut beat one long line (fal)
Kling 3.0[Black-suited Agent, raspy, deep voice]: "Don't move." with the action before the line; Immediately, between speakers or the lines merge (fal / Kling rules)10 s clips and at most 2 speakers work best (Kling 2.6 guide)
Veo 3.1Man: (Hand on his hunting knife) "That's no ordinary bear."SFX: and Ambient noise: as separate cues; timestamped segments for multi-shot (Google Cloud)
Sora 2a Dialogue: block under the visual prose, one label per speakerA 4 s shot holds one or two short exchanges
Any model, lip-sync laterprompt "the character should be speaking naturally" so there is mouth movement (sync.)Feed the lip-sync model one clean voice, no music

If the native voice drifts or sounds off, keep the take and replace the voice: clone a clean line, re-voice, re-sync (Daubrez), or run the clip audio through a voice changer to one target voice (MetaPuppet). Check every generated line against the script by transcript: models add gibberish words.

Sound effects: few, exact, on the right frame

Pros use far fewer effects than agents do. In measured launch films about 85% of hard cuts have no sound of their own and there are about 2 whooshes per film (Cinetic). earshot found that a strong accent lands on only 4-25% of cuts (study of 8 films). Every effect needs a visible cause, lands by its loudest point, and sits in its own band. "Few and exact: every effect has a cause on screen, and too many read as a template" (HyperFrames).

What gets a sound, and when

On screenSoundTimingSource
A real cursor clickshort click10-30 ms before the visible pressCinetic, earshot (measured)
Text typing inhigh-passed noise ticks at 3-12 kHz, nothing under 2 kHz, about 87 ms apart, level and timing jitteredwith each characterearshot (measured)
A badge, chip or number landinglow thump at 80-150 Hz with a faint tick25-40 ms after its cutearshot (measured)
A real scene changeone soft whoosh; big whooshes only for big momentspeak within about ±150 ms of the cutearshot (measured), Nate Curtiss
The payoff or the logoriser into impact, once per filmthe music stops on a bar line, 0.2-0.7 s of near-silence, then one soft element on the wordmark, a 0.5-2.5 s tailCinetic

Sources disagree on the lead: HyperFrames puts a peak one frame after its picture, OpenMontage starts a whoosh 10-20 ms before, earshot and Cinetic measured onsets 100-450 ms ahead with the apex on the visual peak. All agree the loudest point belongs on the event. Trailer editor Derek Lieu adds that a hit two frames off sometimes looks better, so measure first, then let a person nudge.

Layer a scene the way film sound designers do

Prompt and generate effects

ElevenLabs SFX v3 (eleven_sound_effects_v3 on varg): "describe the action, setting, atmosphere and performance you want rather than a broad sound category": "Footsteps on gravel approaching slowly". Chain events with "then". Counts like "triangle hit 4 times" are not followed, so make one hit and repeat it yourself. Set loop for ambience beds, and raise prompt_influence above its 0.3 default for literal UI sounds (docs).

We generated five effects ($0.07). One came back unusable:

PromptAskedCame back
"Single crisp click on a laptop trackpad, close-mic'd, dry, quiet room, one-shot"1 snear silence, peak -54 dBFS
"Loud, crisp computer mouse click, close-up, one-shot"model picks0.48 s, peak -9.3 dBTP: usable
"Fast airy whoosh that rises and swells into a hit, clean and modern, close, one-shot"2 s0.40 s of silent lead-in, loudest point at 1.21 s
"Deep cinematic impact: a punchy low boom with a short sub-bass drop and a soft airy tail, one-shot"3 sstarts at 0.00 s, 2.6 s of sound
"Quiet modern studio room tone, soft air conditioning hum..." (loop)10 s-53.5 LUFS, needed +18 dB

So measure every effect before placing it, and place a whoosh by its peak: start = hit time - 1.21 s.

Inside video models. Veo: SFX: thunder cracks in the distance and Ambient noise: the quiet hum of a starship bridge as separate cues (Google Cloud). Kling: [Object: Wooden Door] suddenly [Action: Slams] + [Sound Effect: Bang] (Kling 2.6 guide). Seedance: list every sound in <...> and ban generic transition sounds: "Also no "whoosh" or "zoom" sound effects" (PJ Ace). For a phone-video look: "phone-mic quality... No music, narrator, or cinematic SFX".

Foley from the picture. MMAudio works on 8 s windows and syncs to the centre square of the frame; it can mumble speech-like sounds and its weights are non-commercial (README). A video-only foley model with no text "stays almost silent" (Heather Cooper), so describe the sounds too.

Library or code? HyperFrames retrieves only CC0 and CC BY sounds and bans synthesized tones. earshot and Cinetic synthesize sounds in code, yet list "code-synthesized beeps, sine pings" among the cheap tells and ask for a loudness-matched human blind test. Use code-made sounds for UI ticks and pops; use recordings or generated sounds for anything that should feel physical. Check licences: Freesound CC BY-NC bans money-making use; Sonniss GDC sounds may not be used to train AI.

The cheap tells (audit before delivery)

From earshot's checklist (source): code beeps and sine pings, one sample machine-gunned, a whoosh on every cut, a boom on every logo, reverb on tiny sounds, random pitch on typing, clicks at cuts, music chopped mid-note, an audible ambience loop, a different room on every sound, an ending that stops mid-note instead of a tail into silence.

Music: decide it, map it to the beats, finish it in the edit

Answer genre, mood, instruments, tempo and era yourself, plan where the music enters, stops and hits, then measure what came back. Make the stops, the drop and the exact length in the edit. In our test the model kept 120 BPM in 3 of 3 takes but never made the planned silence.

How much music, and where

Tempo, length and the beat map

The beat map of our 20 s launch example, 120 BPM, one bar = 2 s:

TimeBarsMusicVoice and pictureEffects
0-8 s1-4just a muted pulse and hi-hatL01 at 0.5 s: "Your week... is already full."click on the 4.0 s downbeat cut
8-12 s5-6warm sub bass entersL02 at 8.25 s rides the bass entry
12-13 shalf a barstopped on the bar line; room tone staysthe build holdswhoosh swells, peak at 13.0 s
13 s7re-enters on the downbeatlogo landsimpact on 13.0 s
14-20 s7-10groove, then rings outL03 at 14.0 s: "This is Tide. Your week, handed back."

Waveform and spectrogram of the designed 20 s launch soundtrack with voice lines, effects and cuts marked

Watch: the designed launch soundtrack, sound on (MP4)

The designed launch soundtrack: orange bars are voice lines, white lines effects, grey lines cuts. Sound on: waveform on top, spectrogram below; orange bars are voice lines, white lines effects, grey lines cuts.

Prompt the music model

Model (varg id)What worksKnown failures
ElevenLabs Music v2.5 (eleven_music_v2_5)Answer the five questions: "French house, 122 BPM, filtered disco sample chops, sidechained bass...". Narrate the arrangement: "start with just a shuffled drum loop, add a warm sub bassline after four bars, then bring in...". "Without the just, the model fills the silence." BPM and key as numbers. The ElevenLabs API also takes a composition plan of up to 30 sections, each with its own duration (guide)Naming an artist or song returns an error
Lyria 3.5 (lyria_3_5)[Genre & style] + [Mood] + [Instrumentation] + [Tempo & rhythm] + [Vocal style & language] + [Lyrics]; timed sections like [0:12 - 0:24] Verse 1: ... Intensity: 3/10; "Build to a beat drop at 12 seconds" (Google)No negative prompt on 3.5; in our October run it ignored the asked length (20 s asked, about 90 s back)
Suno v6Use the lyrics field as a timeline of bracketed events: [A heavy, distorted 808 sub-bass drops in] ... [Fade to silence] (INK); Exclude field for unwanted partsEvery model before v6 is retired, so older tag tricks need re-testing; tempo can speed up across a song (composer Tomoya Kinoshita); an unwanted sound often comes from one style word, so delete it
Mureka 9.5 (mureka_9_5)instrumental from a text description up to 1,024 charactersThe model picks the length
ACE-Step 1.5 (open)bpm, key and duration as parameters, not in the caption"Final result might be 118 or 122, not exactly 120"

Our test on eleven_music_v2_5, 20 s asked, a detailed prompt written from the ElevenLabs guide, 3 takes at $0.05 each, measured with sound.py beats:

InstructionFollowed
120 BPM3 of 3 (120.0, 120.05, 119.97)
"start with just ... then bring in a warm sub bass" after four bars2 of 3
"half a bar of complete silence", then an impact and a kick0 of 3
20 s length2 of 3 (one came back 29 s)

librosa's own tempo estimate read 117.45 BPM for these 120 BPM takes, and its downbeat guess was half a bar off. The bass entry showed the real bar line. The average beat gap gives the true tempo; check downbeats against where instruments enter.

Edit music to length

Licences

ServiceCommercial use
ElevenLabs Musiconline and offline commercial use on paid plans, except film, TV, radio and studio games (terms); no artist, song or label names in prompts
Sunoonly songs downloaded on a paid plan; free-plan songs belong to Suno; 20 or 60 downloads a month; fully AI music is not copyrightable in the US (help)
Udiodownloads disabled since the UMG deal
LyriaGoogle claims no ownership but may make similar music for others; SynthID watermark
Mureka APIpaid calls carry commercial authorization
TikTok adsuse the Commercial Music Library, not any trending song

Mix and master: voice first, measured, checked on a phone

Mix the voice first and keep the music 10-20 dB under it where words live. Keep room tone under everything. Master social video to about -14 LUFS with true peak at or below -1 dBTP, then listen on a phone speaker. Evidence that sound adds views comes mostly from platforms that sell sound-on ads, so make the video work muted too.

Loudness targets

Where it playsIntegrated loudnessTrue peakSource
YouTubeno published number; turns loud videos down, never quiet ones up; -14 LUFS is the common reading-1 dBTPIan Shepherd, YouTube Help
TikTok, Instagram Reelsno official spec; practitioners deliver -14 LUFS-1 dBTPOpenMontage
Spotify-14 LUFS-1 dBTP, or -2 if the master is louder than -14Spotify for Artists
Apple Podcasts-16 LKFS ±1-1 dBFSApple
Streaming speech (AES TD1008)-18 LUFS speech, -16 musicIan Shepherd
European broadcast-23 LUFS-1 dBTPEBU R128
US TV-24 LKFSAlex Knickerbocker
Netflix-27 LKFS ±2, measured on dialogue-2 dBTPNetflix spec v1.5
Audiobook (ACX)-23 to -18 dB RMS; noise floor under -60 dB-3 dB peakACX

Mastering engineer Ian Shepherd's advice: don't squash a mix to chase a number, since platforms normalise; keep -1 dBTP because "Lossy encoding for streaming or mp3 can also generate extra peak information above zero". earshot found that 6 of 7 pro launch films clip after AAC decoding, so don't copy their peaks. ffmpeg's loudnorm defaults to -24 LUFS and -2 dBTP, so always pass your own target.

Levels, ducking, space

Designed vs plain, measured

Same three voice lines, both mastered to -14 LUFS, graded by sound.py check:

CheckDesignedPlain
Loudness, true peak-14.2 LUFS, -1.5 dBTP-14.0 LUFS, -1.5 dBTP
Voice above music, 300 Hz-4 kHz, worst line+15.9 dB-6.6 dB: the music is louder than the voice
Effects on their moments3 of 3 within one frameno effects
Cuts on the beat4 of 41 of 4 (44-125 ms off)
Dead airnone; room tone runs through the stop1.2 s of digital silence at the end

Waveform and spectrogram of the plain 20 s launch soundtrack: music at full level, no effects

Watch: the plain version of the launch soundtrack, sound on (MP4)

The plain version: vague music at full level, no ducking, no effects, no room tone. Sound on: waveform on top, spectrogram below; orange bars are voice lines, white lines effects, grey lines cuts.

Does sound add views?

No public study measures watch time for the same video with and without sound design. Every number we found was paid for by a platform that sells sound-on ads, or is a survey:

FindingWho and how
88% of TikTok users call sound essential; 73% stop for ads with audioTikTok-commissioned Kantar survey, 2021 (report)
2.4x more attention to ads on sight-and-sound platformsNielsen lab study for Google, 400 people, 2016 (PDF)
About 80% of Reels watched with sound onEssenceMediacom for Meta, sponsored, 2024 (Campaign)
92% watch mobile video with sound off; captions lifted ad recall 8%Verizon Media and Publicis survey, 2019 (Next TV)
A brand's own sonic cues beat borrowed music and celebritiesIpsos meta-analysis of 2,000+ ads, independent (Ipsos)
Fury Road's preview score rose 8 points after a sound-only redesignMark Mangini, one film (BTL)
"I've seen multiple videos fail because the music was too loud, the music was too chaotic, or the sound came through distorted"Mario Joos, retention director for MrBeast and others; opinion (post)

What follows from it: build for sound on TikTok and Reels, caption for the sound-off viewer, own a recognisable sound instead of borrowing a song, and never let music bury words.

Doing it with code: the agent's sound toolkit

An agent can't hear, so give it data and checks. Keep every sound's time in a cue sheet, build the mix from it with a script, grade the mix with numbers, and render a picture of the sound to look at. The same steps serve a 100% code video (HyperFrames, Remotion) and a generated film (Seedance, Kling, Veo). "Agents can't listen. Flag music endings and loop points for a human ear-check" (Elie Steinbock's launch-film prompt).

The pipeline

  1. Voice first. Render the lines, get word timings from the same call (ElevenLabs /with-timestamps) or a transcript, and let them size the timeline. Remotion's skill: generate TTS per scene, then calculateMetadata sizes the composition to the audio (Remotion skills).
  2. Music. Generate or pick, then sound.py beats for tempo and bar lines, sound.py fit to cut it to length on downbeats, and snap the picture's cuts to the grid.
  3. Effects. List the on-screen events, generate or retrieve one sound per event, measure each one, and give it an at time (and align: peak for whooshes and risers).
  4. Mix and grade. sound.py mix cues.json places everything, ducks the music under the voice band, adds room tone and masters to the platform. sound.py check grades it and exits non-zero on a failure.
  5. Look, then let a person listen. sound.py preview makes a video of the waveform and spectrogram with a playhead, voice lines, effects and cuts marked. A person listens on headphones and a phone. Then sound.py mux puts the mix on the picture.
  6. Generated films: Seedance 2.5 always outputs audio. Keep good native dialogue, strip the rest, and rebuild music and effects on top; re-voice drifting lines and lip-sync with a clean, single voice.

The two scripts in the skill

CommandWhat it does
sound.py measure FILEloudness, true peak, loudness range, silences
sound.py beats MUSIC -o beats.jsonBPM from the average beat gap, beats, estimated downbeats, strongest hits, quiet spans (librosa)
sound.py fit MUSIC --to 30 -o bed.wavkeeps the opening and the real ending, removes whole bars from the middle
sound.py mix cues.jsonbuilds the mix and stems from the cue sheet; fast-down, slow-up ducking in the voice band; 30 ms fades on every clip; master to target with a 4x-oversampled limiter
sound.py check MIX --cues cues.jsonpass or fail: loudness, true peak, length, voice above music in 300 Hz-4 kHz, effects within one frame of their moment, cuts on the beat, no dead air, mono fold-down
sound.py sheet / previewthe sound as a picture, or a video with a playhead
audition.py plan / render / score / pagethe same lines in several voices: renders through the varg API, measures pace, pauses and pitch spread, builds a page with a Pick button
{ "duration": 20.0, "fps": 30, "platform": "youtube", "duck": { "depth_db": 8 },
  "beats": "music/bed_beats.json", "cuts": [4.0, 8.0, 13.0, 16.0],
  "silences": [{ "start": 12.0, "end": 13.0, "why": "stop before the logo hit" }],
  "tracks": [
    { "id": "L01", "role": "voice", "file": "vo/L01.mp3", "start": 0.5 },
    { "id": "bed_a", "role": "music", "file": "music/bed.mp3", "start": 0, "length": 12.0, "gain_db": -4 },
    { "id": "bed_b", "role": "music", "file": "music/bed.mp3", "start": 13.0, "trim_start": 12.0, "length": 7.0, "gain_db": -4, "fade_out": 1.5 },
    { "id": "room", "role": "ambience", "file": "sfx/room.mp3", "loop": true, "gain_db": 18 },
    { "id": "whoosh", "role": "sfx", "file": "sfx/whoosh.mp3", "at": 13.0, "align": "peak" },
    { "id": "impact", "role": "sfx", "file": "sfx/impact.mp3", "start": 13.0 } ] }

Code-video specifics

ffmpeg traps

Casting candidates by code

Open models worth knowing

JobModelLicence note
Narration, free and offlineKokoro-82MApache-2.0; no cloning
CloningChatterbox (27.5k stars)MIT; watermark on every output
Two-speaker dialogueDiaApache-2.0; a fixed seed does not hold the voice (open issue #109)
Directed deliveryFish Audio Drama 3research licence, not OSI
Foley from videoMMAudio, HunyuanVideo-FoleyMMAudio weights are non-commercial
SongsACE-Step 1.5, YuEMIT, Apache-2.0
StemsDemucs v4MIT
Word timingswhisperX, faster-whisperBSD, MIT
Beats and downbeatsBeat This!, All-In-OneMIT

F5-TTS, MusicGen, MMAudio, Higgs Audio v3 and madmom's model files are non-commercial: check before a paid project.

Ranking takes with an LLM judge, without bias

Use an LLM judge to shortlist takes and check the words, never to measure levels or to pick by taste. Write one yes/no question per known failure, let a script answer every question a script can, and compute the winner in code. Compare takes two at a time in both orders, and let a person make the final pick. In our test a vague "which sounds better?" judge chose the plain mix over the designed one every time it answered. It also picked confidently between two identical takes.

The criteria: one question per failure, one checker per question

Modern TTS "stopped failing on artifacts. It fails on subtle things now: prosody, breath placement, expressive overshoot" (harrrshall). So each criterion names one of those failures and has one checker. A criterion is clear when "two domain experts would independently reach the same pass/fail verdict" (Anthropic).

WhatPasses whenChecked byWhy that checker
Voice: wordsthe words match the script line, nothing added, dropped or said wrongscript (transcript diff), then the judge on namesGemini judged pronunciation with "negligible hallucinations, leveraging Gemini's robust ASR" (EmergentTTS-Eval). A transcript lifts audio-judge pronunciation accuracy from 46% to 63% (AudioJudge)
Voice: tagsno stage direction ("sigh", "exhales") spoken as a wordscript (transcript), the judge as backupSpeech models sometimes "directly read the words 'sigh' out loud" (Chiang et al.). It happened 0 times in 16 in our Eleven v4 test
Voice: endingthe last syllable is not cut offthe judge, on that take aloneElie Steinbock's judge scores "pacing, mispronunciations, odd stresses and clipped endings" (post)
Voice: pace140-170 words a minute for narration (dialogue may sit outside on purpose)script (audition.py score)Audio judges told which of two clips was faster 47-55% of the time (AudioJudge). The window is from digitalsamba's pick_takes
Voice: pauseseach planned pause is there and at least as long as the cue saysscript (silence detection)Gemini judges were weaker on pause length (EmergentTTS-Eval); ours got 3 of 4
Voice: pitchpitch spread of at least 2 semitonesscript (audition.py score)"a spread under 2 semitones reads flat" (digitalsamba pick_takes)
Voice: deliverysounds like the brief ("apologising, not reading")the judge in pairs, both orders, as a shortlist; a person picksThe best audio judge "still only achieves less than 70% agreement with human preferences" on naturalness (SpeechJudge)
Dialogue in videoeach line said by the right character, in order, nothing extrathe judge on that clip alone, with MM:SS timesGemini's documented audio uses include transcription and speaker diarization (Gemini docs)
Lip syncmouths match the wordsa person, or a judge at a raised frame rateGemini samples video at 1 frame a second by default, and "fast action sequences might lose detail" (Gemini video docs)
Music: tempo, lengthBPM and length match the planscript (sound.py beats)right 3 of 3 times in our test
Music: quiet partthe planned quiet section is therescript (loudness over time)the model delivered it 0 of 3 times in our test
Music: vocalsno vocals in an instrumental bedthe judge: "describe what you hear", then a text check"most are speech-trained and return a transcript, not the music" (wolfram)
Music: fitfits the video's mooda personA music-quality predictor reached 0.926 rank correlation over whole systems but 0.633 per clip (MusicEval). Audiobox Aesthetics predicted people's pick 62.3% of the time; chance is 50% (Zhang et al.)
Effect: there and on timenot silent, its peak within 1 frame of the cuescript (sound.py check)1 of 5 effects came back silent in our test
Effect: identityit is the sound asked forthe judge: describe first, compare in codeAudio LLMs "tend to give affirmative answers" to "is there a sound of X?" (Kuan et al.)
Mix: levelsplatform loudness ±1 LUFS, true peak, voice 10 dB or more above music in 300 Hz-4 kHz during every linescript (sound.py check)Judges kept 73-93% of verdicts unchanged at 1 dB signal-to-noise (AudioJudge): they hear words through noise you want flagged. Ours misread the plain mix's levels
Mix: feelthe soundtrack carries the videoa personno judge in our sources was validated for this

Our test: an audio judge on our own soundtracks

We used varg's analyze_video, which runs Gemini 3.8 Flash on a video and hears its soundtrack: 30 calls, $1.50. Each pair played as one video with neutral "CLIP A" / "CLIP B" cards, in both orders. We scored it against measured truth on three pairs. The first was the designed vs the plain launch mix. The second was a bare vs a breath-tagged take, for two voices. The control was two takes of the same line.

SetupMix: designed vs plainVoices: bare vs tagged (2 pairs)Control: two near-identical takes
Vague question, 2 runs x 2 ordersthe plain mix every time it answered (3 of 3; once no answer)the tagged take 7 of 8; once "B" in both orders (the slot, not the take)a confident pick every time; take 2 in run 1, take 1 in run 2
5 yes/no questions, winner by a fixed ruleright in one order, tie in the otherright in one order each, ties in the othertie in both orders
The same questions, each clip alonedesigned 4 of 5 answers right, plain 1 of 5pause question 3 of 4 rightnot needed

Twelve rules for an unbiased judge

  1. Write criteria from real failures, one judge per failure. "In practice, we recommend building a separate judge for each failure mode" (Hamel Husain). Grade a batch of real takes before you write the prompt: people's criteria change as they grade (Shankar et al.). Lotte Verheyden says 10-20 labelled cases are enough to build a judge (post); Eugene Yan found 20 too noisy and labels 50-100 (AlignEval).
  2. Ask yes/no questions, never for a 1-5 score. "The difference between adjacent points (like 3 vs 4) is subjective" (Husain and Shankar). Splitting criteria into yes/no checklist questions raised agreement across judge models by 0.45 (CheckEval). Each question names one thing and what to ignore: "The response cites at least one source document. Ignore formatting issues" (Verheyden).
  3. Ask the judge to describe, not to confirm. Audio models answer "yes" to "is there a sound of X?" when X is missing (Kuan et al.). Ask for a transcript and a list of non-word sounds with MM:SS times, and check them in code. Allow "unknown": "give the LLM a way out" (Anthropic). End the prompt as Woody does: "Don't call the sound good. Report what the checks show" (post).
  4. Reason first, but don't expect reasoning to remove bias. "Ask the judge to output its explanation prior to (instead of after) outputting the score" (Cameron Wolfe). For audio, a chain-of-thought rubric "just inverted the bias from always-B to always-A" (harrrshall).
  5. Compute the winner in code from the answers. Never make "which is better" the deciding question. In our test that change alone removed every wrong pick.
  6. Compare two at a time, in both orders, and count only verdicts that agree. Off-the-shelf audio models "pick whichever clip is in slot B, then write a confident rationale for why B was better" (harrrshall). The median model flips 45% of decisive pairs (Lech Mazur). Position bias grows when there are three or four options in one prompt (CALM), so never rank five takes in one call. "Call the judge twice with swapped positions. If evaluation results are inconsistent, then return False" (Jerry Liu). Google's autorater swaps half its pairs for this (Vertex AI).
  7. Judge each clip alone for yes/no checks; stitch pairs into one file. In our test the first clip leaked into answers about the second. For pairs, one file with both takes beat two uploads: pronunciation accuracy rose from 46% to 66% (AudioJudge).
  8. Make the takes comparable and hide where they came from. Every audio judge AudioJudge tested preferred the longer clip when people rated the two equal. Voice timbre shifts emotion verdicts (EmergentTTS-Eval), so judge delivery in the same voice. Judges also score "refined" answers higher without saying why (CALM). Name files CLIP A / CLIP B, never "v2" or "tagged".
  9. Split dimensions. Three narrow audio judges (words, delivery, quality) with a majority vote reached 0.912 rank correlation with people (AudioJudge). One judge scoring a voice "kept inferring tone from the words, inventing vocal cues that weren't there" and correlated 0.62 with people; scoring content and voice apart reached 0.78 (Neethu Joy).
  10. Repeat, vote, and send unstable calls to a person. A majority of 5 runs helped; averaging the 5 runs or keeping the best did not (Gu et al.). Sending only the 20% most order-unstable pairs to people matched human accuracy at 39% lower cost (Wang et al.).
  11. Judge with a different model family, or prove it is safe. One judge "marked 38% of its own failed unit tests as passed, vs 1.9% for other models' code" (Sword). Anthropic's docs call a different grading model best practice (docs). Husain and Shankar say the same model "is usually fine" if it matches human labels. Either way, the human-label check in rule 12 decides.
  12. Measure the judge against people, and against people's own agreement. Validate on about 100 labelled examples per failure, with both passes and fails (Husain). Report how many defects it catches, apart from how many good takes it passes. One judge passed over 95% of good outputs but caught only 30-60% of the bad ones (Eugene Yan). Use Cohen's kappa, not correlation: the same judge scored kappa 0.3-0.5 but correlation 0.8-0.9 (Yan). A judge "can agree with itself every time and still be wrong in the same direction every time" (DAIR.AI). Aim at the human ceiling, not 100%. Expert annotators agreed with the majority 71.7% of the time (Wang et al.). About 70% of TTS pairs had full or weak human agreement (SpeechJudge). People agreed on dialogue style at only r = 0.253 (Chiang et al.). Pin the judge model and log raw outputs. Check every claim it makes: "Video and audio judges hallucinate problems, so verify every claim against frames" (Steinbock).

Ranking five takes, step by step

  1. Filter by script. Drop any take whose transcript misses or adds words, reads a tag aloud, clips, or falls outside the pace window.
  2. Check each remaining take alone with the prompt below. A take passes when every answer is "no" and its sounds list has the breath the cue asks for. A take with an "unknown" goes to a person.
  3. Compare the survivors two at a time, in both orders, one question per call. Use one stitched file per pair. A win counts only when both orders agree; anything else is a tie. Five takes make 10 pairs, so 20 calls per question: about $1 at 5 credits a call on analyze_video.
  4. Rank by wins. Send pairs that split to a person.
  5. A person listens to the top 2-3 on headphones and on a phone, and picks. Their pick and reason become labels for the next check of the judge (rule 12).
You are checking ONE voice take. The script line is: "<line exactly as written>".
Listen to the whole take. Do not compare it with anything and do not say whether it is good.
1. Write down exactly what you hear, word for word.
2. List every sound that is not a word (breath, sigh, sniff, laugh, click, noise) with its MM:SS time, or "none".
Then for each question write one short reason, then answer yes, no or unknown (unknown when the audio does not tell you):
Q1 Is any word missing, added or changed compared with the script line?
Q2 Is a stage direction spoken as a word (for example "sigh", "exhales", "quietly")?
Q3 Is any name or brand word said wrongly? Give its MM:SS.
Q4 Is the last syllable cut off?
Ignore the accent, the voice's gender and any music. Don't call the take good. Report what you hear.
Return JSON: {"heard": "...", "sounds": [{"t": "MM:SS", "what": "..."}],
 "Q1": {"reason": "...", "answer": "yes|no|unknown"}, "Q2": {...}, "Q3": {...}, "Q4": {...}}
This file holds two takes of the same line: "CLIP A", a short tone, then "CLIP B".
Question: <one narrow question, e.g. "Which take sounds more like someone apologising than reading?">
First describe each take in one sentence about this question only. Then answer A, B or tie.
Answer tie if you are not sure. Return JSON: {"A": "...", "B": "...", "answer": "A|B|tie"}

Run the second prompt again with the takes swapped. A win counts only when both runs name the same take.

The video-sound skill runs steps 2-4 with scripts/judge_takes.py. prompt prints both prompts, and check turns a one-take reply into pass, fail or review in code. make builds level-matched blind pair files in both orders, and tally counts a win only when both orders agree. The criteria, rules and test are in references/judge.md (vargHQ/skills #95). All 143 judge findings with their quotes (73 from X, 70 from papers and docs) and the raw judge replies are in vargHQ/aeo #42.

Variations by video type

The routine stays the same; what changes is who leads. Music leads a launch film and a music video, the voice leads an explainer and a UGC clip, the room and the silences lead a cinematic film, and a trailer alternates between them.

TypeLeadsMusicEffectsSignature move
Launch / product filmmusicunder nearly all of it, 108-136 BPM3-6 accents: reveal, key UI action, logothe music stops on a bar line before the logo
Cinematic AI filmroom and silencelittle, often hiddenoff-screen, at real distancessilence before the big hit, 3-4 times a film
Music videothe songthe song, uneditedfew, under the songpicture cut to the song, never the reverse
Trailer / adalternatesthree acts with gapsbuilt from the film's own soundsa line, then a hit
UGC / talking charactervoicenone, or the room'sthe room and the handsphone-mic sound that matches the camera
Narrated explainer / code demovoicea carved bedone big hit every 7-10 severy cut on a word or a pause

Launch and product films (code-made)

Cinematic AI films

Music videos

Trailers and ads

UGC and talking characters

Narrated explainers and code demos

The skill and the copy-paste prompt

Everything above ships as the video-sound skill (GitHub, merged in vargHQ/skills #79) and as a varg how to card (varg.ai/how-to/skills/video-sound). The card is #1 in how-to search for "make my AI voice sound real" and "add sound design to my video".

AgentInstall
Claude Codeclaude plugin marketplace add vargHQ/skills && claude plugin install varg@varg
Codexnpx skills add vargHQ/skills --skill video-sound -a codex -y
Cursornpx skills add vargHQ/skills --skill video-sound -a cursor -y
Gemini CLInpx skills add vargHQ/skills --skill video-sound -a gemini-cli -y

What is in it: SKILL.md (the routine with STOPs), scripts/sound.py and scripts/audition.py, one reference file per section of this guide, references/evidence.md with all 642 findings and their quotes, and examples/launch-20s/, which rebuilds the designed and plain soundtracks from this guide with one fetch.sh. It hands off to video-script (lines and the sound column of script.md) and character-passport (each character's voice block).

The full prompt

The whole skill as one prompt, for any agent with a shell. Replace the brief line:

You are the sound designer, voice director and music editor for this video. You cannot hear, so you plan in writing,
measure everything with numbers, and a person listens at the STOPs. Never say you heard something.

Brief: {what the video is, type (launch / cinematic / music video / trailer or ad / UGC / explainer), platform,
length, who speaks, budget cap in dollars}

If the video-sound skill is installed, follow it. Otherwise fetch its scripts and references first:
https://github.com/vargHQ/skills/tree/main/video-sound

1. Sound plan before picture. Write sound/plan.md: one row per beat with time, picture, voice line, music, effects,
   ambience and planned silence. For each music cue: start, stop, emotion, intensity, why now. Pick only 3-6 moments
   that get an accent and one silence. Never more than 5 sound layers in any 5 seconds. Say what leads: music,
   voice, or room and silence.

2. Lines for the ear. Write sound/lines.md: id, speaker, text exactly as sent, direction, pause after, shot. Plain
   spoken words, one idea per sentence, one breath per sentence; cut every line the picture already says. Direct
   with behaviour, not emotion words. Mark pauses, breaths and small actions with the model's documented tags
   (Eleven v4: ..., line breaks, [pause], [sighs], [exhales], [whispers], a bracketed stage note; no SSML <break>).
   STOP: show plan.md and lines.md.

3. Cast from candidates. Render the hardest line in 3-5 voices, 2 takes each. Measure every take (length, words
   per minute, pauses, pitch spread; transcript must match) and build one audition page. Quote the price first.
   STOP: I listen and pick.

4. Record all lines in the chosen voice. Transcribe with word timestamps, re-render only wrong lines, keep quiet
   breaths. Every cut lands on a word boundary or a pause.

5. Music. Prompt genre, mood, instruments, BPM and key as numbers, and era; narrate the arrangement ("start with just
   ... then bring in ..."); never name an artist or song. Measure what came back, fit it to length on downbeats
   keeping the real ending, and make stops, drops and exact length in the edit.

6. Effects only for events with a visible cause; about 2 whooshes per film. Measure every file; align impacts by
   onset and whooshes by their loudest point. Room tone under everything.

7. Mix and check from sound/cues.json: music ducked hardest in 300 Hz-4 kHz, fast down and slow up, faded edges,
   mastered to -14 LUFS / -1 dBTP for social. Fix every failure: loudness, true peak, voice 10 dB over music in the
   speech band, effects within one frame, cuts on the beat, no dead air, mono safe. Audit the cheap tells.

8. STOP: make a preview video of the sound (waveform, spectrogram, playhead) and the muxed video. I listen on
   headphones and a phone. Apply my notes, re-check, deliver.

Never: an effect on every cut, music louder than words, lyrics under narration, a fade-out ending, dead digital
silence, cloning a voice without consent, claiming you listened.

Sources

All 642 findings, each with its verbatim quote, author, credentials and link, are in the skill's references/evidence.md. The raw research, the six reports, the expert roster with engagement flags and the test files are in vargHQ/aeo #22. The pages this guide quotes most:

Film, trailer, ad and radio craft

Model and platform docs

Agent tools and repos

Practitioners on X (verified credits): Henry Daubrez, PJ Ace, Uncanny Harry, Mac Smith, Gossip Goblin, MetaPuppet, bruno (tvnxty), Mario Joos

LLM-as-a-judge

Our tests (2026-11-03, about $0.60 on varg): the voice line test, the music prompt test, five generated effects and the designed vs plain launch soundtrack. Files and numbers: RESULTS.md; the soundtrack rebuilds from examples/launch-20s.

Go deeper

All 10 steps

The route, with the one line that hands it to an agent: How to make a video with an AI agent in ten steps

  1. How to find references for an AI video
  2. How to make a moodboard for an AI video
  3. How to write a video script with an AI agent
  4. How to design sound and voice for an AI video (this page)
  5. How to design characters and props for AI video
  6. How to previz a video in Three.js or Blender
  7. How to animate a previz with AI video models
  8. How to edit a video with HyperFrames and AI agents
  9. How to judge an AI video: measure, don't grade
  10. How to keep an AI video project in GitHub