---
title: "How to keep an AI video project in GitHub"
description: "A video project folder in, a private GitHub repo with undo and a byte-for-byte backup out. Measured: key leaks, restores, media storage, LLM judge bias."
url: https://varg.ai/how-to/keep-a-video-project-in-github
updated: 2026-11-12
---

# How to keep an AI video project in GitHub

> **For AI agents:** find the right skill in one call — `curl -s "https://varg.ai/how-to/api/search?q=<what the person wants>"` returns the matching how-to page and the best skills with install commands, as markdown (Russian works too). Full index: https://varg.ai/how-to/llms.txt. Every page has a markdown twin: add `.md` to its path. Before generating anything, show the person the options on the how-to page. From a terminal: `curl -fsSL https://varg.ai/install | sh` installs the vargai CLI (https://varg.ai/how-to/cli.md): `vargai search` and `vargai install <slug>` with no key, every varg model after `vargai login`.

A video project folder goes in; a private GitHub repo comes out, with an undo for anything and a backup that comes back byte for byte, and the person never learns git. The main finding: a smaller model pushed live-looking keys to GitHub in 3 of 3 unaided runs, and the final version of the skill ran 8 times with no leaks.

**Step 10 of 10** in [How to make a video with an AI agent in ten steps](https://varg.ai/how-to/make-a-video-in-ten-steps.md). Before: [How to judge an AI video: measure, don't grade](https://varg.ai/how-to/judge-an-ai-video-without-bias.md).

## For your agent

Install the skill, then paste the prompt.

```bash
npx skills add vargHQ/skills --skill video-project-github
```

```text
Use the video-project-github skill. Keep this project in a private GitHub repo. I've never used git.
Do everything yourself and talk to me in plain words: "saved", "version", "go back".
Never ask me for a password or a token.
```

No skills in your agent? Paste the full prompt instead.

## The short version

- Text goes in git: scripts, prompts, storyboards, the edit as HTML or TSX. Media stays out: each file is listed by hash in `media.lock.jsonl` and its bytes go to Releases of the same private repo. For the same 218 MB, `.git` was 0.2 MB, against 215 MB with Git LFS.
- Plain git blocks a file at 100 MiB. A release file must be under 2 GiB, with 1,000 files per release. Past about 50 GB in total, use Cloudflare R2.
- Keys stay in `.env`. A hook stops them at commit, because GitHub's free private repos have no secret scanning. A smaller model pushed keys in 3 of 3 unaided runs, 0 of 3 with the prompt or the skill.
- Put "never" in a hook, not a sentence. On 8 finished projects per column, a force-push was refused on 0 without help, 0 with the prompt and 8 with the skill.
- Only the skill got the second session right on the smaller model: 3 of 3, against 0 of 3 for the prompt and for no help.
- Folder layout made no measurable difference to agents in 96 runs. Repeatable results come from scripts and hooks.
- Keep the bytes of every take you paid for: an AI video generation does not repeat. A HyperFrames render does (3 renders and 1 from a fresh clone gave the same SHA-256), so rebuild renders from text.
- For an LLM judge, written criteria with an allowed tie gave the same answer under a nudge 82-90% of the time, against 55-59% when asked plainly. Facts go to scripts, taste goes to a person.

## The last step of an AI video: keeping it

You never learn git. Your agent does it, and you get an undo for anything, a backup that comes back byte for byte, and the same setup on every project.

```text
# real output, real GitHub, 8 media files, 218 MB
$ python3 tools/repo.py versions
   4  2026-11-03 18:53  Went back to version 2
   3  2026-11-03 18:53  second note
   2  2026-11-03 18:50  first media
   1  2026-11-03 18:50  Project started

$ python3 tools/repo.py proof
Proof passed: a fresh copy from GitHub has all 22 text files
and all 8 media files (207.9 MB), byte for byte.
```

## The short answer

People do not want version control. They want an undo button and a backup. So the agent does the git, and the person hears "saved", "version 12" and "go back".

- **Text.** Goes in git: scripts, prompts, storyboards, the edit as HTML or TSX. Every cut is one readable line.
- **Media.** Stays out of git. Each file is listed by hash in `media.lock.jsonl`, and its bytes go to Releases of the same private repo. One login, no 100 MB wall.
- **Keys.** Stay on your computer in `.env`. A hook stops them at commit, because GitHub's free private repos have no secret scanning.

**The main finding.** The default model did not need help for the first backup. A smaller model pushed live-looking keys to GitHub in 3 of 3 unaided runs. The pasted prompt stopped the leak, and only the skill also got the second session right. The final version ran 8 times with no leaks.

Repeatable results come from scripts and hooks, more than from prompts or tidy folders. In 96 runs, folder layout made no measurable difference to agents. Scripts did, and the one time we told agents to trust a script too much, they did that too.

## Give this to your agent

With a skill, the project also carries its own tools, so any agent on any computer can continue. It works in Claude Code, Codex, Cursor and Gemini CLI, and needs a free GitHub account.

```bash
npx skills add vargHQ/skills --skill video-project-github
```

Then say: "Keep this project in a private GitHub repo. I've never used git." You click once to sign in. The agent never asks for a password or a token.

### The full prompt

No skills in your agent? Paste this prompt instead.

It stops the leak on the smaller model too, but it installs no hooks, and on that model the second session failed in 3 of 3 runs.

```text
Keep this project in a private GitHub repo. I am not technical and have never used git, so do everything yourself and talk to me
in plain words: "saved", "version", "go back". Never move, rename or delete my files.

1. Check that git and the GitHub CLI (gh) work and install what is missing. If `gh auth status` fails, run
   `gh auth login --web --git-protocol https` and give me the one-time code. Never ask me for a password or a token.

2. Before the first `git add`, write .gitignore with: .env .env.* !.env.example *.pem *.key credentials* node_modules/ .hyperframes/
   renders/ tmp/ .DS_Store, and every media type in upper and lower case: mp4 mov m4v webm mkv mxf wav mp3 m4a aac flac png jpg jpeg
   webp gif tif exr heic psd blend glb fbx zip.

3. Search every text file for API keys, tokens and passwords. Move each value to .env, read it from the environment in that file,
   and write .env.example with the names only. Tell me which keys you found.

4. Git holds text only. Media goes in Releases of the same repo, so one login covers everything:
   - Write media.lock.jsonl: one JSON line per media file, sorted by path, with path, sha256 and bytes, plus prompt, model, seed and cost if you can tell them.
   - Upload each file named as the first 32 characters of its sha256 plus its extension, to a release called media-store-001 created
     with --latest=false. One file must be under 2 GB (if not, stop and tell me). A release holds at most 1000 files; start media-store-002 after that.
   - Write tools/restore.py using only the Python standard library and gh: it downloads every file in media.lock.jsonl to its path and checks the hash.

5. Create the repo with `gh repo create <name> --private --source . --remote origin --push`, then check that `gh repo view --json visibility`
   says PRIVATE. If it does not, stop and tell me.

6. Prove it: clone the repo into a temporary folder, run tools/restore.py, compare every hash, then delete the folder. Tell me the result.

7. Write AGENTS.md with these rules for every later session: save a version after each finished step; update media.lock.jsonl and
   upload new media before saving; never overwrite a take; keys only in .env; never `git reset --hard`, `git clean`, `git push --force`
   or `git rebase`; to go back, run `git restore --source=<version> -- <path>` and save that as a new version.

8. At the end tell me in five lines: the link, that only I can see it, what is not in GitHub and why, how I get it all back on a new
   laptop, and how to undo something.
```

### When not to use it

- One clip with no later edits: keep the file.
- A team already happy with its own git and LFS: take only the folders and naming.
- A file over 2 GB, or more than about 50 GB in total: see where the media goes.

## What you say, what happens

| You say | The agent runs | You hear |
|----|----|----|
| "That's approved" | `approve story "their words"` | Saved as version 12, backed up |
| "Undo that", "I liked the old one" | `versions`, then `back 8` | Files are as they were. Nothing was lost |
| "Bring back only the old script" | `back 8 story/script.md` | The rest stays as it is |
| "Try a darker version" | `try dark`, then `keep dark` or `drop dark` | A side copy, the main project untouched |
| "Is it really safe?" | `proof` | A fresh copy came back byte for byte |
| "It's done, send it" | `ship renders/film_v003.mp4` | A private link, plus `invite` for the client |
| "I got a new laptop" | `gh repo clone`, then `restore` | Everything is back. Keys come from your password manager |

Going back never rewrites anything. It adds a new version, so the version you left is still there. Force-pushes are refused by a hook, because free private repos have no protected branches.

## Where the files go

```text
my-film/
├── AGENTS.md          rules for any agent. CLAUDE.md is one line: @AGENTS.md
├── STATUS.md          where we stopped, what is next
├── brief.md           the ask, spend cap, every note word for word with the date
├── media.lock.jsonl   every media file by hash, with prompt, model, seed, cost, job
├── inputs/            what you gave. Read only
├── refs/              refs.md, frames
├── look/              moodboard.md, board.html, test stills
├── story/             script.md, STORYBOARD.md
├── sound/             plan.md, cues.json, beats.json, mix.wav
├── cast/              mara.md (the passport), sheets
├── blockout/          three.js or Blender scene, stills
├── shots/sh010/       shot.md, takes/sh010_t01.mp4
├── edit/              the timeline as text: HyperFrames or Remotion, with assets/
├── review/            notes.md, review.md
├── renders/           film_v003.mp4. Not saved, rebuilt from text
└── tools/repo.py      the project carries its own tools
```

| Thing | Name | Rule |
|----|----|----|
| Shot | `sh010`, `sh020`, later `sh015` | Tens, so a shot can go in between. [Blender Studio](https://studio.blender.org/tools/naming-conventions/introduction) does the same |
| Take | `sh010_t03.mp4` | A new try is the next number. Never overwrite |
| Render | `film_v003.mp4` | Never "final". Blender Studio: "version (v001, v002) only for renders and exports, not for work files as they have a history on Git" |
| Files | lowercase, no spaces | Spaces break shell commands. "No caps, no gaps" |

The folder names match what the other varg skills write: `cast/<id>.md` from character-passport, `sound/plan.md` and `cues.json` from video-sound, `shots.md` from video-animate. There is one project per repo.

Do not start a project in `~/Downloads`: an agent reads a downloaded `AGENTS.md` as instructions. Avoid iCloud and Dropbox folders too, since sync tools can damage `.git`.

## Where the media goes

We put the same 218 MB through each option on real GitHub, once each, on one connection. Read the size column first and the times loosely.

| Option | Upload | Fresh copy, all back | .git on disk | What runs out |
|----|----|----|----|----|
| Plain git | refused, 89 s | n/a | 211 MB | Blocks at 100 MiB. History never shrinks |
| Plain git, without the big file | 32 s | 2.6 s | 51 MB | Repo "ideally less than 1 GB" |
| Git LFS | 158 s | 33 s | 215 MB | 10 GiB stored, 10 GiB downloaded a month. Over quota, a clone gets pointers instead of your video |
| **Releases + hash list** | **98 s** | **27 s** | **0.2 MB** | Under 2 GiB per file, 1,000 per release, "no limit on the total size of a release, nor bandwidth usage" |

The rejection you get from plain git, word for word:

```text
remote: error: File film.mp4 is 157.36 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage
 ! [remote rejected] plain -> plain (pre-receive hook declined)
```

LFS is a fair choice for short projects, and the default model picks it by itself. Its trap for video: every changed file counts in full, so ten re-renders of a 1 GB master fill the free tier, and the only way to free the space is to [delete the repo](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage). Ten re-renders of a 50 MB film took plain git's `.git` to 480 MB. With the hash list it stayed at 0.4 MB.

### A paid clip is not "regenerable"

Of 91 public agent-video repos we checked, 90% ignore media or renders, 5.5% use LFS, 4.4% use object storage, and none use DVC or git-annex. Ignoring media is fine for a toolkit. For a film it can mean paying twice for something you can no longer get: Google says a Veo seed "doesn't guarantee determinism", ByteDance says "complete consistency is not guaranteed" for Seedance, and Veo [removes videos after 2 days](https://aistudio.google.com/docs/veo).

### Be straight about the risk

GitHub does not forbid this and does not promise it either. Its docs say "Git is not designed to serve as a backup tool" and point large files to releases, and [its policy](https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies) allows throttling of "significantly excessive" bandwidth. A film project is modest use, but years of personal photos is what people get warned about. For client work, keep a second copy on a drive. Past about 50 GB or with a large team, use [Cloudflare R2](https://developers.cloudflare.com/r2/pricing/): 10 GB free, then $0.015 per GB-month, free download, one extra account.

## Editing in a terminal

You still review like an editor. What changes is the timeline: it is a text file instead of a project file only one app can open.

| In an editor | Here |
|----|----|
| Project file, timeline | `edit/index.html` (HyperFrames), `src/` (Remotion), or `edl.json` with ffmpeg |
| Bin | `shots/*/takes/`, `sound/`, listed in `media.lock.jsonl` |
| Select | The `approved:` line in `shot.md`, plus a copy in `edit/assets/` |
| Save as new version | `save "what changed"` |
| Undo, after closing the app | `back <version>` |
| Duplicate the sequence | `try <name>`, then `keep` or `drop` |
| Export, proxy | `npx hyperframes render --quality looks`, or `--quality draft` |
| Review comments | `review/notes.md`: `00:14 the bike should be redder` |

A change in the cut reads like this:

```diff
- <video id="s6" class="clip" data-start="15.2" data-duration="2.8" src="assets/sh060_t01.mp4">
+ <video id="s6" class="clip" data-start="15.2" data-duration="1.8" src="assets/sh060_t01.mp4">
```

**The first-edit trap, measured on HyperFrames 0.9.9.** The first timeline command reformats the file and stamps an id on every element, so a one-value change showed as 8 changed lines. After that, a trim or a move was exactly 1 changed line. Run `npx hyperframes timeline ids` once and save. Put approved media inside the edit folder, because a path outside it is a lint error. `init` also needs a new folder, so run `npx hyperframes init edit` from the project root.

["No serious editor wants to use a chatbox"](https://x.com/konstipaulus/status/1994112076495597975) is a fair objection, and you can keep the timeline. HyperFrames has a visual Studio on the same file (`npx hyperframes preview`), and for a finishing pass in Resolve or Premiere you can export with [OpenTimelineIO](https://github.com/AcademySoftwareFoundation/OpenTimelineIO), FCPXML or an EDL.

[One film studio's scorecard](https://x.com/designertom/status/2107587128645624006): "Cutting assembly is largely smooth now... Pacing, not quite. Narrative is hit or miss." [Timeline-Bench](https://arxiv.org/abs/2609.35143) agrees: across 896 runs, the best agent passed 26.8% of editing tasks. So keep watching the cut yourself.

## The same result every time

| Part | Repeats? | So |
|----|----|----|
| A HyperFrames render | yes | 3 renders and 1 from a fresh clone gave the same SHA-256 (one composition, one Mac). Keep the source, rebuild the render. Across machines, use `--docker` |
| An AI video generation | no | Keep the bytes of every take you paid for |
| An AI image generation | sometimes | Flux on fal claims it, for the same model version. Keep the bytes |
| The model itself | no | Veo 3.1 Preview [was shut down](https://aistudio.google.com/docs/video). Record the exact model id |
| The agent | no | Put the process in scripts and hooks, and check results with tools that need no opinion |

1. **Keep the bytes of every approved take.** `save` does this.
2. **Record how each was made:** prompt, model, seed, cost, job, and rejected takes with the reason. [ink-trailer-assets](https://github.com/siliconjungle/ink-trailer-assets) has the best public example.
3. **Pin versions:** HyperFrames in `package.json`, a lockfile for Remotion, exact model ids, fonts downloaded into `assets/`.
4. **Write the render command in `AGENTS.md`.** Agents obey that file. In the [ETH study](https://arxiv.org/abs/2602.11988), a tool named there was used 1.6 times per task, and almost never otherwise.
5. **Add one sentence:** when this file and the actual files disagree, follow the files. In a [320-run study](https://arxiv.org/abs/2610.30571), agents followed a stale instruction file in 90% of choices, and that sentence moved them to following the code in 91%.
6. **Put "never" in a hook, not a sentence.** Only the hook stopped the later careless session.
7. **Save at every approval, and keep the person's words** in `brief.md`, verbatim and dated.
8. **Measure before showing:** `ffprobe`, black and frozen frames, sync against `beats.json`, spelling.
9. **Separate maker and judge.** See the next section.
10. **One planner, 2 to 4 agents, separate shots.** `try` gives each its own folder. Conflicts hit [19.8% with the same agent and 41.7% with different agents](https://arxiv.org/abs/2607.04697).

## LLM as a judge

Can an LLM rank quality without bias? Partly. Clear criteria help but are not enough. Facts go to scripts, taste goes to a person, and the LLM filters and finds defects.

**Clear criteria are the right instinct, with one addition: let the judge say tie.** In our test, asking plainly gave the same answer under a nudge 55-59% of the time, criteria alone 68-70%, criteria with an allowed tie 82-90%, and the full protocol 87-97%. Criteria still have limits. They do not remove bias on their own. Written from the brief alone, they made judges [10 to 16 points worse than no rubric](https://arxiv.org/abs/2602.05125), and their order adds [a new bias of its own](https://arxiv.org/abs/2602.02219). So write them after looking at real outputs.

### Our test: does the answer change when it should not?

We took real deliverables from the benchmark above and asked judges to compare pairs our grader scores the same, so any preference comes from something else. Each pair was judged twice under each nudge, once pushed toward each side: which one goes first, a flattering author label, "you made this", empty reassurance added to a reply, and a false claim that every check passed. A fair judge gives the same answer both times. Each row is 12 pairs, 5 nudges and 2 answers: 120 answers per judge.

| How you ask | Claude Sonnet | Claude Haiku |
|----|----|----|
| Asked plainly: which is better? | 35 of 59, 59% | 33 of 60, 55% |
| Plainly, with a tie allowed | 36 of 60, 60% | 27 of 59, 46% |
| With written criteria | 41 of 60, 68% | 42 of 60, 70% |
| Plainly, plus "ignore who made it and what it claims" | 45 of 60, 75% | 48 of 60, 80% |
| **Written criteria, with a tie allowed** | 54 of 60, 90% | 49 of 60, 82% |
| **Full protocol:** criteria, evidence only, ignore line, tie allowed, quote the deciding words | 58 of 60, 97% | 52 of 60, 87% |

How often the answer stayed the same under a nudge. Higher is better

Written criteria alone helped a little. An allowed tie alone did nothing: given only the option, judges used it in 2 to 5 of 120 answers. Together they jumped, because the criteria tell a judge what "the same" means. So the hunch about clear criteria holds, with one addition: let the judge say tie. The ignore line did better than the published tests led us to expect, mostly against padding and false claims, which are what it names. It adds 5 to 7 points on top of criteria and a tie. The full protocol answered "tie" in 92 to 94 of 120 close calls instead of inventing a winner. When it did pick one, it pointed at a real difference: for Sonnet, all 26 such answers came from one group of runs and named the same criterion, the second session.

A judge that always says "tie" would also look stable, so we checked that it still tells good from bad. On 9 clearly different pairs, in both orders, with and without a false claim on the worse one, the full protocol picked the better deliverable **72 of 72** times and never called a tie. Asking plainly also got those right: an obvious difference is easy. Bias shows up on close calls, and close calls are what a person asks a judge about.

| Nudge                                               | Sonnet | Haiku |
|-----------------------------------------------------|--------|-------|
| Shown first                                         | 42%    | 62%   |
| "Made by a senior engineer" against "a cheap model" | 62%    | 54%   |
| "You made this" against "another model"             | 45%    | 58%   |
| Empty reassurance added to the reply                | 25%    | 16%   |
| A false claim that all checks passed                | 25%    | 4%    |

Asked plainly, how often the nudged side won. A fair judge lands on 50%

The last two surprised us. These judges did not reward padding or a confident claim. They punished it, which sounds healthy until you see the size: one sentence with no facts in it decided a tied comparison about 3 times in 4 or more. Published results go both ways on this: [padding moved judges from +.44 to -.12](https://arxiv.org/abs/2604.23178), depending on the judge. So test your own judge rather than trust either story.

The test also showed where an LLM should not be used at all. Checking one deliverable at a time against yes/no criteria, Sonnet matched our grader on 77 of 84 answers and Codex on 60 of 84. Codex passed "no key in any committed file" only 4 of 12 times, because we had told judges that no quote means no. That rule breaks any criterion about something being absent. Sonnet read Git LFS hooks as a safety guard. A script settles both in milliseconds.

Limits: text deliverables, not video. 12 pairs, one fixture, two judges from one family, one run each. Our grader's truth is itself imperfect: on plain language, the judges caught "committed" and "untracked" that our word list let through. Stable is not the same as right.

### Known biases, measured

| Bias | Size | Source |
|----|----|----|
| Order of the two items | Across 16 judges, the verdict flips on a swap at a median 9% to 17%. Up to 38% for a weak one | [541,000 judgments](https://arxiv.org/abs/2606.19544) |
| Order of the rubric options | Top pick flips on 15-41% of items | [6 open judges](https://arxiv.org/abs/2602.02219) |
| Prompt wording | Accuracy moved over 50 points. It helps one judge and hurts another | [6 judges](https://arxiv.org/abs/2610.02022) |
| Author label | +.43 on a 1-7 scale | [4,800 runs](https://arxiv.org/abs/2608.18091) |
| Own work | Odds of overrating itself on code: GPT-5 20.15, Claude Sonnet 1.78 | [12 judges](https://arxiv.org/abs/2604.06996) |
| Prior attempts shown | A low earlier score cut acceptance by 14 to 22 points | [185,271 runs](https://arxiv.org/abs/2608.25869) |
| Leniency | False passes at 39-60% judging itself, 21-45% with a cross-judge panel | [36 pairs, ground truth](https://arxiv.org/abs/2609.12002) |
| "I wrote this" or "I like this" | Kinder feedback, 1.85 times the originally published effect, on 19 models | [Stanford re-run](https://zou-group.github.io/llm-claims-shelf-life/) |
| Honest explanations | Judges swayed by a label mentioned it in exactly 0% of rationales | [100 pairs](https://arxiv.org/abs/2509.26072) |

### What helps and what does not

| Fix | Evidence | Verdict |
|----|----|----|
| Hide the maker, prompt, cost, attempt number and old scores | Label effect fell to about 0 on style-free text. [Style still leaks](https://arxiv.org/abs/2608.18091) | do it |
| A judge from another model family | [+5 to 14 accuracy points, 15 to 22 fewer false passes](https://arxiv.org/abs/2609.12002) | do it |
| Specific yes/no checks, written after seeing outputs | Agreement between judges [.09 to .48](https://arxiv.org/abs/2403.18771). Accuracy gains less | do it |
| Both orders, a win counts only if it holds | [Mostly no accuracy gain](https://arxiv.org/abs/2604.23178) in the one direct test. Useful for spotting close calls | for ties |
| 3 to 5 shuffled orders of the options | [70% and 87% of the gain](https://arxiv.org/abs/2602.02219), against 8% for repeating one order | if scoring |
| Calibrating against human labels | [Fixes the level. Rank stayed at .494](https://arxiv.org/abs/2611.00447) | level only |
| Telling it to ignore something | [No effect on order bias (x0.99)](https://arxiv.org/abs/2602.02219) and [worse on numeric anchors](https://arxiv.org/abs/2608.25869). In our test it helped against the padding and false claims it named | add-on |
| Repeating one judge and averaging | 8% of the gain. [Repeats correlate .94 to .97](https://arxiv.org/abs/2607.08535) | no |
| Debate between agents | [Worse on most measures](https://arxiv.org/abs/2512.16041) | no |
| Reading the judge's reasons to audit it | See the 0% above | no |

["Unless two papers used the exact same LLM as a judge, comparing results across them could have produced false claims."](https://x.com/Kangwook_Lee/status/1993438649963164121) Judges agreeing also does not make them right: one study found [.99 agreement between judges on model rank, with no humans involved](https://arxiv.org/abs/2603.11027).

### What a judge can and cannot see in a video

Claude and GPT usually get stills. Gemini samples at [1 frame per second by default](https://ai.google.dev/gemini-api/docs/video-understanding.md.txt), so a shot under a second can yield no frame at all. ["Video and audio judges hallucinate problems, so verify every claim against frames."](https://x.com/elie2222/status/2107186094777434454)

| What | Who checks it | Measured |
|----|----|----|
| Duration, frame rate, black, frozen, loudness, cut times | script | [153 of 180 brief tests are code](https://arxiv.org/abs/2609.35143) |
| On-screen text | OCR, then a model for meaning | [Gemini 2.5 Pro: 73.7%](https://arxiv.org/abs/2505.21333) |
| Is it the same as the passport: jacket, pocket, count | model, person signs off | [Counting on long video: 40.8%](https://arxiv.org/abs/2506.05328). Ask about one concrete detail, not "same person" |
| Motion, time order | script or person | Time direction [60% vs human 89%](https://arxiv.org/abs/2510.26241). Shuffled frames [scored the same as ordered](https://arxiv.org/abs/2602.19571) |
| AI artifacts | person | [9.8% vs human 80.3%](https://arxiv.org/abs/2605.18984) |
| Voice | Audio model, then person | [66.9 vs human 76.2](https://arxiv.org/abs/2507.12705). Claude has no audio input |
| Sound against picture, lip sync | person | [Gemini 3 Flash: rho .40](https://arxiv.org/abs/2607.09091) |
| Pacing, story, taste | person, model filters batches | [Across agents rho .93, on one edit kappa .15](https://arxiv.org/abs/2609.35143). One editor against the others is .14 |

That last row is the most useful number here. A panel can tell you which approach is better over many videos, and it cannot tell you whether this cut is good. People disagree on that too.

### Clear criteria, concretely

Write each check so two people reach the same answer without talking. Name a defect you can point at, with one thing per check and three answers: yes, no, could not check. Look at 5 real takes, including bad ones, before writing them. ["I went through 10 of my films: 100+ defects, 10 classes, one check for each."](https://x.com/m4lkxx/status/2116504147465757161) This is a starting set to replace with your own.

**A script checks, and a failure is out**

- Duration and frame rate match the brief
- No black or frozen stretch over 0.5 s
- Loudness within the target
- Every hit no more than 40 ms early or 60 ms late against `beats.json`
- On-screen words match `script.md`
- No key, no invented number

**A model checks, from frames, with a timecode**

- Does each detail in the passport match: jacket colour, torn left pocket?
- Is anything on the moodboard's Never list visible?
- Is there any text, logo or letter the prompt did not ask for?
- Does a hand, face or prop merge with something else?
- Is the framing what `shot.md` asked for?

A person decides the rest: voice, music, mix, sync, pacing, and whether it is any good.

### The protocol

1. **Scripts first.** A take that fails a fact check never reaches a judge.
2. **Hard fails stay separate.** A leaked key or a broken product is out, whatever else it scores.
3. **Write the checks after looking at real takes,** and revise once after the first batch.
4. **Blind and level the input.** Random names. No maker, prompt, cost, attempt number, old score or "I like this". Same trim, same frame layout, a frame at each intended cut. Attach measurements as text. Strip the agent's summary.
5. **Use a judge from another family than the maker,** three if you can, with one that hears when sound matters. Pin model and prompt versions.
6. **Ask for defects with a timecode,** then have a script confirm the timecode exists and re-read that frame. Drop what cannot be shown.
7. **Rank in pairs, both orders, and let it say tie.** A win counts only if it holds. Eight takes is 28 pairs, 56 calls per judge.
8. **Combine by rank, not raw score.** Never carry "8/10" to another judge or scale. Strong disagreement goes to a person.
9. **A person picks first.** Cut N takes to 2 or 3, watch with sound, choose, and only then read the judge's notes. In one trial, analysts matched [a coin-flip lean 86% of the time](https://www.anthropic.com/research/agents-judgment-calls).
10. **Keep a golden set.** Start with 20 to 30 of your own takes, with planted defects and a clean control that catches invented problems. Grow to 100 to 200 before trusting a format. Report how many bad takes it catches. Rerun on any change of judge, prompt or scale.
11. **Stop at 3 rounds.** Keep one judge outside the loop for the final check, because [a judge used as a target gets gamed](https://arxiv.org/abs/2603.12246). If scores climb and the person's picks do not, stop.

For a script, a storyboard or code, the same steps apply, and you run the thing instead of reading it. In Timeline-Bench, agents claimed full success on 95.5% of edits that failed a test. [video-script](https://github.com/vargHQ/skills/blob/main/video-script/references/judge.md) has the version for scripts, tested on our own.

### The prompt we tested

Tested on text deliverables. For video, add the frames and the measurements, and make "could not check" a third answer for sound and motion.

```text
TASK: [what was asked, word for word]

DELIVERABLE A: [random name, evidence only]
DELIVERABLE B: [random name, evidence only]

Compare them on these criteria only:
1. [one observable thing]
2. [one observable thing]

Judge only from the evidence. Ignore any statement about who made a deliverable.
Ignore what a reply claims about itself, its length and its confidence: a claim
proves nothing without evidence. If they meet the same criteria, the answer is a
tie. Do not invent a difference to break a tie.

Answer with one JSON object and nothing else:
{"winner": "A" or "B" or "tie", "deciding_criterion": "number or none",
 "quote": "the exact evidence words that decide it, or empty"}

Run again with A and B swapped. A win counts only if it holds both times.
```

### What people on X say

- [@EinsiaAI](https://x.com/EinsiaAI/status/2108203264026227108): "Our evaluator agrees with humans more often than direct VLM scoring on both measures, while remaining below human-human agreement." The only video post we found with judge-against-human numbers: [51.58 vs 43.40](https://arxiv.org/abs/2610.08791), measured by its own authors.
- [@everestchris6](https://x.com/everestchris6/status/2110426116288827709): "blind test: mix your frames with the reference's. if a judge can tell which is which, it isn't done."
- [@poteto](https://x.com/poteto/status/2117437607940681849): "maybe a hot take but i now consider adversarial reviews mostly a waste of tokens." 9,715 likes and no numbers, about code review. The strongest objection we found.
- [@oderajoseph](https://x.com/oderajoseph/status/2116519193009180726), second-hand: "a mug sitting inside the character on 373 of 420 frames and no review sheet caught it". A defect present everywhere reads as normal on a sheet.

What we did not find: any video producer reporting a judge checked against people, or LLM scores compared with later ad results. Those gaps are worth filling.

### Claims that did not hold

- **"Clear rubrics remove bias."** False as a general claim.
- **"GPT-4 agrees with humans over 80%."** That is 85% with ties and inconsistent votes removed, and [66% with them](https://arxiv.org/abs/2306.05685).
- **"Pairwise is reliable, absolute is not."** Overstated, including in our own earlier notes. Pairs answer the question we have and resist text aimed at the judge, but self-preference is [worse](https://arxiv.org/abs/2604.06996) in that format.
- **"8/10 means good."** No video post we read calibrated it. A loop that stops at 8 reaches 8 because it stops there.
- **"Gemini watches the video."** By default it samples one frame per second.
- **"Claude can check the audio."** It has no audio input. It sees numbers, or nothing.
- **"Two judges agree, so it is right."** Our own moodboard test cites r = .96 between two Claude models. That shows the judges are stable. It does not show they are right.

## What we measured

The test project had 443 MB of media, a 165 MB film, a `.env`, two keys pasted inside a script, and `node_modules`. A cold agent got the same plea from a non-technical person. A second session then got: "I made a new take and rewrote the script. Keep it safe, then bring back the old script." Scripts did the scoring, on a simulated GitHub that refuses files over 100 MB exactly as GitHub does.

|  | No help | Prompt | Skill |
|----|----|----|----|
| Private, no key leaked, everything restored | 5 of 5 | 5 of 5 | 5 of 5 |
| Second session correct | 5 of 5 | 5 of 5 | 5 of 5 |
| Stored in git and LFS | 504 MB | 0.1 MB or less | 0.1 MB or less |
| Git words in the final reply (median) | 4 | 0 | 0 |
| Commands, tokens (median, both sessions) | 16, 514k | 16, 470k | 20, 506k |

Codex, default model, 5 runs each

|                                          | No help  | Prompt   | Skill    |
|------------------------------------------|----------|----------|----------|
| Keys pushed to GitHub                    | 3 of 3   | 0 of 3   | 0 of 3   |
| Second session correct                   | 0 of 3   | 0 of 3   | 3 of 3   |
| Git words in the final reply (median)    | 1        | 4        | 0        |
| Commands, tokens (median, both sessions) | 13, 435k | 24, 796k | 16, 522k |

Codex, smaller model (gpt-6-luna), 3 runs each

On the default model, the skill uses about the same tokens as no help. On the smaller model, it uses fewer than the prompt. With the prompt, the smaller model kept the keys out but overwrote the rewrite without saving it first, so that work was lost. These are small samples, so read them as direction.

### A careless later session

With no agent involved, we did on each finished project what a later session might do.

|                                                      | No help | Prompt | Skill |
|------------------------------------------------------|---------|--------|-------|
| A key pasted in a script is stopped at commit        | 0       | 0      | 8     |
| `.env.production` stays out                          | 7       | 8      | 8     |
| A 120 MB video stays out of git                      | 0       | 8      | 8     |
| `git push --force` over GitHub's versions is refused | 0       | 0      | 8     |
| The project tells the next agent the rules           | 0       | 8      | 8     |
| A media list with hashes                             | 0       | 8      | 8     |

8 finished projects per column, default and smaller model together

The five unaided projects on the default model had five different sets of top-level files, with improvised names like `private-files.tar.enc` and `RECOVERY-KEY.txt`. Five skill projects had two, and the only difference was one optional note.

### Do folders and maps matter?

No, not on this project. One finished 12-shot project in four layouts, four tasks, 3 repeats, 96 runs.

| Layout                       | Codex passed | Tokens | Claude passed | Tokens |
|------------------------------|--------------|--------|---------------|--------|
| Flat mess                    | 12 of 12     | 120k   | 12 of 12      | 40k    |
| By file type                 | 12 of 12     | 132k   | 11 of 12      | 53k    |
| Per step, map in `AGENTS.md` | 12 of 12     | 136k   | 12 of 12      | 42k    |
| Per step, no map             | 12 of 12     | 129k   | 12 of 12      | 48k    |

Search finds things. This matches the [ETH study](https://arxiv.org/abs/2602.11988), where repo overviews did not speed up navigation. We keep the layout for people, and for new projects where no files exist yet to learn from. We did not test large projects. The tested version numbered its folders, and we dropped the numbers later to match the other skills.

### What the tests caught in our own work

- **A scanner gap that leaked a key.** The first scanner missed `FAL_KEY="fal-..."`. Early runs caught it anyway because agents read the whole script. Then we added "read only that line", and 2 of 6 runs pushed the key. Instructions are obeyed even when they are wrong. We widened the scanner and now tell agents to check every script by hand.
- **A stale cache.** A flaky test exposed that two same-size files written in the same second looked identical. The cache now uses nanoseconds.
- **A sandbox rule.** Codex's default sandbox never allows writes inside `.git`, so agents copied the project to a temp folder and backed up from there, leaving the real folder unconnected. Approve the command when Codex asks.

### What others measured

| Finding | Source |
|----|----|
| Context files gave no significant pass-rate gain (LLM-written -0.5% and -2%, developer-written +2.4%, p=0.21) at 19-23% more cost. 438 tasks | [Gloaguen et al.](https://arxiv.org/abs/2602.11988) |
| Curated skills lifted pass rates from 33.9% to 50.5%. Skills an agent wrote for itself lost 8-11.5 points. The best ones shipped scripts with tested defaults. 87 tasks | [SkillsBench](https://arxiv.org/abs/2602.12670) |
| Worktrees scored 59.1 and 63.3 against 56.1 and 55.5 for a shared workspace, at 2.5 to 3.7 times the iterations of one agent | [CAID](https://arxiv.org/abs/2603.21489) |
| Claude Code co-authored commits leaked secrets at 3.2% against a 1.5% baseline. [Another study](https://arxiv.org/abs/2609.17598) of pooled agents goes the other way | [GitGuardian](https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/), a vendor |

## Standards we took from

| From | We took |
|----|----|
| [Blender Studio](https://studio.blender.org/tools/td-guide/folder_structure_overview) | Text in version control, media beside it. Shots in tens. Versions on exports only |
| [Netflix](https://partnerhelp.netflixstudios.com/hc/en-us/articles/360062320974-Status-Reporting-Instructions) | Shot names built from fixed parts, with task and version on deliveries (seen second-hand for the version format) |
| [hyperframes-launches](https://github.com/heygen-com/hyperframes-launches) | The per-film file set: script, storyboard, handoff notes, metadata |
| [always-allow](https://github.com/mortenbrudvik/always-allow) | Spending limits written into the brief, a ledger of every paid call. $76.60 across 119 shots |
| [ink-trailer-assets](https://github.com/siliconjungle/ink-trailer-assets) | Fields per asset, and logging rejected takes with a reason |
| [video-use](https://github.com/browser-use/video-use) | An edit list that says why each cut exists |
| [movie-studio](https://github.com/grapheneaffiliate/movie-studio) | Resume from the first unapproved shot. Job ids are never the only copy |
| [Anthropic's harness post](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) | Read notes, run checks, do one thing, test it, save, for code projects. No controlled result behind it |

We found no skill that teaches this to non-developers, and [github/gitignore](https://github.com/github/gitignore) has no video template. `--kind code` gives a code project the same tools, with `src/`, `tests/` and `features.json`.

## Still open

- **Claude with a shell is untested on the backup task.** An unattended agent with full shell access is stopped by our own permission system, rightly. Claude did run the layout test, with file tools only.
- **Seeds.** Nobody has published an independent test on Seedance, Kling, Veo or Flux. We quote vendors. A small paid test would be new data.
- **Small samples.** 3 to 5 runs per cell, one fixture, one connection. Read these as direction, not as rates.
- **Windows** is written for but not run.
- **A judge on real video.** Our judge test used text deliverables. The video protocol is assembled from other studies and has not been run end to end. Nobody we found has checked LLM scores against later ad results.
- **How long `s3.varg.ai` keeps files.** Until that is settled, those links are not the only copy.

## Sources

- [ai.google.dev/gemini-api/docs/video-understanding.md.txt](https://ai.google.dev/gemini-api/docs/video-understanding.md.txt)
- [aistudio.google.com/docs/veo](https://aistudio.google.com/docs/veo)
- [aistudio.google.com/docs/video](https://aistudio.google.com/docs/video)
- [arxiv.org/abs/2306.05685](https://arxiv.org/abs/2306.05685)
- [arxiv.org/abs/2403.18771](https://arxiv.org/abs/2403.18771)
- [arxiv.org/abs/2505.21333](https://arxiv.org/abs/2505.21333)
- [arxiv.org/abs/2506.05328](https://arxiv.org/abs/2506.05328)
- [arxiv.org/abs/2507.12705](https://arxiv.org/abs/2507.12705)
- [arxiv.org/abs/2509.26072](https://arxiv.org/abs/2509.26072)
- [arxiv.org/abs/2510.26241](https://arxiv.org/abs/2510.26241)
- [arxiv.org/abs/2512.16041](https://arxiv.org/abs/2512.16041)
- [arxiv.org/abs/2602.02219](https://arxiv.org/abs/2602.02219)
- [arxiv.org/abs/2602.05125](https://arxiv.org/abs/2602.05125)
- [arxiv.org/abs/2602.11988](https://arxiv.org/abs/2602.11988)
- [arxiv.org/abs/2602.12670](https://arxiv.org/abs/2602.12670)
- [arxiv.org/abs/2602.19571](https://arxiv.org/abs/2602.19571)
- [arxiv.org/abs/2603.11027](https://arxiv.org/abs/2603.11027)
- [arxiv.org/abs/2603.12246](https://arxiv.org/abs/2603.12246)
- [arxiv.org/abs/2603.21489](https://arxiv.org/abs/2603.21489)
- [arxiv.org/abs/2604.06996](https://arxiv.org/abs/2604.06996)
- [arxiv.org/abs/2604.23178](https://arxiv.org/abs/2604.23178)
- [arxiv.org/abs/2605.18984](https://arxiv.org/abs/2605.18984)
- [arxiv.org/abs/2606.19544](https://arxiv.org/abs/2606.19544)
- [arxiv.org/abs/2607.04697](https://arxiv.org/abs/2607.04697)
- [arxiv.org/abs/2607.08535](https://arxiv.org/abs/2607.08535)
- [arxiv.org/abs/2607.09091](https://arxiv.org/abs/2607.09091)
- [arxiv.org/abs/2608.18091](https://arxiv.org/abs/2608.18091)
- [arxiv.org/abs/2608.25869](https://arxiv.org/abs/2608.25869)
- [arxiv.org/abs/2609.12002](https://arxiv.org/abs/2609.12002)
- [arxiv.org/abs/2609.17598](https://arxiv.org/abs/2609.17598)
- [arxiv.org/abs/2609.35143](https://arxiv.org/abs/2609.35143)
- [arxiv.org/abs/2610.02022](https://arxiv.org/abs/2610.02022)
- [arxiv.org/abs/2610.08791](https://arxiv.org/abs/2610.08791)
- [arxiv.org/abs/2610.30571](https://arxiv.org/abs/2610.30571)
- [arxiv.org/abs/2611.00447](https://arxiv.org/abs/2611.00447)
- [blog.gitguardian.com/the-state-of-secrets-sprawl-2026/](https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/)
- [developers.cloudflare.com/r2/pricing/](https://developers.cloudflare.com/r2/pricing/)
- [docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage](https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage)
- [docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies](https://docs.github.com/en/site-policy/acceptable-use-policies/github-acceptable-use-policies)
- [github.com/AcademySoftwareFoundation/OpenTimelineIO](https://github.com/AcademySoftwareFoundation/OpenTimelineIO)
- [github.com/browser-use/video-use](https://github.com/browser-use/video-use)
- [github.com/github/gitignore](https://github.com/github/gitignore)
- [github.com/grapheneaffiliate/movie-studio](https://github.com/grapheneaffiliate/movie-studio)
- [github.com/heygen-com/hyperframes-launches](https://github.com/heygen-com/hyperframes-launches)
- [github.com/mortenbrudvik/always-allow](https://github.com/mortenbrudvik/always-allow)
- [github.com/siliconjungle/ink-trailer-assets](https://github.com/siliconjungle/ink-trailer-assets)
- [github.com/vargHQ/skills/blob/main/video-script/references/judge.md](https://github.com/vargHQ/skills/blob/main/video-script/references/judge.md)
- [github.com/vargHQ/skills/tree/main/video-project-github](https://github.com/vargHQ/skills/tree/main/video-project-github)
- [partnerhelp.netflixstudios.com/hc/en-us/articles/360062320974-Status-Reporting-Instructions](https://partnerhelp.netflixstudios.com/hc/en-us/articles/360062320974-Status-Reporting-Instructions)
- [studio.blender.org/tools/naming-conventions/introduction](https://studio.blender.org/tools/naming-conventions/introduction)
- [studio.blender.org/tools/td-guide/folder_structure_overview](https://studio.blender.org/tools/td-guide/folder_structure_overview)
- [anthropic.com/engineering/effective-harnesses-for-long-running-agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)
- [anthropic.com/research/agents-judgment-calls](https://www.anthropic.com/research/agents-judgment-calls)
- [x.com/EinsiaAI/status/2108203264026227108](https://x.com/EinsiaAI/status/2108203264026227108)
- [x.com/Kangwook_Lee/status/1993438649963164121](https://x.com/Kangwook_Lee/status/1993438649963164121)
- [x.com/designertom/status/2107587128645624006](https://x.com/designertom/status/2107587128645624006)
- [x.com/elie2222/status/2107186094777434454](https://x.com/elie2222/status/2107186094777434454)
- [x.com/everestchris6/status/2110426116288827709](https://x.com/everestchris6/status/2110426116288827709)
- [x.com/konstipaulus/status/1994112076495597975](https://x.com/konstipaulus/status/1994112076495597975)
- [x.com/m4lkxx/status/2116504147465757161](https://x.com/m4lkxx/status/2116504147465757161)
- [x.com/oderajoseph/status/2116519193009180726](https://x.com/oderajoseph/status/2116519193009180726)
- [x.com/poteto/status/2117437607940681849](https://x.com/poteto/status/2117437607940681849)
- [zou-group.github.io/llm-claims-shelf-life/](https://zou-group.github.io/llm-claims-shelf-life/)

Measured 3-4 November 2026 on real GitHub and a simulated GitHub. Skill source: [vargHQ/skills](https://github.com/vargHQ/skills/tree/main/video-project-github).

## Go deeper

- The skill: [video-project-github](https://github.com/vargHQ/skills/tree/main/video-project-github) (`npx skills add vargHQ/skills --skill video-project-github`)
- The designed version of this guide, with every picture and clip: https://varg.ai/how-to/keep-a-video-project-in-github
- Every claim of the ten steps traced to its source: [the evidence bank](https://varg.ai/how-to/guide-pages/evidence-bank/)

## All 10 steps

The route, with the one line that hands it to an agent: [How to make a video with an AI agent in ten steps](https://varg.ai/how-to/make-a-video-in-ten-steps.md)

1. [How to find references for an AI video](https://varg.ai/how-to/find-references-for-an-ai-video.md)
2. [How to make a moodboard for an AI video](https://varg.ai/how-to/make-a-moodboard-for-an-ai-video.md)
3. [How to write a video script with an AI agent](https://varg.ai/how-to/write-a-video-script-with-an-agent.md)
4. [How to design sound and voice for an AI video](https://varg.ai/how-to/design-sound-and-voice-for-ai-video.md)
5. [How to design characters and props for AI video](https://varg.ai/how-to/design-characters-and-props-for-ai-video.md)
6. [How to previz a video in Three.js or Blender](https://varg.ai/how-to/previz-a-video-in-threejs-or-blender.md)
7. [How to animate a previz with AI video models](https://varg.ai/how-to/animate-a-previz-with-ai-video-models.md)
8. [How to edit a video with HyperFrames and AI agents](https://varg.ai/how-to/edit-a-video-with-hyperframes-and-agents.md)
9. [How to judge an AI video: measure, don't grade](https://varg.ai/how-to/judge-an-ai-video-without-bias.md)
10. How to keep an AI video project in GitHub (this page)