Agentic Video Editing
AI agents that operate After Effects: they edit, render a frame, look at it, and fix their own mistakes.
Part of the same system as TikTok Agent Control. Two halves of one system. The agent that watches the account decides what to make, then hands the source and the narration over here to be edited.
- Role
- Solo: research, PRD, architecture, build
- Year
- 2026
- Type
- AI & Agents
- Status
- In progress
$ cat brief.md# In short
- The problem
- Matching the editing style of a reference video is expert work done frame by frame, and After Effects has no API an agent can call.
- What I did
- Solo: the research, PRD, architecture and build, including the ExtendScript bridge that lets an agent drive After Effects.
- Result
- In progress, with the core loop proven: the agent edits, renders a frame, looks at it with vision and corrects itself, tested with a deliberately misplaced layer it had to catch.
- Research build
- In progress; runs against a local After Effects install
$ stats --summary# In numbers
- 2
- agents: an executor, and a critic that never sees its reasoning
- Vision
- check on a rendered frame before any edit is accepted
- 6
- design docs: PRD, architecture and specs
- 23
- experiment runs so far
$ ./demo --play# Watch it run (with sound)
$ ls desktop/# On desktop
$ cat README.md# Overview
The goal is to take the editing style of a reference video (pacing, typography, motion intensity, where the emphasis lands) and apply it to new footage in After Effects, without an expert doing it frame by frame.
Agents drive After Effects through a bridge I wrote, render frames, look at them with vision, and correct what's wrong. It is also the editor behind TikTok Agent Control: that agent picks the story, the source and the narration, then hands the footage here. What comes back is not a finished file but an edit sitting in an After Effects project, ready for a person to review and export.
$ tree architecture/# How it fits together
Input
Reference video
The style to match.
New footage
The material to edit.
Agents (TypeScript + Anthropic SDK)
Style analyst
Built solo
Turns measured signals into a style profile.
Executor
Built solo
Plans the edit and applies it in After Effects.
Critic
Built solo
Isolated. Judges rendered frames against the brief and profile.
After Effects
File-IPC bridge
Built solo
ExtendScript: the only way agents can change the project. Every change is journaled.
$ cat flow.log# How it flows
- 1.
Read the reference · ffmpeg + analyst
Shots, beats and transcript are measured, then turned into a style profile.
- 2.
Plan the edit · Executor
Choices go into the Edit Plan, each with a reason.
- 3.
Apply in After Effects · Bridge
Operations run through the bridge and come back as structured results.
- 4.
Render and look · Critic
A frame is rendered and judged with vision, without the executor's reasoning.
- 5.
Revise or stop · Loop
Fix what's wrong and try again, or stop when it no longer improves.
$ git log --author=amar# What I built
- Built an ExtendScript file-IPC bridge so an agent can run After Effects operations unattended and get structured results back.
- Proved the loop with an injected fault: a text layer is secretly placed off-centre, and the only way to catch it is to render a frame and look. The agent finds and fixes it.
- Architecture uses a blackboard with typed messages, with the Edit Plan as its backbone. Every change is journaled with its rationale, for audit and rollback.
- The Critic runs in an isolated context and never sees the executor's reasoning, so it judges the output rather than the argument.
- Style profiles extracted from reference videos, with deterministic signal processing (shots, beats, transcripts) kept out of the model. Agents only handle judgment.
- No agent framework: peer-to-peer coordination is the product, so it's a few hundred lines on the Anthropic SDK.
- Wrote the PRD, architecture and specs first, and ran spikes to answer the riskiest questions before building.
$ cat fixed-bugs.md# Challenges I solved
After Effects has no API an agent can call.
An ExtendScript bridge watches a folder for requests, runs the operations inside After Effects and writes back structured results.
Tool results said everything was fine even when the frame looked wrong.
The agent has to render a frame and look at it. A test with a deliberately misplaced layer showed it catches problems that only show up on screen.
An agent judging its own work tends to approve it.
The Critic runs separately and only sees the brief, the style profile and the frames, never the executor's reasoning.
Loops that never converge waste time and money.
Plateau detection is part of the design: the system says when it stopped improving instead of quietly shipping a twelfth attempt.
$ cat learned.txt# What I learned
- Writing the PRD and architecture before code, and running spikes to rule out risky ideas early.
- Where AI models help and where plain code wins: measuring signals versus making judgments.
- Coordinating several agents without a framework.
$ cat package.json# Stack
TypeScript · Anthropic SDK · Zod · After Effects · ExtendScript · ffmpeg