AI tools continue to reshape how films are made, offering new workflows that integrate text, image, and video generation. Joey breaks down the current landscape of AI filmmaking, exploring practical steps, key tools, and challenges creators face in this evolving domain. This overview highlights how to maintain visual consistency, develop characters, and leverage emerging video-to-video models to streamline production.
Understanding the AI Filmmaking Workflow
The typical AI filmmaking workflow follows a logical progression from ideation to final video output. Creators begin with story development and shot design, move through visual style and character consistency, generate initial frames, and then transform those into moving images. Despite rapid tool evolution, the fundamental workflow remains stable, adaptable across tools and needs.
Joey points out that many creators in competitions like Cinema Synthetica use some variation of this flow: text prompts lead to images, which then become videos. This approach allows filmmakers to create compelling visuals even without cameras or physical resources, relying solely on AI-driven generation.
Story and Shot Design: The Starting Point
Every film begins with a solid story. Once the narrative is clear, shot design becomes the next focus. AI language models (LLMs) like Google Gemini 2.5, Claude, or ChatGPT assist by brainstorming shot ideas and generating detailed prompts for image generation.
For example, if you envision an action sequence—a character chased through a mine tunnel on a mine cart—LLMs can help define what shots are needed and provide precise text prompts to feed into image generation tools.
Establishing a Consistent Visual Style
Maintaining a consistent style across shots is crucial to avoid jarring visual shifts. There are several strategies to achieve this:
Descriptive Text Prompts: Use detailed language to specify the desired style.
LoRA Models: Train lightweight models on a set of images representing your style, such as a cyberpunk cityscape, to guide generation.
Midjourney SREFs: Utilize style reference codes to recall specific visual aesthetics.
Reference Images: Tools like Runway References and Flux Kontext can apply the aesthetic of an existing image to new content using style transfer techniques.
Joey emphasizes creating original styles ethically by generating images specifically for training, rather than borrowing copyrighted movie imagery. Style transfer, a technique familiar from projects replicating styles like Miyazaki’s Spirited Away, applies consistent filters to maintain a unified look.
Character Consistency and Development
Characters must appear consistent across shots, especially when faces and performances are central. AI workflows often combine these approaches:
Descriptive Text Prompts: For generic characters, detailed prompts can suffice.
LoRA Models: Train character-specific LoRA models to maintain facial and physical consistency.
Character Sheets: Create pose sheets with front, side, and expression views to guide generation across angles.
Photo Inputs: Some tools like Runway can recreate a character from a single photo.
Joey explains that the LoRA model is trained first to establish the character's base look, followed by a pose sheet that guides the character’s appearance in different positions and angles. This layered approach helps AI maintain the character’s identity through various shots.
He warns about the "uncanny valley" effect: slight imperfections in faces can break audience immersion, so creators must invest time ensuring natural and believable character appearances.
