He demos OpenArt where you can train a consistent character from:
a text prompt,
a single image, or
multiple images
He says, “The character weight slider controls how strongly your character’s features are preserved in the generated image. At higher values like 0.8 or 0.9 your character’s features will be strongly preserved, resulting in very consistent appearances…. Next is the preserve key features toggle that when turned on instructs the AI to maintain a very consistent appearance, particularly for elements like clothing, hairstyle and accessories. When turned off you can change their clothing and environment while keeping their face consistent.”
And concludes:
“I’ve tested pretty much every AI platform out there and I can honestly say that OpenArt is by far the best for creating consistent characters. Nothing else even comes close.”
My take: one of the neat things on the OpenArt home page is the “See what others are creating” section that lets you know the models and prompts other artists are using. I do wish Roboverse’s text on screen didn’t flicker – cuz it tires my eyes.
“Gems let you customize Gemini to create your own personal AI expert on any topic, and are starting to roll out for everyone at no cost in the Gemini app. Get started with one of our premade Gems or quickly create your own custom Gems, like a translator, meal planner or math coach. Just go to the “Gems manager” on desktop, write instructions, give it a name and then chat with it whenever you want. You can also upload files when creating a custom Gem, so it can reference even more helpful information.”
Some of the pre-made Gems:
Brainstormer: Helps generate ideas and concepts.
Career guide: Assists with career planning and job searches.
Coding partner: Provides support for coding tasks.
Learning coach: Helps with studying and learning new topics.
Writing editor: Assists with grammar, style, and clarity.
Google suggests using this format when writing instructions: Persona / Task / Context / Format. For instance, this is their prompt for Brainstormer:
Persona
Your purpose is to inspire and spark creativity. You’ll help me brainstorm ideas for all sorts of things: gifts, party themes, story ideas, weekend activities, and more.
Task
Act like my personal idea generation tool coming up with ideas that are relevant to the prompt, original, and out-of-the-box.
Collaborate with me and look for input to make the ideas more relevant to my needs and interests.
Context
Ask questions to find new inspiration from the inputs and perfect the ideas.
Use an energetic, enthusiastic tone and easy to understand vocabulary.
Keep context across the entire conversation, ensuring that the ideas and responses are related to all the previous turns of conversation.
If greeted or asked what you can do, please briefly explain your purpose. Keep it concise and to the point, giving some short examples.
Format
Understand my request: Before you start throwing out ideas, clarify my request by asking pointed questions about interests, needs, themes, location, or any other detail that might make the ideas more interesting or tailored. For example, if the prompt is around gift ideas, ask for the interests and needs of the person that is receiving the gift. If the question includes some kind of activity or experience, ask about budget or any other constraint that needs to be applied to the idea.
Show me options: Offer at least three ideas tailored to the request, numbering each one of them so it’s easy to pick a favorite.
Share the ideas in an easy-to-read format, giving a short introduction that invites me to explore further.
Location-related ideas: If the ideas imply a location and, from the previous conversation context, the location is unclear, ask if there’s a particular geographic area where the idea should be located or a particular interest that can help discern a related geographic area.
Traveling ideas: When it comes to transportation, ask what is the preferred transportation to a location before offering options. If the distance between two locations is large, always go with the fastest option.
Check if I have something to add: Ask if there are any other details that need to be added or if the ideas need to be taken in a different direction. Incorporate any new details or changes that are made in the conversation.
Ask me to pick an idea and then dive deeper: If one of the ideas is picked, dive deeper. Add details to flesh out the theme but make it to the point and keep the responses concise.
My take: Google’s Gems are similar to OpenAI’s CustomGPTs. I’ve made a few for my own use and they work very well. Even in a free Google account. Canada now has a federal government Minister of AI and Digital Innovation – maybe it’s time to bite the bullet and start exploring?
He has a list of 18 categories of things he feels filmmakers need to specify and,
“Number one and number two for me are gaze control and expression control.”
He explains:
“The reason you need gaze control or eye control is because where a character is looking in a story tells you everything about what they want or what they don’t want, what they’re afraid of. It shows you what their desires are, what their hopes, their dreams, their aspirations, the thing that they’re working towards. The thing that is most important to them in any given moment is revealed through what they are looking at.”
Dzine to the rescue! See their new Face Kit Expression Edit in action below.
He squeals, “She’s looking at the guy. She’s looking at the guy. She’s looking at the guy!”
Here’s the full tutorial:
My take: Hayden is right. More control is critical for all AI filmmakers.
“We’re already to the point where you can make videos indistinguishable from reality or create entire short films and this will only keep getting better.”
My take: Very interesting to see where we are today — and arguably these are not the latest cutting-edge tools.
He completes the package by taking us behind the scenes to reveal his workflow:
The software or services he used and their cost per month (or for this project)? See below:
Midjourney – $30 (images)
Gemini – free (prompts)
ElevenLabs – $22 (voice)
Hume – free (voice)
Udio – $10 (music)
Hedra – $10 (lip sync)
Premiere – $60 (NLE)
RunwayML – $30 (stylize)
Magnific – $40 (creative upscale)
Veo 2 – $1,500 (video at 50 cents/second)
Topaz – $300 (upscale) TOTAL – $2,002 (plus 40 hours of Tim’s time)
In addition to the great AI news and advice, Tim is actually funny:
“At some point in the process Gemini and I definitely got into a bit of a groove and I just ended up ditching the reference images entirely. I have often said that working this way kind of feels a bit like being a writer/producer/director working remotely with a film crew in like let’s say Belgium and then your point of contact speaks English but none of the other department heads do. But like with all creative endeavours you know somehow it gets done.”
My take: Tim’s “shooting” ratio worked out to about 10:1 and there are many, many steps in this work flow. Basically, it’s a new form of animation — kinda takes me back to the early days of Machinima, that, in hindsight, was actually more linear than this process.
1/ If you’re not using a LLM (Gemini, ChatGPT, whatever), you’re doing it wrong.
VEO 2 currently has a sweet spot when it comes to prompt length: too short is poor, too long drops information, action, description etc. I did a lot of back and forth to find my sweet spot, but once I got in a place I thought felt right, I used a LLM to help me keep my structure, length, and help me draft actions. I would then spent an extensive amount of time tweaking, iterating, removing words, changing order, adding others, but the draft would come from a LLM and a conversation I built and trained to understand what my structure looked like, what was a success, or a failure. I would also share the prompts working well for further reference, and sharing the failures also for further reference. This would ensure my LLM conversation became a true companion.
2/ Structure, structure, structure
Structure is important. Each recipe is different but same as any GenAI text-to something, it looks like the “higher on the prompt has more weight” rule applies. So, in my case I would start by describing the aesthetics I am looking for, time of day, colors, mood, then move to camera, subject, action, and all the rest. Once again, you might have a different experience but what is important is to stick to whatever structure you have as you move forward. Keeping it organized also makes it easier to edit later.
3/ Only describe what you see in the frame
If you have a character you want to keep consistent, but you want a close-up on the face for example, your reflex will be to describe the character from head to toe and then mention you want a close-up…It’s not that simple. If I tell VEO I want a face close-up but then proceed to describe the character’s feet, the close-up mention will be dropped by VEO… Once again, the LLM can help you in this by giving it the instruction to only describe what is in the frame.
4/ Patience
Well, it can get costly to be patient, but even if you repeat the same structure, sometimes changing one word can still throw the entire thing out and totally change the aesthetics of your scene. It is by nature extremely consistent if you conserve most words, but sometimes it happens. In those situations, trace your steps back and try to figure out which words are triggering a larger change.
5/ Documenting
When I started “Kitsune” (and did the same for all others), the first thing I did was start a Figjam file so I could save the successful prompts and come back to them for future reference. Why Figjam? So I could also upload 1 to 4 generations from this prompt, and browse through them in the future.
6/ VEO is the Midjourney of video
Currently, no text-to-video tool (Minimax being the closest behind) gave me a feeling I could provide strong art directions and actually get them. I have been a designer for nearly 20 years, and art direction to me has been one of the strongest foundations of most of my work. Dark, light, happy, sad, colorful or not, it doesn’t matter as long as you have a point of view and please…have a point of view. Recently watched a great video about the slow death of art direction in film (link in comments) and oh boy, did VEO 2 deliver on giving me the feeling I was listened. Try starting your prompts with different kinds of medium (watercolor for example), the mood you are trying to achieve, the kind of lighting you want, the dust in the rays of light, etc… which gets me to the next one
7/ You can direct your colors in VEO
It’s as simple as mentioning the hues you want to have in the final result, in which quantity, and where. When I direct shots, I am constantly describing colors for two reasons: 1. Well, having a point of view and 2. reaching better consistency through text-to-video. If I have a strong and consistent mood but my character is slightly different because of text-to-video, the impact won’t be dramatic because a strong art direction helps a lot with consistency.
8/ Describe your life away
Some people asked me how I achieved a good consistency between shots knowing it’s only text-to-video and the answer is simple: I describe my characters, their unique traits, their clothing, their haircut, etc..anything which could help someone visually impaired have a very precise mental representation of the subject.
9/ But don’t describe too much either…
It would be magical if you could stuff 3000 words in the window and have exactly what you asked for, right? Well, it turns out VEO is amazing with its prompt adherence, but there is always a moment where it starts dropping animations or visual elements when your prompt stretches for a tad too long. This actually happens way before the character limit allowed by VEO is reached, so don’t overdo it, it’s no use and will play against the results. For info, 200-250 words seems like a sweet spot!
10/ Natural movements but…
VEO is great with natural movements and this is also one of the reasons why I used it so extensively: people walking don’t walk in slow-motion. That being said, don’t try to be too ambitious on some of the expected movements: multiple camera movements won’t work, full 360 revolutions around a subject won’t work, anime-style crazy camera movements won’t work, etc… what it can do is already great, but there are still some limitations…
Rory Flynn has shared a workflow that uses a combination of AI tools to create aerial clips.
CLAUDE + MAGNIFIC + RUNWAY WORKFLOW
3D-to-Video (full tutorial) ↓
PROCESS:
01. Build 3D Renders in Claude 3.7
02. Program camera movements
03. Screen record render
04. Upload video to Runway Gen-3
05. Extract 1st frame
06. Magnific Struct. Ref. 1st frame
07. Upload in Runway… pic.twitter.com/PriqPhnKif
Apply a Magnific Structure Reference to the first frame
Upload this new first first frame in Runway
Apply the new first frame to the initially rendered video using Runway Restyle.
The Claude prompt he used in Step 1 is: “can you code a 3d version of [subject + env] in three.js?” E.g. “can you code a 3d version of an epic castle atop a mountain plateau in a valley in three.js?”
The Magnific Structure Reference he used in Step 6 is: “editorial photo, epic castle on a plateau, intricate rocky textures and fine details, immaculate New Zealand landscape, white marble castle, high precision photography” with these settings:
