Cost, Control, Quality, and Lessons from a Real Production
Artificial intelligence has rapidly changed the way music videos can be designed and produced. Visual concepts that once required large crews, complex sets, expensive locations, and extensive post-production can now be explored with a smaller team and a more flexible workflow.
However, AI is not a complete replacement for traditional visual effects.
In many professional productions, the strongest results come from combining AI-generated imagery with established VFX techniques, 3D production, compositing, rotoscoping, and careful art direction.
A recent music video project at Epiccells was a clear example of why a hybrid workflow can be more effective than choosing either AI or traditional VFX alone.
The Creative Concept
The music video was set inside a large roller-skating and arcade venue.
The story begins when the singer enters the stage and starts performing. At first, the people inside the venue appear tired, disengaged, and uninterested in the performance.
As the music progresses, their behavior gradually changes.
The customers, arcade players, and skaters become increasingly energetic. Their movements become faster, their expressions become more animated, and the entire atmosphere of the venue transforms from dull and lifeless into vibrant and celebratory.
This gradual emotional transformation was one of the core storytelling elements of the video.
It also became one of the most technically demanding parts of the production.
The Production Material We Received
The only live-action footage available for the project was the singer performing against a blue-green screen.
The chroma key footage was not recorded under ideal conditions. The background was uneven, and the separation between the performer and the screen was not clean enough for a standard keying process.
Because of this, automatic chroma keying alone could not produce a reliable result.
The singer had to be extracted from the original footage using extensive rotoscoping and some frames manual cleanup.
This required careful attention to:
- Body movement
- Motion blur
- Clothing details
- Shadows
- Fast gestures
- Frame-by-frame consistency
Building the Environment in Unreal Engine
The main location did not exist as a filmed set.
The skating hall, performance stage, arcade machines, architectural elements, lighting, and spatial layout were created digitally in Unreal Engine.
Using Unreal Engine gave us precise control over several important production elements:
- Camera position
- Lens angle
- Stage dimensions
- Lighting direction
- Perspective
- Arcade placement
- Floor layout
- Background depth
- Spatial continuity between shots
This 3D environment became the visual foundation of the project.
This consistency was extremely important because the singer needed to appear naturally positioned inside the same venue from different camera angles.
Creating the Crowd with AI
The background characters were created using artificial intelligence.
These included:
- Arcade players
- Audience members
- Customers inside the venue
- Roller skaters
- Groups interacting with each other
- Characters reacting to the singer
At first glance, generating background characters may seem simpler than creating the main performer. In reality, it was one of the largest challenges in the project.
The production required a large number of secondary characters, but they could not appear random or disconnected.
We needed to control:
- The number of people in each shot
- Their location in the environment
- Their clothing and appearance
- Their direction of attention
- Their emotional state
- Their movement
- Their consistency between related shots
- Their gradual transformation throughout the video
The characters also needed to feel like they belonged in the same location and the same visual world.
Without careful control, AI models tend to change faces, clothing, body proportions, and crowd density between generations.
This meant that every shot required repeated testing, prompt refinement, image selection, and manual correction.
Designing the Character Transformation
The crowd did not simply need to appear in the scene.
They also had to change behavior as the song developed.
At the beginning of the music video, the people inside the venue were intentionally designed to appear:
- Tired
- Passive
- Emotionally distant
- Uninterested
- Low-energy
Later, they had to become:
- Active
- Engaged
- Expressive
- Energetic
- Excited by the performance
This was not a single visual change.
It was a progression that had to remain consistent with the pacing of the music.
The transformation needed to happen gradually across multiple shots, without making the audience feel that completely different characters had suddenly appeared.
Maintaining that progression required a combination of visual planning, prompt engineering, editing decisions, and controlled AI generation.
Combining AI Characters with the 3D Environment
Once the Unreal Engine environment and AI-generated characters were created, they needed to be visually integrated.
The crowd images were combined with the digital location through an AI-assisted image-generation process.
This required careful alignment of:
- Perspective
- Camera height
- Character scale
- Lighting direction
- Floor contact
- Background depth
- Scene composition
- Viewing angle
A character generated from the wrong camera perspective would immediately feel disconnected from the environment.
The same was true for lighting.
If the characters were illuminated differently from the Unreal Engine scene, the final composite would look artificial, even if each individual element looked good on its own.
For this reason, the AI process was not treated as a simple image-generation step. It was treated as a controlled extension of the 3D production pipeline.
Animating the AI Images in ComfyUI
After the still images were approved, they were animated in ComfyUI.
We created a custom workflow specifically for this project, using models and control methods that gave us greater influence over:
- Character identity
- Camera movement
- Shot duration
- Frame composition
- Image proportions
- Subject placement
- Motion intensity
- Direction of movement
Each image had to be converted into a video with a specific camera angle and a controlled type of movement.
This was especially important because the generated footage needed to match both the Unreal Engine environment and the live-action performance of the singer.
The objective was not simply to make the images move.
The movement had to support the story, preserve the characters, maintain the composition, and fit the rhythm of the song.
The Main Limitation: Controlling AI
The most significant technical challenge was the limited predictability of AI-generated video.
AI models can create impressive results, but they do not always follow production instructions precisely.
Common problems included:
- Characters changing appearance
- Background people disappearing
- New characters appearing unexpectedly
- Clothing changing between frames
- Faces becoming distorted
- Camera movement becoming too aggressive
- Scene composition drifting
- Objects changing shape
- Lighting shifting during the shot
- Character actions not matching the requested direction
These problems become more serious when working on a narrative sequence rather than a single independent shot.
In a music video, every shot must connect to the next.
The visual energy, character behavior, camera direction, and emotional progression must remain understandable across the entire edit.
To overcome these limitations, we used:
- Detailed prompts
- Controlled camera descriptions
- Reference images
- Multiple generation tests
- Carefully designed ComfyUI workflows
- Character control models
- Shot-by-shot visual planning
- Selective regeneration
- Traditional compositing and cleanup
The goal was to reduce unpredictability and move the AI output closer to a directed production rather than a random generation process.
Matching the Singer with the AI-Generated Footage
After the backgrounds and crowd shots were created, the rotoscoped singer had to be composited into the final scenes.
This was another major challenge.
The singer’s movements were already fixed in the original footage. The digital environments and AI-generated videos therefore had to be designed around the performance.
We needed to match:
- Camera angle
- Performer scale
- Body direction
- Stage position
- Floor perspective
- Lighting
- Motion
- Timing
- Visual energy
The singer could not simply be placed over the background.
The environment had to feel as though it had been filmed together with the performance.
The pacing of the AI-generated shots also had to match the singer’s gestures and the structure of the music.
For example, when the performer moved with greater energy, the background characters and camera movement also needed to reflect that change.
This relationship between the singer, the crowd, and the progression of the song required careful editing and repeated compositing adjustments.
Why Traditional VFX Was Still Essential
Although AI was used extensively, the project could not have been completed professionally using AI alone.
Traditional VFX remained necessary for:
- Rotoscoping the singer
- Cleaning the original footage
- Edge refinement
- Compositing
- Color correction
- Perspective matching
- Masking
- Stabilization
- Layer integration
- Shot continuity
- Final editing
- Timing adjustments
- Visual cleanup
AI generated important parts of the visual world, but traditional post-production made those elements usable within a controlled production.
This distinction is important.
AI can generate content, but professional VFX is still required to make that content consistent, intentional, and production-ready.
The Hybrid Workflow
The final production followed a hybrid pipeline:
- Creative concept development
- Story and character progression planning
- Live-action footage review
- Rotoscoping the singer
- Building the venue in Unreal Engine
- Designing the stage and arcade environment
- Creating secondary characters with AI
- Generating crowd variations
- Combining characters with the 3D environment
- Animating still images in ComfyUI
- Controlling camera movement and character motion
- Selecting and refining usable AI shots
- Compositing the singer into the generated scenes
- Matching lighting, scale, and perspective
- Editing the crowd transformation to the music
- Final color grading and cleanup
This workflow allowed us to use the strengths of each production method.
Unreal Engine provided environmental consistency.
AI provided visual flexibility and a large number of characters.
Traditional VFX provided control, precision, integration, and final quality.
AI vs Traditional VFX
| Production Factor | AI-Based Workflow | Traditional VFX |
|---|---|---|
| Concept exploration | Very fast | Slower |
| Crowd generation | Efficient for large variations | Expensive and time-consuming |
| Character consistency | Difficult to maintain | Highly controllable |
| Camera control | Limited and unpredictable | Precise |
| Environment consistency | Requires strong references | Reliable in 3D |
| Late-stage revisions | Fast but unpredictable | Slower but precise |
| Complex compositing | Limited | Essential |
| Performer integration | Requires VFX support | Highly controllable |
| Production cost | Potentially lower | Usually higher |
| Final reliability | Depends on cleanup | More predictable |
When AI Is a Good Choice for a Music Video
AI can be highly effective when the project includes:
- Surreal environments
- Large crowds
- Stylized background characters
- Rapid concept development
- Limited production budgets
- Short production schedules
- Visual experimentation
- Scenes that would be expensive to film
- Environments that do not exist physically
It is especially useful when creative flexibility is more important than perfect physical continuity.
When Traditional VFX Is the Better Choice
Traditional VFX remains the better option when:
- The main performer must remain perfectly consistent
- Product accuracy is critical
- Camera movement must be exact
- Lip-sync must be preserved
- The client requires precise revisions
- Character continuity is essential
- Complex physical interaction is required
- Multiple shots must match perfectly
- The production has strict visual standards
For many professional projects, a fully AI-generated workflow still introduces too much uncertainty.
So, Which Approach Is Right?
The answer depends on the needs of the music video.
AI is not automatically better because it is faster.
Traditional VFX is not automatically better because it offers more control.
The correct approach depends on:
- The creative concept
- The budget
- The schedule
- The required level of realism
- The importance of character consistency
- The number of revisions
- The complexity of the performance
- The amount of camera control required
In this project, neither AI nor traditional VFX would have been sufficient on its own.
The final result depended on combining several methods into a single controlled workflow.
Final Thoughts
AI is transforming music video production, but it has not eliminated the need for experienced directors, VFX artists, editors, and technical artists.
In fact, AI-based production often requires more planning than clients expect.
The technology can generate impressive images and movement, but professional results still depend on:
- Strong art direction
- Technical knowledge
- Shot planning
- Prompt engineering
- Visual consistency
- Compositing
- Editing
- Traditional VFX experience
The most effective workflow is often not AI versus traditional VFX.
It is AI combined with traditional VFX.
That combination makes it possible to create ambitious visual worlds while maintaining the control required for a professional music video production.
Planning a Music Video with AI or VFX?
Epic Cells develops hybrid production workflows that combine Unreal Engine, AI-generated visuals, custom ComfyUI pipelines, and traditional VFX.
Each project is designed around its creative goals, production limitations, and required level of visual control.
Contact Epiccells to discuss the right production approach for your music video.