Google Demonstrates Natural Language Replacement of Characters and Objects in Videos
Google has published a demonstration of Gemini Omni, which can replace objects or characters in videos using natural language while maintaining scene coherence. Examples include replacing a spaceship with a specified object, aligning characters with movements and dialogues.
This capability comes from Gemini Omni, positioned to generate content from any input, starting with video. Users can modify step-by-step through dialogue, with character consistency, physics, and previous shots being written as memory. Omni 1.1 Flash is an accelerated version that supports continuous shots, reference image angle changes, style transfer, and allows stateful editing through a developer interface.
Runway announced on August 28 that it has integrated Google Omni 1.1 Flash, launching alongside its image and video models. When Google launched Omni in May, it emphasized integrating Gemini's world knowledge into video generation, allowing combinations of text, images, audio, and video as input.
Official prompts also include changing lighting, modifying sign text, inserting events at specific timestamps, and using reference images to integrate clothing or architecture into original shots. Outputs can reach 1080p and support higher resolution scaling. Google compares it to conversational editing for video rather than one-time production.
In market mechanics, the buyers are producers who want to change characters and props in a completed video without reshooting, while the selling is based on the number of generation instances charged per prompt. Events are driven by platform listings and official demonstrations, with funds flowing from video tool subscriptions to model calls; benefiting are editing platforms that integrate foundational models into workflows, while traditional angle-changing processes relying on keyframes and masks are under pressure.
Source: Public Information
ABAB AI Insight
Changing angles has shifted from masks and timelines to a single sentence. A spaceship turns into a fruit basket, while the person continues on the original trajectory. Google sells scene memory: changing one thing without disrupting lighting and ground. Runway's launch of Flash embeds experimental models into timelines with existing paying users. Whoever controls the state of "continuing to change within a single video" charges by conversation rather than by export count.
The capital path is to write Gemini's multimodal reasoning into a video API, then reach editing rooms through distribution layers like Runway. Money flows from subscriptions and tokens into training and inference clusters. Reference image slots turn product images and actor images into replaceable layers, initially utilized by e-commerce and commercials. The dynamic version corresponds to static "banana" photo editing: the same conversational habits transition from posters to shots.
It benchmarks against Runway's own Gen series, OpenAI's video generation, and Adobe's integration of Firefly into Premiere. Generated videos are at a stage where "being able to produce a video must also mean being able to modify it": a stunning first frame no longer constitutes a workflow; being able to change the protagonist without it being obvious forms a workflow. World knowledge is used to supplement physics, addressing the audience's eyes, not the on-set script supervisor.
Structurally, it belongs to technological substitution. The mechanism is: when the model and the original footage share a timeline, replacement no longer equals reshooting. Pricing power shifts from the photographer's schedule to the iteration count of prompts. Coherence is the product; changing angles is merely a button.
ABAB News · Cognitive Laws
After being able to produce a video, being able to modify it constitutes a workflow
If changing the protagonist is not obvious, masks will turn into a single sentence
Scenes remember previous shots, allowing generation to enter the timeline