Patreon
Home / Videos / Type One Line, Change the Whole Scene: All 12 OpenAI Sora Vi

Type One Line, Change the Whole Scene: All 12 OpenAI Sora Video-to-Video Demos (2024)

I have combined all 12 Video To Video #SORA demos released by #OpenAI into 1 video with their used prompts and a amazing background music. You won't believe how this Video to Video will change entire movie, animation, social media industries forever. The results are just simply astonishing.

Published Feb 17, 2024 · 9,530 views · Watch on YouTube

About this video

I have combined all 12 Video To Video #SORA demos released by #OpenAI into 1 video with their used prompts and a amazing background music. You won't believe how this Video to Video will change entire movie, animation, social media industries forever. The results are just simply astonishing.

Our Discord Channel ⤵️

https://discord.com/servers/software-engineering-courses-secourses-772774097734074388

Our Patreon With Amazing AI Scripts & Tutorials ⤵️

https://www.patreon.com/SECourses

Prompts Of Each Demo Video (Public Post) ⤵️

https://www.patreon.com/posts/98564118

Official Site ⤵️

https://openai.com/sora

[AI video generation] Sora element technology explanation

Sora's technical configuration

Although the paper has not been published, OpenAI has published an explanation page for the elemental technology, so I will refer to that page.

If you would like to see the original text, please click here

overall structure

Sora is said to consist of the following technical elements.

Turning visual data into patches

Video compression network

Spacetime latent patches

Scaling transformers for video generation

Variable durations, resolutions, aspect ratios

Sampling flexibility

Improved framing and composition

Language understanding

To summarize very simply, there are four main elements:

A technology that compresses video data into latent space and then converts it into a "spatiotemporal latent patch" that Transformer can use as a token.

Transformer-based video diffusion model

Dataset creation using high-precision video captioning using DALLE3

Looking at it this way, it doesn't seem like they're using particularly new technology.

Raise your level and hit it physically. You can clearly understand the importance of level (money/calculation resources) rather than small techniques.

Turning visual data into patches

First, let's look at how to create a "space-time potential patch."

(Source: https://openai.com/research/video-generation-models-as-world-simulators)

As a pre-process to create a spatiotemporal latent patch, the input video (video data) is compressed into a latent space.

If you think of it as equivalent to VAE in image generation, I think it's mostly correct.

(In fact, since the paper on VAE is cited, I think it's safe to assume that it's just VAE.)

This greatly reduces the amount of calculation, and Sora trains with this compressed latent space. Masu.

In image generation, training begins immediately after conversion to VAE, but Sora includes another conversion process to create what is called a spatiotemporal latent patch.

This seems to correspond to a text token in LLM.

An image is worth 16x16 words: Transformers for image recognition at scale.

The patching method divides the image based on position (patching) and converts it into a one-dimensional vector (flatten/smoothing).