OpenAI, an artificial intelligence (AI) research and deployment company, has launched Sora, an AI-based text-to-video model. “Sora can generate videos up to a minute long while maintaining visual quality and adherence to the user’s prompt,” an OpenAI official blog post stated.
According to an official release, Sora is present for usage by read teamers to detect critical areas for harms or risks. It’s believed that OpenAI has made Sora available to designers, filmmakers, and visual artists for being able to get feedback on how the model’s progression can benefit creative professionals. “We’re sharing our research progress early to start working with and getting feedback from people outside of OpenAI and to give the public a sense of what AI capabilities are on the horizon,” OpenAI mentioned.
Reportedly, Sora is capable of developing complex scenes with multiple characters, certain motion types, and correct attributes of the subject and background. Seemingly, the model has the ability to understand not only what the user’s requirement is in the prompt, but also how those things thrive in the physical world. “The model has a deep understanding of language, enabling it to accurately interpret prompts and generate compelling characters that express vibrant emotions. Sora can also create multiple shots within a single generated video that accurately persist characters and visual style,” OpenAI added.
“The current model has weaknesses. It may struggle with accurately simulating the physics of a complex scene, and may not understand specific instances of cause and effect. For example, a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark. The model may also confuse spatial details of a prompt, for example, mixing up left and right, and may struggle with precise descriptions of events that take place over time, like following a specific camera trajectory,” Open AI highlighted.
In terms of safety, OpenAI stated how it’ll take important safety measures before making Sora available in OpenAI’s products, such as working with red teamers, building tools for finding deceptive content, which involves the future inclusion of C2PA metadata, leveraging current safety techniques developed for products which utilise DALL·E 3, development of robust image classifiers to review frames of videos for adherence to their usage policies, and engaging global policymakers, educators and artists to find positive use cases for the technology.
Moreover, Sora is understood to be a diffusion model, which is capable of generating a video by starting off with one that looks like static noise and gradually transforms it by removing the noise. Official data suggests that Sora is capable of generating entire videos all at once or extending generating videos to make them longer. Reportedly, Sora makes use of a transformer architecture to unravel superior scaling performance, as seen in GPT models. “Sora builds on past research in DALL·E and GPT models. It uses the recaptioning technique from DALL·E 3, which involves generating highly descriptive captions for the visual training data. As a result, the model is able to follow the user’s text instructions in the generated video more faithfully. In addition to being able to generate a video solely from text instructions, the model is able to take an existing still image and generate a video from it, animating the image’s contents with accuracy and attention to small detail. The model can also take an existing video and extend it or fill in missing frames,” OpenAI concluded.
