Nvidia plans a more robust Omniverse with avatars, synthetic data
As businesses prepare to bring more of their businesses and operations to the virtual world, Nvidia is developing Omniverse, its platform to extend workflows into the virtual sphere. The latest updates to the platform, introduced during GTC 2021, include Omniverse Avatar, an embodied AI creation tool, as well as Omniverse Replicator, a synthetic data generation engine.
Nvidia rolled out Omniverse in open beta last December – nearly a year before Facebook embarked on the concept of “metaverse” by renaming itself Meta. Omniverse offers 3D designers a shared virtual world from which they can collaborate on different software applications and from different geographic locations.
Since its launch in December, Omniverse has been downloaded by over 70,000 individual creators. It is also used by professionals in more than 700 companies, including BMW Group, CannonDesign, Epigraph, Ericsson, architectural firms HKS and KPF, Lockheed Martin and Sony Pictures Animation.
“Virtual worlds are essential for the next era of innovation,” Richard Kerris, Omnverse vice president for Nvidia, told reporters last week.
Nvidia’s version of the virtual world has largely focused on building “digital twins” – an accurate digital replica of physical entities. Here’s how Rev Lebaredian, vice president of simulation technology and Omniverse engineering at Nvidia, explained the concept of a digital twin:
“For me, digital twins are basically a way to take things in the real world and represent them in the virtual world so that we can give ourselves superpowers. Once you have an accurate representation … and that you can simulate the behavior of this world, you can do some pretty amazing things. You can teleport – if you have a representation of your factory, or a model of the Earth, or a city with cars driving around, you can jump at any point in this world and feel and perceive it as if you were there.With simulation we also have the potential to time travel – we can record what happened in the past and go back to replay what happened in your factory, in a situation on the roads. You also have the potential to move forward quickly. Not only can you go into the future, but you can also go into the future. alternative futures by changing the parameters within this world. It allows us to plan for much brighter futures, to optimize our businesses, to create a better Earth and a future for ourselves. “
Omniverse Replicator is a tool that should ultimately help organizations create better digital twins – and therefore, better AI-powered tools in the real world. Nvidia presents two different apps built with Replicator, which illustrate some of its use cases: The first app is Nvidia Drive Sim, a virtual world to host the vehicles digital twin. Then, Nvidia Isaac Sim is a virtual world for the digital twin of manipulation robots.
Data is a necessary prerequisite for building AI models, but “you never have enough data, and never of enough quality and diversity to make your system as smart as you want it to be.” , explained Lebaredian. “But if you can synthesize your data, you effectively have an unlimited amount … of a quality that cannot be extracted from the real world.”
Autonomous vehicles and robots built using this Replicator-generated data can master skills in a range of virtual environments before applying them in the physical world.
While its first two use cases are in robotics and automotive, “this general problem of data creation for AI is a problem that everyone is having,” Lebaredian said. Omniverse Replicator will be available next year for developers to build domain-specific data generation engines.
To further demonstrate the value of digital twins, Nvidia featured two customer testimonials. First, Ericsson uses Omniverse to create digital twins for 5G networks. The telecommunications equipment maker is building city-scale digital twins to help accurately simulate the interaction between 5G cells and the environment. This should help optimize performance and 5G coverage.
Next, Nvidia works with Lockheed Martin, as well as the US Department of Agriculture Forest Service and the Colorado Division of Fire Prevention & Control, to run simulated forest fires with Omniverse. The team will use variables such as wind direction, topography, and any other available information to create a digital twin of a forest fire and predict its course.
While there is clear business value in creating digital twins, Nvidia is taking Omniverse beyond real-world replications with the new Omniverse Avatar platform. Avatar is a complete end-to-end platform for creating embodied AI that humans interact with. It connects Nvidia’s technologies in voice AI, computer vision, natural language understanding, recommendation engines and simulation technologies. The avatars created on the platform are interactive characters with ray-traced 3D graphics. They can see and speak on a wide range of subjects and understand spoken intention naturally.
While there is a lot of justified skepticism about the use of avatars – given that efforts like Second Life have failed to make their voice heard – Nvidia’s Lebaredian argued that “today, there are many examples of people who use avatars on a daily basis ”. He pointed out video games like Fortnite and Roblox.
“You can go to Twitch and see players streaming games where they have a virtual avatar in a game engine that represents themselves,” he said. “It’s very natural for this generation who grew up with video games and virtual worlds like air.”
So far, Nvidia has launched a few initiatives targeting Avatar for specific use cases: Project Tokkio uses Avatar to create customer support agents, Nvidia Drive Concierge focuses on smart services in vehicles, and Project Maxine will help customers. to create avatars – which may or may not look like themselves – for video conferencing. The company said it has seen significant interest in the Tokkio project from the retail sector.
The many Nvidia technologies behind Avatar include Riva, a massive new software development kit for handling advanced voice AI. It recognizes speech in multiple languages and can generate human-like voice responses using text-to-speech capabilities. The natural language understanding of the Avatar platform is based on the Megatron 530B large language model which can recognize, understand and generate human language.
Nvidia also integrated Metropolis for computer vision and perception capabilities – so avatars can see and understand the humans they interact with – and the Nvidia Merlin Framework for recommendations. Avatar animation is powered by Nvidia Video2Face and Audio2Face, 2D and 3D facial animation and rendering technologies. Avatar applications are processed in real time using the Nvidia Unified Compute Framework. Customers can use Fleet Command to deploy avatars in the field.
Beyond Replicator and Avatar, Nvidia announced a slew of other Omniverse updates, including new AR, VR, and multi-GPU rendering features. There are also new integrations for infrastructure applications and industrial digital twins with software from Bentley Systems and Esri.