Using Virtual Reality and Machine Learning to Create 3D Content

Creating 3D content such as video games, simulated scenarios and many other forms of content requires proficiency with many different types of software. However, even if you are able to master a particular tool, it is often slow and does not allow you to fully bring your imagination to life. This is one of the biggest problems discussed by Jonathan Gagne, CEO of Masterpiece Studio, at the recent GTC 2021 conference. In fact, he cites data that 99% of the 152 million professionals worldwide are struggling to create 3D content because the software is too slow and complex. Let’s first take a look at why this problem exists and then explore how VR and ML can be used to solve this problem. 

Why is it so Difficult to Use the Traditional Software? 

The big challenge is that there is a mismatch between interfaces. Basically, you are trying to create something in 3D, but you are using a 2D interface i.e. the flat screen monitor and your mouse. This means that you will need to buy a lot of complex tools to help you bridge that gap between your 2D tools and the 3D world. This also slows down the development process since it often takes years to fully learn and master the use of such software. This is where virtual reality can be very useful in helping content creators replace such expensive and complex software. 

You can put on your headset and you can see everything in 3D and immerse yourself in the content that you are creating. Also, with hand-tracking controllers, you can also interact with that environment, which leads to a fast and natural user experience. You can create content a lot faster using VR, 10X faster than traditional software, and it is also easier to use because we know how to use our hands and how to interact with the 3D world that we live in. All of these things make VR a significant upgrade over the traditional tools being used to create 3D content. 

Even though VR is a great tool as we have established above, can we go a step further and improve it even more? The answer is definitely yes, but we will need to bring machine learning to help us. 

How Does Machine Learning Assist Content Creators? 

Machine learning has a powerful capability to add intelligence to the content we are trying to create. This includes things like automating slow and difficult processes, removing technical and creative barriers, and many other benefits. For example, if we take a look at a tool like Unity ArtEngine which is used for adding texture to an image it automates the process of photoconversion to physically-based rendered (PBR) materials. This means that you can take a photo of any object and it will be automatically digitized so you can use it in your own work. Even if the image is low-resolution, the machine learning can fix this as well by inferring what details should be there. 

The AI also automates a very time-consuming and labor-intensive process known as rigging and skinny. Rigging refers to creating the skeleton for the animated person or creature while the skinning aspect refers to actually drawing all of the skin and exterior around the skeleton. This would then be placed into a gaming engine like Unity or Unreal for animation. This is not something that a lot of content craters look forward to doing, but it is an essential part of the animation process. Deep learning can automate this process and input the skeleton into the mesh just the right way.

NVIDIA itself has an AI-based 3D content creation tool called Audio2Face. This is cutting-edge software that generates expressive facial animation from just an audio source. This significantly simplifies the creation of any animated 3D character to match the voice-over track. Another great tool is NVIDIA DeepStream which allows artists and developers to use computer vision to create human pose estimation. Such technology is used for creating animated characters in movies and also in video games. 

In addition to the AI software, human data annotators need to annotate video sequences and frames and draw various lines that annotate human poses. This includes things like multi-purpose pose estimation and multi-purpose pose-tracking. Mindy Support has recently worked on such a project. We needed to annotate the human skeleton with key points on each frame in a video which is very challenging because the person is constantly moving in the video and therefore the combinations of points are constantly changing. Moreover, the people in the video can be in various positions i.e. vertical (running, jumping, squatting), horizontal (push-ups, laying down, plank position). Moreover, the algorithm evaluates key points differently when a person is in different positions.

Check some more data annotation projects we worked on.

We assembled a team of 115 full-time data annotators to perform all of the needed labeling tasks. In total we annotated 12,000 videos which is 1,200,000 frames. All of this work was done within the specified time frame of 60 days and the quality score was >95%.

The Practical Applications of VR and ML in 3D Content Creation

All kinds of content can be created with this technology to assist artists and developers create life-like images faster and easier. For example, an artist working on 3D storytelling can create illustrations that they can render from any direction i.e. if they want to change the angle, they do not have to redraw everything from scratch. All they have to do is change the camera view and render a new scene. 

VR and ML can be very useful to gaming studios, especially ones that are just getting started and don’t have a full 3D modeling team or have very talented content creators and they would like to accelerate that 3D modeling process. On the same note, it can also be useful for creating animations even if a company or individual artist does not have the expertise on staff to create them. Also, there are some applications in the education area as well since graphic designers and other artists usually need to spend a lot of time mastering the content creation tools. The AI and ML would allow them to come into the classroom and start creating their concepts right away.

The VR and ML take the technological burden off your shoulders so you are limited only by your imagination. The only hardware you will need is a computer with the necessary amount of computing power, so at least 1060 GPU. You will also need a VR headset like an HTC Vive, Oculus Rift, or pretty much any other one. 

Mindy Support Offers Comprehensive Data Annotation 3D Content Creation

If you are working on 3D content for video games, animations, or any other medium that requires data annotation, Mindy Support can provide you with comprehensive data annotation services. We are one of the largest BPO companies in Eastern Europe with more than 2,000 employees in six locations all over Ukraine. Our size and location allow us to source and recruit the necessary number of candidates quickly and we will be able to scale your team without sacrificing the quality of the data annotation. Contact us today to learn more about how we can help you.