Vid2Game, an artificial intelligence (AI) has been created by the Research team of Facebook which will allow extracting playable characters from real people’s videos. It is considered to be a higher-tech version of the full-motion video (FMV) games of the 80s.
Videos in which people do some specific action will be analyzed by the neural networks, which will then recreate that character and its action in a new environment and let the user control its actions using a joystick.
Two neural networks, Pose2Pose and Pose2Frame, were used by the team. In the first stage, video is fed to Pose2Pose neural network, which is made for certain actions as dancing, tennis or fencing. The system identifies and distinguishes the person from its background and isolate him and his pose from it. In the second stage, Pose2Frame inserts the person along with its movements and whatever object it is holding, into a new scene having little artifacts. Then, joystick or keyboard is used to control the movement of the character.
In the training of the system, a few short videos were used to analyze the activities like Fencing, dancing, and Tennis, after which it was capable of filtering out unwanted people and incorporate different camera angles.
This research is much like the “content-aware fill” feature of Adobe Premier Pro that removes different elements from video using AI. Also, NVIDIA and a few other companies have also worked on AI that is capable of transforming real videos into virtual landscapes, making it suitable for games.
As the research is still at the beginning level, a problem has been identified in 3D animation, called “Foot slide” through which it appears like a character is playing on ice. Also, there is a limited range of motion. Despite all this, the new character extraction efforts are better and more realistic than the previous ones. It is expected that with time team will be able to resolve the issues related to motion.
The gaming experience will be personalized with the introduction of Facebook’s Vid2Games synthesis, allowing you to have characters of your choice, be it a YouTube personality.
The team working on the research said that new computational problems have been addressed and also a new era of video games with realistic graphics is about to begin. Along with this, characters extracted from YouTube videos will soon be part of the virtual world and augmented realities.
Read next: Facebook to Give Tough Competition to Amazon, Apple and Google with its New Voice Assistant
Videos in which people do some specific action will be analyzed by the neural networks, which will then recreate that character and its action in a new environment and let the user control its actions using a joystick.
Two neural networks, Pose2Pose and Pose2Frame, were used by the team. In the first stage, video is fed to Pose2Pose neural network, which is made for certain actions as dancing, tennis or fencing. The system identifies and distinguishes the person from its background and isolate him and his pose from it. In the second stage, Pose2Frame inserts the person along with its movements and whatever object it is holding, into a new scene having little artifacts. Then, joystick or keyboard is used to control the movement of the character.
In the training of the system, a few short videos were used to analyze the activities like Fencing, dancing, and Tennis, after which it was capable of filtering out unwanted people and incorporate different camera angles.
This research is much like the “content-aware fill” feature of Adobe Premier Pro that removes different elements from video using AI. Also, NVIDIA and a few other companies have also worked on AI that is capable of transforming real videos into virtual landscapes, making it suitable for games.
As the research is still at the beginning level, a problem has been identified in 3D animation, called “Foot slide” through which it appears like a character is playing on ice. Also, there is a limited range of motion. Despite all this, the new character extraction efforts are better and more realistic than the previous ones. It is expected that with time team will be able to resolve the issues related to motion.
The gaming experience will be personalized with the introduction of Facebook’s Vid2Games synthesis, allowing you to have characters of your choice, be it a YouTube personality.
The team working on the research said that new computational problems have been addressed and also a new era of video games with realistic graphics is about to begin. Along with this, characters extracted from YouTube videos will soon be part of the virtual world and augmented realities.
Read next: Facebook to Give Tough Competition to Amazon, Apple and Google with its New Voice Assistant
