Google DeepMind has unveiled SIMA (Scalable Instructable Multiworld Agent), a generalist AI agent designed to follow natural language instructions and perform tasks in a variety of 3D video game environments. This breakthrough marks a significant milestone in AI research, as SIMA can understand and act across multiple game worlds, similar to how a human would.
(AI generated- Copilot)
Importance of Video Games in AI
Video games are rich and dynamic learning environments, ideal for testing AI systems. DeepMind has long used video games as a testing ground for its AI research, both with Atari games and StarCraft II.
SIMA’s Goal
For DeepMind, the goal of SIMA is not to achieve high scores, but to learn to follow instructions in different environments/games, thereby unlocking more useful AI agents for any other environment.
Collaboration and Training
To evolve SIMA, DeepMind collaborated with eight game studios, training and testing SIMA in nine different games. Each game offers a new interactive world with a variety of skills to learn, from simple navigation and menu use, to resource mining, spaceship piloting, and crafting.
Training Method
To train SIMA, human players were recorded playing the game, accompanied by another player who observed and gave instructions. They were also able to view these recordings later to add comments on other actions they would carry out.
SIMA Components
SIMA is composed of pre-trained vision models and a main model that includes a memory and generates keyboard and mouse actions. This agent can perceive and understand a variety of environments, and then take actions to achieve an instructed goal.
Impact and Future
The work with SIMA demonstrates how video games can serve to better understand how AI systems can become more useful. The ability to follow instructions in multiple game environments could translate into practical applications in the real world, improving human-AI interaction.
At the level of video game development, this type of AI could help in certain phases of game testing, especially in open-world games, where testing and trying out all the possibilities can be very complicated.
If you want more information about SIMA: https://deepmind.google/discover/blog/sima-generalist-ai-agent-for-3d-virtual-environments/