Two students from Carnegie Mellon University recently placed second in an artificial intelligence competition for their submission of a program that was able to learn the game Doom the same way humans do – by playing.
Whereas most such programs have traditionally learned by following a set of instructions and subsets therein if this, do that), the program learns based on what’s happening on the screen. In one round, the AI even learned to duck, a favored tactic of human players.
Guillaume Lample and Devendra Singh Chaplot spent four months creating a program that utilized many different facets of AI technology. The first problem they solved was that of “rewarding” the AI. The AI must have a framework to learn what is good and bad behavior. In their paper on the subject, the students note that if the reward is based on score alone, the learning is slow because that reward does not appear until the end of the round.
Instead, they introduced a concept learned about from a 2003 paper by Baidu Research Chief Scientist Andrew Ng called “reward shaping.” Reward shaping allows their bot to gain intermediate rewards that build up to a complete reward. Their reward table looked like this: positive reward for picking up objects, negative reward for losing health, and negative reward for shooting/loosing ammunition.
We used different rewards for the navigation network. Since it evolves on a map without enemies and its goal is just to gather items, we simply give it a positive reward when it picks up an item, and a negative reward when it’s walking on lava.
We also found it very helpful to give the network a small positive reward proportional to the distance it travelled [sic] since the last step. That way, the agent is faster to explore the map, and avoids turning in circles.
Other important factors go into the development of the bot, including map learning and a concept called “frame skip.”
Having placed second in a competition that sees a lot of heavy hitters entering it, the two are moving onto more advanced games like Quake. The development of advanced artificial intelligence in gaming could prove to be a very important aspect of the future of gaming as a whole. From a developer’s perspective, more advanced AI means that people will have more fun and be more challenged when playing first-person shooter games in single-player mode. From a player’s perspective, the prospect of cheating arises.
As more and more games see the introduction of paid-to-play models and lucrative competitions, game developers and publishers will have to become hyper-aware of how to differentiate between a bot and a human player. Both Doom and Quake are very old games, but the concept of the first person shooter has only evolved so much since their introduction, and the technology behind these bots isn’t so many leaps in logic behind them.
Featured image from Flickr.