Modelling Human-like Behavior through Reward-based Approach in a First-Person Shooter Game

Makarov, Ilya and Zyuzin, Peter and Polyakov, Pavel and Tokmakov, Mikhail and Gerasimova, Olga and Guschenko-Cheverda, Ivan and Uriev, Maxim (2016): Modelling Human-like Behavior through Reward-based Approach in a First-Person Shooter Game. Published in: CEUR Workshop Proceeding , Vol. 1627, No. Experimental Economics and Machine Learning (25 July 2016): pp. 24-33.

Abstract

We present two examples of how human-like behavior can be implemented in a model of computer player to improve its characteristics and decision-making patterns in video game. At first, we describe a reinforcement learning model, which helps to choose the best weapon depending on reward values obtained from shooting combat situations.Secondly, we consider an obstacle avoiding path planning adapted to the tactical visibility measure. We describe an implementation of a smoothing path model, which allows the use of penalties (negative rewards) for walking through \bad" tactical positions. We also study algorithms of path nding such as improved I-ARA* search algorithm for dynamic graph by copying human discrete decision-making model of reconsidering goals similar to Page-Rank algorithm. All the approaches demonstrate how human behavior can be modeled in applications with significant perception of intellectual agent actions.

• Item Type: MPRA Paper
• Language: English
• Keywords: Human-like Behavior, Game Arti cial Intelligence, Reinforcement Learning, Path Planning, Graph-based Search, Video Game
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C57 - Econometrics of Games and Auctions; C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C91 - Laboratory, Individual Behavior

References

Wang, D., Tan, A.H.: Creating autonomous adaptive agents in a real-time first-person shooter computer game. IEEE Transactions on Computational Intelligence and AI in Games 7 (2) (June 2015) 123-138

Hingston, P.: A turing test for computer game bots. IEEE Transactions on Computational Intelligence and AI in Games 1 (3) (Sept 2009) 169-186

Karpov, I.V., Schrum, J., Miikkulainen, R. In: Believable Bot Navigation via Playback of Human Traces. Springer Berlin Heidelberg, Berlin, Heidelberg (2012) 151-170

van Hoorn, N., Togelius, J., Schmidhuber, J.: Hierarchical controller learning in a first-person shooter. In: 2009 IEEE Symposium on Computational Intelligence and Games. (Sept 2009) 294-301

da Silva, F.S.C., Vasconcelos, W.W. In: Rule Schemata for Game Arti cial Intelligence. Springer Berlin Heidelberg, Berlin, Heidelberg (2006) 451-461

Cole, N., Louis, S.J., Miles, C.: Using a genetic algorithm to tune first-person shooter bots. In: Evolutionary Computation, 2004. CEC2004. Congress on. Volume 1. (June 2004) 139{145 Vol.1

Smith, M., Lee-Urban, S., Mu~noz-Avila, H.: RETALIATE: learning winning policies in rst-person shooter games. In: Proceedings of the Twenty-Second AAAI Conference on Arti cial Intelligence, July 22-26, 2007, Vancouver, British Columbia, Canada, AAAI Press (2007) 1801-1806

Stanley, K.O., Bryant, B.D., Miikkulainen, R.: Real-time neuroevolution in the nero video game. IEEE Transactions on Evolutionary Computation 9 (6) (Dec 2005) 653-668

Veldhuis, M.O.: Arti cial intelligence techniques used in rst-person shooter and real-time strategy games. human media interaction seminar 2010/2011: Designing entertainment interaction (2011)

McPartland, M., Gallagher, M.: Reinforcement learning in rst person shooter games. IEEE Transactions on Computational Intelligence and AI in Games 3 (1) (March 2011) 43-56

McPartland, M., Gallagher, M.: Interactively training rst person shooter bots. In: 2012 IEEE Conference on Computational Intelligence and Games (CIG). (Sept 2012) 132-138

McPartland, M., Gallagher, M. In: Game Designers Training First Person Shooter Bots. Springer Berlin Heidelberg, Berlin, Heidelberg (2012) 397{408

Tan, A.H.: Falcon: a fusion architecture for learning, cognition, and navigation. In: Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on. Volume 4. (July 2004) 3297-3302 vol.4

Bhattacharya, P., Gavrilova, Marina L.: Voronoi diagram in optimal path planning. In: 4th IEEE International Symposium on Voronoi Diagrams in Science and Engineering. (2007) 38-47

Choi, J.w., Curry, Renwick E., Elkaim, Gabriel H.: Obstacle avoiding real-time trajectory generation and control of omnidirectional vehicles. In: American Control Conference. (2009)

Gulati, S., Kuipers, B.: High performance control for graceful motion of an intelligent wheelchair. In: IEEE International Conference on Robotics and Automation. (2008) 3932-3938

Guechi, E.H., Lauber, J., Dambrine, M.: On-line moving-obstacle avoidance using piecewise bezier curves with unknown obstacle trajectory. In: 16th Mediterranean Conference on Control and Automation. (2008) 505-510 Modelling Human-like Behavior in FPS Game 33

Nagatani, K., Iwai, Y., Tanaka, Y.: Sensor based navigation for car-like mobile robots using generalized voronoi graph. In: IEEE International Conference on Intelligent Robots and Systems. (2001) 1017-1022

Mohammadi, S., Hazar, N.: A voronoi-based reactive approach for mobile robot navigation. Advances in Computer Science and Engineering 6 (2009) 901-904

Eren, H., Fung, C.C., Evans, J.: Implementation of the spline method for mobile robot path control. In: 16th IEEE Instrumentation and Measurement Technology Conference. Volume 2. (1999) 739-744

Magid, E., Keren, D., Rivlin, E., Yavneh, I.: Spline-based robot navigation. In: International Conference on Intelilgent Robots and Systems. (2006) 2296-2301

Hwang, J.H., Arkin, R.C., Kwon, D.S.: Mobile robots at your ngertip: Bezier curve on-line trajectory generation for supervisory control. In: IEEE International Conference on Intelligent Robots and Systems. Volume 2. (2003) 1444-1449

Skrjanc, I., Klan�car, G.: Cooperative collision avoidance between multiple robots based on bezier curves. In: 29th International Conference on Information Technology Interfaces. (2007) 451-456

Ho, Y.J., Liu, J.S.: Smoothing voronoi-based obstacle-avoiding path by length-minimizing composite bezier curve. In: International Conference on Service and Interactive Robotics. (2009)

Ho, Y.J., Liu, J. S.: Collision-free curvature-bounded smooth path planning using composite bezier curve based on voronoi diagram. In: IEEE International Symposium on Computational Intelligence in Robotics and Automation. (2009) 463-468

Koenig, S., Sun, X., Uras, T., Yeoh, W.: Incremental ARA : An incremental anytime search algorithm for moving-target search. In: Proceedings of the Twenty-Second International Conference on Automated Planning and Scheduling. (2012)

Makarov, I., Tokmakov, M., Tokmakova, L.: Imitation of human behavior in 3D-shooter game. In Khachay, M.Y., Konstantinova, N., Panchenko, A., Delhibabu, R., Spirin, N., Labunets, V.G., eds.: 4th International Conference on Analysis of Images, Social Networks and Texts. Volume 1452 of CEUR Workshop Proceedings., CEUR-WS.org (2015) 64-77

Pohl, I.: First results on the e ect of error in heuristic search. Machine Learning 5 (1970) 219-236

Likhachev, M., Gordon, G., Thrun, S.: ARA*: Anytime A* search with provable bounds on sub-optimality. In Thrun, S., Saul, L., Scholkopf, B., eds.: Proceedings of Conference on Neural Information Processing Systems (NIPS), MIT Press (2003)

Sun, X., Yeoh, W., Uras, T., Koenig, S.: Incremental ara*: An incremental anytime search algorithm for moving-target search. In: ICAPS. (2012)

Makarov, I., Polyakov, P.: Smoothing voronoi-based path with minimized length and visibility using composite bezier curves. In Khachay, M.Y., Vorontsov, K., Loukachevitch, N., Panchenko, A., Ignatov, D., Nikolenko, S., Savchenko, A., eds.: 5th International Conference on Analysis of Images, Social Networks and Texts. CEUR Workshop Proceedings, CEUR-WS.org, In Print (2016)