Project Participants

Victor Lesser (Principal Investigator)

Chongjie Zhang (Post-Doctoral Research Associate)

Bruno Silva (Graduate Research Assistant)

Overview of Project

The project is focused on developing coordination policies for large-scale multi-agent systems operating in uncertain environments through the use of multi-agent reinforcement learning (MARL). Existing MARL techniques do not scale well. This research addresses the scaling issue by using coordination technology to "coordinate" the individual agent learning so as to speed up convergence and lead to learned policies that better reflect overall system objectives. This novel idea is being implemented using an emergent supervisory organization with low overhead that exploits non-local information to dynamically coordinate and shape the learning processes of individual agents while still allowing agents to react autonomously to local feedback. A key question is how to automate the development of the supervisory control process (including supervisory information generation and organization formation). One approach to automation is using a formal model of interactions among agents that also includes a model of global system objectives and policy space of agents to derive the information necessary for appropriate supervisory control. Another approach is the formulation of the supervision problem as a distributed constraint optimization problem.

The results of this work provide a necessary component for the development of a wide variety of next-generation adaptive applications, such as smart power grids, cloud computing, and large-scale sensor networks. The broader impact stems from the wide applicability of the resulting learning technology for distributed control, undergraduate and graduate educational activities at UMass, dissemination efforts that make the experimental domain and algorithms publically available, and the development of international collaborations.

Top | Publications

Publications Supported by this Project

Shanjun Cheng, Anita Raja, and Victor Lesser (2013). “Using Conflict Resolution to Inform Decentralized Learning” Proceedings of 12th International Conference on Autonomous Agents and Multiagent Systems. Ito; Jonker; Gini; Shehory (eds.), IFAAMAS, pp. 893-900.

Chongjie Zhang and Victor Lesser (2013). “Coordinating Multi-Agent Reinforcement Learning with Limited Communication” Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems. Ito; Jonker; Gini; Shehory (eds.), IFAAMAS, pp. 1101-1108.

Chongjie Zhang and Victor Lesser (2012). “Coordinated Multi-Agent Learning for Decentralized POMDPs” Proceedings of 7th Workshop on Multiagent Sequential Decision Making Under Uncertainty; held in conjunction with 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-12), pp. 72-78.

The materials above are based upon work supported by the National Science Foundation under Grant No. 1116078. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Other Related Work by the Multi-Agent Systems Lab

Chongjie Zhang and Victor Lesser (2011). “Coordinated Multi-Agent Reinforcement Learning in Networked Distributed POMDPs” Proceedings of the 25th AAAI Conference on Artificial Intelligence, pp. 764–770.

Chongjie Zhang; Victor Lesser; Sherief Abdallah (2010). “Self-Organization for Coordinating Decentralized Reinforcement Learning” Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, van der Hoek, Kaminka, Lespérance, Luck and Sen, ed., pp. 739–746.

Chongjie Zhang and Victor Lesser (2010). “Multi-Agent Learning with Policy Prediction” Proceedings of the 24th AAAI Conference on Artificial Intelligence, pp. 927–934.

Chongjie Zhang; Victor Lesser; Sherief Abdallah (2009). “Integrating Organizational Control into Multi-Agent Learning” Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems, Vol. 2. Decker, Sichman, Sierra and Castelfranchi (eds.), pp. 757–764.