Graph-based Formation Control with Collision Avoidance via Policy-Guided Reinforcement Learning

Abstract

This letter addresses the challenge of formation control with collision avoidance (FCCA) in multi-agent systems under limited communication, using a multi-agent reinforcement learning (MARL) approach. We propose a novel training framework that combines Graph Transformer-based information aggregation, a consensus theory-based guide policy, and hybrid curriculum learning regulators to enhance the scalability and sample efficiency of the system. The key contributions include: (1) the design of a MARL framework tailored for FCCA, where agents are trained using local information only, (2) the incorporation of a guide policy and curriculum learning techniques to promote effective training, and (3) we conducted extensive experiments in obstacle-cluttered environments with varying agent numbers, including simulations and real-world applications on omnidirectional vehicles, to showcase the robustness and generalizability of our method. Besides, our method is compared with the latest works on the FCCA problem and demonstrates significant superiority. The findings in this letter establish a strong foundation for further research on policy-guided MARL tasks.

Index Terms—Formation control, Graph neural networks, Collision avoidance, Consensus theory, Guide policy.

Contributions

1. We propose a MARL training framework that leverages a Graph Transformer for formation control with collision avoidance under restricted communication, while encoding different types of entities accordingly.

2. We integrate guide policy and curriculum learning techniques to enhance sample efficiency and model performance, and incorporate these strategies into multiple training algorithms to assess the robustness.

3. We conduct experiments with varying numbers of agents in cluttered obstacle environments to showcase the scalability of our method. Additionally, real-world experiments are carried out using omnidirectional vehicles with different formation shapes.

Visualization

Here is the bird’s-eye view of the formation control with collision avoidance (FCCA) scenario in our experiments and the environmental settings.

Here is the ablation test of different curriculum learning regulators and training algorithms in 4-agent and 5-agent FCCA task.

Here are the GIFs illustrating the formation control with collision avoidance process of multiple agents trained using our proposed method in environments with cluttered static obstacles.