Introduction
The literature relevant to this work can be organized into three main parts: ROS 2 frameworks and tools for multi-robot systems, toolboxes in specific domains, and game engine and mixed-reality tools for robotics.
When real deployments are costly, risky, or difficult to access, high-fidelity simulation provides an effective complement to physical experiments. Unity-based environments have been integrated with ROS to enable immersive teleoperation, realistic mobile-robot simulation, and mixed-reality or digital-twin validation frameworks. To the best of our knowledge, however, none of these approaches provides a ROS 2–Unity toolkit explicitly tailored to mixed-reality fleet management of autonomous mobile robots executing distributed coordination algorithms.
Unity-Based Interface for Virtual Sensing and Mixed-Reality Simulations
Unity is a versatile 3D development platform for creating realistic simulations and visualizations, making it suitable for virtualizing environments for multi-robot systems. Its graphics and physics engines enable the development of dynamic, interactive environments where robots can navigate and interact with objects in real time.
Together, these tools enable Unity to serve as an interactive, high-fidelity simulation front end, while ROS manages backend robotic processes such as sensor-data handling, path planning, and control algorithms. This setup allows Unity to simulate complex environments and robotic behaviors under ROS control, enhancing the realism and utility of virtual testing and validation in multi-robot systems.
The Unity-based interface focuses on two main capabilities:
- Dynamic generation of parametric environments.
- Integration of robot models and virtual sensors for ROS 2-based experiments.
Mixed-Reality Experiments
In the experiment, the Jackal robot communicates with the workstation using the ROS 2 publisher-subscriber protocol over a standard 5 GHz Wi-Fi network. The real and virtual experiments are dockerized and run on Ubuntu 24.04.
The workstation is equipped with an AMD Ryzen 7 7800X3D processor and an Nvidia GeForce RTX 5090 GPU. The ROS 2 version used is Jazzy Jalisco, while Unity is version 6000.0.74f1.
During the mixed-reality experiments, the robots’ poses—position and orientation—are provided by the Vicon motion-capture system. A second workstation, equipped with an Intel i9-9900K processor and Windows 11, runs Vicon Tracker 3.10.
Scenario 1: Dynamic Task Assignment in Virtual Indoor Environments with Obstacles
In this scenario, a dynamically evolving set of tasks must be performed cooperatively by robots in a distributed fashion, leveraging their communication capabilities. This setup can be modeled as a dynamic task assignment problem, a well-known combinatorial optimization problem in multi-robot systems.
The distributed optimization problem is handled by the optimization-based management system. Once each robot determines its task assignment, the single-robot control and navigation system manages task planning and execution.
A task is considered accomplished when the robot reaches a specific location in the facility. As in real-world applications, the assignment problem is dynamic: new tasks are introduced as others are completed. Consequently, the robots must continuously re-optimize and adjust their local plans in response to new data.
The experiments show robots servicing tasks in different settings. In the virtual experiment, a robot navigates within the Unity environment, maps its surroundings, and avoids collisions with obstacles and other robots. A video of the experiment is available as supplementary material.
Scenario 2: Real Robot in a Virtual Environment
The Jackal navigates in the real world while sensing the virtual environment. To highlight the mixed-reality interconnection demonstrated in this experiment, the robot is tasked with reaching a series of goal poses within the virtual environment in the presence of virtual obstacles.
Frequently Asked Questions
What is the purpose of the ROS 2–Unity toolkit? It supports mixed-reality fleet management of autonomous mobile robots executing distributed coordination algorithms.
Why is Unity used in the toolkit? Unity provides realistic 3D simulation and visualization through interactive graphics and physics engines.
How does the Jackal communicate with the workstation? The Jackal uses the ROS 2 publisher-subscriber protocol over a standard 5 GHz Wi-Fi network.
What happens in the second mixed-reality scenario? A real Jackal navigates while sensing a virtual environment and reaches goal poses among virtual obstacles.