Related Work and State of the Art
This section reviews existing approaches in hardware abstraction, the current state of mixed-traffic simulation, and the technical challenges associated with distributed physics and latency management in multi-agent environments to contextualize the framework of the CARLAverse ecosystem.
Synthesis: The CARLAverse Differentiation
The literature indicates that centralized physics architectures relying on intermediary middleware may introduce latency and jitter challenges when supporting high-frequency, globally connected force-feedback simulators. The CARLAverse ecosystem is engineered specifically to address these structural trade-offs. By bypassing middleware translation layers for active control in favor of native orchestration, the CARLAverse distinguishes itself through three foundational pillars.
CARLAverse Architecture
To facilitate a high-fidelity, multi-agent mixed-traffic environment without compromising the latency required for human-in-the-loop immersion, the CARLAverse relies on a specialized, distributed software and hardware architecture.
System Architecture Overview
The CARLAverse ecosystem’s core topology follows CARLA’s strict server-client paradigm, but is extended to support multiple, heterogeneous simulator clients simultaneously. At the center of the architecture is a dedicated central server running the CARLA Unreal Engine environment. Various client machines are connected to this central server via a network. Each local node hosts a specific multimodal simulator setup, such as a DrivoCARLA driving simulator, a CycloCARLA bicycle simulator, or a WalkoCARLA pedestrian setup.
Cross-Cutting Frameworks and Orchestration
Creating a cohesive, shared virtual environment for multiple human actors requires more than just distributed physics. To ensure high modularity and valid experimental results across all nodes, the CARLAverse relies on cross-cutting technologies that manage system orchestration, telemetry visualization, and sensory augmentation uniformly across the framework. In accordance with the system-level focus of this paper, the following subsections outline the software architecture, abstraction layers, and integration interfaces of these cross-cutting subsystems, such as spatial audio, force feedback, and motion cueing, whereas their domain-specific mathematical modeling and validation are addressed in companion works.
The CARLAverse Tools Library
SetupManager: Handles the lifecycle of the virtual environment, including world state management and the safe spawning and destruction of actors across the distributed network.
SharedMemory: Manages interprocess communication between the CARLAverse modules for state variables that must bypass the CARLA server.
Vehicle Physics and Control: Provides standardized wrappers to extract latency-critical telemetry, such as velocity, engine RPM, and wheelbase dimensions, and seamlessly manage actuator commands, such as vehicle lighting and turn signals.
Utilities: A suite of helper functions designed to streamline actor and sensor management. This includes fail-safes for retrieving specific actors, ensuring the unique spawning of sensors to prevent duplicate data streams or API crashes, and robust error handling during runtime.
Rapid Prototyping and YAML-Based Orchestration
A frequent limitation of custom simulator software is the tight coupling of hardware mappings, where transitioning from a basic desktop setup to a high-fidelity motion platform necessitates extensive source code modifications. To address this, the CARLAverse adheres to the principle of separation of concerns by implementing a Single Source of Truth architecture. Executable modules, such as control, force feedback, and visualization, are fundamentally hardware-agnostic. Instead, the instantiation of each simulator node is governed entirely by a comprehensive YAML configuration file, facilitating rapid prototyping and streamlined reconfiguration across multiple domains.
Furthermore, strict experimental reproducibility is a fundamental requirement for behavioral and human factors research. By archiving this central YAML configuration file alongside the experimental rosbag data logs, researchers can ensure that the specific hardware state, physics models, environmental profiles, and control mappings are accurately preserved and can be consistently reconstructed long after an experiment concludes.
Spatial Audio Rendering
Geo-Spatial Sound Generation: The engine extracts metadata from the active OpenDRIVE map and the Unreal Engine environment to generate context-aware ambient noise. For example, it calculates the proximity to virtual vegetation to generate wind rustling, or proximity to water bodies for ocean acoustics.
Dynamic Emitters: Localized events, such as construction noise, sirens, or the tire-road friction of surrounding AI traffic, are spawned as dynamic audio emitters in the 3D space.
Client-Specific Rendering: The central server calculates the spatial coordinates of all sound emitters and transmits them to the respective local nodes. Each simulator node then renders the audio mix locally. This can be done from simple stereo systems, such as headsets, to multi-channel systems.
While this geo-spatial sound generation significantly enhances immersion, the detailed acoustic modeling, frequency masking, and subjective validation of the CARLAcoustics module are highly complex and will be addressed in a dedicated future publication.
Conclusion and Future Work
Developing automated driving systems for urban environments requires detailed insight into mixed-traffic dynamics. Human-in-the-loop simulators provide a safe and repeatable method for investigating such interactions without exposing participants to physical risk. However, a key limitation of traditional simulator setups has often been their operational isolation; studying mixed traffic benefits fundamentally from the synchronous interaction of multiple human actors, including vulnerable road users.
Future Work
While the foundational architecture of the CARLAverse has been successfully implemented and deployed, several avenues for future research, validation, and system enhancement remain.
First, the physical models underlying the vulnerable road user nodes require empirical validation. The longitudinal bicycle dynamics and force feedback implemented in CycloCARLA have been mathematically validated in a dedicated study, to appear. Similarly, the spatial audio propagation and frequency masking algorithms within the CARLAcoustics module will be evaluated in an upcoming publication to quantify their impact on participant immersion and localization accuracy.
To further elevate the physical immersion of the simulator nodes, several hardware extensions are currently under development. For the bicycle simulator, a motion platform is being built to simulate uneven surfaces and slopes, alongside a dedicated wind simulation system. Concurrently, an omnidirectional treadmill is being integrated to support natural, unconstrained locomotion for the pedestrian node.
Enhancing visual fidelity and performance is another critical focus area. A rigorous evaluation of the current virtual-reality render pipeline is planned, which will systematically compare it against different rendering methodologies. As part of this effort, the potential of using neural rendering techniques as an alternative to the default CARLA render engine is being actively investigated to achieve higher levels of photorealism.
Finally, in alignment with the principles of open science, while core modules and tooling are already publicly available, the ongoing effort focuses on completing full system integration and releasing all remaining components of the CARLAverse framework. By providing the complete suite of hardware-agnostic Python modules, YAML templates, and synchronization scripts to the global research community, the authors aim to lower the barrier to entry for immersive, connected simulator research and foster collaborative advancements in the field of automated mobility.
Acknowledgments
This work was developed in the projects KIIWI, reference number 16DHBKI060, which is funded by the German Federal Ministry of Research, Technology and Space, and Real2Sim, reference number BW7_2199/2, which is funded by the Baden-Wuerttemberg Ministry of Economic Affairs, Labor and Tourism.
During the preparation of this work, the authors used Google Gemini for language editing and to assist in generating visual assets, such as project logos. The authors reviewed and edited the content and take full responsibility for the publication.
Frequently Asked Questions
What is the CARLAverse? The CARLAverse is a distributed software and hardware ecosystem for multimodal, multi-agent human-in-the-loop simulation.
How does CARLAverse connect simulator nodes? A central CARLA Unreal Engine server connects over a network to heterogeneous local simulator nodes, including driving, bicycle, and pedestrian setups.
How does CARLAverse support rapid simulator reconfiguration? Hardware-agnostic executable modules are configured through comprehensive YAML files that define each simulator node.
What future hardware developments are planned? Planned developments include a bicycle motion platform, dedicated wind simulation, and an omnidirectional treadmill for pedestrian simulation.