We present GS-Agent, a multi-agent framework that integrates a physics engine in the loop to generate dynamic and physically plausible 4D worlds from natural language descriptions. GS-Agent interacts with the physics engine through executable code, actively seeks multimodal feedback, and collaborates with other agents to iteratively construct coherent and controllable worlds. Experimental results demonstrate that GS-Agent produces physically plausible worlds and offers fine-grained controllability beyond existing methods.
Physics Engine
A physics engine provides a unified computational framework for constructing and simulating dynamic worlds. Its core components include:
Renderer. The renderer component displays the results of the physics simulation on the screen, and it is typically a system separate from the physics engine itself. Renderer parameters include the Camera (its position, angle, and field of view), Lighting (the type, position, color, and intensity of light sources) and Shadows (their quality and draw distance).
Manager Agent
Task decomposition and delegation. Given a natural language description, the Manager Agent interprets the user intent, reasons about the required scene components, and formulates an actionable multi-step plan. It assigns one sub-task at a time to the Entity Agent or Render Agent based on their capabilities—for example, asset creation with appropriate geometry and material properties, spatial placement under geometric constraints, or camera trajectory refinement for a desired cinematic effect. After each sub-task is completed, the Manager Agent critically verifies the result and triggers iterative refinement until the specification is met.
Scene Configuration. The Manager Agent configures the simulation and visualization options both before and during creation. It sets global simulation parameters (e.g., time step, substeps, gravity), solver-specific settings (e.g., MPM grid resolution and domain bounds, SPH particle size), and visualization options (shadows, background color, particle rendering mode, segmentation levels). These configurations ensure stable physics, consistent domain scales, and clear visual diagnostics for downstream reasoning.
Timeline Control. To assess the evolving dynamics of the constructed world, the Manager Agent regulates the temporal progression of the simulation. It can advance or reset the scene to probe intermediate states, evaluate the outcomes of modifications, or perform multi-stage rollouts. After verifying that all components satisfy the language specification, the agent initiates the final execution and triggers video capture, producing the completed 4D world.
Entity Agent
3D Model Asset Generation. If the agent fails to retrieve appropriate 3D model assets that meet the requirement, it will try to generate the 3D model asset by calling a text-to-3D model service and calculate the same metadata of the assets, saving it to the curated assets library. Based on the generation results, the agent may adjust the text query accordingly. After a maximum number of failed attempts, the agent will try to fall back to using primitives to construct the required morphologies.
Entity Placement. Based on 3D model assets metadata, the Entity Agent computes plausible scale, orientation, and position. It aligns assets using canonical orientation, rescales meshes from axis-aligned bounding box bounds, and places them to satisfy world-space constraints such as resting surfaces, non-interpenetration, and adequate spacing. The Entity Agent queries the scene state and asks the Render Agent for visual feedback; if misalignment or collisions are detected, it corrects and iterates.
Entity Motion. The Entity Agent controls motion by editing per-step control functions executed during simulation. Supported controls include PD controllers, direct position/velocity/force commands for rigid bodies, and programmable emitters for particle-based materials. This interface allows scripting kinematic trajectories and transient particle events within the same simulation timeline.
Render Agent
Camera Positioning. The Render Agent determines camera placement and intrinsics for both intermediate inspection and final rendering. It interprets verbal directives, such as requests for specific viewpoints or framings, by grounding them in the world coordinate system. When necessary, the agent queries the spatial state of entities (e.g., positions and extents) to position the camera in a manner consistent with the requested visual perspective.
Camera Motion and Video Capturing. To generate temporally coherent visualizations, the Render Agent represents camera motion as explicit control code that is executed over the course of the simulation. This code specifies camera trajectories (e.g., orbits, dolly movements, or entity-tracking motions) and the corresponding frame-capture schedule. Because trajectories are coded rather than baked into pixels, they can be further refined, for example, through smoothing or reparameterization, without changing the underlying physical simulation. Camera updates are synchronized with simulation time steps, ensuring that recorded frames reflect physically consistent temporal evolution.
Lighting. The Render Agent configures illumination to satisfy the descriptive requirements of a scene while maintaining visual and physical coherence. Depending on the rendering backend, it may adopt directional lights, environment maps, or local light sources. The agent adjusts lighting placement and intensity to achieve the requested mood or emphasis, while avoiding artifacts such as visible light emitters or inconsistent illumination. This allows the generated scenes to reflect the intended lighting conditions in a stable and interpretable manner.
Experimental Setup
To systematically evaluate GS-Agent's ability to construct 4D physical worlds, we conduct experiments across two evaluation suites. The first comprises 24 scenes from the NewtonGen benchmark, specifically designed to test 12 distinct physical laws (e.g., parabolic motion, deformation). To assess the novel capabilities unlocked by our method, we collected a second suite of 30 complex scenes featuring non-continuous dynamics in multi-object interactions (e.g., collisions) and dynamic camera controls.
Implementation Details. We employ Genesis as our underlying physics engine, leveraging its capacity to simulate diverse materials and its fast, photo-realistic rendering system. Unless otherwise specified, we use gpt-5 as the foundation model backbone for both our method and the baselines. To ensure a fair comparison, all output is rendered at 720p resolution to match the constraints of the baseline video generative models, although GS-Agent inherently supports rendering at much higher resolutions.
Emergent Capabilities
Thanks to its flexible end-to-end agentic design, GS-Agent can autonomously detect and resolve unexpected generation failures that would typically require human intervention. Handcrafting robust rules for such edge cases is largely impractical. When an imported 3D bathtub asset was not perfectly waterproof, causing water leakage, GS-Agent recognized the physics failure, proposed a programmatic fix by applying rigid material patches to the geometry's corners, and automatically verified the corrected output. This highlights the system's capacity to intelligently debug and adapt to unforeseen simulation states.
Discussions
GS-Agent generates more than videos. While our quantitative metrics focus on rendered video output, GS-Agent fundamentally constructs a multimodal 4D environment. It naturally yields rich, aligned data structures including metric depth maps, precise segmentation masks, surface normals, and particle-level dynamics. Because the final output is an executable simulation script, it can be seamlessly integrated with robotic embodiments for downstream reinforcement learning and evaluation. Consequently, GS-Agent serves not merely as a video generator, but as an engine for embodied data synthesis.
Conclusion
In this work, we presented GS-Agent, a multi-agent framework that integrates a physics engine in the loop to generate dynamic and physically plausible 4D worlds from natural language. GS-Agent interacts with the physics engine through executable code, actively seeks multimodal feedback, and collaborates with other agents to iteratively construct coherent and controllable worlds. Experimental results demonstrate that GS-Agent produces physically plausible worlds and offers fine-grained controllability beyond existing methods. Looking forward, we envision our approach as a foundation for building large-scale, interactive, and consistent world simulators, bridging the gap between generative models, embodied intelligence, and physics-based simulation for scalable multimodal data creation and new formats of experience.
Frequently Asked Questions
What is GS-Agent and what does it generate? GS-Agent is a multi-agent framework that generates dynamic, physically plausible 4D worlds from natural language descriptions by integrating a physics engine, producing executable simulation scripts rather than just videos.
How does GS-Agent handle generation errors autonomously? The system can detect physics failures, such as water leakage from a non-watertight 3D asset, and automatically propose and verify programmatic fixes without human intervention.
What types of control does the Render Agent provide? The Render Agent controls camera positioning, motion trajectories (orbits, dollies, entity-tracking), lighting configuration, and video capture, all synchronized with simulation time steps.
Can the output of GS-Agent be used for robotics applications? Yes, because the final output is an executable simulation script, it can be directly integrated with robotic embodiments for downstream reinforcement learning and evaluation.