We propose ARDY, an autoregressive diffusion model that enables interactive and controllable human motion generation. ARDY natively supports online text prompting and flexible kinematic goal constraints tailored to interactive applications, including long-horizon goals that extend beyond a single generation window. We present a real-time demonstration of interactive and instructable motion generation, underscoring the potential of generative models for future animation systems. We validate our architectural decisions through extensive ablation studies on the large-scale, studio-quality Bones Rigplay dataset. Furthermore, experiments on the public HumanML3D benchmark demonstrate that ARDY outperforms existing methods in terms of both motion fidelity and control accuracy.
Hybrid Motion Representation
To balance the representational compactness required for efficient generative learning with the need for direct, precise control via explicit feature overwriting, we propose a hybrid motion representation that decouples root motion from body motion. Specifically, root trajectories are represented in an explicit, interpretable form, while body motion is encoded in a compact latent space.
Our hybrid representation builds on an explicit motion representation. Each frame of a motion that uses this explicit representation is defined as a tuple of root and body skeleton joint features. This approach provides the compactness needed for generative learning while enabling direct control through explicit feature manipulation.
Training and Implementation Details
In practice, our motion tokenizer uses a patch size of frames per patch. Both the encoder and decoder are implemented as 8-layer transformers with a latent dimension of 512, utilizing causal self-attention to preserve temporal consistency. The tokenizer is trained on motion clips of varying lengths (1–10 seconds) using a reconstruction loss and additional loss penalizing foot skating.
Interactive Motion Generation Demo
To showcase ARDY's versatility, we developed an interface using Viser to interactively generate motion with our model. The system enables real-time character control through a combination of streaming text prompts and interactive spatial constraints provided via mouse and keyboard inputs.
Demo Results
Our system supports diverse locomotion interfaces: users can define target root trajectories in real time using mouse-based waypoints or modulate real-time velocity via keyboard commands. For mouse-based root path control, we derive the target trajectory by linearly interpolating between mouse-click waypoints and smoothing the resulting path. For keyboard-based root velocity control, we compute a target velocity from user input and the current velocity, then linearly interpolate between the two and integrate the resulting per-frame velocities to derive the root trajectory input to the model.
Analysis on Large-Scale MoCap Data
We thoroughly analyze key design choices of ARDY along with the effects of various hyperparameter settings.
Ablation Study
Ablation results are presented on three key design choices: the hybrid motion representation, the global-to-local root conversion within the tokenizer decoder, and the two-stage denoiser design.
Hyperparameter and Tokenizer Type Analysis
An analysis of the generation horizon length, the number of diffusion steps, and the tokenizer configurations is provided.
Benchmark Evaluation
We evaluate ARDY against both offline and online state-of-the-art baselines for text-constrained generation on the standard HumanML3D dataset. For these experiments, our model is trained with a 40-frame generation horizon using 10 diffusion steps and a vanilla autoencoder tokenizer. ARDY outperforms existing methods in terms of both motion fidelity and control accuracy.
Discussion
We propose ARDY, an autoregressive motion diffusion model that enables interactive and controllable human motion generation. ARDY natively supports online text prompting and flexible kinematic goal constraints tailored to interactive applications, including long-horizon goals that extend beyond a single generation window. We present a real-time demonstration of interactive and instructable motion generation, underscoring the potential of generative models for future animation systems. We validate our architectural decisions through extensive ablation studies on the large-scale, studio-quality Bones Rigplay dataset. Furthermore, experiments on the public HumanML3D benchmark demonstrate that ARDY outperforms existing methods in terms of both motion fidelity and control accuracy.
Frequently Asked Questions
What makes ARDY different from other human motion generation models? ARDY is an autoregressive diffusion model that supports interactive, real-time motion generation with online text prompting and flexible kinematic goal constraints, including long-horizon goals.
How does ARDY handle root motion and body motion separately? ARDY uses a hybrid motion representation that decouples root motion (represented explicitly) from body motion (encoded in a compact latent space), balancing compactness for generative learning with direct control.
What datasets were used to evaluate ARDY? ARDY was validated on the large-scale Bones Rigplay dataset for ablation studies and evaluated on the public HumanML3D benchmark for comparison against state-of-the-art methods.
Can ARDY generate motion in real time? Yes, ARDY supports real-time interactive motion generation through keyboard and mouse inputs, enabling root trajectory control, velocity modulation, and streaming text prompts.