Vision models can estimate a tennis ball’s position, track a player, and anticipate the next stroke from video recorded from the player’s moving viewpoint. This study compares image, video, and vision-language models across seven perception and prediction tasks, then uses visual estimates to generate whole-body motion—an early test of how visual AI can help robots respond in fast-changing environments.
What did the researchers build?
The researchers created a study setup for teaching and testing visual models on tennis footage filmed from a player’s point of view. The dataset pairs egocentric video—video from a camera moving with the player—with allocentric video, meaning views from cameras positioned around the court. Those external views provide a broader reference for where players and the ball are, while the wearable viewpoint captures what a robot would see while moving and acting.
The team aligned the recordings using a visible synchronization signal: the player clapped at the beginning and end of a session. They also calibrated the cameras, allowing image locations to be related to the court and to physical positions. Annotated information includes the ball’s position, the player’s position, and whether the player’s next stroke is a forehand or backhand. The ball’s location in the player’s view is labeled for 97.4% of frames; the remaining frames contain occlusions.
The work tests several types of pretrained vision model, including image encoders, video encoders, and vision-language models (VLMs). It goes beyond scoring predictions: the researchers also use visual estimates to condition a system that generates whole-body movement.
What did the experiments show?
The main result is that no single model family stands out as the answer for every task. Image encoders and vision-language models deliver broadly competitive performance, but their relative strengths change depending on what the system needs to perceive or predict. That matters for robot designers: choosing a model for one skill, such as locating a ball, does not establish that it will be best at forecasting a player’s next action.
The study examines seven tasks involving spatial understanding and anticipation from dynamic tennis video. One practical finding is that adding depth information substantially improves ball localization. In other words, visual appearance alone is not as useful for locating the ball as appearance paired with information about how far away it is.
More visual history is not automatically better. Experiments that varied the number and spacing of input frames found that longer histories did not consistently improve future ball-position predictions. The dataset’s 97.4% ball-label coverage is a measurement of annotation availability, not a model’s prediction accuracy. The supplied results do not give per-task error scores or detailed head-to-head margins, so a numerical model ranking cannot be reported accurately. The takeaway is the comparison across model types and the specific effects of depth and history, not a single headline accuracy.
How does the vision-to-motion pipeline work?
The system starts with a pretrained visual encoder: a model that converts images or video into a stream of numerical features. The researchers test image-based, video-based, and vision-language encoders to see how useful their existing visual representations are for tennis tasks. Rather than retraining each encoder from scratch, they attach small prediction heads to its output. These heads read the encoder’s features and produce task-specific estimates.
Each head uses learnable queries—compact representations of the outputs the model needs to make—to attend to the encoder’s tokens. Tokens are the feature units representing parts of the input, such as image patches and, for VLMs, language-related features. A two-layer neural network then converts the attended information into predictions. The heads are trained together, with a shared positional embedding to help the system keep track of where features came from. Continuous predictions, such as positions, use an L2 loss; stroke categories use a cross-entropy loss.
For future ball position, the team varies how much video history the model receives and how far apart the frames are. The history tests include SigLIP 2 and V-JEPA 2. V-JEPA 2 is not tested with a single-frame input because its video processing requires groups of at least two frames.
Finally, the researchers feed visual estimates into conditional motion diffusion. Diffusion is a process that generates a movement sequence by progressively refining motion toward one that fits the conditioning information. Here, the conditioning comes from the tennis scene estimates. This links perception and prediction to a possible action output, although the experiment demonstrates generated motion rather than a robot physically playing tennis.
Why does this matter for robotics?
Many robots must make decisions from cameras that move with the robot rather than from fixed, overhead viewpoints. A mobile robot needs to interpret changing images while it turns, walks, or reaches. Tennis is an unusually demanding test of that challenge: the ball moves quickly, can disappear from view, and must be understood in relation to both the player and the court.
The study gives robot developers a structured way to ask whether a vision model can do more than recognize objects in a still image. Can it estimate where a moving object is? Can it predict where that object will go? Can it identify what action a person is about to take? The findings also make depth and input history practical design choices to evaluate, rather than assuming that more frames or a larger model always produce better results.
The motion-generation step points toward a longer-term path from perception to control: visual estimates can inform a proposed movement, which a robot’s control system could then execute and refine. That makes the work relevant to agile humanoids, including systems that need to react to objects or people in motion. Readers can browse humanoid robots on Robot Overflow. The same questions about moving viewpoints and timely reactions also apply to mobile platforms in busy facilities, including warehouse robots.
What are the limitations and open questions?
Tennis is a useful stress test, but it is only one environment. The results do not establish that the same models or adaptation strategy will work equally well for catching, throwing, team sports, or navigation around fast-moving people. Those settings differ in object appearance, motion, occlusion, and the actions a robot must take.
The reported experiments also stop short of a physical robot controlling a real match. Generating whole-body motion from visual estimates is a step toward action, not proof of safe, reliable real-time control. A robot would also need to handle sensing delays, prediction errors, balance, contact forces, and safety around people.
Finally, the supplied results do not include detailed numerical scores for each task or model. That makes it impossible to judge exact error levels, compare margins, or assess how much better one model is than another from this summary alone. Further testing should report those scores and check how well the approach transfers beyond tennis footage.
What questions do readers ask most often?
Does the system make a tennis-playing robot? No. It predicts information from tennis video and demonstrates generated body motion, but it does not show a robot physically playing tennis.
How often is the ball labeled in the player’s view? The ball’s two-dimensional position is annotated for 97.4% of frames, with occlusions accounting for the remainder.
Do longer video histories improve predictions? Not consistently. The experiments report that adding more visual history does not necessarily improve future ball-position prediction.
What helped ball localization most? Adding depth information substantially improved localization, according to the study’s reported findings.
What is the main takeaway?
This work tests vision models on the connected challenges of locating a ball, understanding a player’s state, and anticipating what happens next from a moving camera. It finds that model strengths depend on the task, that depth can improve localization, and that more video history is not automatically more useful.