Controlled Experimental Testbed
The robot receives two sources of state information. Wheel odometry provides incremental motion measurements, while range-bearing measurements to known landmarks provide intermittent absolute corrections. The odometry stream is corrupted by multiplicative wheel slip and additive gyroscope bias. Landmark sensing is degraded systematically along four axes: range-measurement noise, the number of visible landmarks, gyroscope bias, and scheduled sensing outages during which all landmark observations are unavailable. These perturbations allow us to study estimator behavior under gradually worsening sensing conditions while keeping the robot dynamics and control law fixed.
The controller does not have access to the ground-truth pose. Instead, each control input is computed from the pose estimate produced by the active observer. Estimation errors therefore affect the applied control command, which changes the subsequent robot trajectory and the measurements encountered later in the run. This coupling is central to the experimental design. An estimator that appears accurate when evaluated on a fixed trajectory may produce different behavior once its estimates are used directly in the feedback loop.
Predictive State Models
Dead Reckoning and Extended Kalman Filtering
Dead reckoning (DR) propagates the robot pose using odometry alone and does not apply measurement-based corrections. The extended Kalman filter (EKF) uses the same unicycle motion model together with analytic Jacobians for the landmark range-bearing observation model. Between landmark observations, the EKF propagates the state and covariance using the motion model; when a valid landmark measurement becomes available, it performs a standard measurement update.
Pose composition combines the predicted pose with the correction computed from the residual measurement.
The six primary observers considered in the study are DR, EKF, GRU-DR, SSM-DR, GRU-EKF, and SSM-EKF. The recurrent models are trained by backpropagation through time using 200 domain-randomized expert trajectories that span variations in sensing quality and motion uncertainty.
Training with Extended Sensing Outages
To examine whether exposure to longer periods without landmark measurements improves robustness, additional versions of SSM-EKF, GRU-EKF, and SSM-DR are trained. These models are denoted by extended-blackout variants. They are trained on 300 trajectories, half of which contain an additional sensing outage lasting between 32 and 58 time steps. The standard training data contain outages lasting between 6 and 27 steps. The network architecture, training objective, and optimizer are otherwise unchanged.
The sensing outage is introduced by masking the landmark-measurement channels while leaving the simulated vehicle dynamics and odometry sequence unchanged. Because landmark sensing does not alter the physical evolution of the simulated robot, this procedure isolates the effect of missing measurement updates while preserving the underlying motion. The resulting models allow us to test whether training on longer periods of prediction without external correction translates to improved closed-loop performance.
Evaluation Protocols
Each observer is evaluated under three protocols that differ primarily in the availability of future measurements and in whether the estimated state is allowed to influence the robot trajectory. Using the same set of observers under all three protocols allows comparison of conventional offline estimation accuracy, measurement-free prediction accuracy, and actual closed-loop control performance.
Trajectory Replay with Measurement Updates
In the replay evaluation, every observer processes the same recorded trajectory. The input sequence contains the realized odometry, landmark range-bearing measurements, and the corresponding measurement-availability masks. Because the robot trajectory is fixed, differences between observers do not alter the future inputs presented to the estimator.
The resulting pose estimates are evaluated against the ground-truth trajectory using position RMSE and heading RMSE. This protocol represents the conventional offline evaluation of localization or state-estimation accuracy.
Measurement-Free Rollout
The second protocol evaluates the ability of each observer to propagate its state estimate in the absence of future landmark corrections. The recorded trajectory is first replayed through the observer, and its complete internal state is saved at regularly spaced checkpoints. At each checkpoint, a copy of the observer is propagated forward for a specified rollout horizon using only the realized odometry sequence. Landmark measurements are withheld for the entire rollout.
Position RMSE is measured at the final rollout horizon and aggregated across checkpoints and ten evaluation seeds.
This evaluation measures how rapidly each observer accumulates state error when it must rely entirely on motion propagation for two seconds. Unlike the replay protocol, the measurement-free rollout explicitly tests multi-step prediction without external correction, while still using the same realized odometry sequence for every observer.
Closed-Loop Path Tracking
In the closed-loop evaluation, the observer operates online inside the simulator and supplies the pose estimate used by the pure-pursuit controller. The resulting control commands are applied to the simulated robot, so estimator errors directly affect the subsequent vehicle trajectory.
Landmark measurements continue to arrive according to the sensing condition being evaluated, and the observer incorporates those measurements whenever they are available. Cross-track RMSE is used as the primary measure of path-tracking performance. Control effort, recovery time following sensing outages, and the fraction of runs that diverge are also recorded.
The three protocols therefore expose each observer to different information and feedback structures:
- Replay permits repeated measurement correction but holds the robot trajectory fixed.
- The measurement-free rollout removes landmark corrections for a prescribed prediction horizon while still replaying the realized odometry sequence.
- Closed-loop evaluation combines intermittent measurement updates with feedback, allowing estimation errors to alter future control inputs, robot motion, and sensing geometry.
The central question of the paper is whether either offline protocol preserves the ranking of observers obtained when those same observers are used for closed-loop control.
For each of the 24 sensing conditions, every offline metric produces a ranking over the six primary observers. This ranking is compared with the ordering obtained from closed-loop cross-track RMSE under the same sensing condition. In addition to the overall Spearman rank correlation across the 144 observer-condition pairs, the frequency with which an offline criterion selects an observer other than the closed-loop optimum is reported. The resulting selection regret relative to the best closed-loop observer for that condition is also quantified.
Implications for World-Model Evaluation in Robotics
Although the experimental platform considered in this study is intentionally simple, the underlying evaluation issue applies more broadly to predictive models used in robotic systems. The information available during an offline rollout often differs from the information available when the same model is used online.
An offline benchmark may evaluate a long prediction from a fixed initial context, whereas a deployed robot may receive new camera images, proprioceptive measurements, force measurements, or other sensor updates after only a few control cycles. The state estimate is then corrected and the controller is evaluated again from the updated state. As a result, the usefulness of a predictive model depends not only on how accurately it propagates the system forward, but also on the sensing and feedback schedule under which those predictions are used.
Prediction Accuracy and Closed-Loop Agreement
State-prediction error remains an important measure of model quality, but it does not directly describe how estimation errors affect the controlled system. Offline evaluation can therefore be strengthened by reporting quantities that compare model rankings at the prediction and control levels, such as estimator-selection accuracy, policy-ranking agreement, closed-loop regret, or task success.
In the experiments, the 20-step measurement-free rollout provides a meaningful measure of state propagation, yet it produces a substantially different estimator ranking from the one obtained in closed-loop path tracking. Reporting rank agreement and selection regret makes this distinction visible even when average state-prediction errors appear reasonable.
Separating State Propagation from Measurement Correction
State propagation and measurement correction should be evaluated separately. Predictive models used in robotics increasingly combine multiple sensing modalities, including vision, proprioception, force sensing, and other external measurements. The arrival of a new measurement changes the state estimate and can substantially alter the subsequent control action.
A model that accumulates drift during a long period of open-loop propagation may still perform well if it incorporates the next measurement effectively. Conversely, a model with accurate short-term propagation can perform poorly if its state corrections introduce errors that are unfavorable for feedback control. Evaluation should therefore characterize both how the model propagates the system between measurements and how its state estimate evolves when new measurements become available.
Matching Evaluation to the Intended Application
These observations do not imply that short-horizon evaluation is generally preferable to long-horizon prediction. Predictive models used for planning, model-predictive control, or policy learning may rely on substantially longer periods of model propagation than the state estimators considered here. In those applications, long-horizon prediction accuracy may be directly relevant to performance.
The results instead indicate that the relationship between rollout accuracy and closed-loop utility depends on how the predictive model is incorporated into the robotic system. Evaluation protocols should therefore reproduce the sensing, state-update, and control structure of the intended application as closely as possible.
Limitations
The amount of training replication also differs across experiments. The four principal EKF-anchored recurrent estimators—SSM-EKF, GRU-EKF, and their extended-blackout counterparts—are each evaluated using three independent training seeds, as reported in the appendix. Five or more independent training runs would provide a stronger estimate of training variability, particularly for the SSM-EKF model trained with extended sensing outages, which exhibits noticeably larger variation across initializations.
Some secondary experiments involving DR-anchored observers and ablations use fewer independent training runs and should therefore be interpreted primarily as supporting evidence rather than as independent statistical conclusions.
The matched-data study controls for training-set size while changing the distribution of sensing-outage durations. In that experiment, matching the training blackout distribution to the evaluation conditions does not provide a consistent closed-loop advantage. The result should therefore be interpreted as evidence that blackout-duration matching alone is insufficient, rather than as evidence for a particular training curriculum.
These limitations define the scope of the conclusions. The experiments do not establish a universal evaluation metric for large visual world models, nor do they claim that measurement-free rollouts are inappropriate for robotic planning in general.
They demonstrate a more specific result: when a predictive state model is used inside a feedback controller and receives intermittent measurement corrections, an offline rollout can produce a misleading model ranking if its measurement schedule differs substantially from the one encountered during closed-loop operation.
Extending the same controlled comparison to visual predictive models, additional robot dynamics, different controllers, and physical closed-loop experiments is an important direction for future work.
Conclusion
This paper examined how offline prediction accuracy relates to closed-loop control performance when a predictive state estimator is used inside a robotic feedback loop. In a differential-drive path-tracking task with biased odometry and intermittent landmark sensing, replay-based estimation error, multi-step measurement-free rollout error, and closed-loop tracking performance were compared across six observers and 24 sensing conditions.
The 20-step measurement-free rollout was less consistent with the closed-loop observer ranking than replay position RMSE, despite explicitly evaluating multi-step state propagation.
The horizon and measurement-update experiments explain this result more precisely. Long prediction horizons remain informative when the estimator continues to receive regular measurement corrections, while the correspondence with closed-loop performance deteriorates substantially when long propagation is combined with infrequent or absent measurement updates. The relevant evaluation variable is therefore not prediction horizon alone, but the combination of prediction horizon and measurement-update schedule.
Taken together, the results suggest a practical principle for evaluating predictive models in robotics. Prediction error should be measured under an information pattern that reflects how the model will actually be used. For models operating inside a feedback controller, this means specifying when new measurements become available, when the state estimate is updated, and when new control inputs are computed.
A useful rollout benchmark should therefore reproduce the sensing and feedback structure of the intended robotic application, rather than treating prediction horizon as the sole measure of temporal difficulty.
Acknowledgments
This project began as a final project for ECE 6562 (Autonomous Control of Robotic Systems), Georgia Institute of Technology, Summer 2026, and was extended with the blind-rollout evaluation and long-blackout training experiments reported here.
Frequently Asked Questions
What robot task was used in the study? The study used a differential-drive path-tracking task with biased odometry and intermittent landmark sensing.
Which observers were compared? The six primary observers were DR, EKF, GRU-DR, SSM-DR, GRU-EKF, and SSM-EKF.
What evaluation protocols were used? The observers were evaluated through trajectory replay with measurement updates, measurement-free rollouts, and closed-loop path tracking.
What is the main evaluation lesson? Prediction accuracy should be measured under a sensing and feedback schedule that reflects how the predictive model will actually be used.