RoboRMBench
RoboRMBench evaluates whether language-conditioned reward models assign consistent scores to the same robot trajectory under semantically equivalent goal descriptions. The benchmark pairs each trajectory with linguistically diverse paraphrases of the same goal, allowing reward instability caused by instruction wording to be measured while keeping the visual evidence and task intent fixed.
Problem Formulation
A language-conditioned reward model assigns a progress score to a robot trajectory given a textual goal description. Let the trajectory denote the sequence of robot states and actions, and let the paraphrase set denote a collection of semantically equivalent goal descriptions for the same underlying goal. For each description, the reward model produces a predicted progress score.
Because the trajectory and task intent are fixed, a paraphrase-robust reward model should produce stable predictions across the paraphrase set. Variation among the scores should reflect differences in robot behavior, not differences in how the same goal is worded. RoboRMBench evaluates this property directly by holding the trajectory fixed and varying only the goal description within a verified paraphrase set.
Main Results
Many general-purpose vision-language models exhibit high score crossing rates and failure rates, indicating that reward judgments can shift substantially depending on how the task is phrased. This instability is not a minor numerical fluctuation: under adversarially generated paraphrases, models such as Gemini2.5-flash-lite and Llama4-scout cross the failure–success boundary on more than half of all trajectories. The same behavior can therefore be labeled as success or failure depending solely on instruction wording.
Instability also tends to increase with the degree of linguistic divergence. In most cases, score crossing rates and failure rates are higher under structural rewrites and adversarially generated paraphrases than under lexical substitutions. This suggests that vision-language models are more sensitive to structural and perspective-level rewrites than to lexical variation.
For example, GPT-5.1 has a score crossing rate of 0.153 under lexical substitutions, which rises to 0.204 under structural rewrites and 0.300 under adversarially generated paraphrases.
Additional Analysis
The study investigates whether explicit reasoning mitigates prompt variance by comparing reasoning-enabled and non-reasoning variants. The main analysis compares the Instruct and Thinking versions of Qwen3-vl-235B-A22B, as well as Gemini3-flash with reasoning disabled and enabled.
Mitigation
A natural way to reduce sensitivity to instruction wording is to average out the textual surface form. Ideally, a language-conditioned reward model should estimate reward from the trajectory and the underlying task intent while remaining invariant to which semantically equivalent instruction expresses that intent. This invariance is approximated by treating paraphrases as multiple textual realizations of the same task intent and averaging the model’s predictions across them.
For each trajectory, each paraphrase in the evaluation set is treated as a seed prompt, and five additional semantically equivalent variants are generated from it. The model’s predicted scores within the variants generated from the same seed are then averaged, yielding one aggregated reward for each seed prompt. This procedure marginalizes over local wording choices while keeping the trajectory and task intent fixed.
Aggregation consistently reduces both score crossing rates and prediction error across the evaluated models and paraphrasing strategies. Nevertheless, aggregation only partially mitigates the problem: it requires multiple reward-model queries, and the aggregated predictions still exhibit non-negligible score crossings.
Conclusion
Vision-language reward models can assign different rewards to the same trajectory under semantically equivalent goal descriptions. RoboRMBench measures this failure mode with verified paraphrases and robustness metrics.
Across models, paraphrase-induced instability is widespread and severe, worsens under stronger rewrites, and persists despite scale or explicit reasoning. It also affects offline best-of-n selection, where more stable models choose trajectories with higher ground-truth progress. These findings establish paraphrase robustness as necessary for reliable language-conditioned reward modeling.
Limitations
RoboRMBench evaluates reward predictions under paraphrased task descriptions generated through controlled strategies and filtered for semantic equivalence by an ensemble of large language model validators. This pipeline provides scalable and consistent paraphrase quality, but all instructions in the benchmark are in English. Paraphrase robustness in multilingual or code-switched settings is therefore unexamined.
The experiments also focus on end-of-episode reward predictions. Other reward-modeling setups, such as step-level shaping or trajectory comparison, may exhibit different robustness properties.
Acknowledgment
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation grants funded by the Korean government through the Ministry of Science and ICT:
- RS-2026-25522885
- RS-2021-II211343, Development of a World Foundation Model for Training and Development of Physical AI Systems
- RS-2020-II201361, Artificial Intelligence Graduate School Program, Yonsei University
The work was also supported by the Artificial Intelligence Graduate School Program at Seoul National University and the National Research Foundation of Korea grant funded by the Korean government through the Ministry of Science and ICT in 2026:
- RS-2026-25612350, Development of an End-to-End Design and Validation Framework for Mathematically Principled Next-Generation AI Architectures and Learning Algorithms
Model Access and API Configuration
For proprietary and hosted open-source models, the OpenRouter Chat Completions API was used when available. The models evaluated through OpenRouter were:
- GPT-4o
- GPT-5.1
- Gemini2.5-flash-lite
- Gemini2.5-flash
- Gemini3-flash
- Claude-haiku-4.5
- Claude-sonnet-4.6
- Qwen3-vl-2B, 4B, 8B, and 32B
- Qwen3-vl-235B-A22B
- Gemma3-4B, 12B, and 27B
- Llama4-scout
- Llama4-maverick
RoboReward-4B and RoboReward-8B were evaluated locally using their released checkpoints and inference code.
Unless otherwise specified, the provider- or model-specified sampling configuration was used for each model family rather than imposing a single universal decoding setup:
- OpenAI models: temperature = 1.0 and top_p = 1.0
- Gemini models: temperature = 1.0 and topP = 0.95
- Claude models: temperature = 1.0 and top_p = 0.999
- Qwen3-vl models: temperature = 0.7 and top_p = 0.8
- Gemma3 models: temperature = 1.0 and top_p = 0.95
- Llama 4 Scout and Maverick: temperature = 0.6 and top_p = 0.9
For locally evaluated RoboReward models, inference used PyTorch 2.5.1, Transformers 5.0.0, CUDA 12.1, and bfloat16 precision. All local experiments were run on NVIDIA L40S 48 GB GPUs. Unless otherwise specified by the released checkpoint, local decoding used greedy generation with temperature = 0.0, do_sample = false, and max_new_tokens = 32.
Input Construction and Frame Sampling
Each evaluation example consists of a robot trajectory, a task instruction, and a ground-truth progress label on a five-point ordinal scale. For each trajectory, the visual input is kept fixed and only the task instruction is varied. Any variation in predicted reward scores across paraphrases therefore reflects sensitivity to instruction wording rather than changes in visual evidence.
One RGB frame per second is uniformly sampled over the episode duration, always including the first and last frames. If the sampled sequence exceeds a model’s image capacity, frames are uniformly subsampled to fit the model-specific limit while preserving the first and last frames.
Reward Scoring Prompt
All general-purpose vision-language models are queried using the same scoring prompt template. The prompt asks the model to evaluate the progress of the robot trajectory with respect to the provided task instruction and return a single integer score from 1 to 5.
For all paraphrases of the same trajectory, the visual frames and all prompt text other than {instruction} are held fixed.
Decoding Stochasticity and Repeatability
The main experiments use one model call per trajectory–instruction pair. Because hosted APIs may apply provider-side defaults and exact determinism is not guaranteed across providers, the measured instability is interpreted as deployment-realistic behavior rather than as a purely deterministic property of a fixed local model.
To verify that paraphrase-induced instability exceeds ordinary repeated-call variation, a repeatability analysis was conducted on a stratified subset of 300 trajectory–instruction pairs. The subset was balanced across paraphrase strategies and ground-truth progress labels. Each selected pair was queried ten times using the same visual frames, prompt, API configuration, and instruction.
Repeated-call variation within the same instruction was compared against variation across paraphrases of the same trajectory. The average repeated-call standard deviation across all evaluated models was 0.09, whereas the average across-paraphrase standard deviation was 0.52. Paraphrase-induced variation was therefore substantially larger than ordinary repeated-call variation and was the dominant source of reward instability in the evaluation.
Range and Standard Deviation
Prediction range and standard deviation across paraphrases were additionally reported as complementary measures of score dispersion. These statistics capture the overall variability of reward predictions but do not directly reflect category-level contradictions measured by score crossing rate.
A model may exhibit small numerical variability while still producing failure–success reversals, or show larger dispersion without crossing the failure–success boundary. These observations suggest that score crossing rate captures a different aspect of paraphrase robustness beyond numerical variability.
Additional Reasoning Model Results
Reasoning-enabled variants of GPT-5.1 and Claude-sonnet-4.6 were additionally evaluated. The effect of reasoning is model-dependent.
For GPT-5.1, enabling reasoning reduces both score crossing rate and mean error, indicating improved robustness and accuracy. For Claude-sonnet-4.6, however, reasoning substantially increases mean error and increases score crossing rate, indicating greater instability to paraphrased instructions.
Together with the main Qwen3-vl and Gemini3-flash results, these comparisons show that explicit reasoning does not provide a consistent improvement in paraphrase robustness across models.
Frequently Asked Questions
What does RoboRMBench evaluate? It evaluates whether reward models assign consistent scores to the same robot trajectory when the goal is expressed through semantically equivalent descriptions.
Why are paraphrases used in the benchmark? Paraphrases keep the task intent fixed while testing whether instruction wording changes the model’s reward prediction.
Does explicit reasoning consistently improve paraphrase robustness? No. Its effect is model-dependent: it improves robustness for some models but increases instability for others.
How does paraphrase aggregation mitigate reward instability? The method generates five additional equivalent variants for each seed prompt and averages their predicted scores, reducing but not eliminating instability.