Learning robot policies from human preferences traditionally forces annotators to give a single binary score comparing two trajectories, collapsing complex judgments into a single number. Freeform Preference Learning (FPL) instead asks annotators to describe the dimensions they care about in natural language—such as speed, safety, or task completion—and provide a separate preference for each axis, preserving the richness of human feedback.
Introduction
To preserve these axes of judgment, the key insight is to allow annotators to provide preference labels on any task-relevant axes of their choosing. This multidimensional supervision can then be used to learn a policy optimized for the combination of these dimensions. FPL asks annotators to specify relevant judgment dimensions in natural language and to provide a binary preference for each axis. The axes can be defined up front or during the annotation process. A multi-axis reward function is learned that produces a scalar reward score when conditioned on a natural language description of the axis. Finally, a promptable policy is trained to optimize the combination of axes described during reward training. This framework simultaneously improves both the ease of providing unambiguous supervision and the density of supervision for downstream policy optimization.
Learning from Freeform Preferences
The key idea behind FPL is to learn from natural language and open-ended feedback instead of traditional binary preferences. Rather than asking annotators for one overall preference between two trajectories, they are asked to describe the axes along which to compare them, such as speed, safety, smoothness, or subtask completion. This yields feedback that is more granular and less ambiguous. These freeform preferences are used to learn a language-conditioned, multi-dimensional reward function that scores trajectories along each specified axis. A policy conditioned on multiple preference dimensions is then trained to optimize the behavior with respect to each axis.
Analysis and Experimental Evaluation
The experimental section is designed to answer the following questions: (a) Does FPL learn effective policies through freeform human preferences? (b) Do policies learned through FPL exhibit compositionality of behaviors unseen in the data? (c) Does FPL exhibit steerability of rewards at test time? (d) Does FPL learn denser reward functions for long-horizon tasks?
Experimental Setup
To empirically respond to the questions above, four real-world manipulation tasks and two simulation tasks are considered. All tasks start from offline demonstrations with varying quality and strategies. Mean and standard error are reported, using 20 rollouts per real-world method and three seeds in simulation.
Qualitative Analysis
Freeform preferences axes naturally change throughout iterations. As the policy performance improves at each iteration, annotators change from more coarse feedback to more concrete and perfectionist feedback. In the early iterations of a fold shorts task, the annotator focuses on whether each fold happens at all; later, once folding is reliable, whether there are wrinkles and final alignment are perfected.
Freeform preferences reduce annotation time per label. Collecting freeform preferences is approximately 50% faster per label than collecting single binary preferences. Because annotators provide multiple axis-specific judgments for each trajectory pair, the cost of viewing the videos is shared across several labels, which reduces the annotation overhead.
Limitations and Conclusion
FPL introduces a method for learning robot policies from freeform human preferences. By collecting preferences with natural-language axes, FPL provides denser and less ambiguous supervision than single binary preferences. Across four real-world tasks and two simulation settings, FPL outperforms the baselines. Preserving the multi-dimensional structure of human feedback enables compositionality of behaviors and test-time steerability of the learned policy, as well as qualitatively learning reward models with better credit assignment.
Several limitations remain. Preference learning requires collecting human preferences, which is more expensive than fully unsupervised approaches. Reward-conditioned policy learning requires selecting appropriate reward values at test time; automating this selection is an important direction for future work. The current policy is conditioned on a fixed set of preference axes, and extending this method to handle variable axes is a promising direction as VLAs become more capable.
Real-World Tasks
All four real-world tasks use the DROID setup on a Franka robot, with two camera views (wrist and third person) as observations and joint-velocity control. Each task starts from offline demonstrations of varying quality and strategy.
The robot is tasked with plating a piece of toast, either by scooping it with a spatula or picking it up directly with the gripper from a tray and placing it onto a plate. The initial position and orientation of the toast on the tray vary between episodes, as does the position of the spatula. Unlike the structured tasks, the preference axes were not fixed in advance; instead, the annotators freely specified the dimensions most relevant to them. The collected axes include speed, smoothness of motion, cleanliness, quality of plating, and whether the robot damaged the toast.
Simulation Tasks
Two simulation tasks and one additional variation are proposed, which are extensions from the robomimic original tasks. In one task, two objects are present in the scene—the bread and the can. The goal is to place them in the correct box, in the correct order (first bread then can), and without dropping them. The offline dataset contains a mix of behaviors: data that places in the wrong order or that drops the objects, as well as perfect demonstrations.
Preference Collection
Preferences are collected through a custom web interface in which annotators are presented with two side-by-side video recordings of robot rollouts and asked to indicate which rollout is preferred along each axis or indicate equivalence. For some tasks (Setup Table, Fold Shorts, and Place Cube into Bowl), the axes are predefined and vary by task. For others, like the Plate Toast task, annotators additionally specify their own preference axes in free-form text prior to providing their preferences. Collecting freeform preferences is cheaper in time than collecting single preferences when looking at the time spent per label. The annotator can provide multiple labels instead of only one when watching a single pair of videos of two trajectories, while for single preferences they can only provide one label. The speedup observed is around 1.85 times faster to provide freeform preferences than single preferences.
Implementation Details
For reward-model learning, annotators are allowed to specify that two trajectories are equivalent in some axis; in those cases the label is dropped.
Compute Details
The compute resources used were one H100 GPU for full finetuning of the policy and finetuning of the reward model. For the simulation experiments, smaller GPUs were used, generally NVIDIA RTX 4500.
Frequently Asked Questions
What makes freeform preference learning different from standard preference learning? Instead of collapsing all judgment into a single binary score, FPL asks annotators to specify judgment axes in natural language and provide separate preferences for each axis, giving denser and less ambiguous supervision.
Does FPL work for long-horizon tasks? Yes, FPL learns denser reward functions for long-horizon tasks because annotators can assign credit along specific axes (like subtask completion) rather than only an overall comparison.
How much faster is freeform preference annotation? Collecting freeform preferences is approximately 1.85 times faster per label than collecting single binary preferences, because viewing costs are shared across multiple axis-specific judgments.
Can the learned policy handle different preference dimensions at test time? Yes, the policy is conditioned on multiple preference dimensions, enabling test-time steerability where users can specify which axes to optimize for.