In recent years, reinforcement learning from human feedback (RLHF) has become the dominant technique for aligning decision-making agents with human intentions. The ease of providing preference feedback has been a crucial factor in its popularity as a feedback type for reward modeling, but since each feedback provides at most one bit of information, preference feedback is also known for poor sample efficiency. Asking a human thousands of comparison questions to learn a reward model is often not scalable.
To the best of our knowledge, this is the first work to leverage subspace filtering to train neural network reward models from preference feedback. We compare PreferenceEKF to four widely used Bayesian deep learning methods for active preference-based reward learning in continuous control tasks. We further test whether the learned reward models are useful for policy optimization in offline reinforcement learning tasks. Experiments are conducted in the D4RL and V-D4RL benchmarks.
The findings are as follows:
- PreferenceEKF performs on par with or better than all Bayesian deep learning baselines in terms of sample efficiency and calibration in preference modeling tasks.
- PreferenceEKF’s runtime is not only much faster, but also scales better with both model size and number of posterior samples compared to all other methods.
- When used for policy optimization, reward models learned using PreferenceEKF result in policies with performance competitive with those optimized using reward models learned with other methods.
Does PreferenceEKF Lead to Better Model Calibration?
PreferenceEKF has the lowest expected calibration error (ECE) among all methods, and the second-lowest Brier score behind active DeepEnsemble. This highlights the quality of the posterior approximation achieved by subspace inference methods compared to the other Bayesian deep learning baselines. Further calibration experiment details and reliability diagrams are provided in Section A.2.12.
Ablation Study on Subspace Construction
In the case where no initial dataset is available, belief initialization using the random projection approach is often sufficient for good reward-learning performance. This result decouples PreferenceEKF’s reliance on stochastic gradient descent altogether.
For consistency, unless otherwise stated, the main PreferenceEKF experiments use the singular value decomposition (SVD)-based approach that relies on stochastic gradient descent.
Can PreferenceEKF Learn from Image Data and Sparse Preference Feedback?
The main experiments showcase the effectiveness of PreferenceEKF in state-based control tasks, where preference labels are synthetically generated by comparing the sum of dense rewards between two trajectories.
The method is also applied to two additional challenging yet common settings:
- Sparse comparative feedback.
- Pixel-based control.
Broader Impacts
This work presents an algorithm for active learning in preference-based reward modeling, enhancing the efficiency and accuracy of neural network training in applications requiring subjective human evaluations, such as natural language processing, personalized recommendations, and human-robot interaction.
By optimizing data collection around uncertain or high-impact preferences, the approach can reduce labeling costs and improve model alignment with human intentions.
However, working in a reduced subspace and performing inference with the extended Kalman filter may introduce suboptimalities in preference modeling, such as bias amplification or neglect of minority preferences. To mitigate these risks, future research should investigate the robustness of PreferenceEKF and potential information loss caused by subspace reduction.
Acknowledgments
The authors thank Aleyna Kara for initial discussions about the method.
Yutai Zhou was partially supported by a fellowship from the USC–Capital One Center for Responsible AI and Decision Making in Finance (CREDIF). Erdem Bıyık acknowledges funding from the Airbus Institute for Engineering Research (AIER).
LLM Usage
LLMs were used primarily for writing Python visualization scripts, typesetting figures and tables in LaTeX, finding related work on subspace construction methods, and debugging JAX compilation and model-loading errors.
LLMs were not used for paper writing, research ideation, or implementing the core algorithm components.
Frequently Asked Questions
What is PreferenceEKF?
PreferenceEKF is a method that uses subspace filtering to train neural network reward models from preference feedback.
How does PreferenceEKF compare with other Bayesian deep learning methods?
It performs on par with or better than the evaluated Bayesian deep learning baselines in sample efficiency and preference-model calibration.
What calibration results does PreferenceEKF achieve?
PreferenceEKF has the lowest expected calibration error and the second-lowest Brier score among the evaluated methods.
Can PreferenceEKF work with sparse preferences and image data?
The method is applied to sparse comparative feedback and pixel-based control in addition to state-based control tasks.