Learning low-level robot control policies is challenging when task success signals are extremely sparse. Exploration may produce many unsuccessful trajectories before encountering a successful one, leaving little feedback to distinguish useful behavior or identify which actions matter. For example, a robot may need to coordinate several movements before completing an assembly, or strike an object so that it reaches a distant target. Ordering, timing, and simultaneous constraints can further restrict the set of successful trajectories.
For these sparse-completion tasks, informative feedback from unsuccessful trajectories is essential to practical learning efficiency; otherwise, policy improvement must wait for rare successful exploration. Signal Temporal Logic (STL) specifies logical and temporal requirements over real-valued signals. Its robustness semantics provide a quantitative measure of a trajectory’s margin to success instead of a binary signal.
To amortize this computation, prior work trains a feedback policy using STL robustness as a reward. However, this leaves temporal credit assignment to exploration, while differentiable predictive controllers still rely on online trajectory prediction and backup mechanisms. These limitations motivate learning low-level feedback policies by differentiating trajectory-level robustness, where unsuccessful rollouts are still valuable training signals.
The goal of this work is to use STL task specifications to provide improvement signals even in unsuccessful rollouts, and thereby train low-level robot control policies for tasks with sparse success signals. Empirically, the framework achieves the highest mean success among a set of reinforcement learning baselines with shaped dense rewards on five of six benchmarks. Policies trained with STL-SVPG in simulation also transfer zero-shot to physical hardware.
The contributions are:
- STL-SVPG, a specification-driven policy-learning framework that turns overall task requirements into actionable policy gradients for low-level feedback control, including from unsuccessful trajectories.
- A policy-optimization method that shares information among a population of policy parameters, combining task-directed improvements with population-based exploration.
- An investigation of how STL robustness gradients guide improvement and how population interactions expand exploration for better policies.
- Validation of simulation-only-trained STL-SVPG policies on two real-world tasks.
Related Work
Learning with sparse task-success signals can be assisted by shaped rewards, explicit task-progress representations, or quantitative specifications. The paper discusses existing work in each of these directions to contextualize its contribution.
Stein Variational Gradient Descent
The two portions of the update directions represent an attraction force pulling particles toward areas of higher expected score, with the kernel’s value enabling information sharing between particles, and a repulsion force that prevents mode collapse.
Experimental Setup
The experiments select sparse-success-signal tasks on two different embodiments. These tasks combine continuous control with requirements for ordered actions, event-triggered responses, and outcomes based on physical contact. The scenarios test whether the algorithm can learn successful physical behavior while satisfying the logical and temporal specifications that result in extremely sparse success signals.
The learning algorithm is tested mainly on the Crazyflie platform, with additional experiments showing the same algorithm generalizing to a Franka Emika Panda manipulator.
Robot Simulations
Crazyflie rigid-body dynamics are simulated in MuJoCo MJX, using a cascaded body-rate PID controller, first-order motor dynamics, and rotor-dependent aerodynamic drag. Actions in the simulation are normalized collective thrust and body-rate commands, expected at 100 Hz, while control and physics run at 500 Hz.
The Panda environment uses the seven-joint model for forward kinematics and bounded joint-velocity control provided by MuJoCo. Joint positions are integrated at 5 ms, and each action is held for five steps, giving a 25 ms control interval.
Tasks
The Crazyflie must stay in region A for 0.6 seconds to unlock, enter region Q within 2 seconds of unlocking, and then return home to region H within 10 seconds. Entering Q before activation is forbidden.
Safety requires staying within world bounds. The 2-second entry deadline starts at the end of the dwell.
Evaluation Protocol
Performance is measured by complete task success: a rollout succeeds when the full task specification has positive exact robustness. For each main benchmark, policies from six independent training seeds are evaluated on a shared set of 1,000 held-out initial conditions. The reported values are the mean success rate and sample standard deviation across training seeds.
Policy evaluation uses deterministic actions, while TD-MPC2 uses its evaluation-mode planner. The Branching study additionally evaluates the policy population to measure coverage of alternative successful routes.
Results
The experiments address the following questions:
- Can STL-SVPG learn policies for tasks with extremely sparse success signals?
- What does direct STL robustness optimization contribute compared with a dense reward?
- How much does maintaining a policy population contribute?
- Does STL-SVPG learn multi-modal solutions to the task?
- Does a policy learned in simulation transfer to real robots?
Diverse Successful Policy Populations
The Branching task is used to visualize whether the proposed algorithm learns multiple solutions. STL-SVPG learns a diverse population of successful solutions, retaining both Branching routes in every one of six runs.
On 128 held-out initial conditions, the mean success rate of the final chosen policy is 99.65%. Inspecting all policies in the population shows an average split of 34 A–C and 30 B–D policies out of 64 particles. Thus, high task success coexists with distinct, consistently executed strategies.
Sim-to-Real Transfer on Crazyflie
Crazyflie Random Cue and Ping-pong are tested on real hardware. Because Ping-pong requires physical contact with the ball, a 3D-printed cage is attached around the brushless version of the Crazyflie.
New deployment policies are trained using updated scene and drone-cage geometries and brushless motor dynamics. For Ping-pong, task success is also changed so that the ball only counts as successful when it enters the target through the top face, simulating a box. These changes significantly decrease simulated success in Ping-pong because of more unpredictable contact physics and a stricter success requirement. A launcher throws the ball, and its trajectory is tracked with a RealSense depth camera.
Acknowledgment
OpenAI Codex was used to improve clarity in non-technical sections of the manuscript and to assist with implementing experimental code and plotting. All technical content, experimental design, results, and conclusions were developed, verified, and carefully vetted by the authors.
Frequently Asked Questions
What is STL-SVPG designed to do? It uses Signal Temporal Logic task specifications to produce policy gradients for low-level robot control, including from unsuccessful trajectories.
How does STL robustness help with sparse success signals? STL robustness gives a quantitative measure of a trajectory’s margin to success rather than only a binary success signal.
Which robot platforms are included in the experiments? The experiments mainly use the Crazyflie and also include a Franka Emika Panda manipulator.
What does the Branching-task population study show? Across six runs, the policy population retains both routes, with an average split of 34 A–C and 30 B–D policies out of 64.