Simulating the Robotic Tree Pruning Task
To support visuomotor policy learning, we construct a simulator for robotic tree pruning that mimics the geometry of an orchard scene. As our policy relies on optical flow rather than image texture, photorealism is not required. Our goal is to generate diverse pruning scenarios with variation in cutpoint locations, branch orientations, and lighting, while maintaining structural similarity to modern orchard training systems.
Learning Setup for Visuomotor Pruning
Achieving robust real-world deployment requires exposing the policy to conditions it would face in the real world. We first define the task space and introduce a sampling procedure to generate sufficiently diverse cutpoints. Next, we specify the state and action representations used for learning and deployment. Finally, using the simulation environment with access to complete state information, we outline our method for automated collection of successful trajectories.
Generating Diverse Training Episodes
An episode consists of a simulation scene with a sampled cutpoint defined on one of the tertiary branches. At the start of a new episode, the lighting, robot, and camera placement are randomized as outlined in Section 3.
Task Space
We define a likely reachable region for the UR5e robot that includes all points 75 cm to 105 cm from the robot base, excluding points behind the vertical plane of the robot’s base. This region is defined heuristically based on the robot’s maximum reach with the cutters (125 cm), the maximum outward-facing branch length (40 cm), and the visibility of the tree for obstacle avoidance. Positions closer than 30 cm yielded poor views.
This region would allow the robot to reach pruning points on up to four trellis wires when placed in front of the tree in an orchard.
Observation and Action Spaces
While the simulator provides full information about tree geometry and pose, the policy observation space is limited to:
- A goal that specifies the cutpoint.
- Optical-flow imagery.
- Robot proprioception.
A summary of the state space is shown below.
Proprioception
We include the six-degree-of-freedom pose, the end-effector’s current velocity, and the robot’s joint angles in the state space. The end-effector’s pose is defined by its location as Cartesian position coordinates and its orientation by a six-dimensional parameterization. The joint angles are represented using their sine and cosine values. This results in a 27-dimensional proprioception state.
Actions
The policy outputs linear and angular velocities in the end-effector frame. Joint velocities are then computed using the damped least-squares controller and applied for 0.5 seconds, corresponding to a control frequency of 2 Hz.
Policy Architecture
We consider an LSTM-based actor–critic architecture with the addition of a privileged critic that has access to training-time-only features and an encoder for visual feature learning.
Privileged Critic
The critic shares the same image encoder as the actor and has an identical architecture, except that:
- It outputs a single scalar.
- It receives two additional scalar features corresponding to the pointing cosine similarity and perpendicular cosine similarity between the end-effector and the branch to be pruned.
These inputs rely on privileged knowledge of the tree geometry and pose, but they are not available to the policy and are used only during training.
Generating Synthetic Data
Collecting a large dataset of successful pruning trajectories manually—on the order of thousands—would be prohibitively expensive because of the human effort required. To address this, we automate the generation process by leveraging the simulation environment with access to perfect state information.
Specifically, we use RRT-Connect, a sampling-based motion-planning method that produces waypoints satisfying the success criterion for pruning. These waypoints are then post-processed into smoother trajectories and converted into transition tuples comprising observation, action, reward, and next observation, making them compatible with the reinforcement-learning setup.
Generating Goal Configurations
Each pruning point can have multiple end-effector poses satisfying the success criterion based on pointing direction and the tolerances present in reaching, pointing, and perpendicularity requirements. While some of these poses are reachable through collision-free trajectories, others are infeasible because of collisions with surrounding orchard clutter. To identify feasible solutions, we generate a set of end-effector configurations that satisfy the pruning criterion and use RRT-Connect to compute collision-free trajectories to these poses.
To sample valid end-effector poses, we begin with the ideal end-effector pose: one that is perfectly aligned to the branch to be pruned, is closest in orientation to the initial end-effector frame, and satisfies the success criterion.
Next, the resulting pose is perturbed randomly around the pointing and perpendicular vectors. These perturbations are constrained within task-defined thresholds, ensuring that all sampled goal poses fall within the acceptable range for successful pruning. Finally, each pose is converted into a robot configuration using inverse kinematics so it can be passed to RRT-Connect.
Generating Trajectories
For each pruning point, we run RRT-Connect by sampling goal configurations until a valid trajectory is found or until 100 attempts have been made, after which the point is discarded. Directly sampling goal poses on the branch surface often caused the planner to fail because of proximity to collisions. To mitigate this, we offset the goal pose by 5 cm along the branch normal while keeping the end-effector oriented toward the branch. Once a valid path is found, the final goal pose is appended as the last configuration.
RRT-Connect finds feasible paths composed of discrete robot-configuration waypoints. These paths are not smooth and typically require post-processing. We apply shortcutting, a post-processing technique that samples random points along the path and attempts to connect them directly to form shorter, smoother, yet feasible paths.
The resulting smoothed paths are then subsampled every 5 cm to create dense waypoint sequences. These dense waypoints are converted into trajectories by computing end-effector velocity commands at a control frequency of 2 Hz. For each pair of consecutive waypoints, a velocity vector is computed and scaled to ensure that the end-effector reaches the next point within the fixed control interval.
Creating the Dataset
To generate transition tuples comprising observation, action, reward, and next observation, we replay the velocity-action trajectories in simulation and record transitions at each time step. Only trajectories that end in successful configurations are retained.
To construct the trajectory-level dataset, we uniformly sample pruning points across the workspace, as described in the task-space section. In total, we generate 5,000 unique pruning points across 1,000 different trees and apply the synthetic-data generation pipeline for each of UFO and V-Trellis.
After filtering unsuccessful attempts, the final dataset contains:
| Architecture | Successful pruning trajectories |
|---|---|
| V-Trellis | 4,282 |
| UFO | 4,424 |
Learning a Visuomotor Pruning Policy
Proximal Policy Optimization (PPO) has been used successfully to train visuomotor pruning policies. Recent work has shown that combining reinforcement learning with an offline dataset of successful trajectories can improve performance compared with using reinforcement learning alone. Motivated by these findings, we augment the learning process using a synthetically generated dataset of successful trajectories.
The synthetic-data generation pipeline uses the full environment state available in simulation, while the learned policy operates under partial observability. As a result, directly imitating such trajectories can lead to suboptimal actions; for instance, the policy may begin rotating the end-effector before the relevant branch is visible.
Directly applying behavior cloning to this dataset reproduces such suboptimal behaviors. Our goal, therefore, is to develop an algorithm that can stitch together offline and online trajectories while efficiently moving away from poor actions present in the offline data.
Methods such as PPO with behavior cloning can partially mitigate this issue by adding a behavioral-cloning loss to online updates, but they do not explicitly reject suboptimal actions. The competing losses can degrade performance, as demonstrated by hybrid reinforcement-learning algorithms often outperforming baselines that add a behavioral-cloning loss.
We propose a Hybrid Proximal Policy Optimization method that incorporates per-decision importance sampling to correct for off-policy data and integrate it into the PPO framework. This allows us to leverage simulated rollouts and offline data concurrently.
This advantage-weighted formulation enables the critic to reweight offline actions. By assigning lower advantage estimates to suboptimal actions, the policy avoids favoring these actions in the offline dataset.
The algorithm is compared against an online-only PPO baseline, an offline-only behavior-cloning baseline, and a simple PPO-plus-behavior-cloning baseline on the pruning task.
Problem Definition
The learned policy is a squashed Gaussian policy parameterized by a neural network. Its mean is predicted conditioned on the policy observation, and the output is passed through a squashing function. The standard deviation is state-independent and is a learned parameter.
Deployment
For experiments, high-level decision-making and task sequencing were implemented using a behavior-tree architecture built with PyTreesROS. User interaction and manual overrides were provided through a PlayStation controller integrated with the behavior tree for ease of operation during field tests.
Field Setup
For the physical evaluations, target pruning points were constrained to the bottom three trellis wires in the commercial UFO and V-Trellis orchards. The laboratory setup used only the bottom two wires.
Experimental Setup
We design our experimental evaluation to answer four core questions.
Algorithmic Validation
How does the proposed Hybrid-PPO algorithm compare against behavior cloning, standard PPO, and PPO-plus-behavior-cloning baselines on simulated V-Trellis structures?
Policy Evaluation Across Architectures in Simulation
In exhaustive and diverse simulation trials, what is the overall performance of the learned policy across both V-Trellis and Upright Fruiting Offshoots (UFO) canopy architectures?
Real-World Sim-to-Real Transfer
Can the trained policies transfer zero-shot to physical commercial V-Trellis and experimental UFO orchards under operational field conditions?
RRT-Connect Baseline Comparison
How does the reinforcement-learning policy perform against a classical RRT-Connect motion-planning baseline on physical hardware in a V-Trellis tree in a laboratory setting?
Policy Evaluation Across Architectures in Simulation
To evaluate the performance of our trained policy within our workspace, we conduct exhaustive simulation trials with pruning points uniformly sampled across both location and orientation for V-Trellis and UFO architectures.
We also run an oracle RRT-Connect planner equipped with perfect environment information and a large computational budget to establish the upper bound on success rate. These comprehensive simulation experiments allow us to perform exhaustive evaluation across all possible branch configurations that would be logistically prohibitive and pose hardware risks to evaluate in physical field trials.
Obtaining precise, noise-free three-dimensional meshes in the real world is extremely difficult, making this setup unrealistic for practical field applications. However, this rigorous evaluation establishes the absolute best-case performance limit for a classical planner given perfect perception.
Reinforcement-Learning Policy
We execute the learned reinforcement-learning policy by performing the most likely action predicted by the network. For each cutpoint, similarly to training, the policy is run for 100 simulation steps at 2 Hz or until the termination condition is reached.
Real-World Sim-to-Real Transfer
To validate the sim-to-real transfer of our learned policies, we evaluate our system across the reported number of real-world trials split between two settings:
- The reported number of controlled laboratory trials on a constructed V-Trellis setup.
- The reported number of outdoor field trials across commercial V-Trellis apple and experimental UFO cherry orchards under operational field conditions.
The orchards where the experiments were performed are located in Prosser, Washington, USA. The outdoor field trials were conducted under diverse environmental and atmospheric conditions, including overcast skies, fog, mild rain, partly cloudy conditions, and direct bright sunlight.
Because of the distinct geometries of the V-Trellis and UFO architectures, separate policies were trained and deployed for each. Whether joint training could leverage shared structural representations across canopy types remains to be investigated.
Results
This section presents the findings addressing each of the four research questions.
Algorithmic Validation
Hybrid-PPO achieves higher mean success rates compared with behavior cloning, standard PPO, and PPO-plus-behavior-cloning baselines.
Policy Evaluation Across Architectures in Simulation
In exhaustive simulation trials, the policy achieves the reported success rate on V-Trellis and the reported success rate on UFO architectures, improving upon prior benchmark performance.
Real-World Sim-to-Real Transfer
The learned policies successfully execute zero-shot across the reported number of field trials and the reported number of laboratory trials under variable outdoor environmental conditions.
RRT-Connect Baseline Comparison
On physical hardware, the reinforcement-learning controller outperforms a classical RRT-Connect planner using reconstructed point clouds in overall success rate.
The following subsection details these findings.
Algorithmic Validation
Training reward curves for each method across three random seeds show that standard PPO exhibits noticeably higher variance across seeds. Hybrid-PPO and PPO-plus-behavior-cloning consistently achieve higher mean rewards, despite overlapping reward distributions across methods.
Because raw reward magnitude does not always correlate directly with task execution—often because of misaligned reward shaping or potential reward hacking—we ground our primary evaluation on task success rates, which provide a direct measure of physical pruning success.
Frequently Asked Questions
What robot is used for the pruning task? The task uses a UR5e robot equipped with pruning cutters.
What does the policy observe? The policy observes a cutpoint goal, optical-flow imagery, and robot proprioception.
How is synthetic training data generated? RRT-Connect generates collision-free trajectories from sampled end-effector configurations, which are replayed in simulation to create transition tuples.
Why does the method use a privileged critic? The privileged critic uses tree-geometry features available only during training to reweight offline actions while keeping the deployed policy partially observable.