PAC-ACT Uses Reinforcement Learning to Sharpen Pretrained Robot Policies

Researchers from Wuhan University introduce PAC-ACT, a reinforcement learning (RL) post-training framework that fine-tunes pretrained Action Chunking Transformer (ACT) policies to dramatically improve task success and force safety. By adapting the original ACT architecture into an Actor-Critic structure with hybrid KL constraints, the method boosts a 60% Contour success rate to 100% and reduces peak contact forces by over 98% — all while preserving the low-latency, chunked action generation that makes ACT appealing for industrial deployment.

What the Researchers Built

PAC-ACT is a post-training framework that applies reinforcement learning to an already-pretrained Action Chunking Transformer policy. The original ACT is a behavior cloning method that generates fixed-length sequences of future actions (chunks) in a single forward pass, enabling smooth, low-latency control. But like all imitation-learned policies, it can struggle in out-of-distribution states or produce unsafe forces.

PAC-ACT addresses this by removing the CVAE (Conditional Variational Autoencoder) latent module from the pretrained ACT backbone — simplifying the architecture for RL optimization — and constructing an Actor-Critic network from the remaining encoder and decoder. A hybrid KL divergence constraint keeps the fine-tuned policy close to the original behavior prior, preventing catastrophic forgetting. The researchers also reformulate the Markov Decision Process (MDP) to resolve the mismatch between step-wise RL and chunk-level action generation, so the RL agent can learn from sparse or dense rewards without breaking the temporal structure of movements.

The experiments focus on two precision contact tasks: Metal Touch Contour (tracing a path without exceeding force limits) and Square Assembly (inserting a peg into a square hole). These tasks demand both high success rates and strict force safety, making them ideal for testing the tradeoff between performance and safety.

Key Results

The numbers speak for themselves. On the Contour task, pretrained ACT achieves a 60% success rate. After PAC-ACT fine-tuning, success jumps to 100%. On Square Assembly, the improvement is from 51.2% to 98.2%. For comparison, a 3B-parameter VLA foundation model (π0.5) achieves only 79% on Contour and 62.6% on Square Assembly, while consuming 44.15 GB of GPU memory — making it impractical for edge deployment.

Beyond success rates, the force-safety improvements are striking. The peak force dropped from 8,452.5 N to 120.9 N — a reduction of over 98%. Since such extreme forces occur only in a minority of failure episodes, they can easily be missed in standard behavior cloning evaluation. PAC-ACT systematically eliminates these hidden hazards.

How It Works

PAC-ACT builds on the LeRobot implementation of ACT and follows a three-step pipeline:

1. Architecture conversion. The pretrained ACT model (encoder + CVAE + decoder) is modified by removing the CVAE latent module. The encoder and decoder are repurposed as the Actor network. A separate Critic (value function) network is added, sharing the same encoder but with its own MLP head. This design preserves the fast, chunked generation of the original policy while enabling RL training.

1. MDP reformulation. Standard RL treats each timestep as a state-action pair, but ACT outputs chunks of future actions (e.g., 16 steps). PAC-ACT reformulates the MDP so that the RL agent chooses which chunk to execute at each time step. This resolves the structural mismatch: the policy still generates chunks, but the RL reward is computed over the entire chunk, and the critic evaluates the value of being in a given state before executing the chunk.

1. Hybrid KL constraint. To prevent the fine-tuned policy from deviating too far from the pretrained behavior, two KL divergence constraints are applied. The first keeps the Actor close to the original frozen policy, and the second regularizes the latent representation. These constraints are critical when rewards are sparse — they ensure that the policy doesn't collapse into erratic or unsafe movements.

The training environment uses MuJoCo with two RGB cameras (front and wrist views at 512×512) and joint-angle encodings. Force/torque readings are used only for reward computation and safety evaluation, not as policy inputs. The control frequency is 10 Hz, and each RL episode runs up to 512 steps (~51 seconds). During validation, the maximum duration is 600 steps, with early termination upon successful completion.

Why This Matters for Robotics

PAC-ACT demonstrates a practical path to customize pretrained robot policies using RL without starting from scratch. For industrial applications — where a single task might require thousands of demonstrations — the ability to fine-tune an existing policy with a moderate number of RL episodes can save weeks of data collection and training time.

The force-safety improvements are especially relevant for manufacturing and assembly robots that must work near humans or delicate parts. A 98.6% reduction in peak forces means fewer damaged components, lower maintenance costs, and safer human-robot collaboration. The method also preserves the low-latency inference of ACT (approximately 0.01 seconds for an 80M-parameter model on a single RTX 2080 Ti), making it suitable for real-time deployment on edge hardware.

For warehouse and logistics applications, where robots perform repetitive but variable tasks like bin picking or kitting, this approach could enable quick retraining for new product geometries. Similarly, humanoid robots and used cobots that rely on imitation learning could benefit from RL-based fine-tuning to improve robustness in unstructured environments.

Limitations and Open Questions

The current experiments are conducted entirely in simulation. Sim-to-real transfer of the fine-tuned policy has not yet been validated, so real-world performance may vary due to sensor noise, calibration errors, or dynamics mismatches. The robustness evaluation is also limited: it covers only object initial-position perturbations and multi-trajectory generalization, not visual changes like lighting variation or physical perturbations like dynamic parameter shifts.

Removing the CVAE module simplifies RL optimization but may limit the policy's ability to represent highly multi-modal action distributions — a known strength of the original ACT. Future work could explore regularized CVAE integration that preserves multimodality while maintaining training stability. Additionally, the framework still requires task-specific reward design, which is an engineering burden. More automated reward specification methods would make this approach more accessible to practitioners.

Frequently Asked Questions

What is an Action Chunking Transformer (ACT)? ACT is a behavior cloning method that generates a fixed-length sequence of future actions in a single forward pass, enabling smooth, low-latency robot control from demonstration data.

How does PAC-ACT differ from other RL fine-tuning approaches? PAC-ACT specifically targets pretrained chunking policies, reformulating the MDP to align step-wise RL with chunk-level action generation and using hybrid KL constraints to preserve the original behavior prior.

What tasks were used to evaluate PAC-ACT? The method was tested on two MuJoCo-simulated tasks: Metal Touch Contour (path tracing with force limits) and Square Assembly (peg-in-hole insertion).

Does PAC-ACT require designing a reward function for each task? Yes, the current framework relies on task-specific dense rewards for best performance, though sparse rewards can still work when combined with the behavior-prior constraints.

Conclusion

PAC-ACT shows that a relatively lightweight RL post-training step can transform a decent imitation-learned policy into an exceptional one, doubling success rates and nearly eliminating dangerous force spikes. For industrial robotics where both reliability and safety are paramount, this approach offers a practical way to upgrade existing ACT-based deployments without retraining from scratch.