Related Work
STEP is inspired by research on shared autonomy and long-term action anticipation using large language models. It addresses some limitations of prior work by extending methods that track a state representation of the environment for long-term action anticipation.
Metrics
Executability
Executability ensures that the predicted actions follow a logical order and satisfy execution constraints, such as proper object poses and action preconditions. Checks are performed for executability, and the predicted action sequence is truncated at the first action that fails the checks.
Executability is reported as the percentage of predicted actions that pass all checks.
Final-State Error
Final-state error measures the error between the ground-truth task-completion state and the final state that can be successfully achieved by executing the predicted actions.
For the final achievable state, the resulting state from the last predicted action that can be executed successfully is selected. This metric measures the difference between the result and the expected goal.
Task Correctness
Task correctness measures the accuracy of task estimation. It is assigned 1 if the predicted task matches any of the possible tasks at inference time, and 0 otherwise.
Current State Error
Current state error compares the distance between the current state output of state-representation generation and the ground-truth state at the time of inference.
Results and Discussion
The GPT-4o model offered through the OpenAI API was used for the main results. Each time the multimodal large language model was prompted, multiple outputs were sampled, and the output with the maximum log-probability returned by the model was selected.
Comparison with the Baseline
To model variations in user delegations to STEP, experiments were conducted at task-completion percentages of 30%, 50%, 70%, and 90%. A task completion of a given percentage signifies that the initial corresponding percentage of actions from the recorded action sequence was used as input for evaluation.
With increasing task-completion percentage, both methods achieved a better final error because the robot started from a state closer to the goal state as more of the assembly was completed by the operator.
With increasing task completion, task correctness first declined because of increasing noise in longer input action sequences and more occlusion of the block structures in the images. It then increased as the structure being built became more discernible.
The current state error appeared to increase as the task progressed because the structure became more complicated and the blocks became closer together in the image.
Conclusion
STEP is introduced as a method to enhance long-term action anticipation with multimodal large language models by explicitly modeling state transitions. This leads to more accurate and executable robot action plans in collaborative industrial tasks.
The method uses a multistage pipeline. It first estimates the overall task that a robot operator is trying to perform, together with a structured representation of the current state of the robot’s workstation. This information is then passed to later stages to predict the actions required to complete the task and propagate the state representation to obtain the assistance parameters required to successfully execute the predicted actions for the desired task.
STEP also tracks the distance between propagated states and the goal state and guides predicted actions toward the goal.
Additional work is required to make STEP operate in real time before it can be adopted for practical use. To propagate states and guide the generation of action plans toward the goal, STEP queries a multimodal large language model several more times than the baseline. This causes STEP to trade off speed for performance accuracy.
In the experiments, the baseline had a shorter runtime, with a mean of 18.24 seconds and a standard deviation of 4.23 seconds, than STEP, which had a mean of 148.07 seconds and a standard deviation of 55.03 seconds.
Future research could investigate hosting a multimodal large language model locally or fine-tuning a smaller model to reduce the latency of each query. This work encourages further investigation into the potential of multimodal large language models for long-horizon reasoning in human-robot collaboration.
Frequently Asked Questions
What is STEP designed to improve?
STEP is designed to improve long-term action anticipation with multimodal large language models for collaborative industrial tasks.
How does STEP represent the robot’s situation?
STEP estimates the operator’s overall task and generates a structured representation of the current state of the robot’s workstation.
What happens to task correctness as task completion increases?
Task correctness first declines because of noise and image occlusion, then increases as the structure being built becomes more discernible.
How does STEP’s runtime compare with the baseline?
The baseline had a mean runtime of 18.24 seconds, while STEP had a mean runtime of 148.07 seconds.