We scale feed-forward 3D object articulation by using vision-language models to pseudo-label synthetic 3D assets, enabling a transformer-based architecture that outputs part segmentation and joint motion parameters in a single forward pass. Our method supports kinematic prompting for direct control over articulation parameters and is robust to AI-generated meshes, facilitating image-conditioned articulated 3D object generation for simulation and embodied AI training.
Pseudo-Labeling 3D Articulated Parts with Vision-Language Models
We label AI-generated assets from HY3D-Bench-Synthetic using this pipeline, excluding objects that are either fully rigid or exhibit soft, non-rigid deformations. This yields thousands of synthetic 3D assets spanning dozens of categories.
Augmenting Part-Segmented 3D Models
When selecting assets from HY3D-Bench-Part-Level, we filter out assets with more than a certain threshold of visible parts, as dense segmentation maps make it difficult for the VLM to reliably distinguish colors and assign accurate captions. We caption the remaining thousands of objects and include them in training.
Attention Mechanism
The query tokens are meant to infer the association of each query surface point to the corresponding part. They are updated by cross-attending the shape tokens to obtain information on the object shape, and to the (updated) part tokens to obtain information on the parts. Crucially, we do not use self-attention between query tokens, for speed, and to avoid making the result dependent on the number of query points decoded together.
Decoders
Once the final versions of the part and query tokens are obtained, decoder heads extract the part segmentation and joint motion parameters.
Articulated 3D Object Generation and Kinematic Prompting
While our model takes existing 3D assets as input, it is robust to AI-generated meshes, and thus enables image-conditioned articulated 3D object generation via an off-the-shelf 3D generator. This pipeline can support the creation of diverse simulation assets for embodied AI training.
Failure Cases and Limitations
While our model is generally robust to AI-generated meshes, segmentation artifacts can still occur, such as on the right knob of the CD player. Because each joint axis is estimated by aggregating votes from query points on the predicted part, such local errors can propagate to joint-motion estimation. Artifacts in the generated meshes can also degrade prediction quality, as shown by the floating component in the CD player and the missing part separation in the refrigerator.
Frequently Asked Questions
What types of 3D assets does Instruct-Particulate support? The model supports both existing 3D assets and AI-generated meshes, making it suitable for image-conditioned articulated 3D object generation via off-the-shelf 3D generators.
How does the model avoid performance issues with dense segmentation maps? Assets with too many visible parts are filtered out during training, as dense segmentation maps make it difficult for the vision-language model to reliably distinguish colors and assign accurate captions.
Why does the model avoid self-attention between query tokens? Self-attention between query tokens is omitted for speed and to prevent the result from depending on the number of query points decoded together.
What are the main failure modes of Instruct-Particulate? Segmentation artifacts in AI-generated meshes can cause local errors that propagate to joint-motion estimation, and artifacts in the generated meshes themselves can degrade prediction quality overall.