We introduce AFUN, an affordance foundation model that learns a unified representation of object functionality from visual data. Our model leverages a frozen vision-language model (VLM) with learnable MetaQuery tokens to extract task-relevant features, which are then used to predict affordance segmentation masks and future motion trajectories. We propose a three-stage training scheme that progressively aligns the MetaQuery interface with a segmentation foundation model, learns reliable task-conditioned affordance segmentation, and finally fine-tunes motion prediction. The model is trained on a new large-scale affordance dataset spanning multiple robot interaction domains. Experimental results demonstrate that AFUN generalizes across diverse manipulation tasks and outperforms prior affordance detection methods in both segmentation accuracy and motion prediction quality.
Network Architecture
The MetaQuery approach serves as an interface to connect a frozen VLM with downstream models. A small set of learnable special tokens is appended to the VLM's input prompt and processed through the transformer. The hidden states in the final layer of the VLM serve as a compact conditioning feature for the downstream model. This approach can extract detailed visual conditions and transfer reasoning capabilities to multimodal generation tasks such as image editing.
Training Scheme
Directly training the full model is unstable: randomly initialized MetaQuery tokens provide a poor conditioning signal for the segmentation model, and noisy mask predictions would in turn make motion supervision ambiguous. We therefore train our model in three stages:
- Stage I: Aligning the MetaQuery interface with the segmentation foundation model
- Stage II: Learning reliable task-conditioned affordance segmentation
- Stage III: Fine-tuning motion prediction when the model is already robust in segmentation prediction
The pretrained priors — including Qwen-VL, the segmentation model, and the motion prediction model — are kept frozen throughout the training.
Cross-Dataset Preprocessing
Our data preprocessing pipeline is built around source-specific adapters that share the same processing logic and structure. Every adapter writes the same interval-level schema: observation/contact frames, RGB-D data, task language, camera calibration, and the video span. What differs is how these fields are recovered from each raw dataset. For example, the AgiBotWorld-Beta dataset provides action-interval annotations, and we use each annotated action as one interval. We load the paired head RGB/depth frames for the interval and use the calibrated non-fisheye head camera.
Gallery on the Affordance Dataset
We sampled 48 entries from the AFUN dataset and visualize each as the observation frame with the affordance mask and the Bézier spline curve fitted 3D trajectory. The language instruction for each sample is shown directly below its image.
Converting Other Datasets for Segmentation Training
The part-level prompt is used only to obtain the mask; it is not used as the training query, since directly naming the contacted part would leak the answer. We therefore run a second query pass on the original image and the selected-mask overlay. This pass rewrites the sample into a natural task-level instruction that implies the affordance without naming the highlighted region explicitly.
Frequently Asked Questions
How does AFUN handle training instability caused by randomly initialized MetaQuery tokens? The model uses a three-stage training scheme that progressively aligns MetaQuery tokens with the segmentation model before learning task-conditioned affordances and then fine-tuning motion prediction.
What pretrained models are kept frozen during AFUN training? The Qwen-VL vision-language model, the SAM3 segmentation model, and the Sonata motion prediction model remain frozen throughout all training stages.
How does the cross-dataset preprocessing pipeline ensure consistency across different datasets? Source-specific adapters share the same processing logic and write data into a common interval-level schema containing observation/contact frames, RGB-D data, task language, camera calibration, and video span information.
Why is a second query pass needed when converting datasets for segmentation training? The second pass rewrites the sample into a natural task-level instruction that implies the affordance without explicitly naming the highlighted region, preventing the model from leaking the answer during training.