Sliding-Window Inference and De-Duplication
At the end of each window, the feature cloud is voxelized based on predicted target positions. Unlike the per-frame voxelization used during encoding, this operation merges points across multiple origin frames.
This cross-frame merging is valid because all points are now tracked to the same target timestep. The method therefore leverages the rule that two entities cannot occupy the same voxel at the same time.
Implementation Details
The predicted target positions and trajectories are supervised with an L2 loss against 3D ground truth. Visibility and static/dynamic classification are each supervised with cross-entropy loss.
Because the training datasets provide ground-truth tracks for only a subset of points, losses are computed only over the labelled points. To save compute, full trajectories are decoded for a subset of dynamic points during training.
Experiments
TrackEverything is evaluated on standard 3D point-tracking benchmarks and compared with both sparse and dense tracking baselines. Beyond benchmarking the model, the evaluation aims to provide a comprehensive comparison across method paradigms and elucidate the trade-offs in this space.
Conclusion
This material is based upon work supported in part by an NSF Career award, ONR award N00014-23-1-2415, and AFOSR Grant FA9550-23-1-0257. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Ayush Jain was a summer intern at Meta for the initial duration of the project and is supported in part also by the Meta AI Mentorship Fellowship. The authors thank Yash Jangir, Jay Karhade, Nikhil Keetha, Ananya Bal, Gabriel Sarch, Naitik Khandelwal, and Brian Yang for their helpful discussions and support.
Additional Method and Implementation Details
Two components of the model are visually illustrated: sliding-window de-duplication and 3D WAFT.
Fixes to the PointOdyssey Dataset
During training runs, significant loss spikes were observed and traced back to the PointOdyssey dataset. Several critical annotation errors were identified, and a programmatic correction pipeline was developed.
Temporally Incoherent Sequences
The sequence scene_recording_20210910_S05_S06_0_ego1 exhibits sub-frame camera jitter that cannot be resolved by an integer frame shift. It is excluded from the training and evaluation splits.
Object segmentation masks were also briefly explored as a way to remove incorrect point-track labels. The approach relied on the fact that a point born within an object's segmentation mask should always remain within that mask.
However, the segmentation mask also contained bugs, making this approach infeasible.
The corrections to the PointOdyssey dataset will be released with the codebase. These fixes reduced loss spikes, although some noisy points remain in the dataset and could not be corrected.
Complexity Analysis
The complexity analysis was repeated using the APD-P metric instead of APD-M. The results show that all conclusions remain similar.
Frequently Asked Questions
What does sliding-window de-duplication do? It voxelizes the feature cloud using predicted target positions and merges points across multiple origin frames.
Why is cross-frame merging valid? All points are tracked to the same target timestep, where two entities cannot occupy the same voxel simultaneously.
Which losses supervise TrackEverything? Predicted target positions and trajectories use an L2 loss, while visibility and static/dynamic classification use cross-entropy losses.
What issue was found in the PointOdyssey dataset? The dataset contained annotation errors, including a sequence with unresolvable sub-frame camera jitter and additional noisy points.