Abstract

---

Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to maximize model capability gains? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap.

Real-World Post-Training

---

To validate Any3D-VLA's generalization capabilities on new tasks including specific rules and new language instructions, we design two challenging evaluation scenarios.

Notations: For the training dataset, Setting 2 incorporates both sensor-based and point clouds estimated by multiple models, while Setting 3 utilizes only the sensor-based point cloud. During inference, RealSense denotes the use of the sensor-based point cloud, whereas DA3 refers to the point cloud derived from Depth Anything 3 depth predictions.

Task 1: Grasp a flower and place it into a vase.

Task 2: Place a transparent condiment cup into a specific slot of a cup carrier.

Zero-Shot Comparisons in the Real World

---

To evaluate Any3D-VLA's zero-shot generalization ability and robustness in the real world, we design four challenging test sets.

Notations: For the training dataset, Setting 1 utilizes only the simulator-based point cloud, whereas Setting 2 incorporates both simulator-based and point clouds estimated by multiple models. During inference, RealSense denotes the use of the sensor-based point cloud, while DA3 refers to the point cloud derived from Depth Anything 3 depth predictions.

1. Standard

Relatively simple scenes, with no more than six objects on the tabletop, and target objects mostly of conventional shapes and scales.

2. Scale & Shape Challenge

Scenes with substantial intra-class variations in size and shape, e.g., dogs and bottles of different sizes and appearances; this set also includes geometrically challenging target objects, such as elongated objects (pen, fork, spoon, etc.) and small objects (diameter < 3cm, e.g., bottle cap).

3. Viewpoint Challenge

While keeping the coordinate-system origin fixed, we rotate the camera viewpoint around the z-axis (perpendicular to the tabletop) by 5°, 15°, and 30°, respectively.

4. Appearance-Deprived Challenge

Scenes designed to weaken informative 2D cues, including transparent objects, textureless objects (solid white, solid green, solid blue, etc.), and visual camouflage (objects with the same color as the tabletop), forcing the model to rely more on 3D geometry rather than 2D color and texture information.