Query-Conditioned Articulation Estimation from a Single Image

QueryArt is a model designed to estimate kinematic parameters of articulated objects from only a single RGB image and a 2D query point, which enables robots to infer object structure before physical interaction. The core contribution is decoupling where to interact from how the selected point moves by predicting articulation parameters in units of the query point's depth, thus making the target invariant to uniform scaling and allowing for metric recovery using a single depth measurement.

How it works

The model estimates three primary articulation parameters: the joint type (static, revolute, or prismatic), the 3D motion axis direction, and for revolute joints, the 3D offset of that axis relative to the query point. The formulation is designed to predict a joint type from a single calibrated RGB image and a pixel coordinate query. For moving parts, it predicts a unit direction vector (axis) and an offset vector.

Model Architecture

QueryArt employs an architecture that combines high-resolution frozen visual foundation features with lightweight prediction heads. Specifically, it utilizes a frozen DINOv3 ViT-B/16 image encoder to produce 768-channel patch features. These features are fused using a three-scale feature pyramid, downsampling the fused features to resolutions S, S/2, and S/4. A query embedding is constructed by combining three signals: a Fourier positional encoding of the normalized pixel query, a bilinear sample from the final DINOv3 block feature at the query location (a 768-channel descriptor), and a hand-crafted calibration vector derived from camera intrinsics. This combined signal forms the Query embedding, which is then added to two randomly initialized learned task tokens: a TYPE token and a LINE token. These tokens pass through two decoder layers where they undergo deformable cross-attention to the three-scale image pyramid, followed by self-attention between the TYPE and LINE tokens, and finally a token-wise feedforward network.

Articulation Heads

The model utilizes four distinct output heads. A single MLP classifier reads the TYPE token to predict the joint type, denoted as p(j I, q, K). Two independent MLPs read the LINE token to produce raw vectors for revolute and prismatic directions, vˆrev and vˆpris, respectively. These raw vectors are normalized to yield aˆrev and aˆpris. For revolute joints specifically, the fourth head predicts the axis offset relative to the query point. To ensure this prediction is invariant to axis sign, an outer-product embedding ψ(a) is used for the axis direction vector aˆrev. A separate MLP projects both the LINE token and ψ(aˆrev) into 64 dimensions, which are summed and passed through a linear layer to produce a raw vector vr. The final offset prediction is then enforced via the equation: ˆr = I3 − aˆrevaˆ⊤ rev vr, where I3 is the identity matrix. This projection guarantees that the resulting offset vector is orthogonal to the predicted axis direction and points from the unprojected query point to the nearest point on the predicted axis line.

Loss Functions

The model is trained using a composite loss function L = Ltype + 2Lrev axis + 2Lpris axis + 2Loffset. The joint type prediction is supervised using a three-class cross-entropy loss (Ltype). For each moving type, an unsigned cosine loss supervises the axis direction vector (Laxis). For the revolute joint, the normalized offset target r∗ is defined as r∗ = I3 − a∗a∗⊤ o∗ zq − p˜. This target is designed to be invariant to reversing a star direction, shifting o star along its axis, and jointly scaling the metric scene and query depth. The predicted offset ˆr is supervised using Smooth L1 loss applied to the Euclidean prediction error: Loffset = SmoothL1β(∥ˆr − r∗∥2), with β = 0.02. Eligibility for geometric supervision depends on ground-truth labels, not the predicted type, ensuring that incorrect type predictions do not suppress geometric learning.

Evaluation and Results

QueryArt was trained on a curated mixture of synthetic and real-world articulation datasets, including Articulate3D [9], SceneFun3D [38], MultiScan [37], PartNet-Mobility [39], and HSSD [40]. Evaluation metrics include axis error in degrees, type recall in %, depth error (dh), RMSE in meters, and the overall success rate (S). On held-out splits of four training datasets and two excluded out-of-distribution datasets (Arti4D [10] and HOI! [41]), QueryArt outperformed recent single-image baselines on most articulation metrics. In real-robot experiments on a mobile manipulator, QueryArt achieved a 70.

Here are the specific improvements for AI systems based on the QueryArt paper, along with what those improved systems can achieve:

1. The core improvement is a shift from coupled detection and segmentation methods to a decoupled query-conditioned regression formulation for articulation estimation. This decouples localization errors from kinematic prediction errors.

2. This system can estimate the full 3D kinematic parameters (joint type, motion axis direction, and query-relative axis location) of an articulated object solely from a single RGB image and a 2D query point, without requiring prior scene exploration or multi-view observations.

3. The system utilizes a normalized coordinate space representation for the articulation geometry relative to the query point's depth.

4. This normalization makes the target invariant to uniform scaling of the scene, meaning the model doesn't need to infer absolute metric size from appearance alone (like traditional methods that regress raw 3D geometry).

5. The system explicitly predicts a direction vector for both revolute and prismatic joints, along with an offset vector.

6. For revolute joints, the predicted offset is expressed in units of the query point's depth, allowing for deterministic lifting of the prediction to metric 3D space once the actual depth is measured at that query point (a single measurement suffices).

7. The architecture leverages a frozen foundation model encoder (DINOv3 ViT-B/16) combined with specialized, lightweight articulation-specific heads and a feature pyramid structure.

8. This combination allows the system to effectively combine high-resolution global semantic context (from the foundation model) with fine-grained local detail necessary for precise joint localization and classification.

9. The training objective is multi-task: it supervises the joint type classification, the axis direction using an unsigned cosine loss, and crucially, uses a Smooth L1 loss on a depth-normalized offset target that is invariant to axis reversal and global scaling.

10. This rigorous geometric supervision ensures that the predicted kinematic parameters are physically consistent (e.g., ensuring the predicted offset vector is orthogonal to the predicted axis direction), leading to highly accurate and reliable predictions, even on out-of-distribution data (like unseen furniture).

11. The resulting system can drive real-world manipulation tasks by generating analytic motion trajectories directly from its predictions.

12. Specifically, a robot equipped with this system can autonomously open previously unseen furniture (drawers, cabinets) or interact with objects in cluttered environments by first querying the model at a grasp point to determine the required joint parameters and motion path before executing physical interaction.

13. The system's performance is robust across various data modalities (synthetic and real-world) and viewpoint classes, demonstrating strong generalization capabilities to out-of-distribution object instances without requiring further retraining or fine-tuning.

Enabling robots to estimate the kinematic parameters of articulated objects unlocks a wide range of capabilities for interaction and manipulation. The estimation has to happen from the information the robot currently observes, often just a single RGB image of an object it has never seen before. Current single-image approaches couple articulation part segmentation with articulation estimation, making their predictions vulnerable to missed detections and incorrect part associations, and they regress metric 3D geometry that a single view fixes only up to scale. We present QueryArt, a model that estimates articulation parameters from a single RGB image, a 2D query point, and camera intrinsics. QueryArt is trained to estimate the 3D articulation geometry relative to the queried point and in units of its depth, which keeps its target identifiable from the image alone. A single depth measurement at the query point then supplies the scale and recovers the metric parameters. We train QueryArt on a curated mixture of synthetic and real-world articulation datasets. We evaluate QueryArt on several benchmarks and compare it against recent baselines. QueryArt outperforms recent baselines on most articulation metrics, including on out-of-distribution data. To demonstrate the model's capabilities in real-world settings, we evaluate QueryArt on a mobile manipulator across 57 manipulation trials spanning 16 object parts and five viewpoint classes, achieving a 70.2% success rate. We provide code and videos at: https://abwerby.github.io/queryart/