Reachability Is Not Generalization: Understanding Verb--Noun Decomposition in Assembly Action Recognition

Assembly actions are compositional, combining a manipulation (verb) with a part or tool (noun), yet traditional atomic classifiers assign zero probability to unseen combinations by construction. The central finding is that while verb–noun decomposition makes unseen compositions reachable, its generalization extends only partially beyond it, revealing that performance is governed by training-data structure rather than unconstrained recombination.

The Gist

Decomposition makes unseen compositions reachable but not necessarily generalizable because success remains strongly conditioned on primitive support and vocabulary asymmetry.

Testing the Assumptions of Decomposition

The paper systematically tests several implicit assumptions required for decomposition to generalize, using three representative formulations: atomic prediction (the unreachable baseline), shared-encoder multi-head architecture (the field standard), and independent recombination of separately trained component models. The analysis proceeds from phenomenon to localization, mechanism, and intervention across four research questions.

1. Whether unseen-composition success reflects genuine recombination or merely interpolation within densely observed regions of the compositional space (RQ1).

2. Where failures occur and whether the dominant source of error can be predicted before training (RQ2).

3. Why the field-standard shared encoder generalizes worse than independent recombination (RQ3).

4. Whether these limitations can be alleviated through alternative factorization granularities, architectures, or objectives (RQ4).

What Must Hold for Decomposition to Generalize?

Decomposition succeeds only if several conditions hold. The first is visual separability: the assumption that the visual evidence required to recognize one component can be represented without being inseparably tied to the other, meaning a component head's score should remain stable across different noun contexts. Failure of this leads to shared representations introducing a shortcut, quantified by the metric: ∆ent = dropjoint seen→unseen −dropsolo seen→unseen. The second condition is statistical factorization, requiring that component predictions provide a useful approximation to the joint action distribution: P(v,n x) ≈ Pˆ(v x)Pˆ(n x). This is tested by comparing unseen accuracy with the amount of training support available for its constituent primitives, measured by ρ = corrAcc(vu,nu), f(vu)· f(nu), where a high ρ suggests success is driven by interpolation near densely observed regions.

Which Component Becomes the Bottleneck?

The analysis localized failures to specific components based on error decomposition. The hypothesis is that unseen errors systematically concentrate on the larger-vocabulary component. This asymmetry is predictable based on dataset structure, not component type; for instance, in IMPACT, where the verb vocabulary exceeds its noun vocabulary, the bottleneck reverses with it across all five views. This is quantified by comparing verb-only and object-only errors. Furthermore, shared training introduces component entanglement, which encourages reliance on co-occurrence patterns that transfer poorly to novel compositions. This entanglement is measured by the excess unseen drop: a large ∆ent value indicates that shared training provides little benefit on seen compositions while substantially increasing the unseen generalization gap.

Why Shared Training Underperforms (RQ3)

The gap between shared and independent decomposition is attributed to component entanglement rather than capacity alone. While independent recombination generally generalizes better, the shared formulation amplifies coverage limitations. This is evidenced by comparing the seen margin (joint minus independent accuracy on seen compositions) against the excess unseen drop ∆ent. The fact that seen margins remain close to zero while ∆ent is consistently large suggests that joint training encourages reliance on co-occurrence structure that benefits familiar combinations but transfers poorly to novel ones. This implies that decomposition, in its current shared form, behaves more like structured retrieval than genuine compositional reasoning.

Can the Limitation Be Engineered Away? (RQ4)

The final question evaluates interventions: factorization granularity, architecture, and training objectives. The results show that finer factorizations do not improve compositional generalization, as the standard two-component decomposition (k=2) provides the best unseen performance. Increasing granularity beyond k=2 primarily improves seen accuracy while degrading transfer to unseen compositions. Architecture has the largest effect; Independent recombination consistently outperforms the shared formulation, supporting the entanglement mechanism identified in RQ3. Crucially, none of these interventions change the asymmetry identified in RQ2: the largervocabulary component remains the dominant source of unseen error across architectures. The remaining headroom is largely a ranking problem rather than a representation problem. The paper suggests that subtracting the factorized training prior at inference, without retraining, improves unseen accuracy by up to +4.2pp on IMPACT.

Discussion: Implications for Compositional Recognition

The overall conclusion is that decomposition improves reachability but not generalization because performance is shaped by training co-occurrence structure. The diagnosis links primitive support, vocabulary asymmetry, and component entanglement to these failures. This suggests current methods rely more on "structured reuse of observed co-occurrences than on genuine compositional reasoning.

Here are the specific improvements to AI systems derived from this research, categorized by the mechanism of improvement:

) Primitive Support & Vocabulary Asymmetry Allocation:

Allocate model capacity (parameters, training data augmentation budget) asymmetrically toward the larger-vocabulary component (verb or noun, depending on which is larger in a given dataset). This targets the known bottleneck identified in RQ2 and Table 3.

The improved system will exhibit higher unseen accuracy by focusing more learning resources on the component that is inherently harder to learn or cover, rather than treating both components with equal weight.

) Architectural Choice Over Objective Design:

Prioritize architectural designs that enforce component separation (e.g., independent recombination of trained sub-models) over simply adding compositional objectives (like C2C).

The improved system will be more robust to compositional shift because it preserves the necessary component separation, mitigating the performance degradation caused by component entanglement identified in RQ3.

) Inference-Time Prior Subtraction:

Implement a technique where, at inference time, the factorized training prior is subtracted from the final recombined score without requiring retraining.

The improved system will achieve an immediate boost in unseen accuracy (up to +4.2pp on IMPACT) by correcting the bias introduced during factorization, effectively moving it closer to genuine compositional reasoning rather than structured retrieval.

) Enhanced Recombination Strategy (Addressing Interpolation vs. Composition):

When dealing with unseen compositions, the system should employ a strategy that explicitly tests for genuine recombination rather than relying solely on interpolation within densely observed regions of the training grid.

The improved system will be more sensitive to true compositional structure, leading to better performance on low-support/novel combinations where current models default to interpolation.

) Granularity Optimization (Finding the Pareto Frontier):

Instead of sticking to a standard two-way decomposition, the system should dynamically select a factorization granularity (number of heads, k) that maximizes unseen accuracy while minimizing degradation in seen accuracy.

The improved system will adapt its internal representation strategy based on the specific dataset structure, selecting the optimal level of verb/noun/tool decomposition to maximize generalization capability for unseen actions.

This overall improved AI system can perform:

1. Maximize performance on assembly actions that have never been explicitly seen during training (i.e., improve compositional zero-shot learning).

2. Be more robust to changes in the underlying data distribution (compositional shift) by correctly identifying and mitigating the sources of failure (primitive support, vocabulary imbalance, and entanglement).

3. Operate as a more sophisticated compositional reasoning engine rather than a simple structured retrieval system.