SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

The paper addresses a core limitation in embodied visuomotor models. As stated in the abstract: "Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures."

The authors critique prior memory-augmented approaches: "Existing methods lack an explicit mechanism for memorizing behaviorally salient skill-level features. Consequently, they tend to exhibit inefficient reuse of prior experience and diminished capability for compositional generalization in unseen tasks." Specifically, they note that MemoryVLA introduces a dual-stream Perceptual-Cognitive Memory Bank for context-aware decision making. HAMLET addresses non-Markovian dependencies by encoding interaction history into temporal moment tokens, but these typically store unstructured or holistic visual-semantic representations. They lack the capacity to capture and reuse fine-grained, skill-level behavioral patterns.

To overcome this, the authors propose the Skill-Based Memory (SkillMemo) framework, which implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic memory bank for solving compositional tasks.

The first component decomposes whole trajectories into latent atomic skills. The authors argue that "Conventional VLA models typically learn entire behavioral trajectories using a single monolithic network, which leads to limited compositionality and poor generalization over heterogeneous, high-dimensional action sequences."

The decomposition is implemented using a Mixture-of-Experts (MoE) architecture consisting of N expert networks E1,...,E N and a gating network G. Given an intermediate feature representation h t, the MoE layer computes:

y t = Σi gi(h t) · Ei(h t)

As the authors explain: "Intuitively, each expert captures a recurring local motion primitive, while the gating network g implicitly defines the temporal segmentation boundaries by modulating expert activations over time. This implicit partitioning mechanism enables adaptive and data-driven skill discovery, ensuring that the identified skills align with emergent task-level structures."

To encourage expert specialization, they introduce a synergy-based information loss derived from Partial Information Decomposition (PID) theory. For expert pair (A, B) and task target G:

L PID = −(I(A, B; G) − I(A; G) − I(B; G))

Minimizing this loss drives each expert to specialize in complementary skills while enabling higher-order coordination, thereby facilitating the learning of distinct, non-overlapping skill primitives. Additionally, the authors employ expert load-balancing losses and top-k routing strategies to prevent expert collapse.

The second component stores, retrieves, and composes the discovered skill primitives. The memory bank M is organized as key-value pairs:

Memory Key: For each atomic skill segment, temporal aggregation extracts a compact centroid feature: k m = (1/L) Σ h t over the segment's timesteps. This preserves the most representative characteristics of the skill trajectory, ensuring the efficiency of the retrieval process.

Memory Value: Since the gating weights g(h t) govern expert activation and implicitly define the category of the skill, the value stores the complete sequence of gating weights across all time steps within the segment. The memory is formalized as M = (k m, v m) v m = g(h t) t∈τ m m=1 M.

Memory Retrieval: During inference, cosine similarity is computed between the current latent feature q t and every key: s(q t, k m) = (q t · k m)/(q t k m). The top-N entries are retrieved, subject to a memory reliability check comparing average similarity against a predefined confidence threshold δ. The gating weights are then refined via:

g'(q t) = λg(q t) + (1−λ)(1/N) Σn vn[t′]

where λ controls memory intervention strength and vn[t′] is the aligned timestep within the retrieved skill sequence.

Memory Update: For finite capacity M max, a prioritized pruning strategy removes: (1) memories with the most outdated timestamps, mimicking the human cognitive process of forgetting distant, non-reinforced details; (2) memories that have remained unused, i.e., not successfully retrieved; (3) memories that consistently fail to meet the reliability threshold δ."

Component-wise analysis on the LIBERO benchmark shows progressive improvement from incorporating EGTS and SLMA. For example, DP success on the Goal suite rises from 68.7 (baseline) to 71.1 (+EGTS) to 73.4 (+both); on Spatial from 72.1 to 78.6 to 80.2; on Object from 75.7 to 79.3 to 79.8; on Push-T from 52.9 to 53.6 to 55.2; on BlockPush from 73.6 to 76.5 to 78.1; and on Kitchen from 57.1 to 59.1 to 60.6.

Expert number sensitivity analysis (N ∈ 1, 2, 3, 5, 8, 10) shows "the model's ability to represent diverse skill primitives improves, leading to a steady rise in success rates. However, we observe diminishing returns beyond N=5; specifically, increasing the expert count to 10 provides only marginal accuracy improvement while substantially increasing inference time. Therefore, we adopt N=5 as the default configuration."

Expert activation analysis (Figure 3) visualizes gating coefficients over time, showing distinct dynamic shifts aligned with critical motion transitions: "when the robotic arm initiates significant semantic actions such as grasping a bowl or turning on a stove, the activation weights of specific experts demonstrate pronounced peaks. This temporal correlation confirms that the MoE routing mechanism successfully extracts and specializes in distinct latent atomic skills, even in the complete absence of explicit action labels."

DP backbone benchmarks (Push-T, UR3 Block Push, Franka Kitchen): SkillMemo outperforms all baselines (DP, SDP, CP, IMLE Policy, STEP), achieving the highest average success rate of 64.6%, surpassing DP by 3.4% and IMLE Policy by 1.8%.

VLA backbone benchmarks (LIBERO): SkillMemo consistently enhances various VLA backbones across Goal, Spatial, Object, and Long suites. Notably:

• UniAct (0.5B): improved from 68.7→73.4 (Goal), 72.1→80.2 (Spatial), 75.7→79.8 (Object), 51.4→57.2 (Long), average 67.2→72.7

• UniVLA (8.5B): improved from 91.8→94.5, 96.5→98.1, 95.6→97.8, 92.0→93.2, average 93.9→95.9

• π0.5-SkillMemo establishes an average success rate of 98.0%, yielding a 1.2% (98.0% vs. 96.8%) improvement over π0.5 and decisively surpassing MemoryVLA by 1.5% (98.0% vs. 96.5%), demonstrating that skill-level memory is crucial for maximizing VLA performance.

Unseen task generalization (zero-shot cross-suite): In Table 4, models trained on one LIBERO suite are evaluated on others. Notably, the SkillMemo framework trained on Goal achieves 75.6% on unseen Spatial suite, which even outperforms the in-distribution baseline with 3.5% (75.6% vs. 72.1%) trained explicitly on Spatial. This strongly supports our hypothesis that the memory bank enables compositional reuse of skill primitives, allowing the model to generalize beyond the training distribution by recombining known skills.

Deployed on a 7-DoF UR5e robotic manipulator with a Weiss WSG-50 parallel-jaw gripper and ORBBEC Femto Bolt RGB-D camera, with 50 expert demonstrations per task and 40 rollouts per evaluation.

Single-task performance (Table 6): SkillMemo consistently outperforms DP across all five tasks, with particularly notable improvements on more challenging tasks such as Corn in Pot (+7.5%) and Butter in Pot (+12.5%). Results: Strawberry in Bowl 77.5→82.5, Lemon on Plate 77.5→80.0, Corn in Pot 82.5→90.0, Butter in Pot 62.5→75.0, Chip Bucket on Table 67.5→77.5.

Unseen compositional generalization (Table 7): Models trained on limited source-task subsets are evaluated on novel object-receptacle pairings. "SkillMemo significantly outperforms DP in these compositional scenarios, improving Strawberry on Plate from 65.0% to 75.0%. This confirms that our framework effectively retrieves and fuses atomic skills from disparate training experiences to facilitate robust compositional generalization on unseen task configurations."

The paper concludes: "We presented SkillMemo, a skill memory framework for compositional embodied manipulation that addresses the limitations of existing visuomotor models in capturing and reusing structured behavioral patterns. By coupling expert-guided trajectory segmentation with a skill-level episodic memory architecture, SkillMemo enables the explicit decomposition, storage, retrieval, and composition of atomic skill primitives. Extensive experiments on both simulation benchmarks and real-world robotic tasks demonstrate that SkillMemo consistently enhances the performance of diverse backbone architectures while exhibiting zero-shot compositional generalization to unseen task compositions."

Keywords: Skill Learning, Memory-Augmented Policy, Robot Manipulation.

• Add expert-guided skill decomposition: Replace monolithic policy networks with a Mixture-of-Experts layer containing N specialized expert networks and a gating network. The gate softly partitions long-horizon trajectories into latent atomic skills, learning to activate different experts at different temporal phases without requiring explicit skill labels.

• Enforce expert specialization via synergy-based information loss: Use Partial Information Decomposition (PID) to penalize overlapping information between expert pairs relative to the task target. This drives each expert to capture complementary, non-redundant skill primitives while preserving higher-order coordination among experts.

• Prevent expert collapse and balance utilization: Apply top-k routing strategies and expert load-balancing losses so all experts remain active and no single expert dominates, ensuring the discovered skill set is diverse and fully covers the behavioral space.

• Store discovered skills in an episodic memory bank: Maintain a dynamic memory of atomic skills as key-value pairs. Keys are compact temporal centroids of latent features for each skill segment; values are the full sequences of gating weights over that segment, preserving the temporal activation pattern that defines the skill.

• Add memory retrieval with reliability checking: At inference, compute cosine similarity between the current latent feature and all memory keys, retrieve the top-N most similar skills, and only use them if the average similarity exceeds a predefined confidence threshold δ. This filters out irrelevant memories that could harm performance.

• Blend retrieved skills with current predictions: Refine the current gating weights by combining the live gating output with the average of retrieved skill gating sequences, controlled by an adaptive interpolation weight λ. This injects structured prior experience into the decision-making process instead of relying on holistic, unstructured memory.

• Implement prioritized memory pruning and update: Bound memory capacity and regularly remove (1) outdated memories, (2) memories that are rarely or never retrieved, and (3) memories that consistently fail the reliability threshold. This mimics human forgetting of non-reinforced details and keeps the memory bank fresh, efficient, and trustworthy.

• Incorporate this as a plug-in module for any visuomotor backbone: Apply the SkillMemo components externally to Diffusion Policy (DP), VLA models such as UniAct, UniVLA, and π0.5, or other policies, enabling them to reuse learned skill structures without retraining the underlying architecture from scratch.

• Discover reusable skill primitives automatically: It decomposes long-horizon demonstrations into latent atomic skills (e.g., grasping, placing, stirring, turning on) purely from observation and action data, without requiring manual skill annotations or pre-defined task decompositions.

• Compose known skills to handle unseen tasks: It can recombine skills from its memory bank to solve novel task configurations never seen during training. For example, a model trained on put strawberry in bowl and put lemon on plate can generalize to put strawberry on plate by retrieving and blending the relevant skill memories.

• Outperform in-distribution training for new task suites: Its memory-driven composition can achieve higher success on unseen task distributions than a baseline explicitly trained on those tasks, as shown by the Goal-trained model reaching 75.6% on the Spatial suite versus 72.1% for the Spatial-only baseline.

• Maintain strong performance in out-of-distribution scenarios: It improves success rates across diverse backbones and benchmarks—e.g., π0.5 with SkillMemo reaches 98.0% average on LIBERO, surpassing MemoryVLA by 1.5% and the baseline π0.5 by 1.2%—by explicitly capturing and reusing skill-level behavioral patterns instead of only storing unstructured visual-semantic features.

• Act robustly in real-world manipulation with limited data: With only 50 expert demonstrations per task, it improves real-robot success rates by up to +12.5% on challenging tasks (e.g., Butter in Pot) and +10% on Chip Bucket on Table, demonstrating practical deployability in physical environments.

• Generalize zero-shot across simulation and real-world compositional settings: It reliably adapts to novel object-receptacle pairings in the real world (e.g., Strawberry on Plate improves from 65.0% to 75.0%) by retrieving and fusing atomic skills from disparate training experiences.

• Scale gracefully with model capacity: Using five expert primitives (N=5) gives most of the benefit with acceptable inference cost, while adding more experts only marginally improves accuracy—so the system can be tuned for speed or performance depending on the application.

• Provide interpretable skill activation patterns: The gating network shows distinct, pronounced peaks when the robot initiates significant semantic actions (e.g., grasping a bowl, turning on a stove), making it possible to inspect and reason about which skill primitive the system is currently executing.

Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures. To address this limitation, we propose Skill-Based Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic memory bank for solving compositional tasks. Specifically, we first introduce an expert-guided trajectory segmentation module built upon a Mixture-of-Experts (MoE) architecture, which implicitly partitions trajectories into distinct skill primitives represented by learned gating coefficients. We further design a skill-level episodic memory architecture that stores compact skill representations as retrievable key-value pairs. During inference, the memory bank retrieves the most relevant skill primitives which are subsequently fused with the model's current gating distribution, providing a robust contextual prior to refine action predictions. Extensive experiments on the simulation benchmark and real-world manipulation tasks demonstrate that SkillMemo consistently enhances both DP and VLA backbones, achieving state-of-the-art performance and outperforming pi 0.5, while exhibiting strong compositional generalization to unseen task configurations.