Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

The Segment Anything Model (SAM) currently relies heavily on massive manual annotations, creating a fundamental bottleneck for model scaling. This paper introduces Motion-Grounded Segment Anything (MoSA), a highly scalable unsupervised framework that learns a transferable objectness prior from unlabeled videos to achieve segmentation performance comparable to the fully supervised SAM without requiring manual annotations.

How it works

The MoSA framework operates in three progressive stages:

1. Automatically generating multi-granularity motion pseudo-labels from large-scale video data via a Multi-Granularity Motion Segmentation (MGMS) pipeline.

2. Training a Perceptual Grouping Model (PGM) via contrastive learning to internalize a generalized, appearance-driven concept of objects from these motion pseudo-labels.

3. Transferring this learned prior into a prompt-guided architecture for segment-anything style inference on images, enabling strong zero-shot performance.

Multi-Granularity Motion Segmentation (MGMS)

The MGMS pipeline is designed to generate high-quality motion pseudo-labels from unlabeled videos using bidirectional optical flow. It comprises two complementary modules:

1. Rigid Object Segmentation Module (ROSM): This module adapts the SOLOv2 architecture to segment individual rigidly moving objects, leveraging parallax for static foreground objects and discerning object parts.

2. Motion Foreground Segmentation Module (MFSM): This module employs a U-Net architecture to delineate the entire moving foreground from background, capturing complex non-rigid deformations.

To ensure high fidelity, candidate masks are filtered using a Mask Quality Score (Squality), which combines two components: Maskness (Smaskness)—the mean prediction value for pixels exceeding a confidence threshold—and Boundary Sharpness (Ssharpness)—the proportion of boundary pixels with a gradient magnitude above a threshold. Masks with an Squality below 0.85 are discarded, and Non-Maximum Suppression (NMS) is applied to yield the final set of pseudo-labels.

Perceptual Grouping Model (PGM)

The PGM is built upon a Vision Transformer (ViT) backbone and incorporates K parallel linear projection heads to capture objects at various scales. A novel contrastive training strategy, Perceptual Grouping Contrastive Learning (PGCL), is employed to learn a generalizable notion of “objectness” independent of explicit object count or categories.

The PGCL loss contrasts positive pairs (patches within the same mask) against negative ones by modeling the probability that a patch belongs to the same object as an anchor patch, using a sigmoid function with a learnable temperature τk:

p t,j k = σ (τ k · s t k), where s t k is the cosine similarity between anchor and patch features from head k.

Adaptation to Segment Anything

The learned objectness prior from the PGM is transferred into two specialized models for inference:

1. Whole-Image Segmentation Model: This model employs a DINO pre-trained ResNet-50 backbone with a Mask2Former decoder, configured with 1000 learnable object queries, to output automatic segmentation masks for high-resolution images.

2. Promptable Segmentation Model: This model is based on the Semantic-SAM framework and uses a SwinTransformer Tiny backbone to generate hierarchical masks from user prompts, simulating user clicks during training by sampling points within the foreground of each mask as positive point prompts.

Evaluation and Results

MoSA achieves state-of-the-art performance across seven challenging benchmarks (e.g., COCO and ADE20K). For whole-image segmentation, MoSA achieves an average Average Recall (AR) of 42.1% on the SA-1B dataset, significantly outperforming existing unsupervised methods like UnSAM [38]. For point-based promptable segmentation on COCO Val2017, MoSA achieves a MaxIoU of 41.6% and an OracleIoU of 63.4%, surpassing the previous SOTA, UnSAM [38], by +1.3% and +3.9% respectively. Ablation studies confirm that the combination of ROSM and MFSM is essential, as their individual contributions are complementary for capturing multi-scale motion cues, leading to a final score of 36.6% AR1000 when both are included in the MGMS pipeline. The study concludes that learning general object perception from large-scale unlabeled motion is a feasible and scalable alternative to annotation-driven pipelines.

Limitations

A practical failure mode identified is degraded motion estimation in lowlight environments, such as night scenes, where unreliable optical flow can weaken supervision for fine structures. Future research will focus on improving robustness to these flow-degraded conditions. The current work focuses on extracting general segmentation knowledge from motion cues with the goal of enabling zero-shot segmentation on static images without relying on explicit motion information during inference.

Here are the specific improvements that can be made to AI systems by implementing Motion-Grounded Segment Anything (MoSA), along with a description of what these improved systems will be capable of:

The implementation of MoSA enables the creation of a foundational, scalable segmentation system capable of achieving state-of-the-art performance comparable to fully supervised models like SAM, but trained entirely without manual annotations.

Specific improvements and capabilities include:

1. textbf>Zero-Shot Multi-Granularity Segmentation from Unlabeled Video Data (Core Capability):

Based on the three-stage MoSA framework (MGMS pipeline, PGM contrastive learning, and Adaptation), the improved system can perform segmentation on completely unseen static images in a zero-shot manner. It learns a generalized concept of objectness by observing motion patterns in massive, unlabeled videos (Kinetics-700, BDD100K) rather than relying on specific object labels.

2. textbf>Generalization to Static and Weakly Moving Entities (Addressing Unsupervised Bias):

The Perceptual Grouping Model (PGM), trained via Perceptual Grouping Contrastive Learning (PGCL), overcomes the limitation of motion-only models by internalizing appearance-driven concepts. This allows the system to successfully segment objects that are completely static or exhibit only slight motion (e.g., roof tiles, stationary furniture) where traditional motion-based methods fail, significantly expanding its object vocabulary beyond moving entities.

3. textbf>Handling Complex Multi-Granularity Objects (Enhanced Structural Understanding):

The Multi-Granularity Motion Segmentation (MGMS) pipeline uses two complementary modules: the Rigid Object Segmentation Module (ROSM) and the Motion Foreground Segmentation Module (MFSM). This allows the system to simultaneously capture:

• Precise, instance-level masks of rigidly moving objects and their articulated parts.

• Coarse, encompassing binary masks for complex, non-rigid entities like pedestrians or animals.

This duality ensures a comprehensive understanding of objects across different scales and structural complexities within a single video frame.

4. textbf>Promptable Interaction with High-Resolution Fidelity (Interactive Segmentation):

By transferring the learned object priors into prompt-guided architectures (like Mask2Former for whole-image segmentation and Semantic-SAM for promptable segmentation), the system gains interactive capabilities. Users can interact with the model using point prompts, leading to high-quality, fine-grained masks across multiple granularity levels (e.g., segmenting a person's arm vs. their entire torso) directly from high-resolution images.

5. textbf>Scalability and Efficiency (Reduced Annotation Bottleneck):

The system drastically reduces the dependence on costly manual annotations (SA-1B). The MoSA framework is shown to achieve performance comparable to SAM while requiring significantly less training data (1% of SA-1B) and fewer model parameters, making it a highly scalable alternative for deploying general-purpose segmentation models in resource-constrained environments.

6. textbf>Robustness via Multi-Stage Filtering (High Confidence Outputs):

The inclusion of a rigorous Mask Quality Score (Squality), which combines maskness and boundary sharpness metrics, ensures that the system produces outputs with high pixel confidence and clear boundaries, minimizing segmentation noise—a critical improvement over methods relying solely on raw pseudo-labels.