ActiveSAM: Fast and Accurate Open-Vocabulary Semantic Segmentation with Frozen SAM 3

ActiveSAM introduces a training-free, zero-shot inference framework that adapts the frozen SAM 3 model into an active-vocabulary segmenter for open-vocabulary semantic segmentation (OVSS). This method addresses the inefficiency of running full-resolution decoding over an entire vocabulary when only a small subset of classes is relevant to a specific image. By leveraging SAM 3’s presence signal before full decoding and employing class pruning, ActiveSAM significantly improves the speed-accuracy tradeoff compared to existing state-of-the-art methods, achieving up to +1.4 mIoU improvement over SegEarth-OV3 while running up to 5.5× faster on large datasets, making it highly suitable for deployment in noisy, real-world domains.

Key Contributions and Framework Overview

ActiveSAM is a training-free inference framework that keeps all SAM 3 weights frozen, transforming the model into an image-adaptive segmenter at test time. The core idea is to use SAM 3’s presence signal before full-resolution decoding rather than only as a post-decoding filtering score. This changes the computational structure by invoking the expensive prompt-conditioned decoder only for an image-specific active vocabulary, which scales with the active set size rather than the full vocabulary size.

The framework consists of three main stages:

1. A low-resolution presence preview that estimates an image-conditioned active set, denoted as A(I).

2. Full-resolution decoding, where only prompts from the retained classes are decoded using bucketed prompt multiplexing.

3. Margin-aware background calibration to suppress low-confidence pixels and assign the background label when applicable.

Contextual Prompt Expansion (CPE)

To ensure robust alignment, ActiveSAM employs Contextual Prompt Expansion (CPE) to enrich raw class names with lexical and vocabulary context before decoding. This module performs two main tasks:

1. Lexical canonicalization: It applies a fixed, image-independent map, such as repairing compound labels like windowpane7 to window pane, ensuring deterministic label artifacts are resolved before text encoding.

2. Contextual enrichment: It retrieves class-level semantic context by finding the nearest other class names in the frozen pooled text-embedding space (Ms nearest neighbors) and incorporating WordNet noun hypernyms. The final prompt for each class is constructed by concatenating the canonicalized tokens with these cached neighbor and hypernym tokens, resulting in a contextual prompt, denoted as Z¯c.

Preview-driven Class Selection

The framework uses SAM 3’s low-resolution presence head as a preview to select the active vocabulary A(I). This stage is crucial for reducing latency by skipping unnecessary segmentation-head computation. The selection process involves:

1. A gating threshold, Vgate = 40, which bypasses the preview stage for small vocabularies (V ≤ Vgate).

2. For large vocabularies (V > Vgate), SAM 3 is run at a low resolution of rp = 672 to estimate a class-presence score qc for every class.

3. The final active set A(I) is determined by an image-adaptive quantile, defined as A(I) = (A0(I), A0(I) ≤ Vgate, arg TopKVgate c∈A0(I) qc).

Bucketed Full-Resolution Decoding and Calibration

Once the active set A(I) is selected, the full-resolution SAM 3 image encoder is evaluated once. The active classes are then partitioned into buckets (B1,..., BJ), where each bucket contains at most K=32 classes. The frozen grounding decoder processes these buckets using shared image features and the corresponding contextual prompts Z¯c. Scores from the instance and semantic heads are fused pixel-wise to obtain Sc(p) = max S inst c(p), Ssem c(p) for c ∈ A(I). Finally, Margin-aware Background Calibration (MABC) is applied:

1. It computes a margin score m(p) = s1(p) − s2(p), where s1 and s2 are the top two active-class scores at pixel p.

2. The MABC-calibrated confidence is cMABC(p) = s1(p) + max(s1 - s2, 0).

3. The final prediction yˆ(p) is set to background (bg) if cMABC(p) < tγ, where tγ is the dataset's background threshold and γ = 1.25.

Performance and Robustness

ActiveSAM achieves state-of-the-art performance across eight OVSS benchmarks, outperforming SegEarth-OV3 by +1.4 mIoU on average while running up to 5.5× faster on large datasets.

As a fastidious researcher, I have analyzed the ActiveSAM paper and identified several concrete, high-impact areas for improving current AI systems, particularly in the domain of Open-Vocabulary Semantic Segmentation (OVSS).

Here are the specific improvements and what the resulting system can achieve:

) 1. Inference Efficiency and Scalability Improvement

The most immediate improvement is a fundamental shift in computational architecture from full-vocabulary decoding to active-set decoding.

[Source: Abstract, Section 3.4, Computational Analysis]

ActiveSAM drastically reduces inference time by skipping the expensive full-resolution grounding decoder for classes not present in the image (via the low-resolution presence preview).

[Specific Improvement]: Implement a presence-first routing mechanism where a frozen backbone's presence head dictates which prompts receive full computational load. This transforms prompt decoding from an operation scaling with vocabulary size to one scaling only with active class set size, achieving up to 5.5× faster inference on large datasets compared to SegEarth-OV3.

[System Capability]: Enables real-time or near real-time semantic segmentation for high-resolution imagery (e.g., autonomous driving feeds, high-throughput surveillance) where latency is critical, without sacrificing the accuracy achieved by the frozen SAM 3 backbone.

) 2. Robustness to Distribution Shift and Environmental Noise

The system explicitly shows superior performance when inputs are corrupted (noise, blur, compression), which is vital for real-world deployment.

[Source: Abstract, Section 4 (Robustness)]

ActiveSAM maintains high performance under Gaussian noise, motion blur, JPEG compression, and fog where CLIP-based methods degrade severely.

[Specific Improvement]: Integrate the margin-aware background calibration (MABC) mechanism as a primary noise filter during the final prediction stage. By using the margin between the top two scores to decide on background assignment rather than a simple absolute threshold, it enhances local consistency and fine-grained separation in complex scenes corrupted by visual artifacts.

[System Capability]: Creates more reliable perception systems for autonomous vehicles and embodied AI operating in unpredictable real-world conditions (e.g., adverse weather, sensor noise) by maintaining high mIoU stability compared to state-of-the-art baselines like CorrCLIP.

) 3. Enhanced Semantic Prompt Quality via Contextual Expansion

The method moves beyond raw class names by enriching them with lexical and hierarchical context before decoding.

[Source: Section 3.1, Contextual Prompt Expansion (CPE)]

ActiveSAM uses the frozen SAM 3 text encoder to generate prompts that are augmented by semantic-neighbor tokens (nearest neighbors in embedding space) and WordNet hypernyms.

[Specific Improvement]: Systematically cache and retrieve class context (neighbor embeddings and WordNet hypernyms) once per dataset, ensuring every image prompt is rich with both immediate visual semantic context and broader hierarchical knowledge. This prevents poor alignment caused by isolated or ambiguous class names (e.g., correctly expanding windowpane into related concepts like window pane).

[System Capability]: Improves the precision of segmentation in complex, fine-grained scenes (e.g., distinguishing between different types of furniture or architectural elements) where standard text prompts fail due to low semantic density.

) 4. Optimized Prompt Multiplexing and Grounding

The system uses bucketed prompt multiplexing to efficiently feed multiple active class prompts into the decoder in a single pass, rather than sequentially decoding each one.

[Source: Section 3.3, Bucketed Full-Resolution Decoding]

ActiveSAM groups active classes into buckets and feeds the shared image features and contextual prompts for that bucket simultaneously to the grounded decoder.

[Specific Improvement]: Implement dynamic prompt batching based on image content complexity or object density, rather than a fixed bucket size (K=32). This ensures that highly complex regions receive sufficient computational resources (shared cross-attention) without wasting cycles on simple regions.

[System Capability]: Increases throughput for high-density scenes by maximizing the efficiency of the frozen SAM 3 decoder, ensuring that the computational savings from pruning are fully realized in terms of speed and accuracy trade-off.

This set of improvements transforms a general OVSS pipeline into a highly efficient, robust, and contextually aware perception system capable of operating reliably in dynamic, real-world environments.

Segment Anything Model 3 (SAM 3) provides a strong frozen backbone for concept-prompted segmentation, but applying it directly to open-vocabulary semantic segmentation (OVSS) is inefficient: full-resolution decoding is typically run over the entire dataset vocabulary, whereas each image contains only a small active subset of classes. We introduce ActiveSAM, a training-free inference framework that turns SAM 3 into an active-vocabulary segmenter. ActiveSAM first canonicalizes and expands class prompts, then uses evidence-proportional grounding to estimate an image-conditioned active set from a low-resolution presence preview. Only retained prompts receive full-resolution mask prediction, using bucketed prompt multiplexing with the frozen SAM 3 decoder. The preview stage uses only class-presence evidence and skips unnecessary segmentation-head computation. To resolve overlapping concept responses, exclusive concept decoding compares each pixel's joint score vector with class signatures estimated once per vocabulary from unlabeled images. ActiveSAM requires no weight updates, no oracle class-presence labels and no per-dataset hyperparameter tuning. Across eight OVSS benchmarks, ActiveSAM improves the speed-accuracy tradeoff of training-free open-vocabulary semantic segmentation, outperforming the current state-of-the-art SegEarth-OV3 by +2.1 mIoU on average while running much faster, with 7.3-12.2x speedups on large-vocabulary datasets. ActiveSAM also achieves the highest accuracy under image corruptions that simulate real-world distribution shift, making it well-suited for deployment in noisy-input domains such as autonomous driving and embodied AI. Code is available at https://github.com/VILA-Lab/ActiveSAM