SAM3 for Cross-Prompt Detection: A Practical Walkthrough

Segmentation used to mean clicks and boxes. You picked a point. The model masked one object. That was the whole interaction.

SAM3 changes this. Meta built it to understand concepts, not just coordinates. You give it a phrase like "yellow school bus" or a single boxed example, and it finds every matching object in the frame. This guide walks through how SAM3 works, why cross-prompt detection matters, and how a Labellerr project applies these ideas in real code.

Official paper

What Is SAM3?

SAM3 stands for Segment Anything Model 3. Meta released it as a unified model for detecting, segmenting, and tracking objects in both images and video. It accepts three prompt types: short text phrases, image exemplars, and visual clicks such as points or boxes.

This task has a name. Meta calls it Promptable Concept Segmentation, or PCS. Compared to its predecessor SAM 2, SAM 3 introduces the ability to exhaustively segment all instances of an open-vocabulary concept specified by a short text phrase or exemplars. That single sentence explains the entire leap forward. Older models found one object per prompt. SAM3 finds every instance that matches your concept, in a single pass.

SAM 3 can handle a vastly larger set of open-vocabulary prompts, achieving 75 to 80 percent of human performance on Meta's SA-Co benchmark, which contains 270,000 unique concepts. That is over fifty times larger than older segmentation benchmarks. The scale here is not a marketing number. It reflects how much broader the model's vocabulary has become.

The Architecture Behind the Model

  SAM3 architecture

SAM 3 consists of a detector and a tracker that share a vision encoder, with 848 million parameters total. The detector is a DETR-based model conditioned on text, geometry, and image exemplars. The tracker portion carries over the transformer encoder-decoder design from SAM2, so it supports video segmentation and lets you refine results interactively as you go.

One detail matters more than the rest for cross-prompt work. The model consists of an image-level detector and a memory-based video tracker that share a single backbone. Because the backbone stays the same regardless of what you type into the prompt, SAM3 can reuse extracted image features across multiple detection passes. This is the technical seed behind cross-prompt and cross-image workflows.

Meta also added something new to the architecture. A presence token helps the model discriminate between closely related text prompts, such as "a player in white" versus "a player in red." Recognition and localization run as separate steps now. This split reduces the confusion that happens when two prompts sound almost identical.

Text Prompts vs Exemplar Prompts

SAM3 supports two main ways to define a concept.

Text prompts work like a search query. You type a noun phrase, and the model scans the image for every match. SAM3 performs Promptable Concept Segmentation on images, taking text and image exemplars as input, and predicts instance and semantic masks for every object matching the concept. Type "car," and every car in the frame gets a mask. No manual clicking required.

Exemplar prompts work differently. You draw a box around one example object. SAM3 then treats that box as a visual definition of the concept and finds every similar object elsewhere in the same image. Providing bounding box examples lets you segment similar objects, and multiple bounding boxes can serve as exemplars of the same visual concept.

This second mode is where cross-prompt detection becomes genuinely useful. Instead of describing an object in words, which can be awkward for unusual shapes or textures, you just point at one and say "more like this."

What "Cross-Prompt" Means in Practice

Project : Github

Cross-prompt detection, as demonstrated in the Labellerr project, refers to applying one defined prompt (a text phrase, or an exemplar drawn on a reference image) consistently across a batch of different images. Rather than manually re-labeling every photo in a folder, you set the concept once and let SAM3 apply it everywhere.

  Project workflow

This pattern mirrors an established Labellerr workflow. A batch inference script accepts a list of text prompts like "Red Flower," "Yellow Flower," "White Flower," and "Violet Flower," then runs SAM3 across an entire folder of images to detect and segment each matching category. The output gets exported in COCO-JSON format, ready for upload back into Labellerr for review and refinement.

The efficiency gain here is real. The script uses prompt-sequential processing to keep GPU memory usage low, which matters if you're running this on a single consumer card rather than a data center GPU.

Worth noting: SAM3's official documentation currently treats each image independently at inference time. A community discussion on passing exemplars from one image to a different image confirms this directly.

For non-tracked image-to-image prompting, the documented cross-image workflow today runs each image independently rather than through the video tracker, since SAM 3 currently documents exemplar prompts for concept segmentation on the image being processed.

In other words, "cross-prompt" in this context means reusing the same defined concept across many images in a loop, not literally copying a mask from image A onto image B without rerunning detection.

Setting Up SAM3 for Your Own Project

Getting started follows a familiar pattern if you've used earlier SAM versions.

You install the notebook dependencies, then start Jupyter to run the example predictor notebook. The basic image inference code looks like this:

from PIL import Image
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

model = build_sam3_image_model()
processor = Sam3Processor(model)

image = Image.open("your_image.jpg")
inference_state = processor.set_image(image)

output = processor.set_text_prompt(state=inference_state, prompt="your_prompt")
masks, boxes, scores = output["masks"], output["boxes"], output["scores"]

This same structure applies whether you load an image and prompt it with text, or set up the video predictor for tracking across frames.

For a cross-prompt batch job, you wrap this loop around every file in a folder, keeping the prompt (or list of prompts) fixed while the image variable changes. Multi-class detection support means the same script can carry several text prompts to detect different object categories across the whole folder in one run.

Performance You Can Expect

Numbers help set realistic expectations before you commit GPU hours to a project.

In zero-shot evaluations on the LVIS dataset, SAM3 achieves a mask AP of 48.8, a significant jump over the previous best score of 38.5. On the SA-Co dataset, SAM3 reaches a cgF1 score of 37.2, more than double the performance of strong baselines like OWLv2.

Speed is a separate consideration. If your project involves video and multiple tracked objects, Meta's newer SAM 3.1 update is worth knowing about. SAM 3.1 introduces Object Multiplex, a shared-memory approach for joint multi-object tracking that delivers up to seven times faster inference without sacrificing accuracy, and it requires no API changes if your pipeline already runs SAM3.

Where This Fits Into a Labeling Workflow

Cross-prompt detection is not just a research demo. It solves a real bottleneck in data annotation.

Uploading pre-annotations directly to Labellerr projects for review lets teams skip the slowest part of labeling: drawing every mask from scratch. Instead, SAM3 produces a first pass across your entire image set, and a human reviewer corrects the mistakes. This is faster than manual annotation and more accurate than fully automated labeling with no review step.

For dense or repetitive scenes, exemplar-based cross-prompting removes even more manual work. Click one instance of an object as your reference, and let the model handle the rest of the matching instances in that frame, one image at a time.

Conclusion

SAM3 moves segmentation from a manual, click-by-click task to a concept-driven pipeline. You define what you want once, whether through text or a boxed example, and the model applies that definition consistently. The cross-prompt pattern demonstrated in the Labellerr project shows exactly how this scales: one prompt, one folder of images, one batch job, and a COCO-JSON file ready for review.

If you're building an annotation pipeline, this is the direction worth testing first. Start small with a single text prompt on a handful of images. Confirm the masks look right. Then scale up to the full batch script once you trust the output.

FAQs