Introducing Meta SAM 3 & SAM 3D

Image segmentation and annotation underpin countless computer vision applications, from autonomous vehicles detecting road users to medical imaging highlighting tumors.

Yet producing pixel-accurate masks at scale has been painfully expensive and time-consuming.

Traditional vision models often rely on fixed class taxonomies and require manual bounding-box or mask annotations, struggling to “link language to specific visual elements”

Meta’s Segment Anything initiative began to address this by treating segmentation as a promptable task: SAM 1 (Apr 2023) and SAM 2 (Jul 2024) demonstrated zero-shot image (and video) segmentation using visual prompts (points, boxes, masks).

However, these versions still required manual prompts and only handled a fixed set of labels.

The latest SAM 3 and its 3D counterpart SAM 3D dramatically extend this paradigm by enabling open-vocabulary promptable segmentation, multi-instance detection, and even 3D reconstruction from a single image.

In this blog, we take a closer look at SAM 3 and how it's reshaping not just the data annotation landscape but also unlocking new frontiers across industries like robotics, autonomous systems, and VR.

What's New in SAM 3 And SAM 3D?

The release of SAM 3 and SAM 3D marks a milestone in the evolution of promptable segmentation and 3D perception.

Unlike its predecessors, SAM 3 introduces the ability to perform open-vocabulary segmentation using text or image exemplars.

This means users can input a simple prompt like "dog" and SAM 3 will return segmentation masks for all relevant instances in the image or video. It’s not just segmenting what you click on, it’s understanding the concept itself.

In parallel, SAM 3D expands SAM's utility into the third dimension. SAM 3D Objects and SAM 3D Body reconstruct textured 3D meshes from a single image.

SAM 3D Objects handles general scenes and objects, while SAM 3D Body specializes in human form, capturing body pose and shape with remarkable accuracy.

With this dual evolution, SAM now bridges 2D segmentation and 3D modeling, offering a powerful toolkit for developers working across a variety of visual understanding applications.

Limitations Of Previous Generation Model?

While SAM 1 and SAM 2 were revolutionary in introducing promptable segmentation, they were still largely dependent on user interaction. SAM 1 required points or boxes to be manually placed, segmenting only one object per prompt. SAM 2 improved speed and video capability with memory transformers but was still limited in its ability to generalize.

Notably, both models lacked semantic understanding. They couldn’t interpret or act on textual descriptions. This severely limited their usefulness in scenarios like automated labeling or dynamic object detection.

Moreover, they offered no support for 3D reconstruction, leaving a gap for applications that required spatial or volumetric understanding.

Inference Result of SAM 3

You can check SAM 3 at its official GitHub or Try playground provided by Meta.

Here are some example inference:

SAM 3: Prompt based Video Segmentation

SAM 3D

Architecture and Capability

SAM 3 is architected with a dual-branch design: one branch handles DETR-like detection based on text prompts, while the other uses memory-based tracking for temporal consistency in videos.

This allows the model to detect, segment, and track multiple instances of a concept simultaneously and with high fidelity.

Architecture Diagram

The presence head is a key innovation, decoupling recognition from localization. This enables SAM 3 to answer both "what is in the image?" and "where is it?" more effectively than prior models.

The open-vocabulary engine is powered by a massive dataset of over 4 million unique noun phrases and 1.4 billion synthetic masks, enabling robust generalization.

SAM 3D complements this with neural rendering techniques to create high-fidelity 3D outputs from a single image. SAM 3D Objects produces textured meshes, while SAM 3D Body incorporates a new rigging system for anatomically accurate human reconstruction.

These models function with minimal user input and are tuned for real-world variability, such as occlusion and complex poses.

Industry Impact and Use Cases

SAM 3 and SAM 3D will make waves across various industries:

• Robotics: Robots can use open-vocabulary segmentation to identify and manipulate objects described in natural language, greatly improving autonomy in complex environments.
• Autonomous Vehicles: The ability to recognize novel or unusual objects on the road can enhance the safety and adaptability of self-driving systems.
• Media and Entertainment: Video editors can segment and track objects via text, making complex editing tasks more intuitive and efficient.
• Healthcare: Medical imaging tools can use promptable segmentation to highlight anomalies like tumors or organs, reducing the burden on radiologists.
• E-commerce and AR/VR: SAM 3D enables quick generation of 3D models from product photos, supporting virtual try-ons and immersive shopping experiences.

Conclusion

SAM 3 and SAM 3D represent a significant leap toward universal visual understanding. They address key limitations of prior models by introducing semantic prompts, multi-instance support, and 3D reconstruction from a single image. Their impact spans from simplifying data annotation workflows to enabling advanced robotics and AR/VR applications.

For ML/Computer Vision Engineers, these models offer both a new set of tools and a new paradigm: vision models that understand language and space, not just pixels. As these models become more widely adopted, they are likely to redefine how we build and deploy computer vision systems at scale.

FAQ