sam3.cpp : Segment Anything 3/2/2.1/EdgeTAM using ggml

[PABannier]

Hi everyone,

sam3.cpp is a C++ port of Meta's Segment Anything models (SAM 2, SAM 2.1, SAM 3, EdgeTAM) using ggml. It runs real-time video segmentation at 30 FPS on CUDA, with no Python, no PyTorch, and no heavy dependencies.

What it does:

• Point, box, and text-prompted segmentation ("find every cat in this image")
• Video object tracking with a memory bank across frames
• 4-bit quantization: EdgeTAM fits in 15 MB, SAM 2.1 Tiny in 22 MB
• CUDA and Apple Metal GPU acceleration
• Single-file C++ library (sam3.cpp + sam3.h), C++14, zero dependencies beyond ggml

The full SAM 3 model includes a ViT backbone, text encoder, and DETR decoder for open-vocabulary detection. The lighter models (EdgeTAM, SAM 2 Tiny) are small enough to run on a phone.

I went from PyTorch's ~2 FPS to 30 FPS by rewriting the entire pipeline in C++ with ggml, using fused Metal/CUDA kernels, aggressive quantization, and careful memory management to avoid unnecessary copies between pipeline stages.

52 pretrained models are available on Hugging Face in multiple precisions (f32, f16, q8_0, q4_1, q4_0).

Code: https://github.com/PABannier/sam3.cpp
Models: https://huggingface.co/PABannier/sam3.cpp

--

Along the way I needed several Metal operations that upstream only has on CPU/Vulkan/CUDA. Happy to upstream these if there's interest.

What I'd contribute:

1. Metal CONV_2D_DW: first Metal impl, needed for depthwise separable convolutions
2. Metal WIN_PART/WIN_UNPART: first Metal impl, needed for window-based vision transformers
3. Flash attention head_dim=56: new template instantiations for Hiera models
4. Flash attention head_dim=16 whitelist fix: templates existed, supports_op didn't list them
5. Improved conv_transpose_2d: rewritten gather algorithm, F16 vectorization
6. CONT fast path: float4-vectorized contiguous copy

Would any of these be welcome as PRs? Happy to split into separate PRs per operation.

[smidm]

You mention here CUDA acceleration, but your repo seems to support just Apple Metal. How is it with CUDA right now?