Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Accepted to the European Conference on Computer Vision (ECCV), 2026.

Overview

Segment Anything models are typically trained with massive collections of manually annotated masks. We present Motion-Grounded Segment Anything (MoSA), a scalable unsupervised framework that instead learns a transferable objectness prior from unlabeled videos.

MoSA learns in three progressive stages:

1. Motion pseudo-label generation. Multi-Granularity Motion Segmentation (MGMS) extracts object- and part-level masks from unlabeled videos.
2. Learning objectness from motion. The Perceptual Grouping Model (PGM), trained with Perceptual Grouping Contrastive Learning (PGCL), transfers sparse motion supervision into an appearance-driven objectness prior.
3. Adaptation to Segment Anything. PGM generates pseudo-labels on static images, which are used to train whole-image and point-promptable segmentation models.

This release contains the following two source directories:

Directory	Contents
PGM/	PGM training and single-image sliding-window inference with DenseCRF refinement
Adaption_to_Segment_Anything/	Whole-image and point-promptable adaptation, based on the training and evaluation pipeline of UnSAM

Highlights

• Fully unsupervised: no human-provided labels, boxes, or masks are used for training.
• Scalable motion supervision: MGMS produces approximately 21 million multi-granularity pseudo-labels from 10,000 hours of unlabeled video.
• Generalizable perceptual grouping: PGCL learns object coherence beyond moving regions and transfers to unseen static objects and domains.
• Multi-granularity segmentation: MoSA handles objects, instances, and fine-grained parts in automatic and promptable settings.

Installation

The reference environment uses PyTorch 2.2.1, torchvision 0.17.1, CUDA 12.1, and a Detectron2 0.6 build compiled for that PyTorch/CUDA combination. Install PyTorch first so that CUDA-dependent packages are compiled against the correct version.

conda create -n mosa python=3.10 -y
conda activate mosa

pip install torch==2.2.1 torchvision==0.17.1 \
  --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

Install Detectron2 and the COCO panoptic API separately:

pip install 'git+https://github.com/facebookresearch/detectron2.git@v0.6'
pip install 'git+https://github.com/cocodataset/panopticapi.git'

Finally, compile the multi-scale deformable-attention CUDA operator. The two adaptation branches use the same extension name, so one build is sufficient.

REPO_ROOT=$(pwd)
cd Adaption_to_Segment_Anything/whole_image_segmentation/mask2former/modeling/pixel_decoder/ops
sh make.sh
cd "$REPO_ROOT"

Checkpoints

Download the pretrained weights from Hugging Face and place them at the following paths:

checkpoints/
├── pgm.pth
├── moby_swin_t_300ep_pretrained.pth
├── promptable_segmentation.pth
└── whole_image_segmentation.pth

Perceptual Grouping Model

PGM training data

PGM is trained from RGB video frames and motion pseudo-masks generated by MGMS. Frames and packed mask files are matched by basename:

/path/to/video_images/000001.jpg
/path/to/video_masks/000001.png

For M square masks of size W x W, the grayscale PNG must pack the masks vertically and therefore have shape (M * W) x W. Pixel values are divided by 255 by the loader.

Pretrained encoder

The paper uses an MAE-pretrained ViT-B/8 encoder. Supply it with --pretrained-checkpoint.

Train PGM

Run from PGM/. The paper setting uses 8 GPUs, a global batch size of 32, and 10 epochs:

cd PGM

torchrun --standalone --nproc_per_node=8 train.py \
  --video-image-dir /path/to/video_images \
  --video-mask-dir /path/to/video_masks \
  --pretrained-checkpoint /path/to/mae_vit_base_patch8.pth \
  --output-dir outputs/pgm_vitb8

Single-image PGM inference

The demo applies the paper's global-context pass and multi-scale sliding windows, mask-quality filtering, window-boundary rejection, within-/cross-window NMS, and DenseCRF refinement.

cd PGM

python demo.py \
  --image /path/to/image.jpg \
  --checkpoint ../checkpoints/pgm.pth \
  --output-dir outputs/example \
  --device cuda:0

Building the Adaptation Dataset with PGM

Following the UnSAM data preparation pipeline, select 1% of SA-1B and run the trained PGM on these images to generate pseudo-masks. Organize the resulting pseudo-labels in the UnSAM training-data format and use them to train the promptable and whole-image segmentation models below.

Adaptation to Segment Anything

The adaptation code follows the UnSAM training and evaluation pipeline. Please refer to the UnSAM repository for dataset preparation, training, and evaluation instructions, and configure the paths in the following scripts:

• Promptable segmentation training: Adaption_to_Segment_Anything/promptable_segmentation/train.sh
• Promptable segmentation evaluation: Adaption_to_Segment_Anything/promptable_segmentation/eval.sh
• Whole-image segmentation training: Adaption_to_Segment_Anything/whole_image_segmentation/train.sh
• Whole-image segmentation evaluation: Adaption_to_Segment_Anything/whole_image_segmentation/eval.sh

Main Results

Whole-image segmentation

MoSA achieves an average recall of 42.1 AR across seven benchmarks.

Method	COCO	LVIS	ADE20K	EntitySeg	SA-1B	PartImageNet	PACO	Average
CutLER	28.1	20.2	26.3	23.1	17.0	28.7	8.9	21.8
SOHES	30.5	29.1	31.1	33.5	33.3	36.0	17.1	30.1
UnSAM	40.5	37.7	35.7	39.6	41.9	51.6	27.5	39.2
MoSA (ours)	43.5	42.2	38.4	41.1	48.2	52.7	28.4	42.1

All values are Average Recall. MoSA uses a ResNet-50 backbone and 1% of SA-1B images during adaptation.

Point-promptable segmentation

Results on COCO Val2017 use a Swin-Tiny backbone and 1% of SA-1B images.

Method	Supervision	MaxIoU	OracleIoU
SAM	Supervised	52.1	68.2
UnSAM	Not strictly unsupervised*	40.3	59.5
MoSA (ours)	Unsupervised	41.6	63.4

_{* UnSAM uses CascadePSP, which is trained with manual annotations, for mask refinement.}

Acknowledgements

The adaptation implementation is derived from the UnSAM pipeline and builds on Detectron2, Mask2Former, Semantic-SAM, MaskDINO, OpenSeed, Deformable DETR, and their dependencies. Please retain the upstream notices and follow the licenses and citation requirements of the corresponding projects.

Updates

• 2026/06: The paper was accepted to ECCV 2026.
• 2026/09: PGM and Segment Anything adaptation code released.