SAMURAI Tracker

Object tracking using Grounding DINO for text-based detection and SAMURAI (SAM2 with motion-aware memory) for video segmentation.

Features

• Text-based detection: Describe what you want to track in natural language
• Motion-aware tracking: SAMURAI uses Kalman filtering for robust tracking through occlusions
• Batch processing: Process entire video directories automatically
• Periodic re-detection: Catches new objects entering the scene

Installation

Prerequisites

• NVIDIA GPU with CUDA support (recommended)
• Anaconda or Miniconda

Step 1: Create and activate conda environment

conda create -n samurai_tracker python=3.10 -y
conda activate samurai_tracker

Step 2: Install PyTorch with CUDA

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Step 3: Install Python dependencies

pip install opencv-python matplotlib tqdm transformers huggingface_hub supervision hydra-core iopath loguru

Step 4: Clone and install SAMURAI (SAM2 fork)

git clone https://github.com/yangchris11/samurai.git
cd samurai/sam2
pip install -e .
cd ../..

Step 5: Download SAM2 checkpoints

Linux/Mac:

cd samurai/sam2/checkpoints
./download_ckpts.sh
cd ../../..

Windows (PowerShell):

cd samurai/sam2/checkpoints
Invoke-WebRequest -Uri "https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_base_plus.pt" -OutFile "sam2.1_hiera_base_plus.pt"
cd ../../..

Note: Only sam2.1_hiera_base_plus.pt is required. The Linux script downloads all sizes but you only need this one.

Usage

Track objects in a frame sequence

python samurai_tracker.py --input_dir path/to/frames --text_prompt "fish"

Batch process multiple videos

python batch_samurai_videos.py \
    --input_root /path/to/videos \
    --output_root /path/to/output \
    --text_prompt "turtle"

All options

Argument	Description	Default
--input_dir	Directory containing input frames	Required
--text_prompt	Text description of object(s) to track	Required
--output_dir	Directory to save outputs	output
--models_dir	Directory to cache model weights	models
--box_threshold	Detection confidence threshold	0.30
--text_threshold	Text matching threshold	0.25
--max_image_size	Maximum image dimension	1280
--redetect_interval	Re-run detection every N frames	30
--device	Device to use (cuda/cpu)	Auto-detect
--no-samurai	Disable SAMURAI, use vanilla SAM2	Enabled

SAMURAI vs Vanilla SAM2

By default, SAMURAI mode is enabled which uses Kalman filtering for motion-aware tracking. This is better for tracking through occlusions but tracks one object at a time when multiple are detected.

Use --no-samurai to disable SAMURAI mode and use vanilla SAM2:

• Faster when tracking multiple objects simultaneously
• No motion prediction (may lose objects through occlusions)

# SAMURAI mode (default) - better tracking, slower with multiple objects
python samurai_tracker.py --input_dir frames/ --text_prompt "fish"

# Vanilla SAM2 - faster multi-object, no motion prediction
python samurai_tracker.py --no-samurai --input_dir frames/ --text_prompt "fish"

Tips for text prompts

• Use periods to separate multiple objects: "fish. turtle. dolphin"
• Try variations if detection fails: "fish" vs "a fish"
• Lower --box_threshold to 0.15 if objects are missed

Output

output/
├── visualizations/         # Annotated frames with masks
│   ├── vis_000000.jpg
│   └── ...
├── tracked_output.mp4      # Final output video
└── tracking_metadata.json  # Object IDs per detection interval

Troubleshooting

CUDA Out of Memory: Reduce --max_image_size to 720 or use --device cpu

No Objects Detected: Lower --box_threshold to 0.15-0.20

Import Errors: Ensure environment is active and SAMURAI installed:

conda activate samurai_tracker
cd samurai/sam2 && pip install -e .

Credits

• Grounding DINO - IDEA Research
• SAMURAI - University of Washington
• SAM2 - Meta AI