🎬 dedupr

GPU-Accelerated, Mirror-Aware Video Deduplication & Smart Library Organizer

100% Local • Zero Cloud Egress • Sub-Second Rescans with SQLite Caching

---

⚡ Why dedupr?

Finding duplicate videos is hard when files have been:

• 🔄 Horizontally flipped / mirrored (common on social media platforms)
• 📐 Rescaled to different resolutions (e.g. 4K original vs 1080p / 720p clips)
• 🗜️ Re-encoded with different bitrates or codecs (H.264, HEVC, AV1, VP9)
• ⏱️ Slightly trimmed or adjusted

Traditional byte-hashing (MD5/SHA256) fails on all of these. dedupr extracts normalized perceptual keyframe hashes across both standard and mirrored orientations, detects near-duplicates with sub-frame accuracy, elects the best master video, and organizes your media library according to customizable metadata rules.

---

🚀 Key Features

• 🪞 Mirror & Flip Invariant: Detects videos even when flipped horizontally.
• ⚡ GPU & Multi-Core Acceleration: Leverages your NVIDIA RTX GPU (CUDA / NVDEC / D3D11VA) and multi-threaded CPU worker pools for lightning-fast scanning.
• 👑 Automatic Master Election: Elects the highest quality copy (best resolution, highest bitrate, original orientation) and isolates lower-quality dupes to a quarantine folder.
• 📁 Smart Template Routing: Dynamically organize collections using rich metadata templates: {resolution}/{codec}/{year}-{month} or custom regex group patterns.
• ⚡ Instant Incremental Rescans: SQLite database caching ensures already-processed videos are skipped on subsequent runs.
• 🛡️ Safe Dry-Run Simulation: Preview every duplicate match and file action before writing a single byte to disk.
• 🔒 100% Local & Private: No cloud APIs, zero telemetry, no network calls.

---

🏗️ Architecture & Pipeline

flowchart LR
    A["🎥 Video Library"] --> B["⚡ Multi-Threaded Scanner"]
    B --> C["🔍 Metadata & GPU Decoder"]
    C --> D[("💾 SQLite Cache (.dedupr.db)")]
    C --> E["🖼️ Dual-Orientation Perceptual Fingerprinter"]
    E --> F["📊 Duplicate Clusterer & Master Election"]
    F --> G["🛡️ Quarantine Duplicates"]
    F --> H["🗂️ Template & Regex Router"]
    H --> I["✨ Clean Organized Media Library"]

Loading

---

📦 Installation

Prerequisites

• Python 3.10+
• FFmpeg (recommended for GPU HW acceleration)

Install via pip / git

# Clone the repository
git clone https://github.com/kaiser62/dedupr.git
cd dedupr

# Create and activate virtual environment
python -m venv .venv

# Windows (PowerShell)
.\.venv\Scripts\Activate.ps1

# Linux / macOS
source .venv/bin/activate

# Install in editable mode
pip install -e .

---

📖 CLI Usage & Examples

1. 🔍 Scan & Cache Fingerprints

Fast multi-threaded scan of your video library. Fingerprints are indexed in .dedupr.db.

# Basic scan (auto-detects CPU cores & GPU)
dedupr scan "C:/Videos"

# High-performance scan with 16 workers and CUDA hardware decoding
dedupr scan "C:/Videos" --workers 16 --hwaccel cuda

---

2. ✂️ Find & Quarantine Duplicates

Scans for duplicate groups (exact, resized, mirrored, re-encoded) and isolates lower quality copies into duplicates/.

# 1. Preview duplicate clusters safely without moving any files
dedupr dedup "C:/Videos" --dry-run

# 2. Execute deduplication and quarantine duplicates
dedupr dedup "C:/Videos" --workers 8 --hwaccel cuda

# Export duplicate report to JSON
dedupr dedup "C:/Videos" --json > duplicates_report.json

📊 Example Duplicate Group Output:

┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃ Status              ┃ Filename           ┃ Relationship ┃ Resolution  ┃ Size     ┃ Bitrate  ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│ MASTER (Keep)       │ clip_master.mp4    │ Original     │ 4K (3840x2160)│ 1.25 GB  │ 45000 kbps│
│ DUPLICATE           │ clip_mirror_720.mp4│ MIRRORED     │ 720p (1280x720)│ 142 MB  │ 4200 kbps│
│ DUPLICATE           │ clip_1080p_reenc.mp4│ RESIZED     │ 1080p (1920x1080)│ 450 MB│ 8000 kbps│
└─────────────────────┴────────────────────┴──────────────┴─────────────┴──────────┴──────────┘

---

3. 🗂️ Sort & Organize Library

Organize messy video folders into structured directories using metadata templates.

# Organize by Resolution, Codec, and Date
dedupr sort "C:/Videos" "C:/SortedVideos" \
  --template "{resolution}/{codec}/{year}-{month}" \
  --mode move \
  --dry-run

Available Template Variables:

Variable	Description	Example
{resolution}	Resolution category	4K, 1440p, 1080p, 720p, 480p
{codec}	Video compression codec	h264, hevc, vp9, av1
{audio_codec}	Audio compression codec	aac, mp3, opus, none
{fps}	Frames per second	60, 30, 24
{aspect}	Aspect ratio	16-9, 9-16, 4-3, 1-1
{year}	Creation / modification year	2026
{month}	Creation / modification month	08
{day}	Creation / modification day	29
{ext}	Original file extension	mp4, mkv, mov

---

4. 🌟 Full Pipeline (all)

Runs the complete scan, deduplication, quarantine, and library organization in a single command:

dedupr all "C:/RawVideos" "C:/CleanMediaLibrary" --workers 8 --hwaccel cuda

---

🛠️ CLI Options Reference

Usage: dedupr [OPTIONS] COMMAND [ARGS]...

Commands:
  scan    Scan video files, extract metadata & perceptual fingerprints.
  dedup   Find and isolate near-duplicates (mirrored, re-scaled, re-encoded).
  sort    Sort and organize videos into structured subdirectories.
  all     Full pipeline: scan + dedup + sort clean masters.

Global Options:
  -w, --workers INTEGER   Number of worker threads (default: auto CPU count).
  -g, --hwaccel TEXT      GPU hardware acceleration: auto, cuda, d3d11va, none.
  -n, --dry-run           Simulate actions without touching disk.
  --json                  Export results as JSON.

---

🧪 Running Tests

pytest -v

---

📄 License

Distributed under the MIT License. See LICENSE for more information.

---

Built with ❤️ by kaiser62