Skip to content

how_to_find_duplicates.md

Latest commit

 

History

History
52 lines (38 loc) · 1.79 KB

how_to_find_duplicates.md

File metadata and controls

52 lines (38 loc) · 1.79 KB
Raw

Duplicate & Near-Duplicate Detection

Added two scripts in preprocess/post_anomaly_audit/:

Script Purpose
find_duplicates.py Detect duplicates using MD5 + pHash DCT, save report in JSON
quarantine_duplicates.py Move detected duplicates to TrainImages/duplicates/ based on generated JSON report

How To Use

Duplicate Detection

python3 preprocess/post_anomaly_audit/find_duplicates.py
  --data-dir <path_to_dataset>
  --output-report <path_to_report>
  --threshold <hamming_distance>
  --cross-class <trigger_cross_class_detection>
  --workers <number_of_threads>

Note

Valid hamming distance value is 0-64

Quarantine Detected Duplicates

This is a preview run (dry-run, SAFE – no files will be moved)

python3 preprocess/post_anomaly_audit/quarantine_duplicates.py
  --report <path_to_json_report>
  --data-dir <path_to_dataset>
  --quarantine-dir <path_to_quarantine_folder>
  --dry-run

Permanent Quarantine

Run this to actually move the detected dupliates

python3 preprocess/post_anomaly_audit/quarantine_duplicates.py
  --report <path_to_json_report>
  --data-dir <path_to_dataset>
  --quarantine-dir <path_to_quarantine_folder>

Important

Before running quarantine without --dry-run, make sure to review the duplicates_report.json to validate whether the threshold meets the needs. Increase the threshold for more quarantines, decrease it for more selectivity.