Skip to content

CONVERSATION_SUMMARY.md

Latest commit

 

History

History
432 lines (317 loc) · 25.5 KB

CONVERSATION_SUMMARY.md

File metadata and controls

432 lines (317 loc) · 25.5 KB
Raw

PhotoCuller — Conversation Summary

This document captures the full build journey of the PhotoCuller macOS app across multiple conversation sessions — from initial planning through performance optimization.

Session 1: Planning & Core Build

Chat: PhotoCuller Native Build

Starting Point

  • User had an existing React/TypeScript web app (~/Scripts/Claude_test) for photo culling
  • Web app worked but had critical limitations:
    • Browser cannot natively decode RAW files (NEF, CR2, etc.)
    • Thumbnails were blurry due to memory constraints
    • No real file system access (delete/move was sandboxed)
  • Goal: Build a native macOS app replicating and improving the web version

Clarification Questions & Decisions

Question Decision
Project setup (Xcode vs SPM) No preference → chose SPM for simplicity
MVP vs full features Full feature parity with web app + improvements
Extra features Blur detection, exposure scoring, EXIF metadata display
Architecture MVVM (best for single-user SwiftUI app)
macOS target macOS 15 Sequoia
Auto-marking behavior Auto-mark best by quality, but display in file order
Photo volume 500–2000 photos per session

Implementation Plan

Built incrementally in 8 phases, each producing a working app:

  1. Project scaffold + Models + Folder scanning
  2. Image loading + Thumbnails (including RAW via CIRAWFilter)
  3. pHash engine + Similarity grouping
  4. Keep/Reject UI + ComparisonView
  5. Quality scoring (blur + exposure) + Auto-marking
  6. EXIF metadata reading + MetadataPanel
  7. File operations (trash, move, export JSON)
  8. Polish (keyboard shortcuts, progress indicators, edge cases)

Issues Encountered & Fixes

1. Swift tools version error

  • Problem: Package.swift used swift-tools-version: 5.9 with .macOS(.v15), which requires 6.0
  • Fix: Changed to swift-tools-version: 6.0

2. Swift 6 concurrency warnings

  • Problem: Multiple warnings about mutable global state, actor-isolated calls, and Sendable conformance
  • Fix: Systematic approach:
    • All service singletons → final class X: @unchecked Sendable
    • Photo → @unchecked Sendable (contains NSImage which isn't Sendable)
    • PhotoMetadata, PhotoStatus → Sendable
    • FileOperationsService → @MainActor (uses NSOpenPanel/NSSavePanel)
    • Photo processing → nonisolated static method to avoid main-actor isolation

3. Info.plist warning

  • Problem: SPM warned about unhandled Info.plist file in sources
  • Fix: Added exclude: ["Info.plist"] to the target in Package.swift

4. App not launching as GUI

  • Problem: swift run built and ran the binary but no window appeared (SwiftUI needs a proper app bundle to show GUI)
  • Fix: Created build_app.sh script that:
    • Runs swift build
    • Creates a .app bundle directory structure
    • Copies the binary into Contents/MacOS/
    • Writes a minimal Info.plist into Contents/
    • Launches with open

5. await warning on synchronous function

  • Problem: processPhotoStatic was called with await inside TaskGroup but is synchronous
  • Fix: Removed await — the function was already nonisolated

Post-Core Features (Still Session 1)

After the core app was working, several feature requests were made:

ComparisonView generalization

  • Request: Enable ComparisonView for "All Photos" and "Unique Photos" tabs, not just duplicate groups
  • Change: Added comparisonPhotoIDs: [UUID]? to ViewModel; generalized openComparison to accept either a groupID or a direct UUID array
  • Impact: AllPhotosView and UniquePhotosView now pass their photo lists to the comparison view

Arrow key navigation fix

  • Problem: Arrow keys didn't work in ComparisonView
  • Fix: Added @FocusState + .focusable() + .focused() + .onAppear { isFocused = true } to ensure keyboard focus was captured

Delete functionality

  • Request: Add delete button + keyboard delete key in ComparisonView
  • Initial problem: Delete key not working
  • Fix: Added multiple .onKeyPress handlers for .delete, .deleteForward, and explicit character set \u{08}\u{7F} to catch macOS's various delete key codes
  • UX improvement: One-time confirmation dialog — first delete shows alert, after confirming, subsequent deletes skip the prompt (skipDeleteConfirm state flag)

Kept photos actions

  • Request: Add same action menus (Move to Folder, Move to Trash, Export JSON) for Kept photos, not just Rejected
  • Change: Added keptPhotos computed property, trashKept(), moveKept(), exportKeptJSON() methods, and a "Kept" menu in ToolbarView

Advanced quality metrics research

  • Discussion: Researched what professional photo culling software considers:
    • Face detection + count
    • Eye blink detection
    • Smile detection
    • Face-region sharpness (more important than whole-image sharpness for portraits)
    • ISO-based noise penalty
  • Decision: Implement all of these using CIDetector (no external dependencies needed)
  • Quality formula change: Two variants — with-faces (face-aware weights) and without-faces (landscape/general weights)

Quality metrics implementation

  • Extended QualityAnalyzer with:
    • analyzeFaces() — CIDetector for face count, eye blink (leftEyeClosed/rightEyeClosed), smile
    • computeRegionSharpness() — Laplacian variance on cropped face bounding box
    • computeNoisePenalty() — ISO mapping (≤400 → 0, linear to 12800 → 1.0)
  • Extended Photo model with new fields
  • Updated MetadataPanel to display face count, face sharpness, eyes open ratio, smiles
  • Dynamic quality formula based on face presence

Session 2: Polish Features

Chat: Polish Features

Features Requested

  1. Remember last folder — auto-reopen on launch
  2. Sort options — name, date, quality, size
  3. Batch selection — Cmd+Click, Shift+Click in grid and thumbnail strip
  4. Per-photo scan progress — show filename + count

Implementation Details

1. Remember Last Folder

  • Approach: Security-scoped bookmark stored in UserDefaults
  • Key: "lastOpenedFolderBookmark" for bookmark data, fallback "lastOpenedFolderBookmarkPath" for plain path
  • Behavior: On init, ViewModel calls restoreLastFolder() which resolves the bookmark, starts security-scoped access, sets folderURL, and triggers scanFolder()
  • Edge case: Falls back to plain file path if bookmark creation fails

2. Sort Options

  • Added: SortOrder enum (.name, .date, .quality, .size) to ViewModel
  • Computed properties: sortedPhotos and sortedUniquePhotos that apply the sort
  • UI: Segmented picker in ToolbarView (4-segment, 260px width)
  • Views updated: AllPhotosView and UniquePhotosView now use vm.sortedPhotos / vm.sortedUniquePhotos

3. Batch Selection

  • ViewModel additions:
    • selectedPhotoIDs: Set<UUID> — tracks selected photos
    • toggleSelection(for:) — Cmd+Click handler
    • selectRange(from:to:in:) — Shift+Click handler (selects all photos between anchor and target)
    • setStatusForSelected(to:) and deleteSelected() — batch actions
    • clearSelection() — deselect all
  • PhotoCardView changes:
    • Detects NSEvent.modifierFlags on tap: .command → toggle select, .shift → range select, plain → open comparison
    • Blue border + checkmark overlay when selected
    • Context menu includes batch actions when selection is active
    • New onShiftClick callback parameter
  • ComparisonView thumbnail strip:
    • Same Cmd+Click (toggle) and Shift+Click (range from current photo) behavior
    • Blue border + mini checkmark for selected thumbnails
    • Toolbar shows selection count + batch action buttons when selection exists
  • ToolbarView:
    • Shows selection actions bar (count + Keep/Reject/Delete/Clear buttons) when photos are selected
  • Grid views: Track lastClickedIndex state for Shift+Click range anchor

4. Per-Photo Progress

  • ViewModel additions: scanCurrentFile, scanProcessedCount, scanTotalCount published properties
  • Scan loop: Updates these after each batch completes
  • Status message: Now shows "Processing 42/500..." instead of generic "Processing 500 photos..."
  • ContentView scanningView: Displays processed / total count, current filename, and status message

Architecture Decisions Log

Decision Choice Rationale
Build system SPM (not Xcode project) Simpler, no .xcodeproj file management
swift-tools-version 6.0 Required for .macOS(.v15) platform
Concurrency safety @unchecked Sendable on services Services have CIContext (thread-safe in practice but not marked Sendable)
Photo ID references DuplicateGroup.photoIDs: [UUID] Avoids duplicating Photo structs; lookup by ID in main array
DCT implementation Manual (not vDSP) More control over the exact algorithm matching the web app's behavior
Quality scoring Adaptive formula (with/without faces) Prevents face metrics from penalizing landscape photos
App launch method Custom build_app.sh swift run doesn't create proper app bundle for SwiftUI GUI
Folder persistence Security-scoped bookmark Required for sandboxed access; falls back to plain path
Selection model Set<UUID> in ViewModel Simple, O(1) lookup, works across views
Sort implementation Computed properties, not mutating sort Preserves original photo array order; sort only affects display

Files Modified Per Feature

Core Build (Session 1)

All files created from scratch:

  • Package.swift, build_app.sh, Info.plist, PhotoCullerApp.swift
  • All Models, Services, ViewModels, and Views

ComparisonView Generalization

  • PhotoLibraryViewModel.swift — added comparisonPhotoIDs, generalized openComparison
  • ComparisonView.swift — removed group parameter, uses vm.comparisonPhotos
  • AllPhotosView.swift, UniquePhotosView.swift — changed onTap to use openComparison(photoIDs:)
  • ContentView.swift — uses vm.isComparisonOpen instead of checking group

Delete Functionality

  • ComparisonView.swift — delete button, keyboard handlers, one-time confirm dialog
  • PhotoLibraryViewModel.swift — deletePhoto(id:) method

Kept Photos Actions

  • PhotoLibraryViewModel.swift — keptPhotos, trashKept(), moveKept(), exportKeptJSON(), removePhotos(ids:) helper
  • ToolbarView.swift — "Kept" menu with 3 actions

Advanced Quality Metrics

  • Photo.swift — new fields: faceCount, eyesOpenRatio, faceSharpness, noisePenalty, hasSmiles
  • QualityAnalyzer.swift — analyzeFaces(), computeRegionSharpness(), computeNoisePenalty(), updated formula
  • PhotoLibraryViewModel.swift — passes ISO to analyzer, stores new scores
  • MetadataPanel.swift — displays face section with all new metrics

Session 2 Features

  • PhotoLibraryViewModel.swift — remember folder, sort order, selection state, scan progress properties
  • ContentView.swift — per-photo progress display
  • ToolbarView.swift — sort picker, selection action bar
  • PhotoCardView.swift — selection UI (blue border, checkmark), modifier key detection, onShiftClick
  • AllPhotosView.swift — sorted display, shift-click anchor tracking
  • UniquePhotosView.swift — sorted display, shift-click anchor tracking
  • ComparisonView.swift — thumbnail strip selection, batch actions in toolbar

Session 3: Performance Debugging & Two-Phase Processing

Chat: NEF Performance Fix

Problem

Scanning a folder of 94 JPG photos hung for 15+ minutes. With 7 photos it worked fine.

Debugging Round 1: CIDetector Hang (JPG)

Instrumentation: Added per-step timing logs (step_thumbnail, step_hash, step_metadata, step_quality) and sub-step quality logs (qa_blur, qa_exposure, qa_face_start, qa_face_done).

Finding: CIDetector face detection with CIDetectorAccuracyHigh on full-resolution images hung when 8 tasks ran concurrently. Logs showed qa_face_start but never qa_face_done.

Fix:

  • Downscale input image to 1024px max before CIDetector in analyzeFaces()
  • Reduce batch size from 8 to 4
  • Changed CIDetectorAccuracyHigh → CIDetectorAccuracyLow (later reverted for background processing)

Result: 94-photo JPG folder completed successfully.

Two-Phase Processing Architecture

Motivation: Even with the CIDetector fix, quality analysis was the dominant cost. User wanted the UI responsive quickly with quality computed in the background.

Implementation:

  • Phase 1 (Fast): Thumbnail + metadata + pHash only → UI becomes interactive, duplicate groups shown
  • Phase 2 (Background): Quality analysis runs as a Task in the background, scores update progressively, auto-marking happens after Phase 2 completes
  • CIDetectorAccuracyHigh restored for Phase 2 (safe since it runs with batch size 4, not 8)

Debugging Round 2: NEF Performance (1700+ files)

Problem: Two-phase processing worked on JPG folders, but a 1737-NEF folder stuck on the first batch of Phase 1.

Instrumentation: Added per-photo timing in processPhase1() — p1_thumb, p1_meta, p1_preview, p1_hash, p1_done.

Issue 1: Double file decode

Finding: Each NEF was opened via CGImageSource twice — once for display thumbnail (300px) and once for hash preview (512px). Each decode took 500–1300ms.

Fix: Created loadFastThumbnailAndPreview() in ImageLoaderService — opens CGImageSource once, extracts preview (512px), derives thumbnail from same source. Cut I/O in half.

Result: All 1737 photos completed Phase 1, but still took ~9 minutes (~535 seconds).

Issue 2: Full RAW decode for thumbnails

Finding: kCGImageSourceCreateThumbnailFromImageAlways was forcing a full RAW decode for every NEF file even though NEF files embed a JPEG preview. Each decode: 500–1300ms.

Fix: For RAW files, switched to kCGImageSourceCreateThumbnailFromImageIfAbsent which extracts the embedded JPEG preview directly (no RAW decode). Per-NEF time dropped from 500–1300ms to 50–230ms.

Result: Phase 1 for 1737 NEF files: ~36 seconds (down from ~9 minutes, originally 15+ minutes).

Performance Summary

Stage Before After
Phase 1 (1737 NEFs) 15+ min (hung) ~36 seconds
Per-NEF thumbnail+hash 1000–2600ms (double decode) 50–230ms (embedded JPEG)
Phase 1 batch size 10 → 6 6
Quality analysis Blocked UI Background (Phase 2)

Files Modified

  • ImageLoaderService.swift — Added loadFastThumbnailAndPreview() (single-pass extraction), FastLoadResult struct; uses kCGImageSourceCreateThumbnailFromImageIfAbsent for RAW files
  • PhotoLibraryViewModel.swift — Refactored scanFolder() into Phase 1 (fast) + Phase 2 (background quality); added processPhase1() static method, startBackgroundQualityAnalysis(), qualityTask, progress tracking (isAnalyzingQuality, qualityAnalyzedCount); batch size reduced to 6
  • HashingEngine.swift — computeHash() now accepts CIImage directly (avoids reloading)
  • QualityAnalyzer.swift — analyze() accepts CIImage directly; analyzeFaces() downscales to 1024px before CIDetector; CIDetectorAccuracyHigh used (safe in Phase 2 with batch size 4)
  • ContentView.swift — Added quality progress bar showing background analysis progress

Architecture Decisions Log (Continued)

Decision Choice Rationale
Two-phase scan Phase 1 (UI) + Phase 2 (quality) User sees photos immediately; quality computed in background
RAW thumbnail extraction kCGImageSourceCreateThumbnailFromImageIfAbsent Extracts embedded JPEG from NEF/CR2 files instead of decoding full RAW (~20x faster)
Single-pass file load loadFastThumbnailAndPreview() Opens CGImageSource once per file, produces both thumbnail and preview for hashing
Phase 1 batch size 6 Balances parallelism with disk I/O contention for large RAW files
Phase 2 batch size 4 Prevents CIDetector from hanging under high concurrency
CIDetector downscale 1024px before face detection Prevents hang on full-resolution images; accuracy preserved

Session 4: Similarity Matrix, Memory Management & Performance

Chat: Similarity Matrix & LRU Cache

Problem 1: Slow Re-Grouping on Threshold Change

When the user adjusted the similarity threshold slider, the app re-ran the full O(n²) pairwise comparison every time. With 1600+ photos, this caused noticeable lag.

Options Considered

  1. Re-run O(n²) each time — simple but slow (~seconds per change for 1600 photos)
  2. Precompute similarity matrix + filter on threshold change — compute all pairwise similarities once, then filter edges

Decision: Precomputed Similarity Matrix + Union-Find

  • buildSimilarityMatrix() runs once after Phase 1 scan completes, producing a SimilarityMatrix struct containing all SimilarityEdge pairs with ≥50% similarity (the minimum possible slider value)
  • groupFromMatrix() uses Union-Find with path compression to regroup from the precomputed edges, filtering by the current threshold — essentially instant
  • Burst relaxation (burstGap=3): When two photos are similar and within 3 positions of each other by filename order, all intermediate photos are merged into the same group. This handles camera burst sequences where alternating frames may be similar to each other but not to every other frame
  • Threshold slider fires onThresholdChanged() on release (not during drag), which runs groupFromMatrix() on a background thread with isGrouping flag
  • Groups are sorted by first photo name for stable display order

Files Modified

  • SimilarityEngine.swift — Added SimilarityMatrix, SimilarityEdge structs; buildSimilarityMatrix() for one-time O(n²) precompute; groupFromMatrix() for instant Union-Find regrouping with burst relaxation
  • PhotoLibraryViewModel.swift — Stores similarityMatrix after Phase 1; regroupFromMatrix() and onThresholdChanged() for threshold slider; isGrouping published state; background thread dispatch

Problem 2: Cross-Group Navigation

User needed to compare images across different duplicate groups since grouping is imperfect. Added Up/Down arrow keys and chevron buttons in ComparisonView toolbar to jump between groups.

Files Modified

  • PhotoLibraryViewModel.swift — comparisonGroupIndex, canNavigateToPrevGroup, canNavigateToNextGroup, navigateToPrevGroup(), navigateToNextGroup()
  • ComparisonView.swift — .onKeyPress(.upArrow), .onKeyPress(.downArrow), group navigation buttons in toolbar with group counter display

Problem 3: Unbounded Memory in ComparisonView (22GB+)

The fullImages: [UUID: NSImage] dictionary in ComparisonView grew without bound. Each full NEF decode takes ~10–25MB. Navigating through groups accumulated decoded images, reaching 22GB+ and causing heavy disk swap (~5GB increase in macOS storage from swap files).

Options Considered for Memory Management

  1. LRU cache with fixed image count cap — keep max N most-recently-used images, evict oldest when cap reached
  2. NSCache — Foundation's auto-evicting cache, drops items under memory pressure automatically
  3. Two-tier cache — current group at full resolution, previous groups downscaled to 1024px
  4. Group-based cap — cache full images for last N groups (e.g., last 5 groups)

Decision: LRU Cache with 40-Image Cap

  • Why LRU over NSCache: NSCache's eviction is unpredictable and can drop items mid-use. LRU gives deterministic behavior with a predictable memory ceiling.
  • Why 40 images: User stated max 20 images needed for cross-group reference, 40 provides generous buffer. Worst-case memory: ~1GB (40 × 25MB) vs unbounded 22GB.
  • Why not two-tier: Added complexity of downscaling, cache invalidation, and resolution switching wasn't justified for the use case. User explicitly preferred simplicity.
  • Why not group-based cap: Groups vary in size; image count cap is more predictable for memory.
  • Implementation: imageAccessOrder: [UUID] array tracks recency. cacheImage() appends + evicts oldest. touchImage() bumps to most-recent on navigation. evictIfNeeded() trims when count > 40.

Swap/Disk Space Issue

User observed ~5GB macOS storage increase after using the app (before LRU fix). This was confirmed as swap files from the 22GB RAM usage. Resolution: reboot clears swap; the LRU fix prevents future swap accumulation.

Problem 4: ComparisonView Hangs When Opened from All Photos

Opening a photo from "All Photos" tab caused the app to hang for 30+ seconds (sometimes 2+ minutes), while opening the same photo from Duplicates tab was instant.

Root Cause (Confirmed by Debug Instrumentation)

Two compounding issues discovered via runtime logs:

Issue A — O(n²) comparisonPhotos lookup (610ms × 100+ calls): comparisonPhotos computed property did ids.compactMap { id in photos.first { $0.id == id } }. With 1632 photos, each call was O(1632 × 1632) = 2.6M iterations, taking ~610ms. SwiftUI called this 100+ times during render cycles, accumulating ~60+ seconds of main-thread time.

Issue B — Eager full-image loading (loadAllFullImages): The old approach tried to decode every photo in the list sequentially before the view became responsive. For "All Photos" with 1632 NEFs, this meant decoding 1632 images.

Issue C — Eager thumbnail strip (HStack with 1632 views): The thumbnail strip used HStack which created all 1632 views immediately.

Issue D — Array index out-of-bounds crash: photos was a computed property re-evaluated on every access. In loadWindowAroundCurrent, between capturing count = photos.count and accessing photos[i], the underlying data could change (deletion, group navigation), causing a crash. Both crash reports showed identical stack traces at ComparisonView.loadWindowAroundCurrent() line 540.

Fixes Applied

Fix Before After
comparisonPhotos lookup O(n×m) linear scan, ~610ms/call O(n) dictionary lookup via photosByID, ~2ms/call (300× faster)
Full image loading loadAllFullImages() — decode all photos loadWindowAroundCurrent() — current photo first, then ±5 nearby
Thumbnail strip HStack — 1632 views created eagerly LazyHStack — only visible views created, .frame(height: 76)
Crash fix photos re-evaluated on each access let photosCopy = photos snapshot once at function entry

Debug Methodology

  • Added NDJSON file-based logging to comparisonPhotos (timing per call, count), loadWindowAroundCurrent (entry/exit timing, index, cache size), loadSingleImage (per-image decode timing), and thumbnailStrip (render count)
  • First run logs showed: 100 calls to comparisonPhotos at 610ms each, first loadWindowAroundCurrent took 137 seconds total, first image decode 8.7 seconds (contending with background quality analysis CIRAWFilter)
  • Post-fix logs confirmed: comparisonPhotos dropped to ~2ms/call, total function time dropped proportionally

Files Modified

  • PhotoLibraryViewModel.swift — Added photosByID computed dictionary; comparisonPhotos and photosForGroup() now use dictionary lookup
  • ComparisonView.swift — LRU cache state (imageAccessOrder, maxCachedImages=40), cacheImage/touchImage/evictIfNeeded helpers, loadWindowAroundCurrent() with snapshot + prefetch window, LazyHStack with fixed height, removed loadAllFullImages()

Architecture Decisions Log (Continued — Session 4)

Decision Choice Rationale
Similarity matrix Precompute once, Union-Find regroup Threshold slider changes are instant; no O(n²) recomputation
Burst relaxation burstGap=3 in Union-Find Handles camera bursts where alternating frames match
Image memory management LRU cache, 40-image cap Predictable ~1GB ceiling vs unbounded 22GB; simpler than NSCache or two-tier
Full image loading Window-based prefetch (current + ±5) Instant first render; background prefetch for smooth navigation
Photo lookup [UUID: Photo] dictionary O(1) vs O(n) per lookup; 300× speedup for 1600+ photos
Thumbnail strip LazyHStack + fixed height Only visible thumbnails rendered; explicit height prevents layout issues
Crash prevention Snapshot photos array once Computed property re-evaluation during async work caused index out-of-bounds

Potential Future Work (Discussed but Not Implemented)

  • Apple Vision framework — scene classification, object detection for smarter categorization
  • Undo for file deletion — maintain a deletion history to allow reversal
  • Thumbnail caching — persist thumbnails to disk to avoid re-generating on each launch
  • SQLite-based result caching — cache quality scores to skip reanalysis on re-open (discussed, deferred due to cache invalidation complexity)
  • Keyboard shortcut overlay — help dialog showing available shortcuts
  • Multi-select drag — drag to select a rectangular region of photos
  • Star rating — 1–5 star rating in addition to keep/reject
  • Color label / tags — categorize photos beyond binary keep/reject
  • LSH for large libraries — locality-sensitive hashing for O(n) duplicate detection at 10K+ scale