fix(worker): runtime orphan recovery for stuck OCR jobs

[kvithayathil]

Problem

When the worker process dies mid-execution (after setting job status to OCR_STARTED but before the OpenAI batch is submitted), the job is permanently stuck:

1. provider_job_id remains NULL — no remote batch to poll
2. Worker only picks up NOT_STARTED jobs (worker.py:174)
3. Orphan detection only runs at startup (_detect_orphaned_jobs in worker.py:159) — if the server stays up, the stuck job is never recovered
4. The orphan detector terminates stuck jobs to error states rather than retrying them

Reproduction

1. Start a job with a real OpenAI provider
2. Kill the worker process after job transitions to OCR_STARTED but before batch submission completes
3. If server stays up: job stuck forever in OCR_STARTED
4. If server restarts: orphan detector moves it to OCR_FAILED (data lost, no retry)

Impact

• Jobs can get stuck in OCR_STARTED, OCR_COMPLETED, MATCHING_PENDING, or MATCHING states with no recovery path while the server is running
• The only recovery is a server restart, which terminates the job to an error state rather than retrying

Suggested Fix

Add periodic runtime orphan detection to the worker loop — every N seconds check for jobs stuck in non-terminal states beyond a threshold, and either retry (reset to NOT_STARTED) or terminate gracefully.

Alternatively, add a POST /api/jobs/{id}/retry endpoint that resets a stuck job.

Related

• Discovered during session fix: address OS-level CVEs in Docker base image (ncurses, systemd) #16 while investigating job 1 stuck at OCR_STARTED

[kvithayathil]

Broader Scope: Graceful Interruption Recovery

This issue also covers the general case of the worker being interrupted at any point in the job pipeline. The worker currently has no crash-consistency guarantees — it mutates DB state incrementally across multiple steps, and an interruption at any point leaves partial state:

Vulnerable Interruption Points

Phase	Step	State Left Behind	Recovery Needed
Job start	Set OCR_STARTED, create OcrJob	Job in OCR_STARTED, no provider batch	Reset to NOT_STARTED or terminate
Batch OCR	create_batch_job succeeds, polling starts	provider_job_id set, worker dies mid-poll	Resume polling or fetch latest status
Batch OCR	get_ocr_results partially consumed	Some OcrResult rows written, rest lost	Resume from last processed page or restart
Sequential OCR	Processing crops one-by-one	Partial OcrResult rows	Resume from first missing crop
Matching	Processing OCR results one-by-one	Partial MatchResult rows	Resume from first unprocessed result
Finalization	Writing MATCHING_COMPLETED	All work done but status not set	Just flip status

Principles for Recovery

1. Idempotent operations — re-processing an already-processed item should be a no-op (check before write)
2. Atomic state transitions — consider wrapping multi-step mutations in a transaction or using a "claim" model
3. Checkpointing — track progress within a phase (e.g., ocr_pages_processed, matching_results_processed) so the worker can resume rather than restart
4. Retry over terminate — prefer resetting to the start of the current phase over failing the entire job