Rider Intelligence Bot — MVP Project Plan

A Telegram bot that lets UK delivery riders send a screenshot of a Deliveroo order offer and get back a 5-second verdict: accept / borderline / skip + reasoning, grounded in a real cycling ETA from the rider's current location.

Local-first, laptop-hosted. Parsing is OCR + deterministic field extraction; a local VLM (Qwen3-VL via Ollama) is the fallback. Routing uses Mapbox Directions API (cycling profile) — cloud but free for our volume. Personal/single-user MVP; privacy hardening deferred.

Goals & non-goals for this build

Goals

End-to-end pipeline: rider sends screenshot → bot replies with a verdict grounded in real cycling time from current location
OCR runs locally; routing uses Mapbox; no AI inference in the cloud by default
Logs every screenshot + parse + route + verdict for future tuning
Survives sleep/restart, runs on a single laptop
Codebase clean enough to migrate to a server later without rewriting

Non-goals (out of scope for MVP)

Multi-tenant privacy hardening (single user / known testers only — full postcodes stored)
User accounts / payments / web UI
Postcode/restaurant friction database (Phase 2)
Multi-platform (Uber Eats, Just Eat) — Deliveroo only
Native mobile app
Cloud hosting

Tech stack

Layer	Choice	Why
Language	Python 3.11+	Best Telegram + OCR + LLM ecosystem
Bot framework	`python-telegram-bot` v21+	Async, well-maintained, native location-sharing support
Primary parser	RapidOCR (`rapidocr-onnxruntime`)	Pure-Python, no system deps, ~200–500ms/image
VLM fallback (local)	Qwen3-VL via Ollama	Local, free, used when OCR confidence is low
VLM fallback (cloud)	`anthropic` SDK (Claude Haiku)	Switchable via env var for benchmarking
Routing	Mapbox Directions API (cycling profile)	Free 100k/mo, real cycling ETAs, reliable
Geocoding	Mapbox Geocoding API	Same vendor, postcode → lat/lng, free tier
Location source	Telegram location sharing	Native button, no Apple-specific glue, future-proof
Storage	SQLite via `sqlmodel`	Zero-config, file-based
Config	`.env` + `pydantic-settings`	Standard, typed
Logging	`structlog`	JSON logs, easy to grep
Process manager	`pm2` or plain `tmux`	Survives terminal close

Key architectural choices:

Long-polling Telegram bot — no public URL needed.
OCR first, VLM only on failure — Deliveroo's offer screen is consistent; regex + spatial rules solve the happy path in <1s.
Parser behind a single abstraction — Parser protocol with OCRParser, OllamaParser, AnthropicParser, HybridParser implementations.
Routing on real geometry — Mapbox cycling ETA from rider's current location through every pickup and dropoff is the basis of the score. No fake "ETA + buffer" arithmetic.

What Deliveroo's offer screen actually contains

Based on real screenshots: Deliveroo shows payout, restaurant address(es), and customer address(es) — and not distance or ETA. That makes our routing module non-optional.

A single offer consists of:

£X.XX headline → payout
N order(s) label → number of items across all pickups (not equal to pickup count)
One or more pickup blocks (shopping-bag icon, Nx quantity label): restaurant name + full address + postcode
A ↓ separator
One or more dropoff blocks (person icon): customer address + postcode

Trip-shape taxonomy

Shape	Pickups	Dropoffs	Notes
Single	1 restaurant	1 customer	Baseline
Multi-item, single pickup	1 restaurant, qty >1	1 customer	One bag, same effort
Stacked (classic)	2 restaurants	2 customers	Two pickup stops + two drop stops
Multi-drop from one restaurant	1 restaurant	2+ customers	One pickup, two drops
Multi-pickup to one customer	2+ restaurants	1 customer	Two pickups, one drop
Co-located drops	2 restaurants	2 customers, same building	Effectively one drop stop
Co-located pickups	2 restaurants, same food hall	2 customers	Effectively one pickup stop

Co-location is detected by postcode equality at extraction time and folded into the route as a single stop.

Architecture

┌─────────────┐
│   Rider     │
│ (Telegram)  │
└──────┬──────┘
       │ screenshot + shared location
       ▼
┌─────────────────────────────────────────────────────────────┐
│                  Bot process (laptop)                       │
│                                                             │
│  ┌──────────┐   ┌──────────────────┐                       │
│  │ Telegram │──▶│  Parser pipeline │                       │
│  │ handler  │   │  OCR → (VLM)     │                       │
│  └────┬─────┘   └────────┬─────────┘                       │
│       │ location          │ ParsedOrder                     │
│       ▼                   ▼                                 │
│  ┌──────────────┐   ┌────────────────┐   ┌──────────────┐ │
│  │ Last-known   │──▶│  Geocode +     │──▶│   Scoring    │ │
│  │ location     │   │  Mapbox route  │   │   engine     │ │
│  └──────────────┘   │  (cycling)     │   └──────┬───────┘ │
│                     └────────────────┘          │         │
│                                                 ▼         │
│                                          ┌───────────┐    │
│                                          │ SQLite DB │    │
│                                          └───────────┘    │
└─────────────────────────────────────────────────────────────┘

Logical components:

Telegram handler — receives image, receives location updates, sends reply
Parser pipeline — OCR + deterministic extraction; VLM fallback for low confidence
Geocoding + Routing — postcode → lat/lng → cycling route through all legs
Scoring engine — pure-Python rules on (payout, real_duration_minutes, stop_count, peak_hour)

Data model

class PickupLeg(BaseModel):
    restaurant_name: str | None
    address_line: str | None        # full address as OCR'd
    postcode: str | None            # full postcode, e.g. "W1B 5PJ"
    item_count: int = 1

class DropoffLeg(BaseModel):
    address_line: str | None
    postcode: str | None

class ParsedOrder(BaseModel):
    payout_gbp: float | None
    order_count_label: int | None   # the "N orders" text
    pickups: list[PickupLeg]
    dropoffs: list[DropoffLeg]
    confidence: float
    not_a_deliveroo_order: bool
    raw_ocr: list[str] | None       # for debugging

# orders table
class Order(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    telegram_user_id: int
    received_at: datetime
    image_path: str

    parsed_json: str | None         # full ParsedOrder serialised
    parser_used: str | None         # "ocr" | "ocr+ollama:qwen3-vl:4b" | ...
    parse_confidence: float | None

    # Routing result (set after parse)
    route_distance_m: float | None
    route_duration_s: float | None
    route_legs_json: str | None     # leg-by-leg breakdown
    rider_lat: float | None
    rider_lon: float | None

    # Verdict
    verdict: str | None             # "accept" | "borderline" | "skip"
    pounds_per_hour: float | None
    reasoning: str | None

    # Telemetry
    ocr_ms: int | None
    vlm_ms: int | None
    routing_ms: int | None
    total_ms: int | None
    error: str | None

class User(SQLModel, table=True):
    telegram_user_id: int = Field(primary_key=True)
    first_seen_at: datetime
    last_seen_at: datetime
    total_screenshots: int = 0

class UserLocation(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    telegram_user_id: int = Field(index=True)
    lat: float
    lon: float
    captured_at: datetime
    source: str                     # "telegram_share" | "telegram_live"

Last-known location is the most recent UserLocation row per user. Telegram supports both one-shot share and live-location streaming; we accept both.

Project structure

rider-bot/
├── .env                       # secrets (gitignored)
├── .env.example
├── .gitignore
├── README.md
├── PROJECT_PLAN.md
├── pyproject.toml
├── data/
│   ├── bot.db                 # SQLite (gitignored)
│   └── screenshots/           # raw images (gitignored)
├── src/
│   └── rider_bot/
│       ├── __init__.py
│       ├── __main__.py
│       ├── config.py
│       ├── logging_setup.py
│       ├── bot.py             # telegram handlers
│       ├── db.py
│       ├── models.py
│       ├── parser/
│       │   ├── __init__.py    # Parser protocol + factory
│       │   ├── base.py        # ParsedOrder, PickupLeg, DropoffLeg
│       │   ├── ocr.py         # RapidOCR + field extraction
│       │   ├── ollama.py
│       │   ├── anthropic.py
│       │   └── hybrid.py
│       ├── geocoding.py       # Mapbox geocoding client (postcode → lat/lng)
│       ├── routing.py         # Mapbox Directions client (cycling)
│       ├── scoring.py
│       └── prompts.py
├── tests/
│   ├── test_scoring.py
│   ├── test_parser_ocr.py
│   ├── test_parser_hybrid.py
│   ├── test_routing.py
│   └── fixtures/
│       └── screenshots/
└── scripts/
    ├── test_ocr.py            # Phase B smoke test
    ├── test_routing.py        # Mapbox smoke test
    ├── benchmark_parsers.py
    └── export_data.py

OCR parser — primary path

Engine

from rapidocr_onnxruntime import RapidOCR
engine = RapidOCR()
result, _ = engine(image_path)
# result = list of [bbox, text, confidence]

Field extraction — the new structure

Deliveroo blocks are visually grouped; OCR gives us text + bounding boxes. The extractor:

Find the payout — match £X.XX (or bare X.XX near the top if £ got dropped); pick the largest such number in the upper portion of the image.
Find the "N orders" label — N\s+order(s)? pattern.
Split pickup vs dropoff regions by the ↓ separator (a single isolated arrow block) or, failing that, by the icon glyphs (shopping bag vs person — these usually OCR to junk but their bounding boxes are reliable column anchors).
For each pickup block: nearest restaurant-name text (bold, larger font, no postcode) + the address line (contains a postcode).
For each dropoff block: the address line.
Co-location detection: pickups/dropoffs with identical postcodes count as one stop in the route.

Full UK postcode regex (kept loose because OCR sometimes drops the space):

POSTCODE_RE = re.compile(
    r"\b([A-Z]{1,2}\d[A-Z\d]?)\s?(\d[A-Z]{2})\b",
    re.IGNORECASE,
)

confidence = function of (required fields present) × (mean OCR per-block confidence). Required = at least one pickup with a postcode, at least one dropoff with a postcode, and a payout.

Hybrid parser — OCR with VLM fallback

class HybridParser:
    async def parse(self, image_path: str) -> ParsedOrder:
        parsed = await self.ocr.parse(image_path)
        if parsed.confidence >= self.floor and parsed.has_required_fields():
            return parsed.with_parser("ocr")
        vlm_parsed = await self.vlm.parse(image_path)
        return merge(parsed, vlm_parsed).with_parser(f"ocr+{self.vlm.provider_tag}")

Routing — Mapbox

Geocoding

Postcode → lat/lng via Mapbox Geocoding API. Cache per-postcode in SQLite (postcode_cache table) so repeat addresses cost nothing.

class Geocoder:
    async def geocode_postcode(self, postcode: str) -> tuple[float, float]:
        # Mapbox: /geocoding/v5/mapbox.places/{postcode}.json?country=gb&types=postcode
        ...

Cycling route

Build the ordered stop list from the rider's current location:

[rider_now, pickup_1, pickup_2, ..., dropoff_1, dropoff_2, ...]

After collapsing co-located stops, call Mapbox Directions cycling profile with all waypoints:

class Router:
    async def route(self, waypoints: list[tuple[float, float]]) -> RouteResult:
        # GET /directions/v5/mapbox/cycling/{lon1,lat1};{lon2,lat2};...
        # ?annotations=duration,distance
        ...

class RouteResult(BaseModel):
    total_distance_m: float
    total_duration_s: float
    legs: list[Leg]  # per-segment distance + duration

The score uses total_duration_s. Per-leg data is stored for later analysis (e.g. "the first pickup leg was 60% of the time").

Mapbox env config

MAPBOX_TOKEN=pk.eyJ...
MAPBOX_DIRECTIONS_PROFILE=cycling

Telegram location flow

On /start, bot replies with a ReplyKeyboardMarkup containing a KeyboardButton(request_location=True) and a brief instruction.
Rider taps "Share location" → bot stores UserLocation row.
Rider sends a screenshot → bot uses the most recent UserLocation for that user (with a freshness warning if older than e.g. 30 minutes).
Optional: /here command resends the location request anytime.
Live-location messages (Telegram's continuous-share) are accepted and update the row on each tick — best UX for an actively-riding user.

If no location is on file when a screenshot arrives: bot replies "I need your location first — tap the button below," and re-sends the request.

Scoring engine — real-ETA edition

Inputs from routing:

total_duration_s — actual cycling time end-to-end
total_distance_m
stop count = unique pickup postcodes + unique dropoff postcodes

Pure-Python rules:

def score_order(parsed: ParsedOrder, route: RouteResult, now: datetime) -> Verdict:
    base_minutes = route.total_duration_s / 60

    # Per-stop friction: each pickup/dropoff adds parking, lift, hand-off time
    stop_count = unique_pickups(parsed) + unique_dropoffs(parsed)
    friction_min = 2.5 * stop_count  # tunable

    # Peak-hour restaurant wait (only applies if any pickup is during peak)
    if is_peak_hour(now):
        friction_min += 3 * unique_pickups(parsed)

    total_minutes = base_minutes + friction_min
    pounds_per_minute = parsed.payout_gbp / total_minutes
    pounds_per_hour = pounds_per_minute * 60

    # Thresholds (initial; calibrate from feedback)
    if pounds_per_hour >= 18:
        verdict = "accept"
    elif pounds_per_hour >= 13:
        verdict = "borderline"
    else:
        verdict = "skip"

    return Verdict(
        verdict=verdict,
        pounds_per_hour=pounds_per_hour,
        total_minutes=total_minutes,
        cycling_minutes=base_minutes,
        friction_minutes=friction_min,
        reasoning=build_reasoning(...),
    )

Every constant (per-stop friction, peak bump, thresholds) is a tunable parameter as feedback arrives.

User flow

Rider opens bot → /start
Bot asks to share location (button).
Rider taps share → confirmation.
Rider sends screenshot.
Bot: "Analysing… 🔍" within 1s.
Within ~2–4s on the happy path: verdict card.
Optional: 👍 / 👎 inline buttons.

Verdict card format:

⚠️ BORDERLINE — £14.20/hr

£7.52 · 2 pickups · 2 drops (same building)
Cycling: 17 min from your location
Plus stops + peak wait: ~31 min total

Pickup 1: La Maritxu (W1B 5PJ)
Pickup 2: Cô Thành (WC2E 8QH)
Drop:     19 Sun St, EC2A 2FJ

Was this accurate? 👍 👎

Commands:

/start — intro + location request
/help — usage
/here — re-share current location
/stats — your last 7 days
/forget — delete all your data

Build plan — phased

Phase A — Scaffolding ✅ DONE

Phase B — OCR smoke test with new schema (CURRENT)

Implement the new ParsedOrder / PickupLeg / DropoffLeg extractor against the actual Deliveroo layout (no £ symbol survives OCR; multiple pickup/dropoff blocks; full postcodes). Run on the provided fixtures (test.PNG, test1.PNG) and print parsed JSON. No Telegram, no DB, no routing.

Done when: both fixtures return correct payout, all pickup restaurants with their postcodes, all dropoffs with their postcodes, and the right pickup/dropoff counts. test.PNG → 2 pickups (different restaurants) + 2 dropoffs (same postcode → 1 effective stop). test1.PNG → 1 pickup + 1 dropoff.

Phase C — Telegram bot: photo + location (1 evening)

Telegram handlers for /start, /help, photo messages, location messages, live-location updates. /start sends the location-request keyboard. Photos save to data/screenshots/{user_id}/{timestamp}.jpg. Locations update the in-memory store. For now reply "Got photo + location at {ts}, will analyse next phase".

Done when: sharing location works, photo+location both persist, replies arrive.

Phase D — Database (half evening)

SQLModel models per this doc. SQLite at data/bot.db. Wire handlers to persist Order, User, UserLocation. Add /forget command for full deletion.

Done when: schema looks right; /forget wipes user's rows.

Phase E — Parser integration (1 evening)

Plumb the Phase-B extractor into the bot. On photo receipt, parse → save ParsedOrder JSON to Order row → reply with the structured fields (no verdict yet). Wire VLM fallback (HybridParser). Switching VISION_PROVIDER=anthropic swaps fallback to Claude with no code change.

Done when: real screenshot returns the new schema in <1s via OCR; hybrid fallback triggers when OCR confidence is low.

Phase F — Mapbox geocoding + routing (1 evening)

Implement geocoding.py (Mapbox postcode → lat/lng, SQLite-cached) and routing.py (Mapbox Directions, cycling profile, multi-waypoint). Add scripts/test_routing.py for offline smoke testing. Wire into bot: after parse, geocode every postcode, build waypoint list from rider's last-known location, call Mapbox, save route to Order. Reply with raw "cycling X min, Y miles" — no verdict yet.

Done when: test fixture screenshots produce sensible cycling ETAs from a hard-coded test location; bot replies with real numbers in <3s end-to-end.

Phase G — Scoring engine (half evening)

Implement scoring.py per this doc. Pure functions, no IO. Unit tests covering: high £/short cycle, low £/long cycle, borderline, peak vs off-peak, single vs stacked vs co-located drops, low-confidence parse. Wire into bot reply, replacing raw fields with the verdict card.

Done when: verdict card renders; pytest tests/test_scoring.py passes 10+ cases.

Phase H — Feedback loop (half evening)

Inline keyboard 👍 / 👎 on verdict replies. Persist to feedback table. /stats shows last 7 days.

Done when: feedback persists; /stats accurate.

Phase I — Benchmark (half evening)

scripts/benchmark_parsers.py runs all fixtures through OCR-only, OCR+Ollama hybrid, and Anthropic-only; outputs CSV with field-by-field agreement and latency.

Done when: CSV produced; we know the OCR-alone hit rate.

Phase J — Polish + run (half evening)

Graceful shutdown, friendly errors, daily log rotation, README, scripts/run.sh, Mapbox/Ollama health checks on startup.

Done when: survives sleep/wake; one command launches everything.

Privacy stance (relaxed for now)

Personal project, single user, known testers only. Full postcodes and addresses stored. /forget still implemented for clean state. If this ever opens to public riders, revisit:

Postcode redaction to outward-code-only
Address-line redaction in OCR output
PRIVACY.md disclosure on /start
Auto-delete of screenshots older than N days

Until then, the model is: it's my data, I'll handle it carefully.

Costs

Item	Cost
RapidOCR	£0
Ollama + Qwen3-VL	£0 (fallback only)
Mapbox (Directions + Geocoding)	£0 well within 100k/mo free tier
Telegram bot	£0
SQLite	£0
Laptop electricity	~£3–8/month
Total	~£3–8/month

Appendix A — VLM prompt (Ollama / Anthropic fallback)

You are parsing a screenshot from the Deliveroo Rider app's order offer screen.

Return ONLY a JSON object with this exact schema. No prose, no markdown, just JSON.

{
  "payout_gbp": number or null,
  "pickups": [
    {"restaurant_name": string, "postcode": string, "item_count": integer}
  ],
  "dropoffs": [
    {"postcode": string}
  ],
  "confidence": number between 0 and 1,
  "not_a_deliveroo_order": boolean
}

RULES:
- payout_gbp: the rider fee in pounds, e.g. 7.52
- pickups: one entry per restaurant shown (could be 1, 2, or more); item_count is the "Nx" quantity label, default 1
- dropoffs: one entry per customer address shown
- postcode: full UK postcode if visible, e.g. "W1B 5PJ"; null if not visible
- confidence: 0–1, how sure you are this parsing is correct
- not_a_deliveroo_order: true if the image is not a Deliveroo rider order offer screen
- If you cannot see a value clearly, use null. Never invent values.

Return only the JSON object.

Ollama setup (one-time, for fallback)

brew install ollama        # Mac
ollama serve               # if not auto-started
ollama pull qwen3-vl:4b
export OLLAMA_KEEP_ALIVE=24h

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rider Intelligence Bot — MVP Project Plan

Goals & non-goals for this build

Tech stack

What Deliveroo's offer screen actually contains

Trip-shape taxonomy

Architecture

Data model

Project structure

OCR parser — primary path

Engine

Field extraction — the new structure

Hybrid parser — OCR with VLM fallback

Routing — Mapbox

Geocoding

Cycling route

Mapbox env config

Telegram location flow

Scoring engine — real-ETA edition

User flow

Build plan — phased

Phase A — Scaffolding ✅ DONE

Phase B — OCR smoke test with new schema (CURRENT)

Phase C — Telegram bot: photo + location (1 evening)

Phase D — Database (half evening)

Phase E — Parser integration (1 evening)

Phase F — Mapbox geocoding + routing (1 evening)

Phase G — Scoring engine (half evening)

Phase H — Feedback loop (half evening)

Phase I — Benchmark (half evening)

Phase J — Polish + run (half evening)

Privacy stance (relaxed for now)

Costs

Appendix A — VLM prompt (Ollama / Anthropic fallback)

Ollama setup (one-time, for fallback)

FilesExpand file tree

PROJECT_PLAN.md

Latest commit

History

PROJECT_PLAN.md

File metadata and controls

Rider Intelligence Bot — MVP Project Plan

Goals & non-goals for this build

Tech stack

What Deliveroo's offer screen actually contains

Trip-shape taxonomy

Architecture

Data model

Project structure

OCR parser — primary path

Engine

Field extraction — the new structure

Hybrid parser — OCR with VLM fallback

Routing — Mapbox

Geocoding

Cycling route

Mapbox env config

Telegram location flow

Scoring engine — real-ETA edition

User flow

Build plan — phased

Phase A — Scaffolding ✅ DONE

Phase B — OCR smoke test with new schema (CURRENT)

Phase C — Telegram bot: photo + location (1 evening)

Phase D — Database (half evening)

Phase E — Parser integration (1 evening)

Phase F — Mapbox geocoding + routing (1 evening)

Phase G — Scoring engine (half evening)

Phase H — Feedback loop (half evening)

Phase I — Benchmark (half evening)

Phase J — Polish + run (half evening)

Privacy stance (relaxed for now)

Costs

Appendix A — VLM prompt (Ollama / Anthropic fallback)

Ollama setup (one-time, for fallback)