Fetch and cache live Models.dev catalog into ProviderLake

[Hmbown]

Parent: #4184

Depends on: #4185 and #4186

Problem

CodeWhale already has lower-level Models.dev parsing and a provider_lake live overlay, but there is no OpenCode-style producer that fetches Models.dev, caches it, compiles it, and publishes the result into the TUI catalog path.

OpenCode's pattern is the right rough model:

• Fetch Models.dev with a timeout and user-agent.
• Cache under the app cache dir.
• Use stale disk cache or bundled snapshot when the network fails.
• Allow an override URL/path for testing and dogfooding.
• Refresh manually and opportunistically in the background.

Browser/live verification

Checked live on 2026-07-07:

• https://models.dev/catalog.json is available as application/json and about 3.13 MB.
• It contains both provider endpoint rows and canonical model metadata.
• It has enough data to populate model picker metadata: names, wire ids, context/output limits, costs, env vars, API base URLs, npm/protocol hints, modalities, reasoning, tool support, attachment support.

Likely files

• New module likely under crates/tui/src/, for example models_dev_live.rs or catalog_refresh/models_dev.rs
• crates/config/src/models_dev.rs
• crates/config/src/catalog.rs
• crates/tui/src/provider_lake.rs
• crates/tui/src/client.rs
• crates/tui/src/tui/model_picker.rs
• crates/tui/src/tui/provider_picker.rs

Proposed implementation

Add a secret-free live refresh layer that:

• Defaults to https://models.dev/catalog.json.
• Supports an env/config override for base URL or file path, useful for tests and local snapshots.
• Uses a bounded timeout, explicit user-agent, and no credentials.
• Writes cache atomically via temp file + rename.
• Reads cache on startup before attempting network refresh.
• Falls back to bundled snapshot if there is no fresh/stale cache.
• Compiles parsed ModelsDevCatalog into CatalogOffering rows.
• Publishes the compiled live snapshot with provider_lake::set_live_snapshot.
• Emits freshness/provenance metadata so UI can distinguish bundled, live, stale, and failed states.

Suggested cache policy

• Manual refresh command forces a network attempt.
• Background refresh is best-effort and should never block startup or model selection.
• TTL can start short during development, then settle around 24h for release.
• Failed refresh keeps prior cache visible when available.
• 4xx/5xx/network errors should produce quiet status/freshness signals, not break the picker.

Acceptance

• Fresh install/offline startup still has bundled catalog rows.
• Online startup eventually publishes live Models.dev rows into provider_lake.
• Manual refresh updates the cache and visible picker data.
• Refresh failure keeps prior/bundled rows and does not make model selection fail.
• No secrets, auth headers, or raw provider error bodies are persisted.
• Tests cover: fresh fetch success, parse failure fallback, network failure fallback, stale cache fallback, override path/base URL behavior, and ProviderLake publication.
• cargo test -p codewhale-config models_dev catalog and focused TUI picker/provider-lake tests pass.

[Hmbown]

Catalog lane context

Models.dev live catalog chain (Wave 2 / #4109):

#4184 → #4185 → #4186 → #4187 → #4188

• Depends on: Normalize Models.dev provider IDs onto CodeWhale provider kinds #4186 (provider ID normalization)
• Blocks: Demote curated bundled model data after Models.dev live catalog lands #4188 (demote bundled snapshot)

Related catalog lane

• Parent: v0.9.0 remaining: model catalog consolidation and live refresh #4109 · v0.9.0 1.2: catalog refresh policy for live model metadata #4114 (refresh policy) · v0.9.0 1.5: add catalog refresh and bundled snapshot automation #4117 (refresh automation)
• v0.9.0 3.1: wire live catalog cache into provider and model pickers #4139 (wire live cache into pickers)

References

• Architecture: v0.9.2 architecture: Fleet / Workflow / Lane / Runtime product model (canonical tracker) #4175 · Playbook: docs/AGENT_WORKFLOWS_0868.md

[Hmbown]

Closed: implemented on main

#4247 added the OpenCode-style Models.dev producer:

• Fetch with timeout + user-agent
• Secret-free disk cache under CodeWhale state
• Stale cache / bundled fallback on failure
• Path/URL overrides for dogfood
• Background refresh (spawn_background_refresh) + manual /model refresh (AppAction::RefreshModelsDevCatalog)

Supporting demotion of bundled-as-fallback: #4248 (#4188). Live cache in pickers: #4250 (#4139).