Skip to content

Commit 7f1016b

Browse files
AndrewAvery7claude
andauthored
fix(sync): fold re-keyed upstream duplicates into their existing id (#124)
* fix(sync): fold re-keyed upstream duplicates into their existing id LiteLLM started listing Mistral Medium 3.5 under a second key that differs only by punctuation (a dot instead of a dash). mergeCatalog treated it as a brand-new model, and its id collided with the already- published one once both were turned into a URL slug -- crashing check-pages and failing every Sync pricing run on main since. Reconcile any freshly-matched feed id that isn't already published against the slug of ids we do publish, and fold it into the existing id instead of minting a second, colliding one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: update published test counts for the new regression test The new pipeline.test.ts case bumps this package to 434 tests and the published total to 997 -- update the README badge, the README docs table, and docs/TESTING.md's total and per-file row to match, per check:test-badge. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * chore(mobile): align Expo SDK patch dependencies expo, expo-constants and expo-font had drifted a patch version behind what the installed Expo SDK 57 expects, tripping expo-doctor in the mobile verify job -- unrelated to this branch's pricing-sync fix, but failing on main regardless. `expo install --fix` resolves it; all 21 expo-doctor checks, the mobile test suite (115/115), and the iOS/ Android/web export all pass afterward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent d71ea91 commit 7f1016b

6 files changed

Lines changed: 116 additions & 38 deletions

File tree

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT license"></a>
1515
<img src="https://img.shields.io/badge/models-74-2456E6.svg" alt="74 models tracked">
1616
<img src="https://img.shields.io/badge/providers-12-2456E6.svg" alt="12 providers">
17-
<img src="https://img.shields.io/badge/tests-996-blue.svg" alt="996 tests">
17+
<img src="https://img.shields.io/badge/tests-997-blue.svg" alt="997 tests">
1818
<img src="https://img.shields.io/badge/initial%20payload-87%20KB%20gzip-blue.svg" alt="87 KB gzip initial payload">
1919
<a href="https://github.com/AndrewAvery7/promptspend/actions/workflows/ci.yml"><img src="https://github.com/AndrewAvery7/promptspend/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
2020
<a href="https://github.com/AndrewAvery7/promptspend/actions/workflows/sync-pricing.yml"><img src="https://github.com/AndrewAvery7/promptspend/actions/workflows/sync-pricing.yml/badge.svg" alt="Sync pricing"></a>
@@ -377,7 +377,7 @@ there is a `Ctrl`/`Cmd`+`K` command palette.
377377
| Document | What is in it |
378378
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
379379
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | How the pipeline, the engine and the state layer work, and **why** each is shaped that way |
380-
| [docs/TESTING.md](docs/TESTING.md) | What the 996 tests cover, the uneven coverage thresholds, and what the suite deliberately does not cover |
380+
| [docs/TESTING.md](docs/TESTING.md) | What the 997 tests cover, the uneven coverage thresholds, and what the suite deliberately does not cover |
381381
| [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) | "The estimate does not match my bill", flagged prices, missing models, running it locally |
382382
| [docs/PAGES.md](docs/PAGES.md) | The 163 generated pages: what is built, why the comparison set is curated, and the IndexNow pipeline |
383383
| [docs/API.md](docs/API.md) | The public pricing API on `promptspend.dev` — endpoints, why it fetches rather than bundles, going live |

apps/mobile/package-lock.json

Lines changed: 32 additions & 32 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

apps/mobile/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
"@expo/vector-icons": "^15.0.2",
99
"@promptspend/core": "file:../../packages/core",
1010
"@react-native-async-storage/async-storage": "2.2.0",
11-
"expo": "~57.0.17",
11+
"expo": "~57.0.18",
1212
"expo-clipboard": "~57.0.1",
1313
"expo-constants": "~57.0.12",
1414
"expo-file-system": "~57.0.4",

docs/TESTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ to publish past it.
1515

1616
## What the suite covers
1717

18-
**996 tests in all**: 433 unit and integration tests in this package, 115 in
18+
**997 tests in all**: 434 unit and integration tests in this package, 115 in
1919
`apps/mobile/`, 108 in `worker/`, 44 in `api/`, 47 in `mcp/`, 137 in `vscode/`,
2020
and 112 browser tests across four viewports. Those seven figures are the whole suite and they sum to the
2121
total. Two earlier revisions did not: one claimed 533, which was neither the sum
@@ -47,7 +47,7 @@ anyone adding them up.
4747

4848
| Suite | Tests | Guards |
4949
| -------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
50-
| `scripts/lib/pipeline.test.ts` | 62 | The whole pipeline against fixtures: the trust ladder, both sanity thresholds, cold start, stale retention, flag-once semantics — including a standing flag whose _figures_ drift, which is what re-opened a pull request every morning — full-field diffing, coverage versus a price move versus a source correction, the content hash, changelog rendering. |
50+
| `scripts/lib/pipeline.test.ts` | 63 | The whole pipeline against fixtures: the trust ladder, both sanity thresholds, cold start, stale retention, flag-once semantics — including a standing flag whose _figures_ drift, which is what re-opened a pull request every morning — full-field diffing, coverage versus a price move versus a source correction, the content hash, changelog rendering. |
5151
| `scripts/lib/readme-badges.test.ts` | 3 | The daily catalog sync updates both badge URLs and accessible alt text, leaves unrelated prose untouched, and fails closed if a README redesign removes or duplicates a derived claim. |
5252
| `src/lib/contrast.test.ts` | 49 | Every accent × theme × canvas combination against every surface it can appear on, plus the status colours on their own soft fills — the pairing the accent matrix does not reach. |
5353
| `src/lib/engine/cost.test.ts` | 44 | Compounding history, per-turn long-context tiers, cache reads and writes, promotional windows, reasoning multipliers, margins, break-even, and the scenarios that cannot physically run. |

scripts/lib/merge.ts

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
*/
2121
import type { Model, PricingCatalog } from '../../src/lib/pricing/types';
2222
import { SCHEMA_VERSION } from '../../src/lib/pricing/types';
23+
import { modelSlug } from '../../src/lib/seo/slug';
2324
import { pricingChanged } from './diff';
2425
import {
2526
comparisonKey,
@@ -105,7 +106,20 @@ export function mergeCatalog(input: MergeInput): MergeResult {
105106
}
106107
const overrideById = new Map(overrides.map((o) => [o.id, o]));
107108
const previousById = new Map((previous?.models ?? []).map((m) => [m.id, m]));
108-
const feedById = new Map(litellm.map((r) => [r.id, r]));
109+
110+
// Upstream occasionally re-keys a model with different punctuation (a dot
111+
// where a dash used to be, or vice versa) without it being a new model at
112+
// all. Ids are permanent once published (see src/lib/seo/slug.ts), so when
113+
// a feed row we don't already publish collides on URL slug with one we do,
114+
// treat it as that same model rather than minting a second, colliding id.
115+
const publishedIdBySlug = new Map<string, string>();
116+
for (const id of previousById.keys()) publishedIdBySlug.set(modelSlug(id), id);
117+
const reconciledLitellm = litellm.map((rate) => {
118+
if (previousById.has(rate.id)) return rate;
119+
const knownId = publishedIdBySlug.get(modelSlug(rate.id));
120+
return knownId ? { ...rate, id: knownId } : rate;
121+
});
122+
const feedById = new Map(reconciledLitellm.map((r) => [r.id, r]));
109123
const review: ReviewItem[] = [];
110124
const corrections: CorrectionItem[] = [];
111125
const staleIds: string[] = [];

scripts/lib/pipeline.test.ts

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,70 @@ describe('mergeCatalog — the trust ladder', () => {
257257
expect(warm.review.map((item) => item.id)).toEqual(['moonshot-kimi-k2.6']);
258258
});
259259

260+
it('folds a re-keyed upstream duplicate into the existing id instead of colliding on URL slug', () => {
261+
// LiteLLM occasionally lists the same model under a differently-punctuated
262+
// key (a dot where a dash used to be). Both are valid catalog ids on their
263+
// own, but they collapse to the identical URL slug — assertUniqueSlugs in
264+
// src/lib/seo/slug.ts is what catches that at build time — so a same-priced
265+
// dot-form arriving later must be folded into the id already published,
266+
// not minted as a second "new" model.
267+
const mistralAllowlist: Allowlist = {
268+
...ALLOWLIST,
269+
providers: [...ALLOWLIST.providers, { id: 'mistral', name: 'Mistral AI', country: 'FR' }],
270+
families: [
271+
...ALLOWLIST.families,
272+
{
273+
id: 'mistral',
274+
providerId: 'mistral',
275+
include: ['^mistral/mistral-medium-\\d(\\.\\d|-\\d)?$'],
276+
stripPrefix: 'mistral/',
277+
tokenizer: { kind: 'approx', charsPerToken: 3.7, cjkCharsPerToken: 1.8 },
278+
capabilities: { reasoning: false, vision: true },
279+
},
280+
],
281+
};
282+
const previous: PricingCatalog = {
283+
schemaVersion: SCHEMA_VERSION,
284+
generatedAt: '2026-08-01T06:00:00.000Z',
285+
providers: mistralAllowlist.providers,
286+
models: [
287+
{
288+
id: 'mistral-mistral-medium-3-5',
289+
providerId: 'mistral',
290+
displayName: 'Mistral Medium 3.5',
291+
status: 'current',
292+
contextWindow: 262_144,
293+
pricing: { input: 1.5, output: 7.5 },
294+
tokenizer: { kind: 'approx', charsPerToken: 3.7, cjkCharsPerToken: 1.8 },
295+
capabilities: { reasoning: false, vision: true },
296+
provenance: { source: 'litellm', lastVerified: '2026-08-01' },
297+
},
298+
],
299+
};
300+
const feed = fromLiteLLM(
301+
{
302+
'mistral/mistral-medium-3.5': {
303+
mode: 'chat',
304+
input_cost_per_token: 1.5e-6,
305+
output_cost_per_token: 7.5e-6,
306+
},
307+
},
308+
mistralAllowlist,
309+
);
310+
const { catalog, review } = mergeCatalog({
311+
litellm: feed,
312+
openrouter: new Map(),
313+
allowlist: mistralAllowlist,
314+
overrides: [],
315+
previous,
316+
generatedAt,
317+
});
318+
const mistralModels = catalog.models.filter((m) => m.providerId === 'mistral');
319+
expect(mistralModels).toHaveLength(1);
320+
expect(mistralModels[0]!.id).toBe('mistral-mistral-medium-3-5');
321+
expect(review).toEqual([]);
322+
});
323+
260324
it('keeps a previously published model when the feed stops listing it', () => {
261325
// One truncated upstream response used to be enough to delete most of the
262326
// catalog — and because removals did not trip the review rule, the deletion

0 commit comments

Comments
 (0)