Skip to content

Refresh stored embedding rows when store or metadata changes#1513

Open
dpersek wants to merge 1 commit into
simonw:mainfrom
dpersek:codex/embedding-store-refresh
Open

Refresh stored embedding rows when store or metadata changes#1513
dpersek wants to merge 1 commit into
simonw:mainfrom
dpersek:codex/embedding-store-refresh

Conversation

@dpersek

@dpersek dpersek commented Jul 4, 2026

Copy link
Copy Markdown

Summary

  • Refresh stored content, content_blob, and metadata when repeat embedding calls target an existing row with the same ID and content hash
  • Avoid recalculating embeddings for exact existing rows while preserving normal insert/replace behavior for new IDs or changed content
  • Add regression coverage for store/metadata refresh, binary content, duplicate-content rows, new IDs sharing content, and changed-content batch cases

Why

Fixes #224.

The existing embed() and embed_multi_with_metadata() paths can notice that a content_hash is already present and skip work before updating stored content or metadata. That makes rerunning embed / embed-multi with --store or new metadata leave the stored columns stale.

Tests

  • uv run --group dev pytest tests/test_embed.py -k "updates_store_and_metadata_for_existing_hash or updates_content_blob_for_existing_hash or embed_inserts_new_id_for_existing_hash or existing_hash_refresh_only_updates_matching_id or embed_multi_inserts_new_id_for_existing_hash or embed_multi_updates_changed_id_when_old_hash_is_in_batch": 7 passed, 15 deselected
  • uv run --group dev pytest tests/test_embed.py: 22 passed
  • uv run --group dev pytest tests/test_embed_cli.py: 91 passed
  • uv run --group dev ruff check llm/embeddings.py tests/test_embed.py: passed
  • uv run --group dev black --target-version py312 --check llm/embeddings.py tests/test_embed.py: 2 files unchanged
  • git diff --check: passed
  • uv run --group dev pytest: 789 passed

Scope

Refresh stored content, content_blob, and metadata for exact existing embedding rows without recomputing embeddings. Preserve normal insert and replace behavior for new IDs or changed content, with regression coverage for the edge cases found during review.

Fixes simonw#224.
@dpersek
dpersek marked this pull request as ready for review July 4, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant