Right now every speaker upsert, bill mention insert, topic insert, and contributor link in ingest_sitting is a separate round trip. For a typical sitting with 20+ speakers and 10+ bills that is easily 100+ queries per sitting. ingest_members has the same pattern: one upsert_member per row in a loop.
What to change
- Add bulk variants (or make bulk the defacto operation) to the
DataStore trait for the hot paths:
upsert_speakers_bulkupsert_bill_mentions_bulk / link_speakers_to_bill_mentions_bulkupsert_topics_bulk / link_speakers_to_topics_bulkupsert_members_bulk
- Implement them in
postgres.rs using UNNEST-based bulk upserts
- Update
IngestPipeline to collect extracted data per sitting and flush in one call per entity type instead of looping
The enrich store_*_summary methods are one-at-a-time by design (each summary is an independent AI call), so those can stay as-is.
Right now every speaker upsert, bill mention insert, topic insert, and contributor link in
ingest_sittingis a separate round trip. For a typical sitting with 20+ speakers and 10+ bills that is easily 100+ queries per sitting.ingest_membershas the same pattern: oneupsert_memberper row in a loop.What to change
DataStoretrait for the hot paths:upsert_speakers_bulkupsert_bill_mentions_bulk/link_speakers_to_bill_mentions_bulkupsert_topics_bulk/link_speakers_to_topics_bulkupsert_members_bulkpostgres.rsusingUNNEST-based bulk upsertsIngestPipelineto collect extracted data per sitting and flush in one call per entity type instead of loopingThe enrich
store_*_summarymethods are one-at-a-time by design (each summary is an independent AI call), so those can stay as-is.