Commit 66a89f5
perf(reset_budget_job): elect one sweeper per tick and bound the window scan (#36497)
Every pod schedules the budget reset job, so a fleet re-read the whole due
population and wrote it back against one Postgres at the same calendar
boundary, multiplying a single sweep by its replica count. The job now takes
the shared PodLockManager lease, so one pod sweeps per tick. A deployment with
no Redis keeps its previous behavior, and a Redis that cannot answer sweeps
unguarded rather than stranding every expired budget at its cap.
The per-window scan read every row carrying budget_limits in one statement, so
its cost grew with the deployment's key count. It is now keyset-paginated and
walks to the end of the table on every sweep. A per-run cap would need a resume
position, and no pod can hold one because the lease rotates between ticks, so
the strictly advancing cursor is what terminates the walk.
Found and updated rows were also JSON-serialized into the service hook's
metadata and into debug lines on every chunk, on the event loop, whether or not
anything consumed them. The hooks now carry counts, and the debug payload is
deferred until a record is actually emitted.
Resolves LIT-4793
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>1 parent e98b9fd commit 66a89f5
5 files changed
Lines changed: 596 additions & 111 deletions
File tree
- litellm
- proxy
- common_utils
- tests
- litellm_utils_tests
- test_litellm/proxy/common_utils
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1537 | 1537 | | |
1538 | 1538 | | |
1539 | 1539 | | |
| 1540 | + | |
| 1541 | + | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
1540 | 1545 | | |
1541 | 1546 | | |
1542 | 1547 | | |
| |||
0 commit comments