Skip to content

Commit 66a89f5

Browse files
perf(reset_budget_job): elect one sweeper per tick and bound the window scan (#36497)
Every pod schedules the budget reset job, so a fleet re-read the whole due population and wrote it back against one Postgres at the same calendar boundary, multiplying a single sweep by its replica count. The job now takes the shared PodLockManager lease, so one pod sweeps per tick. A deployment with no Redis keeps its previous behavior, and a Redis that cannot answer sweeps unguarded rather than stranding every expired budget at its cap. The per-window scan read every row carrying budget_limits in one statement, so its cost grew with the deployment's key count. It is now keyset-paginated and walks to the end of the table on every sweep. A per-run cap would need a resume position, and no pod can hold one because the lease rotates between ticks, so the strictly advancing cursor is what terminates the walk. Found and updated rows were also JSON-serialized into the service hook's metadata and into debug lines on every chunk, on the event loop, whether or not anything consumed them. The hooks now carry counts, and the debug payload is deferred until a record is actually emitted. Resolves LIT-4793 Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent e98b9fd commit 66a89f5

5 files changed

Lines changed: 596 additions & 111 deletions

File tree

litellm/constants.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1537,6 +1537,11 @@
15371537
PROXY_BUDGET_RESCHEDULER_MIN_TIME: Final = int(os.getenv("PROXY_BUDGET_RESCHEDULER_MIN_TIME", 597))
15381538
RESET_BUDGET_JOB_BATCH_SIZE: Final = max(1, int(os.getenv("RESET_BUDGET_JOB_BATCH_SIZE", "500")))
15391539
RESET_BUDGET_JOB_MAX_CHUNKS_PER_RUN: Final = max(1, int(os.getenv("RESET_BUDGET_JOB_MAX_CHUNKS_PER_RUN", "100")))
1540+
RESET_BUDGET_JOB_NAME: Final = "reset_budget_job"
1541+
# Comfortably longer than one PROXY_BUDGET_RESCHEDULER_MIN_TIME tick, so a healthy
1542+
# leader keeps the lease across its own run, and a crashed one strands the sweep for
1543+
# at most a single tick.
1544+
RESET_BUDGET_JOB_LOCK_TTL_SECONDS: Final[int] = 900
15401545
PROXY_BATCH_POLLING_INTERVAL: Final = int(os.getenv("PROXY_BATCH_POLLING_INTERVAL", 3600))
15411546
MAX_OBJECTS_PER_POLL_CYCLE: Final = max(1, int(os.getenv("MAX_OBJECTS_PER_POLL_CYCLE", 50)))
15421547
MANAGED_OBJECT_STALENESS_CUTOFF_DAYS: Final = max(1, int(os.getenv("MANAGED_OBJECT_STALENESS_CUTOFF_DAYS", 7)))

0 commit comments

Comments
 (0)