fix: ship buffered job logs on interval even when output is idle#315
Draft
wesleyjellis wants to merge 1 commit into
Draft
fix: ship buffered job logs on interval even when output is idle#315wesleyjellis wants to merge 1 commit into
wesleyjellis wants to merge 1 commit into
Conversation
The append log processors only evaluated the max-time condition inside Process, which runs when a new line arrives. A job that printed output and then went quiet had those logs held until the next line, maxBytes, or job completion - breaking the contract of job-pod-log-max-interval. The log streamer's Run loop now calls an optional Tick() on processors every ticker interval, and both append processors ship their buffer from Tick once maxTime has elapsed since the last submit. This replaces the arrival-driven elapsed-time accounting (and removes the `elapsed = time.Since(time.Now())` typo in the faktory processor). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
job-pod-log-max-intervalpromises logs are shipped to OpsLevel at most N seconds apart, but the max-time check lived insideProcess, which only runs when a new line arrives. A job that prints a burst of output and then goes quiet (e.g. a long compile or terraform apply) has those buffered logs held until the next line, the maxBytes threshold, or job completion — potentially the entire job duration.Fix
LogStreamer.Runnow calls an optionalTick()(newtickableinterface, asserted per processor) on every ticker interval, so processors get a periodic hook even with no output.Tickruns on the same goroutine asProcess, so no new synchronization is needed.OpsLevelAppendLogProcessorandFaktoryAppendJobLogProcessorimplementTick: if the buffer is non-empty andmaxTimehas elapsed since the last submit, they ship.elapsed/lastTimeaccounting with a singlelastSubmitTime, and incidentally removes thes.elapsed = time.Since(time.Now())typo in the Faktory processor (which assigned a tiny negative duration instead of resetting to zero).Testing
TestOpsLevelAppendLogProcessorTickShipsIdleLogs— buffered lines ship viaTickaftermaxTimewith no new output, and not before.TestLogStreamerCallsTickWhileIdle— the streamer ticks processors even when no lines are ever written.go test -race ./pkg/...passes.Note: touches
logs.goalongside #313, so whichever merges second will need a trivial rebase.🤖 Generated with Claude Code