Skip to content

fix: ship buffered job logs on interval even when output is idle#315

Draft
wesleyjellis wants to merge 1 commit into
mainfrom
fix/log-max-interval-idle-jobs
Draft

fix: ship buffered job logs on interval even when output is idle#315
wesleyjellis wants to merge 1 commit into
mainfrom
fix/log-max-interval-idle-jobs

Conversation

@wesleyjellis

Copy link
Copy Markdown

Problem

job-pod-log-max-interval promises logs are shipped to OpsLevel at most N seconds apart, but the max-time check lived inside Process, which only runs when a new line arrives. A job that prints a burst of output and then goes quiet (e.g. a long compile or terraform apply) has those buffered logs held until the next line, the maxBytes threshold, or job completion — potentially the entire job duration.

Fix

  • LogStreamer.Run now calls an optional Tick() (new tickable interface, asserted per processor) on every ticker interval, so processors get a periodic hook even with no output. Tick runs on the same goroutine as Process, so no new synchronization is needed.
  • Both OpsLevelAppendLogProcessor and FaktoryAppendJobLogProcessor implement Tick: if the buffer is non-empty and maxTime has elapsed since the last submit, they ship.
  • This replaces the arrival-driven elapsed/lastTime accounting with a single lastSubmitTime, and incidentally removes the s.elapsed = time.Since(time.Now()) typo in the Faktory processor (which assigned a tiny negative duration instead of resetting to zero).

Testing

  • TestOpsLevelAppendLogProcessorTickShipsIdleLogs — buffered lines ship via Tick after maxTime with no new output, and not before.
  • TestLogStreamerCallsTickWhileIdle — the streamer ticks processors even when no lines are ever written.
  • go test -race ./pkg/... passes.

Note: touches logs.go alongside #313, so whichever merges second will need a trivial rebase.

🤖 Generated with Claude Code

The append log processors only evaluated the max-time condition inside
Process, which runs when a new line arrives. A job that printed output
and then went quiet had those logs held until the next line, maxBytes,
or job completion - breaking the contract of job-pod-log-max-interval.

The log streamer's Run loop now calls an optional Tick() on processors
every ticker interval, and both append processors ship their buffer
from Tick once maxTime has elapsed since the last submit. This replaces
the arrival-driven elapsed-time accounting (and removes the
`elapsed = time.Since(time.Now())` typo in the faktory processor).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@wesleyjellis wesleyjellis self-assigned this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant