fix(sqs): stop close() busy-loop duplicate sends - #7301
Open
zoewangg wants to merge 2 commits into
Open
Conversation
SqsAsyncBatchManager.close() extracted one ready batch once and re-flushed that same map in a loop that only the async completion callback could clear, so on a real client the closing thread re-sent the identical SendMessageBatch thousands of times. It now drains the buffer by re-extracting and flushing each batch (including partials) exactly once, terminating when the buffer is empty. Shutdown is also graceful: DefaultSqsAsyncBatchManager.close() first dispatches every buffered batch across the three write managers (dispatchPending), then does a single bounded wait for the in-flight sends so callers receive real results, and finally cancels only the stragglers past the deadline (cancelPending). The wait is one shared grace period (internal, default 5s) across all three managers, not one each, and close() is idempotent and guarded against concurrent calls. Post-close submissions fail fast with IllegalStateException. Internal only; no public API change.
dagnir
reviewed
Aug 21, 2026
Fred1155
approved these changes
Aug 21, 2026
Renamed the internal shutdown-timeout terminology from "grace period" to "timeout": DEFAULT_SHUTDOWN_TIMEOUT, the shutdownTimeout field, accessor, and builder seam, and the SqsAsyncBatchManager.close() Javadoc. Also renamed RequestBatchManager.dispatchPending() to closeAndDispatch(). Both per PR review. Updated the changelog wording accordingly. Internal @SdkInternalApi only; no public API change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
This fixes two shutdown issues in
SqsAsyncBatchManager.close(), both in the shared internal base classRequestBatchManager(moduleservices/sqs).Redundant re-sends of a buffered batch on close. When
close()was called while other threads were still submitting on a real asynchronous client, a buffered batch could be sent more than once:close()waited for the in-flight response by re-sending the same batch, because the drain loop stopped only after an asynchronous completion callback cleared it rather than after draining the buffer. The effect was a bounded number of duplicateSendMessageBatchcalls for the same messages during that singleclose().close()now drains each buffered batch and sends it exactly once, stopping when the buffer is empty (partial batches included).In-flight requests cancelled on shutdown. Previously
close()cancelled in-flight batch sends immediately, so a request that had already been dispatched could complete withCancellationExceptioninstead of its realSendMessageResponse.close()now waits a bounded grace period for in-flight sends to finish so their callers receive the real result, and cancels only the requests still outstanding when that period expires. This mirrors the graceful bounded-await shutdown already used byCloudWatchMetricPublisher.close().Modifications
Drain each buffered batch exactly once:
RequestBatchManagernow drains each buffer by re-extracting the next batch withextractEntriesForScheduledFlush(the drain-any-non-empty primitiveperformScheduledFlushalready uses) and flushing each distinct batch exactly once, terminating when the buffer is empty. Full and partial batches are now sent exactly once (previously a residual partial batch was never sent and its callers were cancelled).Graceful bounded-await shutdown (dispatch-all-first, then a single shared wait):
RequestBatchManagerexposes a two-phase internal shutdown surface:dispatchPending()(phase 1, non-blocking) marks the manager closed, drains/dispatches every buffered batch, and returns aCompletableFuture<Void>that completes when the in-flight sends it dispatched complete;cancelPending()(phase 2) cancels anything still pending and releases the buffers. The wait targets the caller-facing futures (pendingResponses), so a completing send delivers the real result rather than being cancelled.DefaultSqsAsyncBatchManager.close()orchestrates the shutdown: it callsdispatchPending()on all three write managers first (every buffered batch goes on the wire at once), then performs a single bounded wait viaCompletableFuture.allOf(...)up to one shared grace period, then callscancelPending()on all three, then closes the receive manager. Because all three dispatch before the single wait, the whole shutdown is bounded by ONE grace period, not one per manager. The wait helper never throws (handlesTimeoutException/ExecutionException/InterruptedException, restoring the interrupt flag).RequestBatchConfiguration(@SdkInternalApi), default 5 seconds. There is no public knob.DefaultSqsAsyncBatchManager.close()is guarded by anAtomicBooleanso it is idempotent and safe under concurrent calls: only the first caller runs the dispatch/await/cancel orchestration, and a concurrent or second call is a clean no-op (this avoids a concurrent loser cancelling the winner's still-awaited in-flight sends).RequestBatchManagergains anAtomicBooleanclosed guard: asendMessage/deleteMessage/changeMessageVisibilitysubmitted after close returns a future completed exceptionally withIllegalStateException, matching the receive path.Documentation:
@Override void close()Javadoc on the publicSqsAsyncBatchManagerdocumenting that close flushes buffered requests, blocks up to a bounded grace period for in-flight sends so callers receive real results, then cancels stragglers.Testing
services/sqsbuilt with full static analysis (Checkstyle, SpotBugs, PMD, japicmp) - BUILD SUCCESS, no public API break. New and existing tests pass.Added new test cases
DefaultSqsAsyncBatchManagerTest- close/shutdown behavior against the REALDefaultSqsAsyncBatchManager(and, through it, the real write batch managers) over a mockSqsAsyncClientwith controllable batch-send futures:SqsAsyncBatchManagerTest(WireMock) - real end-to-end batching over HTTP.Types of changes
Checklist
mvn installsucceeds (built the affected moduleservices/sqs, not a full-repomvn install)License