fix(dns): back off on TCP accept errors instead of spinning - #77
Merged
Conversation
`uninlined_format_args` fires on the current stable clippy across the certs, dns, http and tcp modules. Applied `cargo clippy --fix`; no behavior change. Separated from the DNS accept fix so that change stays reviewable on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the process runs out of file descriptors, `accept()` returns EMFILE and leaves the pending connection queued. The error arm logged and looped straight back into `accept()`, which failed on the same connection immediately - an unbounded hot loop. Observed in production on a 1 OCPU host: ~1,600 error lines/second, the container pegged at 149% CPU, and 65-72% CPU steal. TLS handshakes, the most CPU-hungry part of the request path, went from ~90ms to 12-18 seconds, with roughly 1 in 5 connections timing out entirely. Plain HTTP returned empty replies and sshd could not be scheduled reliably. The spin also starved the very tasks that would have closed descriptors, so the process could not recover on its own. Back off 100ms after any accept error. A transient failure now costs a brief pause rather than the CPU the server needs to recover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A DNS-over-TCP
accept()failure caused an unbounded hot loop that took production down to a crawl. This adds a 100ms backoff on accept errors.Root cause
When the process exhausts its file descriptors,
accept()returnsEMFILEand leaves the pending connection queued. The error arm logged and looped straight back intoaccept(), which failed on the same connection immediately:The spin then starved the very tasks that would have closed descriptors, so the process could not recover on its own.
Production impact
Observed on a 1 OCPU host with the container's soft
RLIMIT_NOFILEat the 1024 default (1018 of those FDs were sockets):Because TLS handshakes are the most CPU-hungry part of the request path, they degraded first and worst. Plain HTTP returned
Empty reply from serverandsshdcould not be scheduled reliably — the box looked network-broken when it was purely CPU-starved.The fix
Back off 100ms after any accept error, not just resource exhaustion. Checking for
EMFILE/ENFILEspecifically would need alibcdependency for two constants, and a brief pause is the right response to any unexpected accept failure. The existing semaphore-exhaustion path already dropsstreamcorrectly and is unchanged.Note on the first commit
394fbabappliescargo clippy --fixforuninlined_format_argsacross certs/dns/http/tcp. Those fire on current stable clippy on a cleanmainand blocked the pre-commit hook, so they had to be resolved to commit at all. Kept as a separate commit so the actual fix stays reviewable.Testing
cargo test— 84 passed, 0 failed (1 ignored: ACME staging, needs a real domain)cargo clippy --all-targets -- -D warnings— cleancargo fmt --check— cleanFollow-up worth considering (not in this PR)
nofilelimit indocker-compose.yml; the 1024 default is low for a process with four listeners. Already applied to the running production host.max-size: 10mrotation it destroys the diagnostics you need.🤖 Generated with Claude Code