Skip to content

fix(dns): back off on TCP accept errors instead of spinning - #77

Merged
adrgs merged 2 commits into
mainfrom
fix/dns-tcp-accept-backoff
Aug 25, 2026
Merged

fix(dns): back off on TCP accept errors instead of spinning#77
adrgs merged 2 commits into
mainfrom
fix/dns-tcp-accept-backoff

Conversation

@adrgs

@adrgs adrgs commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Summary

A DNS-over-TCP accept() failure caused an unbounded hot loop that took production down to a crawl. This adds a 100ms backoff on accept errors.

Root cause

When the process exhausts its file descriptors, accept() returns EMFILE and leaves the pending connection queued. The error arm logged and looped straight back into accept(), which failed on the same connection immediately:

Err(e) => {
    error!("Error accepting DNS TCP connection: {}", e);
}   // <- back to accept(), same connection, same error

The spin then starved the very tasks that would have closed descriptors, so the process could not recover on its own.

Production impact

Observed on a 1 OCPU host with the container's soft RLIMIT_NOFILE at the 1024 default (1018 of those FDs were sockets):

Metric During incident After raising the FD limit
Error log rate ~1,600 lines/sec 0
Container CPU 149% 1.05%
CPU steal 65-72% negligible
TLS handshake 12-18s, ~1 in 5 timing out ~90ms

Because TLS handshakes are the most CPU-hungry part of the request path, they degraded first and worst. Plain HTTP returned Empty reply from server and sshd could not be scheduled reliably — the box looked network-broken when it was purely CPU-starved.

The fix

Back off 100ms after any accept error, not just resource exhaustion. Checking for EMFILE/ENFILE specifically would need a libc dependency for two constants, and a brief pause is the right response to any unexpected accept failure. The existing semaphore-exhaustion path already drops stream correctly and is unchanged.

Note on the first commit

394fbab applies cargo clippy --fix for uninlined_format_args across certs/dns/http/tcp. Those fire on current stable clippy on a clean main and blocked the pre-commit hook, so they had to be resolved to commit at all. Kept as a separate commit so the actual fix stays reviewable.

Testing

  • cargo test — 84 passed, 0 failed (1 ignored: ACME staging, needs a real domain)
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo fmt --check — clean

Follow-up worth considering (not in this PR)

  • Raise the container's nofile limit in docker-compose.yml; the 1024 default is low for a process with four listeners. Already applied to the running production host.
  • Rate-limit that error log. 100k lines/minute is its own denial of service, and with max-size: 10m rotation it destroys the diagnostics you need.
  • Investigate why FDs reached 1024 — this PR stops the spin, but the exhaustion itself is a separate question.

🤖 Generated with Claude Code

adrgs and others added 2 commits August 25, 2026 20:25
`uninlined_format_args` fires on the current stable clippy across the
certs, dns, http and tcp modules. Applied `cargo clippy --fix`; no
behavior change. Separated from the DNS accept fix so that change stays
reviewable on its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the process runs out of file descriptors, `accept()` returns EMFILE
and leaves the pending connection queued. The error arm logged and looped
straight back into `accept()`, which failed on the same connection
immediately - an unbounded hot loop.

Observed in production on a 1 OCPU host: ~1,600 error lines/second, the
container pegged at 149% CPU, and 65-72% CPU steal. TLS handshakes, the
most CPU-hungry part of the request path, went from ~90ms to 12-18
seconds, with roughly 1 in 5 connections timing out entirely. Plain HTTP
returned empty replies and sshd could not be scheduled reliably.

The spin also starved the very tasks that would have closed descriptors,
so the process could not recover on its own.

Back off 100ms after any accept error. A transient failure now costs a
brief pause rather than the CPU the server needs to recover.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@adrgs
adrgs merged commit 6a5b089 into main Aug 25, 2026
5 checks passed
@adrgs
adrgs deleted the fix/dns-tcp-accept-backoff branch August 25, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant