Skip to content

Commit 2645b81

Browse files
adrgsclaude
andcommitted
fix(dns): back off on TCP accept errors instead of spinning
When the process runs out of file descriptors, `accept()` returns EMFILE and leaves the pending connection queued. The error arm logged and looped straight back into `accept()`, which failed on the same connection immediately - an unbounded hot loop. Observed in production on a 1 OCPU host: ~1,600 error lines/second, the container pegged at 149% CPU, and 65-72% CPU steal. TLS handshakes, the most CPU-hungry part of the request path, went from ~90ms to 12-18 seconds, with roughly 1 in 5 connections timing out entirely. Plain HTTP returned empty replies and sshd could not be scheduled reliably. The spin also starved the very tasks that would have closed descriptors, so the process could not recover on its own. Back off 100ms after any accept error. A transient failure now costs a brief pause rather than the CPU the server needs to recover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 394fbab commit 2645b81

1 file changed

Lines changed: 12 additions & 1 deletion

File tree

src/src/dns/mod.rs

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,11 @@ const TCP_IDLE_TIMEOUT: Duration = Duration::from_secs(30);
3636
/// Bound DNS-over-TCP tasks and their socket buffers under connection floods.
3737
const DNS_TCP_MAX_CONCURRENT_CONNECTIONS: usize = 256;
3838

39+
/// Backoff applied after a resource-exhaustion `accept()` failure (EMFILE/ENFILE).
40+
/// Without it the failed connection stays queued and `accept()` fails again
41+
/// immediately, spinning the loop at 100% CPU and starving every other task.
42+
const TCP_ACCEPT_BACKOFF: Duration = Duration::from_millis(100);
43+
3944
pub(crate) struct DnsRateLimiter {
4045
limits: Mutex<HashMap<IpAddr, (Instant, u32)>>,
4146
max_per_second: u32,
@@ -173,7 +178,13 @@ impl Server {
173178
});
174179
}
175180
Err(e) => {
176-
error!("Error accepting DNS TCP connection: {}", e);
181+
// Resource exhaustion (EMFILE/ENFILE) leaves the pending connection
182+
// queued, so returning straight to `accept()` fails on the same
183+
// connection immediately. That spins at 100% CPU and starves the
184+
// very tasks that would release descriptors. Backing off on every
185+
// accept error keeps a transient failure from becoming an outage.
186+
error!("Error accepting DNS TCP connection, backing off: {e}");
187+
tokio::time::sleep(TCP_ACCEPT_BACKOFF).await;
177188
}
178189
}
179190
}

0 commit comments

Comments
 (0)