
Sometimes infrastructure creates the illusion of perfect health. CPU utilization graphs look great with their low numbers — but Load Average is skyrocketing, and the connection pool is completely exhausted. The system stops accepting new incoming traffic for a simple reason: worker slots are clogged with old, stuck connections.
Three hidden points of failure:
- Outbound calls without timeouts. The service goes down not because of incoming clients, but because of a stuck external partner that never returned a response over the network in time.
- Synchronous pool blocking. Classic synchronous workers or thread pools replace asynchronous waiting with a full block of the worker slot. A dozen slow clients can easily exhaust the entire limit.
- Weak load balancer tuning. Default timeouts in Nginx allow slow clients to burn through descriptors if you forget to raise worker_rlimit_nofile in the global config section to match the system ulimit.
How to diagnose and fix:
- Move timeouts for incoming and outgoing sockets into configs, environment variables, or a Service Mesh — don't hardcode them.
- Analyze dumps with ss. A growing CLOSE_WAIT count means the client sent a FIN, but the application is stuck and never calls close(). A growing FIN_WAIT_2 means we closed the socket first and are hanging while waiting for a response.
- Strictly control file descriptor limits at both the OS and load balancer levels.
How do you deal with stuck connections? Do you set limits on the proxy, or at the framework level?