Nginx Upstream Keepalive Starvation: Why Omitting keepalive Chokes Backends at 10k RPS

Problem Description

Under 8k+ RPS, Nginx edge proxies started returning 502 Bad Gateway to clients. Nginx error logs reported:

connect() failed (99: Cannot assign requested address) while connecting to upstream

At the same time, p99 latency spiked from 8ms to 140ms.

Socket State Inspection

Checking socket stats via ss -s:

Total: 34100
TCP:   32800 (estab 4200, closed 27500, orphaned 12, timewait 27480)

Over 27,000 sockets were sitting in TIME_WAIT state, depleting ip_local_port_range (32768-60999). Nginx was creating and tearing down an individual TCP connection for every single HTTP request.

Proper Nginx Upstream Configuration

To persist TCP connections to upstreams, Nginx requires both keepalive inside upstream and HTTP/1.1 headers inside location:

upstream backend_cluster {
    server 127.0.0.1:8080;
    keepalive 128;
    keepalive_requests 10000;
}

server {
    listen 80;
    location / {
        proxy_pass http://backend_cluster;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Kernel Network Optimizations

net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_fin_timeout = 15

Results

• TIME_WAIT sockets plummeted from 27,500 to under 350.
• p99 latency settled at 4.5ms at 12,000 RPS.
• Nginx worker CPU utilization dropped by 35% by eliminating redundant TCP handshakes.