Consider a high-throughput edge routing architecture where an OpenResty / Nginx L7 Ingress Gateway routes incoming HTTPS traffic to a cluster of downstream microservices running inside Kubernetes (EKS/GKE). External clients connect over heterogeneous mobile networks (4G/5G and high-latency Wi-Fi), while the internal VPC backbone operates on 10Gbps+ low-latency virtual interfaces.
During a simulated 50,000 QPS flash-sale load test, external clients experienced intermittent HTTP 502 Bad Gateway errors and unexpected TCP connection resets.
# 1. Nginx Worker Error Log 2026/08/04 14:22:01 [warn] 28914#0: *18492001 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000842, client: 10.244.3.18, server: api.zhabrosima.com 2026/08/04 14:22:03 [error] 28914#0: *18492005 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.244.3.18, upstream: "http://10.244.12.91:8080/v1/stream" # 2. Linux Kernel dmesg Alert [10482.110294] TCP: request_sock_TCP: Possible SYN flooding on port 80. Sending cookies. Check SNMP counters. [10485.491204] net_ratelimit: 842 callbacks suppressed
To understand why recv() failed (104: Connection reset by peer) occurs, we must trace how Linux handles TCP socket memory buffers between upstream fast producers (internal microservices) and downstream slow consumers (mobile clients).
[ Downstream Client ] <─────── [ Nginx Memory Buffer ] <─────── [ Upstream Microservice ]
(Slow Mobile 2MB/s) (1MB Pure RAM Cap) (Fast VPC 1GB/s)
│ │
└───────────── Zero Disk Temp Spooling Enabled ──────────────┘
(TCP Flow Control Throttles Upstream)
Nginx acts as a dual-socket bridge. When an upstream service returns a 2MB JSON payload at 1GB/s over local VPC interface, Nginx reads the bytes into memory. However, if the client network consumes data at only 2MB/s, Nginx worker memory fills up immediately.
When socket receive memory (sk_rcvbuf) hits its hard ceiling defined by net.ipv4.tcp_rmem, the Linux kernel invokes tcp_prune_queue() inside net/ipv4/tcp_input.c. If memory remains exhausted, the kernel purges out-of-order packets and sends a TCP RST frame to the upstream peer, instantly killing the connection and emitting an HTTP 502 error.
When Nginx's in-memory proxy_buffers are exhausted, Nginx falls back to spooling payload chunks to disk at /var/cache/nginx/client_temp/. Under 50,000 QPS, this causes heavy NVMe/SSD write amplification. Because Nginx uses an asynchronous, single-threaded event loop per worker, blocking I/O calls to disk freeze all other client sockets multiplexed on that worker thread!
To resolve this without collapsing host memory, we must tune the Linux Kernel's TCP Window Scaling (RFC 1323) and memory ring allocation limits.
The theoretical maximum throughput of a single TCP connection is determined by the Bandwidth-Delay Product (BDP):
$\text{BDP (Bytes)} = \text{Bandwidth (Bytes/sec)} \times \text{RTT (Seconds)}$
For a cross-region connection with 100ms RTT on a 1Gbps link, $\text{BDP} = (10^9 / 8) \times 0.1 = 12.5\text{ MB}$. Without TCP Window Scaling enabled (which limits window size to 64KB), maximum throughput per connection drops to less than 5Mbps!
# Apply via /etc/sysctl.d/99-sre-network.conf
# 1. Enable TCP Window Scaling (RFC 1323)
net.ipv4.tcp_window_scaling = 1
# 2. Auto-tuning TCP Receive/Write Buffers (min, default, max in Bytes)
# Allocates up to 16MB per socket under high throughput demands
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# 3. Maximum OS Socket Receive/Write Buffer Caps
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# 4. Listen Backlog Queues for High Concurrency SYN Bursts
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 32768
# 5. Enable BBR Congestion Control & Fair Queueing (Requires Kernel 4.9+)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbrBelow is the production-tested Nginx configuration designed to completely eliminate disk temporary file writes while maintaining fast upstream keepalive connections:
http {
# ----------------------------------------------------------------------
# Core I/O & Socket Offloading
# ----------------------------------------------------------------------
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65s;
keepalive_requests 100000;
# ----------------------------------------------------------------------
# In-Memory Proxy Buffering (Zero Disk Spooling Setup)
# ----------------------------------------------------------------------
proxy_buffering on;
# Header buffer allocation
proxy_buffer_size 16k;
# Allocate 16 buffers of 64KB = 1MB pure RAM buffer per request
proxy_buffers 16 64k;
# Expand active transmit buffer threshold
proxy_busy_buffers_size 128k;
# CRITICAL: Setting max_temp_file_size to 0 disables disk temp files entirely
proxy_max_temp_file_size 0;
# ----------------------------------------------------------------------
# Upstream Persistent Connection Pool
# ----------------------------------------------------------------------
upstream backend_microservices {
server 10.244.12.91:8080 max_fails=3 fail_timeout=10s;
server 10.244.12.92:8080 max_fails=3 fail_timeout=10s;
# Keep 256 idle HTTP/1.1 connections alive per worker thread
keepalive 256;
keepalive_time 1h;
keepalive_timeout 60s;
}
server {
listen 80 reuseport;
server_name api.zhabrosima.com;
location /v1/ {
proxy_pass http://backend_microservices;
# Enforce HTTP 1.1 protocol for upstream keepalive recycling
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}When diagnosing TCP socket window collapses or buffer stalls on live production edge nodes, use the following low-overhead CLI tools:
ss)# Display TCP socket internal state, send/receive queues, and window sizes
ss -t -i -a 'sport = :http or sport = :https'
# Key outputs to inspect:
# rcv_space: Current receive window size
# cwnd: Congestion window size
# bytes_acked: Total bytes successfully acknowledgedeBPF)# Trace kernel kfree_skb functions to identify silent TCP buffer prunes
sudo bpftrace -e 'kprobe:kfree_skb { @[stack] = count(); }'
A 10-minute wrk synthetic benchmark was executed against the gateway before and after applying the kernel and buffer configuration:
| Performance Metric | Default Nginx Config | Tuned Zero-Disk Buffer Config | Impact Delta |
|---|---|---|---|
| Sustained QPS | 14,210 QPS | 52,480 QPS | +269.3% |
| Latency p99 | 248.50 ms | 12.10 ms | -95.1% |
| Disk I/O Wait (iowait) | 38.2% CPU wait | 0.00% (Zero disk spool) | -100.0% |
| TCP RST / 502 Rate | 5.68% connection drops | 0.000% drops | Resolved |
proxy_max_temp_file_size 0; is set across all high-throughput location blocks.net.ipv4.tcp_window_scaling = 1 using sysctl net.ipv4.tcp_window_scaling.proxy_http_version 1.1; and proxy_set_header Connection "";.# Alert when Nginx 502 Bad Gateway rate exceeds 0.1% over a 5m window
expr: (sum(rate(nginx_http_requests_total{status="502"}[5m])) / sum(rate(nginx_http_requests_total[5m]))) * 100 > 0.1
for: 2m
labels:
severity: critical
annotations:
summary: "High HTTP 502 rate on {{ $labels.instance }} - Check TCP buffer prunes"