Resolving Kubernetes Ingress-Nginx 502 Bad Gateway Under High Load

1. Diagnostic Symptom & Environmental Context

During traffic spikes exceeding 15,000 QPS, microservices deployed behind ingress-nginx begin exhibiting intermittent HTTP 502 Bad Gateway errors. Strangely, application container logs show HTTP 200 OK responses, indicating the error originates purely within the reverse proxy layer during upstream response decoding.

2. Root Cause Analysis: The Three Fail-Modes

[ Ingress-Nginx Proxy ] ─── Keep-Alive 60s Window ───> [ Backend Pod ]
         │                                                    │
         │  1. Request Reuse (Sends HTTP GET)                 │
         │───────────────────────────────────────────────────>│ (Pod TCP Keep-Alive 50s Timeout!)
         │                                                    │ [ Backend Closes Socket -> Sends TCP RST ]
         │<─────────────── TCP RST Received ──────────────────│
         ▼
[ Return 502 Bad Gateway to User! ]
            

Cause A: Upstream Response Header Memory Exhaustion

In modern microservices utilizing OpenTelemetry, Istio service meshes, or heavy OAuth2 JWT tokens, response headers frequently exceed Nginx's default proxy_buffer_size of 4k/8k. When Nginx encounters a header larger than its buffer allocation, it abruptly closes the connection and emits the upstream sent too big header log error.

Cause B: TCP Keep-Alive Timeout Race Condition

Ingress-Nginx maintains a persistent connection pool to backend Pods. If the backend application (e.g., Node.js or Gunicorn) drops idle TCP connections faster than Nginx's idle keep-alive timeout (default 60s), Nginx may send an incoming HTTP request over a connection the backend is closing. The backend responds with a TCP RST packet, forcing Nginx to return a 502.

Cause C: Pod Rolling Updates & Endpoint Stale TTL

When Kubernetes terminates a Pod during deployments or HPA downscaling, the Pod IP removal from the Endpoints list is asynchronous. If Ingress-Nginx routes traffic to a terminating Pod before its TCP socket finishes draining, requests hit closed ports and return 502 errors.

3. Step-by-Step Remediation Plan

Step 1: Increase Proxy Buffer Memory Allocations & Failover Retry

Apply these annotations directly to your Kubernetes Ingress resource to expand memory buffers and enable graceful retry on stale sockets:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: core-api-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: "nginx"
    # Increase single response header buffer to 16k
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    # Allocate 8 buffers of 16k for total response payload buffering
    nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: "32k"
    # Automatic retry on transient upstream errors
    nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout invalid_header http_502 http_503"
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "3"

Step 2: Align Connection Keep-Alive Timestamps

Ensure backend container keep-alive timeouts are configured to at least 65 seconds, guaranteeing that Ingress-Nginx initiates connection closures before the backend Pod drops idle sockets:

# Example Gunicorn Production Configuration
bind = "0.0.0.0:8080"
workers = 4
keepalive = 75  # Must exceed Ingress-Nginx 60s default timeout

Step 3: Add Container PreStop Lifecycle Hook

Add a preStop sleep delay to your application Pod Deployment spec to ensure active connections drain before SIGTERM:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]

4. Verification & Load Testing

Validate the fix using hey or wrk to generate artificial QPS surges and verify 0% error rates:

# Generate 20,000 requests with 200 concurrent TCP connections
hey -n 20000 -c 200 -m GET https://api.zhabrosima.com/v1/user/profile