Conduit 02: Dynamic Upstream DNS Re-Resolution in Edge Proxying

1. Production Incident Context: Stale Pod IPs & Flapping 502s

In dynamic Kubernetes microservice environments, Pods are ephemeral entities. Horizontal Pod Autoscalers (HPA) continuously scale application Pods up or down based on CPU/Memory thresholds, while rolling updates replace old Pod instances with new ones. Each new Pod receives a dynamic IP assigned from the CNI network plugin (e.g. Cilium or AWS-VPC CNI).

During a deployment rollout of an upstream payment processing service, our edge OpenResty API gateway began throwing intermittent HTTP 502 Bad Gateway errors, despite all Kubernetes Pod health checks reporting 1/1 Running Ready status.

2026/08/04 15:10:04 [error] 31204#0: *2048122 connect() failed (111: Connection refused) while connecting to upstream, client: 172.16.4.102, server: api.zhabrosima.com, upstream: "http://10.244.8.41:8080/v1/payment"
2026/08/04 15:10:06 [error] 31204#0: *2048128 no live upstreams while connecting to upstream, client: 172.16.4.102, server: api.zhabrosima.com, upstream: "http://payment-service.prod.svc.cluster.local/v1/payment"

2. Deep Architecture Mechanics: Static DNS Caching in Nginx Core

To understand why Nginx continues routing traffic to terminated Pod IP addresses (such as 10.244.8.41 in the error log above), we must analyze Nginx's startup configuration parsing phase.

[ 1. Standard Static Upstream Resolution ]
 Nginx Master Start ──> gethostbyname() ──> IP 10.244.8.41 (Cached Indefinitely)
 (Pod Terminated ──> Nginx keeps sending SYN to 10.244.8.41 ──> 502 BAD GATEWAY!)

[ 2. Variable-Driven Dynamic Resolver ]
 Incoming Request ──> $upstream_var ──> Cosocket Query CoreDNS (TTL=5s) ──> Fresh Pod IP
 (Pod Terminated ──> CoreDNS Updates ──> Proxy Routes to New Pod Instantly!)
            

The Startup DNS Resolution Pitfall

When Nginx parses a standard upstream block containing domain names:

# Standard Nginx Config (Static Parsing)
upstream payment_backend {
    server payment-service.prod.svc.cluster.local:8080;
}

Nginx invokes the system gethostbyname() glibc function only once during master process startup or configuration reload (SIGHUP). It resolves payment-service.prod.svc.cluster.local to its current set of IP addresses and bakes those IP addresses directly into the in-memory ngx_http_upstream_rr_peer_t data structure.

3. Non-Blocking Async DNS Resolution via OpenResty Cosocket

To force Nginx to respect CoreDNS TTL records dynamically without restarting workers, we must trigger runtime DNS re-resolution using variables in the proxy_pass directive alongside OpenResty's non-blocking Lua cosocket engine.

When proxy_pass uses a variable name (e.g. proxy_pass http://$upstream_endpoint;), Nginx evaluates the variable for every incoming HTTP request and invokes its internal resolver engine according to the TTL parameter.

# Production OpenResty / Nginx Dynamic Upstream Architecture Config

http {
    # 1. Point to Kubernetes Internal CoreDNS Cluster IP
    # 'valid=5s' forces re-resolution every 5 seconds regardless of TTL
    resolver 10.96.0.10 valid=5s ipv6=off;
    resolver_timeout 2s;

    # Shared memory zone for caching resolved upstream sockets across workers
    lua_shared_dict dns_cache 10m;

    server {
        listen 80 reuseport;
        server_name api.zhabrosima.com;

        location /v1/payment/ {
            # 2. Assign upstream FQDN to a variable to enable runtime re-resolution
            set $upstream_endpoint "payment-service.prod.svc.cluster.local";
            
            # 3. Dynamic proxy pass forcing runtime DNS lookup
            proxy_pass http://$upstream_endpoint:8080;

            # HTTP 1.1 Persistent connection parameters
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;

            # 4. Graceful retry on transient socket errors during Pod termination
            proxy_next_upstream error timeout http_502 http_503;
            proxy_next_upstream_tries 3;
            proxy_next_upstream_timeout 5s;
        }
    }
}

4. Real-World SRE Live Diagnostic Toolkit

Use the following CLI commands on your Ingress nodes to diagnose CoreDNS lookup latencies and active TCP socket states during Pod rolling updates:

1. Inspect CoreDNS A-Record TTL & IPs

# Trace CoreDNS resolution and verify TTL counter (e.g., 5 seconds)
dig @10.96.0.10 payment-service.prod.svc.cluster.local +nocmd +noall +answer

# Expected Output:
# payment-service.prod.svc.cluster.local. 5 IN A 10.244.12.105
# payment-service.prod.svc.cluster.local. 5 IN A 10.244.14.88

2. Trace Socket Connection Refused Events with eBPF

# Monitor TCP connection failures to stale Pod IPs in real time
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_connect { $sa = (struct sockaddr_in *)args->uservaddr; if ($sa->sin_family == AF_INET) { @dest[ntohs($sa->sin_port)] = count(); } }'

5. Keep-Alive Drain & Stale Socket Eviction Strategy

Dynamic DNS re-resolution alone is insufficient if existing HTTP Keep-Alive connections remain open in Nginx's connection pool.

• Upstream Connection Max Age: Set keepalive_timeout 60s; and keepalive_requests 1000; to periodically cycle persistent sockets.
• Graceful Pre-Stop Delay: Configure preStop: exec: command: ["/bin/sleep", "10"] on upstream Pods to allow edge proxies to drain connections before final SIGTERM signal.

6. Verified Benchmark Results: Static vs Dynamic Resolver

We simulated a 20-minute continuous HPA scaling event (Pods scaling from 5 → 25 → 5) under 20,000 QPS load to measure error rates:

7. Prometheus Observability (PromQL Queries)

Monitor CoreDNS query rates and Nginx upstream response codes to ensure dynamic re-resolution functions flawlessly:

• CoreDNS Query Volume per Proxy: sum(rate(coredns_dns_requests_total{zone="cluster.local"}[5m])) by (plugin)
• Upstream Connect Failure Alert: rate(nginx_http_requests_total{status=~"502|503"}[1m]) > 0