Kubernetes Advanced Operations: Packet Paths, Rollouts, DNS, and Operators

1. Diagnose the pod lifecycle before changing probes

Startup, readiness, and liveness probes answer different questions. Startup gives a slow application time to initialize before liveness checks begin. Readiness controls whether a pod should receive traffic. Liveness asks whether restarting the process is a reasonable recovery. A liveness probe that depends on a temporarily slow downstream service can restart healthy processes in a cascading failure.

During a rolling update, intermittent 502s can come from readiness turning true before the listener is ready, termination beginning before endpoint withdrawal propagates, insufficient terminationGracePeriodSeconds, a missing preStop drain interval, or an aggressive maxUnavailable. Align readiness, graceful shutdown, ingress/load-balancer connection draining, and rollout surge/unavailable settings; inspect endpoint changes and timestamps rather than tuning one field in isolation.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          memory: 512Mi
      startupProbe:
        httpGet: { path: /startup, port: 8080 }
        failureThreshold: 30
        periodSeconds: 2
      readinessProbe:
        httpGet: { path: /ready, port: 8080 }
        periodSeconds: 5
      livenessProbe:
        httpGet: { path: /live, port: 8080 }
        periodSeconds: 10

Tune thresholds from startup and shutdown measurements. A readiness endpoint should reflect whether the instance can serve its contract; liveness should not fail merely because a downstream dependency is briefly slow.

2. CPU throttling despite idle nodes

CPU limits are enforced per container quota. A latency-sensitive container can exhaust its quota and be throttled even when the node has spare CPU, because unrelated capacity is not automatically borrowed beyond the configured quota. Compare cgroup throttling counters, container CPU limit, request/limit, and application latency. Decide whether to raise/remove the limit, adjust requests and HPA targets, or reduce CPU work based on the workload contract and cluster policy.

3. Service packet path and node exposure

A client reaches a NodePort on a node address. The node dataplane—iptables rules or IPVS, depending on kube-proxy mode and cluster configuration—selects a Service endpoint and performs destination translation toward a pod IP/port. CNI routes the packet into the pod network namespace; policy and return-path translation determine whether the response succeeds. Exact paths differ across eBPF dataplanes, local versus cluster endpoints, and external traffic policies.

Inspect kubectl get pods -o wide, Service endpoints, node routes, kube-proxy mode/rules, CNI policy, and packet captures. kubectl port-forward is a useful diagnostic bypass but does not prove the normal Service or NodePort path. kubectl debug can add an ephemeral diagnostic container when the workload image lacks tools; follow cluster authorization policy.

kubectl get pods -o wide -n production
kubectl get endpointslice -l kubernetes.io/service-name=api -n production
kubectl port-forward svc/api 8080:80 -n production
kubectl debug pod/api-abc -it --image=nicolaka/netshoot -n production

Use approved diagnostic images and RBAC; ephemeral containers can access sensitive network and process context.

4. CoreDNS and network troubleshooting

Intermittent DNS failures can come from overloaded CoreDNS, upstream resolver delays, conntrack pressure, node-local resolver configuration, or application-side caching. Inspect CoreDNS request/error/latency metrics and logs, pod resource pressure, endpoint distribution, and node resolver settings. Distinguish NXDOMAIN from timeout and SERVFAIL; they indicate different failures.

5. CRDs and controller conflict handling

An operator reconciles observed resources toward a declared desired state. This should be level-triggered: each loop should be safe to repeat and derive required actions from current state, not rely on seeing every event exactly once. Kubernetes resource versions implement optimistic concurrency. If an update conflicts because another writer changed the object, refetch the latest version, recompute the intended patch, and retry with a bounded policy. Do not blindly replay a stale full-object update that overwrites other controllers' fields.

6. Ingress, mesh, and traffic shifts

An Ingress controller implements HTTP routing; a Service provides discovery and endpoint selection inside the cluster. A service mesh sidecar may add mTLS, retries, telemetry, and traffic policy, but it also introduces another hop and configuration plane. Canary traffic shifting should include health gates and rollback criteria; retries can multiply load if the destination is already unhealthy.

Interview and troubleshooting drills

Interview: Why can CPU throttling occur at 40% node utilization? Explain container CFS quota exhaustion, per-container limits, and the difference between node-wide spare capacity and the container's allowed budget.

Troubleshooting: During a rollout, correlate 502 timestamps with pod readiness, endpoint removal, termination grace, preStop behavior, and ingress upstream errors. Fix the drain/readiness mismatch and validate with connection reuse and slow-start traffic.

Challenge: Trace a NodePort packet through node translation, endpoint selection, CNI routing, network policy, pod namespace, and return path. State which assumptions change for IPVS, eBPF, externalTrafficPolicy=Local, and a service mesh.