coderain guide

Debugging Kubernetes: Tools and Techniques for Troubleshooting

Kubernetes (K8s) has become the de facto orchestration platform for containerized applications, enabling scalability, resilience, and automation. However, its distributed nature—with components like pods, nodes, services, and the control plane—introduces complexity when things go wrong. Pods crash, services fail to route traffic, nodes run out of resources, or the control plane becomes unresponsive. Debugging these issues requires a systematic approach, combined with the right tools and techniques. In this blog, we’ll demystify Kubernetes troubleshooting by breaking down common scenarios, essential tools, and proven techniques to diagnose and resolve issues efficiently. Whether you’re a developer deploying apps or an SRE managing clusters, this guide will help you navigate the complexities of Kubernetes debugging.

Table of Contents

  1. Understanding Kubernetes Architecture: A Primer
  2. Common Kubernetes Troubleshooting Scenarios
  3. Essential Debugging Tools
  4. Proven Troubleshooting Techniques
  5. Best Practices for Minimizing Downtime
  6. Conclusion
  7. References

1. Understanding Kubernetes Architecture: A Primer

Before diving into debugging, it’s critical to grasp Kubernetes’ core components. A typical cluster includes:

  • Control Plane: Manages the cluster state. Components:

    • kube-apiserver: Frontend for the Kubernetes API.
    • etcd: Distributed key-value store for cluster data.
    • kube-scheduler: Assigns pods to nodes.
    • kube-controller-manager: Runs controller processes (e.g., node controller, replication controller).
    • cloud-controller-manager: Integrates with cloud providers (optional).

  • Nodes: Worker machines running containers. Components:

    • kubelet: Ensures pods run as per manifests.
    • kube-proxy: Maintains network rules on nodes.
    • Container Runtime (e.g., containerd, CRI-O): Executes containers.

  • Pods: Smallest deployable units, containing one or more containers.

  • Services: Expose pods as network services (e.g., ClusterIP, NodePort, LoadBalancer).

  • ConfigMaps/Secrets: Store configuration and sensitive data.

2. Common Kubernetes Troubleshooting Scenarios

Let’s explore the most frequent issues and how to diagnose them.

2.1 Pod Issues

Pods are the workhorses of Kubernetes, and issues here are the most common.

Pending Pods

A pod stays in Pending state when Kubernetes cannot schedule it to a node.

Causes:

  • Insufficient resources (CPU/memory) on nodes.
  • Node taints (no matching tolerations on the pod).
  • Image pull failures (invalid image name, registry access denied).
  • PersistentVolume (PV) unavailability (no matching PersistentVolumeClaim (PVC)).

Debugging Steps:

  1. Check pod details with kubectl describe pod <pod-name>. Focus on the Events section at the bottom: 
    kubectl describe pod my-pending-pod
    Example output snippet: 
    Events:  
  Type     Reason            Age                  From               Message  
  ----     ------            ----                 ----               -------  
  Warning  FailedScheduling  2m (x12 over 5m)     default-scheduler  0/3 nodes are available: 3 Insufficient cpu.
  2. If resources are the issue, check node capacity with kubectl top nodes and adjust pod resource requests/limits.
  3. For taints, check node taints with kubectl describe node <node-name> | grep Taint and add tolerations to the pod manifest.

Crashing Pods

Pods in CrashLoopBackOff, Error, or Failed states indicate container failures.

Causes:

  • Application errors (e.g., unhandled exceptions).
  • Missing dependencies (e.g., config files, environment variables).
  • Liveness/readiness probe failures.
  • Out-of-memory (OOM) kills.

Debugging Steps:

  1. Check logs for the failed container: 
    kubectl logs <pod-name>  # Latest logs  
kubectl logs <pod-name> --previous  # Logs from the last crash
  2. Inspect exit codes (e.g., 137 = OOM kill, 1 = application error) with kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'.
  3. For probe failures, check kubectl describe pod <pod-name> under Liveness Probe or Readiness Probe sections. Adjust probe thresholds (e.g., initialDelaySeconds).

Image Pull Errors

Pods fail with ImagePullBackOff if the container runtime cannot fetch the image.

Causes:

  • Invalid image name/tag (e.g., my-app:latest vs. my-app:v1).
  • Private registry access denied (missing imagePullSecrets).
  • Network issues on the node (cannot reach the registry).

Debugging Steps:

  1. Verify the image exists: docker pull <image-name> (run on a node with Docker, or use a debug pod).
  2. Check for imagePullSecrets in the pod manifest: 
    spec:  
  imagePullSecrets:  
    - name: my-registry-secret  # Must exist in the namespace
  3. Test registry connectivity from a node: curl -I https://index.docker.io/v2/ (for Docker Hub).

2.2 Network Connectivity Problems

Kubernetes networking is notoriously complex, with issues ranging from service misconfigurations to CNI (Container Network Interface) failures.

Service Not Routing to Pods

Services fail to forward traffic to pods.

Causes:

  • Service selector mismatch (labels on pods don’t match the service’s selector).
  • Pods not ready (readiness probe failed).
  • Network policies blocking traffic.

Debugging Steps:

  1. Check if the service has endpoints (pods matching the selector): 
    kubectl get endpoints <service-name>
    If endpoints are empty, fix the service selector or pod labels.
  2. Test direct pod connectivity with kubectl port-forward <pod-name> 8080:80 and access localhost:8080.
  3. Inspect network policies with kubectl get networkpolicy and ensure they allow traffic to the pod.

Inter-Pod/External Connectivity Issues

Pods cannot communicate with other pods or external services.

Causes:

  • CNI plugin issues (e.g., Calico, Flannel not running).
  • Firewall rules blocking pod IP ranges.
  • DNS resolution failures (common with kube-dns/coredns).

Debugging Steps:

  1. Check CNI pods in kube-system: kubectl get pods -n kube-system | grep calico (or your CNI).
  2. Test DNS resolution from a pod: 
    kubectl exec -it <pod-name> -- nslookup kubernetes.default.svc.cluster.local
    If DNS fails, check coredns logs: kubectl logs -n kube-system -l k8s-app=kube-dns.

2.3 Control Plane Failures

Control plane issues (e.g., API server downtime) can bring the entire cluster to a halt.

Causes:

  • etcd data corruption or disk full.
  • kube-apiserver crashes (e.g., misconfigured flags).
  • Node resource exhaustion (control plane nodes run out of CPU/memory).

Debugging Steps:

  1. Check control plane pods in kube-system (for managed clusters like EKS/GKE) or systemd services (for self-managed clusters): 
    # For systemd-managed clusters (e.g., kubeadm)  
sudo systemctl status kube-apiserver  
sudo journalctl -u kube-apiserver -f  # Live logs
  2. For etcd issues, check disk usage on the etcd node: df -h /var/lib/etcd.
  3. Verify API server health: kubectl get --raw=/healthz (should return ok).

2.4 Resource Constraints

Pods or nodes may suffer from CPU throttling, OOM kills, or disk pressure.

Causes:

  • Missing resource limits/requests (pods starve or hog resources).
  • Nodes reaching MemoryPressure or DiskPressure taints.

Debugging Steps:

  1. Monitor resource usage with kubectl top pods and kubectl top nodes.
  2. Check node conditions for pressure: 
    kubectl describe node <node-name> | grep -A 10 "Conditions"
  3. For OOM kills, check kernel logs on the node: sudo dmesg | grep -i 'out of memory'.

2.5 Stateful Application Issues

StatefulSets (e.g., databases, message brokers) face unique challenges like network identity and storage persistence.

Causes:

  • PV/PVC binding failures (no available PVs).
  • Headless service misconfiguration (required for stable network identities).
  • Data corruption in PVs.

Debugging Steps:

  1. Check PV/PVC status: kubectl get pv and kubectl get pvc.
  2. For StatefulSets, verify the headless service exists and matches the serviceName in the StatefulSet manifest.

3. Essential Debugging Tools

Equipping yourself with the right tools accelerates troubleshooting.

3.1 kubectl: The Swiss Army Knife

kubectl is Kubernetes’ command-line tool—learn these critical commands:

CommandUse Case
kubectl get pods -o wideList pods with IP, node, and status.
kubectl describe pod <pod-name>Detailed pod metadata, events, and status.
kubectl logs <pod-name> -c <container-name>Fetch logs for a specific container.
kubectl exec -it <pod-name> -- /bin/bashRun a shell in a container.
kubectl debug <pod-name> --image=busyboxCreate a temporary debug pod with tools.
kubectl get events --sort-by='.lastTimestamp'View cluster events (sorted by time).

3.2 Logging Tools

Centralized logging aggregates logs from pods, nodes, and control plane components.

  • ELK Stack (Elasticsearch, Logstash, Kibana): Ingest, store, and visualize logs.
  • Loki + Grafana: Lightweight logging inspired by Prometheus, with Grafana for dashboards.
  • ** stern**: Tail logs from multiple pods matching a pattern (e.g., stern my-app).

3.3 Monitoring and Observability

Proactive monitoring helps catch issues before they escalate.

  • Prometheus + Grafana: Collect metrics (e.g., pod CPU/memory, API server latency) and build dashboards.
  • kube-state-metrics: Expose Kubernetes object metrics (e.g., pod status, deployment replicas).
  • Datadog/New Relic: Managed observability platforms with Kubernetes integrations.

3.4 Advanced Debugging Tools

  • k9s: Terminal UI for Kubernetes—browse clusters, view logs, exec into pods, and more.
  • kubectx/kubens: Switch between clusters (kubectx) and namespaces (kubens) quickly.
  • kube-ps1: Add Kubernetes context/namespace to your shell prompt (e.g., (prod:default)).
  • kubectl-debug: Plugin to create ephemeral debug pods with tools like tcpdump, curl, and nslookup.

4. Proven Troubleshooting Techniques

4.1 Reproduce the Issue

Reproduce problems in a staging environment (e.g., Minikube, Kind) to avoid disrupting production. Use kubectl apply -f with the same manifests to isolate variables.

4.2 Check Events

Kubernetes events provide a timeline of cluster activity. Use kubectl get events -w to stream events in real time, or kubectl get events --field-selector involvedObject.kind=Pod to filter by pod.

4.3 Inspect Resource Details

Always start with kubectl describe for pods, services, or nodes. It reveals hidden issues like failed mounts, probe timeouts, or taint/toleration mismatches.

4.4 Debug with Temporary Pods

Use kubectl debug to create a pod with debugging tools (e.g., busybox, ubuntu) in the same namespace as the problematic pod. Example:

kubectl debug -it --image=nicolaka/netshoot my-debug-pod  # netshoot has tools like tcpdump, curl

4.5 Network Debugging

  • Port Forwarding: Test pod endpoints locally with kubectl port-forward <pod-name> 8080:80.
  • Service Testing: Use kubectl exec to run curl <service-name>:<port> from another pod.
  • DNS Debugging: Deploy a dnsutils pod to test resolution: 
    kubectl run dnsutils --image=k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.3 -- sleep 3600  
kubectl exec -it dnsutils -- nslookup <service-name>

5. Best Practices for Minimizing Downtime

  • Version Control Manifests: Store YAML files in Git to track changes and roll back quickly.
  • Use Probes: Define liveness/readiness probes to detect and restart unhealthy pods.
  • Set Resource Limits: Prevent resource starvation with resources.limits and resources.requests.
  • Centralize Logs/Metrics: Use ELK/Loki and Prometheus to avoid hunting for logs across nodes.
  • Test Changes: Validate manifests with kubectl apply --dry-run=client before deployment.

Conclusion

Debugging Kubernetes is a blend of understanding architecture, using the right tools, and following systematic techniques. By mastering kubectl, leveraging logging/monitoring tools, and practicing proactive troubleshooting, you can resolve issues faster and keep your cluster resilient. Remember: the key is to start with the basics (pod logs, events) before diving into advanced tools.

References