coderain guide

Kubernetes Troubleshooting 101: Common Issues and Solutions

Kubernetes (K8s) has become the de facto orchestration platform for containerized applications, offering scalability, resilience, and flexibility. However, its complexity—with components like pods, services, deployments, and networks—means even seasoned engineers encounter issues. Whether it’s a pod stuck in `Pending`, a service failing to route traffic, or a node suddenly going `NotReady`, troubleshooting Kubernetes requires a systematic approach, familiarity with key components, and the right tools. This blog is your go-to guide for resolving the most common Kubernetes issues. We’ll break down symptoms, root causes, and step-by-step solutions, equipping you to diagnose and fix problems efficiently. Let’s dive in!

Table of Contents

  1. Understanding the Kubernetes Troubleshooting Workflow
  2. Common Kubernetes Issues and Solutions
  3. Essential Troubleshooting Tools
  4. Best Practices for Effective Troubleshooting
  5. Conclusion
  6. References

1. Understanding the Kubernetes Troubleshooting Workflow

Before diving into specific issues, it’s critical to adopt a systematic workflow. Kubernetes is a distributed system, so problems often stem from interactions between components (e.g., pods, services, nodes). Follow this steps:

  1. Identify the affected component: Is it a pod, service, node, or ingress? Start with the most granular object (e.g., a pod) before moving up the stack.
  2. Check the status: Use kubectl get <resource> (e.g., kubectl get pods) to see the current state (e.g., Pending, CrashLoopBackOff).
  3. Inspect events and details: Run kubectl describe <resource> <name> to view events, configuration, and errors (e.g., kubectl describe pod my-pod).
  4. Check logs: For pods, use kubectl logs <pod> (or kubectl logs <pod> -p to view previous logs).
  5. Validate dependencies: Ensure supporting resources (e.g., services, ConfigMaps, Secrets, StorageClasses) exist and are correctly configured.

2. Common Kubernetes Issues and Solutions

2.1 Pods Not Starting or Stuck in Pending/CrashLoopBackOff

Pods are the smallest deployable units in Kubernetes, so they’re often the first place to look. Common symptoms include:

  • Pending: Pod is not scheduled on any node.
  • CrashLoopBackOff: Pod starts but crashes repeatedly.
  • ImagePullBackOff: Kubernetes fails to pull the container image.

Common Causes

  • Insufficient resources: Nodes lack CPU/memory to meet pod requests.
  • Invalid container image: Image name/tag is incorrect, or the registry is unreachable.
  • Misconfigured probes: Liveness/readiness probes fail, causing the pod to restart or stay unready.
  • Invalid YAML: Syntax errors in the pod/deployment manifest (e.g., missing commas, incorrect indentation).

Troubleshooting Steps & Solutions

Step 1: Check pod status and events
Run kubectl describe pod <pod-name> to view events (under the “Events” section at the bottom). For example:

kubectl describe pod my-pod

Look for messages like:

  • FailedScheduling: 0/3 nodes are available: 3 Insufficient cpu. (Resource issue)
  • Failed to pull image "my-app:v1": rpc error: code = Unknown desc = failed to pull and unpack image (Image pull error)

Step 2: Resolve resource constraints
If events show Insufficient cpu or Insufficient memory, check the pod’s resources in its manifest. Nodes must have enough free resources to satisfy the pod’s requests.

Example fix: Reduce requests or add more nodes to the cluster.

resources:  
  requests:  
    cpu: 100m  # Lower from 500m if nodes are constrained  
    memory: 256Mi  
  limits:  
    cpu: 500m  
    memory: 512Mi

Step 3: Fix image pull issues

  • Verify the image name/tag (e.g., nginx:latest vs. nginx:1.21).
  • Ensure the cluster has access to the registry (e.g., add imagePullSecrets for private registries).
  • Test image pull manually with docker pull <image> (on a node) to confirm the image exists.

Step 4: Debug probes
If probes are failing, check logs to see why the container isn’t responding. For example, a liveness probe might expect a /health endpoint that isn’t implemented.

View probe configuration in kubectl describe pod:

Liveness:   http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3

Fix: Update the probe path/port, or disable probes temporarily (not recommended for production).

2.2 Services Not Routing Traffic Correctly

Services expose pods internally or externally, but traffic may fail to route. Symptoms include:

  • Unable to access pods via the service’s ClusterIP or DNS name.
  • curl <service-name> from another pod returns connection refused or no route to host.

Common Causes

  • Selector mismatch: Service selector doesn’t match pod labels, so no endpoints are created.
  • Port misconfiguration: Service port doesn’t match the container’s containerPort.
  • Pods are unready: Readiness probes fail, so pods aren’t added to the service’s endpoints.

Troubleshooting Steps & Solutions

Step 1: Verify the service and endpoints
Check if the service has endpoints (pods that match its selector):

kubectl describe service <service-name>  # Look for "Endpoints" section  
kubectl get endpoints <service-name>     # List healthy pods attached to the service

If endpoints are empty, the selector is likely incorrect.

Step 2: Fix selector/label mismatch
Ensure the service’s selector matches pod labels. For example:

# Service manifest (incorrect selector)  
apiVersion: v1  
kind: Service  
metadata:  
  name: my-service  
spec:  
  selector:  
    app: my-app  # Pods must have label "app: my-app"  
  ports:  
    - port: 80  
      targetPort: 8080

If pods have app: wrong-name, update the selector or pod labels.

Step 3: Test port connectivity

  • Confirm the service’s targetPort matches the container’s listening port (e.g., container listens on 8080, but targetPort is set to 80).
  • Test directly from a pod: 
    kubectl run test-pod --image=busybox --rm -it -- sh  
wget -qO- <service-name>:<service-port>  # Should return a response if routing works

Step 4: Debug DNS resolution
If the service isn’t reachable via its DNS name (e.g., my-service.default.svc.cluster.local), test DNS with:

kubectl run dns-test --image=busybox --rm -it -- sh  
nslookup my-service.default.svc.cluster.local  # Should resolve to the service’s ClusterIP

If DNS fails, check the coredns pods in the kube-system namespace (they may be crashing or misconfigured).

2.3 Resource Constraints (CPU/Memory Issues)

Pods rely on CPU and memory, and misconfigured requests/limits can cause instability. Symptoms include:

  • Evicted pods (due to node pressure).
  • OOMKilled (Out-of-Memory) errors in pod logs.
  • Slow performance or unresponsive applications.

Common Causes

  • Limits too low: Pod exceeds its memory limit, triggering an OOM kill.
  • No requests set: Pods without requests may be scheduled on nodes with insufficient resources.
  • Node resource exhaustion: Nodes run out of CPU/memory due to overcommitment.

Troubleshooting Steps & Solutions

Step 1: Identify resource-hungry pods
Use kubectl top to check pod resource usage:

kubectl top pods --sort-by=cpu  # Sort pods by CPU usage  
kubectl top pods --sort-by=memory  # Sort by memory usage

Step 2: Adjust requests/limits
Set requests (minimum resources required) and limits (maximum allowed) to prevent starvation and OOM kills:

resources:  
  requests:  
    cpu: 200m    # 200 millicores (0.2 CPU)  
    memory: 256Mi  
  limits:  
    cpu: 500m    # Prevent CPU throttling if exceeded  
    memory: 512Mi  # OOM kill if exceeded

Step 3: Use auto-scaling
For dynamic workloads, use:

  • Horizontal Pod Autoscaler (HPA): Scale pod count based on CPU/memory usage.
  • Vertical Pod Autoscaler (VPA): Automatically adjust requests/limits (requires the VPA add-on).

2.4 Network Policy Blocks (Unexpected Connectivity Issues)

Network Policies control traffic flow between pods, but overly restrictive policies can block legitimate traffic. Symptoms include:

  • Pods unable to communicate with each other suddenly.
  • curl <pod-ip> from another pod returns connection timed out.

Common Causes

  • Overly restrictive policies: A policy denies all ingress/egress by default, blocking necessary traffic.
  • Incorrect CIDR ranges: Policies target the wrong IP ranges (e.g., blocking the cluster’s internal CIDR).

Troubleshooting Steps & Solutions

Step 1: Check network policies
List all network policies in the namespace and describe them:

kubectl get networkpolicy -n <namespace>  
kubectl describe networkpolicy <policy-name> -n <namespace>

Step 2: Test connectivity
Use a test pod to simulate traffic and identify blocked paths:

kubectl run test-pod --image=busybox --rm -it -- sh  
telnet <target-pod-ip> <port>  # Check if the port is reachable

Step 3: Relax the policy
If a policy is too strict, update it to allow necessary traffic. For example, allow ingress from pods with label app: frontend:

apiVersion: networking.k8s.io/v1  
kind: NetworkPolicy  
metadata:  
  name: allow-frontend  
spec:  
  podSelector:  
    matchLabels:  
      app: backend  # Pods to protect  
  ingress:  
  - from:  
    - podSelector:  
        matchLabels:  
          app: frontend  # Allow traffic from frontend pods

2.5 StatefulSet Failures (e.g., Volume Claims, DNS Issues)

StatefulSets manage stateful applications (e.g., databases) with stable network identities and persistent storage. Common issues include:

  • Pods failing to create (e.g., 0/x nodes are available: x pod has unbound immediate PersistentVolumeClaims).
  • DNS resolution failures for stateful pod names (e.g., my-statefulset-0.my-service.default.svc.cluster.local).

Common Causes

  • Missing headless service: StatefulSets require a headless service (no ClusterIP) to assign stable DNS names.
  • StorageClass unavailable: No StorageClass exists to provision the PersistentVolumeClaim (PVC).
  • PVCs stuck in Pending: The storage backend (e.g., AWS EBS, GCP PD) can’t provision volumes.

Troubleshooting Steps & Solutions

Step 1: Check PVC status
StatefulSets create PVCs with names like <statefulset-name>-<ordinal>-<volume-name>. Verify they’re bound:

kubectl get pvc -n <namespace>

If PVCs are Pending, check the StorageClass:

kubectl get sc  # Ensure a default StorageClass exists

Step 2: Verify the headless service
StatefulSets require a headless service with the same serviceName as specified in the StatefulSet:

# Headless service (required)  
apiVersion: v1  
kind: Service  
metadata:  
  name: my-stateful-service  
spec:  
  clusterIP: None  # Headless (no ClusterIP)  
  selector:  
    app: my-statefulset  
  ports:  
    - port: 80

Step 3: Debug DNS for stateful pods
Stateful pod DNS names follow the format <pod-name>.<service-name>.<namespace>.svc.cluster.local. Test resolution:

kubectl run dns-test --image=busybox --rm -it -- sh  
nslookup my-statefulset-0.my-stateful-service.default.svc.cluster.local

2.6 Ingress Controller Problems (404/503 Errors)

Ingress controllers (e.g., NGINX, Traefik) route external traffic to services. Common issues:

  • 404 Not Found: Ingress rule doesn’t match the request path.
  • 503 Service Unavailable: Backend service has no healthy endpoints.

Common Causes

  • Ingress controller not running: The controller pod (e.g., nginx-ingress-controller) is CrashLoopBackOff.
  • Path/host mismatch: Ingress rules don’t match the request’s host/path.
  • TLS misconfiguration: Invalid certificates or missing tls section in the ingress manifest.

Troubleshooting Steps & Solutions

Step 1: Check the ingress controller
Ensure the controller is running and has healthy pods:

kubectl get pods -n <ingress-namespace>  # e.g., ingress-nginx

Step 2: Verify ingress rules
Describe the ingress to check backend services and paths:

kubectl describe ingress <ingress-name>

Step 3: Check controller logs
Ingress controllers log traffic and errors. For NGINX:

kubectl logs -n ingress-nginx <nginx-controller-pod>  # Look for 404/503 entries

2.7 Node-Level Issues (Unhealthy Nodes, Disk Pressure)

Nodes are the worker machines in Kubernetes. Issues here affect all pods on the node. Symptoms include:

  • Nodes marked NotReady.
  • DiskPressure or MemoryPressure taints on nodes (pods are evicted).

Common Causes

  • Kubelet failure: The kubelet (agent on nodes) is down or misconfigured.
  • Resource exhaustion: Node disk/CPU/memory is full.
  • Network issues: Node can’t communicate with the API server.

Troubleshooting Steps & Solutions

Step 1: Check node status and events

kubectl get nodes  
kubectl describe node <node-name>  # Look for "Conditions" (e.g., DiskPressure)

Step 2: Fix disk pressure
If DiskPressure is detected, free up space on the node:

# On the node (via SSH)  
df -h  # Check disk usage  
du -sh /var/lib/docker  # Docker often uses significant space  
docker system prune -a  # Clean up unused containers/images (use with caution!)

Step 3: Restart the kubelet
If the kubelet is unresponsive:

# On the node  
sudo systemctl restart kubelet  
sudo journalctl -u kubelet -f  # Check kubelet logs for errors

3. Essential Troubleshooting Tools

  • kubectl: The primary CLI for interacting with Kubernetes clusters (e.g., kubectl describe, kubectl logs).
  • stern: Tail logs from multiple pods/containers (e.g., stern my-deployment).
  • k9s: A terminal UI for managing Kubernetes clusters (simplifies viewing pods, logs, and events).
  • kube-ps1: A shell prompt plugin that shows the current Kubernetes context/namespace.
  • Prometheus + Grafana: Monitor cluster metrics (CPU, memory, pod status) to identify trends.
  • kube-debug: A plugin for debugging pods with ephemeral debug containers (e.g., kubectl debug -it <pod> --image=busybox).

4. Best Practices for Effective Troubleshooting

  • Reproduce the issue: Replicate the problem in a staging environment to avoid disrupting production.
  • Document everything: Log commands run, error messages, and fixes for future reference.
  • Monitor proactively: Use tools like Prometheus to set alerts for common issues (e.g., high pod restarts, OOM kills).
  • Version control manifests: Store YAML files in Git to track changes and roll back quickly.
  • Understand the architecture: Learn how components (pods, services, CNI, CSI) interact to diagnose root causes faster.

5. Conclusion

Kubernetes troubleshooting can be challenging, but with a systematic approach, the right tools, and an understanding of common issues, you can resolve problems efficiently. Start with the pod level, check events and logs, and validate dependencies to isolate root causes. Remember: Kubernetes is a distributed system—issues rarely exist in isolation, so always consider the bigger picture!

6. References