coderain guide

The Ultimate Guide to Kubernetes Monitoring and Logging

Kubernetes (K8s) has revolutionized how we deploy, scale, and manage containerized applications. Its distributed nature, dynamic orchestration, and self-healing capabilities make it a powerhouse for modern infrastructure. However, this complexity comes with a trade-off: visibility. As clusters grow—spanning nodes, pods, services, and custom resources—tracking performance, diagnosing issues, and ensuring reliability becomes exponentially harder. This is where **monitoring** and **logging** come into play. Monitoring provides real-time insights into the health and performance of your cluster, while logging captures historical data to debug issues and audit activity. Together, they form the backbone of Kubernetes observability, enabling teams to proactively identify problems, optimize resource usage, and maintain system stability. Whether you’re a developer, DevOps engineer, or SRE, this guide will demystify Kubernetes monitoring and logging. We’ll cover key concepts, tools, best practices, and real-world examples to help you build a robust observability stack.

Table of Contents

  1. Why Kubernetes Monitoring and Logging Matter
  2. Key Components to Monitor in Kubernetes
  3. Types of Metrics in Kubernetes
  4. Top Kubernetes Monitoring Tools
  5. Logging Fundamentals in Kubernetes
  6. Log Collection: Tools and Techniques
  7. Log Storage and Analysis
  8. Visualization and Alerting
  9. Best Practices for Kubernetes Monitoring and Logging
  10. Conclusion
  11. References

Why Kubernetes Monitoring and Logging Matter

Kubernetes clusters are dynamic and ephemeral by design. Containers start, stop, and scale automatically; nodes fail; and configurations change frequently. Without robust monitoring and logging, you’re flying blind. Here’s why they’re critical:

1. Reliability & Uptime

Monitoring tracks the health of critical components (e.g., API server, etcd) and alerts you to issues before they cause outages. Logs help diagnose root causes when failures occur (e.g., “Why did my pod crash?”).

2. Troubleshooting

Distributed systems like Kubernetes make debugging hard. Logs from containers, nodes, and control plane components provide a trail of events. Metrics (e.g., high CPU usage) pinpoint bottlenecks.

3. Cost Optimization

Unmonitored clusters often waste resources (e.g., over-provisioned pods). Metrics like container_cpu_usage_seconds_total help right-size resources and reduce cloud bills.

4. Compliance & Auditing

Regulations (e.g., GDPR, HIPAA) require tracking access and changes. Logs of API server requests (kube-apiserver-audit.log) or node access help meet compliance standards.

5. Performance Engineering

Metrics like request latency, throughput, and error rates (e.g., for a microservice) enable you to optimize application performance and user experience.

Key Components to Monitor in Kubernetes

To gain full visibility, you need to monitor every layer of the Kubernetes stack. Here’s a breakdown of critical components:

1. Control Plane

The brain of Kubernetes, consisting of:

  • API Server: The frontend for Kubernetes API. Monitor request latency (apiserver_request_latency_seconds), error rates (apiserver_request_total{code=~"5.."}), and throughput.
  • etcd: The distributed key-value store for cluster state. Track disk usage (etcd_disk_backend_commit_duration_seconds), CPU/memory, and leader election status.
  • Scheduler: Assigns pods to nodes. Monitor pending pods (kube_scheduler_pending_pods) and scheduling failures (kube_scheduler_scheduling_attempts_total{result="failure"}).
  • Controller Manager: Runs controllers (e.g., Deployment, Node). Track reconciliation loops and errors.

2. Nodes

Physical/virtual machines hosting pods. Monitor:

  • Resource Usage: CPU, memory, disk I/O, and network bandwidth (node_cpu_usage_seconds_total, node_memory_usage_bytes).
  • Health: Node status (Ready/NotReady), disk pressure, and out-of-memory (OOM) kills.
  • Kernel Metrics: Load average, context switches, and filesystem usage.

3. Pods & Containers

The smallest deployable units. Track:

  • Status: Running, Pending, CrashLoopBackOff, or Terminated.
  • Resource Usage: CPU/memory limits vs. requests (container_cpu_usage_seconds_total, container_memory_usage_bytes).
  • Startup Time: How long containers take to initialize.

4. Services & Ingress

Network entry points. Monitor:

  • Traffic: Request count (istio_requests_total for Istio, nginx_ingress_controller_requests for NGINX Ingress).
  • Latency: P95/P99 request duration.
  • Error Rates: 4xx/5xx status codes.

5. Custom Resources

Applications or operators (e.g., Prometheus, cert-manager) may expose custom metrics (e.g., certmanager_certificate_ready_status).

Types of Metrics in Kubernetes

Metrics are numerical data points collected at regular intervals. Kubernetes uses three primary types:

1. Resource Metrics

Track CPU, memory, disk, and network usage. Collected by the metrics-server (a lightweight aggregator) or Prometheus. Examples:

  • node_cpu_usage_seconds_total: Cumulative CPU time used by a node.
  • container_memory_usage_bytes: Current memory usage of a container.

2. Custom Metrics

Application-specific or business metrics (e.g., “user signups per minute”). Exposed via APIs (e.g., Prometheus exporters) or the Kubernetes Custom Metrics API. Examples:

  • http_requests_total: Total HTTP requests to a microservice.
  • database_query_latency_seconds: Time taken for database queries.

3. Health Metrics

Indicate liveness/readiness of pods or nodes. Collected via liveness/readiness probes. Examples:

  • kube_pod_status_ready: 1 if pod is ready, 0 otherwise.
  • node_condition: 1 if a node condition (e.g., “DiskPressure”) is true.

4. Network Metrics

Track traffic flow, latency, and errors. Collected via CNI plugins (e.g., Calico, Flannel) or service meshes (e.g., Istio). Examples:

  • container_network_receive_bytes_total: Bytes received by a container.
  • istio_tcp_connections_opened_total: TCP connections opened via Istio.

Top Kubernetes Monitoring Tools

Choosing the right tools depends on your needs (self-hosted vs. managed, complexity, budget). Here are the most popular options:

1. Prometheus + Grafana (Self-Hosted)

  • Prometheus: The de facto standard for metrics collection. Pulls metrics from exporters (e.g., node-exporter for nodes, kube-state-metrics for Kubernetes objects) and stores them in a time-series database.

    • Key Features: Powerful query language (PromQL), alerting rules, and integration with Grafana.
    • Use Case: Self-managed clusters needing flexibility.

  • Grafana: Visualization platform for metrics. Ships with pre-built Kubernetes dashboards (e.g., “Kubernetes Cluster Monitoring”).

    • Example Dashboard: Node CPU/memory usage, pod status, and control plane health.

2. Managed Tools (Datadog, New Relic, Dynatrace)

  • Datadog: Cloud-native monitoring with auto-discovery of Kubernetes resources. Integrates with logs, APM, and security.
  • New Relic: Full-stack observability with Kubernetes-specific dashboards and AI-driven alerting.
  • Dynatrace: AI-powered monitoring with automatic root-cause analysis for Kubernetes and microservices.

3. OpenTelemetry

A CNCF project for unified observability (metrics, logs, traces). Collects data from applications and infrastructure, then exports it to tools like Prometheus or Jaeger.

Logging Fundamentals in Kubernetes

Logs are unstructured or structured records of events. Unlike metrics, they capture what happened (e.g., “Pod X crashed due to OOM”).

1. Log Sources

  • Container Logs: stdout/stderr of containers, stored on nodes at /var/log/pods/<pod_id>/<container_name>/*.log (Docker/containerd).
  • Node Logs: System logs (e.g., /var/log/syslog, /var/log/kern.log).
  • Control Plane Logs: API server, etcd, and controller manager logs (stored in /var/log/containers/ or via systemd).

2. Log Structure

  • Unstructured: Plain text (e.g., 2024-05-20 12:00:00 [INFO] User login). Hard to query.
  • Structured: JSON, CSV, or key-value pairs (e.g., {"timestamp": "2024-05-20T12:00:00Z", "level": "INFO", "message": "User login"}). Easy to filter/aggregate.

3. Challenges

  • Ephemerality: Containers/pods are short-lived; logs are lost if not collected.
  • Scale: Thousands of pods generate massive log volumes.
  • Distributed Systems: Logs span multiple nodes/pods, requiring correlation (e.g., trace IDs).

Log Collection: Tools and Techniques

Log collection tools aggregate logs from sources and forward them to storage/analysis systems.

1. Fluentd & Fluent Bit

  • Fluentd: A popular open-source log collector with a plugin ecosystem. Runs as a DaemonSet to collect node/pod logs.
    • Example Use Case: Collect container logs from /var/log/pods, parse JSON, and send to Elasticsearch.
  • Fluent Bit: A lightweight alternative to Fluentd (written in C), optimized for performance. Ideal for edge environments.

2. Log Collection Patterns

  • DaemonSet: One collector per node (e.g., Fluentd) to scrape all pod logs on that node.
  • Sidecar Container: A dedicated container in a pod to collect logs from the main container (e.g., for apps writing to files instead of stdout).
  • Operator-Based: Tools like OpenTelemetry Operator automate deployment of collectors.

3. Log Forwarding

Collectors send logs to backends via outputs (plugins). Common outputs:

  • Elasticsearch
  • Loki
  • AWS CloudWatch Logs
  • GCP Cloud Logging

Log Storage and Analysis

Once collected, logs need to be stored and analyzed. Key tools:

1. Elasticsearch + Kibana (ELK Stack)

  • Elasticsearch: Distributed search engine for storing and indexing logs. Scales horizontally and supports fast queries.
  • Kibana: Visualization tool for Elasticsearch. Build dashboards, search logs, and create alerts (e.g., “Show all 5xx errors in the last hour”).

2. Loki + Grafana

  • Loki: Open-source log aggregator by Grafana Labs, optimized for Kubernetes. Stores logs as unstructured data with labels (like Prometheus metrics), reducing storage costs.
  • Grafana: Visualize logs alongside metrics. Use LogQL to query logs (e.g., {job="kube-apiserver"} |= "error").

3. Cloud-Native Solutions

  • AWS CloudWatch Logs: Centralized logging for AWS EKS. Integrates with CloudWatch Alarms.
  • Azure Monitor Logs: Collects logs from AKS and Azure resources.
  • GCP Cloud Logging: Native logging for GKE with integration to BigQuery for analysis.

Visualization and Alerting

Observability isn’t just about collecting data—it’s about acting on it.

1. Visualization

  • Grafana Dashboards: Create custom dashboards for metrics (e.g., “Cluster Health” with node CPU, pod status, and etcd disk usage).
  • Kibana Dashboards: Build log-centric dashboards (e.g., “API Error Rate” with 4xx/5xx trends).
  • Unified Dashboards: Tools like Datadog or New Relic combine metrics, logs, and traces in one view.

2. Alerting

  • Prometheus Alertmanager: Define rules in Prometheus (e.g., node_memory_usage_bytes / node_memory_total_bytes > 0.9 for high memory), then send alerts via email, Slack, or PagerDuty.
  • Grafana Alerts: Set alerts on Loki logs (e.g., “Alert on 10+ CrashLoopBackOff events in 5 minutes”).
  • Best Practices: Avoid alert fatigue by prioritizing critical alerts (e.g., “Control plane down”) over warnings (e.g., “Pod restarted”).

Best Practices for Kubernetes Monitoring and Logging

Follow these guidelines to build a robust observability stack:

1. Define SLIs/SLOs

Service Level Indicators (SLIs) (e.g., “99.9% uptime”) and Service Level Objectives (SLOs) guide what to monitor. Alerts should trigger when SLOs are at risk.

2. Use Structured Logs

Format logs as JSON to enable easy filtering (e.g., {"level": "error", "service": "payment"}). Tools like Fluentd or Loki can parse structured logs for better querying.

3. Set Retention Policies

Avoid storing logs indefinitely. Use tools like Elasticsearch ILM (Index Lifecycle Management) or Loki retention rules to delete old data and reduce costs.

4. Monitor the Monitoring Stack

Prometheus, Fluentd, and Elasticsearch are critical—monitor their health (e.g., “Is Prometheus ingesting metrics?”) to avoid blind spots.

5. Secure Sensitive Data

Encrypt logs in transit (TLS) and at rest (e.g., Elasticsearch encryption). Redact PII (e.g., passwords) from logs.

6. Iterate and Improve

Start with core metrics/logs (control plane, nodes), then expand to applications. Review dashboards and alerts regularly to remove noise.

Conclusion

Kubernetes monitoring and logging are foundational to operating reliable, efficient clusters. By combining metrics (for “what’s broken”) and logs (for “why it’s broken”), you gain full visibility into your system.

Start small: Deploy Prometheus + Grafana for metrics, Fluent Bit + Loki for logs, and build dashboards for critical components. As your cluster grows, adopt managed tools or OpenTelemetry for unified observability.

Remember: Observability is a journey, not a destination. Continuously refine your stack to match your team’s needs and the evolving complexity of your applications.

References