Networking in Kubernetes: Understanding CNI Plugins

Table of Contents

1. What is Kubernetes Networking?
2. The Role of CNI in Kubernetes
3. How CNI Works: Core Concepts
4. Types of CNI Plugins
5. Selecting the Right CNI Plugin
6. Installing and Configuring a CNI Plugin
7. Troubleshooting CNI Plugins
8. Conclusion
9. References

1. What is Kubernetes Networking?

Kubernetes networking addresses the unique challenges of containerized environments, where pods (the smallest deployable units) are created, destroyed, and migrated dynamically. The Kubernetes network model defines four core requirements:

• Pod-to-Pod Communication: All pods must communicate with each other without Network Address Translation (NAT), regardless of the node they run on.
• Pod IP Uniqueness: Each pod gets a unique IP address within the cluster.
• Node-to-Pod Communication: Nodes must communicate with pods, and vice versa.
• External Network Access: Pods must access external networks (e.g., the internet), and external clients must access pods (via Services).

To meet these requirements, Kubernetes relies on CNI plugins to manage network configuration. Without CNI, pods would lack network connectivity, rendering the cluster useless.

2. The Role of CNI in Kubernetes

CNI (Container Network Interface) is not a plugin itself but a specification developed by the Cloud Native Computing Foundation (CNCF). It standardizes how container runtimes and network plugins interact to configure networking for containers.

Why CNI?

Before CNI, container runtimes (e.g., Docker) used proprietary networking tools (e.g., docker0 bridge), leading to vendor lock-in and inconsistent behavior across environments. CNI solves this by defining a common interface for network plugins, ensuring interoperability between runtimes (containerd, CRI-O) and plugins (Calico, Flannel, etc.).

How Kubernetes Uses CNI

Kubernetes delegates network configuration to CNI via the Kubelet (the agent running on each node). When a pod is scheduled, the Kubelet:

1. Creates the pod’s network namespace.
2. Invokes the CNI plugin specified in the cluster’s CNI configuration.
3. The plugin assigns an IP to the pod, sets up network interfaces (e.g., eth0), and configures routes.
4. The pod is then ready to communicate with other pods and external networks.

3. How CNI Works: Core Concepts

To understand CNI, let’s break down its workflow, components, and key operations.

CNI Workflow

When a pod is created or deleted, the CNI plugin performs the following steps:

Pod Creation (CNI ADD Command)

1. The Kubelet creates a network namespace for the pod.
2. It calls the CNI plugin with the ADD command, passing:
   • The pod’s network namespace path (e.g., /var/run/netns/<pod-id>).
   • The pod’s name and namespace (metadata for context).
3. The plugin:
   • Creates a virtual interface (e.g., veth pair) connecting the pod’s namespace to the host’s network.
   • Assigns an IP address to the pod (via an IP Address Management (IPAM) plugin, e.g., host-local).
   • Configures routes on the host to forward traffic to/from the pod.
   • (Optional) Enforces network policies (e.g., Calico’s policy engine).

Pod Deletion (CNI DEL Command)

When a pod is deleted, the Kubelet calls the CNI plugin with the DEL command to:

• Remove the pod’s virtual interface.
• Release the IP address (via IPAM).
• Clean up routes and network rules.

CNI Components

CNI relies on three key components:

1. CNI Plugins: Binaries (e.g., calico, flannel) that implement the CNI specification. Plugins are stored in /opt/cni/bin/ on each node.
2. CNI Configuration Files: JSON/YAML files in /etc/cni/net.d/ that define which plugin to use and its settings (e.g., IP range, MTU). Only the lexicographically first config file is used (e.g., 10-calico.conflist).
3. IPAM Plugins: Sub-plugins responsible for IP address assignment (e.g., host-local for static IP pools, dhcp for dynamic IPs from a DHCP server). IPAM is specified in the CNI config under the ipam field.

Example CNI Configuration File

A typical CNI configuration file (e.g., /etc/cni/net.d/10-calico.conflist) looks like this:

{  
  "cniVersion": "0.3.1",  
  "name": "calico",  
  "plugins": [  
    {  
      "type": "calico",  
      "log_level": "info",  
      "datastore_type": "kubernetes",  
      "nodename": "node-1",  
      "ipam": {  
        "type": "calico-ipam"  
      },  
      "policy": {  
        "type": "k8s"  
      }  
    },  
    {  
      "type": "portmap",  
      "capabilities": {"portMappings": true}  
    }  
  ]  
}  

• cniVersion: The CNI spec version (e.g., 0.3.1).
• name: The network name (e.g., calico).
• type: The CNI plugin to use (e.g., calico).
• ipam: IP address management plugin (e.g., calico-ipam for Calico’s IP pool).
• plugins: A chain of plugins (e.g., calico for networking, portmap for port forwarding).

4. Types of CNI Plugins

CNI plugins vary in architecture, performance, and features. Below are the most popular categories and examples:

1. Overlay Network Plugins

Overlay plugins create a virtual network on top of the host network (underlay), using encapsulation (e.g., VXLAN) to route traffic between nodes.

Flannel

• Architecture: Uses VXLAN (default) or host-gw (direct routing) to connect nodes. A flanneld daemon on each node manages IP allocation via etcd.
• Pros: Simple to install, lightweight, ideal for small to medium clusters.
• Cons: Limited features (no network policies), VXLAN adds latency.
• Use Case: Dev/test environments, clusters with basic networking needs.

Weave Net

• Architecture: Uses a mesh of routers on each node, with its own encapsulation protocol (Weave WireGuard for encryption).
• Pros: Built-in encryption, automatic IP allocation, easy to deploy.
• Cons: Higher resource usage than Flannel, less mature than Calico.
• Use Case: Clusters requiring secure pod-to-pod communication.

2. Underlay Network Plugins

Underlay plugins use the existing physical network (e.g., BGP, SR-IOV) to route traffic, avoiding encapsulation overhead.

Calico

• Architecture: Uses BGP (Border Gateway Protocol) to route traffic between nodes. A calico-node daemon on each node manages BGP peers and network policies.
• Pros: High performance (no encapsulation by default), rich network policies (L3/L4/L7 rules), scalable to 1000+ nodes.
• Cons: Requires BGP knowledge for advanced configuration.
• Use Case: Production clusters, multi-tenant environments, clusters needing fine-grained security.

SR-IOV CNI

• Architecture: Uses Single Root I/O Virtualization (SR-IOV) to assign direct physical network interfaces (VF) to pods, bypassing the host OS kernel.
• Pros: Ultra-low latency, high throughput (near-bare-metal performance).
• Cons: Requires SR-IOV-capable hardware, complex to configure.
• Use Case: High-performance workloads (e.g., AI/ML, financial trading).

3. eBPF-Based Plugins

eBPF (extended Berkeley Packet Filter) plugins use Linux kernel hooks to process traffic in-kernel, enabling high performance and advanced features.

Cilium

• Architecture: Uses eBPF to enforce L3/L4/L7 network policies, route traffic, and provide load balancing. Integrates with Envoy for service mesh capabilities.
• Pros: High performance (eBPF is faster than user-space proxies), deep observability ( Hubble UI), L7 policies (HTTP, gRPC).
• Cons: Requires Linux kernel ≥ 5.4 (for advanced eBPF features).
• Use Case: Microservices, service meshes, clusters needing L7 security.

5. Selecting the Right CNI Plugin

Choosing a CNI plugin depends on your cluster’s requirements. Key factors to consider:

Recommendation Summary:

• Start Simple: Use Flannel for small clusters or dev/test.
• Production-Grade: Use Calico for scalability and network policies.
• High Performance: Use Cilium (eBPF) or SR-IOV for low-latency workloads.

6. Installing and Configuring a CNI Plugin

Let’s walk through installing Calico, one of the most popular production-grade CNI plugins.

Prerequisites

• A Kubernetes cluster (v1.21+ recommended).
• kubectl configured to access the cluster.

Step 1: Install Calico

Calico is deployed via a Kubernetes manifest. Run:

kubectl apply -f https://docs.projectcalico.org/v3.26/manifests/calico.yaml  

This deploys:

• calico-node daemonset (runs on every node).
• calico-kube-controllers deployment (manages network policies).
• CRDs for network policies, BGP configurations, etc.

Step 2: Verify Installation

Check if Calico pods are running:

kubectl get pods -n calico-system  

Output should show calico-node-* and calico-kube-controllers-* pods in Running state.

Check node status (Calico-specific):

kubectl exec -it -n calico-system calico-node-<pod-id> -- calicoctl node status  

You should see BGP peers (nodes) connected.

Step 3: CNI Configuration

Calico’s CNI config is stored in /etc/cni/net.d/10-calico.conflist on each node. Example snippet:

{  
  "cniVersion": "0.3.1",  
  "name": "calico",  
  "plugins": [  
    {  
      "type": "calico",  
      "ipam": { "type": "calico-ipam" },  
      "policy": { "type": "k8s" }  
    }  
  ]  
}  

• ipam: calico-ipam: Uses Calico’s IP pool (configured via IPPool CRD).
• policy: k8s: Enables Kubernetes NetworkPolicy support.

7. Troubleshooting CNI Plugins

Common CNI issues and how to resolve them:

Issue 1: Pods Stuck in ContainerCreating

Pods fail to start because the CNI plugin didn’t configure the network.

Debug Steps:

• Check Kubelet logs for CNI errors:

  journalctl -u kubelet | grep -i cni  

• Verify CNI config exists in /etc/cni/net.d/ (lexicographically first file is used).
• Ensure CNI plugin binaries are present in /opt/cni/bin/.

Issue 2: Pods Have No Network Connectivity

Pods are running but cannot ping other pods or external IPs.

Debug Steps:

• Check pod network interfaces:

  kubectl exec -it <pod-name> -- ip addr  

  Ensure eth0 has an IP (e.g., 10.244.1.5/32).
• Check host routes (Calico example):

  ip route show | grep cali  

• Verify BGP connectivity (Calico):

  calicoctl get bgppeer  

Issue 3: Network Policies Not Enforced

Pod traffic isn’t blocked/allowed as per NetworkPolicy rules.

Debug Steps:

• Ensure the CNI plugin supports network policies (Calico, Cilium, Weave).
• Check policy status with plugin-specific tools:

  # Calico  
  calicoctl get networkpolicy  

  # Cilium  
  cilium policy get  

8. Conclusion

CNI plugins are the backbone of Kubernetes networking, enabling pods to communicate dynamically across clusters. From simple overlays (Flannel) to high-performance eBPF (Cilium) or underlay (Calico) solutions, choosing the right plugin depends on your cluster’s size, performance needs, and security requirements.

By understanding CNI’s workflow, plugin types, and troubleshooting techniques, you can build a reliable, secure, and scalable Kubernetes network.

9. References

• CNI Specification
• Kubernetes Networking Documentation
• Calico Documentation
• Cilium Documentation
• Flannel GitHub
• Weave Net Documentation