Kubernetes and Terraform: Managing Infrastructure as Code

Introduction to Infrastructure as Code (IaC)
What is Terraform?
What is Kubernetes?
Why Combine Terraform and Kubernetes?
How Terraform and Kubernetes Work Together
Step-by-Step Guide: Provisioning a Kubernetes Cluster with Terraform
Managing Kubernetes Resources with Terraform
Best Practices for Terraform + Kubernetes
Challenges and Limitations
Conclusion
References

Introduction to Infrastructure as Code (IaC)

Infrastructure as Code (IaC) is the practice of defining and managing infrastructure (servers, networks, databases, etc.) using machine-readable configuration files rather than manual processes. These files are treated like application code: version-controlled, tested, and deployed programmatically.

Key Benefits of IaC:

Consistency: Eliminates “works on my machine” issues by ensuring environments (dev, staging, prod) are identical.
Scalability: Automatically provision infrastructure for new projects or growing workloads without manual intervention.
Version Control: Track changes to infrastructure over time, enabling rollbacks and audits.
Collaboration: Teams can collaborate on infrastructure changes via Git, with code reviews and pull requests.
Speed: Reduce time-to-market by automating infrastructure provisioning.

IaC tools fall into two categories: declarative (define “what” you want, not “how” to build it) and imperative (define “how” to build it step-by-step). Terraform is declarative, while tools like Ansible (though often used for configuration management) can be imperative. Kubernetes, too, is declarative: you define the desired state of your applications, and Kubernetes ensures the actual state matches.

What is Terraform?

Developed by HashiCorp, Terraform is an open-source, cloud-agnostic IaC tool for building, changing, and versioning infrastructure safely and efficiently. It uses a declarative language called HashiCorp Configuration Language (HCL) to define infrastructure resources, and it works with over 200+ cloud providers (AWS, Azure, GCP, Kubernetes, etc.) via providers.

Core Concepts of Terraform:

Providers: Plugins that interface with external APIs (e.g., aws for Amazon Web Services, kubernetes for Kubernetes).
Resources: The building blocks of infrastructure (e.g., aws_instance for an EC2 VM, kubernetes_deployment for a Kubernetes Deployment).
State: A JSON file (terraform.tfstate) that tracks the current state of your infrastructure, enabling Terraform to plan and apply changes.
Modules: Reusable packages of Terraform configurations (e.g., a module to provision a VPC, or a Kubernetes cluster).
Variables/Outputs: variables.tf for input parameters (e.g., cluster size), outputs.tf for exposing values (e.g., a Kubernetes cluster endpoint).

Why Terraform?

Cloud-Agnostic: Write once, deploy to any cloud or on-prem environment.
Idempotent: Running terraform apply multiple times has the same effect as running it once (no unintended side effects).
State Management: Tracks infrastructure changes, making it easy to destroy or update resources.
Strong Ecosystem: A rich library of community-contributed modules (via the Terraform Registry) and providers.

What is Kubernetes?

Kubernetes (often called “K8s”) is an open-source container orchestration platform designed to automate the deployment, scaling, and management of containerized applications. It groups containers into pods (the smallest deployable units) and manages their lifecycle across a cluster of machines (nodes).

Core Concepts of Kubernetes:

Cluster: A set of nodes (physical or virtual machines) managed by Kubernetes.
Control Plane: The “brain” of the cluster (components like kube-apiserver, etcd, scheduler, controller-manager).
Nodes: Worker machines that run pods.
Pods: Groups of one or more containers sharing network and storage resources.
Deployments: Declarative updates for pods (e.g., “run 3 replicas of my app”).
Services: Expose pods to internal/external traffic (e.g., a NodePort or LoadBalancer service).
ConfigMaps/Secrets: Store configuration data or sensitive information (e.g., API keys) for pods.

Why Kubernetes?

Scalability: Automatically scale applications up/down based on demand (via HorizontalPodAutoscaler).
Self-Healing: Restarts failed pods, replaces nodes, and reschedules containers.
Portability: Run applications consistently across clouds, data centers, or edge environments.
Extensibility: Extend functionality with custom resources (CRDs), operators, or add-ons (e.g., Prometheus for monitoring).

Why Kubernetes for IaC?

Kubernetes itself is declarative: you define the desired state of your applications (e.g., “3 replicas of my app, with 2GB RAM each”) in YAML/JSON manifests, and Kubernetes’ control plane ensures the actual state matches. This aligns perfectly with IaC principles, making Kubernetes a natural fit for managing application infrastructure.

Why Combine Terraform and Kubernetes?

While Terraform and Kubernetes excel in different domains, their strengths complement each other:

Terraform	Kubernetes
Provisions infrastructure (VMs, networks, storage).	Orchestrates containerized applications.
Manages “lower-level” infrastructure (e.g., VPCs, load balancers).	Manages “higher-level” application resources (pods, services).
Cloud-agnostic.	Container-agnostic (works with Docker, containerd, etc.).

Together, they enable full-stack IaC: from provisioning the underlying infrastructure (with Terraform) to deploying applications (with Kubernetes), all defined as code.

Key Benefits of Combining Them:

End-to-End Automation: Provision infrastructure (Terraform) → deploy Kubernetes (Terraform) → run apps (Kubernetes) → update everything (Terraform + Kubernetes).
Consistency: Define infrastructure and application state in code, ensuring dev/staging/prod environments are identical.
Version Control: Track changes to infrastructure (Terraform code) and application deployments (Kubernetes manifests) in Git.
Scalability: Use Terraform to scale the cluster (add nodes) and Kubernetes to scale apps (add replicas).

How Terraform and Kubernetes Work Together

The workflow typically involves two phases:

Phase 1: Provision the Kubernetes Cluster with Terraform

Terraform provisions the underlying infrastructure (e.g., VMs, networks) and deploys the Kubernetes cluster itself. This can be:

A managed Kubernetes service (e.g., AWS EKS, Azure AKS, GCP GKE).
A self-managed cluster (e.g., on EC2 VMs, using tools like kubeadm).

Terraform uses cloud-specific providers (e.g., aws for EKS) to create the cluster and outputs critical details (e.g., cluster endpoint, kubeconfig) needed to interact with Kubernetes.

Phase 2: Deploy Kubernetes Resources with Terraform

Once the cluster is running, Terraform can deploy Kubernetes resources (Deployments, Services, ConfigMaps) using the Kubernetes Provider. This provider interacts with the Kubernetes API (via kubectl or directly) to apply manifests defined in HCL.

High-Level Workflow:

Define Infrastructure: Write Terraform configs to provision VPCs, subnets, and a Kubernetes cluster (e.g., EKS).
Provision Cluster: Run terraform apply to create the cluster. Terraform updates the terraform.tfstate file with the cluster’s state.
Deploy Apps: Use the kubernetes provider in Terraform to deploy Kubernetes resources (e.g., a nginx Deployment and LoadBalancer Service).
Update/Scale: Modify Terraform configs to scale the cluster (e.g., add nodes) or update Kubernetes resources (e.g., increase replicas), then run terraform apply.

Step-by-Step Guide: Provisioning a Kubernetes Cluster with Terraform

Let’s walk through a practical example: provisioning an AWS EKS cluster with Terraform, then deploying a sample nginx app to it using the Kubernetes Provider.

Prerequisites

An AWS account with permissions to create EKS clusters.
Terraform installed (v1.3+ recommended).
AWS CLI configured with credentials.
kubectl installed (to verify cluster access).

Step 1: Project Setup

Create a new directory for your Terraform project:

mkdir terraform-eks-demo && cd terraform-eks-demo

Step 2: Define Terraform Configs for EKS

We’ll use the official AWS EKS Module (a community module) to simplify cluster provisioning. Create the following files:

`main.tf` (Core Configuration)

# Configure the AWS Provider  
provider "aws" {  
  region = var.aws_region  
}  

# Use the AWS EKS Module  
module "eks" {  
  source          = "terraform-aws-modules/eks/aws"  
  version         = "~> 20.0"  
  cluster_name    = var.cluster_name  
  cluster_version = "1.28"  
  vpc_id          = module.vpc.vpc_id  
  subnet_ids      = module.vpc.private_subnets  

  # Node group configuration (worker nodes)  
  eks_managed_node_groups = {  
    main = {  
      desired_size = 2  
      min_size     = 1  
      max_size     = 3  

      instance_types = ["t3.medium"]  
      capacity_type  = "ON_DEMAND"  
    }  
  }  
}  

# Create a VPC (using the AWS VPC Module)  
module "vpc" {  
  source  = "terraform-aws-modules/vpc/aws"  
  version = "~> 3.0"  
  name    = "${var.cluster_name}-vpc"  
  cidr    = "10.0.0.0/16"  

  azs             = ["${var.aws_region}a", "${var.aws_region}b"]  
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]  
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]  

  enable_nat_gateway = true  
  single_nat_gateway = true  
}  

# Configure the Kubernetes Provider to use the EKS cluster  
provider "kubernetes" {  
  host                   = module.eks.cluster_endpoint  
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)  
  token                  = data.aws_eks_cluster_auth.eks.token  
}  

# Get EKS cluster auth token  
data "aws_eks_cluster_auth" "eks" {  
  name = module.eks.cluster_name  
}  

# Deploy a sample nginx app using the Kubernetes Provider  
resource "kubernetes_deployment" "nginx" {  
  metadata {  
    name      = "nginx"  
    namespace = "default"  
  }  

  spec {  
    replicas = 2  

    selector {  
      match_labels = {  
        app = "nginx"  
      }  
    }  

    template {  
      metadata {  
        labels = {  
          app = "nginx"  
        }  
      }  

      spec {  
        container {  
          image = "nginx:1.23"  
          name  = "nginx"  

          port {  
            container_port = 80  
          }  
        }  
      }  
    }  
  }  
}  

# Expose nginx via a LoadBalancer Service  
resource "kubernetes_service" "nginx" {  
  metadata {  
    name      = "nginx"  
    namespace = "default"  
  }  

  spec {  
    selector = {  
      app = kubernetes_deployment.nginx.spec[0].template[0].metadata[0].labels.app  
    }  

    port {  
      port        = 80  
      target_port = 80  
    }  

    type = "LoadBalancer"  
  }  
}

`variables.tf` (Input Variables)

variable "aws_region" {  
  description = "AWS region for the EKS cluster"  
  type        = string  
  default     = "us-west-2"  
}  

variable "cluster_name" {  
  description = "Name of the EKS cluster"  
  type        = string  
  default     = "my-eks-cluster"  
}

`outputs.tf` (Expose Values)

output "cluster_endpoint" {  
  description = "EKS cluster endpoint"  
  value       = module.eks.cluster_endpoint  
}  

output "kubeconfig" {  
  description = "Kubeconfig for the EKS cluster"  
  value       = module.eks.kubeconfig_raw  
}  

output "nginx_service_ip" {  
  description = "Public IP of the nginx LoadBalancer Service"  
  value       = kubernetes_service.nginx.status[0].load_balancer[0].ingress[0].hostname  
}

Step 3: Initialize Terraform

Run terraform init to download providers and modules:

terraform init

Step 4: Review the Execution Plan

Run terraform plan to preview the infrastructure Terraform will create:

terraform plan

Step 5: Apply the Configuration

Run terraform apply to provision the EKS cluster and deploy nginx:

terraform apply

Type yes when prompted. This will take 10–15 minutes (EKS clusters take time to provision).

Step 6: Verify the Cluster and App

Once complete, use the output kubeconfig to configure kubectl:

terraform output -raw kubeconfig > ~/.kube/eks-config  
export KUBECONFIG=~/.kube/eks-config

Check nodes:

kubectl get nodes  
# Output should show 2 worker nodes (from the node group)

Check pods:

kubectl get pods  
# Output should show 2 nginx pods

Check the nginx service:

kubectl get svc nginx  
# The EXTERNAL-IP will be the LoadBalancer hostname/IP

Visit the EXTERNAL-IP in a browser to see the nginx welcome page!

Step 7: Cleanup (Optional)

To destroy the cluster and resources when done:

terraform destroy

Managing Kubernetes Resources with Terraform

The Kubernetes Provider allows you to define Kubernetes resources (Deployments, Services, ConfigMaps, etc.) in HCL, alongside your infrastructure code. This is ideal for:

Infrastructure-Critical Resources: Namespaces, RBAC roles, persistent volumes, or network policies that are part of the cluster’s core setup.
Version-Controlled Workflows: Ensuring Kubernetes resources are tracked in Terraform state, alongside the infrastructure that hosts them.

When to Use Terraform vs. `kubectl`/Helm?

Terraform: Best for resources that are tightly coupled to infrastructure (e.g., a PV that depends on an AWS EBS volume provisioned by Terraform).
kubectl/Helm: Better for application-specific resources that change frequently (e.g., deploying a new app version). Helm charts are also great for packaging complex applications with dependencies.

Example: Deploying a ConfigMap with Terraform

Add this to main.tf to create a ConfigMap for nginx:

resource "kubernetes_config_map" "nginx_config" {  
  metadata {  
    name      = "nginx-config"  
    namespace = "default"  
  }  

  data = {  
    "nginx.conf" = <<-EOT  
      server {  
        listen 80;  
        server_name localhost;  

        location / {  
          root /usr/share/nginx/html;  
          index index.html;  
        }  
      }  
    EOT  
  }  
}

Update the nginx Deployment to mount this ConfigMap:

resource "kubernetes_deployment" "nginx" {  
  # ... (previous config)  

  spec {  
    template {  
      spec {  
        container {  
          image = "nginx:1.23"  
          name  = "nginx"  

          port {  
            container_port = 80  
          }  

          volume_mount {  
            name       = "nginx-config-volume"  
            mount_path = "/etc/nginx/conf.d"  
            read_only  = true  
          }  
        }  

        volume {  
          name = "nginx-config-volume"  
          config_map {  
            name = kubernetes_config_map.nginx_config.metadata[0].name  
          }  
        }  
      }  
    }  
  }  
}

Run terraform apply to update the Deployment with the new ConfigMap.

Best Practices for Terraform + Kubernetes

1. Use Modules for Reusability

Leverage Terraform modules (e.g., the AWS EKS Module) to avoid重复代码 and enforce consistency. Create custom modules for common Kubernetes resources (e.g., a “base namespace” module with RBAC).

2. Remote State Management

Store terraform.tfstate remotely (e.g., AWS S3, Terraform Cloud) instead of locally. Enable state locking (e.g., with DynamoDB for S3) to prevent conflicts when multiple users run terraform apply.

Example S3 backend config (add to main.tf):

terraform {  
  backend "s3" {  
    bucket         = "my-terraform-state-bucket"  
    key            = "eks-cluster/terraform.tfstate"  
    region         = "us-west-2"  
    encrypt        = true  
    dynamodb_table = "terraform-state-lock"  
  }  
}

3. Separate State Files by Environment

Use separate state files for dev, staging, and prod environments to avoid accidental changes to production. Organize projects in directories like environments/dev and environments/prod.

4. Validate Configs Before Applying

Run terraform validate to check for syntax errors, and terraform plan to review changes before running terraform apply.

5. Least Privilege for Terraform

Restrict the AWS (or cloud provider) IAM role used by Terraform to the minimum permissions needed (e.g., only allow creating EKS clusters and Kubernetes resources).

6. Document Everything

Add comments to HCL files explaining why resources are configured a certain way. Use README.md to document prerequisites, variables, and deployment steps.

Challenges and Limitations

While Terraform and Kubernetes are powerful together, they have limitations:

1. State Management Complexity

Terraform state files can grow large and become hard to manage, especially in multi-team environments. Remote state and modules help, but require careful planning.

2. Conflict with `kubectl` Changes

If someone uses kubectl edit to modify a resource managed by Terraform, Terraform will overwrite those changes on the next apply. Use RBAC to restrict direct kubectl access to Terraform-managed resources.

3. Performance with Large Kubernetes Deployments

Terraform can be slow when managing hundreds of Kubernetes resources. For application deployments, consider Helm or Kubernetes Operators instead.

4. Learning Curve

Both Terraform and Kubernetes have steep learning curves. Teams new to IaC may need time to master HCL, Kubernetes concepts, and provider-specific quirks.

5. Provider Lag

The Kubernetes Provider may not support the latest Kubernetes features immediately (e.g., new API versions). Check the provider docs for compatibility.

Conclusion

Terraform and Kubernetes are a match made in IaC heaven. Terraform automates infrastructure provisioning (VMs, networks, Kubernetes clusters), while Kubernetes orchestrates containerized applications. Together, they enable end-to-end automation, consistency, and scalability—critical for modern DevOps workflows.

By following best practices like using modules, remote state, and separating environments, teams can overcome the challenges of managing infrastructure and applications as code. Whether you’re running a small app or a global enterprise system, this duo will help you deliver faster, more reliably, and with less overhead.

Table of Contents