Kubernetes DaemonSets and Jobs: An In-Depth Exploration

Table of Contents

1. What Are DaemonSets?
2. Key Components of DaemonSets
3. DaemonSets: Use Cases
4. How DaemonSets Work
5. DaemonSet Configuration Example
6. Best Practices for DaemonSets
7. What Are Jobs?
8. Types of Jobs
9. Key Components of Jobs
10. Jobs: Use Cases
11. How Jobs Work
12. Job Configuration Examples
13. Best Practices for Jobs
14. DaemonSets vs. Jobs: Key Differences
15. Conclusion
16. References

What Are DaemonSets?

A DaemonSet is a Kubernetes workload resource that ensures a copy of a pod runs on all nodes in the cluster (or a subset of nodes, if filtered). Unlike Deployments or StatefulSets— which scale based on replicas—DaemonSets scale with the cluster: when a new node joins, a DaemonSet pod is automatically scheduled on it; when a node is removed, the pod is terminated.

DaemonSets are designed for node-centric tasks—services that need to run locally on each node to function.

Key Components of DaemonSets

A DaemonSet manifest (daemonset.yaml) includes the following core fields:

Node Selection

By default, DaemonSets run on all nodes. To target specific nodes, use:

• nodeSelector: Matches nodes with specific labels (e.g., disk: ssd).
• affinity/anti-affinity: More advanced rules (e.g., “avoid nodes with label env: prod”).
• tolerations: Allows pods to run on nodes with taints (e.g., control plane nodes).

DaemonSets: Use Cases

DaemonSets excel at scenarios where a service must run locally on nodes. Common use cases include:

1. Logging/Monitoring: Tools like Fluentd (log collection) or Prometheus Node Exporter (metrics) run on every node to aggregate data.
2. Network Plugins: CNI plugins (e.g., Calico, Flannel) use DaemonSets to manage node networking.
3. Storage Daemons: Local storage drivers (e.g., local-volume-provisioner) run on nodes to manage persistent storage.
4. Security Agents: Antivirus or compliance tools (e.g., Falco) monitor node-level activity.

How DaemonSets Work

The DaemonSet controller (a built-in Kubernetes controller) manages DaemonSet pods. Its workflow:

1. Desired State: The DaemonSet manifest defines the desired pod template and node selection rules.
2. Current State: The controller watches nodes, pods, and the DaemonSet itself to track the current state.
3. Reconciliation: If a node lacks a DaemonSet pod (or has an outdated one), the controller creates/updates the pod to match the desired state.

Key behaviors:

• Node Additions: When a new node joins the cluster, the controller immediately schedules a DaemonSet pod on it.
• Node Removals: When a node is deleted, the controller terminates the associated DaemonSet pod.
• Updates: With RollingUpdate, old pods are replaced with new ones one node at a time (default). With OnDelete, pods are only updated when manually deleted.

DaemonSet Configuration Example

Here’s a DaemonSet for Prometheus Node Exporter (a metrics collector):

# daemonset-node-exporter.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.5.0
        ports:
        - containerPort: 9100
          name: metrics
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      tolerations:  # Allow running on control plane nodes
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule

Explanation:

• selector and template.metadata.labels ensure the DaemonSet manages pods with label app: node-exporter.
• volumes mount host directories (/proc, /sys) into the pod, enabling Node Exporter to collect node metrics.
• tolerations allow the pod to run on control plane nodes (which are typically tainted to prevent non-essential workloads).

Best Practices for DaemonSets

1. Limit Resources: DaemonSets run on every node—set resources.requests and resources.limits to prevent resource starvation.

   resources:
     requests:
       cpu: 100m
       memory: 128Mi
     limits:
       cpu: 250m
       memory: 256Mi

2. Use Node Affinity Sparingly: Prefer nodeSelector for simple node targeting; use affinity only for complex rules (e.g., “run on nodes with GPU”).

3. Tolerate Taints Judiciously: Only add tolerations for nodes that require the DaemonSet (e.g., control plane nodes for network plugins).

4. Monitor DaemonSets: Use tools like Prometheus to track pod status, restart counts, and resource usage across nodes.

5. Test Updates: Use RollingUpdate with maxUnavailable: 1 (default) to minimize downtime during upgrades.

What Are Jobs?

A Job is a Kubernetes workload resource that manages pods to run a task to completion. Unlike DaemonSets (which run indefinitely), Jobs ensure pods execute a task (e.g., a script, batch job) and exit successfully. Once all pods in a Job complete, the Job itself is marked as Completed.

Jobs are ideal for one-off or recurring batch tasks that don’t need to run continuously.

Types of Jobs

Kubernetes Jobs have three primary flavors:

1. One-Off Jobs

Runs a single pod to completion. If the pod fails, the Job restarts it until it succeeds (or hits a retry limit).

2. Parallel Jobs

Runs multiple pods in parallel to complete a task. Two sub-types:

• Indexed Jobs: Each pod processes a distinct subset of data (e.g., “pod 1 processes IDs 1-100, pod 2 processes 101-200”).
• Non-Indexed Jobs: All pods process the same task (e.g., rendering a video in parallel).

3. CronJobs

A time-based Job that runs on a schedule (e.g., “daily at 2 AM”). CronJobs create Jobs at specified intervals using cron syntax (e.g., 0 2 * * *).

Key Components of Jobs

A Job manifest (job.yaml) includes these core fields:

For CronJobs, additional fields include:

• spec.schedule: Cron syntax for recurrence (e.g., 0 3 * * * for 3 AM daily).
• spec.concurrencyPolicy: How to handle overlapping Jobs (Allow, Forbid, or Replace).

Jobs: Use Cases

Jobs shine for tasks that need to finish and exit. Common use cases:

1. Batch Processing: Analyzing logs, generating reports, or transforming data (e.g., CSV to Parquet).
2. Backups: Running daily database backups (via CronJob).
3. Database Migrations: Executing schema updates during deployments.
4. One-Off Scripts: Cleaning up old files, syncing data between services, or testing APIs.

How Jobs Work

The Job controller manages Job lifecycle:

1. Pod Creation: The controller creates pods based on spec.template until spec.completions are met.
2. Failure Handling: If a pod fails (e.g., exits with non-zero code), the controller restarts it (up to backoffLimit retries).
3. Completion Check: Once completions successful pods exist, the Job is marked Completed.
4. Cleanup: If ttlSecondsAfterFinished is set, the Job is auto-deleted after the TTL expires.

For CronJobs:

• The CronJob controller checks the schedule at fixed intervals (every 10 seconds by default).
• When the schedule triggers, it creates a Job, which in turn creates pods.

Job Configuration Examples

Example 1: One-Off Job (Batch Processing)

A Job to process a CSV file and exit:

# job-batch-processing.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: csv-processor
spec:
  template:
    spec:
      containers:
      - name: processor
        image: my-csv-processor:1.0
        command: ["/app/process.sh", "/data/input.csv"]
        volumeMounts:
        - name: data-volume
          mountPath: /data
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: input-data-pvc
      restartPolicy: Never  # Jobs typically use Never or OnFailure
  backoffLimit: 3  # Retry up to 3 times on failure
  activeDeadlineSeconds: 3600  # Fail if not done in 1 hour

Example 2: Parallel Job

A Job with 4 completions and 2 parallel pods (e.g., processing 4 data chunks):

# job-parallel.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-data-processor
spec:
  completions: 4  # 4 total successful pods
  parallelism: 2  # Run 2 pods at a time
  template:
    spec:
      containers:
      - name: processor
        image: my-parallel-processor:1.0
        env:
        - name: CHUNK_ID
          valueFrom:
            fieldRef:
              fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']  # Unique chunk ID
      restartPolicy: Never
  backoffLimit: 2

Example 3: CronJob (Daily Backup)

A CronJob to run a daily database backup at 2 AM:

# cronjob-backup.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-backup
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  concurrencyPolicy: Forbid  # Don't run if previous backup is still running
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: my-db-backup:1.0
            command: ["/backup.sh", "--db=prod"]
            env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secrets
                  key: password
          restartPolicy: OnFailure
      backoffLimit: 1
      ttlSecondsAfterFinished: 86400  # Keep Job for 1 day

Best Practices for Jobs

1. Set restartPolicy: Never or OnFailure: Jobs are not meant to restart indefinitely. Use Never for one-off tasks and OnFailure to retry failed pods.

2. Limit Runtime with activeDeadlineSeconds: Prevent stuck Jobs from running forever (e.g., 3600 seconds = 1 hour).

3. Clean Up Old Jobs: Use ttlSecondsAfterFinished to auto-delete completed Jobs (e.g., 604800 = 7 days).

4. Avoid Over-Parallelism: Too many parallel pods can strain the cluster—test with parallelism: 1 first, then scale up.

5. Secure Sensitive Data: Use secrets (not plain text) for credentials in CronJobs/Jobs.

DaemonSets vs. Jobs: Key Differences

Conclusion

DaemonSets and Jobs are indispensable Kubernetes workload resources, each solving unique orchestration challenges.

• DaemonSets ensure node-level services (e.g., logging, networking) run consistently across the cluster, scaling with node additions/removals.
• Jobs (and CronJobs) manage finite tasks, from one-off scripts to recurring backups, ensuring tasks complete reliably.

By mastering these resources, you can build resilient, efficient Kubernetes clusters that handle both continuous services and batch workloads with ease.

References

• Kubernetes DaemonSet Documentation
• Kubernetes Job Documentation
• Kubernetes CronJob Documentation
• Prometheus Node Exporter
• Kubernetes Workload Resources Best Practices