coderain guide

Exploring Kubernetes CRDs: A Guide to Custom Resources

Kubernetes has revolutionized container orchestration by providing a robust, extensible platform for managing containerized applications. At its core, Kubernetes relies on **resources**—API objects like `Pods`, `Deployments`, and `Services`—to model and manage cluster state. However, every organization and application has unique needs: maybe you need to define a "DatabaseInstance" for your microservices, a "MLModel" for machine learning workflows, or a "FeatureFlag" for dynamic configuration. This is where **Custom Resource Definitions (CRDs)** come into play. CRDs extend the Kubernetes API to let you define and use custom resources (CRs) tailored to your specific use case, without modifying Kubernetes source code. In this guide, we’ll demystify CRDs, walk through creating and using them, and explore advanced features, best practices, and real-world applications.

Table of Contents

  1. Understanding Kubernetes Resources
  2. What Are Custom Resource Definitions (CRDs)?
  3. Components of a CRD
  4. Creating Your First CRD
  5. Working with Custom Resources (CRs)
  6. Advanced CRD Features
  7. Use Cases for CRDs
  8. Best Practices
  9. Challenges and Limitations
  10. Conclusion
  11. References

1. Understanding Kubernetes Resources

Before diving into CRDs, let’s recap Kubernetes resources. A resource is a persistent API object that represents a cluster entity. Built-in resources (e.g., Pod, Service) are defined by the Kubernetes API and managed via kubectl or client libraries.

Key characteristics of resources:

  • API Endpoints: Each resource is exposed via a RESTful API endpoint (e.g., /api/v1/pods for Pods).
  • Schema Validation: Resources have a schema (via OpenAPI) that enforces structure (e.g., a Pod requires a spec.containers field).
  • Lifecycle Management: Kubernetes controllers (e.g., the Deployment controller) reconcile the desired state (defined in the resource) with the actual cluster state.

2. What Are Custom Resource Definitions (CRDs)?

A Custom Resource Definition (CRD) is a Kubernetes API resource that lets you define custom resources (CRs)—new resource types not built into Kubernetes. CRDs act as a “blueprint” for your custom resource, specifying its name, structure, validation rules, and behavior.

CRD vs. Custom Resource (CR)

  • CRD: The definition/schema (e.g., “Define a Book resource with title, author, and pages fields”).
  • CR: An instance of the CRD (e.g., “A specific book: The Great Gatsby by F. Scott Fitzgerald, 180 pages”).

3. Components of a CRD

A CRD is defined as a YAML file with metadata and a spec. Let’s break down its key components:

Metadata

  • name: Unique name for the CRD (format: <plural>.<group>; e.g., books.example.com).
  • group: DNS-style group name for your custom resources (e.g., example.com).
  • version: API version (e.g., v1alpha1, v1).

Spec

The spec defines the behavior and structure of the custom resource:

scope

  • Namespaced: CRs exist in a namespace (like Pods).
  • Cluster: CRs are cluster-scoped (like Nodes).

names

Defines how the resource is referenced in kubectl and YAML:

  • plural: Plural name (used in API endpoints and kubectl get <plural>; e.g., books).
  • singular: Singular name (e.g., book).
  • kind: CamelCase name for the resource (used in YAML kind field; e.g., Book).
  • shortNames: Shortcuts for kubectl (e.g., bk for books).

validation (OpenAPIv3 Schema)

Enforces structure and constraints on CRs using an OpenAPIv3 schema. Define required fields, data types, enums, and more.

subresources

Optional: Enable features like status (separate spec/status for runtime data) or scale (integration with HPA).

additionalPrinterColumns

Customize columns displayed in kubectl get <plural> output (e.g., show author and pages for books).

Example CRD Snippet

Here’s a partial CRD for a Book resource:

apiVersion: crd.k8s.io/v1  
kind: CustomResourceDefinition  
metadata:  
  name: books.example.com  # Format: <plural>.<group>  
spec:  
  group: example.com  
  versions:  
    - name: v1  
      served: true  # Expose this version in the API  
      storage: true  # Store CRs in this version  
      schema:  
        openAPIV3Schema:  
          type: object  
          properties:  
            spec:  
              type: object  
              properties:  
                title:  
                  type: string  
                  minLength: 1  
                author:  
                  type: string  
                  minLength: 1  
                pages:  
                  type: integer  
                  minimum: 1  
              required: ["title", "author"]  # Enforce required fields  
  scope: Namespaced  
  names:  
    plural: books  
    singular: book  
    kind: Book  
    shortNames: ["bk"]  
  additionalPrinterColumns:  
    - name: Title  
      type: string  
      jsonPath: .spec.title  
    - name: Author  
      type: string  
      jsonPath: .spec.author  
    - name: Pages  
      type: integer  
      jsonPath: .spec.pages

4. Creating Your First CRD

Let’s build a working CRD step-by-step. We’ll create a Book resource to track books in a library.

Step 1: Define the CRD

Create a file book-crd.yaml with the full CRD (expanding on the snippet above):

apiVersion: crd.k8s.io/v1  
kind: CustomResourceDefinition  
metadata:  
  name: books.example.com  
spec:  
  group: example.com  
  versions:  
    - name: v1  
      served: true  
      storage: true  
      schema:  
        openAPIV3Schema:  
          type: object  
          properties:  
            spec:  
              type: object  
              properties:  
                title:  
                  type: string  
                  minLength: 1  
                author:  
                  type: string  
                  minLength: 1  
                pages:  
                  type: integer  
                  minimum: 1  
                genre:  
                  type: string  
                  enum: ["fiction", "non-fiction", "sci-fi", "biography"]  # Restrict to allowed values  
              required: ["title", "author"]  # Enforce title and author  
  scope: Namespaced  
  names:  
    plural: books  
    singular: book  
    kind: Book  
    shortNames: ["bk"]  
  additionalPrinterColumns:  
    - name: Title  
      type: string  
      jsonPath: .spec.title  
    - name: Author  
      type: string  
      jsonPath: .spec.author  
    - name: Pages  
      type: integer  
      jsonPath: .spec.pages  
    - name: Genre  
      type: string  
      jsonPath: .spec.genre

Step 2: Apply the CRD

Deploy the CRD to your cluster:

kubectl apply -f book-crd.yaml

Verify the CRD is created:

kubectl get crds  
# Output:  
# NAME               CREATED AT  
# books.example.com  2024-05-20T12:00:00Z

Step 3: Verify the API Endpoint

Kubernetes automatically creates an API endpoint for your CRD. Check if the books resource is available:

kubectl api-resources | grep books  
# Output:  
# books         example.com/v1  Namespaced  true       Book

5. Working with Custom Resources (CRs)

Now that the CRD is deployed, create and manage instances of your custom resource (CRs).

Step 1: Create a CR Instance

Define a CR for a specific book (e.g., the-great-gatsby.yaml):

apiVersion: example.com/v1  
kind: Book  
metadata:  
  name: the-great-gatsby  
  namespace: default  
spec:  
  title: "The Great Gatsby"  
  author: "F. Scott Fitzgerald"  
  pages: 180  
  genre: "fiction"

Apply the CR:

kubectl apply -f the-great-gatsby.yaml

Step 2: Retrieve CRs

List all Book CRs:

kubectl get books  # or kubectl get bk (shortName)  
# Output:  
# NAME                  TITLE               AUTHOR                  PAGES  GENRE  
# the-great-gatsby      The Great Gatsby    F. Scott Fitzgerald     180    fiction

Describe a CR to see full details:

kubectl describe book the-great-gatsby

Step 3: Validate CRs

CRDs enforce validation via the OpenAPI schema. Try creating an invalid CR (e.g., missing author or invalid genre):

# Invalid CR: Missing 'author' and invalid 'genre'  
apiVersion: example.com/v1  
kind: Book  
metadata:  
  name: invalid-book  
spec:  
  title: "Invalid Book"  
  pages: "not-a-number"  # Wrong type (string instead of integer)  
  genre: "comedy"  # Not in enum ["fiction", ...]

Applying this will fail with validation errors:

kubectl apply -f invalid-book.yaml  
# Error: Book.invalid.example.com "invalid-book" is invalid:  
# * spec.author: Required value  
# * spec.pages: Invalid value: "string": spec.pages in body must be of type integer: "string"  
# * spec.genre: Unsupported value: "comedy": supported values: "fiction", "non-fiction", "sci-fi", "biography"

Step 4: Update and Delete CRs

Update a CR by editing its YAML or using kubectl edit book <name>. For example, update pages to 182:

kubectl patch book the-great-gatsby --type=merge -p '{"spec":{"pages": 182}}'

Delete a CR:

kubectl delete book the-great-gatsby

6. Advanced CRD Features

CRDs offer powerful features to tailor behavior to your needs:

Validation with OpenAPIv3

Use the validation field to enforce strict constraints. Example rules:

  • minimum/maximum for numbers.
  • minLength/maxLength for strings.
  • enum for allowed values.
  • pattern for regex (e.g., ^[A-Za-z0-9]+$ for alphanumeric titles).

Default Values

Set default values for optional fields. For example, default genre to “fiction” if unspecified:

spec:  
  validation:  
    openAPIV3Schema:  
      properties:  
        spec:  
          properties:  
            genre:  
              type: string  
              enum: ["fiction", ...]  
              default: "fiction"  # Default value

Subresources: status

The status subresource separates desired state (.spec) from observed state (.status). Useful for controllers/operators to report runtime data (e.g., “ready: true” or “error: DatabaseUnavailable”).

Enable status in the CRD:

spec:  
  subresources:  
    status: {}  # Enable status subresource

Update status via kubectl patch (or a controller):

kubectl patch book the-great-gatsby --subresource=status -p '{"status":{"available": true, "lastChecked": "2024-05-20T14:30:00Z"}}'

Subresources: scale

Enable horizontal scaling by adding the scale subresource. This integrates with the Horizontal Pod Autoscaler (HPA) to scale based on CR metrics.

Versioning

Manage multiple API versions (e.g., v1alpha1, v1beta1, v1) for backward compatibility. Use served: true to expose versions and storage: true to mark the version for storing CRs.

Conversion Webhooks

For non-backward-compatible version changes, use conversion webhooks to automatically convert CRs between versions (e.g., from v1alpha1 to v1).

7. Use Cases for CRDs

CRDs shine in scenarios where built-in resources are insufficient:

1. Operator Pattern

The Operator pattern uses CRDs + controllers to automate application lifecycle management. For example:

  • Prometheus Operator: Manages Prometheus CRs to deploy and configure Prometheus instances.
  • etcd Operator: Orchestrates EtcdCluster CRs to manage etcd clusters.

How it works: A controller watches CRs and reconciles the cluster state to match the CR’s .spec.

2. Domain-Specific Resources

Model business logic directly in Kubernetes. Examples:

  • DatabaseInstance: Define database parameters (size, engine, backup policy).
  • FeatureFlag: Manage feature flags for your application (e.g., enabled: true, rolloutPercentage: 50).

3. Custom Configuration

Replace static ConfigMaps with typed, validated CRs. For example, a PaymentGateway CR with encrypted secrets and validation for API keys.

8. Best Practices

1. Use Namespaced Scope Unless Necessary

Prefer Namespaced scope for CRs to isolate resources across teams or environments. Use Cluster scope only for cluster-wide resources (e.g., ClusterIssuer in Cert-Manager).

2. Enforce Strict Validation

Use OpenAPI schemas to catch errors early. Require critical fields and restrict data types/enums to prevent invalid CRs.

3. Version CRDs Carefully

Follow Kubernetes versioning conventions:

  • v1alpha1: Experimental, breaking changes allowed.
  • v1beta1: Stable enough for testing, minor changes only.
  • v1: GA (General Availability), no breaking changes.

4. Document CRDs

Add description fields in the OpenAPI schema and maintain docs for users to understand how to use your CRD. Tools like kube-openapi auto-generate docs.

5. Avoid Overengineering

Use built-in resources (e.g., ConfigMap, Secret) if they meet your needs. CRDs add complexity—reserve them for unique, structured data.

9. Challenges and Limitations

1. No Built-In Behavior

CRDs define the API, but do not add functionality on their own. To make CRs “do” something, you need a controller (e.g., an Operator) to watch and act on CR changes.

2. Validation Complexity

Writing OpenAPI schemas can be error-prone. Use tools like kubeconform to validate CRD YAMLs.

3. Upgrades and Migrations

Changing CRD schemas (e.g., adding/removing fields) requires careful planning. Use conversion webhooks for version upgrades, and back up CRs before modifying CRDs.

4. Performance Overhead

Large numbers of CRs (10k+) can strain the Kubernetes API server. Use tools like kubectl get --raw to monitor API performance.

10. Conclusion

Custom Resource Definitions (CRDs) unlock Kubernetes’ extensibility, letting you model domain-specific resources tailored to your applications. By defining schemas, validation rules, and subresources, CRDs transform Kubernetes into a platform for managing not just containers, but entire application lifecycles.

Whether you’re building an Operator, modeling business logic, or customizing configuration, CRDs empower you to extend Kubernetes to fit your needs. Start small (e.g., a Book CRD), experiment with validation, and gradually integrate controllers to add behavior.

11. References