Kubernetes CRDs & Operators: Building Cloud-Native Applications#

Introduction#

Custom Resource Definitions (CRDs) and Operators represent the pinnacle of Kubernetes extensibility. This comprehensive guide teaches you how to extend the Kubernetes API, build production-ready operators, and implement sophisticated application management patterns.

Why CRDs and Operators?#

🎯 API Extension: Add domain-specific resources to Kubernetes
🤖 Automation: Encode operational knowledge into software
🔄 Reconciliation: Maintain desired state automatically
📈 Scalability: Manage complex applications declaratively
🛠️ Standardization: Use Kubernetes patterns for custom resources

CRD Architecture & Design#

CRD Lifecycle Flow#

1
graph TB
2
    subgraph "CRD Definition"
3
        A[Define CRD Schema] --> B[Apply CRD to Cluster]
4
        B --> C[API Server Registration]
5
    end
6

7
    subgraph "Custom Resource Management"
8
        D[Create CR Instance] --> E[Validation via OpenAPI]
9
        E --> F[Store in etcd]
10
        F --> G[Available via API]
11
    end
12

13
    subgraph "Operator Controller"
14
        H[Watch for Changes] --> I[Reconciliation Loop]
15
        I --> J[Compare Desired vs Actual]
16
        J --> K[Take Action]
17
        K --> L[Update Status]
18
        L --> H
19
    end
20

21
    C --> D
22
    G --> H

CRD Components Overview#

Component	Purpose	Example
Group	API grouping	`apps.example.com`
Version	API version	`v1alpha1`, `v1beta1`, `v1`
Kind	Resource type	`Database`, `Application`
Resource	Plural name	`databases`, `applications`
Scope	Namespaced/Cluster	`Namespaced`
Schema	OpenAPI validation	JSON Schema
Subresources	Status, Scale	`/status`, `/scale`

Building Your First CRD#

1. Database CRD Definition#

1
apiVersion: apiextensions.k8s.io/v1
2
kind: CustomResourceDefinition
3
metadata:
4
  name: databases.apps.example.com
5
spec:
6
  group: apps.example.com
7
  versions:
8
  - name: v1
9
    served: true
10
    storage: true
11
    schema:
12
      openAPIV3Schema:
13
        type: object
14
        properties:
15
          spec:
16
            type: object
17
            required:
18
            - engine
19
            - version
20
            - size
21
            properties:
22
              engine:
23
                type: string
24
                enum:
25
                - postgres
26
                - mysql
27
                - mongodb
28
                description: "Database engine type"
29
              version:
30
                type: string
31
                pattern: '^[0-9]+\.[0-9]+(\.[0-9]+)?$'
32
                description: "Database version (e.g., 14.5)"
33
              size:
34
                type: string
35
                enum:
36
                - small
37
                - medium
38
                - large
39
                description: "Database size tier"
40
              replicas:
41
                type: integer
42
                minimum: 1
43
                maximum: 9
44
                default: 1
45
                description: "Number of replicas"
46
              backup:
47
                type: object
48
                properties:
49
                  enabled:
50
                    type: boolean
51
                    default: true
52
                  schedule:
53
                    type: string
54
                    pattern: '^(\*|([0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9])|\*\/([0-9]|1[0-9]|2[0-9]|3[0-9]|4[0-9]|5[0-9])) (\*|([0-9]|1[0-9]|2[0-3])|\*\/([0-9]|1[0-9]|2[0-3])) (\*|([1-9]|1[0-9]|2[0-9]|3[0-1])|\*\/([1-9]|1[0-9]|2[0-9]|3[0-1])) (\*|([1-9]|1[0-2])|\*\/([1-9]|1[0-2])) (\*|([0-6])|\*\/([0-6]))$'
55
                    default: "0 2 * * *"
56
                  retention:
57
                    type: integer
58
                    minimum: 1
59
                    maximum: 365
60
                    default: 30
61
              storage:
62
                type: object
63
                properties:
64
                  size:
65
                    type: string
66
                    pattern: '^[0-9]+[GM]i$'
67
                    default: "10Gi"
68
                  class:
69
                    type: string
70
                    default: "standard"
71
              monitoring:
72
                type: object
73
                properties:
74
                  enabled:
75
                    type: boolean
76
                    default: true
77
                  interval:
78
                    type: string
79
                    default: "30s"
80
          status:
81
            type: object
82
            properties:
83
              phase:
84
                type: string
85
                enum:
86
                - Pending
87
                - Creating
88
                - Running
89
                - Failed
90
                - Terminating
91
              message:
92
                type: string
93
              endpoint:
94
                type: string
95
              conditions:
96
                type: array
97
                items:
98
                  type: object
99
                  properties:
100
                    type:
101
                      type: string
102
                    status:
103
                      type: string
104
                    lastTransitionTime:
105
                      type: string
106
                    reason:
107
                      type: string
108
                    message:
109
                      type: string
110
    subresources:
111
      status: {}
112
      scale:
113
        specReplicasPath: .spec.replicas
114
        statusReplicasPath: .status.replicas
115
    additionalPrinterColumns:
116
    - name: Engine
117
      type: string
118
      jsonPath: .spec.engine
119
    - name: Version
120
      type: string
121
      jsonPath: .spec.version
122
    - name: Status
123
      type: string
124
      jsonPath: .status.phase
125
    - name: Endpoint
126
      type: string
127
      jsonPath: .status.endpoint
128
    - name: Age
129
      type: date
130
      jsonPath: .metadata.creationTimestamp
131
  scope: Namespaced
132
  names:
133
    plural: databases
134
    singular: database
135
    kind: Database
136
    shortNames:
137
    - db

2. Creating a Custom Resource#

1
apiVersion: apps.example.com/v1
2
kind: Database
3
metadata:
4
  name: postgres-production
5
  namespace: production
6
spec:
7
  engine: postgres
8
  version: "14.5"
9
  size: large
10
  replicas: 3
11
  backup:
12
    enabled: true
13
    schedule: "0 2 * * *"
14
    retention: 30
15
  storage:
16
    size: "100Gi"
17
    class: "fast-ssd"
18
  monitoring:
19
    enabled: true
20
    interval: "30s"

3. CRD Versioning Strategy#

1
apiVersion: apiextensions.k8s.io/v1
2
kind: CustomResourceDefinition
3
metadata:
4
  name: databases.apps.example.com
5
spec:
6
  group: apps.example.com
7
  versions:
8
  # v1 - stable version
9
  - name: v1
10
    served: true
11
    storage: true
12
    schema:
13
      openAPIV3Schema:
14
        # ... v1 schema
15
  # v1beta1 - deprecated version
16
  - name: v1beta1
17
    served: true
18
    storage: false
19
    deprecated: true
20
    deprecationWarning: "apps.example.com/v1beta1 Database is deprecated; use apps.example.com/v1"
21
    schema:
22
      openAPIV3Schema:
23
        # ... v1beta1 schema
24
  # v2alpha1 - experimental version
25
  - name: v2alpha1
26
    served: true
27
    storage: false
28
    schema:
29
      openAPIV3Schema:
30
        # ... v2alpha1 schema with new features
31
  conversion:
32
    strategy: Webhook
33
    webhook:
34
      clientConfig:
35
        service:
36
          namespace: system
37
          name: webhook-service
38
          path: /convert
39
      conversionReviewVersions:
40
      - v1
41
      - v1beta1

Operator Development with Kubebuilder#

1. Initialize Operator Project#

1
#!/bin/bash
2
# Install Kubebuilder
3
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
4
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/
5

6
# Create new operator project
7
mkdir database-operator && cd database-operator
8
kubebuilder init --domain example.com --repo github.com/example/database-operator
9

10
# Create API
11
kubebuilder create api --group apps --version v1 --kind Database
12

13
# Generate manifests
14
make manifests

2. Implement Controller Logic#

1
package controllers
2

3
import (
4
    "context"
5
    "fmt"
6
    "time"
7

8
    appsv1 "k8s.io/api/apps/v1"
9
    corev1 "k8s.io/api/core/v1"
10
    "k8s.io/apimachinery/pkg/api/errors"
11
    "k8s.io/apimachinery/pkg/api/resource"
12
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
13
    "k8s.io/apimachinery/pkg/runtime"
14
    "k8s.io/apimachinery/pkg/types"
15
    ctrl "sigs.k8s.io/controller-runtime"
16
    "sigs.k8s.io/controller-runtime/pkg/client"
17
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
18
    "sigs.k8s.io/controller-runtime/pkg/log"
19

20
    appsv1alpha1 "github.com/example/database-operator/api/v1"
21
)
22

23
type DatabaseReconciler struct {
24
    client.Client
25
    Scheme *runtime.Scheme
26
}
27

28
//+kubebuilder:rbac:groups=apps.example.com,resources=databases,verbs=get;list;watch;create;update;patch;delete
29
//+kubebuilder:rbac:groups=apps.example.com,resources=databases/status,verbs=get;update;patch
30
//+kubebuilder:rbac:groups=apps.example.com,resources=databases/finalizers,verbs=update
31
//+kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
32
//+kubebuilder:rbac:groups=core,resources=services;persistentvolumeclaims;configmaps;secrets,verbs=get;list;watch;create;update;patch;delete
33

34
func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
35
    log := log.FromContext(ctx)
36

37
    // Fetch the Database instance
38
    database := &appsv1alpha1.Database{}
39
    err := r.Get(ctx, req.NamespacedName, database)
40
    if err != nil {
41
        if errors.IsNotFound(err) {
42
            log.Info("Database resource not found. Ignoring since object must be deleted")
43
            return ctrl.Result{}, nil
44
        }
45
        log.Error(err, "Failed to get Database")
46
        return ctrl.Result{}, err
47
    }
48

49
    // Add finalizer for cleanup
50
    finalizerName := "database.apps.example.com/finalizer"
51
    if database.ObjectMeta.DeletionTimestamp.IsZero() {
52
        if !controllerutil.ContainsFinalizer(database, finalizerName) {
53
            controllerutil.AddFinalizer(database, finalizerName)
54
            if err := r.Update(ctx, database); err != nil {
55
                return ctrl.Result{}, err
56
            }
57
        }
58
    } else {
59
        // Database is being deleted
60
        if controllerutil.ContainsFinalizer(database, finalizerName) {
61
            // Perform cleanup
62
            if err := r.cleanupDatabase(ctx, database); err != nil {
63
                return ctrl.Result{}, err
64
            }
65

66
            // Remove finalizer
67
            controllerutil.RemoveFinalizer(database, finalizerName)
68
            if err := r.Update(ctx, database); err != nil {
69
                return ctrl.Result{}, err
70
            }
71
        }
72
        return ctrl.Result{}, nil
73
    }
74

75
    // Reconcile Database resources
76
    if err := r.reconcileSecret(ctx, database); err != nil {
77
        return ctrl.Result{}, err
78
    }
79

80
    if err := r.reconcileConfigMap(ctx, database); err != nil {
81
        return ctrl.Result{}, err
82
    }
83

84
    if err := r.reconcileService(ctx, database); err != nil {
85
        return ctrl.Result{}, err
86
    }
87

88
    if err := r.reconcileStatefulSet(ctx, database); err != nil {
89
        return ctrl.Result{}, err
90
    }
91

92
    if err := r.reconcileBackupCronJob(ctx, database); err != nil {
93
        return ctrl.Result{}, err
94
    }
95

96
    // Update status
97
    if err := r.updateStatus(ctx, database); err != nil {
98
        return ctrl.Result{}, err
99
    }
100

101
    // Requeue after 1 minute for status checks
102
    return ctrl.Result{RequeueAfter: time.Minute}, nil
103
}
104

105
func (r *DatabaseReconciler) reconcileStatefulSet(ctx context.Context, db *appsv1alpha1.Database) error {
106
    log := log.FromContext(ctx)
107

108
    statefulSet := &appsv1.StatefulSet{}
109
    err := r.Get(ctx, types.NamespacedName{
110
        Name:      db.Name + "-sts",
111
        Namespace: db.Namespace,
112
    }, statefulSet)
113

114
    if err != nil && errors.IsNotFound(err) {
115
        // Create new StatefulSet
116
        sts := r.constructStatefulSet(db)
117
        if err := controllerutil.SetControllerReference(db, sts, r.Scheme); err != nil {
118
            return err
119
        }
120

121
        log.Info("Creating StatefulSet", "name", sts.Name)
122
        if err := r.Create(ctx, sts); err != nil {
123
            return err
124
        }
125
        return nil
126
    } else if err != nil {
127
        return err
128
    }
129

130
    // Update existing StatefulSet if needed
131
    desired := r.constructStatefulSet(db)
132
    if !r.isStatefulSetEqual(statefulSet, desired) {
133
        statefulSet.Spec = desired.Spec
134
        log.Info("Updating StatefulSet", "name", statefulSet.Name)
135
        if err := r.Update(ctx, statefulSet); err != nil {
136
            return err
137
        }
138
    }
139

140
    return nil
141
}
142

143
func (r *DatabaseReconciler) constructStatefulSet(db *appsv1alpha1.Database) *appsv1.StatefulSet {
144
    labels := map[string]string{
145
        "app":      "database",
146
        "database": db.Name,
147
        "engine":   db.Spec.Engine,
148
    }
149

150
    replicas := int32(1)
151
    if db.Spec.Replicas != nil {
152
        replicas = *db.Spec.Replicas
153
    }
154

155
    // Determine image based on engine and version
156
    image := r.getDatabaseImage(db.Spec.Engine, db.Spec.Version)
157

158
    // Determine resources based on size
159
    resources := r.getResourceRequirements(db.Spec.Size)
160

161
    return &appsv1.StatefulSet{
162
        ObjectMeta: metav1.ObjectMeta{
163
            Name:      db.Name + "-sts",
164
            Namespace: db.Namespace,
165
            Labels:    labels,
166
        },
167
        Spec: appsv1.StatefulSetSpec{
168
            Replicas:    &replicas,
169
            ServiceName: db.Name + "-svc",
170
            Selector: &metav1.LabelSelector{
171
                MatchLabels: labels,
172
            },
173
            Template: corev1.PodTemplateSpec{
174
                ObjectMeta: metav1.ObjectMeta{
175
                    Labels: labels,
176
                },
177
                Spec: corev1.PodSpec{
178
                    Containers: []corev1.Container{
179
                        {
180
                            Name:            "database",
181
                            Image:           image,
182
                            ImagePullPolicy: corev1.PullIfNotPresent,
183
                            Ports: []corev1.ContainerPort{
184
                                {
185
                                    Name:          "db",
186
                                    ContainerPort: r.getDatabasePort(db.Spec.Engine),
187
                                    Protocol:      corev1.ProtocolTCP,
188
                                },
189
                            },
190
                            EnvFrom: []corev1.EnvFromSource{
191
                                {
192
                                    SecretRef: &corev1.SecretEnvSource{
193
                                        LocalObjectReference: corev1.LocalObjectReference{
194
                                            Name: db.Name + "-secret",
195
                                        },
196
                                    },
197
                                },
198
                                {
199
                                    ConfigMapRef: &corev1.ConfigMapEnvSource{
200
                                        LocalObjectReference: corev1.LocalObjectReference{
201
                                            Name: db.Name + "-config",
202
                                        },
203
                                    },
204
                                },
205
                            },
206
                            Resources: resources,
207
                            VolumeMounts: []corev1.VolumeMount{
208
                                {
209
                                    Name:      "data",
210
                                    MountPath: "/var/lib/database",
211
                                },
212
                            },
213
                            LivenessProbe: &corev1.Probe{
214
                                ProbeHandler: corev1.ProbeHandler{
215
                                    TCPSocket: &corev1.TCPSocketAction{
216
                                        Port: intstr.FromString("db"),
217
                                    },
218
                                },
219
                                InitialDelaySeconds: 30,
220
                                PeriodSeconds:       10,
221
                            },
222
                            ReadinessProbe: &corev1.Probe{
223
                                ProbeHandler: corev1.ProbeHandler{
224
                                    TCPSocket: &corev1.TCPSocketAction{
225
                                        Port: intstr.FromString("db"),
226
                                    },
227
                                },
228
                                InitialDelaySeconds: 5,
229
                                PeriodSeconds:       5,
230
                            },
231
                        },
232
                    },
233
                },
234
            },
235
            VolumeClaimTemplates: []corev1.PersistentVolumeClaim{
236
                {
237
                    ObjectMeta: metav1.ObjectMeta{
238
                        Name: "data",
239
                    },
240
                    Spec: corev1.PersistentVolumeClaimSpec{
241
                        AccessModes: []corev1.PersistentVolumeAccessMode{
242
                            corev1.ReadWriteOnce,
243
                        },
244
                        StorageClassName: &db.Spec.Storage.Class,
245
                        Resources: corev1.ResourceRequirements{
246
                            Requests: corev1.ResourceList{
247
                                corev1.ResourceStorage: resource.MustParse(db.Spec.Storage.Size),
248
                            },
249
                        },
250
                    },
251
                },
252
            },
253
        },
254
    }
255
}
256

257
func (r *DatabaseReconciler) getDatabaseImage(engine, version string) string {
258
    imageMap := map[string]string{
259
        "postgres": fmt.Sprintf("postgres:%s", version),
260
        "mysql":    fmt.Sprintf("mysql:%s", version),
261
        "mongodb":  fmt.Sprintf("mongo:%s", version),
262
    }
263
    return imageMap[engine]
264
}
265

266
func (r *DatabaseReconciler) getResourceRequirements(size string) corev1.ResourceRequirements {
267
    sizeMap := map[string]corev1.ResourceRequirements{
268
        "small": {
269
            Requests: corev1.ResourceList{
270
                corev1.ResourceCPU:    resource.MustParse("500m"),
271
                corev1.ResourceMemory: resource.MustParse("1Gi"),
272
            },
273
            Limits: corev1.ResourceList{
274
                corev1.ResourceCPU:    resource.MustParse("1"),
275
                corev1.ResourceMemory: resource.MustParse("2Gi"),
276
            },
277
        },
278
        "medium": {
279
            Requests: corev1.ResourceList{
280
                corev1.ResourceCPU:    resource.MustParse("1"),
281
                corev1.ResourceMemory: resource.MustParse("4Gi"),
282
            },
283
            Limits: corev1.ResourceList{
284
                corev1.ResourceCPU:    resource.MustParse("2"),
285
                corev1.ResourceMemory: resource.MustParse("8Gi"),
286
            },
287
        },
288
        "large": {
289
            Requests: corev1.ResourceList{
290
                corev1.ResourceCPU:    resource.MustParse("2"),
291
                corev1.ResourceMemory: resource.MustParse("8Gi"),
292
            },
293
            Limits: corev1.ResourceList{
294
                corev1.ResourceCPU:    resource.MustParse("4"),
295
                corev1.ResourceMemory: resource.MustParse("16Gi"),
296
            },
297
        },
298
    }
299
    return sizeMap[size]
300
}
301

302
func (r *DatabaseReconciler) updateStatus(ctx context.Context, db *appsv1alpha1.Database) error {
303
    log := log.FromContext(ctx)
304

305
    // Get StatefulSet to check status
306
    sts := &appsv1.StatefulSet{}
307
    err := r.Get(ctx, types.NamespacedName{
308
        Name:      db.Name + "-sts",
309
        Namespace: db.Namespace,
310
    }, sts)
311

312
    if err != nil {
313
        db.Status.Phase = "Failed"
314
        db.Status.Message = fmt.Sprintf("StatefulSet not found: %v", err)
315
    } else {
316
        if sts.Status.ReadyReplicas == *sts.Spec.Replicas {
317
            db.Status.Phase = "Running"
318
            db.Status.Message = "All database replicas are running"
319
            db.Status.Endpoint = fmt.Sprintf("%s-svc.%s.svc.cluster.local:%d",
320
                db.Name, db.Namespace, r.getDatabasePort(db.Spec.Engine))
321
        } else {
322
            db.Status.Phase = "Creating"
323
            db.Status.Message = fmt.Sprintf("Waiting for replicas: %d/%d ready",
324
                sts.Status.ReadyReplicas, *sts.Spec.Replicas)
325
        }
326
    }
327

328
    // Update conditions
329
    db.Status.Conditions = []appsv1alpha1.Condition{
330
        {
331
            Type:               "Available",
332
            Status:             metav1.ConditionTrue,
333
            LastTransitionTime: metav1.Now(),
334
            Reason:             "DatabaseReady",
335
            Message:            "Database is available",
336
        },
337
    }
338

339
    log.Info("Updating Database status", "phase", db.Status.Phase)
340
    if err := r.Status().Update(ctx, db); err != nil {
341
        log.Error(err, "Failed to update Database status")
342
        return err
343
    }
344

345
    return nil
346
}
347

348
func (r *DatabaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
349
    return ctrl.NewControllerManagedBy(mgr).
350
        For(&appsv1alpha1.Database{}).
351
        Owns(&appsv1.StatefulSet{}).
352
        Owns(&corev1.Service{}).
353
        Owns(&corev1.Secret{}).
354
        Owns(&corev1.ConfigMap{}).
355
        Complete(r)
356
}

Advanced Controller Patterns#

1. Leader Election for HA#

1
// main.go - Leader election setup
2
package main
3

4
import (
5
    "flag"
6
    "os"
7

8
    "k8s.io/apimachinery/pkg/runtime"
9
    utilruntime "k8s.io/apimachinery/pkg/util/runtime"
10
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
11
    ctrl "sigs.k8s.io/controller-runtime"
12
    "sigs.k8s.io/controller-runtime/pkg/healthz"
13
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
14

15
    appsv1 "github.com/example/database-operator/api/v1"
16
    "github.com/example/database-operator/controllers"
17
)
18

19
var (
20
    scheme   = runtime.NewScheme()
21
    setupLog = ctrl.Log.WithName("setup")
22
)
23

24
func init() {
25
    utilruntime.Must(clientgoscheme.AddToScheme(scheme))
26
    utilruntime.Must(appsv1.AddToScheme(scheme))
27
}
28

29
func main() {
30
    var metricsAddr string
31
    var enableLeaderElection bool
32
    var probeAddr string
33

34
    flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
35
    flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
36
    flag.BoolVar(&enableLeaderElection, "leader-elect", false,
37
        "Enable leader election for controller manager. "+
38
        "Enabling this will ensure there is only one active controller manager.")
39
    flag.Parse()
40

41
    ctrl.SetLogger(zap.New(zap.UseDevMode(true)))
42

43
    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
44
        Scheme:                 scheme,
45
        MetricsBindAddress:     metricsAddr,
46
        Port:                   9443,
47
        HealthProbeBindAddress: probeAddr,
48
        LeaderElection:         enableLeaderElection,
49
        LeaderElectionID:       "database-operator-leader-election",
50
        // LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
51
        // when the Manager ends. This requires the binary to immediately end when the
52
        // Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
53
        // speeds up voluntary leader transitions as the new leader don't have to wait
54
        // LeaseDuration time first.
55
        LeaderElectionReleaseOnCancel: true,
56
    })
57

58
    if err != nil {
59
        setupLog.Error(err, "unable to start manager")
60
        os.Exit(1)
61
    }
62

63
    if err = (&controllers.DatabaseReconciler{
64
        Client: mgr.GetClient(),
65
        Scheme: mgr.GetScheme(),
66
    }).SetupWithManager(mgr); err != nil {
67
        setupLog.Error(err, "unable to create controller", "controller", "Database")
68
        os.Exit(1)
69
    }
70

71
    if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
72
        setupLog.Error(err, "unable to set up health check")
73
        os.Exit(1)
74
    }
75
    if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
76
        setupLog.Error(err, "unable to set up ready check")
77
        os.Exit(1)
78
    }
79

80
    setupLog.Info("starting manager")
81
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
82
        setupLog.Error(err, "problem running manager")
83
        os.Exit(1)
84
    }
85
}

2. Event Recording and Metrics#

1
package controllers
2

3
import (
4
    "github.com/prometheus/client_golang/prometheus"
5
    "sigs.k8s.io/controller-runtime/pkg/metrics"
6
)
7

8
var (
9
    reconciliationTotal = prometheus.NewCounterVec(
10
        prometheus.CounterOpts{
11
            Name: "database_operator_reconciliations_total",
12
            Help: "Total number of reconciliations per database",
13
        },
14
        []string{"database", "namespace"},
15
    )
16

17
    reconciliationErrors = prometheus.NewCounterVec(
18
        prometheus.CounterOpts{
19
            Name: "database_operator_reconciliation_errors_total",
20
            Help: "Total number of reconciliation errors per database",
21
        },
22
        []string{"database", "namespace", "error"},
23
    )
24

25
    databasePhase = prometheus.NewGaugeVec(
26
        prometheus.GaugeOpts{
27
            Name: "database_operator_database_phase",
28
            Help: "Current phase of the database",
29
        },
30
        []string{"database", "namespace", "phase"},
31
    )
32
)
33

34
func init() {
35
    // Register custom metrics
36
    metrics.Registry.MustRegister(
37
        reconciliationTotal,
38
        reconciliationErrors,
39
        databasePhase,
40
    )
41
}
42

43
// RecordMetrics records controller metrics
44
func RecordMetrics(database, namespace, phase string, err error) {
45
    reconciliationTotal.WithLabelValues(database, namespace).Inc()
46

47
    if err != nil {
48
        reconciliationErrors.WithLabelValues(database, namespace, err.Error()).Inc()
49
    }
50

51
    // Reset all phases to 0
52
    for _, p := range []string{"Pending", "Creating", "Running", "Failed", "Terminating"} {
53
        databasePhase.WithLabelValues(database, namespace, p).Set(0)
54
    }
55
    // Set current phase to 1
56
    databasePhase.WithLabelValues(database, namespace, phase).Set(1)
57
}

3. Webhook Validation#

1
package v1
2

3
import (
4
    "fmt"
5
    "k8s.io/apimachinery/pkg/runtime"
6
    ctrl "sigs.k8s.io/controller-runtime"
7
    logf "sigs.k8s.io/controller-runtime/pkg/log"
8
    "sigs.k8s.io/controller-runtime/pkg/webhook"
9
)
10

11
var databaselog = logf.Log.WithName("database-resource")
12

13
func (r *Database) SetupWebhookWithManager(mgr ctrl.Manager) error {
14
    return ctrl.NewWebhookManagedBy(mgr).
15
        For(r).
16
        Complete()
17
}
18

19
//+kubebuilder:webhook:path=/validate-apps-example-com-v1-database,mutating=false,failurePolicy=fail,sideEffects=None,groups=apps.example.com,resources=databases,verbs=create;update,versions=v1,name=vdatabase.kb.io,admissionReviewVersions=v1
20

21
var _ webhook.Validator = &Database{}
22

23
// ValidateCreate implements webhook.Validator
24
func (r *Database) ValidateCreate() error {
25
    databaselog.Info("validate create", "name", r.Name)
26

27
    // Validate engine-specific requirements
28
    switch r.Spec.Engine {
29
    case "postgres":
30
        if r.Spec.Replicas != nil && *r.Spec.Replicas > 1 && !r.hasReplicationConfig() {
31
            return fmt.Errorf("postgres with replicas > 1 requires replication configuration")
32
        }
33
    case "mysql":
34
        if r.Spec.Version < "5.7" {
35
            return fmt.Errorf("mysql version must be >= 5.7")
36
        }
37
    case "mongodb":
38
        if r.Spec.Replicas != nil && *r.Spec.Replicas%2 == 0 {
39
            return fmt.Errorf("mongodb replicas must be odd number for proper quorum")
40
        }
41
    }
42

43
    // Validate backup configuration
44
    if r.Spec.Backup.Enabled && r.Spec.Backup.Schedule == "" {
45
        return fmt.Errorf("backup schedule must be specified when backup is enabled")
46
    }
47

48
    return nil
49
}
50

51
// ValidateUpdate implements webhook.Validator
52
func (r *Database) ValidateUpdate(old runtime.Object) error {
53
    databaselog.Info("validate update", "name", r.Name)
54

55
    oldDB := old.(*Database)
56

57
    // Prevent engine changes
58
    if oldDB.Spec.Engine != r.Spec.Engine {
59
        return fmt.Errorf("database engine cannot be changed")
60
    }
61

62
    // Prevent major version downgrades
63
    if r.Spec.Version < oldDB.Spec.Version {
64
        return fmt.Errorf("database version downgrade is not allowed")
65
    }
66

67
    // Prevent storage size reduction
68
    oldSize := resource.MustParse(oldDB.Spec.Storage.Size)
69
    newSize := resource.MustParse(r.Spec.Storage.Size)
70
    if newSize.Cmp(oldSize) < 0 {
71
        return fmt.Errorf("storage size cannot be reduced")
72
    }
73

74
    return nil
75
}
76

77
// ValidateDelete implements webhook.Validator
78
func (r *Database) ValidateDelete() error {
79
    databaselog.Info("validate delete", "name", r.Name)
80

81
    // Add any cleanup validation here
82
    // For example, check if there are any dependent resources
83

84
    return nil
85
}
86

87
//+kubebuilder:webhook:path=/mutate-apps-example-com-v1-database,mutating=true,failurePolicy=fail,sideEffects=None,groups=apps.example.com,resources=databases,verbs=create;update,versions=v1,name=mdatabase.kb.io,admissionReviewVersions=v1
88

89
var _ webhook.Defaulter = &Database{}
90

91
// Default implements webhook.Defaulter
92
func (r *Database) Default() {
93
    databaselog.Info("default", "name", r.Name)
94

95
    // Set default replicas
96
    if r.Spec.Replicas == nil {
97
        replicas := int32(1)
98
        r.Spec.Replicas = &replicas
99
    }
100

101
    // Set default storage size based on database size
102
    if r.Spec.Storage.Size == "" {
103
        sizeMap := map[string]string{
104
            "small":  "10Gi",
105
            "medium": "50Gi",
106
            "large":  "100Gi",
107
        }
108
        r.Spec.Storage.Size = sizeMap[r.Spec.Size]
109
    }
110

111
    // Set default backup retention
112
    if r.Spec.Backup.Enabled && r.Spec.Backup.Retention == 0 {
113
        r.Spec.Backup.Retention = 30
114
    }
115
}

Production Best Practices#

1. Operator Deployment#

1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: database-operator-system
5
---
6
apiVersion: apps/v1
7
kind: Deployment
8
metadata:
9
  name: database-operator-controller-manager
10
  namespace: database-operator-system
11
spec:
12
  replicas: 3  # HA setup
13
  selector:
14
    matchLabels:
15
      control-plane: controller-manager
16
  template:
17
    metadata:
18
      labels:
19
        control-plane: controller-manager
20
    spec:
21
      serviceAccountName: database-operator-controller-manager
22
      containers:
23
      - name: manager
24
        image: example.com/database-operator:latest
25
        args:
26
        - --leader-elect
27
        - --metrics-bind-address=:8080
28
        - --health-probe-bind-address=:8081
29
        env:
30
        - name: WATCH_NAMESPACE
31
          value: ""  # Watch all namespaces
32
        ports:
33
        - containerPort: 9443
34
          name: webhook-server
35
          protocol: TCP
36
        - containerPort: 8080
37
          name: metrics
38
          protocol: TCP
39
        - containerPort: 8081
40
          name: health
41
          protocol: TCP
42
        livenessProbe:
43
          httpGet:
44
            path: /healthz
45
            port: 8081
46
          initialDelaySeconds: 15
47
          periodSeconds: 20
48
        readinessProbe:
49
          httpGet:
50
            path: /readyz
51
            port: 8081
52
          initialDelaySeconds: 5
53
          periodSeconds: 10
54
        resources:
55
          limits:
56
            cpu: 500m
57
            memory: 128Mi
58
          requests:
59
            cpu: 100m
60
            memory: 64Mi
61
        securityContext:
62
          allowPrivilegeEscalation: false
63
          capabilities:
64
            drop:
65
            - ALL
66
          readOnlyRootFilesystem: true
67
          runAsNonRoot: true
68
          runAsUser: 65532
69
      terminationGracePeriodSeconds: 10
70
      affinity:
71
        podAntiAffinity:
72
          preferredDuringSchedulingIgnoredDuringExecution:
73
          - weight: 100
74
            podAffinityTerm:
75
              labelSelector:
76
                matchExpressions:
77
                - key: control-plane
78
                  operator: In
79
                  values:
80
                  - controller-manager
81
              topologyKey: kubernetes.io/hostname

2. RBAC Configuration#

1
apiVersion: v1
2
kind: ServiceAccount
3
metadata:
4
  name: database-operator-controller-manager
5
  namespace: database-operator-system
6
---
7
apiVersion: rbac.authorization.k8s.io/v1
8
kind: ClusterRole
9
metadata:
10
  name: database-operator-manager-role
11
rules:
12
- apiGroups:
13
  - apps.example.com
14
  resources:
15
  - databases
16
  verbs:
17
  - create
18
  - delete
19
  - get
20
  - list
21
  - patch
22
  - update
23
  - watch
24
- apiGroups:
25
  - apps.example.com
26
  resources:
27
  - databases/finalizers
28
  verbs:
29
  - update
30
- apiGroups:
31
  - apps.example.com
32
  resources:
33
  - databases/status
34
  verbs:
35
  - get
36
  - patch
37
  - update
38
- apiGroups:
39
  - apps
40
  resources:
41
  - statefulsets
42
  verbs:
43
  - create
44
  - delete
45
  - get
46
  - list
47
  - patch
48
  - update
49
  - watch
50
- apiGroups:
51
  - ""
52
  resources:
53
  - configmaps
54
  - persistentvolumeclaims
55
  - secrets
56
  - services
57
  verbs:
58
  - create
59
  - delete
60
  - get
61
  - list
62
  - patch
63
  - update
64
  - watch
65
- apiGroups:
66
  - ""
67
  resources:
68
  - events
69
  verbs:
70
  - create
71
  - patch
72
- apiGroups:
73
  - batch
74
  resources:
75
  - cronjobs
76
  - jobs
77
  verbs:
78
  - create
79
  - delete
80
  - get
81
  - list
82
  - patch
83
  - update
84
  - watch
85
---
86
apiVersion: rbac.authorization.k8s.io/v1
87
kind: ClusterRoleBinding
88
metadata:
89
  name: database-operator-manager-rolebinding
90
roleRef:
91
  apiGroup: rbac.authorization.k8s.io
92
  kind: ClusterRole
93
  name: database-operator-manager-role
94
subjects:
95
- kind: ServiceAccount
96
  name: database-operator-controller-manager
97
  namespace: database-operator-system

Testing and Validation#

1. Unit Testing Controllers#

1
package controllers
2

3
import (
4
    "context"
5
    "testing"
6
    "time"
7

8
    . "github.com/onsi/ginkgo"
9
    . "github.com/onsi/gomega"
10
    appsv1 "k8s.io/api/apps/v1"
11
    corev1 "k8s.io/api/core/v1"
12
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
13
    "k8s.io/apimachinery/pkg/types"
14
    "sigs.k8s.io/controller-runtime/pkg/client"
15

16
    appsv1alpha1 "github.com/example/database-operator/api/v1"
17
)
18

19
func TestDatabaseController(t *testing.T) {
20
    RegisterFailHandler(Fail)
21
    RunSpecs(t, "Controller Suite")
22
}
23

24
var _ = Describe("Database Controller", func() {
25
    const (
26
        DatabaseName      = "test-database"
27
        DatabaseNamespace = "default"
28
        timeout           = time.Second * 10
29
        interval          = time.Millisecond * 250
30
    )
31

32
    Context("When creating Database", func() {
33
        It("Should create StatefulSet", func() {
34
            ctx := context.Background()
35
            database := &appsv1alpha1.Database{
36
                TypeMeta: metav1.TypeMeta{
37
                    APIVersion: "apps.example.com/v1",
38
                    Kind:       "Database",
39
                },
40
                ObjectMeta: metav1.ObjectMeta{
41
                    Name:      DatabaseName,
42
                    Namespace: DatabaseNamespace,
43
                },
44
                Spec: appsv1alpha1.DatabaseSpec{
45
                    Engine:  "postgres",
46
                    Version: "14.5",
47
                    Size:    "small",
48
                },
49
            }
50

51
            Expect(k8sClient.Create(ctx, database)).Should(Succeed())
52

53
            // Check if StatefulSet is created
54
            statefulSet := &appsv1.StatefulSet{}
55
            Eventually(func() bool {
56
                err := k8sClient.Get(ctx, types.NamespacedName{
57
                    Name:      DatabaseName + "-sts",
58
                    Namespace: DatabaseNamespace,
59
                }, statefulSet)
60
                return err == nil
61
            }, timeout, interval).Should(BeTrue())
62

63
            Expect(statefulSet.Spec.Replicas).Should(Equal(int32Ptr(1)))
64
            Expect(statefulSet.Spec.Template.Spec.Containers[0].Image).Should(Equal("postgres:14.5"))
65
        })
66

67
        It("Should create Service", func() {
68
            ctx := context.Background()
69

70
            service := &corev1.Service{}
71
            Eventually(func() bool {
72
                err := k8sClient.Get(ctx, types.NamespacedName{
73
                    Name:      DatabaseName + "-svc",
74
                    Namespace: DatabaseNamespace,
75
                }, service)
76
                return err == nil
77
            }, timeout, interval).Should(BeTrue())
78

79
            Expect(service.Spec.Type).Should(Equal(corev1.ServiceTypeClusterIP))
80
        })
81

82
        It("Should update Database status", func() {
83
            ctx := context.Background()
84

85
            database := &appsv1alpha1.Database{}
86
            Eventually(func() string {
87
                err := k8sClient.Get(ctx, types.NamespacedName{
88
                    Name:      DatabaseName,
89
                    Namespace: DatabaseNamespace,
90
                }, database)
91
                if err != nil {
92
                    return ""
93
                }
94
                return database.Status.Phase
95
            }, timeout, interval).Should(Equal("Creating"))
96
        })
97
    })
98
})
99

100
func int32Ptr(i int32) *int32 {
101
    return &i
102
}

2. Integration Testing#

1
#!/bin/bash
2
# Install CRDs
3
kubectl apply -f config/crd/bases/
4

5
# Deploy operator
6
kubectl apply -f config/manager/
7

8
# Create test database
9
cat <<EOF | kubectl apply -f -
10
apiVersion: apps.example.com/v1
11
kind: Database
12
metadata:
13
  name: test-postgres
14
  namespace: default
15
spec:
16
  engine: postgres
17
  version: "14.5"
18
  size: small
19
  replicas: 1
20
EOF
21

22
# Wait for database to be ready
23
kubectl wait --for=condition=Available database/test-postgres --timeout=300s
24

25
# Verify resources
26
echo "Checking created resources..."
27
kubectl get database test-postgres -o yaml
28
kubectl get sts test-postgres-sts
29
kubectl get svc test-postgres-svc
30
kubectl get secret test-postgres-secret
31
kubectl get cm test-postgres-config
32

33
# Test scaling
34
kubectl patch database test-postgres --type='json' \
35
  -p='[{"op": "replace", "path": "/spec/replicas", "value": 3}]'
36

37
# Wait for scaling
38
sleep 30
39
kubectl get sts test-postgres-sts
40

41
# Cleanup
42
kubectl delete database test-postgres

Real-World Examples#

1. Multi-Tenant Database Operator#

1
apiVersion: apps.example.com/v1
2
kind: Database
3
metadata:
4
  name: tenant-db-prod
5
  namespace: tenant-a
6
  labels:
7
    tenant: tenant-a
8
    environment: production
9
    compliance: pci-dss
10
spec:
11
  engine: postgres
12
  version: "14.5"
13
  size: large
14
  replicas: 3
15
  multiTenant:
16
    enabled: true
17
    isolation: strict
18
    resourceQuota:
19
      maxConnections: 100
20
      maxStorage: "500Gi"
21
  encryption:
22
    atRest: true
23
    inTransit: true
24
    keyProvider: vault
25
  backup:
26
    enabled: true
27
    schedule: "0 */6 * * *"
28
    retention: 90
29
    destination: s3://backups/tenant-a/
30
  monitoring:
31
    enabled: true
32
    interval: "15s"
33
    exporters:
34
    - postgres_exporter
35
    alerts:
36
    - name: HighConnections
37
      threshold: 80
38
    - name: LowDiskSpace
39
      threshold: 20
40
  maintenance:
41
    window: "SUN 02:00-06:00 UTC"
42
    autoUpgrade: true
43
    autoBackup: true

2. GitOps-Managed CRDs#

1
apiVersion: argoproj.io/v1alpha1
2
kind: Application
3
metadata:
4
  name: database-operator
5
  namespace: argocd
6
spec:
7
  project: default
8
  source:
9
    repoURL: https://github.com/example/database-operator
10
    targetRevision: HEAD
11
    path: config/
12
  destination:
13
    server: https://kubernetes.default.svc
14
    namespace: database-operator-system
15
  syncPolicy:
16
    automated:
17
      prune: true
18
      selfHeal: true
19
    syncOptions:
20
    - CreateNamespace=true
21
    retry:
22
      limit: 5
23
      backoff:
24
        duration: 5s
25
        factor: 2
26
        maxDuration: 3m
27
---
28
apiVersion: argoproj.io/v1alpha1
29
kind: Application
30
metadata:
31
  name: databases
32
  namespace: argocd
33
spec:
34
  project: default
35
  source:
36
    repoURL: https://github.com/example/databases
37
    targetRevision: HEAD
38
    path: manifests/
39
  destination:
40
    server: https://kubernetes.default.svc
41
  syncPolicy:
42
    automated:
43
      prune: false  # Don't delete databases automatically
44
      selfHeal: true

Monitoring and Observability#

1. Prometheus ServiceMonitor#

1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
  name: database-operator-metrics
5
  namespace: database-operator-system
6
spec:
7
  endpoints:
8
  - interval: 30s
9
    path: /metrics
10
    port: metrics
11
  selector:
12
    matchLabels:
13
      control-plane: controller-manager
14
---
15
apiVersion: monitoring.coreos.com/v1
16
kind: PrometheusRule
17
metadata:
18
  name: database-operator-alerts
19
  namespace: database-operator-system
20
spec:
21
  groups:
22
  - name: database-operator.rules
23
    interval: 30s
24
    rules:
25
    - alert: DatabaseOperatorDown
26
      expr: up{job="database-operator-metrics"} == 0
27
      for: 5m
28
      labels:
29
        severity: critical
30
      annotations:
31
        summary: "Database Operator is down"
32
        description: "Database Operator has been down for more than 5 minutes"
33

34
    - alert: HighReconciliationErrors
35
      expr: |
36
        rate(database_operator_reconciliation_errors_total[5m]) > 0.1
37
      for: 10m
38
      labels:
39
        severity: warning
40
      annotations:
41
        summary: "High reconciliation error rate"
42
        description: "Database {{ $labels.database }} has error rate of {{ $value }}"
43

44
    - alert: DatabaseNotReady
45
      expr: |
46
        database_operator_database_phase{phase="Failed"} == 1
47
      for: 15m
48
      labels:
49
        severity: warning
50
      annotations:
51
        summary: "Database in failed state"
52
        description: "Database {{ $labels.database }} has been in failed state for 15 minutes"

2. Grafana Dashboard#

1
{
2
  "dashboard": {
3
    "title": "Database Operator Metrics",
4
    "panels": [
5
      {
6
        "title": "Reconciliations per Second",
7
        "targets": [
8
          {
9
            "expr": "rate(database_operator_reconciliations_total[5m])"
10
          }
11
        ]
12
      },
13
      {
14
        "title": "Error Rate",
15
        "targets": [
16
          {
17
            "expr": "rate(database_operator_reconciliation_errors_total[5m])"
18
          }
19
        ]
20
      },
21
      {
22
        "title": "Database Status",
23
        "targets": [
24
          {
25
            "expr": "database_operator_database_phase"
26
          }
27
        ]
28
      },
29
      {
30
        "title": "Controller Memory Usage",
31
        "targets": [
32
          {
33
            "expr": "container_memory_usage_bytes{pod=~\"database-operator-.*\"}"
34
          }
35
        ]
36
      }
37
    ]
38
  }
39
}

Conclusion#

Building Kubernetes CRDs and Operators enables you to extend the platform with domain-specific functionality. By following these patterns and best practices, you can create robust, production-ready operators that automate complex application management.

Key Takeaways#

Design CRDs with comprehensive validation and versioning
Implement idempotent reconciliation logic
Use proper error handling and retry mechanisms
Enable high availability with leader election
Implement comprehensive testing strategies
Monitor operator performance and resource health

Next Steps#

Explore advanced operator patterns (state machines, finalizers)
Implement operator upgrades and migrations
Build multi-cluster operator management
Integrate with service mesh and observability tools
Develop operator certification strategies