CoreOS Kubernetes Deployment: Production-Ready Multi-Node Cluster#

This comprehensive guide provides detailed instructions for deploying production-ready Kubernetes clusters on CoreOS infrastructure. Learn to implement high availability, robust networking, persistent storage, comprehensive security, and operational excellence for enterprise-grade container orchestration.

Table of Contents#

Introduction to Production Kubernetes on CoreOS#

Architecture Overview#

A production Kubernetes cluster on CoreOS typically consists of:

Control Plane Nodes: Multiple masters for high availability
Worker Nodes: Scalable compute resources for workloads
Load Balancers: Traffic distribution and API server access
Storage Layer: Persistent storage for stateful applications
Network Layer: Pod-to-pod and service communication
Security Layer: RBAC, network policies, and encryption

CoreOS Advantages for Production#

Container-Optimized OS:

Minimal attack surface with essential components only
Automatic updates with rollback capabilities
Immutable infrastructure for consistent deployments

Built-in Security:

SELinux enforcement by default
Secure boot and verified boot chain
Container isolation and resource constraints

Operational Excellence:

Systemd integration for service management
Journald for centralized logging
Update strategies that minimize downtime

Infrastructure Planning and Prerequisites#

Hardware Requirements#

Control Plane Nodes (3 minimum for HA):

CPU: 4 cores minimum (8 recommended)
Memory: 8GB minimum (16GB recommended)
Storage: 100GB SSD minimum (NVMe preferred)
Network: 10Gbps interfaces for production

Worker Nodes (3+ for production):

CPU: 8+ cores (varies by workload)
Memory: 32GB+ (varies by workload)
Storage: 200GB+ SSD for OS, separate storage for applications
Network: 10Gbps interfaces for high-throughput workloads

Load Balancer Nodes (2 for HA):

CPU: 4 cores
Memory: 8GB
Storage: 50GB SSD
Network: High-bandwidth interface for cluster traffic

Network Architecture Design#

1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: network-topology
5
data:
6
  cluster_cidr: "10.244.0.0/16"
7
  service_cidr: "10.96.0.0/12"
8
  dns_domain: "cluster.local"
9

10
  control_plane_subnet: "192.168.10.0/24"
11
  worker_subnet: "192.168.20.0/24"
12
  storage_subnet: "192.168.30.0/24"
13

14
  api_server_lb: "192.168.10.100"
15
  ingress_lb: "192.168.10.101"
16

17
  network_policies:
18
    enabled: true
19
    default_deny: true
20
    inter_namespace_communication: false

CoreOS Infrastructure Setup#

Ignition Configuration for Control Plane#

1
variant: fcos
2
version: 1.4.0
3

4
passwd:
5
  users:
6
    - name: core
7
      ssh_authorized_keys:
8
        - ssh-rsa AAAAB3NzaC1yc2EAAAA... # Your SSH public key
9
      groups:
10
        - sudo
11
        - docker
12
      shell: /bin/bash
13
    - name: k8s-admin
14
      ssh_authorized_keys:
15
        - ssh-rsa AAAAB3NzaC1yc2EAAAA... # Admin SSH key
16
      groups:
17
        - sudo
18
      shell: /bin/bash
19

20
systemd:
21
  units:
22
    - name: docker.service
23
      enabled: true
24
    - name: kubelet.service
25
      enabled: true
26
      contents: |
27
        [Unit]
28
        Description=Kubernetes Kubelet
29
        Documentation=https://kubernetes.io/docs/
30
        After=docker.service
31
        Requires=docker.service
32

33
        [Service]
34
        ExecStart=/usr/local/bin/kubelet
35
        Restart=always
36
        StartLimitInterval=0
37
        RestartSec=10
38

39
        [Install]
40
        WantedBy=multi-user.target
41

42
    - name: setup-kubernetes.service
43
      enabled: true
44
      contents: |
45
        [Unit]
46
        Description=Setup Kubernetes Master
47
        After=docker.service network-online.target
48
        Requires=docker.service network-online.target
49

50
        [Service]
51
        Type=oneshot
52
        ExecStart=/usr/local/bin/setup-master.sh
53
        RemainAfterExit=yes
54

55
        [Install]
56
        WantedBy=multi-user.target
57

58
    - name: etcd-backup.service
59
      enabled: true
60
      contents: |
61
        [Unit]
62
        Description=etcd Backup Service
63

64
        [Service]
65
        Type=oneshot
66
        ExecStart=/usr/local/bin/backup-etcd.sh
67

68
    - name: etcd-backup.timer
69
      enabled: true
70
      contents: |
71
        [Unit]
72
        Description=etcd Backup Timer
73
        Requires=etcd-backup.service
74

75
        [Timer]
76
        OnCalendar=*-*-* 02:00:00
77
        Persistent=true
78

79
        [Install]
80
        WantedBy=timers.target
81

82
storage:
83
  directories:
84
    - path: /opt/kubernetes
85
      mode: 0755
86
    - path: /var/lib/etcd
87
      mode: 0700
88
    - path: /etc/kubernetes
89
      mode: 0755
90
    - path: /etc/kubernetes/pki
91
      mode: 0700
92
    - path: /var/log/pods
93
      mode: 0755
94
    - path: /opt/cni/bin
95
      mode: 0755
96
    - path: /etc/cni/net.d
97
      mode: 0755
98

99
  files:
100
    - path: /etc/hostname
101
      mode: 0644
102
      contents:
103
        inline: k8s-master-01 # Change for each master node
104

105
    - path: /etc/hosts
106
      mode: 0644
107
      contents:
108
        inline: |
109
          127.0.0.1 localhost
110
          192.168.10.10 k8s-master-01
111
          192.168.10.11 k8s-master-02
112
          192.168.10.12 k8s-master-03
113
          192.168.10.100 k8s-api-lb
114
          192.168.20.10 k8s-worker-01
115
          192.168.20.11 k8s-worker-02
116
          192.168.20.12 k8s-worker-03
117

118
    - path: /usr/local/bin/setup-master.sh
119
      mode: 0755
120
      contents:
121
        inline: |
122
          #!/bin/bash
123
          set -euxo pipefail
124

125
          KUBERNETES_VERSION="1.28.0"
126
          NODE_NAME=$(hostname)
127

128
          # Install Kubernetes components
129
          curl -L --remote-name-all https://dl.k8s.io/release/v${KUBERNETES_VERSION}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
130
          chmod +x {kubeadm,kubelet,kubectl}
131
          mv {kubeadm,kubelet,kubectl} /usr/local/bin/
132

133
          # Setup kubelet systemd service
134
          curl -sSL "https://raw.githubusercontent.com/kubernetes/release/v0.15.1/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:/usr/local/bin:g" > /etc/systemd/system/kubelet.service
135
          mkdir -p /etc/systemd/system/kubelet.service.d
136
          curl -sSL "https://raw.githubusercontent.com/kubernetes/release/v0.15.1/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:/usr/local/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
137

138
          # Configure kubelet
139
          cat > /etc/default/kubelet << EOF
140
          KUBELET_EXTRA_ARGS="--container-runtime=docker --cgroup-driver=systemd --fail-swap-on=false"
141
          EOF
142

143
          systemctl daemon-reload
144
          systemctl enable kubelet
145

146
          # Install CNI plugins
147
          CNI_VERSION="v1.3.0"
148
          mkdir -p /opt/cni/bin
149
          curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz
150

151
          echo "Kubernetes components installed successfully"
152

153
    - path: /etc/kubernetes/kubeadm-config.yaml
154
      mode: 0644
155
      contents:
156
        inline: |
157
          apiVersion: kubeadm.k8s.io/v1beta3
158
          kind: InitConfiguration
159
          localAPIEndpoint:
160
            advertiseAddress: "192.168.10.10"  # Change for each master
161
            bindPort: 6443
162
          nodeRegistration:
163
            criSocket: "/var/run/dockershim.sock"
164
            kubeletExtraArgs:
165
              cloud-provider: ""
166
              container-runtime: "docker"
167
              cgroup-driver: "systemd"
168
              fail-swap-on: "false"
169
          ---
170
          apiVersion: kubeadm.k8s.io/v1beta3
171
          kind: ClusterConfiguration
172
          kubernetesVersion: "v1.28.0"
173
          clusterName: "production-cluster"
174
          controlPlaneEndpoint: "k8s-api-lb:6443"
175
          networking:
176
            serviceSubnet: "10.96.0.0/12"
177
            podSubnet: "10.244.0.0/16"
178
            dnsDomain: "cluster.local"
179
          etcd:
180
            local:
181
              dataDir: "/var/lib/etcd"
182
              extraArgs:
183
                listen-metrics-urls: "http://0.0.0.0:2381"
184
          apiServer:
185
            bindPort: 6443
186
            extraArgs:
187
              authorization-mode: "Node,RBAC"
188
              enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction"
189
              audit-log-path: "/var/log/audit.log"
190
              audit-log-maxage: "30"
191
              audit-log-maxbackup: "3"
192
              audit-log-maxsize: "100"
193
              audit-policy-file: "/etc/kubernetes/audit-policy.yaml"
194
              enable-swagger-ui: "false"
195
              profiling: "false"
196
              repair-malformed-updates: "false"
197
              service-cluster-ip-range: "10.96.0.0/12"
198
              service-node-port-range: "30000-32767"
199
          controllerManager:
200
            extraArgs:
201
              bind-address: "0.0.0.0"
202
              service-cluster-ip-range: "10.96.0.0/12"
203
              cluster-cidr: "10.244.0.0/16"
204
              profiling: "false"
205
          scheduler:
206
            extraArgs:
207
              bind-address: "0.0.0.0"
208
              profiling: "false"
209
          ---
210
          apiVersion: kubelet.config.k8s.io/v1beta1
211
          kind: KubeletConfiguration
212
          failSwapOn: false
213
          containerRuntimeEndpoint: "unix:///var/run/dockershim.sock"
214
          cgroupDriver: "systemd"
215
          clusterDNS:
216
            - "10.96.0.10"
217
          clusterDomain: "cluster.local"
218
          authentication:
219
            anonymous:
220
              enabled: false
221
            webhook:
222
              enabled: true
223
          authorization:
224
            mode: "Webhook"
225
          readOnlyPort: 0
226
          protectKernelDefaults: true
227
          makeIPTablesUtilChains: true
228
          eventRecordQPS: 0
229
          rotateCertificates: true
230
          serverTLSBootstrap: true
231

232
    - path: /etc/kubernetes/audit-policy.yaml
233
      mode: 0644
234
      contents:
235
        inline: |
236
          apiVersion: audit.k8s.io/v1
237
          kind: Policy
238
          rules:
239
          - level: Metadata
240
            resources:
241
            - group: ""
242
              resources: ["secrets", "configmaps"]
243
          - level: RequestResponse
244
            resources:
245
            - group: ""
246
              resources: ["pods", "services", "nodes"]
247
          - level: Request
248
            resources:
249
            - group: "rbac.authorization.k8s.io"
250
              resources: ["*"]
251
          - level: Metadata
252
            omitStages:
253
            - "RequestReceived"
254

255
    - path: /usr/local/bin/backup-etcd.sh
256
      mode: 0755
257
      contents:
258
        inline: |
259
          #!/bin/bash
260
          set -euo pipefail
261

262
          BACKUP_DIR="/opt/kubernetes/backups"
263
          DATE=$(date +%Y%m%d_%H%M%S)
264
          BACKUP_FILE="$BACKUP_DIR/etcd-snapshot-$DATE.db"
265

266
          mkdir -p $BACKUP_DIR
267

268
          # Create etcd snapshot
269
          ETCDCTL_API=3 etcdctl snapshot save $BACKUP_FILE \
270
            --endpoints=https://127.0.0.1:2379 \
271
            --cacert=/etc/kubernetes/pki/etcd/ca.crt \
272
            --cert=/etc/kubernetes/pki/etcd/server.crt \
273
            --key=/etc/kubernetes/pki/etcd/server.key
274

275
          # Verify snapshot
276
          ETCDCTL_API=3 etcdctl snapshot status $BACKUP_FILE \
277
            --write-out=table
278

279
          # Cleanup old backups (keep last 7 days)
280
          find $BACKUP_DIR -name "etcd-snapshot-*.db" -mtime +7 -delete
281

282
          echo "etcd backup completed: $BACKUP_FILE"
283

284
    - path: /etc/docker/daemon.json
285
      mode: 0644
286
      contents:
287
        inline: |
288
          {
289
            "exec-opts": ["native.cgroupdriver=systemd"],
290
            "log-driver": "journald",
291
            "log-opts": {
292
              "max-size": "100m",
293
              "max-file": "5"
294
            },
295
            "storage-driver": "overlay2",
296
            "storage-opts": [
297
              "overlay2.override_kernel_check=true"
298
            ],
299
            "live-restore": true,
300
            "userland-proxy": false,
301
            "no-new-privileges": true,
302
            "seccomp-profile": "/etc/docker/seccomp.json",
303
            "default-ulimits": {
304
              "nofile": {
305
                "Hard": 64000,
306
                "Name": "nofile",
307
                "Soft": 64000
308
              }
309
            }
310
          }

Ignition Configuration for Worker Nodes#

1
variant: fcos
2
version: 1.4.0
3

4
passwd:
5
  users:
6
    - name: core
7
      ssh_authorized_keys:
8
        - ssh-rsa AAAAB3NzaC1yc2EAAAA... # Your SSH public key
9
      groups:
10
        - sudo
11
        - docker
12
      shell: /bin/bash
13

14
systemd:
15
  units:
16
    - name: docker.service
17
      enabled: true
18
    - name: kubelet.service
19
      enabled: true
20
      contents: |
21
        [Unit]
22
        Description=Kubernetes Kubelet
23
        Documentation=https://kubernetes.io/docs/
24
        After=docker.service
25
        Requires=docker.service
26

27
        [Service]
28
        ExecStart=/usr/local/bin/kubelet
29
        Restart=always
30
        StartLimitInterval=0
31
        RestartSec=10
32

33
        [Install]
34
        WantedBy=multi-user.target
35

36
    - name: setup-worker.service
37
      enabled: true
38
      contents: |
39
        [Unit]
40
        Description=Setup Kubernetes Worker
41
        After=docker.service network-online.target
42
        Requires=docker.service network-online.target
43

44
        [Service]
45
        Type=oneshot
46
        ExecStart=/usr/local/bin/setup-worker.sh
47
        RemainAfterExit=yes
48

49
        [Install]
50
        WantedBy=multi-user.target
51

52
storage:
53
  directories:
54
    - path: /opt/kubernetes
55
      mode: 0755
56
    - path: /etc/kubernetes
57
      mode: 0755
58
    - path: /var/log/pods
59
      mode: 0755
60
    - path: /opt/cni/bin
61
      mode: 0755
62
    - path: /etc/cni/net.d
63
      mode: 0755
64

65
  files:
66
    - path: /etc/hostname
67
      mode: 0644
68
      contents:
69
        inline: k8s-worker-01 # Change for each worker node
70

71
    - path: /etc/hosts
72
      mode: 0644
73
      contents:
74
        inline: |
75
          127.0.0.1 localhost
76
          192.168.10.10 k8s-master-01
77
          192.168.10.11 k8s-master-02
78
          192.168.10.12 k8s-master-03
79
          192.168.10.100 k8s-api-lb
80
          192.168.20.10 k8s-worker-01
81
          192.168.20.11 k8s-worker-02
82
          192.168.20.12 k8s-worker-03
83

84
    - path: /usr/local/bin/setup-worker.sh
85
      mode: 0755
86
      contents:
87
        inline: |
88
          #!/bin/bash
89
          set -euxo pipefail
90

91
          KUBERNETES_VERSION="1.28.0"
92

93
          # Install Kubernetes components
94
          curl -L --remote-name-all https://dl.k8s.io/release/v${KUBERNETES_VERSION}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
95
          chmod +x {kubeadm,kubelet,kubectl}
96
          mv {kubeadm,kubelet,kubectl} /usr/local/bin/
97

98
          # Setup kubelet systemd service
99
          curl -sSL "https://raw.githubusercontent.com/kubernetes/release/v0.15.1/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:/usr/local/bin:g" > /etc/systemd/system/kubelet.service
100
          mkdir -p /etc/systemd/system/kubelet.service.d
101
          curl -sSL "https://raw.githubusercontent.com/kubernetes/release/v0.15.1/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:/usr/local/bin:g" > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
102

103
          # Configure kubelet
104
          cat > /etc/default/kubelet << EOF
105
          KUBELET_EXTRA_ARGS="--container-runtime=docker --cgroup-driver=systemd --fail-swap-on=false"
106
          EOF
107

108
          systemctl daemon-reload
109
          systemctl enable kubelet
110

111
          # Install CNI plugins
112
          CNI_VERSION="v1.3.0"
113
          mkdir -p /opt/cni/bin
114
          curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz
115

116
          echo "Worker node setup completed"
117

118
    - path: /etc/docker/daemon.json
119
      mode: 0644
120
      contents:
121
        inline: |
122
          {
123
            "exec-opts": ["native.cgroupdriver=systemd"],
124
            "log-driver": "journald",
125
            "log-opts": {
126
              "max-size": "100m",
127
              "max-file": "5"
128
            },
129
            "storage-driver": "overlay2",
130
            "storage-opts": [
131
              "overlay2.override_kernel_check=true"
132
            ],
133
            "live-restore": true,
134
            "userland-proxy": false,
135
            "no-new-privileges": true,
136
            "default-ulimits": {
137
              "nofile": {
138
                "Hard": 64000,
139
                "Name": "nofile",
140
                "Soft": 64000
141
              }
142
            }
143
          }

High Availability Load Balancer Setup#

HAProxy Configuration for API Server#

1
#!/bin/bash
2
# setup-haproxy.sh - Load balancer setup for Kubernetes API
3

4
# Install HAProxy
5
dnf install -y haproxy keepalived
6

7
# Configure HAProxy
8
cat > /etc/haproxy/haproxy.cfg << 'EOF'
9
global
10
    log stdout len 65536 local0 info
11
    chroot /var/lib/haproxy
12
    stats socket /run/haproxy/admin.sock mode 660 level admin
13
    stats timeout 30s
14
    user haproxy
15
    group haproxy
16
    daemon
17

18
defaults
19
    mode http
20
    log global
21
    option httplog
22
    option dontlognull
23
    option log-health-checks
24
    option forwardfor except 127.0.0.0/8
25
    option redispatch
26
    retries 3
27
    timeout http-request 10s
28
    timeout queue 20s
29
    timeout connect 10s
30
    timeout client 1m
31
    timeout server 1m
32
    timeout http-keep-alive 10s
33
    timeout check 10s
34

35
# Statistics
36
listen stats
37
    bind *:8404
38
    stats enable
39
    stats uri /stats
40
    stats refresh 5s
41
    stats admin if TRUE
42

43
# Kubernetes API Server
44
frontend k8s_api_frontend
45
    bind *:6443
46
    mode tcp
47
    option tcplog
48
    default_backend k8s_api_backend
49

50
backend k8s_api_backend
51
    mode tcp
52
    balance roundrobin
53
    option tcp-check
54

55
    # Health check
56
    tcp-check connect
57
    tcp-check send-binary 474554202f20485454502f312e310d0a0d0a
58
    tcp-check expect binary 485454502f312e31
59

60
    # Master nodes
61
    server k8s-master-01 192.168.10.10:6443 check inter 5s rise 3 fall 3
62
    server k8s-master-02 192.168.10.11:6443 check inter 5s rise 3 fall 3
63
    server k8s-master-03 192.168.10.12:6443 check inter 5s rise 3 fall 3
64

65
# Ingress Controller (if needed)
66
frontend k8s_ingress_http
67
    bind *:80
68
    mode http
69
    redirect scheme https code 301 if !{ ssl_fc }
70

71
frontend k8s_ingress_https
72
    bind *:443
73
    mode tcp
74
    default_backend k8s_ingress_backend
75

76
backend k8s_ingress_backend
77
    mode tcp
78
    balance roundrobin
79
    option tcp-check
80

81
    # Worker nodes (where ingress controllers run)
82
    server k8s-worker-01 192.168.20.10:443 check inter 5s rise 3 fall 3
83
    server k8s-worker-02 192.168.20.11:443 check inter 5s rise 3 fall 3
84
    server k8s-worker-03 192.168.20.12:443 check inter 5s rise 3 fall 3
85
EOF
86

87
# Configure Keepalived for HA
88
cat > /etc/keepalived/keepalived.conf << 'EOF'
89
vrrp_script chk_haproxy {
90
    script "/bin/curl -f http://localhost:8404/stats || exit 1"
91
    interval 3
92
    weight -2
93
    fall 3
94
    rise 2
95
}
96

97
vrrp_instance VI_1 {
98
    state MASTER
99
    interface eth0
100
    virtual_router_id 51
101
    priority 110  # Set to 100 on backup node
102
    advert_int 1
103
    authentication {
104
        auth_type PASS
105
        auth_pass changeme123
106
    }
107
    virtual_ipaddress {
108
        192.168.10.100/24
109
    }
110
    track_script {
111
        chk_haproxy
112
    }
113
}
114
EOF
115

116
# Enable and start services
117
systemctl enable haproxy keepalived
118
systemctl start haproxy keepalived
119

120
echo "HAProxy and Keepalived configured for Kubernetes API HA"

Cluster Initialization and Bootstrap#

Master Node Initialization Script#

1
#!/bin/bash
2
# initialize-cluster.sh - Initialize the first control plane node
3

4
set -euo pipefail
5

6
CLUSTER_NAME="production-cluster"
7
POD_SUBNET="10.244.0.0/16"
8
SERVICE_SUBNET="10.96.0.0/12"
9
API_SERVER_ENDPOINT="k8s-api-lb:6443"
10

11
log() {
12
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
13
}
14

15
# Initialize the first control plane node
16
initialize_first_master() {
17
    log "Initializing first control plane node..."
18

19
    # Pre-pull images to speed up initialization
20
    kubeadm config images pull --kubernetes-version=v1.28.0
21

22
    # Initialize cluster
23
    kubeadm init --config=/etc/kubernetes/kubeadm-config.yaml --upload-certs --v=5
24

25
    # Setup kubectl for root
26
    mkdir -p $HOME/.kube
27
    cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
28
    chown $(id -u):$(id -g) $HOME/.kube/config
29

30
    # Setup kubectl for core user
31
    mkdir -p /home/core/.kube
32
    cp -i /etc/kubernetes/admin.conf /home/core/.kube/config
33
    chown core:core /home/core/.kube/config
34

35
    log "First control plane node initialized successfully"
36
}
37

38
# Install Calico CNI
39
install_calico_cni() {
40
    log "Installing Calico CNI..."
41

42
    # Download Calico manifests
43
    curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
44
    curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml
45

46
    # Modify custom resources for our pod CIDR
47
    sed -i "s|192.168.0.0/16|$POD_SUBNET|g" custom-resources.yaml
48

49
    # Apply Calico
50
    kubectl create -f tigera-operator.yaml
51
    kubectl create -f custom-resources.yaml
52

53
    log "Calico CNI installed successfully"
54
}
55

56
# Generate join commands
57
generate_join_commands() {
58
    log "Generating join commands..."
59

60
    # Control plane join command
61
    CERT_KEY=$(kubeadm init phase upload-certs --upload-certs | tail -1)
62
    MASTER_JOIN_CMD=$(kubeadm token create --print-join-command)
63

64
    echo "=== CONTROL PLANE JOIN COMMAND ==="
65
    echo "$MASTER_JOIN_CMD --control-plane --certificate-key $CERT_KEY"
66
    echo
67
    echo "=== WORKER JOIN COMMAND ==="
68
    echo "$MASTER_JOIN_CMD"
69
    echo
70

71
    # Save commands to files
72
    echo "$MASTER_JOIN_CMD --control-plane --certificate-key $CERT_KEY" > /opt/kubernetes/master-join-command.sh
73
    echo "$MASTER_JOIN_CMD" > /opt/kubernetes/worker-join-command.sh
74
    chmod +x /opt/kubernetes/*-join-command.sh
75

76
    log "Join commands saved to /opt/kubernetes/"
77
}
78

79
# Configure RBAC
80
setup_rbac() {
81
    log "Setting up RBAC..."
82

83
    # Create admin user
84
    cat > /tmp/admin-user.yaml << EOF
85
apiVersion: v1
86
kind: ServiceAccount
87
metadata:
88
  name: admin-user
89
  namespace: kube-system
90
---
91
apiVersion: rbac.authorization.k8s.io/v1
92
kind: ClusterRoleBinding
93
metadata:
94
  name: admin-user
95
roleRef:
96
  apiGroup: rbac.authorization.k8s.io
97
  kind: ClusterRole
98
  name: cluster-admin
99
subjects:
100
- kind: ServiceAccount
101
  name: admin-user
102
  namespace: kube-system
103
EOF
104

105
    kubectl apply -f /tmp/admin-user.yaml
106

107
    # Create read-only user
108
    cat > /tmp/readonly-user.yaml << EOF
109
apiVersion: v1
110
kind: ServiceAccount
111
metadata:
112
  name: readonly-user
113
  namespace: kube-system
114
---
115
apiVersion: rbac.authorization.k8s.io/v1
116
kind: ClusterRole
117
metadata:
118
  name: readonly-cluster-role
119
rules:
120
- apiGroups: [""]
121
  resources: ["*"]
122
  verbs: ["get", "list", "watch"]
123
- apiGroups: ["apps", "extensions"]
124
  resources: ["*"]
125
  verbs: ["get", "list", "watch"]
126
---
127
apiVersion: rbac.authorization.k8s.io/v1
128
kind: ClusterRoleBinding
129
metadata:
130
  name: readonly-user
131
roleRef:
132
  apiGroup: rbac.authorization.k8s.io
133
  kind: ClusterRole
134
  name: readonly-cluster-role
135
subjects:
136
- kind: ServiceAccount
137
  name: readonly-user
138
  namespace: kube-system
139
EOF
140

141
    kubectl apply -f /tmp/readonly-user.yaml
142

143
    log "RBAC configured successfully"
144
}
145

146
# Main execution
147
main() {
148
    log "Starting Kubernetes cluster initialization"
149

150
    initialize_first_master
151
    install_calico_cni
152
    setup_rbac
153

154
    # Wait for cluster to be ready
155
    log "Waiting for cluster to be ready..."
156
    kubectl wait --for=condition=Ready nodes --all --timeout=300s
157
    kubectl wait --for=condition=Available deployments --all -n kube-system --timeout=300s
158

159
    generate_join_commands
160

161
    log "Cluster initialization completed successfully"
162
    log "Cluster status:"
163
    kubectl get nodes -o wide
164
    kubectl get pods --all-namespaces
165
}
166

167
main "$@"

Additional Master Node Setup#

1
#!/bin/bash
2
# join-master.sh - Join additional control plane nodes
3

4
set -euo pipefail
5

6
MASTER_JOIN_COMMAND="$1"
7

8
if [ -z "$MASTER_JOIN_COMMAND" ]; then
9
    echo "Usage: $0 '<master-join-command>'"
10
    echo "Get the join command from the first master node"
11
    exit 1
12
fi
13

14
log() {
15
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
16
}
17

18
# Join as control plane node
19
join_control_plane() {
20
    log "Joining as control plane node..."
21

22
    # Execute join command
23
    eval "$MASTER_JOIN_COMMAND"
24

25
    # Setup kubectl for root
26
    mkdir -p $HOME/.kube
27
    cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
28
    chown $(id -u):$(id -g) $HOME/.kube/config
29

30
    # Setup kubectl for core user
31
    mkdir -p /home/core/.kube
32
    cp -i /etc/kubernetes/admin.conf /home/core/.kube/config
33
    chown core:core /home/core/.kube/config
34

35
    log "Successfully joined as control plane node"
36
}
37

38
# Verify cluster health
39
verify_cluster() {
40
    log "Verifying cluster health..."
41

42
    # Wait for node to be ready
43
    kubectl wait --for=condition=Ready node/$(hostname) --timeout=300s
44

45
    # Check cluster status
46
    kubectl get nodes
47
    kubectl get pods --all-namespaces
48

49
    log "Cluster verification completed"
50
}
51

52
main() {
53
    log "Starting control plane node join process"
54

55
    join_control_plane
56
    verify_cluster
57

58
    log "Control plane node join completed successfully"
59
}
60

61
main "$@"

Worker Node Setup#

1
#!/bin/bash
2
# join-worker.sh - Join worker nodes to the cluster
3

4
set -euo pipefail
5

6
WORKER_JOIN_COMMAND="$1"
7

8
if [ -z "$WORKER_JOIN_COMMAND" ]; then
9
    echo "Usage: $0 '<worker-join-command>'"
10
    echo "Get the join command from a master node"
11
    exit 1
12
fi
13

14
log() {
15
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
16
}
17

18
# Prepare worker node
19
prepare_worker() {
20
    log "Preparing worker node..."
21

22
    # Ensure Docker is running
23
    systemctl enable docker
24
    systemctl start docker
25

26
    # Configure system settings
27
    modprobe br_netfilter
28
    echo 'net.bridge.bridge-nf-call-iptables = 1' >> /etc/sysctl.conf
29
    echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
30
    sysctl -p
31

32
    log "Worker node preparation completed"
33
}
34

35
# Join cluster as worker
36
join_worker() {
37
    log "Joining cluster as worker node..."
38

39
    # Execute join command
40
    eval "$WORKER_JOIN_COMMAND"
41

42
    log "Successfully joined cluster as worker node"
43
}
44

45
# Configure worker-specific settings
46
configure_worker() {
47
    log "Configuring worker node settings..."
48

49
    # Label node based on its role/purpose
50
    NODE_NAME=$(hostname)
51

52
    # Wait for node to be ready
53
    sleep 30
54

55
    # Apply node labels (run from master node)
56
    cat > /tmp/label-worker.sh << 'EOF'
57
#!/bin/bash
58
NODE_NAME="$1"
59
kubectl label node "$NODE_NAME" node-role.kubernetes.io/worker=worker
60
kubectl label node "$NODE_NAME" node.kubernetes.io/instance-type=worker
61
EOF
62

63
    chmod +x /tmp/label-worker.sh
64
    echo "Run on master node: /tmp/label-worker.sh $NODE_NAME"
65

66
    log "Worker node configuration completed"
67
}
68

69
main() {
70
    log "Starting worker node join process"
71

72
    prepare_worker
73
    join_worker
74
    configure_worker
75

76
    log "Worker node join completed successfully"
77
}
78

79
main "$@"

Storage Configuration#

Persistent Storage Setup#

1
apiVersion: storage.k8s.io/v1
2
kind: StorageClass
3
metadata:
4
  name: fast-ssd
5
  annotations:
6
    storageclass.kubernetes.io/is-default-class: "true"
7
provisioner: kubernetes.io/no-provisioner
8
volumeBindingMode: WaitForFirstConsumer
9
reclaimPolicy: Delete
10
---
11
apiVersion: storage.k8s.io/v1
12
kind: StorageClass
13
metadata:
14
  name: bulk-storage
15
provisioner: kubernetes.io/no-provisioner
16
volumeBindingMode: WaitForFirstConsumer
17
reclaimPolicy: Retain
18
---
19
# Local storage provisioner
20
apiVersion: apps/v1
21
kind: DaemonSet
22
metadata:
23
  name: local-volume-provisioner
24
  namespace: kube-system
25
spec:
26
  selector:
27
    matchLabels:
28
      app: local-volume-provisioner
29
  template:
30
    metadata:
31
      labels:
32
        app: local-volume-provisioner
33
    spec:
34
      serviceAccountName: local-storage-admin
35
      containers:
36
        - image: "quay.io/external_storage/local-volume-provisioner:v2.5.0"
37
          name: provisioner
38
          securityContext:
39
            privileged: true
40
          env:
41
            - name: MY_NODE_NAME
42
              valueFrom:
43
                fieldRef:
44
                  fieldPath: spec.nodeName
45
            - name: MY_NAMESPACE
46
              valueFrom:
47
                fieldRef:
48
                  fieldPath: metadata.namespace
49
            - name: JOB_CONTAINER_IMAGE
50
              value: "quay.io/external_storage/local-volume-provisioner:v2.5.0"
51
          volumeMounts:
52
            - mountPath: /etc/provisioner/config
53
              name: provisioner-config
54
              readOnly: true
55
            - mountPath: /mnt/fast-ssd
56
              name: fast-ssd
57
              mountPropagation: "HostToContainer"
58
            - mountPath: /mnt/bulk-storage
59
              name: bulk-storage
60
              mountPropagation: "HostToContainer"
61
      volumes:
62
        - name: provisioner-config
63
          configMap:
64
            name: local-provisioner-config
65
        - name: fast-ssd
66
          hostPath:
67
            path: /mnt/fast-ssd
68
        - name: bulk-storage
69
          hostPath:
70
            path: /mnt/bulk-storage
71
      nodeSelector:
72
        kubernetes.io/os: linux
73
---
74
apiVersion: v1
75
kind: ServiceAccount
76
metadata:
77
  name: local-storage-admin
78
  namespace: kube-system
79
---
80
apiVersion: rbac.authorization.k8s.io/v1
81
kind: ClusterRoleBinding
82
metadata:
83
  name: local-storage-provisioner-pv-binding
84
roleRef:
85
  apiGroup: rbac.authorization.k8s.io
86
  kind: ClusterRole
87
  name: system:persistent-volume-provisioner
88
subjects:
89
  - kind: ServiceAccount
90
    name: local-storage-admin
91
    namespace: kube-system
92
---
93
apiVersion: rbac.authorization.k8s.io/v1
94
kind: ClusterRole
95
metadata:
96
  name: local-storage-provisioner-node-clusterrole
97
rules:
98
  - apiGroups: [""]
99
    resources: ["nodes"]
100
    verbs: ["get"]
101
---
102
apiVersion: rbac.authorization.k8s.io/v1
103
kind: ClusterRoleBinding
104
metadata:
105
  name: local-storage-provisioner-node-binding
106
roleRef:
107
  apiGroup: rbac.authorization.k8s.io
108
  kind: ClusterRole
109
  name: local-storage-provisioner-node-clusterrole
110
subjects:
111
  - kind: ServiceAccount
112
    name: local-storage-admin
113
    namespace: kube-system
114
---
115
apiVersion: v1
116
kind: ConfigMap
117
metadata:
118
  name: local-provisioner-config
119
  namespace: kube-system
120
data:
121
  storageClassMap: |
122
    fast-ssd:
123
       hostDir: /mnt/fast-ssd
124
       mountDir: /mnt/fast-ssd
125
       blockCleanerCommand:
126
         - "/scripts/shred.sh"
127
         - "2"
128
       volumeMode: Filesystem
129
       fsType: ext4
130
    bulk-storage:
131
       hostDir: /mnt/bulk-storage
132
       mountDir: /mnt/bulk-storage
133
       blockCleanerCommand:
134
         - "/scripts/shred.sh"
135
         - "2"
136
       volumeMode: Filesystem
137
       fsType: ext4

Backup Storage Configuration#

1
#!/bin/bash
2
# setup-backup-storage.sh - Configure backup storage
3

4
set -euo pipefail
5

6
# Create backup storage directories
7
create_backup_directories() {
8
    echo "Creating backup storage directories..."
9

10
    # Local backup storage
11
    mkdir -p /opt/kubernetes/backups/{etcd,configs,applications}
12
    chmod 750 /opt/kubernetes/backups
13

14
    # NFS backup mount (if using NFS)
15
    mkdir -p /mnt/nfs-backup
16

17
    # Add to fstab for persistent mounting
18
    # echo "nfs-server:/backup/kubernetes /mnt/nfs-backup nfs defaults 0 0" >> /etc/fstab
19
}
20

21
# Install and configure backup tools
22
install_backup_tools() {
23
    echo "Installing backup tools..."
24

25
    # Install restic for application backups
26
    RESTIC_VERSION="0.16.0"
27
    wget -O /tmp/restic.bz2 "https://github.com/restic/restic/releases/download/v${RESTIC_VERSION}/restic_${RESTIC_VERSION}_linux_amd64.bz2"
28
    bunzip2 /tmp/restic.bz2
29
    chmod +x /tmp/restic
30
    mv /tmp/restic /usr/local/bin/
31

32
    # Install velero for Kubernetes-native backups
33
    VELERO_VERSION="1.11.1"
34
    wget -O /tmp/velero.tar.gz "https://github.com/vmware-tanzu/velero/releases/download/v${VELERO_VERSION}/velero-v${VELERO_VERSION}-linux-amd64.tar.gz"
35
    tar -xzf /tmp/velero.tar.gz -C /tmp/
36
    mv /tmp/velero-v${VELERO_VERSION}-linux-amd64/velero /usr/local/bin/
37
    chmod +x /usr/local/bin/velero
38
}
39

40
# Configure Velero for cluster backups
41
setup_velero() {
42
    echo "Setting up Velero for cluster backups..."
43

44
    # Create Velero namespace and configuration
45
    kubectl create namespace velero || true
46

47
    # Configure backup storage location (example with MinIO)
48
    cat > /tmp/velero-config.yaml << 'EOF'
49
apiVersion: velero.io/v1
50
kind: BackupStorageLocation
51
metadata:
52
  name: default
53
  namespace: velero
54
spec:
55
  provider: aws
56
  objectStorage:
57
    bucket: kubernetes-backups
58
    prefix: velero
59
  config:
60
    region: us-east-1
61
    s3ForcePathStyle: "true"
62
    s3Url: http://minio.backup.svc.cluster.local:9000
63
---
64
apiVersion: velero.io/v1
65
kind: VolumeSnapshotLocation
66
metadata:
67
  name: default
68
  namespace: velero
69
spec:
70
  provider: aws
71
  config:
72
    region: us-east-1
73
EOF
74

75
    kubectl apply -f /tmp/velero-config.yaml
76
}
77

78
create_backup_directories
79
install_backup_tools
80
setup_velero
81

82
echo "Backup storage configuration completed"

Security Hardening#

Network Policies Implementation#

1
# Default deny all traffic
2
apiVersion: networking.k8s.io/v1
3
kind: NetworkPolicy
4
metadata:
5
  name: default-deny-all
6
  namespace: default
7
spec:
8
  podSelector: {}
9
  policyTypes:
10
    - Ingress
11
    - Egress
12
---
13
# Allow DNS traffic
14
apiVersion: networking.k8s.io/v1
15
kind: NetworkPolicy
16
metadata:
17
  name: allow-dns
18
  namespace: default
19
spec:
20
  podSelector: {}
21
  policyTypes:
22
    - Egress
23
  egress:
24
    - to: []
25
      ports:
26
        - protocol: UDP
27
          port: 53
28
        - protocol: TCP
29
          port: 53
30
---
31
# Allow traffic to kube-system
32
apiVersion: networking.k8s.io/v1
33
kind: NetworkPolicy
34
metadata:
35
  name: allow-kube-system
36
  namespace: default
37
spec:
38
  podSelector: {}
39
  policyTypes:
40
    - Egress
41
  egress:
42
    - to:
43
        - namespaceSelector:
44
            matchLabels:
45
              name: kube-system
46
---
47
# Kube-system network policy
48
apiVersion: networking.k8s.io/v1
49
kind: NetworkPolicy
50
metadata:
51
  name: kube-system-default-deny
52
  namespace: kube-system
53
spec:
54
  podSelector: {}
55
  policyTypes:
56
    - Ingress
57
    - Egress
58
  egress:
59
    - {} # Allow all egress for system components
60
  ingress:
61
    - from:
62
        - namespaceSelector: {}
63
      ports:
64
        - protocol: TCP
65
          port: 53
66
        - protocol: UDP
67
          port: 53
68
    - from:
69
        - namespaceSelector: {}
70
        - podSelector: {}
71
---
72
# Production namespace network policy
73
apiVersion: v1
74
kind: Namespace
75
metadata:
76
  name: production
77
  labels:
78
    name: production
79
    environment: production
80
---
81
apiVersion: networking.k8s.io/v1
82
kind: NetworkPolicy
83
metadata:
84
  name: production-network-policy
85
  namespace: production
86
spec:
87
  podSelector: {}
88
  policyTypes:
89
    - Ingress
90
    - Egress
91
  egress:
92
    - to: []
93
      ports:
94
        - protocol: UDP
95
          port: 53
96
        - protocol: TCP
97
          port: 53
98
    - to:
99
        - namespaceSelector:
100
            matchLabels:
101
              name: kube-system
102
    - to:
103
        - namespaceSelector:
104
            matchLabels:
105
              name: production
106
  ingress:
107
    - from:
108
        - namespaceSelector:
109
            matchLabels:
110
              name: production
111
    - from:
112
        - namespaceSelector:
113
            matchLabels:
114
              name: ingress-nginx
115
      ports:
116
        - protocol: TCP
117
          port: 8080

Pod Security Standards#

1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: secure-workloads
5
  labels:
6
    pod-security.kubernetes.io/enforce: restricted
7
    pod-security.kubernetes.io/audit: restricted
8
    pod-security.kubernetes.io/warn: restricted
9
---
10
# Resource quotas and limits
11
apiVersion: v1
12
kind: ResourceQuota
13
metadata:
14
  name: secure-workloads-quota
15
  namespace: secure-workloads
16
spec:
17
  hard:
18
    requests.cpu: "10"
19
    requests.memory: 20Gi
20
    limits.cpu: "20"
21
    limits.memory: 40Gi
22
    pods: "50"
23
    persistentvolumeclaims: "10"
24
    services: "10"
25
    secrets: "20"
26
    configmaps: "20"
27
---
28
apiVersion: v1
29
kind: LimitRange
30
metadata:
31
  name: secure-workloads-limits
32
  namespace: secure-workloads
33
spec:
34
  limits:
35
    - default:
36
        cpu: 500m
37
        memory: 512Mi
38
      defaultRequest:
39
        cpu: 100m
40
        memory: 128Mi
41
      type: Container
42
    - max:
43
        cpu: 2
44
        memory: 4Gi
45
      min:
46
        cpu: 50m
47
        memory: 64Mi
48
      type: Container
49
---
50
# Security policies
51
apiVersion: v1
52
kind: ServiceAccount
53
metadata:
54
  name: restricted-service-account
55
  namespace: secure-workloads
56
---
57
apiVersion: rbac.authorization.k8s.io/v1
58
kind: Role
59
metadata:
60
  name: restricted-role
61
  namespace: secure-workloads
62
rules:
63
  - apiGroups: [""]
64
    resources: ["pods", "configmaps", "secrets"]
65
    verbs: ["get", "list", "watch"]
66
  - apiGroups: ["apps"]
67
    resources: ["deployments", "replicasets"]
68
    verbs: ["get", "list", "watch"]
69
---
70
apiVersion: rbac.authorization.k8s.io/v1
71
kind: RoleBinding
72
metadata:
73
  name: restricted-binding
74
  namespace: secure-workloads
75
subjects:
76
  - kind: ServiceAccount
77
    name: restricted-service-account
78
    namespace: secure-workloads
79
roleRef:
80
  kind: Role
81
  name: restricted-role
82
  apiGroup: rbac.authorization.k8s.io

Security Scanning and Monitoring#

1
#!/bin/bash
2
# security-scanning.sh - Implement security scanning
3

4
set -euo pipefail
5

6
# Install Falco for runtime security
7
install_falco() {
8
    echo "Installing Falco for runtime security monitoring..."
9

10
    # Add Falco repository
11
    curl -s https://falco.org/repo/falcosecurity-packages.asc | apt-key add -
12
    echo "deb https://download.falco.org/packages/deb stable main" | tee -a /etc/apt/sources.list.d/falcosecurity.list
13
    apt-get update -qq
14
    apt-get install -y falco
15

16
    # Configure Falco
17
    cat > /etc/falco/falco_rules.local.yaml << 'EOF'
18
- rule: Kubernetes Client Tool Launched in Container
19
  desc: Detect kubernetes client tool launched in container
20
  condition: >
21
    spawned_process and container and
22
    (proc.name in (kubectl, oc))
23
  output: >
24
    Kubernetes client tool launched in container (user=%user.name container_id=%container.id
25
    image=%container.image.repository proc=%proc.cmdline)
26
  priority: NOTICE
27
  tags:
28
  - process
29
  - mitre_execution
30

31
- rule: Suspicious Network Activity in Container
32
  desc: Detect suspicious network activity in containers
33
  condition: >
34
    spawned_process and container and
35
    proc.name in (nc, ncat, netcat, socat, ss, netstat)
36
  output: >
37
    Suspicious network tool launched in container (user=%user.name container_id=%container.id
38
    image=%container.image.repository proc=%proc.cmdline)
39
  priority: WARNING
40
  tags:
41
  - network
42
  - mitre_discovery
43
EOF
44

45
    systemctl enable falco
46
    systemctl start falco
47
}
48

49
# Install Trivy for vulnerability scanning
50
install_trivy() {
51
    echo "Installing Trivy for vulnerability scanning..."
52

53
    # Install Trivy
54
    curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
55

56
    # Create scan script
57
    cat > /usr/local/bin/scan-cluster-images.sh << 'EOF'
58
#!/bin/bash
59
# Scan all images in the cluster for vulnerabilities
60

61
NAMESPACE="${1:-default}"
62
OUTPUT_DIR="/opt/kubernetes/security-scans/$(date +%Y%m%d_%H%M%S)"
63

64
mkdir -p "$OUTPUT_DIR"
65

66
echo "Scanning images in namespace: $NAMESPACE"
67

68
# Get all images in the namespace
69
kubectl get pods -n "$NAMESPACE" -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u > "$OUTPUT_DIR/images.txt"
70

71
# Scan each image
72
while read -r image; do
73
    echo "Scanning $image..."
74
    trivy image --severity HIGH,CRITICAL --format json "$image" > "$OUTPUT_DIR/$(echo $image | tr '/' '_' | tr ':' '_').json"
75
done < "$OUTPUT_DIR/images.txt"
76

77
echo "Scan results saved to: $OUTPUT_DIR"
78
EOF
79

80
    chmod +x /usr/local/bin/scan-cluster-images.sh
81
}
82

83
# Install kube-bench for CIS compliance
84
install_kube_bench() {
85
    echo "Installing kube-bench for CIS compliance checking..."
86

87
    # Download and install kube-bench
88
    KUBE_BENCH_VERSION="0.6.15"
89
    wget -O /tmp/kube-bench.tar.gz "https://github.com/aquasecurity/kube-bench/releases/download/v${KUBE_BENCH_VERSION}/kube-bench_${KUBE_BENCH_VERSION}_linux_amd64.tar.gz"
90
    tar -xzf /tmp/kube-bench.tar.gz -C /tmp/
91
    mv /tmp/kube-bench /usr/local/bin/
92
    chmod +x /usr/local/bin/kube-bench
93

94
    # Create compliance check script
95
    cat > /usr/local/bin/compliance-check.sh << 'EOF'
96
#!/bin/bash
97
# Run CIS compliance checks
98

99
REPORT_DIR="/opt/kubernetes/compliance-reports/$(date +%Y%m%d_%H%M%S)"
100
mkdir -p "$REPORT_DIR"
101

102
echo "Running CIS compliance checks..."
103

104
# Run kube-bench
105
kube-bench --json > "$REPORT_DIR/cis-compliance.json"
106
kube-bench > "$REPORT_DIR/cis-compliance.txt"
107

108
# Generate summary
109
jq '.Totals' "$REPORT_DIR/cis-compliance.json" > "$REPORT_DIR/summary.json"
110

111
echo "Compliance report saved to: $REPORT_DIR"
112
echo "Summary:"
113
cat "$REPORT_DIR/summary.json"
114
EOF
115

116
    chmod +x /usr/local/bin/compliance-check.sh
117
}
118

119
# Setup security monitoring
120
setup_security_monitoring() {
121
    echo "Setting up security monitoring..."
122

123
    # Create security monitoring namespace
124
    kubectl create namespace security-monitoring || true
125

126
    # Deploy security monitoring stack
127
    cat > /tmp/security-monitoring.yaml << 'EOF'
128
apiVersion: apps/v1
129
kind: DaemonSet
130
metadata:
131
  name: security-monitor
132
  namespace: security-monitoring
133
spec:
134
  selector:
135
    matchLabels:
136
      name: security-monitor
137
  template:
138
    metadata:
139
      labels:
140
        name: security-monitor
141
    spec:
142
      hostPID: true
143
      hostNetwork: true
144
      serviceAccountName: security-monitor
145
      containers:
146
      - name: security-monitor
147
        image: alpine:latest
148
        command: ["/bin/sh"]
149
        args: ["-c", "while true; do sleep 3600; done"]
150
        securityContext:
151
          privileged: true
152
        volumeMounts:
153
        - name: proc
154
          mountPath: /host/proc
155
          readOnly: true
156
        - name: sys
157
          mountPath: /host/sys
158
          readOnly: true
159
        - name: var-run
160
          mountPath: /host/var/run
161
          readOnly: true
162
      volumes:
163
      - name: proc
164
        hostPath:
165
          path: /proc
166
      - name: sys
167
        hostPath:
168
          path: /sys
169
      - name: var-run
170
        hostPath:
171
          path: /var/run
172
      tolerations:
173
      - effect: NoSchedule
174
        operator: Exists
175
---
176
apiVersion: v1
177
kind: ServiceAccount
178
metadata:
179
  name: security-monitor
180
  namespace: security-monitoring
181
---
182
apiVersion: rbac.authorization.k8s.io/v1
183
kind: ClusterRole
184
metadata:
185
  name: security-monitor
186
rules:
187
- apiGroups: [""]
188
  resources: ["nodes", "pods", "namespaces"]
189
  verbs: ["get", "list", "watch"]
190
- apiGroups: ["apps"]
191
  resources: ["deployments", "daemonsets", "replicasets"]
192
  verbs: ["get", "list", "watch"]
193
---
194
apiVersion: rbac.authorization.k8s.io/v1
195
kind: ClusterRoleBinding
196
metadata:
197
  name: security-monitor
198
roleRef:
199
  apiGroup: rbac.authorization.k8s.io
200
  kind: ClusterRole
201
  name: security-monitor
202
subjects:
203
- kind: ServiceAccount
204
  name: security-monitor
205
  namespace: security-monitoring
206
EOF
207

208
    kubectl apply -f /tmp/security-monitoring.yaml
209
}
210

211
# Main execution
212
main() {
213
    echo "Setting up security scanning and monitoring..."
214

215
    install_falco
216
    install_trivy
217
    install_kube_bench
218
    setup_security_monitoring
219

220
    echo "Security setup completed successfully"
221
    echo "Run compliance check: /usr/local/bin/compliance-check.sh"
222
    echo "Scan cluster images: /usr/local/bin/scan-cluster-images.sh"
223
}
224

225
main "$@"

Monitoring and Observability#

Prometheus and Grafana Stack#

1
apiVersion: v1
2
kind: Namespace
3
metadata:
4
  name: monitoring
5
---
6
# Prometheus ConfigMap
7
apiVersion: v1
8
kind: ConfigMap
9
metadata:
10
  name: prometheus-config
11
  namespace: monitoring
12
data:
13
  prometheus.yml: |
14
    global:
15
      scrape_interval: 15s
16
      evaluation_interval: 15s
17
      external_labels:
18
        cluster: 'production-cluster'
19
        region: 'us-east-1'
20

21
    rule_files:
22
      - "/etc/prometheus/rules/*.yml"
23

24
    scrape_configs:
25
      - job_name: 'prometheus'
26
        static_configs:
27
          - targets: ['localhost:9090']
28

29
      - job_name: 'kubernetes-apiservers'
30
        kubernetes_sd_configs:
31
        - role: endpoints
32
        scheme: https
33
        tls_config:
34
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
35
          insecure_skip_verify: true
36
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
37
        relabel_configs:
38
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
39
          action: keep
40
          regex: default;kubernetes;https
41

42
      - job_name: 'kubernetes-nodes'
43
        kubernetes_sd_configs:
44
        - role: node
45
        scheme: https
46
        tls_config:
47
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
48
          insecure_skip_verify: true
49
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
50
        relabel_configs:
51
        - action: labelmap
52
          regex: __meta_kubernetes_node_label_(.+)
53
        - target_label: __address__
54
          replacement: kubernetes.default.svc:443
55
        - source_labels: [__meta_kubernetes_node_name]
56
          regex: (.+)
57
          target_label: __metrics_path__
58
          replacement: /api/v1/nodes/${1}/proxy/metrics
59

60
      - job_name: 'kubernetes-cadvisor'
61
        kubernetes_sd_configs:
62
        - role: node
63
        scheme: https
64
        tls_config:
65
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
66
          insecure_skip_verify: true
67
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
68
        relabel_configs:
69
        - action: labelmap
70
          regex: __meta_kubernetes_node_label_(.+)
71
        - target_label: __address__
72
          replacement: kubernetes.default.svc:443
73
        - source_labels: [__meta_kubernetes_node_name]
74
          regex: (.+)
75
          target_label: __metrics_path__
76
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
77

78
      - job_name: 'kubernetes-service-endpoints'
79
        kubernetes_sd_configs:
80
        - role: endpoints
81
        relabel_configs:
82
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
83
          action: keep
84
          regex: true
85
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
86
          action: replace
87
          target_label: __metrics_path__
88
          regex: (.+)
89
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
90
          action: replace
91
          regex: ([^:]+)(?::\d+)?;(\d+)
92
          replacement: $1:$2
93
          target_label: __address__
94
        - action: labelmap
95
          regex: __meta_kubernetes_service_label_(.+)
96
        - source_labels: [__meta_kubernetes_namespace]
97
          action: replace
98
          target_label: kubernetes_namespace
99
        - source_labels: [__meta_kubernetes_service_name]
100
          action: replace
101
          target_label: kubernetes_name
102

103
      - job_name: 'kubernetes-pods'
104
        kubernetes_sd_configs:
105
        - role: pod
106
        relabel_configs:
107
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
108
          action: keep
109
          regex: true
110
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
111
          action: replace
112
          target_label: __metrics_path__
113
          regex: (.+)
114
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
115
          action: replace
116
          regex: ([^:]+)(?::\d+)?;(\d+)
117
          replacement: $1:$2
118
          target_label: __address__
119
        - action: labelmap
120
          regex: __meta_kubernetes_pod_label_(.+)
121
        - source_labels: [__meta_kubernetes_namespace]
122
          action: replace
123
          target_label: kubernetes_namespace
124
        - source_labels: [__meta_kubernetes_pod_name]
125
          action: replace
126
          target_label: kubernetes_pod_name
127

128
  alert_rules.yml: |
129
    groups:
130
    - name: kubernetes-cluster
131
      rules:
132
      - alert: KubernetesNodeReady
133
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
134
        for: 10m
135
        labels:
136
          severity: critical
137
        annotations:
138
          summary: Kubernetes Node ready (instance {{ $labels.instance }})
139
          description: "Node {{ $labels.node }} has been unready for a long time"
140

141
      - alert: KubernetesMemoryPressure
142
        expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
143
        for: 2m
144
        labels:
145
          severity: critical
146
        annotations:
147
          summary: Kubernetes memory pressure (instance {{ $labels.instance }})
148
          description: "Node {{ $labels.node }} has MemoryPressure condition"
149

150
      - alert: KubernetesDiskPressure
151
        expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
152
        for: 2m
153
        labels:
154
          severity: critical
155
        annotations:
156
          summary: Kubernetes disk pressure (instance {{ $labels.instance }})
157
          description: "Node {{ $labels.node }} has DiskPressure condition"
158

159
      - alert: KubernetesPodCrashLooping
160
        expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
161
        for: 2m
162
        labels:
163
          severity: warning
164
        annotations:
165
          summary: Kubernetes pod crash looping (instance {{ $labels.instance }})
166
          description: "Pod {{ $labels.pod }} is crash looping"
167

168
      - alert: KubernetesPersistentvolumeclaimPending
169
        expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
170
        for: 2m
171
        labels:
172
          severity: warning
173
        annotations:
174
          summary: Kubernetes PersistentVolumeClaim pending (instance {{ $labels.instance }})
175
          description: "PersistentVolumeClaim {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} is pending"
176
---
177
# Prometheus Deployment
178
apiVersion: apps/v1
179
kind: Deployment
180
metadata:
181
  name: prometheus
182
  namespace: monitoring
183
spec:
184
  replicas: 1
185
  selector:
186
    matchLabels:
187
      app: prometheus
188
  template:
189
    metadata:
190
      labels:
191
        app: prometheus
192
    spec:
193
      serviceAccountName: prometheus
194
      containers:
195
        - name: prometheus
196
          image: prom/prometheus:v2.45.0
197
          args:
198
            - "--config.file=/etc/prometheus/prometheus.yml"
199
            - "--storage.tsdb.path=/prometheus/"
200
            - "--web.console.libraries=/etc/prometheus/console_libraries"
201
            - "--web.console.templates=/etc/prometheus/consoles"
202
            - "--storage.tsdb.retention.time=30d"
203
            - "--web.enable-lifecycle"
204
            - "--web.enable-admin-api"
205
          ports:
206
            - containerPort: 9090
207
          resources:
208
            requests:
209
              cpu: 500m
210
              memory: 1Gi
211
            limits:
212
              cpu: 2
213
              memory: 4Gi
214
          volumeMounts:
215
            - name: prometheus-config
216
              mountPath: /etc/prometheus/
217
            - name: prometheus-storage
218
              mountPath: /prometheus/
219
      volumes:
220
        - name: prometheus-config
221
          configMap:
222
            name: prometheus-config
223
        - name: prometheus-storage
224
          persistentVolumeClaim:
225
            claimName: prometheus-storage
226
---
227
# Prometheus PVC
228
apiVersion: v1
229
kind: PersistentVolumeClaim
230
metadata:
231
  name: prometheus-storage
232
  namespace: monitoring
233
spec:
234
  accessModes:
235
    - ReadWriteOnce
236
  storageClassName: fast-ssd
237
  resources:
238
    requests:
239
      storage: 50Gi
240
---
241
# Prometheus Service
242
apiVersion: v1
243
kind: Service
244
metadata:
245
  name: prometheus
246
  namespace: monitoring
247
spec:
248
  type: ClusterIP
249
  ports:
250
    - port: 9090
251
      targetPort: 9090
252
  selector:
253
    app: prometheus
254
---
255
# Prometheus ServiceAccount and RBAC
256
apiVersion: v1
257
kind: ServiceAccount
258
metadata:
259
  name: prometheus
260
  namespace: monitoring
261
---
262
apiVersion: rbac.authorization.k8s.io/v1
263
kind: ClusterRole
264
metadata:
265
  name: prometheus
266
rules:
267
  - apiGroups: [""]
268
    resources:
269
      - nodes
270
      - nodes/proxy
271
      - services
272
      - endpoints
273
      - pods
274
    verbs: ["get", "list", "watch"]
275
  - apiGroups:
276
      - extensions
277
    resources:
278
      - ingresses
279
    verbs: ["get", "list", "watch"]
280
  - nonResourceURLs: ["/metrics"]
281
    verbs: ["get"]
282
---
283
apiVersion: rbac.authorization.k8s.io/v1
284
kind: ClusterRoleBinding
285
metadata:
286
  name: prometheus
287
roleRef:
288
  apiGroup: rbac.authorization.k8s.io
289
  kind: ClusterRole
290
  name: prometheus
291
subjects:
292
  - kind: ServiceAccount
293
    name: prometheus
294
    namespace: monitoring

Node Exporter and Kube-State-Metrics#

1
# Node Exporter DaemonSet
2
apiVersion: apps/v1
3
kind: DaemonSet
4
metadata:
5
  name: node-exporter
6
  namespace: monitoring
7
spec:
8
  selector:
9
    matchLabels:
10
      name: node-exporter
11
  template:
12
    metadata:
13
      labels:
14
        name: node-exporter
15
      annotations:
16
        prometheus.io/scrape: "true"
17
        prometheus.io/port: "9100"
18
    spec:
19
      hostPID: true
20
      hostIPC: true
21
      hostNetwork: true
22
      containers:
23
        - name: node-exporter
24
          image: prom/node-exporter:v1.6.0
25
          ports:
26
            - containerPort: 9100
27
          args:
28
            - "--path.sysfs=/host/sys"
29
            - "--path.rootfs=/host/root"
30
            - "--path.procfs=/host/proc"
31
            - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
32
            - "--collector.systemd"
33
            - "--collector.processes"
34
          resources:
35
            requests:
36
              memory: 30Mi
37
              cpu: 100m
38
            limits:
39
              memory: 50Mi
40
              cpu: 200m
41
          volumeMounts:
42
            - name: dev
43
              mountPath: /host/dev
44
            - name: proc
45
              mountPath: /host/proc
46
            - name: sys
47
              mountPath: /host/sys
48
            - name: rootfs
49
              mountPath: /host/root
50
              readOnly: true
51
      tolerations:
52
        - operator: Exists
53
      volumes:
54
        - name: proc
55
          hostPath:
56
            path: /proc
57
        - name: dev
58
          hostPath:
59
            path: /dev
60
        - name: sys
61
          hostPath:
62
            path: /sys
63
        - name: rootfs
64
          hostPath:
65
            path: /
66
---
67
# Kube State Metrics
68
apiVersion: apps/v1
69
kind: Deployment
70
metadata:
71
  name: kube-state-metrics
72
  namespace: monitoring
73
spec:
74
  replicas: 1
75
  selector:
76
    matchLabels:
77
      app: kube-state-metrics
78
  template:
79
    metadata:
80
      labels:
81
        app: kube-state-metrics
82
      annotations:
83
        prometheus.io/scrape: "true"
84
        prometheus.io/port: "8080"
85
    spec:
86
      serviceAccountName: kube-state-metrics
87
      containers:
88
        - name: kube-state-metrics
89
          image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.9.2
90
          ports:
91
            - containerPort: 8080
92
              name: http-metrics
93
            - containerPort: 8081
94
              name: telemetry
95
          livenessProbe:
96
            httpGet:
97
              path: /healthz
98
              port: 8080
99
            initialDelaySeconds: 5
100
            timeoutSeconds: 5
101
          readinessProbe:
102
            httpGet:
103
              path: /
104
              port: 8081
105
            initialDelaySeconds: 5
106
            timeoutSeconds: 5
107
          resources:
108
            requests:
109
              memory: 100Mi
110
              cpu: 100m
111
            limits:
112
              memory: 200Mi
113
              cpu: 200m
114
---
115
apiVersion: v1
116
kind: ServiceAccount
117
metadata:
118
  name: kube-state-metrics
119
  namespace: monitoring
120
---
121
apiVersion: rbac.authorization.k8s.io/v1
122
kind: ClusterRole
123
metadata:
124
  name: kube-state-metrics
125
rules:
126
  - apiGroups: [""]
127
    resources:
128
      - configmaps
129
      - secrets
130
      - nodes
131
      - pods
132
      - services
133
      - resourcequotas
134
      - replicationcontrollers
135
      - limitranges
136
      - persistentvolumeclaims
137
      - persistentvolumes
138
      - namespaces
139
      - endpoints
140
    verbs: ["list", "watch"]
141
  - apiGroups: ["apps"]
142
    resources:
143
      - statefulsets
144
      - daemonsets
145
      - deployments
146
      - replicasets
147
    verbs: ["list", "watch"]
148
  - apiGroups: ["batch"]
149
    resources:
150
      - cronjobs
151
      - jobs
152
    verbs: ["list", "watch"]
153
  - apiGroups: ["autoscaling"]
154
    resources:
155
      - horizontalpodautoscalers
156
    verbs: ["list", "watch"]
157
  - apiGroups: ["authentication.k8s.io"]
158
    resources:
159
      - tokenreviews
160
    verbs: ["create"]
161
  - apiGroups: ["authorization.k8s.io"]
162
    resources:
163
      - subjectaccessreviews
164
    verbs: ["create"]
165
  - apiGroups: ["policy"]
166
    resources:
167
      - poddisruptionbudgets
168
    verbs: ["list", "watch"]
169
  - apiGroups: ["certificates.k8s.io"]
170
    resources:
171
      - certificatesigningrequests
172
    verbs: ["list", "watch"]
173
  - apiGroups: ["storage.k8s.io"]
174
    resources:
175
      - storageclasses
176
      - volumeattachments
177
    verbs: ["list", "watch"]
178
  - apiGroups: ["admissionregistration.k8s.io"]
179
    resources:
180
      - mutatingwebhookconfigurations
181
      - validatingwebhookconfigurations
182
    verbs: ["list", "watch"]
183
  - apiGroups: ["networking.k8s.io"]
184
    resources:
185
      - networkpolicies
186
      - ingresses
187
    verbs: ["list", "watch"]
188
  - apiGroups: ["coordination.k8s.io"]
189
    resources:
190
      - leases
191
    verbs: ["list", "watch"]
192
---
193
apiVersion: rbac.authorization.k8s.io/v1
194
kind: ClusterRoleBinding
195
metadata:
196
  name: kube-state-metrics
197
roleRef:
198
  apiGroup: rbac.authorization.k8s.io
199
  kind: ClusterRole
200
  name: kube-state-metrics
201
subjects:
202
  - kind: ServiceAccount
203
    name: kube-state-metrics
204
    namespace: monitoring
205
---
206
apiVersion: v1
207
kind: Service
208
metadata:
209
  name: kube-state-metrics
210
  namespace: monitoring
211
spec:
212
  ports:
213
    - name: http-metrics
214
      port: 8080
215
      targetPort: http-metrics
216
    - name: telemetry
217
      port: 8081
218
      targetPort: telemetry
219
  selector:
220
    app: kube-state-metrics

Operational Procedures#

Automated Cluster Operations#

1
#!/bin/bash
2
# cluster-operations.sh - Automated cluster management
3

4
set -euo pipefail
5

6
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
7
LOG_FILE="/var/log/cluster-operations.log"
8

9
log() {
10
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
11
}
12

13
# Health check function
14
health_check() {
15
    log "Performing cluster health check..."
16

17
    # Check node status
18
    nodes_ready=$(kubectl get nodes --no-headers | grep -c "Ready")
19
    nodes_total=$(kubectl get nodes --no-headers | wc -l)
20

21
    log "Nodes: $nodes_ready/$nodes_total Ready"
22

23
    if [ "$nodes_ready" -lt "$nodes_total" ]; then
24
        log "WARNING: Not all nodes are ready"
25
        kubectl get nodes
26
    fi
27

28
    # Check system pods
29
    system_pods_not_ready=$(kubectl get pods -n kube-system --no-headers | grep -v "Running\|Completed" | wc -l)
30

31
    if [ "$system_pods_not_ready" -gt 0 ]; then
32
        log "WARNING: System pods not ready: $system_pods_not_ready"
33
        kubectl get pods -n kube-system --field-selector=status.phase!=Running,status.phase!=Succeeded
34
    fi
35

36
    # Check API server health
37
    if ! kubectl cluster-info > /dev/null 2>&1; then
38
        log "ERROR: API server not accessible"
39
        return 1
40
    fi
41

42
    # Check etcd health
43
    etcd_endpoints=$(kubectl get endpoints -n kube-system etcd -o jsonpath='{.subsets[0].addresses[*].ip}' | tr ' ' ',')
44
    if [ -n "$etcd_endpoints" ]; then
45
        for endpoint in $(echo "$etcd_endpoints" | tr ',' ' '); do
46
            if ! ETCDCTL_API=3 etcdctl endpoint health --endpoints="$endpoint:2379" --insecure-skip-tls-verify > /dev/null 2>&1; then
47
                log "WARNING: etcd endpoint $endpoint not healthy"
48
            fi
49
        done
50
    fi
51

52
    log "Health check completed"
53
}
54

55
# Backup function
56
backup_cluster() {
57
    log "Starting cluster backup..."
58

59
    BACKUP_DIR="/opt/kubernetes/backups/$(date +%Y%m%d_%H%M%S)"
60
    mkdir -p "$BACKUP_DIR"
61

62
    # Backup etcd
63
    log "Backing up etcd..."
64
    ETCDCTL_API=3 etcdctl snapshot save "$BACKUP_DIR/etcd-snapshot.db" \
65
        --endpoints=https://127.0.0.1:2379 \
66
        --cacert=/etc/kubernetes/pki/etcd/ca.crt \
67
        --cert=/etc/kubernetes/pki/etcd/server.crt \
68
        --key=/etc/kubernetes/pki/etcd/server.key
69

70
    # Backup cluster resources
71
    log "Backing up cluster resources..."
72
    kubectl get all --all-namespaces -o yaml > "$BACKUP_DIR/all-resources.yaml"
73
    kubectl get persistentvolumes -o yaml > "$BACKUP_DIR/persistent-volumes.yaml"
74
    kubectl get persistentvolumeclaims --all-namespaces -o yaml > "$BACKUP_DIR/persistent-volume-claims.yaml"
75
    kubectl get configmaps --all-namespaces -o yaml > "$BACKUP_DIR/configmaps.yaml"
76
    kubectl get secrets --all-namespaces -o yaml > "$BACKUP_DIR/secrets.yaml"
77

78
    # Backup certificates
79
    log "Backing up certificates..."
80
    cp -r /etc/kubernetes/pki "$BACKUP_DIR/"
81

82
    # Backup configuration
83
    log "Backing up configuration..."
84
    cp /etc/kubernetes/*.conf "$BACKUP_DIR/" 2>/dev/null || true
85

86
    # Create backup manifest
87
    cat > "$BACKUP_DIR/manifest.json" << EOF
88
{
89
    "backup_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
90
    "cluster_name": "$(kubectl config current-context)",
91
    "kubernetes_version": "$(kubectl version --short --client | grep 'Client Version')",
92
    "node_count": $(kubectl get nodes --no-headers | wc -l),
93
    "namespace_count": $(kubectl get namespaces --no-headers | wc -l),
94
    "backup_size": "$(du -sh $BACKUP_DIR | cut -f1)"
95
}
96
EOF
97

98
    # Compress backup
99
    tar -czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
100
    rm -rf "$BACKUP_DIR"
101

102
    log "Backup completed: $BACKUP_DIR.tar.gz"
103
}
104

105
# Cleanup function
106
cleanup_cluster() {
107
    log "Starting cluster cleanup..."
108

109
    # Clean up completed pods
110
    kubectl delete pods --all-namespaces --field-selector=status.phase=Succeeded --ignore-not-found=true
111
    kubectl delete pods --all-namespaces --field-selector=status.phase=Failed --ignore-not-found=true
112

113
    # Clean up orphaned resources
114
    kubectl get events --all-namespaces --sort-by='.lastTimestamp' | head -n -1000 | awk '{print $1" "$2}' | xargs -r kubectl delete events -n
115

116
    # Clean up old replicasets
117
    kubectl get replicasets --all-namespaces -o json | jq -r '.items[] | select(.spec.replicas==0) | "\(.metadata.namespace) \(.metadata.name)"' | xargs -r -n2 sh -c 'kubectl delete replicaset -n $0 $1'
118

119
    # Clean up old backups (keep last 7 days)
120
    find /opt/kubernetes/backups -name "*.tar.gz" -mtime +7 -delete
121

122
    log "Cleanup completed"
123
}
124

125
# Update function
126
update_cluster() {
127
    local target_version="$1"
128

129
    if [ -z "$target_version" ]; then
130
        log "ERROR: Target version not specified"
131
        return 1
132
    fi
133

134
    log "Starting cluster update to version $target_version..."
135

136
    # Backup before update
137
    backup_cluster
138

139
    # Update control plane nodes
140
    log "Updating control plane nodes..."
141

142
    # Drain and update each master node
143
    for node in $(kubectl get nodes -l node-role.kubernetes.io/control-plane -o name); do
144
        node_name=$(echo "$node" | cut -d'/' -f2)
145
        log "Updating control plane node: $node_name"
146

147
        # Drain node
148
        kubectl drain "$node_name" --ignore-daemonsets --delete-emptydir-data --force
149

150
        # Update kubeadm, kubelet, kubectl on the node
151
        ssh "$node_name" "
152
            apt-mark unhold kubeadm kubelet kubectl
153
            apt-get update
154
            apt-get install -y kubeadm=$target_version-00 kubelet=$target_version-00 kubectl=$target_version-00
155
            apt-mark hold kubeadm kubelet kubectl
156
            kubeadm upgrade apply $target_version --yes
157
            systemctl daemon-reload
158
            systemctl restart kubelet
159
        "
160

161
        # Uncordon node
162
        kubectl uncordon "$node_name"
163

164
        # Wait for node to be ready
165
        kubectl wait --for=condition=Ready node/"$node_name" --timeout=300s
166
    done
167

168
    # Update worker nodes
169
    log "Updating worker nodes..."
170

171
    for node in $(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o name); do
172
        node_name=$(echo "$node" | cut -d'/' -f2)
173
        log "Updating worker node: $node_name"
174

175
        # Drain node
176
        kubectl drain "$node_name" --ignore-daemonsets --delete-emptydir-data --force
177

178
        # Update kubeadm, kubelet, kubectl on the node
179
        ssh "$node_name" "
180
            apt-mark unhold kubeadm kubelet kubectl
181
            apt-get update
182
            apt-get install -y kubeadm=$target_version-00 kubelet=$target_version-00 kubectl=$target_version-00
183
            apt-mark hold kubeadm kubelet kubectl
184
            kubeadm upgrade node
185
            systemctl daemon-reload
186
            systemctl restart kubelet
187
        "
188

189
        # Uncordon node
190
        kubectl uncordon "$node_name"
191

192
        # Wait for node to be ready
193
        kubectl wait --for=condition=Ready node/"$node_name" --timeout=300s
194
    done
195

196
    log "Cluster update to $target_version completed successfully"
197
}
198

199
# Certificate renewal
200
renew_certificates() {
201
    log "Starting certificate renewal..."
202

203
    # Backup current certificates
204
    cp -r /etc/kubernetes/pki /opt/kubernetes/pki-backup-$(date +%Y%m%d_%H%M%S)
205

206
    # Renew certificates
207
    kubeadm certs renew all
208

209
    # Restart control plane components
210
    systemctl restart kubelet
211

212
    # Wait for API server to be ready
213
    kubectl wait --for=condition=Available deployment/coredns -n kube-system --timeout=300s
214

215
    log "Certificate renewal completed"
216
}
217

218
# Usage function
219
usage() {
220
    echo "Usage: $0 {health-check|backup|cleanup|update|renew-certs}"
221
    echo
222
    echo "Commands:"
223
    echo "  health-check    Perform cluster health check"
224
    echo "  backup         Create cluster backup"
225
    echo "  cleanup        Clean up cluster resources"
226
    echo "  update <ver>   Update cluster to specified version"
227
    echo "  renew-certs    Renew cluster certificates"
228
    exit 1
229
}
230

231
# Main execution
232
case "${1:-}" in
233
    health-check)
234
        health_check
235
        ;;
236
    backup)
237
        backup_cluster
238
        ;;
239
    cleanup)
240
        cleanup_cluster
241
        ;;
242
    update)
243
        update_cluster "${2:-}"
244
        ;;
245
    renew-certs)
246
        renew_certificates
247
        ;;
248
    *)
249
        usage
250
        ;;
251
esac

Disaster Recovery Procedures#

1
#!/bin/bash
2
# disaster-recovery.sh - Cluster disaster recovery procedures
3

4
set -euo pipefail
5

6
BACKUP_LOCATION="/opt/kubernetes/backups"
7
RECOVERY_LOG="/var/log/disaster-recovery.log"
8

9
log() {
10
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$RECOVERY_LOG"
11
}
12

13
# etcd disaster recovery
14
recover_etcd() {
15
    local backup_file="$1"
16

17
    if [ ! -f "$backup_file" ]; then
18
        log "ERROR: Backup file not found: $backup_file"
19
        return 1
20
    fi
21

22
    log "Starting etcd disaster recovery from: $backup_file"
23

24
    # Stop etcd and API server
25
    systemctl stop kubelet
26

27
    # Backup current etcd data
28
    mv /var/lib/etcd /var/lib/etcd.backup.$(date +%Y%m%d_%H%M%S)
29

30
    # Restore from snapshot
31
    ETCDCTL_API=3 etcdctl snapshot restore "$backup_file" \
32
        --data-dir=/var/lib/etcd \
33
        --initial-cluster-token=etcd-cluster-1 \
34
        --initial-advertise-peer-urls=https://$(hostname -i):2380 \
35
        --name=$(hostname) \
36
        --initial-cluster=$(hostname)=https://$(hostname -i):2380
37

38
    # Fix permissions
39
    chown -R etcd:etcd /var/lib/etcd
40

41
    # Restart services
42
    systemctl start kubelet
43

44
    # Wait for cluster to be ready
45
    sleep 30
46
    kubectl wait --for=condition=Ready nodes --all --timeout=600s
47

48
    log "etcd recovery completed successfully"
49
}
50

51
# Full cluster recovery
52
recover_cluster() {
53
    local backup_archive="$1"
54

55
    if [ ! -f "$backup_archive" ]; then
56
        log "ERROR: Backup archive not found: $backup_archive"
57
        return 1
58
    fi
59

60
    log "Starting full cluster recovery from: $backup_archive"
61

62
    # Extract backup
63
    TEMP_DIR=$(mktemp -d)
64
    tar -xzf "$backup_archive" -C "$TEMP_DIR"
65
    BACKUP_DIR=$(find "$TEMP_DIR" -maxdepth 1 -type d -name "2*" | head -1)
66

67
    if [ ! -d "$BACKUP_DIR" ]; then
68
        log "ERROR: Invalid backup archive structure"
69
        return 1
70
    fi
71

72
    # Recover etcd
73
    if [ -f "$BACKUP_DIR/etcd-snapshot.db" ]; then
74
        recover_etcd "$BACKUP_DIR/etcd-snapshot.db"
75
    fi
76

77
    # Restore certificates
78
    if [ -d "$BACKUP_DIR/pki" ]; then
79
        log "Restoring certificates..."
80
        cp -r "$BACKUP_DIR/pki"/* /etc/kubernetes/pki/
81
        chown -R root:root /etc/kubernetes/pki
82
        chmod -R 600 /etc/kubernetes/pki
83
        find /etc/kubernetes/pki -name "*.crt" -exec chmod 644 {} \;
84
    fi
85

86
    # Restore configuration
87
    if ls "$BACKUP_DIR"/*.conf > /dev/null 2>&1; then
88
        log "Restoring configuration..."
89
        cp "$BACKUP_DIR"/*.conf /etc/kubernetes/
90
    fi
91

92
    # Restart kubelet
93
    systemctl restart kubelet
94

95
    # Wait for cluster to be ready
96
    kubectl wait --for=condition=Ready nodes --all --timeout=600s
97

98
    # Restore resources
99
    if [ -f "$BACKUP_DIR/all-resources.yaml" ]; then
100
        log "Restoring cluster resources..."
101
        kubectl apply -f "$BACKUP_DIR/all-resources.yaml" --force
102
    fi
103

104
    if [ -f "$BACKUP_DIR/persistent-volumes.yaml" ]; then
105
        log "Restoring persistent volumes..."
106
        kubectl apply -f "$BACKUP_DIR/persistent-volumes.yaml"
107
    fi
108

109
    # Cleanup
110
    rm -rf "$TEMP_DIR"
111

112
    log "Full cluster recovery completed successfully"
113
}
114

115
# Node recovery
116
recover_node() {
117
    local node_name="$1"
118
    local node_type="${2:-worker}"  # worker or master
119

120
    log "Starting node recovery for: $node_name ($node_type)"
121

122
    # Remove node from cluster if it exists
123
    kubectl delete node "$node_name" --ignore-not-found=true
124

125
    if [ "$node_type" = "master" ]; then
126
        log "Recovering master node..."
127

128
        # Generate join command for master
129
        CERT_KEY=$(kubeadm init phase upload-certs --upload-certs | tail -1)
130
        JOIN_CMD=$(kubeadm token create --print-join-command)
131
        MASTER_JOIN_CMD="$JOIN_CMD --control-plane --certificate-key $CERT_KEY"
132

133
        # Execute join on the target node
134
        ssh "$node_name" "
135
            kubeadm reset --force
136
            $MASTER_JOIN_CMD
137
            mkdir -p /root/.kube
138
            cp -i /etc/kubernetes/admin.conf /root/.kube/config
139
        "
140

141
    else
142
        log "Recovering worker node..."
143

144
        # Generate join command for worker
145
        JOIN_CMD=$(kubeadm token create --print-join-command)
146

147
        # Execute join on the target node
148
        ssh "$node_name" "
149
            kubeadm reset --force
150
            $JOIN_CMD
151
        "
152
    fi
153

154
    # Wait for node to be ready
155
    kubectl wait --for=condition=Ready node/"$node_name" --timeout=300s
156

157
    log "Node recovery completed for: $node_name"
158
}
159

160
# Validate backup
161
validate_backup() {
162
    local backup_archive="$1"
163

164
    if [ ! -f "$backup_archive" ]; then
165
        log "ERROR: Backup archive not found: $backup_archive"
166
        return 1
167
    fi
168

169
    log "Validating backup: $backup_archive"
170

171
    # Extract and check structure
172
    TEMP_DIR=$(mktemp -d)
173
    tar -xzf "$backup_archive" -C "$TEMP_DIR"
174
    BACKUP_DIR=$(find "$TEMP_DIR" -maxdepth 1 -type d -name "2*" | head -1)
175

176
    # Check for required files
177
    REQUIRED_FILES=("etcd-snapshot.db" "all-resources.yaml" "manifest.json")
178
    for file in "${REQUIRED_FILES[@]}"; do
179
        if [ ! -f "$BACKUP_DIR/$file" ]; then
180
            log "ERROR: Missing required file in backup: $file"
181
            rm -rf "$TEMP_DIR"
182
            return 1
183
        fi
184
    done
185

186
    # Validate etcd snapshot
187
    if ! ETCDCTL_API=3 etcdctl snapshot status "$BACKUP_DIR/etcd-snapshot.db" > /dev/null 2>&1; then
188
        log "ERROR: Invalid etcd snapshot"
189
        rm -rf "$TEMP_DIR"
190
        return 1
191
    fi
192

193
    # Check manifest
194
    if ! jq . "$BACKUP_DIR/manifest.json" > /dev/null 2>&1; then
195
        log "ERROR: Invalid manifest.json"
196
        rm -rf "$TEMP_DIR"
197
        return 1
198
    fi
199

200
    rm -rf "$TEMP_DIR"
201
    log "Backup validation successful"
202
}
203

204
# List available backups
205
list_backups() {
206
    log "Available backups:"
207
    find "$BACKUP_LOCATION" -name "*.tar.gz" -type f -exec ls -lh {} \; | sort -k6,7
208
}
209

210
# Usage
211
usage() {
212
    echo "Usage: $0 {recover-etcd|recover-cluster|recover-node|validate-backup|list-backups}"
213
    echo
214
    echo "Commands:"
215
    echo "  recover-etcd <backup-file>              Recover etcd from snapshot"
216
    echo "  recover-cluster <backup-archive>        Full cluster recovery"
217
    echo "  recover-node <node-name> [master|worker] Recover individual node"
218
    echo "  validate-backup <backup-archive>        Validate backup integrity"
219
    echo "  list-backups                           List available backups"
220
    exit 1
221
}
222

223
# Main execution
224
case "${1:-}" in
225
    recover-etcd)
226
        recover_etcd "${2:-}"
227
        ;;
228
    recover-cluster)
229
        recover_cluster "${2:-}"
230
        ;;
231
    recover-node)
232
        recover_node "${2:-}" "${3:-worker}"
233
        ;;
234
    validate-backup)
235
        validate_backup "${2:-}"
236
        ;;
237
    list-backups)
238
        list_backups
239
        ;;
240
    *)
241
        usage
242
        ;;
243
esac

Best Practices and Recommendations#

Production Readiness Checklist#

Infrastructure:

Multi-node control plane (minimum 3 nodes)
Dedicated etcd cluster or HA etcd setup
Load balancer for API server
Network redundancy and monitoring
Sufficient resource allocation

Security:

RBAC properly configured
Network policies implemented
Pod security standards enforced
Regular security scanning
Certificate management automated

Monitoring:

Backup and Recovery:

Operations:

Performance Optimization#

Node Configuration:

Optimize kernel parameters for container workloads
Configure appropriate CPU and memory limits
Use fast storage for etcd and container runtime
Implement proper network configuration

Cluster Configuration:

Tune API server parameters for scale
Configure appropriate pod and service subnets
Implement resource quotas and limits
Use horizontal pod autoscaling

Application Best Practices:

Design stateless applications when possible
Implement proper health checks
Use resource requests and limits
Follow 12-factor app principles

Conclusion#

This comprehensive guide provides the foundation for deploying production-ready Kubernetes clusters on CoreOS. The implementation covers:

High Availability: Multi-master setup with load balancing
Security: Comprehensive RBAC, network policies, and hardening
Monitoring: Full observability with Prometheus and Grafana
Operations: Automated backup, recovery, and maintenance procedures
Scalability: Infrastructure designed for growth and expansion

Key benefits of this deployment approach:

Reliability: Built for enterprise-grade uptime requirements
Security: Defense-in-depth security implementation
Operational Excellence: Automated operations and monitoring
Disaster Recovery: Comprehensive backup and recovery capabilities
Maintainability: Clear procedures for updates and maintenance

By following this guide, organizations can establish a robust Kubernetes platform that serves as the foundation for modern containerized applications and microservices architectures.

Remember to customize configurations for your specific environment and regularly review and update security practices based on the latest Kubernetes security guidelines.