A Comprehensive Guide to Upgrading a Multi-Node Kubernetes Cluster with Kubeadm

Kubernetes cluster upgrades are a critical aspect of maintaining a healthy and secure container orchestration environment. In this guide, we’ll walk through the process of upgrading a multi-node Kubernetes cluster using kubeadm, focusing on best practices and potential pitfalls.

Prerequisites

Before beginning the upgrade process, ensure:

1. Your cluster is in a healthy state
2. You have root or sudo access to all nodes
3. You have a recent backup of your cluster and important data
4. You’ve reviewed the Kubernetes release notes for any breaking changes

Understanding Kubernetes Versioning

Kubernetes follows a semantic versioning scheme (MAJOR.MINOR.PATCH). It’s crucial to note:

• ETCD and CoreDNS have separate version numbers from other Kubernetes components
• Always upgrade one minor version at a time (e.g., 1.29.x to 1.30.x)

Step 1: Upgrading the Control Plane Node

Let’s start by upgrading the control plane (master) node:

1. Find the latest 1.30.x version:

   sudo apt update
   sudo apt-cache madison kubeadm

2. Upgrade kubeadm:

   sudo apt-mark unhold kubeadm && \
   sudo apt-get update && sudo apt-get install -y kubeadm='1.30.2-1.1' && \
   sudo apt-mark hold kubeadm

3. Plan the upgrade:

   sudo kubeadm upgrade plan

4. Apply the upgrade:

   sudo kubeadm upgrade apply v1.30.2

5. Drain the control plane node:

   kubectl drain <control-plane-node> --ignore-daemonsets

6. Upgrade kubelet and kubectl:

   sudo apt-mark unhold kubelet kubectl && \
   sudo apt-get update && sudo apt-get install -y kubelet='1.30.2-1.1' kubectl='1.30.2-1.1' && \
   sudo apt-mark hold kubelet kubectl

7. Restart kubelet:

   sudo systemctl daemon-reload
   sudo systemctl restart kubelet

8. Uncordon the control plane node:

   kubectl uncordon <control-plane-node>

Step 2: Upgrading Worker Nodes

Repeat the following steps for each worker node:

1. Drain the node:

   kubectl drain <worker-node> --ignore-daemonsets

2. SSH into the worker node and upgrade kubeadm:

   sudo apt-mark unhold kubeadm && \
   sudo apt-get update && sudo apt-get install -y kubeadm='1.30.2-1.1' && \
   sudo apt-mark hold kubeadm

3. Upgrade the kubelet configuration:

   sudo kubeadm upgrade node

4. Upgrade kubelet and kubectl:

   sudo apt-mark unhold kubelet kubectl && \
   sudo apt-get update && sudo apt-get install -y kubelet='1.30.2-1.1' kubectl='1.30.2-1.1' && \
   sudo apt-mark hold kubelet kubectl

5. Restart kubelet:

   sudo systemctl daemon-reload
   sudo systemctl restart kubelet

6. Uncordon the node:

   kubectl uncordon <worker-node>

Step 3: Verifying the Upgrade

After upgrading all nodes, verify the cluster state:

1. Check node versions:

   kubectl get nodes

2. Verify pod health:

   kubectl get pods --all-namespaces

3. Run a sample workload to ensure cluster functionality

Key Takeaways and Best Practices

1. Incremental Upgrades: Always upgrade one minor version at a time to minimize risks.

2. Control Plane First: Upgrade the control plane before worker nodes to ensure compatibility.

3. Node Draining: Always drain nodes before upgrading to minimize workload disruption.

4. Backup: Create backups before starting the upgrade process.

5. Testing: Have a staging environment to test upgrades before applying to production.

6. Component Versions: Pay attention to component-specific versions (e.g., ETCD, CoreDNS).

7. Rolling Upgrades: For larger clusters, consider upgrading worker nodes in batches to maintain availability.

8. Monitoring: Keep a close eye on cluster health metrics during and after the upgrade.

9. Documentation: Keep detailed notes of the upgrade process for future reference and troubleshooting.

Conclusion

Upgrading a Kubernetes cluster requires careful planning and execution. By following this guide and adhering to best practices, you can ensure a smooth upgrade process with minimal disruption to your workloads. Remember, the key to successful upgrades is thorough preparation, careful execution, and comprehensive verification.

As Kubernetes continues to evolve, staying up-to-date with the latest stable versions is crucial for maintaining security, performance, and access to new features. Regular, well-planned upgrades should be part of your ongoing Kubernetes maintenance strategy.