A Comprehensive Guide to Upgrading a Multi-Node Kubernetes Cluster with Kubeadm#

Prerequisites#

Before beginning the upgrade process, ensure:

1. Your cluster is in a healthy state
2. You have root or sudo access to all nodes
3. You have a recent backup of your cluster and important data
4. You’ve reviewed the Kubernetes release notes for any breaking changes

Understanding Kubernetes Versioning#

Kubernetes follows a semantic versioning scheme (MAJOR.MINOR.PATCH). It’s crucial to note:

• ETCD and CoreDNS have separate version numbers from other Kubernetes components
• Always upgrade one minor version at a time (e.g., 1.29.x to 1.30.x)

Step 1: Upgrading the Control Plane Node#

Let’s start by upgrading the control plane (master) node:

1. Find the latest 1.30.x version:

   1
   sudo apt update
   2
   sudo apt-cache madison kubeadm
   

2. Upgrade kubeadm:

   1
   sudo apt-mark unhold kubeadm && \
   2
   sudo apt-get update && sudo apt-get install -y kubeadm='1.30.2-1.1' && \
   3
   sudo apt-mark hold kubeadm
   

3. Plan the upgrade:

   1
   sudo kubeadm upgrade plan
   

4. Apply the upgrade:

   1
   sudo kubeadm upgrade apply v1.30.2
   

5. Drain the control plane node:

   1
   kubectl drain <control-plane-node> --ignore-daemonsets
   

6. Upgrade kubelet and kubectl:

   1
   sudo apt-mark unhold kubelet kubectl && \
   2
   sudo apt-get update && sudo apt-get install -y kubelet='1.30.2-1.1' kubectl='1.30.2-1.1' && \
   3
   sudo apt-mark hold kubelet kubectl
   

7. Restart kubelet:

   1
   sudo systemctl daemon-reload
   2
   sudo systemctl restart kubelet
   

8. Uncordon the control plane node:

   1
   kubectl uncordon <control-plane-node>
   

Step 2: Upgrading Worker Nodes#

Repeat the following steps for each worker node:

1. Drain the node:

   1
   kubectl drain <worker-node> --ignore-daemonsets
   

2. SSH into the worker node and upgrade kubeadm:

   1
   sudo apt-mark unhold kubeadm && \
   2
   sudo apt-get update && sudo apt-get install -y kubeadm='1.30.2-1.1' && \
   3
   sudo apt-mark hold kubeadm
   

3. Upgrade the kubelet configuration:

   1
   sudo kubeadm upgrade node
   

4. Upgrade kubelet and kubectl:

   1
   sudo apt-mark unhold kubelet kubectl && \
   2
   sudo apt-get update && sudo apt-get install -y kubelet='1.30.2-1.1' kubectl='1.30.2-1.1' && \
   3
   sudo apt-mark hold kubelet kubectl
   

5. Restart kubelet:

   1
   sudo systemctl daemon-reload
   2
   sudo systemctl restart kubelet
   

6. Uncordon the node:

   1
   kubectl uncordon <worker-node>
   

Step 3: Verifying the Upgrade#

After upgrading all nodes, verify the cluster state:

1. Check node versions:

   1
   kubectl get nodes
   

2. Verify pod health:

   1
   kubectl get pods --all-namespaces
   

3. Run a sample workload to ensure cluster functionality

Key Takeaways and Best Practices#

1. Incremental Upgrades: Always upgrade one minor version at a time to minimize risks.

2. Control Plane First: Upgrade the control plane before worker nodes to ensure compatibility.

3. Node Draining: Always drain nodes before upgrading to minimize workload disruption.

4. Backup: Create backups before starting the upgrade process.

5. Testing: Have a staging environment to test upgrades before applying to production.

6. Component Versions: Pay attention to component-specific versions (e.g., ETCD, CoreDNS).

7. Rolling Upgrades: For larger clusters, consider upgrading worker nodes in batches to maintain availability.

8. Monitoring: Keep a close eye on cluster health metrics during and after the upgrade.

9. Documentation: Keep detailed notes of the upgrade process for future reference and troubleshooting.

Conclusion#

Upgrading a Kubernetes cluster requires careful planning and execution. By following this guide and adhering to best practices, you can ensure a smooth upgrade process with minimal disruption to your workloads. Remember, the key to successful upgrades is thorough preparation, careful execution, and comprehensive verification.

As Kubernetes continues to evolve, staying up-to-date with the latest stable versions is crucial for maintaining security, performance, and access to new features. Regular, well-planned upgrades should be part of your ongoing Kubernetes maintenance strategy.