Mastering Taints and Tolerations in Kubernetes - A Hands-On Guide

Kubernetes taints and tolerations are powerful features that give you fine-grained control over which pods can be scheduled on specific nodes. In this hands-on guide, we’ll explore these concepts through a series of practical exercises. Let’s dive in!

1. Tainting Worker Nodes

First, let’s taint our worker nodes:

kubectl taint nodes worker01 gpu=true:NoSchedule
kubectl taint nodes worker02 gpu=false:NoSchedule

These commands add taints to our worker nodes, effectively telling the scheduler not to place pods on these nodes unless they have matching tolerations.

2. Creating a Pod Without Tolerations

Now, let’s create a simple nginx pod and see what happens:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  containers:
    - name: nginx
      image: nginx

Save this as nginx-pod.yaml and apply it:

kubectl apply -f nginx-pod.yaml

Check the pod status:

kubectl get pods

You’ll notice that the pod is in a “Pending” state. Let’s investigate why:

kubectl describe pod nginx-pod

In the events section, you should see messages indicating that the pod can’t be scheduled due to the taints on the worker nodes.

3. Adding Toleration to the Pod

Let’s modify our pod definition to include a toleration for the gpu=true:NoSchedule taint:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod-with-toleration
spec:
  containers:
    - name: nginx
      image: nginx
  tolerations:
    - key: "gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

Apply this new configuration:

kubectl apply -f nginx-pod-with-toleration.yaml

Now, check the pod status again:

kubectl get pods -o wide

You should see that the pod is now scheduled on worker01, which has the matching taint.

4. Removing Taint from Control Plane Node

By default, Kubernetes taints control plane nodes to prevent regular workloads from running on them. Let’s remove this taint:

kubectl taint nodes $(kubectl get nodes --selector=node-role.kubernetes.io/control-plane -o jsonpath='{.items[*].metadata.name}') node-role.kubernetes.io/control-plane:NoSchedule-

5. Creating a Redis Pod

Now that we’ve removed the taint from the control plane, let’s create a Redis pod:

apiVersion: v1
kind: Pod
metadata:
  name: redis-pod
spec:
  containers:
    - name: redis
      image: redis

Apply this configuration:

kubectl apply -f redis-pod.yaml

Check where the pod is scheduled:

kubectl get pods -o wide

You should see that the Redis pod is now scheduled on the control plane node.

6. Re-adding Taint to Control Plane Node

Finally, let’s add the taint back to the control plane node:

kubectl taint nodes $(kubectl get nodes --selector=node-role.kubernetes.io/control-plane -o jsonpath='{.items[*].metadata.name}') node-role.kubernetes.io/control-plane=:NoSchedule

Conclusion

Through these exercises, we’ve explored the practical application of taints and tolerations in Kubernetes. We’ve seen how taints can prevent pods from being scheduled on specific nodes, and how tolerations allow pods to bypass these restrictions.

Key takeaways:

1. Taints are applied to nodes and prevent pods from being scheduled unless they have matching tolerations.
2. Tolerations are defined in pod specifications and allow pods to be scheduled on nodes with matching taints.
3. Control plane nodes are typically tainted by default to prevent regular workloads from running on them.
4. Removing taints from control plane nodes allows regular pods to be scheduled there, which can be useful in development or testing scenarios.

Remember, while taints and tolerations provide powerful control over pod scheduling, they should be used judiciously. In production environments, it’s generally recommended to keep the control plane dedicated to system tasks and schedule application workloads on worker nodes.

As you continue your Kubernetes journey, experiment with different combinations of taints and tolerations to see how they affect pod scheduling in your cluster. Happy Kuberneting!