Exploring Resource Requests and Limits in Kubernetes - A Practical Guide

Resource management is a crucial aspect of running applications in Kubernetes. In this hands-on guide, we’ll explore how to set up and use resource requests and limits in a Kubernetes cluster. Let’s dive in!

Prerequisites

• A running Kubernetes cluster
• kubectl command-line tool configured to communicate with your cluster

Step 1: Create a New Namespace

First, let’s create a new namespace for our experiments:

kubectl create namespace mem-example

Step 2: Install Metrics Server

The Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API. Let’s install it using the provided YAML:

kubectl apply -f metrics-server.yaml

Verify that the Metrics Server is running:

kubectl get pods -n kube-system | grep metrics-server

You should see a pod running with the name metrics-server-*.

Step 3: Specify Memory Request and Limit

Now, let’s create a Pod with specific memory requests and limits:

apiVersion: v1
kind: Pod
metadata:
  name: memory-demo
  namespace: mem-example
spec:
  containers:
    - name: memory-demo-ctr
      image: polinux/stress
      resources:
        requests:
          memory: "100Mi"
        limits:
          memory: "200Mi"
      command: ["stress"]
      args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"]

Save this as memory-demo.yaml and apply it:

kubectl apply -f memory-demo.yaml

Step 4: Verify Pod Status

Check the status of the pod:

kubectl get pod memory-demo -n mem-example

You should see the pod in a Running state.

Step 5: View Detailed Pod Information

Let’s look at the detailed information about the pod:

kubectl get pod memory-demo -n mem-example --output=yaml

In the output, you should see the memory request and limit specified in the resources section.

Step 6: View Pod Metrics

Now, let’s check the actual memory usage of our pod:

kubectl top pod memory-demo -n mem-example

This command will show you the current CPU and memory usage of the pod.

Step 7: Exceed Memory Limits

Let’s create another pod that tries to use more memory than its limit:

apiVersion: v1
kind: Pod
metadata:
  name: memory-demo-2
  namespace: mem-example
spec:
  containers:
    - name: memory-demo-2-ctr
      image: polinux/stress
      resources:
        requests:
          memory: "50Mi"
        limits:
          memory: "100Mi"
      command: ["stress"]
      args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]

Save this as memory-demo-2.yaml and apply it:

kubectl apply -f memory-demo-2.yaml

Step 8: Observe OOM Kill

Check the status of the new pod:

kubectl get pod memory-demo-2 -n mem-example

You might see that the pod is in a CrashLoopBackOff state. Let’s check the logs:

kubectl logs memory-demo-2 -n mem-example

You should see an error message indicating that the pod was killed due to an Out of Memory (OOM) condition.

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
  - apiGroups:
      - metrics.k8s.io
    resources:
      - pods
      - nodes
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/metrics
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - pods
      - nodes
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
  - kind: ServiceAccount
    name: metrics-server
    namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
        - args:
            - --cert-dir=/tmp
            - --secure-port=10250
            - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
            - --kubelet-use-node-status-port
            - --kubelet-insecure-tls
            - --metric-resolution=15s
          image: registry.k8s.io/metrics-server/metrics-server:v0.7.1
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /livez
              port: https
              scheme: HTTPS
            periodSeconds: 10
          name: metrics-server
          ports:
            - containerPort: 10250
              name: https
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /readyz
              port: https
              scheme: HTTPS
            initialDelaySeconds: 20
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 200Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
            seccompProfile:
              type: RuntimeDefault
          volumeMounts:
            - mountPath: /tmp
              name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
        - emptyDir: {}
          name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

Conclusion

In this practical guide, we’ve explored how to set and use resource requests and limits in Kubernetes. We’ve seen how Kubernetes schedules pods based on their resource requests and how it enforces resource limits to prevent pods from consuming more resources than allocated.

Key takeaways:

1. Resource requests help Kubernetes schedule pods efficiently.
2. Resource limits prevent pods from consuming excessive resources.
3. Pods that exceed their memory limits are terminated with an OOM kill.
4. The Metrics Server is crucial for monitoring actual resource usage in your cluster.

Remember, setting appropriate resource requests and limits is crucial for maintaining a stable and efficient Kubernetes cluster. Always monitor your applications’ actual resource usage and adjust these values as needed.

Happy Kuberneting!