Exploring Resource Requests and Limits in Kubernetes - A Practical Guide

Resource management is a crucial aspect of running applications in Kubernetes. In this hands-on guide, we’ll explore how to set up and use resource requests and limits in a Kubernetes cluster. Let’s dive in!

Prerequisites#

A running Kubernetes cluster
kubectl command-line tool configured to communicate with your cluster

Step 1: Create a New Namespace#

First, let’s create a new namespace for our experiments:

1
kubectl create namespace mem-example

Step 2: Install Metrics Server#

The Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API. Let’s install it using the provided YAML:

1
kubectl apply -f metrics-server.yaml

Verify that the Metrics Server is running:

1
kubectl get pods -n kube-system | grep metrics-server

You should see a pod running with the name metrics-server-*.

Step 3: Specify Memory Request and Limit#

Now, let’s create a Pod with specific memory requests and limits:

1
apiVersion: v1
2
kind: Pod
3
metadata:
4
  name: memory-demo
5
  namespace: mem-example
6
spec:
7
  containers:
8
    - name: memory-demo-ctr
9
      image: polinux/stress
10
      resources:
11
        requests:
12
          memory: "100Mi"
13
        limits:
14
          memory: "200Mi"
15
      command: ["stress"]
16
      args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"]

Save this as memory-demo.yaml and apply it:

1
kubectl apply -f memory-demo.yaml

Step 4: Verify Pod Status#

Check the status of the pod:

1
kubectl get pod memory-demo -n mem-example

You should see the pod in a Running state.

Step 5: View Detailed Pod Information#

Let’s look at the detailed information about the pod:

1
kubectl get pod memory-demo -n mem-example --output=yaml

In the output, you should see the memory request and limit specified in the resources section.

Step 6: View Pod Metrics#

Now, let’s check the actual memory usage of our pod:

1
kubectl top pod memory-demo -n mem-example

This command will show you the current CPU and memory usage of the pod.

Step 7: Exceed Memory Limits#

Let’s create another pod that tries to use more memory than its limit:

1
apiVersion: v1
2
kind: Pod
3
metadata:
4
  name: memory-demo-2
5
  namespace: mem-example
6
spec:
7
  containers:
8
    - name: memory-demo-2-ctr
9
      image: polinux/stress
10
      resources:
11
        requests:
12
          memory: "50Mi"
13
        limits:
14
          memory: "100Mi"
15
      command: ["stress"]
16
      args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]

Save this as memory-demo-2.yaml and apply it:

1
kubectl apply -f memory-demo-2.yaml

Step 8: Observe OOM Kill#

Check the status of the new pod:

1
kubectl get pod memory-demo-2 -n mem-example

You might see that the pod is in a CrashLoopBackOff state. Let’s check the logs:

1
kubectl logs memory-demo-2 -n mem-example

You should see an error message indicating that the pod was killed due to an Out of Memory (OOM) condition.

1
apiVersion: v1
2
kind: ServiceAccount
3
metadata:
4
  labels:
5
    k8s-app: metrics-server
6
  name: metrics-server
7
  namespace: kube-system
8
---
9
apiVersion: rbac.authorization.k8s.io/v1
10
kind: ClusterRole
11
metadata:
12
  labels:
13
    k8s-app: metrics-server
14
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
15
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
16
    rbac.authorization.k8s.io/aggregate-to-view: "true"
17
  name: system:aggregated-metrics-reader
18
rules:
19
  - apiGroups:
20
      - metrics.k8s.io
21
    resources:
22
      - pods
23
      - nodes
24
    verbs:
25
      - get
26
      - list
27
      - watch
28
---
29
apiVersion: rbac.authorization.k8s.io/v1
30
kind: ClusterRole
31
metadata:
32
  labels:
33
    k8s-app: metrics-server
34
  name: system:metrics-server
35
rules:
36
  - apiGroups:
37
      - ""
38
    resources:
39
      - nodes/metrics
40
    verbs:
41
      - get
42
  - apiGroups:
43
      - ""
44
    resources:
45
      - pods
46
      - nodes
47
    verbs:
48
      - get
49
      - list
50
      - watch
51
---
52
apiVersion: rbac.authorization.k8s.io/v1
53
kind: RoleBinding
54
metadata:
55
  labels:
56
    k8s-app: metrics-server
57
  name: metrics-server-auth-reader
58
  namespace: kube-system
59
roleRef:
60
  apiGroup: rbac.authorization.k8s.io
61
  kind: Role
62
  name: extension-apiserver-authentication-reader
63
subjects:
64
  - kind: ServiceAccount
65
    name: metrics-server
66
    namespace: kube-system
67
---
68
apiVersion: rbac.authorization.k8s.io/v1
69
kind: ClusterRoleBinding
70
metadata:
71
  labels:
72
    k8s-app: metrics-server
73
  name: metrics-server:system:auth-delegator
74
roleRef:
75
  apiGroup: rbac.authorization.k8s.io
76
  kind: ClusterRole
77
  name: system:auth-delegator
78
subjects:
79
  - kind: ServiceAccount
80
    name: metrics-server
81
    namespace: kube-system
82
---
83
apiVersion: rbac.authorization.k8s.io/v1
84
kind: ClusterRoleBinding
85
metadata:
86
  labels:
87
    k8s-app: metrics-server
88
  name: system:metrics-server
89
roleRef:
90
  apiGroup: rbac.authorization.k8s.io
91
  kind: ClusterRole
92
  name: system:metrics-server
93
subjects:
94
  - kind: ServiceAccount
95
    name: metrics-server
96
    namespace: kube-system
97
---
98
apiVersion: v1
99
kind: Service
100
metadata:
101
  labels:
102
    k8s-app: metrics-server
103
  name: metrics-server
104
  namespace: kube-system
105
spec:
106
  ports:
107
    - name: https
108
      port: 443
109
      protocol: TCP
110
      targetPort: https
111
  selector:
112
    k8s-app: metrics-server
113
---
114
apiVersion: apps/v1
115
kind: Deployment
116
metadata:
117
  labels:
118
    k8s-app: metrics-server
119
  name: metrics-server
120
  namespace: kube-system
121
spec:
122
  selector:
123
    matchLabels:
124
      k8s-app: metrics-server
125
  strategy:
126
    rollingUpdate:
127
      maxUnavailable: 0
128
  template:
129
    metadata:
130
      labels:
131
        k8s-app: metrics-server
132
    spec:
133
      containers:
134
        - args:
135
            - --cert-dir=/tmp
136
            - --secure-port=10250
137
            - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
138
            - --kubelet-use-node-status-port
139
            - --kubelet-insecure-tls
140
            - --metric-resolution=15s
141
          image: registry.k8s.io/metrics-server/metrics-server:v0.7.1
142
          imagePullPolicy: IfNotPresent
143
          livenessProbe:
144
            failureThreshold: 3
145
            httpGet:
146
              path: /livez
147
              port: https
148
              scheme: HTTPS
149
            periodSeconds: 10
150
          name: metrics-server
151
          ports:
152
            - containerPort: 10250
153
              name: https
154
              protocol: TCP
155
          readinessProbe:
156
            failureThreshold: 3
157
            httpGet:
158
              path: /readyz
159
              port: https
160
              scheme: HTTPS
161
            initialDelaySeconds: 20
162
            periodSeconds: 10
163
          resources:
164
            requests:
165
              cpu: 100m
166
              memory: 200Mi
167
          securityContext:
168
            allowPrivilegeEscalation: false
169
            capabilities:
170
              drop:
171
                - ALL
172
            readOnlyRootFilesystem: true
173
            runAsNonRoot: true
174
            runAsUser: 1000
175
            seccompProfile:
176
              type: RuntimeDefault
177
          volumeMounts:
178
            - mountPath: /tmp
179
              name: tmp-dir
180
      nodeSelector:
181
        kubernetes.io/os: linux
182
      priorityClassName: system-cluster-critical
183
      serviceAccountName: metrics-server
184
      volumes:
185
        - emptyDir: {}
186
          name: tmp-dir
187
---
188
apiVersion: apiregistration.k8s.io/v1
189
kind: APIService
190
metadata:
191
  labels:
192
    k8s-app: metrics-server
193
  name: v1beta1.metrics.k8s.io
194
spec:
195
  group: metrics.k8s.io
196
  groupPriorityMinimum: 100
197
  insecureSkipTLSVerify: true
198
  service:
199
    name: metrics-server
200
    namespace: kube-system
201
  version: v1beta1
202
  versionPriority: 100

Conclusion#

In this practical guide, we’ve explored how to set and use resource requests and limits in Kubernetes. We’ve seen how Kubernetes schedules pods based on their resource requests and how it enforces resource limits to prevent pods from consuming more resources than allocated.

Key takeaways:

Resource requests help Kubernetes schedule pods efficiently.
Resource limits prevent pods from consuming excessive resources.
Pods that exceed their memory limits are terminated with an OOM kill.
The Metrics Server is crucial for monitoring actual resource usage in your cluster.

Remember, setting appropriate resource requests and limits is crucial for maintaining a stable and efficient Kubernetes cluster. Always monitor your applications’ actual resource usage and adjust these values as needed.

Happy Kuberneting!