Lab 8 - Container Lifecycle Management
Introduction¶
Welcome to the lab 8. In this session, the following topics are covered:
- Container liveness and readiness probes;
- Affinity and anti-affinity;
- Introduction to Helm tool.
Container probes¶
As we already know. Kubernetes manages lifecycle of the Pods in the cluster. But what exactly reports a container's state? By default, Kubernetes considers a Pod as running when an initial process started. An example: when a user starts an application server, it takes some time to boot up, apply migrations and start listening on a port. During this phases, the server can't process external requests, but Kubernetes doesn't know about it. In order to report container status, three types of probes exist:
- liveness - application is healthy and is considered as successfully running. When a probe fails, Kubernetes restarts a Pod;
- readiness - application is healthy and ready to process external requests. When a probe fails, Kubernetes don't forward requests sent to the app through a Service;
- startup - application has started successfully and initial actions performed well. If defined, the probe starts before the previous two. Also unlike them, this one performs only once and then is replaced by a liveness probe.
A probe can be a simple shell command (cat /var/run/app.pid
) or a network check (TCP, HTTP, RPC). When an action finishes with successful code 0, container is marked as healthy by Kubernetes.
Liveness
Probes¶
Complete
Let's delete the existing pod for echoserver:
kubectl delete pod echoserver
apiVersion: apps/v1
kind: Deployment
metadata:
name: echoserver
spec:
selector:
matchLabels:
app: echoserver
template:
metadata:
name: echoserver
labels:
app: echoserver
spec:
containers:
- name: echoserver
image: registry.hpc.ut.ee/mirror/ealen/echo-server:latest
ports:
- containerPort: 80
env:
- name: PORT
value: "80"
livenessProbe: #(1)
initialDelaySeconds: 10 #(2)
periodSeconds: 5 #(3)
failureThreshold: 5 #(4)
httpGet:
port: 80
path: /
- The probe checks if a process listens on port 80 and responses with 200 HTTP OK code.
- Initial delay before first probe performs.
- A time period between checks in seconds.
- Number of failed checks before the Pod marked as unhealthy and gets restarted.
Validate
You can check the Pod status using this command (-w
stands for "wait"):
kubectl get pods -l app=echoserver -w
Let's see how Kubernetes acts when a Pod fails liveness probes more then failureThreshold
. For this, change the port in the probe config to 81
and apply the new manifest. You should see a similar result to this one:
kubectl get pods -l app=echoserver -w
# NAME READY STATUS RESTARTS AGE
# echoserver-7f655fd6f5-c4gjq 1/1 Running 0 8s
# echoserver-7f655fd6f5-c4gjq 1/1 Running 1 (2s ago) 37s
# echoserver-7f655fd6f5-c4gjq 1/1 Running 2 (2s ago) 72s
# echoserver-7f655fd6f5-c4gjq 1/1 Running 3 (2s ago) 107s
# echoserver-7f655fd6f5-c4gjq 1/1 Running 4 (2s ago) 2m22s
# echoserver-7f655fd6f5-c4gjq 0/1 CrashLoopBackOff 4 (1s ago) 2m56s
CrashLoopBackOff
status indicates the Pod doesn't pass the checks and Kubernetes restarts it infinitely. Please, restore the correct port value for the probe for further sections.
Readiness Probes¶
Complete
Let's setup the readiness probe for the same Deployment:
apiVersion: apps/v1
kind: Deployment
...
livenessProbe:
...
readinessProbe:
initialDelaySeconds: 15
periodSeconds: 5
failureThreshold: 5
httpGet:
port: 80
path: /
It has the same content but a different impact when checks fail.
Validate
The echoserver Pod should be up and running. You can view the Deployment status and ensure there is one available Pod:
kubectl get deployment echoserver
# NAME READY UP-TO-DATE AVAILABLE AGE
# echoserver 1/1 1 1 37m
Let's change the port for the probe (e.g. 81), apply the new manifest and list the echoserver Pods.
kubectl get pod -l app=echoserver
# NAME READY STATUS RESTARTS AGE
# echoserver-6d87869565-7fp6z 0/1 Running 0 15s
# echoserver-7c7746c59b-mwz87 1/1 Running 0 7m53s
As you can see, there are two Pods: the old one is running and has all containers ready, while the new one is hanging. Kubernetes doesn't remove existing Pods from a Deployment, when new ones are not ready. Even though there is a working Pod, the Deployment is still marked as unavailable:
kubectl get deployment echoserver
# NAME READY UP-TO-DATE AVAILABLE AGE
# echoserver 0/1 1 0 38m
Let's also check how Services behave when a Pod is not ready. For this, remove the deployment and apply the manifest with readiness probe checking an incorrect port (e.g. 81).
kubectl get pods -l app=echoserver
# NAME READY STATUS RESTARTS AGE
# echoserver-84b767ff89-jfdl8 0/1 Running 0 84s
There is only one Pod now. If you try to curl this Pod through the Service we created in networking lab, you should get "connection refused" status:
kubectl get service echoserver
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# echoserver ClusterIP 10.106.242.153 <none> 80/TCP 33d
curl 10.106.242.153
# curl: (7) Failed to connect to 10.106.242.153 port 80: Connection refused
This indicates Kubernetes doesn't forward any traffic to the Pod through the Service, because the container isn't ready. You can still access the Pod using it's IP, while other application relying on the Service can't.
ECHOSERVER_IP=$(kubectl get pod -l app=echoserver -o jsonpath={.items[0].status.podIP})
curl $ECHOSERVER_IP
# {"host":{"hostname":"10.0.1.184"
# ...
In real-life production systems, it can be crucial to avoid external communication, when an application isn't ready.
Complete
For this section, you need to define liveness and readiness probes for the ghostfolio deployment similar to the ones we created before.
Pods scheduling¶
Kubernetes allows setting a requirement for scheduler, so that a Pod runs on a subset of nodes with/without particular labels.
nodeSelector
field¶
The simplest way to restrict set of nodes for a Pod is usage of nodeSelector
field, which accepts a map of node labels.
Complete
First of all, let's view node labels in the Kubernetes cluster:
kubectl get nodes --show-labels
# NAME STATUS ROLES AGE VERSION LABELS
# k8s-worker.cloud.ut.ee Ready <none> 34d v1.27.5 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker.cloud.ut.ee,kubernetes.io/os=linux
# labs-test.cloud.ut.ee Ready control-plane 34d v1.27.5 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=labs-test.cloud.ut.ee,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
The output contain standard set of labels for each node. Let's add a label to a one, which is not a control plane:
kubectl label nodes k8s-worker.cloud.ut.ee custom-role=worker
NB: use the hostname of your VM.
Verify
Check if the node get the label using the command a kubectl get nodes --show-labels
Complete
In this task, we need to make echoserver run on a worker node only. You can view the current node, where the Pod run, via the command:
kubectl get pods -l app=echoserver -o wide
# NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
# echoserver-7b4bbb5784-hgwpf 1/1 Running 0 26h 10.0.1.200 k8s-worker.cloud.ut.ee <none> <none>
In this example, the Pod is already running on the worker node, but this can change when after the redeployment. Set nodeSelector
for the Pod template in the echoserver deployment do ensure this requirement:
apiVersion: apps/v1
kind: Deployment
metadata:
name: echoserver
spec:
selector:
matchLabels:
app: echoserver
template:
metadata:
name: echoserver
labels:
app: echoserver
spec:
nodeSelector: #(1)
custom-role: worker #(2)
...
Apply the new manifest to the node and check the result with the command we used before.
Node affinity and anti-affinity¶
In case if you need to define advanced rules for node selection, affinity/anti-affinity can help. The main differences of this approach from nodeSelector
are:
- Selectors can be soft requirements instead of hard ones, so Pods will run on a node even if it doesn't satisfy the rules;
- Support for selection for other Pods, so that two Pod sets can be scheduled on the same node or vice versa.
There are two rule types for node affinity: requiredDuringSchedulingIgnoredDuringExecution
and preferredDuringSchedulingIgnoredDuringExecution
. The first one is a requirement working like a nodeSelector
, the second one - same, but a preference. IgnoredDuringExecution
part means the Pods with the constraints will continue running on a node even if the node labels are changed.
Complete
In this section, you need to replace nodeSelector
field with affinity
for the existing echoserver deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: echoserver
spec:
selector:
matchLabels:
app: echoserver
template:
metadata:
name: echoserver
labels:
app: echoserver
spec:
affinity: #(1)
nodeAffinity: #(2)
requiredDuringSchedulingIgnoredDuringExecution: #(3)
nodeSelectorTerms: #(4)
- matchExpressions:
- key: custom-role
operator: In
values:
- worker
...
- Affinity section.
- Specifies the node affinity.
- This is a requirement, not a preference.
- A set of expressions with subject-operator-object(s) style. In this case the
custom-role
label of a node should haveworker
value
As you can see, the syntax is much more expressive and provides better flexibility over node selection.
Verify
Apply the manifest and check that the Pod is still running on the worker node. Example:
# kubectl get pods -l app=echoserver -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
echoserver-75894b9c47-75n6b 1/1 Running 0 85s 10.0.1.94 k8s-worker.cloud.ut.ee <none> <none>
Complete
Let's setup a Pod anti-affinity for a second echoserver deployment. For this, create a similar manifest with a different affinity config:
apiVersion: apps/v1
kind: Deployment
metadata:
name: echoserver-2
spec:
selector:
matchLabels:
app: echoserver-2
template:
metadata:
name: echoserver-2
labels:
app: echoserver-2
spec:
affinity:
podAntiAffinity: #(1)
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector: #(2)
matchExpressions:
- key: app
operator: In
values:
- echoserver
topologyKey: "kubernetes.io/hostname" #(3)
...
- Inter-pod anti-affinity section for the deployment
- In this case, Kubernetes can't schedule a workload on a node, where Pods with the
app=echoserver
label are running. - Specifies the context where rule applies:
hostname
means selection for nodes. In case ofzone
value, the Pod would be scheduled in a different set of nodes.
Verify
Apply the manifest and check if the new Pod is running on a different node then the previous one. Example:
kubectl get pods -o wide | grep "echoserver"
# echoserver-2-56bdc76fdd-8pl8w 1/1 Running 0 8m41s 10.0.0.227 labs-test.cloud.ut.ee <none> <none>
# echoserver-75894b9c47-75n6b 1/1 Running 0 27m 10.0.1.94 k8s-worker.cloud.ut.ee <none> <none>
Complete
Now you need to create an affinity rule for ghostfolio
deployment: it should run on a node with custom-role=worker
label
Introduction to Helm¶
In this section we cover Helm installation and basic management capabilities. Helm is a package manager for Kubernetes, which is used in the future labs, and provides:
- Manifests templating using
golang
templates. A set of manifest templates callsChart
. The installed manifests on a Kubernetes cluster after templating performed isRelease
. - Chart packaging, dependency management and publication.
- Release versioning and rollback.
Complete
Install Helm tool on your control plane VM:
curl https://get.helm.sh/helm-v3.13.1-linux-amd64.tar.gz -o ./helm.tar.gz
tar -zxvf ./helm.tar.gz
cp linux-amd64/helm /usr/bin/helm
rm -rf linux-amd64/
Create a sample chart and have a look at the contents:
helm create sample
ls -lh sample/
# total 8.0K
# drwxr-xr-x. 2 root root 6 Oct 27 08:34 charts #(1)
# -rw-r--r--. 1 root root 1.2K Oct 27 08:34 Chart.yaml #(2)
# drwxr-xr-x. 3 root root 162 Oct 27 08:34 templates #(3)
# -rw-r--r--. 1 root root 2.2K Oct 27 08:34 values.yaml #(4)
- directory with dependency charts
- main file with chart settings: name, description, type (library or application), version, app version
- Kubernetes manifest templates
- values for templating
Remove the templates/
folder as we create templates ourself. Let's add the nginx deployment to the empty templates
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-chart
labels:
app: nginx-chart
helm.sh/chart: sample
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: nginx-chart
helm.sh/chart: sample
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
app: nginx-chart
helm.sh/chart: sample
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
ports:
- name: http
containerPort: {{ .Values.service.port }}
protocol: TCP
env:
- name: "KEY"
value: "VALUE"
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
As you can see, lot of values in the manifest are templated. This simplifies changing of common parts through a single values.yaml
file.
Now, let's install the release:
helm install nginx-release sample/
# NAME: nginx-release
# LAST DEPLOYED: Fri Oct 27 09:09:37 2023
# NAMESPACE: default
# STATUS: deployed #(1)
# REVISION: 1 #(2)
# TEST SUITE: None
- Indicates the release is installed successfully.
- Each revision has its number and you can rollback to the desired one if needed.
Validate
After installation, you can list releases within the namespace:
helm ls
# NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
# nginx-release default 1 2023-10-27 09:09:37.022773145 -0400 EDT deployed nginx-test-0.1.0 1.16.0
and view the created Pod:
kubectl get pods -l helm.sh/chart=sample
# NAME READY STATUS RESTARTS AGE
# nginx-chart-84f7fd9546-crcwt 1/1 Running 0 15s
Complete
Let's add a service to the deployment:
apiVersion: v1
kind: Service
metadata:
name: "nginx-chart"
labels:
app: nginx-chart
helm.sh/chart: sample
spec:
type: {{ .Values.service.type }} #(1)
ports:
- port: {{ .Values.service.port }} #(2)
targetPort: http
protocol: TCP
name: http
selector:
app: nginx-chart
helm.sh/chart: sample
- Service type,
ClusterIP
in this case - The same port as we use in the deployment
Then upgrade the chart:
helm upgrade nginx-release sample/
# Release "nginx-release" has been upgraded. Happy Helming!
# NAME: nginx-release
# LAST DEPLOYED: Fri Oct 27 09:24:16 2023
# NAMESPACE: default
# STATUS: deployed
# REVISION: 2 #(1)
# TEST SUITE: None
Validate
You can view the created service and check it works:
kubectl get service -l helm.sh/chart=sample
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# nginx-chart ClusterIP 10.107.82.204 <none> 80/TCP 62s
curl 10.107.82.204
# <!DOCTYPE html>
# <html>
# ...