Skip to content

Lab 8 - Container Lifecycle Management

Introduction

Welcome to the lab 8. In this session, the following topics are covered:

  • Container liveness and readiness probes;
  • Affinity and anti-affinity;
  • Introduction to Helm tool.

Container probes

As we already know. Kubernetes manages lifecycle of the Pods in the cluster. But what exactly reports a container's state? By default, Kubernetes considers a Pod as running when an initial process started. An example: when a user starts an application server, it takes some time to boot up, apply migrations and start listening on a port. During this phases, the server can't process external requests, but Kubernetes doesn't know about it. In order to report container status, three types of probes exist:

  1. liveness - application is healthy and is considered as successfully running. When a probe fails, Kubernetes restarts a Pod;
  2. readiness - application is healthy and ready to process external requests. When a probe fails, Kubernetes don't forward requests sent to the app through a Service;
  3. startup - application has started successfully and initial actions performed well. If defined, the probe starts before the previous two. Also unlike them, this one performs only once and then is replaced by a liveness probe.

A probe can be a simple shell command (cat /var/run/app.pid) or a network check (TCP, HTTP, RPC). When an action finishes with successful code 0, container is marked as healthy by Kubernetes.

Liveness Probes

Complete

Let's delete the existing pod for echoserver:

kubectl delete pod echoserver
and create a new echoserver deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      name: echoserver
      labels:
        app: echoserver
    spec:
      containers:
      - name: echoserver
        image: registry.hpc.ut.ee/mirror/ealen/echo-server:latest
        ports:
        - containerPort: 80
        env:
        - name: PORT
          value: "80"
        livenessProbe: #(1)
          initialDelaySeconds: 10 #(2)
          periodSeconds: 5 #(3)
          failureThreshold: 5 #(4)
          httpGet:
            port: 80
            path: /
  1. The probe checks if a process listens on port 80 and responses with 200 HTTP OK code.
  2. Initial delay before first probe performs.
  3. A time period between checks in seconds.
  4. Number of failed checks before the Pod marked as unhealthy and gets restarted.

Validate

You can check the Pod status using this command (-w stands for "wait"):

kubectl get pods -l app=echoserver -w

Let's see how Kubernetes acts when a Pod fails liveness probes more then failureThreshold. For this, change the port in the probe config to 81 and apply the new manifest. You should see a similar result to this one:

kubectl get pods -l app=echoserver -w
# NAME                          READY   STATUS    RESTARTS   AGE
# echoserver-7f655fd6f5-c4gjq   1/1     Running   0          8s
# echoserver-7f655fd6f5-c4gjq   1/1     Running   1 (2s ago)   37s
# echoserver-7f655fd6f5-c4gjq   1/1     Running   2 (2s ago)   72s
# echoserver-7f655fd6f5-c4gjq   1/1     Running   3 (2s ago)   107s
# echoserver-7f655fd6f5-c4gjq   1/1     Running   4 (2s ago)   2m22s
# echoserver-7f655fd6f5-c4gjq   0/1     CrashLoopBackOff   4 (1s ago)   2m56s

CrashLoopBackOff status indicates the Pod doesn't pass the checks and Kubernetes restarts it infinitely. Please, restore the correct port value for the probe for further sections.

Readiness Probes

Complete

Let's setup the readiness probe for the same Deployment:

apiVersion: apps/v1
kind: Deployment
...
        livenessProbe:
        ...
        readinessProbe:
          initialDelaySeconds: 15
          periodSeconds: 5
          failureThreshold: 5
          httpGet:
            port: 80
            path: /

It has the same content but a different impact when checks fail.

Validate

The echoserver Pod should be up and running. You can view the Deployment status and ensure there is one available Pod:

kubectl get deployment echoserver
# NAME         READY   UP-TO-DATE   AVAILABLE   AGE
# echoserver   1/1     1            1           37m

Let's change the port for the probe (e.g. 81), apply the new manifest and list the echoserver Pods.

kubectl get pod -l app=echoserver
# NAME                          READY   STATUS    RESTARTS   AGE
# echoserver-6d87869565-7fp6z   0/1     Running   0          15s
# echoserver-7c7746c59b-mwz87   1/1     Running   0          7m53s

As you can see, there are two Pods: the old one is running and has all containers ready, while the new one is hanging. Kubernetes doesn't remove existing Pods from a Deployment, when new ones are not ready. Even though there is a working Pod, the Deployment is still marked as unavailable:

kubectl get deployment echoserver
# NAME         READY   UP-TO-DATE   AVAILABLE   AGE
# echoserver   0/1     1            0           38m

Let's also check how Services behave when a Pod is not ready. For this, remove the deployment and apply the manifest with readiness probe checking an incorrect port (e.g. 81).

kubectl get pods -l app=echoserver
# NAME                          READY   STATUS    RESTARTS   AGE
# echoserver-84b767ff89-jfdl8   0/1     Running   0          84s

There is only one Pod now. If you try to curl this Pod through the Service we created in networking lab, you should get "connection refused" status:

kubectl get service echoserver
# NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
# echoserver   ClusterIP   10.106.242.153   <none>        80/TCP    33d

curl 10.106.242.153
# curl: (7) Failed to connect to 10.106.242.153 port 80: Connection refused

This indicates Kubernetes doesn't forward any traffic to the Pod through the Service, because the container isn't ready. You can still access the Pod using it's IP, while other application relying on the Service can't.

ECHOSERVER_IP=$(kubectl get pod -l app=echoserver -o jsonpath={.items[0].status.podIP})
curl $ECHOSERVER_IP
# {"host":{"hostname":"10.0.1.184"
# ...

In real-life production systems, it can be crucial to avoid external communication, when an application isn't ready.

Complete

For this section, you need to define liveness and readiness probes for the ghostfolio deployment similar to the ones we created before.

Pods scheduling

Kubernetes allows setting a requirement for scheduler, so that a Pod runs on a subset of nodes with/without particular labels.

nodeSelector field

The simplest way to restrict set of nodes for a Pod is usage of nodeSelector field, which accepts a map of node labels.

Complete

First of all, let's view node labels in the Kubernetes cluster:

kubectl get nodes --show-labels
# NAME                     STATUS   ROLES           AGE   VERSION   LABELS
# k8s-worker.cloud.ut.ee   Ready    <none>          34d   v1.27.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-worker.cloud.ut.ee,kubernetes.io/os=linux
# labs-test.cloud.ut.ee    Ready    control-plane   34d   v1.27.5   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=labs-test.cloud.ut.ee,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=

The output contain standard set of labels for each node. Let's add a label to a one, which is not a control plane:

kubectl label nodes k8s-worker.cloud.ut.ee custom-role=worker

NB: use the hostname of your VM.

Verify

Check if the node get the label using the command a kubectl get nodes --show-labels

Complete

In this task, we need to make echoserver run on a worker node only. You can view the current node, where the Pod run, via the command:

kubectl get pods -l app=echoserver -o wide
# NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE                     NOMINATED NODE   READINESS GATES
# echoserver-7b4bbb5784-hgwpf   1/1     Running   0          26h   10.0.1.200   k8s-worker.cloud.ut.ee   <none>           <none>

In this example, the Pod is already running on the worker node, but this can change when after the redeployment. Set nodeSelector for the Pod template in the echoserver deployment do ensure this requirement:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      name: echoserver
      labels:
        app: echoserver
    spec:
      nodeSelector: #(1)
        custom-role: worker #(2)
...
1. Setting for selecting a node within the cluster with the provided labels. 2. The label with the required value.

Apply the new manifest to the node and check the result with the command we used before.

Node affinity and anti-affinity

In case if you need to define advanced rules for node selection, affinity/anti-affinity can help. The main differences of this approach from nodeSelector are:

  1. Selectors can be soft requirements instead of hard ones, so Pods will run on a node even if it doesn't satisfy the rules;
  2. Support for selection for other Pods, so that two Pod sets can be scheduled on the same node or vice versa.

There are two rule types for node affinity: requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. The first one is a requirement working like a nodeSelector, the second one - same, but a preference. IgnoredDuringExecution part means the Pods with the constraints will continue running on a node even if the node labels are changed.

Complete

In this section, you need to replace nodeSelector field with affinity for the existing echoserver deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      name: echoserver
      labels:
        app: echoserver
    spec:
      affinity: #(1)
        nodeAffinity: #(2)
          requiredDuringSchedulingIgnoredDuringExecution: #(3)
            nodeSelectorTerms: #(4)
            - matchExpressions:
              - key: custom-role
                operator: In
                values:
                - worker
...
  1. Affinity section.
  2. Specifies the node affinity.
  3. This is a requirement, not a preference.
  4. A set of expressions with subject-operator-object(s) style. In this case the custom-role label of a node should have worker value

As you can see, the syntax is much more expressive and provides better flexibility over node selection.

Verify

Apply the manifest and check that the Pod is still running on the worker node. Example:

# kubectl get pods -l app=echoserver -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP          NODE                     NOMINATED NODE   READINESS GATES
echoserver-75894b9c47-75n6b   1/1     Running   0          85s   10.0.1.94   k8s-worker.cloud.ut.ee   <none>           <none>

Complete

Let's setup a Pod anti-affinity for a second echoserver deployment. For this, create a similar manifest with a different affinity config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver-2
spec:
  selector:
    matchLabels:
      app: echoserver-2
  template:
    metadata:
      name: echoserver-2
      labels:
        app: echoserver-2
    spec:
      affinity:
        podAntiAffinity: #(1)
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector: #(2)
              matchExpressions:
              - key: app
                operator: In
                values:
                - echoserver
            topologyKey: "kubernetes.io/hostname" #(3)
...
  1. Inter-pod anti-affinity section for the deployment
  2. In this case, Kubernetes can't schedule a workload on a node, where Pods with the app=echoserver label are running.
  3. Specifies the context where rule applies: hostname means selection for nodes. In case of zone value, the Pod would be scheduled in a different set of nodes.

Verify

Apply the manifest and check if the new Pod is running on a different node then the previous one. Example:

kubectl get pods -o wide | grep "echoserver"
# echoserver-2-56bdc76fdd-8pl8w   1/1     Running   0             8m41s   10.0.0.227   labs-test.cloud.ut.ee    <none>           <none>
# echoserver-75894b9c47-75n6b     1/1     Running   0             27m     10.0.1.94    k8s-worker.cloud.ut.ee   <none>           <none>

Complete

Now you need to create an affinity rule for ghostfolio deployment: it should run on a node with custom-role=worker label

Introduction to Helm

In this section we cover Helm installation and basic management capabilities. Helm is a package manager for Kubernetes, which is used in the future labs, and provides:

  1. Manifests templating using golang templates. A set of manifest templates calls Chart. The installed manifests on a Kubernetes cluster after templating performed is Release.
  2. Chart packaging, dependency management and publication.
  3. Release versioning and rollback.

Complete

Install Helm tool on your control plane VM:

curl https://get.helm.sh/helm-v3.13.1-linux-amd64.tar.gz -o ./helm.tar.gz
tar -zxvf ./helm.tar.gz
cp linux-amd64/helm /usr/bin/helm
rm -rf linux-amd64/

Create a sample chart and have a look at the contents:

helm create sample
ls -lh sample/
# total 8.0K
# drwxr-xr-x. 2 root root    6 Oct 27 08:34 charts #(1)
# -rw-r--r--. 1 root root 1.2K Oct 27 08:34 Chart.yaml #(2)
# drwxr-xr-x. 3 root root  162 Oct 27 08:34 templates #(3)
# -rw-r--r--. 1 root root 2.2K Oct 27 08:34 values.yaml #(4)
  1. directory with dependency charts
  2. main file with chart settings: name, description, type (library or application), version, app version
  3. Kubernetes manifest templates
  4. values for templating

Remove the templates/ folder as we create templates ourself. Let's add the nginx deployment to the empty templates:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-chart
  labels:
    app: nginx-chart
    helm.sh/chart: sample
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: nginx-chart
      helm.sh/chart: sample
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        app: nginx-chart
        helm.sh/chart: sample
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          env:
            - name: "KEY"
              value: "VALUE"
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

As you can see, lot of values in the manifest are templated. This simplifies changing of common parts through a single values.yaml file.

Now, let's install the release:

helm install nginx-release sample/
# NAME: nginx-release
# LAST DEPLOYED: Fri Oct 27 09:09:37 2023
# NAMESPACE: default
# STATUS: deployed #(1)
# REVISION: 1 #(2)
# TEST SUITE: None
  1. Indicates the release is installed successfully.
  2. Each revision has its number and you can rollback to the desired one if needed.

Validate

After installation, you can list releases within the namespace:

helm ls
# NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
# nginx-release   default         1               2023-10-27 09:09:37.022773145 -0400 EDT deployed        nginx-test-0.1.0        1.16.0

and view the created Pod:

kubectl get pods -l helm.sh/chart=sample
# NAME                           READY   STATUS    RESTARTS   AGE
# nginx-chart-84f7fd9546-crcwt   1/1     Running   0          15s

Complete

Let's add a service to the deployment:

apiVersion: v1
kind: Service
metadata:
  name: "nginx-chart"
  labels:
    app: nginx-chart
    helm.sh/chart: sample
spec:
  type: {{ .Values.service.type }} #(1)
  ports:
    - port: {{ .Values.service.port }} #(2)
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: nginx-chart
    helm.sh/chart: sample
  1. Service type, ClusterIP in this case
  2. The same port as we use in the deployment

Then upgrade the chart:

helm upgrade nginx-release sample/
# Release "nginx-release" has been upgraded. Happy Helming!
# NAME: nginx-release
# LAST DEPLOYED: Fri Oct 27 09:24:16 2023
# NAMESPACE: default
# STATUS: deployed
# REVISION: 2 #(1)
# TEST SUITE: None

Validate

You can view the created service and check it works:

kubectl get service -l helm.sh/chart=sample
# NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
# nginx-chart   ClusterIP   10.107.82.204   <none>        80/TCP    62s
curl 10.107.82.204
# <!DOCTYPE html>
# <html>
# ...