Introduction¶

Welcome to the lab 10. In this session, the following topics are covered:

• Different deployment strategies for the Kubernetes applications;
• First steps in using operators.

Deployment strategies¶

For upgrade strategy practice, we shall follow an excellent tutorial partially replicated and adjusted here for convenience of following.

First, let's review what common strategies exist in the first place.

• recreate: terminate the old version and release the new one;
• ramped: release a new version on a rolling update fashion, one after the other;
• blue/green: release a new version alongside the old version then switch traffic;
• canary: release a new version to a subset of users, then proceed to a full new deployment;
• a/b testing: release a new version to a subset of users in a precise way (HTTP headers, cookie, weight, etc.). This doesn’t come out of the box with Kubernetes, it implies extra work to setup a smarter load-balancing system (Istio, Linkerd, Traeffik, custom NGINX/HAproxy, etc);
• shadow: release a new version alongside the old version. Incoming traffic is mirrored to the new version and doesn't impact the response.

Monitoring setup and demo application.¶

We are going to reuse Prometheus and Grafana applications deployed in Lab 7 for monitoring impact of different deployment strategies and load a custom dashboard for tracking demo application request processing. Make sure that dashboard is using existing Prometheus data source. After import it uses a hard-coded data source identifier, so you need to update dashboard and set a correct one.

Create a new custom Grafana dashboard by creating a new empty dashboard and overwriting it's JSON to the following custom dashboard JSON. Make sure that dashboard is using an existing Prometheus data source. If you run into issues when importing, change the version number (for example, to 1), and you can also try changing the uid. If you keep the current uid, it is possible a second new dashboard is created instead of modifying your newly created one.

Check that a new Grafana dashboard with name k8s-deployment-strategies dashboard was created.

sum(rate(http_requests_total{app="my-app"}[20s])) by (version)

A demo application is an echo server exposing additional endpoints for serving Prometheus metrics and live and readiness endpoints. Please check the code of the demo application as well. Demo application accept environment variable VERSION that emulates version of the application.

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

kubectl rollout restart statefulset prometheus -n monitoring

kubectl logs -n monitoring prometheus-0 --tail 50
# check if you see any message about disk being full

Recreate strategy¶

The first strategy is very straightforward. To upgrade a deployment, we first remove old and then add a new one.

Let's first deploy a v1 app and wait for it to finish loading. Please note that we create 3 replicas of a service, and we shall deploy it on a NodePort service - dedicate the host port 30111 for this.

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: http
    nodePort: 30111
  selector:
    app: my-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9101"
    spec:
      containers:
      - name: my-app
        image: containersol/k8s-deployment-strategies
        ports:
        - name: http
          containerPort: 8080
        - name: probe
          containerPort: 8086
        env:
        - name: VERSION
          value: v1.0.0
        livenessProbe:
          httpGet:
            path: /live
            port: probe
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: probe
          periodSeconds: 5

Please save the configuration from above as app-v1.yaml. Let's create a service and a deployment and wait till they are working.

After that, let's generate some basic load on the system (replace k8s_node_ip with IP of your Kubernetes node). Let the generator run during the whole time in a separate shell.

for i in `seq 100000`; do curl http://k8s_node_ip:30111/; sleep 0.1; done

Now create a copy of the application manifest, for example app-v2.yaml, change manifest and set VERSION variable to v2.0.0 and app version label to v2.0 and deploy again.

$ kubectl apply -f app-v2.yaml 
service/my-app unchanged
deployment.apps/my-app configured

Host: my-app-599bdf7545-q7j42, Version: v1.0.0
curl: (7) Failed to connect to 172.17.89.232 port 30111 after 101 ms: Couldn't connect to server
curl: (28) Failed to connect to 172.17.89.232 port 30111 after 43643 ms: Couldn't connect to server
Host: my-app-7cdf7685d9-qpskd, Version: v2.0.0

As a result of this, your Grafana chart should show something similar to this:

Ramped strategy¶

We saw that plain removal and deployment of a new version leads to a downtime. However, in ideal case we want our application to be highly available at all times.

Ramped strategy is an approach where instances of a new version are gradually loading and replacing old versions.

Generally speaking there are 3 parameters that can be tuned:

• Parallelism, max batch size: Number of concurrent instances to roll out.
• Max surge: How many instances to add in addition of the current amount.
• Max unavailable: Number of unavailable instances during the rolling update procedure.

kubectl get pods |grep my-app | wc -l
10

  ...
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  ...

# In case you discover some issue with the new version, you can undo the rollout
$ kubectl rollout undo deploy my-app

# If you can also pause the rollout if you want to run the application for a
# subset of users
$ kubectl rollout pause deploy my-app

# Then if you are satisfied with the result, resume rollout
$ kubectl rollout resume deploy my-app

Monitoring in Grafana should also show a more gradual transition from old version to a new one.

Blue-green deployment¶

The blue/green deployment strategy differs from a ramped deployment, version B (green) is deployed alongside version A (blue) with exactly the same amount of instances. After testing that the new version meets all the requirements the traffic is switched from version A to version B at the load balancer level.

In this lab we shall try out an approach for blue-green update strategy using targeting of traffic flow in Service.

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: http

  # Note here that we match both the app and the version
  selector:
    app: my-app
    version: v1.0.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-v1
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: v1.0.0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9101"
    spec:
      containers:
      - name: my-app
        image: containersol/k8s-deployment-strategies
        ports:
        - name: http
          containerPort: 8080
        - name: probe
          containerPort: 8086
        env:
        - name: VERSION
          value: v1.0.0
        livenessProbe:
          httpGet:
            path: /live
            port: probe
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: probe
          periodSeconds: 5

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-v2
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: v2.0.0
  template:
    metadata:
      labels:
        app: my-app
        version: v2.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9101"
    spec:
      containers:
      - name: my-app
        image: containersol/k8s-deployment-strategies
        ports:
        - name: http
          containerPort: 8080
        - name: probe
          containerPort: 8086
        env:
        - name: VERSION
          value: v2.0.0
        livenessProbe:
          httpGet:
            path: /live
            port: probe
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: probe
          periodSeconds: 5

# kubectl get pods | grep my-app
my-app-v1-599bdf7545-2t8sm      1/1     Running   0             7m36s
my-app-v1-599bdf7545-5wvnc      1/1     Running   0             7m36s
my-app-v1-599bdf7545-fzdfh      1/1     Running   0             7m36s
my-app-v2-7cdf7685d9-452qz      1/1     Running   0             46s
my-app-v2-7cdf7685d9-8ttps      1/1     Running   0             46s
my-app-v2-7cdf7685d9-hkx49      1/1     Running   0             46s

Host: my-app-v1-599bdf7545-2t8sm, Version: v1.0.0
Host: my-app-v1-599bdf7545-fzdfh, Version: v1.0.0
Host: my-app-v1-599bdf7545-2t8sm, Version: v1.0.0
Host: my-app-v1-599bdf7545-2t8sm, Version: v1.0.0
Host: my-app-v1-599bdf7545-fzdfh, Version: v1.0.0

selector:
  app: my-app
  version: v1.0.0

kubectl patch service my-app -p '{"spec":{"selector":{"version":"v2.0.0"}}}'
````

Note the effect on the load generator:

```bash
...
Host: my-app-v1-599bdf7545-fzdfh, Version: v1.0.0
Host: my-app-v1-599bdf7545-2t8sm, Version: v1.0.0
Host: my-app-v2-7cdf7685d9-8ttps, Version: v2.0.0
Host: my-app-v2-7cdf7685d9-8ttps, Version: v2.0.0
...

Canary deployment¶

Canary deployment is a strategy of rollout where application traffic is split between already deployed instances and a new version. New version can be targeted for specific users, in this case strategy is also referred to as A/B testing, or can be used to gradually introduce new logic by limiting the damage it can cause by reducing the number of requests served by a new version (aka canary).

Canary deployment allows to check if new version is buggy and provides easy way of removing it from serving the load.

Let's use the same manifests as for the blue-green deployment, but adjust service and remove version from the selector. Don't forget to remove previous deployments for clean testing.

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app: my-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-v1
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
      version: v1.0.0
  template:
    metadata:
      labels:
        app: my-app
        version: v1.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9101"
    spec:
      containers:
      - name: my-app
        image: containersol/k8s-deployment-strategies
        ports:
        - name: http
          containerPort: 8080
        - name: probe
          containerPort: 8086
        env:
        - name: VERSION
          value: v1.0.0
        livenessProbe:
          httpGet:
            path: /live
            port: probe
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: probe
          periodSeconds: 5

After applying the manifest, we shall get 3 instances of app v1 working and processing requests. Let's now introduce 1 instance of app v2.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-v2
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
      version: v2.0.0
  template:
    metadata:
      labels:
        app: my-app
        version: v2.0.0
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9101"
    spec:
      containers:
      - name: my-app
        image: containersol/k8s-deployment-strategies
        ports:
        - name: http
          containerPort: 8080
        - name: probe
          containerPort: 8086
        env:
        - name: VERSION
          value: v2.0.0
        livenessProbe:
          httpGet:
            path: /live
            port: probe
          initialDelaySeconds: 5
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: probe
          periodSeconds: 5

Your pods should now look something like this:

$ kubectl get pods | grep my-app
my-app-v1-599bdf7545-bfjmf      1/1     Running   0             5m55s
my-app-v1-599bdf7545-k886m      1/1     Running   0             5m55s
my-app-v1-599bdf7545-z29vd      1/1     Running   0             5m55s
my-app-v2-7cdf7685d9-gxmtr      1/1     Running   0             12s

And in Grafana you can observe that traffic now reaches instances with 2 different versions. Note that ~25% of the traffic goes to a new version.

Let's image that version is OK, that is canary is alive, and let's scale v2 to 10 instances:

kubectl scale --replicas=10 deploy my-app-v2

And finally, remove the v1:

kubectl delete deployment/my-app-v1
deployment.apps "my-app-v1" deleted

A/B and shadow¶

There are two more strategies used for rollout: A/B testing and shadow.

A/B testing can be seen as an advanced version of canary deployment, where users are segmented based, for example, on their geolocation or configured cookies.

Shadow deployment approach consists of mirroring incoming traffic and sending it both to old version and new one, whereas requests to the new version are not used for serving the responses to users, but rather observing behaviour of real traffic on a new version.

Both of these strategies require advanced capabilities to be present in the cluster, like service mesh, and are not used with native Kubernetes tools. You are welcome to investigate them on your own.

Ghostfolio task¶

Operator framework¶

Kubernetes Operators are Kubernetes native applications that aim to capture (and automate) the responsibilities of "human" operators who are responsible for managing and monitoring some specific software (for example, Kafka or PostgreSQL clusters) running in Kubernetes. Operators should include everything necessary to control the full lifecycle of Kubernetes native applications, from packaging, deploying to managing (logging, upgrading, backing up, etc.). They act as a control plane for a specific Kubernetes application, often extending Kubernetes base API and implement additional custom resources.

Using PostgreSQL operator¶

We will use the CloudNativePG operator for setting up PostgreSQL clusters and will configure our Ghostfolio to use it instead of the PostgreSQL StatefulSet we set up in lab4.

CloudNativePG extends the Kubernetes API defining the following custom resources: Backup, Cluster, PoolerScheduled, Backup. You will find additional info about the API here: https://cloudnative-pg.io/documentation/1.15/api_reference/.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-postgresql
spec:
  instances: 2
  primaryUpdateStrategy: unsupervised # Automated updating of primary once all replicas have been upgraded
  imageName: ghcr.io/cloudnative-pg/postgresql:15.4

https://github.com/cloudnative-pg/cloudnative-pg/blob/main/docs/src/quickstart.md

Building an operator¶

Operators can be created using Operator SDK. You are welcome to try out https://sdk.operatorframework.io/docs/building-operators/helm/quickstart/ on your own.