Lab 13 - Kubernetes Horizontal Pod Autoscaler

Complete

Follow the steps in Lab 3 to add a new worker node.

Use m1.r6c4 for the VM type and set the System Volume size to be 50 GB.

Make sure to install the same Kubernetes version you used in lab 11 (1.28.4) when upgrading Kubernetes. Also make sure to assign label custom-role=worker to the new worker node. We did this in lab 8. NB! Also make sure to select system volume type to be scratch (IOPS intensive SSD). Otherwise may run into difficult to debug disk errors.

Also, complete the Preparing the nodes for Longhorn part of Lab 6 to make sure the node becomes available for Longhorn storage.

Complete

Download the metrics server deployment manifest:

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Before we deploy it, we need to change some parameters inside the manifest.

By default, the Metrics server is configured to connect to node kublet through verified TLS.

However, as we generated our server certificate ourselves during installation, they are not verifiable through globally trusted root certificates. We will enable insecure tls for the metrics server to get pass this, but in production clusters, a verifiable certificates should be used or the used certificate authority should be added to the listed of trusted root certificates in the cluster.

Also, we need to modify how the Metrics server generates the connection urls, and enable only Hostname based address types, as other options will likely result in failed requests in our setup.

Modify the downloaded components.yaml file and change the container args in the Deployment kind template.

Add --kubelet-insecure-tls to the container args.

Modify --kubelet-preferred-address-types to only include Hostname option.

Apply the components.yaml manifest with kubectl.

Complete

The metrics server will be deployed inside the kube-system namespace.

Verify that the mertics server Pod starts up properly.

Complete

Create a new namespace hpa-test.

Deploy the following manifest in the hpa-test namespace:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      run: php-apache
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: registry.k8s.io/hpa-example
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 200m
          requests:
            cpu: 100m
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  ports:
  - port: 80
  selector:
    run: php-apache

This will deploy a simple PHP web app in a Apache web server container. One thing to note is that it has no ReplicaSet configured.

Complete

Next, we will create a HPA configuration for enabling automatic scaling of this deployment.

Create the folowing manifest for the HPA and apply it.

This will configure the HPA controller to track the performance of the php-apache deployment.

We configure the Target metric to be CPU utilization and configure the automatic scaling target to be avaerage 50% CPU utilization.

The HPA controller will try to keep the average CPU utilization below 50%, while adding Pod replicas when it increases above this target and removing replicas when the average CPU utilization drops below it.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: hpa-test
spec:
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
       averageUtilization: 50
        type: Utilization
    type: Resource
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache

Complete

Create a new HPA for the Ghostfolio deployment we have earlier set up in the default namespace. NB! Not for the Ghostfolio Helm deployment in the lab9 namespace!

Set the number of minimum replicas to 1, maximum replicas to 10, and average CPU utilization to 60%.

If you notice that Ghostfolio scaling is too agressive, and want to configure the behaviour of scaling, check the Configuring scaling behavior guide in the Kubernetes HPA documentation.

One reason for default configuration not working perfectly is that it takes some time before Ghostfolio is verified to be in ready state (we configured readyness probes).

You can change the default values of the HPA Policy, including how long the scaling trigger event (CPU higher than 60% in our case) must last before the HPA controller takes action, what is the duration that is used for computing the average, how many Pods can be added or removed at once, how often scaling descisions can be initiated, and several other things.

Complete

Cleanup the hpa-test namespace, remove the deployment, service and php-apache HPA configuration.