Lab 13 - Kubernetes Horizontal Pod Autoscaler

Complete

Export the default Helm values of the Prometheus adapter: helm show values prometheus-community/prometheus-adapter > adapter-all-values.yaml

Adjust the values inside the Yaml file. Change the prometheus.url value to correct service address in the prometheus namespace.

Also, uncomment all the lines in the resource: block. This will expose cpu and memory metrics for the HPA through the normal Kubernetes Metrics API metrics.k8s.io.

Install the prometheus-community/prometheus-adapter Helm chart. Set the version to 4.11.0, use prometheus namespace and use the prepared values file.

Complete

Create a new namespace hpa-test.

Deploy the following manifest in the hpa-test namespace:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      run: php-apache
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
      - name: php-apache
        image: registry.k8s.io/hpa-example
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 200m
          requests:
            cpu: 100m
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  ports:
  - port: 80
  selector:
    run: php-apache

This will deploy a simple PHP web app in a Apache web server container. One thing to note is that it has no ReplicaSet configured.

Complete

Next, we will create a HPA configuration for enabling automatic scaling of this deployment.

Create the folowing manifest for the HPA and apply it.

This will configure the HPA controller to track the performance of the php-apache deployment.

We configure the target metric to be CPU utilization and configure the automatic scaling target to be avaerage 50% CPU utilization.

The HPA controller will try to keep the average CPU utilization below 50%, while adding Pod replicas when it increases above this target and removing replicas when the average CPU utilization drops below it.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: hpa-test
spec:
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
       averageUtilization: 50
       type: Utilization
    type: Resource
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache

Complete

Double check that your History Server deployment has resoruce request and limit values set.

Create a new HPA for the History Server deployment we have earlier set up in the production namespace. Set the number of minimum replicas to 1, maximum replicas to 10, and average CPU utilization to 60%.

If you notice that History Server scaling is too agressive, and want to configure the behaviour of scaling, check the Configuring scaling behavior guide in the Kubernetes HPA documentation.

One reason for default configuration might not work perfectly is that it takes some time before History Server is verified to be in ready state (we configured readyness probes).

You can change the default values of the HPA Policy, including how long the scaling trigger event (CPU higher than 60% in our case) must last before the HPA controller takes action, what is the duration that is used for computing the average, how many Pods can be added or removed at once, how often scaling descisions can be initiated, and several other things.

Complete

Configure a new Custom metric in the Prometheus Adapter Helm values file under the rules.custom YAML block:

  custom:
  - seriesQuery: '{namespace!="",__name__=~"^application_server_requests_total$"}'
    resources:
      template: <<.Resource>>
    name:
      matches: ""
      as: "application_server_requests_per_second"
    metricsQuery: avg(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)    

This configuration will look for Prometheus metrics named application_server_requests_total, apply average and rate functions on them to compute average requests per second in a 2 minute window and register it as a new custom metric named application_server_requests_per_second. This custom metric will be available through the Prometheus adapter custom.metrics.k8s.io API.

Complete

Create a HPA manifest for the Application Server deployment and apply it. Because we are now using Custom metric, set the metric name to application_server_requests_per_second and metric type to be Pods. Use AverageValue type target for metric and assign desired metric value to 10 (10 requests per second).

Complete

Cleanup the hpa-test namespace, remove the deployment, service and php-apache HPA configuration.