Lab 11 - Cluster administration

Overview¶

• Tearing down existing monitoring
• Installing Helm versions of Prometheus, Grafana and Loki
• Kubernetes Audit logging
• Upgrading the cluster
• Using Kubeadm patches

Introduction¶

In this, Cluster administration lab, you'll delve into key aspects of monitoring, logging, and cluster management. You'll be destroying the manually installed monitoring layer, and leverage the power of new, fancy tools like operators and Helm to setup new, better versions. You'll be doing this, as understanding proper visibility into your cluster and applications is a large portion of complexity in microservices environments, and without it, it's impossible to understand your infrastructure.

This lab is going to also guide you through the process of upgrading a Kubernetes cluster, while trying to rely on best practices and real-world experience. While updates are generally important, they're especially so in Kubernetes, due to the frequency of new releases, and the way the whole Kubernetes ecosystem moves together with the versions.

Tearing down existing monitoring¶

As you are going to reinstall your monitoring stack in this lab, you can delete the old ones to make some space.

Installing Prometheus and Grafana with Helm¶

Initial setup of the Prometheus stack using Helm is fairly straightforward. You'll be installing a version called kube-prometheus-stack, which is a fairly complex system of Prometheus, Grafana, Alertmanager, node-exporter, kube-state-metrics and an adapter for Kubernetes Metrics API-s, last of which you'll need for Horizontal Pod Autoscaler, or HPA, in the last lab in the course.

The repository for the stack is located here: https://github.com/prometheus-operator/kube-prometheus

You can also take a look at the Helm repository: https://prometheus-community.github.io/helm-charts

helm show values prometheus-community/kube-prometheus-stack > prometheus.yaml

NAME: prometheus
LAST DEPLOYED: Mon Nov 20 09:12:12 2023
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace prometheus get pods -l "release=prometheus"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

Taking a look inside¶

When you now take a look inside your new Grafana instance, something you'll see is that Helm automatically setup quite a few rules, dashboards and alerts.

This is the base monitoring capacity a Kubernetes cluster should have nowadays, due to the simplicity of setting it up.

And this should be useful for answering any questions related to Kubernetes and it's related hosts.

As an exercise, you can try optimizing the workloads running in your cluster, to reduce CPU and memory consumption, by comparing actual resource consumption to the requests and limits.

Replicating previous lab's configuration with new stack¶

In the previous lab, you setup pod scrape monitoring with Prometheus to be able to detect and view how your application updates. As deleting the stack also deleted this configuration, replicate this with the new system.

In the new system, you could just take the Prometheus scrape configuration given to you in the previous lab, and shove it into the Helm values file. While this would work, the new Prometheus stack provides interfaces for doing this without writing any scrape configuration.

The kube-prometheus-stack introduces two new CRD-s for this purpose - ServiceMonitor and PodMonitor. These are automatically configured probes, that either go to probe services or pods. In previous lab's case, you configured Prometheus to directly scrape Pods, so this is what you need to configure now.

There's a middle step as well - because the kube-prometheus-stack is slightly opinionated on how things are supposed to work, you need to specify the scrape port with a port name, and move the scrape annotation to labels. In the previous lab, you did not configure a name for the scrape port (oversight on teachers part), so you'll need to do it now.

Configure Prometheus to scrape your application, by creating the PodMonitor object.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: port-9101
spec:
  selector:
    matchLabels:
      "prometheus.io/scrape": "true"
  podMetricsEndpoints:
  - port: metrics
    interval: 10s
    path: /metrics
    relabelings:
    - action: replace
      sourceLabels:
        - __meta_kubernetes_pod_label_app
      targetLabel: app
    - action: replace
      sourceLabels:
        - __meta_kubernetes_pod_label_version
      targetLabel: version
  namespaceSelector:
    any: true

- name: metrics
  containerPort: 9101

Install Loki with Helm¶

Loki is also much more straightforward to setup using Helm. It has been automated to install in a sensible fashion, and takes a lot of the difficult configuration out of the equation. From the Loki Helm chart documentation https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/:

  schemaConfig:
    configs:
      - from: 2024-01-01
        object_store: filesystem
        store: tsdb
        schema: v13
        index:
          prefix: index_
          period: 24h

loki:
  persistence:
    enabled: true
    storageClassName: longhorn
    size: 10Gi
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: 'filesystem'
  schemaConfig:
    configs:
      - from: 2024-01-01
        object_store: filesystem
        store: tsdb
        schema: v13
        index:
          prefix: index_
          period: 24h
deploymentMode: SingleBinary
singleBinary:
  replicas: 1
monitoring:
  selfMonitoring:
    enabled: false
test:
  enabled: false
serviceMonitor:
  enabled: false
write:
  # -- Number of replicas for the read
  replicas: 0
read:
  # -- Number of replicas for the read
  replicas: 0
backend:
  # -- Number of replicas for the backend
  replicas: 0
chunksCache:
  allocatedMemory: 1500

Installing Promtail¶

Installing Promtail is made simple with Helm, as well. The chart is in the same repository as Loki's, so it'll be very easy to setup.

level=error ts=2023-11-20T14:34:29.141649402Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): 12 errors like: entry for stream '{app=\"teacher-test.cloud.ut.ee\", component=\"kube-apiserver\", container=\"kube-apiserver\", filename=\"/var/log/pods/kube-system_kube-apiserver-teacher-test.cloud.ut.ee_ff01467ad8a05d6aad9e55f6149157dc/kube-apiserver/14.log\", job=\"kube-system/teacher-test.cloud.ut.ee\", namespace=\"kube-system\", node_name=\"teacher-test.cloud.ut.ee\", pod=\"kube-apiserver-teacher-test.cloud.ut.ee\", stream=\"stderr\"}' has timestamp too old: 2023-09-19T05:44:10Z, oldest acceptable timestamp is: 2023-11-13T14:34:29Z; 9 errors like: entry for stream '{app=\"teacher-test.cloud.ut.ee\", component=\"kube-apiserver\", container=\"kube-apiserver\", filename=\"/var/log/pods/kube-system_kube-apiserver-teacher-test.cloud.ut.ee_ff01467ad8a05d6aad9e55f6149157dc/kube-apiserver/14.log\", job=\"kube-system/teacher-test.cloud.ut.ee\", namespace=\"kube-system\", node_name=\"teacher-test.cloud.ut.ee\", pod=\"kube-apiserver-teacher-test.cloud.ut.ee\", stream=\"stderr\"}' has timestamp too old: 2023-09-19T00:44:52Z, oldest acceptable timesta"

Configuring Grafana to use Loki data source¶

Out of the gate, Grafana does not know how to connect to Loki, to query the logs. Thankfully, the developers of the kube-prometheus-stack Helm chart have thought about this problem, and have exposed a Helm values file setting called grafana.additionalDataSources, where you can configure other data sources Grafana should connect to.

Setting up audit logs¶

Kubernetes makes some things very easy with it's REST API based architecture, because HTTP protocol is inherently transactional. Transactionality makes it, theoretically, also very auditable, as you can always tell who did what kind of operation in the cluster. This kind of tracking is called auditing or audit logging, and is very useful for both debugging issues for cluster users (for example, why can't I access this?), and also security (for example, which user brought down the production database pod).

Audit logging needs to be configured manually - Kubernetes does not default to any kind of audit logging. This is because Kubernetes API (which is what writes the audit logs) gets a lot of requests, and logging all of them is necessary only in the most secure of installations, and is going to consume a lot of storage and IO.

Instead, Kubernetes allows you to write a policy, defining what and how to audit. In this section, you'll be setting up audit logging with the Google Kubernetes Engine audit policy, which is shown here: GitHub.

Now, this policy is definitely not perfect, but through their countless hours of running Kubernetes clusters, this is what they've settled on.

Enabling audit logging requires you to re-configure the cluster partially, as you'll need to feed some settings to the Kubernetes API server. This needs to be done only on control plane nodes.

kubectl get cm -n kube-system kubeadm-config

kubeadm init phase control-plane apiserver --config /root/clusterconfiguration.yaml

kubectl create cm kubeadm-config --from-file=ClusterConfiguration=/root/clusterconfiguration.yaml -n kube-system --dry-run=client -o yaml | kubectl apply -f -

Sending audit logs to Loki¶

These kinds of logs are only mildly useful when they stay on the host that they were emitted from. This is both because of security reasons - where if your node gets compromised, the attacker can delete the logs - and because of the fact, that these kinds of logs usually need to be stored for years.

In a real-world environment, you'd also not run the log aggregation system on the same system which produces the logs, for the same reasons as listed.

Configure Promtail to send the Kubernetes audit logs to Loki.

config:
  snippets:
    extraScrapeConfigs: |
      - job_name: kubernetes-audit
        pipeline_stages:
        - json: 
            expressions:
                user: user
                verb: verb
                objectRef:
                annotations:
                timestamp: requestReceivedTimestamp
        - timestamp:
            source: timestamp
            format: RFC3339
        - json:
            expressions: 
                resource:
                namespace:
                apiGroup:
            source: objectRef
        - json:
            expressions:
                decision: '"authorization.k8s.io/decision"'
                reason: '"authorization.k8s.io/reason"'
            source: annotations
        - json:
            expressions:
                username:
                groups: 
            source: user
        - labels:
            username:
            groups: 
            verb: 
            resource: 
            namespace: 
            apiGroup: 
            decision: 
            reason: 
        static_configs:
        - targets:
            - localhost
          labels:
            job: auditlog
            __path__: /var/log/kubernetes/audit.log

Tracing (optional)¶

Tracing is a method of monitoring and observing the flow of requests as they traverse through a distributed system. It records details of individual operations (called spans) that collectively form a trace, showing the entire lifecycle of a request across multiple services.

So, to reiterate:

• Span: A single unit of work in a system (For example an HTTP request, database query).
• Trace: A collection of spans that together represent the full lifecycle of a request.

Tracing is usually done in the context of an application, but more and more infrastructure level software also provide interfaces to trace their activities nowadays. Utilizing tracing requires, similar to other monitoring, to have something that sends or publishes the data, and a server that receives and allows to query the data.

There are a lot of systems that provide this functionality, but the one you'll be setting up is Jaeger, an open source, distributed tracing platform. The benefits of Jaeger for us is that it is meant to do tracing only, not a million other things on top of it, which keeps it simple and understandable.

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest

Once you have a running Jaeger server, the other step is to configure your applications to send data to Jaeger. This is called instrumenting your application, which is usually done via some libraries, depending on the programming language used: https://www.jaegertracing.io/docs/1.63/client-features/

Jaeger subscribes to the OpenTelemetry telemetry data model, which is a set of API, SDK, and tools to improve interoperability between different telemetry systems, including tracing. This means you need to use OpenTelemetry libraries to send traces to Jaeger.

OpenTelemetry supports two main instrumentation methods - manual and automatic:

• Manual is used to instrument your application, and export custom spans with custom data. Very useful in production environments where you want to have a lot of visibility into the context of an application, as you can control it very well.
• Automatic is an easier way to get the baseline amount of traces out of the system. It's easier to setup and get running, and provides a lot of information out of the gate, but providing custom visibility is a bit more difficult.

You will need to see whether automatic instrumentation is provided for your language, and decide whether to use that. You will need to import the correct libraries, and set the environment variables to export your traces. For example, in Python and Flask:

opentelemetry-instrumentation
opentelemetry-distro
opentelemetry-exporter-otlp
opentelemetry-instrumentation-flask
opentelemetry-exporter-jaeger

Upgrading Kubernetes¶

Kubernetes follows a 4-month release schedule, which means that not constantly upgrading Kubernetes means, you'll be behind in upgrades very soon. As new functionalities and existing software updates (for example Ingress controllers) are written to match with the latest Kubernetes version, you can very quickly find yourself in a situation where you can't install new software to the cluster because you're a version or two behind.

This was especially true with, for example, going from version 1.24 to 1.25, as a very massive and important security feature in Kubernetes, called PodSecurityPolicy, got fully deprecated, as old software that tried to deploy to new cluster couldn't, due to trying to deploy manifests to an API, that did not exist any more. Same with new software deploying to old clusters. This is why, in several Helm charts, you can still see references to PSP, which is short for PodSecurityPolicy.

This is a general theme with Kubernetes - each version might deprecate or remove specific resource definitions and API versions, and it's up to the cluster admins and software developers to manage this.

Generally, it's easiest to handle this by staying on the latest Kubernetes version, but you need to be careful and make sure all the software is capable of running on the latest version. Some software moves slower, either due to complexity or because the developers do not have enough man-power. One such example is Longhorn, which usually doesn't have a new version out by every new Kubernetes minor version release. Installing new version in that case can, potentially irrevocably (without backups), corrupt the data.

Kubernetes follows semantic versioning, where upgrades inside the same minor version are straightforward and compatible, and do not break API. Your cluster is currently on version 1.30.X. You'll be upgrading it to a newer minor version 1.31.Y (latest version currently), to go through and learn the process.

To see whether an API gets deprecated, carefully read through the Kubernetes v1.31: Planternetes release notes.

The documentation for upgrading in Kubernetes documentation is here: kubernetes.io

You're recommended to try to upgrade using this document, instead of the lab materials, as it gives a better real-world experience. Also, it might explain some concepts better.

The upgrade workflow at high level is the following:

1. Upgrade a primary control plane node.
2. Upgrade additional control plane nodes.
3. Upgrade worker nodes.

You need to only do steps 1. and 3., as you only have a single control plane node.

yum list --showduplicates kubeadm --disableexcludes=kubernetes

yum install -y kubeadm-'<version>-*' --disableexcludes=kubernetes

[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

These were the necessary steps to upgrade the control plane components. You'd run this on all the control plane nodes, before continuing forward, but this did nothing for the nodes themselves - your kubectl and kubelet are still not upgraded on the nodes.

Upgrading the node-level components in a way, where workloads don't experience downtime, requires all workloads and system level components to have at least 2 replicas. On top of this, you'd need at least multiple control planes, and enough space on other workers to completely be able to migrate the workloads from one node to the other ones.

The steps are usually the following:

1. Drain the node with kubectl drain <nodename> --ignore daemonsets. This is the so-called migration step, which just kills the pods on the node and starts them up on another. This is why workloads need at least 2 replicas, which are on different nodes.
2. Update the binaries, restart the kubelet. Verify everything works.
3. Uncordon (resume) the node with kubectl uncordon <nodename>.

You can play this through on the worker node, but in your case, upgrading control plane node is going to make the cluster experience downtime.

yum install -y kubelet-'<version>-*' kubectl-'<version>-*' --disableexcludes=kubernetes

systemctl daemon-reload
systemctl restart kubelet

Now, upgrade the worker node properly.

yum install -y kubelet-'<version>-*' kubectl-'<version>-*' --disableexcludes=kubernetes

systemctl daemon-reload
systemctl restart kubelet

kubectl uncordon <worker-node-name>