Lab 3 - Kubernetes setup

Overview¶

Welcome to the third lab of the Cloud Native Applications on Kubernetes course. This is the lab where you finally get started on Kubernetes:

This week's tasks:

• Install Kubernetes
  • Setup the cluster management software
  • Install control plane
  • Install client tools
  • Install the CNI
• Setup two VMs for the worker nodes
  • Install the worker nodes
• Test Kubernetes by creating a single container
• Test networking by creating a single service
• Detailed usage of client tools

Kubernetes¶

What's Kubernetes?¶

The containerization lab showed that running a container is a relatively simple task. However, running containers across multiple hosts, scaling them up or down, managing, and deploying, and publishing them to the outside network reliably, among other things, can be difficult without some extra tools.

Kubernetes is a tool that aims to solve all these problems by defining a set of API endpoints and resource objects, and asking all software, hardware and tools to confer to those definitions. This allows the integrations between tools be fully intended and documented out of the gate, as the integrations follow the same API definitions.

Then, Kubernetes defines a set of components it needs, to provide these API endpoints and track the defined resource objects, convert them into actionable items on a lower abstraction layer (container engine and OS), and run these workloads and tasks on those.

Terminology¶

There is a set of terms that Kubernetes basically redefines, and some new ones. As a way to prevent confusion, here's a list of the worst offenders together with most used definitions in Kubernetes:

• Pod - group of one or more containers that share a network. Lowest tier of workloads in Kubernetes.
• CP or control plane - administrative Kubernetes layer that runs the most important Kubernetes components.
• worker - a compute unit that doesn't run any control plane components.
• Node - in Kubernetes, a node is a physical or virtual machine that can run a workload. It doesn't matter if it's control plane or worker.
• cluster - in Kubernetes context, this means a set of nodes (at least 1 CP), that have been aggregated into the same failure and trust domain by the Kubernetes components.
• CNI - Container Network Interface. A standard that facilitates communication between container run-times and network plugins. Often also used to mean the networking software that uses the standard to provide networking for the Kubernetes cluster.
• Service - A Kubernetes resource object that defines a logical set of Pods and a policy to access them.
• Namespace - A Kubernetes resource object that allows for the segregation of cluster resources between multiple users or environments.
• Ingress - in Kubernetes context, usually means the Kubernetes resource object Ingress, which defines IngressController routes.
• API server - in Kubernetes context, this usually means the Kubernetes control plane component called kube-apiserver.
• Labels and Selectors - in Kubernetes context, allow for organizing and selecting subsets of objects based on key-value pairs.
• kubectl - a command-line tool for interacting with Kubernetes clusters.
• scheduler - in Kubernetes context, this usually means the Kubernetes control plane component called scheduler.

Kubernetes components¶

As shown in the lecture, a working Kubernetes cluster consists of multiple components working in unison.

kube-apiserver¶

Acts as the main management point of the entire cluster. Handles and processes REST requests and updates object states in etcd. You interact with this using kubectl, and all other cluster components and utilities are tied together by this component.

In your case, it's going to run on only the first node you set up - control plane node - as a container, exposed on port 6443.

etcd¶

Stores configuration data of the cluster, acting as the “source of truth” for the cluster. Usually run in a distributed method, where you have multiples of the component in a single cluster, so a single failure could not bring the cluster down.

In your case, it's going to run on only the first node you set up - control plane node - as a container.

scheduler¶

Constantly probes the API server for unscheduled workflows (pods) and assigns them onto worker nodes, based on specific criteria. Considers factors like individual and collective resource requirements, hardware/software/policy constraints, data locality, inter-workload interference, and deadlines while scheduling.

In your case, it's going to run on only the first node you set up - control plane node - as a container.

controller-manager¶

Constantly probes the API server for the defined cluster state, and compares it against the current cluster state. If a change is detected, works to rectify any divergences between the states.

In your case, it's going to run on only the first node you set up - control plane node - as a container.

kube-proxy¶

Responsible for handling network communication both cluster internally, and also externally. Usually handles firewall rules, node-to-node routing, pod-to-pod routing, OS port to pod routing.

In your case, it's going to run on all the nodes as a container.

kubelet¶

Responsible for listening for commands from the control plane, and “converting” the commands into local container runtime or operating system (network, filesystem) tasks. Also handles node registration, pod creation/deletion, and status reporting, by continuously collecting and sending runtime metrics to control plane.

In your case, it's going to run on all the nodes, as a systemd unit.

Installing Kubernetes¶

In the labs, you're going to use the so-called "vanilla" Kubernetes distribution, which uses Kubeadm as management tooling.

Using this tooling, you can install, configure, update, and manage your Kubernetes cluster.

Preparing the nodes¶

Before you can go onward with installing Kubernetes, there's a few you need to complete on the existing node.

You're going to use the machine you setup in labs 1 and 2 as the Control Plane node, meaning it runs all the cluster components. Later on, you'll be adding a two more VMs to act as worker nodes.

In the previous labs you setup containerd in a kind-of tied down form. You'll now need it to work with Kubernetes, so there's a few things to change.

containerd config default | sudo tee /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  BinaryName = ""
  CriuImagePath = ""
  CriuPath = ""
  CriuWorkPath = ""
  IoGid = 0
  IoUid = 0
  NoNewKeyring = false
  NoPivotRoot = false
  Root = ""
  ShimCgroup = ""
  SystemdCgroup = true // (1)

[plugins."io.containerd.grpc.v1.cri"]

   sandbox_image = "registry.k8s.io/pause:3.9"

Once this has been completed, you can continue with setting up the rest of the node, preparing it for Kubernetes.

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Now that the node has been prepared, you can continue with setting up the Kubernetes management software - kubelet.

Installing Kubeadm, kubelet and kubectl¶

Kubeadm is a tool built to provide best-practice fast paths for setting up Kubernetes clusters. It performs the necessary steps to get a minimum viable cluster.

By design, it cares only about bootstrapping the cluster - it's not meant for provisioning machines, installing plugins (mandatory nor nice-to-have), nor monitoring solutions.

This methodology is perfect for running a self-contained bloatware-less cluster, which follows the best practices and is understandable for the admins.

You'll need to install the main three binaries to get Kubeadm ready to create a cluster:

• kubeadm - the Kubeadm binary itself.
• kubelet - the node agent that each Kubernetes node needs to run.
• kubectl - the Kubernetes command-line tool for simplifying administrative access to the cluster.

# This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

dnf install kubelet-1.30.5-150500.1.1.x86_64 kubeadm-1.30.5-150500.1.1.x86_64 kubectl-1.30.5-150500.1.1.x86_64 --disableexcludes=kubernetes

Once you have a running (well, technically crashlooping) kubelet service, you can start with creating a Kubernetes cluster.

Create a Kubernetes cluster¶

There are two ways to initialize a Kubernetes cluster - a quickfire way over the command line, or by specifying a configuration file.

You'll be using a configuration file, as that option is more reproducible and understandable.

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
controlPlaneEndpoint: "<VM external IP>:6443"
networking:
  dnsDomain: cluster.local
  podSubnet: "10.217.0.0/16"
  serviceSubnet: "10.96.0.0/12"
apiServer:
  extraArgs:
    advertise-address: "<VM external IP>"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  name: "<your vm name>.cloud.ut.ee"
  criSocket: "unix:///run/containerd/containerd.sock"
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: "systemd"
shutdownGracePeriod: 5m0s
shutdownGracePeriodCriticalPods: 5m0s
---

kubeadm init --config=<config location>/<config file>.yaml

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join 172.17.1.1:6443 --token abcd12.r7vsu1keh193typx \
        --discovery-token-ca-cert-hash sha256:831d55931749cb150f60594ecf4631b877a865e8fea2873834de2b35ec2ec002 \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.17.1.1:6443 --token abcd12.r7vsu1keh193typx \
        --discovery-token-ca-cert-hash sha256:831d55931749cb150f60594ecf4631b877a865e8fea2873834de2b35ec2ec002

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

You should now follow the three first commands in the output so you could use the kubectl tool without too much problems. After doing the mkdir, cp and chown commands, you should be able to run kubectl get nodes, and see a response something like this:

[root@teacher-test ~]# kubectl get nodes
NAME                       STATUS   ROLES           AGE   VERSION
teacher-test.cloud.ut.ee   Ready    control-plane   39s   v1.30.5

Basics of the command line tool¶

You'll mainly use the kubectl command line tool, at least in the first part of the course, to manage the cluster. This is because kubectl is the default and main way Kubernetes is interacted with.

kubectl is a command-line tool that interacts with the API server of a Kubernetes cluster and allows you to deploy and manage applications, inspect and manage cluster resources, and view logs. It uses a configuration file, usually located at ~/.kube/config, to determine the cluster's details and authentication information.

The basic operations include, but are not limited by:

• kubectl get - get an object in the API, for example kubectl get nodes.
• kubectl describe - describe an object and their metadata in the API, for example kubectl describe nodes.
• kubectl apply - apply a resource manifest into the API, for example kubectl apply -f manifest.yaml.
• kubectl delete - delete a resource from the API, for example kubectl delete -f manifest.yaml.
• kubectl edit - edit an existing resource in the API, for example kubectl edit service/kubernetes.

Another thing to keep in mind that Kubernetes has two versions of resources - cluster wide, and namespaced resources.

Cluster wide resources are usable across the whole cluster. Good examples are ClusterRoles - roles which give access to cluster-level resources. You can view them with kubectl get clusterrole. By default, there's quite a few, which are already required for base Kubernetes to work.

On the other hand, if you try to view pods with kubectl get pods, the output is going to be No resources found in default namespace.. But you already have control plane components running inside the cluster. How can that be?

The answer is, that the control plane components live inside a namespace called kube-system. You can view the pods in this namespace by doing kubectl get pods -n kube-system. This lists you all the main components a Kubernetes consists of, and because of this, kube-system is the most important namespace for Kubernetes.

Namespaces are a way to divide cluster resources between multiple users, groups, and systems. By utilizing namespaces, the cluster administrators can "hide away" all system level resources from users, and give them access only to the ones they care about. This is what makes Kubernetes such a wonderful orchestration tool - you can give each user or team their own namespace, and they will feel like they're alone in the cluster.

Installing a CNI¶

You setup a container networking interface (CNI) in the second lab to enable networking for your containers. A Kubernetes CNI is similar, but way more complicated. The main purpose of CNI in Kubernetes is:

• Giving a network address to all the resources like pods, services, endpoints.
• Connecting these networks together across different hosts, by utilizing complicated network technologies like overlay networks or encapsulation.
• Making the resource addresses routable - connectable - for all other resources and hosts in the Kubernetes cluster.
• Usually also allowing for firewall rules, in Kubernetes with NetworkPolicy objects, which are going to be covered in later labs.
• Providing egress, meaning outgoing, network connectivity for all containers.

There are many different CNI systems available for Kubernetes, and the one you use depends on your Kubernetes distribution, OS, hardware, software, requirements and many other things. Good examples are:

• flannel - most simple CNI, easiest to get running.
• calico - most configurable, secure and performant CNI for quite a long time, but now superseded by..
• cilium - an extremely secure, configurable and performant CNI that also supports many advanced features like border gateway protocol (BGP), direct server return, cluster meshing, and so on.

You'll be using cilium in this course, as it provides some very good observability tools, which are going to make understanding and debugging the cluster easier.

Cilium quick installation isn't a very pleasant way of installing software into Kubernetes, nor is it the recommended way, but as the main way of packaging software, called helm, basically requires it's own weekly topic, you'll start off with the ugly - but quick - way.

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

/usr/local/bin/cilium install --version 1.16.1 --set hubble.ui.enabled=true --set hubble.relay.enabled=true --set operator.replicas=1 --set hubble.relay.tls.server.enabled=false --set hubble.tls.enabled=false

Setup worker VM¶

Similarly to how you setup a VM in ETAIS self-service, you'll also have to setup another two worker nodes. You don't have install consul or go through with giving scoring access though.

Joining the worker nodes to the cluster¶

Similarly to the initialization command, you're going to use Kubeadm to join the nodes to the cluster as well. But before that, you need to obtain some kind of authentication method for the worker nodes to authenticate to the control plane.

There are several methods a cluster administrator can use to do this, but you'll use the default one - Token-based discovery with CA pinning. This means that the control plane generates a token, with which the worker node makes an initial connection. The worker node validates the control plane node with the Kubernetes control plane CA certificate hash. If everything works, and initial connection succeeds, all nodes agree to a TLS client certificate, which to use for further operations.

Steps taken by a joining node, which are the worker nodes in your case:

• Joining node's Kubeadm downloads necessary cluster information from the API server. It uses the bootstrap token and the CA key hash to verify the authenticity of that data.
• Once the cluster information is known, kubelet can start the TLS bootstrapping process.
• The TLS bootstrap uses the shared token to temporarily authenticate with the Kubernetes API server to submit a certificate signing request (CSR); by default the control plane signs this CSR request automatically.
• Finally, Kubeadm configures the local kubelet to connect to the API server with the definitive identity assigned to the node.

If instead of a worker node, you'd be adding a control plane node, the following extra steps would be performed:

• Downloading certificates shared among control-plane nodes from the cluster.
• Generating control-plane component manifests, certificates and kubectl configuration file.
• Adding a new local etcd data store member.

kubeadm token create --print-join-command

kubeadm config print join-defaults

apiVersion: kubeadm.k8s.io/v1beta3
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: kube-apiserver:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: teacher-worker-a.openstacklocal
  taints: null

discovery:
  bootstrapToken:
    caCertHashes:
    - sha256:abcasdasdasd-98adfsg-09uasdoiphasoip0gha0s9dygu

---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: "systemd"
shutdownGracePeriod: 5m0s
shutdownGracePeriodCriticalPods: 5m0s
---

kubeadm join --config join.yaml

Something to notice, also, is that the worker node has no access to the cluster-wide administrative kubectl configuration file. This is by design, and this means that while you can do a lot of bad with a compromised Kubernetes worker, you can't fully take over the control plane, at least not without any extra vulnerabilities.

When you now check which pods are created in the cluster, kubectl get pods --all-namespaces to view all namespaces, you can see that some pods are instantly started on the worker node, the most important of which are cilium for networking, and kube-proxy to allow for service access, if some workload would require it.

Test Kubernetes¶

As a way to test that all the parts of Kubernetes work together, you'll run a basic container - echo - inside a pod. The container is simple - it listens for HTTP requests, and prints back all the information it receives as the response to the HTTP request. It'll be helpful later on to debug the Kubernetes, networking and also the applications.

Deploying objects to Kubernetes can be done in two ways - in an ad-hoc, single kubectl command kind of way, for example: kubectl run echoserver --image=registry.hpc.ut.ee/mirror/ealen/echo-server:latest --port=80.

That's not a good way though, as it's not declarative. Instead, do it like intended - create an object manifest, a YAML file, with the resource definition inside, and use kubectl apply on it. This makes also managing the workload easier, as if you make a mistake, you can edit the file, and run kubectl apply again, to bring the cluster into correct state.

apiVersion: v1
kind: Pod
metadata:
  name: echoserver
  labels: # (1)
    app: echoserver
spec:
  containers:
    - name: echoserver
      image: registry.hpc.ut.ee/mirror/ealen/echo-server:latest
      ports:
        - containerPort: 80
      env:
        - name: PORT
          value: "80"

While now you have a working singular workload, it's very hard-coded. Every time you delete and re-create the pod, it'll receive a new IP address. That's not a very pleasant way to use anything.

Solving this requires you to also define another resource, called a Service. A service is a logical abstraction deployed on top of pods, which allows a group of pods to be given a single IP address, and domain name. You'll be using Service type ClusterIP, which gives the pods it points to a single static IP address (and domain name) inside the cluster.

apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: ClusterIP
  selector:
    app: echoserver # (1)

Test networking¶

As a test to see whether the networking works properly, you can create a way to access cilium's hubble component from outside the machine. This component draws a graphical UI about traffic inside the cluster.

Doing this requires you to edit a Kubernetes service object, that was already created during the installation of cilium. The problem is, it was created in a service type ClusterIP, which makes the service available only inside the local Kubernetes cluster.

You need to change it into a NodePort service type, so each Kubernetes node allocates a service for this component, and internally routes traffic so that whichever node you connect to, the traffic always hits the component - even if you connect to the worker node, while the component actually runs on control plane.

Advanced client tools¶

As preparation, you'll also go over more useful kubectl functionalities, and also have a look into more intuitive ways to manage the cluster.

kubectl¶

As said before, kubectl is the main management tool for Kubernetes clusters. While it's just basically a translator, converting the command line calls into REST API queries, and appending the authentication configuration to the queries, it's still an extremely powerful and extensible tool.

You can start exploring the whole REST API with kubectl by doing: kubectl get --raw /

The --raw option, instead of converting commands, basically becomes glorified curl which sends requests to specific paths you specify. As a response, you can see the list of all the valid endpoints. Going further down, you'll be able to see the different endpoints and settings you can query from the API.

You can also see which commands are run by appending --v=9 to any kubectl command, similarly to kubeadm. This is going to show you the full curl command executed by the kubectl binary for any command, together with the request and response headers.

You can get the whole list of resources manageable with kubectl with the command kubectl api-resources.

You can also:

• Copy files into containers with: kubectl cp --tar /my/local/directory mypod:/my/pod/directory
• Create kubectl plugins, by creating files in $PATH with the prefix kubectl-.
• For example, a file /usr/local/bin/kubectl-hello could be executed through kubectl with the command kubectl hello.
• Read container logs: kubectl logs pod/<podname>
• Compare current resource state to a file: kubectl diff -f <filename>.yaml
• Port forward a pod or service from the cluster to your local machine: kubectl port-forward svc/my-service 8080:80
• This is useful only if you run the kubectl on your personal computer, which you can easily do.
• Then you can access resources in the cluster easily, without having to expose them to the world.

Kubernetes permission system¶

Even more advanced, kubectl can tell you whether the user you're using has enough permissions to access a resource:

kubectl auth can-i list pods --namespace=default

Currently you're using the cluster administrator user, so obviously you can, but if you have normal user permissions, then it's very easy to use this methodology, without changing anything.

And in case you're, instead, a cluster administrator, and trying to show or prove to a user that they can access, you can impersonate their user or group.

The following command checks whether a random, non-existent user username could query the pods in default namespace when in a group called system:authenticated. This group is something anyone, who has authenticated with the cluster, has. You'll notice that it says it can't list pods in the default namespace:

kubectl auth can-i list pods --namespace=default --as=username --as-group=system:authenticated

Instead, check if it can create a SelfSubjectRulesReview object:

kubectl auth can-i create SelfSubjectRulesReview --as=username --as-group=system:authenticated

grep client-certificate-data <kubectl-config>.conf

grep "client-certificate-data" <kubectl-config>.conf | awk '{print $2}' | base64 -d

openssl x509 -in <certificate_file> -noout -text

k9s¶

kubectl is a nice and powerful tool, but Kubernetes tends to grow very quickly. Larger clusters nowadays have up to 100 thousand nodes. That's too much text for a person to handle normally.

Thankfully, there are visual methods available, as well. One of them is still a command-line based, but visual tool, called k9s. This is a tool that still runs inside the terminal, but makes searching and viewing the cluster much more simple. A demonstration (click twice to play):

The problem with this tool, as with most command-line tools, is that you need to know the shortcuts and commands to be efficient with it, but once you are, you'll be unstoppable. Installation is fairly straightforward, and if everything else fails, you can just download the binary from their GitHub page, and run that.

OpenLens¶

Another, even more graphically pleasant way is something called Lens. This is a full Kubernetes IDE, which runs as separate software inside your personal computer.

lens does not make it much easier to create workflows, but finding/viewing/searching/managing/editing/scaling/deleting resources is much more intuitive than through kubectl, especially if you have a bunch of them. It also supports useful plugins and integrations, like directly integrating with the cluster's monitoring system (if you have one) to make metrics easily available.

Due to licensing reasons, there's two versions of it. Lens, which came earlier, but is proprietary, and OpenLens, which is a free open source fork of it. Installation requires you to go to the GitHub page, and install it from there.