Lab 5 - Kubernetes Networking

Complete

Implement the HTTP API server application. You can use any programming language or technologies you want. If you are unsure which approach to use, it is recommended to use either Python FastAPI or Flask frameworks.

An OpenAPI specification for the API has also been prepared, so feel free to generate the HTTP API implementation in any suported languages and implement the logic of the HTTP methods. You can find the API specification here: History data service OpenAPI specification

The main requirement is that you use the exact endpoints that have been specifed and that the price data message JSON format is the following:

[
  {
    "estonian_time": "2024-10-01T00:00:00+03:00",
    "price": 5.01,
    "timestamp": 1727730000
  },
  {
    "estonian_time": "2024-10-01T01:00:00+03:00",
    "price": 3.21,
    "timestamp": 1727733600
  },
  {
    "estonian_time": "2024-10-02T00:00:00+03:00",
    "price": 78.57,
    "timestamp": 1727816400
  }
]

For the MinIO integration, use the official MinIO libraries: Software Development Kits. For example, the Python Quickstart Guide provides reasonable examples to start with if you chose to use Python. MinIO library methods put_object, get_object, bucket_exists, and make_bucket should be enough to implement everything needed.

MinIO connection authentication variables (ACCESS_KEY, SECRET_KEY, and MinIO URI should be read in from the enviorenment variables and not hard coded). Also, when storing the data in MinIO, you can use the value of the day variable (For example 2024-09-29) as part of the filename. Data should be stored in the price-data MinIo bucket. It would be best if the application also checks that the price-data bucket exists and creates it if it does not exist.

After the code is ready and tested, publish the Docker image to the Docker hub. Name of the Docker image in DockerHub should be history-server.

Complete

Configure the price fetcher script to read the address of the History data server from system enviorenment variable named HISTORY_SERVER_URI. We will later configure this value through the Kubernetes CronJob manifest when we create a Kubernetes Service for the history data server.

Inside the script, once the data has been collected from Elering API, prepare a JSON, and send a POST request (For example using the requests.post() method in Python) at the History data server API endpoint address HISTORY_SERVER_URI/api/prices?day={tomorrow}, where the tomorrow variable format is 2024-09-29.

Prepare the data in the same JSON format as shown in the previous section and attach it as the request body to the HTTP POST request. Hint: It is ok to replace Elering CSV API request with a similiar JSON request to avoid CSV to JSON parsing steps.

Complete

Inspect an IP address of the echoserver Service (Test Kubernetes from lab3):

kubectl describe service/echoserver -n default
# Name:              echoserver
# Namespace:         default
# ...
# IP:                10.106.242.153
# IPs:               10.106.242.153
# Port:              <unset>  80/TCP
# ...

The IP above is cluster-scoped IP meaning it is available only within the cluster.

Complete

In the production namespace:

1. create a history-server Deployment for the history server app using the image you have built previously;
2. create a ClusterIP-type Service (name - history-server, listening port - 80) for the Deployment and validate if it works using curl;
3. build the adjusted price-fetcher app, which sends price data to the history-server endpoint, and update the existing CronJob.

NB:

1. Please, use the following labels for the history-server Deployment: app=electricity-calculator, microservice=history-server. Make sure to add them both to the metadata of the Deployment and to the metadata of the pod template;
2. The history-server Deployment should use the MinIO credentials from the minio-secret to set up necessary system enviorenment variables, similarily how they were defined for the MinIO StatefulSet int he previous lab.
3. Update the price-fetcher CronJob specification and add a system enviorenment variable (env:) named HISTORY_SERVER_URI, which defines how to connect to the history server. Its value should be the Service address of the history server: http://history-server.production.svc.cluster.local:80

Hint: To reference the MinIO service use either minio:9000 or http://minio:9000, depending on whether the minio library (that you used) requires specifying protocol before the hostname or not. You can find the explanation in the service discovery section.

Complete

Let's create a NodePort service for the echoserver in the default namespace:

apiVersion: v1
kind: Service
metadata:
  name: echoserver-service-nodeport
  namespace: default
spec:
  type: NodePort
  selector:
    app: echoserver
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30001

It works as a previous service selecting Pods by app: echoserver label, but uses a different hostname (echoserver-service-nodeport) for discovery.

After inspecting it, you can see the network-specific data:

kubectl describe service/echoserver-service-nodeport
# ...
# IPs:                      10.109.11.56
# Port:                     <unset>  80/TCP
# TargetPort:               80/TCP
# NodePort:                 <unset>  30001/TCP
# ...

Complete

Let's create a PostgreSQL StatefulSet with this config:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql-hl-test
  namespace: test
spec:
  selector:
    matchLabels:
      app: postgresql-hl-test
  serviceName: postgresql-hl #(1)
  replicas: 2 #(2)
  template:
    metadata:
      labels:
        app: postgresql-hl-test
    spec:
      containers:
      - name: postgresql
        image: bitnami/postgresql:15.4.0
        env: #(3)
          - name: POSTGRESQL_USERNAME
            value: "test"
          - name: POSTGRESQL_DATABASE
            value: "test"
          - name: POSTGRESQL_PASSWORD
            value: "postgres-password"
        ports:
        - containerPort: 5432
          name: postgres-port
        volumeMounts:
        - name: postgresql-data
          mountPath: /bitnami/postgresql
      volumes: #(4)
        - name: postgresql-data
          emptyDir: {}

1. A name of the headless service created later
2. Number of PostgreSQL replicas
3. Environment variables for PostgreSQL database setup
4. PostgreSQL doesn't use any persistent storage for now

Also, let's create a headless service with the aforementioned name.

apiVersion: v1
kind: Service
metadata:
  name: postgresql-hl
  namespace: test
spec:
  clusterIP: None
  selector:
    app: postgresql-hl-test
  ports:
  - port: 5432
    targetPort: 5432

The description of the service shows 2 endpoints for the postgresql Pods, meaning, you can access each one separately.

...
IP:                None
IPs:               None
Port:              <unset>  5432/TCP
TargetPort:        5432/TCP
Endpoints:         10.0.1.181:5432,10.0.1.42:5432
...

For testing, let's use one of 2 pods to connect to another:

kubectl exec -it postgresql-hl-test-0 -n test -- /bin/bash
# ...
export PGPASSWORD='postgres-password'
psql -h postgresql-hl-test-0.postgresql-hl -U test
# psql (15.4)
# Type "help" for help.

# test=>

After testing, you should see successful connection to the database.

Complete

First of all, let's install the controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/baremetal/deploy.yaml

All the required resources (Deployment, Service, etc.) should be available in the ingress-nginx namespace.

Complete

Let's create an Ingress for echoserver:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echoserver-ingress
  labels:
    name: echoserver-ingress
spec:
  ingressClassName: "nginx"
  rules: #(5)
  - host: echoserver.PUBLIC_NODE_IP.nip.io #(1)
    http:
      paths:
      - pathType: Prefix
        path: "/" #(2)
        backend:
          service:
            name: echoserver #(3)
            port:
              number: 80 #(4)

1. Hostname for the service. Please, replace the PUBLIC_NODE_IP with actual IP of your control-plane node.
2. Ingress allows different paths for services, in this lab we use only root
3. Service name to lookup
4. Service port to target
5. nip.io forwards traffic to the publicly available IP on third domain level (PUBLIC_NODE_IP in the example)

Complete

Let's create a new namespace called k8s-lab5:

kubectl create namespace k8s-lab5

and deploy an additional echoserver Pod and Service with the same config as before (Service should have ClusterIP type).

kubectl apply -f echoserver-pod.yaml -n k8s-lab5
kubectl apply -f echoserver-service.yaml -n k8s-lab5

Complete

An example policy allowing access to echoserver in default namespace is:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: echoserver-network-policy
spec:
  podSelector:
    matchLabels:
      app: echoserver
  policyTypes:
    - Ingress
  ingress:
    - from:
      - namespaceSelector:
          matchLabels: #(1)
            kubernetes.io/metadata.name: k8s-lab5
      - podSelector:
          matchLabels: #(2)
            name: nginx
      ports:
        - protocol: TCP
          port: 80

1. All the Pods from k8s-lab5 namespace can access the Pod
2. All the Pods from the same namespace and labels name=nginx can access the Pod

Create a network policy via the manifest above in the default namespace.

NB: This policy could break the lab3-05 check in the scoring site. If this happens, please follow cleanup instructions after completion of the main content of this lab.

Complete

You need to create network policy for MinIO Pods. For this policy, isolate the MinIO Pods from all connections except ones from history-server Pod within the same production namespace.

NB: please use minio-network-policy name.