Lab 9 - Monitoring

Overview¶

• Metrics
  • node_exporter
  • Prometheus
• Visualizations
  • Grafana for Prometheus
• Kubernetes specific metrics
  • Kubernetes state metrics
  • cAdvisor
• Centralized logging
  • Loki
  • Promtail
  • Grafana for Loki
• Instrumenting your application

Introduction¶

Monitoring is the process of collecting, analyzing, and displaying data about the performance, availability, and health of systems, applications, and network infrastructure. By continuously observing the state of these components, monitoring enables operators and administrators to gain insights into the behavior of their environment, detect potential issues, and proactively resolve problems before they impact users or critical business processes.

The need for monitoring arises from the increasing complexity and scale of modern IT environments, which require effective tools and techniques to ensure their reliability, performance, and security. Monitoring helps organizations maintain a high level of service quality, adhere to service level agreements (SLAs), and optimize resource usage, all of which contribute to better user experience and overall operational efficiency.

In practice, monitoring involves gathering various types of data, such as metrics, logs, and traces, from different sources and presenting them in a meaningful way. This data is then used to analyze trends, detect anomalies, generate alerts, and drive informed decision-making. Additionally, monitoring supports capacity planning, performance tuning, incident management, and root cause analysis efforts, which are essential for maintaining the availability and performance of IT services.

Metrics¶

Metrics are numerical measurements that represent the behavior, performance, and health of systems, applications, and network infrastructure. They provide quantitative data that helps operators and administrators understand the current state of their environment and make data-driven decisions. By continuously collecting and analyzing metrics, sysadmins can identify trends, detect anomalies, and optimize their infrastructure to ensure a high level of service quality and performance.

A simple example of a metric is the amount of time it took for the web server to load the content of this page, or how much CPU the container running the web server takes.

In practice, metrics are gathered from different sources, such as operating systems, hardware components, and application instrumentation. This data is collected at regular intervals and stored in time-series databases, which enable efficient querying, aggregation, and analysis of historical and real-time data. Metrics can be visualized in dashboards, charts, and graphs, providing a clear and concise view of the monitored systems' performance and health.

In this part of the lab, you'll be setting up Prometheus - a time-series database made for primarily querying and storing metrics - and an exporter, called node_exporter, which Prometheus can query for operating system level information. Then, as Kubernetes publishes several types of metrics natively in Prometheus format, you'll query those metrics to give better visibility into your managed cluster and applications.

node_exporter¶

First you'll be setting up the node_exporter tool to become familiar with the general methodology, and also the format of metrics used by Prometheus. Node Exporter is a widely used, open source monitoring tool that collects hardware and operating system metrics from Linux-based systems.

Developed as part of the Prometheus project, Node Exporter exposes these metrics in a format that can be easily scraped and ingested by a Prometheus server.

You'll be using a resource type called DaemonSet, which is very similar to Deployment - the difference is in the fact, that while with Deployment, you can choose how many replicas are going to run run, a DaemonSet always runs one replica on each node of the cluster. This is perfect for distributing monitoring tools across the cluster, as monitoring tools usually require a single agent. Additionally, when you add another node to the cluster, DaemonSet will cause the monitoring agent to be scheduled on the node automatically, reducing management overhead.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        args:
        - "--path.procfs=/host/proc"
        - "--path.sysfs=/host/sys"
        - "--path.rootfs=/host/root"
        - "--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)"
        ports:
        - containerPort: 9100
          name: http-metrics
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host/root
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

If you now query /metrics, you'll see a format like this:

...
# HELP node_filefd_allocated File descriptor statistics: allocated.
# TYPE node_filefd_allocated gauge
node_filefd_allocated 2048
# HELP node_filefd_maximum File descriptor statistics: maximum.
# TYPE node_filefd_maximum gauge
node_filefd_maximum 9.223372036854776e+18
# HELP node_filesystem_avail_bytes Filesystem space available to non-root users in bytes.
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{device="/dev/sda1",fstype="xfs",mountpoint="/"} 4.5791588352e+10
node_filesystem_avail_bytes{device="none",fstype="ramfs",mountpoint="/run/credentials/systemd-sysctl.service"} 0
...

This is the OpenMetrics format, which Prometheus and it's exporters use for describing metrics. The format short description:

<metric name>{<label_key>=<label_value>} <metric_value>

The result is a list of metrics at the time of query. For example, it might be the response contains node_cpu_seconds_total metric with labels cpu="0" and mode="idle", with the metric value 2.19401742e+06.

Additionally, there's four important elements to the metric:

• The HELP comment, that provides a human-readable description of the metric which follows (is lower in the list).
• The TYPE comment, this specifies the metric's type, such as gauge, counter, histogram. It is mainly useful during querying and visualizing the metrics, as different types need to be visualized differently.
• For example, gauge is a value that can go up or down, like temperature. You can use this for visualization without extra elements.
• counter, on the other hand, is a metric that either grows forever or can be reset to zero. They are often used to represent cumulative metrics, like total number of bytes transferred. As this metric can get into billions or trillions very easily, it's not human readable without some tricks.
  • One of the easiest tricks is to, instead of using the value itself, use it's rate of change. For example, how much has the bytes transferred metric changed in the past minute (derivative). This is usually much more human readable.
• The metric itself, in the format of metric_name numeric_value.
• Some systems are more complicated, and the usable monitoring requires more context - for that you can add labels to every metric. These are key-value pairs enclosed in curly braces, and they provide additional dimensions for the metric.
• In case of node_filesystem_avail_bytes{device="/dev/sda1",fstype="xfs",mountpoint="/"} 4.5791588352e+10 - device, fstype and mountpoint are metrics. This allows you to see the available bytes based on specific mount points and filesystem types, not only as an aggregate over the system.

You'll also see, that there's thousands of lines of these metrics. This is normal - the more complicated the system, the more metrics there is going to be. It might also seem like a tremendous amount of information, but it's actually not, at least not for computers:

$ curl 10.0.1.76:9100/metrics | wc --bytes # (1)
129519

$ echo "129519.0/1024.0" | bc -l # (2)
126.48339843750000000000

1. Count the bytes in the response.
2. Convert the bytes into kilobytes.

This is less than the first response by google.com, not to mention any other websites with actual graphics.

Prometheus¶

Now that you have an exporter in place, setup a system to query that exporter, and provide a way to store and aggregate this data.

Prometheus is an open source monitoring and alerting toolkit designed for reliability and scalability, primarily used for monitoring containerized and distributed systems. Developed at SoundCloud and now a part of the Cloud Native Computing Foundation (CNCF), Prometheus has become a widely adopted monitoring solution in modern cloud-native environments. Its powerful query language, data model, and efficient storage design make it an ideal choice for organizations seeking to monitor their infrastructure, applications, and services.

Prometheus operates on a pull-based model, where it actively scrapes metrics from target systems or applications at regular intervals. This approach simplifies the overall architecture and makes it easier to scale and manage. You'll be configuring Prometheus to query the Node Exporter every 15 seconds.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data: # (1)
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

    scrape_configs:
      - job_name: 'kubernetes-node-exporter'
        static_configs:
          - targets: ["10.0.1.72:9100"]
          - targets: ["10.0.0.139:9100"]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
spec:
  serviceName: prometheus
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      initContainers: # (2)
      - name: prometheus-data-permission-setup
        image: busybox
        command: ["/bin/chown","-R","65534:65534","/data"]
        volumeMounts:
        - name: data
          mountPath: /data
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--storage.tsdb.path=/data"
        - "--web.enable-lifecycle"
        ports:
        - containerPort: 9090
          name: http-metrics
        volumeMounts:
        - name: config
          mountPath: /etc/prometheus
        - name: data
          mountPath: /data
      volumes:
      - name: config
        configMap:
          name: prometheus-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
---

Now, back to the issue of non-dynamic node-exporter targets. Your first reaction might be to just create a service for node-exporter pods, and use that address in the scrape target configuration of Kubernetes.

The problem with this is, remember, that service forwards the request to a random backend pod, but makes no guarantees about which one. So you can have a situation, when the next 15 requests go to only the pod on the worker node, completely removing control plane node from visibility.

As powerful as Kubernetes might be, there's no way to easily solve this with Kubernetes' own tools - Kubernetes does not provide a resource that automatically allows for that. Thankfully, Kubernetes and Prometheus allow for this using another methodology.

Prometheus allows querying Kubernetes API to automatically find specific resources. This is called service discovery (SD). Doing this requires creating a Kubernetes service account, mounting it to the Prometheus container, giving the service account appropriate permissions, and using it to query Kubernetes API.

---
apiVersion: v1
kind: ServiceAccount # (1)
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole # (2)
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/metrics
  - nodes/proxy
  - nodes/stats
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding # (3)
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data: # (4)
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

    scrape_configs:
      - job_name: 'kubernetes-node-exporter'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: node-exporter
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: prometheus
spec:
  serviceName: prometheus
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus # (5)
      initContainers:
      - name: prometheus-data-permission-setup
        image: busybox
        command: ["/bin/chown","-R","65534:65534","/data"]
        volumeMounts:
        - name: data
          mountPath: /data
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--storage.tsdb.path=/data"
        - "--web.enable-lifecycle"
        ports:
        - containerPort: 9090
          name: http-metrics
        volumeMounts:
        - name: config
          mountPath: /etc/prometheus
        - name: data
          mountPath: /data
      volumes:
      - name: config
        configMap:
          name: prometheus-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
---

Visualizations with Grafana¶

While you can already use Prometheus to do some basic queries, using it is not very pleasant - you need to go to a page, type in the query, and so on. This would be a nightmare to use in a team, as you can't expect everyone to know the queries. Also, this methodology does not allow for incremental improvements by team members.

This is why, usually, monitoring systems come with some way to build visualisations, dashboards, graphs, and diagrams, as humans are way better at seeing and correlating patterns in images, than in textual representation of numbers.

Grafana¶

Grafana is an open source, feature-rich visualization and analytics platform used for monitoring and observability. It provides an intuitive and customizable interface for visualizing time-series data from various data sources, such as Prometheus, InfluxDB, Elasticsearch, and many more. Grafana's popularity stems from its flexibility, ease of use, and extensive plugin ecosystem.

You'll be using Grafana as a visualisation frontend to Prometheus, and later, Loki, mainly because Grafana is so widely used, feature rich and open source.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi  # adjust size as needed
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      initContainers:
      - name: grafana-data-permission-setup
        image: busybox
        command: ["/bin/chown","-R","472:472","/var/lib/grafana"]
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/lib/grafana
      containers:
      - name: grafana
        image: grafana/grafana:latest
        env:
        - name: GF_SECURITY_ADMIN_USER
          value: "admin"
        - name: GF_SECURITY_ADMIN_PASSWORD
          value: "password" # (1)
        ports:
        - name: http
          containerPort: 3000
        readinessProbe:
          httpGet:
            path: /api/health
            port: http
          initialDelaySeconds: 30
          timeoutSeconds: 5
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/lib/grafana
      volumes:
      - name: grafana-storage
        persistentVolumeClaim:
          claimName: grafana-pvc
---

Once you have logged into the system, you should have a fairly empty Grafana with no visualisations or dashboards. Fix this issue by importing a dashboard to view your metrics.

After clicking import, you should have a nice screen with a lot of metrics visible to you. If something is broken, these metrics are going to be empty, in which case it's a good idea to ask teachers for help.

You can now browse the dashboard, and see the different node specific metrics about the different nodes in the cluster.

Kubernetes specific metrics¶

In this section, you'll be adding some metrics which allow obtaining information about Kubernetes as well. With microservices based architectures, like Kubernetes is and is mainly meant to accommodate, good monitoring is of upmost importance, and often the only way to receive a full picture of the cluster.

Kubernetes state metrics¶

kube-state-metrics is a service that listens to the Kubernetes API server and generates metrics about the state of Kubernetes objects like Deployments, Pods, and Services.

For example, it can tell you the number of desired replicas for a Deployment, the current CPU requests or limits for a Pod, or even the number of nodes in specific conditions. These metrics are often essential for scaling, monitoring, and alerting purposes.

It exposes these metrics natively in the Prometheus format, making integration with Prometheus fairly simple. The only problem is that because it tries to give information about the whole state of the cluster, it needs to have permissions to the whole cluster.

The repository for it is located here in GitHub.

Now that you have the metrics, you need to also configure Prometheus to query this exporter.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-node-exporter'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app]
            action: keep
            regex: node-exporter
          - source_labels: [__meta_kubernetes_pod_label_app]
            target_label: kubernetes_pod_label_app

      - job_name: 'kubernetes-kube-state-metrics'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
            action: keep
            regex: kube-state-metrics
          - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
            target_label: kubernetes_pod_label_app

Now you have a monitoring system that shows metrics about all the workloads managed by Kubernetes. This is the basis of proper monitoring for Kubernetes - in a production Kubernetes cluster metrics like the state of nodes are of crucial value, and you can build whole monitoring systems based on these.

Adding useful labels¶

When you start deep-diving into the metrics, one of the more annoying things you're going to notice, is that the metrics do not have good labels with them. For example, the Host dropdown in the Node Exporter dashboard has the internal IP addresses and ports of the node-exporter pods.

This is not very useful, as you cannot directly correlate which IP is about which node. Additionally, the IP changes on pod restart, meaning you cannot have a static graph about a node.

Thankfully, Prometheus relabel_configs, which you have already used to add the kubernetes_pod_label_app label, can also be used to fix this problem. Specifically, if you check the service discovery page in UI, it shows all the fields and their values that Prometheus scrapes from Kubernetes.

One such field is __meta_kubernetes_pod_node_name.

  - source_labels: [__meta_kubernetes_pod_node_name]
    target_label: instance

cAdvisor¶

cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of running containers. Integrated into the Kubelet agent in every Kubernetes node, cAdvisor collects and exposes metrics about CPU, memory, network, and disk usage for all containers running on that node. These metrics are useful for monitoring and performance tuning.

You might be asking what is the difference between node-exporter and cAdvisor - both of them show CPU, memory, network, and so on. The problem is, that node-exporter shows these metrics aggregated over the whole host. With cAdvisor, you can see which pod is using how much CPU at a point in time, which is immensely useful for debugging, but also for billing.

Kubernetes nodes expose cAdvisor metrics on the path <node_name>:<node_port>/metrics/cadvisor. This endpoint requires a valid TLS certificate, and also uses HTTPS. The only secure method is to both use the Kubernetes CA to validate the certificate, and some kind of service account or KUBECONFIG to authenticate against the endpoint. Thankfully, Kubernetes makes this rather simple.

 - job_name: 'kubernetes/cadvisor'
   scheme: https # (1)
   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token # (2)
   tls_config:
     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt # (3)
   kubernetes_sd_configs:
     - role: node # (4)
   relabel_configs:
     - target_label: __address__
       replacement: kubernetes.default.svc:443 # (5)
     - source_labels: [__meta_kubernetes_node_name]
       regex: (.+)
       target_label: __metrics_path__
       replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor # (6)
     - source_labels: [__meta_kubernetes_node_name]
       target_label: kubernetes_node_name
     - source_labels: [pod]
       target_label: pod_name
       action: replace
     - action: labelmap
       regex: __meta_kubernetes_node_label_(.+) # (7)

Centralized logging¶

Centralized logging is the practice of aggregating logs from multiple sources, such as applications, services, and infrastructure components, into a single, centralized location. A logging solution deployed in a distributed fashion prevents from operators losing their visibility into the systems from a single failure in a monitoring solution.

Newer logging solutions also allow for ways to aggregate logs with other information from their systems, like metrics. This kind of centralized logging aggregated with other systems also simplifies the troubleshooting process by providing a single, unified view of log data, reducing the time spent searching for relevant information across disparate systems. You'll be brushing past this very quickly, but if you want, you can play around inside the Grafana once you have both Prometheus and Loki connected with it.

Loki¶

Loki is an open source, horizontally scalable log aggregation system developed by Grafana Labs. Inspired by Prometheus, Loki is designed to be cost-effective, easy to operate, and efficient in handling large volumes of log data. It provides a simple yet powerful solution for centralizing logs from various sources, such as applications, services, and infrastructure components, making it easier to search, analyze, and visualize log data. Loki integrates seamlessly with the Grafana visualization platform, enabling users to explore logs alongside metrics for a comprehensive view of their systems.

The need for a tool like Loki arises from the challenges posed by modern, distributed environments, where logs are generated by numerous components running across multiple nodes. Traditional log management solutions can struggle to cope with the volume and complexity of log data produced by these systems, leading to increased operational overhead, storage costs, and difficulty in extracting meaningful insights. Loki addresses these challenges with a unique approach that indexes only the metadata of log data (for example labels and timestamps), rather than the log content itself. This results in a more efficient and cost-effective storage solution, while still providing fast and accurate log querying capabilities.

Even though Loki is designed to be simple to deploy and manage, it's not. It is simpler than other such distributed logging solutions, but it is definitely not simple. For simplicity sake, you'll be deploying this application in a single-binary fashion - as a monolithic application, and you'll take no benefit from it being a distributed logging solution, but it's a good stepping stone, and way easier to do if you have general knowledge of the system.

As logging systems usually follow the push methodology, where an agent pushes logs to the central system, then you need to start from setting up the central system.

apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-config
  namespace: loki
data:
  loki.yaml: |
    auth_enabled: false

    server:
      http_listen_port: 3100

    common:
      ring:
        instance_addr: 127.0.0.1
        kvstore:
          store: inmemory
      replication_factor: 1
      path_prefix: /data/loki

    ingester:
      lifecycler:
        address: 127.0.0.1
        ring:
          kvstore:
            store: inmemory
          replication_factor: 1
      chunk_idle_period: 15m
      chunk_retain_period: 5m
      wal:
        dir: /data/loki/wal

    schema_config:
      configs:
        - from: 2024-01-01
          schema: v13
          store: tsdb
          object_store: filesystem
          index:
            period: 24h
            prefix: kubernetes_

    storage_config:
      filesystem:
        directory: /data/loki/chunks
      tsdb_shipper:
        active_index_directory: /data/loki/index
        cache_location: /data/loki/index_cache

    limits_config:
      reject_old_samples: true
      reject_old_samples_max_age: 168h

    table_manager:
      retention_deletes_enabled: true
      retention_period: 168h

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: loki
  namespace: loki
spec:
  serviceName: loki
  replicas: 1
  selector:
    matchLabels:
      app: loki
  template:
    metadata:
      labels:
        app: loki
    spec:
      initContainers:
      - name: init-chown-data
        image: busybox:1.33.1
        command: ["sh", "-c", "chown -R 10001:10001 /data/loki"]
        volumeMounts:
        - name: data
          mountPath: /data/loki
      containers:
      - name: loki
        image: grafana/loki:latest
        args:
        - "-config.file=/etc/loki/loki.yaml"
        ports:
        - containerPort: 3100
          name: http-metrics
        volumeMounts:
        - name: config
          mountPath: /etc/loki
        - name: data
          mountPath: /data/loki
      volumes:
      - name: config
        configMap:
          name: loki-config
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi

Once you have the central system in place, you can continue with installing the log exporter called promtail.

Setup Promtail¶

Promtail is an open source log collection agent developed by Grafana Labs, specifically designed to integrate with the Loki log aggregation system. As a crucial component of the Loki ecosystem, Promtail is responsible for gathering log data from various sources, such as files, systemd/journald, or syslog, and forwarding it to a Loki instance.

First, while keeping it simple, you'll just send the container logs from /var/log/pods/*/*/*.log on Kubernetes hosts to Loki using Promtail. When combined with Kubernetes service discovery, it already becomes fairly powerful.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail-service-account
  namespace: loki
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: promtail-role
subjects:
- kind: ServiceAccount
  name: promtail-service-account
  namespace: loki
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: loki
data:
  promtail.yaml: |
    server:
      http_listen_port: 3101

    positions:
      filename: "/tmp/positions.yaml"

    clients:
      - url: http://loki-svc.loki.svc.cluster.local:3100/loki/api/v1/push

    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - cri: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          replacement: $1
          separator: /
          source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - replacement: /var/log/pods/*$1/$2/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: loki
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      serviceAccountName: promtail-service-account
      containers:
      - name: promtail
        image: grafana/promtail:latest
        ports:
        - name: http-metrics
          containerPort: 3101
          protocol: TCP
        args:
        - "-config.file=/etc/promtail/promtail.yaml"
        env:
        - name: 'HOSTNAME'
          valueFrom:
            fieldRef:
              fieldPath: 'spec.nodeName'
        volumeMounts:
        - name: config
          mountPath: /etc/promtail
        - name: varlog
          mountPath: /var/log
      volumes:
      - name: config
        configMap:
          name: promtail-config
      - name: varlog
        hostPath:
          path: /var/log

Grafana¶

As the last thing, also make these logs visible in the Grafana, by adding Loki as a data source and importing a new dashboard.

You are now free to play around with your newfound monitoring capability.

Instrumenting your application¶

One of the more useful applications of observability is getting runtime information from your own developed software. The Prometheus ecosystem makes it easy by publishing client libraries, that can easily be used to publish Prometheus metrics from applications on a single port: Prometheus client libraries

# HELP application_server_start_time Unix timestamp of when the application started
# TYPE application_server_start_time gauge
application_server_start_time{version="v1.2.3",container="application-server-1"} 1699113600
# HELP application_server_requests_total Count of requests to the Application Server
# TYPE application_server_requests_total counter
application_server_requests_total{version="v1.2.3",container="application-server-1",status_code="200"} 1500
application_server_requests_total{version="v1.2.3",container="application-server-1",status_code="404"} 35
application_server_requests_total{version="v1.2.3",container="application-server-1",status_code="500"} 7