HKOSCon 2024

Metrics, Logs, Traces and Profiles with Grafana LGTM

Wong Hoi Sing, Edison

2024-07-06

Introduction

Get the Code

About Me

What is Grafana LGTM Stack?

Prometheus + LGTM Stack + Pyroscope

  • Monitoring: Collects and stores metrics as time series data
  • Logs: What is happening?
  • Metrics: Is something happening?
  • Traces: Where is it happening?
  • Profiles: How do I fix it?
  • Dashboard: Data visualization and monitoring

Prometheus

  • Prometheus is a systems monitoring and alerting toolkit
  • Multi-dimensional data model
  • PromQL, a flexible query language to leverage this dimensionality
  • 2nd CNCF graduated project after Kubernetes
  • Apache License 2.0

Grafana

  • Grafana is a multi-platform open source analytics and interactive visualization web application
  • It can produce charts, graphs, and alerts for the web when connected to supported data sources
  • AGPLv3

  • Support multiple installation style, e.g. monolithic, microservice, or even manually
  • Here we are going to demo with distributed Helm chart style

Loki

  • Loki is a backend store for long-term log retention
  • Designed to be very cost effective and highly scalable
  • AGPLv3

  • Suport multiple installation style, e.g. monolithic, microservice, or even manually
  • Here we are going to demo with distributed Helm chart style

Tempo

  • Tempo is a distributed tracing backend
  • Designed to be cost-efficient, requiring only object storage to operate
  • Deeply integrated with Grafana, Prometheus, and Loki
  • AGPLv3

  • Suport multiple installation style, e.g. monolithic, microservice, or even manually
  • Here we are going to demo with distributed Helm chart style

Mimir

  • Mimir is a TSDB for long-term storage for Prometheus
  • Designed to be horizontally scalable, highly available, multi-tenant
  • 100% Prometheus compatible
  • AGPLv3

  • Suport multiple installation style, e.g. monolithic, microservice, or even manually
  • Here we are going to demo with distributed Helm chart style
  • EXCLUDE: due to not enough VM memory

Pyroscope

  • Pyroscope is a continuous profiling database
  • Designed to be fast, scalable, highly available, and efficient storage and querying
  • AGPLv3

  • Suport multiple installation style, e.g. monolithic, microservice, or even manually
  • Here we are going to demo with distributed Helm chart style
  • EXCLUDE: due to not enough VM memory

Demo with AlviStack Vagrant Box

System Requirements

  • Windows 10+
  • 4 Core CPU (for running a 2 Core CPU VM)
  • 16GB Memory (for running a 8GB VM)
  • 50GB SSD

Install VirtualBox

Install Vagrant

Prepare Vagrantfile

vagrant up

vagrant provision

vagrant ssh

kubectl get node

root@kubernetes-1:~# kubectl get node
NAME           STATUS   ROLES           AGE     VERSION
kubernetes-1   Ready    control-plane   9m18s   v1.29.2

kubectl get pod

root@kubernetes-1:~# kubectl get pod --all-namespaces
NAMESPACE      NAME                                   READY   STATUS    RESTARTS   AGE
csi-hostpath   csi-hostpath-socat-0                   1/1     Running   0          6m36s
csi-hostpath   csi-hostpathplugin-0                   8/8     Running   0          6m36s
kube-system    coredns-76f75df574-2tdbp               1/1     Running   0          9m2s
kube-system    coredns-76f75df574-g687d               1/1     Running   0          9m2s
kube-system    kube-addon-manager-kubernetes-1        1/1     Running   0          6m48s
kube-system    kube-apiserver-kubernetes-1            1/1     Running   0          9m18s
kube-system    kube-controller-manager-kubernetes-1   1/1     Running   0          9m18s
kube-system    kube-flannel-ds-kmq79                  1/1     Running   0          6m36s
kube-system    kube-proxy-vd4qj                       1/1     Running   0          9m2s
kube-system    kube-scheduler-kubernetes-1            1/1     Running   0          9m18s
kube-system    snapshot-controller-7b6f9cf9b4-rj5v5   1/1     Running   0          6m36s

Verify with Browser

  • Check the result (from host, with VirtualBox port mapping enabled in Vagrantfile):
    • http://localhost:8080
    • INCLUDE: Prometheus, Grafana, Loki, Tempo
    • EXCLUDE: Mimir (long-term storage for Prometheus), pyroscope (continuous profiling)

Create a Grafana Dashboard

Connect with Data Sources

  • Before show some data with Grafana, we need some data source
  • Let’s add the Prometheus endpoint for it
  • Set the Prometheus server URL with http://prometheus-operated.prometheus.svc.cluster.local:9090
  • Don’t forget click the Save & test button

Create Dashboard

Q&A

References

Contact Me