How To Setup Prometheus and Grafana Monitoring on Kubernetes | by Guillaume Vincent | Mar, 2022

Tourist viewer so they can see the city from afar
Photograph by Matthijs van Schuppen on Unsplash

Timeline

2012 — built at SoundCloud 2013 — use in SoundCloud’s production May 2016 — CNCF incubated Prometheus July 2016 — Prometheus 1.0 released Nov 2017 — Prometheus 2.0 released Aug 2018 — CNCF graduated Prometheus production-ready
The Prometheus mission timeline

Ideas

Example of a metric exposes to Prometheus over HTTP
Instance of a metric exposes to Prometheus over HTTP

Structure

The Prometheus structure from https://prometheus.io/
  • Prometheus server scraping and shops time-series information. It additionally gives a person interface to question the metrics.
  • Client libraries for instrumenting utility code.
  • Pushgateway helps metrics assortment from short-lived jobs
  • Exporters for providers that don’t instrument Prometheus metrics straight.
  • Alertmanager handles real-time alerts based mostly on triggers

Putting in Helm

$ brew set up helm

Creating the devoted monitoring namespace

$ kubectl create namespace monitoring

Putting in kube-prometheus-stack with Helm

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
$ helm repo replace
$ helm improve --namespace monitoring --install kube-stack-prometheus prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount.enabled=false
$ kubectl get -n monitoring crds                                                           
NAME CREATED AT
alertmanagerconfigs.monitoring.coreos.com 2022-03-15T10:54:41Z
alertmanagers.monitoring.coreos.com 2022-03-15T10:54:42Z
podmonitors.monitoring.coreos.com 2022-03-15T10:54:42Z
probes.monitoring.coreos.com 2022-03-15T10:54:42Z
prometheuses.monitoring.coreos.com 2022-03-15T10:54:42Z
prometheusrules.monitoring.coreos.com 2022-03-15T10:54:42Z
servicemonitors.monitoring.coreos.com 2022-03-15T10:54:42Z
thanosrulers.monitoring.coreos.com 2022-03-15T10:54:42Z
$ kubectl get pods -n monitoring                                                                    
NAME READY STATUS RESTARTS AGE
alertmanager-kube-stack-prometheus-kube-alertmanager-0 2/2 Working 0 2m36s
kube-stack-prometheus-grafana-6994bd6c69-h6s9z 3/3 Working 0 13h
kube-stack-prometheus-kube-operator-86667b5cdf-cqndt 1/1 Working 0 13h
kube-stack-prometheus-kube-state-metrics-fc9878699-dpgh6 1/1 Working 0 13h
kube-stack-prometheus-prometheus-node-exporter-vrjsl 1/1 Working 0 13h
prometheus-kube-stack-prometheus-kube-prometheus-0 2/2 Working 0 13h
$ kubectl port-forward --namespace monitoring svc/kube-stack-prometheus-kube-prometheus 9090:9090
Dashboard in the Prometheus web UI showing time-series metrics
Dashboard within the Prometheus net UI displaying time-series metrics
The metric endpoints discovered by Prometheus are visible in Status>Targets menu
The metric endpoints found by Prometheus are seen in Standing>Targets menu
Few target endpoints discovered by Prometheus server
Few goal endpoints found by Prometheus server
$ kubectl get secret --namespace monitoring kube-stack-prometheus-grafana -o jsonpath='.information.admin-user' | base64 -d 
$ kubectl get secret --namespace monitoring kube-stack-prometheus-grafana -o jsonpath='.information.admin-password' | base64 -d
$ kubectl port-forward --namespace monitoring svc/kube-stack-prometheus-grafana 8080:80
The Grafana login interface
The Grafana login interface
List of the provisioned Grafana dashboards
Checklist of the provisioned Grafana dashboards
A Grafana dashboard displaying Kubernetes pod compute sources

More Posts