a few words about prometheus

A few days ago I implemented the horizontal pod auto scaling for our application based on customer metrics in the Kubernetes. During the implementation, I setup a Prometheus server to gather metrics from the application and setup a Prometheus-Adapter to convert those application metrics into Kubernetes metrics for pod auto scaling. In this post I will summarize a few tips about Prometheus.

1. Prometheus is an open source, system monitoring and alerting tool. The naming conventions of the metrics that are destined for Prometheus to gather are as below:

1) a metrics name usually has a prefix for the domain, e.g. prometheus_notifications_total, process_cpu_seconds_total, http_request_duration_seconds.

2) a metrics name must have a suffix for the unit, e.g. http_request_duration_seconds, node_memory_usage_bytes.

3) a metrics name should have a suffix called ‘total’ for the accumulating count. e.g. process_cpu_seconds_total.

The more detailed description can be found in the below URL link:

https://prometheus.io/docs/practices/naming/

2. The results of the expression in Prometheus can be divided into four types:

1) instant vector: a set of time series, containing a single sample for each time series.

2) range vector: a set of time series, containing a range of data for each time series.

3) scalar: a numeric floating value

4) string: a string value, currently not used now.

The more detailed description can be found in the below URL link:

https://prometheus.io/docs/prometheus/latest/querying/basics/

3. The below operators can be used when labels are used for instant vector selector.

1) =: Select labels that are exactly equal to the provided string.

2) !=: Select labels that are not equal to the provided string.

3) =~: Select labels that regex-match the provided string.

4) !~: Select labels that do not regex-match the provided string.

A special internal label named ‘__name__’ can be applied to metric names.

The below notation should be used for range vector selector. The square bracket after the metric name donates the time range.

http_requests_total{job=”prometheus”}[5m]

The more detailed description can be found in the below URL link:

https://prometheus.io/docs/prometheus/latest/querying/basics/

4. The syntax for aggregation operator is as below:

<aggr-op>([parameter,] <vector expression>) [without|by (<label list>)]

for example:

sum by (application, group) (http_requests_total)

The more detailed description can be found in the below URL link:

https://prometheus.io/docs/prometheus/latest/querying/operators/

5. Calculate the http requests that the server receives per second as below:

sum(rate(http_server_requests_seconds_total[5m]))

The reference materials are as below:

https://tomgregory.com/monitoring-a-spring-boot-application-part-2-prometheus/

https://medium.com/expedia-group-tech/creating-monitoring-dashboards-1f3fbe0ae1ac

6. The below rules can be defined in Prometheus-Adapter for converting metrics in Prometheus to metrics in Kubernetes:

seriesQuery: '{__name__=~"^.*_seconds_count",container!="POD",namespace!="",pod!=""}'
resources:
  template: <<.Resource>>
name:
  matches: "^(.*)_seconds_count$"
  as: "$1"
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

The reference materials are as below:

https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config-walkthrough.md

https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/docs/config.md

7. For horizontal pod auto scaling to work in Kubernetes, the below components should be installed

https://github.com/kubernetes-sigs/prometheus-adapter

https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

The Prometheus-Adapter registers custom metrics API in Kubernetes. These metrics can be used by Kubernetes HPA to scaling pods horizontally.

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md

The kube-prometheus-stack can be used to install a Prometheus server and a Prometheus-Operator.

8. The below materials can be referenced for horizontal pod auto scaling.

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/