We should be able to gather high level metrics about response time of request.

As it's far from trivial to do that with Prometheus (there are no data type for duration), it should be done with an histogram type (or summary but its more complex, lets start with summary). That means we should define an histogram of possible values (buckets) and then prom will keep track of the number of requests that matched each category.

Given an histogram set with the following buckets (correspond to microseconds of request duration)

• 0.05
• 0.5
• 1

I have then 3 requests

• 250ms
• 750ms
• 3s

I will get the following results in Prometheus

• 0.05: 0  => no requests took less than 50ms
• 0.5: 1   => 1 request took less than 500ms
• 1: 2     => 2 requests took less than 1s
• +Inf: 3  => All requests took less than +Inf
• count: 3 => There were 3 requests
• sum: 4   => The total of all requests took 4s

+Inf, count and sum are automatically generated.

Example inspired by https://povilasv.me/prometheus-tracking-request-duration/

Other useful references:

• http://linuxczar.net/blog/2017/06/15/prometheus-histogram-2/
• https://alex.dzyoba.com/blog/go-prometheus-service/