Links

PromQL Cheat Sheet: A Quick Guide to Prometheus Query Language

Prometheus is an open-source monitoring and alerting toolkit that has gained significant popularity in DevOps and systems monitoring. At the core of Prometheus lies PromQL (Prometheus Query Language), a powerful and flexible query language used to extract valuable insights from the collected metrics. In this guide, we will explore the basics of PromQL and provide query examples for an example use case.

Example Scenario

You have a high availability web app that you maintain. You'd like to have some observability into the traffic of your application. Your environment consists of 3 production web servers and 1 staging web server. Below is a table of instance vectors for your servers.
Server Instance
environment
http_requests_total(t0)
http_requests_total(t1)
web_prod_1
production
100
110
web_prod_2
production
200
220
web_prod_3
production
300
330
web_stg_1
staging
10
20

Querying Time Series

PromQL allows you to query time series data, which consists of metrics and their corresponding labels. The basic syntax for querying time series is as follows:
<metric_name>{<label_name>=<label_value>, ...}
Example:
To query the total HTTP requests metric for a your fleet of servers, you would use:
http_requests_total
The above example would return an instance vector for each server in your fleet.
{__name__="http_requests_total", server="web_prod_1", environment="production"} = [100, 110]
{__name__="http_requests_total", server="web_prod_2", environment="production"} = [200, 220]
{__name__="http_requests_total", server="web_prod_3", environment="production"} = [300, 330]
{__name__="http_requests_total", server="web_stg_1", environment="staging"} = [10, 20]

Instance Vector Selectors

Instance vector selectors allow you to filter and focus on specific labels to extract relevant metrics. To filter the time series, append a comma-separated list of label matchers in curly braces {}
http_requests_total{environment="production"}
The above example would return an instance vector for each production server in your fleet.
{__name__="http_requests_total", server="web_prod_1", environment="production"} = [100, 110]
{__name__="http_requests_total", server="web_prod_2", environment="production"} = [200, 220]
{__name__="http_requests_total", server="web_prod_3", environment="production"} = [300, 330]

Label Matching Operators

Additionally, PromQL provides the following label matching operators:
  • =: Select labels that are exactly equal to the provided string.
  • !=: Select labels that are not equal to the provided string.
  • =~: Select labels that regex-match the provided string.
  • !~: Select labels that do not regex-match the provided string.
Regex matches are fully anchored. A match of env=~"foo" is treated as env=~"^foo$". You can test your regex matches here using the Golang flavor.
So to select all of our staging servers we could use the following query:
http_requests_total{server=~".*_stg_.*"}

Aggregation Functions

PromQL provides various aggregation functions to summarize and aggregate time series data. Here are a few commonly used functions:
  • sum: Calculates the sum of all matching time series.
  • avg: Computes the average value of matching time series.
  • min: Returns the minimum value among all matching time series.
  • max: Returns the maximum value among all matching time series.
Example:
To calculate the average HTTP requests across all production instances, you can use:
avg(http_requests_total{environment="production"})
The above would first return the instance vectors then generate the average:
{__name__="http_requests_total", server="web_prod_1", environment="production"} = [100, 110]
{__name__="http_requests_total", server="web_prod_2", environment="production"} = [200, 220]
{__name__="http_requests_total", server="web_prod_3", environment="production"} = [300, 330]
[100, 200]
[200, 400]
[300, 600]
+ ----------
[600, 1200]
÷ 3, 3
-----------
= [200, 400]
{__name__="avg(http_requests_total{environment="production"})"} = [200, 400]

Range Vectors and Functions

PromQL allows you to work with range vectors, which represent time series data over a specified time range. This is particularly useful for analyzing trends and patterns. Here are a few important range functions:
  • rate: Calculates the per-second average rate of increase of a time series.
  • increase: Computes the increase in a time series value over a specified time range.
  • irate: Similar to rate, but instantly adjusts for counter resets.
Please note rate and irate calculate the per-second average rate of increase. This will calculate it based on the time increment of your graph x-value.
PromQL increase vs irate
Example:
To calculate the number of HTTP requests you are getting for your entire production fleet.
increase(sum(http_requests_total{environment="production"}))
The above would first return the instance vectors, then sum them, then calculate the difference between the vector values t1 - t0
{__name__="http_requests_total", server="web_prod_1", environment="production"} = [100, 110]
{__name__="http_requests_total", server="web_prod_2", environment="production"} = [200, 220]
{__name__="http_requests_total", server="web_prod_3", environment="production"} = [300, 330]
[100, 200]
[200, 400]
[300, 600]
+ ----------
= [600, 1200]
[t1-t0] = 1200 - 600 = 600
{__name__="increase(sum(http_requests_total{environment="production"}))"} = [600]

Conclusion

PromQL is a versatile and powerful query language that empowers users to extract valuable insights from Prometheus metrics. By mastering the basics covered in this cheat sheet, you'll be well-equipped to explore and analyze your monitoring data effectively. Remember, this blog post only scratches the surface. Make sure to experiment with different functions and operators to make the most of PromQL's capabilities.
By keeping this cheat sheet handy, you'll be able to navigate PromQL queries efficiently and unlock the full potential of Prometheus for monitoring and alerting in your systems.

PromQL Queries Used At PagerTree

At PagerTree we monitor our systems extensively; here are some of the common queries we use. The metrics (and metric names) we use are provided by the discord/prometheus_exporter gem or own own metric label name.
HTTP Response Count By Status Code
Alerts Total
Notifications Total By Channel
Sidekiq Queue Latency By Queue

HTTP Response Count by Status Code

Query:
sum(increase(fly_app_http_responses_count{app="pt4-[[environment]]-web"})) by (status) > 0
Graphed Result
Query:
sum(increase(ruby_pagertree_alerts_total{app=~"pt4-[[environment]]-.*"}))
Graphed Result:

Notifications Total By Channel

Query:
sum(increase(ruby_pagertree_message_notifications_total{app=~"pt4-[[environment]]-.*"})) by (channel)
Graphed Result:

Sidekiq Queue Latency By Queue

Query:
avg(ruby_sidekiq_queue_latency_seconds{app="pt4-[[environment]]-worker"}) by (queue)
Graphed Result: