Prometheus Monitoring Tutorial

In this post, I will walk you through creating a simple Prometheus monitoring stack, connecting it to Grafana for pretty dashboards, and finally configuring alerts via PagerTree.

If you would like a video to follow along instead, you can see it on YouTube. You can find all the code for this stack on Github.

Setup Docker Environment

The first thing we’ll do is a machine up and running for this solution. This tutorial assumes you will be using Ubuntu 16.04.

I like Digital Ocean for small tutorials like this one. If you don’t already have an account, use this link to create an account and get $10 in credits.

If you don’t know how to create a Digital Ocean droplet or SSH into the machine you can follow this article on Medium.

Build the Prometheus Stack

Once you’ve created the Ubuntu server, run the following command in the shell terminal:

curl https://raw.githubusercontent.com/PagerTree/prometheus-grafana-alertmanager-example/master/install.sh -H 'Cache-Control: no-cache' | sudo sh; cd prometheus-grafana-alertmanager-example;

At this point you’ll have automagically deployed the entire Prometheus, Grafana, and Alert Manager stack. You can now access the Grafana dashboard from your browser at: Address: http://<Host IP Address>:3000, Username: admin, Password: 9uT46ZKE.

Add the Prometheus Datasource to Grafana

As of this writing Grafana still does not support auto importing data sources, so we’ll have to do it manually.

  • Click the Grafana Menu at the top left corner (looks like a fireball)
  • Click Data Sources
  • Click the green button Add Data Source.
  • Input the following parameters exactly as shown
    • Name - Prometheus, Default - checked
    • Type - Prometheus
    • HTTP settings
      • URL - http://prometheus:9090
      • Access - proxy
  • Click Save & Test

Add Prometheus Datasource to Grafana

Grafana Dashboards

Awesome! Now if you navigate to the Dashboards in Grafana you will see data populating and some nice looking graphs.

  • Click the Grafana Menu at the top left corner
  • Click Dashboards
  • Click Home

Grafana Dashboards

At this point you’ll then 2 dashboards. They are pretty cool. Check them out. When your ready, head down to the Configure Alerts Section.

This dashboard monitors a couple websites for uptime. Ping Dashboard
This dashboard monitors the load on the machine that is running your Prometheus stack. System Monitor Dashboard

Configure Alerts

Now while the dashboards are cool, it would be even cooler if we were able to get alerted when something went wrong. Luckily for us, this project will create an alert after 30 seconds of high CPU. So let’s try to make use of it.

Create the PagerTree Webhook

In your PagerTree Dashboard

  • Click the a Team that should receive alerts for the demo
Warning Ensure that for this team you are the Layer 1 on-call and that you have at least 1 notification method setup.
  • Click the Integrations tab
  • Click the + Integration button
    • Leave all the settings as default, and change following settings
      • Name - Prometheus Stack Example
      • Integration Type - Prometheus
      • Options - Check 'Notify From User on Events'
    • Click the Create button Create the PagerTree Prometheus Integration
  • Copy the PagerTree Webhook URL by clicking the copy button Copy the PagerTree Webhook URL

Modify the Alert Manager Configuration

Now we want to modify the alert manager configuration to make use of our PagerTree Webhook. Run the following command and make sure to replace <Your PagerTree Webhook URL> with the you copied.

./util/alertmanager-configure-pagertree.sh <Your PagerTree Webhook URL>;

After you have run the configuration script, restart the stack with the following command:

./util/restart.sh;

Simulate an Alert Worthy Incident

In order for us to get an alert we’ll wan to simulate some sort of Alert Worthy Incident. From the shell terminal, run the following command:

./util/high-load.sh;

Now we’ll wait for 30 seconds or so, and if you’ve followed all the steps correctly you should get a notification saying something like Instance under high load.

Congrats! You’re done!

If you are reading this give yourself a pat on the back. Good job! You’ve successfully deployed a Prometheus monitoring system, hooked it up to Grafana, and configured and alerts to go to your PagerTree account.

This project is intended just to be a quick tutorial. Before being production worthy, several security considerations should be implemented.

Cheers,

Austin