Kapacitor Integration Guide

Kapacitor Integration

Kapacitor is an open source data processing framework that makes it easy to create alerts, run ETL jobs and detect anomalies. It can process both stream and batch data from InfluxDB. It lets you plug in your own custom logic or user-defined functions to process alerts with dynamic thresholds, match metrics for patterns, compute statistical anomalies, and perform specific actions based on these alerts like dynamic load rebalancing.

How Kapacitor users benefit from PagerTree

Kapacitor triggers alerts when alert rules are triggered. PagerTree acts as the dispatcher for these alerts, determining the right people to notify based on on-call schedules. PagerTree notifies them via email, sms, voice, and/or push notifications until the incident is acknowledged.

Functionality of Integration

The Kapacitor integration is one-way, meaning:

  • When an alert is not OK (level !== 'OK') in Kapacitor, an incident is created in PagerTree automatically.
  • When an alert is OK (level === 'OK') in Kapacitor, the incident is resolved in PagerTree automatically.

Tutorial Overview

In this integration tutorial we will show you how to send alerts from Kapacitor into PagerTree. The estimated time for this integration is 5 minutes. We assume that you already have a PagerTree and Kapacitor setup. We also assume you are familiar with the TICK stack since we will be using several of these components to setup the Kapacitor integration.

In PagerTree

  1. From the navigation menu, click the “+ Integration” button. Click Plus Integration Button
  2. In the “Create Integration” box, fill out the following:
    1. Name of the Integration,
    2. Select the Integration Type to Kapacitor
    3. Select an appropriate urgency for when this integration is triggered
    4. Select the team this incident will be assigned to Click Create
  3. Click “Create”
  4. Copy the Endpoint URL Copy Endpoint URL

In Kapacitor

  1. From your Chronograph dashboard, in the left hand menu, select Alerting -> Manage Tasks. Alerting Manage Tasks
  2. Click the Write TICKscript button. Write TICKscript button
  3. In the TICKscript editor:
    1. Select a database. Select a database
    2. Title your TICKscript (e.g. “high_cpu”).
    3. Copy and paste the following code, replacing the pagertree_url value with the PagerTree Endpoint URL you copied earlier.
     var pagertree_url = 'https://api.pagertree.com/integration/int_SkJhDKwTm'
     var period = 1m
     var crit = 90
     var warn = 80
     var info = 70
    
    
     var data = stream
         |from()
             .database('telegraf')
             .retentionPolicy('autogen')
             .measurement('win_cpu')
         |window()
             .period(period)
             .every(period)
         |mean('Percent_User_Time')
         |eval(lambda: "mean")
             .as('value')
    
     var trigger = data
         |alert()
             .info(lambda: "value" > info)
             .warn(lambda: "value" > warn)
             .crit(lambda: "value" > crit)
             .stateChangesOnly()
             .message(' is  value: ')
             .id('high_cpu')
             .idTag('alertID')
             .levelTag('level')
             .messageField('message')
             .post(pagertree_url)
    
    1. Click the Save New TICKscript button Click Save New TICKscript

This tick script, uses InfluxDB’s sample Telegraph collector that merely collects CPU usage on your local machine. You’ll need to apply the alert to your own tickscripts to alert on metrics you wish to monitor. The important pieces to this script are:

...
// will look at windows of data so you are not spammed by every datapoint
|window()
    .period(period)
    .every(period)
...
// send the alert to PagerTree with different levels. Only report state changes.
|alert()
    .info(lambda: "value" > info)
    .warn(lambda: "value" > warn)
    .crit(lambda: "value" > crit)
    .stateChangesOnly() // *very important*
    .message(' is  value: ')
    .id('high_cpu')
    .idTag('alertID')
    .levelTag('level')
    .messageField('message')
    .post(pagertree_url)

You have successfully completed the Kapacitor Integration. Now when Kapacitor creates an alert, PagerTree will create an incident and route it to the correct team member that is on-call. If you have any questions please contact support@pagertree.com


Additional Resources

  • cpu_alert.tick - Sample TICKscript to alert PagerTree using HTTP POST